File corruptions/immutability guarantees under OmeroFs

[jamoore]

Another use case discussion for OmeroFs related to: #860 and #890.

sha1

The sha1 is currently computed on import (and should be by all clients). In Beta4, my plan was to have the server check all uploads against the sha1 and throw an exception if they don't match. This is important both for noticing physical corruption and tampering with scripts.

immutability

However, if files are mutable, then the sha1 isn't worth its wait in hex. Which means I was also going to suggest the romio library support some sort of UNFINISHED and IMMUTABLE flags. Not sure what that entire workflow would look like, or if it would be necessary on all files, but now with OmeroFs it's even a bit more complicated.

Ideas

One possible solution would be to allow "snapshots" of OmeroFs files. This would let users keep their files mutable, but once they left the UNFINISHED status, the server could take a snapshot if they were used in a particular way, e.g. an administrator marking them as "trusted" for scripting. The question becomes which OriginalFile entries do the clients work with: the OmeroFs one or the snapshot.

Similar to #941, this may be a case of OmeroFs configuration in which one must start the process with:

  ./omerofs --add /my/trusted/path --trusted --snapshots

I realize this is getting away from the original purpose of OmeroFs which was to prevent data/file duplication, but since OmeroFs functionality may become very popular, we need to make sure we can still control the data to some extent when necessary.

[cblackburn]

Moved to Future to clear from 4.0. Needs to be rescheduled.

[cblackburn]

Is this addressed/made redundant by the OMERO 5 import workflow?

[jamoore]

This is covering a couple of larger items than we've covered so far with FS. This should be migrated to trello and closed "duplicate". I'll do that during my next round of ticket cleanups.

[jamoore]

Moved to ​https://trello.com/c/vSjjHous/269-file-corruptions-immutability