BUG: Import failure - checksum mismatch

[rkferguson]

test_images_good > bruker

acqp import failed - off local disc and off squig

When viewing data tree - shows up as partial import 15 out of 21 look normal, 4 are very pale and washed out and one is just noise - see screenshot.

Same in Insight and Web, and ImageJ

java.io.IOException: file checksum mismatch on upload: /Users/rkferguson/omero/tmp/omero_rkferguson/13966@ls25532.dyn.lifesci.dundee.ac.uk/ORIGINAL_METADATA_TXT3774015435956551614cc8dbf12-9c49-4c40-a9c4-8d2f5c78c229 (client has f9708cd6b86ffe386df3d2fbf089cd77aaab1d42, server has a5c2bf0dc86e6ee228129880780522b1008f3419)
     at ome.formats.OMEROMetadataStoreClient.writeFilesToFileStore(OMEROMetadataStoreClient.java:1859)
     at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:694)
     at org.openmicroscopy.shoola.env.data.OMEROGateway.importImage(OMEROGateway.java:6634)
     at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importCandidates(OmeroImageServiceImpl.java:227)
     at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1443)
     at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:85)
     at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:54)
     at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:110)
     at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
     at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
     at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
     at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
     at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
     at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
     at java.lang.Thread.run(Thread.java:680)

     at org.openmicroscopy.shoola.env.data.OMEROGateway.importImage(OMEROGateway.java:6688)
     at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importCandidates(OmeroImageServiceImpl.java:227)
     at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1443)
     at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:85)
     at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:54)
     at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:110)
     at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
     at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
     at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
     at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
     at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
     at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
     at java.lang.Thread.run(Thread.java:680)
Caused by: java.io.IOException: file checksum mismatch on upload: /Users/rkferguson/omero/tmp/omero_rkferguson/13966@ls25532.dyn.lifesci.dundee.ac.uk/ORIGINAL_METADATA_TXT3774015435956551614cc8dbf12-9c49-4c40-a9c4-8d2f5c78c229 (client has f9708cd6b86ffe386df3d2fbf089cd77aaab1d42, server has a5c2bf0dc86e6ee228129880780522b1008f3419)
     at ome.formats.OMEROMetadataStoreClient.writeFilesToFileStore(OMEROMetadataStoreClient.java:1859)
     at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:694)
     at org.openmicroscopy.shoola.env.data.OMEROGateway.importImage(OMEROGateway.java:6634)
     ... 12 more

[rkferguson]

Sorry about the FS confusion - reset ticket as non-FS.

[mtbcarroll]

The IO code is a bit fast and loose but once that's tidied it becomes apparent that the problem is that OMEROMetadataStoreClient.writeFilesToFileStore is given a set of distinct files but, in mapping them to original files, some of them map to the same original file ID. Then, the checksum logic gets confused because server-side the file is being appended and the earlier data from a previous upload gets included in the server-side checksum calculation but not client-side.

I would guess that this file-appending predates the recent checksum PRs, but I don't know if it is intentional: the answer to that determines what needs to be fixed.

(Edit: actually, I'm not sure that "appending" is quite what happens in the file-combining, now I think back on the resulting files ...)

[mtbcarroll]

Yes, it's not appending, it's writing the data from the start, but leaving the rest. That's clearly not intentional, but right now I've no idea what's going on with this intimidating originalFileMap.

For instance, in one example (importing the same data as Gus used), we upload the large /Users/mark/Desktop/wis/RA_060110.051/9/fid then the small /Users/mark/omero/tmp/omero_mark/54806@ls27175.dyn.lifesci.dundee.ac.uk/ORIGINAL_METADATA_TXT23536318138100681208ec83dad-b5b9-4edd-b52d-9a404b35a0b9 and the resulting server-side /Users/mark/var/omero/Files/Dir-001/1937 ends up the same size as the former but starting with the latter's data and, of course, the checksum match then rightly fails.

[mtbcarroll]

Fixed by ​https://github.com/openmicroscopy/openmicroscopy/pull/966

[Josh Moore <josh@…>]

(In [a4dec2a9802127b4b7f18ba6afcd80724f7713f3/ome.git] on branch develop) Merge pull request #997 from mtbc/trac-10602

Fix #10602 and clean up IO code. (rebased)