Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Task #10602 (closed)

Opened 7 years ago

Closed 7 years ago

Last modified 7 years ago

BUG: Import failure - checksum mismatch

Reported by: rkferguson Owned by: mtbcarroll
Priority: major Milestone: OMERO-4.4.7
Component: Services Version: n.a.
Keywords: n.a. Cc: bpindelski
Resources: n.a. Referenced By: n.a.
References: n.a. Remaining Time: 0.0d
Sprint: 2013-04-09 (7))

Description (last modified by rkferguson)

test_images_good > bruker

acqp import failed - off local disc and off squig

When viewing data tree - shows up as partial import 15 out of 21 look normal, 4 are very pale and washed out and one is just noise - see screenshot.

Same in Insight and Web, and ImageJ

java.io.IOException: file checksum mismatch on upload: /Users/rkferguson/omero/tmp/omero_rkferguson/13966@ls25532.dyn.lifesci.dundee.ac.uk/ORIGINAL_METADATA_TXT3774015435956551614cc8dbf12-9c49-4c40-a9c4-8d2f5c78c229 (client has f9708cd6b86ffe386df3d2fbf089cd77aaab1d42, server has a5c2bf0dc86e6ee228129880780522b1008f3419)
     at ome.formats.OMEROMetadataStoreClient.writeFilesToFileStore(OMEROMetadataStoreClient.java:1859)
     at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:694)
     at org.openmicroscopy.shoola.env.data.OMEROGateway.importImage(OMEROGateway.java:6634)
     at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importCandidates(OmeroImageServiceImpl.java:227)
     at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1443)
     at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:85)
     at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:54)
     at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:110)
     at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
     at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
     at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
     at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
     at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
     at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
     at java.lang.Thread.run(Thread.java:680)

     at org.openmicroscopy.shoola.env.data.OMEROGateway.importImage(OMEROGateway.java:6688)
     at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importCandidates(OmeroImageServiceImpl.java:227)
     at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1443)
     at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:85)
     at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:54)
     at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:110)
     at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
     at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
     at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
     at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
     at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
     at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
     at java.lang.Thread.run(Thread.java:680)
Caused by: java.io.IOException: file checksum mismatch on upload: /Users/rkferguson/omero/tmp/omero_rkferguson/13966@ls25532.dyn.lifesci.dundee.ac.uk/ORIGINAL_METADATA_TXT3774015435956551614cc8dbf12-9c49-4c40-a9c4-8d2f5c78c229 (client has f9708cd6b86ffe386df3d2fbf089cd77aaab1d42, server has a5c2bf0dc86e6ee228129880780522b1008f3419)
     at ome.formats.OMEROMetadataStoreClient.writeFilesToFileStore(OMEROMetadataStoreClient.java:1859)
     at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:694)
     at org.openmicroscopy.shoola.env.data.OMEROGateway.importImage(OMEROGateway.java:6634)
     ... 12 more

Attachments (1)

bruker import failure.jpg (230.6 KB) - added by rkferguson 7 years ago.

Download all attachments as: .zip

Change History (12)

Changed 7 years ago by rkferguson

comment:1 Changed 7 years ago by rkferguson

comment:2 Changed 7 years ago by rkferguson

  • Description modified (diff)

comment:3 Changed 7 years ago by rkferguson

  • Keywords fs removed
  • Milestone changed from OMERO-5 to OMERO-4.5
  • Sprint changed from FS Demo 3 to 2013-04-09 (7))
  • Summary changed from BUG: FS Import failure - checksum mismatch to BUG: Import failure - checksum mismatch

Sorry about the FS confusion - reset ticket as non-FS.

comment:4 Changed 7 years ago by bpindelski

  • Component changed from Insight to Services
  • Owner changed from jburel to bpindelski

comment:5 Changed 7 years ago by bpindelski

  • Cc mtbcarroll added; bpindelski removed

comment:6 Changed 7 years ago by bpindelski

  • Owner changed from bpindelski to mtbcarroll

comment:7 Changed 7 years ago by bpindelski

  • Cc bpindelski added; mtbcarroll removed

comment:8 Changed 7 years ago by mtbcarroll

The IO code is a bit fast and loose but once that's tidied it becomes apparent that the problem is that OMEROMetadataStoreClient.writeFilesToFileStore is given a set of distinct files but, in mapping them to original files, some of them map to the same original file ID. Then, the checksum logic gets confused because server-side the file is being appended and the earlier data from a previous upload gets included in the server-side checksum calculation but not client-side.

I would guess that this file-appending predates the recent checksum PRs, but I don't know if it is intentional: the answer to that determines what needs to be fixed.

(Edit: actually, I'm not sure that "appending" is quite what happens in the file-combining, now I think back on the resulting files ...)

Last edited 7 years ago by mtbcarroll (previous) (diff)

comment:9 Changed 7 years ago by mtbcarroll

Yes, it's not appending, it's writing the data from the start, but leaving the rest. That's clearly not intentional, but right now I've no idea what's going on with this intimidating originalFileMap.

For instance, in one example (importing the same data as Gus used), we upload the large /Users/mark/Desktop/wis/RA_060110.051/9/fid then the small /Users/mark/omero/tmp/omero_mark/54806@ls27175.dyn.lifesci.dundee.ac.uk/ORIGINAL_METADATA_TXT23536318138100681208ec83dad-b5b9-4edd-b52d-9a404b35a0b9 and the resulting server-side /Users/mark/var/omero/Files/Dir-001/1937 ends up the same size as the former but starting with the latter's data and, of course, the checksum match then rightly fails.

comment:10 Changed 7 years ago by mtbcarroll

  • Resolution set to fixed
  • Status changed from new to closed

comment:11 Changed 7 years ago by Josh Moore <josh@…>

  • Remaining Time set to 0

(In [a4dec2a9802127b4b7f18ba6afcd80724f7713f3/ome.git] on branch develop) Merge pull request #997 from mtbc/trac-10602

Fix #10602 and clean up IO code. (rebased)

Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.222838 sec.)

We're Hiring!