Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Task #11850 (closed)

Opened 6 years ago

Closed 5 years ago

Bug: OME.tiff import slow

Reported by: wmoore Owned by: mlinkert
Priority: major Milestone: 5.0.1
Component: Import Version: n.a.
Keywords: n.a. Cc: fs@…
Resources: n.a. Referenced By: n.a.
References: n.a. Remaining Time: n.a.
Sprint: n.a.

Description

I have compared import of 150707_WTstack.lsm and an ome.tiff file generated by exporting the same lsm. So, both files are same size, planes etc.

The lsm import takes 5 seconds but the ome.tiff takes a minute and the log file generated by the ome.tiff import is 33556 lines, compared with 164 lines for the lsm.

Seems like ome.tiff import could do with some optimisation.

Log files attached.

Attachments (3)

lsm-import.log​ (24.4 KB) - added by wmoore 6 years ago.
ome-tiff-import.log​ (4.9 MB) - added by wmoore 6 years ago.
ss-11850-ome-tiff-import.png​ (117.4 KB) - added by jamoore 6 years ago.
jvisualvm screenshot of the pixeldata phase

Change History (11)

Changed 6 years ago by wmoore

Changed 6 years ago by wmoore

comment:1 Changed 6 years ago by jamoore

A few random stack traces:

	at java.util.Arrays.copyOfRange(Arrays.java:2697)
	at java.lang.String.<init>(String.java:204)
	at java.lang.String.toLowerCase(String.java:2501)
	at java.lang.String.toLowerCase(String.java:2524)
	at ome.scifio.io.BZip2Handle.isBZip2File(BZip2Handle.java:91)
	at ome.scifio.io.Location.getHandle(Location.java:311)
	at ome.scifio.io.Location.getHandle(Location.java:286)
	at ome.scifio.io.Location.getHandle(Location.java:276)
	at ome.scifio.io.RandomAccessInputStream.<init>(RandomAccessInputStream.java:108)
	at loci.common.RandomAccessInputStream.<init>(RandomAccessInputStream.java:73)
	at loci.formats.in.OMETiffReader.openBytes(OMETiffReader.java:266)
	at loci.formats.ImageReader.openBytes(ImageReader.java:435)
	at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:207)
	at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:226)
	at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:332)
	at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:332)
	at loci.formats.MinMaxCalculator.openBytes(MinMaxCalculator.java:275)
	at ome.services.blitz.repo.ManagedImportRequestI.parseDataByPlane(Manage

	at loci.formats.tiff.IFD.getStripByteCounts(IFD.java:792)
	at loci.formats.tiff.TiffParser.getSamples(TiffParser.java:822)
	at loci.formats.tiff.TiffParser.getSamples(TiffParser.java:745)
	at loci.formats.in.OMETiffReader.openBytes(OMETiffReader.java:268)
	at loci.formats.ImageReader.openBytes(ImageReader.java:435)
	at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:207)
	at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:226)
	at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:332)
	at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:332)
	at loci.formats.MinMaxCalculator.openBytes(MinMaxCalculator.java:275)
	at ome.services.blitz.repo.ManagedImportRequestI.parseDataByPlane(ManagedImportRequestI.java:643)

	at java.lang.String.charAt(String.java:660)
	at java.io.UnixFileSystem.normalize(UnixFileSystem.java:88)
	at java.io.File.<init>(File.java:254)
	at ome.scifio.io.NIOFileHandle.<init>(NIOFileHandle.java:150)
	at ome.scifio.io.Location.getHandle(Location.java:315)
	at ome.scifio.io.Location.getHandle(Location.java:286)
	at ome.scifio.io.Location.getHandle(Location.java:276)
	at ome.scifio.io.RandomAccessInputStream.<init>(RandomAccessInputStream.java:108)
	at loci.common.RandomAccessInputStream.<init>(RandomAccessInputStream.java:73)
	at loci.formats.in.OMETiffReader.openBytes(OMETiffReader.java:266)
	at loci.formats.ImageReader.openBytes(ImageReader.java:435)
	at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:207)
	at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:226)
	at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:332)
	at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:332)
	at loci.formats.MinMaxCalculator.openBytes(MinMaxCalculator.java:275)
	at ome.services.blitz.repo.ManagedImportRequestI.parseDataByPlane(ManagedImportRequestI.java:643)

	at java.lang.Character.toLowerCase(Character.java:6148)
	at java.lang.String.toLowerCase(String.java:2469)
	at java.lang.String.toLowerCase(String.java:2524)
	at ome.scifio.io.BZip2Handle.isBZip2File(BZip2Handle.java:91)
	at ome.scifio.io.Location.getHandle(Location.java:311)
	at ome.scifio.io.Location.getHandle(Location.java:286)
	at ome.scifio.io.Location.getHandle(Location.java:276)
	at ome.scifio.io.RandomAccessInputStream.<init>(RandomAccessInputStream.java:108)
	at loci.common.RandomAccessInputStream.<init>(RandomAccessInputStream.java:73)
	at loci.formats.in.OMETiffReader.openBytes(OMETiffReader.java:266)
	at loci.formats.ImageReader.openBytes(ImageReader.java:435)
	at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:207)
	at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:226)
	at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:332)
	at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:332)
	at loci.formats.MinMaxCalculator.openBytes(MinMaxCalculator.java:275)
	at ome.services.blitz.repo.ManagedImportRequestI.parseDataByPlane(ManagedImportRequestI.java:643)

	at sun.text.normalizer.ReplaceableString.charAt(ReplaceableString.java:98)
	at sun.text.normalizer.ReplaceableUCharacterIterator.next(ReplaceableUCharacterIterator.java:133)
	at sun.text.normalizer.NormalizerBase.next(NormalizerBase.java:1427)
	at sun.text.normalizer.NormalizerBase.nextNormalize(NormalizerBase.java:1492)
	at sun.text.normalizer.NormalizerBase.next(NormalizerBase.java:815)
	at java.text.CollationElementIterator.next(CollationElementIterator.java:226)
	at java.text.RuleBasedCollator.compare(RuleBasedCollator.java:398)
	- locked <0x00000000e4c6dcf0> (a java.text.RuleBasedCollator)
	at ome.formats.OMEXMLModelComparator.compare(OMEXMLModelComparator.java:80)
	at ome.formats.OMEXMLModelComparator.compare(OMEXMLModelComparator.java:31)
	at java.util.TreeMap.getEntryUsingComparator(TreeMap.java:370)
	at java.util.TreeMap.getEntry(TreeMap.java:342)
	at java.util.TreeMap.get(TreeMap.java:274)
	at ome.formats.OMEROMetadataStoreClient.getSourceObject(OMEROMetadataStoreClient.java:1473)
	at ome.formats.OMEROMetadataStoreClient.setChannelGlobalMinMax(OMEROMetadataStoreClient.java:2205)
	at loci.formats.MinMaxCalculator.updateMinMax(MinMaxCalculator.java:379)
	at ome.formats.importer.OMEROWrapper.updateMinMax(OMEROWrapper.java:259)
	at loci.formats.MinMaxCalculator.openBytes(MinMaxCalculator.java:276)
	at ome.services.blitz.repo.ManagedImportRequestI.parseDataByPlane(ManagedImportRequestI.java:643)

comment:2 Changed 6 years ago by jamoore

All the above were from the PIXELDATA_PROCESSED phase (under profiler):

2013-12-17 12:53:54,519 23008      [      main] INFO   ormats.importer.cli.LoggingImportMonitor  - FILESET_UPLOAD_END
2013-12-17 12:55:01,596 90085      [l.Client-1] INFO   ormats.importer.cli.LoggingImportMonitor  - METADATA_IMPORTED Step: 1 of 5  Logfile: 8703
2013-12-17 13:03:13,934 582423     [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor  - PIXELDATA_PROCESSED Step: 2 of 5  Logfile: 8703

Changed 6 years ago by jamoore

jvisualvm screenshot of the pixeldata phase

comment:3 Changed 6 years ago by jamoore

  • Cc fs@… added
  • Owner changed from jamoore to mlinkert

Passing to Melissa for review. There's likely nothing that can/should be one for RC1.

comment:4 Changed 6 years ago by mlinkert

  • Milestone changed from 5.0.0-rc1 to 5.0.0
  • Priority changed from critical to major

The difference in log file size is due to a difference in optimal tile height (and likely the height of pixel data strips stored in the files). I would guess that this also has some performance impact.

Probably the first thing to do after RC1 is to look at the export code; there are ways to make the strips larger when appropriate, which may help matters.

comment:5 Changed 6 years ago by mlinkert

  • Resolution set to fixed
  • Status changed from new to closed

PRs opened:

​https://github.com/openmicroscopy/bioformats/pull/871
​https://github.com/openmicroscopy/openmicroscopy/pull/2029

In my tests, the import times are now roughly the same for the original vs. exported files.

comment:6 Changed 5 years ago by wmoore

  • Milestone changed from 5.0.0 to 5.0.1
  • Resolution fixed deleted
  • Status changed from closed to reopened

Another report of "OME.tiff import slow" http://www.openmicroscopy.org/community/viewtopic.php?f=6&t=7442&p=13590 so I'm re-opening this (please re-assign if necessary). yuriy created these OME-XMLs himself, but I don't have the details.

Files were submitted to QA: 7942

Last edited 5 years ago by wmoore (previous) (diff)

comment:8 Changed 5 years ago by mlinkert

  • Resolution set to fixed
  • Status changed from reopened to closed

PR opened: ​https://github.com/openmicroscopy/bioformats/pull/1009

Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.81438 sec.)

© 2000-2014 University of Dundee & Open Microscopy Environment. Creative Commons Attribution 3.0 Unported License
OME source code is available under the GNU General public license or through commercial license from Glencoe Software

We're Hiring!