Task #5552 (closed)
RFE: Speed up Nikon-nef import
Reported by: | jamoore | Owned by: | mlinkert |
---|---|---|---|
Priority: | critical | Milestone: | OMERO-Beta4.3 |
Component: | Bio-Formats | Version: | n.a. |
Keywords: | n.a. | Cc: | cxallan, jburel |
Resources: | n.a. | Referenced By: | n.a. |
References: | n.a. | Remaining Time: | 0.0d |
Sprint: | 2011-06-02 (13) |
Description
From the insight log (which is still running):
2035275 ms since start (2011-05-25 11:43:09,413) ** DEBUG ** [thread: Thread-110][class: loci.formats.FormatHandler] - loci.formats.in.NikonReader.initFile(/Users/moore/NoBackup/SampleImages/nikon-nef/melissa/DJC_D2X_DOF_0004.nef) ... 4304245 ms since start (2011-05-25 12:20:58,383) ** DEBUG ** [thread: Thread-110][class: ome.formats.importer.ImportLibrary] - Plane:1 X:1792 Y:2816 TileWidth:256 TileHeight:52 arrayBuf.length:26624
i.e. 35+ minutes for the first plane of the 11M file
Pseudo-histo gram of the per tile times to date:
~/omero/log $ grep -B2 TileWidth omeroinsight.log.1 omeroinsight.log | grep ms | cut -d- -f2 | cut -d" " -f1 | python diff.py | sort -n | uniq -c 93 0 s 2 1 s 4 4 s 304 5 s 39 6 s 22 7 s 14 8 s 5 9 s 2 10 s 3 13 s 1 14 s 1 365 s
Almost all checks of the stack show:
"Thread-110" prio=6 tid=102b57000 nid=0x12678e000 runnable [12678c000] java.lang.Thread.State: RUNNABLE at loci.formats.codec.BitBuffer.getBits(BitBuffer.java:176) at loci.formats.in.NikonReader.openBytes(NikonReader.java:223) at loci.formats.ImageReader.openBytes(ImageReader.java:394) at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:183) at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:175) at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:180) at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:304) at loci.formats.MinMaxCalculator.openBytes(MinMaxCalculator.java:260) at ome.formats.importer.ImportLibrary.writeDataTileBased(ImportLibrary.java:756) at ome.formats.importer.ImportLibrary.importData(ImportLibrary.java:681) at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:477) at org.openmicroscopy.shoola.env.data.OMEROGateway.importImage(OMEROGateway.java:6142) at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importCandidates(OmeroImageServiceImpl.java:179) at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1185) at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:85) at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:54) at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:110) at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144) at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226) at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126) at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165) at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:274) at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91) at java.lang.Thread.run(Thread.java:680)
Change History (5)
comment:1 Changed 13 years ago by jmoore
comment:2 Changed 13 years ago by mlinkert
- Sprint set to 2011-06-02 (13)
comment:3 Changed 13 years ago by Melissa Linkert <melissa@…>
- Remaining Time set to 0
- Resolution set to fixed
- Status changed from new to closed
(In [f4f937586a32fc6ed79a8d3bdc341482b66b8614/bioformats.git]) Cache each plane after its been decoded.
That way, we don't have to do a full decode for every single tile.
Closes #5552.
comment:4 Changed 13 years ago by Melissa Linkert <melissa@…>
(In [f4f937586a32fc6ed79a8d3bdc341482b66b8614/bioformats.git]) Cache each plane after its been decoded.
That way, we don't have to do a full decode for every single tile.
Closes #5552.
comment:5 Changed 13 years ago by jmoore
Not more than 15 minutes for the whole thing rather than 30+ minutes for plane 1. Kudos.
One more data point. Here's a bash-based profiling of the writeDataTileBased method: