Warning: Can't synchronize with repository "(default)" (/home/git/ome.git does not appear to be a Git repository.). Look in the Trac log for more information.
Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Task #5552 (closed)

Opened 13 years ago

Closed 13 years ago

Last modified 13 years ago

RFE: Speed up Nikon-nef import

Reported by: jamoore Owned by: mlinkert
Priority: critical Milestone: OMERO-Beta4.3
Component: Bio-Formats Version: n.a.
Keywords: n.a. Cc: cxallan, jburel
Resources: n.a. Referenced By: n.a.
References: n.a. Remaining Time: 0.0d
Sprint: 2011-06-02 (13)

Description

From the insight log (which is still running):

2035275 ms since start (2011-05-25 11:43:09,413) ** DEBUG **
 [thread: Thread-110][class: loci.formats.FormatHandler] -
 loci.formats.in.NikonReader.initFile(/Users/moore/NoBackup/SampleImages/nikon-nef/melissa/DJC_D2X_DOF_0004.nef)


...

4304245 ms since start (2011-05-25 12:20:58,383) ** DEBUG **
 [thread: Thread-110][class: ome.formats.importer.ImportLibrary] -
 Plane:1 X:1792 Y:2816 TileWidth:256 TileHeight:52 arrayBuf.length:26624

i.e. 35+ minutes for the first plane of the 11M file

Pseudo-histo gram of the per tile times to date:

~/omero/log $ grep -B2 TileWidth omeroinsight.log.1 omeroinsight.log | grep ms | cut -d- -f2 | cut -d" " -f1 | python diff.py  | sort -n | uniq -c
  93    0 s
   2    1 s
   4    4 s
 304    5 s
  39    6 s
  22    7 s
  14    8 s
   5    9 s
   2    10 s
   3    13 s
   1    14 s
   1    365 s

Almost all checks of the stack show:

"Thread-110" prio=6 tid=102b57000 nid=0x12678e000 runnable [12678c000]
   java.lang.Thread.State: RUNNABLE
	at loci.formats.codec.BitBuffer.getBits(BitBuffer.java:176)
	at loci.formats.in.NikonReader.openBytes(NikonReader.java:223)
	at loci.formats.ImageReader.openBytes(ImageReader.java:394)
	at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:183)
	at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:175)
	at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:180)
	at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:304)
	at loci.formats.MinMaxCalculator.openBytes(MinMaxCalculator.java:260)
	at ome.formats.importer.ImportLibrary.writeDataTileBased(ImportLibrary.java:756)
	at ome.formats.importer.ImportLibrary.importData(ImportLibrary.java:681)
	at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:477)
	at org.openmicroscopy.shoola.env.data.OMEROGateway.importImage(OMEROGateway.java:6142)
	at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importCandidates(OmeroImageServiceImpl.java:179)
	at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1185)
	at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:85)
	at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:54)
	at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:110)
	at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
	at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
	at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
	at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
	at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:274)
	at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
	at java.lang.Thread.run(Thread.java:680)

Change History (5)

comment:1 Changed 13 years ago by jmoore

One more data point. Here's a bash-based profiling of the writeDataTileBased method:

~/omero/log $ sort 14578.log | uniq -c
 203 	at loci.formats.MinMaxCalculator.openBytes(MinMaxCalculator.java:260)
 107 	at ome.formats.OMEROMetadataStoreClient.setTile(OMEROMetadataStoreClient.java:2479)

comment:2 Changed 13 years ago by mlinkert

  • Sprint set to 2011-06-02 (13)

comment:3 Changed 13 years ago by Melissa Linkert <melissa@…>

  • Remaining Time set to 0
  • Resolution set to fixed
  • Status changed from new to closed

(In [f4f937586a32fc6ed79a8d3bdc341482b66b8614/bioformats.git]) Cache each plane after its been decoded.

That way, we don't have to do a full decode for every single tile.
Closes #5552.

comment:4 Changed 13 years ago by Melissa Linkert <melissa@…>

(In [f4f937586a32fc6ed79a8d3bdc341482b66b8614/bioformats.git]) Cache each plane after its been decoded.

That way, we don't have to do a full decode for every single tile.
Closes #5552.

comment:5 Changed 13 years ago by jmoore

Not more than 15 minutes for the whole thing rather than 30+ minutes for plane 1. Kudos.

Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.68338 sec.)

We're Hiring!