Warning: Can't synchronize with repository "(default)" (/home/git/ome.git does not appear to be a Git repository.). Look in the Trac log for more information.
Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Task #12930 (closed)

Opened 9 years ago

Closed 9 years ago

Last modified 9 years ago

Bug: TIFF parser performance regression

Reported by: rleigh Owned by: mlinkert
Priority: major Milestone: 5.1.3
Component: Bio-Formats Version: 5.1.2
Keywords: performance Cc: jan.eglinger@…
Resources: n.a. Referenced By: n.a.
References: n.a. Remaining Time: 0.0d
Sprint: n.a.

Description

File: Jumping_dom_brain3_slide2_s4.ome.tiff from QA #11125

This is 11313x9450 with no tiling (1 strip/row). For some reason this is performing extremely poorly on Windows, by at least three orders of magnitude.

ImageJ:
MacOSX10.10/ImageJ 1.49u/BF 5.1.2: Load time is ~5s (hot cache)
Windows/Fiji?/BF 5.1.2: Load time is ~30mins

bfconvert:
Reports tile size as 11313 x 46
MacOS10.10/develop: 308.12s elapsed (766.0+152378.0ms per plane, 1202ms overhead)
FreeBSD10.1/develop: 1779.888s elapsed (4408.5+883832.5ms per plane, 3250ms overhead)
Windows 8.1x64/develop: 1713.207s elapsed (4443.5+851598.0ms per plane, 1090ms overhead)

While the bfconvert performance is not as good (despite this being a beefy 4GHz 8 core system, much better than all the others--the BSD system is much slower), this doesn't explain why the ImageJ plugin performance is so bad, or why it's platform-specific. Is the long strip size interacting very badly with how the plugin calls openBytes or something?

Looking at stacks of the frozen Fiji import, it's stuck on IO. Note that the disk light was mostly inactive; this is not actually blocked on physical IO. It's on a SATA III SSD and likely cached in memory as well.

"main" #1 prio=5 os_prio=0 tid=0x0000000002aed800 nid=0x1654 runnable [0x0000000002c9e000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.FileDispatcherImpl.pread0(Native Method)
        at sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:66)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:220)
        at sun.nio.ch.IOUtil.read(IOUtil.java:197)
        at sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:708)
        at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:691)
        - locked <0x00000000e004bcc8> (a java.lang.Object)
        at loci.common.NIOByteBufferProvider.allocateDirect(NIOByteBufferProvider.java:127)
        at loci.common.NIOByteBufferProvider.allocate(NIOByteBufferProvider.java:112)
        at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:600)
        at loci.common.NIOFileHandle.seek(NIOFileHandle.java:291)
        at loci.common.RandomAccessInputStream.seek(RandomAccessInputStream.java:174)
        at loci.formats.tiff.OnDemandLongArray.get(OnDemandLongArray.java:67)
        at loci.formats.tiff.OnDemandLongArray.toArray(OnDemandLongArray.java:80)
        at loci.formats.tiff.IFD.getIFDLongArray(IFD.java:408)
        at loci.formats.tiff.IFD.getStripByteCounts(IFD.java:805)
        at loci.formats.tiff.TiffParser.getTile(TiffParser.java:686)
        at loci.formats.tiff.TiffParser.getSamples(TiffParser.java:990)
        at loci.formats.tiff.TiffParser.getSamples(TiffParser.java:785)
        at loci.formats.in.OMETiffReader.openBytes(OMETiffReader.java:314)
        at loci.formats.FormatReader.openBytes(FormatReader.java:920)
        at loci.formats.ImageReader.openBytes(ImageReader.java:437)
        at loci.formats.tools.ImageConverter.convertTilePlane(ImageConverter.java:681)
        at loci.formats.tools.ImageConverter.convertPlane(ImageConverter.java:617)
        at loci.formats.tools.ImageConverter.testConvert(ImageConverter.java:549)
        at loci.formats.tools.ImageConverter.main(ImageConverter.java:874)

Change History (7)

comment:1 Changed 9 years ago by rleigh

  • Component changed from General to Bio-Formats
  • Owner set to mlinkert

comment:2 Changed 9 years ago by mlinkert

  • Milestone changed from Unscheduled to 5.1.4

Original thread was: ​http://lists.openmicroscopy.org.uk/pipermail/ome-users/2015-June/005427.html

comment:3 Changed 9 years ago by rleigh

Some further details. The MacOS test was with Fiji, and this doesn't correlate with plain ImageJ and bioformats_package.jar, which shows:

5.1.0 ~10s (Windows and MacOS)
5.1.1 ~10s (Windows and MacOS)
5.1.2 22.5 min (Windows) and ~5 min (MacOS)
develop 22.5 min (Windows) and ~5 min (MacOS)

So it looks like the regression was introduced between 5.1.1 and 5.1.2. It does affect MacOS, but not to the same degree as Windows.

comment:4 Changed 9 years ago by imunro-x

I have been looking at the file uploaded to QA #11133 using the command-line tools.

xmlvalid passed, tiffcomment looked as I'd expect.

showinf Using the 5.1.2 tools gave me:

eading pixel data (0-1739)

Read 4/1740 planes (0%)
Read 10/1740 planes (0%)
Read 17/1740 planes (0%)
Read 23/1740 planes (1%)
Read 31/1740 planes (1%)
Read 38/1740 planes (2%)
Read 46/1740 planes (2%)
Read 53/1740 planes (3%)
Read 60/1740 planes (3%)
Read 67/1740 planes (3%)
Read 74/1740 planes (4%)
Read 83/1740 planes (4%)
Read 92/1740 planes (5%)
Read 101/1740 planes (5%)
Read 110/1740 planes (6%)
Read 119/1740 planes (6%)
Read 128/1740 planes (7%)
Read 137/1740 planes (7%)
Read 146/1740 planes (8%)
Read 155/1740 planes (8%)
Read 164/1740 planes (9%)
Read 173/1740 planes (9%)
Read 182/1740 planes (10%)
Read 191/1740 planes (10%)
Read 200/1740 planes (11%)
Read 209/1740 planes (12%)
Read 218/1740 planes (12%)
Read 227/1740 planes (13%)
Read 236/1740 planes (13%)
Read 245/1740 planes (14%)
Read 254/1740 planes (14%)
Read 263/1740 planes (15%)
Read 272/1740 planes (15%)
Read 281/1740 planes (16%)
Read 290/1740 planes (16%)
Read 299/1740 planes (17%)
Read 308/1740 planes (17%)
Read 317/1740 planes (18%)
Read 326/1740 planes (18%)
Read 335/1740 planes (19%)
Read 344/1740 planes (19%)
Read 353/1740 planes (20%)
Read 362/1740 planes (20%)
Read 371/1740 planes (21%)
Read 380/1740 planes (21%)
Read 389/1740 planes (22%)
Read 398/1740 planes (22%)
Read 407/1740 planes (23%)
Read 415/1740 planes (23%)
Read 424/1740 planes (24%)
Read 433/1740 planes (24%)
Read 442/1740 planes (25%)
Read 451/1740 planes (25%)
Read 460/1740 planes (26%)
Read 469/1740 planes (26%)
Read 478/1740 planes (27%)
Read 487/1740 planes (27%)
Read 496/1740 planes (28%)
Read 505/1740 planes (29%)
Read 514/1740 planes (29%)
Read 523/1740 planes (30%)
Read 532/1740 planes (30%)
Read 540/1740 planes (31%)
Read 548/1740 planes (31%)
Read 556/1740 planes (31%)
Read 565/1740 planes (32%)
Read 574/1740 planes (32%)
Read 582/1740 planes (33%)
Read 591/1740 planes (33%)
Read 600/1740 planes (34%)
Read 608/1740 planes (34%)
Read 617/1740 planes (35%)
Read 626/1740 planes (35%)
Read 635/1740 planes (36%)
Read 643/1740 planes (36%)
Read 652/1740 planes (37%)
Read 661/1740 planes (37%)
Read 670/1740 planes (38%)
Read 679/1740 planes (39%)
Read 688/1740 planes (39%)
Read 697/1740 planes (40%)
Read 706/1740 planes (40%)
Read 715/1740 planes (41%)
Read 724/1740 planes (41%)
Read 733/1740 planes (42%)
Read 742/1740 planes (42%)
Read 751/1740 planes (43%)
Read 760/1740 planes (43%)
Read 769/1740 planes (44%)
Read 778/1740 planes (44%)
Read 787/1740 planes (45%)
Read 796/1740 planes (45%)
Read 805/1740 planes (46%)
Read 814/1740 planes (46%)
Read 823/1740 planes (47%)
Read 832/1740 planes (47%)
Read 841/1740 planes (48%)
Read 850/1740 planes (48%)
Read 859/1740 planes (49%)
Read 868/1740 planes (49%)
Read 877/1740 planes (50%)
Read 886/1740 planes (50%)
Read 895/1740 planes (51%)
Read 904/1740 planes (51%)
Read 913/1740 planes (52%)
Read 922/1740 planes (52%)
Read 931/1740 planes (53%)
Read 940/1740 planes (54%)
Read 949/1740 planes (54%)
Read 958/1740 planes (55%)
Read 967/1740 planes (55%)
Read 976/1740 planes (56%)
Read 985/1740 planes (56%)
Read 994/1740 planes (57%)
Read 1003/1740 planes (57%)
Read 1012/1740 planes (58%)
Read 1021/1740 planes (58%)
Read 1030/1740 planes (59%)
Read 1039/1740 planes (59%)
Read 1048/1740 planes (60%)
Read 1057/1740 planes (60%)
Read 1066/1740 planes (61%)
Read 1075/1740 planes (61%)
Read 1084/1740 planes (62%)
Read 1093/1740 planes (62%)
Read 1101/1740 planes (63%)
Read 1110/1740 planes (63%)
Read 1119/1740 planes (64%)
Read 1128/1740 planes (64%)
Read 1137/1740 planes (65%)
Read 1146/1740 planes (65%)
Read 1155/1740 planes (66%)
Read 1164/1740 planes (66%)
Read 1173/1740 planes (67%)
Read 1182/1740 planes (67%)
Read 1191/1740 planes (68%)
Read 1200/1740 planes (68%)
Read 1209/1740 planes (69%)
Read 1218/1740 planes (70%)
Read 1227/1740 planes (70%)
Read 1236/1740 planes (71%)
Read 1245/1740 planes (71%)
Read 1254/1740 planes (72%)
Read 1263/1740 planes (72%)
Read 1272/1740 planes (73%)
Read 1281/1740 planes (73%)
Read 1290/1740 planes (74%)
Read 1299/1740 planes (74%)
Read 1308/1740 planes (75%)
Read 1317/1740 planes (75%)
Read 1326/1740 planes (76%)
Read 1335/1740 planes (76%)
Read 1344/1740 planes (77%)
Read 1353/1740 planes (77%)
Read 1362/1740 planes (78%)
Read 1371/1740 planes (78%)
Read 1380/1740 planes (79%)
Read 1389/1740 planes (79%)
Read 1398/1740 planes (80%)
Read 1407/1740 planes (80%)
Read 1416/1740 planes (81%)
Read 1425/1740 planes (81%)
Read 1434/1740 planes (82%)
Read 1443/1740 planes (82%)
Read 1452/1740 planes (83%)
Read 1461/1740 planes (83%)
Read 1469/1740 planes (84%)
Read 1477/1740 planes (84%)
Read 1486/1740 planes (85%)
Read 1495/1740 planes (85%)
Read 1504/1740 planes (86%)
Read 1512/1740 planes (86%)
Read 1521/1740 planes (87%)
Read 1530/1740 planes (87%)
Read 1539/1740 planes (88%)
Read 1548/1740 planes (88%)
Read 1556/1740 planes (89%)
Read 1564/1740 planes (89%)
Read 1573/1740 planes (90%)
Read 1582/1740 planes (90%)
Read 1591/1740 planes (91%)
Read 1599/1740 planes (91%)
Read 1608/1740 planes (92%)
Read 1617/1740 planes (92%)
Read 1626/1740 planes (93%)
Read 1635/1740 planes (93%)
Read 1644/1740 planes (94%)
Read 1653/1740 planes (95%)
Read 1662/1740 planes (95%)
Read 1671/1740 planes (96%)
Read 1680/1740 planes (96%)
Read 1689/1740 planes (97%)
Read 1698/1740 planes (97%)
Read 1707/1740 planes (98%)
Read 1716/1740 planes (98%)
Read 1725/1740 planes (99%)
Read 1734/1740 planes (99%)
Read 1740/1740 planes (100%)

[done]
210.746s elapsed (121.11839ms per plane)

By contrast the 5.0.8 tools gave me:

Reading pixel data (0-1739)

Read 56/1740 planes (3%)
Read 178/1740 planes (10%)
Read 288/1740 planes (16%)
Read 432/1740 planes (24%)
Read 553/1740 planes (31%)
Read 697/1740 planes (40%)
Read 830/1740 planes (47%)
Read 974/1740 planes (55%)
Read 1114/1740 planes (64%)
Read 1249/1740 planes (71%)
Read 1390/1740 planes (79%)
Read 1534/1740 planes (88%)
Read 1666/1740 planes (95%)
Read 1740/1740 planes (100%)

[done]
13.587s elapsed (7.8086205ms per plane)

Both on OSX.

comment:5 Changed 9 years ago by mlinkert

  • Milestone changed from 5.1.4 to 5.1.3
  • Summary changed from Bug: TIFF parser/ImageJ plugin performance regression on Windows to Bug: TIFF parser performance regression

comment:6 Changed 9 years ago by mlinkert

  • Resolution set to fixed
  • Status changed from new to closed

Should be fixed with: ​https://github.com/openmicroscopy/bioformats/pull/1848

comment:7 Changed 9 years ago by Melissa Linkert <melissa@…>

  • Remaining Time set to 0

(In [16d85f66343b8ed49d2cff4e88739cb00e87d310/bioformats.git] on branch develop) TIFF: reduce number of seeks in OnDemandLongArray?

This should reduce the seek count to O(1) instead of O(n) when calling
toArray().

Fixes #12930.

Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.66251 sec.)

© 2000-2014 University of Dundee & Open Microscopy Environment. Creative Commons Attribution 3.0 Unported License
OME source code is available under the GNU General public license or through commercial license from Glencoe Software

We're Hiring!