Warning: Can't synchronize with repository "(default)" (/home/git/ome.git does not appear to be a Git repository.). Look in the Trac log for more information.
Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Task #5965 (closed)

Opened 10 years ago

Closed 10 years ago

Last modified 10 years ago

Investigate improving write performance in loci.common.*

Reported by: mlinkert Owned by: mlinkert
Priority: major Milestone: OMERO-Beta4.3.2
Component: Bio-Formats Version: n.a.
Keywords: n.a. Cc: ruben.munoz@…, jamoore, cxallan
Resources: n.a. Referenced By: n.a.
References: n.a. Remaining Time: n.a.
Sprint: 2011-09-01 (4)

Description

See this thread on ome-devel: http://lists.openmicroscopy.org.uk/pipermail/ome-devel/2011-June/001989.html

The root problem is that random access writes to a network share are (predictably) slow, which means that converting files to TIFF over a network share is quite painful. It has been suggested that better buffering, and potentially the use of a StreamHandle? that wraps a BufferedOutputStream?, would improve performance.

Investigation is needed to determine whether that approach makes sense, given the overhead of closing and re-opening streams to achieve random access. It may also be worthwhile to improve NIOFileHandle's buffering to accomplish the same thing (maybe use a larger default buffer if the access mode is READ_WRITE?).

Change History (9)

comment:1 Changed 10 years ago by jmoore

  • Cc jmoore added

comment:2 Changed 10 years ago by mlinkert

  • Sprint set to 2011-09-01 (4)

comment:3 Changed 10 years ago by mlinkert

  • Status changed from new to accepted

comment:4 Changed 10 years ago by Melissa Linkert <melissa@…>

(In [9d2a6867540469c55870e8feb2fed299ff190e61/bioformats.git]) Make the default buffer sizes adjustable.

See #5965.

comment:5 Changed 10 years ago by mlinkert

  • Resolution set to fixed
  • Status changed from accepted to closed

After quite a bit of investigation, the current solution is to have a *much* smaller reading buffer for writable files. Because of the random access nature of writing TIFF files, the buffer must be reset quite often, so having a smaller buffer means less time spent on (mostly useless) reads. The difference when writing small files is negligible, but I noticed a ~25% speedup when converting a full plate.

I've also added two new static methods to NIOFileHandle, so that calling code can adjust the default buffer sizes as desired.

Note that I did try implementing a StreamHandle? backed by a BufferedOutputStream?, but this was incredibly slow due to the fact that the stream must be closed and re-opened any time there is a backwards seek (and there are many of these when writing TIFF files).

comment:6 Changed 10 years ago by Melissa Linkert <melissa@…>

(In [e1bcc7082f0103c9d848454e58c3d2a365f51b8e/bioformats.git]) Make the default buffer sizes adjustable.

See #5965.
(cherry picked from commit 9d2a6867540469c55870e8feb2fed299ff190e61)

comment:7 Changed 10 years ago by Melissa Linkert <melissa@…>

(In [e1bcc7082f0103c9d848454e58c3d2a365f51b8e/bioformats.git]) Make the default buffer sizes adjustable.

See #5965.
(cherry picked from commit 9d2a6867540469c55870e8feb2fed299ff190e61)

comment:8 Changed 10 years ago by jmoore

  • Cc cxallan added

Melissa, since buffer size is probably something that is not often modified, the static solution is probably ok, but in general, this works against the multi-threaded requirement that's been a goal for the last several releases. If (and I admit it might be far-fetched) 2 omero threads both have some specific buffer-size requirement, then there is either a race condition or each thread must take out a lock on the NIOFileHandle class itself which introduces a different performance bottleneck.

Looking forward, I think we're going to need a configuration object in any situation where one would want modifiable static state. The configuration object would need to get passed to all newly created objects in the threaded case. (Something akin to the service factory).

comment:9 Changed 10 years ago by Melissa Linkert <melissa@…>

(In [9d2a6867540469c55870e8feb2fed299ff190e61/bioformats.git]) Make the default buffer sizes adjustable.

See #5965.

Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.65678 sec.)

We're Hiring!