Investigate improving write performance in loci.common.*

[mlinkert]

See this thread on ome-devel: ​http://lists.openmicroscopy.org.uk/pipermail/ome-devel/2011-June/001989.html

The root problem is that random access writes to a network share are (predictably) slow, which means that converting files to TIFF over a network share is quite painful. It has been suggested that better buffering, and potentially the use of a StreamHandle? that wraps a BufferedOutputStream?, would improve performance.

Investigation is needed to determine whether that approach makes sense, given the overhead of closing and re-opening streams to achieve random access. It may also be worthwhile to improve NIOFileHandle's buffering to accomplish the same thing (maybe use a larger default buffer if the access mode is READ_WRITE?).

[Melissa Linkert <melissa@…>]

(In [9d2a6867540469c55870e8feb2fed299ff190e61/bioformats.git]) Make the default buffer sizes adjustable.

See #5965.

[mlinkert]

After quite a bit of investigation, the current solution is to have a *much* smaller reading buffer for writable files. Because of the random access nature of writing TIFF files, the buffer must be reset quite often, so having a smaller buffer means less time spent on (mostly useless) reads. The difference when writing small files is negligible, but I noticed a ~25% speedup when converting a full plate.

I've also added two new static methods to NIOFileHandle, so that calling code can adjust the default buffer sizes as desired.

Note that I did try implementing a StreamHandle? backed by a BufferedOutputStream?, but this was incredibly slow due to the fact that the stream must be closed and re-opened any time there is a backwards seek (and there are many of these when writing TIFF files).

[Melissa Linkert <melissa@…>]

(In [e1bcc7082f0103c9d848454e58c3d2a365f51b8e/bioformats.git]) Make the default buffer sizes adjustable.

See #5965.
(cherry picked from commit 9d2a6867540469c55870e8feb2fed299ff190e61)

[Melissa Linkert <melissa@…>]

(In [e1bcc7082f0103c9d848454e58c3d2a365f51b8e/bioformats.git]) Make the default buffer sizes adjustable.

See #5965.
(cherry picked from commit 9d2a6867540469c55870e8feb2fed299ff190e61)

[jmoore]

Melissa, since buffer size is probably something that is not often modified, the static solution is probably ok, but in general, this works against the multi-threaded requirement that's been a goal for the last several releases. If (and I admit it might be far-fetched) 2 omero threads both have some specific buffer-size requirement, then there is either a race condition or each thread must take out a lock on the NIOFileHandle class itself which introduces a different performance bottleneck.

Looking forward, I think we're going to need a configuration object in any situation where one would want modifiable static state. The configuration object would need to get passed to all newly created objects in the threaded case. (Something akin to the service factory).

[Melissa Linkert <melissa@…>]

(In [9d2a6867540469c55870e8feb2fed299ff190e61/bioformats.git]) Make the default buffer sizes adjustable.

See #5965.