Task #6520 (closed)
Opened 13 years ago
Closed 13 years ago
Bug: bigtiff fails on export
Reported by: | jamoore | Owned by: | jamoore |
---|---|---|---|
Priority: | major | Milestone: | OMERO-Beta4.3.2 |
Component: | Services | Version: | n.a. |
Keywords: | n.a. | Cc: | jburel, cxallan, a.herbert@…, mlinkert |
Resources: | n.a. | Referenced By: | 6589 |
References: | n.a. | Remaining Time: | 0.0d |
Sprint: | 2011-09-15 (5) |
Description
Export of images over 4 GB causes a "Negative position" error in NIO's FileChannelImpl during OMETiffWriter.close.
See: https://www.openmicroscopy.org/community/viewtopic.php?f=4&t=785
(stack trace below)
Change History (17)
comment:1 Changed 13 years ago by jmoore
comment:2 Changed 13 years ago by mlinkert
- Sprint set to 2011-09-01 (4)
comment:3 Changed 13 years ago by mlinkert
- Status changed from new to accepted
comment:4 Changed 13 years ago by mlinkert
There are two possible solutions to this as I see it:
- Update ExporterI to call TiffWriter?.setBigTiff(true) if it looks like a > 4 GB file will be generated.
- Update TiffWriter? to magically switch to BigTIFF if it looks like > 4 GB of data will be written.
Updating ExporterI is maybe less invasive, but updating TiffWriter? is more friendly for anyone who is using Bio-Formats. Any opinions on which option to pursue would be welcome.
comment:5 Changed 13 years ago by jmoore
- Cc mlinkert added
- Owner changed from mlinkert-x to jmoore
From today's conf call, fixing in 4.3.2 in exporter but will add an RFE for a more generic bioformats fix in 4.4+.
comment:6 Changed 13 years ago by mlinkert
- Referenced By set to #6589
comment:7 Changed 13 years ago by jmoore
- Referenced By changed from #6589 to 6589
comment:8 follow-up: ↓ 11 Changed 13 years ago by jmoore
Opinions:
~/git $ git diff components/blitz/src/ome/services/blitz/impl/ diff --git a/components/blitz/src/ome/services/blitz/impl/ExporterI.java b/components/blitz/src/ome/services/blitz/impl/ExporterI.java index e660408..a46285f 100644 --- a/components/blitz/src/ome/services/blitz/impl/ExporterI.java +++ b/components/blitz/src/ome/services/blitz/impl/ExporterI.java @@ -24,6 +24,7 @@ import loci.formats.ImageWriter; import loci.formats.MetadataTools; import loci.formats.meta.IMetadata; import loci.formats.meta.MetadataRetrieve; +import loci.formats.out.TiffWriter; import loci.formats.services.OMEXMLService; import ome.api.RawPixelsStore; import ome.conditions.ApiUsageException; @@ -349,6 +350,13 @@ public class ExporterI extends AbstractAmdServant implements writer.setMetadataRetrieve(retrieve); writer.setId(file.getAbsolutePath()); + final boolean bigtiff = + ( raw.getTotalSize() > Integer.MAX_VALUE * 2); + + if (bigtiff) { + ((TiffWriter) writer.getWriter()).setBigTiff(true); + } + int planeCount = reader.planes; int planeSize = raw.getPlaneSize(); byte[] plane = new byte[planeSize];
There is no case where this could not be a TiffWriter?, right, Melissa?
comment:9 Changed 13 years ago by jmoore
- Owner jmoore deleted
- Status changed from accepted to new
comment:10 Changed 13 years ago by jmoore
- Owner set to jmoore
Also, do we try to setup an automated test for this???
comment:11 in reply to: ↑ 8 Changed 13 years ago by mlinkert
Replying to jmoore:
Opinions:
~/git $ git diff components/blitz/src/ome/services/blitz/impl/ diff --git a/components/blitz/src/ome/services/blitz/impl/ExporterI.java b/components/blitz/src/ome/services/blitz/impl/ExporterI.java index e660408..a46285f 100644 --- a/components/blitz/src/ome/services/blitz/impl/ExporterI.java +++ b/components/blitz/src/ome/services/blitz/impl/ExporterI.java @@ -24,6 +24,7 @@ import loci.formats.ImageWriter; import loci.formats.MetadataTools; import loci.formats.meta.IMetadata; import loci.formats.meta.MetadataRetrieve; +import loci.formats.out.TiffWriter; import loci.formats.services.OMEXMLService; import ome.api.RawPixelsStore; import ome.conditions.ApiUsageException; @@ -349,6 +350,13 @@ public class ExporterI extends AbstractAmdServant implements writer.setMetadataRetrieve(retrieve); writer.setId(file.getAbsolutePath()); + final boolean bigtiff = + ( raw.getTotalSize() > Integer.MAX_VALUE * 2); + + if (bigtiff) { + ((TiffWriter) writer.getWriter()).setBigTiff(true); + } + int planeCount = reader.planes; int planeSize = raw.getPlaneSize(); byte[] plane = new byte[planeSize];
You might consider checking for (Integer.MAX_VALUE * 1.99) (or something else slightly smaller than 4 GB). The total size of the file must be < 4 GB for non-BigTIFF, and there is some overhead for writing the file metadata (~256-512 bytes/plane + the OME-XML metadata).
There is no case where this could not be a TiffWriter?, right, Melissa?
Correct.
comment:12 follow-up: ↓ 13 Changed 13 years ago by jmoore
Ok, so that's about 20 MB of overhead, right? How often for a 4+ GB file could the OME-XML metadata be larger than that? Can I get the XML string before hand, say in a tmp file and calculate the size? Then, in some cases, the available space would actually only be 4 MB for the nightshade files:
omero3=> select distinct (21474836 - (512 * sizet * sizec * sizez)) as leftover_bytes from pixels order by sizet*sizec*sizez desc; leftover_bytes ---------------- 4333076 9770516 9924116 12243476 ...
comment:13 in reply to: ↑ 12 Changed 13 years ago by mlinkert
Replying to jmoore:
Ok, so that's about 20 MB of overhead, right? How often for a 4+ GB file could the OME-XML metadata be larger than that?
It depends largely upon the configuration of the pixel data (lots of little planes will have more overhead than a few really big planes). I think if you allow 512 bytes per plane plus the size of the XML you should be OK.
Can I get the XML string before hand, say in a tmp file and calculate the size?
You should be able to get the XML string in advance like this:
ServiceFactory factory = new ServiceFactory(); OMEXMLService service = factory.getInstance(OMEXMLService.class); String xml = service.getOMEXML(retrieve);
comment:14 Changed 13 years ago by jburel
- Sprint changed from 2011-09-01 (4) to 2011-09-15 (5)
Moved from sprint 2011-09-01 (4)
comment:15 Changed 13 years ago by jmoore <josh@…>
(In [c037e050b579f031a53ebba957877b5a2123591a/ome.git] on branch develop) Set bigtiff for 4GB+ exports (See #6520)
comment:16 Changed 13 years ago by jmoore <josh@…>
(In [a8b80849eb8d21fbdf08eb76b70fef380f3e6ed2/ome.git] on branch develop) Check xml+plane size for big-tiff in export (See #6520)
comment:17 Changed 13 years ago by jmoore <josh@…>
- Remaining Time set to 0
- Resolution set to fixed
- Status changed from new to closed
(In [6e09514503d87f9b680be6f88afc665ec1be514c/ome.git] on branch develop) Merge branch 'feature/export-bugs' into develop (Fix #6520, #6701, #6713)