Warning: Can't synchronize with repository "(default)" (/home/git/ome.git does not appear to be a Git repository.). Look in the Trac log for more information.
Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Task #6520 (closed)

Opened 10 years ago

Closed 10 years ago

Bug: bigtiff fails on export

Reported by: jamoore Owned by: jamoore
Priority: major Milestone: OMERO-Beta4.3.2
Component: Services Version: n.a.
Keywords: n.a. Cc: jburel, cxallan, a.herbert@…, mlinkert
Resources: n.a. Referenced By: 6589
References: n.a. Remaining Time: 0.0d
Sprint: 2011-09-15 (5)

Description

Export of images over 4 GB causes a "Negative position" error in NIO's FileChannelImpl during OMETiffWriter.close.

See: https://www.openmicroscopy.org/community/viewtopic.php?f=4&t=785

(stack trace below)

Change History (17)

comment:1 Changed 10 years ago by jmoore

2011-08-19 10:45:26,154 WARN  [              ome.xml.model.OMEModelImpl] (l.Server-1) ome.xml.model.Image@1afca609 reference to urn:lsid:export.openmicroscopy.org:Annotation:62ca249c-966a-41e2-b2ef-26393603daa2_454:19039 missing from object hierarchy.
2011-08-19 10:45:26,154 WARN  [              ome.xml.model.OMEModelImpl] (l.Server-1) ome.xml.model.Image@1afca609 reference to urn:lsid:export.openmicroscopy.org:Annotation:62ca249c-966a-41e2-b2ef-26393603daa2_454:19039 missing from object hierarchy.
2011-08-19 10:45:27,753 ERROR [       ome.services.blitz.impl.ExporterI] (l.Server-1) Error closing writer
java.lang.RuntimeException: java.lang.IllegalArgumentException: Negative position
	at loci.formats.out.OMETiffWriter.close(OMETiffWriter.java:126)
	at loci.formats.ImageWriter.close(ImageWriter.java:428)
	at ome.services.blitz.impl.ExporterI$2.cleanup(ExporterI.java:405)
	at ome.services.blitz.impl.ExporterI$2.doWork(ExporterI.java:381)
	at sun.reflect.GeneratedMethodAccessor259.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
	at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:440)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at ome.security.basic.EventHandler.invoke(EventHandler.java:150)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:231)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:116)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
	at $Proxy64.doWork(Unknown Source)
	at ome.services.util.Executor$Impl.execute(Executor.java:371)
	at ome.services.blitz.impl.ExporterI.do_tiff(ExporterI.java:316)
	at ome.services.blitz.impl.ExporterI.generateTiff_async(ExporterI.java:186)
	at omero.api._ExporterTie.generateTiff_async(_ExporterTie.java:71)
	at omero.api._ExporterDisp.___generateTiff(_ExporterDisp.java:175)
	at omero.api._ExporterDisp.__dispatch(_ExporterDisp.java:248)
	at IceInternal.Incoming.invoke(Incoming.java:159)
	at Ice.ConnectionI.invokeAll(ConnectionI.java:2037)
	at Ice.ConnectionI.message(ConnectionI.java:972)
	at IceInternal.ThreadPool.run(ThreadPool.java:577)
	at IceInternal.ThreadPool.access$100(ThreadPool.java:12)
	at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:971)
Caused by: java.lang.IllegalArgumentException: Negative position
	at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:651)
	at loci.common.NIOByteBufferProvider.allocateDirect(NIOByteBufferProvider.java:120)
	at loci.common.NIOByteBufferProvider.allocate(NIOByteBufferProvider.java:105)
	at loci.common.NIOFileHandle.buffer(NIOFileHandle.java:482)
	at loci.common.NIOFileHandle.seek(NIOFileHandle.java:204)
	at loci.common.RandomAccessOutputStream.seek(RandomAccessOutputStream.java:69)
	at loci.formats.tiff.TiffSaver.overwriteLastIFDOffset(TiffSaver.java:693)
	at loci.formats.out.OMETiffWriter.saveComment(OMETiffWriter.java:237)
	at loci.formats.out.OMETiffWriter.close(OMETiffWriter.java:110)
	... 35 more
2011-08-19 10:45:27,755 INFO  [                 org.perf4j.TimingLogger] (l.Server-1) start[1313745756266] time[1371489] tag[omero.call.success.ome.services.blitz.impl.ExporterI$2.doWork]
2011-08-19 10:45:27,755 INFO  [        ome.services.util.ServiceHandler] (l.Server-1)  Rslt:	null
2011-08-19 10:45:27,755 ERROR [        ome.services.util.ServiceHandler] (l.Server-1) Method interface ome.services.util.Executor$Work.doWork invocation took 1371489

comment:2 Changed 10 years ago by mlinkert

  • Sprint set to 2011-09-01 (4)

comment:3 Changed 10 years ago by mlinkert

  • Status changed from new to accepted

comment:4 Changed 10 years ago by mlinkert

There are two possible solutions to this as I see it:

  1. Update ExporterI to call TiffWriter?.setBigTiff(true) if it looks like a > 4 GB file will be generated.
  2. Update TiffWriter? to magically switch to BigTIFF if it looks like > 4 GB of data will be written.

Updating ExporterI is maybe less invasive, but updating TiffWriter? is more friendly for anyone who is using Bio-Formats. Any opinions on which option to pursue would be welcome.

comment:5 Changed 10 years ago by jmoore

  • Cc mlinkert added
  • Owner changed from mlinkert-x to jmoore

From today's conf call, fixing in 4.3.2 in exporter but will add an RFE for a more generic bioformats fix in 4.4+.

comment:6 Changed 10 years ago by mlinkert

comment:7 Changed 10 years ago by jmoore

comment:8 follow-up: Changed 10 years ago by jmoore

Opinions:

~/git $ git diff components/blitz/src/ome/services/blitz/impl/
diff --git a/components/blitz/src/ome/services/blitz/impl/ExporterI.java b/components/blitz/src/ome/services/blitz/impl/ExporterI.java
index e660408..a46285f 100644
--- a/components/blitz/src/ome/services/blitz/impl/ExporterI.java
+++ b/components/blitz/src/ome/services/blitz/impl/ExporterI.java
@@ -24,6 +24,7 @@ import loci.formats.ImageWriter;
 import loci.formats.MetadataTools;
 import loci.formats.meta.IMetadata;
 import loci.formats.meta.MetadataRetrieve;
+import loci.formats.out.TiffWriter;
 import loci.formats.services.OMEXMLService;
 import ome.api.RawPixelsStore;
 import ome.conditions.ApiUsageException;
@@ -349,6 +350,13 @@ public class ExporterI extends AbstractAmdServant implements
                                 writer.setMetadataRetrieve(retrieve);
                                 writer.setId(file.getAbsolutePath());
 
+                                final boolean bigtiff =
+                                    ( raw.getTotalSize() > Integer.MAX_VALUE * 2);
+
+                                if (bigtiff) {
+                                    ((TiffWriter) writer.getWriter()).setBigTiff(true);
+                                }
+
                                 int planeCount = reader.planes;
                                 int planeSize = raw.getPlaneSize();
                                 byte[] plane = new byte[planeSize];

There is no case where this could not be a TiffWriter?, right, Melissa?

comment:9 Changed 10 years ago by jmoore

  • Owner jmoore deleted
  • Status changed from accepted to new

comment:10 Changed 10 years ago by jmoore

  • Owner set to jmoore

Also, do we try to setup an automated test for this???

comment:11 in reply to: ↑ 8 Changed 10 years ago by mlinkert

Replying to jmoore:

Opinions:

~/git $ git diff components/blitz/src/ome/services/blitz/impl/
diff --git a/components/blitz/src/ome/services/blitz/impl/ExporterI.java b/components/blitz/src/ome/services/blitz/impl/ExporterI.java
index e660408..a46285f 100644
--- a/components/blitz/src/ome/services/blitz/impl/ExporterI.java
+++ b/components/blitz/src/ome/services/blitz/impl/ExporterI.java
@@ -24,6 +24,7 @@ import loci.formats.ImageWriter;
 import loci.formats.MetadataTools;
 import loci.formats.meta.IMetadata;
 import loci.formats.meta.MetadataRetrieve;
+import loci.formats.out.TiffWriter;
 import loci.formats.services.OMEXMLService;
 import ome.api.RawPixelsStore;
 import ome.conditions.ApiUsageException;
@@ -349,6 +350,13 @@ public class ExporterI extends AbstractAmdServant implements
                                 writer.setMetadataRetrieve(retrieve);
                                 writer.setId(file.getAbsolutePath());
 
+                                final boolean bigtiff =
+                                    ( raw.getTotalSize() > Integer.MAX_VALUE * 2);
+
+                                if (bigtiff) {
+                                    ((TiffWriter) writer.getWriter()).setBigTiff(true);
+                                }
+
                                 int planeCount = reader.planes;
                                 int planeSize = raw.getPlaneSize();
                                 byte[] plane = new byte[planeSize];

You might consider checking for (Integer.MAX_VALUE * 1.99) (or something else slightly smaller than 4 GB). The total size of the file must be < 4 GB for non-BigTIFF, and there is some overhead for writing the file metadata (~256-512 bytes/plane + the OME-XML metadata).


There is no case where this could not be a TiffWriter?, right, Melissa?

Correct.

comment:12 follow-up: Changed 10 years ago by jmoore

Ok, so that's about 20 MB of overhead, right? How often for a 4+ GB file could the OME-XML metadata be larger than that? Can I get the XML string before hand, say in a tmp file and calculate the size? Then, in some cases, the available space would actually only be 4 MB for the nightshade files:

omero3=> select distinct (21474836 - (512 * sizet * sizec * sizez)) as leftover_bytes from pixels order by sizet*sizec*sizez desc;

 leftover_bytes 
----------------
        4333076
        9770516
        9924116
       12243476
...

comment:13 in reply to: ↑ 12 Changed 10 years ago by mlinkert

Replying to jmoore:

Ok, so that's about 20 MB of overhead, right? How often for a 4+ GB file could the OME-XML metadata be larger than that?


It depends largely upon the configuration of the pixel data (lots of little planes will have more overhead than a few really big planes). I think if you allow 512 bytes per plane plus the size of the XML you should be OK.

Can I get the XML string before hand, say in a tmp file and calculate the size?

You should be able to get the XML string in advance like this:

ServiceFactory factory = new ServiceFactory();
OMEXMLService service = factory.getInstance(OMEXMLService.class);
String xml = service.getOMEXML(retrieve);

comment:14 Changed 10 years ago by jburel

  • Sprint changed from 2011-09-01 (4) to 2011-09-15 (5)

Moved from sprint 2011-09-01 (4)

comment:15 Changed 10 years ago by jmoore <josh@…>

(In [c037e050b579f031a53ebba957877b5a2123591a/ome.git] on branch develop) Set bigtiff for 4GB+ exports (See #6520)

comment:16 Changed 10 years ago by jmoore <josh@…>

(In [a8b80849eb8d21fbdf08eb76b70fef380f3e6ed2/ome.git] on branch develop) Check xml+plane size for big-tiff in export (See #6520)

comment:17 Changed 10 years ago by jmoore <josh@…>

  • Remaining Time set to 0
  • Resolution set to fixed
  • Status changed from new to closed

(In [6e09514503d87f9b680be6f88afc665ec1be514c/ome.git] on branch develop) Merge branch 'feature/export-bugs' into develop (Fix #6520, #6701, #6713)

Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.67161 sec.)

We're Hiring!