Task #12485 (closed)
Opened 10 years ago
Closed 10 years ago
Bug: reduce the number of companion files reported by VolocityReader
| Reported by: | mlinkert | Owned by: | mlinkert |
|---|---|---|---|
| Priority: | critical | Milestone: | 5.1.0-m3 |
| Component: | Bio-Formats | Version: | 5.0.2 |
| Keywords: | n.a. | Cc: | johnbillheimer@…, jamoore, mtbcarroll |
| Resources: | n.a. | Referenced By: | n.a. |
| References: | n.a. | Remaining Time: | n.a. |
| Sprint: | n.a. |
Description
See QA 9407. Large Volocity datasets throw Ice MemoryLimitExceptions? on import, as the "companion" binary data files can be very large. Adjusting getSeriesUsedFiles to identify most/all files as pixels files should fix this.
Change History (7)
comment:1 Changed 10 years ago by mlinkert
- Cc jamoore added
comment:2 Changed 10 years ago by jamoore
- Cc mtbcarroll added
- Priority changed from major to critical
Looking at that stacktrace, this is a matter of the number of files as opposed to their size, right? (Do you know how many?) If so, I'd think:
- short-term solution (5.0) : save the annotations links in batches
- middle-term solution (5.1): evaluate not creating these annotations. With FS, it's really unnecessary.
comment:3 Changed 10 years ago by mlinkert
Should be 994 files, ranging from 512 bytes to 61MB - guess I assumed size was the issue based upon the size of the largest files.
comment:4 Changed 10 years ago by jamoore
https://github.com/openmicroscopy/openmicroscopy/pull/2853 has been merged which should correct the most immediate error for 5.0.3. The Volocity fix itself and the middle-term removal of these companion files are still outstanding.
comment:5 Changed 10 years ago by jmoore <josh@…>
(In [e1589b08186a99a882a7a6145aeb09b472f5d39c/ome.git] on branch develop) Add batching to attachCompanionFilesToImage (See #12485)
comment:6 Changed 10 years ago by Josh Moore <josh@…>
(In [4652484245472e6d31e497ee320d14c318e86dd9/ome.git] on branch develop) Merge pull request #2867 from joshmoore/rebased/develop/12485-companion-loop
Add batching to attachCompanionFilesToImage (See #12485) (rebased onto develop)
comment:7 Changed 10 years ago by mlinkert
- Resolution set to fixed
- Status changed from new to closed
PR opened to address companion file classification: https://github.com/openmicroscopy/bioformats/pull/1267
#12519 opened to address the broader issue of companion attachment creation.
Full stack trace on import with 5.0.3-rc1:
2014-07-22 11:10:12,041 1002091 [l.Client-4] ERROR ome.formats.importer.cli.ErrorHandler - INTERNAL_EXCEPTION: /home/melissa/data/volocity/qa-9407/Imaging.mvd2 java.lang.RuntimeException: Failure response on import! Category: ::omero::grid::ImportRequest Name: import-request-failure Parameters: {message=, stacktrace=Ice.MemoryLimitException reason = "requested 67108865 bytes, maximum allowed is 67108864 bytes (see Ice.MessageSizeMax)" at IceInternal.Ex.throwMemoryLimitException(Ex.java:23) at IceInternal.BasicStream.expand(BasicStream.java:2049) at IceInternal.BasicStream.writeInt(BasicStream.java:888) at IceInternal.BasicStream.writeObject(BasicStream.java:1362) at omero.model.OriginalFile.__write(OriginalFile.java:1477) at IceInternal.BasicStream.writeInstance(BasicStream.java:1721) at IceInternal.BasicStream.writePendingObjects(BasicStream.java:1639) at omero.api._IUpdateDelM.saveCollection(_IUpdateDelM.java:348) at omero.api.IUpdatePrxHelper.saveCollection(IUpdatePrxHelper.java:994) at omero.api.IUpdatePrxHelper.saveCollection(IUpdatePrxHelper.java:967) at ome.formats.OMEROMetadataStoreClient.attachCompanionFilesToImage(OMEROMetadataStoreClient.java:794) at ome.services.blitz.repo.ManagedImportRequestI.importMetadata(ManagedImportRequestI.java:514) at ome.services.blitz.repo.ManagedImportRequestI.step(ManagedImportRequestI.java:379) at omero.cmd.HandleI.steps(HandleI.java:436) at omero.cmd.HandleI$1.doWork(HandleI.java:365) at omero.cmd.HandleI$1.doWork(HandleI.java:361) at sun.reflect.GeneratedMethodAccessor239.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:576) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at ome.security.basic.EventHandler.invoke(EventHandler.java:154) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:241) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:116) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at com.sun.proxy.$Proxy69.doWork(Unknown Source) at ome.services.util.Executor$Impl.execute(Executor.java:457) at omero.cmd.HandleI.run(HandleI.java:359) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at ome.services.util.Executor$Impl$1.call(Executor.java:498) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) } at ome.formats.importer.ImportLibrary$ImportCallback.onFinished(ImportLibrary.java:627) at omero.cmd.CmdCallbackI.finished(CmdCallbackI.java:301) at omero.cmd._CmdCallbackDisp.___finished(_CmdCallbackDisp.java:130) at omero.cmd._CmdCallbackDisp.__dispatch(_CmdCallbackDisp.java:157) at IceInternal.Incoming.invoke(Incoming.java:159) at Ice.ConnectionI.invokeAll(ConnectionI.java:2357) at Ice.ConnectionI.dispatch(ConnectionI.java:1208) at Ice.ConnectionI.message(ConnectionI.java:1163) at IceInternal.ThreadPool.run(ThreadPool.java:302) at IceInternal.ThreadPool.access$300(ThreadPool.java:12) at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:643) at java.lang.Thread.run(Thread.java:744) java.lang.RuntimeException: Failure response on import! Category: ::omero::grid::ImportRequest Name: import-request-failure Parameters: {message=, stacktrace=Ice.MemoryLimitException reason = "requested 67108865 bytes, maximum allowed is 67108864 bytes (see Ice.MessageSizeMax)" at IceInternal.Ex.throwMemoryLimitException(Ex.java:23) at IceInternal.BasicStream.expand(BasicStream.java:2049) at IceInternal.BasicStream.writeInt(BasicStream.java:888) at IceInternal.BasicStream.writeObject(BasicStream.java:1362) at omero.model.OriginalFile.__write(OriginalFile.java:1477) at IceInternal.BasicStream.writeInstance(BasicStream.java:1721) at IceInternal.BasicStream.writePendingObjects(BasicStream.java:1639) at omero.api._IUpdateDelM.saveCollection(_IUpdateDelM.java:348) at omero.api.IUpdatePrxHelper.saveCollection(IUpdatePrxHelper.java:994) at omero.api.IUpdatePrxHelper.saveCollection(IUpdatePrxHelper.java:967) at ome.formats.OMEROMetadataStoreClient.attachCompanionFilesToImage(OMEROMetadataStoreClient.java:794) at ome.services.blitz.repo.ManagedImportRequestI.importMetadata(ManagedImportRequestI.java:514) at ome.services.blitz.repo.ManagedImportRequestI.step(ManagedImportRequestI.java:379) at omero.cmd.HandleI.steps(HandleI.java:436) at omero.cmd.HandleI$1.doWork(HandleI.java:365) at omero.cmd.HandleI$1.doWork(HandleI.java:361) at sun.reflect.GeneratedMethodAccessor239.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:576) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at ome.security.basic.EventHandler.invoke(EventHandler.java:154) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:241) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:116) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at com.sun.proxy.$Proxy69.doWork(Unknown Source) at ome.services.util.Executor$Impl.execute(Executor.java:457) at omero.cmd.HandleI.run(HandleI.java:359) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at ome.services.util.Executor$Impl$1.call(Executor.java:498) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) } at ome.formats.importer.ImportLibrary$ImportCallback.onFinished(ImportLibrary.java:627) ~[blitz.jar:na] at omero.cmd.CmdCallbackI.finished(CmdCallbackI.java:301) [blitz.jar:na] at omero.cmd._CmdCallbackDisp.___finished(_CmdCallbackDisp.java:130) [blitz.jar:na] at omero.cmd._CmdCallbackDisp.__dispatch(_CmdCallbackDisp.java:157) [blitz.jar:na] at IceInternal.Incoming.invoke(Incoming.java:159) [ice.jar:na] at Ice.ConnectionI.invokeAll(ConnectionI.java:2357) [ice.jar:na] at Ice.ConnectionI.dispatch(ConnectionI.java:1208) [ice.jar:na] at Ice.ConnectionI.message(ConnectionI.java:1163) [ice.jar:na] at IceInternal.ThreadPool.run(ThreadPool.java:302) [ice.jar:na] at IceInternal.ThreadPool.access$300(ThreadPool.java:12) [ice.jar:na] at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:643) [ice.jar:na] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]