Task #10841 (accepted)
Opened 11 years ago
Last modified 8 years ago
BUG: Export as jpeg
Reported by: | pwalczysko | Owned by: | jamoore |
---|---|---|---|
Priority: | major | Milestone: | Unscheduled |
Component: | Services | Version: | n.a. |
Keywords: | n.a. | Cc: | |
Resources: | n.a. | Referenced By: | n.a. |
References: | n.a. | Remaining Time: | n.a. |
Sprint: | n.a. |
Description (last modified by pwalczysko)
When trying to run Batch image export on big images with "jpeg" option selected, you end up with an error, both in Web and Insight (user-4 on Gretzky, using latest develop branch Inisght build for Insight).
Images tested include e.g. leica-scn/mihaela/...., and .svs images.
Found out during the testing of PR https://github.com/ome/scripts/pull/30.
See similar problematics in ticket on this PR, concerning export as one-tiff option.
WARNING:omero.gateway:InternalException on <class 'omero.gateway.OmeroGatewaySafeCallWrapper'> to <86:24:41:e3:-51f141ba:13e7811cad3:-7ba4omero.api.RenderingEngine> renderCompressed((object #0 (::omero::romio::PlaneDef) { slice = 0 x = 0 y = 0 z = 0 t = 0 region = <nil> stride = 0 }, <ServiceOptsDict: {'omero.session.uuid': '9d092919-574d-4f7c-9504-3f661cee3da8', 'omero.client.uuid': 'e6657582-f2cc-4d9e-85f2-b31acf0cfa4b'}>), {}) Traceback (most recent call last): File "/home/omero/slave/workspace/OMERO-merge-develop/src/dist/lib/python/omero/gateway/__init__.py", line 3481, in __call__ return self.f(*args, **kwargs) File "/home/omero/slave/workspace/OMERO-merge-develop/src/dist/lib/python/omero_api_RenderingEngine_ice.py", line 196, in renderCompressed return _M_omero.api.RenderingEngine._op_renderCompressed.invoke(self, ((_def, ), _ctx)) InternalException: exception ::omero::InternalException { serverStackTrace = ome.conditions.InternalException: Wrapped Exception: (java.lang.NegativeArraySizeException): null at omeis.providers.re.RGBIntBuffer.<init>(RGBIntBuffer.java:69) at omeis.providers.re.RenderingStrategy.getIntBuffer(RenderingStrategy.java:188) at omeis.providers.re.HSBStrategy.renderAsPackedInt(HSBStrategy.java:274) at omeis.providers.re.Renderer.renderAsPackedInt(Renderer.java:511) at ome.services.RenderingBean.renderAsPackedInt(RenderingBean.java:478) at ome.services.RenderingBean.renderCompressed(RenderingBean.java:532) at sun.reflect.GeneratedMethodAccessor1366.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:592) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:116) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy105.renderCompressed(Unknown Source) at sun.reflect.GeneratedMethodAccessor1366.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:592) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:98) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy105.renderCompressed(Unknown Source) at sun.reflect.GeneratedMethodAccessor1402.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:592) at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:179) at ome.services.throttling.Callback.run(Callback.java:56) at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56) at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:150) at ome.services.blitz.impl.RenderingEngineI.renderCompressed_async(RenderingEngineI.java:344) at sun.reflect.GeneratedMethodAccessor1401.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:592) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at omero.cmd.CallContext.invoke(CallContext.java:59) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy107.renderCompressed_async(Unknown Source) at omero.api._RenderingEngineTie.renderCompressed_async(_RenderingEngineTie.java:309) at omero.api._RenderingEngineDisp.___renderCompressed(_RenderingEngineDisp.java:583) at omero.api._RenderingEngineDisp.__dispatch(_RenderingEngineDisp.java:1700) at IceInternal.Incoming.invoke(Incoming.java:159) at Ice.ConnectionI.invokeAll(ConnectionI.java:2037) at Ice.ConnectionI.message(ConnectionI.java:972) at IceInternal.ThreadPool.run(ThreadPool.java:577) at IceInternal.ThreadPool.access$100(ThreadPool.java:12) at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:971) serverExceptionClass = ome.conditions.InternalException message = Wrapped Exception: (java.lang.NegativeArraySizeException): null } Traceback (most recent call last): File "./script", line 527, in <module> runScript() File "./script", line 513, in runScript fileAnnotation, message = batchImageExport(conn, scriptParams) File "./script", line 384, in batchImageExport zRange, tRange, greyscale, imgWidth, projectZ=projectZ, format=format, folder_name=folder_name) File "./script", line 233, in savePlanesForImage savePlane(image, format, cName, (defaultZ,), projectZ, t, c, gScale, imgWidth, folder_name) File "./script", line 127, in savePlane plane.save(imgName) AttributeError: 'NoneType' object has no attribute 'save'
Further, when trying to use the download as jpeg option in Web on a big image, a new window in browser is navigated, saying
Page not found. (404) The requested object or URL was not found on this server. The link you followed is either outdated, inaccurate, or the server has been instructed not to let you have it.
Change History (14)
comment:1 Changed 11 years ago by pwalczysko
- Description modified (diff)
comment:2 Changed 11 years ago by pwalczysko
- Description modified (diff)
- Summary changed from BUG: Batch image export as jpeg to BUG: Export as jpeg
comment:3 Changed 11 years ago by pwalczysko
- Priority changed from critical to blocker
comment:4 Changed 11 years ago by jamoore
- Sprint changed from FS demo 4.1 to FS demo 4.x
comment:5 Changed 11 years ago by jburel
- Sprint changed from FS demo 4.x to FS Demo 4.3
comment:6 Changed 11 years ago by mtbcarroll
- Owner set to mtbcarroll
- Status changed from new to accepted
I can at least try to pin down what's going on even if I don't know what to do about it!
comment:7 Changed 11 years ago by wmoore
- Owner changed from mtbcarroll to wmoore
Since this is Batch_Image_Export script, probably my responsibility to work out what's going on. Probably something that's not working with Big Images.
comment:8 Changed 11 years ago by wmoore
It seems like we get the error above when we call re.renderCompressed() where (sizeX * sizeY) > Integer.MAX_VALUE (2147483647).
E.g. Batch_Image_Export.py works with a 4k * 4k Big Image, but not 48979 * 64574 (just tested).
Not sure how to write a test for this since after doing:
$ omero import "test&sizeX=48979&sizeY=64574.fake" ... 2013-06-03 13:54:49,900 13478 [l.Client-0] INFO ormats.importer.cli.LoggingImportMonitor - IMPORT_DONE Imported file: /Users/will/Desktop/OMERO/test&sizeX=48979&sizeY=64574.fake Imported pixels: 572 Other imported objects: Fileset:210 Image:572
I then do:
... >>> i = conn.getObject("Image", 572) <_ImageWrapper id=572> >>> i.renderJpeg() WARNING:omero.gateway:MissingPyramidException
I get the "wrong" exception, so it looks like this won't work in an integration test.
Possibly the Batch_Image_Export.py should limit the size of planes that it works with (returns a message and does nothing, in the same way as for OME-TIFF export).
OR, it could Export Tiles (either 'small' tiles E.g. 256 x 256 or 'big' tiles E.g. 4096 x 4096.
comment:9 Changed 11 years ago by jamoore
This would benefit fake subresolutions. See #11004
comment:10 Changed 11 years ago by mtbcarroll
I wonder if there should be some general code review of where dimensions get multiplied in case these value overflow bugs exist elsewhere.
comment:11 Changed 11 years ago by wmoore
OK, now that the Pyramid has been generated for my fake image, I get a different error from above: ApiUsageException vv InternalException . Not sure the reason for this discrepancy.
>>> i.renderJpeg() WARNING:omero.gateway:ApiUsageException on <class 'omero.gateway.OmeroGatewaySafeCallWrapper'> to <b24ee084-5287-4561-aa38-03c7fa21f7e5omero.api.RenderingEngine> renderCompressed((object #0 (::omero::romio::PlaneDef) { slice = 0 x = 0 y = 0 z = 0 t = 0 region = <nil> stride = 0 }, <ServiceOptsDict: {'omero.session.uuid': '9217bcd0-4271-4e5d-b553-6c1934aaccc2', 'omero.client.uuid': 'ffda7148-39ca-462d-b830-bd0dcea37879'}>), {}) Traceback (most recent call last): File "/Users/will/Desktop/OMERO/dist/lib/python/omero/gateway/__init__.py", line 3459, in __call__ return self.f(*args, **kwargs) File "/Users/will/Desktop/OMERO/dist/lib/python/omero_api_RenderingEngine_ice.py", line 326, in renderCompressed return _M_omero.api.RenderingEngine._op_renderCompressed.invoke(self, ((_def, ), _ctx)) ApiUsageException: exception ::omero::ApiUsageException { serverStackTrace = ome.conditions.ApiUsageException: Converting Long 3162769946 to Integer is an overflow. at ome.io.nio.RomioPixelBuffer.safeLongToInteger(RomioPixelBuffer.java:123) at ome.io.bioformats.BfPixelBuffer.getPlane(BfPixelBuffer.java:183) at ome.io.bioformats.BfPyramidPixelBuffer.getPlane(BfPyramidPixelBuffer.java:755) at omeis.providers.re.data.PlaneFactory.createPlane(PlaneFactory.java:230) at omeis.providers.re.GreyScaleStrategy.renderAsPackedInt(GreyScaleStrategy.java:153) at omeis.providers.re.Renderer.renderAsPackedInt(Renderer.java:511) at ome.services.RenderingBean.renderAsPackedInt(RenderingBean.java:478) at ome.services.RenderingBean.renderCompressed(RenderingBean.java:532) at sun.reflect.GeneratedMethodAccessor1152.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:116) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy90.renderCompressed(Unknown Source) at sun.reflect.GeneratedMethodAccessor1152.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:98) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy90.renderCompressed(Unknown Source) at sun.reflect.GeneratedMethodAccessor1164.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:179) at ome.services.throttling.Callback.run(Callback.java:56) at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56) at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:150) at ome.services.blitz.impl.RenderingEngineI.renderCompressed_async(RenderingEngineI.java:344) at sun.reflect.GeneratedMethodAccessor1163.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at omero.cmd.CallContext.invoke(CallContext.java:59) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy92.renderCompressed_async(Unknown Source) at omero.api._RenderingEngineTie.renderCompressed_async(_RenderingEngineTie.java:320) at omero.api._RenderingEngineDisp.___renderCompressed(_RenderingEngineDisp.java:666) at omero.api._RenderingEngineDisp.__dispatch(_RenderingEngineDisp.java:1784) at IceInternal.Incoming.invoke(Incoming.java:159) at Ice.ConnectionI.invokeAll(ConnectionI.java:2357) at Ice.ConnectionI.dispatch(ConnectionI.java:1208) at Ice.ConnectionI.message(ConnectionI.java:1163) at IceInternal.ThreadPool.run(ThreadPool.java:302) at IceInternal.ThreadPool.access$300(ThreadPool.java:12) at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:643) at java.lang.Thread.run(Thread.java:680) serverExceptionClass = ome.conditions.ApiUsageException message = Converting Long 3162769946 to Integer is an overflow. }
comment:12 Changed 11 years ago by jamoore
@mtbc: likely. Certainly add it to the list. I've done at least one before, and @chris-allan as well. In fact, the error that @will-moore is now getting is due to exactly one such review/refactoring.
comment:13 Changed 11 years ago by wmoore
- Cc ux@… fs@… removed
- Component changed from Export to Services
- Milestone changed from OMERO-5 to OMERO-4.4.9
- Owner changed from wmoore to jamoore
- Priority changed from blocker to major
- Sprint FS Demo 4.3 deleted
Script doesn't try to export any Big images - https://github.com/ome/scripts/pull/31
This fixes the reported issue from the user's perspective.
Re-assigning for any work on the long -> int Exception itself, but lowering the priority.
comment:14 Changed 8 years ago by jamoore
- Milestone changed from 5.x to Unscheduled
Moved from sprint FS demo 4.1