Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Task #10841 (accepted)

Opened 7 years ago

Last modified 4 years ago

BUG: Export as jpeg

Reported by: pwalczysko Owned by: jamoore
Priority: major Milestone: Unscheduled
Component: Services Version: n.a.
Keywords: n.a. Cc:
Resources: n.a. Referenced By: n.a.
References: n.a. Remaining Time: n.a.
Sprint: n.a.

Description (last modified by pwalczysko)

When trying to run Batch image export on big images with "jpeg" option selected, you end up with an error, both in Web and Insight (user-4 on Gretzky, using latest develop branch Inisght build for Insight).

Images tested include e.g. leica-scn/mihaela/...., and .svs images.
Found out during the testing of PR https://github.com/ome/scripts/pull/30.
See similar problematics in ticket on this PR, concerning export as one-tiff option.

WARNING:omero.gateway:InternalException on <class 'omero.gateway.OmeroGatewaySafeCallWrapper'> to <86:24:41:e3:-51f141ba:13e7811cad3:-7ba4omero.api.RenderingEngine> renderCompressed((object #0 (::omero::romio::PlaneDef)
{
    slice = 0
    x = 0
    y = 0
    z = 0
    t = 0
    region = <nil>
    stride = 0
}, <ServiceOptsDict: {'omero.session.uuid': '9d092919-574d-4f7c-9504-3f661cee3da8', 'omero.client.uuid': 'e6657582-f2cc-4d9e-85f2-b31acf0cfa4b'}>), {})
Traceback (most recent call last):
  File "/home/omero/slave/workspace/OMERO-merge-develop/src/dist/lib/python/omero/gateway/__init__.py", line 3481, in __call__
    return self.f(*args, **kwargs)
  File "/home/omero/slave/workspace/OMERO-merge-develop/src/dist/lib/python/omero_api_RenderingEngine_ice.py", line 196, in renderCompressed
    return _M_omero.api.RenderingEngine._op_renderCompressed.invoke(self, ((_def, ), _ctx))
InternalException: exception ::omero::InternalException
{
    serverStackTrace = ome.conditions.InternalException:  Wrapped Exception: (java.lang.NegativeArraySizeException):
null
	at omeis.providers.re.RGBIntBuffer.<init>(RGBIntBuffer.java:69)
	at omeis.providers.re.RenderingStrategy.getIntBuffer(RenderingStrategy.java:188)
	at omeis.providers.re.HSBStrategy.renderAsPackedInt(HSBStrategy.java:274)
	at omeis.providers.re.Renderer.renderAsPackedInt(Renderer.java:511)
	at ome.services.RenderingBean.renderAsPackedInt(RenderingBean.java:478)
	at ome.services.RenderingBean.renderCompressed(RenderingBean.java:532)
	at sun.reflect.GeneratedMethodAccessor1366.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:592)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
	at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:116)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
	at $Proxy105.renderCompressed(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor1366.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:592)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
	at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:98)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
	at $Proxy105.renderCompressed(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor1402.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:592)
	at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:179)
	at ome.services.throttling.Callback.run(Callback.java:56)
	at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56)
	at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:150)
	at ome.services.blitz.impl.RenderingEngineI.renderCompressed_async(RenderingEngineI.java:344)
	at sun.reflect.GeneratedMethodAccessor1401.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:592)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
	at omero.cmd.CallContext.invoke(CallContext.java:59)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
	at $Proxy107.renderCompressed_async(Unknown Source)
	at omero.api._RenderingEngineTie.renderCompressed_async(_RenderingEngineTie.java:309)
	at omero.api._RenderingEngineDisp.___renderCompressed(_RenderingEngineDisp.java:583)
	at omero.api._RenderingEngineDisp.__dispatch(_RenderingEngineDisp.java:1700)
	at IceInternal.Incoming.invoke(Incoming.java:159)
	at Ice.ConnectionI.invokeAll(ConnectionI.java:2037)
	at Ice.ConnectionI.message(ConnectionI.java:972)
	at IceInternal.ThreadPool.run(ThreadPool.java:577)
	at IceInternal.ThreadPool.access$100(ThreadPool.java:12)
	at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:971)

    serverExceptionClass = ome.conditions.InternalException
    message =  Wrapped Exception: (java.lang.NegativeArraySizeException):
null
}
Traceback (most recent call last):
  File "./script", line 527, in <module>
    runScript()
  File "./script", line 513, in runScript
    fileAnnotation, message = batchImageExport(conn, scriptParams)
  File "./script", line 384, in batchImageExport
    zRange, tRange, greyscale, imgWidth, projectZ=projectZ, format=format, folder_name=folder_name)
  File "./script", line 233, in savePlanesForImage
    savePlane(image, format, cName, (defaultZ,), projectZ, t, c, gScale, imgWidth, folder_name)
  File "./script", line 127, in savePlane
    plane.save(imgName)
AttributeError: 'NoneType' object has no attribute 'save'

Further, when trying to use the download as jpeg option in Web on a big image, a new window in browser is navigated, saying

Page not found. (404)
The requested object or URL was not found on this server. The link you followed is either outdated, inaccurate, or the server has been instructed not to let you have it.

Change History (14)

comment:1 Changed 7 years ago by pwalczysko

  • Description modified (diff)

comment:2 Changed 7 years ago by pwalczysko

  • Description modified (diff)
  • Summary changed from BUG: Batch image export as jpeg to BUG: Export as jpeg

comment:3 Changed 7 years ago by pwalczysko

  • Priority changed from critical to blocker

comment:4 Changed 7 years ago by jamoore

  • Sprint changed from FS demo 4.1 to FS demo 4.x

Moved from sprint FS demo 4.1

comment:5 Changed 6 years ago by jburel

  • Sprint changed from FS demo 4.x to FS Demo 4.3

comment:6 Changed 6 years ago by mtbcarroll

  • Owner set to mtbcarroll
  • Status changed from new to accepted

I can at least try to pin down what's going on even if I don't know what to do about it!

comment:7 Changed 6 years ago by wmoore

  • Owner changed from mtbcarroll to wmoore

Since this is Batch_Image_Export script, probably my responsibility to work out what's going on. Probably something that's not working with Big Images.

comment:8 Changed 6 years ago by wmoore

It seems like we get the error above when we call re.renderCompressed() where (sizeX * sizeY) > Integer.MAX_VALUE (2147483647).
E.g. Batch_Image_Export.py works with a 4k * 4k Big Image, but not 48979 * 64574 (just tested).

Not sure how to write a test for this since after doing:

$ omero import "test&sizeX=48979&sizeY=64574.fake" 
...
2013-06-03 13:54:49,900 13478      [l.Client-0] INFO   ormats.importer.cli.LoggingImportMonitor  - IMPORT_DONE Imported file: /Users/will/Desktop/OMERO/test&sizeX=48979&sizeY=64574.fake
Imported pixels:
572
Other imported objects:
Fileset:210
Image:572

I then do:

...
>>> i = conn.getObject("Image", 572)
<_ImageWrapper id=572>
>>> i.renderJpeg()
WARNING:omero.gateway:MissingPyramidException

I get the "wrong" exception, so it looks like this won't work in an integration test.

Possibly the Batch_Image_Export.py should limit the size of planes that it works with (returns a message and does nothing, in the same way as for OME-TIFF export).
OR, it could Export Tiles (either 'small' tiles E.g. 256 x 256 or 'big' tiles E.g. 4096 x 4096.

comment:9 Changed 6 years ago by jamoore

This would benefit fake subresolutions. See #11004

Last edited 6 years ago by jamoore (previous) (diff)

comment:10 Changed 6 years ago by mtbcarroll

I wonder if there should be some general code review of where dimensions get multiplied in case these value overflow bugs exist elsewhere.

comment:11 Changed 6 years ago by wmoore

OK, now that the Pyramid has been generated for my fake image, I get a different error from above: ApiUsageException vv InternalException . Not sure the reason for this discrepancy.

>>> i.renderJpeg()
WARNING:omero.gateway:ApiUsageException on <class 'omero.gateway.OmeroGatewaySafeCallWrapper'> to <b24ee084-5287-4561-aa38-03c7fa21f7e5omero.api.RenderingEngine> renderCompressed((object #0 (::omero::romio::PlaneDef)
{
    slice = 0
    x = 0
    y = 0
    z = 0
    t = 0
    region = <nil>
    stride = 0
}, <ServiceOptsDict: {'omero.session.uuid': '9217bcd0-4271-4e5d-b553-6c1934aaccc2', 'omero.client.uuid': 'ffda7148-39ca-462d-b830-bd0dcea37879'}>), {})
Traceback (most recent call last):
  File "/Users/will/Desktop/OMERO/dist/lib/python/omero/gateway/__init__.py", line 3459, in __call__
    return self.f(*args, **kwargs)
  File "/Users/will/Desktop/OMERO/dist/lib/python/omero_api_RenderingEngine_ice.py", line 326, in renderCompressed
    return _M_omero.api.RenderingEngine._op_renderCompressed.invoke(self, ((_def, ), _ctx))
ApiUsageException: exception ::omero::ApiUsageException
{
    serverStackTrace = ome.conditions.ApiUsageException: Converting Long 3162769946 to Integer is an overflow.
	at ome.io.nio.RomioPixelBuffer.safeLongToInteger(RomioPixelBuffer.java:123)
	at ome.io.bioformats.BfPixelBuffer.getPlane(BfPixelBuffer.java:183)
	at ome.io.bioformats.BfPyramidPixelBuffer.getPlane(BfPyramidPixelBuffer.java:755)
	at omeis.providers.re.data.PlaneFactory.createPlane(PlaneFactory.java:230)
	at omeis.providers.re.GreyScaleStrategy.renderAsPackedInt(GreyScaleStrategy.java:153)
	at omeis.providers.re.Renderer.renderAsPackedInt(Renderer.java:511)
	at ome.services.RenderingBean.renderAsPackedInt(RenderingBean.java:478)
	at ome.services.RenderingBean.renderCompressed(RenderingBean.java:532)
	at sun.reflect.GeneratedMethodAccessor1152.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
	at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:116)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
	at $Proxy90.renderCompressed(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor1152.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
	at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:98)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
	at $Proxy90.renderCompressed(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor1164.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:179)
	at ome.services.throttling.Callback.run(Callback.java:56)
	at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56)
	at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:150)
	at ome.services.blitz.impl.RenderingEngineI.renderCompressed_async(RenderingEngineI.java:344)
	at sun.reflect.GeneratedMethodAccessor1163.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
	at omero.cmd.CallContext.invoke(CallContext.java:59)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
	at $Proxy92.renderCompressed_async(Unknown Source)
	at omero.api._RenderingEngineTie.renderCompressed_async(_RenderingEngineTie.java:320)
	at omero.api._RenderingEngineDisp.___renderCompressed(_RenderingEngineDisp.java:666)
	at omero.api._RenderingEngineDisp.__dispatch(_RenderingEngineDisp.java:1784)
	at IceInternal.Incoming.invoke(Incoming.java:159)
	at Ice.ConnectionI.invokeAll(ConnectionI.java:2357)
	at Ice.ConnectionI.dispatch(ConnectionI.java:1208)
	at Ice.ConnectionI.message(ConnectionI.java:1163)
	at IceInternal.ThreadPool.run(ThreadPool.java:302)
	at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
	at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:643)
	at java.lang.Thread.run(Thread.java:680)

    serverExceptionClass = ome.conditions.ApiUsageException
    message = Converting Long 3162769946 to Integer is an overflow.
}

comment:12 Changed 6 years ago by jamoore

@mtbc: likely. Certainly add it to the list. I've done at least one before, and @chris-allan as well. In fact, the error that @will-moore is now getting is due to exactly one such review/refactoring.

comment:13 Changed 6 years ago by wmoore

  • Cc ux@… fs@… removed
  • Component changed from Export to Services
  • Milestone changed from OMERO-5 to OMERO-4.4.9
  • Owner changed from wmoore to jamoore
  • Priority changed from blocker to major
  • Sprint FS Demo 4.3 deleted

Script doesn't try to export any Big images - https://github.com/ome/scripts/pull/31

This fixes the reported issue from the user's perspective.

Re-assigning for any work on the long -> int Exception itself, but lowering the priority.

comment:14 Changed 4 years ago by jamoore

  • Milestone changed from 5.x to Unscheduled
Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.79330 sec.)

We're Hiring!