Task #9124 (closed)
Bug: Import failures for multiple formats: failure to generate thumbnails and to view imported images
| Reported by: | rleigh | Owned by: | jburel |
|---|---|---|---|
| Priority: | major | Milestone: | OMERO-4.4 |
| Component: | Import | Version: | n.a. |
| Keywords: | n.a. | Cc: | sbesson, mlinkert, jburel, cneves, jamoore, cxallan |
| Resources: | n.a. | Referenced By: | n.a. |
| References: | n.a. | Remaining Time: | 0.0d |
| Sprint: | 2012-07-03 (18) |
Description (last modified by jburel)
Formats affected are at least the following:
- ariol
- avi
- CellSens?
- gatan
- gif
- hamamatsu-vms
- jpk
- jpeg
- jpeg2000
- khoros
- li-flim
- nd2
- png
- targa
- tif
- ubm
Files:
- ariol/04CBE9A7-FA1B-4556-9900-6D869E85FFC7.jpg
- avi/*.avi
- CellSens?/Image_01_V76 K2 11_lung.overview.jpg
- gatan/biosamplefullframetif.tif
- gif/*.gif
- incell3000/samples/SBS-bmp/*.bmp
- hamamatsu-vms/openslide/CMU-1/CMU-1-40x - 2010-01-12 13.24.05.vms [CMU-1-40x - 2010-01-12 13.24.05.vms full resolution] [CMU-1-40x - 2010-01-12 13.24.05.vms map] ([CMU-1-40x - 2010-01-12 13.24.05.vms macro] pending pyramid generation?)
- hamamatsu-vms/openslide/CMU-3/CMU-3-40x - 2010-01-12 13.57.09.vms [CMU-3-40x - 2010-01-12 13.57.09.vms full resolution] [CMU-3-40x - 2010-01-12 13.57.09.vms map] ([CMU-3-40x - 2010-01-12 13.57.09.vms macro] pending pyramid generation?)
- jpg/8kx8k.jpg
- jpeg2000/2kx2k.jpf
- jpeg2000/4kx4k.jpf
- jpeg2000/8kx8k.jpf
- khoros/kitten.xv
- li-flim/lifetime-extended.fli
- nd2/N202-11 20x20.nd2
- png/8kx8k.png
- targa/CCM8.TGA
- tif-big-images/8kx8k.tif
- ubm/SG2.PR3 [Expected 32-bit not currently supported by RE]
The above all import "successfully", but unlike all the other images, a thumbnail was not created successfully (error: "Cannot create thumbnail"). The images can also not be opened, failing with the error:
"The image count not be opened.
The image is not a valid image."
Now fixed:
- jpg/4kx4k.jpg
- tif-big-images/4kx4k.tif
- png/4kx4k.png
Attachments (2)
Change History (41)
comment:1 Changed 7 years ago by rleigh
comment:2 Changed 7 years ago by sbesson
Same bug also affects gif/23srp5dksub.gif, gif/dice_1.gif, gif/DMSOHigh_001.gif & gif/raw_a_rg.gif
comment:3 Changed 7 years ago by sbesson
- Cc sbesson added
comment:4 Changed 7 years ago by jburel
Did you get a black thumbnail or the "clock" icon that is used for pyramid?
comment:5 Changed 7 years ago by sbesson
For gifs, I get a series of black thumbnails.
comment:6 Changed 7 years ago by rleigh
Importing InCell? 3000 data:
Images with thumbnails (test_images_good/incell3000/samples/SBS/*.frm): pixel type is uint16, 0 channels (but one channel is displayed in greyscale)
Images without thumbnails (test_images_good/incell3000/samples/SBS-BMP): pixel type is uint8, 1 channel (red)
Images without thumbnails don't display at all, with the same "The image could not be opened. The image is not a valid image" error.
It looks like the .frm files are missing channel information, and the .bmp files are not being imported correctly at all.
comment:7 Changed 7 years ago by rleigh
- Description modified (diff)
- Summary changed from Bug: Import failure for big jpeg and jpeg2000 images to Bug: Import failures for multiple formats: failure to generate thumbnails and to view imported images
comment:8 Changed 7 years ago by rleigh
- Description modified (diff)
comment:9 Changed 7 years ago by jburel
I can view the .bmp images using directly B-F in ImageJ.
comment:10 Changed 7 years ago by rleigh
- Description modified (diff)
comment:11 Changed 7 years ago by jburel
- Cc mlinkert-x added
- Component changed from Insight to Import
The statistics i.e. min max generated by min max calculator are not saved and linked to the channels.
We need to check what is happening for those formats.
comment:12 Changed 7 years ago by jburel
- Remaining Time set to 0.35
- Sprint set to 2012-07-03 (18)
- Status changed from new to accepted
TODO:
- Adjust rendering engine so the image is displayed even if the stats are not saved (not required)
- Review why the stats are not saved.
comment:13 Changed 7 years ago by jburel
- Cc jburel added
- Owner changed from jburel to mlinkert-x
Fixed the display problem
Passing to Melissa to check the calculation of the stats in B-F.
comment:14 Changed 7 years ago by rleigh
The recent change fixes the thumbnails and image viewing for the following formats:
avi, gif, jpk, khoros and targa
There is an issue with the following big images:
jpeg2000 (2k/4k/8k), jpg (4k/8k) and tif-big-images (4k/8k)
These have no thumbnails and are not viewable as before.
comment:15 Changed 7 years ago by mlinkert
- Owner changed from mlinkert-x to jburel
Stats population should be fixed with: https://github.com/openmicroscopy/bioformats/commit/6d8d2f7621a99c04afb31959392d8e7a27e5a501
comment:16 Changed 7 years ago by jburel
- Cc cneves-x added
comment:17 Changed 7 years ago by jburel
- Cc jmoore added
- Owner changed from jburel to jmoore
for the 4x4 the error is actually different
syncProcessor$Runner.run(AsyncProcessor.java:91)
at java.lang.Thread.run(Thread.java:680)
Caused by: org.openmicroscopy.shoola.env.data.DSAccessException: Cannot access data.
Cannot set the rendering defaults.
at org.openmicroscopy.shoola.env.data.OMEROGateway.handleException(OMEROGateway.java:878)
at org.openmicroscopy.shoola.env.data.OMEROGateway.needDefault(OMEROGateway.java:1897)
at org.openmicroscopy.shoola.env.data.OMEROGateway.createRenderingEngine(OMEROGateway.java:3229)
... 9 more
Caused by: omero.ApiUsageException
serverStackTrace = "ome.conditions.ApiUsageException: Not intended for copying!
at ome.model.internal.Permissions.<init>(Permissions.java:156)
at ome.security.basic.CurrentDetails.createDetails(CurrentDetails.java:365)
at ome.security.basic.OmeroInterceptor.newTransientDetails(OmeroInterceptor.java:551)
at ome.security.basic.OmeroInterceptor.onSave(OmeroInterceptor.java:157)
at org.hibernate.event.def.AbstractSaveEventListener.substituteValuesIfNecessary(AbstractSaveEventListener.java:414)
at org.hibernate.event.def.AbstractSaveEventListener.performSaveOrReplicate(AbstractSaveEventListener.java:293)
at org.hibernate.event.def.AbstractSaveEventListener.performSave(AbstractSaveEventListener.java:204)
at org.hibernate.event.def.AbstractSaveEventListener.saveWithGeneratedId(AbstractSaveEventListener.java:144)
at org.hibernate.event.def.DefaultMergeEventListener.saveTransientEntity(DefaultMergeEventListener.java:415)
at org.hibernate.event.def.DefaultMergeEventListener.mergeTransientEntity(DefaultMergeEventListener.java:341)
at org.hibernate.event.def.DefaultMergeEventListener.entityIsTransient(DefaultMergeEventListener.java:303)
at org.springframework.orm.hibernate3.support.IdTransferringMergeEventListener.entityIsTransient(IdTransferringMergeEventListener.java:59)
at ome.security.basic.MergeEventListener.entityIsTransient(MergeEventListener.java:161)
at org.hibernate.event.def.DefaultMergeEventListener.onMerge(DefaultMergeEventListener.java:258)
Passing it to Josh for clarification
comment:18 Changed 7 years ago by rleigh
- Description modified (diff)
comment:19 Changed 7 years ago by rleigh
- Description modified (diff)
comment:20 Changed 7 years ago by jmoore
Can I get the entire stack trace please?
comment:21 Changed 7 years ago by jburel
- retested with new server, 4kx4k works fine
- 8kx8k.jpg
2012-06-26 09:52:03,287 ERROR [ ome.io.nio.PixelsService] (l.Server-3) Error instantiating pixel buffer: /OMERO/Pixels/Dir-003/3052_pyramid ome.conditions.ResourceError: FormatException: /OMERO/Pixels/Dir-003/3052_pyramid Invalid TIFF file at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:72) at ome.io.bioformats.BfPixelBuffer.isLittleEndian(BfPixelBuffer.java:91) at ome.io.bioformats.BfPyramidPixelBuffer.initializeReader(BfPyramidPixelBuffer.java:192) at ome.io.bioformats.BfPyramidPixelBuffer.<init>(BfPyramidPixelBuffer.java:171) at ome.io.bioformats.BfPyramidPixelBuffer.<init>(BfPyramidPixelBuffer.java:141) at ome.io.nio.PixelsService.createPyramidPixelBuffer(PixelsService.java:635) at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:410) at ome.services.RawPixelsBean.setPixelsId(RawPixelsBean.java:278) at sun.reflect.GeneratedMethodAccessor743.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at ome.security.basic.EventHandler.invoke(EventHandler.java:154) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at ome.tools.hibernate.SessionHandler.doStateful(SessionHandler.java:182) at ome.tools.hibernate.SessionHandler.invoke(SessionHandler.java:166) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:241) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:116) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy89.setPixelsId(Unknown Source) at sun.reflect.GeneratedMethodAccessor743.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:98) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy89.setPixelsId(Unknown Source) at sun.reflect.GeneratedMethodAccessor768.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:179) at ome.services.throttling.Callback.run(Callback.java:56) at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56) at ome.services.blitz.impl.AbstractAmdServant.callInvokerOnRawArgs(AbstractAmdServant.java:137) at ome.services.blitz.impl.RawPixelsStoreI.setPixelsId_async(RawPixelsStoreI.java:193) at sun.reflect.GeneratedMethodAccessor767.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at omero.cmd.CallContext.invoke(CallContext.java:59) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy90.setPixelsId_async(Unknown Source) at omero.api._RawPixelsStoreTie.setPixelsId_async(_RawPixelsStoreTie.java:274) at omero.api._RawPixelsStoreDisp.___setPixelsId(_RawPixelsStoreDisp.java:375) at omero.api._RawPixelsStoreDisp.__dispatch(_RawPixelsStoreDisp.java:1297) at IceInternal.Incoming.invoke(Incoming.java:159) at Ice.ConnectionI.invokeAll(ConnectionI.java:2037) at Ice.ConnectionI.message(ConnectionI.java:972) at IceInternal.ThreadPool.run(ThreadPool.java:577) at IceInternal.ThreadPool.access$100(ThreadPool.java:12) at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:971)
comment:22 Changed 7 years ago by jmoore
- Cc cxallan added
- Owner changed from jmoore to jburel
Sorry, but that's something completely different again. Are there multiple issues causing problems here?
comment:23 Changed 7 years ago by jburel
We are trying to narrow down the problems. Depending on the format or size, we have found different issues, e.g. statistics not being calculated, RE adjustment etc.
The problem reported yesterday previously happened for the 4kx4k and did not happen today. We will try again to see if it occurs.
comment:24 Changed 7 years ago by jburel
- Owner changed from jburel to mlinkert-x
Passing to melissa, see B-F error while importing the 8kx8k.jpg
comment:25 Changed 7 years ago by rleigh
- Description modified (diff)
comment:26 Changed 7 years ago by rleigh
- Description modified (diff)
comment:27 Changed 7 years ago by jburel
- Description modified (diff)
comment:28 Changed 7 years ago by jburel
- Description modified (diff)
Same error for png/8kx8k.png jpeg2000/2kx2k, 4kx4k, 8kx8k.jpf i.e.
Invalid TIFF file at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:72) at ome.io.bioformats.BfPixelBuffer.isLittleEndian(BfPixelBuffer.java:91) at ome.io.bioformats.BfPyramidPixelBuffer.initializeReader(BfPyramidPixelBuffer.java:192) at ome.io.bioformats.BfPyramidPixelBuffer.<init>(BfPyramidPixelBuffer.java:171) at ome.io.bioformats.BfPyramidPixelBuffer.<init>(BfPyramidPixelBuffer.java:141) at ome.io.nio.PixelsService.createPyramidPixelBuffer(PixelsService.java:635) at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:410) at ome.services.RawPixelsBean.setPixelsId(RawPixelsBean.java:278) at sun.reflect.GeneratedMethodAccessor777.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at ome.security.basic.EventHandler.invoke(EventHandler.java:154) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at ome.tools.hibernate.SessionHandler.doStateful(SessionHandler.java:182) at ome.tools.hibernate.SessionHandler.invoke(SessionHandler.java:166) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:241) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:116) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy91.setPixelsId(Unknown Source) at sun.reflect.GeneratedMethodAccessor777.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at ome.security.basic.BasicSecurityWiring.invoke(BasicSecurityWiring.java:98) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at ome.services.blitz.fire.AopContextInitializer.invoke(AopContextInitializer.java:43) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at $Proxy91.setPixelsId(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at ome.services.blitz.util.IceMethodInvoker.invoke(IceMethodInvoker.java:179) at ome.services.throttling.Callback.run(Callback.java:56) at ome.services.throttling.InThreadThrottlingStrategy.callInvokerOnRawArgs(InThreadThrottlingStrategy.java:56)
comment:29 Changed 7 years ago by rleigh
Still failing:
- jpeg2000/2kx2k.jpf
- jpeg2000/4kx4k.jpf
- jpeg2000/8kx8k.jpf
- li-flim/lifetime-extended.fli
Now OK with recent changes:
- ariol/04CBE9A7-FA1B-4556-9900-6D869E85FFC7.jpg
- avi/*.avi
- gatan/biosamplefullframetif.tif
- gif/*.gif
- incell3000/samples/SBS-bmp/*.bmp
- jpg/8kx8k.jpg
- khoros/kitten.xv
- nd2/N202-11 20x20.nd2
- png/8kx8k.png
- targa/CCM8.TGA
- tif-big-images/8kx8k.tif
To test
- CellSens/Image??_01_V76 K2 11_lung.overview.jpg
Not testing:
- hamamatsu-vms/openslide/CMU-1/CMU-1-40x - 2010-01-12 13.24.05.vms [CMU-1-40x - 2010-01-12 13.24.05.vms full resolution] [CMU-1-40x - 2010-01-12 13.24.05.vms map] ([CMU-1-40x - 2010-01-12 13.24.05.vms macro] pending pyramid generation?)
- hamamatsu-vms/openslide/CMU-3/CMU-3-40x - 2010-01-12 13.57.09.vms [CMU-3-40x - 2010-01-12 13.57.09.vms full resolution] [CMU-3-40x - 2010-01-12 13.57.09.vms map] ([CMU-3-40x - 2010-01-12 13.57.09.vms macro] pending pyramid generation?)
- ubm/SG2.PR3 [Expected 32-bit not currently supported by RE]
comment:30 Changed 7 years ago by jburel
- li-flim/lifetime-extended.fli This is due to a current limitation of RE. Not a B-F/import problem
i.e.
java.lang.ArrayIndexOutOfBoundsException: 65536 at omeis.providers.re.quantum.Quantization_8_16_bit.buildLUT(Quantization_8_16_bit.java:255) at omeis.providers.re.quantum.Quantization_8_16_bit.onWindowChange(Quantization_8_16_bit.java:283) at omeis.providers.re.quantum.QuantumStrategy.setWindow(QuantumStrategy.java:323) at omeis.providers.re.QuantumManager.initStrategies(QuantumManager.java:118) at omeis.providers.re.Renderer.<init>(Renderer.java:352) at ome.services.ThumbnailBean.load(ThumbnailBean.java:344) at ome.services.ThumbnailBean.errorIfInvalidState(ThumbnailBean.java:695)
comment:31 Changed 7 years ago by rleigh
JPEG2000 issues are fixed by a one line patch to JPEG2000Reader (attached)
comment:32 Changed 7 years ago by rleigh
8kx8k PNG is broken again today.
comment:33 Changed 7 years ago by rleigh
JPEG 2000 issues fixed with bioformat commit d1ee10e
All other large image issues fixed by increasing the size of pixeldata to 1GiB
comment:34 Changed 7 years ago by rleigh
- Remaining Time changed from 0.35 to 0
- Resolution set to fixed
- Status changed from accepted to closed
comment:35 Changed 7 years ago by jburel
- Resolution fixed deleted
- Status changed from closed to reopened
comment:36 Changed 7 years ago by jburel
- Owner changed from mlinkert-x to jburel
- Status changed from reopened to accepted
comment:37 Changed 7 years ago by jburel
- Resolution set to fixed
- Status changed from accepted to closed
comment:38 Changed 7 years ago by jburel <j.burel@…>
(In [b6fb2c5bc24c643b9a182a6d9bed388fb37bbcc2/ome.git] on branch develop) Modify RE if no stats created so image can be viewed (see #9124)
comment:39 Changed 7 years ago by jburel <j.burel@…>
(In [2286df149d9c3a9e008df2ea17db7d946267e17f/ome.git] on branch develop) Handle cases of image w/o stats (see #9124)
The same bug also appears to affect
jpk/Virus.jpk [Series 1 only]
khoros/kitten.xv