Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Task #10893 (closed)

Opened 6 years ago

Closed 6 years ago

Testing data FS import

Reported by: jburel Owned by:
Priority: major Milestone: 5.0.0-beta1
Component: General Version: n.a.
Keywords: n.a. Cc: ux@…, fs@…, ajppatterson
Resources: n.a. Referenced By: n.a.
References: n.a. Remaining Time: n.a.
Sprint: FS demo 4.2

Description

We have 3 main areas we want to test and create failures, under each section, options we can put in place.

  • Scanning using B-F (pre-upload for now).
    • For that we can use Mike.dv on test_images_bad on squig
  • Checksum
    • Use an alternative server returning invalid checksum
  • Processing
    • Generate ome.tiff files with incorrect metadata.
    • Depending on which processing steps we want to test, we might have to use an alternative server e.g. Thumbnails generation steps.

Change History (12)

comment:1 Changed 6 years ago by jburel

Assigning to blazej first for checksum section

comment:2 Changed 6 years ago by bpindelski

  • Status changed from new to accepted

comment:3 Changed 6 years ago by bpindelski

Jean-Marie, Josh: Unless there is a better way to do it, I was planning on using ome-adhoc-test. I'd deploy the server there manually (from OMERO-merge-develop-ice34) and add a class to the classpath that extends the checksum provider. The child class would return a bogus value and make every import fail.

comment:4 Changed 6 years ago by bpindelski

A quick solution is on ​https://github.com/bpindelski/openmicroscopy/commits/chksm-failure. This will get rebased against develop, once PR 1183 gets merged. Then the server will be deployed on ome-adhoc-test, with lib/server/blizt.jar and lib/client/blitz.jar substituted with versions built locally and failing the checksums. Assigning back to Jean-Marie.

comment:5 Changed 6 years ago by bpindelski

  • Owner changed from bpindelski to jburel

comment:6 Changed 6 years ago by jburel

  • Owner changed from jburel to ajpatterson

Passing it to Andrew. Could you generate some xml/ome-tiff files that will fail during the processing steps?

comment:7 Changed 6 years ago by ajpatterson

Tested all files in test_images_bad and test_images_broken

One failure when selecting folder:

File: test_images_broken/volocity/3 Colour.acff

The file uploads and metadata extracts, then thrown an exception.

java.lang.RuntimeException: Failure response on import!
Category: ::omero::grid::ImportRequest
Name: import-file-exception
Parameters: {message=Attempting to read beyond end of file., filename=user-4_5/2013-05/17/13-34-35.871/3 Colour.acff, stacktrace=java.io.EOFException: Attempting to read beyond end of file.
	at ome.scifio.io.NIOFileHandle.readByte(NIOFileHandle.java:272)
	at ome.scifio.io.RandomAccessInputStream.readByte(RandomAccessInputStream.java:364)
	at ome.scifio.io.RandomAccessInputStream.read(RandomAccessInputStream.java:481)
	at loci.common.RandomAccessInputStream.read(RandomAccessInputStream.java:347)
	at loci.formats.codec.LZOCodec.decompress(LZOCodec.java:78)
	at loci.formats.in.VolocityClippingReader.openBytes(VolocityClippingReader.java:90)
	at loci.formats.ImageReader.openBytes(ImageReader.java:416)
	at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:197)
	at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:226)
	at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:320)
	at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:320)
	at loci.formats.MinMaxCalculator.openBytes(MinMaxCalculator.java:273)
	at ome.services.blitz.repo.ManagedImportRequestI.parseDataByPlane(ManagedImportRequestI.java:642)
	at ome.services.blitz.repo.ManagedImportRequestI.parseData(ManagedImportRequestI.java:573)
	at ome.services.blitz.repo.ManagedImportRequestI.pixelData(ManagedImportRequestI.java:446)
	at ome.services.blitz.repo.ManagedImportRequestI.step(ManagedImportRequestI.java:311)
	at omero.cmd.HandleI.steps(HandleI.java:435)
	at omero.cmd.HandleI$1.doWork(HandleI.java:365)
	at omero.cmd.HandleI$1.doWork(HandleI.java:363)
	at sun.reflect.GeneratedMethodAccessor243.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
	at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:518)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at ome.security.basic.EventHandler.invoke(EventHandler.java:154)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:241)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:116)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
	at $Proxy66.doWork(Unknown Source)
	at ome.services.util.Executor$Impl.execute(Executor.java:416)
	at omero.cmd.HandleI.run(HandleI.java:359)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at ome.services.util.Executor$Impl$1.call(Executor.java:447)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)
Caused by: java.nio.BufferUnderflowException
	at java.nio.Buffer.nextGetIndex(Buffer.java:474)
	at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:117)
	at ome.scifio.io.NIOFileHandle.readByte(NIOFileHandle.java:270)
	... 47 more
}

	at ome.formats.importer.ImportLibrary.getImportResponse(ImportLibrary.java:504)
	at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:446)
	at org.openmicroscopy.shoola.env.data.OMEROGateway.importImageNew(OMEROGateway.java:6816)
	at org.openmicroscopy.shoola.env.data.OMEROGateway.importImage(OMEROGateway.java:6799)
	at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importCandidates(OmeroImageServiceImpl.java:234)
	at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1496)
	at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:77)
	at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:53)
	at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:102)
	at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
	at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
	at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
	at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
	at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
	at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
	at java.lang.Thread.run(Thread.java:680)

	at org.openmicroscopy.shoola.env.data.OMEROGateway.importImageNew(OMEROGateway.java:6868)
	at org.openmicroscopy.shoola.env.data.OMEROGateway.importImage(OMEROGateway.java:6799)
	at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importCandidates(OmeroImageServiceImpl.java:234)
	at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.importFile(OmeroImageServiceImpl.java:1496)
	at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.importFile(ImagesImporter.java:77)
	at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter.access$000(ImagesImporter.java:53)
	at org.openmicroscopy.shoola.env.data.views.calls.ImagesImporter$1.doCall(ImagesImporter.java:102)
	at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
	at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
	at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
	at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
	at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
	at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
	at java.lang.Thread.run(Thread.java:680)
Caused by: java.lang.RuntimeException: Failure response on import!
Category: ::omero::grid::ImportRequest
Name: import-file-exception
Parameters: {message=Attempting to read beyond end of file., filename=user-4_5/2013-05/17/13-34-35.871/3 Colour.acff, stacktrace=java.io.EOFException: Attempting to read beyond end of file.
	at ome.scifio.io.NIOFileHandle.readByte(NIOFileHandle.java:272)
	at ome.scifio.io.RandomAccessInputStream.readByte(RandomAccessInputStream.java:364)
	at ome.scifio.io.RandomAccessInputStream.read(RandomAccessInputStream.java:481)
	at loci.common.RandomAccessInputStream.read(RandomAccessInputStream.java:347)
	at loci.formats.codec.LZOCodec.decompress(LZOCodec.java:78)
	at loci.formats.in.VolocityClippingReader.openBytes(VolocityClippingReader.java:90)
	at loci.formats.ImageReader.openBytes(ImageReader.java:416)
	at loci.formats.ChannelFiller.openBytes(ChannelFiller.java:197)
	at loci.formats.ChannelSeparator.openBytes(ChannelSeparator.java:226)
	at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:320)
	at loci.formats.ReaderWrapper.openBytes(ReaderWrapper.java:320)
	at loci.formats.MinMaxCalculator.openBytes(MinMaxCalculator.java:273)
	at ome.services.blitz.repo.ManagedImportRequestI.parseDataByPlane(ManagedImportRequestI.java:642)
	at ome.services.blitz.repo.ManagedImportRequestI.parseData(ManagedImportRequestI.java:573)
	at ome.services.blitz.repo.ManagedImportRequestI.pixelData(ManagedImportRequestI.java:446)
	at ome.services.blitz.repo.ManagedImportRequestI.step(ManagedImportRequestI.java:311)
	at omero.cmd.HandleI.steps(HandleI.java:435)
	at omero.cmd.HandleI$1.doWork(HandleI.java:365)
	at omero.cmd.HandleI$1.doWork(HandleI.java:363)
	at sun.reflect.GeneratedMethodAccessor243.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
	at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:518)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at ome.security.basic.EventHandler.invoke(EventHandler.java:154)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:108)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at ome.tools.hibernate.ProxyCleanupFilter$Interceptor.invoke(ProxyCleanupFilter.java:241)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at ome.services.util.ServiceHandler.invoke(ServiceHandler.java:116)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
	at $Proxy66.doWork(Unknown Source)
	at ome.services.util.Executor$Impl.execute(Executor.java:416)
	at omero.cmd.HandleI.run(HandleI.java:359)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at ome.services.util.Executor$Impl$1.call(Executor.java:447)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)
Caused by: java.nio.BufferUnderflowException
	at java.nio.Buffer.nextGetIndex(Buffer.java:474)
	at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:117)
	at ome.scifio.io.NIOFileHandle.readByte(NIOFileHandle.java:270)
	... 47 more
}

	at ome.formats.importer.ImportLibrary.getImportResponse(ImportLibrary.java:504)
	at ome.formats.importer.ImportLibrary.importImage(ImportLibrary.java:446)
	at org.openmicroscopy.shoola.env.data.OMEROGateway.importImageNew(OMEROGateway.java:6816)
	... 13 more
Last edited 6 years ago by ajpatterson (previous) (diff)

comment:8 Changed 6 years ago by ajpatterson

File: test_images_bad/mike1_R3D.dv gives "File not valid" on scanning.

Last edited 6 years ago by ajpatterson (previous) (diff)

comment:9 Changed 6 years ago by jburel

Keeping it opened for now as a reference point if we need more information while adjusting scenario and testing material.
The steps are now covered. Thanks Blazej and Andrew

Last edited 6 years ago by jburel (previous) (diff)

comment:10 Changed 6 years ago by jburel

  • Owner ajpatterson deleted
  • Status changed from accepted to new

comment:11 Changed 6 years ago by jburel

  • Sprint changed from FS demo 4.2 to FS Demo 4.3

Moved from sprint FS demo 4.2

comment:12 Changed 6 years ago by jburel

  • Resolution set to fixed
  • Sprint changed from FS Demo 4.3 to FS demo 4.2
  • Status changed from new to closed

Data listed in import scenario. Closing

Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.84398 sec.)

© 2000-2014 University of Dundee & Open Microscopy Environment. Creative Commons Attribution 3.0 Unported License
OME source code is available under the GNU General public license or through commercial license from Glencoe Software

We're Hiring!