Bug: svs import?

[wmoore]

Some strange behaviour during svs import. See logs linked from e-mails below.

E.g. Blitz.log

2011-11-20 16:00:03,497 INFO  [         ome.security.basic.EventHandler] (l.Server-0)  Auth:	user=0,group=0,event=null(User),sess=9a37cc94-286c-4f0a-ba73-1fe822886ded,share=-1
2011-11-20 16:00:03,499 INFO  [                 org.perf4j.TimingLogger] (l.Server-0) start[1321765203495] time[4] tag[omero.call.success.ome.logic.QueryImpl.projection]
2011-11-20 16:00:03,499 INFO  [        ome.services.util.ServiceHandler] (l.Server-0)  Rslt:	([Ljava.lang.Object;@509bc8f7, [Ljava.lang.Object;@27448938, [Ljava.lang.Object;@38962789, ... 22 more)
2011-11-20 16:00:03,514 INFO  [        ome.services.util.ServiceHandler] (l.Server-6)  Meth:	interface ome.api.IQuery.projection
2011-11-20 16:00:03,514 INFO  [        ome.services.util.ServiceHandler] (l.Server-6)  Args:	[select o.id from OriginalFile as o where o.id in (:ids), PARAMS:ids=ArrayList(25) ]
2011-11-20 16:00:03,515 INFO  [  ome.security.basic.BasicSecuritySystem] (l.Server-6) Setting share id to -1
2011-11-20 16:00:03,515 INFO  [         ome.security.basic.EventHandler] (l.Server-6)  Auth:	user=0,group=0,event=null(User),sess=9a37cc94-286c-4f0a-ba73-1fe822886ded,share=-1
2011-11-20 16:00:03,517 INFO  [                 org.perf4j.TimingLogger] (l.Server-6) start[1321765203514] time[3] tag[omero.call.success.ome.logic.QueryImpl.projection]
2011-11-20 16:00:03,517 INFO  [        ome.services.util.ServiceHandler] (l.Server-6)  Rslt:	([Ljava.lang.Object;@42c42928, [Ljava.lang.Object;@3c361280, [Ljava.lang.Object;@6e03f2f0, ... 22 more)
2011-11-20 16:00:03,530 INFO  [        ome.services.util.ServiceHandler] (l.Server-8)  Meth:	interface ome.api.IQuery.projection
2011-11-20 16:00:03,530 INFO  [        ome.services.util.ServiceHandler] (l.Server-8)  Args:	[select o.id from OriginalFile as o where o.id in (:ids), PARAMS:ids=ArrayList(25) ]
2011-11-20 16:00:03,531 INFO  [  ome.security.basic.BasicSecuritySystem] (l.Server-8) Setting share id to -1

OMERO.web log

Thu, 20 Oct 2011 16:00:03 omero.util.Resources INFO     Starting
Thu, 20 Oct 2011 16:00:23 omero.util.Resources INFO     Halted
Thu, 20 Oct 2011 17:01:49 omero.util.Resources INFO     Starting
Thu, 20 Oct 2011 17:05:52 omero.util.Resources INFO     Halted
Thu, 20 Oct 2011 18:00:43 omero.util.Resources INFO     Starting
Thu, 20 Oct 2011 18:04:32 omero.util.Resources INFO     Halted
Thu, 20 Oct 2011 19:00:04 omero.util.Resources INFO     Starting

I've tested again with:
237M - 3066.svs

I've tested this file on another OMERO server with Insight client.
Import went fine but I can't see the images (see the snapshots).
It's definetely not 8 min ;(

Logs are here - ​https://cloudstor.aarnet.edu.au/filesender/?vid=09D4E374-EF48-AF26-D9E6-C49263129BD7

May be it takes a reeeeally long time.

Thanks,

On Wed, Nov 16, 2011 at 09:41, Will Moore <will@…> wrote:
Hi Kim (& Leon),

I'm not sure what the "Version 5.0" logging statement means.

Could you show us that part of the log? Our roadmap for when and what OMERO 5.0 is has evolved over time. We probably need to clear that up.

Could you give us some more details on how slow you're finding the import?
I just tried it again with your sample file, running the server on my laptop (Recent Macbook Pro) and it took about 8 minutes to generate image pyramids for all the images apart from the largest (which fails).
Is it taking you longer than this?

I have also noticed incidentally that my laptop seems to be working hard at the failing image (fan going etc) until I delete it using Insight.
Haven't investigated this fully yet. My logs are attached.

Cheers,

Will.

On 14 Nov 2011, at 03:30:

Hi Will,

Thanks for looking into this issue for us. So all the functionality is there i.e. it will load all the images but it is very slow. It says in the log that it will be fixed in Version 5.0. Is there an approximate timeframe for this?

Thanks,

On 11 November 2011 21:13, Will Moore wrote:

The file seems to import OK, except for the largest of the 7 images. 95k x 45k (see screen-shots).

The largest image doesn't import - This is a known issue: ​http://trac.openmicroscopy.org.uk/ome/ticket/6500
"SizeX * SizeY exceeds Integer.MAX_VALUE."

When we give you the warning "Please try again later", this is when the image pyramids are being built on the server. Just takes time.

Will.

[jmoore]

Moved from sprint 2011-11-29 (3)

[jburel]

Moved from sprint 2011-12-13 (4)

[jburel]

Moved from sprint 2012-01-03 (5)

[cxallan]

Fundamentally this comes down to two things:

1. User expectation and user server performance
2. Bugs to do with very large images (#6500)

Quite simply the import of even moderate size SVS files currently on standard hardware is a multi-hour procedure and while that process is being undertaken no other pyramids will be generated. It is a first come, first serve, single element queue that processes images in the background. Specifically with respect to SVS we are attempting to address this by having more logic available server side and not having to generate pyramids for billions of pixels as outlined in #909.

On the issue of integer overflow, specifically related to SIZE_X * SIZE_Y this is also in the process of being addressed in #6967 and the server side part of the work has already been completed as part of #7297.

Closing this as duplicate.