Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Task #12313 (new)

Opened 5 years ago

Last modified 4 years ago

Bug: OME-TIFF export with XML annotation

Reported by: wmoore Owned by: jburel
Priority: critical Milestone: OME-Files
Component: Export Version: n.a.
Keywords: n.a. Cc: ux@…, mlinkert, hflynn
Resources: n.a. Referenced By: n.a.
References: n.a. Remaining Time: n.a.
Sprint: n.a.

Description

See OME-TIFF files under QA:9249.

After import on 5.0.2 and attempt to "Export as OME-TIFF" in Insight, this fails as described

https://www.openmicroscopy.org/community/viewtopic.php?f=6&t=7442&p=14009#p14009

and we get:

java.lang.Exception: org.openmicroscopy.shoola.env.data.DSAccessException: Cannot export the image as an OME-TIFF
	at org.openmicroscopy.shoola.env.data.OMEROGateway.exportImageAsOMEObject(OMEROGateway.java:7055)
	at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.exportImageAsOMEFormat(OmeroImageServiceImpl.java:1453)
	at org.openmicroscopy.shoola.env.data.views.calls.ExportLoader$1.doCall(ExportLoader.java:87)
	at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
	at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
	at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
	at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
	at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
	at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
	at java.lang.Thread.run(Thread.java:695)
Caused by: org.openmicroscopy.shoola.env.data.DSAccessException: Cannot export the image as an OME-formats 
	at org.openmicroscopy.shoola.env.data.OMEROGateway.exportImageAsOMEObject(OMEROGateway.java:7044)
	... 9 more
Caused by: omero.InternalException
    serverStackTrace = "java.lang.RuntimeException: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 142; The prefix "xsi" for attribute "xsi:schemaLocation" associated with an element type "OME" is not bound.
                        	at ome.xml.model.XMLAnnotation.asXMLElement(XMLAnnotation.java:261)
                        	at ome.xml.model.XMLAnnotation.asXMLElement(XMLAnnotation.java:228)
                        	at ome.xml.model.StructuredAnnotations.asXMLElement(StructuredAnnotations.java:625)
                        	at ome.xml.model.StructuredAnnotations.asXMLElement(StructuredAnnotations.java:605)
                        	at ome.xml.model.OME.asXMLElement(OME.java:835)
                        	at ome.xml.model.OME.asXMLElement(OME.java:720)
                        	at ome.xml.meta.AbstractOMEXMLMetadata.dumpXML(AbstractOMEXMLMetadata.java:113)
                        	at ome.xml.meta.OMEXMLMetadataImpl.dumpXML(OMEXMLMetadataImpl.java:106)
                        	at loci.formats.services.OMEXMLServiceImpl.getOMEXML(OMEXMLServiceImpl.java:415)
                        	at ome.services.blitz.impl.ExporterI.getMetadataBytes(ExporterI.java:566)
                        	at ome.services.blitz.impl.ExporterI.access$400(ExporterI.java:86)
                        	at ome.services.blitz.impl.ExporterI$2.doWork(ExporterI.java:388)
                        	at sun.reflect.GeneratedMethodAccessor272.invoke(Unknown Source)
                        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                        	at java.lang.reflect.Method.invoke(Method.java:606)
                        	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
                        	at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:576)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

Change History (27)

comment:1 Changed 5 years ago by jburel

  • Cc ux@… added
  • Owner changed from jburel to ajpatterson

Problem in the XML itself. Passing it andrew to check

comment:2 Changed 5 years ago by ajpatterson

These files are really really broken. They sorta look like two files inside each other. What details do you want?

comment:3 Changed 5 years ago by ajpatterson

On closer inspection they are 4 files:

  • the first outer file
  • a second file inside the image description of the first file
  • a third file inside an XML annotation on the first file
  • a fourth file inside the image description of the third file

comment:4 Changed 5 years ago by ajpatterson

This is probably what was meant:

<?xml version="1.0" encoding="utf-8"?>
<OME xmlns="http://www.openmicroscopy.org/Schemas/OME/2013-06"
	xmlns:SA="http://www.openmicroscopy.org/Schemas/SA/2013-06"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	UUID="urn:uuid:0369b00c-8a22-4682-8788-723916388cc2"
	xsi:schemaLocation="http://www.openmicroscopy.org/Schemas/OME/2013-06 http://www.openmicroscopy.org/Schemas/OME/2013-06/ome.xsd">
	<Image ID="Image:0">
		<Pixels BigEndian="true" DimensionOrder="XYZCT" ID="Pixels:0" SizeC="1" SizeT="6"
			SizeX="672" SizeY="512" SizeZ="1" Type="uint16">
			<Channel ID="Channel:0:0" SamplesPerPixel="1">
				<LightPath/>
			</Channel>
			<TiffData FirstC="0" FirstT="0" FirstZ="0" IFD="0" PlaneCount="1">
				<UUID FileName="A-1-XY_00000.OME.tiff"
					>urn:uuid:0369b00c-8a22-4682-8788-723916388cc2</UUID>
			</TiffData>
			<TiffData FirstC="0" FirstT="1" FirstZ="0" IFD="1" PlaneCount="1">
				<UUID FileName="A-1-XY_00000.OME.tiff"
					>urn:uuid:0369b00c-8a22-4682-8788-723916388cc2</UUID>
			</TiffData>
			<TiffData FirstC="0" FirstT="2" FirstZ="0" IFD="2" PlaneCount="1">
				<UUID FileName="A-1-XY_00000.OME.tiff"
					>urn:uuid:0369b00c-8a22-4682-8788-723916388cc2</UUID>
			</TiffData>
			<TiffData FirstC="0" FirstT="3" FirstZ="0" IFD="3" PlaneCount="1">
				<UUID FileName="A-1-XY_00000.OME.tiff"
					>urn:uuid:0369b00c-8a22-4682-8788-723916388cc2</UUID>
			</TiffData>
			<TiffData FirstC="0" FirstT="4" FirstZ="0" IFD="4" PlaneCount="1">
				<UUID FileName="A-1-XY_00000.OME.tiff"
					>urn:uuid:0369b00c-8a22-4682-8788-723916388cc2</UUID>
			</TiffData>
			<TiffData FirstC="0" FirstT="5" FirstZ="0" IFD="5" PlaneCount="1">
				<UUID FileName="A-1-XY_00000.OME.tiff"
					>urn:uuid:0369b00c-8a22-4682-8788-723916388cc2</UUID>
			</TiffData>
		</Pixels>
		<SA:AnnotationRef ID="Annotation:0"/>
	</Image>
	<SA:StructuredAnnotations>
		<SA:XMLAnnotation ID="Annotation:0" Namespace="openmicroscopy.org/omero/dimension/modulo">
			<SA:Value>
				<Modulo namespace="http://www.openmicroscopy.org/Schemas/Additions/2011-09">
					<ModuloAlongT Type="lifetime" TypeDescription="Gated" Unit="ps">
						<Label>0</Label>
						<Label>4000</Label>
						<Label>5500</Label>
						<Label>7000</Label>
						<Label>8500</Label>
						<Label>10000</Label>
					</ModuloAlongT>
				</Modulo>
			</SA:Value>
		</SA:XMLAnnotation>
	</SA:StructuredAnnotations>
</OME>

A lot of the structure is correct, once I removed the extra copies and added a namespace definition I could get it to validate.

comment:5 Changed 5 years ago by jamoore

  • Cc mlinkert added

Could this be an issue in how we're escaping/embedding XML within XML?

comment:6 Changed 5 years ago by ajpatterson

The third file inside the XML annotation is not an XML fragment but a full top level XML node with a definition of another xsi:schemaLocation, I suspect this is confusing things a bit. The two files embedded inside the Description string looks to be full of &lt; and &gt; so look to have been processed correctly.

comment:7 Changed 5 years ago by jamoore

But we need to support that, no? If someone gives us an XML documentation, we need to preserve it via CDATA/escaping.

comment:8 Changed 5 years ago by ajpatterson

Just checked my big xml book, it says it is usual to only have the xsi:schemaLocation at the top but is valid to have it lower down. Only requirement is it precedes any use of the namespaces it is for, which makes sense. Course the issue might be related to something else in the file.

comment:9 Changed 5 years ago by jamoore

Andrew: next steps here?

comment:10 Changed 5 years ago by ajpatterson

The reason the file is failing is because the characters "<?xml" are only allowed as the first 5 characters in an xml file. In this case that also occur inside the <value> element of the XmlAnnotation. I have made a small file with this problem to go into data_repo/test_images_bad/Embed-XML-File.ome.tif.

comment:11 Changed 5 years ago by jamoore

Incl. inside of CDATA, etc?

comment:12 Changed 5 years ago by ajpatterson

If the characters are escaped in some form, either using &gt; or a CDATA block then they are fine. Though, as an XmlAnnotation Value elements can only contain elements, a CDATA block is not valid there. You could wrap in in a dummy node as well but that is getting a bit messy:

<SA:Value><a><![CDATA[<?xml version="1.0" encoding="UTF-8" standalone="no"?><DummyElement abcd="1234"/>]]></a></SA:Value>

comment:13 Changed 5 years ago by ajpatterson

You would also have to make sure the XML you were wrapping did not use CDATA blocks or you might have a problem. You could always escape that too, and so on...

comment:14 Changed 5 years ago by jamoore

  • Priority changed from minor to major

Andrew: do you have any work in progress on this? What's the next step?

comment:15 Changed 5 years ago by ajpatterson

Not that I am aware of. It is the writer that would need updated to escape the xml. It was a very, very broken file. Is the onus on us to handle reading a file that is this broken?

comment:16 Changed 5 years ago by jamoore

I would think the measure would be: IF we can import a valid OME-TIFF which contains an entire XML document as an annotation, THEN it should be possible to re-export it.

comment:17 Changed 5 years ago by ajpatterson

Then the only option would be the CDATA block approach, but this will change what was stored (slightly) as the input would not technically be valid xml, but the output should be.

comment:18 Changed 5 years ago by jamoore

  • Cc hflynn added

Minimally we're going to need a clear page on what we suggest. If you a few alternatives (some of which pass and some fail) then that would be a good start.

comment:19 Changed 5 years ago by jburel

  • Owner changed from ajpatterson to jburel

comment:20 Changed 5 years ago by jamoore

  • Milestone changed from 5.1.2 to 5.1.1
  • Priority changed from major to critical

comment:21 Changed 5 years ago by jburel

  • Milestone changed from 5.1.1 to 5.1.3

comment:22 Changed 4 years ago by jamoore

  • Milestone changed from 5.1.4 to OMERO-5.1.4

Splitting 5.1.4 due to milestone decoupling

comment:23 Changed 4 years ago by jburel

  • Milestone changed from OMERO-5.1.4 to OMERO-5.2.0
  • Owner changed from jburel to mtbcarroll

Passing this ticket to Mark who is currently working on OME-TIFF export
and move it to 5.2.0

comment:24 Changed 4 years ago by mtbcarroll

This looks like it would be a major effort for me given my present level of ignorance. If it really is critical for 5.2.0 then I can push other things aside accordingly, just want to confirm that before I do so.

comment:25 Changed 4 years ago by jburel

  • Owner changed from mtbcarroll to jburel

not critical.
I take it back. I will try to have a look.

comment:26 Changed 4 years ago by jburel

  • Milestone changed from OMERO-5.2.0 to 5.x

comment:27 Changed 4 years ago by jamoore

  • Milestone changed from 5.x to OMEFiles
Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.414715 sec.)

We're Hiring!