Bug: OME-TIFF export with XML annotation

[wmoore]

See OME-TIFF files under QA:9249.

After import on 5.0.2 and attempt to "Export as OME-TIFF" in Insight, this fails as described

​https://www.openmicroscopy.org/community/viewtopic.php?f=6&t=7442&p=14009#p14009

and we get:

java.lang.Exception: org.openmicroscopy.shoola.env.data.DSAccessException: Cannot export the image as an OME-TIFF
	at org.openmicroscopy.shoola.env.data.OMEROGateway.exportImageAsOMEObject(OMEROGateway.java:7055)
	at org.openmicroscopy.shoola.env.data.OmeroImageServiceImpl.exportImageAsOMEFormat(OmeroImageServiceImpl.java:1453)
	at org.openmicroscopy.shoola.env.data.views.calls.ExportLoader$1.doCall(ExportLoader.java:87)
	at org.openmicroscopy.shoola.env.data.views.BatchCall.doStep(BatchCall.java:144)
	at org.openmicroscopy.shoola.util.concur.tasks.CompositeTask.doStep(CompositeTask.java:226)
	at org.openmicroscopy.shoola.env.data.views.CompositeBatchCall.doStep(CompositeBatchCall.java:126)
	at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.exec(ExecCommand.java:165)
	at org.openmicroscopy.shoola.util.concur.tasks.ExecCommand.run(ExecCommand.java:276)
	at org.openmicroscopy.shoola.util.concur.tasks.AsyncProcessor$Runner.run(AsyncProcessor.java:91)
	at java.lang.Thread.run(Thread.java:695)
Caused by: org.openmicroscopy.shoola.env.data.DSAccessException: Cannot export the image as an OME-formats 
	at org.openmicroscopy.shoola.env.data.OMEROGateway.exportImageAsOMEObject(OMEROGateway.java:7044)
	... 9 more
Caused by: omero.InternalException
    serverStackTrace = "java.lang.RuntimeException: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 142; The prefix "xsi" for attribute "xsi:schemaLocation" associated with an element type "OME" is not bound.
                        	at ome.xml.model.XMLAnnotation.asXMLElement(XMLAnnotation.java:261)
                        	at ome.xml.model.XMLAnnotation.asXMLElement(XMLAnnotation.java:228)
                        	at ome.xml.model.StructuredAnnotations.asXMLElement(StructuredAnnotations.java:625)
                        	at ome.xml.model.StructuredAnnotations.asXMLElement(StructuredAnnotations.java:605)
                        	at ome.xml.model.OME.asXMLElement(OME.java:835)
                        	at ome.xml.model.OME.asXMLElement(OME.java:720)
                        	at ome.xml.meta.AbstractOMEXMLMetadata.dumpXML(AbstractOMEXMLMetadata.java:113)
                        	at ome.xml.meta.OMEXMLMetadataImpl.dumpXML(OMEXMLMetadataImpl.java:106)
                        	at loci.formats.services.OMEXMLServiceImpl.getOMEXML(OMEXMLServiceImpl.java:415)
                        	at ome.services.blitz.impl.ExporterI.getMetadataBytes(ExporterI.java:566)
                        	at ome.services.blitz.impl.ExporterI.access$400(ExporterI.java:86)
                        	at ome.services.blitz.impl.ExporterI$2.doWork(ExporterI.java:388)
                        	at sun.reflect.GeneratedMethodAccessor272.invoke(Unknown Source)
                        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                        	at java.lang.reflect.Method.invoke(Method.java:606)
                        	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
                        	at ome.services.util.Executor$Impl$Interceptor.invoke(Executor.java:576)
                        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

[jburel]

Problem in the XML itself. Passing it andrew to check

[ajpatterson]

These files are really really broken. They sorta look like two files inside each other. What details do you want?

[ajpatterson]

On closer inspection they are 4 files:

• the first outer file

• a second file inside the image description of the first file

• a third file inside an XML annotation on the first file

• a fourth file inside the image description of the third file

[ajpatterson]

This is probably what was meant:

<?xml version="1.0" encoding="utf-8"?>
<OME xmlns="http://www.openmicroscopy.org/Schemas/OME/2013-06"
	xmlns:SA="http://www.openmicroscopy.org/Schemas/SA/2013-06"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	UUID="urn:uuid:0369b00c-8a22-4682-8788-723916388cc2"
	xsi:schemaLocation="http://www.openmicroscopy.org/Schemas/OME/2013-06 http://www.openmicroscopy.org/Schemas/OME/2013-06/ome.xsd">
	<Image ID="Image:0">
		<Pixels BigEndian="true" DimensionOrder="XYZCT" ID="Pixels:0" SizeC="1" SizeT="6"
			SizeX="672" SizeY="512" SizeZ="1" Type="uint16">
			<Channel ID="Channel:0:0" SamplesPerPixel="1">
				<LightPath/>
			</Channel>
			<TiffData FirstC="0" FirstT="0" FirstZ="0" IFD="0" PlaneCount="1">
				<UUID FileName="A-1-XY_00000.OME.tiff"
					>urn:uuid:0369b00c-8a22-4682-8788-723916388cc2</UUID>
			</TiffData>
			<TiffData FirstC="0" FirstT="1" FirstZ="0" IFD="1" PlaneCount="1">
				<UUID FileName="A-1-XY_00000.OME.tiff"
					>urn:uuid:0369b00c-8a22-4682-8788-723916388cc2</UUID>
			</TiffData>
			<TiffData FirstC="0" FirstT="2" FirstZ="0" IFD="2" PlaneCount="1">
				<UUID FileName="A-1-XY_00000.OME.tiff"
					>urn:uuid:0369b00c-8a22-4682-8788-723916388cc2</UUID>
			</TiffData>
			<TiffData FirstC="0" FirstT="3" FirstZ="0" IFD="3" PlaneCount="1">
				<UUID FileName="A-1-XY_00000.OME.tiff"
					>urn:uuid:0369b00c-8a22-4682-8788-723916388cc2</UUID>
			</TiffData>
			<TiffData FirstC="0" FirstT="4" FirstZ="0" IFD="4" PlaneCount="1">
				<UUID FileName="A-1-XY_00000.OME.tiff"
					>urn:uuid:0369b00c-8a22-4682-8788-723916388cc2</UUID>
			</TiffData>
			<TiffData FirstC="0" FirstT="5" FirstZ="0" IFD="5" PlaneCount="1">
				<UUID FileName="A-1-XY_00000.OME.tiff"
					>urn:uuid:0369b00c-8a22-4682-8788-723916388cc2</UUID>
			</TiffData>
		</Pixels>
		<SA:AnnotationRef ID="Annotation:0"/>
	</Image>
	<SA:StructuredAnnotations>
		<SA:XMLAnnotation ID="Annotation:0" Namespace="openmicroscopy.org/omero/dimension/modulo">
			<SA:Value>
				<Modulo namespace="http://www.openmicroscopy.org/Schemas/Additions/2011-09">
					<ModuloAlongT Type="lifetime" TypeDescription="Gated" Unit="ps">
						<Label>0</Label>
						<Label>4000</Label>
						<Label>5500</Label>
						<Label>7000</Label>
						<Label>8500</Label>
						<Label>10000</Label>
					</ModuloAlongT>
				</Modulo>
			</SA:Value>
		</SA:XMLAnnotation>
	</SA:StructuredAnnotations>
</OME>

A lot of the structure is correct, once I removed the extra copies and added a namespace definition I could get it to validate.

[jamoore]

Could this be an issue in how we're escaping/embedding XML within XML?

[ajpatterson]

The third file inside the XML annotation is not an XML fragment but a full top level XML node with a definition of another xsi:schemaLocation, I suspect this is confusing things a bit. The two files embedded inside the Description string looks to be full of < and > so look to have been processed correctly.

[jamoore]

But we need to support that, no? If someone gives us an XML documentation, we need to preserve it via CDATA/escaping.

[ajpatterson]

Just checked my big xml book, it says it is usual to only have the xsi:schemaLocation at the top but is valid to have it lower down. Only requirement is it precedes any use of the namespaces it is for, which makes sense. Course the issue might be related to something else in the file.

[jamoore]

Andrew: next steps here?

[ajpatterson]

The reason the file is failing is because the characters "<?xml" are only allowed as the first 5 characters in an xml file. In this case that also occur inside the <value> element of the XmlAnnotation. I have made a small file with this problem to go into data_repo/test_images_bad/Embed-XML-File.ome.tif.

[jamoore]

Incl. inside of CDATA, etc?

[ajpatterson]

If the characters are escaped in some form, either using &gt; or a CDATA block then they are fine. Though, as an XmlAnnotation Value elements can only contain elements, a CDATA block is not valid there. You could wrap in in a dummy node as well but that is getting a bit messy:

<SA:Value><a><![CDATA[<?xml version="1.0" encoding="UTF-8" standalone="no"?><DummyElement abcd="1234"/>]]></a></SA:Value>

[ajpatterson]

You would also have to make sure the XML you were wrapping did not use CDATA blocks or you might have a problem. You could always escape that too, and so on...

[jamoore]

Andrew: do you have any work in progress on this? What's the next step?

[ajpatterson]

Not that I am aware of. It is the writer that would need updated to escape the xml. It was a very, very broken file. Is the onus on us to handle reading a file that is this broken?

[jamoore]

I would think the measure would be: IF we can import a valid OME-TIFF which contains an entire XML document as an annotation, THEN it should be possible to re-export it.

[ajpatterson]

Then the only option would be the CDATA block approach, but this will change what was stored (slightly) as the input would not technically be valid xml, but the output should be.

[jamoore]

Minimally we're going to need a clear page on what we suggest. If you a few alternatives (some of which pass and some fail) then that would be a good start.

[jburel]

Scheduled for 5.1.3
Ref card ​https://trello.com/c/22XfjucX/159-export-re-import

[jamoore]

Splitting 5.1.4 due to milestone decoupling

[jburel]

Passing this ticket to Mark who is currently working on OME-TIFF export
and move it to 5.2.0

[mtbcarroll]

This looks like it would be a major effort for me given my present level of ignorance. If it really is critical for 5.2.0 then I can push other things aside accordingly, just want to confirm that before I do so.

[jburel]

not critical.
I take it back. I will try to have a look.