ImageJ plugged into scripting engine

[jamoore]

2011 Discussion

The current plan is to make use of pre/post plugins in ImageJ2 for parsing out the parameters and filling in the input values.

2008 Discussion

– macro? Aware of OMERO? 
– Start with ImageJ talking to OMERO 
– More: grab jar from web and drop it into a directory 
– OMERO downloader done; still working on uploader 
– Can we get help from ImageJ community? Wayne. 
– Jason calls Sebastian. 

[jmoore]

Changing from Jason ownership for evaluation. We might want to move this out of the #1856 requirement so that it's visible in the top-level tickets report.

[jmoore]

Adding another Curtis alias.

[crueden-x]

A couple of weeks ago, I spent most of the week digging in to the OMERO scripting framework. I bookmarked a few places in the source code (using comments) but then it sat on my hard drive for the next week or two. I have finally pushed it to a ​new branch, along with a ​WIP commit whose message expresses my current thoughts, which for posterity I duplicate here as well:

These are some of the key places defining how OMERO scripting works, and what the ImageJ scripting support will potentially need to touch.

My biggest concern right now is that the entire scripting infrastructure is Python-specific, and has a one-to-one mapping between original files and scripts. Both of those assumptions are untrue for server-side ImageJ execution: a JAR file might provide one or a multitude of ImageJ commands which should be exposed via the OMERO scripting service.

As a consequence, the method ScriptRepoHelper#loadAll?(boolean) which returns a List<OriginalFile?> cannot really be hacked to include ImageJ scripts, because OriginalFile? is not a proper metadata-rich object for actually describing an ImageJ command.

What makes the most sense to me would be to create a new ICE interface with fields that describe each script in slightly more detail. This would include an OriginalFile? but also other metadata to facilitate execution of ImageJ and potentially other scripting paradigms as well.

Unfortunately, due to ImageJ deadlines, it will be a while longer before I can pursue any details of the approach above. In the meantime, comments welcome!

[crueden-x]

I am going to try a much easier approach, which is what jmoore suggested last time we spoke in person: wrap each ImageJ command in its own Python script, which executes it using a ​NativeWrapper-based approach.

Steps

1. Create a native wrapper script that executes a single chosen ImageJ command.
2. Once that is working, generalize the script to pull out boilerplate into a common ImageJScript superclass.
3. Write some utility code to autogenerate Python wrappers for all ImageJ commands on the Java classpath.

JAR files will need to live on the server in a hardcoded (relative?) location for now.

Advantages

• Reuses OMERO's existing scripting framework.
• Isolates ImageJ execution from the OMERO server itself, in case of bugs, etc.

Disadvantages

• Going Java (OMERO server) -> Python (scripting framework) -> Java again (ImageJ script) seems very convoluted to me.
• Logic to autogenerate ImageJ commands from uploaded JAR file(s) needs to live somewhere and get integrated for users.

[crueden-x]

This morning I talked to Lee Kamentsky about using the CellProfiler?-ImageJ integration (an in-process solution) for the Python scripts to invoke ImageJ. The good news is that the integration is a quite general ​Python-Java bridge with ​higher-level utility access and ​ImageJ2-specific usage. The bad news is that it is not completely well-separated from the CellProfiler? core right now, so there would be some effort to split it out (which Lee plans to do eventually anyway). I ​chatted about this with leekamentsky and dscho on #imagejdev.

Josh Moore also later chimed in, stating that he is in favor of a solution that does not require jumping back and forth from Python. We are planning to chat about how best to accomplish that tomorrow.

[crueden-x]

After talking to Josh, we decided on a more direct Java-Java interoperability approach using Ice. This will make it much easier to translate ImageJ's image data structures back and forth from OMERO/Ice image data structures, since all the code for doing so will be Java-based. At various points, the OMERO server will spawn an ImageJ process with three main functions:

1. Generate a list of stub files on disk (which can be in whatever format ImageJ likes) from which ImageJ can later reconstruct metadata about each available command/script. The vital pieces of information for the stub are: command class, input presets, and version number.
2. Parse a stub file to produce metadata about that command/script and report it to OMERO (presumably via Ice...?)
3. Execute a command as specified by a particular stub file, receiving the inputs from the OMERO server via Ice, and sending the outputs to the OMERO server via Ice afterwards.

I created an ​imagej-omero repository which will house all this code.

[crueden-x]

First cut at a skeleton implementation is ​here.

[crueden-x]

As of last night, I completed an initial version that does the ImageJ half of things; i.e.: it implements the three features discussed above (generate, parse and execute), providing a simple command line interface to specify which operation(s) to perform. It generates the stubs into an "ij-stubs" folder (configurability of the location to be added later). It can be told to parse a particular stub, from which it will determine the correct ModuleInfo and report its details (but not yet feed them to OMERO). It can also be given a stub for execution, and it will execute it, though no communication with OMERO is in place yet.

Still to be done in ij-omero-server:

• Add a dependency to omero_client library. Use it to feed parse results to OMERO, to request input parameter values from OMERO, and to feed output parameter values back to OMERO.
• In order to successfully request and feed parameter values to and from OMERO, a conversion between OMERO and ImageJ data structures will need to be implemented.
• NEXT STEP: I would like some guidance on the API to use for communication with OMERO: which services to call, which Ice interfaces are relevant, etc. And it is not clear to me whether we need to invent any new ones.

To be done on the OMERO side:

• Add ImageJ awareness, so that an ImageJ installation on the server side (or one on another server) can be detected and/or configured. OMERO needs to launch an ij-omero-server process and tell it to refresh the stubs. It also needs to query the metadata for each stub at some point, for storage into the OMERO DB. Once that is done, the available ImageJ commands will be available for use from OMERO clients. Lastly, it needs to be able to spawn a new ij-omero-server process whenever execution of an ImageJ command is desired.

(Technical side-note: we actually operate on ImageJ Modules rather than ImageJ Commands because a Command is simply one type of Module. There may be others. So we use the ImageJ ModuleService to do the heavy lifting. As long as the Module's ModuleInfo implements ImageJ's Identifiable interface, it can be represented as a stub on disk for later reference by OMERO.)

joshmoore: Please speak up if anything I've said above is incorrect!

[jamoore]

PR opened with support for Jython: ​https://github.com/openmicroscopy/openmicroscopy/pull/1324

Either Curtis, you can take that forward and add specific IJ support, or we can workout what changes you need (and file endings you would like supported) and add them on gh-1324.

[crueden-x]

Using ​PR #1324 as a starting point, it is now possible to call ImageJ commands as OMERO scripts. The required glue code lives in the ​ij-omero-server Java library. Each ImageJ command is wrapped as a Jython script.

The following ImageJ tutorial commands are tested and working:

• ​HelloWorld: a basic example with one string input, and one string output.
• ​WidgetDemo: an example exercising many different parameter types, providing a good illustration of how type conversion works going back and forth between ImageJ and OMERO.

To try it out, follow these steps:

1. Build OMERO from joshmoore's ​jy-scripts branch.

2. Download ​ImageJ2 and unpack into $OMERO_HOME/dist/lib (it will create a subfolder called ImageJ.app).

3. Download the ​ij-omero-server interoperability library into $OMERO_HOME/dist/lib/ImageJ.app/jars.

4. Download the ​simple-command and ​widget-demo tutorial plugins into $OMERO_HOME/dist/lib/ImageJ.app/plugins.

5. Download the latest ​pre-built standalone version of Jython.

6. Create a jython launch script on your path:

   #!/bin/sh
   export OMERO_HOME="$HOME/code/ome/openmicroscopy/dist"
   java -cp "$HOME/bin/jython-standalone-2.5.3.jar:$OMERO_HOME/lib/ImageJ.app/jars/*:$OMERO_HOME/lib/ImageJ.app/plugins/*:$CLASSPATH" org.python.util.jython $@
   

7. Run jython and verify the following lines execute without error:

   import imagej.ImageJ
   import HelloWorld
   import WidgetDemo
   

8. Download ​HelloWorld.jy into $OMERO_HOME/lib/scripts/imagej (create the folder).

9. omero admin start (if you haven't already)

10. omero script list

You should see HelloWorld.jy as an available option.

11. omero script params $(omero script list | grep HelloWorld | sed 's/|.*//')

12. omero script launch $(omero script list | grep HelloWorld | sed 's/|.*//')

13. To test WidgetDemo or any other ImageJ2 command, just copy HelloWorld.jy to e.g. WidgetDemo.jy and make the relevant edits to call the desired command instead.

The next step is to support image parameters (i.e., pixels IDs). That work is pending on the ​scifio-format branch. The goal is to implement an "OMERO" format for SCIFIO which supports transparent pixels access to and from OMERO via the SCIFIO API (and hence via the ImgLib2 API).

[crueden-x]

The ImageJ-OMERO code has been cleaned up and refactored. The above instructions have been added to the README. See:

​https://github.com/imagej/imagej-omero

I also implemented the SCIFIO format for accessing OMERO pixels, both read and write:

​https://github.com/scifio/scifio-omero

There are now ImageJ commands for working with OMERO pixels from within ImageJ, too.

Unfortunately, there are still bugs and nothing related to images fully works yet.

[crueden-x]

Downloading from OMERO to ImageJ is now working. So you can now connect to OMERO from the ImageJ application, and visualize a given image, with cells downloaded on demand from the server thanks to the SCIFIOCellImg functionality.

However, the server-side ij-omero module still cannot do it as part of a script execution, likely due to time performance hangups. Will keep investigating.

[crueden-x]

Performance issue was largely addressed in commit ​ef4b63be. In my tests, executing the HelloWorld and WidgetDemo examples improved from >60s to <20s each. And the ComputeStats example (which takes an image as input) now also works in under 20s with a small single-plane image. So the OMERO -> ImageJ direction is working for images!

Previously, though the ImageJ command itself was finished executing, OMERO was waiting for sessions to be closed before finishing script execution. The new ScriptRunner.main code explicitly disposes the ImageJ context then shuts down the JVM, which addresses the issue.

Next up: uploading images back to OMERO from ImageJ. Main challenge there is that SCIFIO's ImgSaver only supports writing certain types of Img objects. It needs to be expanded to handle any Img including SCIFIOCellImg.

[Josh Moore <josh@…>]

(In [c6cccddf8a7b72faf7d563c45ade2423562d6b52/ome.git] on branch develop) Merge pull request #1324 from joshmoore/jy-scripts

Jython scripts (See #918)

[crueden-x]

The issues relating to ImageJ-OMERO are now all filed in ​its issue tracker.

What is left in this story are only tasks relating to the OMERO side of things. Should I still own this? Should we split them to a separate story?

[jamoore]

I've relinked all existing tickets under #1343 and scheduled it for late-summer/fall. Closing. Thanks, Curtis!

[jamoore]

(Should have adjusted the milestone)