RAPID: Rapid as an OMERO service

[dzmacdonald]

It is now possible to have RAPID as an OMERO service, this will allow clients to submit/manage jobs via OMERO, qith RAPID as an job management engine.

[dzmacdonald]

RAPID in OMERO

RAPID is to be integrated into OMERO as a service. This system will allow the submission of jobs to a cluster though OMERO using the RAPID job submission engine.

Meeting on RAPID in OMERO 11/4

Present : Jos Koetsier, Donald MacDonald

The meeting discussed the integration of RAPID into OMERO, the workflow for job submission and the requirements on the job and cluster, depending on whether the job submitted was OMERO aware, or OMERO agnostic. We also decided on a common nomenclature:

• A job is the running process on the cluster
• A script is the collection of files residing on the server; defines the method for deploying a job, what data is extracted from OMERO, how it is copied onto the cluster, how the job is monitored and the results copied back to OMERO.
• Staging is the act of copying data from OMERO to the cluster.
• Unstaging it the act of copying data from the cluster back to OMERO.
• The executable is the code that runs on the cluster.
• OMERO Aware scripts are those scripts that communicate directly to OMERO.
• OMERO agnostic scripts are those scripts that have no knowledge of OMERO.

Items discussed included:

• How to get RAPID into OMERO; we decided that we needed to create a service for RAPID that would allow users to see scripts, upload script, monitor jobs and submit the jobs to a cluster.
• How do we describe the parameters of a job.
  • There are already methods in OMERO for defining script parameters, and insight can process these methods, we to will use the same framework.
• Should a script executable be aware of OMERO, we decided that it might be simpler for existing, centralised cluster to deploy scripts that did not have any dependancies on OMERO; these script we called OMERO agnostic. Some scripts may be OMERO aware, and so we also would like to have the facility to deploy OMERO aware scripts as well.
• Defining the general job submission structure:
  • A job will be defined in the RAPID xml format, which has methods for data staging, unstaging, job submission and HTML UI elements defined.
  • We need to extend the XML to allow more complex UI element, for instance using Processing.
• Staging/Unstaging data, how do we get data from OMERO to the script.
  • RAPID has methods for staging and unstaging data post job completion, we can use this to run user defined scripts to wrap and unwrap data from the server.
  • We discussed using the existing scripting infrastructure, to perform this part of the job submission.
  • These scripts would be handrolled.
• First point of action was to try and create the service, and rewrite the FLIM script to be OMERO agnostic. This requires:
  • The decoupling of the data staging and execution elements of the script.
    • Writing of Staging scripts to transform the OMERO data types to script data types, in this case Raw files, IRF data.
    • Writing of Unstaging script to transform the csv file data from the script to OMERO and attaching to the correct OMERO data objects(image, dataset).
  • The rewriting of the script to use local file copies, reverting the script to its original format.

[dzmacdonald]

Meeting on RAPID in OMERO 20/4

Present: Jos Koetsier, Donald MacDonald

Managing OMERO Agnostic Scripts

The level of agnosticism an be:

• Cluster and script are both OMERO agnostic.
• The Cluster is OMERO aware, the script is OMERO agnostic.
• The Cluster and script are OMERO aware.

Depending on the level that is available, affects what possible solutions we can present for staging and unstaging data.

Cluster and script are both OMERO agnostic

This means that the prestaging and post staging elements of job submission must convert the data in OMERO to a format required by the script. This presents several difficulties: There needs to be methods available for scripts to use to convert the data in OMERO to the correct format, these transformations have to occur in the OMERO server.

Cluster is OMERO Aware and script is OMERO agnostic

This means that the prestaging and post staging elements of job submission could install some scripts on the cluster to download and convert the data to a format required by the script. This could also be performed by an OMERO.fs install on the cluster.

Cluster and script are OMERO aware

This is the simplest of methods and the script deployed to the cluster is able to access the data on the OMERO server.

[dzmacdonald]

Security and Scripts

Currently the Rapid XML allows the user to specify access to the cluster in 2 different ways:

• The xml has the username and password for the cluster embedded in it.
• The xml requires the user to specify a username and password.

The first could be used by an admin user to allow job submission to a cluster without allowing users to have access to it.

The second means that each user will have their own username, this means that users could upload their own scripts to be run on the cluster as they would need to have submit permission.

[jburel]

Mark tickets as invalid. Tickets can be re-opened if required.