Introduce ImportSet to API

[jamoore]

Currently the hierarchy is:

  /Fileset
     -> /OriginalFile collection
     -> /Image collection

The issue is that on import clients may not know what all is contained in the fileset, and therefore there will need to be a "pulling apart" (or the use of a DirectoryReader in Bio-Formats which reads everything a la ImportCandidates).

If the import action of the clients creates a ImportSet rather than a Fileset, there will be fewer overall top-level containers, and the server can adjust the Filesets as necessary. (NB: It would be important the clients could delete a single fileset from that import set)

The new hierarchy might look something like:

  /ImportSet
     -> /OriginalFile collection
     -> /Fileset collection
           -> /Image collection
           -> /OriginalFile collection (reused pointers?)

Jean-Marie has raised the issue that we will need to also examine how we deal with multi-image filesets, possibly linking Fileset under Dataset, though this cannot be done for the demo. In which case, we may want to omit ImportSet (i.e. this ticket) as well.

Note: while introducing this API, importFileset or similar should also be made into an async request, since for Plates it can take *significant* time.

[jmoore]

Not included in fs demo 2 tag. Removing from sprint

[jburel]

Need to create a new ticket to examine the concept of possibly linking Fileset under Dataset, before closing that task.

[mtbcarroll]

ManagedRepositoryI.replacementMap's importSetId term requires correction

[jamoore]

mtbcarroll: agreed. My thinking was to use a nicely formatted timestamp, since "import-set" will become just a API concept (as opposed to a model concept).

[mtbcarroll]

Okay, addressed the timestamp issue under ​https://github.com/openmicroscopy/openmicroscopy/pull/998

[jamoore]

in-progress

[jamoore]

Pushing to "cleanup" bit of next demo.

[jamoore]

This is my remaining cleanup tasks. It needs to be in-place for the UI/workflow work, and will continue to become less and less rebase-able.

[jburel]

I understood the idea of importSet was no longer in the pipeline. So we will have ImportSet and FileSet.

[jamoore]

ImportSet is only being introduced at the API level. I.e. you can upload a bunch of files and they will turn into multiple filesets. Otherwise, these filesets are unrelated (i.e. no import set object in the DB)

[jamoore]

Didn't have time to work on this. Will do so while the import UI is being re-organized.

[jburel]

Moved from sprint FS demo 4.2

[jamoore]

This will require a lot of coordination and will likely de-stabilize initially. Pushing.

[jamoore]

Pushed branch with origin/develop merged in to ​https://github.com/joshmoore/openmicroscopy/tree/10325-import-set-api

Next steps:

• Finish implementations and final API modifications
• Merge in Blazej's branch from ​https://github.com/openmicroscopy/openmicroscopy/pull/1679 once merged.

[jamoore]

Referencing ticket #10310 has changed sprint.

[jamoore]

API / DB changes should be out asap.

[jamoore]

Would definitely be a nice to have for 5.1 but I'm not optimistic that it will happen.

[mtbcarroll]

"Priority: critical": aim for 5.2 or perhaps push further? Happy to try to help with this if I can.