Delete Refactoring and Performance improvements

[jamoore]

See #2911 for the phase II work on this requirement and ​http://trac.openmicroscopy.org.uk/omero/wiki/Delete for user-level info on Delete.

---

Problem

The current (4.2.0) implementation of IDelete leaves much to be desired:

• Not enough methods to transactionally handle all deletes
• Too many objects are loaded into memory
• Too few EventLogs are created for some deletes

Solution

The intended solution is to have a central definition of "delete specifications" which can be used directly, embedded into one another, or extended by third parties via Spring. These specifications describe the ordered steps (or "delete entries") which will be performed for every id passed to the server. Clients pass any number of "delete commands" to the server for deletion in a single transaction. If any exception is thrown during processing, then all of the deletions are rolled back. Based on client-provided options, some form of exceptions can be safely ignored, so called "SOFT" deletes. Because deletions can take some time, the service is asynchronous and provides a report method as well as a cancellation method.

Options

Each delete command can also pass a string/string map of options. The key values consists of either the absolute path to a particular type like /Image/ImageAnnotationLink/Annotation or a simple path /Annotation, which will then either modify the options for all annotations attached to images or all annotations period. The value consists of one of the option settings below:

• HARD: default option which specifies that a path should be deleted if the user is the owner of the object or its group, or is an admin.
• SOFT: option for paths which should be deleted if possible, but no ConstraintViolationException should be thrown if it's not.
• FORCE: option for paths which should be deleted regardless of permissions and optimistic locks. The only thing which will prevent such a delete is a fatal DB error which will throw a FailedDeleteExcception. Note: it is not possible for non-admin users to specify the FORCE option.

Additionally for annotations, the value of the map can include namespace types which are to be excluded from the delete: "SOFT;excludes=openmicroscopy.org/example".

See also

• The official definition of the delete specifications is available in spec.xml
• Examples for using the new delete API are provided in all supported languages under trunk/examples/Delete
• For information on having your delete specifications recognized by the server, see ExtendingOmero.

[jmoore]

Notes from discussion (Jean-Marie, Josh, Chris):

• Intro
  • Josh: looked at gateway, there seem
  • Josh: there was also something about changing the workflow.
  • Jean-Marie: no, misunderstanding. Need to better handle the feedback. (long time)
  • Josh: need to figure out how much feedback if something wasn't deleted, e.g.
  • Chris: how much deleting is happening in Insight?
  • Quite a bit is done via IUpdate.deleteObject

• General behavior
  • Delete everything possible, or unlink if its linked more than once
  • Need to immediately free disk-space
  • Chris: How to specify the configuration?
    • Jean-Marie: only option is for "tags"
    • only need for certain kinds of comments. (tag, attachments)
    • Chris: Linked once it goes? Jean-Marie: that could be disturbing.
      • e.g. ontology term and/or attached file might have own value
  • Orphans (report of things that need to be cleaned up)
    • Binary data and Tags / attachments
    • Another window like Activity window
  • Group deletes
    • Datasets need the same functionality, so even images go
    • Image, Dataset, Project, Plate, Screen
    • bool of whether or not to delete me if I'm an orphan
  • May not need to handle something like CommentAnnotation specially
  • Things that are typically not shared (rating, comments, ...) should just be wiped out

• Steps
  • Making a list of graphs that we want to delete (users' expectations?)
  • Encoding those in python to test against objects
  • ...

• ...Coffee...

• API
  • Project, Dataset, Image, Plate, Screen, ROI
  • DeleteHandle deleteObject(Class klass, List<Long> ids, DeleteParams parameters);
  • List<Boolean> isInDeleteQueue(Class klass, List<Long> ids);
  • AlreadyDeletedException

• State
  • Getting a report of a on delete
  • Getting a notification of delete/failure/(de-)queueing
  • Managing expectations

[cxallan]

Basic entity-relationship diagram for the Image graph

[jmoore]

Notes from discussion (Jean-Marie, Josh, Chris):

• Ordered Steps
  • Queue-based service (interface)
  • Writing of fixtures (sample XML with all the objects)
    • also use reports to see what was imported?
    • way to remember to re-evaluate tests, say, when model is changed
    • re-use loci code
    • ​http://www.loci.wisc.edu/trac/java/browser/trunk/components/bio-formats/test/loci/formats/utests
    • create a new suite for delete
  • Report-blob for later re-attachment
  • Gui stories
    • From tree, user can delete any node
    • From annotation panel, user can delete any annotations
    • ...
  • EventLogs via triggers
  • Simple API prototype (possibly table)

[jburel]

Looking at the possible cases, ROI measurements are file annotations so "attachements"
they should be removed when the ROI are removed.
We will need to be able to specify the NameSpace

[jburel]

(In [7904]) added test for updated scenario (see #2615)

[jburel]

Looking at deleting plate now (from client), we probably need to add
/PlateAcquisition? to the list of /Roots

[jmoore]

(In [8128]) Log message improvements (faster/smarter) in BaseDeleteSpec (See #2615)

[jmoore]

(In [8159]) Adding delete Perf4j and other logging (See #2615)

[jmoore]

(In [8181]) Various delete test changes (See #2615)

[jmoore]

(In [8244]) DeleteException now always cancels the transation (See #2615)

[jmoore]

4.2.1 released. Closing.