Delete Refactoring and Performance improvements

[jamoore]

The current implementation of IDelete leaves much to be desired:

• Not enough methods to transactionally handle all deletes
• Too many objects are loaded into memory
• Too few EventLogs are created for some deletes

Notation

Below are listed several possible workflows for what a user might want to delete. Each scenario provides the following information:

• REQUEST: the user request for what type should be deleted (ids are omitted). Types are represented as xpath-like statements, some which don't actually exist as types "Comments" represent a combination of fields, like type=Annotation + namespace="comment".
• DELETES: the types which would be deleted regardless of permissions and optimistic locks. The only thing which will prevent such a delete is a fatal DB error which will throw a FailedDeleteExcception
• OPTIONS: the paths listed under "OPTIONS" can be handled in on of the following ways:
  • ORPHAN - the objects are left alone, i.e. unlinked from the target.
  • REAP - delete the objects if they would be orphaned.
  • SOFT - delete the objects, but continue if delete is not possible.
  • HARD - delete the objects at all costs. Transaction will fail if not possible.
• BLOCKER: if this path is filled, then the delete will fail with a BlockedDeleteException

Types

Currently types are separated into two categories. When providing reports to users it should suffice to start at these roots.

/Roots

• /Projects and /Datasets
• /Screens and /Plates
• /Images (includes Pixels, binary data, etc.)
• /Rois (if not already under /Images)
• /FS - each FS mount is also a root

• Note: Delete TagSet. TagSet is, from a user perspective, a way of grouping tags. When the user selects the TagSet, he/she expects an object to be tagged with all the tags "contained" in the TagSet. When deleting a TagSet, we should follow the same approach i.e. if the user selects a TagSet, all the Tags "contained" in the TagSet should also be deleted.

/Orphanable

• /Tags
• /Terms
• /FileAnnotations?

NB: We may want to consider whether or not there should be two types of annotations. Ones which are orphanable and ones which are not.

Pseudo-Scenarios

Note: the official definition of the scenarios is now available in spec.xml

REQUEST:    /Root/Images
DELETES:    /Root/Images/ImageAnnotationLinks
            /Root/Images/DatasetImageLinks
            /Root/Images/WellSamples  (???)
            /Root/Images/Acquisitions
            /Root/Images/Rois
            /Root/Images/{Comments, Ratings, ...}
            /Root/Images/Renderings
            /Root/Images/Thumbnails
OPTIONS:    /Root/Images/Tags
            /Root/Images/Terms
            /Root/Images/FileAnnotations

REQUEST:    /Root/Datasets
DELETES:    /Root/Datasets/DatasetImageLinks
            /Root/Datasets/DatasetAnnotationLinks
OPTIONS:    /Root/Datasets/Tags
            /Root/Datasets/Terms
            /Root/Datasets/FileAnnotations
            /Root/Datasets/Images - option to delete contents (similar to Eclipse)
            + all options from /Root/Images

REQUEST:    /Root/Rois
DELETES:    /Root/Rois/Shapes
            /Root/Rois/Shapes/Masks
BLOCKER:    /Root/Images/Rois

[jmoore]

Notes from discussion (Jean-Marie, Josh, Chris):

• Intro
  • Josh: looked at gateway, there seem
  • Josh: there was also something about changing the workflow.
  • Jean-Marie: no, misunderstanding. Need to better handle the feedback. (long time)
  • Josh: need to figure out how much feedback if something wasn't deleted, e.g.
  • Chris: how much deleting is happening in Insight?
  • Quite a bit is done via IUpdate.deleteObject

• General behavior
  • Delete everything possible, or unlink if its linked more than once
  • Need to immediately free disk-space
  • Chris: How to specify the configuration?
    • Jean-Marie: only option is for "tags"
    • only need for certain kinds of comments. (tag, attachments)
    • Chris: Linked once it goes? Jean-Marie: that could be disturbing.
      • e.g. ontology term and/or attached file might have own value
  • Orphans (report of things that need to be cleaned up)
    • Binary data and Tags / attachments
    • Another window like Activity window
  • Group deletes
    • Datasets need the same functionality, so even images go
    • Image, Dataset, Project, Plate, Screen
    • bool of whether or not to delete me if I'm an orphan
  • May not need to handle something like CommentAnnotation specially
  • Things that are typically not shared (rating, comments, ...) should just be wiped out

• Steps
  • Making a list of graphs that we want to delete (users' expectations?)
  • Encoding those in python to test against objects
  • ...

• ...Coffee...

• API
  • Project, Dataset, Image, Plate, Screen, ROI
  • DeleteHandle deleteObject(Class klass, List<Long> ids, DeleteParams parameters);
  • List<Boolean> isInDeleteQueue(Class klass, List<Long> ids);
  • AlreadyDeletedException

• State
  • Getting a report of a on delete
  • Getting a notification of delete/failure/(de-)queueing
  • Managing expectations

[cxallan]

Basic entity-relationship diagram for the Image graph

[jmoore]

Notes from discussion (Jean-Marie, Josh, Chris):

• Ordered Steps
  • Queue-based service (interface)
  • Writing of fixtures (sample XML with all the objects)
    • also use reports to see what was imported?
    • way to remember to re-evaluate tests, say, when model is changed
    • re-use loci code
    • ​http://www.loci.wisc.edu/trac/java/browser/trunk/components/bio-formats/test/loci/formats/utests
    • create a new suite for delete
  • Report-blob for later re-attachment
  • Gui stories
    • From tree, user can delete any node
    • From annotation panel, user can delete any annotations
    • ...
  • EventLogs via triggers
  • Simple API prototype (possibly table)