Warning: Can't synchronize with repository "(default)" (/home/git/ome.git does not appear to be a Git repository.). Look in the Trac log for more information.
Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Requirement #2615 (new)

Opened 14 years ago

Last modified 13 years ago

Delete Refactoring and Performance improvements — at Version 22

Reported by: jamoore Owned by:
Priority: critical Milestone: OMERO-Beta4.2.1
Component: n.a. Keywords: n.a.
Cc: cxallan, jburel, wmoore, atarkowska Business Value: n.a.
Total Story Points: n.a. Roif: n.a.
Mandatory Story Points: n.a.

Description (last modified by wmoore)

See #2911 for the phase II work on this requirement and http://trac.openmicroscopy.org.uk/omero/wiki/Delete for user-level info on Delete.


The current implementation of IDelete leaves much to be desired:

  • Not enough methods to transactionally handle all deletes
  • Too many objects are loaded into memory
  • Too few EventLogs are created for some deletes

Notation

Below are listed several possible workflows for what a user might want to delete. Each scenario provides the following information:

  • REQUEST: the user request for what type should be deleted (ids are omitted). Types are represented as xpath-like statements, some which don't actually exist as types "Comments" represent a combination of fields, like type=Annotation + namespace="comment".
  • DELETES: the types which would be deleted regardless of permissions and optimistic locks. The only thing which will prevent such a delete is a fatal DB error which will throw a FailedDeleteExcception
  • OPTIONS: the paths listed under "OPTIONS" can be handled in on of the following ways:
    • ORPHAN - the objects are left alone, i.e. unlinked from the target.
    • REAP - delete the objects if they would be orphaned.
    • SOFT - delete the objects, but continue if delete is not possible.
    • HARD - delete the objects at all costs. Transaction will fail if not possible.
  • BLOCKER: if this path is filled, then the delete will fail with a BlockedDeleteException

Types

Currently types are separated into two categories. When providing reports to users it should suffice to start at these roots.

/Roots

  • /Project and /Dataset
  • /Screen and /Plate, /PlateAcquisition?
  • /Image (includes Pixels, binary data, etc.)
  • /Roi (if not already under /Images)
  • /FS - each FS mount is also a root
  • Note: Delete TagSet. TagSet is, from a user perspective, a way of grouping tags. When the user selects the TagSet, he/she expects an object to be tagged with all the tags "contained" in the TagSet. When deleting a TagSet, we should follow the same approach i.e. if the user selects a TagSet, all the Tags "contained" in the TagSet should also be deleted.

/Orphanable

NB: We may want to consider whether or not there should be two types of annotations. Ones which are orphanable and ones which are not.

Pseudo-Scenarios

Note: the official definition of the scenarios is now available in spec.xml

REQUEST:    /Root/Images
DELETES:    /Root/Images/ImageAnnotationLinks
            /Root/Images/DatasetImageLinks
            /Root/Images/WellSamples  (???)
            /Root/Images/Acquisitions
            /Root/Images/Rois
            /Root/Images/{Comments, Ratings, ...}
            /Root/Images/Renderings
            /Root/Images/Thumbnails
OPTIONS:    /Root/Images/Tags
            /Root/Images/Terms
            /Root/Images/FileAnnotations
REQUEST:    /Root/Datasets
DELETES:    /Root/Datasets/DatasetImageLinks
            /Root/Datasets/DatasetAnnotationLinks
OPTIONS:    /Root/Datasets/Tags
            /Root/Datasets/Terms
            /Root/Datasets/FileAnnotations
            /Root/Datasets/Images - option to delete contents (similar to Eclipse)
            + all options from /Root/Images
REQUEST:    /Root/Rois
DELETES:    /Root/Rois/Shapes
            /Root/Rois/Shapes/Masks
BLOCKER:    /Root/Images/Rois


Change History (23)

comment:1 Changed 14 years ago by jmoore

  • Priority changed from major to critical

comment:2 Changed 14 years ago by jmoore

Notes from discussion (Jean-Marie, Josh, Chris):

  • Intro
    • Josh: looked at gateway, there seem
    • Josh: there was also something about changing the workflow.
    • Jean-Marie: no, misunderstanding. Need to better handle the feedback. (long time)
    • Josh: need to figure out how much feedback if something wasn't deleted, e.g.
    • Chris: how much deleting is happening in Insight?
    • Quite a bit is done via IUpdate.deleteObject
  • General behavior
    • Delete everything possible, or unlink if its linked more than once
    • Need to immediately free disk-space
    • Chris: How to specify the configuration?
      • Jean-Marie: only option is for "tags"
      • only need for certain kinds of comments. (tag, attachments)
      • Chris: Linked once it goes? Jean-Marie: that could be disturbing.
        • e.g. ontology term and/or attached file might have own value
    • Orphans (report of things that need to be cleaned up)
      • Binary data and Tags / attachments
      • Another window like Activity window
    • Group deletes
      • Datasets need the same functionality, so even images go
      • Image, Dataset, Project, Plate, Screen
      • bool of whether or not to delete me if I'm an orphan
    • May not need to handle something like CommentAnnotation specially
    • Things that are typically not shared (rating, comments, ...) should just be wiped out
  • Steps
    • Making a list of graphs that we want to delete (users' expectations?)
    • Encoding those in python to test against objects
    • ...
  • ...Coffee...
  • API
    • Project, Dataset, Image, Plate, Screen, ROI
    • DeleteHandle deleteObject(Class klass, List<Long> ids, DeleteParams parameters);
    • List<Boolean> isInDeleteQueue(Class klass, List<Long> ids);
    • AlreadyDeletedException
  • State
    • Getting a report of a on delete
    • Getting a notification of delete/failure/(de-)queueing
    • Managing expectations

comment:3 Changed 14 years ago by jmoore

  • Cc cxallan jburel added
  • Milestone changed from Unscheduled to OMERO-Beta4.2.1

comment:4 Changed 14 years ago by jburel

  • Cc wmoore added
  • Description modified (diff)

comment:5 Changed 14 years ago by jburel

  • Cc ome-nitpick@… added

comment:6 Changed 14 years ago by jmoore

  • Description modified (diff)

comment:7 Changed 14 years ago by jmoore

  • Description modified (diff)

comment:8 Changed 14 years ago by jburel

  • Cc ome-nitpick@… removed

Changed 14 years ago by cxallan

Basic entity-relationship diagram for the Image graph

comment:9 Changed 14 years ago by jmoore

Notes from discussion (Jean-Marie, Josh, Chris):

  • Ordered Steps
    • Queue-based service (interface)
    • Writing of fixtures (sample XML with all the objects)
    • Report-blob for later re-attachment
    • Gui stories
      • From tree, user can delete any node
      • From annotation panel, user can delete any annotations
      • ...
    • EventLogs via triggers
    • Simple API prototype (possibly table)

comment:10 Changed 14 years ago by jburel

  • Description modified (diff)

comment:11 Changed 14 years ago by jmoore

  • Description modified (diff)

comment:12 Changed 14 years ago by jburel

  • Cc atarkowska added

Looking at the possible cases, ROI measurements are file annotations so "attachements"
they should be removed when the ROI are removed.
We will need to be able to specify the NameSpace

comment:13 Changed 14 years ago by jburel

(In [7904]) added test for updated scenario (see #2615)

comment:14 Changed 14 years ago by jburel

Looking at deleting plate now (from client), we probably need to add
/PlateAcquisition? to the list of /Roots

comment:15 Changed 14 years ago by jburel

  • Description modified (diff)

comment:16 Changed 14 years ago by jburel

  • Description modified (diff)

comment:17 Changed 14 years ago by jmoore

(In [8128]) Log message improvements (faster/smarter) in BaseDeleteSpec (See #2615)

comment:18 Changed 14 years ago by jmoore

  • Description modified (diff)

comment:19 Changed 14 years ago by jmoore

(In [8159]) Adding delete Perf4j and other logging (See #2615)

comment:20 Changed 13 years ago by jmoore

(In [8181]) Various delete test changes (See #2615)

comment:21 Changed 13 years ago by jmoore

(In [8244]) DeleteException now always cancels the transation (See #2615)

comment:22 Changed 13 years ago by wmoore

  • Description modified (diff)
Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.144596 sec.)

We're Hiring!