Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Requirement #2615 (closed)

Opened 10 years ago

Closed 10 years ago

Delete Refactoring and Performance improvements

Reported by: jamoore Owned by:
Priority: critical Milestone: OMERO-Beta4.2.1
Component: n.a. Keywords: n.a.
Cc: cxallan, jburel, wmoore, atarkowska Business Value: n.a.
Total Story Points: n.a. Roif: n.a.
Mandatory Story Points: n.a.

Description (last modified by jmoore)

See #2911 for the phase II work on this requirement and http://trac.openmicroscopy.org.uk/omero/wiki/Delete for user-level info on Delete.


Problem

The current (4.2.0) implementation of IDelete leaves much to be desired:

  • Not enough methods to transactionally handle all deletes
  • Too many objects are loaded into memory
  • Too few EventLogs are created for some deletes

Solution

The intended solution is to have a central definition of "delete specifications" which can be used directly, embedded into one another, or extended by third parties via Spring. These specifications describe the ordered steps (or "delete entries") which will be performed for every id passed to the server. Clients pass any number of "delete commands" to the server for deletion in a single transaction. If any exception is thrown during processing, then all of the deletions are rolled back. Based on client-provided options, some form of exceptions can be safely ignored, so called "SOFT" deletes. Because deletions can take some time, the service is asynchronous and provides a report method as well as a cancellation method.

Options

Each delete command can also pass a string/string map of options. The key values consists of either the absolute path to a particular type like /Image/ImageAnnotationLink/Annotation or a simple path /Annotation, which will then either modify the options for all annotations attached to images or all annotations period. The value consists of one of the option settings below:

  • HARD: default option which specifies that a path should be deleted if the user is the owner of the object or its group, or is an admin.
  • SOFT: option for paths which should be deleted if possible, but no ConstraintViolationException should be thrown if it's not.
  • FORCE: option for paths which should be deleted regardless of permissions and optimistic locks. The only thing which will prevent such a delete is a fatal DB error which will throw a FailedDeleteExcception. Note: it is not possible for non-admin users to specify the FORCE option.

Additionally for annotations, the value of the map can include namespace types which are to be excluded from the delete: "SOFT;excludes=openmicroscopy.org/example".

See also

  • The official definition of the delete specifications is available in spec.xml
  • Examples for using the new delete API are provided in all supported languages under trunk/examples/Delete
  • For information on having your delete specifications recognized by the server, see ExtendingOmero.

Attachments (1)

ImageGraph.graffle (252.5 KB) - added by cxallan 10 years ago.
Basic entity-relationship diagram for the Image graph

Download all attachments as: .zip

Change History (25)

comment:1 Changed 10 years ago by jmoore

  • Priority changed from major to critical

comment:2 Changed 10 years ago by jmoore

Notes from discussion (Jean-Marie, Josh, Chris):

  • Intro
    • Josh: looked at gateway, there seem
    • Josh: there was also something about changing the workflow.
    • Jean-Marie: no, misunderstanding. Need to better handle the feedback. (long time)
    • Josh: need to figure out how much feedback if something wasn't deleted, e.g.
    • Chris: how much deleting is happening in Insight?
    • Quite a bit is done via IUpdate.deleteObject
  • General behavior
    • Delete everything possible, or unlink if its linked more than once
    • Need to immediately free disk-space
    • Chris: How to specify the configuration?
      • Jean-Marie: only option is for "tags"
      • only need for certain kinds of comments. (tag, attachments)
      • Chris: Linked once it goes? Jean-Marie: that could be disturbing.
        • e.g. ontology term and/or attached file might have own value
    • Orphans (report of things that need to be cleaned up)
      • Binary data and Tags / attachments
      • Another window like Activity window
    • Group deletes
      • Datasets need the same functionality, so even images go
      • Image, Dataset, Project, Plate, Screen
      • bool of whether or not to delete me if I'm an orphan
    • May not need to handle something like CommentAnnotation specially
    • Things that are typically not shared (rating, comments, ...) should just be wiped out
  • Steps
    • Making a list of graphs that we want to delete (users' expectations?)
    • Encoding those in python to test against objects
    • ...
  • ...Coffee...
  • API
    • Project, Dataset, Image, Plate, Screen, ROI
    • DeleteHandle deleteObject(Class klass, List<Long> ids, DeleteParams parameters);
    • List<Boolean> isInDeleteQueue(Class klass, List<Long> ids);
    • AlreadyDeletedException
  • State
    • Getting a report of a on delete
    • Getting a notification of delete/failure/(de-)queueing
    • Managing expectations

comment:3 Changed 10 years ago by jmoore

  • Cc cxallan jburel added
  • Milestone changed from Unscheduled to OMERO-Beta4.2.1

comment:4 Changed 10 years ago by jburel

  • Cc wmoore added
  • Description modified (diff)

comment:5 Changed 10 years ago by jburel

  • Cc ome-nitpick@… added

comment:6 Changed 10 years ago by jmoore

  • Description modified (diff)

comment:7 Changed 10 years ago by jmoore

  • Description modified (diff)

comment:8 Changed 10 years ago by jburel

  • Cc ome-nitpick@… removed

Changed 10 years ago by cxallan

Basic entity-relationship diagram for the Image graph

comment:9 Changed 10 years ago by jmoore

Notes from discussion (Jean-Marie, Josh, Chris):

  • Ordered Steps
    • Queue-based service (interface)
    • Writing of fixtures (sample XML with all the objects)
    • Report-blob for later re-attachment
    • Gui stories
      • From tree, user can delete any node
      • From annotation panel, user can delete any annotations
      • ...
    • EventLogs via triggers
    • Simple API prototype (possibly table)

comment:10 Changed 10 years ago by jburel

  • Description modified (diff)

comment:11 Changed 10 years ago by jmoore

  • Description modified (diff)

comment:12 Changed 10 years ago by jburel

  • Cc atarkowska added

Looking at the possible cases, ROI measurements are file annotations so "attachements"
they should be removed when the ROI are removed.
We will need to be able to specify the NameSpace

comment:13 Changed 10 years ago by jburel

(In [7904]) added test for updated scenario (see #2615)

comment:14 Changed 10 years ago by jburel

Looking at deleting plate now (from client), we probably need to add
/PlateAcquisition? to the list of /Roots

comment:15 Changed 10 years ago by jburel

  • Description modified (diff)

comment:16 Changed 10 years ago by jburel

  • Description modified (diff)

comment:17 Changed 10 years ago by jmoore

(In [8128]) Log message improvements (faster/smarter) in BaseDeleteSpec (See #2615)

comment:18 Changed 10 years ago by jmoore

  • Description modified (diff)

comment:19 Changed 10 years ago by jmoore

(In [8159]) Adding delete Perf4j and other logging (See #2615)

comment:20 Changed 10 years ago by jmoore

(In [8181]) Various delete test changes (See #2615)

comment:21 Changed 10 years ago by jmoore

(In [8244]) DeleteException now always cancels the transation (See #2615)

comment:22 Changed 10 years ago by wmoore

  • Description modified (diff)

comment:23 Changed 10 years ago by jmoore

  • Description modified (diff)

comment:24 Changed 10 years ago by jmoore

  • Resolution set to fixed
  • Status changed from new to closed

4.2.1 released. Closing.

Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.55493 sec.)

We're Hiring!