Warning: Can't synchronize with repository "(default)" (/home/git/ome.git does not appear to be a Git repository.). Look in the Trac log for more information.
Notice: In order to edit this ticket you need to be either: a Product Owner, The owner or the reporter of the ticket, or, in case of a Task not yet assigned, a team_member"

Task #11779 (closed)

Opened 11 years ago

Closed 10 years ago

Alt. impl. of graph processing

Reported by: jamoore Owned by: mtbcarroll
Priority: critical Milestone: 5.1.0-m3
Component: Services Version: 5.0.0-beta1
Keywords: n.a. Cc: java@…, sethur2@…, k.h.gillen@…, spli@…
Resources: n.a. Referenced By: n.a.
References: n.a. Remaining Time: n.a.
Sprint: n.a.


Since the focus of much of beta3 will be graph processing, it is likely a good time to discuss an alternative implementation for processing graphs. It will be critical to judge what alternatives are possible and how long each might take to know whether or not this will be feasible for beta3.

Current "Rollback" strategy

Currently, the ome.services.graphs package uses `SAVEPOINT ...'` to mark sub-graphs which if not successful should not completely rollback the transaction. It's assumed that the overhead of these savepoints is one of the main limiting factors for DeleteI, ChgrpI, etc.

"Compiler" strategy

One alternative would be to read the graph specs and generate a list of SELECTs which will calculate whether or not to even enter into the subgraphs. If the known-to-fail UPDATEs (or DELETEs) are not called, then there'll be no reason to create a savepoint.

Change History (22)

comment:1 Changed 11 years ago by mtbcarroll

Filed #11780 in case the ideas there could help with testing here. Either way, substantial discussion is warranted.

comment:2 Changed 11 years ago by jamoore

In discussion with Mark, we discussed a few things this refactoring should achieve:

  • Improved performance over rollback
  • Added API of restarting an operation with new user feedback ("delete this annotation link...")
  • Integrating Preprocessor steps so that a DoAll is not necessary for MIFs.

comment:3 Changed 10 years ago by mtbcarroll

Conversation on Jabber:

  1. Unfortunately there's no general spec, i.e. you can't do "/%(random_class)s" and expect it to work. Would be nice if it would.
  2. At the moment, only well-defined Graphs are in spec.xml; everything else has to be deleted differently. Certainly something to look at for 5.1.0-m1 or so.

comment:4 Changed 10 years ago by jamoore

If the new implementation lowers the memory requirements sufficiently, then a full user chown/deletion would be possible. See http://www.openmicroscopy.org/community/viewtopic.php?f=4&t=747&p=13314#p13314

comment:5 Changed 10 years ago by jamoore

  • Cc sethur2@… added

comment:6 Changed 10 years ago by khgillen

  • Cc k.h.gillen@… added

comment:7 Changed 10 years ago by spli

  • Cc spli@… added

comment:8 Changed 10 years ago by mtbcarroll

comment:9 Changed 10 years ago by mtbcarroll

  • Milestone changed from 5.1.0 to 5.1.0-m3

comment:10 Changed 10 years ago by mtbcarroll

  • Status changed from new to accepted

comment:11 Changed 10 years ago by mtbcarroll

My present confusion is the data from the aclVoter bean. For instance, for working on an Image, it reports that the user may update and delete the Pixels and Fileset, but, despite having the same ownership, the five associated Jobs have the bean say that the user may neither update nor delete them.

Last edited 10 years ago by mtbcarroll (previous) (diff)

comment:12 Changed 10 years ago by mtbcarroll

Ah, SystemTypes.isSystemType is the culprit. Not wishing to risk consequences of changing this, for now I will work around it in the new traversal code.

comment:13 Changed 10 years ago by jamoore

Yes, we assume that the Job is a record of the provenance so it belongs to the system. Reviewable certainly.

comment:14 Changed 10 years ago by mtbcarroll

Good news: Switching from the "rollback" to the "compiler" strategy for a dataset with many small images makes chgrp enormously faster. (The implementation may not be quite right yet, but given that draft permissions checking is now in it's unlikely to change much in performance.)

comment:15 Changed 10 years ago by jamoore

Kudos, Mark. How many orders of magnitude are we speaking? :)

comment:16 Changed 10 years ago by mtbcarroll

Well, not clear about "orders" exactly! I gave up waiting for Chgrp after forty minutes or so; goodness knows how long it would have actually taken. Chgrp2 took less than four minutes to move a dataset with 750 images in it (at over 100 model objects per second).

(It is possible that it could get rather faster still if a slow part is loading the Hibernate objects: we do that only to determine their real mapped subclass, so some kind of caching could assist there.)

comment:17 Changed 10 years ago by jburel

  • Milestone changed from 5.1.0-m3 to 5.1.0-m4

comment:18 Changed 10 years ago by mtbcarroll

One puzzle is why the current state of my branch doesn't run faster still. After all the SELECTing and calculation, the result is a series of HQL statements constructed from the templates,

Delete2I: "DELETE FROM " + className + " WHERE id IN (:ids)"
Chgrp2I: "UPDATE " + className + " SET details.group = :group WHERE id IN (:ids)"
Chown2I: "UPDATE " + className + " SET details.owner = :user WHERE id IN (:ids)"

where each one seems to take a fair while to actually execute so the full list of them can take many seconds. So far I haven't been able to find out why they take so long.

comment:19 Changed 10 years ago by jamoore

Mark: how large are the (:ids) collections? PostgreSQL historically has had quite poor performance with them.

comment:20 Changed 10 years ago by mtbcarroll

Good question. Upon further investigation I've tracked down the issue; it turns out to continue a previous thread on the Trello card so I'll add more information there.

comment:21 Changed 10 years ago by mtbcarroll

  • Milestone changed from 5.1.0-m4 to 5.1.0-m3

comment:22 Changed 10 years ago by mtbcarroll

  • Resolution set to fixed
  • Status changed from accepted to closed

Fixed by https://github.com/openmicroscopy/openmicroscopy/pull/3228. Further delete performance work happening on #11084.

Note: See TracTickets for help on using tickets. You may also have a look at Agilo extensions to the ticket.

1.3.13-PRO © 2008-2011 Agilo Software all rights reserved (this page was served in: 0.67164 sec.)

We're Hiring!