Alt. impl. of graph processing

[jamoore]

Since the focus of much of beta3 will be graph processing, it is likely a good time to discuss an alternative implementation for processing graphs. It will be critical to judge what alternatives are possible and how long each might take to know whether or not this will be feasible for beta3.

Current "Rollback" strategy

Currently, the ome.services.graphs package uses ​`SAVEPOINT ...'` to mark sub-graphs which if not successful should not completely rollback the transaction. It's assumed that the overhead of these savepoints is one of the main limiting factors for DeleteI, ChgrpI, etc.

"Compiler" strategy

One alternative would be to read the graph specs and generate a list of SELECTs which will calculate whether or not to even enter into the subgraphs. If the known-to-fail UPDATEs (or DELETEs) are not called, then there'll be no reason to create a savepoint.

[mtbcarroll]

Filed #11780 in case the ideas there could help with testing here. Either way, substantial discussion is warranted.

[jamoore]

In discussion with Mark, we discussed a few things this refactoring should achieve:

• Improved performance over rollback
• Added API of restarting an operation with new user feedback ("delete this annotation link...")
• Integrating Preprocessor steps so that a DoAll is not necessary for MIFs.

[mtbcarroll]

Conversation on Jabber:

1. Unfortunately there's no general spec, i.e. you can't do "/%(random_class)s" and expect it to work. Would be nice if it would.
2. At the moment, only well-defined Graphs are in spec.xml; everything else has to be deleted differently. Certainly something to look at for 5.1.0-m1 or so.

[jamoore]

If the new implementation lowers the memory requirements sufficiently, then a full user chown/deletion would be possible. See http://www.openmicroscopy.org/community/viewtopic.php?f=4&t=747&p=13314#p13314

[mtbcarroll]

Progress noted in ​https://trello.com/c/sUeLuByS/.

[mtbcarroll]

My present confusion is the data from the aclVoter bean. For instance, for working on an Image, it reports that the user may update and delete the Pixels and Fileset, but despite having the same ownership, the five associated Jobs have the bean say that the user may neither update nor delete them.

[mtbcarroll]

Ah, SystemTypes.isSystemType is the culprit. Not wishing to risk consequences of changing this, for now I will work around it in the new traversal code.

[jamoore]

Yes, we assume that the Job is a record of the provenance so it belongs to the system. Reviewable certainly.

[mtbcarroll]

Good news: Switching from the "rollback" to the "compiler" strategy for a dataset with many small images makes chgrp enormously faster. (The implementation may not be quite right yet, but given that draft permissions checking is now in it's unlikely to change much in performance.)

[jamoore]

Kudos, Mark. How many orders of magnitude are we speaking? :)

[mtbcarroll]

Well, not clear about "orders" exactly! I gave up waiting for Chgrp after forty minutes or so; goodness knows how long it would have actually taken. Chgrp2 took less than four minutes to move a dataset with 750 images in it (at over 100 model objects per second).

(It is possible that it could get rather faster still if a slow part is loading the Hibernate objects: we do that only to determine their real mapped subclass, so some kind of caching could assist there.)

[mtbcarroll]

One puzzle is why the current state of my branch doesn't run faster still. After all the SELECTing and calculation, the result is a series of HQL statements constructed from the templates,

Delete2I: "DELETE FROM " + className + " WHERE id IN (:ids)"
Chgrp2I: "UPDATE " + className + " SET details.group = :group WHERE id IN (:ids)"
Chown2I: "UPDATE " + className + " SET details.owner = :user WHERE id IN (:ids)"

where each one seems to take a fair while to actually execute so the full list of them can take many seconds. So far I haven't been able to find out why they take so long.

[jamoore]

Mark: how large are the (:ids) collections? PostgreSQL historically has had quite poor performance with them.

[mtbcarroll]

Good question. Upon further investigation I've tracked down the issue; it turns out to continue a previous thread on the Trello card so I'll add more information there.

[mtbcarroll]

Fixed by ​https://github.com/openmicroscopy/openmicroscopy/pull/3228. Further delete performance work happening on #11084.