Make PixelData-0 encoding multi-threaded

[cxallan]

There are several ways to have more through-put in the pixel data system.

1. Run multiple PixelData processes: long-term this option would be beneficial to have more protection between the individual runs. However, at the moment, each process needs a full Hibernate SessionFactory which amounts to 512M-1G of overhead. There may also be some contention on the eventlog loading.
2. Have each PixelData process be multi-threaded: this is a fairly straight-forward change which should probably come first. One caveat is the error-handling. If a single thread either throws some specific exception or brings down the whole service, all concurrently processed pyramids may have errors.
3. Have the makePyramid call be (more) multi-threaded: though more complicated to implement, having a multi-threaded call deeper in the stack will better pipeline IO.

[cxallan]

Referencing ticket #6220 has changed sprint.

[jmoore]

Currently pushed a version of solution 2 to ​https://github.com/joshmoore/openmicroscopy/tree/mt-pixeldata. Colin: if possible, it would be good to make sure this is generally working and have it in sprint6-server-backlog so we can do more testing on ome-pfelts.

[jmoore]

Updated branch with better single-user MT-handling.

[jburel]

Moved from sprint 2012-01-17 (6)

[jmoore <josh@…>]

(In [8aebe44db417c946b50ab2ea8d8b89512dc365b3/ome.git] on branch develop) Increase each-user throughput in MT PIXELDATA (See #6224)

The previous implementation would take at most one event log
per user even if there is only one user active. Now, up to at
least numThreads event logs are taken regardless of what user.

[jmoore <josh@…>]

(In [fe47c714ccec44b40e535c5b2da3586d505e9f6e/ome.git] on branch develop) Fix missing 'group by el.entityid' for PG 9.0.4 (See #6224)

[jmoore]

Current implementation (solution 2) seems to be working. The major caveats are that:

1. PG 8.3 support has been (temporarily) dropped. See #7813
2. for each batch, there will be some down-time per thread. For example, if there are 2 threads (the default) for pixel data creation and one big image takes 10 minutes and the other only 1 minute, the second thread will have 9 minutes of down-time. There are various ways to fix this, but currently there is an issue with transaction safety and the storage of the "currentEventLogId". A re-working of the general eventlog handling system would make this much easier.

Taking out of "in-progress". If one of the two caveats is too serious to be ignored for the moment, let me know.

[jmoore <josh@…>]

(In [105d4074a9f327fc835e96f424f61476cc43a3d4/ome.git] on branch develop) Properly handle pixeldata dupe eventlogs (See #6224)

Having two row_number functions at the same subquery level
led to missed events. Pulled the dupe up a level.

[jmoore <josh@…>]

(In [516c05f317f46dfd2f5d18bf36b0884dfb2d3c77/ome.git] on branch develop) Fix cut-n-paste error of SqlAction string (See #6224)

commit 105d4074a9f327fc835e96f424f61476cc43a3d4 was
incorrectly copied from the pfelts box.

[jmoore]

New development required that won't get done in 4.4.0. Pushing (and considering that we'll be dropping 8.3 support with #7813 not as urgent)

[sbesson]

Referencing ticket #6220 has changed sprint.

[sbesson]

Referencing ticket #6220 has changed sprint.