Update indexer progress during batch processing

[jballanco-x]

Currently, Indexer worker threads process events in batches and only update their progress as a batch completes. Instead, we should update the progress marker as each event is processed so that, should a worker thread encounter an unrecoverable error during batch processing, the entire batch does not need to be reprocessed.

[jballanco-x]

The strategy we'll take for resolving this issue will be to record (in a file on disk) the last event ID that was attempted to be indexed, and how many attempts have been made. If more than a preconfigured number of attempts have been made, then indexing for the event in question will be skipped in favor of continuing to process later events in the event stream, and a warning will be logged.

[jballanco-x]

Pull request sent: ​https://github.com/openmicroscopy/openmicroscopy/pull/2086

However, an issue remains. Since the event currently being processed is recorded as the event is pulled off of the log, we never get the chance to retry events that failed to process.

[jamoore]

The only thing I can think of is:

version=1
id=100
retries=0
state=inprogress

followed by

version-1
id=100
retries=0
state=failed

or similar. But perhaps that's getting too complicated.

[jballanco-x]

The problem stems more from the fact that the EventLogLoader is only involved in pulling out events for processing. Failures have been typically occurring inside of the loop processing retrieved events, in FullTextIndexer#doIndexing. So, we'd have to add code that deals with the event ID log to FullTextIndexer? as well, breaking through what little abstraction we currently have around pulling events out of the backlog...

[jamoore]

That makes sense. So a "failed" state, fundamentally, doesn't make sense. Whatever ID is in the file on startup, then, needs to be considered "inprogress", right? Unless, perhaps, on shutdown, a marker was set to say that all was well? Perhaps simple a close() on the log loader API would suffice?

[jballanco-x]

Referencing ticket #11936 has changed sprint.

[jballanco-x]

Updated strategy described in pull-request...still not a complete solution, however.

[spli]

Excerpt from gretzky (dev_5_0 merge) Indexer-0.log this morning

[spli]

I'm not sure if you received a notification- I've attached part of the log.

[jballanco-x]

Thanks!

[jballanco-x]

Referencing ticket #11936 has changed sprint.

[jamoore]

Pushing.

[jamoore]

Better or at least easier solution will likely be to have an event log loader which better handles failures by:

• not throwing on exceptions
• retrying failures
• adding annotations to failed objects

At the same time, a partitioning query will be used to reduce the total number of objects indexed.

[jamoore]

PR opened: ​https://github.com/openmicroscopy/openmicroscopy/pull/2639

Errors will be marked with annotations and the indexer will continue.

[jmoore <josh@…>]

(In [fb6eec1d9aa778c4a2c89b54ffb80e4bf558d246/ome.git] on branch develop) EventLogQueue?: new persistent event log loader (See #11948)

This PELL should significantly improve the performance of
the background indexer. Making use of the new partition
method in SqlAction, it should not re-try multiple objects.
And by adding a failure queue, it should not get stuck when
an object is unindexable.