FullText BackLog can get stuck

[jamoore]

Under certain conditions (e.g. custom SearchBridges), the indexing backlog (in EventLogLoader) can be continually filled. The List<EventLog> currently in use should be replaced with something smarter.

[jmoore]

This consists of two issues:

1. The first is that there needs to be a test for EventLog equivalency in the backlog. This is fairly straight-forward and requires adding our own collection type which checks on add(EventLog).

2. The other problem is the possibility of recursion in bridges. For example, if a custom bridge wants to index dataset names with contained images and image names with the containing datasets, then to keep the index in sync, it is necessary to call reindex(Dataset) while indexing an Image. However, if reindex(Image) is also called while indexing the Dataset, then the indexing runs perpetually. The planned solution is to disallow adding any objects to the backlog while clearing the backlog. This means that a bridge should be careful to add ALL objects to the backlog which should be processed for a given object and not rely on transitivity. (i.e. if image names should also be indexed with projects, then the bridge should call reindex(Dataset) and reindex(Project). This will be documented under SearchBridges.

[jmoore]

r2796 contains this fix. This needs some testing, but this will most likely have to happen in non-standard builds, since there are no cases of recursive reindex()ing in trunk. (The only objects which are reindex()ed are the parents of an AnnotationLink).

Closing for now, and will report back from the ​Glencoe use of this code.

[jmoore]

Apologies, r2798 contains the fix.

[jmoore]

Grrr...and r2800 contains the actual implementation of EventBacklog

[jmoore]

r2804 adds better logging. Some bridges can still get stuck. Re-opening.

[jmoore]

r2805 has a possible fix.

[jmoore]

3. Another issue with the EventLogLoader/EventBacklog pair is even more fundamental: nothing gets added to the backlog until flush time. This uniquely Hibernate behavior completely changes how these classes have to function. They were designed with the assumption that:

  for (EventLog log : loader) {
     flush(log); // <-- backlog added here
  }

but it's much more:

  for (EventLog log : loader) {
     flush(log); // no-op
  }
  commit(); // <-- backlog added here!

[jmoore]

r2806 contains the changes to the state machine. Each batch can now only either be in eventlog or in backlog state. Not both. While the backlog is being worked on, nothing can be added to the backlog, which is why SearchBridges must be careful to add everything which needs to be synchronized in one shot.

One further fallout of this change is that some batches may be significantly smaller than the set batch size.