RFE: Lucene to index Original Metadata

[wmoore]

I don't seem to get any image results when I search by field :annotation.

Tried this manually using current Insight search, only using 'annotation' checked.

Also in my own web code I see the same.

Wrote a small Python script that reads every key, value pair from image.loadOriginalMetadata() using every value as a search term with annotation field, and I get no results from any.
This was using my current Blitz Gateway code, so not valid to add here as a test yet, but I'll prepare this using the search service directly.

[jamoore]

There are a couple of things here that we'll likely need to break down:

1) I just did the following on octopus:

• added a tag "e74eed78e043" to ​http://octopus.openmicroscopy.org/merge/webclient/?show=image-21006 as user-1
• opened a immediately and ran successfully:

s = client.sf.createSearchService()
s.onlyType("Image")
s.byFullText("e74eed78e043")
s.hasNext()
s.next().id.val
s.byFullText("annotation:e74eed78e043")
s.hasNext()
s.next().id.val

If you're seeing issues, we're likely going to have to be very specific. A failing test would be best.

2) None of the original metadata is currently parsed into the full text search. For me this falls under the "testing" banner that we mentioned. Somewhere we need an encoding of what you're expecting, but that probably needs a single ticket or perhaps even a story and probably a good deal of discussion. Note: each time we update what needs to be in the index, we will have to re-index the entire database, so if we can get a good estimation early on, our lives will be much simpler. I'm currently working on something that will hopefullly speed up indexing, but it will still take up to days on some of our DBs.

[wmoore]

OK, indexing of Original Metadata will have to wait, but I would imagine that companion files, E.g. DV log files, would be indexed.

[jamoore]

This is currently not happening and so would require a server-side change and a full re-indexing.

[jamoore]

This really should be quite straight-forward to do. The only question is what format would you/we/one expect? Just the original metadata as it was formatted in 4.4, i.e. like an .ini file? Each value as a key/value pair? With some prefix?

[jamoore]

Probably best to do along with #12318 (mapannotation index)

[jamoore]

Pushing beyond 5.1.