BUG: Searching returns no results for wildcard searches

[atarkowska]

This is caused by the FullTextAnalyzer (ticket:1010) not being used for wildcard searches:

• ​http://wiki.apache.org/lucene-java/LuceneFAQ#What_wildcard_search_support_is_available_from_Lucene.3F
• ​http://jira.atlassian.com/browse/JRA-15006
• ​http://stackoverflow.com/questions/2432486/lucene-wildcard-queries
• ​http://markmail.org/message/nfpoofjlyu4fkgcl#query:+page:1+mid:bkbjgtjvp5xz5igo+state:results

There's a proposed workaround in lucene-misc-2.4.1.jar called the AnalyzingQueryParser which does pass the search string to the analyzer, even with wildcards. To use it, however, we may need more investigation. (Especially the JIRA link above illustrates some of the issues one can run into)

Extending Ola's test with the following tests:

        texts = ("*earch", "*h", "search tif", "search",\
                 "test", "tag", "t*", "search_test",\
                 "*test*.tif", "search*tif", "s .tif",\
                 ".tif", "tif", "*tif",\
                 "s*.tif", "*.tif")

I see the following terms fail for the default QueryParser:

• *.tif
• search*tif
• s*.tif
• *test*.tif

For the new AnalyzingQueryParser:

• *earch
• *h
• search*tif
• s*.tif
• *test*.tif

So, we can get "*.tif" back, but at the cost of "*earch" and "*h". With further investigation, we can probably come up with something that makes each of these cases pass, but other searches may then start to fail.

Possibly related is #1011 which would not use an analyzer at all on some fields like Image.name so that the underscores in "search_test_1.tif" don't get removed.

I'll commit the extended test and the lucene-misc jars and we can discuss further.

Update

This issue is apparently not only restricted to leading wildcards, but other forms of wildcard searches. Moving to 4.3 for review.

Matching: test-project-a-b-c
=============================================
                         Query Found  Ok?
                          test    21 GOOD
                  test-project    21 GOOD
                 test\-project    21 GOOD
                         test-    21 GOOD
                 test-project-    21 GOOD
               test\-project\-    21 GOOD
                         test*    21 GOOD
                 test-project*     0 FAIL
                test\-project*     0 FAIL
                        test-*     0 FAIL
                test-project-*     0 FAIL
              test\-project\-*     0 FAIL
                    name:test*    21 GOOD
            name:test-project*     0 FAIL
           name:test\-project*     0 FAIL
                    name:test*    21 GOOD
        name:test name:project    21 GOOD
                  test project    21 GOOD
                test* project*    21 GOOD
                test- project-    21 GOOD
              test-* project-*     0 FAIL
            test-project-a-b-c    21 GOOD
                         a-b-c    21 GOOD
                         a b c    21 GOOD
                            t*    21 GOOD
                            p*    21 GOOD
                            a*    21 GOOD
                            b*    21 GOOD
                            c*    21 GOOD
                         t* p*    21 GOOD
                         proj*    21 GOOD
                    tes* proj*    21 GOOD

[atarkowska]

(In [8385]) test, see #3164

[jmoore]

(In [8386]) More tests of wildcard searching; and lucene-misc (See #3164)

[atarkowska]

I'm not sure if it was known, but in collaborative group I am able to use "*tif", etc. If I switch to read-only or private group no results is returned.

[jmoore]

(In [8393]) Adding search.py to integration_suite.py (See #3164)

[jmoore]

(In [8396]) Testing various groups and disabling broken strings (See #3164)

[jburel]

Moved from sprint 2010-10-28 (18)

[jmoore]

Not making any code modifications to support this for 4.2.1. I've linked this under #2097 (4.2+ search fixes) and am moving to "unscheduled". Hopefully, we will have a large search review after big images.