Wildcard queries on text fields don't obey rules for case sensitivity

[markharwood]

A wildcard query for *Butterflies* should not match a text field with the value I like butterflies.

This used to be the behaviour before 7.8 but then this fix to wildcard queries on keyword fields inadvertently changed the behaviour for text fields. Now it always normalizes the search string due to changes in the common base class (StringFieldType).

The solution should be to only normalize the search string when the new case_insensitive query flag is set to true.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[cbuescher]

Tied into this (or as a follow up) we should probably also check the documentation again around any discrepancies around this question. For example, I wanted to check what we are supposed to do currently with the wildcard query according to the docs and found this on 7.12.:

 case_insensitive [7.10.0] Added in 7.10.0.
    (Optional, Boolean) Allows case insensitive matching of the pattern with the indexed field values when set to true. Default is false which means the case sensitivity of matching depends on the underlying field’s mapping. 

The last past here ("... false which means the case sensitivity of matching depends on the underlying field’s mapping.") somehow sounds to me as if I'd get case-insensitivity also if my analyzer contains a lowercase filter. On the other hand in the main Term Level query docs we emphasize that "term-level queries do not analyze search terms". I'd suggest we also go over this again while looking into this issue.

[markharwood]

sounds to me as if I'd get case-insensitivity also if my analyzer contains a lowercase filter.

I think it's written as "it depends" to cater for the fact that keyword fields broke our rule that all term-level queries are exact-match-only because keyword fields always apply a normalizer. That's an unfortunate inconsistency compared to text etc but one I think we chose to live with because of the BWC issues in changing it.

Keyword fields are also the reason we couldn't call the boolean flag "case_sensitive" - the choices queries can offer are only "case insensitive" or "indeterminate" because of the whole text/keyword inconsistencies.

[cbuescher]

I think it's written as "it depends" to cater for the fact that keyword fields broke our rule

That makes sense to me now. Maybe still something we could make clearer here with some minimal rephrasing.

[markharwood]

That makes sense to me now. Maybe still something we could make clearer here with some minimal rephrasing.

I'm not sure how to get to a similar "aha" level of understanding for users without pointing out that discrepancy with the keyword field. Previously we considered making the flag an enum with 3 options:

1. Indeterminate
2. CaseSensitive
3. CaseInsensitive

..where the default matching mode is 1 to reflect what we have today with different fields doing different things.
That's clear but a bit of an ugly admission to build into an API

[cbuescher]

There's another detail that makes the potential solution a bit more difficult here. We use the fieldtypes wildcard() method from the query_string builders as well, but here we expect the normalization to apply (e.g. in tests like this). I just checked the behaviour on text fields on 7.6 and 7.12 and we do apply a text fields normalization (of e.g. the default Standard analyzer) here, but without the case_insensitive flag set because that one is specific to wildcard query. To me it seems necessary to separate the notion of "normalization" and the case_insensitive flag somehow in the usage of the fieldtype method.
As far as I can see, query_string usage of wildcard query patterns on text fields does currently always apply normalization and we need to keep it if we don't want a breaking change. On the other hand wildcard queries on text fields should normalize, but create a case-insensitive query when the flag is set.