Inconsistent wildcard search results when combined with a trim normalizer

[ACassells]

Tested using clean docker 7.12.1

version: Version: 7.12.1, Build: default/docker/3186837139b9c6b6d23c3200870651f10d3343b7/2021-04-20T20:56:39.040728659Z, JVM: 16
on
Linux 0ca876ff72e4 4.19.121-linuxkit #1 SMP Thu Jan 21 15:36:34 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

When using a trim normalizer in combination of a keyword wildcard search, the search fails at the end or beginning of words when using a single wildcard. In those situations you need to add a double wild card.

Steps to reproduce:

1. Create index

PUT /test-normalizer-index
{
  "settings": {
    "analysis": {
      "normalizer": {
        "trimming_normalizer": {
          "filter": [
            "trim"
          ]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "name": {
        "type": "keyword",
        "normalizer": "trimming_normalizer"
      }
    }
  }
}

2. Add doc to index

POST /test-normalizer-index/_doc/
{
  "name": "elastic search"
}

3. Search for an entry that would expect to get a result

GET /test-normalizer-index/_search
{
  "query": {
    "wildcard": {
      "name": {
        "value": "elasti? search"
      }
    }
  }
}

this provides 0 results

These are some other tests results:

elast?c search -> finds a document - EXPECTED
elastic searc? -> finds a document - EXPECTED
?lastic search -> finds a document - EXPECTED
elastic ?earch -> fails to find a document - UNEXPECTED
elastic ??earch -> finds a document - UNEXPECTED
elasti? search -> fails to find a document - UNEXPECTED
elasti?? search -> finds a document - UNEXPECTED

The first three tests are as expected. The last 4 tests seem incorrect.

[svalaskevicius]

more info:

http :9200/test-normalizer-index-3/_explain/lImsjnkBC9Y0I2JGMwZA query:='
                       {
                           "wildcard": {
                             "name": {
                               "value": "elastic search"
                             }
                           }
                       }'

{
    "_id": "lImsjnkBC9Y0I2JGMwZA",
    "_index": "test-normalizer-index-3",
    "_type": "_doc",
    "explanation": {
        "description": "name:elastic search",
        "details": [],
        "value": 1.0
    },
    "matched": true
}

http :9200/test-normalizer-index-3/_explain/lImsjnkBC9Y0I2JGMwZA query:='
                       {
                           "wildcard": {
                             "name": {
                               "value": "elasti? search"
                             }
                           }
                       }'

{
    "_id": "lImsjnkBC9Y0I2JGMwZA",
    "_index": "test-normalizer-index-3",
    "_type": "_doc",
    "explanation": {
        "description": "name:elasti?search doesn't match id 0",
        "details": [],
        "value": 0.0
    },
    "matched": false
}

? ate the space!

[imotov]

It almost looks like wildcard splits the query into tokens on the wildcards and passes them separately through normalizer. So you get two tokens "elasti" and " search" the space before search is getting trimmed and after reassembly the query becomes:

GET /test-normalizer-index/_validate/query?explain=true
{
  "query": {
    "wildcard": {
      "name": {
        "value": "elasti? search"
      }
    }
  }
}

{
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "valid" : true,
  "explanations" : [
    {
      "index" : "test-normalizer-index",
      "valid" : true,
      "explanation" : "name:elasti?search"
    }
  ]
}

Looks like a bug to me.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[cbuescher]

I think this is an unexpected sideeffect of #53127, which fixed another issue around normalization on wildcard queries. The mentioned issue fixes another bug where wildcard characters get potentially removed by normalizers that erase the * and ? characters. We solve this by splitting the input value where we encounter these special characters and normalize the chunks individually. Unfortunately this doesn't play well with the trim filter when these chunks then have leading/trailing whitespaces that otherwise would be inside the string.

Given the wide range of transformations a normalizer can apply to the input and we're not entirely sure what kind of normalizer we apply when building the query, I'm not sure yet how to proceed with this but wanted to add this analysis for further discussion.

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)