Use new terms enum API for autocomplete value suggestions

[lukasolson]

Summary

Requires #100586.
Resolves #92783.
Resolves #97512.

Uses the new terms enum API for a faster autocomplete experience.

Note that currently, the autocomplete only targets warm and hot data tiers. By default, indices are given data_content tier, so to update this, you need to do something like this:

PUT /my-index-*/_settings
{
  "index.routing.allocation.include._tier_preference": "data_hot"
}

To do

• Config for timeout, data tiers, A/B testing?
• Update unit/functional tests

Checklist

• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list

Release note

Value suggestions for KQL autocomplete and creating filters now use the Elasticsearch _terms_enum API for improved performance.

[markharwood]

Note that currently, the autocomplete only targets warm and hot data tiers.

A heads up on how the API evolved - rather than hard-coding that tier filtering logic into the terms_enum API we changed that and made filtering a responsibility of the calling client. The client should filter indices by using a query on the new _tier metadata field so a typical Kibana query would look like:

localhost:9200/myindex/_terms_enum
{
    "field" : "myfield",
    "string" : "Microsof",
    "index_filter": {
        "bool":{
            "must":[
               { "range": "my Kibana time-picker range"},
               { "terms": {"_tier": ["data_warm", "data_hot"]}
             ]
        }
    }
}

Note that the queries used above are a rough filter on whole indices and not individual documents when it comes to where we look for term suggestions (i.e. we might suggest terms originating from docs outside of the provided date range if the docs sit in an index whose time span partially matches the index filter).

[lukasolson]

A heads up on how the API evolved - rather than hard-coding that tier filtering logic into the terms_enum API we changed that and made filtering a responsibility of the calling client. The client should filter indices by using a query on the new _tier metadata field so a typical Kibana query would look like:

@markharwood I've added it as a kibana.yml setting and defaulted it to ["data_hot", "data_warm"]. Does this seem like a reasonable default, or should we include data_content (since nodes default to that tier)?

[markharwood]

defaulted it to ["data_hot", "data_warm"]. Does this seem like a reasonable default, or should we include data_content (since nodes default to that tier)?

adding data_content sounds like it would make sense, covering those indices that don't "age out". Can you confirm @jimczi ?

[jimczi]

+1 to add data_content by default. I also wonder if data_cold should be considered. Are we applying the active date range filter on every request ? If so, then it would be nice to consider data_cold as default too since most of the requests to that tier would be filtered by the range. We have the data locally in that tier so I think it makes sense to provide suggestions as long as we can filtered the amount of queries that we need to handle (with the active range filter).

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[Dosant]

@lukasolson, afaik it is not necessary to update api_docs as part of the pr.

[nreese]

maps changes LGTM
code review

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: b45b464
• Storybooks Preview
• Documentation Changes

Metrics [docs]

✅ unchanged

History

• 💚 Build #133879 succeeded 056c7d0
• 💔 Build #133666 failed 0ef8b94
• 💔 Build #133426 failed ef001ef0e0b3b0c40070213d92a062de52420b64
• 💔 Build #133398 failed 741add8
• 💔 Build #130788 failed 6812011

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[mattkime]

code change look good, work well!

[kibanamachine]

💔 Backport failed

Status	Branch	Result
❌	7.x	Commit could not be cherrypicked due to conflicts

To backport manually run:
node scripts/backport --pr 100174