Formalize dual text/keyword mappings

[jpountz]

Our default dynamic mappings rules create both a text and a keyword field whenever they hit a JSON string:

{
  "type": "text",
  "fields": {
    "keyword": {
      "type": "keyword",
      "ignore_above": 256
    }
  }
}

And over the years, many clients implemented similar logic:

• when an exact query is fired, use the keyword field,
• when an aggregation is used, use the keyword field,
• otherwise use the text field.

Is it logic that we should embed in Elasticsearch? Maybe we can find better ideas, but here is a proposal to get the discussion started:

• Create a new exact_match query, which tries to match against the whole string. It fails for text fields and has the same behavior as match on keyword, numbers, ...
• Create a new text_keyword field, which is essentially a wrapper around a text and a keyword field. Running aggregations or an exact_match query against this field use the sub keyword field while match, query_string, multi_match and simple_query_string queries use the text field.
• Update default dynamic mappings to create this field for strings instead of the current text + sub keyword mapping.

[elasticmachine]

Pinging @elastic/es-search (:Search/Mapping)

[jpountz]

Relates to #53020

[epixa]

Would the solution put into place to address text/keyword multi-fields also handle wildcards? In discussions I've seen it assumed that it would, but I wanted to clarify.

[jpountz]

Hopefully wildcard is going to be less subject to this issue, as I can't think of many reasons to map a field both as wildcard and keyword, since they support the same operations, or wildcard and text, since wildcard can do infix search already. That said, if you have plans to use multi-fields with a wilcard field, we'd be interested to know more.

ECS has some fields that are mapped as keyword / text that we plan to migrate as keyword. I believe we'll need to think about dedicated migration paths for this case since removing the text multi-field would be a breaking change. For instance, one idea that has been raised is whether wildcard fields could create a virtual text subfield that emulates the behavior of a text field to ease this transition. This requires more thought and we might end up with a different approach but I thought sharing this example would help explain the kind of solution that we're considering.

[epixa]

Are wildcard fields just a superset of keyword in terms of features? I may have been mistaken, but I thought simply switching from keyword to wildcard would result in feature loss for aggregations and such. Admittedly I don't have a concrete example.

[jpountz]

@epixa As we've been thinking about how to migrate from existing mappings to wildcard, we agreed that it would be much easier if wildcard supported the same operations keyword using the same semantics. It supports match queries, terms aggregations, sorting, etc. with the same semantics as keyword. However it takes a different approach to indexing that makes it slower at exact queries or aggregations, but faster at wildcard/regexp queries on high-cardinality fields.

[webmat]

I'd like to point out that ECS went with the reverse convention, on how to index strings. Since ECS started around monitoring, rather than full text search, the default datatype is keyword for string fields. Then only a few fields have a .text multi-field added (less than 20, iirc).

I'm pointing this out because here we're talking about potentially building a shorthand notation that encodes the Elasticsearch default. As the proposal stands, it couldn't be used by users who are trying to build ECS-compatible indices.

• Update default dynamic mappings to create this field for strings instead of the current text + sub keyword mapping.

I'm not sure I understand the 3rd point in the body of the issue. "This field": are we talking about wildcard?

[colings86]

@webmat The idea here would be that ECS would not need to define multi-fields at all. ECS would define the field type as text_keyword (or whatever name we come up with) with no multi-fields. Internally Elasticsearch would handle the fact that there is a text field and a keyword field underlying the field type but to the user it would appear as one field with no multi-fields. The idea here is that users should not need to worry about whether they need the keyword or text form of the field and so just reference the field in one way and Elasticsearch should figure out which underlying field (text or keyword) is the right one to use (so whether its used in an aggregation, in a free text query, an exact match query, etc.)

[webmat]

This makes sense, and would indeed be a good simplification. But this would force all string fields defined this way to be indexed both ways?

ECS followed the Beats convention of trying to do keyword only as much as possible, for performance reasons.

[mayya-sharipova]

Instead of introducing a new exact_match query, can we use term query for exact matching and make term query fail on text fields with a message like "use match query instead"?

---

I like the idea of having a new text_keyword field, which is a wrapper around a text and a subfield keyword field and queries/aggs are delegated to one of those fields automatically.

Another way to organize text_keyword is to index both the exact form and tokenized form into the same Lucene field. This is inspired by the new wildcard field. The exact form can be surrounded by some fake symbols, e.g. "0<exact_value>00". If exact and tokenized form are the same, we can only keep the exact value, which will allow us to save space for many solution fields that mostly single-valued.

[jpountz]

this would force all string fields defined this way to be indexed both ways

Only if you want to support both text search (provided by text) and exact match/sorting/aggregations (provided by keyword).

can we use term query for exact matching and make term query fail on text fields with a message like "use match query instead"?

We could.

index both the exact form and tokenized form into the same Lucene field

The space saving idea is interesting. I wonder if that would cause problems. Preserving support for scoring and multi-term queries would be challenging but I believe it could work?

A problem with the proposal of this issue that we identified when discussing the wildcard field is that we might need variants of the fuzzy, wildcard, regexp, ... queries as well if we want to be consistent in the way that we treat matching individual tokens vs. full values, which might not be scalable.