Add option to return ignored values as part of response

[timroes]

If you create an mapping containing an ignore_above field and ingest values that are above that limit, those are not returned by via the fields API. Since we're using the fields API nowadays in Discover this is a rather severe regression we found. From my experience we often have people having fields containing values above those limits, which were with reading from _source available and visible to the users. Using the new fields API, those will simply return as empty and no longer show in Discover.

It would be good if we can include those values in the response (whether behind a request flag, similar to include_unmapped or by default).

Related Kibana issue: elastic/kibana#101232

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[cbuescher]

I started to look into this a bit and think the current behaviour is consistent with what we also do with e.g. numeric or date values that we reject because of an ignore_malformed setting in as much as we don't return values in fields if they are not indexed and cannot be aggregated over. I think this behaviour was intended, but I'll mark this for discussion in the team to see if we can add an option to retrieve these from source.

[cbuescher]

We had a first discussion today to understand the needs better.
One of the things we already have is the "_ignored" field in the the search hit that - if present - records fields that have not been indexed due to being malformed but have the ignore_malformed option set. We currently don't include keyword fields that we don't index because of ignore_above into this, but we will investigate if it is possible to do that.
With that in place Kibana Discover could display if fields are missing and offer ways of going to "source" view in these cases.
We also discussed that mixing these "non-indexed" values into the "fields" output is something we would not like to do.
An option to consider might be to add an (optional) additional section to the fields output where we include values from fields in "_ignored" that we then get directly from source, however we are not fully convinced that this is the best way to go.
I will open separate issues to discuss and track whether we should add keyword fields that exceed ignore_above to the _ignored output and that we might want to include this metadata field in the fields API output.

[timroes]

We've discussed the desired Kibana behavior with our product managers and came to the following conclusion (see also elastic/kibana#101232 (comment)): We want to show ignored values (e.g. because they are above ignore_above) in the Discover table in columns and expanded documents directly. We'd additionally want to add a warning icon with a tooltip beside them explaining that those values are not indexed, and thus cannot be searched through. There should no additional user interaction be required to show those values (e.g. by loading them async, or refering to the _source view we have).

I think the best way we can achieve that technically is if the fields API (or the high level query) would have a include_ignored option that we could specify to make ES return all ignored values in the response - though as a separate part of the response, not to mix in with fields and thus potentially poluting type-purity. That way we can merge them client side and know which values were ignored (in case a field contained an array of values).

The only alternative solution we'd have at the moment: Request _source and fields at the same time and try to merge them client-side and figure out which values are in _source, but not in fields. This would cause multiple problems, e.g.:

• Requesting _source and fields aroundish doubles the size of our response causing us to run into problems with too large responses (that we know we already have nowadays sometimes)
• Discover would be significantly less performant, since we'd need to compare _source and fields for each field of each document to be able to figure out which _source values are missing from fields (and thus were ignored)
• Causing weird side-effects, e.g. in the 64-bit case, were a value in fields might be rounded by JavaScript, but in _source if present as a string, might not be rounded and thus we're not able to detect equality of those, and will present both values, leading the users to believe their document had multiple values in that fields, while it was only one.

Thus I think, given the discussion we had, adding that flag and a separated part of the response to return the ignored values might be the best (and most performant) solution we can end up with. Please let us know, if that's the way we can continue.

cc @rayafratkina @VijayDoshi