Add ability to limit fields added during Mapper#merge

[felixbarny]

This is extracting a piece of independent functionality from #96235

This is the result of this discussion: #96235 (comment)

The mapping merge executed on the master node is the only authoritative place where we have a consistent view of the cluster state as it's guaranteed to be executed on a single thread on the master node. Therefore, there can be no race conditions and we exactly know how many fields are left before hitting the field limit.

Therefore, this is the best place to add as many fields as possible without going over the limit if index.mapping.total_fields.ignore_dynamic_beyond_limit is set to true in the index settings.

This change makes it possible to limit the number of fields added during mapping merger. However, that feature isn't actually used outside of tests.

In #96235, the feature to limit fields during the merge process is used to guarantee that the cluster state update never fails due to dynamic mapping updates that exceed the field limit in case ignore_dynamic_beyond_limit is enabled.

This works because after each dynamic mapping update, the indexing request gets retried (there's no change to that logic in this or related PRs) so that the document parsing takes the new mapping into account.

This only applies to dynamic mapping updates aka auto-updates. When explicitly trying to update the mapping with too many fields, the put mapping operation will still fail, even if ignore_dynamic_beyond_limit is enabled.

Depends on:

• Introduce MapperMergeContext #104512
• Simplify MergeResult#buildMergedMappers #104578

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[felixbarny]

@elasticmachine update branch

[javanna]

I did another round. Thanks for all the progress you made. Don't get discouraged by the amount of comments, these are all details around refining, polishing and adding some tests. We are very close!

Could you also add docs to clarify design choices: why the field limit is checked as part of merge. I could see future readers getting confused about that. And describe how it works, meaning it is already applied in the data node, and if certain fields are allowed there, they are going to be allowed when the master does the merge of the mappings, but there are scenarios where data nodes allowed addition of fields that the master will reject. Am I getting this right? Could you expand on the mechanics of this, perhaps in the javadocs of MapperMergeContext?

[felixbarny]

Could you also add docs to clarify design choices: why the field limit is checked as part of merge. I could see future readers getting confused about that. And describe how it works, meaning it is already applied in the data node, and if certain fields are allowed there, they are going to be allowed when the master does the merge of the mappings, but there are scenarios where data nodes allowed addition of fields that the master will reject. Am I getting this right? Could you expand on the mechanics of this, perhaps in the javadocs of MapperMergeContext?

I've documented most of that in the other PR. See also https://github.com/elastic/elasticsearch/pull/96235/files#diff-32c1e36544e0c0ef775f9e26576ef7df0504cc05466e513ba7b143947c234784R590

[javanna]

I did a last round, this look really good. My final comments are mainly on strengthening the tests further and a tiny bit more docs. Thanks for all the work. No need for another review round, merge when ready.