Fix assertion error when using passthrough with contradicting fields

[felixbarny]

I've added a yml rest test that reproduces an assertion error when using the passthrough field type with conflicts on multiple levels with a top-level match_only_text field.

[2024-07-01T03:48:58,737][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [test-cluster-1] fatal error in thread [elasticsearch[test-cluster-1][write][T#5]], exiting java.lang.AssertionError: Field body should not have docvalues
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.index.mapper.FieldNamesFieldMapper.addFieldNames(FieldNamesFieldMapper.java:177)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.index.mapper.DocumentParserContext.addToFieldNames(DocumentParserContext.java:292)
	at org.elasticsearch.mapper.extras@8.15.0-SNAPSHOT/org.elasticsearch.index.mapper.extras.MatchOnlyTextFieldMapper.parseCreateField(MatchOnlyTextFieldMapper.java:421)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:185)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:450)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:775)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:368)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:319)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:143)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:88)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:112)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:1038)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:979)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:923)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:376)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:234)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
	at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1570)

When changing the top-level field to a different field type, for example, keyword or text, the assertion error disappears, but the ordering is not quite right. Note that sometimes the tests pass and sometimes they fail with the following message: Expected a list containing 0: expected "top-level" but was "resource". Seems like in certain cases, the passthrough field type overrides the top-level field. I'm not sure if the two issues are related or separate.

This is a dump of the search:

[{
  "stash" : {
    "body" : {
      "took" : 35,
      "timed_out" : false,
      "_shards" : {
        "total" : 1,
        "successful" : 1,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : {
          "value" : 1,
          "relation" : "eq"
        },
        "max_score" : 1.0,
        "hits" : [
          {
            "_index" : ".ds-otel-2024.07.01-000001",
            "_id" : "IHWS-EudCPGf0ayGAAABilDXF2o",
            "_score" : 1.0,
            "_source" : {
              "@timestamp" : "2023-09-01T13:03:08.138Z",
              "attributes" : {
                "body" : "attribute"
              },
              "body" : "top-level",
              "metrics" : {
                "data" : 10
              },
              "resource" : {
                "attributes" : {
                  "body" : "resource"
                }
              },
              "scope" : {
                "attributes" : {
                  "body" : "scope"
                }
              }
            },
            "fields" : {
              "metrics.data" : [
                10
              ],
              "@timestamp" : [
                "2023-09-01T13:03:08.138Z"
              ],
              "data" : [
                10
              ],
              "scope.attributes.body" : [
                "scope"
              ],
              "attributes.body" : [
                "attribute"
              ],
              "resource.attributes.body" : [
                "resource"
              ],
              "body" : [
                "resource"
              ]
            }
          }
        ]
      }
    }
  }
}]

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[kkrik-es]

Thanks for catching and sending a fix promptly.. Tricky logic, the new version is much cleaner.

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	8.14	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 110326

[felixbarny]

The way passthrough field types are created and how the priority is configured has significantly changed in 8.15 with #106829. Backporting this fix to 8.14.x doesn't make sense.