[CI] DataStreamsClientYamlTestSuiteIT test {p0=data_stream/240_logs_ecs_mappings/Test Elastic Agent log ECS mappings} failing

[kingherc]

Build scan:
https://gradle-enterprise.elastic.co/s/56ydgs7cr5vyc/tests/:modules:data-streams:yamlRestTest/org.elasticsearch.datastreams.DataStreamsClientYamlTestSuiteIT/test%20%7Bp0=data_stream%2F240_logs_ecs_mappings%2FTest%20Elastic%20Agent%20log%20ECS%20mappings%7D

Reproduction line:

./gradlew ':modules:data-streams:yamlRestTest' --tests "org.elasticsearch.datastreams.DataStreamsClientYamlTestSuiteIT.test {p0=data_stream/240_logs_ecs_mappings/Test Elastic Agent log ECS mappings}" -Dtests.seed=C0FDAA9DB5113BE3 -Dbuild.snapshot=false -Dtests.jvm.argline="-Dbuild.snapshot=false" -Dtests.locale=es-UY -Dtests.timezone=AET -Druntime.java=20

Applicable branches:
8.9

Reproduces locally?:
Didn't try

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.datastreams.DataStreamsClientYamlTestSuiteIT&tests.test=test%20%7Bp0%3Ddata_stream/240_logs_ecs_mappings/Test%20Elastic%20Agent%20log%20ECS%20mappings%7D

Failure excerpt:

java.lang.AssertionError: Failure at [data_stream/240_logs_ecs_mappings:18]: expected [2xx] status code but api [indices.create_data_stream] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"no matching index template found for data stream [logs-generic-default]","stack_trace":"org.elasticsearch.ElasticsearchException$1: no matching index template found for data stream [logs-generic-default]\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:667)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:595)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.rest.RestResponse.build(RestResponse.java:176)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.rest.RestResponse.<init>(RestResponse.java:124)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.rest.RestResponse.<init>(RestResponse.java:103)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:55)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onFailure(NodeClient.java:170)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:217)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:60)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:72)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:60)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:72)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$2(TransportMasterNodeAction.java:231)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:127)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:60)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:132)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:60)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:72)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener.lambda$notifyListenerFailed$1(AllocationActionListener.java:52)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener.executeInContext(AllocationActionListener.java:58)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener.notifyListenerFailed(AllocationActionListener.java:52)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener$1.onFailure(AllocationActionListener.java:84)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.AckedClusterStateUpdateTask.onFailure(AckedClusterStateUpdateTask.java:75)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$ExecutionResult.notifyFailure(MasterService.java:975)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$ExecutionResult.onClusterStateUnchanged(MasterService.java:938)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:245)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1622)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListener.run(ActionListener.java:382)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1619)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1237)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListener.run(ActionListener.java:382)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1216)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1623)\nCaused by: java.lang.IllegalArgumentException: no matching index template found for data stream [logs-generic-default]\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService.lookupTemplateForDataStream(MetadataCreateDataStreamService.java:307)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService.createDataStream(MetadataCreateDataStreamService.java:221)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService.createDataStream(MetadataCreateDataStreamService.java:168)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService$1.execute(MetadataCreateDataStreamService.java:90)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$UnbatchedExecutor.execute(MasterService.java:550)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1039)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1004)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:232)\n\t... 11 more\n"}],"type":"illegal_argument_exception","reason":"no matching index template found for data stream [logs-generic-default]","stack_trace":"java.lang.IllegalArgumentException: no matching index template found for data stream [logs-generic-default]\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService.lookupTemplateForDataStream(MetadataCreateDataStreamService.java:307)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService.createDataStream(MetadataCreateDataStreamService.java:221)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService.createDataStream(MetadataCreateDataStreamService.java:168)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService$1.execute(MetadataCreateDataStreamService.java:90)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$UnbatchedExecutor.execute(MasterService.java:550)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1039)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1004)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:232)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1622)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListener.run(ActionListener.java:382)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1619)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1237)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListener.run(ActionListener.java:382)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1216)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1623)\n"},"status":400}]

  at __randomizedtesting.SeedInfo.seed([C0FDAA9DB5113BE3:48A995471BED561B]:0)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:572)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:515)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
  at java.lang.reflect.Method.invoke(Method.java:578)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1623)

  Caused by: java.lang.AssertionError: expected [2xx] status code but api [indices.create_data_stream] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"no matching index template found for data stream [logs-generic-default]","stack_trace":"org.elasticsearch.ElasticsearchException$1: no matching index template found for data stream [logs-generic-default]\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:667)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:595)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.rest.RestResponse.build(RestResponse.java:176)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.rest.RestResponse.<init>(RestResponse.java:124)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.rest.RestResponse.<init>(RestResponse.java:103)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:55)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onFailure(NodeClient.java:170)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:217)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:60)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:72)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:60)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:72)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$2(TransportMasterNodeAction.java:231)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:127)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:60)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:132)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:60)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:72)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener.lambda$notifyListenerFailed$1(AllocationActionListener.java:52)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener.executeInContext(AllocationActionListener.java:58)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener.notifyListenerFailed(AllocationActionListener.java:52)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener$1.onFailure(AllocationActionListener.java:84)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.AckedClusterStateUpdateTask.onFailure(AckedClusterStateUpdateTask.java:75)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$ExecutionResult.notifyFailure(MasterService.java:975)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$ExecutionResult.onClusterStateUnchanged(MasterService.java:938)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:245)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1622)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListener.run(ActionListener.java:382)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1619)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1237)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListener.run(ActionListener.java:382)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1216)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1623)\nCaused by: java.lang.IllegalArgumentException: no matching index template found for data stream [logs-generic-default]\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService.lookupTemplateForDataStream(MetadataCreateDataStreamService.java:307)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService.createDataStream(MetadataCreateDataStreamService.java:221)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService.createDataStream(MetadataCreateDataStreamService.java:168)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService$1.execute(MetadataCreateDataStreamService.java:90)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$UnbatchedExecutor.execute(MasterService.java:550)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1039)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1004)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:232)\n\t... 11 more\n"}],"type":"illegal_argument_exception","reason":"no matching index template found for data stream [logs-generic-default]","stack_trace":"java.lang.IllegalArgumentException: no matching index template found for data stream [logs-generic-default]\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService.lookupTemplateForDataStream(MetadataCreateDataStreamService.java:307)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService.createDataStream(MetadataCreateDataStreamService.java:221)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService.createDataStream(MetadataCreateDataStreamService.java:168)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService$1.execute(MetadataCreateDataStreamService.java:90)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$UnbatchedExecutor.execute(MasterService.java:550)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1039)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1004)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:232)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1622)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListener.run(ActionListener.java:382)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1619)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1237)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListener.run(ActionListener.java:382)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1216)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1623)\n"},"status":400}]

    at org.junit.Assert.fail(Assert.java:88)
    at org.elasticsearch.test.rest.yaml.section.DoSection.execute(DoSection.java:372)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:552)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:515)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
    at java.lang.reflect.Method.invoke(Method.java:578)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
    at java.lang.Thread.run(Thread.java:1623)

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[gmarouli]

This is a weird failure, I cannot reproduce it locally and I cannot figure out why would the template not be loaded byStackTemplateRegistry:

{
  "index_patterns": ["logs-*-*"],
  "priority": 100,
  "data_stream": {},
  "composed_of": [
    "logs-mappings",
    "logs-settings",
    "logs@custom",
    "ecs@dynamic_templates"
  ],
  "ignore_missing_component_templates": ["logs@custom"],
  "allow_auto_create": true,
  "_meta": {
    "description": "default logs template installed by x-pack",
    "managed": true
  },
  "version": ${xpack.stack.template.version}
}

I haven't found anything on the logs yet, but I will keep looking

[DaveCTurner]

Two more like this today:

• https://gradle-enterprise.elastic.co/s/rdybftm4ch3a2
• https://gradle-enterprise.elastic.co/s/bzzcmqwoszws2

The problem is I think that the index templates get installed asynchronously after the node has started up, and we start running tests before that's all finished:

[2023-09-13T12:11:05,586][INFO ][o.e.c.m.MetadataIndexTemplateService] [test-cluster-0] adding index template [metrics] for index patterns [metrics-*-*]
[2023-09-13T12:11:05,607][INFO ][o.e.c.m.MetadataIndexTemplateService] [test-cluster-0] adding index template [synthetics] for index patterns [synthetics-*-*]
[2023-09-13T12:11:05,621][INFO ][o.e.c.m.MetadataIndexTemplateService] [test-cluster-0] adding index template [.deprecation-indexing-template] for index patterns [.logs-deprecation.*]
[2023-09-13T12:11:05,677][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [.monitoring-8-ilm-policy]
↓↓↓ TEST FAILS HERE ↓↓↓
[2023-09-13T12:11:05,687][INFO ][o.e.d.DataStreamsClientYamlTestSuiteIT] [test] Dump test yaml [/var/lib/jenkins/workspace/elastic+elasticsearch+main+multijob+platform-support-unix/modules/data-streams/build/resources/yamlRestTest/rest-api-spec/test/data_stream/200_logs_datastream_defaults.yml] on failure:
[2023-09-13T12:11:05,695][INFO ][o.e.d.DataStreamsClientYamlTestSuiteIT] [test] Stash dump on test failure [{
↑↑↑ TEST FAILS HERE ↑↑↑
[2023-09-13T12:11:05,767][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [behavioral_analytics-events-default_policy]
[2023-09-13T12:11:05,848][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [ml-size-based-ilm-policy]
[2023-09-13T12:11:05,909][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [metrics]
[2023-09-13T12:11:05,929][INFO ][o.e.x.m.MachineLearning  ] [test-cluster-0] Starting machine learning feature reset
[2023-09-13T12:11:05,950][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [7-days-default]
[2023-09-13T12:11:06,022][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [synthetics]
[2023-09-13T12:11:06,123][INFO ][o.e.x.c.t.IndexTemplateRegistry] [test-cluster-0] adding ingest pipeline behavioral_analytics-events-final_pipeline
[2023-09-13T12:11:06,123][INFO ][o.e.x.c.t.IndexTemplateRegistry] [test-cluster-0] adding ingest pipeline ent-search-generic-ingestion
[2023-09-13T12:11:06,124][INFO ][o.e.x.c.t.IndexTemplateRegistry] [test-cluster-0] adding ingest pipeline logs@json-message
[2023-09-13T12:11:06,130][INFO ][o.e.c.m.MetadataIndexTemplateService] [test-cluster-0] adding component template [behavioral_analytics-events-settings]
[2023-09-13T12:11:06,197][INFO ][o.e.c.m.MetadataIndexTemplateService] [test-cluster-0] adding index template [behavioral_analytics-events-default] for index patterns [behavioral_analytics-events-*]
[2023-09-13T12:11:06,239][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [30-days-default]
[2023-09-13T12:11:06,284][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [logs]
[2023-09-13T12:11:06,327][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [365-days-default]
[2023-09-13T12:11:06,364][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [180-days-default]
[2023-09-13T12:11:06,405][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [90-days-default]
[2023-09-13T12:11:06,447][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [slm-history-ilm-policy]
[2023-09-13T12:11:06,484][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [watch-history-ilm-policy-16]
[2023-09-13T12:11:06,525][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [.fleet-file-tohost-meta-ilm-policy]
[2023-09-13T12:11:06,561][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [.fleet-actions-results-ilm-policy]
[2023-09-13T12:11:06,593][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [.deprecation-indexing-ilm-policy]
[2023-09-13T12:11:06,632][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [.fleet-file-fromhost-data-ilm-policy]
[2023-09-13T12:11:06,667][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [.fleet-file-fromhost-meta-ilm-policy]
[2023-09-13T12:11:06,703][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [.fleet-file-tohost-data-ilm-policy]
[2023-09-13T12:11:06,828][INFO ][o.e.h.n.s.HealthNodeTaskExecutor] [test-cluster-0] Node [{test-cluster-0}{f-utzOFqQduw1rzY1XndlA}] is selected as the current health node.
[2023-09-13T12:11:06,951][INFO ][o.e.l.ClusterStateLicenseService] [test-cluster-0] license [85ae5aec-0054-4359-bf44-e2f047b533cb] mode [basic] - valid
[2023-09-13T12:11:06,953][INFO ][o.e.x.s.a.Realms         ] [test-cluster-0] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2023-09-13T12:11:07,097][INFO ][o.e.x.c.t.IndexTemplateRegistry] [test-cluster-0] adding ingest pipeline logs-default-pipeline
[2023-09-13T12:11:07,102][INFO ][o.e.c.m.MetadataIndexTemplateService] [test-cluster-0] adding component template [logs-settings]
↓↓↓ INDEX TEMPLATE ADDED HERE ↓↓↓
[2023-09-13T12:11:07,157][INFO ][o.e.c.m.MetadataIndexTemplateService] [test-cluster-0] adding index template [logs] for index patterns [logs-*-*]
↑↑↑ INDEX TEMPLATE ADDED HERE ↑↑↑
[2023-09-13T12:11:07,304][INFO ][o.e.x.m.MachineLearning  ] [test-cluster-0] Finished machine learning feature reset

[pgomulka]

another one with the same exception mesage https://gradle-enterprise.elastic.co/s/2iawhjks6motq/tests/task/:modules:data-streams:yamlRestTest/details/org.elasticsearch.datastreams.DataStreamsClientYamlTestSuiteIT/test%20%7Bp0=data_stream%2F220_logs_default_pipeline%2FTest%20default%20logs-*-*%20pipeline%7D?top-execution=1

[luigidellaquila]

One more failure today https://gradle-enterprise.elastic.co/s/igj572po64n6c/tests/task/:modules:data-streams:yamlRestTest/details/org.elasticsearch.datastreams.DataStreamsClientYamlTestSuiteIT/test%20%7Bp0=data_stream%2F240_logs_ecs_mappings%2FTest%20Elastic%20Agent%20log%20ECS%20mappings%7D?page=eyJvdXRwdXQiOnsiMCI6M319&top-execution=1

[masseyke]

Is this a duplicate of #98163? That there's nothing that stops the server from coming up before all templates are ready?