Fix AsyncSearchErrorTraceIT_ testAsyncSearchFailingQueryErrorTraceFalse

[drempapis]

The exception is thrown while attempting to acquire a shard lock. The key stack frames show that it occurs during the after-test checks (assertAfterTest), when ESIntegTestCase runs its “assert after test” cleanup. At that point, it’s trying to lock shard 0 of the .async-search index, but the shard is still in the starting state because the async task hasn’t fully finished. As a result, the lock attempt times out after 5 seconds.

Caused by: org.elasticsearch.env.ShardLockObtainFailedException: [.async-search][0]: obtaining shard lock for [InternalTestCluster assert after test] timed out after [5000ms]; this shard lock is still held by a different instance of the shard and has been in state [starting shard] for [5.3s/5310ms] |  
  | at org.elasticsearch.env.NodeEnvironment.shardLock(NodeEnvironment.java:895) |  
  | at org.elasticsearch.test.InternalTestCluster.assertAfterTest(InternalTestCluster.java:2617) |  
  | at org.elasticsearch.test.ESIntegTestCase.afterInternal(ESIntegTestCase.java:612) |  
  | at org.elasticsearch.test.ESIntegTestCase.cleanUpCluster(ESIntegTestCase.java:2620) |  
  | at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104) |  
  | at java.lang.reflect.Method.invoke(Method.java:565) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1763) |  
  | at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:1004) |  
  | at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) |  
  | at org.junit.rules.RunRules.evaluate(RunRules.java:20) |  
  | at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48) |  
  | at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) |  
  | at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) |  
  | at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) |  
  | at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
...

The REST response returns is_running as a Boolean, but the code reads it as a String and evaluates true.equals("true"), which is always false. As a result, the wait/poll step is skipped, and the test executes the assertion immediately. This creates a race condition where the async search has not yet finished propagating results between the data and coordinating nodes, causing ErrorTraceHelper.assertStackTraceCleared(..) to occasionally run too early and fail.

 if (createAsyncResponseEntity.get("is_running").equals("true")) 

Closes #136986 #136691

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[benchaplin]

Ha, nice catch! Resolves #133010 as well?

[benchaplin]

Hm, another note - I wrote these tests completely wrong! The assertions work in request handlers, meaning they should be set BEFORE the async search starts. I'll open a PR once you merge @drempapis...

[drempapis]

Ha, nice catch! Resolves #133010 as well?

I’ve updated the test method testAsyncSearchFailingQueryErrorTraceDefault in this PR to address the same issue

[elasticsearchmachine]

💚 Backport successful

Status	Branch	Result
✅	9.2