[CI] RetrySearchIntegTests testSearcherId failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #6369 / oraclelinux-7_platform-support-unix
• elasticsearch-pull-request #55708 / part-2

Reproduction Line:

./gradlew ":x-pack:plugin:searchable-snapshots:internalClusterTest" --tests "org.elasticsearch.xpack.searchablesnapshots.RetrySearchIntegTests.testSearcherId" -Dtests.seed=82E925F1012D500C -Dtests.locale=mai -Dtests.timezone=Europe/Saratov -Druntime.java=23

Applicable branches:
8.18

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Index{id='mvyrsywcul/EBVOiEw_T-qnqtPb2XIt0Q/9ar8771OQkS7mduFisu0Zg/2/_0.cfs/@0', seqNo=3, primaryTerm=2, version=1, autoGeneratedIdTimestamp=-1}
Expected: <4L>
     but: was <2L>

Issue Reasons:

• [8.18] 2 failures in test testSearcherId (0.3% fail rate in 689 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test testSearcherId (0.2% fail rate in 967 executions)

Build Scans:

• elasticsearch-intake #14193 / part2
• elasticsearch-periodic #5192 / concurrent-search-tests

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[benwtrent]

This has been failing since november. Here is the first failure: https://gradle-enterprise.elastic.co/s/lh4h2qvz55yv2

Commit: cc13f38

At commit and before (branch 8.x): https://github.com/elastic/elasticsearch/commits/cc13f38b43215098586b5443b680e16d29ca91f3

[benwtrent]

Since this is searchable snapshots, tagging distributed, but also tagged search foundations if I grabbed the wrong team :(

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)