[CI] CohereServiceUpgradeIT testCohereEmbeddings {upgradedNodes=2} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #13011 / 8.16.1_bwc-snapshots
• elasticsearch-pull-request #41379 / 8.16.1_bwc-snapshots
• elasticsearch-pull-request #41231 / 8.16.1_bwc-snapshots
• elasticsearch-pull-request #41252 / 8.16.1_bwc-snapshots
• elasticsearch-pull-request #41252 / 8.16.1_bwc-snapshots
• elasticsearch-pull-request #41231 / 8.16.1_bwc-snapshots
• elasticsearch-pull-request #41252 / 8.16.1_bwc-snapshots
• elasticsearch-pull-request #41242 / 8.16.1_bwc-snapshots
• elasticsearch-pull-request #41236 / 8.16.1_bwc-snapshots
• elasticsearch-pull-request #41217 / 8.16.1_bwc-snapshots

Reproduction Line:

./gradlew ":x-pack:plugin:inference:qa:rolling-upgrade:v8.16.1#bwcTest" -Dtests.class="org.elasticsearch.xpack.application.CohereServiceUpgradeIT" -Dtests.method="testRerank {upgradedNodes=1}" -Dtests.seed=FD550FE191CA1911 -Dtests.bwc=true -Dtests.locale=fr-CF -Dtests.timezone=Europe/Vienna -Druntime.java=23

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.RuntimeException: An error occurred while checking cluster 'test-cluster' status.

Issue Reasons:

• [8.x] 10 failures in test testCohereEmbeddings {upgradedNodes=2} (2.2% fail rate in 453 executions)
• [8.x] 10 failures in step 8.16.1_bwc-snapshots (9.8% fail rate in 102 executions)
• [8.x] 6 failures in pipeline elasticsearch-pull-request (11.8% fail rate in 51 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch 8.x

Mute Reasons:

• [8.x] 10 failures in test testCohereEmbeddings {upgradedNodes=2} (2.2% fail rate in 453 executions)
• [8.x] 10 failures in step 8.16.1_bwc-snapshots (9.8% fail rate in 102 executions)
• [8.x] 6 failures in pipeline elasticsearch-pull-request (11.8% fail rate in 51 executions)

Build Scans:

• elasticsearch-intake #13011 / 8.16.1_bwc-snapshots
• elasticsearch-pull-request #41379 / 8.16.1_bwc-snapshots
• elasticsearch-pull-request #41231 / 8.16.1_bwc-snapshots
• elasticsearch-pull-request #41252 / 8.16.1_bwc-snapshots
• elasticsearch-pull-request #41252 / 8.16.1_bwc-snapshots
• elasticsearch-pull-request #41231 / 8.16.1_bwc-snapshots
• elasticsearch-pull-request #41252 / 8.16.1_bwc-snapshots
• elasticsearch-pull-request #41242 / 8.16.1_bwc-snapshots
• elasticsearch-pull-request #41236 / 8.16.1_bwc-snapshots
• elasticsearch-pull-request #41217 / 8.16.1_bwc-snapshots

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[davidkyle]

The mixed cluster failed to form due to one of the nodes missing a required feature

Caused by: java.lang.IllegalStateException: Node gX0XxROcRlmfqMEXlwfZcw is missing required features [semantic_text.delete_fix]

Low risk as the issue in the test, just need to figure out how to fix it

[dan-rubinstein]

Looks like this test has been fixed and unmuted (see the commit where this was referenced above). Closing the issue now.