[CI] CohereServiceUpgradeIT testCohereEmbeddings {upgradedNodes=1} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #5382 / 8.14.3_bwc
• elasticsearch-periodic #5382 / 8.13.4_bwc
• elasticsearch-pull-request #46623 / 8.13.4_bwc
• elasticsearch-pull-request #46623 / 8.14.3_bwc
• elasticsearch-periodic #5377 / 8.13.4_bwc
• elasticsearch-periodic #5372 / 8.13.4_bwc

Reproduction Line:

./gradlew ":x-pack:plugin:inference:qa:rolling-upgrade:v8.14.3#bwcTest" -Dtests.class="org.elasticsearch.xpack.application.CohereServiceUpgradeIT" -Dtests.method="testCohereEmbeddings {upgradedNodes=1}" -Dtests.seed=1868AB709715B87A -Dtests.bwc=true -Dtests.locale=ta-SG -Dtests.timezone=Europe/Saratov -Druntime.java=23

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.IndexOutOfBoundsException: Index: 0, Size: 0

Issue Reasons:

• [8.x] 2 consecutive failures in step 8.14.3_bwc
• [8.x] 4 consecutive failures in step 8.13.4_bwc
• [8.x] 6 failures in test testCohereEmbeddings {upgradedNodes=1} (4.2% fail rate in 143 executions)
• [8.x] 2 failures in step 8.14.3_bwc (50.0% fail rate in 4 executions)
• [8.x] 4 failures in step 8.13.4_bwc (100.0% fail rate in 4 executions)
• [8.x] 3 failures in pipeline elasticsearch-periodic (100.0% fail rate in 3 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch 8.x

Mute Reasons:

• [8.x] 4 failures in test testCohereEmbeddings {upgradedNodes=1} (0.9% fail rate in 448 executions)
• [8.x] 4 failures in step 8.16.1_bwc-snapshots (4.0% fail rate in 100 executions)
• [8.x] 3 failures in pipeline elasticsearch-pull-request (6.0% fail rate in 50 executions)

Build Scans:

• elasticsearch-intake #13011 / 8.16.1_bwc-snapshots
• elasticsearch-pull-request #41379 / 8.16.1_bwc-snapshots
• elasticsearch-pull-request #41217 / 8.16.1_bwc-snapshots
• elasticsearch-pull-request #41242 / 8.16.1_bwc-snapshots

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[davidkyle]

The mixed cluster failed to form due to one of the nodes missing a required feature

Caused by: java.lang.IllegalStateException: Node gX0XxROcRlmfqMEXlwfZcw is missing required features [semantic_text.delete_fix]

Low risk as the issue in the test, just need to figure out how to fix it

[elasticsearchmachine]

This has been muted on branch 8.x

Mute Reasons:

• [8.x] 4 consecutive failures in step 8.13.4_bwc
• [8.x] 2 consecutive failures in pipeline elasticsearch-periodic
• [8.x] 5 failures in test testCohereEmbeddings {upgradedNodes=1} (3.7% fail rate in 136 executions)
• [8.x] 4 failures in step 8.13.4_bwc (100.0% fail rate in 4 executions)
• [8.x] 3 failures in pipeline elasticsearch-periodic (100.0% fail rate in 3 executions)

Build Scans:

• elasticsearch-periodic #5382 / 8.13.4_bwc
• elasticsearch-pull-request #46623 / 8.13.4_bwc
• elasticsearch-pull-request #46623 / 8.14.3_bwc
• elasticsearch-periodic #5377 / 8.13.4_bwc
• elasticsearch-periodic #5372 / 8.13.4_bwc