[CI] SnapshotBasedRecoveryIT testSnapshotBasedRecovery failing

[ywangd]

Normally I'd skip timeout errors because they are often just due to luck. But this test is new and already failed 38 times in past few days.

Build scan:
https://gradle-enterprise.elastic.co/s/523u2dz3olx3u

Repro line:
./gradlew ':qa:rolling-upgrade:v7.14.1#oneThirdUpgradedTest' -Dtests.class="org.elasticsearch.upgrades.SnapshotBasedRecoveryIT" -Dtests.method="testSnapshotBasedRecovery" -Dtests.seed=15D06BC7AFD0B42F -Dtests.bwc=true -Dtests.locale=ca-ES -Dtests.timezone=Africa/Banjul -Druntime.java=8

Reproduces locally?:
No

Applicable branches:
7.x

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?search.relativeStartTime=P7D&search.timeZoneId=Australia/Melbourne&tests.container=org.elasticsearch.upgrades.SnapshotBasedRecoveryIT&tests.sortField=FAILED&tests.test=testSnapshotBasedRecovery&tests.unstableOnly=true

Failure excerpt:

java.lang.AssertionError: timed out waiting for green state for index [snapshot_based_recovery] cluster state [{ |  
-- | --
  | "cluster_name" : "v7.14.1", |  
  | "cluster_uuid" : "vNuyvMUdTfaVJAKtMymzNQ", |  
  | "version" : 620, |  
  | "state_uuid" : "N5K_VdvhSVupuKwpFqXHBw", |  
  | "master_node" : "2YFxMyn6Rrmw9JqR-zgMRg", |  
  | "blocks" : { |  
  | "indices" : { |  
  | "index_mixed_7140199" : { |  
  | "4" : { |  
  | "description" : "index closed", |  
  | "retryable" : false, |  
  | "levels" : [ |  
  | "read", |  
  | "write" |  
  | ] |  
  | } |  
  | }, |  
  | "closed_index_replica_allocation" : { |  
  | "4" : { |  
  | "description" : "index closed", |  
  | "retryable" : false,

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[henningandersen]

The failures are genuine. My initial analysis points to the primary ending up on the upgraded node. This is surprising (but may have a valid reason once we dig deeper). I muted this selectively in #76601, with that we should still be validating rolling upgrade works with recovery from snapshot (though less frequently).

[henningandersen]

Pasting David's comment from #76601 here:

I suspect the problem is caused by a rebalance moving the primary onto the newly-upgraded node, but I haven't seen a failure in captivity to confirm that yet. If so I think we could do something a bit stronger here, e.g. apply an allocation filter to exclude the solitary upgraded node, then explicitly cancel any shards it holds to promote a replica on the old nodes, and then remove replicas.