Limit RW separation to remote store enabled clusters and update recovery flow

[mch2]

Description

This PR includes multiple changes to search replica recovery to further decouple these shards from primaries.

1. Change to recover as empty store instead of peer. This will run a store recovery that syncs segments from remote store directly and eliminate any primary communication.
2. Remove search replicas from the in-sync allocation ID set and update routing table to exclude them from allAllocationIds. This ensures primaries aren't tracking or validating the routing table for any search replica's presence.
3. Simplify RW separation by limiting to only remote store enabled clusters. There are versions of the above changes that are still possible with primary based node-node replication but they require additional public api changes and I don't think we have the need at this time.

Related Issues

Resolves #15952

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for a932d59:

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for a932d59:

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for a932d59: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 69.35484% with 19 lines in your changes missing coverage. Please review.

Project coverage is 72.10%. Comparing base (f6dc4a6) to head (1c95833).
Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
...in/java/org/opensearch/index/shard/IndexShard.java	44.44%	2 Missing and 3 partials ⚠️
...search/cluster/routing/IndexShardRoutingTable.java	42.85%	2 Missing and 2 partials ⚠️
.../opensearch/cluster/routing/IndexRoutingTable.java	78.57%	2 Missing and 1 partial ⚠️
...llocation/decider/ThrottlingAllocationDecider.java	84.61%	0 Missing and 2 partials ⚠️
...a/org/opensearch/cluster/routing/ShardRouting.java	66.66%	0 Missing and 1 partial ⚠️
...uster/routing/allocation/IndexMetadataUpdater.java	75.00%	0 Missing and 1 partial ⚠️
...org/opensearch/index/seqno/ReplicationTracker.java	0.00%	0 Missing and 1 partial ⚠️
...a/org/opensearch/index/shard/ReplicationGroup.java	50.00%	0 Missing and 1 partial ⚠️
...java/org/opensearch/index/shard/StoreRecovery.java	80.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #16760      +/-   ##
============================================
- Coverage     72.17%   72.10%   -0.08%     
+ Complexity    65251    65197      -54     
============================================
  Files          5301     5301              
  Lines        303662   303723      +61     
  Branches      43989    44006      +17     
============================================
- Hits         219181   218994     -187     
- Misses        66552    66730     +178     
- Partials      17929    17999      +70     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

✅ Gradle check result for 8935bc7: SUCCESS

[mch2]

apologies for the rebase, had to fix DCO check on an old commit

[github-actions]

❌ Gradle check result for cf68380: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for eaa38d9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for eaa38d9: SUCCESS

[github-actions]

❌ Gradle check result for 002323e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 002323e: SUCCESS

[github-actions]

✅ Gradle check result for 1c95833: SUCCESS

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-16760-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 8191de85856d291507d09a7fd425908843ed8675
# Push it to GitHub
git push --set-upstream origin backport/backport-16760-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-16760-to-2.x.