[RW Separation] Enable auto-expand-replica for search replicas

[vinaykpud]

The setting index.auto_expand_replicas determines whether the cluster should automatically adjust the number of replica shards for an index based on the current number of data nodes. More details on this can be found here

Requirement:

• Search replicas should honor index.auto_expand_replicas settings on the index and it has to expand based on the number of search dedicated node

Earlier we decided to disable the auto_expand_replicas for search replicas, which is not applicable now.
#16912

Related component

Search:Performance

Describe alternatives you've considered

No response

Additional context

No response

[vinaykpud]

This is related to Meta issue : #15306

[vinaykpud]

Existing Behavior:

Auto-expand replicas can be enabled for an index using the setting:

index.auto_expand_replicas

The value of this setting defines the minimum and maximum number of replicas:

• Example:
  • 0-3: The index will have a minimum of 0 replicas and a maximum of 3
  • 0-all: The index will have a minimum of 0 replicas and a maximum equal to (total data nodes - 1).
    • For example, in a cluster with 5 data nodes, an index with 1 primary will automatically scale up to 4 replicas as new nodes are added.

Changes with search replica:

1. We will add new setting index.auto_expand_search_replicas
2. Handling New Node Additions:
   • When a data node is added, a new regular replica will be added based on the index.auto_expand_replicas setting.
   • When a search node is added, a new search replica will also be added based on the index.auto_expand_search_replicas setting.

So essentially we will be using the same setting for both regular and search replicas.

2. Handling Scale to Zero:
   • When scale-to-zero is enabled, the system should prevent the automatic expansion of regular replicas to avoid unnecessary resource allocation