Prefer adaptive replica selection over awareness attribute based routing

[malpani]

Is your feature request related to a problem? Please describe.

Search routing preference and decisions enable coordinators to decide which copy of a shard should be picked to serve the shard query. By default, if shard allocation awareness is configured (eg. zone awareness/rack awareness), the coordinator always prefers to route the shard query to a node with local awareness attributes.

This can also result in major skews - eg. an index with 2 replicas on a cluster with 2 zones --> ie. one zone has 2 copies of the shard (say a primary and replica) whereas other zone has single copy(replica). In such situations, assuming round robin coordination - the shard on zone with single copy will see double requests as compared to the other 2 copies

Further, Adaptive Replica Selection kicks in only when allocation awareness attributes is not set. This implies most production grade clusters dont reap the benefits of tail latency reduction from ARS.

Describe the solution you'd like
My proposal is to

1. [Already present] Introduce a new cluster setting (es.search.ignore_awareness_attributes) that prefers ARS over shard allocation awareness attribute based routing
2. Run some experiments with the setting toggled - the intuition here is ARS will be able to make a smarter decision than human inputs on locality
3. If the experiments, look right turn this on by default

[malpani]

I just realized, there is already a cluster setting es.search.ignore_awareness_attributes to toggle the ordering. So part 1 is already present. Updating the description

[dvjyothsna]

Performance numbers from rally tests

All tests are performed on domain with 6 data nodes

• ARS-D-1: ARS disabled with 1 replica per index
• ARS-E-1: ARS enabled with 1 replica per index
• ARS-D-3: ARS disabled with 3 replicas per index
• ARS-E-3: ARS enabled with 3 replicas per index

Latency(ms)

A	ARS-D-1	ARS-E-1	Improvement	ARS-D-3	ARS-E-3	Improvement
p50	547.2	445.5	18.50%	383.9	375.1	2%
p90	661	501.5	24.10%	570.4	441.2	22.60%
p99	687.1	524	23.70%	596.6	568.2	4.60%
p100	772.2	547.5	29.08%	674.5	672	0.20%

Throughput(req/s)

A	ARS-D-1	ARS-E-1	Improvement	ARS-D-3	ARS-E-3	Improvement
p0	1.573	1.745	11.25%	1.733	1.737	0.23%
p50	1.738	1.749	0.63%	1.748	1.749	0.05%
p100	1.758	1.759	0.05%	1.758	1.761	0.17%