Execute field caps per node directly on remote clusters

[dnhatn]

Today, we execute CCS field-caps requests similarly to CCS search requests with ccs_minimize_roundtrips to avoid network delays for each index field-caps request. However, the benefit is no longer significant since #77047 as field-caps requests are executed per node.

This issue proposes fanning out field-caps requests directly to nodes on remote clusters. With this model, we can skip indices on remote clusters if we have identical index mappings on the local cluster. This model also avoids big transport messages between clusters. The disadvantage of this model is the extra call fetching the routing table and index mapping hash from remote clusters.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[javanna]

Good discussion, thanks for opening this. I wonder if you would target this change to 7.x as in main this is practically solved by #83494 ? On the other hand while reviewing the current execution of field_caps I was wondering if we can simplify it, and maybe make the cross-cluster execution more inline with what we do within a single cluster. Having benchmarks around field_caps is becoming more and more important.

[javanna]

@dnhatn do you think that this is still worth doing given the recent as well as upcoming optimizations around index mapping hash? Or maybe we should have benchmark to help us drive this decision?

[dnhatn]

@javanna I was thinking about the optimization that avoids sending field-caps requests to remote clusters if we can resolve them using the indices on the local cluster. However, this optimization might not be significant with recent improvements. We can revisit this issue if we have CCS field-caps performance problems.