CCS: Should `timeout` parameter be honored?

[spinscale]

While testing some Cross Cluster Search behaviour I noticed search timeouts are not being honored.

Note: This might be intended behaviour so feel free to close this issue.

Reproduction, started two one node clusters with different clustername

./bin/elasticsearch -Enode.name=rhincodon -Ecluster.name=alr -Enode.max_local_storage_nodes=2
./bin/elasticsearch -Enode.name=remote01 -Ecluster.name=remote -Enode.max_local_storage_nodes=2

configure CCS

curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "search": {
      "remote": {
        "cluster_remote": {
          "seeds": [
            "127.0.0.1:9301"
          ]
        }
      }
    }
  }
}
'

Now check that CCS works

curl -X PUT localhost:9200/foo
curl -X PUT localhost:9201/bar
curl localhost:9200/cluster_remote:bar,foo/_search

In order to simulate a failure I stopped the remote cluster node, and ran netcat -l 9301, a service that receives data but does not reply anything (i.e. emulating a GC).

Now running a search ends up like this

curl 'localhost:9200/cluster_remote:bar,foo/_search?timeout=5s'
{"error":{"root_cause":[{"type":"connect_transport_exception","reason":"[][127.0.0.1:9301] handshake_timeout[30s]"}],"type":"connect_transport_exception","reason":"[][127.0.0.1:9301] handshake_timeout[30s]"},"status":500}

As you can see the timeout is the 30s handshake timeout and curl also only returns after 30 seconds.

[elasticmachine]

Pinging @elastic/es-search-aggs

[javanna]

The search timeout that's provided in the request is applied to the query phase like in an ordinary search request. But with this simulated network disruption (or anytime a remote cluster is down when the cross-cluster search request starts), the request fails earlier, what fails is the initial search_shards call made to the remote cluster, which is a different type of request compared to search and does not support the timeout parameter. Maybe we should apply the search timeout as a transport timeout when sending the search shards request here ?

[javanna]

We discussed this during fix-it-friday and we decided that we should try and set the timeout as a transport timeout to the search shards request as it is unexpected that the request times out after 30 seconds when the search timeout was set to 10 seconds. Each search_shards request (one per cluster) will have the same timeout as search, which will happen later, so this is best-effort and not really a guarantee that the request will never take longer than the timeout overall. We discussed also tweaking the search timeout to what is left of the initial search timeout once the search_shards phase is completed (given that the coordinating node waits for all the clusters to be back before moving on to the search phase), and we said it may be nice but not strictly needed, we may consider doing this later.

[javanna]

We haven't come around to make the recommended change above. Search and async search support now automatic cancellation on connection close. My thinking is that we should propagate the cancellation to the search shards call that async search (as well as search when not minimizing roundtrips) makes. I believe this would be enough and we would not need to propagate the search timeout to the search shards call. @DaveCTurner do you have thoughts?

[DaveCTurner]

👍 to having one timeout on the client side and doing everything else via cancellation.