Optimize async_search polling with dynamic wait_for_completion

[Dosant]

Describe the feature:

Initially, when using a search strategy with the async search there was a search polling mechanism, where the client polls a search with a fixed 1s interval until the search completes.

We optimized it a bit to avoid excessive polling (every second) by doing the increase the polling interval as searches take longer to finish. The longer the search takes, the less noticeable the delay due to polling intervals.

We can improve it further by optimizing wait_for_completion param.

This needs testing and investigation

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServicesSv)

[Dosant]

The logic that we probably should consider is the following:

When async search start - start it with looong wait_for_completion, e.g. 5 seconds, so that we receive the results as soon as possible if search completes quickly.

If search didn't complete in 5 seconds, then we can continue with increasing the polling interval and shortening the wait_for_completion (can probably keep the current logic) as this isn't that important anymore to give the results as soon as possible, as a couple seconds difference is not the noticeable from user perspective on this stage.

things to consider:

• current poll interval should be adjusted to work nicely with new wait_for_completion
• need to make sure that long wait_for_completion doesn't lead to more risk in race condition bugs when saving the session while search request with long wait_for_completion is still pending. Might be that this search won't be extended.
• how a lot of searches with long wait_for_completion affect Kibana server under load as requests will get longer to complete.

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[kertal]

Closing this because it's not planned to be resolved in the foreseeable future. It will be tracked in our Icebox and will be re-opened if our priorities change. Feel free to re-open if you think it should be melted sooner.