Expose _async_search metadata to kibana admin user

[lizozom]

This feature is part of elastic/kibana#61738

When Kibana runs BackgroundSessions, it will need to have a monitoring service that periodically checks for the status of each session. We will use that information to mark sessions that are complete or have errors. It will also be responsible for sending push notifications back to users.

In order to do that, we need to have access to each _async_search's metadata, using the kibana admin user (rather than the user that created the search). This meta data should include the amount of total shards, amount of shards processed and the is_running and is_partial flags.

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[jimczi]

I don't think we should tied this to the Kibana admin user. IMO the main question is whether we want to expose the metadata associated with an async search to any user. These metadata infos shouldn't contain any sensible information so I don't see why we couldn't. The following question is how should it be exposed, maybe an additional parameter on the get API is enough ?

[lizozom]

@jimczi note that when we do query for meta data - we will always do so for multiple objects (i.e. take all running background searches, and fetch their metadata).

Maybe it's worth adding another route for that altogether?

[lizozom]

@jimczi I implemented the monitoring task I would like to run to track the completion of async requests.
You can see the place where I would like to query that new API here https://github.com/lizozom/kibana/pull/8/files#diff-462f5bdb3bd70cec29416a1967e9119fR44

[mayya-sharipova]

we will always do so for multiple objects

@lizozom Do you need an API that provides metadata about a particular single async request? Or metadata about all async requests available?

Or perhaps you would like to submit to this API a list of several async requests IDs?

---

If you just need metadata about a single async request, as @jimczi suggested we can reuse the same GET request with a metadata parameter:

GET /_async_search/FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=?metadata=true

In response we will get the same response as in GET request, but without data sections (sections hits, aggs, suggestions will not be included).

{
  "id" : "FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=",
  "is_partial" : true, 
  "is_running" : true, 
  "start_time_in_millis" : 1583945890986,
  "expiration_time_in_millis" : 1584377890986, 
  "response" : {
    "took" : 12144,
    "timed_out" : false,
    "num_reduce_phases" : 46, 
    "_shards" : {
      "total" : 562, 
      "successful" : 188,
      "skipped" : 0,
      "failed" : 0
    }
  }
}

---

For an alternative where it is possible to get metadata about several async requests IDs, we can introduce a new endpoint – metadata:

GET /_async_search/metadata
{
  "async_search_ids" : [
      "FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=",
      "DnF1ZXJ5VGhlbkZldGNoBQAAAAAJFUFEaMkZ5QTVSTZSaVN3WlNFVmtlWHJsdzoxMDc="
    ]
}

and a response here will consists of several responses:

{
  "took": 0,
  "responses": [
    {
      "id": "FmRldE8zREVEUzA2ZVpUeGs2ejJFUFEaMkZ5QTVrSTZSaVN3WlNFVmtlWHJsdzoxMDc=",
      "is_partial": true,
      "is_running": true,
      "start_time_in_millis": 1583945890986,
      "expiration_time_in_millis": 1584377890986,
      "response": {
        "took": 12144,
        "timed_out": false,
        "num_reduce_phases": 46,
        "_shards": {
          "total": 562,
          "successful": 188,
          "skipped": 0,
          "failed": 0
        }
      }
    },
    {
      "id": "DnF1ZXJ5VGhlbkZldGNoBQAAAAAJFUFEaMkZ5QTVSTZSaVN3WlNFVmtlWHJsdzoxMDc=",
      "is_partial": true,
      "is_running": true,
      "start_time_in_millis": 1583945890986,
      "expiration_time_in_millis": 1584377890986,
      "response": {
        "took": 12144,
        "timed_out": false,
        "num_reduce_phases": 46,
        "_shards": {
          "total": 562,
          "successful": 188,
          "skipped": 0,
          "failed": 0
        }
      }
    }
  ]
}

[javanna]

thanks for your thoughts @mayya-sharipova ! I have a preference for a specific API, given that the functionality it offers differ quite a bit from that of the get async search API that is used to retrieve results. It is more of a way to poll for the status of the async search. Also, given that we want this functionality to require different permissions, it may make sense to make it its own endpoint and transport action that requires its own role.

[lizozom]

@mayya-sharipova I think there is no single answer to this question:

From the perspective of Kibana, having a bulk API would be better, as that would allow us grouping queries, at least per Background Session, and minimizing the number of requests to be made.
From the perspective of simplicity for Elasticsearch, I assume, a per-async-search API is better, but then Kibana would be hitting it a lot more.

So as long as it's efficient enough for Elasticsearch, and as long as we can get the status of each async search somehow, then it's good.

I do think that having the same route have different permissions based on the metadata=true option is a bit weird. Don't you think?