Get service account credentials should show file tokens from all nodes

[ywangd]

The Get service account credentials API currently only returns file tokens from the local node (the node that handles the request). A more useful and easier to under behaviour is to gather and return file token information from all nodes. Exactly how these information should be consolidated from all nodes and represented in the response is open to discussion. But at a high level, it is suffice to say that these information is needed in the API response.

[elasticmachine]

Pinging @elastic/es-security (Team:Security)

[ywangd]

Elasticsearch has a NodesResponse that is used by APIs gathering information from or perform actions on all nodes, e.g. GET /_nodes and POST /_security/role/*/_clear_cache. The skeleton format of the response is something like the follows:

{
  "_nodes" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 1,
    "failures": [ {} ]
  },
  "cluster_name" : "de3c22e4aa31476f8d419932666a3eb7",
  "nodes" : {
    "IJ2HmeWLSSqA6lb0u-PRAA" : {
      "name" : "instance-0000000000",
      "...": "..."
    },
    "...": {
      "...": "..."
    }
  }
}

Note that the header setion (field "_nodes" and its value) contains overall stats about the execution. Intuitively, the above response format is applicable to the action of gathering file tokens from all nodes. While the header is indeed useful, the nested details section (field "nodes" and its value) is not a very friendly format for showing the file tokens.

Since we generally recommend a consistent set of file tokens to be configured for all nodes, a reasonble question that this API response can help answer is "Are file tokens configured correctly on all nodes". Users can deduce the answer from the nested "nodes" response. But it requires manaully aggregation on the client side. Since these information are already available when processing at the server side, it feels helpful if we could provide a response that answers the question more explicitly. Hence the proposed response format is like the follows:

{
  "service_account": "elastic/fleet-server",
  "count": 3,
  "tokens": {
    "token1": { }
  },
  "file_tokens": {
    "_nodes" : {
      "total" : 3,
      "successful" : 3,
      "failed" : 0
    },
    "my-token-1": {
      "nodes": ["n0", "n1", "n2"]
    },
    "my-token-2": {
      "nodes": ["n1", "n2"]
    }
  }
}

By comparing the "total" number and the size of each "nodes" field, the user can easily tell whether a file token is configured on all nodes.

A few things to note:

1. The NodesResponse header is placed inside the "file_tokens" because it does not apply to index-backed tokens. Since a service token name cannot begin with an underscore, it is safe to have the header under the field "_nodes".
2. It is practically impossible to determine whether two file tokens have the same secret in all cases. We simply do not attempt to check it and reply on the token name for aggregation. For the above sample response, it means the token with name my-token-1 is available on all three nodes. But their secrets may or may not be identical.
3. For index-backed tokens, we could choose to expose more information for each token, such as creation time, creator username and realm name (@legrego). It is not difficult to add them, but should we? Unlike API keys, service tokens do not expire and do not differentiate based on the creators. So these information does not provide much value in the response and feel more of an auditing requirement.

[tvernum]

One question in my head is What type of credentials will we add to service accounts in the future, and will they be node specific?.
I don't have an answer. However, I think there's a chance we'll support PKI (mTLS) for service accounts at some point, and maybe even integration with Cloud IAM services.

If any of those credentials are node specific, then the _nodes object will be apply to each of those credential types too. Should we duplicate it, or should it be top-level?

But if it's top-level, then it looks like applies to all credentials, when it definitely doesn't apply to "tokens".

I don't have any answer, but it's something to consider.

[ywangd]

Thanks for the comments and they are indeed worth considering.

If any of those credentials are node specific, then the _nodes object will be apply to each of those credential types too. Should we duplicate it, or should it be top-level?

I wonder whether there is value in "duplicating" the _nodes field for cases where the read operation for one type of node specific credentials succeeds while the other fails. I don't know how PKI or IAM would be configured for service accounts, maybe they are read from files or maybe they are validated at startup and can be retrieve from memory or maybe IAM should be validate when retrieving. If any of these operations can fail, it can fail independently from reading file_tokens. Even if they always succeeds, reading of file_tokens can still fail on its own. For these scenarios, it could be useful to have _nodes field for each type of node specific credentials. So I lean towards "duplicating" the _nodes field for each type of node specific credentials.

PS: It also prompted me to think a bit more about the implementation. I think it could be viable to have one action for each type of node specific credentials (instead of having one action for all of them) and have the caller to dispatch the requests and assemble the responses. The advantages are that the actions are more composable and failures are easier to propogated back to the caller.

[ywangd]

While working on the PR, I realised one potential issue with the current proposed response format for the file_tokens field. Since the _nodes field is nested under it, this means the value would have a single field with a fixed name and known value structure, while other fields can have arbitrary field name. I am not sure whether we have something like this already in our response. My impresssion is that we normally don't mix fixed and free field names at the same level since it could make parsing harder than it needs to be (I already felt it when working with HLRC).

Using the response for GET _nodes/stats as an example, the free field names (node names) are nested under a fixed nodes field which is at the same level as _nodes, e.g.:

{
  "_nodes" : {
    ...
  },
  "nodes" : {
    "FVAw1MPoQu-JKFVrh1tnKg" : {
    },
    "eaShNBuZQ_C79g9PMgdYEQ: {
    },
    ...
  }

If we follow this same style, the file_tokens field should be structured as:

  "nodes_credentials": {
    "_nodes" : {
      ...
    },
    "file_tokens": {
      "my-token-1": {
        "nodes": ["n0", "n1", "n2"]
      },
      "my-token-2": {
        "nodes": ["n1", "n2"]
      },
      ...
    }
    
  }

What do you think? @tvernum @bytebilly

EDIT (2021-08-03): The PR #75200 made a final tweak on the field names in above response. Instead of file_tokens.tokens, it is now nodes_credentials.file_tokens. This change is to make it easier to accomodate future additions to node local credentail types. Instead of having each node local credentails wrapped inside its own nodes section, we decided to have a single nodes section (nodes_credentials) to contain all of them. This choice is to avoid networking and auth overhead of fanning out multiple small requests, instead these requests (to retrieve different types of node local credentails) should be groupped per node.

[bytebilly]

Since it makes parser simpler and it's more consistent with what we already do in other API, I'm +1 for the latest proposal. It adds some complexity, but it's future proof and it's an acceptable tradeoff.

[tvernum]

I'm fine with the most recent proposal.