Service tokens ignore privileges additionally defined by packages

[simitt]

Removing the username/password and enforcing usage of service tokens (#1006) revealed a bug related to the privileges that the resulting API Keys have. It seems that any additionally defined privileges of the packages are just ignored.

Description of the Problem

0. The apmpackage specifies additional privileges for the traces-apm.sampled-default data stream:

elasticsearch:
  privileges:
    indices: [auto_configure, create_doc, maintenance, monitor, read]

1. Setting up an agent policy that contains the apm integration and enrolling an elastic agent, the state/data/state.yml lists the privileges for the data streams, as configured in the apmpackage:

bash-4.2$ cat state/data/state.yml

action:
    ...
    inputs:
    - apm-server:
       ...
      data_stream:
        namespace: default
      id: c0f3d4d1-acc9-447c-a443-f9049dab8ee5
      meta:
        package:
          name: apm
          version: 8.1.0
      name: apm_systemtest_2-apm
      revision: 1
      type: apm
      use_output: default
    output_permissions:
      default:
        _elastic_agent_checks:
          cluster:
          - monitor
        _elastic_agent_monitoring:
         ...
        apm_systemtest_2-apm:
          cluster:
          - cluster:monitor/main
          indices:
          - names:
            - logs-apm.app-default
            privileges:
            - auto_configure
            - create_doc
          - names:
            - metrics-apm.app.*-default
            privileges:
            - auto_configure
            - create_doc
          - names:
            - logs-apm.error-default
            privileges:
            - auto_configure
            - create_doc
          - names:
            - metrics-apm.internal-default
            privileges:
            - auto_configure
            - create_doc
          - names:
            - metrics-apm.profiling-default
            privileges:
            - auto_configure
            - create_doc
          - names:
            - traces-apm.rum-default
            privileges:
            - auto_configure
            - create_doc
          - names:
            - traces-apm.sampled-default
            privileges:
            - auto_configure
            - create_doc
            - maintenance
            - monitor
            - read
          - names:
            - traces-apm-default
            privileges:
            - auto_configure
            - create_doc
    outputs:
      default:
        api_key: vOL9Cn4BqaHYuqz-hbSb:Iboop-tIQjKH7tLzzFH0Yg
        hosts:
        - http://elasticsearch:9200
        type: elasticsearch
    revision: 2

2. Using this base64 encoded API Key and querying for the actual privileges shows that privileges are missing:

simitt@simmac ~ % curl -H "Authorization: ApiKey dk9MOUNuNEJxYUhZdXF6LWhiU2I6SWJvb3AtdElRaktIN3RMenpGSDBZZw==" -X GET "localhost:9200/_security/user/_has_privileges?pretty" -H 'Content-Type: application/json' -d'
{
  "cluster": [ "cluster:monitor/main" ],
  "index" : [
    {
      "names": [ "logs-apm.app-default", "metrics-apm.app.*-default", "logs-apm.error-default", "metrics-apm.internal-default", "metrics-apm.profiling-default", "traces-apm.rum-default", "traces-apm-default" ],
      "privileges": [ "auto_configure","create_doc" ]
    },
    {
      "names": [ "traces-apm.sampled-default" ],
      "privileges": [ "auto_configure","create_doc","maintenance","monitor","read" ]
    }
  ]
}
'



{
  "username" : "elastic/fleet-server",
  "has_all_requested" : false,
  "cluster" : {
    "cluster:monitor/main" : true
  },
  "index" : {
    "logs-apm.app-default" : {
      "create_doc" : true,
      "auto_configure" : true
    },
    "logs-apm.error-default" : {
      "create_doc" : true,
      "auto_configure" : true
    },
    "metrics-apm.app.*-default" : {
      "create_doc" : true,
      "auto_configure" : true
    },
    "metrics-apm.internal-default" : {
      "create_doc" : true,
      "auto_configure" : true
    },
    "metrics-apm.profiling-default" : {
      "create_doc" : true,
      "auto_configure" : true
    },
    "traces-apm-default" : {
      "create_doc" : true,
      "auto_configure" : true
    },
    "traces-apm.rum-default" : {
      "create_doc" : true,
      "auto_configure" : true
    },
    "traces-apm.sampled-default" : {
      "read" : false,
      "create_doc" : true,
      "auto_configure" : true,
      "monitor" : false,
      "maintenance" : false
    }
  },
  "application" : { }
}

3. Verifying read privileges with a concrete request confirms that they are indeed missing:

simitt@simmac ~ % curl -i -H "Authorization: ApiKey dk9MOUNuNEJxYUhZdXF6LWhiU2I6SWJvb3AtdElRaktIN3RMenpGSDBZZw==" "http://localhost:9200/traces-apm-default/_search"
HTTP/1.1 403 Forbidden
X-elastic-product: Elasticsearch
content-type: application/json;charset=utf-8
content-length: 621

{"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:data/read/search] is unauthorized for API key id [vOL9Cn4BqaHYuqz-hbSb] of user [elastic/fleet-server] on indices [traces-apm-default,.ds-traces-apm-default-2021.12.30-000001], this action is granted by the index privileges [read,all]"}],"type":"security_exception","reason":"action [indices:data/read/search] is unauthorized for API key id [vOL9Cn4BqaHYuqz-hbSb] of user [elastic/fleet-server] on indices [traces-apm-default,.ds-traces-apm-default-2021.12.30-000001], this action is granted by the index privileges [read,all]"},"status":403}%

[michel-laterman]

I'm not sure if this is related, but I've noticed that the service_token that Kibana issues is unable to list policies:

elastic-agent|retrieve-service-token-container⚡ ⇒ curl -XPOST localhost:5601/api/fleet/service-tokens -u elastic:changeme -H "kbn-xsrf: value"
{"name":"token-1640912238005","value":"AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2NDA5MTIyMzgwMDU6ak53LTZHdkFTZnFrN0dTal9SM3BiUQ"}%
elastic-agent|retrieve-service-token-container⚡ ⇒ curl localhost:5601/api/fleet/agent_policies -H "Authorization: Bearer AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2NDA5MTIyMzgwMDU6ak53LTZHdkFTZnFrN0dTal9SM3BiUQ"
{"items":[],"total":0,"page":1,"perPage":20}%
elastic-agent|retrieve-service-token-container⚡ ⇒ curl localhost:5601/api/fleet/agent_policies -u elastic:changeme
{"items":[{"id":"499b5aa7-d214-5b5d-838b-3cd76469844e","namespace":"default","monitoring_enabled":["logs","metrics"],"name":"Default Fleet Server policy","description":"Default Fleet Server agent policy created by Kibana","is_default":false,"is_default_fleet_server":true,"is_preconfigured":true,"status":"active","is_managed":false,"revision":1,"updated_at":"2021-12-31T00:54:04.727Z","updated_by":"system","package_policies":["default-fleet-server-agent-policy"],"agents":1},{"id":"2016d7cc-135e-5583-9758-3ba01f5a06e5","namespace":"default","monitoring_enabled":["logs","metrics"],"name":"Default policy","description":"Default agent policy created by Kibana","is_default":true,"is_preconfigured":true,"status":"active","is_managed":false,"revision":1,"updated_at":"2021-12-31T00:54:02.703Z","updated_by":"system","package_policies":["default-system-policy"],"agents":0}],"total":2,"page":1,"perPage":20}%

[jlind23]

@simitt do you have any logs on fleet-server side while trying to create the token?
cc @blakerouse @michel-laterman

[blakerouse]

@simitt The permissions show up when using username/password, but not with the service token? If that is the case the Fleet Server calls the same API with the same contents no matter basic auth or service token. This makes me think its an issue on the Elasticsearch side.

As @jlind23 ask do you have any Elasticsearch logs for this, it might show errors about not being able to assign those permissions.

[axw]

There is a single log record from Elasticsearch in our test setup:

elasticsearch_1 | {"@timestamp":"2022-01-06T04:25:38.550Z", "log.level": "WARN", "message":"path: /traces-*/_search, params: {size=200, index=traces-*}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[aba19b0b0e12][search][T#7]","log.logger":"rest.suppressed","elasticsearch.cluster.uuid":"6MdsLK9FS22dUqR23UdxOw","elasticsearch.node.id":"fjTJPyHZR7W4kZyC31hPgg","elasticsearch.node.name":"aba19b0b0e12","elasticsearch.cluster.name":"docker-cluster","error.type":"org.elasticsearch.action.search.SearchPhaseExecutionException","error.message":"all shards failed","error.stack_trace":"Failed to execute phase [query], all shards failed; shardFailures {[fjTJPyHZR7W4kZyC31hPgg][.ds-traces-apm.sampled-default-2022.01.06-000001][0]: [.ds-traces-apm.sampled-default-2022.01.06-000001/UFpv0gZsQ4SMVnD9d5Q_2A][[.ds-traces-apm.sampled-default-2022.01.06-000001][0]] org.elasticsearch.action.NoShardAvailableActionException: [aba19b0b0e12][127.0.0.1:9300][indices:data/read/search[phase/query]]\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:544)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:491)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:343)\n\tat org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66)\n\tat org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:48)\n\tat org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:651)\n\tat org.elasticsearch.transport.TransportService$4.handleException(TransportService.java:724)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1349)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1458)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1432)\n\tat org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:50)\n\tat org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:47)\n\tat org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:77)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:775)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n}\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:725)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:412)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:757)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:509)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:343)\n\tat org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66)\n\tat org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:48)\n\tat org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:651)\n\tat org.elasticsearch.transport.TransportService$4.handleException(TransportService.java:724)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1349)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1458)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1432)\n\tat org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:50)\n\tat org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:47)\n\tat org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:77)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:775)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: [.ds-traces-apm.sampled-default-2022.01.06-000001/UFpv0gZsQ4SMVnD9d5Q_2A][[.ds-traces-apm.sampled-default-2022.01.06-000001][0]] org.elasticsearch.action.NoShardAvailableActionException: [aba19b0b0e12][127.0.0.1:9300][indices:data/read/search[phase/query]]\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:544)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:491)\n\t... 18 more\n"}

The error message doesn't explicitly say so, but I'm pretty sure this is due to lack of read privilege on traces-apm.sampled-default.

API Keys have at most the privileges of the authenticated user. When fleet-server is run with the root user (as we were doing with user/pass auth in our tests previously), that's everything, so fleet-server would produce API Keys with the additional privileges we request.

The elastic/fleet-server service account has limited index privileges:

            new RoleDescriptor.IndicesPrivileges[] {
                RoleDescriptor.IndicesPrivileges.builder()
                    .indices(
                        "logs-*",
                        "metrics-*",
                        "traces-*",
                        "synthetics-*",
                        ".logs-endpoint.diagnostic.collection-*",
                        ".logs-endpoint.action.responses-*"
                    )
                    .privileges("write", "create_index", "auto_configure")
                    .build(),

Thus, the maximum set of index privileges API Keys issues by fleet-server is write, create_index, and auto_configure.

@ruflin this undoes elastic/package-spec#203. How would you suggest we proceed?

[jlind23]

@axw It may be a dumb question but couldn't we add the read right to the elastic/fleet-server service account?

[axw]

@jlind23 we could, I'm just not sure if this was intended. If not, then yes we could expand the privileges; if it is intentionally/necessarily limited, we need to go back to the drawing board on elastic/package-spec#203.

[jlind23]

So you can specify privileges in elastic package only if those privileges are already part of fleet-server ones right?
If yes, then I am pretty sure that it should be extended. Who owns the decision there? @ruflin ?

[axw]

So you can specify privileges in elastic package only if those privileges are already part of fleet-server ones right?

Right. The ones in the elastic/fleet-server service account define the maximum set of privileges an agent's API Key can be assigned.

If yes, then I am pretty sure that it should be extended. Who owns the decision there? @ruflin ?

Yes, I think so. If not he can redirect :)