From 7e8c8d07cad6fe50f9de900894b90e5bf3dba863 Mon Sep 17 00:00:00 2001 From: Cao Shufeng Date: Tue, 5 Sep 2017 19:45:05 +0800 Subject: [PATCH] advanced audit beta features --- docs/tasks/debug-application-cluster/audit.md | 175 +++++++++++------- 1 file changed, 113 insertions(+), 62 deletions(-) diff --git a/docs/tasks/debug-application-cluster/audit.md b/docs/tasks/debug-application-cluster/audit.md index a595469b5b913..61ebfcd294384 100644 --- a/docs/tasks/debug-application-cluster/audit.md +++ b/docs/tasks/debug-application-cluster/audit.md @@ -22,7 +22,7 @@ answer the following questions: - from where was it initiated? - to where was it going? -## Audit logs +## Legacy Audit Kubernetes audit is part of [Kube-apiserver][kube-apiserver] logging all requests processed by the server. Each audit log entry contains two lines: @@ -37,8 +37,13 @@ Example output for `admin` user listing pods in the `default` namespace: 2017-03-21T03:57:09.108403639-04:00 AUDIT: id="c939d2a7-1c37-4ef1-b2f7-4ba9b1e43b53" response="200" ``` -Note that this format changes when enabling the `AdvancedAuditing` feature discussed -later in this document. +Note that Kubernetes 1.8 has switched to use the advanced structured audit log by default. +To fallback to this legacy audit, disable the advanced auditing feature +using the `AdvancedAuditing` feature gate on the [kube-apiserver][kube-apiserver]: + +``` +--feature-gates=AdvancedAuditing=false +``` ### Configuration @@ -62,18 +67,12 @@ and `audit-log-maxage` options. ## Advanced audit Kubernetes 1.7 expands auditing with experimental functionality such as event -filtering and a webhook for integration with external systems. The rest of this -document covers features that are __alpha__ and may change in backward incompatible -ways. +filtering and a webhook for integration with external systems. Kubernetes 1.8 +upgrades the advanced audit feature to beta, and some backward incompatible changes +have been committed. -Enable the alpha auditing features using the `AdvancedAuditing` feature gate on -the [kube-apiserver][kube-apiserver]: -``` ---feature-gates=AdvancedAuditing=true -``` - -`AdvancedAuditing`is customizable in two ways. Policy, which determines what's recorded, +`AdvancedAuditing` is customizable in two ways. Policy, which determines what's recorded, and backends, which persist records. Backend implementations include logs files and webhooks. @@ -110,10 +109,17 @@ The policy file holds rules that determine the level of an event. Known audit le When an event is processed, it's compared against the list of rules in order. The first matching rule sets the audit level of the event. The audit policy is defined by the [`audit.k8s.io` API group][audit-api]. +Some new fields are supported in beta version, like `resourceNames` and `omitStages`. + +In Kubernetes 1.8 `kind` and `apiVersion` along with `rules` __must__ be supplied to +the the audit policy file. A policy file with 0 rules, or a policy file doesn't provide +valid `apiVersion` and `kind` will be treated as illgal. -An example audit policy file: +Some example audit policy files: ```yaml +apiVersion: audit.k8s.io/v1beta1 #this is necessary in Kubernetes 1.8 +kind: Policy rules: # Don't log watch requests by the "system:kube-proxy" on endpoints or services - level: None @@ -155,10 +161,46 @@ rules: - level: Metadata ``` +The next audit policy file shows new features Kubernetes 1.8: + +```yaml +apiVersion: audit.k8s.io/v1beta1 +kind: Policy +rules: + # Log pod changes at Request level + - level: Request + resources: + - group: "" + # Resource "pods" no longer matches requests to any subresource of pods, + # This behavior is consistent with the RBAC policy. + resources: ["pods"] + # Log "pods/log", "pods/status" at Metadata level + - level: Metadata + resources: + - group: "" + resources: ["pods/log", "pods/status"] + + # Not log requests to a configmap called "controller-leader" + - level: None + resources: + - group: "" + resources: ["configmaps"] + resourceNames: ["controller-leader"] + + # A catch-all rule to log all other requests at the Metadata level. + # For this rule we use "omitStages" to omit events at "ReqeustReceived" stage. + # events in this stage will not be sent to backend. + - level: Metadata + omitStages: + - "RequestReceived" +``` + You can use a minimal audit policy file to log all requests at the `Metadata` level: ```yaml # Log all requests at the Metadata level. +apiVersion: audit.k8s.io/v1beta1 +kind: Policy rules: - level: Metadata ``` @@ -183,31 +225,34 @@ API can be found [here][audit-api] with more details about the exact fields capt The behavior of the `--audit-log-path` flag changes when enabling the `AdvancedAuditing` feature flag. This includes the cleanups discussed above, such as changes to the `method` values and the introduction of a "stage" for each event. As before, the `id` field of -the log indicates which events were generated from the same request. With default legacy +the log indicates which events were generated from the same request. With default json format, events are formatted as follows: ``` -2017-06-15T21:50:50.259470834Z AUDIT: id="591e9fde-6a98-46f6-b7bc-ec8ef575696d" stage="RequestReceived" ip="10.2.1.3" method="update" user="system:serviceaccount:kube-system:default" groups="\"system:serviceaccounts\",\"system:serviceaccounts:kube-system\",\"system:authenticated\"" as="" asgroups="" namespace="kube-system" uri="/api/v1/namespaces/kube-system/endpoints/kube-controller-manager" response="" -2017-06-15T21:50:50.259470834Z AUDIT: id="591e9fde-6a98-46f6-b7bc-ec8ef575696d" stage="ResponseComplete" ip="10.2.1.3" method="update" user="system:serviceaccount:kube-system:default" groups="\"system:serviceaccounts\",\"system:serviceaccounts:kube-system\",\"system:authenticated\"" as="" asgroups="" namespace="kube-system" uri="/api/v1/namespaces/kube-system/endpoints/kube-controller-manager" response="200" +{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"level":"Metadata","timestamp":"2017-09-05T10:04:55Z","auditID":"77e58433-d345-40ac-b2d8-9866bd355cea","stage":"RequestReceived","requestURI":"/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles","verb":"list","user":{"username":"kubecfg","groups":["system:masters","system:authenticated"]},"sourceIPs":["172.16.116.128"],"objectRef":{"resource":"roles","namespace":"default","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"}} +{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"level":"Metadata","timestamp":"2017-09-05T10:04:55Z","auditID":"77e58433-d345-40ac-b2d8-9866bd355cea","stage":"ResponseComplete","requestURI":"/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles","verb":"list","user":{"username":"kubecfg","groups":["system:masters","system:authenticated"]},"sourceIPs":["172.16.116.128"],"objectRef":{"resource":"roles","namespace":"default","apiGroup":"rbac.authorization.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200}} ``` -Logged events omit the request and response bodies. The `Request` and -`RequestResponse` levels are equivalent to `Metadata` for legacy format. +In alpha version, objectRef.apiVersion holds both the api group and version. +In beta version these were break out into objectRef.apiGroup and objectRef.apiVersion. -Since Kubernetes 1.8, structed json fromat is supported for log backend. -Use the following option to switch log to json format: +Since Kubernetes 1.8, structed json fromat is used for log backend by default. +Use the following option to switch log to legacy format: ``` ---audit-log-format=json +--audit-log-format=legacy ``` -With json format, events are formatted as follows: +With legacy format, events are formatted as follows: ``` -{"kind":"Event","apiVersion":"audit.k8s.io/v1alpha1","metadata":{"creationTimestamp":null},"level":"Metadata","timestamp":"2017-07-12T11:02:43Z","auditID":"2e79f0c9-a941-45ae-a9ce-663a1b19ff14","stage":"RequestReceived","requestURI":"/api/v1/namespaces/default/pods","verb":"list","user":{"username":"kubecfg","groups":["system:masters","system:authenticated"]},"sourceIPs":["172.16.116.128"],"objectRef":{"resource":"pods","namespace":"default","apiVersion":"/v1"}} -{"kind":"Event","apiVersion":"audit.k8s.io/v1alpha1","metadata":{"creationTimestamp":null},"level":"Metadata","timestamp":"2017-07-12T11:02:43Z","auditID":"2e79f0c9-a941-45ae-a9ce-663a1b19ff14","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/default/pods","verb":"list","user":{"username":"kubecfg","groups":["system:masters","system:authenticated"]},"sourceIPs":["172.16.116.128"],"objectRef":{"resource":"pods","namespace":"default","apiVersion":"/v1"},"responseStatus":{"metadata":{},"code":200}} +2017-09-05T06:08:19.885328047-04:00 AUDIT: id="c28a95ad-f9dd-47e1-a617-b6dc152db95f" stage="RequestReceived" ip="172.16.116.128" method="list" user="kubecfg" groups="\"system:masters\",\"system:authenticated\"" as="" asgroups="" namespace="default" uri="/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles" response="" +2017-09-05T06:08:19.885328047-04:00 AUDIT: id="c28a95ad-f9dd-47e1-a617-b6dc152db95f" stage="ResponseComplete" ip="172.16.116.128" method="list" user="kubecfg" groups="\"system:masters\",\"system:authenticated\"" as="" asgroups="" namespace="default" uri="/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles" response="200" ``` +Logged events omit the request and response bodies. The `Request` and +`RequestResponse` levels are equivalent to `Metadata` for legacy format. + #### Webhook backend The audit webhook backend can be used to have [kube-apiserver][kube-apiserver] @@ -255,41 +300,43 @@ Events are POSTed as a JSON serialized `EventList`. An example payload: ```json { - "kind": "EventList", - "apiVersion": "audit.k8s.io/v1alpha1", - "items": [ - { - "metadata": { - "creationTimestamp": null - }, - "level": "Metadata", - "timestamp": "2017-06-15T23:07:40Z", - "auditID": "4faf711a-9094-400f-a876-d9188ceda548", - "stage": "ResponseComplete", - "requestURI": "/apis/rbac.authorization.k8s.io/v1beta1/namespaces/kube-public/rolebindings/system:controller:bootstrap-signer", - "verb": "get", - "user": { - "username": "system:apiserver", - "uid": "97a62906-e4d7-4048-8eda-4f0fb6ff8f1e", - "groups": [ - "system:masters" - ] - }, - "sourceIPs": [ - "127.0.0.1" - ], - "objectRef": { - "resource": "rolebindings", - "namespace": "kube-public", - "name": "system:controller:bootstrap-signer", - "apiVersion": "rbac.authorization.k8s.io/v1beta1" - }, - "responseStatus": { - "metadata": {}, - "code": 200 - } - } - ] + "apiVersion": "audit.k8s.io/v1beta1", + "items": [ + { + "auditID": "24f30caf-d7d4-45d5-b7bd-e7af300d7886", + "level": "Metadata", + "metadata": { + "creationTimestamp": null + }, + "objectRef": { + "apiGroup": "rbac.authorization.k8s.io", + "apiVersion": "v1", + "name": "jane", + "namespace": "default", + "resource": "roles" + }, + "requestURI": "/apis/rbac.authorization.k8s.io/v1/namespaces/default/roles/jane", + "responseStatus": { + "code": 200, + "metadata": {} + }, + "sourceIPs": [ + "172.16.116.128" + ], + "stage": "ResponseComplete", + "timestamp": "2017-09-05T10:20:24Z", + "user": { + "groups": [ + "system:masters", + "system:authenticated" + ], + "username": "kubecfg" + }, + "verb": "get" + } + ], + "kind": "EventList", + "metadata": {} } ``` @@ -422,9 +469,13 @@ Note that in addition to file output plugin, logstash has a variety of outputs t let users route data where they want. For example, users can emit audit events to elasticsearch plugin which supports full-text search and analytics. -[audit-api]: https://github.com/kubernetes/kubernetes/blob/v1.7.0-rc.1/staging/src/k8s.io/apiserver/pkg/apis/audit/v1alpha1/types.go + +[audit-api]: https://github.com/kubernetes/kubernetes/blob/v1.8.0-beta.0/staging/src/k8s.io/apiserver/pkg/apis/audit/v1beta1/types.go [kube-apiserver]: /docs/admin/kube-apiserver -[gce-audit-profile]: https://github.com/kubernetes/kubernetes/blob/v1.7.0/cluster/gce/gci/configure-helper.sh#L490 +[gce-audit-profile]: https://github.com/kubernetes/kubernetes/blob/v1.8.0-beta.0/cluster/gce/gci/configure-helper.sh#L532 [fluentd]: http://www.fluentd.org/ [fluentd_install_doc]: http://docs.fluentd.org/v0.12/articles/quickstart#step1-installing-fluentd [logstash]: https://www.elastic.co/products/logstash