cluster/gce: Add env var to enable apiserver basic audit log. #41211

enisoc · 2017-02-09T19:33:07Z

For now, this is focused on a fixed set of flags that makes the audit
log show up under /var/log/kube-apiserver-audit.log and behave similarly
to /var/log/kube-apiserver.log. Allowing other customization would
require significantly more complex changes.

Audit log rotation is handled the same as for kube-apiserver.log.

What this PR does / why we need it:

Add a knob to enable basic audit logging in GCE.

Which issue this PR fixes:

Special notes for your reviewer:

We would like to cherrypick/port this to release-1.5 also.

Release note:

The kube-apiserver [basic audit log](https://kubernetes.io/docs/admin/audit/) can be enabled in GCE by exporting the environment variable `ENABLE_APISERVER_BASIC_AUDIT=true` before running `cluster/kube-up.sh`. This will log to `/var/log/kube-apiserver-audit.log` and use the same `logrotate` settings as `/var/log/kube-apiserver.log`.

k8s-reviewable · 2017-02-09T19:34:18Z

This change is

cjcullen · 2017-02-09T21:35:59Z

cluster/gce/container-linux/configure-helper.sh

+    # Lumberjack doesn't offer any way to disable size-based rotation. It also
+    # has an in-memory counter that doesn't notice if you truncate the file.
+    # 1000000000 (MiB) is a large number that fits in 31 bits. If the log grows
+    # by 1MiB/s, it will rotate after ~31 years if apiserver never restarts.


It looks like these logs are about 300B/request, so 1MiB/s translates into ~3500 Req/s. That seems reasonable to me, but could we be spammy enough to bring this into the O(days) category? I think it would fill up in 1 day at ~40M req/s, which is almost certainly higher than any one machine can handle, right?

40M QPS does seem higher than you'd reasonably expect. At 1.3M QPS, you could hit the limit in 30 days. Even that seems rather high for a kube-apiserver though. There is room in the 31 bits to double this limit if you want. I only chose 1000000000 so it's obvious to humans that it's an arbitrary large number. 2000000000 would also have mostly the same effect.

The long-term fix is to add a new flag or special value for --audit-log-maxsize that makes it bypass Lumberjack and write directly to a file. However, I was hoping to avoid touching anything in the actual binary to make this change minimally invasive. Maybe I can do that as a separate PR on master only after we get this one in and cherrypicked into 1.5?

Is there a reason you don't configure logrotate for this log? That would solve this comment, I think.

I'm actually pretty concerned with anything log-like that can grow unbounded.

The idea here is to disable the audit log feature's built-in log rotation (Lumberjack) so we can use the same logrotate facility we already have for kube-apiserver.log.

The current logrotate settings rotate everything matching the wildcard /var/log/*.log so this new /var/log/kube-apiserver-audit.log is included. I verified that it rotates as expected.

https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/gci/configure-helper.sh#L124

Oh wait I just realized the container-linux version doesn't share the logrotate config. In that case, I think I should revert my change from container-linux and only update GCI since that's what I am able to test.

Ah, GCI/COS is covered. I was looking at the Salt config, which looks like it needs to be updated? https://github.com/kubernetes/kubernetes/blob/master/cluster/saltbase/salt/logrotate/init.sls

Ah so that's where logrotate is configured for non-GCI. I've pushed an update to that file. Thanks!

zmerlynn · 2017-02-14T22:34:41Z

This more or less LGTM, except the comment above.

eparis

I feel like most of these things are using 'defined' to mean true instead of needing to have the exact magic value/string "true". but I'm ok with this if you really want it this way.

eparis · 2017-02-14T22:37:52Z

cluster/gce/trusty/configure-helper.sh

+    # We currently only support enabling with a fixed path and with built-in log
+    # rotation "disabled" (large value) so it behaves like kube-apiserver.log.
+    # External log rotation should be set up the same as for kube-apiserver.log.
+    params="${params} --audit-log-path=/var/log/kube-apiserver-audit.log"


is it wrong that I like the params += style you used the other 2 times. And is sometimes used in this file?

I like += better too, but I wanted to follow the style used within each file so it's self-consistent.

I'm actually thinking about reverting my change to the trusty and container-linux versions because I don't know how log rotation works on those and updating those other flavors is beyond the scope of what I'm trying to accomplish (getting this working for GCE w/ GCI).

eparis · 2017-02-14T22:45:40Z

/approve

enisoc · 2017-02-14T22:57:12Z

I feel like most of these things are using 'defined' to mean true instead of needing to have the exact magic value/string "true". but I'm ok with this if you really want it this way.

I preferred to require "true" because other env vars like ENABLE_GARBAGE_COLLECTOR have set the precedent that you can set it to "false" to disable it. That works in those cases because the value is passed to a boolean command-line flag if non-empty. In this case, there is no boolean command-line flag to enable/disable audit logging, so I would have to explicitly check for "false" to avoid enabling the log.

Given the choice between checking for "true" and checking for "false", I chose "true" because I saw many examples of doing that: ENABLE_MANIFEST_URL, ENABLE_CLUSTER_AUTOSCALER, etc.

zmerlynn · 2017-02-14T22:58:41Z

Yes, please check for true. I don't know why there are other styles in here at all, but e.g. GKE tends to just pass FOO={true,false} in the kube-env.

For now, this is focused on a fixed set of flags that makes the audit log show up under /var/log/kube-apiserver-audit.log and behave similarly to /var/log/kube-apiserver.log. Allowing other customization would require significantly more complex changes. Audit log rotation is handled externally by the wildcard /var/log/*.log already configured in configure-helper.sh.

k8s-github-robot · 2017-02-14T23:18:53Z

[APPROVALNOTIFIER] This PR is APPROVED

The following people have approved this PR: enisoc, eparis

Needs approval from an approver in each of these OWNERS Files:

~~cluster/OWNERS~~ [eparis]
~~cluster/gce/container-linux/OWNERS~~ [eparis]
~~hack/OWNERS~~ [eparis]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

zmerlynn · 2017-02-14T23:20:29Z

/lgtm

k8s-ci-robot · 2017-02-14T23:20:30Z

@zmerlynn: you can't LGTM a PR unless you are an assignee.

In response to this comment:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

zmerlynn · 2017-02-14T23:20:43Z

/lgtm

zmerlynn · 2017-02-15T00:17:36Z

@k8s-bot cvm gke e2e test this

k8s-github-robot · 2017-02-15T11:25:11Z

Automatic merge from submit-queue (batch tested with PRs 40297, 41285, 41211, 41243, 39735)

Automatic merge from submit-queue fluentd-gcp: Add kube-apiserver-audit.log. **What this PR does / why we need it**: Add `kube-apiserver-audit.log` from #41211 to fluentd config, so the audit log gets sent to the same place as `kube-apiserver.log`. **Which issue this PR fixes**: **Special notes for your reviewer**: We would like to backport this to release-1.5 also. **Release note**: ```release-note The apiserver audit log (`/var/log/kube-apiserver-audit.log`) will be sent through fluentd if enabled. ```

…-upstream-release-1.5 Automatic merge from submit-queue Automated cherry pick of #41211 Cherry pick of #41211 on release-1.5. #41211: cluster/gce: Add env var to enable apiserver basic audit log.

k8s-cherrypick-bot · 2017-02-24T14:13:47Z

Commit found in the "release-1.5" branch appears to be this PR. Removing the "cherrypick-candidate" label. If this is an error find help to get your PR picked.

enisoc added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Feb 9, 2017

enisoc requested a review from cjcullen February 9, 2017 19:33

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Feb 9, 2017

k8s-github-robot assigned eparis Feb 9, 2017

k8s-github-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Feb 9, 2017

cjcullen reviewed Feb 9, 2017

View reviewed changes

enisoc force-pushed the configure-audit-log branch 2 times, most recently from 9ab9f37 to 06533af Compare February 10, 2017 19:34

enisoc mentioned this pull request Feb 13, 2017

fluentd-gcp: Add kube-apiserver-audit.log. #41360

Merged

eparis reviewed Feb 14, 2017

View reviewed changes

k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 14, 2017

enisoc force-pushed the configure-audit-log branch from 06533af to 7500746 Compare February 14, 2017 23:18

zmerlynn self-assigned this Feb 14, 2017

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 14, 2017

k8s-github-robot merged commit 4c02f29 into kubernetes:master Feb 15, 2017

enisoc added this to the v1.5 milestone Feb 15, 2017

enisoc added the cherrypick-candidate label Feb 15, 2017

enisoc mentioned this pull request Feb 16, 2017

Automated cherry pick of #41211 #41577

Merged

k8s-cherrypick-bot removed the cherrypick-candidate label Feb 24, 2017

enisoc deleted the configure-audit-log branch March 27, 2017 18:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cluster/gce: Add env var to enable apiserver basic audit log. #41211

cluster/gce: Add env var to enable apiserver basic audit log. #41211

enisoc commented Feb 9, 2017 •

edited

Loading

k8s-reviewable commented Feb 9, 2017

cjcullen Feb 9, 2017

enisoc Feb 9, 2017

cjcullen Feb 12, 2017

zmerlynn Feb 14, 2017

enisoc Feb 14, 2017

enisoc Feb 14, 2017

zmerlynn Feb 14, 2017

enisoc Feb 14, 2017

zmerlynn commented Feb 14, 2017

eparis left a comment

eparis Feb 14, 2017

enisoc Feb 14, 2017

eparis commented Feb 14, 2017

enisoc commented Feb 14, 2017

zmerlynn commented Feb 14, 2017

k8s-github-robot commented Feb 14, 2017

zmerlynn commented Feb 14, 2017

k8s-ci-robot commented Feb 14, 2017

zmerlynn commented Feb 14, 2017

zmerlynn commented Feb 15, 2017

k8s-github-robot commented Feb 15, 2017

k8s-cherrypick-bot commented Feb 24, 2017

cluster/gce: Add env var to enable apiserver basic audit log. #41211

cluster/gce: Add env var to enable apiserver basic audit log. #41211

Conversation

enisoc commented Feb 9, 2017 • edited Loading

k8s-reviewable commented Feb 9, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zmerlynn commented Feb 14, 2017

eparis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eparis commented Feb 14, 2017

enisoc commented Feb 14, 2017

zmerlynn commented Feb 14, 2017

k8s-github-robot commented Feb 14, 2017

zmerlynn commented Feb 14, 2017

k8s-ci-robot commented Feb 14, 2017

zmerlynn commented Feb 14, 2017

zmerlynn commented Feb 15, 2017

k8s-github-robot commented Feb 15, 2017

k8s-cherrypick-bot commented Feb 24, 2017

enisoc commented Feb 9, 2017 •

edited

Loading