AUTH-2: reenable PodSecurity on privileged level by stlaz · Pull Request #1308 · openshift/cluster-kube-apiserver-operator

stlaz · 2022-02-08T15:07:57Z

This enables the PodSecurity admission with baseline level (for now). I suspect it might break tests, which is where @s-urbaniak's upstream involvement in the e2e framework might come handy.

/cc @s-urbaniak
/cc @openshift/openshift-team-auth-maintainers

s-urbaniak · 2022-02-09T09:43:06Z

yes i see the reason failures in the e2e logs, however audit doesn't contain them, i.e. from https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_cluster-kube-apiserver-operator/1308/pull-ci-openshift-cluster-kube-apiserver-operator-master-e2e-aws-serial/1491066281317634048:

% gzcat *-audit-*log.gz | jq '. | select(.annotations | has("pod-security.kubernetes.io/audit"))' | wc -l
       0

@stlaz can you check why we have no audit entries for pod-security?

s-urbaniak · 2022-02-09T10:39:34Z

xref'ing openshift/kubernetes#1128

this is the backport PR which I am keeping up-to-date with upstream.

s-urbaniak · 2022-02-09T15:02:22Z

@stlaz i see you bumped audit entries to restricted. i was surprised not to see violations against baseline though given e2e tests didn't pass 🤔

stlaz · 2022-02-09T15:38:42Z

I did not notice your earlier comments and was bumping the audit and warn levels in the meantime to see what's going to block us from moving further to the more restrictive level. I'll check the audit logs now and see whether we've got something.

stlaz · 2022-02-09T15:41:31Z

gzcat *-audit-*log.gz | jq '. | select(.annotations | has("pod-security.kubernetes.io/audit"))' | wc -l

@s-urbaniak I see what's the issue - upstream changed the annotation to pod-security.kubernetes.io/audit-violations in the meantime.

s-urbaniak · 2022-02-10T10:01:45Z

changed the annotation to pod-security.kubernetes.io/audit-violations in the meantime.

doh, you're right. now we get nice result of e2e test offenders:

% gzcat *-audit-*log.gz | jq '. | select(.annotations | has("pod-security.kubernetes.io/audit-violations")) | {namespace: .objectRef.namespace, podSecurityReason: .annotations["pod-security.kubernetes.io/audit-violations"] }' | jq -C -s --sort-keys '. | unique' | head -n 20
[
  {
    "namespace": "e2e-disruption-131",
    "podSecurityReason": "would violate PodSecurity \"baseline:latest\": hostPort (container \"donothing\" uses hostPort 5555)"
  },
  {
    "namespace": "e2e-disruption-8214",
    "podSecurityReason": "would violate PodSecurity \"baseline:latest\": hostPort (container \"donothing\" uses hostPort 5555)"
  },
  {
    "namespace": "e2e-persistent-local-volumes-test-4715",
    "podSecurityReason": "would violate PodSecurity \"baseline:latest\": host namespaces (hostNetwork=true), hostPath volumes (volume \"rootfs\"), privileged (container \"agnhost-container\" must not set securityContext.privileged=true)"
  },
  {
    "namespace": "e2e-persistent-local-volumes-test-5428",
    "podSecurityReason": "would violate PodSecurity \"baseline:latest\": host namespaces (hostNetwork=true), hostPath volumes (volume \"rootfs\"), privileged (container \"agnhost-container\" must not set securityContext.privileged=true)"
  },
  {
    "namespace": "e2e-provisioning-2371",
    "podSecurityReason": "would violate PodSecurity \"baseline:latest\": host namespaces (hostNetwork=true), hostPath volumes (volume \"rootfs\"), privileged (container \"agnhost-container\" must not set securityContext.privileged=true)"
...

I am wrapping up the upstream e2e tests, once I have a 👍 from ligitt we can ping on merging the downstream backport and fix the downstream e2e tests.

s-urbaniak · 2022-03-10T17:22:47Z

/retest

s-urbaniak · 2022-03-11T06:28:22Z

/retest

s-urbaniak · 2022-03-11T06:29:28Z

retesting, as the first one most likely didn't have the necessary changes yet. I will:

Verify the most recent retest has UPSTREAM: 106454: SQUASH: test/e2e: let e2e tests specify pod security admiss… kubernetes#1128 included
Iterate over the failures and start fixing tests

s-urbaniak · 2022-03-15T09:22:22Z

/retest

stlaz · 2022-03-22T08:18:12Z

/retest

s-urbaniak · 2022-03-22T09:16:15Z

/retest

stlaz · 2022-03-22T13:26:35Z

/retest

s-urbaniak · 2022-03-22T15:03:37Z

/retest

s-urbaniak · 2022-03-23T11:36:55Z

/retest

s-urbaniak · 2022-03-23T14:49:55Z

/retest

s-urbaniak · 2022-03-24T07:32:08Z

/retest

s-urbaniak · 2022-03-24T14:50:59Z

/retest

stlaz · 2022-03-26T12:05:30Z

/retest
o/origin vendor bump with fixed tests merged, let's see what's gonna happen

s-urbaniak · 2022-03-28T05:02:40Z

/retest

stlaz · 2022-03-29T09:16:32Z

Moving the enforce to privileged so that we can reiterate on tests easier. We'll move to restricted soon(-ish maybe).

stlaz · 2022-03-29T11:52:35Z

/retest
weeee, infra quota issues again

s-urbaniak · 2022-03-30T12:30:03Z

/lgtm

stlaz · 2022-03-30T12:30:37Z

/retest-required

stlaz · 2022-04-13T16:48:10Z

/hold
another o/origin PR is brewing

s-urbaniak · 2022-04-13T17:49:36Z

/retest-required

stlaz · 2022-04-13T20:12:54Z

/hold cancel
/retest-required
let's pick up the latest o/origin changes

openshift-bot · 2022-04-13T20:54:52Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-04-13T22:30:52Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-04-13T23:18:53Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-04-14T00:42:52Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-04-14T00:54:52Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-04-14T02:06:52Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-04-14T05:07:51Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

stlaz · 2022-04-14T07:36:32Z

/hold
@s-urbaniak discovered some discrepancies in the upgrade test code

stlaz · 2022-04-14T19:46:20Z

/hold cancel
/retest-required

openshift-bot · 2022-04-14T20:22:53Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-ci · 2022-04-14T21:38:59Z

@stlaz: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-operator-disruptive-single-node	`77ec527`	link	false	`/test e2e-aws-operator-disruptive-single-node`
ci/prow/e2e-aws-single-node	`77ec527`	link	false	`/test e2e-aws-single-node`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-bot · 2022-04-14T21:58:52Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-04-14T22:23:24Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2022-04-14T23:00:27Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

Issue https://bugzilla.redhat.com/show_bug.cgi?id=2079292 Problem: In Kubernetes v1.23 the PodSecurity feature is enabled by default, OpenShift is making use of this feature and by default some expectations are already set [1], these expectations end up triggering a lot of alarms in the logs of CMO and workload running on the UWM namespace. [1] openshift/cluster-kube-apiserver-operator#1308 Solution: Add PodSecurity namespace labels to allow workloads running on UWM to have full privileges which will silence the alerts

Issue https://bugzilla.redhat.com/show_bug.cgi?id=2079292 Problem: In Kubernetes v1.23 the PodSecurity feature is enabled by default, OpenShift is making use of this feature and by default some expectations are already set [1], these expectations end up triggering a lot of alarms in the logs of CMO and workload running on the UWM namespace. [1] openshift/cluster-kube-apiserver-operator#1308 Solution: Annotate deployments that violate the PodSecurity mode restricted on the UWM namespace

Issue https://bugzilla.redhat.com/show_bug.cgi?id=2079292 Problem: In Kubernetes v1.23 the PodSecurity feature is enabled by default, OpenShift is making use of this feature and by default some expectations are already set [1], these expectations end up triggering a lot of alarms in the logs of CMO and workload running on the UWM namespace. [1] openshift/cluster-kube-apiserver-operator#1308 Solution: Annotate deployments that violate the PodSecurity mode restricted on the UWM namespace and update the SCC used by the SA of the prometheus instance in UWM

openshift-ci Bot requested a review from s-urbaniak February 8, 2022 15:07

s-urbaniak mentioned this pull request Mar 15, 2022

UPSTREAM: 106454: test/e2e: fix e2e tests for restricted policy openshift/kubernetes#1212

Merged

s-urbaniak mentioned this pull request Mar 25, 2022

bump: k8s.io/kubernetes openshift/origin#26935

Merged

reenable PodSecurity on privileged level

77ec527

stlaz force-pushed the psa_baseline branch from dda717f to 77ec527 Compare March 29, 2022 09:15

stlaz changed the title ~~AUTH-2: reenable PodSecurity on baseline level~~ AUTH-2: reenable PodSecurity on privileged level Mar 29, 2022

openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 13, 2022

openshift-ci Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 13, 2022

openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 14, 2022

openshift-ci Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 14, 2022

openshift-merge-robot merged commit 85a0325 into openshift:master Apr 15, 2022

JoaoBraveCoding mentioned this pull request May 5, 2022

Bug 2079292: Adds necessary SecurityContext settings to UWM deployments openshift/cluster-monitoring-operator#1660

Merged

2 tasks

Conversation

stlaz commented Feb 8, 2022

Uh oh!

s-urbaniak commented Feb 9, 2022

Uh oh!

s-urbaniak commented Feb 9, 2022

Uh oh!

s-urbaniak commented Feb 9, 2022

Uh oh!

stlaz commented Feb 9, 2022

Uh oh!

stlaz commented Feb 9, 2022

Uh oh!

s-urbaniak commented Feb 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

s-urbaniak commented Mar 10, 2022

Uh oh!

s-urbaniak commented Mar 11, 2022

Uh oh!

s-urbaniak commented Mar 11, 2022

Uh oh!

s-urbaniak commented Mar 15, 2022

Uh oh!

stlaz commented Mar 22, 2022

Uh oh!

s-urbaniak commented Mar 22, 2022

Uh oh!

stlaz commented Mar 22, 2022

Uh oh!

s-urbaniak commented Mar 22, 2022

Uh oh!

s-urbaniak commented Mar 23, 2022

Uh oh!

s-urbaniak commented Mar 23, 2022

Uh oh!

s-urbaniak commented Mar 24, 2022

Uh oh!

s-urbaniak commented Mar 24, 2022

Uh oh!

stlaz commented Mar 26, 2022

Uh oh!

s-urbaniak commented Mar 28, 2022

Uh oh!

stlaz commented Mar 29, 2022

Uh oh!

stlaz commented Mar 29, 2022

Uh oh!

s-urbaniak commented Mar 30, 2022

Uh oh!

stlaz commented Mar 30, 2022

Uh oh!

stlaz commented Apr 13, 2022

Uh oh!

s-urbaniak commented Apr 13, 2022

Uh oh!

stlaz commented Apr 13, 2022

Uh oh!

openshift-bot commented Apr 13, 2022

Uh oh!

openshift-bot commented Apr 13, 2022

Uh oh!

openshift-bot commented Apr 13, 2022

Uh oh!

openshift-bot commented Apr 14, 2022

Uh oh!

openshift-bot commented Apr 14, 2022

Uh oh!

openshift-bot commented Apr 14, 2022

Uh oh!

openshift-bot commented Apr 14, 2022

Uh oh!

stlaz commented Apr 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stlaz commented Apr 14, 2022

Uh oh!

openshift-bot commented Apr 14, 2022

Uh oh!

openshift-ci Bot commented Apr 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

s-urbaniak commented Feb 10, 2022 •

edited

Loading

stlaz commented Apr 14, 2022 •

edited

Loading

openshift-ci Bot commented Apr 14, 2022 •

edited

Loading