prow-build-trusted: add kubernetes-external-secrets by spiffxp · Pull Request #2148 · kubernetes/k8s.io

spiffxp · 2021-06-07T21:18:13Z

Followup to #2078

In support of kubernetes/test-infra#22298 and kubernetes/test-infra#22293

k8s-ci-robot · 2021-06-07T21:18:23Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: spiffxp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~infra/gcp/clusters/projects/k8s-infra-prow-build-trusted/OWNERS~~ [spiffxp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

spiffxp · 2021-06-07T21:18:35Z

/cc @ameukam @dims

...s/k8s-infra-prow-build-trusted/prow-build-trusted/resources/kubernetes-external-secrets.yaml

Specifically: - add manifests - copy from /kubernetes-external-secrets - move SA from rbac.yaml to .yaml - add label "app: kubernetes-external-secrets" to some resources that didn't have it in rbac - rename the crd to kubernetes-external-secrets_crd.yaml - remove TODO on LOG_LEVEL being debug - add terraform - add kubernetes-external-secrets serviceaccount with correct role and setup for workload identity - add regional ip address for metrics scraping The "create a service account with X role for use by this cluster via WI" boilerplate is getting pretty unwieldy, but I didn't want to refactor (or look for) a module until I got this deployed

spiffxp · 2021-06-07T22:23:42Z

I had to run terraform apply to get an external IP created, but the resources in here haven't yet been deployed. Your call whether I do this pre-merge or post-merge, but I'll wait for approval before doing so.

ameukam · 2021-06-07T22:51:14Z

Let's do this post-merge.

Also pull-k8sio-terraform-prow-build-trusted was supposed to run but nothing happened. It's not blocking for this PR.

/lgtm
/hold
Remove hold when ready to deploy.

spiffxp · 2021-06-08T01:08:16Z

The run_if_changed regex has a typo:prow-build-trusted/*.tf$ would match prow-build-trusted/.tf or prow-build-trusted////.tf but none of the files this PR changes. If I need to make any changes in the jobs due to kubernetes/test-infra#22463 I'll include as a fix

spiffxp · 2021-06-08T01:08:24Z

/hold cancel

spiffxp · 2021-06-08T01:09:02Z

/hold
actually I want to wait for the postsubmits to change...

spiffxp · 2021-06-08T01:33:39Z

/hold cancel
ready to watch this deploy

spiffxp · 2021-06-08T02:07:07Z

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/post-k8sio-deploy-prow-build-trusted-resources/1402077432638345216
First run failed with:

customresourcedefinition.apiextensions.k8s.io/externalsecrets.kubernetes-client.io created
namespace/kubernetes-external-secrets created
namespace/staging-test-pods unchanged
namespace/test-pods unchanged
Error from server (NotFound): error when creating "prow-build-trusted/resources/kubernetes-external-secrets.yaml": namespaces "kubernetes-external-secrets" not found
Error from server (NotFound): error when creating "prow-build-trusted/resources/kubernetes-external-secrets.yaml": namespaces "kubernetes-external-secrets" not found

Re-ran job, namespaces were found (need to redo job to create cluster-scoped resources before creating namespace-scoped resources)

But it also failed, with

Error from server (Forbidden): error when creating "prow-build-trusted/resources/kubernetes-external-secrets_rbac.yaml": clusterroles.rbac.authorization.k8s.io is forbidden: User "prow-deployer@k8s-infra-prow-build-trusted.iam.gserviceaccount.com" cannot create resource "clusterroles" in API group "rbac.authorization.k8s.io" at the cluster scope: requires one of ["container.clusterRoles.create"] permission(s).
Error from server (Forbidden): error when creating "prow-build-trusted/resources/kubernetes-external-secrets_rbac.yaml": clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "prow-deployer@k8s-infra-prow-build-trusted.iam.gserviceaccount.com" cannot create resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope: requires one of ["container.clusterRoleBindings.create"] permission(s).
Error from server (Forbidden): error when creating "prow-build-trusted/resources/kubernetes-external-secrets_rbac.yaml": clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "prow-deployer@k8s-infra-prow-build-trusted.iam.gserviceaccount.com" cannot create resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope: requires one of ["container.clusterRoleBindings.create"] permission(s).

spiffxp · 2021-06-08T02:15:11Z

It looks like there's no (non-service-agent) pre-defined role that contains container.clusterRoles.create except for roles/container.admin, so my options are to either:

manually deploy myself as a cluster admin
give prow-deployer container.admin
create a custom role with container.clusterRoles.* and container.clusterRoleBindings.* and give that to prow-deployer

I'm going to do 2 to confirm the job can deploy, then work on 3

spiffxp · 2021-06-08T02:17:04Z

gcloud projects add-iam-policy-binding \
  k8s-infra-prow-build-trusted \
  --member=serviceAccount:prow-deployer@k8s-infra-prow-build-trusted.iam.gserviceaccount.com \
  --role=roles/container.admin

spiffxp · 2021-06-08T03:05:58Z

Re-run succeeded: https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/post-k8sio-deploy-prow-build-trusted-resources/1402087372769726464

Although, of note:

Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition

spiffxp · 2021-06-08T04:18:20Z

Opened #2156 for option 3

ameukam · 2021-06-08T07:15:59Z

Re-run succeeded: prow.k8s.io/view/gs/kubernetes-jenkins/logs/post-k8sio-deploy-prow-build-trusted-resources/1402087372769726464

Although, of note:
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition

Fixed in the next major release : https://github.com/external-secrets/kubernetes-external-secrets/releases/tag/8.0.0.

If you're not faster than me, I'll plan do the upgrade in the next weeks.

spiffxp · 2021-06-10T21:10:53Z

Now that #2156 has merged, I'm dropping the roles/container.admin binding

gcloud projects remove-iam-policy-binding \
  k8s-infra-prow-build-trusted \
  --member="serviceAccount:prow-deployer@k8s-infra-prow-build-trusted.iam.gserviceaccount.com" \
  --role="roles/container.admin"

k8s-ci-robot requested review from chases2 and jeremyrickard June 7, 2021 21:18

k8s-ci-robot added the wg/k8s-infra label Jun 7, 2021

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 7, 2021

k8s-ci-robot requested review from ameukam and dims June 7, 2021 21:18

ameukam reviewed Jun 7, 2021

View reviewed changes

...s/k8s-infra-prow-build-trusted/prow-build-trusted/resources/kubernetes-external-secrets.yaml Show resolved Hide resolved

ameukam reviewed Jun 7, 2021

View reviewed changes

...s/k8s-infra-prow-build-trusted/prow-build-trusted/resources/kubernetes-external-secrets.yaml Outdated Show resolved Hide resolved

spiffxp added 2 commits June 7, 2021 17:59

prow-build-trusted: merge namespace resources

69864a2

spiffxp force-pushed the k8s-infra-prow-build-trusted-external-secrets branch from 29244c3 to 9df04e0 Compare June 7, 2021 22:16

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 7, 2021

k8s-ci-robot assigned ameukam Jun 7, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 7, 2021

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 8, 2021

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 8, 2021

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 8, 2021

k8s-ci-robot merged commit 5dbcd87 into kubernetes:main Jun 8, 2021

k8s-ci-robot added this to the v1.22 milestone Jun 8, 2021

spiffxp deleted the k8s-infra-prow-build-trusted-external-secrets branch June 8, 2021 02:04

spiffxp mentioned this pull request Jun 8, 2021

roles: add custom org role container.deployer #2156

Merged

spiffxp mentioned this pull request Jun 8, 2021

audit: update as of 2021-06-08 #2157

Merged

This was referenced Jun 10, 2021

prow: use container.deployer role for deployer SA #2190

Merged

prow: update kubernetes-external-secrets to v8.1.2 #2194

Merged

audit: update as of 2021-06-11 #2193

Merged

spiffxp mentioned this pull request Jul 13, 2021

Tool to declaratively, securely manage secrets kubernetes/test-infra#14049

Closed

Conversation

spiffxp commented Jun 7, 2021

Uh oh!

k8s-ci-robot commented Jun 7, 2021

Uh oh!

spiffxp commented Jun 7, 2021

Uh oh!

Uh oh!

Uh oh!

spiffxp commented Jun 7, 2021

Uh oh!

ameukam commented Jun 7, 2021

Uh oh!

spiffxp commented Jun 8, 2021

Uh oh!

spiffxp commented Jun 8, 2021

Uh oh!

spiffxp commented Jun 8, 2021

Uh oh!

spiffxp commented Jun 8, 2021

Uh oh!

spiffxp commented Jun 8, 2021

Uh oh!

spiffxp commented Jun 8, 2021

Uh oh!

spiffxp commented Jun 8, 2021

Uh oh!

spiffxp commented Jun 8, 2021

Uh oh!

spiffxp commented Jun 8, 2021

Uh oh!

ameukam commented Jun 8, 2021

Uh oh!

spiffxp commented Jun 10, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants