Skip to content

prow: use container.deployer role for deployer SA#2190

Merged
k8s-ci-robot merged 1 commit intokubernetes:mainfrom
spiffxp:use-container-deployer-role
Jun 10, 2021
Merged

prow: use container.deployer role for deployer SA#2190
k8s-ci-robot merged 1 commit intokubernetes:mainfrom
spiffxp:use-container-deployer-role

Conversation

@spiffxp
Copy link
Copy Markdown
Contributor

@spiffxp spiffxp commented Jun 10, 2021

prow-deployer was using roles/container.developer which is insufficient to automatically deploy resources related to RBAC and webhooks, so we've created a custom role to allow full control of all in-cluster resources without granting access to control the cluster itself

This is followup to #2156 which created the role

And #2148 (comment) which is where we discovered the insufficiency of roles/container.developer for auto-deploying kubernetes-external-secrets

prow-deployer was using roles/container.developer which is insufficient
to automatically deploy resources related to RBAC and webhooks, so we've
created a custom role to allow full control of all in-cluster resources
without granting access to control the cluster itself
@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 10, 2021
@k8s-ci-robot k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. area/terraform Terraform modules, testing them, writing more of them, code in infra/gcp/clusters/ area/prow Setting up or working with prow in general, prow.k8s.io, prow build clusters sig/testing Categorizes an issue or PR as relevant to SIG Testing. wg/k8s-infra size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jun 10, 2021
@spiffxp
Copy link
Copy Markdown
Contributor Author

spiffxp commented Jun 10, 2021

/cc @BenTheElder @ameukam
I dropped the manual binding

gcloud projects remove-iam-policy-binding \
  k8s-infra-prow-build-trusted \
  --member="serviceAccount:prow-deployer@k8s-infra-prow-build-trusted.iam.gserviceaccount.com" \
  --role="roles/container.admin"

And have already run terraform apply for these changes

Copy link
Copy Markdown
Member

@BenTheElder BenTheElder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 10, 2021
@k8s-ci-robot
Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: BenTheElder, spiffxp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 525b358 into kubernetes:main Jun 10, 2021
@k8s-ci-robot k8s-ci-robot added this to the v1.22 milestone Jun 10, 2021
@ameukam
Copy link
Copy Markdown
Member

ameukam commented Jun 10, 2021

/lgtm

@spiffxp spiffxp deleted the use-container-deployer-role branch June 10, 2021 22:07
@spiffxp
Copy link
Copy Markdown
Contributor Author

spiffxp commented Jun 11, 2021

Failed to auto-deploy kubernetes-external-secrets again: #2194 (comment)

Specific failure:

clusterroles.rbac.authorization.k8s.io "kubernetes-external-secrets" is forbidden: user "prow-deployer@k8s-infra-prow-build-trusted.iam.gserviceaccount.com" (groups=["system:authenticated"]) is attempting to grant RBAC permissions not currently held:
{APIGroups:[""], Resources:["namespaces"], Verbs:["get" "watch" "list"]}
{APIGroups:[""], Resources:["secrets"], Verbs:["create" "update"]}
{APIGroups:["apiextensions.k8s.io"], Resources:["customresourcedefinitions"], ResourceNames:["externalsecrets.kubernetes-client.io"], Verbs:["get" "update"]}
{APIGroups:["kubernetes-client.io"], Resources:["externalsecrets"], Verbs:["get" "watch" "list"]}
{APIGroups:["kubernetes-client.io"], Resources:["externalsecrets/status"], Verbs:["get" "update"]}

I'm guessing the reason this worked when prow-deployer had roles/container.admin is that something implicitly grants that GCP role a K8S role permitting update of K8S resources. I'm going to take a closer look at how prow is configured to talk to its build clusters.

@spiffxp
Copy link
Copy Markdown
Contributor Author

spiffxp commented Jun 30, 2021

tracking issue ref: #2218

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/prow Setting up or working with prow in general, prow.k8s.io, prow build clusters area/terraform Terraform modules, testing them, writing more of them, code in infra/gcp/clusters/ cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants