Revert "add extra role to be able to create gcp vms"

[ameukam]

Revert of #2124 which
introduces an exposure to privilege escalation.

Signed-off-by: Arnaud Meukam ameukam@gmail.com

[spiffxp]

I would replace with ensure_removed_project_role_binding instead of deleting the line

And maybe try adding

ensure_serviceaccount_role_binding "${serviceaccount}" "serviceAccount:${serviceaccount}" "roles/iam.serviceAccountUser"

to see if my wild guess in #2124 (comment) is correct?

[ameukam]

I would replace with ensure_removed_project_role_binding instead of deleting the line

And maybe try adding

ensure_serviceaccount_role_binding "${serviceaccount}" "serviceAccount:${serviceaccount}" "roles/iam.serviceAccountUser"

to see if my wild guess in #2124 (comment) is correct?

Running ensure-staging-storage.sh with this change.

[spiffxp]

/approve
/lgtm
/hold
To remove when ready to deploy

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ameukam, spiffxp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [ameukam,spiffxp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[spiffxp]

/hold cancel

[cpanato]

@spiffxp @ameukam after the changes applied the job start failing again but looks like it is another reason.
Looks like the instance is not coming up, and maybe it is not being created, maybe lacking permissions

I can edit the script to add the debug flag, so we can see why is failing

https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/periodic-image-builder-gcp-all-nightly/1403250364064468992

[cpanato]

@spiffxp @ameukam gently ping here

[spiffxp]

Pulling out of slack:

• https://console.cloud.google.com/home/activity?project=k8s-staging-cluster-api-gcp - look at the VM failures, figure out when they started so we can compare known-good to known-bad
• https://console.cloud.google.com/logs/query;query=logName%3D%2528%22projects%2Fk8s-staging-cluster-api-gcp%2Flogs%2Fcloudaudit.googleapis.com%252Factivity%22%20OR%20%22projects%2Fk8s-staging-cluster-api-gcp%2Flogs%2Fcloudaudit.googleapis.com%252Fsystem_event%22%20OR%20%22projects%2Fk8s-staging-cluster-api-gcp%2Flogs%2Fcompute.googleapis.com%252Fshielded_vm_integrity%22%20OR%20%22projects%2Fk8s-staging-cluster-api-gcp%2Flogs%2Fcloudbuild%22%2529%0AprotoPayload.methodName%3D%22v1.compute.instances.insert%22;timeRange=2021-06-08T12:00:00.000Z%2F2021-06-10T12:26:00.000Z;cursorTimestamp=2021-06-10T11:20:54.864780Z?project=k8s-staging-cluster-api-gcp - google cloud logging with a window around known good to known bad
• https://console.cloud.google.com/logs/query;cursorTimestamp=2021-06-10T05:34:12.293953Z;query=logName%3D%2528%22projects%2Fk8s-staging-cluster-api-gcp%2Flogs%2Fcloudaudit.googleapis.com%252Factivity%22%20OR%20%22projects%2Fk8s-staging-cluster-api-gcp%2Flogs%2Fcloudaudit.googleapis.com%252Fsystem_event%22%20OR%20%22projects%2Fk8s-staging-cluster-api-gcp%2Flogs%2Fcompute.googleapis.com%252Fshielded_vm_integrity%22%20OR%20%22projects%2Fk8s-staging-cluster-api-gcp%2Flogs%2Fcloudbuild%22%2529%0AprotoPayload.methodName%3D%22v1.compute.instances.insert%22%0Atimestamp%3D%222021-06-10T05:34:12.293953Z%22%0AinsertId%3D%22qxdcsqe4127s%22;timeRange=2021-06-08T12:00:00.000Z%2F2021-06-10T12:26:00.000Z?project=k8s-staging-cluster-api-gcp - turns out there is no real difference between the create requests between known good / known bad, this is an example
• The VM failures are coming back with the amazingly helpful error message Invalid argument (HTTP 400): INVALID_PARAMETER. Since we're dealing with service account usage, I'm going to guess maybe it's this:

"serviceAccounts": [
    {
        "email": "default",
        "scopes": [
            "https://www.googleapis.com/auth/userinfo.email",
            "https://www.googleapis.com/auth/compute",
            "https://www.googleapis.com/auth/devstorage.full_control"
        ]
    }
]

• https://github.com/hashicorp/packer-plugin-googlecompute/blob/b3b083f8d71b73dbb89cdfb97ff76a0de2e7a553/builder/googlecompute/driver_gce.go#L450-L462 is the relevant code that is setting "default" because packer hasn't been configured to do otherwise
• I don't know if "default" has a special meaning to GCE or not, but I think our intent here is to have packer run the VM it's using to build the image as the gcb-builder-cluster-api-gcp@k8s-staging-cluster-api-gcp.iam.gserviceaccount.com service account we've been giving permissions to
• Reading https://www.packer.io/docs/builders/googlecompute#optional-1 it seems like you want to set:
  • disable_default_service_account to true
  • service_account_email to gcb-builder-cluster-api-gcp@k8s-staging-cluster-api-gcp.iam.gserviceaccount.com

[spiffxp]

To help with troubleshooting I'm going to temporarily enable all Audit Logs (Admin Activity and Data Access) for IAM and Compute, via the console at https://console.cloud.google.com/iam-admin/audit?folder=&organizationId=&project=k8s-staging-cluster-api-gcp

[cpanato]

looking the audit logs, on 6/10/21

• June 10, 2021 at 8:15:43 AM GMT+2 removed the Removed iam.serviceAccountUser, and after that job starts failing again

[cpanato]

If set both we got

Error: Failed to prepare build: "ubuntu-1804"

1 error(s) occurred:

* you may not specify a 'service_account_email' when
'disable_default_service_account' is true

I will set up only the service_account_email

PR in image-builder: kubernetes-sigs/image-builder#641