Support service account impersonation for wait-for-cluster script

[Dev25]

This is a rebase of #463 so original credit goes to @ericyz

I ran into a issue with upgrading to the v12 release due to lack of impersonation support so the script stalls due to auth errors, noticed there was a old PR for this so rebased that and have tested it in my environment.

module.blue_gke.module.gke.module.gcloud_wait_for_cluster.null_resource.run_command[0]: Provisioning with 'local-exec'...
module.blue_gke.module.gke.module.gcloud_wait_for_cluster.null_resource.run_command[0] (local-exec): Executing: ["/bin/sh" "-c" "PATH=/google-cloud-sdk/bin:$PATH\n.terraform/modules/blue_gke.gke/modules/beta-private-cluster/scripts/wait-for-cluster.sh <redacted> dev-cluster-blue <redacted>iam.gserviceaccount.com\n"]
module.blue_gke.module.gke.module.gcloud_wait_for_cluster.null_resource.run_command[0] (local-exec): Waiting for cluster dev-cluster-blue in project <redacted> to reconcile...
module.blue_gke.module.gke.module.gcloud_wait_for_cluster.null_resource.run_command[0] (local-exec): WARNING: This command is using service account impersonation. All API calls will be executed as [<redacted>.iam.gserviceaccount.com].
module.blue_gke.module.gke.module.gcloud_wait_for_cluster.null_resource.run_command[0] (local-exec): Cluster is ready!
module.blue_gke.module.gke.module.gcloud_wait_for_cluster.null_resource.run_command[0]: Creation complete after 3s [id=6115564123617205323]

[comment-bot-dev]

Thanks for the PR! 🚀
✅ Lint checks have passed.

[bharathkkb]

Thanks @Dev25! LGTM

[reedox]

FYI this breaks Cloudbuild + Terraform with the recommended Google Foundations setup:
https://github.com/terraform-google-modules/terraform-example-foundation/tree/master/0-bootstrap

Yesterday I started getting this error with a previously working deployment (all terraform operations run fine, just not the script) so everything started failing. I spent more time debugging this than I would care to admit so I hope this helps someone out! Pin the version to v12.0.0.

Step #4 - "tf apply": google_service_networking_connection.redis: Modifications complete after 33s [id=https%3A%2F%2Fwww.googleapis.com%2Fcompute%2Fv1%2xxxxxx:servicenetworking.googleapis.com]
Step #4 - "tf apply": 
Step #4 - "tf apply": Error: Error running command 'PATH=/google-cloud-sdk/bin:$PATH
Step #4 - "tf apply": .terraform/modules/gke/modules/beta-private-cluster/scripts/wait-for-cluster.sh prj-rx1-d-xxxx gke-d-xxxx us-east4 
Step #4 - "tf apply": ': exit status 1. Output: Waiting for cluster prj-rx1-d-xxxx/us-east4/gke-d-xxxx to reconcile...
Step #4 - "tf apply": ERROR: (gcloud.container.clusters.list) ResponseError: code=403, message=Required "container.clusters.list" permission(s) for "projects/xxxx".

[bharathkkb]

Hi @reedox
Thanks for the report. Within example foundations, by default the cloudbuild SA does not have permissions to list clusters. So you should be able to pass in terraform_sa_email to var.impersonate_service_account and this feature should make it work. We also recommend creating a separate SA & cloudbuild pipeline to manage other resources for separation of concerns.

The new impersonation feature is off by default so I am wondering how it previously worked as the permissions required still the same. If you could provide previous apply logs I would be happy to dig further.