Create a service a service account in k8s-infra-prow-build cluster.

[dargudear-google]

I am working on-recreating a prow job and these prow jobs deleted during migration to community infra. Discussion ref.

I started re-creation of the Job and submitted https://github.com/kubernetes/test-infra/pull/33340/files
But when Job was triggered it could not find the serviceaccount secrets-store-csi-driver-gcp
job config: https://prow.k8s.io/prowjob?prowjob=3651f2a3-a736-453e-b349-9f29af4a17ce
build_serviceaccounts.yaml has the config for serviceaccount secrets-store-csi-driver-gcp`

Can we create a similar account as of old account to re-create the tests?

[BenTheElder]

We usually use workload identity.

The interesting question isn't the service account, it is what resources the service account enables access to.

We need to know what resources are required so we can figure out how to manage them in the community accounts.

We are NOT permitting dependency to external resources not managed by the project within the infra/CI we operate, to prevent future headaches.

[BenTheElder]

@kubernetes/sig-k8s-infra-leads [to track this discussion about providing resources for secre-store-csi-driver testing, I suspect we will need something similar to https://github.com//pull/6924 + make sure boskos handles it]

[dargudear-google]

Service account needs to access the secrets from a project owned by google internally.

This prow job creates a kind cluster, inside kind cluster, secret driver and provider gets installed. This provider needs to access secrets. The baseline requirement is the workload identity that we usually use should be able to act as [email protected] like it was earlier https://github.com/kubernetes/test-infra/blob/master/config/prow/cluster/build/build_serviceaccounts.yaml#L59-L66

[BenTheElder]

Service account needs to access the secrets from a project owned by google internally.

This is not supported. We do not permit taking dependencies on third party accounts. We have just spent years fixing this.

As previously mentioned and outlined, but again https://groups.google.com/a/kubernetes.io/g/dev/c/p6PAML90ZOU/m/11sDguoxAQAJ / https://groups.google.com/a/kubernetes.io/g/dev/c/qzNYpcN5la4

This prow job creates a kind cluster, inside kind cluster, secret driver and provider gets installed. This provider needs to access secrets.

Surely we can identify what a GCP project would need to have in order to do this with a kubernetes.io GCP project?

[dargudear-google]

What If we configure a job like this which used boskos. In the test, a new GKE cluster will be created (using gcloud) along with secret manager secret. We will test the functionality in the cluster. Since we will have our project, there won't be any permission issues.

[BenTheElder]

What If we configure a job like this which used boskos.

Sure.

In the test, a new GKE cluster will be created (using gcloud) along with secret manager secret.

We don't generally test OSS projects with GKE versus one of the open source tools (like kops) but ....

We will test the functionality in the cluster. Since we will have our project, there won't be any permission issues.

To be clear: You mean a project rented from boskos? Which is one of the shared projects.

If this job creates an additional resources, the project cleanup script needs to be made aware of them (there's no generic way to get all resources AFAIK, and even if there was, there can be ordering issues) https://github.com/kubernetes/test-infra/blob/master/boskos/cmd/janitor/gcp_janitor.py

[dargudear-google]

If this job creates an additional resources, the project cleanup script needs to be made aware of them (there's no generic way to get all resources AFAIK, and even if there was, there can be ordering issues) https://github.com/kubernetes/test-infra/blob/master/boskos/cmd/janitor/gcp_janitor.py

Like this kubernetes/test-infra#33669 ?

[dargudear-google]

@kubernetes/sig-k8s-infra-leads [to track this discussion about providing resources for secre-store-csi-driver testing, I suspect we will need something similar to https://github.com//pull/6924 + make sure boskos handles it]

See if we can have #7416 ?

Also I am planning to test the prow job after above PRs.

[BenTheElder]

Like this kubernetes/test-infra#33669 ?

I don't think we use that copy anymore (need to check with @dims @upodroid), but that looks about right 👍

See if we can have #7416 ?

We should check @upodroid @ameukam, but I think that's fine.

[ameukam]

Like this kubernetes/test-infra#33669 ?

I don't think we use that copy anymore (need to check with @dims @upodroid), but that looks about right 👍

I remember an issue about secrets cleanup: https://github.com/kubernetes-sigs/boskos/pull/204/files.

[dargudear-google]

Like this kubernetes/test-infra#33669 ?

I don't think we use that copy anymore (need to check with @dims @upodroid), but that looks about right 👍

I remember an issue about secrets cleanup: https://github.com/kubernetes-sigs/boskos/pull/204/files.

I have tested my changes on my local setup and this is working.

I can see the issue in the resource there as in gcloud we have gcloud secrets not secretmanager