add script to create conformance buckets, add bucket for capo

[sbueringer]

For context see: kubernetes/test-infra#15081

[dims]

/assign @listx @thockin

cc @BenTheElder

[thockin]

I think this is a bit under-specified - it seems to me that conformance is closer to "prod" than staging in terms of retention/deletion policy?

[BenTheElder]

thanks for the PR!!

[thockin]

sorry, I hit the wrong button - I didn't mean to send, was a note for myself :)

…

On Mon, Nov 11, 2019 at 10:15 AM Stefan Büringer ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In infra/gcp/ensure-conformance-storage.sh <#459 (comment)>: > +fi + +for REPO; do + color 3 "Configuring staging: ${REPO}" + + # The GCP project name. + PROJECT="k8s-conformance-${REPO}" + + # The group that can write to this staging repo. + ***@***.***" + + # The names of the buckets + STAGING_BUCKET="gs://${PROJECT}" # used by humans + ALL_BUCKETS=("${STAGING_BUCKET}") + + # A short expiration - it can always be raised, but it is hard to lower Yup but know. Should start sending comments after I pushed :) — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#459?email_source=notifications&email_token=ABKWAVBYJRYH7X4GRU4PFSLQTGOLNA5CNFSM4JLLWFQKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCLDZILY#discussion_r344842104>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVGNKZGW7MYMR2CL22DQTGOLNANCNFSM4JLLWFQA> .

[thockin]

Question for the group:

Staging requires a GCP project per staging, because it uses GCR and that's implictly 1:1 with project.

This does not use GCR - it coud technically be all in one project, different buckets.

Other than a smaller list of projects, it isn't significantly better or worse. My inclination is to leave this as-is, but I thought I would bring it up so we don't second-guess later.

[thockin]

I'm approving, but @BenTheElder gets final LGTM

/approve

[justaugustus]

@thockin --

This does not use GCR - it coud technically be all in one project, different buckets.

I was coming here to say this!
Seems that we're sprawling a bit here, and I think we'd be just as easily served with a single conformance GCP project with separate buckets or conformance buckets within existing projects e.g., for CAPI, why not store conformance info within their primary GCP project or the respective provider projects?

[sbueringer]

@justaugustus

At least in case of ClusterAPI OpenStack the only GCP project which already exists (afaik) is named k8s-staging-capi-openstack which would probably be not the correct on the add a bucket for conformance test results.

Apart from that I don't really know how Kubernetes GCP projects/buckets are usually organized. So I let you discuss and adjust the script accordingly :)

[BenTheElder]

Currently I maintain a single GCP project with multiple buckets [for conformance test results] and service accounts 1:1, with specific IAM permissions to the buckets matched to service accounts.

I'm not sure which pattern we want going forward.

[thockin]

I'd be OK with a single k8s-conformance or k8s-conformance-results project with N buckets and a mailing list IAM to write to each bucket

[sbueringer]

I'd be OK with a single k8s-conformance or k8s-conformance-results project with N buckets and a mailing list IAM to write to each bucket

Okay so one project with multi buckets. Just that I get it correctly. Only one google group for the project or one per bucket?

[BenTheElder]

let's do one project with a group per bucket, someone will need to own the project as well though.

[sbueringer]

@BenTheElder Is there's already an existing group I can reuse or should I create a new one?

[BenTheElder]

not sure, @dims wdyt?

…

On Fri, Nov 15, 2019 at 8:29 AM Stefan Büringer ***@***.***> wrote: @BenTheElder <https://github.com/BenTheElder> Is there's already an existing group I should reuse or should I create a new one? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#459?email_source=notifications&email_token=AAHADK7IK74QJXCDZEOCR3LQT3E5NA5CNFSM4JLLWFQKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEF7BDA#issuecomment-554430604>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHADK7PKVQG33MEPHYGMPDQT3E5NANCNFSM4JLLWFQA> .

[sbueringer]

@thockin @BenTheElder @dims

I updated the script to use a single project for all buckets. For now I used the "k8s-infra-conform@kubernetes.io" group for the project. If we want to use this group, instead of reusing an existing one I also have to define this group in the groups.yaml (which I haven't done yet).

I'm not sure which rights are necessary on the project. I used empower_group_as_viewer again like in the previous version of the script.

[sbueringer]

@thockin @BenTheElder @dims

I updated the script to use a single project for all buckets. For now I used the "k8s-infra-conform@kubernetes.io" group for the project. If we want to use this group, instead of reusing an existing one I also have to define this group in the groups.yaml (which I haven't done yet).

I'm not sure which rights are necessary on the project. I used empower_group_as_viewer again like in the previous version of the script.

Just wanted to let you know that I'm on vacation until end of Dezember. I would continue this PR then. If anyone wants to takeover in the meantime, be my guest :)

[dims]

/assign @dims

[dims]

@BenTheElder @sbueringer let's use a new one

[BenTheElder]

SGTM

[dims]

/hold

let's wait for @sbueringer to get back :)

[thockin]

Otherwise this looks ok to me

[thockin]

I don't think we need this, do we? What is it for?

[sbueringer]

Seems to be a leftover from copying this script. I removed PROJECT_VIEWER

[sbueringer]

@thockin please take another look :)

[sbueringer]

@thockin please take another look :)

push :)
@dims @BenTheElder

[thockin]

Thanks!

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by: thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[thockin]

I am actuating this. One function got renamed, fix coming.