add job to launch daily shared cluster

[sallyom]

It will be convenient to have a shared cluster (via a bot in channel that will pin a kubeconfig/kubeadmin-pw to that channel?)

For now, I've been manually creating/pinning a kubeconfig for groub-b to share, and I know console team does the same.

/cc @smarterclayton

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sallyom
To complete the pull request process, please assign stevekuznetsov
You can assign the PR to them by writing /assign @stevekuznetsov in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• ci-operator/jobs/openshift/release/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[smarterclayton]

@bparees owns all the long running clusters, you'll need to work with him on this.

[smarterclayton]

This won't work as is because there is a fixed time limit for prow jobs, and if the job is evicted the cluster is leaked. It also absorbs quota.

If we want to have a single cluster up for a day that everyone shares, that was supposed to be the long running clusters.

[bparees]

@sallyom is this tied to https://jira.coreos.com/browse/DPP-1338 ?

[bparees]

@sallyom see also #3887

[sallyom]

@bparees thanks, I'll follow up w/ you on this! I'll close this for now.

[soltysh]

@bparees it's a combination of both, I guess. We have a cluster created by Sally every other day that is shared between all group b members. The idea is to limit the time needed to wait for cluster where you need to check small things, like configs, running a simple app, verify oc, etc.

[bparees]

@soltysh i know @chancez has a cluster like that which he maintains.. you might see if he's willing to let you use it also if you're doing non-destructive things. Alternatively you can get your own set of openshift-llc creds and run your own long-lived cluster like he's doing.

[soltysh]

@bparees truth to be told, I don't mind the fact that this cluster has to be renewed every 2 days, actually that's pretty good b/c we have reasonably fresh cluster, so I don't think we care about LLC here. Although I'd like to have automation around it so that @sallyom (who's graciously doing this for group-b) can be let go of remembering about it.

[sallyom]

Yes @bparees we would like a slack bot to create a new cluster every day or 2, and pin the kubeconfig to our channel/remove the old pin. right now that bot is me. This PR introduces a new job similar to cluster-bot but instead of a few hrs, the cluster remains for a day. My idea was to add this job, then configure a new slack bot to launch it, 1 per channel or something.

[bparees]

maybe @smarterclayton would be willing to let cluster-bot launch long-lived clusters and you could just use a slack reminder to trigger the bot?

[smarterclayton]

The slack bot won't do that, it's too complicated today. If we have a 2 day cluster it's going to be shared by more than a single team, and it's going to end up being continuous. We could do an 8 hour one that just starts during the workday and anyone can share that.

…

On Thu, Oct 31, 2019 at 3:29 PM Ben Parees ***@***.***> wrote: maybe @smarterclayton <https://github.com/smarterclayton> would be willing to let cluster-bot launch long-lived clusters and you could just use a slack reminder to trigger the bot? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5607?email_source=notifications&email_token=AAI37J2DJZER2PJ7QWJCAATQRMWYZA5CNFSM4JFFHO5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECY7HLQ#issuecomment-548533166>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAI37J3QZYMYTQ27RLFUZR3QRMWYZANCNFSM4JFFHO5A> .

[chancez]

What was the reason we didn't want to stand up a long lived cluster for this again? Is it ownership or the need for it to be the latest release? If people were okay with the latest candidate release of OCP then I would be willing to set up a cluster for everyone, and manage it. It would use GH auth and cluster autoscaler to ensure it's mostly hands-off. Everyone would be cluster-admin and we would just have to see if it works out. Longer term we could find a home for it under DPTP or something like that. On Thu, Oct 31, 2019 at 12:33 PM Clayton Coleman <notifications@github.com> wrote:

…

The slack bot won't do that, it's too complicated today. If we have a 2 day cluster it's going to be shared by more than a single team, and it's going to end up being continuous. We could do an 8 hour one that just starts during the workday and anyone can share that. On Thu, Oct 31, 2019 at 3:29 PM Ben Parees ***@***.***> wrote: > maybe @smarterclayton <https://github.com/smarterclayton> would be > willing to let cluster-bot launch long-lived clusters and you could just > use a slack reminder to trigger the bot? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > < #5607?email_source=notifications&email_token=AAI37J2DJZER2PJ7QWJCAATQRMWYZA5CNFSM4JFFHO5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECY7HLQ#issuecomment-548533166 >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AAI37J3QZYMYTQ27RLFUZR3QRMWYZANCNFSM4JFFHO5A > > . > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5607?email_source=notifications&email_token=AAKV4UIELKN4ZNJFY2SY6CLQRMXIBA5CNFSM4JFFHO5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECY7S3Y#issuecomment-548534639>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKV4UNXSMMZWAKZMPL3OFLQRMXIBANCNFSM4JFFHO5A> .

[soltysh]

It doesn't have to be long lived one, but I like @smarterclayton's idea of 8h one, although maybe extended to 12h, that should cover 2 TZ which is my entire team, at least.

[smarterclayton]

I still want Ben's infra to be hooked in so we start growing that (we have to have long lived clusters).

…

On Tue, Nov 5, 2019 at 4:30 PM Maciej Szulik ***@***.***> wrote: It doesn't have to be long lived one, but I like @smarterclayton <https://github.com/smarterclayton>'s idea of 8h one, although maybe extended to 12h, that should cover 2 TZ which is my entire team, at least. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5607?email_source=notifications&email_token=AAI37J4ZO6ATLHT4MKNGFS3QSHQW7A5CNFSM4JFFHO5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDEMR5A#issuecomment-550029556>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAI37J4V7NYNL36YX4R72H3QSHQW7ANCNFSM4JFFHO5A> .

[bparees]

I still want Ben's infra to be hooked in so we start growing that (we have
to have long lived clusters).

I was about to switch gears to baremetal clusters, but i just got the LLC creds so i'm going to try to get the AWS longlived cluster jobs going again.