Set retention for staging images

[stp-ip]

We currently set a 60d retention on staging storage and staging gcb storage, but don't enforce any retention for images.

Staging images should be discouraged from being used and therefore adding a retention policy would help setting the right expectations as well as keep our storage needs lower in the long run.

I am proposing the same 60d retention to keep things the same across all staging retention settings. Happy for other suggestions.

Additional notes:
Currently GCR itself doesn't provide retention settings. We could create the retention on the GCR created bucket, but I assume this could lead to weird issues.
The other option could run a prow job every week to clean up older images.

"Manual" removal script example: https://gist.github.com/ahmetb/7ce6d741bd5baa194a3fac6b1fec8bb7

[rajibmitra]

I would like to work on prow job that will clean up the older images.

[rajibmitra]

/assign fiorm

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[spiffxp]

/reopen

[k8s-ci-robot]

@spiffxp: Reopened this issue.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[spiffxp]

I agree 60d for GCR is reasonable. Staging GCR repos have images older than 60d. They should not, or people are going to assume they can use them in perpetuity.

This came up because @msau42 mentioned CSI images were close to 60d and was concerned they would expire and break kubernetes CI. They won't.

$ export prj=k8s-staging-csi; for b in $prj $prj-gcb artifacts.$prj.appspot.com; do echo $b: $(gsutil lifecycle get gs://$b); done
k8s-staging-csi: {"rule": [{"action": {"type": "Delete"}, "condition": {"age": 60}}]}
k8s-staging-csi-gcb: {"rule": [{"action": {"type": "Delete"}, "condition": {"age": 60}}]}
artifacts.k8s-staging-csi.appspot.com: gs://artifacts.k8s-staging-csi.appspot.com/ has no lifecycle configuration.

We should give SIG Storage time to promote their images to k8s.gcr.io and update tests to use them. Then I think we should implement this.

/assign @thockin @dims @bartsmykla
as a heads up

[thockin]

I don't object in theory. There isn't a good mechanism to do it, short of writing our own daily things that loops over every staging repo and nukes old images.

[bartsmykla]

I can help with our own solution :-)

[msau42]

Is there a recommended way to do canary testing with the 60d removal? For example, in csi, our periodic canary testing tests multiple repos' canary images in one job. But some repos are more active than others, and the inactive ones may not have any merges for > 60d. Is there a way we can keep the canary images around to facilitate this workflow without having to promote the canary tag?

[thockin]

Define canary? If you need them long-term, why not promote them?

…

On Wed, Aug 5, 2020 at 2:10 PM Michelle Au ***@***.***> wrote: Is there a recommended way to do canary testing with the 60d removal? For example, in csi, our periodic canary testing tests multiple repos' canary images in one job. But some repos are more active than others, and the inactive ones may not have any merges for > 60d. Is there a way we can keep the canary images around to facilitate this workflow without having to promote the canary tag? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#525 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVH5FBZLYUJENK6BUTDR7HDDPANCNFSM4J5IUKCA> .

[msau42]

canary in our case is actually images with a "canary" tag. Every pr that merges, we build and repush the "canary" tag. We have a specific canary job that's configured to test using images with the canary tag. We do end up promoting those images with official release tags, and we have separate jobs that test using release images, but we will still have a canary job that tests against head of everything.

[thockin]

Yeah, you would not want to promote those, and tags are not mutable in prod anyway. So the goal is to keep the last N builds (N may be 1), regardless of age or whether that build was promoted or not? That seems like a recipe for flakes, no? If we were building our own retirement, we could (for example) always leave a specific tag or something, I guess? But I am still shaky on the idea.

…

On Wed, Aug 5, 2020 at 2:44 PM Michelle Au ***@***.***> wrote: canary in our case is actually images with a "canary" tag. Every pr that merges, we build and repush the "canary" tag. We have a specific canary job that's configured to test using images with the canary tag. We do end up promoting those images with official release tags, and we have separate jobs that test using release images, but we will still have a canary job that tests against head of everything. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#525 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVHPSXVOCLOUCJEIQP3R7HHFNANCNFSM4J5IUKCA> .

[msau42]

If there's another way we could achieve a "test against latest for all images, even if some of the repos are inactive", open to suggestions.

[pohly]

Perhaps we can add a periodic job which rebuilds canary images once a month? The added bonus is that we'll notice if something breaks in the build environment (shouldn't happen, but one never knows...) before actually trying to build a proper release.

[thockin]

Why do you want to test against head as opposed to the latest release? Even if that release is explicitly a "daily snapshot" or something? I guess I don't fully know what all the images are or what you are testing...

…

On Thu, Aug 6, 2020 at 12:08 AM Patrick Ohly ***@***.***> wrote: Perhaps we can add a periodic job which rebuilds canary images once a month? The added bonus is that we'll notice if something breaks in the build environment (shouldn't happen, but one never knows...) before actually trying to build a proper release. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#525 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVALGQLZ2WSHQRQ6GEDR7JJINANCNFSM4J5IUKCA> .

[msau42]

I think a daily snapshot also works. It's similar to Patrick's suggestion but more frequent. We'll need to redo our tooling to be able to query/find the daily rc, but it's feasible.

I'm curious what other projects are doing, because I can't imagine we're the only ones.

[thockin]

Most don't seem to be doing much, yet, or they are releasing all components together.

…

On Thu, Aug 6, 2020 at 9:53 AM Michelle Au ***@***.***> wrote: I think a daily snapshot also works. It's similar to Patrick's suggestion but more frequent. We'll need to redo our tooling to be able to query/find the daily rc, but it's feasible. I'm curious what other projects are doing, because I can't imagine we're the only ones. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#525 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVDF52AM46CMVNTYOXDR7LNXHANCNFSM4J5IUKCA> .

[pohly]

We'll need to redo our tooling to be able to query/find the daily rc, but it's feasible.

So you want "canary-" tags because then a test failure can be reproduced locally with the exact same images? I'm not sure how important that is. If it's a genuine issue that hasn't been fixed yet, a more recent "canary" should still expose it, and if it doesn't, would we really dig into such a failure when it no longer occurs?

[pohly]

or they are releasing all components together

Built from the same repo? That works because the tip-of-branch components can all be built in the same test job.

But Kubernetes-CSI uses several different repos and then needs to collect the output from different build jobs for a combined test job.

[pohly]

I guess we could set up a canary job which checks out all of the relevant repos and then builds everything anew each time it runs. 🤷

[thockin]

So there are several options here, and I can't know which makes the most sense for your goals on this. IF we NEED to, we could maybe add one label that we always retain, but IMO that suggests something else is fishy. Why would you keep a "canary" for a long time and NOT promote it or replace it?

…

On Fri, Aug 7, 2020 at 3:57 AM Patrick Ohly ***@***.***> wrote: I guess we could set up a canary job which checks out all of the relevant repos and then builds everything anew each time it runs. 🤷 — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#525 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVAHQZJIEEIF4YOLN53R7PMXZANCNFSM4J5IUKCA> .

[msau42]

We do promote our canaries. The main challenge here is we have two test jobs, one configured to only run against canaries, and one configured to run against release images. The one configured to run against canary is going to be prone to expiration of staging images for inactive repos unless we have something that will periodically build new images.

[spiffxp]

/lifecycle frozen

[pohly]

We do promote our canaries. ... The one configured to run against canary is going to be prone to expiration of staging images for inactive repos

I think you meant "we don't promote our canaries", right?

unless we have something that will periodically build new images

Here's a PR which tentatively defines a job which refreshes "canary" for one repo: kubernetes/test-infra#19734

Release candidates are still problematic. We sometimes need those while preparing new sidecars for an upcoming Kubernetes release. On the other hand, the time period where we do need them might be small enough that the normal retention period is okay, so this might not be a problem?

[msau42]

What I meant was we do promote canary builds to official release version tags. We don't promote the "canary" tag.

Yes I think we can treat release candidates separately. We don't want to promote release candidates and merge any tests that depend on release candidates in k/k

[spiffxp]

From https://cloud.google.com/container-registry/docs/managing#deleting_images:

• "Do not apply Cloud Storage retention policies to storage buckets used by Container Registry. These policies to not work for managing images in Container Registry storage buckets." - so we're following the recommended guidance by not setting these
• https://github.com/sethvargo/gcr-cleaner is a not-official-Google-product that could accomplish this

[spiffxp]

/milestone v1.23

[ameukam]

/milestone clear

[ameukam]

/milestone v1.32