dl.k8s.io: Redirect CI URIs to Kubernetes Community infra #1857

justaugustus · 2021-03-31T18:41:54Z

dl.k8s.io: Redirect CI URIs to Kubernetes Community infra

krel ci-build has been successfully building and pushing CI Kubernetes
builds for a time now.

Updating this redirect (dl.k8s.io/ci) to Kubernetes Community infra
moves us closer to no longer relying on the kubernetes-release-dev
(Google-owned) bucket.

ref: ci-kubernetes-build-canary: Migrate from bootstrap to krel test-infra#20663, Deprecate google-hosted kubernetes-build jobs test-infra#20964
dl.k8s.io: Remove ci-cross redirects

All Kubernetes builds that land in the 'ci' directory of k8s-release-dev
are cross builds, so this reference is no longer required.

ref: version-markers: Remove usage of the ci/k8s-master marker test-infra#18517

Signed-off-by: Stephen Augustus [email protected]

Part of kubernetes/release#1711 and #1569

krel ci-build has been successfully building and pushing CI Kubernetes builds for a time now. Updating this redirect (dl.k8s.io/ci) to Kubernetes Community infra moves us closer to no longer relying on the kubernetes-release-dev (Google-owned) bucket. Signed-off-by: Stephen Augustus <[email protected]>

All Kubernetes builds that land in the 'ci' directory of k8s-release-dev are cross builds, so this reference is no longer required. Signed-off-by: Stephen Augustus <[email protected]>

justaugustus · 2021-03-31T18:58:34Z

/hold for Aaron's feedback

/assign @spiffxp @BenTheElder
cc: @kubernetes/release-engineering

@LappleApple @puerco @sladyn98 -- This is the PR I was referring to in today's RelEng triage session.

BenTheElder · 2021-03-31T19:04:50Z

What's the retention on these buckets currently and are the contents equivalent now?
(less concerning for this bucket I think, but in the past when migrating buckets we've tried to ensure the contents will be 1:1)

BenTheElder · 2021-03-31T19:05:28Z

also should this wait for 1.21 wrap up ..?

ameukam · 2021-03-31T19:06:32Z

Ref: #1569

justaugustus · 2021-03-31T19:15:50Z

What's the retention on these buckets currently and are the contents equivalent now?

$ gsutil retention get gs://k8s-release-dev
gs://k8s-release-dev/ has no Retention Policy.
$ gsutil retention get gs://kubernetes-release-dev
gs://kubernetes-release-dev/ has no Retention Policy.

The contents are roughly equivalent for active release branches as the new krel-based and deprecated bootstrap build jobs have been running in tandem for a few months now.

also should this wait for 1.21 wrap up ..?

Absolutely fine for this to wait until 1.21 is wrapped.
It was one of the things on my mind during triage and I figured it'd be good to toss the PR up to discuss. :)

spiffxp · 2021-04-01T00:03:45Z

#525 (comment) re: retention

justaugustus · 2021-04-01T00:12:45Z

#525 (comment) re: retention

Just so we have it in both places:

From https://cloud.google.com/container-registry/docs/managing#deleting_images:

* "Do not apply Cloud Storage retention policies to storage buckets used by Container Registry. These policies to not work for managing images in Container Registry storage buckets." - so we're following the recommended guidance by not setting these

* https://github.com/sethvargo/gcr-cleaner is a not-official-Google-product that could accomplish this

@spiffxp -- This is awesome! Just curious, is this a general FYI or something you'd consider using here?

Mentioning because I think we're only concerned w/ non-container build artifacts for gs://kubernetes-release-dev, right?

justaugustus · 2021-04-15T03:10:47Z

@spiffxp @BenTheElder -- How are we feeling about this now that v1.21.0 has shipped?

spiffxp · 2021-04-22T15:57:30Z

Ah right, brain misfired re: retention. I somehow lumped k8s-staging-ci-images into this.

So my concern with switching right now is possible increased support burden, aka handling "why is my CI job failing?" emails/issues/slack-messages. We still have (flake-induced) diffs between gs://kubernetes-release-dev and gs://k8s-release-dev (ref: #846 (comment)). Ideally, we would either:

address this by syncing from gs://kubernetes-release-dev to gs://k8s-release-dev (periodically or automatically)
automatically measure the diff and make it brightly visible

If I knew someone had bandwidth to actively babysit this or handle an increased support burden, I would feel more comfortable flipping the switch to find out.

FWIW there are also some community jobs that still need to switch over

test-infra$ go test -v -count=1 ./config/tests/jobs/jobs_test.go -run TestKubernetesE2eJobsMustExtractFromK8sInfraBuckets
# ... actual job names ...
    jobs_test.go:1206:  165/2176 jobs should be updated to pull from community-owned gcs buckets

BenTheElder · 2021-04-22T19:04:31Z

I think we should do this in 1.21 👍

@neolit123 can you or one of the other kubadm/kinder contributors / users comment on if this looks ready for your usage?

Otherwise we primarily have kube-up and kops to consider at the moment, I think. In terms of in-tree projects. They largely look the same and I think this should "just work" ™️

BenTheElder · 2021-04-22T19:06:49Z

#846 (comment)

We should either sync (one-way), move dl.k8s.io and other things aggressively (post-v1.21), or make it really really clear when and why this happens

Yes, this should be a prerequisite for flipping the switch. I don't think it should be overly challenging to setup a periodic one-way sync?

neolit123 · 2021-04-22T21:55:51Z

i think this should be fine.

both kinder and kubeadm use:

https://storage.googleapis.com/k8s-release-dev for CI labels and artifacts
https://dl.k8s.io for release labels and artifacts.

maybe we can switch from https://dl.k8s.io to the direct prod bucket eventually (to be more consistent about URLs in these tools).

spiffxp · 2021-04-23T05:00:03Z

Using a regex in these comparisons to exclude some paths:

-bazel: bazel builds in kubernetes-release-dev (there appear to be none in k8s-release-dev, unsure if we should expect some for older branches)
^ci-kubernetes: some coverage jobs dumping data solely in kubernetes-release-dev, these should migrate to k8s-release-dev, no need to sync in two places
fast/: ci/fast is only in k8s-release-dev, no need to sync back to kubernetes-release-dev

If I list one level deep, things look flaky but not too bad:

# date: 2021-04-22T23:20:16
# old: gs://kubernetes-release-dev/ci/
# new: gs://k8s-release-dev/ci/
# exclude_regex: ^(ci-kubernetes|fast)|-bazel
# summary:
#   total  (in either)                       :    901 (100.0%)
#   common (in both)                         :    827 ( 91.8%)
#   only in gs://kubernetes-release-dev/ci/  :     50 (  5.5%)
#   only in gs://k8s-release-dev/ci/         :     24 (  2.7%)

If I list recursively though things look off:

# date: 2021-04-22T23:20:53
# old: gs://kubernetes-release-dev/ci/**
# new: gs://k8s-release-dev/ci/**
# exclude_regex: ^(ci-kubernetes|fast)|-bazel
# summary:
#   total  (in either)                         : 384958 (100.0%)
#   common (in both)                           : 334896 ( 87.0%)
#   only in gs://kubernetes-release-dev/ci/**  :  40443 ( 10.5%)
#   only in gs://k8s-release-dev/ci/**         :   9619 (  2.5%)

Stuff only in k8s-release-dev is 24 flaky builds that line up with above report

$ cat only.new.txt | cut -d/ -f1 | sort | uniq -c | wc -l
      24

Stuff only in kubernetes-release-dev is:

some set of 27 extra files missing for most builds (not all.. so did this start or end within our retention window?)
50 flaky builds that line up with above report

$ cat only.old.txt | cut -d/ -f1 | sort | uniq -c | wc -l
     765
$ cat only.old.txt | cut -d/ -f1 | sort | uniq -c | awk '{print $1}' | sort | uniq -c
 715 27 
  42 417 
   2 444 
   6 456

The 27 files in question:

$ cat only.old.txt | grep v1.22.0-alpha.0.83+669016067d4911
v1.22.0-alpha.0.83+669016067d4911/extra/gce/configure.sh
v1.22.0-alpha.0.83+669016067d4911/extra/gce/configure.sh.sha256
v1.22.0-alpha.0.83+669016067d4911/extra/gce/configure.sh.sha512
v1.22.0-alpha.0.83+669016067d4911/extra/gce/master.yaml
v1.22.0-alpha.0.83+669016067d4911/extra/gce/master.yaml.sha256
v1.22.0-alpha.0.83+669016067d4911/extra/gce/master.yaml.sha512
v1.22.0-alpha.0.83+669016067d4911/extra/gce/node.yaml
v1.22.0-alpha.0.83+669016067d4911/extra/gce/node.yaml.sha256
v1.22.0-alpha.0.83+669016067d4911/extra/gce/node.yaml.sha512
v1.22.0-alpha.0.83+669016067d4911/extra/gce/shutdown.sh
v1.22.0-alpha.0.83+669016067d4911/extra/gce/shutdown.sh.sha256
v1.22.0-alpha.0.83+669016067d4911/extra/gce/shutdown.sh.sha512
v1.22.0-alpha.0.83+669016067d4911/extra/gce/windows/common.psm1
v1.22.0-alpha.0.83+669016067d4911/extra/gce/windows/common.psm1.sha256
v1.22.0-alpha.0.83+669016067d4911/extra/gce/windows/common.psm1.sha512
v1.22.0-alpha.0.83+669016067d4911/extra/gce/windows/configure.ps1
v1.22.0-alpha.0.83+669016067d4911/extra/gce/windows/configure.ps1.sha256
v1.22.0-alpha.0.83+669016067d4911/extra/gce/windows/configure.ps1.sha512
v1.22.0-alpha.0.83+669016067d4911/extra/gce/windows/install-ssh.psm1
v1.22.0-alpha.0.83+669016067d4911/extra/gce/windows/install-ssh.psm1.sha256
v1.22.0-alpha.0.83+669016067d4911/extra/gce/windows/install-ssh.psm1.sha512
v1.22.0-alpha.0.83+669016067d4911/extra/gce/windows/k8s-node-setup.psm1
v1.22.0-alpha.0.83+669016067d4911/extra/gce/windows/k8s-node-setup.psm1.sha256
v1.22.0-alpha.0.83+669016067d4911/extra/gce/windows/k8s-node-setup.psm1.sha512
v1.22.0-alpha.0.83+669016067d4911/extra/gce/windows/user-profile.psm1
v1.22.0-alpha.0.83+669016067d4911/extra/gce/windows/user-profile.psm1.sha256
v1.22.0-alpha.0.83+669016067d4911/extra/gce/windows/user-profile.psm1.sha512

These files were in both buckets as far back as the retention window goes:

$ for v in $(grep -v '^[^/]*.txt$' either.txt | cut -d. -f2 | uniq); do v="v1.${v}"; grep ${v}.*extra/gce/configure.sh.sha512 common.txt | head -n1 | cut -d/ -f1; done;
v1.18.16-rc.0.11+9f76669271b74b
v1.19.8-rc.0.15+b054c92f501386
v1.20.3-rc.0.17+21ad05d5d14004
v1.21.0-alpha.1.268+194e5193c45c8c

Then, they stopped appearing in k8s-release-dev (only appearing in kubernetes-release-dev) as of these builds:

$ for v in $(grep -v '^[^/]*.txt$' either.txt | cut -d. -f2 | uniq); do v="v1.${v}"; grep ${v}.*extra/gce/configure.sh.sha512 only.old.txt | head -n1 | cut -d/ -f1; done;
v1.18.16-rc.0.30+3260181ab19a81
v1.19.10-rc.0.1+2c03527428d49a
v1.20.3-rc.0.15+18194169ac684f
v1.21.0-alpha.1.260+64baf0f73f8231
v1.22.0-alpha.0.1+7146eb593126d7

Is this a change that didn't get ported back to the deprecated builds?

justaugustus · 2021-04-23T06:17:02Z

@kubernetes/release-managers -- Can I ask one of you to help provide feedback on Aaron's investigation?

spiffxp · 2021-06-09T19:35:24Z

Getting current: slightly better (common went from 87.0% to 90.1%), but still not where we need to be

# date: 2021-06-09T15:28:16
# old: gs://kubernetes-release-dev/ci/**
# new: gs://k8s-release-dev/ci/**
# exclude_regex: ^(ci-kubernetes|fast)|-bazel
# summary:
#   total  (in either)                         : 366058 (100.0%)
#   common (in both)                           : 329984 ( 90.1%)
#   only in gs://kubernetes-release-dev/ci/**  :  27151 (  7.4%)
#   only in gs://k8s-release-dev/ci/**         :   8923 (  2.4%)

The questions in #1857 (comment) remain unanswered.

spiffxp · 2021-06-09T19:35:53Z

/sig testing
/sig release
/area artifacts
/milestone v1.22

spiffxp · 2021-07-02T20:25:18Z

A fun wrinkle I realized while asking myself "wait, why don't I just use gustil rsync for this" when trying to clean up my weird diff report script for PR.

The files in each bucket that have matching filenames are not bit-for-bit identical files, nor do they have matching timestamps, since they're built in two different jobs at two slightly different times. So rsync will by default try to overwrite basically everything.

spiffxp · 2021-07-02T20:26:56Z

I am FWIW willing to flip the switch and find out what happens over what will be for some a long holiday weekend. I realize code freeze is coming up, but the next few days should be quiet, and I can be available to revert if need be.

justaugustus · 2021-07-02T22:24:03Z

I am FWIW willing to flip the switch and find out what happens over what will be for some a long holiday weekend. I realize code freeze is coming up, but the next few days should be quiet, and I can be available to revert if need be.

In favor of this.
I can keep an eye out over the weekend as well.

spiffxp

/approve
/lgtm
Going to flip the switch and see what happens. I will be available to revert if things blow up. Mindful of impacting the run up to code freeze.

k8s-ci-robot · 2021-07-02T22:27:27Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: justaugustus, spiffxp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~k8s.io/OWNERS~~ [spiffxp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

justaugustus · 2021-07-02T22:38:10Z

/hold cancel

spiffxp · 2021-07-02T22:39:45Z

Opened #2291 with a script I've been using to keep an eye on the delta between gs://k8s-release-dev and gs://kubernetes-release-dev

spiffxp · 2021-07-03T00:03:52Z

k8s.io/configmap-nginx.yaml

@@ -158,7 +158,7 @@ data:
          # Don't require /release/ if you want to get at the Kubernetes release artifacts, the common case.
          rewrite ^/(v[0-9]+\.[0-9]+\.[0-9]+(-(alpha|beta|rc)\.[0-9]+)?/.*)$   https://storage.googleapis.com/kubernetes-release/release/$1 redirect;
          # Provide a convenient redirect for CI (continuous integration) artifacts as well, which live in a different bucket.
-          rewrite ^/ci(-cross)?/?(.*)$              https://storage.googleapis.com/kubernetes-release-dev/ci$1/$2 redirect;
+          rewrite ^/ci/?(.*)$              https://storage.googleapis.com/k8s-release-dev/ci$1/$2 redirect;


Two things I noticed when trying to deploy:

ci-cross is still referenced here, and cutting .debs may be broken until this code is fixed: https://github.com/kubernetes/release/blob/6e3b30b6dd78b705474a1086f2ed7f7b8364029b/packages/deb/build.go#L291-L302

removing the first capture group (-cross) means the the replacement is incorrect

Fixup PR inbound

Ahhh, regex, my old enemy.
Thanks for the cleanup, Aaron.

I'll fixup the k/release references.

Opened #2292

* `ci-cross` is still referenced here, and cutting .debs may be broken until this code is fixed: https://github.com/kubernetes/release/blob/6e3b30b6dd78b705474a1086f2ed7f7b8364029b/packages/deb/build.go#L291-L302

Fixed in kubernetes/release#2153.

spiffxp · 2021-07-03T00:14:26Z

I deployed using the contents of #2292

cd k8s.io
./deploy.sh canary
# tests passed

Deployed to prod

./deploy.sh prod
# ...
======================================================================
FAIL: test_dl (__main__.RedirTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test.py", line 294, in test_dl
    ver=rand_num(), path=rand_num())
  File "./test.py", line 100, in assert_temp_redirect
    self.assert_multischeme_redirect(partial_url, expected_loc, 302, **kwargs)
  File "./test.py", line 97, in assert_multischeme_redirect
    scheme + '://' + partial_url, expected_loc, expected_code, **kwargs)
  File "./test.py", line 92, in assert_scheme_redirect
    % (url, expected_loc, resp.getheader('location')))
AssertionError: 'https://storage.googleapis.com/kubernetes-release-dev/ci/v3487/3552' != 'https://storage.googleapis.com/k8s-release-dev/ci/v3487/3552'
- https://storage.googleapis.com/kubernetes-release-dev/ci/v3487/3552
?                                 ^^^^^^^^
+ https://storage.googleapis.com/k8s-release-dev/ci/v3487/3552
?                                 ^
 :
GET "http://dl.k8s.io/ci/v3487/3552" got an unexpected redirect location:
 want: https://storage.googleapis.com/k8s-release-dev/ci/v3487/3552
 got:  https://storage.googleapis.com/kubernetes-release-dev/ci/v3487/3552

I'm going to assume this is eventually consistent, waiting a bit

spiffxp · 2021-07-03T00:15:45Z

Yup, good now

----------------------------------------------------------------------
Ran 25 tests in 13.379s

OK

spiffxp · 2021-07-03T00:30:07Z

OK so as of ~2020-07-03 00:15:00, https://dl.k8s.io/ci redirects to gs://k8s-release-dev instead of gs://kubernetes-release-dev

Any breakages, report them here. Broader notification later.

k8s-ci-robot requested review from aojea and nikhita March 31, 2021 18:42

k8s-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Mar 31, 2021

dl.k8s.io: Remove ci-cross redirects

cb12adb

All Kubernetes builds that land in the 'ci' directory of k8s-release-dev are cross builds, so this reference is no longer required. Signed-off-by: Stephen Augustus <[email protected]>

k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Mar 31, 2021

justaugustus changed the title ~~[WIP] dl.k8s.io: Redirect CI URIs to Kubernetes Community infra~~ dl.k8s.io: Redirect CI URIs to Kubernetes Community infra Mar 31, 2021

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 31, 2021

k8s-ci-robot assigned BenTheElder and spiffxp Mar 31, 2021

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 31, 2021

justaugustus mentioned this pull request Mar 31, 2021

Redirect dl.k8s.io traffic to a community-owned GCS bucket instead of kubernetes-release #1569

Open

k8s-ci-robot added sig/testing Categorizes an issue or PR as relevant to SIG Testing. sig/release Categorizes an issue or PR as relevant to SIG Release. labels Jun 9, 2021

k8s-ci-robot added this to the v1.22 milestone Jun 9, 2021

k8s-ci-robot added the area/artifacts Issues or PRs related to the hosting of release artifacts for subprojects label Jun 9, 2021

spiffxp approved these changes Jul 2, 2021

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 2, 2021

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 2, 2021

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 2, 2021

spiffxp mentioned this pull request Jul 2, 2021

hack: add k8s-infra-diff-buckets.sh #2291

Merged

k8s-ci-robot merged commit f262df3 into kubernetes:main Jul 2, 2021

spiffxp reviewed Jul 3, 2021

View reviewed changes

justaugustus mentioned this pull request Jul 3, 2021

packages/deb: Use ci/latest.txt as canonical cross build marker kubernetes/release#2153

Merged

spiffxp mentioned this pull request Jul 12, 2021

use gs://k8s-release-dev for kubernetes CI builds kubernetes/test-infra#22840

Merged

spiffxp mentioned this pull request Jul 29, 2021

Deprecate and migrate away from gs://kubernetes-release-dev and gcr.io/kubernetes-ci-images #2318

Closed

42 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dl.k8s.io: Redirect CI URIs to Kubernetes Community infra #1857

dl.k8s.io: Redirect CI URIs to Kubernetes Community infra #1857

justaugustus commented Mar 31, 2021 •

edited

Loading

justaugustus commented Mar 31, 2021

BenTheElder commented Mar 31, 2021

BenTheElder commented Mar 31, 2021

ameukam commented Mar 31, 2021

justaugustus commented Mar 31, 2021

spiffxp commented Apr 1, 2021

justaugustus commented Apr 1, 2021 •

edited

Loading

justaugustus commented Apr 15, 2021

spiffxp commented Apr 22, 2021

BenTheElder commented Apr 22, 2021

BenTheElder commented Apr 22, 2021 •

edited

Loading

neolit123 commented Apr 22, 2021

spiffxp commented Apr 23, 2021

justaugustus commented Apr 23, 2021

spiffxp commented Jun 9, 2021

spiffxp commented Jun 9, 2021

spiffxp commented Jul 2, 2021

spiffxp commented Jul 2, 2021

justaugustus commented Jul 2, 2021

spiffxp left a comment

k8s-ci-robot commented Jul 2, 2021

justaugustus commented Jul 2, 2021

spiffxp commented Jul 2, 2021

spiffxp Jul 3, 2021 •

edited

Loading

justaugustus Jul 3, 2021

spiffxp Jul 3, 2021

justaugustus Jul 3, 2021

spiffxp commented Jul 3, 2021

spiffxp commented Jul 3, 2021

spiffxp commented Jul 3, 2021 •

edited

Loading

dl.k8s.io: Redirect CI URIs to Kubernetes Community infra #1857

dl.k8s.io: Redirect CI URIs to Kubernetes Community infra #1857

Conversation

justaugustus commented Mar 31, 2021 • edited Loading

justaugustus commented Mar 31, 2021

BenTheElder commented Mar 31, 2021

BenTheElder commented Mar 31, 2021

ameukam commented Mar 31, 2021

justaugustus commented Mar 31, 2021

spiffxp commented Apr 1, 2021

justaugustus commented Apr 1, 2021 • edited Loading

justaugustus commented Apr 15, 2021

spiffxp commented Apr 22, 2021

BenTheElder commented Apr 22, 2021

BenTheElder commented Apr 22, 2021 • edited Loading

neolit123 commented Apr 22, 2021

spiffxp commented Apr 23, 2021

justaugustus commented Apr 23, 2021

spiffxp commented Jun 9, 2021

spiffxp commented Jun 9, 2021

spiffxp commented Jul 2, 2021

spiffxp commented Jul 2, 2021

justaugustus commented Jul 2, 2021

spiffxp left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Jul 2, 2021

justaugustus commented Jul 2, 2021

spiffxp commented Jul 2, 2021

spiffxp Jul 3, 2021 • edited Loading

Choose a reason for hiding this comment

justaugustus Jul 3, 2021

Choose a reason for hiding this comment

spiffxp Jul 3, 2021

Choose a reason for hiding this comment

justaugustus Jul 3, 2021

Choose a reason for hiding this comment

spiffxp commented Jul 3, 2021

spiffxp commented Jul 3, 2021

spiffxp commented Jul 3, 2021 • edited Loading

justaugustus commented Mar 31, 2021 •

edited

Loading

justaugustus commented Apr 1, 2021 •

edited

Loading

BenTheElder commented Apr 22, 2021 •

edited

Loading

spiffxp Jul 3, 2021 •

edited

Loading

spiffxp commented Jul 3, 2021 •

edited

Loading