Update k8s-staging-test-infra GCR images as needed #32863
Conversation
|
@k8s-infra-ci-robot: GitHub didn't allow me to request PR reviews from the following users: nathanperkins. Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
k8s-ci-robot
added
cncf-cla: yes
k8s-infra-ci-robot
force-pushed
the
prowjobs-autobump
branch
from
06e758f
to
f35475e
Compare
k8s-ci-robot
added
size/L
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: k8s-infra-ci-robot The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
area/images
area/release-eng
k8s-infra-ci-robot
force-pushed
the
prowjobs-autobump
branch
3 times, most recently
from
81dab82
to
5ff2aad
Compare
k8s-ci-robot
added
area/provider/azure
k8s-infra-ci-robot
force-pushed
the
prowjobs-autobump
branch
from
5ff2aad
to
7d5487e
Compare
…nd k8s-staging-test-infra AR images and k8s-staging-apisnoop-test-infra and k8s-staging-apisnoop-apisnoop No gcr.io/k8s-testimages/ changes. Multiple distinct gcr.io/k8s-staging-test-infra changes: Commits | Dates | Images --- | --- | --- kubernetes/test-infra@69ac574...6dd397d | 2024‑02‑05 → 2024‑06‑27 | bigquery kubernetes/test-infra@3b134c2...6dd397d | 2024‑03‑08 → 2024‑06‑27 | bootstrap kubernetes/test-infra@597c402...1dde27f | 2024‑06‑11 → 2024‑06‑25 | kubekins-e2e(1.29), kubekins-e2e(master) kubernetes/test-infra@1dde27f...6dd397d | 2024‑06‑25 → 2024‑06‑27 | krte(1.27), krte(1.28), krte(1.29), krte(1.30), krte(experimental), krte(master) No us-central1-docker.pkg.dev/k8s-staging-test-infra/images changes. No gcr.io/k8s-staging-apisnoop/ changes. No gcr.io/k8s-staging-apisnoop/ changes.
k8s-infra-ci-robot
force-pushed
the
prowjobs-autobump
branch
from
7d5487e
to
f0e0047
Compare
dims
added
the
skip-review
|
@k8s-infra-ci-robot: Updated the
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@dims this broke kind. EDIT: this is a useless comment without further context and verification, in the future will refrain from jumping on that assumption so quickly, apologies. |
|
... confirming here kubernetes-sigs/kind#648 (comment) integration tests hitting the same issue as https://kubernetes.slack.com/archives/CEKK1KTN2/p1719537867758879 by the looks of it. |
|
... pretty sure anyhow, appears to be a problematic docker upgrade breaking IPV6 stuff. cc @aojea we're going to have to dig into this, as soon as I get a failure on the no-op PR will revert this and then we can look into rollback looks like major changes to ipv6 in docker. |
|
... ok, now that is weird https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kind/648/pull-kind-test/1807900248492740608 it ... passed? .... but none of the changes in https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kind/3676/pull-kind-test/1807897149162131456 should affect this ... |
|
I think maybe it became racy, with something else having modprobed on the node, but I'm still suspicious of a docker upgrade because:
Let me pull one of these images and confirm what docker version it has ... The other thought is that it is mis-attributed to this change and is instead the build cluster. |
|
Yeah, we picked up docker 27.x: |
|
Tracking the kind side of this at kubernetes-sigs/kind#3677 |
Ugh! do we want to revert? |
|
Trying to figure out if it's causing issues in other CI, but I had to go out. It's only a light flake in kind, but I'm concerned that we're going to find more issues, it's causing the network creation to flake and some behavior change in IPv6 networking. Let's leave it for the moment. |
|
Ok, what it seems to happen is that now docker REQUIRES ip6tables, We had a knob to enable this test-infra/images/bootstrap/runner.sh Lines 46 to 47 in 0f25764 that installed the required module test-infra/images/bootstrap/runner.sh Lines 61 to 62 in 0f25764 and now some jobs does not seem to have it
@BenTheElder we are back in 2019 😄 , my memory may fail, but I think that some images didn't have that module? |
|
is this is a GKE cluster/pool issue? based on |
on the ubuntu nodes it's just not loaded by default but the module is available, since the host nodes are ipv4. previously we added the modprobe only when we intend to use ipv6 and then enable it in docker, but now ipv6 is always enabled |
|
let's use kubernetes-sigs/kind#3677 to track, even though technically this affects docker networks in all jobs, we do not have evidence yet that other jobs are creating networks (I could see this causing issues for the default bridge but we have no proof yet). |
|
I think this is resolved now, if not fully cleanly, will keep an eye out for any other issues. We're ensuring we load the ipv6 NAT module when setting up dind, and having all dind jobs mount /lib/modules. We can do something more clever in the future. There may be other issues from the v27 changes, but I'm not seeing them yet. |
No gcr.io/k8s-testimages/ changes.
Multiple distinct gcr.io/k8s-staging-test-infra changes:
No us-central1-docker.pkg.dev/k8s-staging-test-infra/images changes.
No gcr.io/k8s-staging-apisnoop/ changes.
No gcr.io/k8s-staging-apisnoop/ changes.
/cc @nathanperkins