Moving images away from quay.io?

[xing-yang]

CI is failing because quay.io is down. It's reported in this issue:
kubernetes/kubernetes#91242

Should we move our images to a different repo, i.e., the community infra?

CC @msau42 @pohly

[xing-yang]

One update from https://status.quay.io/ says that "The number of open connections to the Quay.io database increased significantly causing an outage of the service, we are still investigating the root cause of this spike in DB connections. "

[xing-yang]

We have staging repos for CSI in GCR and the image promotion process. It's used for node-driver-registrar only right now. Should we do this for all sidecar repos?
https://github.com/kubernetes-csi/node-driver-registrar/blob/master/cloudbuild.yaml

[justaugustus]

Thanks for opening this, @xing-yang!
I poked @hasheddan to get the conversation started in kubernetes/kubernetes#91242 / https://kubernetes.slack.com/archives/CJH2GBF7Y/p1589887799251500.

Where possible, images that are used by that community should be on community infra.
I'd suggest moving anything currently hosted at quay.io/k8scsi be moved in a CSI staging repository and then promoted to our prod registry.

Let me know how I can help and feel free to assign me in reviews. :)

[xing-yang]

Thanks @justaugustus!

[pohly]

Work is in progress. It's not just about pushing to some other registry, but also about using an entirely different build infrastructure and producing images for multiple different architectures, ideally including Windows.

As a next step it would help if someone could approve kubernetes-csi/node-driver-registrar#82. As I said in the review of that PR, we need to figure out how building of Windows images can be unified with the approach taken for all other platforms (i.e. build the binary inside the multi-stage Dockerfile, using the same base images if possible).

/cc @vitt-bagal @jingxu97 @ddebroy

[pohly]

Previous PRs:

• Multiarch Docker Image support using GCB node-driver-registrar#68
• Fixing cloudbuild.yml for multiarch Docker Image support node-driver-registrar#72

[pohly]

• try out cloud build one last time after merging multiarch image pushing node-driver-registrar#84: https://prow.k8s.io/?job=post-node-driver-registrar-push-images
• move common code and files into csi-release-tools: cloud build: initial set of shared files #87
• update node-driver-registrar with that: master: update release-tools node-driver-registrar#86
• promote multi-arch mock driver v3.1.1 image to from staging repo to final repo as a test: kubernetes-csi: promote first multi-arch mock-driver release kubernetes/k8s.io#940
• update all repos with newest csi-release-tools and ARG binary in their Dockerfile(s):
  • master: update release-tools, enable cloud build csi-driver-host-path#180
  • master: update release-tools, enable cloud build csi-test#270
  • master: update release-tools, enable cloud build external-attacher#231
  • master: update release-tools, enable cloud build external-provisioner#439
  • master: update release-tools, enable cloud build external-resizer#84
  • master: update release-tools, enable cloud build external-snapshotter#319
  • master: update release-tools, enable cloud build livenessprobe#71
• fix some issues in the cloud build rules (Dockerfile.Windows must be optional): various PRs which update csi-release-tools
• create build jobs similar to https://github.com/kubernetes/test-infra/blob/master/config/jobs/image-pushing/k8s-staging-csi.yaml (ideally, through the generator script instead of manually...): image pushing: configure jobs for all Kubernetes-CSI repos kubernetes/test-infra#17741, dashboard https://testgrid.k8s.io/sig-storage-image-build
• promote existing releases to k8s.gcr.io to ease the migration (first change registry, then version): kubernetes-csi: promote existing releases kubernetes/k8s.io#934 was merged, but vanity domain flip still needs to happen before the images become visible under k8s.gcr.io/csi (temporary workaround: use us.gcr.io/k8s-artifacts-prod/csi)
• fix hostpath driver build issue - it uses RUN apk and that fails with failed to solve: rpc error: code = Unknown desc = failed to load LLB: runtime execution on platform linux/ppc64le not supported
• switch from staging-kubernetes-csi (= k8s.gcr.io/csi) to staging-sig-storage (= k8s.gcr.io/sig-storage) - see add staging project for K8s storage kubernetes/k8s.io#943: done in csi-release-tools, test-infra and several repos; some PRs are still pending:
  • master: update release-tools csi-driver-iscsi#27
  • master: update release-tools csi-proxy#60
• change hostpath example deployment files to use k8s.gcr.io:
  Update e2e csi images to k8s.gcr.io kubernetes/kubernetes#93521
• ensure that "canary" images are always available on sig-storage-staging: if we don't rebuild them regularly, they will get removed eventually (Set retention for staging images kubernetes/k8s.io#525)
• disable TravisCI in all repos

[pohly]

/assign

[pohly]

The vanity domain switch has completed: https://groups.google.com/forum/#!msg/kubernetes-dev/Z-y1EK-XBQM/-fehCfLpAAAJ

[pohly]

k8s.gcr.io works for "docker pull" but only redirects to https://cloud.google.com/container-registry/ when used in a web browser. To list images, use https://console.cloud.google.com/gcr/images/k8s-artifacts-prod/US/sig-storage

[jingxu97]

@pohly for the multi-arch image of node-driver-registar, when can it be promoted from staging to release? https://console.cloud.google.com/gcr/images/k8s-staging-sig-storage/GLOBAL/csi-node-driver-registrar?gcrImageListsize=30

[pohly]

node-driver-registrar needs a formal new release. We don't promote rc or canary images to the production repo. According to https://github.com/orgs/kubernetes-csi/projects/29#card-42709613, it's "not ready".

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[xing-yang]

/remove-lifecycle stale

[msau42]

I believe the last thing that's still remaining is to turn down our travis jobs that push to quay. I think we can do this once those minor versions are out of our support window.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[pohly]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[xing-yang]

@msau42 @pohly do we still need to keep this issue open?

[msau42]

Yes travis jobs are disabled, so I think we're done here.

/close

[k8s-ci-robot]

@msau42: Closing this issue.

In response to this:

Yes travis jobs are disabled, so I think we're done here.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.