Skip to content

Block build13 from CI rotation due to CDN connectivity issues - #79962

Merged
openshift-merge-bot[bot] merged 1 commit into
openshift:mainfrom
redhat-chai-bot:block-build13
Jun 2, 2026
Merged

openshift-merge-bot[bot] merged 1 commit into
openshift:mainfrom
redhat-chai-bot:block-build13

Conversation

@redhat-chai-bot

@redhat-chai-bot redhat-chai-bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Block build13 from receiving CI jobs due to persistent CDN connectivity failures on node build13-wg9x2-worker-a-wjgxk.

Details

  • The base-4-22-rhel9 mirror pod (fxlbv) on build13 has been crash-looping with 77+ restarts
  • Logs show upstream timed out, Network is unreachable, and no live upstreams errors reaching cdn.redhat.com
  • The issue affects both x86_64 and ppc64le repos, causing 502 errors for DNF operations in CI image builds
  • The problem is isolated to one node but is persistent and recurring

Change

Sets blocked: true for build13 in core-services/sanitize-prow-jobs/_clusters.yaml to prevent new CI jobs from being dispatched there until the node connectivity issue is resolved.

Summary by CodeRabbit

This PR blocks the build13 node from the OpenShift CI job scheduling rotation by updating the cluster configuration to prevent new CI jobs from being dispatched to it.

What changed: The blocked flag for build13 in core-services/sanitize-prow-jobs/_clusters.yaml is set to true, which stops Prow from assigning new CI workloads to this node.

Why: The node is experiencing persistent CDN connectivity failures that are causing the base-4-22-rhel9 mirror pod to crash repeatedly (77+ restarts). CI image builds on this node are encountering 502 errors and connectivity timeouts when attempting to reach cdn.redhat.com, affecting both x86_64 and ppc64le repositories.

Impact: This is a temporary operational measure to isolate the node from the CI workload queue while the underlying CDN connectivity issue is resolved. Existing jobs on the node are not affected, only new job assignments are prevented.

The base-4-22-rhel9 mirror pod on node build13-wg9x2-worker-a-wjgxk
has persistent CDN connectivity failures (upstream timed out, network
unreachable, no live upstreams) causing 502 errors for DNF operations.
The pod has reached 77+ restarts and continues crash-looping.

Blocking build13 until the node connectivity issue is resolved.
@openshift-merge-bot openshift-merge-bot Bot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Jun 2, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@redhat-chai-bot: no rehearsable tests are affected by this change

Note: If this PR includes changes to step registry files (ci-operator/step-registry/) and you expected jobs to be found, try rebasing your PR onto the base branch. This helps pj-rehearse accurately detect changes when the base branch has moved forward.

@coderabbitai

coderabbitai Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: d59e3464-e25b-4af4-b59f-94d55a284d39

📥 Commits

Reviewing files that changed from the base of the PR and between aac464e and eb1ee7f.

📒 Files selected for processing (1)
  • core-services/sanitize-prow-jobs/_clusters.yaml

Walkthrough

The build13 cluster entry in the cluster configuration file is updated to block the cluster from accepting new jobs by changing its blocked status from false to true.

Changes

Cluster Configuration Update

Layer / File(s) Summary
Build13 Cluster Blocked Status
core-services/sanitize-prow-jobs/_clusters.yaml
The build13 cluster definition is updated to set its blocked field to true, preventing it from accepting new Prow jobs.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~2 minutes

🚥 Pre-merge checks | ✅ 5 | ❌ 10

❌ Failed checks (10 inconclusive)

Check name Status Explanation Resolution
Stable And Deterministic Test Names ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Test Structure And Quality ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Microshift Test Compatibility ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Single Node Openshift (Sno) Test Compatibility ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Topology-Aware Scheduling Compatibility ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Ote Binary Stdout Contract ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Ipv6 And Disconnected Network Test Compatibility ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
No-Weak-Crypto ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
Container-Privileges ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
No-Sensitive-Data-In-Logs ❓ Inconclusive Repository clone failed, so this custom check could not run with code access. Retry the review run. If this persists, inspect pre-merge custom-check logs for infrastructure or agent runtime failures.
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main change: blocking build13 from CI rotation due to specific technical issues (CDN connectivity problems).
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci openshift-ci Bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jun 2, 2026
@openshift-ci

openshift-ci Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Hi @redhat-chai-bot. Thanks for your PR.

I'm waiting for a openshift member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci
openshift-ci Bot requested review from Prucek and deepsm007 June 2, 2026 09:41
@deepsm007

Copy link
Copy Markdown
Contributor

/ok-to-test

@openshift-ci openshift-ci Bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 2, 2026
@stbenjam

stbenjam commented Jun 2, 2026

Copy link
Copy Markdown
Member

/ok-to-test
/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jun 2, 2026
@openshift-ci

openshift-ci Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deepsm007, redhat-chai-bot, stbenjam

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 2, 2026
@stbenjam

stbenjam commented Jun 2, 2026

Copy link
Copy Markdown
Member

/label priority/ci-critical

@openshift-ci

openshift-ci Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

@stbenjam: The label(s) priority/ci-critical cannot be applied or removed, because you are not in one of the allowed teams and are not an allowed user. Must be a member of one of these teams: test-platform

Details

In response to this:

/label priority/ci-critical

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci

openshift-ci Bot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

@redhat-chai-bot: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-merge-bot
openshift-merge-bot Bot merged commit 729b498 into openshift:main Jun 2, 2026
7 checks passed
TimurMP pushed a commit to TimurMP/release that referenced this pull request Jun 4, 2026
krisnababu pushed a commit to oharan2/release that referenced this pull request Jul 3, 2026
…ift#79962)

The base-4-22-rhel9 mirror pod on node build13-wg9x2-worker-a-wjgxk
has persistent CDN connectivity failures (upstream timed out, network
unreachable, no live upstreams) causing 502 errors for DNF operations.
The pod has reached 77+ restarts and continues crash-looping.

Blocking build13 until the node connectivity issue is resolved.

Co-authored-by: Chai Bot <chai-bot@redhat.com>
krisnababu pushed a commit to oharan2/release that referenced this pull request Jul 3, 2026
andrej1991 pushed a commit to andrej1991/release that referenced this pull request Jul 6, 2026
…ift#79962)

The base-4-22-rhel9 mirror pod on node build13-wg9x2-worker-a-wjgxk
has persistent CDN connectivity failures (upstream timed out, network
unreachable, no live upstreams) causing 502 errors for DNF operations.
The pod has reached 77+ restarts and continues crash-looping.

Blocking build13 until the node connectivity issue is resolved.

Co-authored-by: Chai Bot <chai-bot@redhat.com>
andrej1991 pushed a commit to andrej1991/release that referenced this pull request Jul 6, 2026
TimurMP pushed a commit to TimurMP/release that referenced this pull request Jul 8, 2026
…ift#79962)

The base-4-22-rhel9 mirror pod on node build13-wg9x2-worker-a-wjgxk
has persistent CDN connectivity failures (upstream timed out, network
unreachable, no live upstreams) causing 502 errors for DNF operations.
The pod has reached 77+ restarts and continues crash-looping.

Blocking build13 until the node connectivity issue is resolved.

Co-authored-by: Chai Bot <chai-bot@redhat.com>
TimurMP pushed a commit to TimurMP/release that referenced this pull request Jul 8, 2026
SachinNinganure pushed a commit to SachinNinganure/release that referenced this pull request Jul 20, 2026
…ift#79962)

The base-4-22-rhel9 mirror pod on node build13-wg9x2-worker-a-wjgxk
has persistent CDN connectivity failures (upstream timed out, network
unreachable, no live upstreams) causing 502 errors for DNF operations.
The pod has reached 77+ restarts and continues crash-looping.

Blocking build13 until the node connectivity issue is resolved.

Co-authored-by: Chai Bot <chai-bot@redhat.com>
SachinNinganure pushed a commit to SachinNinganure/release that referenced this pull request Jul 20, 2026
srivickynesh pushed a commit to srivickynesh/release that referenced this pull request Aug 5, 2026
…ift#79962)

The base-4-22-rhel9 mirror pod on node build13-wg9x2-worker-a-wjgxk
has persistent CDN connectivity failures (upstream timed out, network
unreachable, no live upstreams) causing 502 errors for DNF operations.
The pod has reached 77+ restarts and continues crash-looping.

Blocking build13 until the node connectivity issue is resolved.

Co-authored-by: Chai Bot <chai-bot@redhat.com>
srivickynesh pushed a commit to srivickynesh/release that referenced this pull request Aug 5, 2026
malingatembo pushed a commit to malingatembo/release that referenced this pull request Aug 5, 2026
…ift#79962)

The base-4-22-rhel9 mirror pod on node build13-wg9x2-worker-a-wjgxk
has persistent CDN connectivity failures (upstream timed out, network
unreachable, no live upstreams) causing 502 errors for DNF operations.
The pod has reached 77+ restarts and continues crash-looping.

Blocking build13 until the node connectivity issue is resolved.

Co-authored-by: Chai Bot <chai-bot@redhat.com>
malingatembo pushed a commit to malingatembo/release that referenced this pull request Aug 5, 2026
TimurMP pushed a commit to TimurMP/release that referenced this pull request Sep 5, 2026
…ift#79962)

The base-4-22-rhel9 mirror pod on node build13-wg9x2-worker-a-wjgxk
has persistent CDN connectivity failures (upstream timed out, network
unreachable, no live upstreams) causing 502 errors for DNF operations.
The pod has reached 77+ restarts and continues crash-looping.

Blocking build13 until the node connectivity issue is resolved.

Co-authored-by: Chai Bot <chai-bot@redhat.com>
TimurMP pushed a commit to TimurMP/release that referenced this pull request Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. rehearsals-ack Signifies that rehearsal jobs have been acknowledged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants