CNTRLPLANE-3509: Resolve N-minor release images via release controller API (v2) - #80360
Conversation
openshift-ci-robot
added
the
jira/valid-reference
|
@jparrill: This pull request references CNTRLPLANE-3509 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
|
Skipping CI for Draft Pull Request. |
openshift-ci
Bot
added
the
approved
|
/pj-rehearse ? |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdd a hypershift-resolve-nodepool-releases step that queries the release-controller for NODEPOOL_N1..N4, writes exports to ${SHARED_DIR}/nodepool_release_images; switch jobs to accept NODEPOOL_N*_VERSION; scripts source the file and only add --e2e.n*-minor-release-image flags when resolved; workflows insert the new pre-step. ChangesHyperShift NodePool Release Resolution Mechanism
Sequence Diagram(s)sequenceDiagram
participant Workflow as CI Workflow
participant ResolveStep as hypershift-resolve-nodepool-releases
participant ReleaseCtrl as release-controller API
participant SharedDir as ${SHARED_DIR}
participant RunE2E as run-e2e / create-guests scripts
Workflow->>ResolveStep: run pre-step (NODEPOOL_N*_VERSION env)
ResolveStep->>ReleaseCtrl: request releasestream for NODEPOOL_NX_VERSION
ReleaseCtrl-->>ResolveStep: return JSON (.pullSpec) or non-200
ResolveStep->>ResolveStep: retry up to 3 on failure
ResolveStep->>SharedDir: write export OCP_IMAGE_NX="pullSpec"
Workflow->>RunE2E: start test step
RunE2E->>SharedDir: source nodepool_release_images (if exists)
RunE2E->>RunE2E: add --e2e.n*-minor-release-image flags only if OCP_IMAGE_N* set and != OCP_IMAGE_LATEST
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Suggested labels
Suggested reviewers
Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error, 1 warning)
✅ Passed checks (13 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
@openshift-ci-robot: your |
|
@openshift-ci[bot]: your |
|
@jparrill: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@jparrill: job(s): , ? either don't exist or were not found to be affected, and cannot be rehearsed |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@ci-operator/config/openshift/hypershift/openshift-hypershift-main.yaml`:
- Around line 529-532: The nodepool resolver always uses the amd64 controller
URL (RELEASE_CONTROLLER_URL) and ignores AWS_MULTI_ARCH, causing amd64-only
pullspecs for multi-arch runs; update the
hypershift-resolve-nodepool-releases-commands.sh logic to check AWS_MULTI_ARCH
(or the equivalent env var) and, when true, set RELEASE_CONTROLLER_URL to the
multi-arch release controller endpoint (or invoke the release lookup with the
multi-arch controller or an arch-appropriate query) before resolving
NODEPOOL_N1_VERSION..NODEPOOL_N4_VERSION so that OCP_IMAGE_N1..N4 are resolved
to multi-arch/appropriate-arch pullspecs consumed by the run-e2e steps. Ensure
the switch references the existing RELEASE_CONTROLLER_URL variable and the
NODEPOOL_N{1-4}_VERSION -> OCP_IMAGE_N{1-4} resolution code paths so no other
behavior changes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 93129d0d-77bb-43ce-9323-e8fa329f3033
📒 Files selected for processing (19)
ci-operator/config/openshift/hypershift/openshift-hypershift-main.yamlci-operator/step-registry/hypershift/aws/e2e/external/hypershift-aws-e2e-external-workflow.yamlci-operator/step-registry/hypershift/aws/e2e/external/oidc/hypershift-aws-e2e-external-oidc-workflow.yamlci-operator/step-registry/hypershift/aws/e2e/metrics/hypershift-aws-e2e-metrics-workflow.yamlci-operator/step-registry/hypershift/aws/e2e/nested/hypershift-aws-e2e-nested-workflow.yamlci-operator/step-registry/hypershift/aws/run-e2e/external/hypershift-aws-run-e2e-external-commands.shci-operator/step-registry/hypershift/aws/run-e2e/nested/hypershift-aws-run-e2e-nested-commands.shci-operator/step-registry/hypershift/azure/aks/e2e/hypershift-azure-aks-e2e-workflow.yamlci-operator/step-registry/hypershift/azure/aks/external-oidc/hypershift-azure-aks-external-oidc-workflow.yamlci-operator/step-registry/hypershift/azure/create-selfmanaged-guests/hypershift-azure-create-selfmanaged-guests-commands.shci-operator/step-registry/hypershift/azure/create-selfmanaged-guests/hypershift-azure-create-selfmanaged-guests-ref.yamlci-operator/step-registry/hypershift/azure/e2e/self-managed/hypershift-azure-e2e-self-managed-workflow.yamlci-operator/step-registry/hypershift/azure/e2e/v2-self-managed/hypershift-azure-e2e-v2-self-managed-workflow.yamlci-operator/step-registry/hypershift/azure/run-e2e-self-managed/hypershift-azure-run-e2e-self-managed-commands.shci-operator/step-registry/hypershift/azure/run-e2e/hypershift-azure-run-e2e-commands.shci-operator/step-registry/hypershift/resolve-nodepool-releases/OWNERSci-operator/step-registry/hypershift/resolve-nodepool-releases/hypershift-resolve-nodepool-releases-commands.shci-operator/step-registry/hypershift/resolve-nodepool-releases/hypershift-resolve-nodepool-releases-ref.metadata.jsonci-operator/step-registry/hypershift/resolve-nodepool-releases/hypershift-resolve-nodepool-releases-ref.yaml
3ecbdb1 to
cb94f24
Compare
|
/lgtm |
|
/pj-rehearse pull-ci-openshift-cloud-network-config-controller-release-4.22-hypershift-e2e-aks |
|
/pj-rehearse pull-ci-openshift-cluster-csi-snapshot-controller-operator-release-4.22-hypershift-aws-e2e-external |
|
/pj-rehearse pull-ci-openshift-aws-karpenter-provider-aws-release-4.22-e2e-hypershift |
|
@jparrill: |
|
@jparrill: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@jparrill: your |
1 similar comment
|
@jparrill: your |
|
@openshift-ci[bot]: your |
|
@openshift-merge-bot[bot]: your |
5 similar comments
|
@openshift-merge-bot[bot]: your |
|
@openshift-merge-bot[bot]: your |
|
@openshift-merge-bot[bot]: your |
|
@openshift-merge-bot[bot]: your |
|
@openshift-merge-bot[bot]: your |
|
@jparrill: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
1 similar comment
|
@jparrill: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
… API
Replace static ci-operator release imports (n1minor through n4minor) with
runtime resolution via the release controller REST API. This eliminates
concurrent import pods that caused race conditions in ci-tools, failing
~40-45% of e2e runs before tests executed.
Changes:
- Remove n1minor..n4minor and unused latest-418/419/420 release definitions
- Replace OCP_IMAGE_NX dependencies with NODEPOOL_NX_VERSION env vars
- Add hypershift-resolve-nodepool-releases step to all 8 e2e workflows
- Add nounset-safe variable expansion (${OCP_IMAGE_NX:-}) in run-e2e scripts
to prevent crashes when variables are unset
- Improve resolve script with retry logic, HTTP status checks, and hard
failure on resolution errors
Fixes the permafail introduced by openshift#79627 where unguarded variable expansion
under set -o nounset crashed run-e2e scripts before tests could execute.
Jira: CNTRLPLANE-3509
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Juan Manuel Parrilla Madrid <jparrill@redhat.com>
0b93c2f to
8a3b0cb
Compare
|
[REHEARSALNOTIFIER]
A total of 604 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs. A full list of affected jobs can be found here Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
/pj-rehearse pull-ci-openshift-hypershift-main-e2e-aws |
|
/pj-rehearse pull-ci-openshift-hypershift-main-e2e-aks |
|
/pj-rehearse pull-ci-openshift-hypershift-main-e2e-aws-4-22 |
|
@jparrill: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
1 similar comment
|
@jparrill: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse pull-ci-openshift-hypershift-main-e2e-aks-4-22 |
|
@jparrill: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
1 similar comment
|
@jparrill: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: csrwng, jparrill The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/pj-rehearse ack |
|
@jparrill: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
openshift-merge-bot
Bot
added
the
rehearsals-ack
|
Errors non related with the PR. |
|
@jparrill: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
76ba986
into
openshift:main
… API (openshift#80360) Replace static ci-operator release imports (n1minor through n4minor) with runtime resolution via the release controller REST API. This eliminates concurrent import pods that caused race conditions in ci-tools, failing ~40-45% of e2e runs before tests executed. Changes: - Remove n1minor..n4minor and unused latest-418/419/420 release definitions - Replace OCP_IMAGE_NX dependencies with NODEPOOL_NX_VERSION env vars - Add hypershift-resolve-nodepool-releases step to all 8 e2e workflows - Add nounset-safe variable expansion (${OCP_IMAGE_NX:-}) in run-e2e scripts to prevent crashes when variables are unset - Improve resolve script with retry logic, HTTP status checks, and hard failure on resolution errors Fixes the permafail introduced by openshift#79627 where unguarded variable expansion under set -o nounset crashed run-e2e scripts before tests could execute. Jira: CNTRLPLANE-3509 Signed-off-by: Juan Manuel Parrilla Madrid <jparrill@redhat.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… API (openshift#80360) Replace static ci-operator release imports (n1minor through n4minor) with runtime resolution via the release controller REST API. This eliminates concurrent import pods that caused race conditions in ci-tools, failing ~40-45% of e2e runs before tests executed. Changes: - Remove n1minor..n4minor and unused latest-418/419/420 release definitions - Replace OCP_IMAGE_NX dependencies with NODEPOOL_NX_VERSION env vars - Add hypershift-resolve-nodepool-releases step to all 8 e2e workflows - Add nounset-safe variable expansion (${OCP_IMAGE_NX:-}) in run-e2e scripts to prevent crashes when variables are unset - Improve resolve script with retry logic, HTTP status checks, and hard failure on resolution errors Fixes the permafail introduced by openshift#79627 where unguarded variable expansion under set -o nounset crashed run-e2e scripts before tests could execute. Jira: CNTRLPLANE-3509 Signed-off-by: Juan Manuel Parrilla Madrid <jparrill@redhat.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… API (openshift#80360) Replace static ci-operator release imports (n1minor through n4minor) with runtime resolution via the release controller REST API. This eliminates concurrent import pods that caused race conditions in ci-tools, failing ~40-45% of e2e runs before tests executed. Changes: - Remove n1minor..n4minor and unused latest-418/419/420 release definitions - Replace OCP_IMAGE_NX dependencies with NODEPOOL_NX_VERSION env vars - Add hypershift-resolve-nodepool-releases step to all 8 e2e workflows - Add nounset-safe variable expansion (${OCP_IMAGE_NX:-}) in run-e2e scripts to prevent crashes when variables are unset - Improve resolve script with retry logic, HTTP status checks, and hard failure on resolution errors Fixes the permafail introduced by openshift#79627 where unguarded variable expansion under set -o nounset crashed run-e2e scripts before tests could execute. Jira: CNTRLPLANE-3509 Signed-off-by: Juan Manuel Parrilla Madrid <jparrill@redhat.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… API (openshift#80360) Replace static ci-operator release imports (n1minor through n4minor) with runtime resolution via the release controller REST API. This eliminates concurrent import pods that caused race conditions in ci-tools, failing ~40-45% of e2e runs before tests executed. Changes: - Remove n1minor..n4minor and unused latest-418/419/420 release definitions - Replace OCP_IMAGE_NX dependencies with NODEPOOL_NX_VERSION env vars - Add hypershift-resolve-nodepool-releases step to all 8 e2e workflows - Add nounset-safe variable expansion (${OCP_IMAGE_NX:-}) in run-e2e scripts to prevent crashes when variables are unset - Improve resolve script with retry logic, HTTP status checks, and hard failure on resolution errors Fixes the permafail introduced by openshift#79627 where unguarded variable expansion under set -o nounset crashed run-e2e scripts before tests could execute. Jira: CNTRLPLANE-3509 Signed-off-by: Juan Manuel Parrilla Madrid <jparrill@redhat.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… API (openshift#80360) Replace static ci-operator release imports (n1minor through n4minor) with runtime resolution via the release controller REST API. This eliminates concurrent import pods that caused race conditions in ci-tools, failing ~40-45% of e2e runs before tests executed. Changes: - Remove n1minor..n4minor and unused latest-418/419/420 release definitions - Replace OCP_IMAGE_NX dependencies with NODEPOOL_NX_VERSION env vars - Add hypershift-resolve-nodepool-releases step to all 8 e2e workflows - Add nounset-safe variable expansion (${OCP_IMAGE_NX:-}) in run-e2e scripts to prevent crashes when variables are unset - Improve resolve script with retry logic, HTTP status checks, and hard failure on resolution errors Fixes the permafail introduced by openshift#79627 where unguarded variable expansion under set -o nounset crashed run-e2e scripts before tests could execute. Jira: CNTRLPLANE-3509 Signed-off-by: Juan Manuel Parrilla Madrid <jparrill@redhat.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… API (openshift#80360) Replace static ci-operator release imports (n1minor through n4minor) with runtime resolution via the release controller REST API. This eliminates concurrent import pods that caused race conditions in ci-tools, failing ~40-45% of e2e runs before tests executed. Changes: - Remove n1minor..n4minor and unused latest-418/419/420 release definitions - Replace OCP_IMAGE_NX dependencies with NODEPOOL_NX_VERSION env vars - Add hypershift-resolve-nodepool-releases step to all 8 e2e workflows - Add nounset-safe variable expansion (${OCP_IMAGE_NX:-}) in run-e2e scripts to prevent crashes when variables are unset - Improve resolve script with retry logic, HTTP status checks, and hard failure on resolution errors Fixes the permafail introduced by openshift#79627 where unguarded variable expansion under set -o nounset crashed run-e2e scripts before tests could execute. Jira: CNTRLPLANE-3509 Signed-off-by: Juan Manuel Parrilla Madrid <jparrill@redhat.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
![openshift-prow[bot]](https://github.githubassets.com/images/gravatars/gravatar-user-420.png){kind=avatar}
… API (openshift#80360) Replace static ci-operator release imports (n1minor through n4minor) with runtime resolution via the release controller REST API. This eliminates concurrent import pods that caused race conditions in ci-tools, failing ~40-45% of e2e runs before tests executed. Changes: - Remove n1minor..n4minor and unused latest-418/419/420 release definitions - Replace OCP_IMAGE_NX dependencies with NODEPOOL_NX_VERSION env vars - Add hypershift-resolve-nodepool-releases step to all 8 e2e workflows - Add nounset-safe variable expansion (${OCP_IMAGE_NX:-}) in run-e2e scripts to prevent crashes when variables are unset - Improve resolve script with retry logic, HTTP status checks, and hard failure on resolution errors Fixes the permafail introduced by openshift#79627 where unguarded variable expansion under set -o nounset crashed run-e2e scripts before tests could execute. Jira: CNTRLPLANE-3509 Signed-off-by: Juan Manuel Parrilla Madrid <jparrill@redhat.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… API (openshift#80360) Replace static ci-operator release imports (n1minor through n4minor) with runtime resolution via the release controller REST API. This eliminates concurrent import pods that caused race conditions in ci-tools, failing ~40-45% of e2e runs before tests executed. Changes: - Remove n1minor..n4minor and unused latest-418/419/420 release definitions - Replace OCP_IMAGE_NX dependencies with NODEPOOL_NX_VERSION env vars - Add hypershift-resolve-nodepool-releases step to all 8 e2e workflows - Add nounset-safe variable expansion (${OCP_IMAGE_NX:-}) in run-e2e scripts to prevent crashes when variables are unset - Improve resolve script with retry logic, HTTP status checks, and hard failure on resolution errors Fixes the permafail introduced by openshift#79627 where unguarded variable expansion under set -o nounset crashed run-e2e scripts before tests could execute. Jira: CNTRLPLANE-3509 Signed-off-by: Juan Manuel Parrilla Madrid <jparrill@redhat.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
4 participants
Summary
n1minorthroughn4minor) with runtime resolution via the release controller REST APIset -o nounsetcrashed run-e2e scripts before tests could executelatest-418/419/420release definitions that were left orphaned after removing NX dependencies from 4.22 jobsNODEPOOL_NX_VERSIONenv vars frome2e-aws-autonode(Karpenter-only job, does not run N-minor tests)Context
${OCP_IMAGE_NX}expansionWhat changed vs #79627
${OCP_IMAGE_NX:-}with-ncheck instead of bare${OCP_IMAGE_NX}— prevents crash when resolve step fails or variable is unsetoc image infobefore accepting it. This catches stale/garbage-collected CI images.nightlystream's latest Accepted release.latest-418,latest-419,latest-420were no longer referenced after removing NX dependencies from 4.22 jobse2e-aws-autonodeonly runs^TestKarpenterwhich does not use N-minor release imagesRCA: Why e2e-*-4-22 jobs were failing (even after revert)
The CI release streams for OCP 4.19, 4.20, and 4.21 have not produced an
Acceptedbuild in 4-8 days. The old ci-operator import mechanism (release:latest-NXX) fails hard when any component image within any release payload is missing from the registry — killing the entire job with zero test coverage. Evidence from PR #8673 (post-revert) confirms identicalmanifest unknownfailures via the old mechanism.The runtime-resolve mechanism is strictly more resilient: a stale image only affects the specific test that uses it, and with the nightly fallback, even stale CI images are recovered automatically.
Full RCA: rca-e2e-4-22-failures.md
Changes
openshift-hypershift-main.yamln1minor..n4minorrelease definitionslatest-418/419/420release definitionsOCP_IMAGE_NXdependency overrides from all jobsNODEPOOL_N1_VERSION..NODEPOOL_N4_VERSIONenv vars to each job (except autonode)New:
hypershift-resolve-nodepool-releasesstepamd64.ocp.releases.ci.openshift.orgAPI with retry logic (3 attempts × 3 curl retries)oc image infowith CI pull credentialsnightlystream when CI stream images are unavailableexport OCP_IMAGE_NX="pullspec"to${SHARED_DIR}/nodepool_release_images8 workflow files
hypershift-resolve-nodepool-releasesstep to all e2e workflows (AWS nested/external/metrics/oidc, Azure AKS/AKS-OIDC/self-managed/v2-self-managed)4 run-e2e scripts + create-guests
source ${SHARED_DIR}/nodepool_release_imagesblock (guarded by file existence check)${OCP_IMAGE_NX}to nounset-safe${OCP_IMAGE_NX:-}expansionTest plan
/pj-rehearsefore2e-aws,e2e-aks,e2e-aws-4-22,e2e-aks-4-22Jira: https://issues.redhat.com/browse/CNTRLPLANE-3509