Skip to content

CNTRLPLANE-3619: add azure CPO overrides for 4.22.0-4.22.3 - #8908

Merged
celebdor merged 4 commits into
openshift:mainfrom
celebdor:cntrlplane-3619/cpo-overrides-4.22
Jul 6, 2026
Merged

CNTRLPLANE-3619: add azure CPO overrides for 4.22.0-4.22.3#8908
celebdor merged 4 commits into
openshift:mainfrom
celebdor:cntrlplane-3619/cpo-overrides-4.22

Conversation

@celebdor

@celebdor celebdor commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Add azure CPO image overrides for 4.22.0-4.22.3 to backport API-driven Azure topology/private connectivity (CNTRLPLANE-3619) and etcd data re-encryption (CNTRLPLANE-3656)
  • Improve verify-pr-in-image.sh: add AUTHFILE support and --no-tags to avoid skopeo timeouts on large repos like ocp-v4.0-art-dev
  • Add create-cpo-override skill with Product Pages MCP integration for development cutoff verification

branch: 4.22 wants: #8721, #8790

4.22.4 does not need an override: both PRs merged 2026-06-26, before the 4.22.4 development cutoff (2026-07-01).

Test plan

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Added a guided workflow for creating control plane override PRs, including automated image discovery, verification of required fixes, pullability checks, and PR preparation.
    • Added Azure override entries for OCP control plane 4.22 (4.22.0–4.22.3).
  • Documentation
    • Expanded contributor guidance with end-to-end override PR instructions and an image source priority reference.
    • Updated validation contract wording to point to the canonical “Validating Override Images Contain Claimed PRs” section.
  • Bug Fixes
    • Improved image verification to support authenticated registry access when credentials are available.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 2, 2026
@openshift-ci-robot

openshift-ci-robot commented Jul 2, 2026

Copy link
Copy Markdown

@celebdor: This pull request references CNTRLPLANE-3619 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target either version "5.0." or "openshift-5.0.", but it targets "openshift-4.22.z" instead.

Details

In response to this:

Summary

  • Add azure CPO image overrides for 4.22.0-4.22.3 to backport API-driven Azure topology/private connectivity (CNTRLPLANE-3619) and etcd data re-encryption (CNTRLPLANE-3656)
  • Improve verify-pr-in-image.sh: add AUTHFILE support and --no-tags to avoid skopeo timeouts on large repos like ocp-v4.0-art-dev
  • Add create-cpo-override skill with Product Pages MCP integration for development cutoff verification

branch: 4.22 wants: #8721, #8790

4.22.4 does not need an override: both PRs merged 2026-06-26, before the 4.22.4 development cutoff (2026-07-01).

Test plan

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci Bot added do-not-merge/needs-area area/ai Indicates the PR includes changes related to AI - Claude agents, Cursor rules, etc. labels Jul 2, 2026
@openshift-ci
openshift-ci Bot requested review from clebs and enxebre July 2, 2026 14:30
@openshift-ci openshift-ci Bot added area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release area/documentation Indicates the PR includes changes for documentation and removed do-not-merge/needs-area labels Jul 2, 2026
@coderabbitai

coderabbitai Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: d2384892-25e5-40a8-84e7-ccfabb0b966a

📥 Commits

Reviewing files that changed from the base of the PR and between 25770e8 and 7ffc7a3.

⛔ Files ignored due to path filters (1)
  • docs/content/reference/aggregated-docs.md is excluded by !docs/content/reference/aggregated-docs.md
📒 Files selected for processing (5)
  • .claude/skills/create-cpo-override/SKILL.md
  • .claude/skills/validate-pr-override-images/SKILL.md
  • .claude/skills/validate-pr-override-images/verify-pr-in-image.sh
  • docs/content/contribute/cpo-overrides.md
  • hypershift-operator/controlplaneoperator-overrides/assets/overrides.yaml
✅ Files skipped from review due to trivial changes (1)
  • .claude/skills/validate-pr-override-images/SKILL.md
🚧 Files skipped from review as they are similar to previous changes (2)
  • .claude/skills/validate-pr-override-images/verify-pr-in-image.sh
  • hypershift-operator/controlplaneoperator-overrides/assets/overrides.yaml

📝 Walkthrough

Walkthrough

This PR adds a new /create-cpo-override Claude Code skill for building CPO override entries, updates the override validation helper and its documentation, expands contributor guidance, and adds a new Azure 4.22 override block in overrides.yaml. The skill covers auth detection, input collection, image resolution, required-PR verification, pullability checks, YAML editing, test execution, and PR preparation. The validator script now supports optional authenticated skopeo inspect calls.

Sequence Diagram(s)

sequenceDiagram
  participant User
  participant Skill as create-cpo-override skill
  participant Cincinnati
  participant ProductPages as Product Pages MCP
  participant VerifyScript as verify-pr-in-image.sh
  participant Skopeo

  User->>Skill: invoke /create-cpo-override
  Skill->>Cincinnati: resolve z-stream ranges
  Skill->>ProductPages: fetch cutoff dates when available
  Skill->>VerifyScript: verify required PRs in image
  VerifyScript->>Skopeo: inspect image with optional authfile
  Skopeo-->>VerifyScript: image metadata
  VerifyScript-->>Skill: verification result
  Skill->>Skopeo: test image pullability
  Skopeo-->>Skill: pullability result
  Skill-->>User: present summary and PR text
Loading

Compact metadata

  • Files changed: 4
  • Lines changed: +463/-1
  • Estimated review effort: Medium

Related issues: None specified.

Related PRs: None specified.

Suggested labels: documentation, skills

Suggested reviewers: None specified.

Poem

A path of images, checks, and keys,
from payloads sought to YAML trees;
one skill to guide the PR in sight,
one script to probe the registry right.

🚥 Pre-merge checks | ✅ 11
✅ Passed checks (11 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: adding Azure CPO overrides for 4.22.0-4.22.3.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed Touched files are docs, a shell script, and YAML; no Ginkgo It/Describe/Context/When titles appear in them, so no unstable test names were introduced.
Test Structure And Quality ✅ Passed PASS: The changed Ginkgo e2e tests use BeforeEach/BeforeAll and DeferCleanup, have explicit Eventually timeouts, and the new CPO test is single-purpose with clear messages.
Topology-Aware Scheduling Compatibility ✅ Passed PR only adds docs/skills and CPO image override data; no pod specs, node selectors, affinity, tolerations, replicas, or scheduling logic were introduced.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed No new Ginkgo e2e tests were added; the PR changes are docs, a shell script, and override YAML.
No-Weak-Crypto ✅ Passed Touched files only add docs, YAML, and a skopeo helper; no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret comparisons found.
Container-Privileges ✅ Passed PASS: Changed manifests/scripts/docs contain no privileged:true, hostPID/hostNetwork/hostIPC, allowPrivilegeEscalation:true, SYS_ADMIN, or root settings.
No-Sensitive-Data-In-Logs ✅ Passed No new sensitive logging found: authfile is միայն passed to skopeo, and output logs only image refs/commit hashes/PR numbers.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Jul 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 43.28%. Comparing base (f69e734) to head (7ffc7a3).
⚠️ Report is 21 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #8908   +/-   ##
=======================================
  Coverage   43.28%   43.28%           
=======================================
  Files         771      771           
  Lines       95503    95503           
=======================================
  Hits        41335    41335           
  Misses      51284    51284           
  Partials     2884     2884           

see 1 file with indirect coverage changes

Flag Coverage Δ
cmd-support 36.67% <ø> (ø)
cpo-hostedcontrolplane 45.31% <ø> (ø)
cpo-other 45.10% <ø> (ø)
hypershift-operator 53.59% <ø> (ø)
other 31.69% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.claude/skills/create-cpo-override/SKILL.md:
- Around line 386-390: The PR description generation in the create-cpo-override
skill currently depends on reading the validate-pr-override-images skill, which
violates skill isolation. Update the logic around the PR description contract so
it embeds the required validation lines directly or reads them from a shared
non-skill source, and remove any reference to inspecting
`.claude/skills/validate-pr-override-images/SKILL.md` from the
create-cpo-override skill.

In @.claude/skills/validate-pr-override-images/verify-pr-in-image.sh:
- Around line 22-25: The authfile setup in verify-pr-in-image.sh only checks
AUTHFILE, so the workflow’s PULL_SECRET value is never used. Update the script’s
authfile handling (the AUTHFILE_ARGS block) to accept PULL_SECRET as well, or
normalize PULL_SECRET to AUTHFILE before this check, so the documented
pull-secret path is reachable.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 24c94d47-8240-453e-bf8c-c682c23e2f55

📥 Commits

Reviewing files that changed from the base of the PR and between 9aeb1f3 and f1ec48f.

⛔ Files ignored due to path filters (1)
  • docs/content/reference/aggregated-docs.md is excluded by !docs/content/reference/aggregated-docs.md
📒 Files selected for processing (4)
  • .claude/skills/create-cpo-override/SKILL.md
  • .claude/skills/validate-pr-override-images/verify-pr-in-image.sh
  • docs/content/contribute/cpo-overrides.md
  • hypershift-operator/controlplaneoperator-overrides/assets/overrides.yaml

Comment thread .claude/skills/create-cpo-override/SKILL.md Outdated
Comment thread .claude/skills/validate-pr-override-images/verify-pr-in-image.sh
@celebdor
celebdor force-pushed the cntrlplane-3619/cpo-overrides-4.22 branch from f1ec48f to 25770e8 Compare July 2, 2026 15:42
@csrwng

csrwng commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

/approve

@csrwng

csrwng commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci

openshift-ci Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: celebdor, csrwng

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 2, 2026
@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 2, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-22
/test e2e-aws-4-22

celebdor and others added 4 commits July 2, 2026 18:17
Interactive skill that automates CPO image override creation: resolves
images from stable/fast payloads or Konflux builds, verifies PRs are
included, edits overrides.yaml, and prepares PR descriptions compatible
with /validate-pr-override-images.

Includes development cutoff verification via the Product Pages MCP
server to prevent upgrade regressions when the override range ends
before the next z-stream ships.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…e repos

- Add AUTHFILE env var support for registries requiring authentication.
- Add --no-tags to skopeo inspect to fetch the manifest directly instead
  of enumerating every tag. Fixes timeouts on repos like ocp-v4.0-art-dev
  which have hundreds of thousands of tags.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
….0-4.22.3

Add azure CPO image overrides for 4.22 to backport API-driven Azure
topology and private connectivity (CNTRLPLANE-3619, PR openshift#8721) and etcd
data re-encryption after encryption key rotation (CNTRLPLANE-3656,
PR openshift#8790).

4.22.4 does not need an override: both PRs merged 2026-06-26, before
the 4.22.4 development cutoff (2026-07-01).

- 4.22.0-4.22.3: CNTRLPLANE-3619 (PR openshift#8721), CNTRLPLANE-3656 (PR openshift#8790)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@ahitacat

ahitacat commented Jul 3, 2026

Copy link
Copy Markdown

/verified by ahitacat

Deployed the new operator

oc get deploy operator -n hypershift -o json | jq '.spec.template.spec.containers[0].image'
"arohcpsvcdev.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-operator/hypershift-operator-main@sha256:56b0e46ce24f3a6ba1224e7ae042e0926b14ab44a26edb0e8eaf9274b7fdc015"

Created a cluster with ocp version "4.22.1" from the hosted cluster

    controlPlaneVersion:
      desired:
        image: arohcpocpdev.azurecr.io/openshift-release-dev/ocp-release@sha256:ef86b297204a9c2bce2fd1811772d0bad23a7f8411fe0e1bb54eae4c269e9217
        version: 4.22.1
      history:
      - completionTime: "2026-07-03T11:44:17Z"
        image: arohcpocpdev.azurecr.io/openshift-release-dev/ocp-release@sha256:ef86b297204a9c2bce2fd1811772d0bad23a7f8411fe0e1bb54eae4c269e9217
        startedTime: "2026-07-03T11:37:35Z"
        state: Completed
        version: 4.22.1
      observedGeneration: 1

The control plane operator shows the annotations

    - name: CONTROL_PLANE_OPERATOR_IMAGE
      value: quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-22@sha256:9584e156f938121c050c2373583998ec78068013d1a94d97913bea3e6416c427
    - name: HOSTED_CLUSTER_CONFIG_OPERATOR_IMAGE
      value: quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-22@sha256:9584e156f938121c050c2373583998ec78068013d1a94d97913bea3e6416c427
    - name: SOCKS5_PROXY_IMAGE
      value: quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-22@sha256:9584e156f938121c050c2373583998ec78068013d1a94d97913bea3e6416c427
    - name: AVAILABILITY_PROBER_IMAGE
      value: quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-22@sha256:9584e156f938121c050c2373583998ec78068013d1a94d97913bea3e6416c427
    - name: TOKEN_MINTER_IMAGE
      value: quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-22@sha256:9584e156f938121c050c2373583998ec78068013d1a94d97913bea3e6416c427

I have also tested the key rotation and worked.

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jul 3, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@ahitacat: This PR has been marked as verified by ahitacat.

Details

In response to this:

/verified by ahitacat

Deployed the new operator

oc get deploy operator -n hypershift -o json | jq '.spec.template.spec.containers[0].image'
"arohcpsvcdev.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-operator/hypershift-operator-main@sha256:56b0e46ce24f3a6ba1224e7ae042e0926b14ab44a26edb0e8eaf9274b7fdc015"

Created a cluster with ocp version "4.22.1" from the hosted cluster

   controlPlaneVersion:
     desired:
       image: arohcpocpdev.azurecr.io/openshift-release-dev/ocp-release@sha256:ef86b297204a9c2bce2fd1811772d0bad23a7f8411fe0e1bb54eae4c269e9217
       version: 4.22.1
     history:
     - completionTime: "2026-07-03T11:44:17Z"
       image: arohcpocpdev.azurecr.io/openshift-release-dev/ocp-release@sha256:ef86b297204a9c2bce2fd1811772d0bad23a7f8411fe0e1bb54eae4c269e9217
       startedTime: "2026-07-03T11:37:35Z"
       state: Completed
       version: 4.22.1
     observedGeneration: 1

The control plane operator shows the annotations

   - name: CONTROL_PLANE_OPERATOR_IMAGE
     value: quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-22@sha256:9584e156f938121c050c2373583998ec78068013d1a94d97913bea3e6416c427
   - name: HOSTED_CLUSTER_CONFIG_OPERATOR_IMAGE
     value: quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-22@sha256:9584e156f938121c050c2373583998ec78068013d1a94d97913bea3e6416c427
   - name: SOCKS5_PROXY_IMAGE
     value: quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-22@sha256:9584e156f938121c050c2373583998ec78068013d1a94d97913bea3e6416c427
   - name: AVAILABILITY_PROBER_IMAGE
     value: quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-22@sha256:9584e156f938121c050c2373583998ec78068013d1a94d97913bea3e6416c427
   - name: TOKEN_MINTER_IMAGE
     value: quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-22@sha256:9584e156f938121c050c2373583998ec78068013d1a94d97913bea3e6416c427

I have also tested the key rotation and worked.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@jparrill

jparrill commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

/test e2e-aks-override

@jparrill

jparrill commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

/test e2e-aws-override

@jparrill

jparrill commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 3, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-22
/test e2e-aws-4-22

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

/retest-required

Remaining retests: 0 against base HEAD 8e1aa48 and 2 for PR HEAD 7ffc7a3 in total

@openshift-ci

openshift-ci Bot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

@celebdor: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-override 7ffc7a3 link true /test e2e-aws-override
ci/prow/e2e-aks-override 7ffc7a3 link true /test e2e-aks-override

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@hypershift-jira-solve-ci

hypershift-jira-solve-ci Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

I now have everything needed for both reports. Both jobs share the TestAutoscalerRespectsNodePoolPause failure, and the AKS job has an additional EnsureGlobalPullSecret failure. Neither is related to PR #8908's changes (which only add Azure CPO override entries for 4.22.x in overrides.yaml).

Test Failure Analysis Complete

Job Information

  • Prow Job 1: pull-ci-openshift-hypershift-main-e2e-aks-override
  • Build ID 1: 2073201687610265600
  • Prow Job 2: pull-ci-openshift-hypershift-main-e2e-aws-override
  • Build ID 2: 2073201690122653696
  • PR: CNTRLPLANE-3619: add azure CPO overrides for 4.22.0-4.22.3 #8908CNTRLPLANE-3619: add azure CPO overrides for 4.22.0-4.22.3
  • PR Changes: Only modifies overrides.yaml (adds Azure CPO image entries for 4.22.0–4.22.3) and documentation/skill files. No test code or controller logic changes.

Test Failure Analysis

Error

[COMMON TO BOTH JOBS] TestAutoscaling/Main/TestAutoscalerRespectsNodePoolPause

  AKS (668.16s): MachineDeployment replicas should not change while paused —
    CAS should not decrement replicas on paused MachineDeployments (regression for OCPBUGS-78152)
    Expected <int32>: 1 to equal <int32>: 3

  AWS (300.07s): autoscaler deployment missing --scale-down-unneeded-time=60s arg
    context deadline exceeded after 5m0s

[AKS ONLY] TestCreateCluster/Main/EnsureGlobalPullSecret (30.17s):
  daemonsets.apps "global-pull-secret-syncer" not found

Summary

Both jobs fail on TestAutoscalerRespectsNodePoolPause — a regression test for OCPBUGS-78152 that validates the cluster-autoscaler (CAS) respects NodePool pause annotations. The test sets HostedCluster.Spec.Autoscaling.ScaleDown with aggressive timers (UnneededDurationSeconds=60, DelayAfterAddSeconds=30) and expects the CAS deployment to be reconciled with matching --scale-down-unneeded-time=60s and --scale-down-delay-after-add=30s args. On AWS, the CAS deployment never received these args within the 5-minute timeout, causing the test to fail before reaching the pause verification. On AKS, the args were applied but CAS scaled the MachineDeployment from 3 to 1 replicas while the NodePool was paused — the exact regression the test was designed to catch. The AKS job has an additional EnsureGlobalPullSecret failure where the global-pull-secret-syncer DaemonSet is not found in the 4.19.10 guest cluster, indicating the 4.19 CPO override image does not include this feature despite the test's AtLeast(Version419) gate. None of these failures are caused by PR #8908, which only adds CPO override image entries for 4.22.x versions in overrides.yaml.

Root Cause

Failure 1: TestAutoscalerRespectsNodePoolPause (both jobs)

Root cause: Pre-existing test flakiness / CAS reconciliation race condition — not caused by PR #8908.

The test (autoscaling_test.go, introduced in commit ec5bc72b56b on 2026-04-16) works as follows:

  1. Sets HostedCluster.Spec.Autoscaling.ScaleDown with aggressive timers
  2. Calls waitForAutoscalerDeploymentReady which polls the cluster-autoscaler Deployment for up to 5 minutes, checking that container args include --scale-down-unneeded-time=60s and --scale-down-delay-after-add=30s
  3. Creates workload to trigger scale-up to 3 replicas
  4. Pauses the NodePool and verifies replicas remain at 3

AWS failure mode: The control-plane-operator never reconciled the ScaleDown config into the CAS deployment args within 5 minutes. The CAS deployment was observed at resource version 112180 still missing the --scale-down-unneeded-time=60s arg after the full timeout. This is a reconciliation timing issue — the operator's loop did not pick up the Autoscaling.ScaleDown spec change fast enough.

AKS failure mode: The args were applied, but during the pause verification window, CAS scaled the MachineDeployment from 3 replicas down to 1. The CAS "discovered a paused node group" log was never detected, and the replica count dropped — indicating CAS acted on the MachineDeployment before the pause annotation was fully propagated, or the aggressive 60s UnneededDurationSeconds timer expired before the test's polling could capture the paused state.

Failure 2: EnsureGlobalPullSecret (AKS only)

Root cause: Test version gate mismatch — the global-pull-secret-syncer DaemonSet feature is not present in the 4.19 CPO override image.

The test uses AtLeast(t, Version419) as its version gate, but the 4.19 CPO Konflux override (quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-19@sha256:88c55ea...) does not deploy the global-pull-secret-syncer DaemonSet. The DaemonSet lookup at line 2008 of util.go times out after 30 seconds because the feature was either not backported to 4.19 or the override image predates the backport. This is a pre-existing test/version-gate issue, not caused by PR #8908.

Recommendations
  1. Re-run the jobs: Since both failures are pre-existing and unrelated to PR CNTRLPLANE-3619: add azure CPO overrides for 4.22.0-4.22.3 #8908's changes (which only modify overrides.yaml for 4.22.x CPO images), a /retest should be attempted. The TestAutoscalerRespectsNodePoolPause failure appears to be a timing-sensitive flake.

  2. File a bug for TestAutoscalerRespectsNodePoolPause flakiness: The 5-minute timeout for CAS deployment reconciliation (waitForAutoscalerDeploymentReady) may be insufficient under load. Consider:

    • Increasing the timeout from 5 minutes to 10 minutes
    • Adding a retry/backoff in the reconciliation check
    • Investigating why the CPO does not reconcile Autoscaling.ScaleDown args promptly
  3. Fix EnsureGlobalPullSecret version gate: The test's AtLeast(t, Version419) gate is too permissive — the global-pull-secret-syncer feature may not be present in all 4.19.z CPO overrides. Consider adding a CPOAtLeast gate or checking for feature availability before asserting the DaemonSet exists.

  4. Merge PR CNTRLPLANE-3619: add azure CPO overrides for 4.22.0-4.22.3 #8908: The PR's actual changes (Azure CPO override entries for 4.22.0–4.22.3) are valid and verified — the CPO override image verification passed (Successfully waited for control-plane-operator pod is running with expected override image). These test failures should not block the merge.

Evidence
Evidence Detail
PR #8908 changes Only modifies overrides.yaml (4.22.x Azure CPO images), docs, and skill files — no test or controller code
AKS CPO override verified util.go:4441: Successfully waited for control-plane-operator pod is running with expected override image (4.19 CPO)
AWS failed test TestAutoscalerRespectsNodePoolPauseautoscaler deployment missing --scale-down-unneeded-time=60s arg after 5m timeout
AKS failed test 1 TestAutoscalerRespectsNodePoolPause — replicas dropped from 3→1 while paused (OCPBUGS-78152 regression)
AKS failed test 2 EnsureGlobalPullSecretdaemonsets.apps "global-pull-secret-syncer" not found (30s timeout)
Test introduction TestAutoscalerRespectsNodePoolPause added in ec5bc72b56b (2026-04-16) for OCPBUGS-78152
AWS test stats 278 tests, 75 skipped, 3 failures (only TestAutoscaling tree)
AKS test stats 214 tests, 59 skipped, 6 failures (TestAutoscaling + TestCreateCluster trees)
CAS reconciliation code deployment.go:153-178ScaleDownArgs() maps API fields to CLI args
Test wait function autoscaling_test.go:389-417waitForAutoscalerDeploymentReady polls deployment args for 5min
Version gate issue EnsureGlobalPullSecret uses AtLeast(Version419) but 4.19 CPO override lacks the feature

@celebdor
celebdor merged commit 203ef79 into openshift:main Jul 6, 2026
34 of 37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/ai Indicates the PR includes changes related to AI - Claude agents, Cursor rules, etc. area/documentation Indicates the PR includes changes for documentation area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants