Switch OSAC CI from kustomize to helm-based refresh - #80807
Merged
openshift-merge-bot[bot] merged 7 commits intoJun 21, 2026
Merged
openshift-merge-bot[bot] merged 7 commits into
openshift-merge-bot[bot] merged 7 commits into
Conversation
Replace the kustomize-based refresh flow with the Python helm refresh script (refresh-after-snapshot.py). The boot step now uses the vmaas-helm snapshot flavor and deploys via helm upgrade instead of oc apply -k. Changes: - Boot ref: E2E_VALUES_FILE replaces E2E_KUSTOMIZE_OVERLAY, flavor defaults to vmaas-helm - Boot script: license mounts to values dir, component overrides use sed on helm values YAML (handles both image:tag and split repository/tag formats), refresh runs python3 -u refresh-after-snapshot.py - osac-installer config: add values/ to run_if_changed trigger - osac-test-infra config: bump vmaas periodic to hourly, drop kustomize full-setup job Depends on osac-project/osac-installer#296 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
openshift-ci
Bot
added
the
approved
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThe PR migrates vmaas cluster boot from kustomize overlays to Helm values files. Step configuration adds Changesvmaas Helm Values Migration and Testing
Sequence DiagramsequenceDiagram
participant BootStep as Boot Step
participant Remote as Remote Boot Script
participant ValuesYAML as Values YAML File
participant Verify as Verification (grep)
participant ClusterTool as cluster-tool boot
participant Container as Refresh Container
participant Python as refresh-after-snapshot.py
BootStep->>BootStep: Set CLONE_NAME=vmaas-helm<br/>Export E2E_VALUES_FILE
BootStep->>Remote: SSH with VALUES_FILE,<br/>E2E_VALUES_FILE, E2E_VALUES_DIR
Remote->>Remote: Derive VALUES_DIR from VALUES_FILE
Remote->>ValuesYAML: sed: rewrite component images<br/>(image:tag and repository/tag forms)
Remote->>Verify: grep: verify component image changed
Verify-->>Remote: match found or exit 1
Remote->>ValuesYAML: sed: rewrite AAP config<br/>(projectGitBranch, eeImage)
Remote->>Verify: grep: verify AAP_SOURCE_SHA
Verify-->>Remote: match found or exit 1
Remote->>ClusterTool: cluster-tool boot --flavor vmaas-helm
BootStep->>Container: Mount license to<br/>VALUES_DIR/license.zip
BootStep->>Container: Set env: VALUES_FILE,<br/>INSTALLER_VM_TEMPLATE,<br/>INSTALLER_NAMESPACE
Container->>Python: python3 scripts/refresh-after-snapshot.py
Python->>ValuesYAML: Read updated values
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Suggested labels
Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error)
✅ Passed checks (14 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/osac-project/cluster-tool/boot/osac-project-cluster-tool-boot-commands.sh`:
- Line 18: The CLONE_NAME variable in the boot step is set to "vmaas-helm" but
the corresponding test and destroy steps in the workflow use "vmaas-kustomize",
causing a mismatch where the test targets the wrong clone and the boot-created
clone is never cleaned up. Change the CLONE_NAME variable in
osac-project-cluster-tool-boot-commands.sh from "vmaas-helm" to
"vmaas-kustomize" to match the clone name used in the test and destroy workflow
steps, ensuring all three phases operate on the same clone throughout the
workflow chain.
- Around line 195-201: The split repository/tag override in the
COMPONENT_OVERRIDE_CMD at line 201 only updates the tag field but does not
update the repository field itself. When COMPONENT_IMAGE points to a different
registry or repository, the repository value remains unchanged, resulting in an
incomplete override. Extract the repository portion of COMPONENT_IMAGE
(everything before the colon) by using parameter expansion similar to how
COMPONENT_TAG is extracted, then add an additional sed command to the
COMPONENT_OVERRIDE_CMD that updates the repository field to this new
registry/path value alongside the existing tag update. Ensure both the
repository and tag fields are updated when handling the split format
configuration.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 29303d4c-3b0d-4930-b458-467397d22eae
📒 Files selected for processing (4)
ci-operator/config/osac-project/osac-installer/osac-project-osac-installer-main.yamlci-operator/config/osac-project/osac-test-infra/osac-project-osac-test-infra-main.yamlci-operator/step-registry/osac-project/cluster-tool/boot/osac-project-cluster-tool-boot-commands.shci-operator/step-registry/osac-project/cluster-tool/boot/osac-project-cluster-tool-boot-ref.yaml
| echo "-------------------------------------------" | ||
|
|
||
| CLONE_NAME="vmaas-kustomize" | ||
| CLONE_NAME="vmaas-helm" |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "=== Clone names in osac cluster-tool scripts ==="
rg -n -C2 '^\s*CLONE_NAME=' ci-operator/step-registry/osac-project/cluster-tool/**/osac-project-cluster-tool-*-commands.sh
echo
echo "=== Workflow/chain wiring for boot/destroy ==="
rg -n -C3 'osac-project-cluster-tool-(boot|destroy)' ci-operator/step-registry/osac-project/cluster-tool -g '*workflow.yaml' -g '*chain.yaml'Repository: openshift/release
Length of output: 3981
Fix critical clone-name mismatch across boot/test/destroy workflow steps.
The workflow osac-project-cluster-tool-vmaas-workflow.yaml chains boot, test, and destroy in sequence, but they use mismatched clone names:
- Boot (pre phase): Creates
vmaas-helm - Test (test phase): Expects
vmaas-kustomize - Destroy (post phase): Destroys
vmaas-kustomize
This causes test operations to target the wrong clone and leaves vmaas-helm orphaned without cleanup. All three steps must use the same clone name, or the workflow will fail.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/step-registry/osac-project/cluster-tool/boot/osac-project-cluster-tool-boot-commands.sh`
at line 18, The CLONE_NAME variable in the boot step is set to "vmaas-helm" but
the corresponding test and destroy steps in the workflow use "vmaas-kustomize",
causing a mismatch where the test targets the wrong clone and the boot-created
clone is never cleaned up. Change the CLONE_NAME variable in
osac-project-cluster-tool-boot-commands.sh from "vmaas-helm" to
"vmaas-kustomize" to match the clone name used in the test and destroy workflow
steps, ensuring all three phases operate on the same clone throughout the
workflow chain.
- Update CLONE_NAME to vmaas-helm in test and destroy steps to match the boot step — prevents kubeconfig-not-found and VM leak - Fix split repository/tag component override to also replace the repository line (not just the tag) so osac-operator PRs get the correct CI registry image - Restore grep -q verification after AAP sed overrides to fail fast if the field name changes in the values file Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fail fast if the sed replacement didn't match anything in the values file, instead of silently testing against the pinned image. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
Author
|
/pj-rehearse pull-ci-osac-project-osac-installer-main-e2e-vmaas pull-ci-osac-project-osac-aap-main-e2e-vmaas pull-ci-osac-project-fulfillment-service-main-e2e-vmaas pull-ci-osac-project-osac-operator-main-e2e-vmaas pull-ci-osac-project-osac-test-infra-main-e2e-vmaas periodic-ci-osac-project-osac-test-infra-main-e2e-vmaas-periodic |
Contributor
|
@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
Collect LVMS/topolvm state (pods, events, logs, LVMCluster, VolumeAttachments), MachineConfig status and pull-secret registry list, CDI importer pod logs from subnet namespaces, and service account imagePullSecrets state. These were missing when debugging CI failures involving volume mount timeouts and image pull errors. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
Author
|
/pj-rehearse pull-ci-osac-project-osac-installer-main-e2e-vmaas pull-ci-osac-project-osac-aap-main-e2e-vmaas pull-ci-osac-project-fulfillment-service-main-e2e-vmaas pull-ci-osac-project-osac-operator-main-e2e-vmaas pull-ci-osac-project-osac-test-infra-main-e2e-vmaas periodic-ci-osac-project-osac-test-infra-main-e2e-vmaas-periodic |
Contributor
|
@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/osac-project/gather/osac-project-gather-commands.sh`:
- Around line 134-136: The oc logs command in the pod logging loop fails
silently on multi-container pods because the || true operator masks errors and
the command only captures logs from a single container per pod. Add the
--all-containers flag to the oc logs command that processes "${pod}" in the
openshift-storage namespace loop to ensure all containers within each pod have
their logs captured and written to the ARTIFACT_DIR without missing any
diagnostic information.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: bb5a6f66-f8ca-4b3b-87af-ad0be7fc5275
⛔ Files ignored due to path filters (2)
ci-operator/jobs/osac-project/osac-installer/osac-project-osac-installer-main-presubmits.yamlis excluded by!ci-operator/jobs/**ci-operator/jobs/osac-project/osac-test-infra/osac-project-osac-test-infra-main-periodics.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (1)
ci-operator/step-registry/osac-project/gather/osac-project-gather-commands.sh
Comment on lines
+134
to
+136
| for pod in $(oc get pods -n openshift-storage -o jsonpath='{.items[*].metadata.name}' 2>/dev/null); do | ||
| oc logs "${pod}" -n openshift-storage > "${ARTIFACT_DIR}/storage/pod-${pod}.log" 2>&1 || true | ||
| done |
Contributor
There was a problem hiding this comment.
Capture all storage pod containers to avoid silent log gaps.
At Line 135, oc logs "${pod}" -n openshift-storage can fail on multi-container pods, and || true masks it. Use --all-containers (or iterate containers) so diagnostics are complete.
Suggested patch
for pod in $(oc get pods -n openshift-storage -o jsonpath='{.items[*].metadata.name}' 2>/dev/null); do
- oc logs "${pod}" -n openshift-storage > "${ARTIFACT_DIR}/storage/pod-${pod}.log" 2>&1 || true
+ oc logs "${pod}" -n openshift-storage --all-containers > "${ARTIFACT_DIR}/storage/pod-${pod}.log" 2>&1 || true
done📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| for pod in $(oc get pods -n openshift-storage -o jsonpath='{.items[*].metadata.name}' 2>/dev/null); do | |
| oc logs "${pod}" -n openshift-storage > "${ARTIFACT_DIR}/storage/pod-${pod}.log" 2>&1 || true | |
| done | |
| for pod in $(oc get pods -n openshift-storage -o jsonpath='{.items[*].metadata.name}' 2>/dev/null); do | |
| oc logs "${pod}" -n openshift-storage --all-containers > "${ARTIFACT_DIR}/storage/pod-${pod}.log" 2>&1 || true | |
| done |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/step-registry/osac-project/gather/osac-project-gather-commands.sh`
around lines 134 - 136, The oc logs command in the pod logging loop fails
silently on multi-container pods because the || true operator masks errors and
the command only captures logs from a single container per pod. Add the
--all-containers flag to the oc logs command that processes "${pod}" in the
openshift-storage namespace loop to ensure all containers within each pod have
their logs captured and written to the ARTIFACT_DIR without missing any
diagnostic information.
Contributor
Author
|
/pj-rehearse |
Contributor
|
@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
Contributor
Author
|
/pj-rehearse more pull-ci-osac-project-osac-installer-main-e2e-vmaas pull-ci-osac-project-osac-aap-main-e2e-vmaas pull-ci-osac-project-fulfillment-service-main-e2e-vmaas pull-ci-osac-project-osac-operator-main-e2e-vmaas pull-ci-osac-project-osac-test-infra-main-e2e-vmaas periodic-ci-osac-project-osac-test-infra-main-e2e-vmaas-periodic |
Contributor
|
@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
Contributor
|
@omer-vishlitzky: job(s): more either don't exist or were not found to be affected, and cannot be rehearsed |
Contributor
|
@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
The old kustomize flow created a quay-pull-secret in the OSAC namespace via secretGenerator, giving AAP job pods credentials to pull CI-built EE images. The helm migration dropped this mount, causing intermittent ErrImagePull when the MCO hasn't propagated the global pull secret to CRI-O before AAP launches automation jobs. Re-add the pull secret as a volume mount at /installer/pull-secret.json so refresh-after-snapshot.py can create the namespace-level secret. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
omer-vishlitzky
force-pushed
the
osac-use-helm-refresh
branch
from
June 21, 2026 09:45
640b416 to
18108ec
Compare
Merged
1 task
Contributor
Author
|
/pj-rehearse more |
Contributor
|
@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
Author
|
/pj-rehearse ack |
Contributor
|
@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
openshift-merge-bot
Bot
added
the
rehearsals-ack
openshift-merge-bot
Bot
removed
the
rehearsals-ack
Contributor
|
[REHEARSALNOTIFIER]
Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
danmanor
approved these changes
Jun 21, 2026
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: danmanor, omer-vishlitzky The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Contributor
Author
|
/pj-rehearse ack |
Contributor
|
@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
openshift-merge-bot
Bot
added
the
rehearsals-ack
This was referenced Jun 22, 2026
krisnababu
pushed a commit
to krisnababu/release
that referenced
this pull request
Jun 29, 2026
* Switch OSAC CI from kustomize to helm-based refresh Replace the kustomize-based refresh flow with the Python helm refresh script (refresh-after-snapshot.py). The boot step now uses the vmaas-helm snapshot flavor and deploys via helm upgrade instead of oc apply -k. Changes: - Boot ref: E2E_VALUES_FILE replaces E2E_KUSTOMIZE_OVERLAY, flavor defaults to vmaas-helm - Boot script: license mounts to values dir, component overrides use sed on helm values YAML (handles both image:tag and split repository/tag formats), refresh runs python3 -u refresh-after-snapshot.py - osac-installer config: add values/ to run_if_changed trigger - osac-test-infra config: bump vmaas periodic to hourly, drop kustomize full-setup job Depends on osac-project/osac-installer#296 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix clone name mismatch, operator override, and AAP verification - Update CLONE_NAME to vmaas-helm in test and destroy steps to match the boot step — prevents kubeconfig-not-found and VM leak - Fix split repository/tag component override to also replace the repository line (not just the tag) so osac-operator PRs get the correct CI registry image - Restore grep -q verification after AAP sed overrides to fail fast if the field name changes in the values file Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add grep verification for component image override Fail fast if the sed replacement didn't match anything in the values file, instead of silently testing against the pinned image. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * make jobs * Add storage, MCO, and subnet pod diagnostics to gather script Collect LVMS/topolvm state (pods, events, logs, LVMCluster, VolumeAttachments), MachineConfig status and pull-secret registry list, CDI importer pod logs from subnet namespaces, and service account imagePullSecrets state. These were missing when debugging CI failures involving volume mount timeouts and image pull errors. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Mount pull secret into installer container for namespace secret creation The old kustomize flow created a quay-pull-secret in the OSAC namespace via secretGenerator, giving AAP job pods credentials to pull CI-built EE images. The helm migration dropped this mount, causing intermittent ErrImagePull when the MCO hasn't propagated the global pull secret to CRI-O before AAP launches automation jobs. Re-add the pull secret as a volume mount at /installer/pull-secret.json so refresh-after-snapshot.py can create the namespace-level secret. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Increase helm full install periodic frequency to every 4 hours Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
krisnababu
pushed a commit
to oharan2/release
that referenced
this pull request
Jul 3, 2026
* Switch OSAC CI from kustomize to helm-based refresh Replace the kustomize-based refresh flow with the Python helm refresh script (refresh-after-snapshot.py). The boot step now uses the vmaas-helm snapshot flavor and deploys via helm upgrade instead of oc apply -k. Changes: - Boot ref: E2E_VALUES_FILE replaces E2E_KUSTOMIZE_OVERLAY, flavor defaults to vmaas-helm - Boot script: license mounts to values dir, component overrides use sed on helm values YAML (handles both image:tag and split repository/tag formats), refresh runs python3 -u refresh-after-snapshot.py - osac-installer config: add values/ to run_if_changed trigger - osac-test-infra config: bump vmaas periodic to hourly, drop kustomize full-setup job Depends on osac-project/osac-installer#296 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix clone name mismatch, operator override, and AAP verification - Update CLONE_NAME to vmaas-helm in test and destroy steps to match the boot step — prevents kubeconfig-not-found and VM leak - Fix split repository/tag component override to also replace the repository line (not just the tag) so osac-operator PRs get the correct CI registry image - Restore grep -q verification after AAP sed overrides to fail fast if the field name changes in the values file Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add grep verification for component image override Fail fast if the sed replacement didn't match anything in the values file, instead of silently testing against the pinned image. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * make jobs * Add storage, MCO, and subnet pod diagnostics to gather script Collect LVMS/topolvm state (pods, events, logs, LVMCluster, VolumeAttachments), MachineConfig status and pull-secret registry list, CDI importer pod logs from subnet namespaces, and service account imagePullSecrets state. These were missing when debugging CI failures involving volume mount timeouts and image pull errors. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Mount pull secret into installer container for namespace secret creation The old kustomize flow created a quay-pull-secret in the OSAC namespace via secretGenerator, giving AAP job pods credentials to pull CI-built EE images. The helm migration dropped this mount, causing intermittent ErrImagePull when the MCO hasn't propagated the global pull secret to CRI-O before AAP launches automation jobs. Re-add the pull secret as a volume mount at /installer/pull-secret.json so refresh-after-snapshot.py can create the namespace-level secret. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Increase helm full install periodic frequency to every 4 hours Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
andrej1991
pushed a commit
to andrej1991/release
that referenced
this pull request
Jul 6, 2026
* Switch OSAC CI from kustomize to helm-based refresh Replace the kustomize-based refresh flow with the Python helm refresh script (refresh-after-snapshot.py). The boot step now uses the vmaas-helm snapshot flavor and deploys via helm upgrade instead of oc apply -k. Changes: - Boot ref: E2E_VALUES_FILE replaces E2E_KUSTOMIZE_OVERLAY, flavor defaults to vmaas-helm - Boot script: license mounts to values dir, component overrides use sed on helm values YAML (handles both image:tag and split repository/tag formats), refresh runs python3 -u refresh-after-snapshot.py - osac-installer config: add values/ to run_if_changed trigger - osac-test-infra config: bump vmaas periodic to hourly, drop kustomize full-setup job Depends on osac-project/osac-installer#296 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix clone name mismatch, operator override, and AAP verification - Update CLONE_NAME to vmaas-helm in test and destroy steps to match the boot step — prevents kubeconfig-not-found and VM leak - Fix split repository/tag component override to also replace the repository line (not just the tag) so osac-operator PRs get the correct CI registry image - Restore grep -q verification after AAP sed overrides to fail fast if the field name changes in the values file Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add grep verification for component image override Fail fast if the sed replacement didn't match anything in the values file, instead of silently testing against the pinned image. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * make jobs * Add storage, MCO, and subnet pod diagnostics to gather script Collect LVMS/topolvm state (pods, events, logs, LVMCluster, VolumeAttachments), MachineConfig status and pull-secret registry list, CDI importer pod logs from subnet namespaces, and service account imagePullSecrets state. These were missing when debugging CI failures involving volume mount timeouts and image pull errors. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Mount pull secret into installer container for namespace secret creation The old kustomize flow created a quay-pull-secret in the OSAC namespace via secretGenerator, giving AAP job pods credentials to pull CI-built EE images. The helm migration dropped this mount, causing intermittent ErrImagePull when the MCO hasn't propagated the global pull secret to CRI-O before AAP launches automation jobs. Re-add the pull secret as a volume mount at /installer/pull-secret.json so refresh-after-snapshot.py can create the namespace-level secret. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Increase helm full install periodic frequency to every 4 hours Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
TimurMP
pushed a commit
to TimurMP/release
that referenced
this pull request
Jul 8, 2026
* Switch OSAC CI from kustomize to helm-based refresh Replace the kustomize-based refresh flow with the Python helm refresh script (refresh-after-snapshot.py). The boot step now uses the vmaas-helm snapshot flavor and deploys via helm upgrade instead of oc apply -k. Changes: - Boot ref: E2E_VALUES_FILE replaces E2E_KUSTOMIZE_OVERLAY, flavor defaults to vmaas-helm - Boot script: license mounts to values dir, component overrides use sed on helm values YAML (handles both image:tag and split repository/tag formats), refresh runs python3 -u refresh-after-snapshot.py - osac-installer config: add values/ to run_if_changed trigger - osac-test-infra config: bump vmaas periodic to hourly, drop kustomize full-setup job Depends on osac-project/osac-installer#296 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix clone name mismatch, operator override, and AAP verification - Update CLONE_NAME to vmaas-helm in test and destroy steps to match the boot step — prevents kubeconfig-not-found and VM leak - Fix split repository/tag component override to also replace the repository line (not just the tag) so osac-operator PRs get the correct CI registry image - Restore grep -q verification after AAP sed overrides to fail fast if the field name changes in the values file Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add grep verification for component image override Fail fast if the sed replacement didn't match anything in the values file, instead of silently testing against the pinned image. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * make jobs * Add storage, MCO, and subnet pod diagnostics to gather script Collect LVMS/topolvm state (pods, events, logs, LVMCluster, VolumeAttachments), MachineConfig status and pull-secret registry list, CDI importer pod logs from subnet namespaces, and service account imagePullSecrets state. These were missing when debugging CI failures involving volume mount timeouts and image pull errors. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Mount pull secret into installer container for namespace secret creation The old kustomize flow created a quay-pull-secret in the OSAC namespace via secretGenerator, giving AAP job pods credentials to pull CI-built EE images. The helm migration dropped this mount, causing intermittent ErrImagePull when the MCO hasn't propagated the global pull secret to CRI-O before AAP launches automation jobs. Re-add the pull secret as a volume mount at /installer/pull-secret.json so refresh-after-snapshot.py can create the namespace-level secret. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Increase helm full install periodic frequency to every 4 hours Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
SachinNinganure
pushed a commit
to SachinNinganure/release
that referenced
this pull request
Jul 20, 2026
* Switch OSAC CI from kustomize to helm-based refresh Replace the kustomize-based refresh flow with the Python helm refresh script (refresh-after-snapshot.py). The boot step now uses the vmaas-helm snapshot flavor and deploys via helm upgrade instead of oc apply -k. Changes: - Boot ref: E2E_VALUES_FILE replaces E2E_KUSTOMIZE_OVERLAY, flavor defaults to vmaas-helm - Boot script: license mounts to values dir, component overrides use sed on helm values YAML (handles both image:tag and split repository/tag formats), refresh runs python3 -u refresh-after-snapshot.py - osac-installer config: add values/ to run_if_changed trigger - osac-test-infra config: bump vmaas periodic to hourly, drop kustomize full-setup job Depends on osac-project/osac-installer#296 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix clone name mismatch, operator override, and AAP verification - Update CLONE_NAME to vmaas-helm in test and destroy steps to match the boot step — prevents kubeconfig-not-found and VM leak - Fix split repository/tag component override to also replace the repository line (not just the tag) so osac-operator PRs get the correct CI registry image - Restore grep -q verification after AAP sed overrides to fail fast if the field name changes in the values file Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add grep verification for component image override Fail fast if the sed replacement didn't match anything in the values file, instead of silently testing against the pinned image. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * make jobs * Add storage, MCO, and subnet pod diagnostics to gather script Collect LVMS/topolvm state (pods, events, logs, LVMCluster, VolumeAttachments), MachineConfig status and pull-secret registry list, CDI importer pod logs from subnet namespaces, and service account imagePullSecrets state. These were missing when debugging CI failures involving volume mount timeouts and image pull errors. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Mount pull secret into installer container for namespace secret creation The old kustomize flow created a quay-pull-secret in the OSAC namespace via secretGenerator, giving AAP job pods credentials to pull CI-built EE images. The helm migration dropped this mount, causing intermittent ErrImagePull when the MCO hasn't propagated the global pull secret to CRI-O before AAP launches automation jobs. Re-add the pull secret as a volume mount at /installer/pull-secret.json so refresh-after-snapshot.py can create the namespace-level secret. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Increase helm full install periodic frequency to every 4 hours Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
amogh-redhat
pushed a commit
to amogh-redhat/release
that referenced
this pull request
Aug 5, 2026
![openshift-prow[bot]](https://github.githubassets.com/images/gravatars/gravatar-user-420.png){kind=avatar}
* Switch OSAC CI from kustomize to helm-based refresh Replace the kustomize-based refresh flow with the Python helm refresh script (refresh-after-snapshot.py). The boot step now uses the vmaas-helm snapshot flavor and deploys via helm upgrade instead of oc apply -k. Changes: - Boot ref: E2E_VALUES_FILE replaces E2E_KUSTOMIZE_OVERLAY, flavor defaults to vmaas-helm - Boot script: license mounts to values dir, component overrides use sed on helm values YAML (handles both image:tag and split repository/tag formats), refresh runs python3 -u refresh-after-snapshot.py - osac-installer config: add values/ to run_if_changed trigger - osac-test-infra config: bump vmaas periodic to hourly, drop kustomize full-setup job Depends on osac-project/osac-installer#296 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix clone name mismatch, operator override, and AAP verification - Update CLONE_NAME to vmaas-helm in test and destroy steps to match the boot step — prevents kubeconfig-not-found and VM leak - Fix split repository/tag component override to also replace the repository line (not just the tag) so osac-operator PRs get the correct CI registry image - Restore grep -q verification after AAP sed overrides to fail fast if the field name changes in the values file Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add grep verification for component image override Fail fast if the sed replacement didn't match anything in the values file, instead of silently testing against the pinned image. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * make jobs * Add storage, MCO, and subnet pod diagnostics to gather script Collect LVMS/topolvm state (pods, events, logs, LVMCluster, VolumeAttachments), MachineConfig status and pull-secret registry list, CDI importer pod logs from subnet namespaces, and service account imagePullSecrets state. These were missing when debugging CI failures involving volume mount timeouts and image pull errors. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Mount pull secret into installer container for namespace secret creation The old kustomize flow created a quay-pull-secret in the OSAC namespace via secretGenerator, giving AAP job pods credentials to pull CI-built EE images. The helm migration dropped this mount, causing intermittent ErrImagePull when the MCO hasn't propagated the global pull secret to CRI-O before AAP launches automation jobs. Re-add the pull secret as a volume mount at /installer/pull-secret.json so refresh-after-snapshot.py can create the namespace-level secret. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Increase helm full install periodic frequency to every 4 hours Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
TimurMP
pushed a commit
to TimurMP/release
that referenced
this pull request
Sep 5, 2026
* Switch OSAC CI from kustomize to helm-based refresh Replace the kustomize-based refresh flow with the Python helm refresh script (refresh-after-snapshot.py). The boot step now uses the vmaas-helm snapshot flavor and deploys via helm upgrade instead of oc apply -k. Changes: - Boot ref: E2E_VALUES_FILE replaces E2E_KUSTOMIZE_OVERLAY, flavor defaults to vmaas-helm - Boot script: license mounts to values dir, component overrides use sed on helm values YAML (handles both image:tag and split repository/tag formats), refresh runs python3 -u refresh-after-snapshot.py - osac-installer config: add values/ to run_if_changed trigger - osac-test-infra config: bump vmaas periodic to hourly, drop kustomize full-setup job Depends on osac-project/osac-installer#296 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Fix clone name mismatch, operator override, and AAP verification - Update CLONE_NAME to vmaas-helm in test and destroy steps to match the boot step — prevents kubeconfig-not-found and VM leak - Fix split repository/tag component override to also replace the repository line (not just the tag) so osac-operator PRs get the correct CI registry image - Restore grep -q verification after AAP sed overrides to fail fast if the field name changes in the values file Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add grep verification for component image override Fail fast if the sed replacement didn't match anything in the values file, instead of silently testing against the pinned image. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * make jobs * Add storage, MCO, and subnet pod diagnostics to gather script Collect LVMS/topolvm state (pods, events, logs, LVMCluster, VolumeAttachments), MachineConfig status and pull-secret registry list, CDI importer pod logs from subnet namespaces, and service account imagePullSecrets state. These were missing when debugging CI failures involving volume mount timeouts and image pull errors. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Mount pull secret into installer container for namespace secret creation The old kustomize flow created a quay-pull-secret in the OSAC namespace via secretGenerator, giving AAP job pods credentials to pull CI-built EE images. The helm migration dropped this mount, causing intermittent ErrImagePull when the MCO hasn't propagated the global pull secret to CRI-O before AAP launches automation jobs. Re-add the pull secret as a volume mount at /installer/pull-secret.json so refresh-after-snapshot.py can create the namespace-level secret. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Increase helm full install periodic frequency to every 4 hours Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
vmaas-helmsnapshot flavor andrefresh-after-snapshot.py(helm upgrade) instead of kustomizesedon helm values YAML instead ofkustomize edit set imageprojectGitBranch,projectGitUri,eeImage)Depends on osac-project/osac-installer#296
Test plan
🤖 Generated with Claude Code
Summary by CodeRabbit
This PR updates the OpenShift as a Code (OSAC) CI “VMaaS” cluster refresh flow in the
osac-projectCI infrastructure to use a Helm-based refresh approach instead of the previous kustomize/overlay workflow, and adjusts the related presubmit/periodic jobs to validate component overrides and higher-frequency VMaaS execution.What changed (practically)
Cluster boot + refresh migrated to Helm
osac-project-cluster-toolsteps to boot with thevmaas-helmflavor (and use a Helm values file viaE2E_VALUES_FILE, defaulting tovalues/vmaas-ci/values.yaml) instead of the kustomize flavor/overlay flow.python3 -u scripts/refresh-after-snapshot.pyfor the Helm-based refresh, including mounting:/installer/${VALUES_DIR}/license.zip/installer/pull-secret.jsonComponent overrides + AAP overrides now edit Helm values directly
sededits of the installer Helm values YAML (handling bothimage:tagand separaterepository/tagforms).projectGitBranch, optionalprojectGitUri, andeeImage) using the same values YAML and guarded with verification checks to fail fast if edits don’t match expectations.Fix for intermittent EE image pull failures
refresh-after-snapshot.pycan create the needed namespace-level secret for AAP job pods (preventingErrImagePullscenarios when the global pull secret hasn’t propagated to CRI-O in time).Kustomize-based full-setup periodic job removed
e2e-vmaas-full-setup-helm); kustomize full-setup targeting is no longer present.CI workflow updates
osac-project/osac-installer: widened thee2e-vmaasrun_if_changedtrigger to include changes undervalues/(in addition to the existing base/overlays/scripts/prerequisites/charts paths).osac-project/osac-test-infra: changed thee2e-vmaas-periodiccron schedule from twice daily to hourly (7 * * * *).Diagnostics improvements
osac-project-gatherstep to collect more CNV and storage/MachineConfig diagnostics (including per-namespace pod inventories/logs and additional storage/MCO-related data) to improve debugging for VMaaS-related failures.