Skip to content

MGMT-22635: add CaaS CI workflow and e2e-caas jobs for OSAC - #79512

Closed
omer-vishlitzky wants to merge 7 commits into
openshift:mainfrom
omer-vishlitzky:caas-ci-jobs
Closed

MGMT-22635: add CaaS CI workflow and e2e-caas jobs for OSAC#79512
omer-vishlitzky wants to merge 7 commits into
openshift:mainfrom
omer-vishlitzky:caas-ci-jobs

Conversation

@omer-vishlitzky

@omer-vishlitzky omer-vishlitzky commented May 20, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Parameterize cluster-tool boot step: add CLUSTER_TOOL_FLAVOR_NAME and E2E_CLUSTER_TEMPLATE env vars, replace hardcoded flavor name
  • Parameterize cluster-tool test step: add E2E_TEST_DIR env var, make KubeVirt wait conditional
  • Add CaaS agent setup step (osac-project-cluster-tool-caas-agents): boots agent VM for HyperShift provisioning
  • Add CaaS workflow (osac-project-cluster-tool-caas): boot + agents + test + cleanup
  • Add e2e-caas presubmit job to all 5 OSAC repos (fulfillment-service, osac-operator, osac-installer, osac-test-infra, osac-aap)
  • Add 2 CaaS periodic jobs to osac-test-infra (Friday 2am + 6am)
  • Update gather step with CaaS/HyperShift diagnostics (ClusterOrders, HostedClusters, NodePools, Agents, MCE, MetalLB)

Test plan

  • Verify VMaaS e2e-vmaas jobs still work (backward compatible — all new env vars have VMaaS defaults)
  • Verify e2e-caas presubmit runs on osac-test-infra PRs
  • Verify CaaS periodic jobs trigger on Friday schedule
  • Verify gather step collects CaaS diagnostics

Summary by CodeRabbit

This PR adds Cluster-as-a-Service (CaaS) CI infrastructure to the OpenShift release repository, enabling automated testing of OSAC (OpenShift Assisted Installer) components in a CaaS environment via HyperShift.

Key Changes

New CaaS Workflow and Steps:

  • Introduces osac-project-cluster-tool-caas workflow that orchestrates CaaS cluster provisioning, including booting from a pre-built snapshot, setting up HyperShift agent infrastructure, and running e2e tests against the provisioned cluster
  • Adds osac-project-cluster-tool-caas-agents step that provisions agent VMs via libvirt, creates InfraEnv resources, and waits for agent registration—essential for HyperShift-based deployments

Parameterization of Existing Steps:

  • Enhances osac-project-cluster-tool-boot step with new environment variables (CLUSTER_TOOL_FLAVOR_NAME, E2E_CLUSTER_TEMPLATE) to support both VMaaS and CaaS cluster profiles without duplication
  • Updates osac-project-cluster-tool-test step to make KubeVirt testing conditional and add E2E_TEST_DIR parameter to run CaaS-specific test suites

New e2e-caas Presubmit Jobs:

  • Adds e2e-caas presubmit test to CI configurations for five OSAC repositories: fulfillment-service, osac-aap, osac-installer, osac-operator, and osac-test-infra
  • Each presubmit reuses component PR images and the shared osac-test-infra image, wired to the new osac-project-cluster-tool-caas workflow

CaaS Flavor Configuration:

  • Configures CaaS testing to use the caas-kustomize cluster flavor and osac.templates.ocp_ci_small cluster template (vs. VMaaS's vmaas-kustomize and osac.templates.ocp_virt_vm)
  • Targets CaaS-specific test directory (tests/caas) instead of VMaaS tests (tests/vmaas)

Backward Compatibility:

  • All new environment variables include sensible VMaaS defaults, ensuring existing VMaaS e2e jobs continue to function without modification
  • Changes are purely additive; no existing workflows or jobs are altered

Impact

This infrastructure enables the OSAC project to validate fulfillment-service, osac-operator, osac-installer, osac-aap, and osac-test-infra changes against a CaaS/HyperShift-based cluster, complementing the existing VMaaS testing pipeline. Presubmit jobs will automatically trigger on PRs to these repositories, providing immediate CI feedback for CaaS-related code changes.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label May 20, 2026
@openshift-ci-robot

openshift-ci-robot commented May 20, 2026

Copy link
Copy Markdown
Contributor

@omer-vishlitzky: This pull request references MGMT-22635 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

  • Parameterize cluster-tool boot step: add CLUSTER_TOOL_FLAVOR_NAME and E2E_CLUSTER_TEMPLATE env vars, replace hardcoded flavor name
  • Parameterize cluster-tool test step: add E2E_TEST_DIR env var, make KubeVirt wait conditional
  • Add CaaS agent setup step (osac-project-cluster-tool-caas-agents): boots agent VM for HyperShift provisioning
  • Add CaaS workflow (osac-project-cluster-tool-caas): boot + agents + test + cleanup
  • Add e2e-caas presubmit job to all 5 OSAC repos (fulfillment-service, osac-operator, osac-installer, osac-test-infra, osac-aap)
  • Add 2 CaaS periodic jobs to osac-test-infra (Friday 2am + 6am)
  • Update gather step with CaaS/HyperShift diagnostics (ClusterOrders, HostedClusters, NodePools, Agents, MCE, MetalLB)

Test plan

  • Verify VMaaS e2e-vmaas jobs still work (backward compatible — all new env vars have VMaaS defaults)
  • Verify e2e-caas presubmit runs on osac-test-infra PRs
  • Verify CaaS periodic jobs trigger on Friday schedule
  • Verify gather step collects CaaS diagnostics

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented May 20, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Adds e2e-caas test entries across five OSAC project CI configuration files, each enabling intranet capability, configuring component-specific dependencies, and routing to the osac-project-cluster-tool-caas workflow for CAAS end-to-end testing.

Changes

CAAS E2E Test Configuration

Layer / File(s) Summary
Add e2e-caas test entries across OSAC components
ci-operator/config/osac-project/fulfillment-service/osac-project-fulfillment-service-main.yaml, ci-operator/config/osac-project/osac-aap/osac-project-osac-aap-main.yaml, ci-operator/config/osac-project/osac-installer/osac-project-osac-installer-main.yaml, ci-operator/config/osac-project/osac-operator/osac-project-osac-operator-main.yaml, ci-operator/config/osac-project/osac-test-infra/osac-project-osac-test-infra-main.yaml
Five CI operator config files each add a new e2e-caas test entry with intranet capability, component-specific dependencies (fulfillment-service-pr, osac-aap-pr, osac-operator-pr where applicable), shared installer and test infra images, matching COMPONENT_IMAGE_NAME environment variables, and routing to osac-project-cluster-tool-caas workflow.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Suggested labels

lgtm, approved, rehearsals-ack

🚥 Pre-merge checks | ✅ 14 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (14 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and specifically describes the main changes: adding CaaS CI workflow and e2e-caas jobs for OSAC projects, which directly aligns with the changeset.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed PR contains no Ginkgo test files (.go) or test definitions; only CI configuration (YAML), shell scripts, and metadata files. Check is not applicable.
Test Structure And Quality ✅ Passed This PR contains no Ginkgo test code; it only modifies CI/CD configuration YAML and shell scripts. The check is not applicable to this PR.
Microshift Test Compatibility ✅ Passed PR adds CI/CD workflow configurations (YAML) for OSAC testing, not new Ginkgo e2e tests. No Go test code files with Ginkgo test declarations are modified.
Single Node Openshift (Sno) Test Compatibility ✅ Passed PR adds only CI configuration and workflow definitions with pytest-based tests, not Ginkgo e2e tests. Check applies only to new Ginkgo tests, so it is not applicable.
Topology-Aware Scheduling Compatibility ✅ Passed PR adds only CI test job configuration (e2e-caas jobs and workflows) with no deployment manifests, operator code, or pod scheduling constraints. Not applicable to this check.
Ote Binary Stdout Contract ✅ Passed PR modifies only CI/CD configuration (YAML) and shell scripts in openshift/release repo, not Go test code. OTE Binary Stdout Contract check applies only to test binaries in test code repositories.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PR adds CI configuration and orchestration scripts only, not new Ginkgo e2e tests. The custom check applies only to Ginkgo tests, making it not applicable here.
No-Weak-Crypto ✅ Passed PR contains only CI/CD config and shell scripts. No weak crypto algorithms, custom crypto implementations, or non-constant-time secret comparisons found.
Container-Privileges ✅ Passed No privileged containers, elevated capabilities, hostPID/hostNetwork/hostIPC, or allowPrivilegeEscalation found in any YAML or script files added by this PR.
No-Sensitive-Data-In-Logs ✅ Passed Scripts use curl -sk flags (silent) and -u auth which don't expose credentials in logs. No echo statements log tokens or passwords.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (2)
ci-operator/step-registry/osac-project/gather/osac-project-gather-commands.sh (1)

84-86: ⚡ Quick win

Capture YAML for NodePools/Agents/InfraEnvs as well.

These three resources are currently captured only with -o wide; adding YAML snapshots would make post-failure triage much easier and keep this section consistent with the other CaaS resources.

Proposed patch
 oc get nodepools -A -o wide > "${ARTIFACT_DIR}/caas/nodepools.txt" 2>&1 || true
+oc get nodepools -A -o yaml > "${ARTIFACT_DIR}/caas/nodepools.yaml" 2>&1 || true
 oc get agents -A -o wide > "${ARTIFACT_DIR}/caas/agents.txt" 2>&1 || true
+oc get agents -A -o yaml > "${ARTIFACT_DIR}/caas/agents.yaml" 2>&1 || true
 oc get infraenvs -A -o wide > "${ARTIFACT_DIR}/caas/infraenvs.txt" 2>&1 || true
+oc get infraenvs -A -o yaml > "${ARTIFACT_DIR}/caas/infraenvs.yaml" 2>&1 || true
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/osac-project/gather/osac-project-gather-commands.sh`
around lines 84 - 86, Add YAML snapshots for the three CaaS resources in
addition to the existing -o wide outputs by running `oc get nodepools -A -o
yaml`, `oc get agents -A -o yaml`, and `oc get infraenvs -A -o yaml` and writing
them to "${ARTIFACT_DIR}/caas/nodepools.yaml",
"${ARTIFACT_DIR}/caas/agents.yaml", and "${ARTIFACT_DIR}/caas/infraenvs.yaml"
respectively (preserve the existing -o wide > ... .txt lines and the "2>&1 ||
true" behavior); update the block containing the existing `oc get ... -o wide`
commands so each resource also has a corresponding `-o yaml` capture using the
same ARTIFACT_DIR and error-tolerant redirection.
ci-operator/step-registry/osac-project/cluster-tool/caas/osac-project-cluster-tool-caas-workflow.yaml (1)

21-21: ⚡ Quick win

Use a team-owned immutable flavor image reference.

Line 21 points to a personal namespace with a tag. This can make CI reproducibility and ownership brittle; prefer an org-owned image and digest pinning.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/osac-project/cluster-tool/caas/osac-project-cluster-tool-caas-workflow.yaml`
at line 21, The CLUSTER_TOOL_FLAVOR_IMAGE value currently points to a personal
namespace and a mutable tag
("quay.io/rh-ee-ovishlit/cluster-flavors:osac-caas"); change the value of
CLUSTER_TOOL_FLAVOR_IMAGE to use a team/org-owned repository and pin to an
immutable digest (e.g., quay.io/<team-or-org>/cluster-flavors@sha256:<digest>)
so CI uses a stable, owned image; update any related image publishing workflow
to publish the digested image into the team repo if needed.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@ci-operator/step-registry/osac-project/cluster-tool/caas-agents/osac-project-cluster-tool-caas-agents-commands.sh`:
- Line 67: The script prints a sensitive discovery ISO URL via the echo "ISO
URL: ${ISO_URL}" statement; remove this direct printing and either delete the
echo or replace it with a non-sensitive check (e.g., log that ISO_URL is
set/empty or print a redacted placeholder/length) in the same script section so
that the ISO_URL variable is not emitted to CI logs; locate and update the echo
line referencing ISO_URL in osac-project-cluster-tool-caas-agents-commands.sh to
implement this change.

In
`@ci-operator/step-registry/osac-project/cluster-tool/test/osac-project-cluster-tool-test-commands.sh`:
- Around line 49-50: Validate TEST_DIR explicitly instead of using loose
substring matching: check TEST_DIR is non-empty and equals the expected value or
matches a strict pattern (e.g., "^tests/caas(/.*)?$") before the KubeVirt
availability branch and before invoking pytest, and replace the unquoted usage
with a quoted expansion when passing to pytest (use "${TEST_DIR}"). Update the
if condition that currently uses [[ "${TEST_DIR}" != *"caas"* ]] to perform the
explicit validation and ensure later references to TEST_DIR (the pytest
invocation) are quoted to avoid word-splitting.

---

Nitpick comments:
In
`@ci-operator/step-registry/osac-project/cluster-tool/caas/osac-project-cluster-tool-caas-workflow.yaml`:
- Line 21: The CLUSTER_TOOL_FLAVOR_IMAGE value currently points to a personal
namespace and a mutable tag
("quay.io/rh-ee-ovishlit/cluster-flavors:osac-caas"); change the value of
CLUSTER_TOOL_FLAVOR_IMAGE to use a team/org-owned repository and pin to an
immutable digest (e.g., quay.io/<team-or-org>/cluster-flavors@sha256:<digest>)
so CI uses a stable, owned image; update any related image publishing workflow
to publish the digested image into the team repo if needed.

In
`@ci-operator/step-registry/osac-project/gather/osac-project-gather-commands.sh`:
- Around line 84-86: Add YAML snapshots for the three CaaS resources in addition
to the existing -o wide outputs by running `oc get nodepools -A -o yaml`, `oc
get agents -A -o yaml`, and `oc get infraenvs -A -o yaml` and writing them to
"${ARTIFACT_DIR}/caas/nodepools.yaml", "${ARTIFACT_DIR}/caas/agents.yaml", and
"${ARTIFACT_DIR}/caas/infraenvs.yaml" respectively (preserve the existing -o
wide > ... .txt lines and the "2>&1 || true" behavior); update the block
containing the existing `oc get ... -o wide` commands so each resource also has
a corresponding `-o yaml` capture using the same ARTIFACT_DIR and error-tolerant
redirection.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 8e07b33e-6abb-494a-b1cc-229a609e0104

📥 Commits

Reviewing files that changed from the base of the PR and between 69d8268 and 9c1e473.

📒 Files selected for processing (17)
  • ci-operator/config/osac-project/fulfillment-service/osac-project-fulfillment-service-main.yaml
  • ci-operator/config/osac-project/osac-aap/osac-project-osac-aap-main.yaml
  • ci-operator/config/osac-project/osac-installer/osac-project-osac-installer-main.yaml
  • ci-operator/config/osac-project/osac-operator/osac-project-osac-operator-main.yaml
  • ci-operator/config/osac-project/osac-test-infra/osac-project-osac-test-infra-main.yaml
  • ci-operator/step-registry/osac-project/cluster-tool/boot/osac-project-cluster-tool-boot-commands.sh
  • ci-operator/step-registry/osac-project/cluster-tool/boot/osac-project-cluster-tool-boot-ref.yaml
  • ci-operator/step-registry/osac-project/cluster-tool/caas-agents/OWNERS
  • ci-operator/step-registry/osac-project/cluster-tool/caas-agents/osac-project-cluster-tool-caas-agents-commands.sh
  • ci-operator/step-registry/osac-project/cluster-tool/caas-agents/osac-project-cluster-tool-caas-agents-ref.metadata.json
  • ci-operator/step-registry/osac-project/cluster-tool/caas-agents/osac-project-cluster-tool-caas-agents-ref.yaml
  • ci-operator/step-registry/osac-project/cluster-tool/caas/OWNERS
  • ci-operator/step-registry/osac-project/cluster-tool/caas/osac-project-cluster-tool-caas-workflow.metadata.json
  • ci-operator/step-registry/osac-project/cluster-tool/caas/osac-project-cluster-tool-caas-workflow.yaml
  • ci-operator/step-registry/osac-project/cluster-tool/test/osac-project-cluster-tool-test-commands.sh
  • ci-operator/step-registry/osac-project/cluster-tool/test/osac-project-cluster-tool-test-ref.yaml
  • ci-operator/step-registry/osac-project/gather/osac-project-gather-commands.sh

sleep 5
done
[[ -z "${ISO_URL}" ]] && { echo "Timed out waiting for ISO URL"; exit 1; }
echo "ISO URL: ${ISO_URL}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Avoid logging the discovery ISO URL.

Line 67 prints a URL derived from cluster state; this can expose sensitive cluster URL/token material in CI logs.

Proposed fix
-echo "ISO URL: ${ISO_URL}"
+echo "Discovery ISO URL acquired."

As per coding guidelines, "Protect sensitive information in step registry scripts - never echo or print passwords, tokens, API keys, cluster URLs, or kubeconfig contents".

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
echo "ISO URL: ${ISO_URL}"
echo "Discovery ISO URL acquired."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/osac-project/cluster-tool/caas-agents/osac-project-cluster-tool-caas-agents-commands.sh`
at line 67, The script prints a sensitive discovery ISO URL via the echo "ISO
URL: ${ISO_URL}" statement; remove this direct printing and either delete the
echo or replace it with a non-sensitive check (e.g., log that ISO_URL is
set/empty or print a redacted placeholder/length) in the same script section so
that the ISO_URL variable is not emitted to CI logs; locate and update the echo
line referencing ISO_URL in osac-project-cluster-tool-caas-agents-commands.sh to
implement this change.

Comment on lines +49 to +50
if [[ "${TEST_DIR}" != *"caas"* ]]; then
echo "Waiting for KubeVirt to be Available..."

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
rg -n 'E2E_TEST_DIR:' ci-operator/config/osac-project/**/osac-project-*-main.yaml

Repository: openshift/release

Length of output: 305


🏁 Script executed:

# View the full script to understand context around lines 49-83
cat -n ci-operator/step-registry/osac-project/cluster-tool/test/osac-project-cluster-tool-test-commands.sh | sed -n '1,90p'

Repository: openshift/release

Length of output: 3739


🏁 Script executed:

# Search for other references to E2E_TEST_DIR in the repository to see if it's sourced elsewhere
rg -n 'E2E_TEST_DIR|TEST_DIR=' --type yaml --type sh ci-operator/ | head -40

Repository: openshift/release

Length of output: 2558


Harden TEST_DIR handling with explicit validation.

Line 49 uses substring matching and line 83 passes ${TEST_DIR} unquoted to pytest. While current configuration strictly sets E2E_TEST_DIR: tests/caas, explicit validation makes the intent clearer and prevents future misconfiguration.

🔧 Proposed fix
 TEST_DIR="$7"
+
+case "${TEST_DIR}" in
+  tests/caas) ;;
+  *)
+    echo "ERROR: unsupported E2E_TEST_DIR: ${TEST_DIR}"
+    exit 1
+    ;;
+esac
@@
-if [[ "${TEST_DIR}" != *"caas"* ]]; then
+if [[ "${TEST_DIR}" != "tests/caas" ]]; then
@@
-    pytest ${TEST_DIR}/ -v --junitxml=/tmp/test-results/junit_e2e.xml
+    pytest -- "${TEST_DIR%/}/" -v --junitxml=/tmp/test-results/junit_e2e.xml
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/osac-project/cluster-tool/test/osac-project-cluster-tool-test-commands.sh`
around lines 49 - 50, Validate TEST_DIR explicitly instead of using loose
substring matching: check TEST_DIR is non-empty and equals the expected value or
matches a strict pattern (e.g., "^tests/caas(/.*)?$") before the KubeVirt
availability branch and before invoking pytest, and replace the unquoted usage
with a quoted expansion when passing to pytest (use "${TEST_DIR}"). Update the
if condition that currently uses [[ "${TEST_DIR}" != *"caas"* ]] to perform the
explicit validation and ensure later references to TEST_DIR (the pytest
invocation) are quoted to avoid word-splitting.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
ci-operator/step-registry/osac-project/gather/osac-project-gather-commands.sh (1)

90-90: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Change ipaddresspool to ipaddresspools on line 90.

The MetalLB resource name is ipaddresspools (plural), not ipaddresspool (singular). The current command fails silently due to || true and does not collect MetalLB diagnostics. Use oc get ipaddresspools -A -o yaml > "${ARTIFACT_DIR}/caas/metallb-pools.yaml" 2>&1 || true instead.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/osac-project/gather/osac-project-gather-commands.sh`
at line 90, Replace the incorrect MetalLB resource name in the gather command:
change the oc invocation that currently uses "ipaddresspool" to the plural
"ipaddresspools" so the command becomes oc get ipaddresspools -A -o yaml >
"${ARTIFACT_DIR}/caas/metallb-pools.yaml" 2>&1 || true; update the line that
writes to "${ARTIFACT_DIR}/caas/metallb-pools.yaml" to use the corrected
resource name so MetalLB diagnostics are actually collected.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In
`@ci-operator/step-registry/osac-project/gather/osac-project-gather-commands.sh`:
- Line 90: Replace the incorrect MetalLB resource name in the gather command:
change the oc invocation that currently uses "ipaddresspool" to the plural
"ipaddresspools" so the command becomes oc get ipaddresspools -A -o yaml >
"${ARTIFACT_DIR}/caas/metallb-pools.yaml" 2>&1 || true; update the line that
writes to "${ARTIFACT_DIR}/caas/metallb-pools.yaml" to use the corrected
resource name so MetalLB diagnostics are actually collected.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 2e889a68-b836-46dd-99af-c48c28fe9d4a

📥 Commits

Reviewing files that changed from the base of the PR and between 9c1e473 and 1577d00.

📒 Files selected for processing (2)
  • ci-operator/step-registry/osac-project/cluster-tool/caas-agents/osac-project-cluster-tool-caas-agents-commands.sh
  • ci-operator/step-registry/osac-project/gather/osac-project-gather-commands.sh

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/pj-rehearse

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-osac-project-osac-installer-main-e2e-caas pull-ci-osac-project-osac-operator-main-e2e-caas pull-ci-osac-project-osac-test-infra-main-e2e-caas pull-ci-osac-project-osac-aap-main-e2e-caas pull-ci-osac-project-fulfillment-service-main-e2e-caas

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-osac-project-fulfillment-service-main-e2e-caas

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-osac-project-osac-installer-main-e2e-caas

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-osac-project-osac-installer-main-e2e-caas pull-ci-osac-project-osac-operator-main-e2e-caas pull-ci-osac-project-osac-test-infra-main-e2e-caas pull-ci-osac-project-osac-aap-main-e2e-caas pull-ci-osac-project-fulfillment-service-main-e2e-caas

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-osac-project-osac-installer-main-e2e-caas pull-ci-osac-project-osac-operator-main-e2e-caas pull-ci-osac-project-osac-test-infra-main-e2e-caas pull-ci-osac-project-osac-aap-main-e2e-caas pull-ci-osac-project-fulfillment-service-main-e2e-caas

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/retest

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-osac-project-osac-installer-main-e2e-caas pull-ci-osac-project-osac-operator-main-e2e-caas pull-ci-osac-project-osac-test-infra-main-e2e-caas pull-ci-osac-project-osac-aap-main-e2e-caas pull-ci-osac-project-fulfillment-service-main-e2e-caas

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-osac-project-osac-installer-main-e2e-caas pull-ci-osac-project-osac-operator-main-e2e-caas pull-ci-osac-project-osac-test-infra-main-e2e-caas pull-ci-osac-project-osac-aap-main-e2e-caas pull-ci-osac-project-fulfillment-service-main-e2e-caas pull-ci-osac-project-osac-operator-main-e2e-vmaas pull-ci-osac-project-fulfillment-service-main-e2e-vmaas pull-ci-osac-project-osac-test-infra-main-e2e-vmaas pull-ci-osac-project-osac-installer-main-e2e-vmaas pull-ci-osac-project-osac-aap-main-e2e-vmaas

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-osac-project-osac-installer-main-e2e-caas pull-ci-osac-project-osac-operator-main-e2e-caas pull-ci-osac-project-osac-test-infra-main-e2e-caas pull-ci-osac-project-osac-aap-main-e2e-caas pull-ci-osac-project-fulfillment-service-main-e2e-caas pull-ci-osac-project-osac-operator-main-e2e-vmaas pull-ci-osac-project-fulfillment-service-main-e2e-vmaas pull-ci-osac-project-osac-test-infra-main-e2e-vmaas pull-ci-osac-project-osac-installer-main-e2e-vmaas pull-ci-osac-project-osac-aap-main-e2e-vmaas

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/retest

@openshift-ci

openshift-ci Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danmanor, omer-vishlitzky

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 10, 2026
@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-osac-project-osac-aap-main-e2e-caas pull-ci-osac-project-osac-aap-main-e2e-vmaas pull-ci-osac-project-fulfillment-service-main-e2e-caas pull-ci-osac-project-fulfillment-service-main-e2e-vmaas pull-ci-osac-project-osac-test-infra-main-e2e-caas pull-ci-osac-project-osac-test-infra-main-e2e-vmaas pull-ci-osac-project-osac-operator-main-e2e-caas pull-ci-osac-project-osac-operator-main-e2e-vmaas pull-ci-osac-project-osac-installer-main-e2e-caas pull-ci-osac-project-osac-installer-main-e2e-vmaas periodic-ci-osac-project-osac-test-infra-main-e2e-caas-full-setup-helm periodic-ci-osac-project-osac-test-infra-main-e2e-caas-full-setup-kustomize periodic-ci-osac-project-osac-test-infra-main-e2e-caas-periodic

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@openshift-ci[bot]: your /pj-rehearse request was not processed because the request waited in queue for longer than 5 minutes. Please retry in a few minutes.

Add storage diagnostics to the gather script to debug intermittent
CaaS boot failures where the fulfillment-database PVC loses its data
after recert changes the node hostname. Collects PV specs (including
nodeAffinity), topolvm LogicalVolume CRs, LVMS operator state, and
openshift-storage pod logs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@openshift-ci openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Jun 10, 2026
@openshift-ci

openshift-ci Bot commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

New changes are detected. LGTM label has been removed.

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-osac-project-osac-aap-main-e2e-caas pull-ci-osac-project-osac-aap-main-e2e-vmaas pull-ci-osac-project-fulfillment-service-main-e2e-caas pull-ci-osac-project-fulfillment-service-main-e2e-vmaas pull-ci-osac-project-osac-test-infra-main-e2e-caas pull-ci-osac-project-osac-test-infra-main-e2e-vmaas pull-ci-osac-project-osac-operator-main-e2e-caas pull-ci-osac-project-osac-operator-main-e2e-vmaas pull-ci-osac-project-osac-installer-main-e2e-caas pull-ci-osac-project-osac-installer-main-e2e-vmaas periodic-ci-osac-project-osac-test-infra-main-e2e-caas-full-setup-helm periodic-ci-osac-project-osac-test-infra-main-e2e-caas-full-setup-kustomize periodic-ci-osac-project-osac-test-infra-main-e2e-caas-periodic

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-osac-project-osac-aap-main-e2e-caas pull-ci-osac-project-osac-aap-main-e2e-vmaas pull-ci-osac-project-fulfillment-service-main-e2e-caas pull-ci-osac-project-fulfillment-service-main-e2e-vmaas pull-ci-osac-project-osac-test-infra-main-e2e-caas pull-ci-osac-project-osac-test-infra-main-e2e-vmaas pull-ci-osac-project-osac-operator-main-e2e-caas pull-ci-osac-project-osac-operator-main-e2e-vmaas pull-ci-osac-project-osac-installer-main-e2e-caas pull-ci-osac-project-osac-installer-main-e2e-vmaas periodic-ci-osac-project-osac-test-infra-main-e2e-caas-full-setup-helm periodic-ci-osac-project-osac-test-infra-main-e2e-caas-full-setup-kustomize periodic-ci-osac-project-osac-test-infra-main-e2e-caas-periodic

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

Collect per-container logs (including --previous) for openshift-storage
pods and topolvm-node CSI driver pods. The vg-manager pod hosts the
topolvm CSI node driver which handles NodeStageVolume (format/mount) —
its previous logs are needed to diagnose cases where the database volume
is mounted with empty data after a snapshot boot.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-osac-project-osac-test-infra-main-e2e-caas pull-ci-osac-project-osac-aap-main-e2e-caas pull-ci-osac-project-osac-operator-main-e2e-caas pull-ci-osac-project-fulfillment-service-main-e2e-caas pull-ci-osac-project-osac-installer-main-e2e-caas

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-osac-project-osac-installer-main-e2e-caas

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-osac-project-osac-test-infra-main-e2e-caas pull-ci-osac-project-osac-aap-main-e2e-caas pull-ci-osac-project-osac-operator-main-e2e-caas pull-ci-osac-project-fulfillment-service-main-e2e-caas pull-ci-osac-project-osac-installer-main-e2e-caas

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-osac-project-osac-test-infra-main-e2e-caas pull-ci-osac-project-osac-aap-main-e2e-caas pull-ci-osac-project-osac-operator-main-e2e-caas pull-ci-osac-project-fulfillment-service-main-e2e-caas pull-ci-osac-project-osac-installer-main-e2e-caas

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@omer-vishlitzky: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-osac-project-osac-test-infra-main-e2e-caas osac-project/osac-test-infra presubmit Presubmit changed
pull-ci-osac-project-osac-test-infra-main-e2e-vmaas osac-project/osac-test-infra presubmit Registry content changed
pull-ci-osac-project-osac-operator-main-e2e-caas osac-project/osac-operator presubmit Presubmit changed
pull-ci-osac-project-osac-operator-main-e2e-vmaas osac-project/osac-operator presubmit Registry content changed
pull-ci-osac-project-fulfillment-service-main-e2e-caas osac-project/fulfillment-service presubmit Presubmit changed
pull-ci-osac-project-fulfillment-service-main-e2e-vmaas osac-project/fulfillment-service presubmit Registry content changed
pull-ci-osac-project-osac-aap-main-e2e-caas osac-project/osac-aap presubmit Presubmit changed
pull-ci-osac-project-osac-aap-main-e2e-vmaas osac-project/osac-aap presubmit Registry content changed
pull-ci-osac-project-osac-installer-main-e2e-caas osac-project/osac-installer presubmit Presubmit changed
pull-ci-osac-project-osac-installer-main-e2e-vmaas osac-project/osac-installer presubmit Registry content changed
periodic-ci-osac-project-osac-test-infra-main-e2e-vmaas-periodic N/A periodic Registry content changed
periodic-ci-osac-project-osac-test-infra-main-e2e-vmaas-full-setup-kustomize N/A periodic Registry content changed
periodic-ci-osac-project-osac-test-infra-main-e2e-vmaas-full-setup-helm N/A periodic Registry content changed
periodic-ci-osac-project-osac-test-infra-main-e2e-caas-full-setup-helm N/A periodic Periodic changed
periodic-ci-osac-project-osac-test-infra-main-e2e-caas-periodic N/A periodic Periodic changed
periodic-ci-osac-project-osac-test-infra-main-e2e-caas-full-setup-kustomize N/A periodic Periodic changed
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-osac-project-osac-test-infra-main-e2e-caas pull-ci-osac-project-osac-aap-main-e2e-caas pull-ci-osac-project-osac-operator-main-e2e-caas pull-ci-osac-project-fulfillment-service-main-e2e-caas pull-ci-osac-project-osac-installer-main-e2e-caas

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/pj-rehearse pull-ci-osac-project-osac-test-infra-main-e2e-caas pull-ci-osac-project-osac-aap-main-e2e-caas pull-ci-osac-project-osac-operator-main-e2e-caas pull-ci-osac-project-fulfillment-service-main-e2e-caas pull-ci-osac-project-osac-installer-main-e2e-caas

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci

openshift-ci Bot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

@omer-vishlitzky: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/osac-project/osac-test-infra/main/e2e-vmaas 1f2059c link unknown /pj-rehearse pull-ci-osac-project-osac-test-infra-main-e2e-vmaas
ci/rehearse/periodic-ci-osac-project-osac-test-infra-main-e2e-caas-full-setup-helm 1f2059c link unknown /pj-rehearse periodic-ci-osac-project-osac-test-infra-main-e2e-caas-full-setup-helm
ci/rehearse/periodic-ci-osac-project-osac-test-infra-main-e2e-caas-full-setup-kustomize 1f2059c link unknown /pj-rehearse periodic-ci-osac-project-osac-test-infra-main-e2e-caas-full-setup-kustomize
ci/rehearse/osac-project/osac-test-infra/main/e2e-caas 9f967f8 link unknown /pj-rehearse pull-ci-osac-project-osac-test-infra-main-e2e-caas
ci/rehearse/osac-project/osac-aap/main/e2e-caas 9f967f8 link unknown /pj-rehearse pull-ci-osac-project-osac-aap-main-e2e-caas
ci/rehearse/osac-project/osac-operator/main/e2e-caas 9f967f8 link unknown /pj-rehearse pull-ci-osac-project-osac-operator-main-e2e-caas

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants