MGMT-24219: replace 32 vmaas presubmits with 4 cluster-tool-based jobs - #79068
omer-vishlitzky wants to merge 14 commits into
Conversation
openshift-ci-robot
added
the
jira/valid-reference
|
@omer-vishlitzky: This pull request references MGMT-24219 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughConsolidates vMAAS E2E CI onto a new cluster-tool workflow: removes assisted-installer-based base_images and installer step refs/scripts, adds cluster-tool boot/test/destroy step refs and workflow, replaces many per-scenario e2e jobs with consolidated e2e-metal-vmaas-all (and a periodic variant), and updates metadata/OWNERS accordingly. ChangesCluster-Tool Migration
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~28 minutes 🚥 Pre-merge checks | ✅ 10 | ❌ 2❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (10 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Tip 💬 Introducing Slack Agent: The best way for teams to turn conversations into code.Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.
Built for teams:
One agent for your entire SDLC. Right inside Slack. Comment |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: omer-vishlitzky The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/pj-rehearse |
|
@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (3)
ci-operator/config/osac-project/osac-test-infra/osac-project-osac-test-infra-main.yaml (1)
19-20: 💤 Low valueConsider adding a brief comment for the scratch image purpose.
The
FROM scratchplaceholder image is a clever approach for repos that don't override a component, but it may confuse future maintainers. A brief inline comment would clarify the intent.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ci-operator/config/osac-project/osac-test-infra/osac-project-osac-test-infra-main.yaml` around lines 19 - 20, Add a brief inline comment explaining why the Dockerfile uses the scratch placeholder so future maintainers understand its intent; update the YAML entry containing "dockerfile_literal: FROM scratch" (associated with "component-image") to include a short comment like "placeholder base image for components that don't provide their own Dockerfile" immediately adjacent to or above that key.ci-operator/step-registry/osac-project/cluster-tool/boot/osac-project-cluster-tool-boot-ref.yaml (1)
36-37: 💤 Low valueConsider moving snapshot image to a team-owned registry namespace.
The default
CLUSTER_TOOL_FLAVOR_IMAGEpoints to a personal namespace (rh-ee-ovishlit). For long-term maintainability, consider migrating this to a team-owned namespace (e.g.,osac-project) to avoid bus-factor concerns and ensure the image remains accessible if the personal account changes.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ci-operator/step-registry/osac-project/cluster-tool/boot/osac-project-cluster-tool-boot-ref.yaml` around lines 36 - 37, The default value for CLUSTER_TOOL_FLAVOR_IMAGE currently uses a personal quay namespace ("quay.io/rh-ee-ovishlit/cluster-flavors:osac-vmaas"); change the default to a team-owned namespace (for example "quay.io/osac-project/cluster-flavors:osac-vmaas") and update any related docs or comments/configs that reference CLUSTER_TOOL_FLAVOR_IMAGE to point to the new repository so the image is maintained under a team account and avoids bus-factor risk.ci-operator/step-registry/osac-project/cluster-tool/boot/osac-project-cluster-tool-boot-commands.sh (1)
26-30: 💤 Low valueConsider using
mktempfor temporary file handling.Using a fixed path
/tmp/license.zipcould conflict with concurrent executions. While unlikely in this CI context (single job per machine), usingmktempwould be more robust.Suggested improvement
-base64 -d /var/run/osac-installer-aap/license > /tmp/license.zip +LICENSE_TMP=$(mktemp --suffix=.zip) +base64 -d /var/run/osac-installer-aap/license > "${LICENSE_TMP}" timeout -s 9 2m scp -F "${SHARED_DIR}/ssh_config" \ - /tmp/license.zip \ + "${LICENSE_TMP}" \ ci_machine:/tmp/license.zip🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@ci-operator/step-registry/osac-project/cluster-tool/boot/osac-project-cluster-tool-boot-commands.sh` around lines 26 - 30, Replace the fixed /tmp/license.zip temporary file with a safe mktemp-managed file: create a local temp file (e.g., via mktemp), write the decoded license into that temp file instead of /tmp/license.zip, use that temp file as the source in the scp command (the remote target can remain /tmp/license.zip), and ensure the temp file is removed after use (add a trap or explicit rm in the same script block). Update the commands around base64 -d (reading /var/run/osac-installer-aap/license), timeout ... scp (which references "${SHARED_DIR}/ssh_config"), and add cleanup to avoid leaving temp files behind.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@ci-operator/step-registry/osac-project/cluster-tool/boot/osac-project-cluster-tool-boot-commands.sh`:
- Around line 230-235: The kustomize image override string built in
COMPONENT_OVERRIDE_CMD uses unquoted expansions of COMPONENT_IMAGE_NAME and
COMPONENT_IMAGE which can allow shell metacharacters to be interpreted when that
string is later executed; update the assignment of COMPONENT_OVERRIDE_CMD so the
image name and tag are shell-quoted (e.g. wrap ${COMPONENT_IMAGE_NAME} and
${COMPONENT_IMAGE} in double quotes inside the command string) to ensure safe
expansion before running the command via bash -c.
In
`@ci-operator/step-registry/osac-project/cluster-tool/test/osac-project-cluster-tool-test-commands.sh`:
- Around line 18-24: The collect_artifacts function currently silences all scp
errors so missing junit_vmaas.xml can pass unnoticed; remove the stderr
suppression and silent-true fallback on the scp command, capture its exit
status, and if the copy fails or the target file ${ARTIFACT_DIR}/junit_vmaas.xml
does not exist, print a clear error to stderr and exit non‑zero (so the pipeline
fails). Specifically modify the scp in collect_artifacts (the scp -F
"${SHARED_DIR}/ssh_config" "ci_machine:${REMOTE_RESULTS_DIR}/junit_vmaas.xml"
"${ARTIFACT_DIR}/junit_vmaas.xml") to not redirect 2>/dev/null || true, then
check its exit code and/or test -f "${ARTIFACT_DIR}/junit_vmaas.xml" and call
echo >&2 "ERROR: failed to collect junit_vmaas.xml" and exit 1 when missing.
---
Nitpick comments:
In
`@ci-operator/config/osac-project/osac-test-infra/osac-project-osac-test-infra-main.yaml`:
- Around line 19-20: Add a brief inline comment explaining why the Dockerfile
uses the scratch placeholder so future maintainers understand its intent; update
the YAML entry containing "dockerfile_literal: FROM scratch" (associated with
"component-image") to include a short comment like "placeholder base image for
components that don't provide their own Dockerfile" immediately adjacent to or
above that key.
In
`@ci-operator/step-registry/osac-project/cluster-tool/boot/osac-project-cluster-tool-boot-commands.sh`:
- Around line 26-30: Replace the fixed /tmp/license.zip temporary file with a
safe mktemp-managed file: create a local temp file (e.g., via mktemp), write the
decoded license into that temp file instead of /tmp/license.zip, use that temp
file as the source in the scp command (the remote target can remain
/tmp/license.zip), and ensure the temp file is removed after use (add a trap or
explicit rm in the same script block). Update the commands around base64 -d
(reading /var/run/osac-installer-aap/license), timeout ... scp (which references
"${SHARED_DIR}/ssh_config"), and add cleanup to avoid leaving temp files behind.
In
`@ci-operator/step-registry/osac-project/cluster-tool/boot/osac-project-cluster-tool-boot-ref.yaml`:
- Around line 36-37: The default value for CLUSTER_TOOL_FLAVOR_IMAGE currently
uses a personal quay namespace
("quay.io/rh-ee-ovishlit/cluster-flavors:osac-vmaas"); change the default to a
team-owned namespace (for example
"quay.io/osac-project/cluster-flavors:osac-vmaas") and update any related docs
or comments/configs that reference CLUSTER_TOOL_FLAVOR_IMAGE to point to the new
repository so the image is maintained under a team account and avoids bus-factor
risk.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 954c1f8f-6f67-4742-8e57-465c1071e5bd
⛔ Files ignored due to path filters (5)
ci-operator/jobs/osac-project/fulfillment-service/osac-project-fulfillment-service-main-presubmits.yamlis excluded by!ci-operator/jobs/**ci-operator/jobs/osac-project/osac-installer/osac-project-osac-installer-main-presubmits.yamlis excluded by!ci-operator/jobs/**ci-operator/jobs/osac-project/osac-operator/osac-project-osac-operator-main-presubmits.yamlis excluded by!ci-operator/jobs/**ci-operator/jobs/osac-project/osac-test-infra/osac-project-osac-test-infra-main-periodics.yamlis excluded by!ci-operator/jobs/**ci-operator/jobs/osac-project/osac-test-infra/osac-project-osac-test-infra-main-presubmits.yamlis excluded by!ci-operator/jobs/**
📒 Files selected for processing (30)
ci-operator/config/osac-project/fulfillment-service/osac-project-fulfillment-service-main.yamlci-operator/config/osac-project/osac-installer/osac-project-osac-installer-main.yamlci-operator/config/osac-project/osac-operator/osac-project-osac-operator-main.yamlci-operator/config/osac-project/osac-test-infra/osac-project-osac-test-infra-main.yamlci-operator/step-registry/osac-project/baremetal/test/osac-project-baremetal-test-commands.shci-operator/step-registry/osac-project/cluster-tool/boot/OWNERSci-operator/step-registry/osac-project/cluster-tool/boot/osac-project-cluster-tool-boot-commands.shci-operator/step-registry/osac-project/cluster-tool/boot/osac-project-cluster-tool-boot-ref.metadata.jsonci-operator/step-registry/osac-project/cluster-tool/boot/osac-project-cluster-tool-boot-ref.yamlci-operator/step-registry/osac-project/cluster-tool/destroy/OWNERSci-operator/step-registry/osac-project/cluster-tool/destroy/osac-project-cluster-tool-destroy-commands.shci-operator/step-registry/osac-project/cluster-tool/destroy/osac-project-cluster-tool-destroy-ref.metadata.jsonci-operator/step-registry/osac-project/cluster-tool/destroy/osac-project-cluster-tool-destroy-ref.yamlci-operator/step-registry/osac-project/cluster-tool/test/OWNERSci-operator/step-registry/osac-project/cluster-tool/test/osac-project-cluster-tool-test-commands.shci-operator/step-registry/osac-project/cluster-tool/test/osac-project-cluster-tool-test-ref.metadata.jsonci-operator/step-registry/osac-project/cluster-tool/test/osac-project-cluster-tool-test-ref.yamlci-operator/step-registry/osac-project/cluster-tool/vmaas/OWNERSci-operator/step-registry/osac-project/cluster-tool/vmaas/osac-project-cluster-tool-vmaas-workflow.metadata.jsonci-operator/step-registry/osac-project/cluster-tool/vmaas/osac-project-cluster-tool-vmaas-workflow.yamlci-operator/step-registry/osac-project/installer/component/osac-project-installer-component-commands.shci-operator/step-registry/osac-project/installer/component/osac-project-installer-component-ref.yamlci-operator/step-registry/osac-project/installer/osac-project-installer-commands.shci-operator/step-registry/osac-project/installer/osac-project-installer-ref.yamlci-operator/step-registry/osac-project/ofcir/OWNERSci-operator/step-registry/osac-project/ofcir/baremetal/OWNERSci-operator/step-registry/osac-project/ofcir/baremetal/component/OWNERSci-operator/step-registry/osac-project/ofcir/baremetal/component/osac-project-ofcir-baremetal-component-workflow.metadata.jsonci-operator/step-registry/osac-project/ofcir/baremetal/component/osac-project-ofcir-baremetal-component-workflow.yamlci-operator/step-registry/osac-project/ofcir/baremetal/osac-project-ofcir-baremetal-workflow.yaml
💤 Files with no reviewable changes (11)
- ci-operator/step-registry/osac-project/ofcir/OWNERS
- ci-operator/step-registry/osac-project/ofcir/baremetal/OWNERS
- ci-operator/step-registry/osac-project/ofcir/baremetal/component/osac-project-ofcir-baremetal-component-workflow.yaml
- ci-operator/step-registry/osac-project/ofcir/baremetal/component/OWNERS
- ci-operator/step-registry/osac-project/installer/component/osac-project-installer-component-commands.sh
- ci-operator/step-registry/osac-project/installer/osac-project-installer-commands.sh
- ci-operator/step-registry/osac-project/ofcir/baremetal/osac-project-ofcir-baremetal-workflow.yaml
- ci-operator/step-registry/osac-project/installer/osac-project-installer-ref.yaml
- ci-operator/step-registry/osac-project/installer/component/osac-project-installer-component-ref.yaml
- ci-operator/step-registry/osac-project/ofcir/baremetal/component/osac-project-ofcir-baremetal-component-workflow.metadata.json
- ci-operator/step-registry/osac-project/baremetal/test/osac-project-baremetal-test-commands.sh
| # --- Phase 4: component override (conditional) --- | ||
| COMPONENT_OVERRIDE_CMD="" | ||
| if [[ -n "${COMPONENT_IMAGE}" ]] && [[ -n "${COMPONENT_IMAGE_NAME}" ]]; then | ||
| echo "=== Component override: ${COMPONENT_IMAGE_NAME} ===" | ||
| COMPONENT_OVERRIDE_CMD="cd /installer && kustomize edit set image ${COMPONENT_IMAGE_NAME}=${COMPONENT_IMAGE} && " | ||
| fi |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Verify that kustomize edit set image handles image names with special characters properly
# Check documentation for expected format
echo "Checking kustomize edit set image usage patterns in the codebase..."
rg -n "kustomize edit set image" --type sh -C2Repository: openshift/release
Length of output: 911
🏁 Script executed:
# Look at the full script context to understand variable sourcing
cat -n ci-operator/step-registry/osac-project/cluster-tool/boot/osac-project-cluster-tool-boot-commands.sh | head -250 | tail -100Repository: openshift/release
Length of output: 4912
🌐 Web query:
kustomize edit set image syntax security special characters
💡 Result:
The syntax for kustomize edit set image is kustomize edit set image [<oldImageName>=][<newImageName>]:<newTag>[@<digest>] [1][2][3]. Multiple image arguments can be provided [3]. The command modifies the images field in kustomization.yaml in-place, adding or updating entries like: images: - name: busybox newName: alpine newTag: 3.6 [1]. For special characters: Image names with colons (e.g., localhost:5000/image) are supported via regexp parsing (fixed in early versions) [4]. Tags are validated to contain only alphanumeric, '.', '_', '-' characters; other characters like '+' cause "invalid format" errors [5]. Kustomize automatically quotes newTag values in YAML if needed (e.g., git hashes starting with numbers) to preserve them as strings [6]. Use single quotes around arguments with shell special characters (e.g., $, @, *) to prevent shell expansion: kustomize edit set image 'my-image=repo/image:*@sha256:digest' [7][2]. The * wildcard preserves existing values: image=*:tag keeps newName, image=newname:* keeps newTag, image=*@digest keeps newName/newTag [2][8]. Digests use @sha256:... format, with recent support for both tag+digest [9][10]. No security vulnerabilities (e.g., injection, command execution) are documented in results; input validation limits tags [5], and edits are confined to kustomization.yaml [1]. Use in CI/CD is common but commit changes to track updates [11]. Run kustomize edit set image --help for exact version-specific details [3].
Citations:
- 1: https://github.com/kubernetes-sigs/kustomize/blob/master/examples/image.md
- 2: "kustomize edit set image" does not retain image tag information on setting newName kubernetes-sigs/kustomize#4375
- 3: kustomize edit set image --help show "image:" instead of the correct "images:" kubernetes-sigs/kustomize#871
- 4: Unable to change image if it contains a colon kubernetes-sigs/kustomize#221
- 5: Error: invalid format of image for tag with plus (+) in it. kubernetes-sigs/kustomize#3314
- 6: edit set image does not quote tags that will become non-string values in YAML kubernetes-sigs/kustomize#1303
- 7: How to pass a variable through kustomize edit set image command? kubernetes-sigs/kustomize#2475
- 8: Set image tag preserving image name kubernetes-sigs/kustomize#3487
- 9: fix edit set image to parse both tag and digest kubernetes-sigs/kustomize#5234
- 10: kustomize edit set image should correctly parse both tag and digest kubernetes-sigs/kustomize#4713
- 11: https://notes.kodekloud.com/docs/Kustomize/Other-Commands/Edit-CICD-Use-Case
Add quotes around image name and tag variables in kustomize command to prevent shell injection.
The unquoted variables in the command string could allow shell metacharacters to break command parsing or execute unintended code when expanded by bash -c at line 250, even though these come from CI environment variables. The kustomize command itself has strict tag validation, but the injection risk occurs at the shell level before kustomize is invoked. Use:
COMPONENT_OVERRIDE_CMD="cd /installer && kustomize edit set image \"${COMPONENT_IMAGE_NAME}\"=\"${COMPONENT_IMAGE}\" && "
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/step-registry/osac-project/cluster-tool/boot/osac-project-cluster-tool-boot-commands.sh`
around lines 230 - 235, The kustomize image override string built in
COMPONENT_OVERRIDE_CMD uses unquoted expansions of COMPONENT_IMAGE_NAME and
COMPONENT_IMAGE which can allow shell metacharacters to be interpreted when that
string is later executed; update the assignment of COMPONENT_OVERRIDE_CMD so the
image name and tag are shell-quoted (e.g. wrap ${COMPONENT_IMAGE_NAME} and
${COMPONENT_IMAGE} in double quotes inside the command string) to ensure safe
expansion before running the command via bash -c.
| function collect_artifacts() { | ||
| echo "Collecting test artifacts..." | ||
| timeout -s 9 2m scp -F "${SHARED_DIR}/ssh_config" \ | ||
| "ci_machine:${REMOTE_RESULTS_DIR}/junit_vmaas.xml" \ | ||
| "${ARTIFACT_DIR}/junit_vmaas.xml" 2>/dev/null || true | ||
| } | ||
| trap collect_artifacts EXIT |
There was a problem hiding this comment.
Don't silently drop JUnit collection failures.
scp errors are fully suppressed here, so a green run can still publish no XML and lose the per-test reporting this workflow is adding. At least surface the copy failure, and preferably treat missing junit_vmaas.xml as a failed successful run.
Suggested change
function collect_artifacts() {
echo "Collecting test artifacts..."
- timeout -s 9 2m scp -F "${SHARED_DIR}/ssh_config" \
- "ci_machine:${REMOTE_RESULTS_DIR}/junit_vmaas.xml" \
- "${ARTIFACT_DIR}/junit_vmaas.xml" 2>/dev/null || true
+ if ! timeout -s 9 2m scp -F "${SHARED_DIR}/ssh_config" \
+ "ci_machine:${REMOTE_RESULTS_DIR}/junit_vmaas.xml" \
+ "${ARTIFACT_DIR}/junit_vmaas.xml"; then
+ echo "Failed to collect junit_vmaas.xml" >&2
+ return 1
+ fi
}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@ci-operator/step-registry/osac-project/cluster-tool/test/osac-project-cluster-tool-test-commands.sh`
around lines 18 - 24, The collect_artifacts function currently silences all scp
errors so missing junit_vmaas.xml can pass unnoticed; remove the stderr
suppression and silent-true fallback on the scp command, capture its exit
status, and if the copy fails or the target file ${ARTIFACT_DIR}/junit_vmaas.xml
does not exist, print a clear error to stderr and exit non‑zero (so the pipeline
fails). Specifically modify the scp in collect_artifacts (the scp -F
"${SHARED_DIR}/ssh_config" "ci_machine:${REMOTE_RESULTS_DIR}/junit_vmaas.xml"
"${ARTIFACT_DIR}/junit_vmaas.xml") to not redirect 2>/dev/null || true, then
check its exit code and/or test -f "${ARTIFACT_DIR}/junit_vmaas.xml" and call
echo >&2 "ERROR: failed to collect junit_vmaas.xml" and exit 1 when missing.
d6e212a to
4862c3d
Compare
|
/pj-rehearse pull-ci-osac-project-fulfillment-service-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-installer-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-test-infra-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-operator-main-e2e-metal-vmaas-all |
|
@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
4862c3d to
c47e7e0
Compare
|
/pj-rehearse pull-ci-osac-project-fulfillment-service-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-installer-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-test-infra-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-operator-main-e2e-metal-vmaas-all |
|
@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
af0a102 to
8602ce8
Compare
|
/pj-rehearse pull-ci-osac-project-fulfillment-service-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-installer-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-test-infra-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-operator-main-e2e-metal-vmaas-all |
|
@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
8602ce8 to
227fc8e
Compare
|
/pj-rehearse pull-ci-osac-project-fulfillment-service-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-installer-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-test-infra-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-operator-main-e2e-metal-vmaas-all |
|
@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
227fc8e to
948f470
Compare
|
/pj-rehearse pull-ci-osac-project-fulfillment-service-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-installer-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-test-infra-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-operator-main-e2e-metal-vmaas-all |
|
@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
948f470 to
c5f50e0
Compare
|
/pj-rehearse pull-ci-osac-project-fulfillment-service-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-installer-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-test-infra-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-operator-main-e2e-metal-vmaas-all |
|
@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse pull-ci-osac-project-fulfillment-service-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-installer-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-test-infra-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-operator-main-e2e-metal-vmaas-all |
|
@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/pj-rehearse pull-ci-osac-project-osac-test-infra-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-installer-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-operator-main-e2e-metal-vmaas-all pull-ci-osac-project-fulfillment-service-main-e2e-metal-vmaas-all |
|
@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
The previous check for Successful.status=="True" was wrong - during reconciliation it stays False even though reason says Successful (from the pre-snapshot run). The correct signal is Running.message=="Awaiting next reconciliation" which flips from "Running reconciliation" when the operator finishes. Verified on live cluster: Running.message transitions after ~100s. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
/pj-rehearse pull-ci-osac-project-osac-test-infra-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-installer-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-operator-main-e2e-metal-vmaas-all pull-ci-osac-project-fulfillment-service-main-e2e-metal-vmaas-all |
|
@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
Deleting routes and relying on the AAP operator to recreate them is fundamentally unreliable — the operator's Ansible reconciliation takes variable time and may not recreate routes if triggered mid-cycle. Instead, patch each route's spec.host in-place by replacing the old snapshot domain with the new cluster domain. This is instant, deterministic, and eliminates the AAP operator dependency entirely. Verified on live cluster: all 5 routes patched in <1s. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
/pj-rehearse pull-ci-osac-project-osac-test-infra-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-installer-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-operator-main-e2e-metal-vmaas-all pull-ci-osac-project-fulfillment-service-main-e2e-metal-vmaas-all |
|
@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
/retest |
1 similar comment
|
/retest |
The AAP controller API returns 500/503 on slow CI machines when the controller isn't fully ready yet, even though the gateway API responds with 200. Add retry loops to the curl calls that query and launch the publish-templates job template. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
/pj-rehearse pull-ci-osac-project-osac-test-infra-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-installer-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-operator-main-e2e-metal-vmaas-all pull-ci-osac-project-fulfillment-service-main-e2e-metal-vmaas-all |
|
@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
The gather step was missing VM/CNV diagnostics, making it impossible to debug slow VM operations in tests. Now collects: - HyperConverged operator status - VirtualMachine and VirtualMachineInstance state - DataVolume and PVC status - ComputeInstance CRD status - Node CPU/memory usage - openshift-cnv namespace events - AAP operator CR status Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
/pj-rehearse pull-ci-osac-project-osac-test-infra-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-installer-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-operator-main-e2e-metal-vmaas-all pull-ci-osac-project-fulfillment-service-main-e2e-metal-vmaas-all |
|
@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
KubeVirt/CNV is OLM-managed, not a ClusterOperator. After snapshot boot, virt-handler re-deploys and re-registers device plugins (kvm, tun, vhost-net). Until that completes, devices.kubevirt.io/kvm is 0 on the node and all VM scheduling fails with ErrorUnschedulable. Confirmed by comparing failing vs passing CI runs: - Failing: devices.kubevirt.io/kvm=0, HyperConverged Available=False - Passing: devices.kubevirt.io/kvm=1k, HyperConverged Available=True Add a wait in the test step (up to 600s) for HyperConverged Available=True before starting pytest. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
/pj-rehearse pull-ci-osac-project-osac-test-infra-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-installer-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-operator-main-e2e-metal-vmaas-all pull-ci-osac-project-fulfillment-service-main-e2e-metal-vmaas-all |
|
@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
@openshift-merge-bot[bot]: your |
1 similar comment
|
@openshift-merge-bot[bot]: your |
|
/retest |
|
/pj-rehearse pull-ci-osac-project-osac-test-infra-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-installer-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-operator-main-e2e-metal-vmaas-all pull-ci-osac-project-fulfillment-service-main-e2e-metal-vmaas-all |
|
@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
Job spec.template is immutable in Kubernetes. When the snapshot has an old aap-bootstrap Job and the installer submodules update its spec, oc apply fails. Delete all Jobs in the namespace first. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
/pj-rehearse pull-ci-osac-project-osac-test-infra-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-installer-main-e2e-metal-vmaas-all pull-ci-osac-project-osac-operator-main-e2e-metal-vmaas-all pull-ci-osac-project-fulfillment-service-main-e2e-metal-vmaas-all |
|
@omer-vishlitzky: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
[REHEARSALNOTIFIER]
Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
@omer-vishlitzky: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
2 participants
Summary
Replaces 32 vmaas presubmit jobs (8 per repo × 4 repos) and 8 periodic jobs with 4 presubmit jobs (1 per repo) + 1 periodic.
Each new job boots a pre-built OSAC cluster from a Quay snapshot via cluster-tool (~15min), runs a refresh script to fix domain-sensitive resources (~4min), then executes all 8 vmaas tests sequentially on the single cluster.
Before: 8 jobs per PR, each provisioning a full OCP cluster + OSAC install (~2.5h each)
After: 1 job per PR, booting from snapshot (~30min total)
What changed
osac-project-cluster-tool-vmaas: ofcir-acquire → cluster-tool boot+refresh → all 8 vmaas tests → gather+destroy+releasecluster-tool-boot(setup machine, pull flavor, boot, refresh),cluster-tool-test(pytest with JUnit XML),cluster-tool-destroy(best-effort cleanup)osac-project-ofcir-baremetalandosac-project-ofcir-baremetal-componentworkflows, old installer/test stepsRepos affected
Dependencies
quay.io/rh-ee-ovishlit/cluster-flavors:osac-vmaasprepare-fulfillment-service.share embedded (from osac-installer PR #95, not yet merged). Once merged, the embedded copies can be removed.Jira: https://issues.redhat.com/browse/MGMT-24219
Summary
This PR consolidates OSAC vmaas CI across four repositories (osac-test-infra, osac-installer, fulfillment-service, osac-operator). It replaces the previous per-scenario OFCIR/baremetal provisioning + test jobs with a single cluster-tool based workflow that boots a pre-built cluster snapshot, refreshes domain-sensitive resources, runs all vmaas scenarios sequentially on that cluster, gathers artifacts, destroys the clone, and releases OFCIR resources.
Practical impact
New CI artifacts added
Removed/changed pieces
Repo-specific outcomes
Dependencies and notes
This change centralizes and simplifies OSAC vmaas CI maintenance while preserving per-scenario test diagnostics in Prow.