Skip to content
This repository was archived by the owner on Sep 9, 2026. It is now read-only.

OSAC-1155: fix CI boot failures in refresh script - #212

Merged
openshift-merge-bot[bot] merged 2 commits into
osac-project:mainfrom
omer-vishlitzky:OSAC-1155-fix-boot-failures
Jun 2, 2026
Merged

openshift-merge-bot[bot] merged 2 commits into
osac-project:mainfrom
omer-vishlitzky:OSAC-1155-fix-boot-failures

Conversation

@omer-vishlitzky

@omer-vishlitzky omer-vishlitzky commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

Summary

https://redhat.atlassian.net/browse/OSAC-1155

Follow-up to PR #206. Addresses remaining boot failure patterns found in post-merge CI analysis:

  • MetalLB webhook TLS: Wait for Service endpoints (not just pod readiness) before patching IPAddressPool. Fixes race where pod is Ready but Service has no endpoints yet.
  • AAP publish-templates: Wait for AAP project sync before launching job. Log HTTP error body on launch failure (was silently swallowed by 2>/dev/null). Fixes Missing a revision to run due to failed project update errors.
  • CDI rollout retry: Wrap oc rollout status calls with retry_command to handle transient API server timeouts under load. Run CDI deployment waits in parallel instead of sequentially.

Test plan

  • bash -n syntax check passes on both scripts
  • CI e2e-vmaas passes
  • Monitor ci-obs for reduced BOOT failure rates

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Improvements
    • Waits for controller API and associated project sync before launching publish jobs.
    • Hardened job launch flow with stronger validation, logging of failure reasons, and bounded retry behavior (fails after 10 attempts if unresolved).
    • More robust readiness checks using concurrent waits and endpoint-based validation for load‑balancer webhook.
    • Updated pull-secret path resolution.

@openshift-ci-robot

openshift-ci-robot commented Jun 1, 2026

Copy link
Copy Markdown

@omer-vishlitzky: This pull request references OSAC-1155 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

https://redhat.atlassian.net/browse/OSAC-1155

Follow-up to PR #206. Addresses remaining boot failure patterns found in post-merge CI analysis:

  • MetalLB webhook TLS: Wait for Service endpoints (not just pod readiness) before patching IPAddressPool. Fixes race where pod is Ready but Service has no endpoints yet.
  • AAP publish-templates: Wait for AAP project sync before launching job. Log HTTP error body on launch failure (was silently swallowed by 2>/dev/null). Fixes Missing a revision to run due to failed project update errors.
  • CDI rollout retry: Wrap oc rollout status calls with retry_command to handle transient API server timeouts under load. Run CDI deployment waits in parallel instead of sequentially.

Test plan

  • bash -n syntax check passes on both scripts
  • CI e2e-vmaas passes
  • Monitor ci-obs for reduced BOOT failure rates

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci
openshift-ci Bot requested review from adriengentil and eranco74 June 1, 2026 11:32
@openshift-ci openshift-ci Bot added the approved label Jun 1, 2026
@coderabbitai

coderabbitai Bot commented Jun 1, 2026

Copy link
Copy Markdown

Review Change Stack

Walkthrough

Two deployment scripts are hardened: the AAP fulfillment launcher now waits for job template and project sync and adds robust launch retries; the post-snapshot refresh script resolves the pull-secret path and improves CDI and MetalLB readiness and application flows.

Changes

Deployment Script Hardening

Layer / File(s) Summary
AAP fulfillment service launch readiness and error handling
scripts/prepare-fulfillment-service.sh
Polls for the osac-publish-templates job template and its project sync status before launching; job launch retry loop captures stderr to a temp file, validates returned job id (non-empty and not "null"), logs .playbook[0] or .detail reason when present, and fails after configured attempts.
Post-snapshot refresh credential and operator readiness
scripts/refresh-after-snapshot.sh
PULL_SECRET now resolves from SCRIPT_DIR/../overlays/${INSTALLER_KUSTOMIZE_OVERLAY}/files/quay-pull-secret.json; refresh_cdi_certificates restarts CDI components and waits using parallel retry_command background waits; MetalLB webhook readiness now requires at least one endpoint IP; generated IPAddressPool YAML is written to a temp file and applied via retry_command.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Risk severity: Low–Moderate — changes affect deployment readiness and retry behavior; incorrect timeouts or checks could cause delayed or failed rollouts, but fixes improve error detection and reduce silent failure modes.

Possibly related PRs

Suggested reviewers

  • adriengentil
  • eranco74

Poem

🔧 Scripts sharper, checks aligned,
Jobs await projects well-defined,
CDI wakes in parallel grace,
MetalLB finds its endpoint place,
Retries speak truth — failures now signed.


Caution

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

  • Ignore

❌ Failed checks (1 error, 2 warnings)

Check name Status Explanation Resolution
No-Sensitive-Data-In-Logs ❌ Error PR logs HTTP response .detail field from AAP API and PULL_SECRET file path if not found, potentially exposing error details and credential file locations. Remove logging of .detail field; sanitize error messages. Avoid logging full paths to secrets files.
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Ai-Attribution ⚠️ Warning PR uses Co-Authored-By for AI (Claude Opus 4.6) in commit 798ce6c, violating Red Hat attribution policy; should use Assisted-by/Generated-by instead. Update commit 798ce6c message to use 'Assisted-by: Claude Opus 4.6' or 'Generated-by: Claude Opus 4.6' trailer instead of Co-Authored-By for AI tools.
✅ Passed checks (8 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title directly references OSAC-1155 and accurately describes the main objective: fixing CI boot failures in the refresh script, which aligns with the changes made to both prepare-fulfillment-service.sh and refresh-after-snapshot.sh.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
No-Hardcoded-Secrets ✅ Passed No hardcoded secrets introduced. AAP_TOKEN retrieved from Kubernetes secret; Keycloak admin/admin pre-existed in main; all changes implement operational reliability without introducing credentials.
No-Weak-Crypto ✅ Passed No weak cryptography (MD5/SHA1/DES/RC4/3DES/Blowfish/ECB), custom crypto implementations, or insecure token comparisons introduced in PR changes; modifications focus on orchestration logic.
No-Injection-Vectors ✅ Passed Check targets Python/SQL/JS patterns (eval, pickle, yaml.load, os.system, dangerouslySetInnerHTML). PR modifies only bash shell scripts; none of these patterns appear.
Container-Privileges ✅ Passed PR modifies shell scripts only. No privileged containers, hostPID/Network/IPC, SYS_ADMIN capabilities, or security context escalations introduced in changes.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/hold

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/prepare-fulfillment-service.sh`:
- Around line 50-57: The check that waits for a valid AAP project revision is a
no-op because jq's '.current_update // empty' never outputs the literal string
"null", so the guard in the retry_until call returns immediately; update the
retry condition in the block that references PROJECT_ID, retry_until, AAP_TOKEN,
AAP_URL and .current_update to test for a non-empty value instead (e.g., ensure
the curl|jq pipeline yields a non-empty string) so retry_until only succeeds
when .current_update is present and non-empty; keep the same retry_until usage
and error/warning handling but change the jq expression or the comparison to
assert non-empty output rather than comparing to the literal "null".
- Around line 61-69: The loop creates LAUNCH_ERR with mktemp on every attempt
but never reads or deletes it; change the launch logic (where LAUNCH_ERR,
JOB_RESPONSE, mktemp, and curl are used) to capture curl's stderr into a single
temp file or a variable, read and print its contents when curl fails or when
JOB_RESPONSE lacks an expected id, and always rm the temp file after each
iteration (use a trap or explicit rm in both success and failure branches) to
avoid leaking /tmp files and to surface connection-level diagnostics alongside
JOB_RESPONSE and HTTP_BODY.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: osac-project/coderabbit/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 14790b3b-b19b-48ce-a2b0-8dc2d736d54a

📥 Commits

Reviewing files that changed from the base of the PR and between 937d369 and 0cbc2f2.

📒 Files selected for processing (2)
  • scripts/prepare-fulfillment-service.sh
  • scripts/refresh-after-snapshot.sh

Comment thread scripts/prepare-fulfillment-service.sh
Comment thread scripts/prepare-fulfillment-service.sh
@omer-vishlitzky
omer-vishlitzky force-pushed the OSAC-1155-fix-boot-failures branch from 0cbc2f2 to 471261e Compare June 1, 2026 11:46
@openshift-ci openshift-ci Bot removed the lgtm label Jun 1, 2026
MetalLB webhook TLS: rotate cert and wait for Service endpoints
before patching IPAddressPool.
Image preflight: verify ghcr.io/osac-project images exist after
kustomize apply. Fail immediately if missing.
AAP publish-templates: wait for project sync before launching job,
log HTTP errors, wait for controller-task deployment stability.
CDI rollout: retry on API server timeouts, run waits in parallel.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@omer-vishlitzky
omer-vishlitzky force-pushed the OSAC-1155-fix-boot-failures branch from 471261e to b03668c Compare June 1, 2026 12:19
@openshift-ci openshift-ci Bot added the lgtm label Jun 1, 2026
@openshift-ci openshift-ci Bot removed the lgtm label Jun 1, 2026
@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/retest

1 similar comment
@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/retest

@openshift-ci

openshift-ci Bot commented Jun 2, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danmanor, jhernand, omer-vishlitzky

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [danmanor,jhernand,omer-vishlitzky]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/retest

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/unhold

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/override ci/prow/images ci/prow/e2e-vmaas

@openshift-ci

openshift-ci Bot commented Jun 2, 2026

Copy link
Copy Markdown

@omer-vishlitzky: Overrode contexts on behalf of omer-vishlitzky: ci/prow/e2e-vmaas, ci/prow/images

Details

In response to this:

/override ci/prow/images ci/prow/e2e-vmaas

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-merge-bot
openshift-merge-bot Bot merged commit 1d64ef7 into osac-project:main Jun 2, 2026
9 checks passed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants