Skip to content
This repository was archived by the owner on Sep 9, 2026. It is now read-only.

Add refresh-after-snapshot script for cluster-tool clones - #95

Merged
openshift-merge-bot[bot] merged 1 commit into
osac-project:mainfrom
omer-vishlitzky:add-refresh-after-snapshot
May 13, 2026
Merged

openshift-merge-bot[bot] merged 1 commit into
osac-project:mainfrom
omer-vishlitzky:add-refresh-after-snapshot

Conversation

@omer-vishlitzky

@omer-vishlitzky omer-vishlitzky commented May 7, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds scripts/refresh-after-snapshot.sh — refreshes all domain-sensitive OSAC resources after booting a cluster from a snapshot via cluster-tool
  • Makes prepare-fulfillment-service.sh idempotent (deletes existing hub before creating)

Context

cluster-tool boots OpenShift SNO clusters from golden snapshots in ~6 minutes via recert-based identity regeneration. After boot, the cluster has a new domain but OSAC application resources (routes, AAP configs, hub kubeconfig, fulfillment controller) still reference the old domain.

This script handles all the domain-sensitive cleanup:

  1. Deletes stale routes (both kustomize-managed and AAP-operator-managed)
  2. Re-applies kustomize overlay (recreates fulfillment-api route with correct domain)
  3. Triggers AAP operator reconciliation via annotation (recreates AAP routes with correct domain)
  4. Waits for AAP controller to be healthy
  5. Recreates AAP API token and sets OSAC_AAP_URL
  6. Regenerates hub kubeconfig and re-registers hub
  7. Restarts fulfillment pods to pick up new hub config
  8. Ensures tenant exists

Usage

cluster-tool boot --flavor osac-64-post-setup --name my-test
export KUBECONFIG=~/.kube/my-test.kubeconfig
INSTALLER_KUSTOMIZE_OVERLAY=vmaas-ci INSTALLER_VM_TEMPLATE=osac.templates.ocp_virt_vm ./scripts/refresh-after-snapshot.sh

Total time: ~10 minutes (6 min boot + 4 min refresh) vs 2+ hours from scratch.

Test plan

  • Boot from osac-64-post-setup snapshot
  • Run refresh-after-snapshot.sh — all 9 steps pass
  • Run make test-vmaas — 8/8 tests pass
  • Verified no stale domain references in routes, configmaps, secrets, or fulfillment logs

Summary by CodeRabbit

  • New Features

    • Added a refresh workflow to re-sync the deployment stack after snapshot boots so components and routes pick up the current cluster domain and configuration.
  • Chores

    • Updated fulfillment setup to delete and recreate the hub during initialization to ensure a clean registration state.

@coderabbitai

coderabbitai Bot commented May 7, 2026

Copy link
Copy Markdown

Warning

Rate limit exceeded

@omer-vishlitzky has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 24 minutes and 56 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 908470d9-0f1c-4112-abdb-31241ee39353

📥 Commits

Reviewing files that changed from the base of the PR and between 20e2eb1 and 8f5707b.

📒 Files selected for processing (2)
  • scripts/prepare-fulfillment-service.sh
  • scripts/refresh-after-snapshot.sh

Walkthrough

This PR modifies the hub registration flow in prepare-fulfillment-service.sh to delete and recreate the hub, ensuring a clean state with the current kubeconfig. It introduces refresh-after-snapshot.sh, a new orchestration script that recovers an OpenShift cluster after snapshot boots when the domain may have changed. The script systematically refreshes routes, reapplies overlays, forces AAP operator reconciliation, verifies controller and gateway readiness, reinitializes AAP and fulfillment services with the updated domain, restarts deployments, and ensures tenant provisioning.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related PRs

Suggested labels

lgtm

Suggested reviewers

  • tzumainn
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Add refresh-after-snapshot script for cluster-tool clones' directly and specifically describes the main change: adding a new script. It is clear, concise, and accurately reflects the primary addition in the changeset.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/prepare-fulfillment-service.sh`:
- Line 21: The unconditional osac delete hub hub call causes the script to fail
under set -o errexit when the hub is absent; update the script to guard the
deletion by checking for the hub's existence (or run the delete and ignore the
specific "not found" failure) before calling osac delete hub hub so the command
is safe and idempotent. Locate the osac delete hub hub invocation in the script
and either wrap it with an existence test using the appropriate osac query/list
command, or execute it conditionally and suppress or handle the exit code 4 (Not
found) so the script does not abort when the hub is already absent. Ensure
existing errexit behavior remains for other failures.

In `@scripts/refresh-after-snapshot.sh`:
- Around line 101-105: The script restarts four fulfillment deployments but only
waits for two; add oc rollout status checks for both fulfillment-rest-gateway
and fulfillment-ingress-proxy (using the same --timeout=120s and -n
"${INSTALLER_NAMESPACE}" pattern) after you restart them so the script blocks
until all four rollouts are complete (keep the existing oc rollout restart lines
for fulfillment-rest-gateway and fulfillment-ingress-proxy and mirror the status
call pattern used for fulfillment-controller and fulfillment-grpc-server).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: c6e5f6ee-b5a5-42da-a0f9-2215c68181a2

📥 Commits

Reviewing files that changed from the base of the PR and between 7d95258 and 5d6b7e9.

📒 Files selected for processing (2)
  • scripts/prepare-fulfillment-service.sh
  • scripts/refresh-after-snapshot.sh

# Login to fulfillment API and ensure hub exists with current kubeconfig
FULFILLMENT_API_URL=https://$(oc get route -n ${INSTALLER_NAMESPACE} fulfillment-api -o jsonpath='{.status.ingress[0].host}')
osac login --insecure --private --token-script "oc create token -n ${INSTALLER_NAMESPACE} admin" --address ${FULFILLMENT_API_URL}
osac delete hub hub

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

For the OSAC CLI, what is the exit-code behavior of osac delete hub when the hub does not exist, and is there an--ignore-not-found (or equivalent) flag?

💡 Result:

No public documentation or sources were found describing an "OSAC CLI" or the specific command osac delete hub <id>. Searches returned results for OSCAL CLI (oscal-cli), OCI OS Management Hub CLI (oci os-management-hub... delete), and osac-project/fulfillment-cli (a "Cloud-in-a-box fulfillment CLI" for managing infrastructure like clusters and hosts via gRPC, but its README mentions general delete commands without specifics on delete hub or exit codes [1][2]). No explicit mention of exit code behavior when the hub does not exist, nor an --ignore-not-found flag or equivalent. The command may be internal, unpublished, or from a private project under osac-project (e.g., Red Hat-related, given GitHub patterns [3]). OCI delete commands use --force for confirmation skipping but do not address not-found cases explicitly [4][5].

Citations:


🏁 Script executed:

#!/bin/bash
# Search for the script file mentioned in the review
find . -name "prepare-fulfillment-service.sh" -type f 2>/dev/null

Repository: osac-project/osac-installer

Length of output: 112


🏁 Script executed:

#!/bin/bash
# Search for any osac-related documentation or code
find . -type f \( -name "*osac*" -o -name "*delete*hub*" \) 2>/dev/null | head -20

Repository: osac-project/osac-installer

Length of output: 291


🏁 Script executed:

#!/bin/bash
# Look for documentation about osac command
find . -name "README*" -o -name "*.md" -type f 2>/dev/null | xargs grep -l "osac" 2>/dev/null | head -10

Repository: osac-project/osac-installer

Length of output: 179


🏁 Script executed:

#!/bin/bash
# Read the script to see the current state at line 21
cat -n ./scripts/prepare-fulfillment-service.sh | head -30

Repository: osac-project/osac-installer

Length of output: 1935


🏁 Script executed:

#!/bin/bash
# Read the OSAC CLI documentation
cat ./OSAC-CLI-HOWTO.md

Repository: osac-project/osac-installer

Length of output: 50385


🏁 Script executed:

#!/bin/bash
# Search for delete command documentation and exit behavior
grep -n -A 10 -B 5 "osac delete" ./OSAC-CLI-HOWTO.md | head -100

Repository: osac-project/osac-installer

Length of output: 2638


🏁 Script executed:

#!/bin/bash
# Search for exit code behavior
grep -n -i "exit.*code\|exit.*status" ./OSAC-CLI-HOWTO.md

Repository: osac-project/osac-installer

Length of output: 94


🏁 Script executed:

#!/bin/bash
# Search for --ignore or --force flags
grep -n -E "\-\-ignore|\-\-force" ./OSAC-CLI-HOWTO.md

Repository: osac-project/osac-installer

Length of output: 159


🏁 Script executed:

#!/bin/bash
# Check the API Reference section for delete command signature
sed -n '/## API Reference/,/^## /p' ./OSAC-CLI-HOWTO.md | grep -A 20 "delete hub"

Repository: osac-project/osac-installer

Length of output: 495


🏁 Script executed:

#!/bin/bash
# Read the Exit Codes section
sed -n '1668,1690p' ./OSAC-CLI-HOWTO.md

Repository: osac-project/osac-installer

Length of output: 614


🏁 Script executed:

#!/bin/bash
# Search for "not found" or similar error messages
grep -n -i "not found\|does not exist\|not exist" ./OSAC-CLI-HOWTO.md

Repository: osac-project/osac-installer

Length of output: 166


🏁 Script executed:

#!/bin/bash
# Look for error behavior in delete command documentation
sed -n '/## API Reference/,/^##/p' ./OSAC-CLI-HOWTO.md | grep -A 30 "delete hub"

Repository: osac-project/osac-installer

Length of output: 53


Guard hub deletion to preserve idempotency.

With set -o errexit, an unconditional osac delete hub hub exits with code 4 ("Not found error") when the hub is absent, which aborts the script and breaks the idempotent flow this PR targets. The --force flag exists but is for bypassing confirmation prompts, not error handling.

Suggested fix
-osac delete hub hub
+osac delete hub hub || true
 osac create hub --kubeconfig=/tmp/kubeconfig.hub-access --id hub --namespace ${INSTALLER_NAMESPACE}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
osac delete hub hub
osac delete hub hub || true
osac create hub --kubeconfig=/tmp/kubeconfig.hub-access --id hub --namespace ${INSTALLER_NAMESPACE}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/prepare-fulfillment-service.sh` at line 21, The unconditional osac
delete hub hub call causes the script to fail under set -o errexit when the hub
is absent; update the script to guard the deletion by checking for the hub's
existence (or run the delete and ignore the specific "not found" failure) before
calling osac delete hub hub so the command is safe and idempotent. Locate the
osac delete hub hub invocation in the script and either wrap it with an
existence test using the appropriate osac query/list command, or execute it
conditionally and suppress or handle the exit code 4 (Not found) so the script
does not abort when the hub is already absent. Ensure existing errexit behavior
remains for other failures.

Comment thread scripts/refresh-after-snapshot.sh
@omer-vishlitzky
omer-vishlitzky force-pushed the add-refresh-after-snapshot branch from 5d6b7e9 to 5230a29 Compare May 7, 2026 16:34
@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/retest

@omer-vishlitzky
omer-vishlitzky force-pushed the add-refresh-after-snapshot branch from 5230a29 to 20e2eb1 Compare May 7, 2026 18:26

@akshaynadkarni akshaynadkarni left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (1)
scripts/prepare-fulfillment-service.sh (1)

24-24: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Guard hub deletion to keep idempotency under errexit (Line 24).

osac delete hub hub can terminate the script when the hub is already absent, which breaks the idempotent flow this PR is aiming for.

Suggested minimal fix
-osac delete hub hub
+osac delete hub hub || true
 osac create hub --kubeconfig=/tmp/kubeconfig.hub-access --id hub --namespace ${INSTALLER_NAMESPACE}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/prepare-fulfillment-service.sh` at line 24, The direct call to "osac
delete hub hub" fails under errexit when the hub is already missing; change it
to a guarded delete by first checking for the hub's existence (e.g., call "osac
get hub hub" or equivalent) and only run "osac delete hub hub" if the check
succeeds, ensuring the delete is a no-op when the hub is absent and preserving
idempotency under errexit.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Duplicate comments:
In `@scripts/prepare-fulfillment-service.sh`:
- Line 24: The direct call to "osac delete hub hub" fails under errexit when the
hub is already missing; change it to a guarded delete by first checking for the
hub's existence (e.g., call "osac get hub hub" or equivalent) and only run "osac
delete hub hub" if the check succeeds, ensuring the delete is a no-op when the
hub is absent and preserving idempotency under errexit.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 93cfd2ff-a4f1-49a5-8078-dcc7226d6ca9

📥 Commits

Reviewing files that changed from the base of the PR and between 5d6b7e9 and 20e2eb1.

📒 Files selected for processing (2)
  • scripts/prepare-fulfillment-service.sh
  • scripts/refresh-after-snapshot.sh

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/retest

2 similar comments
@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/retest

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/retest

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/hold

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/retest

When booting an OSAC cluster from a snapshot (via cluster-tool), the
cluster gets a new domain but application-level resources still reference
the old one. This script refreshes all domain-sensitive resources:

- Deletes and recreates routes (kustomize-managed and AAP-managed)
- Triggers AAP operator reconciliation to regenerate configs
- Waits for AAP controller to be healthy
- Recreates AAP API token with correct URL
- Re-registers hub with fresh kubeconfig
- Restarts fulfillment pods to pick up new hub config
- Ensures tenant exists

Also makes prepare-fulfillment-service.sh idempotent by deleting the
existing hub before creating (osac delete hub returns 0 if not found).
@omer-vishlitzky
omer-vishlitzky force-pushed the add-refresh-after-snapshot branch from 20e2eb1 to 8f5707b Compare May 12, 2026 20:09
@openshift-ci openshift-ci Bot removed the lgtm label May 12, 2026
@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/unhold

exit 1
}
AAP_ROUTE_HOST=$(oc get route osac-aap -n "${INSTALLER_NAMESPACE}" -o jsonpath='{.spec.host}')
retry_until 120 5 '[[ "$(curl -sk -o /dev/null -w %{http_code} https://'"${AAP_ROUTE_HOST}"'/api/gateway/v1/)" == "200" ]]' || {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you want there's an healthcheck endpoint that expose the health of each components

echo "[6/8] Configuring fulfillment service..."
./scripts/prepare-fulfillment-service.sh

echo "[7/8] Restarting fulfillment pods..."

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't we have a common label ? so we don't rely on the name of the Deployments as the fulfilment-service might change again in the future.

@openshift-ci openshift-ci Bot added the lgtm label May 13, 2026
@openshift-ci

openshift-ci Bot commented May 13, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: adriengentil, akshaynadkarni, omer-vishlitzky

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [adriengentil,akshaynadkarni,omer-vishlitzky]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@omer-vishlitzky

Copy link
Copy Markdown
Contributor Author

/hold

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants