Add operator-chaos upgrade analysis workflow

[HumairAK]

Summary

This adds a repo-local operator-chaos knowledge model for MLflow and a PR workflow that:

• validates the knowledge file
• runs operator-chaos preflight --local
• diffs the base and PR knowledge models
• diffs the checked-in MLflow CRD
• previews simulate-upgrade --dry-run

The workflow now fails fast on breaking knowledge or CRD changes, and intentionally uses the workflow logs as the review surface instead of a separate PR comment flow.

Test plan

• go run ./cmd/operator-chaos validate --knowledge /home/hukhan/projects/github/rhoai/mlflow-operator/chaos/knowledge/mlflow.yaml
• go run ./cmd/operator-chaos preflight --knowledge /home/hukhan/projects/github/rhoai/mlflow-operator/chaos/knowledge/mlflow.yaml --local
• go run ./cmd/operator-chaos diff --source /home/hukhan/projects/github/rhoai/mlflow-operator/chaos/knowledge --target /home/hukhan/projects/github/rhoai/mlflow-operator/chaos/knowledge
• go run ./cmd/operator-chaos diff-crds --source-crds /home/hukhan/projects/github/rhoai/mlflow-operator/config/crd/bases --target-crds /home/hukhan/projects/github/rhoai/mlflow-operator/config/crd/bases
• go run ./cmd/operator-chaos simulate-upgrade --source /home/hukhan/projects/github/rhoai/mlflow-operator/chaos/knowledge --target /home/hukhan/projects/github/rhoai/mlflow-operator/chaos/knowledge --dry-run
• go run github.com/rhysd/actionlint/cmd/actionlint@f5adaf2a03a0a2557568caae2546fa04a05460dc .github/workflows/operator-chaos.yml
• bash -n .github/scripts/operator-chaos-summary.sh
• bash -n .github/scripts/operator-chaos-pr-comment.sh

Summary by CodeRabbit

• New Features

  • Added shift-left upgrade validation that detects breaking changes in operator configurations and CRD schemas during pull requests, preventing incompatible changes from being merged.

• Documentation

  • Updated documentation to explain the upgrade validation workflow and requirements for maintaining compatibility with the stable operator topology.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: f8c1efa6-e40f-4a30-9abb-a5adc333f115

📥 Commits

Reviewing files that changed from the base of the PR and between 9b05cb9 and c578b38.

📒 Files selected for processing (4)

• .github/workflows/operator-chaos.yml
• AGENTS.md
• README.md
• chaos/knowledge/mlflow.yaml

---

📝 Walkthrough

Walkthrough

This PR introduces a shift-left upgrade validation gate for the MLflow operator. It adds a chaos/knowledge/mlflow.yaml file that declares the operator's managed resources and reconciliation topology, then implements a GitHub workflow (.github/workflows/operator-chaos.yml) that uses the operator-chaos CLI to validate the knowledge model, detect breaking changes in CRD schemas between base and PR branches, and simulate upgrade scenarios. The workflow installs a pinned operator-chaos binary and runs validation/diff/dry-run steps that fail fast on incompatibilities. Supporting documentation in AGENTS.md and README.md explains the offline validation approach and maintenance expectations.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~22 minutes

---

Security findings

Pinned operator-chaos CLI binary (range_2a4fcc44e103, range_0676aa583abd)
The workflow pins OPERATOR_CHAOS_VERSION by commit hash (9e6ac9668b9aaca2f0f2ddf169867862b7925b80) and installs via go install directly from a remote repository. Verify:

• The operator-chaos repository is maintained and security-reviewed.
• The commit hash is from an authenticated, reviewed source (not an unsigned fork).
• No MITM exposure during go install fetch; ensure module proxy is trusted or pinned via go.sum.

Sparse checkout of base branch (range_2a4fcc44e103)
The workflow performs a second checkout of the PR base SHA using actions/checkout@v4 with sparse-checkout. Confirm that the sparse paths (chaos/knowledge, charts/mlflow/*/values.yaml) are correctly isolated and do not accidentally include unintended files that could influence diff results.

Breaking change detection logic (range_b20c2bc91c34, range_3e9c529a168f)
Both knowledge and CRD diffs rely on jq expressions to parse JSON and count breaking changes, then exit with code 1 if any are found. The jq expressions should be validated:

• In range_b20c2bc91c34, confirm .summary.breakingChanges field exists in all operator-chaos diff output formats and is numeric.
• In range_3e9c529a168f, the reduce expression filters schemaChanges[].severity for "breaking" strings; verify operator-chaos maintains this schema contract and does not change field names in future versions.

Exit behavior & workflow failure semantics
Ensure that exit 1 in a script step correctly propagates workflow failure status. GitHub Actions interprets non-zero exit codes as step failure; verify no subsequent steps run unintended if an earlier validation fails.

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically summarizes the main change: adding an operator-chaos upgrade analysis workflow for the MLflow operator.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/operator-chaos.yml:
- Around line 150-159: The "Update PR comment" step can fail with HTTP 403 for
forked PRs because the reduced GITHUB_TOKEN is read-only; modify the step (named
"Update PR comment") to skip execution when the PR is from a fork by guarding it
with an if condition that checks github.event.pull_request.head.repo.fork (e.g.,
only run when that value is false), keep the existing env (GH_TOKEN, PR_NUMBER)
and the call to .github/scripts/operator-chaos-pr-comment.sh unchanged so
non-fork PRs still update the comment.
- Around line 14-16: The workflow-level permissions block should be moved to the
specific job to follow least-privilege; remove the top-level "permissions:"
block and add the same permissions under the job named "operator-chaos"
(jobs.operator-chaos.permissions) so only that job gets contents: read and
issues: write instead of applying to all jobs.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 220ece1d-aafc-4fde-9e2f-566c043b21fe

📥 Commits

Reviewing files that changed from the base of the PR and between c6974e7 and f76b666.

📒 Files selected for processing (6)

• .github/scripts/operator-chaos-pr-comment.sh
• .github/scripts/operator-chaos-summary.sh
• .github/workflows/operator-chaos.yml
• AGENTS.md
• README.md
• chaos/knowledge/mlflow.yaml

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (2)

.github/workflows/operator-chaos-pr-comment.yml (2)

129-136: 💤 Low value

Add if: always() so PR comment is posted even when upstream steps fail.

If "Write operator-chaos summary" fails unexpectedly, this step is skipped and the PR gets no feedback. Match the pattern used for "Fail on operator-chaos errors" (line 139).

Suggested change

       - name: Update PR comment
+        if: always()
         env:
           GH_TOKEN: ${{ github.token }}

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/operator-chaos-pr-comment.yml around lines 129 - 136, The
"Update PR comment" workflow step is currently skipped when earlier steps fail;
add the conditional so it always runs by inserting if: always() under the step
name to match the "Fail on operator-chaos errors" pattern. Update the step with
name "Update PR comment" (the one that runs bash
.github/scripts/operator-chaos-pr-comment.sh) to include if: always() so the PR
comment is posted even when "Write operator-chaos summary" or other upstream
steps fail.

---

95-101: ⚡ Quick win

Version drift: hardcoded SHA diverges from OPERATOR_CHAOS_VERSION env var in the other workflow.

Line 100 hardcodes 9e6ac9668b9aaca2f0f2ddf169867862b7925b80 while .github/workflows/operator-chaos.yml defines this as env.OPERATOR_CHAOS_VERSION. Future updates will require changing both files, risking inconsistency.

Define a shared env var or anchor this to a single source of truth.

Suggested change

+env:
+  OPERATOR_CHAOS_VERSION: "9e6ac9668b9aaca2f0f2ddf169867862b7925b80"
+
 jobs:
   update-pr-comment:

Then at line 100:

-          GOBIN="${RUNNER_TEMP}/bin" go install "github.com/opendatahub-io/operator-chaos/cmd/operator-chaos@9e6ac9668b9aaca2f0f2ddf169867862b7925b80"
+          GOBIN="${RUNNER_TEMP}/bin" go install "github.com/opendatahub-io/operator-chaos/cmd/operator-chaos@${OPERATOR_CHAOS_VERSION}"

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/operator-chaos-pr-comment.yml around lines 95 - 101, The
workflow hardcodes the operator-chaos commit SHA in the go install command (the
literal "9e6ac9...b80") which diverges from the OPERATOR_CHAOS_VERSION env var
used elsewhere; update the run step so it uses the shared OPERATOR_CHAOS_VERSION
environment variable instead of the hardcoded SHA (adjust the GOBIN/go install
line that references the operator-chaos module), or move OPERATOR_CHAOS_VERSION
into this workflow's env and reference it consistently to keep a single source
of truth.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/scripts/operator-chaos-run-checks.sh:
- Around line 88-99: The script runs bare operator-chaos diff / diff-crds
commands that may exit non-zero and trigger set -e termination (e.g., the
unguarded JSON captures that write to knowledge-diff.json and similar for CRDs);
change these to be guarded so failures don't short-circuit the script: either
run them via the existing run_capture wrapper (use run_capture "VAR" path
operator-chaos ...) or execute the command in a way that preserves its
stdout/stderr to the JSON file but prevents set -e from exiting (capture the
exit code with || true or via a subshell and then record that exit code into a
variable), and ensure the downstream logic uses the guarded exit code rather
than allowing the bare command to abort execution (apply same fix to the
BASE_KNOWLEDGE_FILE block, the JSON redirect to
"${OUTPUT_DIR}/knowledge-diff.json", and the corresponding diff-crds JSON
capture).

In @.github/workflows/operator-chaos-pr-comment.yml:
- Around line 22-26: The workflow currently unsafely indexes
github.event.workflow_run.pull_requests[0] (used in the concurrency group name
and PR_NUMBER) which can be empty; replace that pattern by resolving the PR
number via a lookup using github.event.workflow_run.head_sha (e.g., call the
GitHub REST/GraphQL API to find the PR whose head.sha matches) store the result
into a workflow output/variable (e.g., PR_NUMBER) and use that variable in the
concurrency group name (operator-chaos-pr-comment-${{ env.PR_NUMBER }} or
similar) and anywhere else instead of pull_requests[0]; remove all direct
references to pull_requests[0] so the job no longer assumes the array is
populated.

---

Nitpick comments:
In @.github/workflows/operator-chaos-pr-comment.yml:
- Around line 129-136: The "Update PR comment" workflow step is currently
skipped when earlier steps fail; add the conditional so it always runs by
inserting if: always() under the step name to match the "Fail on operator-chaos
errors" pattern. Update the step with name "Update PR comment" (the one that
runs bash .github/scripts/operator-chaos-pr-comment.sh) to include if: always()
so the PR comment is posted even when "Write operator-chaos summary" or other
upstream steps fail.
- Around line 95-101: The workflow hardcodes the operator-chaos commit SHA in
the go install command (the literal "9e6ac9...b80") which diverges from the
OPERATOR_CHAOS_VERSION env var used elsewhere; update the run step so it uses
the shared OPERATOR_CHAOS_VERSION environment variable instead of the hardcoded
SHA (adjust the GOBIN/go install line that references the operator-chaos
module), or move OPERATOR_CHAOS_VERSION into this workflow's env and reference
it consistently to keep a single source of truth.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: a5e6d9af-6905-4e10-98de-b137c52073af

📥 Commits

Reviewing files that changed from the base of the PR and between f76b666 and 4fd4adf.

📒 Files selected for processing (8)

• .github/scripts/operator-chaos-pr-comment.sh
• .github/scripts/operator-chaos-run-checks.sh
• .github/scripts/operator-chaos-summary.sh
• .github/workflows/operator-chaos-pr-comment.yml
• .github/workflows/operator-chaos.yml
• AGENTS.md
• README.md
• chaos/knowledge/mlflow.yaml

🚧 Files skipped from review as they are similar to previous changes (2)

• .github/scripts/operator-chaos-pr-comment.sh
• chaos/knowledge/mlflow.yaml

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/operator-chaos-pr-comment.yml:
- Around line 14-17: The workflow currently grants workflow-level permissions
including "issues: write" which is broader than necessary; move the top-level
permissions block into the specific job's definition and grant only the
least-privilege scope required (e.g., add a "permissions:" map under the job
that sets "issues: write" and other entries as needed and remove the
workflow-level "permissions:" block) so that "issues: write" is scoped to the
job instead of the entire workflow.
- Around line 84-128: The workflow currently checks out untrusted pr-head and
then runs .github/scripts/operator-chaos-run-checks.sh which reads
pr-head/chaos/knowledge/mlflow.yaml and copies the PR CRD via cp
"${TARGET_CRD_PATH}" (which follows symlinks); add a symlink-rejection step
before invoking operator-chaos-run-checks.sh that scans the pr-head inputs
(e.g., pr-head/chaos/knowledge and pr-head/config/crd/...) for any symbolic
links and fails the job if any are found; implement this check either as a small
shell step in the workflow (run a find ... -type l and exit 1 if results) or
incorporate it at the start of operator-chaos-run-checks.sh (before cp
"${TARGET_CRD_PATH}") to prevent following malicious symlinks.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: a654b2bb-8dc2-40cb-9ab6-895353918321

📥 Commits

Reviewing files that changed from the base of the PR and between 4fd4adf and 7f8f7cc.

📒 Files selected for processing (8)

• .github/scripts/operator-chaos-pr-comment.sh
• .github/scripts/operator-chaos-run-checks.sh
• .github/scripts/operator-chaos-summary.sh
• .github/workflows/operator-chaos-pr-comment.yml
• .github/workflows/operator-chaos.yml
• AGENTS.md
• README.md
• chaos/knowledge/mlflow.yaml

✅ Files skipped from review due to trivial changes (1)

• AGENTS.md

🚧 Files skipped from review as they are similar to previous changes (4)

• chaos/knowledge/mlflow.yaml
• .github/workflows/operator-chaos.yml
• .github/scripts/operator-chaos-summary.sh
• .github/scripts/operator-chaos-run-checks.sh

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/operator-chaos-pr-comment.yml:
- Around line 28-31: The workflow checks out the repo initially without ref and
later runs bash .github/scripts/operator-chaos-*.sh from that initial checkout,
causing scripts to be from the workflow default ref instead of the base commit
in steps.pr_meta.outputs.base_sha; update the "Checkout base branch assets"
sparse checkout (the checkout step used to fetch base-branch/... assets) to also
include the .github/scripts path and change the run steps that call bash
.github/scripts/operator-chaos-*.sh to invoke
base-branch/.github/scripts/operator-chaos-*.sh (i.e., execute the scripts from
the checked-out base_sha) while keeping the initial checkout (actions/checkout)
if still needed for setup-go's go-version-file: go.mod.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: edddf823-561c-45a3-a7ba-a73cc97c4cbe

📥 Commits

Reviewing files that changed from the base of the PR and between 7f8f7cc and 9b05cb9.

📒 Files selected for processing (8)

• .github/scripts/operator-chaos-pr-comment.sh
• .github/scripts/operator-chaos-run-checks.sh
• .github/scripts/operator-chaos-summary.sh
• .github/workflows/operator-chaos-pr-comment.yml
• .github/workflows/operator-chaos.yml
• AGENTS.md
• README.md
• chaos/knowledge/mlflow.yaml

✅ Files skipped from review due to trivial changes (1)

• AGENTS.md

🚧 Files skipped from review as they are similar to previous changes (5)

• .github/scripts/operator-chaos-pr-comment.sh
• .github/workflows/operator-chaos.yml
• chaos/knowledge/mlflow.yaml
• .github/scripts/operator-chaos-summary.sh
• .github/scripts/operator-chaos-run-checks.sh

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mprahl

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [mprahl]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment