payload-analysis: do not recommend revert or force-reject for kube rebase version skew

[redhat-chai-bot]

Summary

When a Kubernetes rebase merges in openshift/kubernetes, there is an expected lag of hours while the kubelet is rebuilt against the new source and delivered via an updated RHCOS image. During this window, kubelet version skew (kubelet on an older kube version than kube-apiserver) causes widespread test failures across all platforms.

The payload agent currently treats this like any other PR-caused regression: it scores the rebase PR at high confidence and recommends reverting it. This is incorrect — the kubelet binary is built from openshift/kubernetes, so the rebase must remain for the kubelet to pick up the new version. Reverting would prevent the kubelet from ever being updated.

Changes

Two additive sections in plugins/ci/skills/payload-analysis/SKILL.md:

Step 6.2 — Special case: Kubernetes rebase version skew

• When the candidate PR is a kube rebase AND failures show kubelet version skew, do NOT propose a revert even at score >= 85
• Classify as "transient build lag" and recommend monitoring for the rebuilt kubelet to land via RHCOS

Step 6.4 — Guard: kubelet version skew from a Kubernetes rebase

• Force-accepting is inappropriate (real test failures, not infra flakes)
• Force-rejecting is also inappropriate (just accelerates the next payload into the same skew)
• Correct action: wait for the RHCOS with the rebuilt kubelet

Context

This was identified during analysis of 5.0.0-0.nightly-2026-07-17-034502 where the agent incorrectly recommended reverting openshift/kubernetes#2653 (Kubernetes 1.36.2 rebase). Feedback from TRT in https://redhat-internal.slack.com/archives/C02K89U2EV8/p1752756363338069.

---

@sosiouxme requested in Slack thread

Summary by CodeRabbit

• Bug Fixes
  • Improved CI recommendations for Kubernetes rebase failures caused by temporary kubelet/apiserver version skew (“transient build lag”).
  • Avoids suggesting revert and prevents force-accept/force-reject for this scenario.
  • Adds guidance to wait for the rebuilt kubelet to arrive via an updated RHCOS image.
• Chores
  • Bumped the CI plugin version to 0.0.70 in marketplace metadata and plugin manifest.
  • Updated CI eval case annotations/input for the nightly kube rebase skew scenario.

[coderabbitai]

Walkthrough

The CI plugin version is synchronized from 0.0.69 to 0.0.70. Payload analysis now treats Kubernetes rebase kubelet version skew as transient build lag and waits for the rebuilt kubelet before recommending action.

Changes

CI plugin updates

Layer / File(s)	Summary
Kubelet skew decision rules plugins/ci/skills/payload-analysis/SKILL.md, plugins/ci/evals/cases/payload-analysis/case-015-5.0-nightly-kube-rebase-skew/*	Kubernetes rebase kubelet version skew is excluded from revert candidates, classified as transient build lag, and prevented from triggering force-accept or force-reject recommendations. The evaluation case expects no candidates or force-accept recommendation.
Plugin version synchronization .claude-plugin/marketplace.json, docs/index.html, plugins/ci/.claude-plugin/plugin.json	The CI plugin version is updated from 0.0.69 to 0.0.70 in marketplace and plugin metadata.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: bentito, rvanderp3

🚥 Pre-merge checks | ✅ 10

✅ Passed checks (10 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: handling kube rebase version skew by avoiding revert and force-reject recommendations.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
No Real People Names In Style References	✅ Passed	The added skill doc and eval files contain no real-person names in instructions or examples; only technical terms and repo/org names appear.
No Assumed Git Remote Names	✅ Passed	PR-added lines only bump versions and add payload-analysis guidance/eval data; no added patch lines hardcode git remote names.
Git Push Safety Rules	✅ Passed	The PR only adds an eval case; no changed file contains git push, main/master push, or force-push behavior.
No Untrusted Mcp Servers	✅ Passed	Diff only changes eval YAMLs; no new MCP server installs, repos, or npx mcp-server entries were added.
Ai-Helpers Overlap Detection	✅ Passed	No HIGH/MODERATE overlap: the only same-path open PR (#580) is about merged-revert detection, while the kubelet skew fix is distinct; other related PRs touch different files.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

plugins/ci/skills/payload-analysis/SKILL.md (1)

402-403: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Make the output-schema mapping explicit.

The downstream failure_type enum only permits test, install, upgrade, or infra. Clarify that "transient build lag" belongs in root_cause_summary/failure_pattern, while retaining the existing failure_type enum, to avoid invalid payload-results output.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/ci/skills/payload-analysis/SKILL.md` around lines 402 - 403, Clarify
the output mapping in the failure-mode guidance: keep failure_type set to one of
the existing test, install, upgrade, or infra enum values, and record “transient
build lag” only in root_cause_summary and/or failure_pattern. Apply this
clarification to the pending kubelet rebuild and updated RHCOS monitoring
instructions.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@plugins/ci/skills/payload-analysis/SKILL.md`:
- Around line 402-403: Clarify the output mapping in the failure-mode guidance:
keep failure_type set to one of the existing test, install, upgrade, or infra
enum values, and record “transient build lag” only in root_cause_summary and/or
failure_pattern. Apply this clarification to the pending kubelet rebuild and
updated RHCOS monitoring instructions.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c856c062-d7b4-479d-a92e-6aa318dc89e4

📥 Commits

Reviewing files that changed from the base of the PR and between 2a19861 and b2e3676.

📒 Files selected for processing (4)

• .claude-plugin/marketplace.json
• docs/index.html
• plugins/ci/.claude-plugin/plugin.json
• plugins/ci/skills/payload-analysis/SKILL.md

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

plugins/ci/skills/payload-analysis/SKILL.md (1)

397-404: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Apply the Kubernetes skew override only when it is the sole relevant cause.

The new exception can mask independent failure modes: it rewrites classification/output as transient build lag and suppresses both revert and force-action guidance even when another regression still requires action.

• plugins/ci/skills/payload-analysis/SKILL.md#L397-L404: preserve all unrelated failure-mode classifications and summaries.
• plugins/ci/skills/payload-analysis/SKILL.md#L433-L439: disable force actions only when every blocking failure is explained by the skew condition.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/ci/skills/payload-analysis/SKILL.md` around lines 397 - 404, The
Kubernetes skew override in the Kubernetes rebase guidance must apply only when
kubelet version skew is the sole relevant blocking failure. At
plugins/ci/skills/payload-analysis/SKILL.md:397-404, preserve classifications
and root-cause summaries for any independent failures while applying the
transient build-lag treatment only to skew-related failures; at
plugins/ci/skills/payload-analysis/SKILL.md:433-439, suppress revert and
force-action guidance only when every blocking failure is explained by the skew
condition, otherwise retain the required action guidance for unrelated
regressions.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@plugins/ci/skills/payload-analysis/SKILL.md`:
- Around line 397-404: The Kubernetes skew override in the Kubernetes rebase
guidance must apply only when kubelet version skew is the sole relevant blocking
failure. At plugins/ci/skills/payload-analysis/SKILL.md:397-404, preserve
classifications and root-cause summaries for any independent failures while
applying the transient build-lag treatment only to skew-related failures; at
plugins/ci/skills/payload-analysis/SKILL.md:433-439, suppress revert and
force-action guidance only when every blocking failure is explained by the skew
condition, otherwise retain the required action guidance for unrelated
regressions.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: bc56eb92-a7f8-4440-8688-fd8adbcbc0a7

📥 Commits

Reviewing files that changed from the base of the PR and between b2e3676 and 5f32e9d.

📒 Files selected for processing (1)

• plugins/ci/skills/payload-analysis/SKILL.md

[sosiouxme]

/test payload-agent

[theobarberbany]

Typically the eval data lives in https://github.com/stbenjam/historical-payload-data IIUC

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@plugins/ci/evals/cases/payload-analysis/case-015-5.0-nightly-kube-rebase-skew/annotations.yaml`:
- Around line 1-14: Add a scored assertion to the case configuration that
requires classification as transient build lag and recommends waiting for the
rebuilt kubelet, rather than relying solely on expected_phase: Rejected. Update
the relevant expected outcome fields near expected_candidates and preserve the
existing no-revert, no-force-accept, and no-force-reject expectations.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 54fa2de5-81a3-4468-a606-30cc35f9fb6d

📥 Commits

Reviewing files that changed from the base of the PR and between 5f32e9d and 25fd368.

📒 Files selected for processing (2)

• plugins/ci/evals/cases/payload-analysis/case-015-5.0-nightly-kube-rebase-skew/annotations.yaml
• plugins/ci/evals/cases/payload-analysis/case-015-5.0-nightly-kube-rebase-skew/input.yaml

[sosiouxme]

payload agent succeeded on equivalent previous commit.

i ran this case locally and it also succeeded.

/lgtm

[theobarberbany]

/approve

/hold

waiting on stbenjam/historical-payload-data#3, although then shouldnt eval-payload-analysis-changed be red?

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: redhat-chai-bot, sosiouxme, theobarberbany

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• plugins/ci/OWNERS [theobarberbany]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[theobarberbany]

The eval-payload-analysis-changed job passed (0 failures in JUnit), but no eval cases actually ran to completion. The artifacts confirm it:

https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift-eng_ai-helpers/627/pull-ci-openshift-eng-ai-helpers-main-eval-payload-analysis-changed/2078148287981948928

• eval-runs.tar (348 bytes) contains only the empty directory structure — no run_result.json, no summary.yaml, no case outputs
• claude-sessions.tar contains only the orchestrator transcript, not any skill execution transcripts

What happened: the eval harness orchestrator launched execute.py (15 cases, parallelism 5) with run_in_background: true, said "I'll continue with collection, scoring, and reporting once execution completes," then immediately ended its turn. The session exited, the background task was killed, and openshift-claude-agent-eval-commands.sh saw exit 0 from the claude CLI and wrote a passing JUnit.

Evidence from claude-eval.log:

• Line 401: execute.py launched with "run_in_background":true
• Line 409: session result after only 18 turns / 90s API time — "stop_reason":"end_turn"
• Line 411: background task status "killed"

Net effect: the eval gate is not actually gating anything on this PR. The job will pass regardless of whether the changed eval cases succeed or fail.

Likely needs a fix in either the eval harness (prevent run_in_background for execution) or openshift-claude-agent-eval-commands.sh (validate that run artifacts exist before writing a passing JUnit), or both.

[theobarberbany]

openshift/release#82091

cc @stbenjam @enxebre

[stbenjam]

/test ci/prow/eval-payload-analysis-changed

[theobarberbany]

/test eval-payload-analysis-changed

[theobarberbany]

So one passes, one fails with the same code / harness.

Claude analysis:

  Both runs used identical harness code (cloned from main) and both backgrounded execute.py. The outcome depends entirely on whether the model happens to issue another tool call.

  Failing run — model decides to wait passively, session exits:

  Line 427: "Execution launched in background ... Let me check initial progress."
  Line 430: "I'll continue with collection, scoring, and reporting once execution completes."
  Line 431: {"type":"result","subtype":"success","stop_reason":"end_turn","terminal_reason":"completed",...}
  Line 433: {"type":"system","subtype":"task_updated","task_id":"brkjy2a5p","patch":{"status":"killed",...}}

  Model hit end_turn. In -p mode that's terminal — session tears down, background task gets killed, claude exits 0. Empty artifacts, false green.

  Passing run — model decides to poll, session stays alive:

  Line 445: "Execution launched in background ... Let me check initial progress."
  Line 446: {"name":"Bash","input":{"command":"sleep 30 && tail -30 .../bapedpvd1.output","timeout":60000}}
  Line 451: {"name":"Bash","input":{"command":"sleep 120 && tail -40 ...","timeout":180000}}
  Line 456: {"name":"Bash","input":{"command":"sleep 180 && tail -40 ...","timeout":240000}}

  Model kept issuing blocking sleep && tail calls — never hit end_turn, so the session stayed up long enough for execute.py to finish. This worked by accident, not by design.