Improve payload analysis depth and coverage

[stbenjam]

Summary

Improvements to the CI payload analysis and install failure investigation skills to produce more thorough and accurate results.

Payload analysis (analyze-payload)

• Analyze accepted payloads: Payloads can be force-accepted despite blocking job failures. Previously, accepted payloads were skipped entirely — now any failed blocking jobs are investigated and reported.
• Full lookback window: The historical lookback now scans all payloads regardless of phase (Rejected, Accepted, Ready) and doesn't stop at the first pass. This captures intermittent failure patterns (e.g., F-F-S-F-F) that were previously invisible.
• Failure pattern tracking: New failure_pattern field records the full pass/fail history across the lookback window, shown in the HTML report and JSON output.
• Thorough subagent analysis: Subagents now always perform full investigation including must-gather extraction — no more fast mode shortcuts.

Install failure analysis (prow-job-analyze-install-failure)

• Systematic bootstrap log analysis: Replaced thin "check for bootkube errors" guidance with a structured 4-step diagnostic flow: build a timeline from bootkube.log, pursue every error through its causation chain, cross-reference supporting logs, and check OS-level problems.
• TechPreview job awareness: Notes that TechPreview jobs enable additional feature gates that may cause failures in code paths not active in Default clusters.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Analyze Accepted payloads in addition to Rejected and in-progress; reporting now tracks failure patterns and offers revert recommendations.

• Documentation

  • Expanded payload command descriptions and enhanced diagnostic guidance for cluster bootstrap failures, including TechPreview-specific considerations and cross-log correlation.

• Chores

  • Plugin version bumped to 0.0.19.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

CI plugin version bumped from 0.0.17 → 0.0.19 in manifests and docs; prow install-failure guidance rewritten into a log-driven diagnostic procedure with TechPreview notes; analyze-payload docs and skill expanded to include Accepted payloads, broaden lookback semantics, and add failure_pattern reporting and reporting/UI enhancements.

Changes

Cohort / File(s)	Summary
Plugin metadata & docs plugins/ci/.claude-plugin/plugin.json, .claude-plugin/marketplace.json, docs/data.json	Bumped CI plugin version from 0.0.17 to 0.0.19 and updated related command descriptions in docs index.
Prow install-failure skill plugins/ci/skills/prow-job-analyze-install-failure/SKILL.md	Rewrote checklist into a timeline-driven diagnostic flow: assemble must-gather logs, correlate bootkube/kubelet/etcd/kube-apiserver/serial logs and stderr/stdout, analyze termination context (signals/OOM/host metrics), validate installer/cloud API/quotas, and add TechPreview notes.
Analyze-payload command docs plugins/ci/commands/analyze-payload.md, PLUGINS.md	Command description broadened to cover payloads in Rejected, Accepted, and Ready states; wording clarifies Accepted payloads may be force-accepted despite blocking failures.
Analyze-payload skill & reporting plugins/ci/skills/analyze-payload/SKILL.md	Expanded to handle Accepted payloads, broaden lookback to include all payload states, introduce failure_pattern (history/streak semantics), fetch PRs per originating payload, require full subagent analysis and structured ANALYSIS_RESULT, and extend HTML/JSON outputs (failure_pattern rows, revert recommendations, updated JSON schema/filenames).

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 8

✅ Passed checks (8 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'Improve payload analysis depth and coverage' accurately reflects the primary changes: enhanced payload analysis (accepting, tracking, and reporting on more payload states with failure patterns) and deeper diagnostic coverage for install failures.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
No Real People Names In Style References	✅ Passed	Comprehensive search of all modified files found no real people's names used as style references or examples.
No Assumed Git Remote Names	✅ Passed	No hardcoded git remote names like 'origin' or 'upstream' found in the PR changes. Repository search confirms no assumed remote names without discovery.
Git Push Safety Rules	✅ Passed	The pull request modifies only documentation and configuration files with no git push commands or executable scripts present in modified files.
No Untrusted Mcp Servers	✅ Passed	The pull request does not introduce any new MCP server installations from untrusted or other sources. Modifications are limited to version updates and documentation enhancements.
Ai-Helpers Overlap Detection	✅ Passed	PR modifies two distinct complementary skills (analyze-payload and prow-job-analyze-install-failure) with different scopes and purposes. No functional overlap detected between them or with other existing code.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@plugins/ci/skills/prow-job-analyze-install-failure/SKILL.md`:
- Around line 322-323: The sentence "A crash is a software bug until proven
otherwise." in SKILL.md (the bullet about reading stderr/stdout for crashed
processes) is too absolute—update that line to require validating termination
reason before labeling a crash as a product bug; replace the sentence with a
phrasing such as "Treat a crash as a potential bug but validate the termination
reason (OOM, host restart, killed by signal, resource limits, etc.) by examining
container exit status, kernel messages, and host metrics before assigning root
cause." Also add a short note to consult surrounding logs and infra signals
(exit codes, ContainerDied event details) to distinguish software defects from
infra/resource-induced terminations.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 0dd6c0a7-8bb1-4ea0-b94f-ecca4e129e71

📥 Commits

Reviewing files that changed from the base of the PR and between 93d2dbc and 79944f9.

📒 Files selected for processing (2)

• plugins/ci/.claude-plugin/plugin.json
• plugins/ci/skills/prow-job-analyze-install-failure/SKILL.md

[coderabbitai]

🧹 Nitpick comments (1)

plugins/ci/skills/analyze-payload/SKILL.md (1)

71-71: Tighten redundant phrasing for readability.

“first started failing” is repetitive; “started failing” is cleaner in both places.

✏️ Suggested wording tweak

-The goal is to determine **when each failing job first started failing** and understand its failure pattern across recent payloads.
+The goal is to determine **when each failing job started failing** and understand its failure pattern across recent payloads.
...
-- **is_new_failure**: Whether the job first started failing in the target payload (streak_length == 1)
+- **is_new_failure**: Whether the job started failing in the target payload (streak_length == 1)

Also applies to: 82-82

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@plugins/ci/skills/analyze-payload/SKILL.md` at line 71, Replace the redundant
phrase "first started failing" with the shorter "started failing" in the
analyze-payload skill description (SKILL.md); locate the occurrences of the
exact phrase "first started failing" (including the goal sentence and the second
instance later in the file) and update both to "started failing" to tighten the
wording while preserving the original meaning.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@plugins/ci/skills/analyze-payload/SKILL.md`:
- Line 71: Replace the redundant phrase "first started failing" with the shorter
"started failing" in the analyze-payload skill description (SKILL.md); locate
the occurrences of the exact phrase "first started failing" (including the goal
sentence and the second instance later in the file) and update both to "started
failing" to tighten the wording while preserving the original meaning.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 64dc9c0e-4556-496c-b399-40ffffb4589f

📥 Commits

Reviewing files that changed from the base of the PR and between 28f5e4e and 2590fb6.

📒 Files selected for processing (2)

• plugins/ci/commands/analyze-payload.md
• plugins/ci/skills/analyze-payload/SKILL.md

[coderabbitai]

🧹 Nitpick comments (1)

PLUGINS.md (1)

61-61: Use canonical payload phase terminology for consistency.

Line 61 says “in-progress,” while the PR objective references phase names including “Ready.” Consider aligning this wording with the canonical phase labels used elsewhere to avoid ambiguity in operator guidance.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@PLUGINS.md` at line 61, The description for the /ci:analyze-payload command
uses “in-progress” which is inconsistent with canonical payload phase
terminology; update the text for **`/ci:analyze-payload` `<payload-tag>
[--lookback N]`** to use the official phase label (e.g., replace “in-progress”
with “Ready” or the exact phase name used elsewhere), ensure capitalization and
punctuation match other docs, and keep the list order and phrasing consistent
with existing references to “Rejected, Accepted, Ready” (or the project's
canonical phase names).

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@PLUGINS.md`:
- Line 61: The description for the /ci:analyze-payload command uses
“in-progress” which is inconsistent with canonical payload phase terminology;
update the text for **`/ci:analyze-payload` `<payload-tag> [--lookback N]`** to
use the official phase label (e.g., replace “in-progress” with “Ready” or the
exact phase name used elsewhere), ensure capitalization and punctuation match
other docs, and keep the list order and phrasing consistent with existing
references to “Rejected, Accepted, Ready” (or the project's canonical phase
names).

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 8f5c7570-3d0c-429f-9a62-f400fc6054be

📥 Commits

Reviewing files that changed from the base of the PR and between 2590fb6 and 078d5c3.

📒 Files selected for processing (2)

• PLUGINS.md
• docs/data.json

🚧 Files skipped from review as they are similar to previous changes (1)

• docs/data.json

[dgoodwin]

/lgtm

[dgoodwin]

We're about to conflict, sorry! :(

[dgoodwin]

Let's race.

[dgoodwin]

/lgtm

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (1)

plugins/ci/skills/analyze-payload/SKILL.md (1)

71-71: Trim redundant wording for readability.

Line 71 and Line 82 use “first started failing”; “started failing” is cleaner.

Also applies to: 82-82

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@plugins/ci/skills/analyze-payload/SKILL.md` at line 71, In SKILL.md update
the wording to remove redundancy by replacing the phrase "first started failing"
with the shorter "started failing" in both occurrences referenced (the sentence
containing "when each failing job first started failing" and the similar
sentence at the other occurrence), ensuring the intent remains the same and
spacing/punctuation remains consistent.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@plugins/ci/skills/analyze-payload/SKILL.md`:
- Line 650: The documentation uses two delimiter styles for the failure_pattern
(e.g., "F-F-F-S-F-F-F" vs "F F F S F F"); choose one canonical format (recommend
space-delimited) and update all occurrences in SKILL.md to that single format,
including the example sentence currently showing "F-F-F-S-F-F-F" and any earlier
examples that use "F F F S F F"; ensure the README’s description of
failure_pattern and any example headings or inline examples consistently
reference the chosen delimiter so downstream parsers/readers see one canonical
representation.

---

Nitpick comments:
In `@plugins/ci/skills/analyze-payload/SKILL.md`:
- Line 71: In SKILL.md update the wording to remove redundancy by replacing the
phrase "first started failing" with the shorter "started failing" in both
occurrences referenced (the sentence containing "when each failing job first
started failing" and the similar sentence at the other occurrence), ensuring the
intent remains the same and spacing/punctuation remains consistent.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: ce16c26e-e8a5-4707-8a03-320df9a6a23d

📥 Commits

Reviewing files that changed from the base of the PR and between 078d5c3 and 0cc689e.

📒 Files selected for processing (7)

• .claude-plugin/marketplace.json
• PLUGINS.md
• docs/data.json
• plugins/ci/.claude-plugin/plugin.json
• plugins/ci/commands/analyze-payload.md
• plugins/ci/skills/analyze-payload/SKILL.md
• plugins/ci/skills/prow-job-analyze-install-failure/SKILL.md

✅ Files skipped from review due to trivial changes (1)

• PLUGINS.md

🚧 Files skipped from review as they are similar to previous changes (3)

• plugins/ci/commands/analyze-payload.md
• .claude-plugin/marketplace.json
• docs/data.json

[dgoodwin]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dgoodwin, stbenjam

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [dgoodwin,stbenjam]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment