Improvements to payload agent

.claude-plugin/marketplace.json

@@ -49,7 +49,7 @@
       "name": "ci",
       "source": "./plugins/ci",
       "description": "A plugin to work with OpenShift CI and analyze Prow job results",
-      "version": "0.0.43",
+      "version": "0.0.44",
       "category": "ci",
       "keywords": [
         "prow",

.skillsaw.yaml

@@ -98,9 +98,10 @@ custom-rules:
 # Exclude patterns (glob format)
 # Use exclude: [] to disable all excludes including defaults
 exclude:
-    # - "**/template/**"
-    # - "**/templates/**"
-    # - "**/_template/**"
+    - "node_modules/**"
+    - "**/template/**"
+    - "**/templates/**"
+    - "**/_template/**"
 
 # Additional markdown files to run content rules on (glob format)
 content-paths: []

plugins/ci/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "ci",
   "description": "Tools for working with OpenShift CI and analyzing Prow job results",
-  "version": "0.0.43",
+  "version": "0.0.44",
   "author": {
     "name": "github.com/openshift-eng"
   }

plugins/ci/commands/payload-analysis.md

@@ -0,0 +1,102 @@
+---
+description: Analyze a payload snapshot to identify root causes of blocking job failures and produce an HTML report with revert recommendations
+argument-hint: "<payload-tag> [--snapshot-dir DIR]"
+---
+
+## Name
+
+ci:payload-analysis
+
+## Synopsis
+
+```
+/ci:payload-analysis <payload-tag> [--snapshot-dir DIR]
+```
+
+## Description
+
+The `ci:payload-analysis` command analyzes a specific payload tag using a local snapshot (from the `payload-snapshot` skill), investigates every failed blocking job, and produces a self-contained HTML report summarizing what went wrong.
+
+It supports **Rejected** payloads (full analysis), **Ready** payloads (early analysis of blocking jobs that have already failed), and **Accepted** payloads (which may have been force-accepted despite blocking failures).
+
+The analysis reads from a pre-built snapshot that contains all release controller, GitHub, and CI data frozen at the time of snapshot creation. This means:
+
+- **Reproducible**: Re-running the analysis on the same snapshot produces consistent results
+- **Historical replay**: Analyze old payloads against their original data, even after the world has moved on
+- **Model-friendly**: Less capable models can perform analysis without orchestrating complex multi-skill data gathering
+
+If no snapshot exists for the requested payload, one is created automatically.
+
+The analysis performs **historical lookback** through the snapshot's payload chain to determine when each failure first appeared. For each originating payload, it identifies the PRs introduced as likely culprits. When a candidate PR can be correlated with high confidence (>= 85 rubric score), the report recommends it for **immediate revert** per OCP policy.
+
+An **adversarial review** subagent checks the analysis for weak correlations, misattributions, and logical gaps before finalizing the report.
+
+The payload results YAML output (`payload-results-{tag}.yaml`) can be consumed by composable downstream commands: `/ci:payload-revert` stages reverts for high-confidence candidates, and `/ci:payload-experiment` opens draft revert PRs for medium-confidence candidates.
+
+### Key Features
+
+- **Snapshot-based**: All data pre-gathered locally — no live API orchestration during analysis
+- **Historical lookback**: Uses the snapshot's chain data to identify when each job first started failing
+- **PR correlation**: Reads local PR diffs from the snapshot to match error messages with code changes
+- **Parallel investigation**: Kicks off subagents for each failed blocking job using the appropriate CI analysis skill
+- **Adversarial review**: A dedicated reviewer subagent checks conclusions before finalizing
+- **Revert recommendations**: Proposes specific PRs to revert when evidence strongly links them to a failure
+- **HTML report**: Self-contained HTML report with collapsible sections, color-coded severity, and executive summary
+
+## Implementation
+
+Load the "payload-analysis" skill and follow its implementation steps. The skill orchestrates:
+
+1. Locating or creating a payload snapshot
+2. Reading failure data, streaks, and candidate PRs from the snapshot
+3. Launching parallel subagents to investigate each failed job
+4. Scoring candidates and identifying revert recommendations
+5. Adversarial review of conclusions
+6. Generating the final HTML report, YAML, and JSON outputs
+
+## Return Value
+
+- **Format**: Self-contained HTML file + payload results YAML + JSON data file saved to the current working directory
+- **Filenames**:
+  - `payload-analysis-{tag}-summary.html` — HTML report
+  - `payload-analysis-{tag}-autodl.json` — JSON data for database ingestion
+  - `payload-results-{tag}.yaml` — Scored candidates for downstream commands
+- **Contents** (all `<a>` links must use `target="_blank"` to open in a new tab):
+  - Executive summary with overall payload health
+  - Summary table of all blocking jobs (pass/fail with streaks and failure patterns)
+  - Per-job failure analysis with root cause, error messages, and logs
+  - Originating payload and candidate PRs for each failure
+  - Recommended reverts section with PR links, rationale, and copy-paste revert text
+  - Adversarial review notes
+  - Color-coded severity and collapsible detail sections
+
+## Examples
+
+1. **Analyze an amd64 nightly payload** (auto-creates snapshot if needed):
+   ```
+   /ci:payload-analysis 4.22.0-0.nightly-2026-02-25-152806
+   ```
+
+2. **Analyze using an existing snapshot directory**:
+   ```
+   /ci:payload-analysis 4.22.0-0.nightly-2026-02-25-152806 --snapshot-dir payload/4.22/nightly
+   ```
+
+3. **Analyze an arm64 payload** (architecture inferred from tag):
+   ```
+   /ci:payload-analysis 4.22.0-0.nightly-arm64-2026-02-25-152806
+   ```
+
+## Arguments
+
+- $1: A full payload tag (e.g., `4.22.0-0.nightly-2026-02-25-152806`). Version, stream, and architecture are parsed from the tag automatically. Tags without an architecture suffix are amd64. (required)
+- `--snapshot-dir DIR`: Path to an existing snapshot directory containing `summary.json` (optional). If not provided, searches standard locations or creates a new snapshot.
+
+## Skills Used
+
+- `payload-snapshot`: Creates the local data snapshot (auto-invoked if needed)
+- `payload-analysis`: Orchestrates the full analysis workflow
+- `payload-results-yaml`: Schema for the results YAML output
+- `payload-autodl-json`: Schema for the JSON data output
+- `prow-job-analyze-install-failure`: Analyzes install failures (used by subagents)
+- `prow-job-analyze-test-failure`: Analyzes test failures (used by subagents)

plugins/ci/commands/payload-experiment.md

@@ -24,7 +24,7 @@ The `ci:payload-experiment` command opens draft revert PRs for medium-confidence
 All state is tracked in the payload results YAML via the `payload-results-yaml` skill — no separate tracking file is created.
 
 This command is one of three composable stages in the payload triage pipeline:
-1. `/ci:analyze-payload` — produces the payload results YAML
+1. `/ci:payload-analysis` — produces the payload results YAML
 2. `/ci:payload-revert` — stages reverts for HIGH confidence candidates
 3. `/ci:payload-experiment` — experimentally tests MEDIUM confidence candidates (this command)
 
@@ -35,12 +35,12 @@ This command is one of three composable stages in the payload triage pipeline:
 
 ## Implementation
 
-1. **Parse the payload tag** from the argument. Extract `version`, `stream`, and `architecture` from the tag (see `analyze-payload` Step 1 for parsing rules).
+1. **Parse the payload tag** from the argument. Extract `version`, `stream`, and `architecture` from the tag (see `payload-analysis` Step 1 for parsing rules).
 
 2. **Read the payload results YAML** using the `payload-results-yaml` skill: Look for `payload-results-{tag}.yaml` in the current working directory. If not found, print an error and exit:
    ```
    Error: Payload results YAML not found for {payload_tag}.
-   Run `/ci:analyze-payload {payload_tag}` first to generate it.
+   Run `/ci:payload-analysis {payload_tag}` first to generate it.
    ```
 
 3. **Detect Phase 2 resume**: If the results YAML contains any action entry with `type: "experiment"` and `status: "pending"`, jump to Phase 2 (step 5). Phase 2 processes only pending experiments — candidates with other statuses are left unchanged.
@@ -60,7 +60,7 @@ This command is one of three composable stages in the payload triage pipeline:
 
 1. **Start experiments after analysis**:
    ```
-   /ci:analyze-payload 4.22.0-0.nightly-2026-02-25-152806
+   /ci:payload-analysis 4.22.0-0.nightly-2026-02-25-152806
    /ci:payload-experiment 4.22.0-0.nightly-2026-02-25-152806
    ```
 
@@ -71,7 +71,7 @@ This command is one of three composable stages in the payload triage pipeline:
 
 ## Arguments
 
-- $1: A full payload tag (e.g., `4.22.0-0.nightly-2026-02-25-152806`). Must match the tag used with `/ci:analyze-payload`. (required)
+- $1: A full payload tag (e.g., `4.22.0-0.nightly-2026-02-25-152806`). Must match the tag used with `/ci:payload-analysis`. (required)
 
 ## Skills Used
 

plugins/ci/commands/payload-revert.md

@@ -1,5 +1,5 @@
 ---
-description: Stage reverts for high-confidence payload candidates identified by analyze-payload
+description: Stage reverts for high-confidence payload candidates identified by payload-analysis
 argument-hint: "<payload-tag>"
 ---
 
@@ -15,12 +15,12 @@ ci:payload-revert
 
 ## Description
 
-The `ci:payload-revert` command reads the payload results YAML produced by `/ci:analyze-payload` and stages reverts for all high-confidence candidates (confidence score >= 85) that have not already been reverted.
+The `ci:payload-revert` command reads the payload results YAML produced by `/ci:payload-analysis` and stages reverts for all high-confidence candidates (confidence score >= 85) that have not already been reverted.
 
 For each qualifying candidate, it creates a TRT JIRA bug, opens a revert PR, and triggers payload validation jobs using the `stage-payload-reverts` skill.
 
 This command is one of three composable stages in the payload triage pipeline:
-1. `/ci:analyze-payload` — produces the payload results YAML
+1. `/ci:payload-analysis` — produces the payload results YAML
 2. `/ci:payload-revert` — stages reverts for HIGH confidence candidates (this command)
 3. `/ci:payload-experiment` — opens draft revert PRs for MEDIUM confidence candidates
 
@@ -33,12 +33,12 @@ When the number of failing jobs across all candidates exceeds these limits, prio
 
 ## Implementation
 
-1. **Parse the payload tag** from the argument. Extract `version`, `stream`, and `architecture` from the tag (see `analyze-payload` Step 1 for parsing rules).
+1. **Parse the payload tag** from the argument. Extract `version`, `stream`, and `architecture` from the tag (see `payload-analysis` Step 1 for parsing rules).
 
 2. **Read the payload results YAML** using the `payload-results-yaml` skill: Look for `payload-results-{tag}.yaml` in the current working directory. If not found, print an error and exit:
    ```
    Error: Payload results YAML not found for {payload_tag}.
-   Run `/ci:analyze-payload {payload_tag}` first to generate it.
+   Run `/ci:payload-analysis {payload_tag}` first to generate it.
    ```
 
 3. **Filter candidates**: Select candidates with `confidence_score >= 85`. Exclude any that already have an action with `status` of `"open"` or `"merged"` (pre-existing revert).
@@ -56,13 +56,13 @@ When the number of failing jobs across all candidates exceeds these limits, prio
 
 1. **Stage reverts after analysis**:
    ```
-   /ci:analyze-payload 4.22.0-0.nightly-2026-02-25-152806
+   /ci:payload-analysis 4.22.0-0.nightly-2026-02-25-152806
    /ci:payload-revert 4.22.0-0.nightly-2026-02-25-152806
    ```
 
 ## Arguments
 
-- $1: A full payload tag (e.g., `4.22.0-0.nightly-2026-02-25-152806`). Must match the tag used with `/ci:analyze-payload`. (required)
+- $1: A full payload tag (e.g., `4.22.0-0.nightly-2026-02-25-152806`). Must match the tag used with `/ci:payload-analysis`. (required)
 
 ## Skills Used
 

plugins/ci/evals/cases/payload-analysis/case-001-4.22-nightly-ci-config-and-product-regression/annotations.yaml

@@ -0,0 +1,37 @@
+expected_phase: Rejected
+expected_failed_job_count: 3
+has_revert_candidates: true
+force_accept_expected: false
+
+expected_candidates:
+  - pr_url: "https://github.com/openshift/cluster-authentication-operator/pull/839"
+    component: "cluster-authentication-operator"
+    min_confidence: 85
+    expected_confidence: 92
+    description: "Ginkgo v2 migration broke test binary stdout output, corrupting JSON test listing in openshift-tests"
+    expected_failing_jobs:
+      - "metal-ipi-ovn-ipv4"
+      - "metal-ipi-ovn-ipv6"
+
+expected_ci_config_issues:
+  - pr_url: "https://github.com/openshift/release/pull/76852"
+    repo: "openshift/release"
+    description: "Changed vSphere VCM default lease to multi-tenant, breaking UPI workflows that require single-tenant leases"
+    expected_failing_jobs:
+      - "install-analysis-all"
+
+notes: >
+  Mixed-root-cause case combining a product regression and a CI configuration
+  issue. Two metal-ipi jobs fail because cluster-authentication-operator#839
+  (Ginkgo v2 migration) introduced a warning message to stdout that corrupts
+  JSON test listing in openshift-tests. The install-analysis-all job fails
+  because openshift/release#76852 changed the default vSphere VCM lease type
+  from single-tenant to multi-tenant, breaking UPI workflows.
+
+  The CI config issue is the key test: openshift/release PRs do NOT appear in
+  payload changelogs. The agent must recognize that install-analysis-all is a
+  CI step failure (not a product bug), dig into the Prow job to find the
+  underlying vsphere-upi error ("SUBNETS.JSON does not contain enough
+  addresses... expected to be a single-tenant lease"), then search
+  openshift/release for recent commits matching the error. This tests the
+  "look beyond the snapshot" capability via the prow-job-analyze sub-skills.

plugins/ci/evals/cases/payload-analysis/case-001-4.22-nightly-ci-config-and-product-regression/input.yaml

@@ -0,0 +1 @@
+payload_tag: "4.22.0-0.nightly-2026-03-26-231124"

plugins/ci/evals/cases/payload-analysis/case-002-5.0-nightly-all-new-failures/annotations.yaml

@@ -0,0 +1,32 @@
+expected_phase: Rejected
+expected_failed_job_count: 13
+has_revert_candidates: true
+force_accept_expected: false
+
+expected_candidates:
+  - pr_url: "https://github.com/openshift/origin/pull/31006"
+    component: "tests"
+    min_confidence: 85
+    expected_confidence: 95
+    description: "add oc-mirror to extensionBinaries — caused widespread test failures across serial, techpreview, metal, and upgrade jobs"
+    expected_failing_jobs:
+      - "aws-ovn-serial-1of2"
+      - "aws-ovn-serial-2of2"
+      - "aws-ovn-techpreview"
+      - "aws-ovn-techpreview-serial-1of3"
+      - "aws-ovn-techpreview-serial-2of3"
+      - "aws-ovn-techpreview-serial-3of3"
+      - "metal-ipi-ovn-ipv4"
+      - "metal-ipi-ovn-ipv6"
+
+notes: >
+  All 13 blocking jobs are new failures (streak=1), all originating from the
+  same payload with 63 candidate PRs. openshift/origin#31006 ("add oc-mirror
+  to extensionBinaries") caused widespread test failures across multiple job
+  families — serial, techpreview-serial, metal-ipi, and potentially upgrade
+  jobs. The agent must identify this PR as the root cause despite the large
+  candidate pool (63 PRs). Key signals: cross-job correlation (same PR blamed
+  for failures in many independent jobs), error messages referencing test
+  binaries/extensions, and new failure mode not present in prior payloads.
+  force_accept_expected is false because hours_since_baseline is 11.7h (under
+  the 18h threshold) and failures are product regressions, not infrastructure.

plugins/ci/evals/cases/payload-analysis/case-002-5.0-nightly-all-new-failures/input.yaml

@@ -0,0 +1 @@
+payload_tag: "5.0.0-0.nightly-2026-05-30-072431"

plugins/ci/evals/cases/payload-analysis/case-003-5.0-ci-cno-networkpolicy-revert/annotations.yaml

@@ -0,0 +1,25 @@
+expected_phase: Rejected
+expected_failed_job_count: 2
+has_revert_candidates: true
+force_accept_expected: false
+
+expected_candidates:
+  - pr_url: "https://github.com/openshift/cluster-network-operator/pull/2959"
+    component: "cluster-network-operator"
+    min_confidence: 80
+    expected_confidence: 90
+    description: "Added remaining CNO NetworkPolicies that blocked egress from cloud-network-config-controller, causing CrashLoopBackOff due to API server timeout"
+    expected_failing_jobs:
+      - "e2e-aws-ovn"
+      - "e2e-azure-ovn"
+      - "e2e-gcp-ovn"
+
+notes: >
+  Textbook true positive. CNO #2959 added NetworkPolicies that blocked egress
+  from cloud-network-config-controller, causing CrashLoopBackOff across all
+  platforms (AWS, Azure, GCP) due to API server connection timeouts. Evidence
+  quality is excellent: 0% pass rate vs 100% historical, cross-platform
+  consistency, precise mechanism match. Reverted by #2999 on 2026-05-08.
+  Tests the agent's ability to identify infrastructure-level networking
+  restrictions causing application failures and to correlate
+  cross-platform failures to a single root cause.

plugins/ci/evals/cases/payload-analysis/case-003-5.0-ci-cno-networkpolicy-revert/input.yaml

@@ -0,0 +1 @@
+payload_tag: "5.0.0-0.ci-2026-05-07-142711"

plugins/ci/evals/cases/payload-analysis/case-004-4.22-two-reverts/annotations.yaml

@@ -0,0 +1,30 @@
+expected_phase: Rejected
+expected_failed_job_count: 3
+has_revert_candidates: true
+force_accept_expected: false
+
+expected_candidates:
+  - pr_url: "https://github.com/openshift/cluster-ingress-operator/pull/1354"
+    component: "cluster-ingress-operator"
+    min_confidence: 85
+    expected_confidence: 95
+    description: "Replaced OLM-based Istio install with Sail Library, causing GatewayAPIController failures"
+    expected_failing_jobs:
+      - "e2e-aws-ovn-techpreview"
+      - "e2e-aws-ovn-techpreview-serial"
+  - pr_url: "https://github.com/openshift/oc/pull/2232"
+    component: "oc"
+    min_confidence: 85
+    expected_confidence: 90
+    description: "Changed manifest reading to in-memory, causing nil pointer dereference in oc adm release mirror"
+    expected_failing_jobs:
+      - "metal-ipi-ovn-ipv6"
+
+notes: >
+  Payload 4.22.0-0.nightly-2026-03-20-053450 was rejected with 3 failed
+  blocking jobs: aws-ovn-techpreview (test failure: GatewayAPI/OSSM Sail
+  Library migration), aws-ovn-techpreview-serial-3of3 (same root cause),
+  and metal-ipi-ovn-ipv6 (install failure: oc mirror nil pointer crash).
+  Two high-confidence revert candidates: cluster-ingress-operator#1354
+  (score 95, caused GatewayAPI failures) and oc#2232 (score 90, caused
+  mirror crash).

plugins/ci/evals/cases/payload-analysis/case-004-4.22-two-reverts/input.yaml

@@ -0,0 +1 @@
+payload_tag: "4.22.0-0.nightly-2026-03-20-053450"

plugins/ci/evals/cases/payload-analysis/case-005-4.22-ci-cco-revert/annotations.yaml

@@ -0,0 +1,22 @@
+expected_phase: Rejected
+expected_failed_job_count: 3
+has_revert_candidates: true
+force_accept_expected: false
+
+expected_candidates:
+  - pr_url: "https://github.com/openshift/cloud-credential-operator/pull/978"
+    component: "cloud-credential-operator"
+    min_confidence: 85
+    expected_confidence: 95
+    description: "Set operator condition to Progressing when pod identity webhook pods are updating, causing CCO Progressing=True during MCO rolling phase"
+    expected_failing_jobs:
+      - "aggregated-aws-ovn-upgrade-4.22-minor"
+      - "aggregated-gcp-ovn-upgrade-4.22-micro"
+
+notes: >
+  Payload 4.22.0-0.ci-2026-03-31-050515 was rejected with 3 failed blocking
+  jobs: aggregated-aws-ovn-upgrade (CCO Progressing invariant violation),
+  aggregated-gcp-ovn-upgrade (same root cause), and hypershift-e2e-aws
+  (infra teardown timeout, not a code regression). One high-confidence
+  revert candidate: cloud-credential-operator#978 (score 95). The hypershift
+  failure should be classified as infrastructure, not product.

plugins/ci/evals/cases/payload-analysis/case-005-4.22-ci-cco-revert/input.yaml

@@ -0,0 +1 @@
+payload_tag: "4.22.0-0.ci-2026-03-31-050515"