Add analyze-payload command and improve CI skills

.claude-plugin/marketplace.json

@@ -26,7 +26,7 @@
       "name": "ci",
       "source": "./plugins/ci",
       "description": "A plugin to work with OpenShift CI and analyze Prow job results",
-      "version": "0.0.11"
+      "version": "0.0.12"
     },
     {
       "name": "teams",

docs/data.json

@@ -503,7 +503,7 @@
           "name": "Triage Regression"
         }
       ],
-      "version": "0.0.11"
+      "version": "0.0.12"
     },
     {
       "commands": [

plugins/ci/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "ci",
   "description": "Tools for working with OpenShift CI and analyzing Prow job results",
-  "version": "0.0.11",
+  "version": "0.0.12",
   "author": {
     "name": "openshift"
   }

plugins/ci/skills/analyze-payload/SKILL.md

@@ -266,10 +266,56 @@ If any revert candidates were identified in Step 6.2, include a prominent sectio
       <td>{confidence_rationale}</td>
     </tr>
   </table>
-  <p class="revert-note">To revert a PR, use: <code>/ci:revert-pr &lt;pr-url&gt; &lt;jira-ticket&gt;</code></p>
+  <!-- For each revert candidate, include a copy-paste block -->
+  <h3>Revert Instructions</h3>
+  <p>Copy and paste the following into Claude Code to initiate each revert:</p>
+  <div class="revert-prompt">
+    <button onclick="navigator.clipboard.writeText(this.nextElementSibling.textContent.trim())">Copy</button>
+    <pre>PR {pr_url} caused {job_name_1}, {job_name_2}, ... to fail starting in payload {originating_payload_tag}.
+
+Evidence:
+- {brief failure summary from subagent analysis for each affected job}
+- The job(s) were passing prior to {originating_payload_tag} and started failing when this PR landed.
+
+/ci:revert-pr {pr_url}</pre>
+  </div>
 </div>
 ```
 
+Add the following styles for the revert prompt block:
+
+```html
+.revert-prompt {
+  position: relative;
+  margin: 12px 0;
+}
+.revert-prompt pre {
+  background: #1e1e1e;
+  color: #d4d4d4;
+  padding: 16px;
+  border-radius: 4px;
+  white-space: pre-wrap;
+  font-family: 'SFMono-Regular', Consolas, monospace;
+  font-size: 13px;
+  overflow-x: auto;
+}
+.revert-prompt button {
+  position: absolute;
+  top: 8px;
+  right: 8px;
+  background: #3c3c3c;
+  color: #d4d4d4;
+  border: 1px solid #555;
+  border-radius: 4px;
+  padding: 4px 10px;
+  cursor: pointer;
+  font-size: 12px;
+}
+.revert-prompt button:hover {
+  background: #505050;
+}
+```
+
 If **no** revert candidates were identified, include a brief note instead:
 
 ```html
@@ -301,17 +347,6 @@ Add the following styles for the revert section:
 .revert-none h2 {
   color: #5f6368;
 }
-.revert-note {
-  margin-top: 12px;
-  font-size: 13px;
-  color: #5f6368;
-}
-.revert-note code {
-  background: #f1f3f4;
-  padding: 2px 6px;
-  border-radius: 3px;
-  font-size: 12px;
-}
 ```
 
 #### 7.5: Styling

plugins/ci/skills/fetch-new-prs-in-payload/SKILL.md

@@ -5,7 +5,7 @@ description: Fetch pull requests that are new in a given OpenShift payload compa
 
 # Fetch New PRs in Payload
 
-This skill fetches the list of pull requests that are new in a given OpenShift payload tag compared to the previous payload. It uses the Sippy payload diff API to identify which PRs were merged between two consecutive payloads.
+This skill fetches the list of pull requests that are new in a given OpenShift payload tag compared to the previous payload. It tries the Sippy payload diff API first, and falls back to the release controller API when Sippy has not yet ingested the payload (e.g., in-progress or very recent payloads).
 
 ## When to Use This Skill
 
@@ -21,9 +21,10 @@ The payload tag can be obtained from the `fetch-prowjob-json` skill (`release.op
 
 ## Prerequisites
 
-1. **Network Access**: Must be able to reach the Sippy API
-   - Check: `curl -s https://sippy.dptools.openshift.org/api/health`
-   - No authentication required for public API endpoints
+1. **Network Access**: Must be able to reach the Sippy API and/or the release controller
+   - Sippy: `curl -s https://sippy.dptools.openshift.org/api/health`
+   - Release controller: `curl -s https://amd64.ocp.releases.ci.openshift.org/api/v1/releasestream/4-stable/latest`
+   - No authentication required for either
 
 2. **Python 3**: Python 3.6 or later
    - Check: `python3 --version`
@@ -129,9 +130,18 @@ python3 fetch_new_prs_in_payload.py
 - `0`: Success
 - `1`: Error (invalid input, API error, network error, etc.)
 
+## Data Sources
+
+The script tries two APIs in order:
+
+1. **Sippy** (primary) — has longer history but only includes completed payloads
+2. **Release controller** (fallback) — available immediately for in-progress and recent payloads, but has shorter retention
+
+The output format is identical regardless of which source is used. The fallback is automatic and transparent.
+
 ## API Details
 
-### Endpoint
+### Sippy Endpoint
 
 ```text
 GET https://sippy.dptools.openshift.org/api/payloads/diff?toPayload={payload_tag}
@@ -251,9 +261,9 @@ python3 "$FETCH_NEW_PRS" "$payload_tag" --format json
 
 ## Notes
 
-- The API does not require authentication for read-only access
+- Neither API requires authentication for read-only access
 - The Python script uses only standard library modules (no external dependencies)
-- The API automatically determines the previous payload to diff against; you only provide the target payload
+- The previous payload is determined automatically; you only provide the target payload
 - Payload tags follow the format: `{version}-0.{stream}-{date}-{time}` (e.g., `4.22.0-0.ci-2026-02-06-195709` or `4.22.0-0.nightly-2026-01-15-114134`)
 - The `component` field (called `name` in the raw API) may contain multiple comma-separated component names for PRs that affect multiple components
 - PRs without an associated bug will have an empty string for `bug_url`

plugins/ci/skills/fetch-new-prs-in-payload/fetch_new_prs_in_payload.py

@@ -1,69 +1,150 @@
 #!/usr/bin/env python3
 """Fetch new pull requests included in an OpenShift payload that were not in the previous one.
 
-Uses the Sippy payload diff API to retrieve PRs that are new in a given payload tag.
+Uses the Sippy payload diff API to retrieve PRs that are new in a given payload tag,
+falling back to the release controller API when Sippy has not yet ingested the payload.
 """
 
 import argparse
 import json
+import re
 import sys
 import urllib.error
 import urllib.parse
 import urllib.request
-from typing import Dict, List
+from typing import Dict, List, Optional
 
 SIPPY_API_BASE = "https://sippy.dptools.openshift.org/api"
+RELEASE_CONTROLLER_BASE = "https://amd64.ocp.releases.ci.openshift.org/api/v1"
 
 
-def fetch_new_prs(payload_tag: str) -> list:
-    """Fetch new PRs in the given payload tag compared to its predecessor."""
-    url = f"{SIPPY_API_BASE}/payloads/diff?toPayload={urllib.parse.quote(payload_tag)}"
+def _http_get_json(url: str, timeout: int = 30) -> dict:
+    """Fetch JSON from a URL. Raises on HTTP or connection errors."""
+    req = urllib.request.Request(url, headers={"Accept": "application/json"})
+    with urllib.request.urlopen(req, timeout=timeout) as resp:
+        return json.loads(resp.read().decode("utf-8"))
+
+
+def _parse_stream(payload_tag: str) -> Optional[str]:
+    """Extract the release stream from a payload tag.
+
+    Examples:
+        4.21.0-0.ci-2026-02-27-125225 -> 4.21.0-0.ci
+        4.22.0-0.nightly-2026-01-15-114134 -> 4.22.0-0.nightly
+    """
+    m = re.match(r"^(.+)-\d{4}-\d{2}-\d{2}-\d{6}$", payload_tag)
+    return m.group(1) if m else None
 
+
+def _find_previous_tag(stream: str, payload_tag: str) -> Optional[str]:
+    """Find the payload immediately before payload_tag in the stream."""
+    url = f"{RELEASE_CONTROLLER_BASE}/releasestream/{urllib.parse.quote(stream)}/tags"
+    data = _http_get_json(url)
+    tags = [t["name"] for t in data.get("tags", [])]
     try:
-        req = urllib.request.Request(url)
-        with urllib.request.urlopen(req, timeout=30) as resp:
-            data = json.loads(resp.read().decode("utf-8"))
-    except urllib.error.HTTPError as e:
-        if e.code == 404:
-            print(
-                f"Error: Payload '{payload_tag}' not found.",
-                file=sys.stderr,
-            )
-            print(
-                "Verify the payload tag exists (e.g., 4.22.0-0.ci-2026-02-06-195709).",
-                file=sys.stderr,
-            )
-            sys.exit(1)
-        else:
-            print(f"Error: HTTP {e.code} from Sippy API: {e.reason}", file=sys.stderr)
-            sys.exit(1)
-    except urllib.error.URLError as e:
-        print(f"Error: Failed to connect to Sippy API: {e.reason}", file=sys.stderr)
-        print("Check network connectivity.", file=sys.stderr)
-        sys.exit(1)
-    except Exception as e:
-        print(f"Error: {e}", file=sys.stderr)
-        sys.exit(1)
+        idx = tags.index(payload_tag)
+    except ValueError:
+        return None
+    if idx + 1 >= len(tags):
+        return None
+    return tags[idx + 1]
+
+
+def fetch_from_sippy(payload_tag: str) -> Optional[list]:
+    """Try fetching PR diff from Sippy. Returns None if the payload is not available."""
+    url = f"{SIPPY_API_BASE}/payloads/diff?toPayload={urllib.parse.quote(payload_tag)}"
+    try:
+        data = _http_get_json(url)
+    except urllib.error.HTTPError:
+        return None
+    except urllib.error.URLError:
+        return None
 
     if not isinstance(data, list):
-        print("Error: Unexpected API response format (expected a JSON array).", file=sys.stderr)
-        sys.exit(1)
+        return None
 
-    # Transform the response into a cleaner structure
     prs = []
     for entry in data:
-        pr = {
+        prs.append({
             "url": entry.get("url", ""),
             "pull_request_id": entry.get("pull_request_id", ""),
             "component": entry.get("name", ""),
             "description": entry.get("description", ""),
             "bug_url": entry.get("bug_url", ""),
-        }
-        prs.append(pr)
+        })
+    return prs
+
+
+def fetch_from_release_controller(payload_tag: str) -> list:
+    """Fetch PR diff from the release controller API."""
+    stream = _parse_stream(payload_tag)
+    if not stream:
+        print(
+            f"Error: Cannot parse release stream from tag '{payload_tag}'.",
+            file=sys.stderr,
+        )
+        sys.exit(1)
+
+    prev_tag = _find_previous_tag(stream, payload_tag)
+    if not prev_tag:
+        print(
+            f"Error: Cannot find previous payload for '{payload_tag}' in stream '{stream}'.",
+            file=sys.stderr,
+        )
+        sys.exit(1)
+
+    url = (
+        f"{RELEASE_CONTROLLER_BASE}/releasestream/{urllib.parse.quote(stream)}"
+        f"/release/{urllib.parse.quote(payload_tag)}"
+        f"?from={urllib.parse.quote(prev_tag)}"
+    )
+
+    try:
+        data = _http_get_json(url, timeout=60)
+    except urllib.error.HTTPError as e:
+        print(f"Error: HTTP {e.code} from release controller: {e.reason}", file=sys.stderr)
+        sys.exit(1)
+    except urllib.error.URLError as e:
+        print(f"Error: Failed to connect to release controller: {e.reason}", file=sys.stderr)
+        sys.exit(1)
+
+    changelog = data.get("changeLogJson", {})
+    updated_images = changelog.get("updatedImages", []) or []
+
+    prs = []
+    for image in updated_images:
+        component = image.get("name", "")
+        for commit in image.get("commits", []):
+            pull_url = commit.get("pullURL", "")
+            pull_id = str(commit.get("pullID", ""))
+            description = commit.get("subject", "")
+            issues = commit.get("issues", {}) or {}
+            bug_url = next(iter(issues.values()), "")
+            prs.append({
+                "url": pull_url,
+                "pull_request_id": pull_id,
+                "component": component,
+                "description": description,
+                "bug_url": bug_url,
+            })
 
     return prs
 
 
+def fetch_new_prs(payload_tag: str) -> list:
+    """Fetch new PRs, trying Sippy first then falling back to the release controller."""
+    prs = fetch_from_sippy(payload_tag)
+    if prs is not None:
+        return prs
+
+    print(
+        f"Sippy does not have payload '{payload_tag}', "
+        "falling back to release controller.",
+        file=sys.stderr,
+    )
+    return fetch_from_release_controller(payload_tag)
+
+
 def format_summary(prs: list, payload_tag: str) -> str:
     """Format PRs as a human-readable summary."""
     lines = []

plugins/ci/skills/prow-job-analyze-test-failure/SKILL.md

@@ -84,6 +84,52 @@ Use the `fetch-prowjob-json` skill to fetch the prowjob.json for this job. See `
 
 ### Step 4: Analyze Test Failure
 
+### Step 4.0: Detect Aggregated Jobs
+
+Aggregated jobs run the same job in parallel (typically 10 times) and perform statistical
+analysis of test results. Detect aggregation by checking for `aggregated-` prefix in the job
+name or an `aggregator` container/step in prowjob.json.
+
+If the job is aggregated, the failure modes are different from normal jobs:
+
+1. **Statistically significant test failure** — The test itself fails frequently enough across
+   runs to be flagged as a regression. This is a real regression that needs investigation.
+
+2. **Insufficient completed runs** — Not enough runs completed successfully to perform the
+   statistical test (e.g., only 5 of 10 jobs produced results). This manifests as mass test
+   failures across many unrelated tests. The root cause is whatever prevented the other runs
+   from completing — this could be infrastructure issues, install failures, or an actual product
+   bug causing crashes. Investigate the underlying job runs that did not complete to determine
+   why.
+
+3. **Non-deterministic test presence** — A test only ran in a small subset of completed jobs,
+   even though the other jobs completed successfully. The failure message will say something like
+   "Passed 1 times, failed 0 times, skipped 0 times: we require at least 6 attempts to have a
+   chance at success". Every test must produce results in every job run; it is a bug if it does
+   not. This is a regression — someone introduced a test that doesn't produce results
+   deterministically. Investigate which test is non-deterministic and why it only runs in some
+   jobs.
+
+When analyzing an aggregated job failure, first determine which failure mode applies before
+diving into individual test analysis. For mode 2, focus on why runs failed rather than
+individual test results. For mode 3, investigate why the test only ran in some jobs.
+
+**Finding underlying job run URLs:**
+
+The aggregated junit XML contains links to every underlying job run. Download it from:
+```
+gs://test-platform-results/{bucket-path}/artifacts/release-analysis-aggregator/openshift-release-analysis-aggregator/artifacts/release-analysis-aggregator/{job-name}/{payload-tag}/junit-aggregated.xml
+```
+
+Each `<testcase>` element has a `<system-out>` with YAML-formatted data including `passes:`,
+`failures:`, and `skips:` lists. Each entry has:
+- `jobrunid`: The build ID of the underlying job run
+- `humanurl`: Prow URL for the job run (e.g., `https://prow.ci.openshift.org/view/gs/test-platform-results/logs/{job-name}/{jobrunid}`)
+- `gcsartifacturl`: Direct link to GCS artifacts
+
+Use the `humanurl` links to investigate individual job run failures with the normal
+(non-aggregated) analysis steps below.
+
 ### Step 4.1: Download build-log.txt
 
 ```bash