Payload analysis skill improvements

.claude-plugin/marketplace.json

@@ -26,7 +26,7 @@
       "name": "ci",
       "source": "./plugins/ci",
       "description": "A plugin to work with OpenShift CI and analyze Prow job results",
-      "version": "0.0.21"
+      "version": "0.0.22"
     },
     {
       "name": "teams",

docs/data.json

@@ -528,9 +528,14 @@
           "description": "Create or update a Component Readiness triage record linking regressions to a JIRA bug",
           "id": "triage-regression",
           "name": "Triage Regression"
+        },
+        {
+          "description": "Trigger payload validation jobs on a PR and collect the resulting Prow job URLs",
+          "id": "trigger-payload-job",
+          "name": "Trigger Payload Job"
         }
       ],
-      "version": "0.0.21"
+      "version": "0.0.22"
     },
     {
       "commands": [

plugins/ci/.claude-plugin/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "ci",
   "description": "Tools for working with OpenShift CI and analyzing Prow job results",
-  "version": "0.0.21",
+  "version": "0.0.22",
   "author": {
     "name": "openshift"
   }

plugins/ci/skills/bisect-payload-suspects/SKILL.md

@@ -30,7 +30,7 @@ Use this skill when the `payload-agent` orchestrator identifies suspect PRs with
 
 ### Phase 1: Set Up Experiments
 
-For each medium-confidence suspect, launch a **parallel subagent** (Task tool, `subagent_type: "general-purpose"`, do NOT set the `model` parameter):
+For each medium-confidence suspect, launch a **parallel subagent** (do NOT set the `model` parameter):
 
 #### 1.1: Check for Merge Conflicts
 
@@ -71,36 +71,21 @@ Load the `revert-pr` skill and follow its workflow with `--draft`:
 
 Record the draft revert PR URL.
 
-#### 1.3: Trigger Payload Jobs
+#### 1.3: Trigger Payload Jobs and Collect Run URLs
 
-Post a comment on the draft revert PR with payload test commands for each failing blocking job attributed to this suspect:
+Use the `trigger-payload-job` skill (`plugins/ci/skills/trigger-payload-job/SKILL.md`) to trigger payload validation jobs on the draft revert PR and collect the resulting URLs. Pass:
 
-- **Aggregated jobs**: Use `/payload-aggregate <underlying-job-name> <count>`
-- **Non-aggregated jobs**: Use `/payload-job <job-name>`
+- `pr_url`: The draft revert PR URL
+- `jobs`: The `failing_jobs` list for this suspect (includes `job_name`, `is_aggregated`, `underlying_job_name` for each job)
 
-```bash
-gh pr comment "{draft_pr_url}" --body "/payload-aggregate {underlying_job_name_1} {count}
-/payload-job {job_name_2}
-..."
-```
-
-#### 1.4: Collect Run URLs
-
-After posting the comment, wait ~30 seconds and poll for the `openshift-ci[bot]` reply:
-
-```bash
-gh api "repos/<org>/<repo>/issues/<pr_number>/comments" \
-  --jq '[.[] | select(.user.login == "openshift-ci[bot]" and (.body | contains("pr-payload-tests")))] | last | .body'
-```
-
-Extract the `pr-payload-tests.ci.openshift.org/runs/ci/<uuid>` URL. Fetch the page to extract individual prow job URLs.
-
-#### 1.5: Record Experiment
+#### 1.4: Record Experiment
 
 Record the experiment data for the tracking YAML (see schema below).
 
 **Throttling**: Never bisect more than 5 suspects. If there are more than 5, test only the top 5 by confidence score. Record the remainder as `deferred_suspects` in the tracking YAML.
 
+**Job triggering limits**: Respect the global limits set by the `payload-agent` orchestrator. Across all bisect experiments combined: trigger at most 5 non-aggregated jobs and at most 1 aggregated job. Prioritize jobs from higher-confidence suspects. Skip excess jobs and record them in the tracking YAML as skipped due to limits.
+
 ### Write Tracking YAML
 
 After all Phase 1 subagents complete, write `<payload-tag>-bisect.yaml` to the current working directory with all experiment data:
@@ -220,5 +205,6 @@ Return results to the caller for inclusion in the final report.
 ## See Also
 
 - Related Skill: `revert-pr` - The git revert workflow (`plugins/ci/skills/revert-pr/SKILL.md`)
+- Related Skill: `trigger-payload-job` - Triggers payload jobs and collects URLs (`plugins/ci/skills/trigger-payload-job/SKILL.md`)
 - Related Skill: `stage-payload-reverts` - Stages high-confidence reverts (`plugins/ci/skills/stage-payload-reverts/SKILL.md`)
 - Related Command: `/ci:payload-agent` - Autonomous orchestrator that uses this skill (`plugins/ci/commands/payload-agent.md`)

plugins/ci/skills/fetch-payloads/SKILL.md

@@ -58,7 +58,7 @@ The script outputs one block per payload to stdout with job details. Present to
 For each payload, the script outputs:
 
 - **Tag line**: `<tag>  (<phase>)  <timestamp>  <url>`
-- **Rejected payloads**: lists each failed blocking job with retry count and Prow link
+- **Rejected payloads**: lists each failed blocking job with retry count, Prow link for the final attempt, and Prow links for all previous attempts
 - **Ready payloads**: summary of succeeded/pending/failed counts
 - **Accepted payloads**: confirmation that all blocking jobs succeeded
 

plugins/ci/skills/fetch-payloads/fetch_payloads.py

@@ -141,6 +141,9 @@ def format_payload(tag: dict, details: dict, architecture: str, stream_name: str
             lines.append(f"    FAILED  {job_name}{retry_str}")
             if prow_url:
                 lines.append(f"            {prow_url}")
+            previous = info.get("previousAttemptURLs") or []
+            for i, prev_url in enumerate(previous, 1):
+                lines.append(f"            attempt {i}: {prev_url}")
 
     return "\n".join(lines)
 

plugins/ci/skills/payload-agent/SKILL.md

@@ -99,6 +99,17 @@ Based on the confidence tiers, take autonomous action:
 
 **Execute HIGH and MEDIUM actions in parallel** when both tiers have candidates. Pass all required context in-memory to each skill.
 
+**Job triggering limits**: The total number of payload jobs triggered across ALL suspects (both HIGH and MEDIUM tiers combined) must respect these limits:
+
+- **Non-aggregated jobs**: Up to 5 total across all suspects
+- **Aggregated jobs**: Up to 1 initially. A second aggregated job may only be triggered if the first one's results are needed to confirm a finding (e.g., during bisect Phase 2 confirmation). Never trigger more than 2 aggregated jobs total.
+
+When the number of failing jobs across all suspects exceeds these limits, prioritize jobs from higher-confidence suspects first. For aggregated jobs, pick the single most important one (highest confidence suspect, most critical job). Record any jobs that were skipped due to limits in the report as "skipped — job trigger limit reached".
+
+**Critical — Aggregated job handling**: When constructing the `failing_jobs` list for each suspect, you MUST correctly populate `is_aggregated` and `underlying_job_name` using the subagent analysis results from Step 2. Jobs with the `aggregated-` prefix are aggregated jobs. The `underlying_job_name` is extracted by the subagent from the junit-aggregated.xml artifacts (see `analyze-payload` Step 5). Both `bisect-payload-suspects` and `stage-payload-reverts` use the `trigger-payload-job` skill to post the correct commands — but they depend on the caller providing accurate `is_aggregated` and `underlying_job_name` values.
+
+**Fail-fast validation**: After assembling the `failing_jobs` list from subagent results and before passing it to `bisect-payload-suspects` or `stage-payload-reverts`, validate each entry: if `is_aggregated` is true but `underlying_job_name` is empty or null, do NOT enqueue that job for payload testing. Instead, record a hard error for that suspect in the report (e.g., "Cannot trigger payload test: aggregated job missing underlying_job_name from subagent analysis") and continue with the remaining suspects. This prevents silent misuse of `trigger-payload-job` with an aggregated job that has no underlying job name.
+
 If there are no HIGH or MEDIUM candidates, skip to Step 6 (report generation).
 
 ### Step 5: Bisect Phase 2 (Resume)
@@ -261,3 +272,4 @@ After report generation:
 - Related Skill: `analyze-payload` - Core analysis logic (`plugins/ci/skills/analyze-payload/SKILL.md`)
 - Related Skill: `stage-payload-reverts` - High-confidence revert staging (`plugins/ci/skills/stage-payload-reverts/SKILL.md`)
 - Related Skill: `bisect-payload-suspects` - Medium-confidence bisect experiments (`plugins/ci/skills/bisect-payload-suspects/SKILL.md`)
+- Related Skill: `trigger-payload-job` - Triggers payload jobs and collects URLs (`plugins/ci/skills/trigger-payload-job/SKILL.md`)

plugins/ci/skills/stage-payload-reverts/SKILL.md

@@ -29,7 +29,7 @@ Use this skill when revert candidates have already been identified with high con
 
 ## Implementation Steps
 
-For each qualifying revert candidate, launch a **parallel subagent** (Task tool, `subagent_type: "general-purpose"`, do NOT set the `model` parameter). Each subagent executes three substeps in order:
+For each qualifying revert candidate, launch a **parallel subagent** (do NOT set the `model` parameter). Each subagent executes three substeps in order:
 
 ### Substep 1: Create TRT JIRA Bug (with idempotency check)
 
@@ -89,50 +89,16 @@ If an open or draft revert PR is found for this PR number, reuse its URL — ski
 
 Record the revert PR URL (created or reused).
 
-### Substep 3: Trigger Payload Jobs and Collect Run URLs (with idempotency check)
+### Substep 3: Trigger Payload Jobs and Collect Run URLs
 
-**Preflight**: Before posting new payload commands, check whether jobs were already triggered on this revert PR:
+Use the `trigger-payload-job` skill (`plugins/ci/skills/trigger-payload-job/SKILL.md`) to trigger payload validation jobs on the revert PR and collect the resulting URLs. Pass:
 
-```bash
-# Check for an existing openshift-ci[bot] reply with a pr-payload-tests URL
-gh api "repos/<org>/<repo>/issues/<revert_pr_number>/comments" \
-  --jq '[.[] | select(.user.login == "openshift-ci[bot]" and (.body | contains("pr-payload-tests")))] | last | .body'
-```
-
-If a `pr-payload-tests.ci.openshift.org/runs/ci/<uuid>` URL is found, reuse it — extract the prow job URLs from that page and skip posting new commands.
-
-**Trigger** (only if no existing payload test run found): Post a comment on the revert PR with payload test commands for each failing blocking job attributed to this PR. Use the correct command based on whether the job is aggregated:
-
-- **Aggregated jobs** (job name has `aggregated-` prefix): Use `/payload-aggregate <underlying-job-name> <count>`. The underlying job name comes from the caller's analysis data (extracted from the aggregated job's junit artifacts — it cannot be derived from the aggregated job name). Choose a count of up to 10 runs — use judgement based on the total number of jobs being triggered (fewer runs per job when triggering many jobs to limit resource consumption; more runs when only one or two jobs need validation).
-- **Non-aggregated jobs**: Use `/payload-job <job-name>`.
-
-```bash
-# Example with a mix of aggregated and non-aggregated jobs:
-gh pr comment "{revert_pr_url}" --body "/payload-aggregate {underlying_job_name_1} {count}
-/payload-job {job_name_2}
-..."
-```
-
-One command per line for each failing blocking job attributed to this PR.
+- `pr_url`: The revert PR URL from Substep 2
+- `jobs`: The `failing_jobs` list for this candidate (includes `job_name`, `is_aggregated`, `underlying_job_name` for each job)
 
-After posting the comment, wait ~30 seconds and poll for the `openshift-ci[bot]` reply containing the `pr-payload-tests` URL:
-
-```bash
-# Poll for the bot reply containing the pr-payload-tests URL
-gh api "repos/<org>/<repo>/issues/<revert_pr_number>/comments" \
-  --jq '[.[] | select(.user.login == "openshift-ci[bot]" and (.body | contains("pr-payload-tests")))] | last | .body'
-```
-
-Extract the `pr-payload-tests.ci.openshift.org/runs/ci/<uuid>` URL from the reply. This URL is the primary endpoint for checking job completion status (the page shows "AllJobsFinished" when done).
-
-Fetch the pr-payload-tests page to extract the actual prow job URL(s):
-
-```bash
-# The page contains prow job links like:
-# https://prow.ci.openshift.org/view/gs/test-platform-results/logs/<job-slug>/<build-id>
-```
+The skill handles idempotency (checking for existing bot replies), correct command selection (`/payload-aggregate` vs `/payload-job`), polling, and URL extraction.
 
-Record both the `payload_test_url` and individual `prow_url`s in the return data.
+Record the `payload_test_url` and individual `prow_url`s from the skill's return data.
 
 ## Subagent Return Format
 
@@ -165,5 +131,6 @@ Collect all subagent results. Return to the caller for inclusion in the report.
 ## See Also
 
 - Related Skill: `revert-pr` - The git revert workflow (`plugins/ci/skills/revert-pr/SKILL.md`)
+- Related Skill: `trigger-payload-job` - Triggers payload jobs and collects URLs (`plugins/ci/skills/trigger-payload-job/SKILL.md`)
 - Related Skill: `analyze-payload` - Identifies revert candidates (`plugins/ci/skills/analyze-payload/SKILL.md`)
 - Related Command: `/ci:payload-agent` - Autonomous orchestrator that uses this skill (`plugins/ci/commands/payload-agent.md`)

plugins/ci/skills/trigger-payload-job/SKILL.md

@@ -0,0 +1,132 @@
+---
+name: Trigger Payload Job
+description: Trigger payload validation jobs on a PR and collect the resulting Prow job URLs
+---
+
+# Trigger Payload Job
+
+This skill triggers payload validation jobs on a GitHub PR by posting the correct Prow commands as PR comments, then polls for the bot response to collect the resulting job URLs.
+
+## When to Use This Skill
+
+Use this skill whenever you need to trigger payload testing on a PR (revert PR, draft bisect PR, or any PR that needs payload validation). It handles the differences between aggregated and non-aggregated jobs and collects the resulting URLs for downstream tracking.
+
+**Inputs** (passed in-context by the caller):
+
+- `pr_url`: The GitHub PR URL to trigger jobs on (e.g., `https://github.com/openshift/ovn-kubernetes/pull/3040`)
+- `org`, `repo`, `pr_number`: Parsed from the PR URL
+- `jobs`: List of jobs to trigger, each with:
+  - `job_name`: The full job name (e.g., `aggregated-hypershift-ovn-conformance-4.22`)
+  - `is_aggregated`: Whether this is an aggregated job (has `aggregated-` prefix)
+  - `underlying_job_name`: For aggregated jobs only — the underlying periodic job name (e.g., `periodic-ci-openshift-hypershift-release-4.22-periodics-e2e-aws-ovn-conformance`). This MUST be extracted from the aggregated job's junit artifacts by the caller's analysis — it cannot be derived from the aggregated job name.
+  - `count`: For aggregated jobs only — how many runs (default: 10). Use judgement: fewer runs when triggering many jobs to limit resource consumption, more when only one or two jobs need validation.
+
+## Prerequisites
+
+1. **GitHub CLI (`gh`)**: Installed and authenticated
+2. **Repository Access**: User must have permission to comment on the PR
+
+## Job Triggering Limits
+
+The caller (typically `payload-agent`) is responsible for enforcing global limits before invoking this skill. However, this skill also enforces the following hard limits per invocation:
+
+- **Non-aggregated jobs**: No more than 5 per comment
+- **Aggregated jobs**: No more than 1 per comment. Aggregated jobs are expensive (they run many iterations), so only trigger one at a time. A second aggregated job should only be triggered after the first completes, and only if confirmation is needed.
+
+If the `jobs` input exceeds these limits, trigger only the allowed number (prioritizing by the order provided) and return the remainder as `skipped` in the return format.
+
+## Implementation Steps
+
+### Step 1: Idempotency Check
+
+Before posting new payload commands, check whether jobs were already triggered on this PR:
+
+```bash
+gh api "repos/<org>/<repo>/issues/<pr_number>/comments?per_page=100&sort=created&direction=desc" \
+  --jq '[.[] | select(.user.login == "openshift-ci[bot]" and (.body | contains("pr-payload-tests")))] | .[0] | .body'
+```
+
+If a `pr-payload-tests.ci.openshift.org/runs/ci/<uuid>` URL is found, reuse it — skip to Step 3 to extract prow job URLs from that page.
+
+### Step 2: Post Payload Commands
+
+First, validate each job entry. If any job has `is_aggregated == true` but `underlying_job_name` is empty or null, skip that job and return an error for it — do NOT post a comment with an incomplete command.
+
+Build the comment body with one command per line. Use the correct command for each job type:
+
+- **Aggregated jobs** (`is_aggregated == true`): `/payload-aggregate <underlying_job_name> <count>`
+- **Non-aggregated jobs** (`is_aggregated == false`): `/payload-job <job_name>`
+
+```bash
+gh pr comment "<pr_url>" --body "<commands>"
+```
+
+**Example** with a mix of job types:
+
+```bash
+gh pr comment "https://github.com/openshift/ovn-kubernetes/pull/3040" --body "/payload-aggregate periodic-ci-openshift-hypershift-release-4.22-periodics-e2e-aws-ovn-conformance 10
+/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-aws-ovn"
+```
+
+**Common mistakes to avoid**:
+- Do NOT use `/payload <version> <stream> <filter>` — this triggers a different workflow and does NOT include the PR's changes in the payload
+- Do NOT use the aggregated job name with `/payload-job` — aggregated jobs are not directly triggerable; you must use `/payload-aggregate` with the underlying job name
+- Do NOT use `/payload-aggregate` for non-aggregated jobs
+
+### Step 3: Poll for Bot Response
+
+After posting the comment, wait ~30 seconds for the `openshift-ci[bot]` to process the commands, then poll for its reply:
+
+```bash
+sleep 30
+gh api "repos/<org>/<repo>/issues/<pr_number>/comments?per_page=100&sort=created&direction=desc" \
+  --jq '[.[] | select(.user.login == "openshift-ci[bot]" and (.body | contains("pr-payload-tests")))] | .[0] | .body'
+```
+
+If no reply is found, retry up to 3 times with 30-second intervals. If still no reply after ~2 minutes, record the comment URL and note that manual checking is required.
+
+### Step 4: Extract URLs
+
+From the bot reply, extract:
+
+1. **`payload_test_url`**: The `pr-payload-tests.ci.openshift.org/runs/ci/<uuid>` URL. This is the primary endpoint for checking overall job completion status — the page shows "AllJobsFinished" when all triggered jobs have completed.
+
+2. **Individual prow job URLs**: Fetch the `payload_test_url` page and extract prow job links:
+
+```bash
+curl -sL "<payload_test_url>" | grep -oE 'https://prow\.ci\.openshift\.org/view/gs/test-platform-results/logs/[^"]+' | sort -u
+```
+
+Each prow URL corresponds to one triggered job run.
+
+## Return Format
+
+Return the collected data in this format:
+
+```
+PAYLOAD_JOB_RESULT:
+  pr_url: <the PR URL>
+  comment_url: <URL of the comment posted>
+  payload_test_url: <pr-payload-tests URL, or empty if not found>
+  prow_jobs:
+    - job_name: <job name>
+      prow_url: <individual prow URL>
+    - ...
+  skipped_jobs:
+    - job_name: <job name>
+      reason: "job trigger limit reached (max 5 non-aggregated, max 1 aggregated per invocation)"
+    - ...
+  status: triggered|reused|no_response|failed
+  error: none|<description>
+```
+
+## Error Handling
+
+- If the bot never replies, return `status: no_response` with the comment URL so the caller can check manually later.
+- If the PR doesn't exist or the user lacks permissions, return `status: failed` with the error message.
+- Do not retry indefinitely — 3 poll attempts is the maximum.
+
+## See Also
+
+- Used by: `bisect-payload-suspects` — triggers payload jobs for draft bisect PRs
+- Used by: `stage-payload-reverts` — triggers payload jobs for revert PRs