diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index d1ef68cb8..71dcd9a3f 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -26,7 +26,7 @@ "name": "ci", "source": "./plugins/ci", "description": "A plugin to work with OpenShift CI and analyze Prow job results", - "version": "0.0.21" + "version": "0.0.22" }, { "name": "teams", diff --git a/docs/data.json b/docs/data.json index cdbf651c2..69942d138 100644 --- a/docs/data.json +++ b/docs/data.json @@ -528,9 +528,14 @@ "description": "Create or update a Component Readiness triage record linking regressions to a JIRA bug", "id": "triage-regression", "name": "Triage Regression" + }, + { + "description": "Trigger payload validation jobs on a PR and collect the resulting Prow job URLs", + "id": "trigger-payload-job", + "name": "Trigger Payload Job" } ], - "version": "0.0.21" + "version": "0.0.22" }, { "commands": [ diff --git a/plugins/ci/.claude-plugin/plugin.json b/plugins/ci/.claude-plugin/plugin.json index 16e428f81..f630933db 100644 --- a/plugins/ci/.claude-plugin/plugin.json +++ b/plugins/ci/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "ci", "description": "Tools for working with OpenShift CI and analyzing Prow job results", - "version": "0.0.21", + "version": "0.0.22", "author": { "name": "openshift" } diff --git a/plugins/ci/skills/analyze-payload/SKILL.md b/plugins/ci/skills/analyze-payload/SKILL.md index 4a133d5be..59f9b93cb 100644 --- a/plugins/ci/skills/analyze-payload/SKILL.md +++ b/plugins/ci/skills/analyze-payload/SKILL.md @@ -62,9 +62,11 @@ Parse the output to extract payload tag names, phases, and job details. Find the **target payload** (the tag from Step 1) in the fetched list. Based on its phase: -- **Rejected**: Extract all failed blocking job names and their Prow URLs. Proceed with full analysis. -- **Ready**: Extract blocking jobs that have already **failed** (with their Prow URLs). These are jobs that will not pass — they indicate the payload is on track for rejection. Proceed with analysis of those failed jobs and note in the report that the payload is still in progress. -- **Accepted**: Extract any failed blocking job names and their Prow URLs. Payloads can be force-accepted despite blocking job failures, so do NOT assume all blocking jobs passed. If there are failed blocking jobs, proceed with full analysis and note in the report that the payload was accepted despite these failures. If there are truly no failed blocking jobs, report "Payload was accepted with all blocking jobs passing, no analysis needed" and exit. +- **Rejected**: Extract all failed blocking job names, their Prow URLs, and any previous attempt URLs. Proceed with full analysis. +- **Ready**: Extract blocking jobs that have already **failed** (with their Prow URLs and previous attempt URLs). These are jobs that will not pass — they indicate the payload is on track for rejection. Proceed with analysis of those failed jobs and note in the report that the payload is still in progress. +- **Accepted**: Extract any failed blocking job names, their Prow URLs, and previous attempt URLs. Payloads can be force-accepted despite blocking job failures, so do NOT assume all blocking jobs passed. If there are failed blocking jobs, proceed with full analysis and note in the report that the payload was accepted despite these failures. If there are truly no failed blocking jobs, report "Payload was accepted with all blocking jobs passing, no analysis needed" and exit. + +The release controller API returns `previousAttemptURLs` for jobs that were retried. For each failed job, collect the final Prow URL and all previous attempt URLs. These are available in the `fetch-payloads` output as `attempt N: ` lines below the main URL. ### Step 3: Build Failure History (Lookback) @@ -99,13 +101,19 @@ Store the PR data keyed by originating payload tag. These PRs are the **suspects ### Step 5: Investigate Each Failed Job in Parallel -For each failed blocking job in the **target payload**, launch a **parallel subagent** (using the Task tool) to investigate the failure. Use the Prow URL from Step 2. +For each failed blocking job in the **target payload**, launch a **parallel subagent** to investigate the failure. Pass the subagent the final Prow URL **and** all previous attempt URLs from Step 2. Each subagent should determine whether the failure is an install failure or a test failure by checking the JUnit results (e.g., look for `install should succeed*` test failures), then use the appropriate analysis skill. Almost all blocking jobs install a cluster and then run tests, so the job name alone does not tell you the failure type. Instruct each subagent as follows: -> Analyze the failure at . First, check the JUnit results or build log to determine whether this is an install failure (look for `install should succeed: overall` or similar install-related test failures) or a test failure (install passed, specific tests failed). +> Analyze the failure at . This job had retries. The previous attempt URLs are: . +> +> **Aggregated jobs**: If this is an aggregated job (has `aggregated-` prefix or an `aggregator` step), retries only re-run the aggregation analysis — they do NOT re-run the underlying test jobs. Therefore, only examine the most recent attempt; previous attempts contain the same underlying results and do not provide additional signal. +> +> **Non-aggregated jobs**: **Examine the final attempt first**, then compare with previous attempts to determine whether all retries failed the same way. If retries show different failure modes, note this — it distinguishes consistent regressions from intermittent/infrastructure issues. Consistent failures across all attempts strongly indicate a product regression rather than flakiness. +> +> First, check the JUnit results or build log to determine whether this is an install failure (look for `install should succeed: overall` or similar install-related test failures) or a test failure (install passed, specific tests failed). > > Based on the failure type, use the appropriate skill: > - **Install failure**: Use the `ci:prow-job-analyze-install-failure` skill. **You MUST download and examine the actual installer log bundle** — do NOT skip this step or make assessments based only on high-level metadata like pass rates or job names. The log bundle contains the actual error messages that reveal the root cause. For metal/bare-metal jobs (job name contains "metal"), perform additional analysis using the `ci:prow-job-analyze-metal-install-failure` skill as needed for dev-scripts, Metal3/Ironic, and BareMetalHost-specific diagnostics. @@ -127,11 +135,13 @@ ANALYSIS_RESULT: - key_error_patterns: - known_symptoms: - underlying_job_name: +- retries_consistent: yes|no|no_retries|only_final_examined +- retry_summary: ``` This structured format enables downstream consumers (like the `payload-agent` skill) to programmatically extract analysis results for confidence scoring. -**Important**: Launch ALL subagents in parallel (single message with multiple Task tool calls) for maximum speed. Each subagent should be given `subagent_type: "general-purpose"`. Do NOT set the `model` parameter — let subagents inherit the parent model, as these analysis tasks require a capable model. +**Important**: Launch ALL subagents in parallel for maximum speed. Do NOT set the `model` parameter — let subagents inherit the parent model, as these analysis tasks require a capable model. #### Cross-Platform and Cross-Job Failure Pattern Recognition @@ -239,7 +249,7 @@ A table showing ALL blocking jobs with columns: - Job Name - Status (color-coded: green for passed, red for failed) - Streak (how many consecutive payloads it has been failing; "N/A" for passed jobs) -- History (the failure_pattern across the lookback window, e.g., "F F F S F F", showing most recent first; use color-coded markers — red for F, green for S) +- History (the failure_pattern across the lookback window, e.g., "F F F S F F", showing most recent first; use color-coded markers — red for F, green for S. Each marker should be a link to that job's Prow URL from that payload, when available from the lookback data) - First Failed In (originating payload tag, linked to release controller) #### 7.3: Failed Job Details @@ -329,166 +339,105 @@ After the revert is merged and payloads are green again, the original author can ``` -Add the following styles for the revert prompt block: +Use `verdict-revert` for the revert section when there are revert candidates, and `verdict-none` when there are none. The revert prompt copy button should use the same variable-based styling: ```html .revert-prompt { position: relative; - margin: 12px 0; + margin: 0.75rem 0; } .revert-prompt pre { - background: #1e1e1e; - color: #d4d4d4; - padding: 16px; - border-radius: 4px; white-space: pre-wrap; - font-family: 'SFMono-Regular', Consolas, monospace; - font-size: 13px; - overflow-x: auto; } .revert-prompt button { position: absolute; top: 8px; right: 8px; - background: #3c3c3c; - color: #d4d4d4; - border: 1px solid #555; + background: var(--surface); + color: var(--text-muted); + border: 1px solid var(--border); border-radius: 4px; padding: 4px 10px; cursor: pointer; font-size: 12px; } .revert-prompt button:hover { - background: #505050; + border-color: var(--blue); + color: var(--text); } ``` If **no** revert candidates were identified, include a brief note instead: ```html -
-

Recommended Reverts

+
+ No Recommended Reverts

No PRs were identified with sufficient confidence for revert recommendation. Failures may be caused by infrastructure issues, flaky tests, or require further investigation.

``` -##### Shared styles for the revert section - -Add the following styles for both modes: - -```html -.revert-recommendations { - background: white; - border-left: 4px solid #d93025; - padding: 16px 20px; - margin: 20px 0; - border-radius: 4px; - box-shadow: 0 1px 3px rgba(0,0,0,0.1); -} -.revert-recommendations h2 { - color: #d93025; - border-bottom: none; -} -.revert-none { - border-left-color: #5f6368; -} -.revert-none h2 { - color: #5f6368; -} -``` - #### 7.5: Styling -The HTML must be fully self-contained with embedded CSS. Use a clean, professional design: +The HTML must be fully self-contained with embedded CSS. Use a GitHub-inspired dark mode design. Wrap all content in a `
`. Use CSS variables for the color palette and the following base styles as a guide: ```html ``` +You may add additional classes as needed (e.g., history markers, timeline items, pattern groups) following the same variable-based color palette. Use `var(--red)` / `var(--green)` for fail/pass indicators, `var(--orange)` for infrastructure issues, and `var(--blue)` for links and highlights. + ### Step 8: Generate JSON Data File After generating the HTML report, produce a single structured JSON data file for database ingestion. The file contains one flat table with **one row per (failed blocking job, revert candidate) pair**: diff --git a/plugins/ci/skills/bisect-payload-suspects/SKILL.md b/plugins/ci/skills/bisect-payload-suspects/SKILL.md index 06f30c23a..9fb285c66 100644 --- a/plugins/ci/skills/bisect-payload-suspects/SKILL.md +++ b/plugins/ci/skills/bisect-payload-suspects/SKILL.md @@ -30,7 +30,7 @@ Use this skill when the `payload-agent` orchestrator identifies suspect PRs with ### Phase 1: Set Up Experiments -For each medium-confidence suspect, launch a **parallel subagent** (Task tool, `subagent_type: "general-purpose"`, do NOT set the `model` parameter): +For each medium-confidence suspect, launch a **parallel subagent** (do NOT set the `model` parameter): #### 1.1: Check for Merge Conflicts @@ -71,36 +71,21 @@ Load the `revert-pr` skill and follow its workflow with `--draft`: Record the draft revert PR URL. -#### 1.3: Trigger Payload Jobs +#### 1.3: Trigger Payload Jobs and Collect Run URLs -Post a comment on the draft revert PR with payload test commands for each failing blocking job attributed to this suspect: +Use the `trigger-payload-job` skill (`plugins/ci/skills/trigger-payload-job/SKILL.md`) to trigger payload validation jobs on the draft revert PR and collect the resulting URLs. Pass: -- **Aggregated jobs**: Use `/payload-aggregate ` -- **Non-aggregated jobs**: Use `/payload-job ` +- `pr_url`: The draft revert PR URL +- `jobs`: The `failing_jobs` list for this suspect (includes `job_name`, `is_aggregated`, `underlying_job_name` for each job) -```bash -gh pr comment "{draft_pr_url}" --body "/payload-aggregate {underlying_job_name_1} {count} -/payload-job {job_name_2} -..." -``` - -#### 1.4: Collect Run URLs - -After posting the comment, wait ~30 seconds and poll for the `openshift-ci[bot]` reply: - -```bash -gh api "repos///issues//comments" \ - --jq '[.[] | select(.user.login == "openshift-ci[bot]" and (.body | contains("pr-payload-tests")))] | last | .body' -``` - -Extract the `pr-payload-tests.ci.openshift.org/runs/ci/` URL. Fetch the page to extract individual prow job URLs. - -#### 1.5: Record Experiment +#### 1.4: Record Experiment Record the experiment data for the tracking YAML (see schema below). **Throttling**: Never bisect more than 5 suspects. If there are more than 5, test only the top 5 by confidence score. Record the remainder as `deferred_suspects` in the tracking YAML. +**Job triggering limits**: Respect the global limits set by the `payload-agent` orchestrator. Across all bisect experiments combined: trigger at most 5 non-aggregated jobs and at most 1 aggregated job. Prioritize jobs from higher-confidence suspects. Skip excess jobs and record them in the tracking YAML as skipped due to limits. + ### Write Tracking YAML After all Phase 1 subagents complete, write `-bisect.yaml` to the current working directory with all experiment data: @@ -220,5 +205,6 @@ Return results to the caller for inclusion in the final report. ## See Also - Related Skill: `revert-pr` - The git revert workflow (`plugins/ci/skills/revert-pr/SKILL.md`) +- Related Skill: `trigger-payload-job` - Triggers payload jobs and collects URLs (`plugins/ci/skills/trigger-payload-job/SKILL.md`) - Related Skill: `stage-payload-reverts` - Stages high-confidence reverts (`plugins/ci/skills/stage-payload-reverts/SKILL.md`) - Related Command: `/ci:payload-agent` - Autonomous orchestrator that uses this skill (`plugins/ci/commands/payload-agent.md`) diff --git a/plugins/ci/skills/fetch-payloads/SKILL.md b/plugins/ci/skills/fetch-payloads/SKILL.md index af985befd..df0430cf1 100644 --- a/plugins/ci/skills/fetch-payloads/SKILL.md +++ b/plugins/ci/skills/fetch-payloads/SKILL.md @@ -58,7 +58,7 @@ The script outputs one block per payload to stdout with job details. Present to For each payload, the script outputs: - **Tag line**: ` () ` -- **Rejected payloads**: lists each failed blocking job with retry count and Prow link +- **Rejected payloads**: lists each failed blocking job with retry count, Prow link for the final attempt, and Prow links for all previous attempts - **Ready payloads**: summary of succeeded/pending/failed counts - **Accepted payloads**: confirmation that all blocking jobs succeeded diff --git a/plugins/ci/skills/fetch-payloads/fetch_payloads.py b/plugins/ci/skills/fetch-payloads/fetch_payloads.py index 54fbdde4f..f1b2e20a2 100644 --- a/plugins/ci/skills/fetch-payloads/fetch_payloads.py +++ b/plugins/ci/skills/fetch-payloads/fetch_payloads.py @@ -141,6 +141,9 @@ def format_payload(tag: dict, details: dict, architecture: str, stream_name: str lines.append(f" FAILED {job_name}{retry_str}") if prow_url: lines.append(f" {prow_url}") + previous = info.get("previousAttemptURLs") or [] + for i, prev_url in enumerate(previous, 1): + lines.append(f" attempt {i}: {prev_url}") return "\n".join(lines) diff --git a/plugins/ci/skills/payload-agent/SKILL.md b/plugins/ci/skills/payload-agent/SKILL.md index e21b533f7..06c305049 100644 --- a/plugins/ci/skills/payload-agent/SKILL.md +++ b/plugins/ci/skills/payload-agent/SKILL.md @@ -99,6 +99,17 @@ Based on the confidence tiers, take autonomous action: **Execute HIGH and MEDIUM actions in parallel** when both tiers have candidates. Pass all required context in-memory to each skill. +**Job triggering limits**: The total number of payload jobs triggered across ALL suspects (both HIGH and MEDIUM tiers combined) must respect these limits: + +- **Non-aggregated jobs**: Up to 5 total across all suspects +- **Aggregated jobs**: Up to 1 initially. A second aggregated job may only be triggered if the first one's results are needed to confirm a finding (e.g., during bisect Phase 2 confirmation). Never trigger more than 2 aggregated jobs total. + +When the number of failing jobs across all suspects exceeds these limits, prioritize jobs from higher-confidence suspects first. For aggregated jobs, pick the single most important one (highest confidence suspect, most critical job). Record any jobs that were skipped due to limits in the report as "skipped — job trigger limit reached". + +**Critical — Aggregated job handling**: When constructing the `failing_jobs` list for each suspect, you MUST correctly populate `is_aggregated` and `underlying_job_name` using the subagent analysis results from Step 2. Jobs with the `aggregated-` prefix are aggregated jobs. The `underlying_job_name` is extracted by the subagent from the junit-aggregated.xml artifacts (see `analyze-payload` Step 5). Both `bisect-payload-suspects` and `stage-payload-reverts` use the `trigger-payload-job` skill to post the correct commands — but they depend on the caller providing accurate `is_aggregated` and `underlying_job_name` values. + +**Fail-fast validation**: After assembling the `failing_jobs` list from subagent results and before passing it to `bisect-payload-suspects` or `stage-payload-reverts`, validate each entry: if `is_aggregated` is true but `underlying_job_name` is empty or null, do NOT enqueue that job for payload testing. Instead, record a hard error for that suspect in the report (e.g., "Cannot trigger payload test: aggregated job missing underlying_job_name from subagent analysis") and continue with the remaining suspects. This prevents silent misuse of `trigger-payload-job` with an aggregated job that has no underlying job name. + If there are no HIGH or MEDIUM candidates, skip to Step 6 (report generation). ### Step 5: Bisect Phase 2 (Resume) @@ -261,3 +272,4 @@ After report generation: - Related Skill: `analyze-payload` - Core analysis logic (`plugins/ci/skills/analyze-payload/SKILL.md`) - Related Skill: `stage-payload-reverts` - High-confidence revert staging (`plugins/ci/skills/stage-payload-reverts/SKILL.md`) - Related Skill: `bisect-payload-suspects` - Medium-confidence bisect experiments (`plugins/ci/skills/bisect-payload-suspects/SKILL.md`) +- Related Skill: `trigger-payload-job` - Triggers payload jobs and collects URLs (`plugins/ci/skills/trigger-payload-job/SKILL.md`) diff --git a/plugins/ci/skills/stage-payload-reverts/SKILL.md b/plugins/ci/skills/stage-payload-reverts/SKILL.md index 36c559a2f..138cc5550 100644 --- a/plugins/ci/skills/stage-payload-reverts/SKILL.md +++ b/plugins/ci/skills/stage-payload-reverts/SKILL.md @@ -29,7 +29,7 @@ Use this skill when revert candidates have already been identified with high con ## Implementation Steps -For each qualifying revert candidate, launch a **parallel subagent** (Task tool, `subagent_type: "general-purpose"`, do NOT set the `model` parameter). Each subagent executes three substeps in order: +For each qualifying revert candidate, launch a **parallel subagent** (do NOT set the `model` parameter). Each subagent executes three substeps in order: ### Substep 1: Create TRT JIRA Bug (with idempotency check) @@ -89,50 +89,16 @@ If an open or draft revert PR is found for this PR number, reuse its URL — ski Record the revert PR URL (created or reused). -### Substep 3: Trigger Payload Jobs and Collect Run URLs (with idempotency check) +### Substep 3: Trigger Payload Jobs and Collect Run URLs -**Preflight**: Before posting new payload commands, check whether jobs were already triggered on this revert PR: +Use the `trigger-payload-job` skill (`plugins/ci/skills/trigger-payload-job/SKILL.md`) to trigger payload validation jobs on the revert PR and collect the resulting URLs. Pass: -```bash -# Check for an existing openshift-ci[bot] reply with a pr-payload-tests URL -gh api "repos///issues//comments" \ - --jq '[.[] | select(.user.login == "openshift-ci[bot]" and (.body | contains("pr-payload-tests")))] | last | .body' -``` - -If a `pr-payload-tests.ci.openshift.org/runs/ci/` URL is found, reuse it — extract the prow job URLs from that page and skip posting new commands. - -**Trigger** (only if no existing payload test run found): Post a comment on the revert PR with payload test commands for each failing blocking job attributed to this PR. Use the correct command based on whether the job is aggregated: - -- **Aggregated jobs** (job name has `aggregated-` prefix): Use `/payload-aggregate `. The underlying job name comes from the caller's analysis data (extracted from the aggregated job's junit artifacts — it cannot be derived from the aggregated job name). Choose a count of up to 10 runs — use judgement based on the total number of jobs being triggered (fewer runs per job when triggering many jobs to limit resource consumption; more runs when only one or two jobs need validation). -- **Non-aggregated jobs**: Use `/payload-job `. - -```bash -# Example with a mix of aggregated and non-aggregated jobs: -gh pr comment "{revert_pr_url}" --body "/payload-aggregate {underlying_job_name_1} {count} -/payload-job {job_name_2} -..." -``` - -One command per line for each failing blocking job attributed to this PR. +- `pr_url`: The revert PR URL from Substep 2 +- `jobs`: The `failing_jobs` list for this candidate (includes `job_name`, `is_aggregated`, `underlying_job_name` for each job) -After posting the comment, wait ~30 seconds and poll for the `openshift-ci[bot]` reply containing the `pr-payload-tests` URL: - -```bash -# Poll for the bot reply containing the pr-payload-tests URL -gh api "repos///issues//comments" \ - --jq '[.[] | select(.user.login == "openshift-ci[bot]" and (.body | contains("pr-payload-tests")))] | last | .body' -``` - -Extract the `pr-payload-tests.ci.openshift.org/runs/ci/` URL from the reply. This URL is the primary endpoint for checking job completion status (the page shows "AllJobsFinished" when done). - -Fetch the pr-payload-tests page to extract the actual prow job URL(s): - -```bash -# The page contains prow job links like: -# https://prow.ci.openshift.org/view/gs/test-platform-results/logs// -``` +The skill handles idempotency (checking for existing bot replies), correct command selection (`/payload-aggregate` vs `/payload-job`), polling, and URL extraction. -Record both the `payload_test_url` and individual `prow_url`s in the return data. +Record the `payload_test_url` and individual `prow_url`s from the skill's return data. ## Subagent Return Format @@ -165,5 +131,6 @@ Collect all subagent results. Return to the caller for inclusion in the report. ## See Also - Related Skill: `revert-pr` - The git revert workflow (`plugins/ci/skills/revert-pr/SKILL.md`) +- Related Skill: `trigger-payload-job` - Triggers payload jobs and collects URLs (`plugins/ci/skills/trigger-payload-job/SKILL.md`) - Related Skill: `analyze-payload` - Identifies revert candidates (`plugins/ci/skills/analyze-payload/SKILL.md`) - Related Command: `/ci:payload-agent` - Autonomous orchestrator that uses this skill (`plugins/ci/commands/payload-agent.md`) diff --git a/plugins/ci/skills/trigger-payload-job/SKILL.md b/plugins/ci/skills/trigger-payload-job/SKILL.md new file mode 100644 index 000000000..1426fc37d --- /dev/null +++ b/plugins/ci/skills/trigger-payload-job/SKILL.md @@ -0,0 +1,132 @@ +--- +name: Trigger Payload Job +description: Trigger payload validation jobs on a PR and collect the resulting Prow job URLs +--- + +# Trigger Payload Job + +This skill triggers payload validation jobs on a GitHub PR by posting the correct Prow commands as PR comments, then polls for the bot response to collect the resulting job URLs. + +## When to Use This Skill + +Use this skill whenever you need to trigger payload testing on a PR (revert PR, draft bisect PR, or any PR that needs payload validation). It handles the differences between aggregated and non-aggregated jobs and collects the resulting URLs for downstream tracking. + +**Inputs** (passed in-context by the caller): + +- `pr_url`: The GitHub PR URL to trigger jobs on (e.g., `https://github.com/openshift/ovn-kubernetes/pull/3040`) +- `org`, `repo`, `pr_number`: Parsed from the PR URL +- `jobs`: List of jobs to trigger, each with: + - `job_name`: The full job name (e.g., `aggregated-hypershift-ovn-conformance-4.22`) + - `is_aggregated`: Whether this is an aggregated job (has `aggregated-` prefix) + - `underlying_job_name`: For aggregated jobs only — the underlying periodic job name (e.g., `periodic-ci-openshift-hypershift-release-4.22-periodics-e2e-aws-ovn-conformance`). This MUST be extracted from the aggregated job's junit artifacts by the caller's analysis — it cannot be derived from the aggregated job name. + - `count`: For aggregated jobs only — how many runs (default: 10). Use judgement: fewer runs when triggering many jobs to limit resource consumption, more when only one or two jobs need validation. + +## Prerequisites + +1. **GitHub CLI (`gh`)**: Installed and authenticated +2. **Repository Access**: User must have permission to comment on the PR + +## Job Triggering Limits + +The caller (typically `payload-agent`) is responsible for enforcing global limits before invoking this skill. However, this skill also enforces the following hard limits per invocation: + +- **Non-aggregated jobs**: No more than 5 per comment +- **Aggregated jobs**: No more than 1 per comment. Aggregated jobs are expensive (they run many iterations), so only trigger one at a time. A second aggregated job should only be triggered after the first completes, and only if confirmation is needed. + +If the `jobs` input exceeds these limits, trigger only the allowed number (prioritizing by the order provided) and return the remainder as `skipped` in the return format. + +## Implementation Steps + +### Step 1: Idempotency Check + +Before posting new payload commands, check whether jobs were already triggered on this PR: + +```bash +gh api "repos///issues//comments?per_page=100&sort=created&direction=desc" \ + --jq '[.[] | select(.user.login == "openshift-ci[bot]" and (.body | contains("pr-payload-tests")))] | .[0] | .body' +``` + +If a `pr-payload-tests.ci.openshift.org/runs/ci/` URL is found, reuse it — skip to Step 3 to extract prow job URLs from that page. + +### Step 2: Post Payload Commands + +First, validate each job entry. If any job has `is_aggregated == true` but `underlying_job_name` is empty or null, skip that job and return an error for it — do NOT post a comment with an incomplete command. + +Build the comment body with one command per line. Use the correct command for each job type: + +- **Aggregated jobs** (`is_aggregated == true`): `/payload-aggregate ` +- **Non-aggregated jobs** (`is_aggregated == false`): `/payload-job ` + +```bash +gh pr comment "" --body "" +``` + +**Example** with a mix of job types: + +```bash +gh pr comment "https://github.com/openshift/ovn-kubernetes/pull/3040" --body "/payload-aggregate periodic-ci-openshift-hypershift-release-4.22-periodics-e2e-aws-ovn-conformance 10 +/payload-job periodic-ci-openshift-release-main-nightly-4.22-e2e-aws-ovn" +``` + +**Common mistakes to avoid**: +- Do NOT use `/payload ` — this triggers a different workflow and does NOT include the PR's changes in the payload +- Do NOT use the aggregated job name with `/payload-job` — aggregated jobs are not directly triggerable; you must use `/payload-aggregate` with the underlying job name +- Do NOT use `/payload-aggregate` for non-aggregated jobs + +### Step 3: Poll for Bot Response + +After posting the comment, wait ~30 seconds for the `openshift-ci[bot]` to process the commands, then poll for its reply: + +```bash +sleep 30 +gh api "repos///issues//comments?per_page=100&sort=created&direction=desc" \ + --jq '[.[] | select(.user.login == "openshift-ci[bot]" and (.body | contains("pr-payload-tests")))] | .[0] | .body' +``` + +If no reply is found, retry up to 3 times with 30-second intervals. If still no reply after ~2 minutes, record the comment URL and note that manual checking is required. + +### Step 4: Extract URLs + +From the bot reply, extract: + +1. **`payload_test_url`**: The `pr-payload-tests.ci.openshift.org/runs/ci/` URL. This is the primary endpoint for checking overall job completion status — the page shows "AllJobsFinished" when all triggered jobs have completed. + +2. **Individual prow job URLs**: Fetch the `payload_test_url` page and extract prow job links: + +```bash +curl -sL "" | grep -oE 'https://prow\.ci\.openshift\.org/view/gs/test-platform-results/logs/[^"]+' | sort -u +``` + +Each prow URL corresponds to one triggered job run. + +## Return Format + +Return the collected data in this format: + +``` +PAYLOAD_JOB_RESULT: + pr_url: + comment_url: + payload_test_url: + prow_jobs: + - job_name: + prow_url: + - ... + skipped_jobs: + - job_name: + reason: "job trigger limit reached (max 5 non-aggregated, max 1 aggregated per invocation)" + - ... + status: triggered|reused|no_response|failed + error: none| +``` + +## Error Handling + +- If the bot never replies, return `status: no_response` with the comment URL so the caller can check manually later. +- If the PR doesn't exist or the user lacks permissions, return `status: failed` with the error message. +- Do not retry indefinitely — 3 poll attempts is the maximum. + +## See Also + +- Used by: `bisect-payload-suspects` — triggers payload jobs for draft bisect PRs +- Used by: `stage-payload-reverts` — triggers payload jobs for revert PRs