From 5dbbde6ef0a0a5677f7e85a5d2b4cf7ed4e31886 Mon Sep 17 00:00:00 2001 From: Xinyuan Tong Date: Mon, 18 May 2026 00:42:44 +0800 Subject: [PATCH 1/2] Update sglang-ci-monitor for current CI shape MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reflect upstream CI changes since the skill was first written: - Stage rename: NVIDIA `stage-{a,b,c}-test-*` → `base-{a,b,c}-test-*` (PR sgl-project/sglang#25420). AMD jobs still use the `stage-*` prefix. - `check-stage-health` action → `check-pr-test-health`. Annotation messages still use `Fast-fail: skipping — root cause job(s): ` and `Fast-fail: lint check failed`, but they are now mutually exclusive (lint is checked first and returns early). - `/rerun-stage` is deprecated (PR #25322): handler replies `-1` plus a deprecation comment, does nothing. Removed from decision tree and command table; documented as a do-not-use. - New `/rerun-group ` command (PR #24023): expands `test/registered//test_*.py` into a `/rerun-test` batch. - New `pr-test-extra.yml` workflow (label-gated extras, stages `extra-a/b-test-*`) double-gated on `run-ci` + `run-ci-extra`. - New labels: `run-ci-extra` opts in to extra CI; `bypass-fastfail` skips the jobs-failed cascade (lint still cascades); `bypass-maintenance` bypasses the rebase-required / maintenance-pause gates inside `/rerun-test` and `check-maintenance`. - New shortcut for grabbing ``: read the PR States block at the bottom of every PR body (maintained by `pr-states.yml`), which links the latest pr-test and pr-test-extra runs and is refreshed on workflow_run events so slash-command reruns refresh it too. - `base-c-test-8-gpu-h20` is excluded from the fast-fail root-cause set (flaky h20 noise); document the implication. - `/rerun-failed-ci` now iterates every workflow run on the head SHA (pr-test, pr-test-extra, pr-test-sgl-kernel, jit-kernel, multimodal-gen…), so sibling-workflow failures are picked up in one command. --- sglang-ci-monitor/SKILL.md | 93 +++++++++++++++++++++++++------------- 1 file changed, 61 insertions(+), 32 deletions(-) diff --git a/sglang-ci-monitor/SKILL.md b/sglang-ci-monitor/SKILL.md index 42706e9..113537f 100644 --- a/sglang-ci-monitor/SKILL.md +++ b/sglang-ci-monitor/SKILL.md @@ -1,7 +1,7 @@ --- name: sglang-ci-monitor description: >- - Check, diagnose, rerun, and wait on CI for a sglang PR. Use whenever the user asks about CI on a sglang PR, wants to monitor a run, diagnose a CI failure, rerun failed jobs, or wait for CI to finish — even when they phrase it as "is the PR green?", "what's happening with the tests?", "something failed, look at it", or "babysit this PR for me". Assumes the user is a sglang CI maintainer, so all slash commands are available. Sharp edges: fast-fail cascades that hide the real root, `/rerun-failed-ci` vs `/tag-and-rerun-ci` vs `/rerun-stage` doing subtly different things, issue_comment-based reruns invisible on the PR page, exit-255 ambiguity — this skill is the default playbook. + Check, diagnose, rerun, and wait on CI for a sglang PR. Use whenever the user asks about CI on a sglang PR, wants to monitor a run, diagnose a CI failure, rerun failed jobs, or wait for CI to finish — even when they phrase it as "is the PR green?", "what's happening with the tests?", "something failed, look at it", or "babysit this PR for me". Assumes the user is a sglang CI maintainer, so all slash commands are available. Sharp edges: fast-fail cascades that hide the real root, `/rerun-failed-ci` vs `/tag-and-rerun-ci` vs `/rerun-test` vs `/rerun-group` doing subtly different things, `/rerun-stage` is **deprecated** (just posts a notice — does nothing), `pr-test.yml` (base CI, `base-a/b/c` stages) vs `pr-test-extra.yml` (label-gated extra CI, `extra-a/b` stages) being two separate runs, issue_comment-based reruns invisible on the PR page, exit-255 ambiguity — this skill is the default playbook. --- # sglang CI monitor @@ -17,10 +17,12 @@ When this skill triggers, the user's ask ("monitor / diagnose / rerun / babysit" Pre-approved (just do them, with one sentence of narration) — **on any sgl-project/sglang PR, regardless of author**. The user is a CI maintainer with permissions across the whole repo: - All read-only `gh` queries (`gh pr view`, `gh api .../runs|jobs|annotations|logs`). -- Posting slash commands: `/rerun-failed-ci`, `/tag-and-rerun-ci`, `/rerun-stage`, `/rerun-test`, `/tag-run-ci-label`. +- Posting slash commands: `/rerun-failed-ci`, `/tag-and-rerun-ci`, `/rerun-test`, `/rerun-group`, `/tag-run-ci-label`. Adding the `run-ci-extra` label to opt the PR into `pr-test-extra.yml` coverage (or `bypass-fastfail` / `bypass-maintenance` when the diagnostic situation warrants it — see §3). - `gh run cancel` as the precursor to a rerun. - `CronCreate` for polling. +Do **not** post `/rerun-stage` — it's deprecated (PR #25322). The handler now replies with a `-1` reaction and a deprecation comment, and does nothing else; posting it costs the user a notification for no result. + Stop and ask only when: - The action is off the pre-approved list (force-push, close, delete branch, edit labels outside the slash flow). @@ -32,12 +34,12 @@ Do **not** pause to ask "PR isn't yours, ok to proceed?" — the user has explic ### Scope: NVIDIA only -Only monitor NVIDIA CI. **Filter out AMD jobs entirely** — anything matching `*-amd*` (e.g. `stage-{a,b,c}-test-*-amd*`, `sgl-kernel-unit-test*-amd`, `PR Test (AMD)` workflow) is out of scope for this skill. +Only monitor NVIDIA CI. **Filter out AMD jobs entirely** — anything matching `*-amd*` is out of scope. Examples to filter: `stage-{a,b,c}-test-*-amd*` (AMD still uses the old `stage-*` prefix — only the NVIDIA workflow was renamed to `base-*`), `sgl-kernel-unit-test*-amd`, `multimodal-gen-test-*-amd*`, the `PR Test (AMD)` and `PR Test ROCm 7.2 (AMD)` workflows. - When pulling the rollup, exclude AMD jobs before splitting failures into root vs. cascade. - Don't report AMD failures, don't diagnose them, don't rerun them. - Cron polling considers CI "terminal" when every **non-AMD** check is terminal — ignore AMD pending state. -- Never use `/rerun-stage` with an AMD stage name. +- AMD has no slash-command rerun — its stage-level dispatch lives only in the Actions UI (`PR Test (AMD)` → *Run workflow* → pick a stage). The deprecated `/rerun-stage` was NVIDIA-only and doesn't apply to AMD either. ### Resolve the PR @@ -51,27 +53,45 @@ If no PR is associated, ask once for a number or URL. Otherwise proceed. Do not ### Pull job-level status -The PR page's Checks list and `gh pr checks` both miss stage-c jobs. Always use `statusCheckRollup`: +The PR page's Checks list and `gh pr checks` both miss base-c jobs. Always use `statusCheckRollup`: ```bash -gh pr view --repo sgl-project/sglang --json statusCheckRollup,labels \ +gh pr view --repo sgl-project/sglang --json statusCheckRollup,labels,body \ --jq '{labels: .labels | map(.name), checks: .statusCheckRollup | group_by(.conclusion) | map({conclusion: .[0].conclusion, count: length})}' ``` +### Fast `` shortcut — read the PR States block + +`pr-states.yml` maintains a CI States block at the bottom of every PR body, between `` and ``. It shows the **latest** run links for both `pr-test.yml` (base CI) and `pr-test-extra.yml` (extra CI), refreshed on `synchronize`, `labeled`, `unlabeled`, and pr-test* workflow_run events (so slash-command-initiated reruns refresh it too). This is usually the cheapest way to grab ``: + +```bash +gh pr view --repo sgl-project/sglang --json body \ + --jq '.body' | sed -n '//,//p' +``` + +You'll see something like `Latest PR Test (Base): [Run #12345678](https://github.com/.../actions/runs/12345678)` and a matching `(Extra):` line — or `:x: Missing 'run-ci' label` / `:warning: Not enabled — add 'run-ci-extra' label to opt in` when a workflow is gated off. Reach for `gh run list --workflow=pr-test.yml --branch --limit 1 --repo sgl-project/sglang` only when the block is missing (very old PRs) or when you need a non-latest run. + ### Always separate real failures from fast-fail victims -sgl CI uses aggressive fast-fail (`.github/actions/check-stage-health`): when any earlier job fails, the stage-health check inside every downstream job calls `core.setFailed('Fast-fail: skipping — root cause job(s): ')`. A raw count of FAILURE in the rollup is misleading — "8 FAILURE" often means "1 root + 7 cascade victims". +sgl CI uses aggressive fast-fail (`.github/actions/check-pr-test-health`; previously `check-stage-health` — same behavior, renamed in the `stage-*` → `base-*` rework). When any earlier job fails, the health check inside every downstream job calls `core.setFailed('Fast-fail: skipping — root cause job(s): ')`. A raw count of FAILURE in the rollup is misleading — "8 FAILURE" often means "1 root + 7 cascade victims". + +Two label-driven exceptions to know about: + +- **`bypass-fastfail` label on the PR** — the health check skips its jobs-failed scan and returns success. The wait-for-base-* jobs also short-circuit to success, so every stage dispatches in parallel. Cascades disappear; every downstream FAILURE in the rollup is a real failure, not a victim. The lint cascade still fires (see Case B) — only the jobs-failed half is bypassed. +- **`base-c-test-8-gpu-h20` is excluded from the root-cause set.** Its failures are treated as flaky-h20 noise and do **not** trigger fast-fail in other jobs. So if h20 is the only FAILURE you see, no cascade will appear — but the job itself is still a real (flaky) failure to diagnose or rerun. **The authoritative way: read the annotation** on any fast-fail job. It names the real roots directly: ```bash gh api repos/sgl-project/sglang/check-runs//annotations --jq '.[].message' -# → "Fast-fail: skipping — root cause job(s): unit-test-backend-4-gpu, lint check failed" -# or → "Fast-fail: lint check failed" (if lint.yml itself is red) +# → "Fast-fail: skipping — root cause job(s): base-b-test-1-gpu-large, base-c-test-4-gpu-h100" +# or → "Fast-fail: lint check failed" (if lint.yml itself is red — checked first, takes priority) ``` +The two annotation shapes are **mutually exclusive**: the health check inspects lint first and returns early if it failed, so you'll see one or the other, never both in the same message. + Pick any FAILURE job whose duration is near-instant, read its annotation, and every other victim gives the same names. Then diagnose those roots only. **Fallback** (cheap overview when annotation isn't needed): split by duration — real jobs almost never complete in <15s, fast-fail skips almost always do: @@ -89,7 +109,7 @@ gh api repos/sgl-project/sglang/actions/runs//jobs --paginate --jq ' Note: a lint failure in the separate `lint.yml` workflow also triggers the fast-fail across the whole pr-test run — check `lint` in the rollup before blaming pr-test. -`` is the PR Test run — find it from the rollup's `detailsUrl` on any non-gate check, or with `gh run list --workflow=pr-test --branch --limit 1 --repo sgl-project/sglang`. +`` is the `pr-test.yml` run. Easiest source: the PR States block (see above). Otherwise pick it from any non-gate rollup check's `detailsUrl`, or `gh run list --workflow=pr-test.yml --branch --limit 1 --repo sgl-project/sglang`. For extra CI (`run-ci-extra` PRs), substitute `--workflow=pr-test-extra.yml` — it's a separate workflow with its own run id. ### Report style @@ -97,9 +117,9 @@ Lead with the conclusion, with cascades collapsed. Counts first, prose second. Good: -> 1 real failure (`unit-test-cpu-2`, exit 255, 4m18s), 7 fast-fail victims, 12 SUCCESS. Diagnosing the real one now. +> 1 real failure (`base-a-test-cpu` partition 2, exit 255, 4m18s), 7 fast-fail victims, 12 SUCCESS. Diagnosing the real one now. -> 2 labels, 8 pr-gate denials (all fast — no `run-ci` label), 50 SKIPPED, 12 SUCCESS. CI never fired. Posting `/tag-run-ci-label`. +> 2 labels (no `run-ci`), pr-gate denied, 50 SKIPPED. CI never fired. Posting `/tag-and-rerun-ci` — just tagging won't retrigger; pr-test doesn't refire on `labeled`. Bad: @@ -113,10 +133,12 @@ Don't narrate queries. Don't report raw FAILURE counts without first collapsing ### Case A: nothing is running (CI never fired) -Symptom (from `pr-gate.yml`): pr-gate exits with `Missing required label 'run-ci'` and every downstream job is SKIPPED. The rollup shows 0 `run-ci` label, pr-gate FAILUREs, then a sea of SKIPPED. +Symptom (from `pr-gate.yml`): pr-gate exits with `Missing required label 'run-ci'` and every downstream job is SKIPPED. The rollup shows no `run-ci` label, pr-gate FAILUREs, then a sea of SKIPPED. The PR States block will say `:x: Missing run-ci label`. Action: `/tag-and-rerun-ci` (adds label and reruns in one shot). No further diagnosis — nothing actually ran yet. +**Related: `pr-test-extra.yml` (the extra CI) is double-gated** — it requires both `run-ci` AND `run-ci-extra`. So a PR can be green on base CI but show `:warning: Not enabled` for extra CI if `run-ci-extra` is absent. That's intentional, not a failure to fix. Only opt in when the user actually wants the nightly-class coverage; merely waiting on extra CI without the label will never become green. + ### Case B: lint failed → everything fast-fails Symptom: `lint` check FAILURE in the rollup; every pr-test job annotation is `Fast-fail: lint check failed`. @@ -138,13 +160,13 @@ Search logs for `FAILED:` to locate the test file and traceback. ### Case E: only victims, no root in the current run -The root lives in a sibling workflow (e.g. `Build Wheel` in `pr-test-sgl-kernel.yml`). `rerun_failed_jobs` is scoped to a single run, so a failed wheel build won't get retried by `/rerun-failed-ci` unless you look at the right run: +The root lives in a sibling workflow — common candidates: `Build Wheel` from `_pr-test-sgl-kernel-build.yml`, JIT-kernel / multimodal-gen test workflows, or `pr-test-extra.yml` when extra CI failed. `rerun_failed_jobs` is scoped to a single run, so the failure won't be retried by `/rerun-failed-ci` unless you look at the right run: ```bash gh run list --branch --limit 5 --repo sgl-project/sglang ``` -Diagnose the parent run's failure directly. +Note that `/rerun-failed-ci` iterates every workflow run on the head SHA with conclusion `failure` or `skipped` (not just `pr-test.yml`), so a failed sibling workflow IS picked up — but you still need to look at its run to read the actual log. Diagnose the parent run's failure directly. ### Exit code quick reference (for Case D log-reading) @@ -169,26 +191,29 @@ Only the first line of the comment is parsed. Handler: `scripts/ci/utils/slash_c | Command | What it actually does | |---|---| -| `/rerun-failed-ci` | For every run on the head SHA with conclusion `failure` or `skipped`: if the PR touches `sgl-kernel/` **and** not all `Build Wheel*` check-runs are green, call `run.rerun()` (full rerun); otherwise `run.rerun_failed_jobs()` (failed jobs + their dependents). Fast-fail victims have conclusion=failure so they're covered. | +| `/rerun-failed-ci` | For every run on the head SHA with conclusion `failure` or `skipped`: if the PR touches `sgl-kernel/` **and** not all `Build Wheel*` check-runs are green (CUDA + ARM), call `run.rerun()` (full rerun); otherwise `run.rerun_failed_jobs()` (failed jobs + their dependents). Iterates across **every** workflow on the head SHA (pr-test, pr-test-extra, pr-test-sgl-kernel, jit-kernel, multimodal-gen…) so sibling-workflow failures are covered. Cancelled / fast-fail-victim runs count as `failure`. | | `/tag-and-rerun-ci` | `/tag-run-ci-label` + sleep 5s + `/rerun-failed-ci`. Use when the PR is missing `run-ci` and also has reds to retry in one shot. | | `/tag-run-ci-label` | Adds the `run-ci` label. Nothing else. | -| `/rerun-stage ` | Targeted retry: `workflow_dispatch` on `PR Test` with `target_stage=`, running only that stage and skipping its dependencies. Stage must be in the whitelist (below). **Refused if PR touches `sgl-kernel/`** — `target_stage` mode skips the wheel build, so tests would run against the PyPI wheel. | -| `/rerun-test ` | `workflow_dispatch` on the separate **`Rerun Test`** workflow (`rerun-test.yml`). Investigation tool — runs the listed tests on a chosen runner, independently of PR Test. Does NOT update PR checks; the original red stays red. | - -**`/rerun-stage` valid stage names** (whitelist in `handle_rerun_stage`, NVIDIA-only — see §1 Scope): +| `/rerun-test ...` | `workflow_dispatch` on the separate **`Rerun Test`** workflow (`rerun-test.yml`). Resolves each file by looking up its `register_cuda_ci(stage=, runner_config=)` (or `register_cpu_ci(...)`) decorator and dispatches one run per test on the appropriate runner. Investigation tool — runs the listed tests independently of pr-test.yml. Does NOT update PR checks; the original red stays red. Blocked entirely on fork PRs. | +| `/rerun-group ...` | Expands each group name to all `test_*.py` files under `test/registered//` and reuses `/rerun-test` dispatch. Use when you know which module is flaky (e.g. `hicache`, `core`, `dsv4`) but don't want to enumerate files yourself. Same permissions and fork-block as `/rerun-test`. | +| `/rerun-stage` | **DEPRECATED** (PR #25322). The handler now replies with a `-1` reaction and a comment pointing to alternatives, and does nothing. Don't post it. | -`stage-a-test-1-gpu-small`, `stage-a-test-cpu`, `stage-b-test-{1,2,4}-gpu-*`, `stage-c-test-{4,8}-gpu-*`, `multimodal-gen-*`. +**Label-level controls** (use `gh pr edit --repo sgl-project/sglang --add-label ` to apply): -Use the full stage name from the rollup, not `stage-a` / `stage-b`. +- `run-ci` — required by `pr-gate.yml`. Without it, base CI never fires. +- `run-ci-extra` — opts the PR into `pr-test-extra.yml` (nightly-class extras: `extra-a-test-*`, `extra-b-test-*`, deepep, etc.). Requires `run-ci` to also be present. +- `bypass-fastfail` — skips the jobs-failed check in `check-pr-test-health` and short-circuits the wait-for-base-* jobs to success. Use when diagnosing intermittent failures and you want every stage to run to completion so you can see all the failures, not just the first one. **Lint cascade is NOT bypassed.** +- `bypass-maintenance` — bypasses the rebase-required gate and the full-maintenance-pause gate (maintenance issue #21065). Use this only when a CI-fix PR genuinely needs to land during maintenance. `/rerun-test` also honors it; the pre-dispatch gate inside the slash handler refuses without the label. ### Decision tree (goal: turn the PR green) 1. No `run-ci` label (Case A) — `/tag-and-rerun-ci`. 2. Lint red (Case B) — fix lint first; reruns are wasted until lint is green. -3. Want to retry one stage in isolation (skip its dependencies), no `sgl-kernel/` changes — `/rerun-stage `. -4. Everything else — `/rerun-failed-ci`. It already handles `sgl-kernel/` correctly (full rerun if the wheel build didn't pass, partial otherwise), so you don't need to pre-upgrade to `/tag-and-rerun-ci` just because the PR touches kernel code. +3. Everything else — `/rerun-failed-ci`. It already handles `sgl-kernel/` correctly (full rerun if any `Build Wheel*` check-run isn't `success`, partial otherwise) and covers sibling workflows on the same SHA, so you don't need to pre-upgrade to `/tag-and-rerun-ci` just because the PR touches kernel code. + +`/rerun-test` and `/rerun-group` are **off the tree** — they're investigation tools that dispatch the separate `Rerun Test` workflow and don't update the PR's pr-test checks. The original red stays red. Run them in parallel with a tree action when you want independent flake evidence or want to confirm a single suspected file before paying for a full rerun. -`/rerun-test` is off the tree — it's an investigation tool, not a way to green the PR. Run it in parallel with a tree action when you want independent flake evidence. +Stage-level retry (the deprecated `/rerun-stage`) is no longer an option — the maintainer position is that stage granularity is too coarse. If you really want only one stage's tests, use `/rerun-test` with the file list, or `/rerun-group ` if the stage maps cleanly to a `test/registered/` directory. ### Issue it @@ -243,10 +268,11 @@ The prompt offloads cadence decisions to the rules in §4 so the cron body stays ### Why prefer slash commands over `gh run rerun` -`gh run rerun` works, but skips two pieces of logic in `slash_command_handler.py` that you want: +`gh run rerun` works, but skips three pieces of logic in `slash_command_handler.py` that you want: -- **Kernel wheel freshness**: the handler inspects `Build Wheel*` check-runs and upgrades to full `run.rerun()` if any didn't pass. `gh run rerun --failed` always does partial — you can silently test against a stale wheel. -- **Label handling**: `/tag-and-rerun-ci` applies `run-ci` before rerunning. `gh run rerun` won't touch labels, so if the PR lacks `run-ci` all reruns fast-deny at pr-gate. +- **Kernel wheel freshness**: the handler inspects `Build Wheel*` check-runs (CUDA + ARM) and upgrades to full `run.rerun()` if any didn't pass. `gh run rerun --failed` always does partial — you can silently test against a stale wheel. +- **Multi-workflow sweep**: `/rerun-failed-ci` iterates **every** workflow run on the head SHA (pr-test, pr-test-extra, pr-test-sgl-kernel, jit-kernel, multimodal-gen…) so a failed sibling workflow gets retried in the same command. `gh run rerun ` retries one run. +- **Label / maintenance handling**: `/tag-and-rerun-ci` applies `run-ci` before rerunning, and `/rerun-test` pre-checks the maintenance gate. `gh run rerun` won't touch labels or honor `bypass-maintenance`, so reruns fast-deny without warning. pr-gate's 120-min rate limit bypasses for maintainers, so it's not a concern here. @@ -276,7 +302,7 @@ Rules (apply the **first** match, then stop): | ≤3 non-terminal jobs left **and** each has been running ≤ its typical wall time | **15 min** — catch terminal-state quickly so the user gets a final report fast. | | Steady progress: ≥1 job flipped to terminal since last tick, queue still draining | keep **30 min**. | | No state changes for 2 consecutive ticks, but jobs are still actively running (not stuck queued) | extend to **45 min**. | -| A long-tail stage-c job is the only blocker (still in its expected runtime window) | **45 min**. | +| A long-tail base-c job (or `pr-test-extra` stage) is the only blocker, still in its expected runtime window | **45 min**. | | Run looks stuck: same job queued >40 min without starting, **or** the same in-progress job has run >2× its typical wall time | extend to **60 min** — and surface the stuck job by name in the next report so the user sees it. | | `pr-gate` denied / no `run-ci` label / nothing actually running | **stop the cron** and post `/tag-and-rerun-ci` instead — polling a non-running PR is wasted ticks. | @@ -293,11 +319,14 @@ Keep the prompt body identical across replacements so subsequent ticks keep appl ## What to not do -- Don't default to `gh run rerun` — it skips the handler's kernel-wheel detection and label handling. +- Don't default to `gh run rerun` — it skips the handler's kernel-wheel detection, multi-workflow sweep, label handling, and maintenance gate. +- Don't post `/rerun-stage` — it's deprecated (PR #25322). The handler replies with `-1` and a comment but does nothing useful. Use `/rerun-failed-ci`, `/rerun-test `, or `/rerun-group ` instead. +- Don't use the old `stage-{a,b,c}-test-*` names when discussing NVIDIA CI — they're now `base-{a,b,c}-test-*`. (AMD CI is the exception: AMD jobs still use the `stage-*` prefix.) +- Don't confuse `pr-test.yml` (base CI) with `pr-test-extra.yml` (extra CI). They are separate workflows with separate run ids, separate label gates, and separate stages (`base-a/b/c` vs `extra-a/b`). - Never background-shell-poll for CI; use `CronCreate`. - Never call exit 255 "runner crash" without reading the log. -- Never investigate fast-fail victims individually — find the root. +- Never investigate fast-fail victims individually — find the root. (Exception: PR carries `bypass-fastfail`, in which case there are no victims — every FAILURE is real.) - Never report raw FAILURE counts from `statusCheckRollup` without first splitting by duration. A cascade of 8 victims + 1 root is not "9 failures"; it's 1. -- Never trust the PR page's Checks list alone — it misses stage-c jobs and issue_comment workflows. +- Never trust the PR page's Checks list alone — it misses base-c jobs and issue_comment workflows. The PR States block at the bottom of the PR body and `statusCheckRollup` are the authoritative views. - Never stop mid-flow to ask "should I?" — see Execution policy. - Never ask "want me to open a cron?" after a rerun — open it directly. The user already opted in by invoking the skill. From 6761a56bd3cf3b102b936c9ff3aa5d9ce1ce1ac9 Mon Sep 17 00:00:00 2001 From: Xinyuan Tong Date: Mon, 18 May 2026 17:25:13 +0800 Subject: [PATCH 2/2] Fix sglang CI monitor rerun guidance --- sglang-ci-monitor/SKILL.md | 36 +++++++++++++++++++----------------- 1 file changed, 19 insertions(+), 17 deletions(-) diff --git a/sglang-ci-monitor/SKILL.md b/sglang-ci-monitor/SKILL.md index 113537f..b249552 100644 --- a/sglang-ci-monitor/SKILL.md +++ b/sglang-ci-monitor/SKILL.md @@ -64,14 +64,14 @@ gh pr view --repo sgl-project/sglang --json statusCheckRollup,labels,body \ ### Fast `` shortcut — read the PR States block -`pr-states.yml` maintains a CI States block at the bottom of every PR body, between `` and ``. It shows the **latest** run links for both `pr-test.yml` (base CI) and `pr-test-extra.yml` (extra CI), refreshed on `synchronize`, `labeled`, `unlabeled`, and pr-test* workflow_run events (so slash-command-initiated reruns refresh it too). This is usually the cheapest way to grab ``: +`pr-states.yml` maintains a CI States block at the bottom of every PR body, between `` and ``. It shows run links for both `pr-test.yml` (base CI) and `pr-test-extra.yml` (extra CI) when the block is current. This is usually the cheapest way to grab an initial ``, but treat it as a shortcut, not the final source of truth after slash-command reruns — workflow-run refresh wiring can lag or drift from workflow names. ```bash gh pr view --repo sgl-project/sglang --json body \ --jq '.body' | sed -n '//,//p' ``` -You'll see something like `Latest PR Test (Base): [Run #12345678](https://github.com/.../actions/runs/12345678)` and a matching `(Extra):` line — or `:x: Missing 'run-ci' label` / `:warning: Not enabled — add 'run-ci-extra' label to opt in` when a workflow is gated off. Reach for `gh run list --workflow=pr-test.yml --branch --limit 1 --repo sgl-project/sglang` only when the block is missing (very old PRs) or when you need a non-latest run. +You'll see something like `Latest PR Test (Base): [Run #12345678](https://github.com/.../actions/runs/12345678)` and a matching `(Extra):` line — or `:x: Missing 'run-ci' label` / `:warning: Not enabled — add 'run-ci-extra' label to opt in` when a workflow is gated off. Reach for `gh run list --workflow=pr-test.yml --branch --limit 1 --repo sgl-project/sglang` when the block is missing, after a slash-command rerun, when the link looks stale, or when you need a non-latest run. ### Always separate real failures from fast-fail victims @@ -191,11 +191,11 @@ Only the first line of the comment is parsed. Handler: `scripts/ci/utils/slash_c | Command | What it actually does | |---|---| -| `/rerun-failed-ci` | For every run on the head SHA with conclusion `failure` or `skipped`: if the PR touches `sgl-kernel/` **and** not all `Build Wheel*` check-runs are green (CUDA + ARM), call `run.rerun()` (full rerun); otherwise `run.rerun_failed_jobs()` (failed jobs + their dependents). Iterates across **every** workflow on the head SHA (pr-test, pr-test-extra, pr-test-sgl-kernel, jit-kernel, multimodal-gen…) so sibling-workflow failures are covered. Cancelled / fast-fail-victim runs count as `failure`. | +| `/rerun-failed-ci` | For every **completed** run on the head SHA with conclusion `failure` or `skipped`: if the PR touches `sgl-kernel/` **and** not all `Build Wheel*` check-runs are green (CUDA + ARM), call `run.rerun()` (full rerun); otherwise `run.rerun_failed_jobs()` (failed jobs + their dependents). Iterates across **every** workflow on the head SHA (pr-test, pr-test-extra, pr-test-sgl-kernel, jit-kernel, multimodal-gen…) so sibling-workflow failures are covered. Fast-fail-victim jobs conclude `failure` and are covered; cancelled workflow runs are ignored by the handler. | | `/tag-and-rerun-ci` | `/tag-run-ci-label` + sleep 5s + `/rerun-failed-ci`. Use when the PR is missing `run-ci` and also has reds to retry in one shot. | | `/tag-run-ci-label` | Adds the `run-ci` label. Nothing else. | -| `/rerun-test ...` | `workflow_dispatch` on the separate **`Rerun Test`** workflow (`rerun-test.yml`). Resolves each file by looking up its `register_cuda_ci(stage=, runner_config=)` (or `register_cpu_ci(...)`) decorator and dispatches one run per test on the appropriate runner. Investigation tool — runs the listed tests independently of pr-test.yml. Does NOT update PR checks; the original red stays red. Blocked entirely on fork PRs. | -| `/rerun-group ...` | Expands each group name to all `test_*.py` files under `test/registered//` and reuses `/rerun-test` dispatch. Use when you know which module is flaky (e.g. `hicache`, `core`, `dsv4`) but don't want to enumerate files yourself. Same permissions and fork-block as `/rerun-test`. | +| `/rerun-test ...` | `workflow_dispatch` on the separate **`Rerun Test`** workflow (`rerun-test.yml`). Resolves each file by looking up its `register_cuda_ci(stage=, runner_config=)` (or `register_cpu_ci(...)`) decorator and dispatches one run per test on the appropriate runner. Investigation tool — runs the listed tests independently of pr-test.yml. Does NOT update PR checks; the original red stays red. Fork PRs are allowed only when the commenter has write/admin permission; untrusted fork authors are blocked. | +| `/rerun-group ...` | Expands each group name to all `test_*.py` files under `test/registered//` and reuses `/rerun-test` dispatch. Use when you know which module is flaky (e.g. `hicache`, `core`, `dsv4`) but don't want to enumerate files yourself. Same fork permission rule as `/rerun-test`: write/admin commenters can run it on fork PRs; untrusted fork authors cannot. | | `/rerun-stage` | **DEPRECATED** (PR #25322). The handler now replies with a `-1` reaction and a comment pointing to alternatives, and does nothing. Don't post it. | **Label-level controls** (use `gh pr edit --repo sgl-project/sglang --add-label ` to apply): @@ -217,35 +217,37 @@ Stage-level retry (the deprecated `/rerun-stage`) is no longer an option — the ### Issue it -If there's an active run on the PR **and** you're about to rerun because it's doomed (flaky root with pending cascade, or in-flight reds you want to replace), cancel first: +If the run is already terminal, post the slash command directly: ```bash -gh run cancel --repo sgl-project/sglang +gh pr comment --repo sgl-project/sglang --body "/rerun-failed-ci" ``` -Cancellation is **not** needed when the run is already terminal — `/rerun-failed-ci` picks up `failure`/`skipped` conclusions directly. Then post the comment: +If the run is still active, `/rerun-failed-ci` cannot pick it up yet because the handler only scans completed `failure` / `skipped` workflow runs. Do **not** cancel an active run and then post `/rerun-failed-ci`: cancelled workflow runs are ignored by the handler, so the comment can no-op. -```bash -gh pr comment --repo sgl-project/sglang --body "/tag-and-rerun-ci" -``` +When an active run is doomed, choose one of these: + +- Normal path: leave the run alone, poll to terminal, then post `/rerun-failed-ci`. +- Urgent infra-budget path: cancel the active run, wait for the cancellation to land, then rerun that same workflow run explicitly with `gh run rerun `. This is the exception to the slash-command preference; it skips the handler's multi-workflow / kernel-wheel logic, so use it only when you intentionally want to replace that exact active run. -### Flaky root + in-flight cascade: cancel, don't wait +### Flaky root + in-flight cascade: don't use `/rerun-failed-ci` until terminal -If the real failure is a known-flaky test (transient, matches a known flake pattern, or an infra wobble) **and** the run still has pending jobs that will fast-fail from the cascade, cancel the in-flight run immediately and rerun — waiting out the queue gives zero signal: +If the real failure is a known-flaky test (transient, matches a known flake pattern, or an infra wobble) **and** the run still has pending jobs that will fast-fail from the cascade, waiting out the queue gives little signal, but the slash-command handler still won't see the run until it is terminal. Either wait/poll until terminal and then post `/rerun-failed-ci`, or use the urgent path above: ```bash gh run cancel --repo sgl-project/sglang -gh pr comment --repo sgl-project/sglang --body "/rerun-failed-ci" +gh run watch --repo sgl-project/sglang +gh run rerun --repo sgl-project/sglang ``` -Cancelled jobs count as failure for `/rerun-failed-ci`, so this retries the root + all cascade victims + anything that was pending, while preserving already-green jobs. That's the right shape for "rerun everything that isn't already passing." +Do not pair `gh run cancel` with `/rerun-failed-ci`; cancelled workflow runs do not match the handler's `failure` / `skipped` filter. When to pick something other than this: - PR missing `run-ci` label → `/tag-and-rerun-ci` (adds label + runs rerun-failed-ci). - Failure is **not** flaky — cancel + rerun just reproduces it and burns budget. -If you want to **confirm** the flake reproduces before burning a full rerun, fire `/rerun-test ` in parallel with the cancel+rerun — it dispatches a separate `Rerun Test` workflow and won't slow the main pipeline. Remember it doesn't green the PR on its own. +If you want to **confirm** the flake reproduces before burning a full rerun, fire `/rerun-test ` in parallel with the wait or urgent rerun — it dispatches a separate `Rerun Test` workflow and won't slow the main pipeline. Remember it doesn't green the PR on its own. ### Confirm it fired @@ -272,7 +274,7 @@ The prompt offloads cadence decisions to the rules in §4 so the cron body stays - **Kernel wheel freshness**: the handler inspects `Build Wheel*` check-runs (CUDA + ARM) and upgrades to full `run.rerun()` if any didn't pass. `gh run rerun --failed` always does partial — you can silently test against a stale wheel. - **Multi-workflow sweep**: `/rerun-failed-ci` iterates **every** workflow run on the head SHA (pr-test, pr-test-extra, pr-test-sgl-kernel, jit-kernel, multimodal-gen…) so a failed sibling workflow gets retried in the same command. `gh run rerun ` retries one run. -- **Label / maintenance handling**: `/tag-and-rerun-ci` applies `run-ci` before rerunning, and `/rerun-test` pre-checks the maintenance gate. `gh run rerun` won't touch labels or honor `bypass-maintenance`, so reruns fast-deny without warning. +- **Label / maintenance handling**: `/tag-and-rerun-ci` applies `run-ci` before rerunning, and `/rerun-test` pre-checks the maintenance gate. `gh run rerun` won't touch labels or pre-check maintenance, so reruns can fast-deny without warning. pr-gate's 120-min rate limit bypasses for maintainers, so it's not a concern here.