[XPU CI] Guardian bot: auto-disable PR on 2-run XPU regression (draft)

[arathi-hlab]

Summary

Draft — putting this up so it can be reviewed / iterated on tomorrow.

Adds a GitHub Actions workflow (.github/workflows/xpu-ci-guardian.yml) that watches PR Test (XPU) on main. When the last two consecutive runs on main fail, it opens a draft PR that inserts disabled="..." on the register_xpu_ci(...) marker for each failing test file and pings the XPU CI owner. A separate job auto-closes any open guardian PR once main goes green again.

Design

• Watch — schedule every 30 min + manual workflow_dispatch with a dry_run toggle.
• Detect (2-run rule) — only acts when the last two runs on main both failed, so single flakes handled by enable_retry in run_suite.py don't trigger it. Also filters known infra-flake log patterns (UR_RESULT_ERROR_OUT_OF_RESOURCES, Server failed to start, XPU out of memory).
• Diagnose — extracts test/registered/**/*.py paths from the failing job's --log-failed.
• Unblock (scope-limited) — inserts disabled="auto-disabled by xpu-ci-guardian: run <id>" into the file's register_xpu_ci(...). Never touches kernel code, test bodies, run_suite.py, or per-method @unittest.skipIf. Skips files that are already disabled and files that don't call register_xpu_ci.
• Dedup — at most one open guardian PR at a time via the xpu-ci-guardian label.
• PR body — lists failing files with the last 5 commits on each, so the reviewer can jump straight to the probable culprit (as PR [Fix] Skip padded state slots in the chunked GDN kernel #33431 was in the current fire).
• Sweep — companion job closes any open guardian PR when the next main run is green.
• Never auto-merges — PR opens as a draft and only a human can promote / merge it.

Required secrets on the fork's Actions

• GITHUB_TOKEN (default; read-only for detection).
• BOT_PR_TOKEN — fine-grained PAT with contents:write on arathi-hlab/sglang and pull-requests:write on sgl-project/sglang.

Enablement checklist (before turning it on)

• Merge (or land into the fork only, if we decide it shouldn't live upstream).
• Add BOT_PR_TOKEN secret on the fork.
• Manually run once with dry_run=true and inspect the workflow's outputs.
• Manually run once with dry_run=false against a known-failing history to confirm the PR body/format looks right.
• Add the xpu-ci-guardian label to the label set on sgl-project/sglang.

Explicitly out of scope for this bot

• Auto-generating actual code / kernel fixes (LLM-authored kernel patches are wrong often enough that reviewing them costs more than writing them from scratch).
• Deep classification (e.g. distinguishing correctness vs. accuracy assertions). The bot just disables; the human writes the fix.
• Alerting for non-XPU CI. This is scoped narrowly on purpose.

Test plan

• Run the workflow manually with dry_run=true — the detect job should classify the current XPU state and set act correctly without opening any PR.
• Run with dry_run=false on a synthetic branch where I forcibly log two failing runs, and confirm the PR opens with the right title / labels / reviewer.
• Trigger sweep-when-green manually after a subsequent green run — it should close the previously-opened guardian PR.
• Confirm the guardian PR does NOT open a second time while another is open (dedup by xpu-ci-guardian label).

Not included yet (defer to follow-up)

• Slack notification hook — trivial to add via slackapi/slack-github-action when a channel + token are chosen.
• Nightly weekly-summary email of caught regressions.

---

🤖 Draft raised now so it can be reviewed tomorrow morning; happy to iterate on scope / triggers / notification medium.

---

CI States

Latest PR Test (Base): ✅ Run #32281257858
Latest PR Test (Extra): ❌ Run #32281257526