[michaelzhang-ai]

What

A maintainer-gated PR test that runs the checks of the PR review guideline on two dedicated MI355X nodes: for each model a PR touches, its unit, kernel and serving tests, the kernel-only timing scripts, sgl-eval accuracy and InferenceX fixed-length speed. Accuracy and speed are compared against the base branch, run on the same node with the same server recipe. The workflow posts the result as an AMD PR test commit status and keeps one PR comment updated with a checklist in the guideline's order (rules, scope, accuracy, kernel microbenchmark, E2E performance, tests, doc).

Why

The guideline asks reviewers to approve only when these checks pass, plus a ≥5% gain for performance work. Today authors run the checks by hand and paste the numbers. A run takes hours and this repo should change rarely, so the test starts only when a maintainer starts it.

How it works

• Trigger: a maintainer adds the run-ci label, or runs the workflow with a PR number. The run removes the label and pins the PR head SHA it saw, so any later push needs a maintainer again.
• Scope (scripts/ci/scope.py): a model whose own package or tests changed runs the full tier. A change to common/, __init__.py or pyproject.toml runs every other model's smoke tier (tests, plus speed at the covered widths only), and common (the unit tests and the guard/dispatch kernel tests).
• Per model (scripts/ci/models.py, run_model.py): the base branch is installed first, then the PR, over the image's sgl-amd-mono. Both sides use the PR's test/e2e recipe with --enable-amd-monokernel. A model that is new in the PR is compared against SGLang's own path (monokernels off). The base accuracy result is cached per image and base commit.
• Gate (report.py): the run fails on any failed test, a server that logs no monokernel routing, accuracy below base by more than the per-benchmark tolerance, or a median-TPOT loss of more than 2%. [Perf] and [Feat] PRs also need a ≥5% median-TPOT gain at one of the model's covered concurrencies. The run also fails on edits to existing .github/CODEOWNERS lines; an added owner line for a new model is only a warning. A missing model README.md, or a check that can't run on these nodes, is a warning.
• Isolation: the workflow uses pull_request_target, and the scripts that run outside the container come from the base branch. PR code runs only inside the SGLang ROCm container. The cluster job fetches both commits itself.

Notes for the reviewer

• Not covered yet: Kimi-K3.1, GLM-5.3-Flash MXFP4 and Qwen3.8-Flash-Next MXFP4 have no checkpoint on the CI nodes; MiniMax-M3.1-Flash runs on MI350X only; Muse-Spark and Qwen3.8-2.4T have no serving recipe. These models still run their unit and kernel tests, and the report marks the rest as author-provided.
• Accuracy is GSM8K, GPQA-Diamond, AIME26, and MMMU-Pro for vision models. GPQA needs an HF token on the runner account, because it is a gated dataset. LiveCodeBench, SWE-bench Verified and OCRBench are not in CI yet.
• Admin setup this needs: register the two runners (scripts/ci/runner.sh, label sgl-amd-mono-mi355x), create the run-ci label, and, if wanted, make AMD PR test a required status.

Test plan

• Ran scope.py against the file lists of [Perf] [MI355X] deepseek_v41: DSpark step glue in fewer launches, Markov draft sampling, faster K1 and K2 #95, [Chore] maintainer: updates #96, [Fix] [MI355X] glm53_flash: Stop the sparse-attention fault on rows past 262,144 tokens #97, [Feat] common: Check each compiled kernel's residency and scratch where a runner builds it #101, [Feat] [MI355X] kimi_k3/kimi_k31: Kimi-K3's gfx950 launches, and Kimi-K3.1 steps of up to 8 rows on them #103 and [Refactor] [MI355X] kimi_k3/kimi_k31: One serving form, no copied helpers, every wide width a bucket #105; the models and rule results match what each PR touched ([Chore] maintainer: updates #96 and [Feat] [MI355X] kimi_k3/kimi_k31: Kimi-K3's gfx950 launches, and Kimi-K3.1 steps of up to 8 rows on them #103 trip the CODEOWNERS rule).
• Ran the full workflow on a fork, with [Refactor] [MI355X] kimi_k3/kimi_k31: One serving form, no copied helpers, every wide width a bucket #105 replayed on its parent and both runners pinned to the two nodes:
  • label gating, label removal, the pending and final status, and the PR comment all work;
  • the kimi_k31 job passed its kernel test (test_kimi_k31_k1_ring.py) and ran layer_bench.py;
  • unit tests on SGLang v0.5.23 nightly 20261104: 579 passed, 11 failed (the kimi_k3/k31 forward-drift and glm53_flash spec tests). These fail on main too, so they are an SGLang-version issue, not this change.
• Not yet verified end to end: a full-tier model with servers (Kimi-K3, then DeepSeek-V4.1-Flash with [Perf] [MI355X] deepseek_v41: DSpark step glue in fewer launches, Markov draft sampling, faster K1 and K2 #95 for the speed gate) is running now. I will post its report here.