[CI Monitor] Daily Report - 2026-05-02

[amd-bot]

CI Daily Health — 2026-05-02

Snapshot: 2026-05-02 22:30 UTC · Only completed runs counted · Auto-updated every 30 min

TL;DR

🔴 RED · 18 active clusters · 2 🆕 NEW today · 3 reactivated · 13 carrying over · 3 in-flight fixes (none directly actionable)
👉 Today's ask: (1) R6 🔥 reactivated, 10 jobs — HuggingFace Hub xet_get / RemoteProtocolError disconnects exploded across pr-test-amd (8 jobs) and nightly-test-amd-rocm720 (2 jobs); engage AMD CI infra to confirm runner network egress / HF Hub connectivity at MI325 pool. (2) R35 (AITER pin bump #24218 → aiter a6bb499) still active day 2 with 6 jobs — symptom morphed from yesterday's mem-fault to a device_synchronize hang during DSv3.2 graph capture on MI35x — revert #24218 on a branch + rerun the 6 affected jobs. (3) Triage R44 🆕 (cb8fbd5+8a53046 suspected — Qwen3-Coder-Next setUpClass DistStoreError).

Workflow status

Workflow	Runs	✅	❌	7d trend (completed only)	Δ vs yesterday
nightly-test-amd	1	0	1	14·11·12·13·10 fails (25257970669)	-3 jobs (better)
nightly-test-amd-rocm720	1	0	1	11·12·11·11·11 fails (25257923871)	0 jobs
release-docker-amd-nightly	0	—	—	(no run today)	n/a
release-docker-amd-rocm720-nightly	0	—	—	(no run today)	n/a
amd-aiter-scout	0	—	—	(cron Mon/Thu — last: Thu Apr 30 25187625716; next Mon May 4)	n/a
pr-test-amd	4	0	4	3·3·2·3·8·7·3 fails across 25239403651, 25246077188, 25251804407, 25258688414 — 21 jobs total	+11 jobs (worse)
pr-test-amd-rocm720	1	0	1	7·9 fails (25257951974)	+2 jobs

Failure clusters (deduplicated across all workflows)

R6 · 🔥 reactivated · HuggingFace Hub xet_get / RemoteProtocolError mass-disconnect during model weight / config load — 10 jobs across 2 workflows, NEW resurgence today

• Status: Cluster R6 (HF Hub 429 / xet_get stall) was inactive in yesterday's board. Today exploded to 10 jobs concentrated across pr-test-amd runs 25246077188 (4 jobs) and 25251804407 (4 jobs), plus 2 in nightly-test-amd-rocm720. Pattern: xet_get hangs or httpx.RemoteProtocolError: Server disconnected during setUpClass weight/config fetch → 1800 s harness timeout or SIGKILL.
• Top hypothesis: [HIGH] Network-side issue on the AMD linux-mi325-1gpu-sglang / linux-mi35x-gpu-1 runner pool — multiple unrelated tests fail at the HF Hub call site within minutes of each other across two consecutive pr-test-amd runs (~12:23 and ~06:44 UTC). Disconfirming: not seen in nightly-test-amd (NVIDIA HF cache hits served from local image), and earlier 00:44 / 18:22 UTC pr-test-amd runs had only 1 R6 hit each — suggests a transient HF Hub or runner-egress incident clustered around the 06:44/12:23 UTC windows.
• In-flight fix: ⚠️ #23400 "Add HF Hub retry/backoff for xet_get" (open since 2026-04-22) is the closest candidate. Searched: repo:sgl-project/sglang is:pr is:open xet_get, huggingface hub retry, RemoteProtocolError — no fresh PRs.
• Suggested triage: ask AMD CI infra to check egress logs (proxy / NAT) for the MI325/MI35x runner pools at 06:44 and 12:23 UTC; chase reviewers on #23400; short-term: add a retry wrapper around popen_launch_server-time HF downloads in test/registered/amd/.../setUpClass.

Workflow	Job (shard)	Test File	Test Function	Error (one line)	Log
nightly-test-amd-rocm720	nightly-perf-2-gpu-vlm-rocm720	test_vlms_perf_amd.py	warmup processor download	HF Hub processor download disconnect	link
nightly-test-amd-rocm720	nightly-8-gpu-grok1-int4-rocm720	test_grok1_int4_eval_amd.py	weight load	xet_get 60-min stall	link
pr-test-amd	stage-b-1gpu-small-amd (13)	(model load)	setUpClass	xet_get hang → 45-min wall-clock timeout	link
pr-test-amd	stage-b-1gpu-small-amd (7)	(model load)	setUpClass	hf_hub_download hang / RemoteProtocolError	link
pr-test-amd	stage-b-1gpu-small-amd (4)	(model load)	setUpClass	xet_get hang / RemoteProtocolError	link
pr-test-amd	stage-b-1gpu-small-amd (2)	(config fetch)	setUpClass	HF Hub HTTPS disconnect	link
pr-test-amd	stage-b-1gpu-small-amd-mi35x	test_srt_backend.py	setUpClass	HF model download timeout	link
pr-test-amd	stage-a-test-1-gpu-small-amd	(model load)	setUpClass	HF stall → TimeoutError	link
pr-test-amd	stage-b-1gpu-small-amd (13)	(model load)	setUpClass	RemoteProtocolError: Server disconnected → SIGKILL/1800s	link
pr-test-amd	stage-b-1gpu-small-amd (1)	tokenizer init	scheduler tokenizer init	HF Hub list_repo_templates/xet_get failure (Llama-3.1-8B-Instruct)	link

R35 · AITER pin bump #24218 regression umbrella — 6 jobs across 2 workflows, day 2 persistent

• Status: Today's symptom morphed: yesterday was "Memory access fault by GPU" mid-graph-capture on DSv3.2 NSA; today the same DSv3.2 NSA jobs hang in c10::hip::device_synchronize for 3600 s in cuda_graph_runner.py:1163 warm-up loop. Same suspect: aiter v0.1.12.post1...a6bb499 (150 commits incl. 22db4ebb gfx950 MLA kernel, d098ae5a revert of CAR graph-capture fix, 52358334 DSv3 GEMM configs).
• Top hypothesis: [MEDIUM] aiter a6bb499 introduces a gfx950 graph-capture regression for DSv3.2 NSA tilelang backend. Disconfirming: still no clean A/B (yesterday's run failed on env-side mem-imbalance at init, never reached graph capture); MiniMax-M2.7 GSM8K accuracy regression (yesterday's R35 row) was NOT seen today, suggesting either intermittent or partially-resolved.
• In-flight fix: ❌ none found. Adjacent: #23581 (default SGLANG_USE_AITER_AR=0, open since 2026-04-23) could mask if AR is the trigger.
• Suggested triage: revert #24218 on a branch + AITER_COMMIT_OVERRIDE=v0.1.12.post1 rerun; bisect aiter delta if implicated. Wait for next AITER Scout (Mon May 4) for definitive A/B against sister baseline.

Workflow	Job (shard)	Test File	Test Function	Symptom	Log
nightly-test-amd	nightly-accuracy-8-gpu-mi35x-deepseek-v32	test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	TIMEOUT 3600 s — hang in cuda graph capture (TP-wide device_synchronize)	link
nightly-test-amd	nightly-4-gpu	test_deepseek_v32_eval_amd.py	test_bench_one_batch	GPU memory access fault during CUDA graph capture (NSA tilelang, bs=136)	link
nightly-test-amd	nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp	test_deepseek_v32_mtp_eval_mi35x.py	test_a_gsm8k	DSv3.2+MTP GSM8K accuracy collapse (drafts accepted, target unparseable)	link
nightly-test-amd	nightly-perf-8-gpu-mi35x-deepseek-v32-basic	test_deepseek_v32_basic_perf_mi35x.py	test_bench_one_batch	CUDA graph capture hang (torch.cuda.synchronize blocked in HSA)	link
nightly-test-amd-rocm720	nightly-perf-8-gpu-mi35x-deepseek-v32-basic-rocm720	test_deepseek_v32_basic_perf_mi35x.py	test_bench_one_batch	GPU mem fault during DSv3.2 NSA cuda-graph capture (TP=8, MI35x ROCm 7.2)	link
nightly-test-amd-rocm720	nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	Multi-rank simultaneous GPU mem fault during CUDA-graph capture	link

R22 · TP-rank GPU memory imbalance at server init on MI35x runners — 6 jobs across 2 workflows, persistent

• Status: down from ~10 jobs yesterday to 6 today (better). Same signature: RuntimeError: memory capacity is unbalanced at model_runner.py:1227; one MI35x rank reports ~13 GB free vs ≥192 GB normal.
• Top hypothesis: [HIGH] MI35x ephemeral runner pool leaks VRAM between tenants (runner-state contamination, not sglang). Disconfirming: also hit on linux-mi325-2gpu-sglang last week, suggests broader runner-pool cleanup gap.
• In-flight fix: ❌ none. Suggested triage: extend ensure_vram_clear.sh to nightly MI35x; recycle runner images.

Workflow	Job	Test File	Log
nightly-test-amd	nightly-8-gpu-mi35x-glm51	test_glm51_eval_mi35x.py	link
nightly-test-amd	nightly-8-gpu-mi35x-qwen35	test_qwen35_eval_mi35x.py	link
nightly-test-amd	nightly-8-gpu-mi35x-glm5-mxfp4	test_glm5_mxfp4_eval_mi35x.py	link
nightly-test-amd-rocm720	nightly-accuracy-8-gpu-mi35x-rocm720	test_gpt_oss_eval_mi35x.py	link
nightly-test-amd-rocm720	nightly-8-gpu-mi35x-glm51-rocm720	test_glm51_eval_mi35x.py	link
nightly-test-amd-rocm720	nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion-rocm720	test_deepseek_r1_mxfp4_ar_fusion_eval_mi35x.py	link

R5 · gpt-oss-20b mxfp4 /v1/responses empty / non-terminating streaming on MI35x — 4 jobs, persistent ≥18 days

• Top hypothesis: [MEDIUM] Test/server contract mismatch — _check_streaming_responses_api_request reads response.output_text.delta; mxfp4 path emits response.reasoning_text.delta. In-flight fix: ⚠️ candidate #23766 (touches /v1/responses SSE).

Workflow	Job	Log
pr-test-amd	stage-b-1gpu-small-mi35x (run 25239403651)	link
pr-test-amd	stage-b-1gpu-small-mi35x (run 25246077188)	link
pr-test-amd	stage-b-1gpu-small-mi35x (run 25258688414)	link
pr-test-amd-rocm720	stage-b-1gpu-small-mi35x-rocm720	link

R36 · aiter allreduce_fusion_kernel_1stage bf16 residual divergence (max_diff 6.25e-2) — 3 jobs (same shard, 3 runs), persistent

• Top hypothesis: [HIGH] aiter 4331ad7a (#2823 "fused AR + RMSNorm + per-group FP8 quant") changed 1-stage fused path to f32-accumulate + round-once vs unfused round-AR-first → 1 ULP / 6.25e-2 bf16 mismatch.
• In-flight fix: ❌ none.

Workflow	Job (shard)	Run	Log
pr-test-amd	stage-c-test-large-8-gpu-amd (8gpu, 2)	25239403651	link
pr-test-amd	stage-c-test-large-8-gpu-amd (8gpu, 2)	25251804407	link
pr-test-amd	stage-c-test-large-8-gpu-amd (8gpu, 2)	25258688414	link

R44 · 🆕 · Qwen3-Coder-Next 8-GPU TP server setUpClass fails with cross-rank DistStoreError + RCCL HIP failure: 'invalid device pointer' — 1 job, NEW today

• Status: test/registered/amd/test_qwen3_coder_next_8gpu.py::setUpClass fails on pr-test-amd-rocm720 stage-c-mi35x-rocm720 (0) — TP1 hits 300 s soft watchdog during cuda-graph capture, all 8 ranks then emit HIP failure: 'invalid device pointer' from RCCL aiter custom_all_reduce.capture(), server SIGKILL exit -9 after 933 s. Last green: 2026-05-01.
• Top hypothesis: [MEDIUM] Commits cb8fbd5 (#24243 "Reserve slot 0 as padding in all req pools") + 8a53046 (#24244 "Size mamba mappings from req pool, not mamba pool") change HybridReqToTokenPool slot indexing; Qwen3-Coder-Next is hybrid-mamba+attention so cuda-graph-padded batches with new dummy-slot routing could mis-route across TP ranks. Disconfirming: not yet observed on NVIDIA shards or AMD MI325 rocm720 sister shards; sister test test_deepseek_r1_mxfp4_8gpu.py in same job passed (runtime healthy).
• In-flight fix: ❌ none. Suggested triage: revert #24243 + #24244 on a branch (in reverse order) and rerun; bisect ece8a1a788..e0474fdd9b (28 commits) on MI35x.

Workflow	Job (shard)	Test File	Test Function	Error	Log
pr-test-amd-rocm720	stage-c-test-large-8-gpu-amd-mi35x-rocm720 (0)	test/registered/amd/test_qwen3_coder_next_8gpu.py	setUpClass (TestQwen3CoderNext)	Server process exited with code -9 (HIP invalid device pointer + DistStoreError)	link

R43 · 🆕 · HIP CUDA-graph replay hang in /generate on ROCm 7.2.0 — 1 job, NEW today

• Top hypothesis: [LOW] Single occurrence on pr-test-amd-rocm720 stage-b-1gpu-small-rocm720 (4); could be R2-family HIP graph-capture invalidation (ROCm 7.2.0). In-flight fix: ⚠️ overlap with #21839 (bumps to ROCm 7.2.2). Triage: rerun once to test for flake; if reproduces, treat as R2 expansion.

Workflow	Job (shard)	Test File	Error	Log
pr-test-amd-rocm720	stage-b-1gpu-small-rocm720 (4)	(cuda-graph replay)	HIP CUDA-graph replay hang in /generate	link

R30 · ♻️ reactivated · HIP memcpy_and_sync MoE FP8 weight-load mutex contention on 8-GPU MI325 — 2 jobs, reactivated

• Status: Yesterday's board marked R30 as "merged into R35"; today re-emerges with distinct signature: TP workers stuck in HIP memcpy_and_sync during MoE FP8 weight load (only 1/8 TPs finishes before job timeout). Hits both DSv3.2 (MI325) and Kimi-K2 (rocm720).
• In-flight fix: ❌ none.

Workflow	Job (shard)	Test File	Log
pr-test-amd	stage-c-test-large-8-gpu-amd (8gpu, 0)	DSv3.2 server-launch hang on MI325	link
pr-test-amd-rocm720	stage-c-test-large-8-gpu-amd-rocm720 (1)	Kimi-K2-Instruct setUpClass timeout (MoE FP8 weight load)	link

R2 · HIP graph-capture invalidation in AiterCustomAllreduce (Qwen3-235B-MXFP4) — 2 jobs, carry-over

• Top hypothesis: [HIGH] AiterCustomAllreduce launches helper kernels during HIP capture; cudaStreamCaptureModeThreadLocal silently ignored on ROCm 7.2.0. In-flight fix: ✅ #21839 (bumps to ROCm 7.2.2) — chase reviewers; also #23581 (default SGLANG_USE_AITER_AR=0).

Workflow	Job	Log
nightly-test-amd	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	link
nightly-test-amd-rocm720	nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	link

Other carry-over and small clusters

ID	Cluster	Today's count	Top hypothesis (confidence)	In-flight fix
R10	Triton-attention offline throughput / silent latency hang on AMD	3 (pr-test-amd 0, 11, pr-test-amd-rocm720 stage-b-large 0)	[MEDIUM] Perf gate above achievable throughput + 1800 s harness timeout	❌ none
R24	Pre-flight ensure_vram_clear.sh exit 1 on AMD runners	3 (pr-test-amd disagg, pr-test-amd-rocm720 disagg, pr-test-amd-rocm720 stage-c-mi35x 1)	[HIGH] Stale GPU allocation from prior tenant	❌ none
R3	VLM MMMU below threshold (AMD MI30x)	1 (nightly-rocm720 vlm)	[HIGH] Thresholds above sustained scores	⚠️ partial #24084
R4	GSM8K Mistral/Mixtral threshold (AMD MI30x)	2 (nightly-amd 2gpu, rocm720 2gpu)	[LOW] Mistral v5 tokenizer formatting on AMD; +Qwen2-72B-FP8 OOM	❌ none
R11	multimodal-gen 1-GPU 60-min step timeout	1 (rocm720 mm-gen)	[HIGH] ltx_2_3_hq_pipeline push past 60-min cap	⚠️ partial #24002
R26	♻️ VLM MMMU benchmark timeout on AMD 4-GPU	1 (nightly-4-gpu-rocm720)	[LOW] test_encoder_dp runtime variance	❌ none
R34	test_topk_sigmoid_renormalize_correction_bias tie-break flake	1 (sgl-kernel-amd)	[HIGH] atol=rtol=0 comparison on tied scores	❌ none
R39	Silent server hang during VLM MMMU eval (Qwen2-VL-7B, no traceback)	1 (nightly-amd 2gpu vlm)	[LOW] aiter bump or detokenizer race; symptom morphed from yesterday's SIGABRT	❌ none
R40	MoriEP + TBO + EAGLE/MTP CUDA-graph mem fault on MI325/ROCm 7.2	1 (rocm720 stage-c 2)	[LOW] Long-standing MoriEP+TBO+EAGLE instability on MI325+ROCm 7.2; today: HSA OOR	❌ none

Known stable / low-impact clusters not seen today · click to expand

ID	Status	Notes
R27	split_spec_info seq_lens_cpu=None on MoRI-EP+spec-v2+TBO	not hit today (MoriEP test path not exercised in any of today's 4 pr-test-amd runs)
R37	DSv3.2 cold-cache HF model-download timeout	not seen today
R38	LoRA /load_lora_adapter 400 / Server disconnected	not seen today
R41	stage-b-1gpu-small-rocm720 shard 12 30-min step timeout	not seen today
R42	self-hosted runner lost communication (infra)	not seen today
R17	HiCache storage accuracy diff at 0.03 boundary	not seen today
R20	MTP+TBO scheduler watchdog _moe_sorting_impl	not seen today
R23	swiglu_with_alpha_and_limit ImportError	resolved Apr 29 (#24069) — no recurrence
R25	AutoConfig ValueError: Unrecognized model deepseek-ai/DeepSeek-V3.2	not seen today
R28	AiterCustomAllreduce IPC hipIpcOpenMemHandle invalid	not seen today
R29	gpt-oss-20b bf16 mmlu accuracy regression (rocm720)	not seen today
R31	AWQ Mixtral GPU Hang exit 134	not seen today
R32	Alibaba OSS network timeout for qwen_image_edit_2509	not seen today
R33	LFM2 tool-call truncated JSON on rocm720	not seen today

Workflow drill-down (per-workflow view)

nightly-test-amd · run [25257970669](https://github.com/sgl-project/sglang/actions/runs/25257970669) · 10 failures

Job (shard)	Test File	Cluster
nightly-8-gpu-mi35x-glm51	test_glm51_eval_mi35x.py	R22
nightly-8-gpu-mi35x-qwen35	test_qwen35_eval_mi35x.py	R22
nightly-accuracy-8-gpu-mi35x-deepseek-v32	test_deepseek_v32_eval_mi35x.py	R35
nightly-4-gpu	test_deepseek_v32_eval_amd.py	R35
nightly-8-gpu-mi35x-qwen3-235b-mxfp4	test_qwen3_instruct_mxfp4.py	R2
nightly-8-gpu-mi35x-glm5-mxfp4	test_glm5_mxfp4_eval_mi35x.py	R22
nightly-accuracy-2-gpu	test_gsm8k_eval_amd.py	R4
nightly-accuracy-2-gpu-vlm	test_vlms_mmmu_eval_amd.py	R39
nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp	test_deepseek_v32_mtp_eval_mi35x.py	R35
nightly-perf-8-gpu-mi35x-deepseek-v32-basic	test_deepseek_v32_basic_perf_mi35x.py	R35

nightly-test-amd-rocm720 · run [25257923871](https://github.com/sgl-project/sglang/actions/runs/25257923871) · 11 failures

Job	Test File	Cluster
nightly-perf-2-gpu-vlm-rocm720	test_vlms_perf_amd.py	R6 ♻️
nightly-accuracy-8-gpu-mi35x-rocm720	test_gpt_oss_eval_mi35x.py	R22
nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	test_qwen3_instruct_mxfp4.py	R2
nightly-accuracy-2-gpu-vlm-rocm720	test_vlms_mmmu_eval_amd.py	R3
nightly-perf-8-gpu-mi35x-deepseek-v32-basic-rocm720	test_deepseek_v32_basic_perf_mi35x.py	R35
nightly-4-gpu-rocm720	test_encoder_dp MMMU	R26 ♻️
nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	test_deepseek_v32_eval_mi35x.py	R35
nightly-8-gpu-mi35x-glm51-rocm720	test_glm51_eval_mi35x.py	R22
nightly-accuracy-2-gpu-rocm720	test_gsm8k_eval_amd.py	R4
nightly-8-gpu-grok1-int4-rocm720	test_grok1_int4_eval_amd.py	R6 ♻️
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion-rocm720	test_deepseek_r1_mxfp4_ar_fusion_eval_mi35x.py	R22

pr-test-amd · 4 runs today · 21 failures

R6 ♻️ ×8 (HF Hub disconnects in runs 25246077188 and 25251804407), R5 ×3 (mxfp4 streaming), R36 ×3 (aiter AR+RMSNorm residual stage-c shard 2 — every run), R30 ♻️ ×1 (DSv3.2 MoE weight-load hang 25239403651), R10 ×2 (triton-attn hang/throughput in 25246077188), R24 ×1 (VRAM cleanup), R34 ×1 (topk_sigmoid flake), R6/silent-mmlu ×1 (74044904944 — silent run_eval(mmlu) hang in same R6 storm window, classified under R6 cluster).

pr-test-amd-rocm720 · run [25257951974](https://github.com/sgl-project/sglang/actions/runs/25257951974) · 9 failures

R5 ×1, R10 ×1, R11 ×1, R24 ×2, R30 ♻️ ×1 (Kimi-K2 MoE FP8 hang), R40 ×1 (MoriEP HSA OOR), R43 🆕 ×1 (HIP cuda-graph replay hang), R44 🆕 ×1 (Qwen3-Coder-Next setUpClass DistStoreError).

How this report is generated

• Only status == "completed" runs counted in trends; in-flight runs would be labelled (IN-FLIGHT) (none in this snapshot).
• Cluster IDs R1-R42 inherited from yesterday; today's NEW IDs: R43, R44. Reactivations marked ♻️: R6 (10 jobs), R26 (1 job), R30 (2 jobs).
• Confidence labels: FACT / HIGH / MEDIUM / LOW / SPECULATION. Default LOW.
• Bot does NOT assign Priority. Status line + cluster size + persistence are the inputs; engineers decide priority.
• In-flight fix lookup performed for every cluster; verified: R2=#21839 (open), #23581 (open 9d), R3=#24084 (open partial), R5=#23766 (candidate), R6=#23400 (open candidate), R11=#24002 (open partial). All R30/R35/R36/R43/R44 + small carry-overs: ❌ no in-flight fix found.
• AMD AITER Scout did NOT run today (Mon/Thu cron — last Thu Apr 30 25187625716). Next scout: Mon May 4 — until then R35 lacks sister-baseline disconfirmation.

---

Generated by amd-bot · last updated 2026-05-02 22:30 UTC

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-05-02 22:30 UTC)

---

CI Monitor — 2026-05-02

Repo: sgl-project/sglang

Monitored Workflows:

• nightly-test-amd.yml
• nightly-test-amd-rocm720.yml
• release-docker-amd-nightly.yml
• release-docker-amd-rocm720-nightly.yml
• amd-aiter-scout.yml
• pr-test-amd.yml
• pr-test-amd-rocm720.yml

Per-workflow failure reports are appended as comments below; the cross-workflow daily status board is rendered above this section.

[amd-bot]

pr-test-amd-rocm720.yml — 16 failure(s)

Run started (UTC): 2026-05-01 17:48 UTC
Last scanned (UTC): 2026-05-02 20:27 UTC
Commits: sglang e0474fd

Summary

Counts: 16 failures · 10 clusters · 5 carrying over (≥3 days) · 3 new in window

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	Pre-flight VRAM cleanup failed (KFD-pinned)	stage-c-test-large-8-gpu-amd-mi35x-rocm720 (mi35x-gpu-8, 1)	N/A (infra)	N/A	recurring infra (2 of last 6 daily runs)	[HIGH] runner-state pollution; KFD-pinned VRAM, no owning process — needs host reset
2	Pre-flight VRAM cleanup failed (KFD-pinned)	stage-c-test-large-8-gpu-amd-mi35x-rocm720 (mi35x-gpu-8, 1)	N/A (infra)	N/A	1 of 6 — alternates with shard 0 by runner	[HIGH] runner-state pollution (5/5 prior runs of shard passed)
3	Pre-flight VRAM cleanup failed (KFD-pinned)	stage-b-test-large-8-gpu-35x-disaggregation-amd-rocm720	N/A (infra)	N/A	new today; prior 3 runs on same runner passed	[MEDIUM] orphaned VRAM from prior workload on this fabric host
4	multimodal-gen 1-GPU shard-0 60-min step timeout	multimodal-gen-test-1-gpu-amd-rocm720 (1gpu, 0)	python/sglang/multimodal_gen/test/server/test_server_1_gpu.py	TestDiffusionServerOneGpu::test_diffusion_generation[ltx_2_3_hq_pipeline]	8 days persistent (since 2026-04-24)	[FACT] LTX-2.3 case added in cd1fa7506 (#23366) pushed 22-case suite past 60-min budget
5	multimodal-gen 1-GPU shard-0 60-min step timeout	multimodal-gen-test-1-gpu-amd-rocm720 (1gpu, 0)	python/sglang/multimodal_gen/test/server/test_server_1_gpu.py	TestDiffusionServerOneGpu::test_diffusion_generation[fast_hunyuan_video]	8 days persistent	[HIGH] cumulative suite runtime exceeds 60 min budget; varies which case is the cutoff victim
6	GPT-OSS-20B mxfp4 streaming returns empty content (mi35x)	stage-b-test-1-gpu-small-amd-mi35x-rocm720 (mi35x-gpu-1)	test/registered/core/test_gpt_oss_1gpu.py	test_mxfp4_20b	Never-passed (10/10 runs since 2026-04-23)	[MEDIUM] AMD aiter-backend gating: needs SGLANG_USE_AITER=1 (per closed #23829)
7	GPT-OSS-20B mxfp4 streaming returns empty content (mi35x)	stage-b-test-1-gpu-small-amd-mi35x-rocm720 (mi35x-gpu-1)	test/registered/core/test_gpt_oss_1gpu.py	test_mxfp4_20b	Never-passed (10/10)	[no candidate ≥ MEDIUM] (endless decode → 1200s harness timeout variant)
8	Triton-attention offline-throughput < 3500 tok/s on rocm720	stage-b-test-1-gpu-large-amd-rocm720 (1gpu, 0)	test/registered/perf/test_bench_serving_1gpu_part1.py	test_offline_throughput_with_triton_attention_backend	Never-passed ≥11 days	[HIGH] rocm720 image runs ~30% slower than rocm700 (sister rocm700 job hits 4812 tok/s, passes); threshold 3500 unrealistic for ROCm 7.2
9	Triton-attention offline-throughput < 3500 tok/s on rocm720	stage-b-test-1-gpu-large-amd-rocm720 (1gpu, 0)	test/registered/perf/test_bench_serving_1gpu_part1.py	test_offline_throughput_with_triton_attention_backend	Never-passed (10/10 measurements 2866–3214 tok/s)	[MEDIUM] AMD MI325 triton path consistently 8-18% under threshold
10	DeepSeek-V3 + mori a2a + TBO MoE crash (8-GPU rocm720)	stage-c-test-large-8-gpu-amd-rocm720 (8gpu, 2)	test/registered/amd/test_moriep_small.py	TestMTPwithTBONormal.test_gsm8k	Never-passed ≥10 days (HSA_STATUS_ERROR_OUT_OF_RESOURCES)	[MEDIUM] HSA symmetric-heap / VRAM exhaustion under MoriEP+MTP+TBO+EAGLE config
11	DeepSeek-V3 + mori a2a + TBO MoE crash (8-GPU rocm720)	stage-c-test-large-8-gpu-amd-rocm720 (8gpu, 2)	test/registered/amd/test_moriep_small.py	TestMTPwithTBONormal.setUpClass	Never-passed ≥30 days (since 2026-04-02; GPU mem fault in aiter._moe_sorting_impl)	[LOW] AMD-untested mori+TBO+EAGLE config; failing class rotates run-to-run
12	Kimi-K2-Instruct setUpClass load timeout (8-GPU FP8 MoE)	stage-c-test-large-8-gpu-amd-rocm720 (8gpu, 1)	test/registered/amd/test_kimi_k2_instruct.py	TestKimiK2Instruct0905.setUpClass	flaky (2/14 runs over 14 days)	[MEDIUM] runner-local I/O variance; TP workers stuck in HIP memcpy_and_sync during MoE FP8 weight load
13	Stage-b partition-12 step timeout (30 min budget)	stage-b-test-1-gpu-small-amd-rocm720 (1gpu, 12)	test/registered/lora/test_lora_overlap_loading.py	TestLoRAOverlapLoading.test_ci_lora_models_batch_splitting	1 day; prior 2 runs green (borderline)	[MEDIUM] noisy runner + chronically borderline 30-min budget; partition 10 also routinely hits the limit
14	GPU Hang during 2-GPU DP prefill (rocm720)	stage-b-test-2-gpu-large-amd-rocm720 (2gpu, 1)	test/registered/distributed/test_data_parallelism.py	TestDataParallelism.test_gsm8k	NEW 2026-05-01 (last green Apr 30)	[MEDIUM] aiter bump in #24218 (v0.1.12.post1 → a6bb49937, 150-commit delta); A/B revert recommended
15	HIP CUDA-graph replay hang in /generate (rocm720)	stage-b-test-1-gpu-small-amd-rocm720 (1gpu, 4)	test/registered/profiling/test_start_profile.py	TestStartProfile.test_start_profile_3	NEW 2026-05-02 (sister rocm700 passes)	[LOW] ROCm 7.2.0 HIP bug class (deadlock in pthread_mutex_lock inside libamdhip64.so.7.2.70200); rocm720-specific
16	Qwen3-Coder-Next 8-GPU TP server fails to start	stage-c-test-large-8-gpu-amd-mi35x-rocm720 (mi35x-gpu-8, 0)	test/registered/amd/test_qwen3_coder_next_8gpu.py	TestQwen3CoderNext.setUpClass	NEW 2026-05-02 (last green May 1)	[MEDIUM] cb8fbd5 (#24243) + 8a53046 (#24244) req-pool slot-0 padding break HybridReqToTokenPool cuda-graph routing

Cluster: Pre-flight VRAM cleanup failed (KFD-pinned)

• Affected: rows 1, 2, 3 (mi35x-gpu-8 shard 1 on May 1 & May 2; mi35x-gpu-8.fabric on May 2). Sister shards on different runners pass.
• Shared facts: VRAM 88-93% at job start across all 8 GPUs; 3× cleanup attempts (pkill + 30 s wait) free zero memory; lsof /dev/kfd finds no owning process; script exits 1 before docker container start. Failure tracks runner instance, not matrix index or SHA.
• Top hypothesis: [HIGH] orphaned KFD-held VRAM from a prior workload that crashed without releasing memory; only host reboot or amdgpu reload can reclaim. Disconfirming: none — classic ROCm shared-runner pattern.
• In-flight fix: ❌ none found.
• Suggested triage: Have AMD CI ops reboot/recycle runners linux-mi35x-gpu-8-4wrmw-runner-7cfm2, …-5bplx, and linux-mi35x-gpu-8.fabric-kbhnc-runner-kh4bf; consider hardening scripts/ci/amd/ensure_vram_clear.sh to escalate (mark runner unhealthy / rocm-smi --gpureset) on attempt-3 failure.

Cluster: multimodal-gen 1-GPU shard-0 60-min step timeout

• Affected: rows 4, 5 (same shard, 2 runs).
• Shared facts: 21–22 cases run sequentially (max-parallel: 1); last green run 24850489147 finished in 56:25 with only 3.5 min slack; timeout-minutes: 60 at .github/workflows/pr-test-amd-rocm720.yml:625.
• Top hypothesis: [FACT] adding ltx_2_3_hq_pipeline in cd1fa7506 (#23366) pushed shard 0 over budget; failure starts on the very next scheduled run after the commit lands. Different cases hit the cutoff in different runs (cache warmth jitter).
• In-flight fix: ❌ none found.
• Suggested triage: deselect ltx_2_3_hq_pipeline from shard 0 (or move it) and re-run — expect ~56 min; alternatives: bump timeout-minutes to 75 or repartition matrix from 4 → 5/6 partitions.

Cluster: GPT-OSS-20B mxfp4 streaming returns empty content (mi35x)

• Affected: rows 6, 7 (same shard, 2 runs). Symptom alternates between empty-stream AssertionError and 1200s file timeout from same retry loop.
• Shared facts: test_bf16_20b in same file passes; only test_mxfp4_20b fails. AMD MI35X-only. Closed PR #23829 (closed unmerged 2026-04-28) had attempted SGLANG_USE_AITER=1 gating.
• Top hypothesis: [MEDIUM] AMD aiter-backend gating: mxfp4 path needs SGLANG_USE_AITER=1. Disconfirming: PR #23829 was rejected — review reason unknown; rejection may reflect a different preferred fix.
• In-flight fix: ❌ none — needs new PR.
• Suggested triage: revisit #23829's gating approach (or check the closed-PR review thread for the rejection rationale); reproduce locally on linux-mi35x-gpu-1 with and without SGLANG_USE_AITER=1. Per memory, the same failure also affects sister pr-test-amd.yml mi35x job.

Cluster: Triton-attention offline-throughput < 3500 tok/s on rocm720

• Affected: rows 8, 9 (same shard, 2 runs). Throughput band 2866–3214 tok/s in 10-of-10 completed runs.
• Shared facts: threshold 3500 is hard-coded under is_in_amd_ci() at test/registered/perf/test_bench_serving_1gpu_part1.py:120. Sister pr-test-amd.yml (rocm700) job hits 4812 tok/s and passes the same test on the same runner type and SHA.
• Top hypothesis: [HIGH] ROCm 7.2 image runs the triton-attention path ~30% slower than ROCm 7.0; the 3500 threshold was carried over by refactor 4fc5ebf0b (#23389) without re-applying the prior 3050 lowering. Disconfirming: none — pre-refactor 3050 threshold also failed on rocm720.
• In-flight fix: ❌ none — adjacent #23957 "[AMD] CDNA4 Gluon Extend Attention" might lift throughput but is not a confirmed fix.
• Suggested triage: split AMD threshold per image variant (~3700 rocm700, ~2700 rocm720) at line 120; or unify to ~2700 with margin.

Cluster: DeepSeek-V3 + mori a2a + TBO MoE crash (8-GPU rocm720)

• Affected: rows 10, 11 (same shard, 2 runs; failing class rotates between TestMTPwithTBONormal setUpClass/test_gsm8k and TestMTPwithTBOLowLatency).
• Shared facts: server config --moe-a2a-backend mori --enable-two-batch-overlap --speculative-algo EAGLE --tp 8 --ep 8 --dp 8 --mem-fraction-static 0.72. Symptoms vary: GPU mem-access fault in aiter._moe_sorting_impl during CUDA-graph capture (May 1) vs HSA_STATUS_ERROR_OUT_OF_RESOURCES … Available Free mem : 0 MB during gsm8k batching (May 2). Earlier classes in same file (TestMTP, TestPureDP) pass with same model. Test code already sets MORI_SHMEM_MODE=ISOLATION "to avoid out of symmetric heap memory" — this config has documented OOM history.
• Top hypothesis: [MEDIUM] MoriEP+MTP+TBO+EAGLE is a memory-stressed AMD-only config that has never had a green run; HSA queue/symmetric-heap exhaustion under high concurrency.
• In-flight fix: ❌ none confirmed — adjacent feature PRs (#22985, #23760, #11150) are not OOM fixes.
• Suggested triage: A/B knob sweep on standalone class — --mem-fraction-static 0.65, MORI_ENABLE_SDMA=false, SGLANG_MORI_NUM_MAX_DISPATCH_TOKENS_PER_RANK=2048, --cuda-graph-max-bs 16, drop --enable-two-batch-overlap — first knob that turns it green isolates the offending dimension. Consider quarantining TestMTPwithTBO* on AMD until pinned.

Cluster: Kimi-K2-Instruct setUpClass load timeout

• Affected: row 12 (single shard; flaky 2/14).
• Shared facts: TP workers blocked in c10::hip::memcpy_and_sync during FP8 MoE weight load; same code path is exercised in 12 passing runs on the same runner pool. No commit in ece8a1a7..e0474fdd9 touches fused_moe_triton/. Sister shards (0, 2) still in-flight at snapshot time.
• Top hypothesis: [MEDIUM] slow runner-local disk / pinned-buffer I/O on linux-mi325-8gpu-sglang pushes the 22-min healthy load past the 60-min step budget. Disconfirming: cannot directly measure runner I/O; identical failure on Apr 29 with an entirely different sglang HEAD argues against a code regression.
• In-flight fix: ❌ none — runner-side investigation.
• Suggested triage: rerun shard 1 up to 2× (if green, classify as runner-flaky); add iostat -x instrumentation to the pre-test step; bump shard's timeout-minutes 60 → 90.

Cluster: Stage-b partition-12 step timeout (30 min budget)

• Affected: row 13.
• Shared facts: partition 12 used 21.8 min yesterday (8 min margin); today every one of the 5 completed tests is 19-50% slower; other partitions today (7, 10) are faster than yesterday on the same image+aiter, so it is not a global regression. Stale FILE_TIMES (test_lora_load_from_tensor.py estimated 90 s vs actual 180-214 s) inflate the partitioner's plan.
• Top hypothesis: [MEDIUM] noisy runner instance + chronically borderline 30-min budget. Disconfirming: 5-for-5 same-direction slowdown is unusual for pure runner noise.
• In-flight fix: ❌ none — needs FILE_TIMES refresh and/or timeout-minutes bump.
• Suggested triage: rerun partition 12; refresh FILE_TIMES; bump per-step timeout 30 → 35-40 min.

Cluster: GPU Hang during 2-GPU DP prefill (rocm720)

• Affected: row 14 (NEW 2026-05-01).
• Shared facts: HW Exception by GPU node-2 reason :GPU Hang mid-prefill on Llama-3.1-8B (TP=1 DP=2); both attempts fail identically. Aiter version is the ONLY environment change since last green: v0.1.12.post1 → dev150+ga6bb49937 (150-commit delta), bumped by sglang #24218 (7e17a7a1e).
• Top hypothesis: [MEDIUM] aiter bump introduced a regression hanging GPU during BF16 GEMM/prefill in 2-GPU DP. Disconfirming: log-line proximity is not causation (CLAUDE.md AITER Scout [CI Monitor] Daily Report - 2026-06-03 #92 lesson); this is the regular nightly, no sister-baseline A/B available.
• In-flight fix: ❌ none.
• Suggested triage: re-run with AITER_COMMIT_OVERRIDE=v0.1.12.post1 on same sglang HEAD — cheapest A/B for the leading hypothesis. If exonerates aiter, bisect 39-commit sglang window starting with ece8a1a78 (#24197).

Cluster: HIP CUDA-graph replay hang in /generate (rocm720)

• Affected: row 15 (NEW 2026-05-02; sister rocm700 shard 4 passed at near-same SHA).
• Shared facts: scheduler watchdog timeout 300 s; py-spy stack pinned at pthread_mutex_lock inside libamdhip64.so.7.2.70200 → at::cuda::CUDAGraph::replay → cuda_graph_runner.py:1328. None of the 28 commits in the regression window touch cuda_graph_runner.py/model_runner.py/tp_worker.py/profiler/watchdog.
• Top hypothesis: [LOW] ROCm 7.2.0 HIP bug class (related to known #24011 family); profile start/stop/start sequence may corrupt graph state. Disconfirming: 4 prior rocm720 runs of same shard passed — could be a low-rate flake.
• In-flight fix: ✅ #24151 (open) — "[Docker][ROCm] Bump ROCm720 base image from 7.2.0 to 7.2.2"; ✅ #24245 (open, companion). Chase reviewers.
• Suggested triage: wait for next rocm720 schedule; if recurs, A/B by rebuilding rocm720 image on ROCm 7.2.2.

Cluster: Qwen3-Coder-Next 8-GPU TP server fails to start

• Affected: row 16 (NEW 2026-05-02; last green May 1).
• Shared facts: TP1 soft-watchdog timeout during CUDA-graph capture → HIP failure: 'invalid device pointer' from RCCL across 7/8 ranks → DistStoreError: wait timeout after 600000ms on aiter_ipc/p0/g0_1_2_3_4_5_6_7/2/r6 → server SIGKILL. Sister test test_deepseek_r1_mxfp4_8gpu.py (also tp=8 + aiter) ran first in same job and passed — runtime/aiter healthy. Of 28 commits in window, only cb8fbd5 and 8a53046 touch req-pool/cuda_graph plumbing.
• Top hypothesis: [MEDIUM] PRs #24243 "Reserve slot 0 as padding in all req pools" + #24244 "Size mamba mappings from req pool" alter HybridReqToTokenPool slot indexing; Qwen3-Coder-Next is a hybrid mamba+attention MoE, off-by-one in dummy-slot routing during cuda-graph-padded batches matches the symptom. Disconfirming: failing all-gather is in _get_logits, not directly in mamba-mixer code; NVIDIA confirmation absent.
• In-flight fix: ❌ none.
• Suggested triage: revert 8a53046 and cb8fbd5 on a branch and re-run on linux-mi35x-gpu-8 rocm720; or bisect ece8a1a788..e0474fdd9b.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Recycle/reboot runners linux-mi35x-gpu-8-4wrmw-runner-{7cfm2,5bplx} and linux-mi35x-gpu-8.fabric-kbhnc-runner-kh4bf; track whether ensure_vram_clear recurs on next schedule	Cluster Pre-flight VRAM (rows 1-3)	Largest cluster (3 jobs); pure ops, fast win, blocks 8-GPU mi35x coverage
2	Re-run with AITER_COMMIT_OVERRIDE=v0.1.12.post1 on sglang HEAD to A/B #24218	row 14 (GPU Hang DP)	Brand-new May-1 regression, single environment variable changed; cheapest A/B with biggest blast-radius implication
3	Revert cb8fbd5 + 8a53046 on a branch and re-run test_qwen3_coder_next_8gpu.py on mi35x rocm720	row 16 (Qwen3-Coder-Next NEW)	Brand-new May-2 regression with clear hypothesis confined to 2 commits
4	Deselect ltx_2_3_hq_pipeline from shard 0 (or repartition matrix 4 → 5/6)	rows 4-5 (multimodal-gen)	Largest persistent code-side cluster (8 days); causal commit identified; one-line workflow change unblocks 21 cases of coverage
5	Lower AMD threshold in test_bench_serving_1gpu_part1.py:120 to ~2700 (or split per image variant)	rows 8-9 (triton throughput)	Never-passed for 11+ days; intrinsic ROCm 7.2 perf gap confirmed via rocm700 sister; obscures real regressions
6	Verify in-flight fixes #24151 / #24245 (ROCm 7.2.2 bump) and chase reviewers	row 15 (test_start_profile hang)	Open PRs already address the underlying ROCm 7.2.0 HIP class; merge resolves at no extra investigation cost
7	Reopen GPT-OSS mxfp4 fix per closed #23829 review thread	rows 6-7 (test_mxfp4_20b)	Never-passed; clear gating hypothesis; previous PR exists with reviewer feedback to act on
8	Knob-sweep MoriEP+MTP+TBO standalone (--mem-fraction-static 0.65, MORI_ENABLE_SDMA=false, etc.)	rows 10-11 (mori HSA OOM)	Persistent 10+ days but config-specific — A/B sweep cheaper than bisecting 30-day window; consider quarantining the TBO classes meanwhile
9	Refresh FILE_TIMES for stage-b suite + bump timeout-minutes 30 → 35	row 13 (partition 12)	Borderline single-day flake; cheap config change to stabilize
10	Rerun Kimi-K2 shard 1 up to 2× and add iostat instrumentation	row 12 (Kimi-K2 timeout)	Flaky 2/14, likely runner I/O — needs measurement before any code action

---

Job	Failed Steps	Started
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1)	Ensure VRAM is clear	2026-05-01T19:13
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run diffusion server tests (1-GPU)	2026-05-01T17:48
stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1)	Run test	2026-05-01T17:48
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run test	2026-05-01T17:48
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 2)	Run test	2026-05-01T18:50
stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 1)	Run test	2026-05-01T17:48
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 12)	Run test	2026-05-01T17:48
stage-b-test-large-8-gpu-35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric)	Ensure VRAM is clear	2026-05-02T17:45
stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1)	Run test	2026-05-02T17:45
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1)	Ensure VRAM is clear	2026-05-02T18:04
stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 4)	Run test	2026-05-02T17:45
stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run test	2026-05-02T17:45
stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 0)	Run test	2026-05-02T17:54
multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0)	Run diffusion server tests (1-GPU)	2026-05-02T17:45
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 1)	N/A	2026-05-02T18:18
stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 2)	Run test	2026-05-02T18:54

Per-Job Analysis

stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1) — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: ece8a1a78
• aiter: N/A (job aborted in pre-test runner cleanup; no [CI-AITER-CHECK] markers in log — container never started)

Failed Tests

Test File	Test Function	Error	Log
N/A (no tests ran)	N/A	=== FAILED: VRAM cleanup unsuccessful after 3 attempts === followed by Process completed with exit code 1	Log line 327

The job failed during the pre-test ensure_vram_clear.sh rocm step — runner had VRAM stuck at 93% on all 8 GPUs from a prior workload, and 3 cleanup attempts (kill SGLang processes, aggressive cleanup, 30 s waits) all failed to free it. No test container was ever started; no test execution occurred.

Failure Summary

The runner linux-mi35x-gpu-8-4wrmw-runner-5bplx started this job with all 8 MI35x GPUs already at 93% VRAM allocation from a prior occupant. The pre-job script scripts/ci/amd/ensure_vram_clear.sh failed three cleanup attempts and exited 1 before any sglang code ran. After the cleanup loop, lsof /dev/kfd reported No processes found accessing /dev/kfd — i.e., VRAM is held by something the script cannot kill (likely a leaked allocation, kernel driver state, or a different container/cgroup the script lacks visibility into).

Regression Status

Infrastructure issue (runner-state pollution, not a code regression).

Recent history of this exact shard stage-c-test-large-8-gpu-amd-mi35x-rocm720 (linux-mi35x-gpu-8, 1) (same branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
May 1	25225640477	shard 1	❌ Failed (pre-test, no tests ran)	—	VRAM cleanup unsuccessful after 3 attempts
Apr 30	25180892243	shard 1	✅ Passed	—	—
Apr 29	25125273850	shard 1	✅ Passed	—	—
Apr 28	25069395852	shard 1	✅ Passed	—	—
Apr 27	25010896602	shard 1	✅ Passed	—	—
Apr 26	24962999668	shard 1	✅ Passed	—	—

This is a single-day, single-job environmental flake on shard 1; the sister shard 0 in the same run also needs to be checked separately (different job ID). All 5 prior scheduled runs of this exact shard passed.

Failure Cluster

Runner pre-test VRAM-cleanup failure (runner-state pollution, no test executed)

Facts (verified observations, no interpretation)

• [FACT] Initial GPU status at job start (line 132–163 of log): GPU[0..7] : GPU Memory Allocated (VRAM%): 93 on all 8 MI35x GPUs.
• [FACT] Three cleanup attempts (lines 167–326) all reported VRAM still not clear; VRAM% never moved off 93 between attempts.
• [FACT] Post-cleanup process audit (lines 367–371): No processes found using /dev/kfd, No processes found accessing /dev/kfd. The VRAM holder is invisible to the script's process scan.
• [FACT] Job exited at line 372 with Process completed with exit code 1 from scripts/ci/amd/ensure_vram_clear.sh. No docker container was started, no sglang process ran, no [CI-AITER-CHECK] markers in log.
• [FACT] Same shard passed on the prior 5 consecutive scheduled runs (Apr 26–30) on the same workflow + branch + event.
• [FACT] Runner hostname: linux-mi35x-gpu-8-4wrmw-runner-5bplx (recorded once, line 121).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Stale GPU memory from a prior workload on the same self-hosted runner that the cleanup script could not reclaim.
  • Supporting: VRAM is at 93% on every GPU at job entry; the script kills sglang-named processes and runs aggressive cleanup but VRAM% never drops; lsof /dev/kfd finds nothing — meaning the holder is either (a) a different container the script does not inspect, (b) leaked allocations the kernel hasn't reclaimed, or (c) a process in another cgroup/namespace.
  • Disconfirming: (none — the symptom is exactly what runner-state pollution looks like, and 5 prior runs of the identical shard on the same workflow passed, ruling out a code-side regression in this job.)
  • How to verify: SSH to runner linux-mi35x-gpu-8-4wrmw-runner-5bplx, run rocm-smi and fuser -v /dev/kfd /dev/dri/*, identify the VRAM holder, and reboot or drain the runner. Re-trigger the job — expect green on a clean runner.
• [LOW] ensure_vram_clear.sh cleanup logic is too narrow (only kills processes named sglang, doesn't restart docker/cgroup, doesn't reset GPU via rocm-smi --gpureset).
  • Supporting: the script gives up after process-kill + 30 s wait × 3 with no escalation to GPU reset or docker daemon restart.
  • Disconfirming: the script worked for 5 consecutive prior runs of this shard, suggesting current logic is sufficient for typical post-job state — today's runner state is anomalous.
  • How to verify: read scripts/ci/amd/ensure_vram_clear.sh and consider whether to add a rocm-smi --gpureset (or docker restart) escalation step. Out of scope for this single-job triage.

No commit-level hypothesis applies — no test code ran, and the failure is in the pre-test runner cleanup script, which is not on the pass_sha..fail_sha code path of this job.

In-flight Fix Check

Searched repo:sgl-project/sglang is:pr is:open VRAM cleanup and ensure_vram_clear — no matching open PRs.

• In-flight fix: ❌ none found — operational issue requiring runner reboot/drain, not a code fix.

Suggested Triage Steps

• Drain/reboot runner linux-mi35x-gpu-8-4wrmw-runner-5bplx and re-queue the failed shard. Expect green on a clean runner (5/5 recent prior runs passed).
• Identify the VRAM holder before draining: ssh to the runner, docker ps -a, fuser -v /dev/kfd /dev/dri/*, rocm-smi --showpids. Whichever container/process owns the 93% allocation is the prior leaker — look at the previous job that ran on this runner for a clue.
• If this VRAM-stuck pattern recurs across multiple runners over the next few days, consider hardening scripts/ci/amd/ensure_vram_clear.sh: add a final escalation step (e.g. docker stop $(docker ps -q) or rocm-smi --gpureset if available on MI35x), and/or check nvidia-smi-style per-process VRAM accounting beyond name-based process kill.
• No bisect needed; no test was executed, and prior 5 runs on the same shard passed.

Status (factual, no priority assignment)

• Persistence: Single occurrence (1 of 6 most recent scheduled runs of this shard). All prior 5 runs (Apr 26–30) passed.
• Scope: One job/shard on one runner (linux-mi35x-gpu-8-4wrmw-runner-5bplx) on May 1 only. Shard 0 of the same workflow run is a separate job/runner and must be evaluated separately.
• Blocked work: Today's pr-test-amd-rocm720 shard 1 large-8-gpu coverage is missing; sister shard 0 outcome unknown without separate inspection.
• In-flight fix: ❌ none found — operational issue requiring runner reboot/drain, not a code fix.

multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — failed step(s): Run diffusion server tests (1-GPU)

Commit Info

• sglang: ece8a1a78
• aiter: a6bb49937 (source: dev image — v0.1.12.post2.dev150+ga6bb49937 preserved per [CI-AITER-CHECK])

Failed Tests

Test File	Test Function	Error	Log
python/sglang/multimodal_gen/test/server/test_server_1_gpu.py	TestDiffusionServerOneGpu::test_diffusion_generation[fast_hunyuan_video]	Job-level action timeout: The action 'Run diffusion server tests (1-GPU)' has timed out after 60 minutes while polling GET /v1/videos for video generation result (no per-test pytest failure emitted)	Log L7294

Failure Summary

The multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0) job hit GitHub's 60-minute step timeout on Run diffusion server tests (1-GPU). 17 of the 18 parametrized cases in TestDiffusionServerOneGpu::test_diffusion_generation passed sequentially; the 18th (fast_hunyuan_video) had its server up and was polling video status when the action timer expired. There is no per-test assertion failure — the failure is the wall-clock budget for the whole shard.

Regression Status

Persistent failure (8 consecutive scheduled runs failing for shard 0 in pr-test-amd-rocm720.yml).

Recent history of multimodal-gen-test-1-gpu-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — same workflow, event=schedule, branch=main, completed runs only:

Date (UTC)	Run	Job (shard 0)	Status	Last test in log	Time-to-timeout
2026-05-01	25225640477	shard 0	❌ timeout	fast_hunyuan_video (polling)	60 min
2026-04-30	25180892243	shard 0	❌ timeout	ltx_2_3_hq_pipeline (after fast_hunyuan_video PASSED in 11 min)	60 min
2026-04-29	25125273850	shard 0	❌ timeout	fast_hunyuan_video (polling)	60 min
2026-04-28	25069395852	shard 0	❌ timeout	ltx_2_3_hq_pipeline (after fast_hunyuan_video PASSED in 11 min)	60 min
2026-04-27	25010896602	shard 0	❌ timeout	fast_hunyuan_video (polling)	60 min
2026-04-26	24962999668	shard 0	❌ timeout	—	60 min
2026-04-25	24936802965	shard 0	❌ timeout	—	60 min
2026-04-24	24903665661	shard 0	❌ timeout	—	60 min
2026-04-23	24850489147	shard 0	✅ Passed	21 passed in 3385.69s (0:56:25)	— (3.5 min headroom)
2026-04-22	24793649109	shard 0	✅ Passed	10 passed in 1802.85s (0:30:02)	—

First observed failure: 2026-04-24 (run 24903665661). Last green: 2026-04-23 (72749317701). Persisting for 8 consecutive scheduled runs.

Failure Cluster

60-min action timeout in multimodal-gen 1-GPU shard 0 (cumulative test-suite runtime exceeds budget; fast_hunyuan_video is the typical timeout point because it lands ~50 min into the shard with ~11 min own runtime).

Facts (verified observations, no interpretation)

• The action Run diffusion server tests (1-GPU) has timeout-minutes: 60 (.github/workflows/pr-test-amd-rocm720.yml:625). All 8 failing runs hit this exact action timeout — no pytest assertion / no exception in any of them.
• The test runner is python3 sglang/multimodal_gen/test/run_suite.py --suite 1-gpu --partition-id ${{ matrix.part }} --total-partitions 4 -k "not flux_2" (.github/workflows/pr-test-amd-rocm720.yml:646-649); max-parallel: 1, so the 4 partitions run sequentially per runner — but the 60-min budget is per-step, not per-partition.
• In the current run (2026-05-01), 17 cases PASSED in 50:06 (17:59:49 → 18:49:56). fast_hunyuan_video server started at 18:49:56, model loaded by 18:51:17, and the test was polling GET /v1/videos from 18:53:28 until the timeout fired at 18:59:22.81 (log L7294). The video gen had not finished within the remaining ~9 min budget.
• On 2026-04-30 (job 73825595365) fast_hunyuan_video passed in 11 min, then wan2_2_ti2v_5b and fastwan2_2_ti2v_5b also passed, and ltx_2_3_hq_pipeline was the test that triggered the timeout. So the timing-out test varies per run depending on cache warmth and runtime jitter.
• Test count in shard 0 grew from 10 (Apr 22, 30:02) → 21 (Apr 23, 56:25) → exceeds 60-min budget (Apr 24+).
• The single commit between the last green (Apr 23 head 80125feb) and first failure (Apr 24 head baa0aa67) is baa0aa670 "[HiCache & HybridModel] 3FS backend support DSA & mamba model" (#23241) — unrelated to diffusion / multimodal_gen.
• AITER override is empty; image had a dev build v0.1.12.post2.dev150+ga6bb49937 based on the Dockerfile-expected SHA (a6bb499) and was preserved.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Test-suite runtime in shard 0 of the 1-GPU multimodal-gen partition has grown past the 60-min timeout-minutes budget, with fast_hunyuan_video (~11 min runtime) being the natural straw that breaks the camel's back when it lands ~50 min in.

  • Supporting: Apr 22 ran 10 tests in 30:02; Apr 23 ran 21 tests in 56:25 (3.5 min headroom); Apr 24+ all 60-min timeouts. Different tests trigger the cutoff in different runs (fast_hunyuan_video in 3 runs, ltx_2_3_hq_pipeline in 2 runs), which is the signature of a budget exhaustion not a per-test bug. No pytest exception in any run.
  • Disconfirming: (none found) — every failing run has identical ##[error]The action ... has timed out after 60 minutes with no other error in the log.
  • How to verify: time individual tests across runs to confirm cumulative runtime > 3600 s; also inspect run_suite.py partitioning for shard 0 vs shards 1-3 and check whether the partition splitter distributes tests by name hash (so shard 0 may have collected the longest videos like Hunyuan/LTX2.3 by chance).

• [LOW] A specific commit in the de962f32..80125feb window slowed down individual tests enough to push the budget over.

  • Supporting: temporal correlation — the budget edge was crossed between Apr 23 (3.5 min headroom) and Apr 24 (over). Touching commits in the prior day's window: 86ed068 (OpenTelemetry tracing on DiffGenerator, #21254), 3c5b1f0 (#23198), 4d3c7e7 (#23517). The Apr 23→Apr 24 window itself contains only baa0aa670 (3FS HiCache, unrelated).
  • Disconfirming: Apr 23 only had 3.5 min headroom — natural runtime jitter of any one of the 21 tests is sufficient to flip the verdict; no commit needed. Also, fast_hunyuan_video itself takes ~11 min on Apr 28/Apr 30 (when it passes) which by itself is ~18% of the budget.
  • How to verify: not actionable until Hypothesis 1 is mitigated (raise budget or repartition). If after that some specific test is still slower than expected, then bisect.

In-flight Fix Check

Searched open PRs with keywords multimodal-gen+timeout, fast_hunyuan, multimodal+partition:

• In-flight fix: ❌ none found — no open PR addressing the budget overrun for the 1-GPU multimodal-gen shard 0. Adjacent PRs (#24015 disagg test timeout, #19114 parallelize pipeline module loading) are unrelated.

Suggested Triage Steps

• Raise timeout-minutes for the Run diffusion server tests (1-GPU) step in .github/workflows/pr-test-amd-rocm720.yml:625 from 60 to e.g. 75 — quickest mitigation and matches the budget headroom shown by the 56:25 last-pass.
• Or: increase --total-partitions from 4 → 5 (or 6) and the matrix part list correspondingly, so shard 0's load drops back under budget. Same change applies to pr-test-amd.yml if the AMD (non-rocm720) job has the same partitioning.
• Or: add explicit per-test runtime balancing in python/sglang/multimodal_gen/test/run_suite.py (e.g. assign fast_hunyuan_video, ltx_2_3_hq_pipeline, wan2_2_ti2v_5b to different partitions) so no shard accumulates all the long video tests.
• Verify by re-running shard 0 alone with the workflow change, expecting 21 passed in <budget> s.
• If after raising the budget shard 0 still times out, instrument per-test wall time (e.g. enable pytest --durations=0) to confirm whether a specific test regressed or all tests are uniformly slower (e.g. JIT cache cold).

Status

• Persistence: Failing every completed scheduled run for 8 days (since 2026-04-24, last green 2026-04-23 with 3.5 min headroom).
• Scope: One job — shard 0 of multimodal-gen-test-1-gpu-amd-rocm720. Other shards (1, 2, 3) are passing in the same runs (verified for 2026-04-30 and 2026-04-29). Likely also affects shard 0 of pr-test-amd.yml's multimodal-gen-test-1-gpu-amd job (not verified in this analysis — separate workflow).
• Blocked work: Continuous red signal on AMD ROCm 7.2 multimodal-gen 1-GPU coverage for shard 0 (the 17 tests preceding fast_hunyuan_video do still execute and pass, so coverage of those is intact).
• In-flight fix: ❌ none found — needs new PR (workflow timeout-minutes bump or partition rebalance).

stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1) — failed step(s): Run test

Commit Info

• sglang: ece8a1a78
• aiter: a6bb49937 (source: Dockerfile default — gfx950 block; dev image v0.1.12.post2.dev150+ga6bb49937 matched expected SHA so no rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/core/test_gpt_oss_1gpu.py	TestGptOss1Gpu.test_mxfp4_20b	Process timeout after 1200s (test never returns; server stays in endless single-request decode)	Log

Failure Summary

The job runs two sub-tests via python3 test_gpt_oss_1gpu.py. test_bf16_20b finished cleanly (all three GPQA reasoning_efforts produced scores ≥ expected). test_mxfp4_20b then loaded openai/gpt-oss-20b, captured CUDA graphs, started decoding, and never returned — Decode batch, #running-req: 1 was still being emitted at 90 tok/s when the 1200 s wall-clock timeout fired and the runner killed the script with exit 255.

Regression Status

Known recurring failure (Never-passed in queryable window). This test has failed on every pr-test-amd-rocm720.yml scheduled run examined (Apr 26 → May 1, 2026). No green run found in the last 10 scheduled runs.

Recent history of test/registered/core/test_gpt_oss_1gpu.py (test_mxfp4_20b) in job stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1) (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
May 01	25225640477	job	❌ Failed	test_mxfp4_20b	TIMEOUT after 1200s
Apr 30	25180892243	job	❌ Failed	test_mxfp4_20b	TIMEOUT after 1200s
Apr 29	25125273850	job	❌ Failed	test_mxfp4_20b	TIMEOUT after 1200s
Apr 28	25069395852	job	❌ Failed	test_mxfp4_20b	AssertionError: False is not true (errors=1)
Apr 27	25010896602	job	❌ Failed	test_mxfp4_20b	AssertionError: False is not true (errors=1)
Apr 26	24962999668	job	❌ Failed	test_mxfp4_20b	AssertionError: False is not true (errors=1)

Lookback range: Apr 22 → May 1, 2026 (10 scheduled runs); no green run for test_mxfp4_20b found. Failure mode shifted around Apr 29: previously the test asserted out quickly (~1.5 min after starting test_mxfp4_20b); now it hangs in endless single-request decode until the 1200 s harness timeout.

Failure Cluster

test_mxfp4_20b on mi35x-rocm720 — endless single-request decode → 1200s harness timeout

Facts (verified observations, no interpretation)

• test_bf16_20b PASSED in this run: GPQA scores 0.379 (low) / 0.394 (medium) / 0.323 (high), all above expected (0.34/0.34/0.27); evaluations completed by 18:07:10 (log).
• test_mxfp4_20b server bring-up succeeded: weights loaded (avail mem=272.76 GB, mem usage=14.44 GB at 18:08:24), KV cache allocated (full=57.7 GB, swa=46.2 GB), CUDA graphs captured for bs=[1..600] starting 18:08:25.
• The hang state shows #running-req: 1, #full token: 84,000+ → 87,000+, full token usage: 0.02, gen throughput ~90 tok/s for the entire ~10 min between end-of-CUDA-graph-capture and the harness timeout — i.e. one request that never produces an EOS / never reaches max_tokens before 1200 s elapses.
• Aiter version is unchanged from the Dockerfile default for gfx950: dev image v0.1.12.post2.dev150+ga6bb49937 matches AITER_COMMIT=a6bb499375849eec45d68c5ccaebc8865fd422c0; [CI-AITER-CHECK] confirms no rebuild (log).
• This is a non-scout pr-test-amd-rocm720.yml schedule run (no AITER_COMMIT_OVERRIDE), so no Failure Origin / Baseline A/B section is required.
• Health-check warning earlier in the bf16 phase: Health check failed. Server couldn't get a response from detokenizer for last 20 seconds at 18:02:27 (log line) — bf16 still passed, so this looks like a transient detokenizer stall and not the mxfp4 hang.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] test_mxfp4_20b was never functional on mi35x-rocm720 and the test simply does not converge to an EOS within the GPQA dataset / max_tokens=4096 window when running in the mxfp4 path on this hardware/runtime.

  • Supporting: previous failure mode (Apr 26-28) was an assertion error in the same test; current failure mode (Apr 29-May 1) is a hang with endless decode of a single request; in either mode, the test has never passed in the 10-run lookback window.
  • Supporting: the previous fix attempt #23829 "[AMD] Update test_gpt_oss_1gpu.py" (which proposed SGLANG_USE_AITER=1) was closed unmerged on 2026-04-28 — same day the failure mode flipped from assert to timeout, suggesting a related sglang change landed around then.
  • Disconfirming: log emits "mxfp4 quantization is not fully optimized yet" warning but otherwise no obvious error — startup, weight load, KV cache, and CUDA graph capture all succeed. Without a stack trace of WHERE the request is stuck, this is symptom-level only.
  • How to verify: git log --oneline 10fd0fac..ece8a1a7 -- python/sglang/srt/models/gpt_oss.py python/sglang/srt/layers/quantization/mxfp4* test/registered/core/test_gpt_oss_1gpu.py test/srt/gpt_oss_common.py to find the change that flipped assert→timeout on Apr 29; reproduce locally on linux-mi35x-gpu-1 runner with the test command.

• [SPECULATION] Endless decode could indicate a sampling / EOS handling regression in the mxfp4 path (e.g. forced-min-tokens or bad eos_token_id for the openai/gpt-oss-20b repo) where the single decoded request never emits an EOS for the chosen GPQA prompt.

  • Supporting: #running-req: 1 and #full token grows monotonically (84k→87k+) for ~10 min straight — only one request, never finishing.
  • Disconfirming: same model+test in the bf16_20b variant does terminate (and hits target accuracy). If this were a generic EOS issue we would expect it to appear in bf16 too — the symptom is specific to mxfp4.
  • How to verify: enable log_requests=true for the mxfp4 server and inspect a single request's sampling params + completion reason; or run a one-off completion against the booted server with very low max_tokens.

No hypothesis above LOW confidence; needs reproduction on the linux-mi35x-gpu-1 runner and a git log walk between baa0aa67 (Apr 24, last "assert" failing run) and HEAD to find what switched assert→timeout.

In-flight Fix Check

Searches:

• is:pr is:open repo:sgl-project/sglang test_mxfp4_20b → 0 results
• is:pr is:open repo:sgl-project/sglang gpt-oss mxfp4 → 10 results, none are AMD/mi35x test fixes for this hang (closest is #23218 "[WIP] feat: add triton_quant_info for mxfp4 gpt-oss moe layer" but it's WIP and unrelated to the test harness)
• Previous attempt #23829 "[AMD] Update test_gpt_oss_1gpu.py" was closed unmerged on 2026-04-28

In-flight fix: ❌ none found in open PRs — needs new PR or investigation

Suggested Triage Steps

• Reproduce locally on a linux-mi35x-gpu-1 runner: launch the booted mxfp4 server (python3 -m sglang.launch_server --model openai/gpt-oss-20b --quantization mxfp4 --attention-backend aiter ...) and send a single short GPQA-style completion request with max_tokens=4096 and inspect whether (a) it ever finishes and (b) what the completion reason is.
• git log --oneline baa0aa67..ece8a1a7 -- python/sglang/srt/models/gpt_oss.py python/sglang/srt/layers/quantization/ test/registered/core/test_gpt_oss_1gpu.py test/srt/gpt_oss_common.py to enumerate sglang changes in the assert→timeout transition window (Apr 24-29).
• Re-evaluate why #23829 was closed without merge — if SGLANG_USE_AITER=1 was the right env wrapping for this test, a corrected PR may unblock CI.
• If reproduction confirms an EOS / sampling issue specific to the mxfp4 path on mi35x, consider quarantining test_mxfp4_20b on mi35x-rocm720 until a real fix lands so this signal stops masking other regressions in the same job.

Status

• Persistence: Failing every completed pr-test-amd-rocm720.yml scheduled run for ≥6 days (verified Apr 26 → May 1, 2026). Likely Never-passed on this job — no green run found in the 10-run lookback.
• Scope: This single job (stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1)). Per memory, the same test_mxfp4_20b is also Never-passed in the sibling pr-test-amd.yml job stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1), so the failure spans two workflows on the same mi35x hardware.
• Blocked work: Blocks the entire test_gpt_oss_1gpu.py signal on linux-mi35x-gpu-1 for both the rocm720 and the rocm-default pr-test-amd* workflows (test_bf16_20b passes but the harness reports the whole script as failed because of the timeout).
• In-flight fix: ❌ none found — previous attempt #23829 was closed unmerged 2026-04-28.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-05-02 20:27 UTC)

[amd-bot]

pr-test-amd-rocm720.yml — Per-Job Analysis (continued 2/4)

stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0) — failed step(s): Run test

Commit Info

• sglang: ece8a1a78
• aiter: a6bb49937 (source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected: v0.1.12.post2.dev150+ga6bb49937 → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/perf/test_bench_serving_1gpu_part1.py	TestBenchServing1GPUPart1.test_offline_throughput_with_triton_attention_backend	AssertionError: 3214.1485368562585 not greater than 3500 (retry: 3206.49…)	Log line

Failure Summary

The AMD-CI throughput assertion output_throughput > 3500 tok/s for test_offline_throughput_with_triton_attention_backend (Llama-3.1-8B + --attention-backend triton) fails on linux-mi325-1gpu-sglang. Both the original attempt (3206.49 tok/s) and the single retry (3214.15 tok/s) come in ~92 % of the threshold, leaving the test at exit 1 → step exit 255.

Stack Trace

File "/sglang-checkout/test/registered/perf/test_bench_serving_1gpu_part1.py", line 120,
    in test_offline_throughput_with_triton_attention_backend
        self.assertGreater(res["output_throughput"], 3500)
AssertionError: 3214.1485368562585 not greater than 3500

Regression Status

Never-passed (no green run found in queryable window).

Recent history of test/registered/perf/test_bench_serving_1gpu_part1.py::test_offline_throughput_with_triton_attention_backend in job stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0) (branch=main, event=schedule, completed runs only):

Date	Run	Test File Status	Failed Function	Measured throughput (tok/s)
May 01	25225640477	❌ Failed	test_offline_throughput_with_triton_attention_backend	3206.49 → 3214.15
Apr 30	25180892243	❌ Failed	same	2936.95 → 2942.88
Apr 29	25125273850	❌ Failed	same	3178.43 → 3186.02
Apr 28	25069395852	❌ Failed	same	2874.22 → 2878.15
Apr 27	25010896602	❌ Failed	same	2925.14 → 2920.85
Apr 26	24962999668	❌ Failed	same	2866.50
Apr 25	24936802965	❌ Failed	same	2919.31
Apr 24	24903665661	❌ Failed	same	2872.74
Apr 23	24850489147	❌ Failed	same	3192.17
Apr 23	24793649109	❌ Failed	same	3203.14

Lookback window: last 10 completed schedule runs (Apr 23 → May 01). No green run found. Measured throughput range = 2866 – 3214 tok/s, consistently 8–18 % below the 3500 threshold. The threshold was introduced in the suite migration commit 989d4b301 ([CI] Migrate nightly tests to test/registered/) and has never matched the MI325 reality on this branch+event combination.

Failure Cluster

AMD MI325 triton-attention offline throughput below 3500 tok/s threshold

Facts (verified observations, no interpretation)

• Direct log evidence (line 8562 of job log): AssertionError: 3214.1485368562585 not greater than 3500. Stack trace points to test_bench_serving_1gpu_part1.py:120.
• Test source (test/registered/perf/test_bench_serving_1gpu_part1.py:120): self.assertGreater(res["output_throughput"], 3500) is gated by is_in_amd_ci(); the NVIDIA branch uses the higher 3700 threshold.
• Test config: model meta-llama/Llama-3.1-8B-Instruct, --attention-backend triton, --context-length 8192, request_rate=inf (offline), 500 prompts, recorded 100%|██████████| 500/500 [02:46<00:00, 3.01it/s].
• Retry behaviour: framework retries once (retry() failed once (0th try, maximum 1 retries). Will delay 1.64s and retry), both attempts fail.
• Reproducibility: same test/file/function fails in 10/10 queryable completed schedule runs spanning 9 calendar days; throughput ranges 2866–3214 tok/s, never crossing 3500.
• AITER unchanged across the window: image-shipped v0.1.12.post2.dev150+ga6bb49937 (Dockerfile expects a6bb499375…, dev build preserved). No AITER_COMMIT_OVERRIDE.
• Workflow file at HEAD shows the threshold value is hard-coded; no recent edits to this test file in the failing window (most recent modifying commit on the parent dir is the original migration 989d4b301 plus mechanical refactors that don't change the assertion).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] AMD MI325 triton-attention path consistently runs at ~85–92 % of the 3500 tok/s threshold; the threshold itself is set too high for this hardware/backend combo.
  • Supporting: 10 consecutive failures, all measurements clustered 2866–3214 tok/s, no outliers crossing 3500. Test gates on is_in_amd_ci() separately from NVIDIA, indicating the maintainer intended a different number for AMD; the chosen 3500 simply doesn't match observed steady-state.
  • Disconfirming: Recent run is the highest of the 10 (3214). It's plausible (though not yet observed) that an upcoming optimisation lifts triton-attention throughput across the 3500 line — e.g. open PR #23957 "[AMD] Add CDNA4 Gluon Extend Attention to Triton Attention Backend" advertises faster triton attention for CDNA. If merged and effective on MI325, the threshold could become reachable without lowering it.
  • How to verify: run python3 test/registered/perf/test_bench_serving_1gpu_part1.py on a clean linux-mi325-1gpu-sglang container with HEAD twice; confirm output_throughput stays below 3500. Then confirm with PR #23957 cherry-picked.
• [LOW] Today's HEAD ece8a1a78 ("Refactor device timer, clean up metrics collector, and add fwd occupancy metric" #24197) touches scheduler.py, model_runner.py, metrics_collector.py, cuda_graph_runner.py — code paths exercised by the offline throughput test.
  • Supporting: HEAD diff is non-trivial (700 +/563 -) and includes runtime hot paths.
  • Disconfirming: throughput today (3214) is the highest value in the 10-run window, not lower. The same test failed on the 9 prior days when ece8a1a78 did not exist, so this commit cannot explain the historical pattern.
  • How to verify: not actionable as a regression hypothesis given the disconfirming history.

No hypothesis above MEDIUM confidence; primary hypothesis (threshold too high for MI325 triton path) is the most consistent with the data.

In-flight Fix Check

Searched open PRs with keywords test_bench_serving_1gpu_part1, output_throughput 3500, amd throughput threshold, triton_attention throughput:

• In-flight fix: ❌ none found in open PRs targeting this assertion threshold or test file.
• Adjacent (potentially relevant, but not a direct fix): #23957 "[AMD] Add CDNA4 Gluon Extend Attention to Triton Attention Backend" — could lift MI325 triton attention throughput; would need re-measurement to know if it crosses 3500. #22299 "[AMD] Enable Piecewise CUDA Graph for AMD GPUs" — different code path.

API call: curl -s -H "Authorization: token $GH_PAT" "https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+test_offline_throughput_with_triton" → 0 results.

Suggested Triage Steps

• Reproduce on a fresh linux-mi325-1gpu-sglang container at HEAD: python3 test/registered/perf/test_bench_serving_1gpu_part1.py TestBenchServing1GPUPart1.test_offline_throughput_with_triton_attention_backend and record output_throughput. If 3 consecutive runs all land in the 2900–3300 band, treat the threshold as miscalibrated.
• If maintainer agrees the bar is unrealistic for MI325: open a PR to lower the AMD branch in test_bench_serving_1gpu_part1.py:120 (e.g. assertGreater(res["output_throughput"], 3000)) and back the choice with the 10-run distribution above. Keep the NVIDIA branch (3700) unchanged.
• Alternatively, cherry-pick PR #23957 onto a branch and re-measure; if it crosses 3500 reproducibly, leave the threshold alone and let the PR landing close the gap.
• If the failure is preferred to remain visible (so the day a kernel regression makes it worse, the bar still catches it), no triage step needed beyond documenting Never-passed status; otherwise consider quarantining the assertion behind an explicit @unittest.expectedFailure guarded by is_in_amd_ci().

Status (factual, no priority assignment)

• Persistence: failing for ≥9 days (10 consecutive completed schedule runs, 2026-04-23 → 2026-05-01). Never-passed in the queryable window.
• Scope: 1 job (stage-b-test-1-gpu-large-amd-rocm720 (linux-mi325-1gpu-sglang, 0)) in 1 workflow (pr-test-amd-rocm720.yml); shard 1 of the same matrix is a different test set and is unaffected.
• Blocked work: Blocks the green signal on pr-test-amd-rocm720.yml schedule runs (the workflow has been failing every day for at least 20 consecutive scheduled runs); this single perf assertion is the gating failure for shard 0 specifically.
• In-flight fix: ❌ none found in open PRs.

stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 2) — failed step(s): Run test

Commit Info

• sglang: ece8a1a78
• aiter: a6bb49937 (source: dev image v0.1.12.post2.dev150+ga6bb49937, matches Dockerfile default — no override)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_moriep_small.py	TestMTPwithTBONormal.setUpClass	Exception: Server process exited with code -9. Check server logs for errors. (server crashed during CUDA graph capture with Memory access fault by GPU node-2 ... Reason: Unknown → Fatal Python error: Aborted)	Log

TestMTP.test_gsm8k and TestMTPwithTBOLowLatency.test_gsm8k ran (3 tests total in the suite); the file as a whole ✗ FAILED with exit code 1 after 3044 s.

Failure Summary

The DeepSeek-V3-0324 server launched by TestMTPwithTBONormal.setUpClass with --moe-a2a-backend mori --enable-two-batch-overlap --speculative-algo EAGLE aborted during CUDA-graph capture for batch size 10. The crashing thread is in aiter/fused_moe.py:_moe_sorting_impl, called from sglang/srt/models/deepseek_v2.py:1096 op_experts → batch_overlap/two_batch_overlap.py:885 _model_forward_tbo → cuda_graph_runner.py:884 _capture_one_stream. The launcher then raised Exception: Server process exited with code -9 from popen_launch_server.

Regression Status

Known recurring failure / Never-passed in queryable window — no green run of test_moriep_small.py has been observed in this job since at least 2026-04-02 (the start of the queryable window).

Recent history of test/registered/amd/test_moriep_small.py in job stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 2) (branch main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
May 1	25225640477	job	❌ Failed	TestMTPwithTBONormal.setUpClass	Mem access fault → server -9
Apr 30	25180892243	job	❌ Failed	TestMTPwithTBOLowLatency.setUpClass	(errors=1)
Apr 29	25125273850	job	❌ Failed	TestMTPwithTBONormal.test_gsm8k	(errors=1)
Apr 28	25069395852	job	❌ Failed	TestMTPwithTBONormal.test_gsm8k	(errors=1)
Apr 27	25010896602	job	❌ Failed	TestMTPwithTBONormal.test_gsm8k	(errors=1)
Apr 26	24962999668	job	❌ Failed	TestMTPwithTBONormal.test_gsm8k	Memory access fault on 4 GPU nodes
Apr 14	24414449645	job	❌ Failed	test_gsm8k (line 294)	(errors=1)
Apr 11	24287992722	job	❌ Failed	—	exit code 1
Apr 8	24150164806	job	❌ Failed	—	TIMEOUT after 3600 s
Apr 2	23914009880	job	❌ Failed	—	exit code 1

Lookback window: 2026-04-02 → 2026-05-01 (30 days). The job name was stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2) before the rocm720 suffix was added — both were checked. No passing run of this test file was found. First observed failure date: ≤ 2026-04-02 (predates the queryable window).

Failure Cluster

GPU memory access fault during CUDA graph capture in DeepSeek-V3 + mori a2a + TBO + EAGLE.

Facts (verified observations, no interpretation)

• Server command (verbatim from log): sglang serve --model-path deepseek-ai/DeepSeek-V3-0324 --tp-size 8 --ep-size 8 --dp-size 8 --enable-dp-attention --moe-a2a-backend mori --moe-dense-tp-size 1 --enable-dp-lm-head --mem-fraction-static 0.72 --chunked-prefill-size 32768 --max-running-requests 128 --context-length 12288 --attention-backend aiter --cuda-graph-max-bs 32 --speculative-algo EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --deepep-mode normal --enable-two-batch-overlap --device cuda (log).
• Crash banner: Memory access fault by GPU node-2 (Agent handle: 0x55c36e113ce0) on address (nil). Reason: Unknown. followed by GPU coredump: handler exited with error (status: 1) and Fatal Python error: Aborted (log).
• Crashing-thread stack (top frames, verbatim): aiter/fused_moe.py:50 _moe_sorting_impl → aiter/fused_moe.py:82 moe_sorting → aiter/fused_moe.py:298 fused_moe_ → sglang/srt/layers/moe/ep_moe/layer.py:764 run_moe_core → sglang/srt/models/deepseek_v2.py:1096 op_experts → batch_overlap/operations.py:108 next → batch_overlap/two_batch_overlap.py:885 _model_forward_tbo → model_executor/cuda_graph_runner.py:884 _capture_one_stream (log).
• Crash occurs during the third bs=10 graph capture: Capturing batches (bs=10 avail_mem=14.06 GB): 38%|███▊ | 3/8 [00:18<00:22, 4.54s/it] immediately before the fault.
• Aiter version inside the CI image is the dev wheel that matches Dockerfile default (a6bb49937), no AITER_COMMIT_OVERRIDE. This is the regular nightly path, not an AITER Scout override.
• The earlier two test classes (TestMTP, TestMTPwithTBOLowLatency) reached [CI Test Method] and produced Accuracy: 0.965 / avg_spec_accept_length=2.997 lines, so the runner GPUs and aiter MoE were functional for those configurations.
• Repeated Pyspy failed (py-spy dump --native --pid …). Error: Failed to get stack traces warnings throughout model loading are noise (TP scheduler watchdog probes), not the cause.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] DeepSeek-V3 + --moe-a2a-backend mori + --enable-two-batch-overlap + EAGLE is an AMD-untested config that has never passed CI. The crash always lands in aiter._moe_sorting_impl reached via op_experts from the TBO path; the two earlier test classes in the same file (which differ in either no-TBO or LowLatency mori mode) finish their gsm8k runs successfully on the same image.
  • Supporting: every queryable run since 2026-04-02 fails this file; the consistently-failing setUpClass/test_gsm8k cases are the TBO-Normal and (more recently) TBO-LowLatency variants; earlier non-TBO TestMTP repeatedly succeeds in the same job.
  • Disconfirming: today's failure is TBONormal.setUpClass whereas Apr 30 was TBOLowLatency.setUpClass and Apr 26-29 was TBONormal.test_gsm8k; the failing class rotates run-to-run, so a single static config bug does not fully fit. Could indicate a memory-pressure / non-deterministic kernel issue rather than a pure config bug.
  • How to verify: launch the exact server command above on an mi325-8gpu runner with HSA_ENABLE_SDMA=0 and HIP_LAUNCH_BLOCKING=1; collect a GPU coredump with ROCM_GPU_CORE_DUMP=1; A/B with --moe-a2a-backend deepep (drop mori) and separately with --enable-two-batch-overlap removed.
• [LOW] Memory pressure under --mem-fraction-static 0.72 with EAGLE draft model + TBO buffers leaves too little for graph capture buffers, manifesting as null-pointer dereference in _moe_sorting_impl.
  • Supporting: Capturing batches (bs=10 avail_mem=14.06 GB) shows only 14 GB free at fault; previous in-file tests already loaded model state on the same GPUs.
  • Disconfirming: TestMTP (no TBO) and TestMTPwithTBOLowLatency complete with the same --mem-fraction-static; Apr 26 fault hit on 4 different GPU nodes simultaneously, which is more consistent with a kernel/launch error than an OOM.
  • How to verify: lower --mem-fraction-static to 0.6 and re-run; if it passes, mem pressure is implicated.
• [SPECULATION] Aiter _moe_sorting_impl interaction with TBO double-buffered expert dispatch under EAGLE produces a stale tensor descriptor.
  • Supporting: the literal crashing kernel is _moe_sorting_impl and aiter just printed using torch solution:0 (BF16 GEMM M:20,N:256,K:7168 fell back) for many calls immediately before the fault.
  • Disconfirming: the same aiter wheel handles TestMTP correctly; no aiter override in play, and the failure has spanned multiple aiter dev wheel versions.
  • How to verify: build aiter with AITER_LOG_MORE=1 and re-run; capture sortedExpertIds / expert_offsets tensors prior to crash.

No hypothesis above LOW confidence — narrowing requires a manual reproduction since CI has no green commit window for git bisect.

In-flight Fix Check

Searched: repo:sgl-project/sglang is:pr is:open test_moriep, +mori +moe, +TBO +deepseek, +two_batch_overlap +EAGLE.

• No PR explicitly targets test_moriep_small.py or DeepSeek-V3 + mori + TBO + EAGLE crash.
• Tangentially-related open PRs (NOT confirmed as fixes for this failure): #23760 "Unify DeepEPMoE+MoriEPMoE through AITER MoeRunner pre/post-permute", #22221 "Add Mori a2a backend path for Qwen-moe", #13082 "fix launching failed when --enable-two-batch-overlap".
• In-flight fix: ❌ none confirmed — needs new PR or investigation.

Suggested Triage Steps

• Reproduce locally on mi325-8gpu with the exact server command from the Facts section; collect GPU coredump (ROCM_GPU_CORE_DUMP=1) and rerun under HIP_LAUNCH_BLOCKING=1 to localize the faulting kernel within _moe_sorting_impl.
• A/B isolate the trigger: drop --moe-a2a-backend mori (use deepep) → if passes, mori is implicated; drop --enable-two-batch-overlap → if passes, TBO + mori interaction is implicated; drop --speculative-algo EAGLE → if passes, EAGLE-induced bs is implicated.
• Lower --mem-fraction-static from 0.72 to 0.6 to rule out (or confirm) memory-pressure as a contributing factor during graph capture.
• If the file is intentionally a "never-passed AMD-only port" tracking integration of mori + TBO, consider quarantining TestMTPwithTBONormal / TestMTPwithTBOLowLatency on AMD until _moe_sorting_impl is hardened — the file consumes the full 1-hour timeout slot every scheduled run.
• Ping owners of the mori-a2a integration (#11150) and the TBO refactor; this test was added but has no green baseline in CI.

Status

• Persistence: Failing for ≥ 30 days (since at least 2026-04-02 18:00 UTC, the start of the queryable window). No green run found.
• Scope: 1 test file (test_moriep_small.py) in 1 job (stage-c-test-large-8-gpu-amd-rocm720 (linux-mi325-8gpu-sglang, 2)) of pr-test-amd-rocm720.yml. Not investigated whether the sister pr-test-amd.yml 8-gpu shard 2 also runs this file.
• Blocked work: 1-hour timeout slot consumed every scheduled run; blocks any pass signal for shard 2 of the 8-GPU large-test stage on rocm720.
• In-flight fix: ❌ none confirmed in open PRs — needs new PR or investigation.

stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 1) — failed step(s): Run test

Commit Info

• sglang: ece8a1a78
• aiter: a6bb49937 (source: Dockerfile default; image carries dev tag v0.1.12.post2.dev150+ga6bb49937, preserved per [CI-AITER-CHECK] "Dev/patched version detected: ... → skipping rebuild")

Failed Tests

Test File	Test Function	Error	Log
test/registered/distributed/test_data_parallelism.py	TestDataParallelism.test_gsm8k	HW Exception by GPU node-2 reason :GPU Hang → Fatal Python error: Aborted (exit 134) on both attempts	Log

Failure Summary

TestDataParallelism.test_gsm8k (Llama-3.1-8B-Instruct, TP=1 DP=2) crashed mid-prefill on the 2-GPU MI325 ROCm 7.2.0 runner. Multiple GPU agents reported GPU Hang, the watchdog (python/sglang/srt/utils/watchdog.py:147) aborted with Fatal Python error: Aborted, and the test harness retried once with HF_HUB_OFFLINE=1; the second attempt failed identically (process exit code 134). test_get_memory_pool_size ran first and succeeded; the hang occurs once the gsm8k workload starts driving real prefill batches across DP0/DP1.

Regression Status

New regression — first observed today (2026-05-01).

History of test/registered/distributed/test_data_parallelism.py (test_gsm8k) in job stage-b-test-2-gpu-large-amd-rocm720 (linux-mi325-2gpu-sglang, 1) (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
May 1	25225640477	job	❌ Failed (both attempts)	test_gsm8k	GPU Hang → exit 134
Apr 30	25180892243	job	✅ Passed	—	—
Apr 29	25125273850	job	✅ Passed	—	—
Apr 28	25069395852	job	✅ Passed	—	—
Apr 27	25010896602	job	✅ Passed	—	—

First observed failure: 2026-05-01 18:10 UTC. Last passing run: 2026-04-30 18:34 UTC.

Failure Cluster

GPU Hang during DP+TP prefill on ROCm720 (Llama-3.1-8B / gsm8k)

Facts (verified observations, no interpretation)

• Verbatim crash sequence (job log line 6888):

  HW Exception by GPU node-2 (Agent handle: 0x55dd4cbb9ad0) reason :GPU Hang
  Fatal Python error: Aborted
  Thread 0x...: File "/sglang-checkout/python/sglang/srt/utils/watchdog.py", line 147 in _watchdog_once
  Thread 0x...: File "/sglang-checkout/python/sglang/srt/managers/data_parallel_controller.py", line 300 in launch_tensor_parallel_group_thread
  

  Log: step 18, line 6888.
• Crash happens during prefill — last log lines before hang are normal prefill batches cuda graph: False on DP0/DP1 with aiter ... using torch solution:0 GEMM logs (line 6886).
• Both attempts (HF online + HF_HUB_OFFLINE=1 retry at line 7025) crash with the same symptom; "First attempt failed with exit code 134" → second attempt also dies.
• Two other completed 2-gpu-large-amd-rocm720 (..., 0) and 1-gpu-small-amd-rocm720 jobs in this same run also failed (job IDs 73968459255, 73968459315, 73968459437) — the failure is not isolated to this single shard.
• Aiter version diff (the only environment-level change vs the last green run):
  • Apr 30 (passing): [CI-AITER-CHECK] AITER version inside CI image: v0.1.12.post1 (line 2180 of yesterday's log).
  • May 1 (failing): [CI-AITER-CHECK] AITER version inside CI image: v0.1.12.post2.dev150+ga6bb49937 (line 2189), driven by Dockerfile bump AITER_COMMIT_DEFAULT="a6bb499..." landed in #24218 (7e17a7a1e, Apr 30 22:26 PT).
• sglang delta between 7bb7f6049a (last passing) and ece8a1a78 (failing) = 39 commits.
• aiter delta v0.1.12.post1...a6bb499 = 150 commits, including d098ae5a91 "Revert 'fix(car): graph capture err (#2638)'" (#2735) — the same revert previously implicated in AITER Scout [CI Monitor] Daily Report - 2026-06-03 #92 (per CLAUDE.md, that attribution turned out to be wrong).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] aiter bump in #24218 (v0.1.12.post1 → a6bb49937) introduced a regression that hangs the GPU during BF16 GEMM/prefill in 2-GPU DP on ROCm720.

  • Supporting: temporal correlation is exact — last passing run had v0.1.12.post1, first failing run has dev150+ga6bb49937; aiter is the only environment variable that changed; failure is on ROCm only; crash log lines immediately preceding the hang are aiter GEMM dispatches ([aiter] shape is M:..., N:..., K:... using torch solution:0); 150 aiter commits in the delta give a large surface for new bugs; and the 1-gpu shards in the same run also broke today, consistent with a global aiter-side regression rather than a DP-specific one.
  • Disconfirming: log-line proximity is not causation (CLAUDE.md AITER Scout [CI Monitor] Daily Report - 2026-06-03 #92 lesson) — [aiter] ... using torch solution:0 is just the previous log line and does not prove the crash is inside aiter; this is the regular nightly workflow, not a Scout, so there is no sister-baseline A/B run with the old aiter to confirm; the same revert blamed in Scout [CI Monitor] Daily Report - 2026-06-03 #92 (d098ae5a91) is in this delta and turned out then to be a red herring.
  • How to verify: temporarily revert #24218 (or set AITER_COMMIT_OVERRIDE=v0.1.12.post1 in the workflow) and re-run stage-b-test-2-gpu-large-amd-rocm720 on linux-mi325-2gpu-sglang. If the test passes, aiter is causal; if it still hangs, look at the 39-commit sglang delta (top suspect: ece8a1a78 #24197 model_runner / scheduler refactor).

• [LOW] sglang ece8a1a78 "Refactor device timer, clean up metrics collector, and add fwd occupancy metric" (#24197) — touches model_runner.py (197 lines), scheduler.py (59), cuda_graph_runner.py (20), and metrics_collector.py (637) on Apr 30/May 1.

  • Supporting: lands in regression window; touches the per-iteration forward-pass instrumentation around the prefill code path that crashed.
  • Disconfirming: the failing test runs with cuda graph: False on prefill, so the cuda_graph_runner edits are not exercised; the crash signature is HW Exception ... GPU Hang, which is a kernel/runtime-level symptom not typically caused by Python-level metrics refactors; same job's 1-GPU non-DP shards (which exercise the same scheduler) failed too, so a metrics/scheduler bug would have to manifest in many independent paths simultaneously.
  • How to verify: only investigate after the aiter revert/override A/B above; if aiter is exonerated, bisect the 39-commit window with this commit at one end.

• [LOW] sglang 108bfd8b6 "[MoE] Add Aiter MoE runner backend and purge aiter.fused_moe from quant methods" (#23597).

  • Supporting: changes how sglang dispatches into aiter for quant/MoE paths, so an aiter-version interaction is plausible.
  • Disconfirming: Llama-3.1-8B-Instruct is dense (no MoE), so the MoE runner code path should not be hit by test_gsm8k.
  • How to verify: not actionable until aiter is ruled in/out.

No hypothesis above MEDIUM confidence; the right next step is the aiter A/B above.

In-flight Fix Check

Searched open sglang PRs for test_data_parallelism, aiter v0.1.12, aiter rocm, revert aiter, gpu hang rocm. No PR matches the symptom (GPU Hang in test_data_parallelism after the #24218 bump).

• In-flight fix: ❌ none found in open PRs — needs new investigation (aiter A/B revert), or a follow-up PR if [#24218](https://github.com/sgl-project/sglang/pull/24218) is implicated.

Suggested Triage Steps

• Re-run stage-b-test-2-gpu-large-amd-rocm720 with AITER_COMMIT_OVERRIDE=v0.1.12.post1 (Scout-style override) on the same sglang HEAD ece8a1a78 — this is the cheapest A/B for the leading hypothesis. If the test passes, the regression is in the aiter delta v0.1.12.post1...a6bb499 and triage shifts to aiter (consider bisecting aiter, starting with d098ae5a91 and the GEMM/MoE commits).
• Cross-check the other failing rocm720 jobs in run 25225640477 (1-gpu-large, 1-gpu-small-mi35x, 1-gpu-small) for the same GPU Hang cluster — if all share the symptom, the aiter hypothesis gets stronger evidence and the fix scope is fleet-wide rather than DP-specific.
• If the aiter override A/B exonerates aiter, bisect the 39-commit sglang window 7bb7f6049a..ece8a1a78, starting with ece8a1a78 (#24197) given the breadth of model_runner.py / scheduler edits.
• File a follow-up issue/PR linking #24218 once the A/B is conclusive (revert vs. pin a known-good aiter commit).

Status

• Persistence: Failing for ~1 day (since 2026-05-01 18:10 UTC), 1 completed run since the regression landed. Both retry attempts crashed in the same place.
• Scope: 4 failed jobs in this run on ROCm720 (stage-b-test-{1,2}-gpu-...-rocm720 shards across MI325 + MI35X) — failure is not test-specific or shard-specific.
• Blocked work: ROCm720 nightly signal — pr-test-amd-rocm720.yml is fully red on the 2-GPU large stage; downstream gating off this workflow blocked.
• In-flight fix: ❌ none found in open PRs — needs new investigation (aiter override A/B is the cheapest first step).

stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 12) — failed step(s): Run test

Commit Info

• sglang: ece8a1a78
• aiter: a6bb49937 (source: dev image — v0.1.12.post2.dev150+ga6bb49937, matches Dockerfile expectation)

Failed Tests

Test File	Test Function	Error	Log
test/registered/lora/test_lora_overlap_loading.py	TestLoRAOverlapLoading.test_ci_lora_models_batch_splitting (in-progress when killed)	step-level timeout: ##[error]The action 'Run test' has timed out after 30 minutes.	Log L16901

This is not a test assertion failure — the GitHub step timed out at 30 min while test 5 of 7 was still running (it had been executing for ~1m 42s when the kill arrived).

Failure Summary

The Run test step in partition 12 of stage-b-test-1-gpu-small-amd-rocm720 exceeded its 30-minute step budget. Tests 0-4 completed (with elapsed times 474s, 550s, 299s, 162s, 214s — totalling ~28.3 min on their own), leaving < 2 min for the remaining 3 tests; the kill landed during test 5 (test_lora_overlap_loading.py). The job's 7-test workload is identical to yesterday's passing run (same partition, same files).

Regression Status

Flaky / borderline timeout, not a fresh regression of any single test.

Recent history of partition 12 in workflow pr-test-amd-rocm720.yml, job stage-b-test-1-gpu-small-amd-rocm720 (linux-mi325-1gpu-sglang, 12) (branch=main, event=schedule, completed runs only):

Date	Run	Job Status	Run test step duration	Note
2026-05-01	25225640477	❌ failure	30.2 min (timed out)	runner 9j59m; test_lora_overlap_loading.py killed mid-run
2026-04-30	25180892243	✅ success	21.8 min	runner kzlmj; same 7 files passed
2026-04-29	25125273850	✅ success	n/a	partition 12 passed
2026-04-28	25069395852	❌ failure	n/a	unrelated — test_llada2_mini.py exit 1 (different test, different symptom)
2026-04-27	25010896602	✅ success	n/a	partition 12 passed
2026-04-26	24962999668	✅ success	n/a	partition 12 passed

test_lora_overlap_loading.py itself completed in 70 s (well under its 75 s estimate) yesterday and has no known failures — it is the workload-tail victim, not a culprit.

Failure Cluster

Stage-b partition-12 step timeout (30 min) — borderline workload + slow runner

Facts (verified observations)

• Step error verbatim (line 16901): ##[error]The action 'Run test' has timed out after 30 minutes. (Log).

• Per-test elapsed times today vs yesterday (same 7 files, same partition):

  #	File	Today (s)	Yesterday (s)	Δ
  0	test_llada2_mini.py	474	316	+50%
  1	test_scheduler_control.py	550	372	+48%
  2	test_openai_server.py	299	216	+38%
  3	test_constrained_decoding.py	162	130	+25%
  4	test_lora_load_from_tensor.py	214	180	+19%
  5	test_lora_overlap_loading.py	killed	70	—

• Run test step duration today across all 14 partitions of the same workflow (Run 25225640477): partitions 7 (-11.3 min), 10 (-9.1 min), 8 (-3.8 min), 9 (-2.9 min) are faster than yesterday; partition 12 is +8.2 min slower; others are within ±3 min. So the slowdown is NOT a global compute regression.

• Yesterday partition 10 already needed 30.2 min for its Run test step (Run 25180892243) — i.e. the 30-min timeout is already routinely hit-or-miss in this suite.

• Stale FILE_TIMES: test_lora_load_from_tensor.py is estimated at 90 s but consistently runs 180-214 s (verified across both runs), inflating partition 12's planned budget.

• Image / aiter delta between yesterday and today (factual, but see disconfirming evidence below):

  • Image: rocm/sgl-dev:v0.5.10.post1-rocm720-mi30x-20260430 → …-20260501
  • AITER: v0.1.12.post1 → a6bb49937 (150 commits delta), bumped by sglang 7e17a7a1e (PR #24218) merged 2026-05-01 01:26 PDT.

• Runner today: linux-mi325-1gpu-sglang-xbmw2-runner-9j59m; yesterday: …-kzlmj.

Hypothesised Causes

• [MEDIUM] Noisy / slow runner instance + already-borderline 30 min budget for partition 12.
  • Supporting: yesterday partition 12 used 21.8 / 30 min (margin only 8.2 min); other partitions today (7, 10) are faster than yesterday on the same image+aiter, ruling out a global regression; partition 10 already hit 30.2 min yesterday on the prior image. The pattern (partition 12 slowdown, others mixed) matches per-runner variance, not a code-wide change.
  • Disconfirming: every one of the 5 completed tests in partition 12 today is 19-50% slower than yesterday — a 5-for-5 pattern is unusual for pure runner noise; could also reflect a small model-loading / memory-bandwidth perf delta on this specific runner.
  • How to verify: rerun partition 12 alone once or twice; if it lands in 22-25 min on a different runner, runner-noise hypothesis is confirmed.
• [LOW] AITER bump in 7e17a7a1e (PR #24218) introduced a small global perf regression.
  • Supporting: AITER moved v0.1.12.post1 → a6bb49937 (150 commits, many touching gemm / moe / mha tuning); this bump landed in the exact run that failed.
  • Disconfirming: 9 of 14 sister partitions (e.g. partition 7: 26.9→15.6 min, partition 10: 30.2→21.1 min) are faster under the new AITER. A global AITER regression would not show net speedups elsewhere.
  • How to verify: re-trigger the workflow with AITER_COMMIT_OVERRIDE=v0.1.12.post1 and check whether partition 12's Run test drops below 28 min on a representative runner.
• [LOW] Stale partition FILE_TIMES (specifically test_lora_load_from_tensor.py estimated 90 s vs actual 180-214 s) put partition 12 too close to the 30 min budget.
  • Supporting: estimate is ~2x off, consistent across both runs; auto-partition planning trusts these estimates.
  • Disconfirming: this same imbalance existed yesterday and the job still passed, so the imbalance alone isn't sufficient — it only matters in combination with runner noise.
  • How to verify: re-run with updated estimated_time for test_lora_load_from_tensor.py (200 s) and test_scheduler_control.py (550 s observed).

No hypothesis above MEDIUM confidence; the failure is most likely runner-noise on a chronically borderline partition rather than a code regression.

In-flight Fix Check

• Searched: repo:sgl-project/sglang is:pr is:open for stage-b-test+timeout, test_lora_overlap_loading, partition+timeout+amd, auto-partition+amd.
• In-flight fix: ❌ none found — needs FILE_TIMES refresh and/or step-timeout bump for stage-b-test-1-gpu-small-amd[-rocm720].

Suggested Triage Steps

• Re-run Run 25225640477 just for partition 12; if it passes on a different runner in < 28 min, this is runner-noise — close as flaky.
• Refresh FILE_TIMES for the AMD stage-b suite (in particular bump test_lora_load_from_tensor.py from ~90 s to ~210 s, and test_scheduler_control.py from 300 s toward 500 s) so the auto-partitioner stops over-packing partition 12.
• Consider raising the per-step timeout for stage-b-test-1-gpu-small-amd-rocm720 from 30 min to 35-40 min (yesterday partition 10 already used 30.2 min — multiple partitions are routinely within 1-2 min of the limit).
• A/B AITER: re-trigger the workflow with AITER_COMMIT_OVERRIDE=v0.1.12.post1 once, compare partition 12's Run test step duration. Only escalate the AITER hypothesis if the rerun is materially faster than today.
• If partition 12 times out twice more in the next 2-3 scheduled runs on different runners, escalate from "flaky runner" to "real perf regression in the AITER bump" and bisect within the 150-commit aiter delta.

Status

• Persistence: 1 failed run (today, 2026-05-01); previous two scheduled runs (4/30, 4/29) were green for this partition. Not yet a multi-day pattern.
• Scope: 1 job (partition 12) in 1 workflow today. Other 13 partitions of the same job/run are green.
• Blocked work: scheduled CI signal for this partition's tests on AMD MI325 ROCm 7.2 (LoRA, llada2, scheduler-control, openai-server, constrained-decoding, evs, swa unittest).
• In-flight fix: ❌ none found — needs FILE_TIMES refresh and/or step-timeout bump.

stage-b-test-large-8-gpu-35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric) — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: e0474fdd
• aiter: N/A (job died at ensure_vram_clear.sh, before docker container start; no [CI-AITER-CHECK] markers in log)

Failed Tests

Test File	Test Function	Error	Log
N/A (no test executed)	N/A	Pre-job step ensure_vram_clear.sh exited with code 1: VRAM cleanup unsuccessful after 3 attempts	step 4 line 372

The job failed at the runner-cleanup pre-check (bash scripts/ci/amd/ensure_vram_clear.sh rocm); no python -m pytest or sglang server ever launched.

Failure Summary

On runner host linux-mi35x-gpu-8.fabric-kbhnc-runner-kh4bf, all 8 MI35x GPUs reported 88-89% VRAM already allocated at job start. Three sequential cleanup attempts (kill sglang processes → rocm-smi --showpids based PID kill + 30 s wait) failed to release any memory, and post-mortem inspection showed No processes found using /dev/kfd — i.e. nobody owns the allocated memory. The job exited 1 before docker container start, before aiter install, and before any test could run.

Regression Status

New runner infrastructure failure (not a code regression for this test).

Recent history of stage-b-test-large-8-gpu-35x-disaggregation-amd-rocm720 (linux-mi35x-gpu-8.fabric) on pr-test-amd-rocm720.yml, branch main, event=schedule, completed runs only:

Date	Run	Job	Test File Status	Failed Function	Error
May 02	25257951974	stage-b…disaggregation	❌ Failed	N/A (pre-job VRAM cleanup)	Cleanup unsuccessful after 3 attempts
May 01	25225640477	stage-b…disaggregation	✅ Passed	—	—
Apr 30	25180892243	stage-b…disaggregation	✅ Passed	—	—
Apr 29	25125273850	stage-b…disaggregation	✅ Passed	—	—

The same job on the same fabric runner passed in the previous three scheduled runs. This is a runner-state failure on May 02, not a test regression. (The parent run 25257951974 is still queued overall — sister jobs stage-c-test-large-8-gpu-amd-mi35x-rocm720 shards 0/1 are still queued at snapshot time; trend data for them is [IN-FLIGHT].)

Failure Cluster

Pre-job VRAM cleanup failed (orphaned VRAM with no owning process on AMD MI35x runner)

Facts (verified observations, no interpretation)

• Step that failed: Run bash scripts/ci/amd/ensure_vram_clear.sh rocm — the second runner step, immediately after actions/checkout@v4 (log line 109-372).
• Initial VRAM (verbatim from log): GPU[0]=88%, GPU[1]=89%, GPU[2]=89%, GPU[3]=89%, GPU[4]=89%, GPU[5]=88%, GPU[6]=88%, GPU[7]=88% (log line 132-163).
• VRAM after attempt 1, 2, 3: identical 88-89% allocation on all 8 GPUs — no memory was reclaimed by any cleanup step (log line 226-287).
• No owning process: pgrep -f 'sglang::|sglang.launch_server|...' returned nothing (no output between "Killing SGLang processes..." and the next step). Post-mortem No processes found using /dev/kfd and No processes found accessing /dev/kfd confirm no userspace process owns the VRAM (log line 367-371).
• rocm-smi --showpids during aggressive cleanup also produced no PIDs to kill (no kill targets logged before each 30s wait).
• Container state: docker stop ci_sglang and docker rm ci_sglang both returned Error response from daemon: No such container: ci_sglang (log line 117-119) — no leftover sglang container is responsible.
• Runner identity: linux-mi35x-gpu-8.fabric-kbhnc-runner-kh4bf (log line 2) — only one runner of its kind in this matrix; the same machine ran the previous 3 successful daily disaggregation jobs.
• No tests executed, no aiter install: log ends at the cleanup-failure exit (line 372) and never reaches docker pull, container start, dependency install, or pytest invocation. No [CI-AITER-CHECK] markers exist.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Orphaned GPU memory left by a prior workload on the same runner host that crashed/was force-killed without releasing VRAM (e.g. an OOM, segfault, or external kill -9 of a previous container). Symptoms match exactly: VRAM is allocated but no userspace process owns it (no PIDs in /dev/kfd, no PIDs in rocm-smi --showpids, no ci_sglang container left), so the userland kill -9 based cleanup script has nothing to kill. Memory will only be reclaimed by amdgpu driver re-init (e.g. host reboot or module reload), neither of which the script attempts.
  • Supporting: identical 88-89% allocation across all 8 GPUs is consistent with a multi-GPU TP=8 / disaggregation workload that died mid-run; the prior scheduled run on this runner (pr-test-amd-rocm720 on May 01) ran the same disaggregation job, and the host has no obvious tenant between scheduled runs.
  • Disconfirming: I have not verified what workload actually ran on this physical host between May 01 17:47 UTC and May 02 17:45 UTC. (External non-CI tenants or a non-sglang workload could have been the culprit; the bot does not have host-level access.)
  • How to verify: SSH to linux-mi35x-gpu-8.fabric host, check dmesg | grep -i amdgpu for fault entries near 2026-05-02 17:00 UTC, check container/podman/docker journalctl for OOMs or kills, and inspect the runner's job history (which workflow last finished here before 17:45 UTC) — likely the previous PR Test (AMD ROCm 7.2) run 25251804407 at 12:22 UTC.
• [LOW] ensure_vram_clear.sh cleanup logic is insufficient: when no process owns the allocated VRAM (orphaned memory in driver state), pgrep/rocm-smi --showpids/kill -9 are no-ops by design.
  • Supporting: the script only attempts process-level kills; it never tries rocm-smi --gpureset, modprobe -r amdgpu && modprobe amdgpu, systemctl restart kubelet, or marking the runner offline. The script's own log shows it found no processes to kill across 3 attempts.
  • Disconfirming: the script is unchanged across the 4-day window where the same job passed 3× and failed 1× (scripts/ci/amd/ensure_vram_clear.sh — git log -- scripts/ci/amd/ensure_vram_clear.sh shows no recent edits in this window). This means the script logic is not the trigger; it is exposed only when the host arrives in a bad state.
  • How to verify: re-run the same job on a freshly rebooted host — if it passes, the cleanup gap matters only when the host arrives dirty.
• [SPECULATION] Hardware/firmware issue on this specific MI35x card or fabric link causing VRAM accounting to be stuck in rocm-smi's view even after free.
  • Supporting: none beyond the symptom; no mention of a known MI35x firmware bug in the log.
  • Disconfirming: no kernel fault, MCE, or amdgpu ECC error appears in the visible log fragment; this is purely speculative.
  • How to verify: check host dmesg and rocm-smi --showvbios / rocm-smi --showserial against AMD's known-bad lots; not bot-actionable.

The aiter override is empty (AITER_COMMIT_OVERRIDE: blank, line 33), and no aiter commit was ever installed. No aiter hypothesis applies to this failure.

In-flight Fix Check

• In-flight fix: ❌ none found — repo:sgl-project/sglang+is:pr+is:open+ensure_vram_clear returned 0 results; …+is:open+VRAM+cleanup returned 3 PRs (#22048, #19412, #13553) but none address runner VRAM-cleanup tooling — they are quantization/grammar/spec-decoding feature PRs that incidentally mention the words. No PR is in flight to harden ensure_vram_clear.sh or to add a host-reboot escalation.

Suggested Triage Steps

• Reboot or recycle the linux-mi35x-gpu-8.fabric runner host (or mark it offline and reimage) — orphaned VRAM with no owning process typically requires amdgpu driver re-init or a host reboot. After recycling, retry the workflow run 25257951974 and confirm the disaggregation job passes.
• On the host, run dmesg | grep -i -E 'amdgpu|kfd|hsa|page fault' for the window 2026-05-02 12:00-17:45 UTC; identify the prior workload that left VRAM allocated. If a previous CI job is the culprit, file a separate ticket against that test's teardown path.
• Inspect the runner's job log between 25225640477 (last passing daily run) and 25257951974 (today's failure) to see what other workflows shared this host (e.g. PR Test (AMD ROCm 7.2) 25251804407 at 12:22 UTC — the most recent failing AMD workflow before today's schedule trigger).
• If orphaned-VRAM incidents become recurrent, consider hardening scripts/ci/amd/ensure_vram_clear.sh to escalate beyond process kills — e.g. add a final attempt that issues rocm-smi --gpureset (or marks the runner offline so the orchestrator sends jobs elsewhere). This is a tooling change, not a directive.
• If hypothesis 1 cannot be verified (no host access available), retry the workflow once and watch whether the failure recurs on the next scheduled run; a single-occurrence orphaned-VRAM event is consistent with a one-off prior crash.

Status

• Persistence: Single occurrence on 2026-05-02 17:45 UTC; the same job on the same runner passed on May 01, Apr 30, and Apr 29.
• Scope: 1 job in 1 workflow (pr-test-amd-rocm720.yml). The two sister jobs in run 25257951974 (stage-c-test-large-8-gpu-amd-mi35x-rocm720 shards 0/1) are still queued at snapshot time and have not yet executed on this runner — [IN-FLIGHT], may be affected if scheduled to the same dirty host.
• Blocked work: blocks today's 8-GPU disaggregation signal on AMD MI35x ROCm 7.2.
• In-flight fix: ❌ none found in open PRs.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-05-02 20:27 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — 22 failure(s)

Run started (UTC): 2026-05-01 17:46 UTC
Last scanned (UTC): 2026-05-02 20:25 UTC
Commits: sglang e0474fd

Summary

Counts: 22 failures · 10 clusters · 4 carrying over (≥3 days) · 6 new today

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	TP memory imbalance on MI35x runners	nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8-rocm720	test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_kv_fp8_eval_mi35x.py	test_deepseek_r1_mxfp4_kv_fp8_accuracy	3 days persistent	[HIGH] runner-pool stale GPU memory on linux-mi35x-gpu-8-4wrmw-*
2	TP memory imbalance on MI35x runners	nightly-accuracy-8-gpu-mi35x-rocm720	test/registered/amd/accuracy/mi35x/test_gpt_oss_eval_mi35x.py	test_gpt_oss_accuracy	2 days persistent	[MEDIUM] runner leftover process holds ~2 GB on one rank
3	TP memory imbalance on MI35x runners	nightly-accuracy-8-gpu-mi35x-rocm720	test/registered/amd/accuracy/mi35x/test_gpt_oss_eval_mi35x.py	test_gpt_oss_accuracy	2 days persistent	[MEDIUM] runner-pool contention on linux-mi35x-gpu-8
4	TP memory imbalance on MI35x runners	nightly-8-gpu-mi35x-qwen35-rocm720	test/registered/amd/accuracy/mi35x/test_qwen35_eval_mi35x.py	TestQwen35EvalMI35x.test_lm_eval	2 days persistent	[LOW] pre-existing leaked ~1.7 GB on rank 1
5	TP memory imbalance on MI35x runners	nightly-8-gpu-mi35x-glm51-rocm720	test/registered/amd/accuracy/mi35x/test_glm51_eval_mi35x.py	TestGLM51EvalMI35x.test_glm51_accuracy	1 day (2 runs)	[MEDIUM] runner leftover state on MI35x
6	TP memory imbalance on MI35x runners	nightly-8-gpu-mi35x-glm51-rocm720	test/registered/amd/accuracy/mi35x/test_glm51_eval_mi35x.py	test_glm51_accuracy	1 day (2 runs)	[HIGH] MI35x runner GPU contamination ~240 GB
7	TP memory imbalance on MI35x runners	nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion-rocm720	test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_ar_fusion_eval_mi35x.py	test_deepseek_r1_mxfp4_ar_fusion_accuracy	new today	[HIGH] stale GPU mem on rank 1 of MI35x runner
8	DeepSeek-V3.2 NSA tilelang CUDA-graph memory fault	nightly-perf-8-gpu-mi35x-deepseek-v32-basic-rocm720	test/registered/amd/perf/mi35x/test_deepseek_v32_basic_perf_mi35x.py	test_bench_one_batch	2 days persistent	[MEDIUM] aiter bump #24218 (7e17a7a1) → gemm_a8w8_blockscale_bpreshuffle
9	DeepSeek-V3.2 NSA tilelang CUDA-graph memory fault	nightly-perf-8-gpu-mi35x-deepseek-v32-basic-rocm720	test/registered/amd/perf/mi35x/test_deepseek_v32_basic_perf_mi35x.py	test_bench_one_batch	2 days persistent	[MEDIUM] commit ece8a1a78 (#24197) device_timer refactor — capture-path rewrite
10	DeepSeek-V3.2 NSA tilelang CUDA-graph memory fault	nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	2 days persistent	[MEDIUM] aiter bump #24218
11	DeepSeek-V3.2 NSA tilelang CUDA-graph memory fault	nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	2 days persistent	[MEDIUM] aiter bump #24218
12	Mistral/Mixtral GSM8K accuracy below threshold (AMD MI30x)	nightly-accuracy-2-gpu-rocm720	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models	Never-passed (≥24 days, since 2026-04-08)	[HIGH] thresholds too aggressive after mgsm_en→gsm8k migration (f6fc39569a, #21931)
13	Mistral/Mixtral GSM8K accuracy below threshold (AMD MI30x)	nightly-accuracy-2-gpu-vlm-rocm720	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	TestNightlyGsm8KEval.test_gsm8k_all_models	Never-passed (≥11 days)	[LOW] aiter MoE backend regression suspected; no green run for bisection
14	Mistral/Mixtral GSM8K accuracy below threshold (AMD MI30x)	nightly-accuracy-2-gpu-rocm720	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models	Never-passed (≥3 weeks)	[LOW] pre-existing AMD MI325 ROCm 7.2 regression; no green in window
15	AITER custom all-reduce HIP graph-capture invalidation (hipErrorCapturedEvent)	nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	21 days persistent (since 2026-04-11)	[HIGH] ROCm 7.2.0 hipEventQuery cross-thread race + AITER AR helper streams
16	AITER custom all-reduce HIP graph-capture invalidation (hipErrorCapturedEvent)	nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	test/registered/amd/test_qwen3_instruct_mxfp4.py	setUpClass (TestQwen3Instruct2507MXFP4)	22 days persistent	[HIGH] ROCm 7.2.0 hipEventQuery race + AITER IPCBufferPool (v0.1.12.post1+)
17	VLM MMMU accuracy below threshold on AMD MI30x	nightly-accuracy-2-gpu-vlm-rocm720	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	TestNightlyVLMMmmuEvalAMD.test_mmmu_vlm_models	≥7 days persistent	[MEDIUM] per-model thresholds calibrated above long-running AMD MI30x measurement
18	MiniMax-M2.7 GSM8K accuracy borderline (0.925 vs 0.93)	nightly-8-gpu-minimax-m27-rocm720	test/registered/amd/accuracy/mi30x/test_minimax_m27_eval_amd.py	TestMiniMaxM27EvalAMD.test_minimax_m27_accuracy	new today (1/7 runs)	[MEDIUM] sampling noise on tight threshold (test design)
19	Self-hosted runner lost communication mid-perf	nightly-8-gpu-glm51-rocm720	N/A (runner died during GLM-5.1 perf step)	N/A	new today (1/5 runs)	[MEDIUM] runner pod OOM-killed/evicted during perf workload
20	HF Hub processor download disconnect (VLM perf warm-up)	nightly-perf-2-gpu-vlm-rocm720	test/registered/amd/perf/mi30x/test_vlms_perf_amd.py	test_bench_one_batch (Qwen2.5-VL-7B)	new today (1/6 runs)	[HIGH] transient HF Hub RemoteProtocolError during AutoProcessor.from_pretrained
21	HF Hub xet_get download stall (Grok-1 weights)	nightly-8-gpu-grok1-int4-rocm720	test/registered/amd/accuracy/mi30x/test_grok1_int4_eval_amd.py	N/A (60-min action timeout before test phase)	new today (1/4 runs)	[MEDIUM] HF/CDN slowdown during 150 GB sharded download
22	VLM test_encoder_dp MMMU benchmark timeout (intermittent)	nightly-4-gpu-rocm720	test/registered/vlm/test_encoder_dp.py	TestVLMEncoderDP.test_vlm_mmmu_benchmark	2/5 runs (intermittent)	[MEDIUM] 3600s budget too tight for 4-model VLM sweep on MI325

Cluster: TP memory imbalance on MI35x runners

• Affected: 7 jobs across both runs (25225581444, 25257923871) — rows 1–7 above. All MI35x 8-GPU jobs sharing the linux-mi35x-gpu-8-4wrmw-* runner pool.
• Shared facts: identical error wording (pre_model_load_memory=6.738, local_gpu_memory=8.441) on multiple distinct runners; only ~3% HBM free at process start; failure raised inside model_runner.py:~1231 init_torch_distributed TP-skew check.
• Top hypothesis: [HIGH] shared runner-pool contamination — leftover/stale GPU memory on MI35x 8-GPU pool. Disconfirming: smaller-model jobs in the same pool sometimes pass (runner-specific, not pool-wide); the asymmetry direction (TP1 having more free than peers in some cases) is opposite of typical leftover-process pattern.
• In-flight fix: ❌ none found — runner-side investigation needed.
• Suggested triage: add rocm-smi --showpids; rocm-smi --showmeminfo vram as an early step in nightly-test-amd-rocm720.yml; SSH into pool runners while idle to identify orphan PIDs; consider re-running on different runner labels.

Cluster: DeepSeek-V3.2 NSA tilelang CUDA-graph memory fault

• Affected: rows 8–11 — nightly-perf-8-gpu-mi35x-deepseek-v32-basic-rocm720 and nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720, both runs. Sister *-deepseek-v32-mtp-rocm720 PASSES on the same SHA + aiter, isolating to the basic (non-MTP) NSA tilelang config.
• Shared facts: 8× simultaneous Memory access fault by GPU node-{2..9} during HIP graph capture; preceding log line is aiter GEMM using torch solution:0 (untuned bf16 fallback); first observed 2026-05-01 18:58 UTC (the run after #24218 merged Apr 30 23:26 UTC); previous 5 scheduled runs green.
• Top hypothesis: [MEDIUM] aiter bump in #24218 (7e17a7a1, v0.1.12.post1 → a6bb499) introduces gemm_a8w8_blockscale_bpreshuffle regression for NSA-indexer FP8 shapes on gfx950. Disconfirming: log-line proximity is not causation; sister MTP variant works with same aiter — could be sglang capture-path commit ece8a1a78 (#24197) instead.
• In-flight fix: ❌ none found.
• Suggested triage: re-run with AITER_COMMIT_OVERRIDE=v0.1.12.post1 on the same sglang SHA; if green, aiter-caused → bisect aiter delta. If still red, revert ece8a1a78 on a branch and retest.

Cluster: Mistral/Mixtral GSM8K accuracy below threshold (AMD MI30x)

• Affected: rows 12–14 — nightly-accuracy-2-gpu-rocm720 (both runs) plus the vlm shard once due to partition collision; same test_gsm8k_eval_amd.py::test_gsm8k_all_models.
• Shared facts: 5 models consistently fail (Mistral-7B/FP8, Mixtral-8x7B/FP8, Llama-3.1-8B-FP8 marginal); scores stable run-over-run (~0.18–0.37) far below 0.46–0.69 thresholds; Never-passed since gsm8k migration #21931 on 2026-04-07.
• Top hypothesis: [HIGH] thresholds set in f6fc39569a as reported - 5% are too aggressive for AMD MI30x; diff itself comments "lower if AMD scores differently". Disconfirming: ~0.30 on Mistral-7B is below published HF baselines (~0.52), suggesting a real serving/correctness bug specific to chat-template + few-shot prompts on AMD rather than just an offset.
• In-flight fix: ❌ none found.
• Suggested triage: cross-check with NVIDIA baseline at same sglang SHA + harness; if NVIDIA also ~0.30, lower thresholds; if NVIDIA ~0.50, file an AMD correctness bug for Mistral chat-template inference.

Cluster: AITER custom all-reduce HIP graph-capture invalidation

• Affected: rows 15–16 — nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 in both runs.
• Shared facts: hipErrorCapturedEvent raised in c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() during CUDA-graph capture inside aiter/.../custom_all_reduce.py::_gather_ipc_meta; persistent for 21–22 days since 2026-04-11.
• Top hypothesis: [HIGH] ROCm 7.2.0 hipEventQuery cross-thread race (NCCL watchdog ignores THREAD_LOCAL capture mode) widened by AITER IPCBufferPool helper-stream kernels in v0.1.12.post1+. Disconfirming: ROCm 7.2.0 image base unchanged across the Apr 9→11 transition, so a sglang commit in 021bd77c..5cb4ea1d widened the exposure window — exact commit not bisected.
• In-flight fix: ✅ #24151 (open since 2026-04-30, ROCm 7.2.0 → 7.2.2 root-cause bump) and ✅ #23581 (open since 2026-04-23, SGLANG_USE_AITER_AR=false workaround); 🟡 #24245 overlapping. Chase reviewers — do NOT duplicate.
• Suggested triage: merge #24151 and rebuild the rocm720 image; confirm test_qwen3_instruct_mxfp4.py reaches Capturing batches.

Cluster: VLM MMMU accuracy below threshold on AMD MI30x

• Affected: row 17 — nightly-accuracy-2-gpu-vlm-rocm720 (74060283077). 5 of 11 VLM models fail per-model thresholds.
• Top hypothesis: [MEDIUM] per-model accuracy_threshold values in test_vlms_mmmu_eval_amd.py:45-133 are calibrated above the long-running AMD MI30x measurement; scores stable (variance ≤0.01 over 7 days). Disconfirming: queryable window only 7 days — cannot prove no green run earlier.
• In-flight fix: ❌ none found.
• Suggested triage: git log -p -- test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py to see when thresholds landed and whether they ever passed.

Cluster: Single-occurrence / infrastructure noise (rows 18–22)

• Row 18 (MiniMax-M2.7 borderline 0.925/0.93): [MEDIUM] sampling noise on tight threshold — 6/6 prior runs passed, 7-run range 0.925–0.955. Triage: rerun 3–5×; if pass rate ~70–90% confirm; or raise GSM8K_NUM_QUESTIONS=500. ❌ no in-flight fix.
• Row 19 (runner disconnect mid-GLM-5.1 perf): [MEDIUM] runner pod OOM/eviction; sister minimax-m27 on same runnerset also died with exit 255. ❌ no in-flight fix; pull kubelet/dmesg for linux-mi325-8gpu-sglang-bjvd5-… 18:30–18:55 UTC.
• Row 20 (HF Hub disconnect): [HIGH] transient HF Hub RemoteProtocolError for Qwen2.5-VL-7B AutoProcessor. ❌ no in-flight fix; rerun + pre-warm HF cache.
• Row 21 (Grok-1 60-min download timeout): [MEDIUM] HF Hub xet_get stall on 150 GB sharded weights; concurrent local Docker registry also unreachable. ❌ no in-flight fix; rerun first.
• Row 22 (test_encoder_dp 3600s timeout): [MEDIUM] 4-model VLM sweep too large for 3600s budget on MI325 (3/5 passes span 365–2845s); aiter using torch solution:0 slows prefill. ⚠️ adjacent #18721 targets a related mm_enable_dp_encoder hang but not this timeout. Triage: rerun in isolation; if passes, raise --timeout-per-file to 5400s or split models.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	Merge in-flight #24151 (ROCm 7.2.0 → 7.2.2) and rebuild image; rerun rocm720 nightlies	Cluster D (rows 15–16, 22-day persistent)	Fix already authored, awaiting reviewers; one merge resolves longest-standing AMD ROCm 7.2 cluster
2	A/B re-run DeepSeek-V3.2 jobs with AITER_COMMIT_OVERRIDE=v0.1.12.post1; if green, bisect aiter delta and consider reverting #24218	Cluster C (rows 8–11)	4-job blast radius from a single recent commit; cheap to disconfirm with one re-run; blocks DeepSeek-V3.2 perf+accuracy signal
3	Run rocm-smi --showpids/--showmeminfo on linux-mi35x-gpu-8-4wrmw runner pool while idle; identify+kill orphan PIDs; add the rocm-smi dump as a workflow pre-step	Cluster A (rows 1–7, biggest cluster)	7 jobs share identical infra symptom; root-cause is runner state, not code — must inspect host directly
4	Cross-check Mistral/Mixtral GSM8K scores against an NVIDIA baseline at same SHA + harness	Cluster B (rows 12–14, Never-passed ≥24 days)	Determines whether to lower thresholds or file an AMD correctness bug; blocks accuracy signal indefinitely
5	Rerun rows 19–21 (runner disconnect, HF disconnects) once before any code action	Rows 19, 20, 21	All single-occurrence transient infra; cheap to disconfirm before opening PRs

---

Job	Failed Steps	Started
nightly-accuracy-2-gpu-rocm720	Nightly Test ROCm 7.2 (2-GPU)	2026-05-01T17:46
nightly-accuracy-2-gpu-vlm-rocm720	Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)	2026-05-01T17:46
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-R1-MXFP4 KV FP8)	2026-05-01T17:46
nightly-8-gpu-minimax-m27-rocm720	Accuracy Test ROCm 7.2 (8-GPU MiniMax-M2.7)	2026-05-01T17:46
nightly-8-gpu-glm51-rocm720	N/A	2026-05-01T17:46
nightly-accuracy-8-gpu-mi35x-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU GPT-OSS)	2026-05-01T18:58
nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2)	2026-05-01T20:05
nightly-perf-8-gpu-mi35x-deepseek-v32-basic-rocm720	Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2 Basic)	2026-05-01T18:12
nightly-8-gpu-mi35x-glm51-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU GLM-5.1 NSA)	2026-05-01T19:31
nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	Accuracy Test + Performance Test MI35x ROCm 7.2 (8-GPU Qwen3-235B-MXFP4)	2026-05-01T18:34
nightly-8-gpu-mi35x-qwen35-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU Qwen 3.5)	2026-05-01T18:32
nightly-perf-2-gpu-vlm-rocm720	Performance Test ROCm 7.2 (2-GPU VLM Models)	2026-05-02T17:43
nightly-accuracy-8-gpu-mi35x-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU GPT-OSS)	2026-05-02T17:43
nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	Accuracy Test + Performance Test MI35x ROCm 7.2 (8-GPU Qwen3-235B-MXFP4)	2026-05-02T17:43
nightly-accuracy-2-gpu-vlm-rocm720	Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)	2026-05-02T17:43
nightly-perf-8-gpu-mi35x-deepseek-v32-basic-rocm720	Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2 Basic)	2026-05-02T17:43
nightly-4-gpu-rocm720	Nightly Test ROCm 7.2 (4-GPU)	2026-05-02T17:43
nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2)	2026-05-02T17:43
nightly-8-gpu-mi35x-glm51-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU GLM-5.1 NSA)	2026-05-02T19:08
nightly-accuracy-2-gpu-rocm720	Nightly Test ROCm 7.2 (2-GPU)	2026-05-02T17:43
nightly-8-gpu-grok1-int4-rocm720	Accuracy Test ROCm 7.2 (8-GPU Grok1-INT4)	2026-05-02T17:43
nightly-8-gpu-mi35x-deepseek-r1-mxfp4-ar-fusion-rocm720	Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-R1-MXFP4 AllReduce Fusion)	2026-05-02T20:00

Per-Job Analysis

nightly-accuracy-2-gpu-rocm720 — failed step(s): Nightly Test ROCm 7.2 (2-GPU)

Commit Info

• sglang: ece8a1a7
• aiter: a6bb499 (source: dev image v0.1.12.post2.dev150+ga6bb49937, rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models	AssertionError: The following models failed: (5 models below GSM8K threshold)	Log

Per-model breakdown (within the single failing test):

Model	Score	Threshold	Note
mistralai/Mistral-7B-Instruct-v0.3	0.365	0.47	recurring miss since gsm8k migration
mistralai/Mixtral-8x7B-Instruct-v0.1	0.313	0.69	recurring miss since gsm8k migration
neuralmagic/Mistral-7B-Instruct-v0.3-FP8	0.317	0.46	recurring miss since gsm8k migration
neuralmagic/Mixtral-8x7B-Instruct-v0.1-FP8	0.185	0.69	recurring miss since gsm8k migration
neuralmagic/Qwen2-72B-Instruct-FP8	0.000	0.86	intermittent — server retries exhausted (passed Apr 11 with 0.93)

Failure Summary

test_gsm8k_all_models aggregates 17 model GSM8K runs and asserts each exceeds its registered threshold. 5 of the 17 models fell below threshold today (12 passed). Four of the five (Mistral/Mixtral pairs) have been below threshold every nightly since the test was migrated from mgsm_en to gsm8k benchmark on 2026-04-08; the Qwen2-72B-FP8 sub-failure is intermittent and looks distinct (server-side retry exhaustion mid-eval).

Regression Status

Known recurring failure (4 of 5 sub-failures) + intermittent flake (Qwen2-72B-FP8).

Recent history of test_gsm8k_eval_amd.py (test_gsm8k_all_models) in job nightly-accuracy-2-gpu-rocm720 (nightly-test-amd-rocm720.yml, event=schedule, branch=main, completed runs only):

Date	Run	Test File Status	Failing sub-models	Notes
May 01	25225581444	❌ Failed	Mistral-7B, Mixtral-8x7B, Mistral-7B-FP8, Mixtral-8x7B-FP8, Qwen2-72B-FP8	this run
Apr 30	25180829975	❌ Failed	Mistral-7B, Mixtral-8x7B, Mistral-7B-FP8, Mixtral-8x7B-FP8	Qwen2-72B-FP8 passed
Apr 29	25125219864	❌ Failed	(server-launch failure earlier in test)	different symptom
Apr 27	25010858144	❌ Failed	Mistral-7B, Mixtral-8x7B, Mistral-7B-FP8, Mixtral-8x7B-FP8
Apr 26	24962968926	❌ Failed	Mistral-7B, gemma-2-27b, Mixtral-8x7B, Mistral-7B-FP8, Mixtral-8x7B-FP8, Qwen2-72B-FP8
Apr 11	24287967867	❌ Failed	Mistral-7B, gemma-2-27b, Mixtral-8x7B, Mistral-7B-FP8, Mixtral-8x7B-FP8	Qwen2-72B-FP8 PASSED with 0.93
Mar 31	23811470402	✅ Passed (17/17)	—	pre-migration: was running mgsm_en benchmark

Last passing date for this test file: 2026-03-31 (under the prior mgsm_en benchmark). First observed failure of this signature: 2026-04-08, the day after PR #21931 (commit f6fc3956) migrated the eval from mgsm_en to gsm8k. Under the gsm8k thresholds, the four Mistral/Mixtral entries are Never-passed.

Failure Cluster

GSM8K accuracy threshold miss — Mistral/Mixtral families on AMD MI30x (rocm 7.2)

Facts (verified observations, no interpretation)

• Verbatim assertion at log line 59542:

  AssertionError: The following models failed:
  - mistralai/Mistral-7B-Instruct-v0.3: score=0.3645, threshold=0.4700
  - mistralai/Mixtral-8x7B-Instruct-v0.1: score=0.3128, threshold=0.6900
  - neuralmagic/Mistral-7B-Instruct-v0.3-FP8: score=0.3166, threshold=0.4600
  - neuralmagic/Mixtral-8x7B-Instruct-v0.1-FP8: score=0.1849, threshold=0.6900
  - neuralmagic/Qwen2-72B-Instruct-FP8: score=0.0000, threshold=0.8600
  

• Pre-migration (Mar 31) the same test file passed 17/17 under mgsm_en. Post-migration nightlies (Apr 11, Apr 26, Apr 27, Apr 30, May 1) all fail with the same Mistral/Mixtral cluster.
• Mistral/Mixtral score stability across runs is high (Mistral-7B: 0.36–0.37; Mixtral-8x7B: 0.29–0.31; Mistral-7B-FP8: 0.31–0.33; Mixtral-8x7B-FP8: 0.17–0.19) — the threshold values are simply above what the AMD MI30x runs reach on gsm8k.
• Threshold rationale recorded in PR #21931 diff: "Thresholds set at 5% below reported GSM8K (5-shot/CoT) scores". The Mixtral-8x7B threshold note in the diff itself flags the risk — # 74.4% - 5% (lower if AMD scores differently).
• 12 of 17 models pass: Llama 3.1/3.2/8B-FP8/70B-FP8, gemma-2-27b, Qwen2 (non-72B), Qwen2.5-7B, Qwen3-8B, Qwen3-30B-A3B-Thinking-2507, DeepSeek-Coder-V2-Lite — so the eval pipeline itself is working.
• Distinct sub-failure: neuralmagic/Qwen2-72B-Instruct-FP8 scored exactly 0.000 with All retry attempts exhausted for request. Returning empty response. (log lines 195–197 of the captured snippet). On Apr 11 it scored 0.93 (passing), on Apr 26 also 0.000 (failing) — intermittent.
• An HSA_STATUS_ERROR_OUT_OF_RESOURCES ... Available Free mem : 128 MB event followed by Subprocess scheduler_0 (pid=43351) crashed with exit code -6. Triggering SIGQUIT for cleanup... occurred at 18:58:32 — after the Mixtral-8x7B-FP8 eval finished and before Qwen2-72B-FP8 startup completed. Could be teardown of the previous server, could be tight host-memory headroom; needs verification.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] The four Mistral/Mixtral failures are threshold-too-aggressive after the mgsm_en→gsm8k migration, not a model regression.

  • Supporting: thresholds were set in f6fc39569a (PR #21931, 2026-04-07) as reported_score - 5%; AMD MI30x scores cluster ~0.30 for Mistral-7B and ~0.18–0.31 for Mixtral, well below the 0.46–0.69 thresholds; scores are stable run-to-run; 12/17 models do pass; the diff itself comments "lower if AMD scores differently" on Mixtral-8x7B.
  • Disconfirming: ~0.30 on Mistral-7B is also lower than published HF/GSM8K scores (~0.52) — could indicate a real serving/correctness issue specific to these chat-template + few-shot prompts on AMD MI30x rather than just an AMD-specific score offset. (i.e. the threshold may be reasonable; the score may be bug-driven.) No A/B against an NVIDIA baseline is available in this run to disconfirm.
  • How to verify: run the same test_gsm8k_eval_amd.py against an NVIDIA backend with identical sglang+gsm8k harness and compare per-model scores. If NVIDIA also gets ~0.30 for Mistral, threshold is wrong; if NVIDIA gets ~0.50, there's a real AMD-side correctness/serving bug for these models.

• [LOW] neuralmagic/Qwen2-72B-Instruct-FP8 0.000 score is caused by per-request retries exhausting mid-eval, not by an accuracy regression.

  • Supporting: log line All retry attempts exhausted for request. Returning empty response. precedes Score: 0.000; Total latency only 148.764s vs ~470s eval time normally; passed with 0.93 on Apr 11.
  • Disconfirming: also failed identically on Apr 26 (run 24962968926); the pattern (HSA OOR on prior server teardown → next launch wedged) recurs at least twice, so it may not be purely random flake.
  • How to verify: look at scheduler/TP logs for Qwen2-72B-FP8's server launch around 19:01–19:12; check whether scheduler is alive and whether requests are timing out at the client (sampler retry) or at the server side.

• [LOW] HSA OOR + scheduler crash on Mixtral-8x7B-FP8 teardown leaves the GPU/host in degraded state, contributing to Qwen2-72B-FP8's failure.

  • Supporting: temporal adjacency (18:58:32 OOR → 18:58:33 scheduler crash → next test starts 18:58:05+181s startup → first prefill batches at 18:58:07 succeed but then retries exhaust by 19:12).
  • Disconfirming: Qwen2-72B-FP8 did launch successfully and did serve the first prefill batches (token usage climbed to 0.06); the failure mode is "all retry attempts exhausted" later in the run, not failed startup.
  • How to verify: check if rocm-smi/amdgpu runner state was reset between tests; check for OOM kills during Qwen2-72B-FP8 generation phase.

• [LOW] aiter a6bb499 (dev build in image) introduces a numerical regression on Mistral/Mixtral attention paths.

  • Supporting: aiter is on a dev/patched version, not the canonical Dockerfile pin.
  • Disconfirming: the Mistral/Mixtral threshold misses precede this aiter version — they appear in Apr 11 as well; multiple aiter image versions over Apr/May yield essentially the same scores. Score stability across weeks argues against a fresh aiter regression for these four models.
  • How to verify: not the priority hypothesis; only worth investigating if the NVIDIA A/B (above) shows AMD-specific divergence.

No hypothesis above LOW confidence for the Qwen2-72B-FP8 sub-failure specifically; the HIGH-confidence threshold hypothesis only addresses the four Mistral/Mixtral entries.

In-flight Fix Check

Searched: is:pr is:open gsm8k threshold amd, is:pr is:open test_gsm8k_eval_amd, is:pr is:open gsm8k migrate, is:pr is:open nightly-accuracy-2-gpu-rocm720.

• In-flight fix: ❌ none found in open PRs — no PR adjusts the AMD MODEL_SCORE_THRESHOLDS for Mistral/Mixtral, and no PR addresses the Qwen2-72B-FP8 retry-exhaustion symptom. Needs a new PR or investigation.

Suggested Triage Steps

• Run test_gsm8k_eval_amd.py (or its NVIDIA sibling if one exists) on an NVIDIA H100/A100 with the same sglang revision to obtain a reference per-model GSM8K score for Mistral-7B, Mixtral-8x7B, and the FP8 variants. Outcome decides whether the right fix is a threshold adjustment or a serving-correctness investigation.
• If NVIDIA also scores ~0.30 on Mistral-7B with this harness: open a PR adjusting MODEL_SCORE_THRESHOLDS in test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py:45 for the four Mistral/Mixtral entries to ~5% below the AMD-observed scores, citing the gap from the GSM8K-reported numbers.
• If NVIDIA scores ~0.50 on Mistral-7B: bisect AMD-side serving correctness for mistralai/Mistral-7B-Instruct-v0.3 (chat template, sampling, attention backend) — start by verifying gsm8k chat-template handling and --temperature 0 determinism, then compare attention backends (--attention-backend triton vs aiter) on this model.
• Separately for Qwen2-72B-FP8 0.000 score: rerun the test in isolation (pytest ... -k Qwen2-72B-Instruct-FP8) to disentangle from the Mixtral teardown crash; if it still fails, capture scheduler logs to determine whether requests time out client-side or server-side.
• Optional: investigate the HSA OUT_OF_RESOURCES (Available Free mem : 128 MB) on Mixtral-8x7B-FP8 teardown — likely host-memory exhaustion at multi-server-restart in a single test process; consider explicit gc.collect() + torch.cuda.empty_cache() between models in the test driver.

Status

• Persistence: Mistral/Mixtral cluster failing every completed nightly since 2026-04-08 (≈24 days, 100% of completed runs in window); Qwen2-72B-FP8 sub-failure intermittent (failed Apr 26 + May 01, passed Apr 11 + Apr 30).
• Scope: this single nightly-accuracy-2-gpu-rocm720 job in nightly-test-amd-rocm720.yml. Not seen in non-AMD nightly accuracy jobs (different test files / different thresholds).
• Blocked work: AMD MI30x rocm 7.2 nightly accuracy signal — the job has not produced a green GSM8K result since the migration.
• In-flight fix: ❌ none found — needs new PR or investigation.

nightly-accuracy-2-gpu-vlm-rocm720 — failed step(s): Nightly Accuracy Test ROCm 7.2 (2-GPU VLM MMMU)

Commit Info

• sglang: ece8a1a7
• aiter: a6bb4993 (source: dev image preserved — v0.1.12.post2.dev150+ga6bb49937 matches Dockerfile default; no override)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	TestNightlyGsm8KEval.test_gsm8k_all_models	AssertionError: The following models failed: (5 models below)	Log

5 model sub-failures inside the single test:

Model	TP	FP8	Score	Threshold	Notes
mistralai/Mistral-7B-Instruct-v0.3	1	no	0.365	0.47	3-retry consensus 0.369/0.372/0.365
mistralai/Mixtral-8x7B-Instruct-v0.1	2	no	0.313	0.69	3-retry consensus 0.318/0.308/0.313
neuralmagic/Mistral-7B-Instruct-v0.3-FP8	1	yes	0.317	0.46	3-retry consensus 0.329/0.336/0.317
neuralmagic/Mixtral-8x7B-Instruct-v0.1-FP8	2	yes	0.185	0.69	3-retry consensus 0.187/0.187/0.185
neuralmagic/Qwen2-72B-Instruct-FP8	2	yes	0.000	0.86	Scheduler crashed mid-decode (HSA OOM, exit -6)

Failure Summary

The single registered test test_gsm8k_all_models in test_gsm8k_eval_amd.py aggregates per-model gsm8k accuracy checks. 5 of the 17 models scored below their threshold. The 4 Mistral/Mixtral models miss by very large margins (~10–50 absolute pts), and neuralmagic/Qwen2-72B-Instruct-FP8 scored 0.000 because the TP=0 scheduler aborted mid-batch with HSA_STATUS_ERROR_OUT_OF_RESOURCES (exit code -6) and all subsequent requests timed out.

Regression Status

Known recurring failure (also Never-passed for the 4 Mistral/Mixtral models in the queryable window).

Recent history of test_gsm8k_eval_amd.py (test_gsm8k_all_models) on nightly-test-amd-rocm720.yml (event=schedule, branch=main, completed runs only). The current job nightly-accuracy-2-gpu-vlm-rocm720 ran this file because today both 2-gpu accuracy shards picked up the same partition; in earlier runs the file consistently ran inside nightly-accuracy-2-gpu-rocm720. Comparison is at the same test file:

Date	Run	Job (where this file ran)	Test File Status	Failed Models	Error
May 1	25225581444	vlm shard	❌ Failed	Mistral×2, Mixtral×2, Qwen2-72B-FP8	accuracy + OOM
Apr 30	25180829975	non-vlm shard	❌ Failed	Mistral×2, Mixtral×2	accuracy
Apr 27	25010858144	non-vlm shard	❌ Failed	Mistral×2, Mixtral×2	accuracy
Apr 26	24962968926	non-vlm shard	❌ Failed	Mistral×2, Mixtral×2, gemma-2-27b, Qwen2-72B-FP8	accuracy + OOM
Apr 22	24793595557	non-vlm shard	❌ Failed	Mistral×2, Mixtral×2	accuracy
Apr 21	24737582959	non-vlm shard	❌ Failed	Mistral×2, Mixtral×2, Qwen2-72B-FP8	accuracy + OOM

No passing run for this test file in the last 6+ scheduled runs (lookback ≥ 11 days). The 4 Mistral/Mixtral failures are stable in magnitude (Mistral-7B ~0.36–0.37, Mixtral 0.29–0.32, Mistral-7B-FP8 ~0.30–0.34, Mixtral-FP8 ~0.17–0.19) — well-controlled accuracy regression, not noise. Qwen2-72B-FP8 OOM is intermittent (3 of 6 runs sampled).

Failure Cluster

• Cluster A: "Mistral/Mixtral gsm8k accuracy below threshold (TP=1/2, BF16 + FP8)" — 4 model sub-failures
• Cluster B: "Qwen2-72B-Instruct-FP8 TP=2 scheduler aborts mid-decode with HSA OOM" — 1 model sub-failure

Facts (verified observations, no interpretation)

• Today's run launches [AITER_COMMIT_OVERRIDE=] empty; image already at v0.1.12.post2.dev150+ga6bb49937 (Dockerfile default = a6bb4993). No aiter override (this is NOT a Scout run; baseline A/B not required).
• All 4 Mistral/Mixtral failures consistently miss the threshold across 3 retries each within the same run (run-internal noise is not the cause):
  • Mistral-7B: 0.369, 0.372, 0.365 ↔ threshold 0.47 (gap ≈ -0.10)
  • Mixtral-8x7B: 0.318, 0.308, 0.313 ↔ threshold 0.69 (gap ≈ -0.38)
  • Mistral-7B-FP8: 0.329, 0.336, 0.317 ↔ threshold 0.46 (gap ≈ -0.13)
  • Mixtral-8x7B-FP8: 0.187, 0.187, 0.185 ↔ threshold 0.69 (gap ≈ -0.50)
• test_gsm8k_eval_amd.py (source) has commented-out entries for these exact models in TRITON_MOE_MODELS, indicating maintainers were already aware of attention/MoE-backend interaction issues for them but did not exclude them or lower thresholds. Mixtral-FP8 also lives in NO_MOE_PADDING_MODELS (line 106) — special-cased, but not enough to recover accuracy.
• Qwen2-72B-FP8 crash signature (verbatim, line 24772): :0:rocdevice.cpp :3582: ... HSA_STATUS_ERROR_OUT_OF_RESOURCES: ... Code: 0x1008 Available Free mem : 128 MB followed by Fatal Python error: Aborted and Subprocess scheduler_0 (pid=43351) crashed with exit code -6. The crash happens during decode at high concurrency: last log line before abort shows #running-req: 778, #queue-req: 237, #pending-token: 207079, token usage: 0.12 (line 24769).
• Stack trace top frames at crash time (line 24784+): static_quant_fp8 (fp8_kernel.py:714) → apply_fp8_linear (fp8_utils.py:1612) → Fp8LinearMethod.apply (fp8.py:772) → qwen2.py:190 → ... → event_loop_overlap → run_scheduler_process.
• Same Qwen2-72B-FP8 score=0.0000 was recorded on Apr 21 and Apr 26 in identical test partition — intermittent OOM, not deterministic.
• After the Qwen2-72B-FP8 scheduler crash, the eval driver retried the gsm8k run twice; both retries returned 0.205 then 0.000 (final). All attempts log All retry attempts exhausted for request. Returning empty response.

Hypothesised Causes (with confidence, with disconfirming evidence)

Cluster A — Mistral/Mixtral accuracy regression

• [LOW] Aiter attention backend (attention_backend='aiter') numerical accuracy issue for Mistral-style and Mixtral MoE architectures on MI325/gfx942.

  • Supporting: Failure is reproducible across retries within a run (low variance) and across days. The commented-out entries for these models in TRITON_MOE_MODELS (line 112-115) suggest maintainers previously considered routing them through Triton MoE — implies a known MoE-backend interaction concern.
  • Disconfirming: This is a Never-passed pattern in the queryable window — no green run found, so we cannot prove "aiter caused it" by bisection without going further back. The failure is not unique to MoE either: dense Mistral-7B-Instruct-v0.3 also fails. Other aiter-using models in the same job (Llama-3.1-8B, Llama-3.1-70B, Qwen2-57B, Qwen3-8B, Qwen3-30B) PASS at high margin, so aiter is not uniformly broken.
  • How to verify: rerun the failing 4 models with --attention-backend triton and --moe-runner-backend triton to A/B against the aiter path.

• [LOW] Thresholds calibrated for NVIDIA reference scores not validated on AMD MI325/ROCm 7.2; Mistral/Mixtral genuinely score lower on AMD due to a long-standing kernel difference (e.g. SwiGLU/RMSNorm fp16-accumulation behavior).

  • Supporting: Comments next to thresholds (lines 45/46/61/65) note "X% - 5%" / "lower if AMD scores differently" — author acknowledged AMD may diverge. Magnitude of failure (Mixtral 0.31 vs 0.69 threshold) is too large to be a simple calibration gap, but Mistral-7B (0.36 vs 0.47 = -0.10) is in calibration-error territory.
  • Disconfirming: Mixtral-FP8 at 0.185 vs 0.69 (-0.50) is far beyond any plausible calibration gap — that is a real correctness regression, not threshold noise.
  • How to verify: run gsm8k on the same models on a known-good NVIDIA reference image and compare; rerun on an older sglang+aiter combo to find a green commit window.

• [LOW] Long-standing AMD-specific MoE kernel correctness issue affecting Mixtral-class models (8 experts × 7B), with Mistral failing for an unrelated reason that just landed in the same window.

  • Supporting: Mixtral-FP8 at 0.185 strongly resembles "garbage output" (random-baseline gsm8k is ~0). NO_MOE_PADDING_MODELS carve-out for Mixtral-FP8 (line 106) hints at a known padding-vs-correctness tradeoff.
  • Disconfirming: dense Mistral-7B is failing too with no MoE involvement, so MoE-only explanations don't cover all 4 failures.
  • How to verify: Toggle --moe-padding for the BF16 Mixtral run; rerun Mistral-7B with --attention-backend triton to isolate attention vs MoE.

• No hypothesis above LOW confidence; needs git bisect over the last green run for this test file. Earliest queryable failure is Apr 21 — to establish a regression window I'd need to walk further back (and possibly older workflow IDs that have rolled out of the API window).

Cluster B — Qwen2-72B-FP8 scheduler crash (HSA OOM)

• [MEDIUM] mem_fraction_static=0.76415 is too aggressive for Qwen2-72B-Instruct-FP8 on TP=2 MI325 (128 GB HBM each); under high concurrency the scheduler runs out of HSA queue/pinned-memory headroom (Available Free mem: 128 MB at crash time).

  • Supporting: Crash signature is HSA_STATUS_ERROR_OUT_OF_RESOURCES with Free mem at exactly 128 MB; happens under heavy decode (#running-req: 778, #queue-req: 237); only this one large 72B-FP8 model crashes; failure is intermittent across runs (3-of-6 sampled), consistent with a memory pressure threshold that depends on prefix-cache state and request mix.
  • Disconfirming: smaller and similarly sized models in the same job (Llama-3.1-70B-FP8 at TP=2, Qwen2-57B-A14B-FP8 at TP=2) PASS without OOM, so it is not a generic TP=2 memory-budget bug — it is specific to Qwen2-72B-FP8's working set.
  • How to verify: rerun Qwen2-72B-FP8 with --mem-fraction-static 0.70 and/or --max-running-requests 256 and check whether the crash disappears; also try --cuda-graph-max-bs 256 to cap working set.

• [LOW] FP8 quantization codepath (apply_fp8_linear → static_quant_fp8) is allocating temporaries that exceed available pinned memory at high batch sizes.

  • Supporting: Stack-trace top frame is static_quant_fp8 in fp8_kernel.py:714. Other FP8 models that pass (Llama-3.1-70B-FP8, Qwen2-57B-A14B-FP8) are smaller — Qwen2-72B is the largest FP8 model in the suite.
  • Disconfirming: same code path runs successfully for the other FP8 models in the same job; the crash log says memory exhaustion, not a numerical or kernel-API failure.
  • How to verify: same as above (--mem-fraction-static).

In-flight Fix Check

Searched open PRs for mistral gsm8k, gsm8k threshold, Mixtral accuracy amd, amd gsm8k nightly, threshold nightly amd (5 queries via https://api.github.com/search/issues?q=...).

• In-flight fix: ❌ none found — no open PR addressing the Mistral/Mixtral gsm8k accuracy regression or the Qwen2-72B-FP8 OOM crash.

Suggested Triage Steps

• Cluster A (Mistral/Mixtral accuracy):
  • Rerun the 4 failing models with --attention-backend triton (and additionally --moe-runner-backend triton for the two MoE Mixtral variants) to A/B test whether the aiter attention/MoE backend is responsible. The commented-out TRITON_MOE_MODELS entries in test_gsm8k_eval_amd.py:112-115 suggest this was previously the workaround.
  • If still failing under triton, walk further back through workflow runs (page 2+ of actions/workflows/nightly-test-amd-rocm720.yml/runs) to find a green run for test_gsm8k_eval_amd.py. Last seen successes on this workflow are Mar 8/9 — bisect commits in 1cb86f51..ece8a1a7 if older runs cannot be queried.
  • If maintainers conclude the gap is genuinely architectural and not a regression, lower thresholds in MODEL_SCORE_THRESHOLDS (lines 45–65) or remove the failing models from the AMD eval set, rather than letting the test-file Never-pass.
• Cluster B (Qwen2-72B-FP8 OOM):
  • Reproduce the crash with sglang serve --model-path neuralmagic/Qwen2-72B-Instruct-FP8 --tp 2 --device cuda followed by python3 bench_serving --num-prompts 1314 --request-rate inf (mirrors the eval driver). Then halve --mem-fraction-static (0.76 → 0.65) and rerun.
  • If the crash is reproducible, file a one-line config change to set per-model mem-fraction-static for Qwen2-72B-FP8 in popen_launch_server_wrapper so the OOM stops masking the underlying accuracy result.

Status

• Persistence: 4 Mistral/Mixtral failures persist for at least 11 days (since Apr 21 06:13 UTC, possibly older — limited by API lookback). Never-passed for test_gsm8k_eval_amd.py in the queryable schedule window.
• Scope: 1 test file (test_gsm8k_eval_amd.py) failing in 1 job in nightly-test-amd-rocm720.yml. Today the file ran in nightly-accuracy-2-gpu-vlm-rocm720 (instead of its usual nightly-accuracy-2-gpu-rocm720 shard) due to a partition collision, but the failures are independent of which shard executes the file.
• Blocked work: Blocks gsm8k accuracy signal on AMD MI325/ROCm 7.2 nightly. The Qwen2-72B-FP8 OOM also blocks signal on that one model independently.
• In-flight fix: ❌ none found in open PRs — needs new PR or investigation.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-05-02 20:25 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — Per-Job Analysis (continued 2/6)

nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-R1-MXFP4 KV FP8)

Commit Info

• sglang: ece8a1a7
• aiter: a6bb499375849eec45d68c5ccaebc8865fd422c0 (source: Dockerfile default, dev image v0.1.12.post2.dev150+ga6bb49937 preserved per [CI-AITER-CHECK])

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_r1_mxfp4_kv_fp8_eval_mi35x.py	test_deepseek_r1_mxfp4_kv_fp8_accuracy (model amd/DeepSeek-R1-MXFP4-Preview kv-fp8)	RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes. pre_model_load_memory=6.73828125, local_gpu_memory=8.44140625 → AssertionError: Failed models: ['amd/DeepSeek-R1-MXFP4-Preview (kv-fp8)']	Log

Failure Summary

The single test in this job failed during server startup at python/sglang/srt/model_executor/model_runner.py:1231 (TP1 rank): init_torch_distributed raised because per-rank free GPU memory differed by >10% across the 8 TP ranks. Server died in 17.4s of init; entire test ran 44.5s before the assert.

Regression Status

Known recurring failure (3 consecutive scheduled runs).

Recent history of test_deepseek_r1_mxfp4_kv_fp8_eval_mi35x.py (test_deepseek_r1_mxfp4_kv_fp8_accuracy) in job nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8-rocm720 (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
May 01	25225581444	73968206660	❌ Failed	test_deepseek_r1_mxfp4_kv_fp8_accuracy	RuntimeError mem unbalanced (pre=6.74, local=8.44)
Apr 30	25180829975	73825313940	❌ Failed	test_deepseek_r1_mxfp4_kv_fp8_accuracy	RuntimeError mem unbalanced (pre=11.46, local=13.17)
Apr 29	25125219864	73718181393	❌ Failed	test_deepseek_r1_mxfp4_kv_fp8_accuracy	RuntimeError mem unbalanced (pre=6.74, local=8.44)
Apr 28	25069314464	—	⚪ Cancelled	—	—
Apr 27	25010858144	73245936866	✅ Passed	—	—

First observed failure: 2026-04-29; last passing: 2026-04-27. Different runners each run (xjflh, 69mkd, x4lxm — all from the linux-mi35x-gpu-8-4wrmw-runner-* pool).

Failure Cluster

GPU memory unbalanced at TP init on mi35x-8gpu pool

Facts (verified observations)

• Server-launch crash: TP1 raises RuntimeError at model_runner.py:1231; guarded by env SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK (which CI evidently sets) — verified from log line 1978 of the extracted job log.
• Available memory at TP init is tiny relative to MI355X capacity: local_gpu_memory ≈ 8.44 GB of free memory. MI355X HBM is ~288 GB, so ≈97% of memory was already occupied before sglang started.
• Identical error fingerprint (pre=6.73828125, local=8.44140625) appears in multiple sibling jobs in the same run: nightly-accuracy-8-gpu-mi35x-rocm720, nightly-8-gpu-mi35x-glm51-rocm720, nightly-8-gpu-mi35x-qwen35-rocm720 — pool-wide problem, not a single bad runner.
• Apr 30 run had different but similarly small numbers (pre=11.46, local=13.17 GB free), Apr 29 matched today's exact numbers — suggests deterministic leftover (a specific cached allocation of ~280 GB / rank).
• [CI-AITER-CHECK] shows aiter unchanged (v0.1.12.post2.dev150+ga6bb49937, dev image preserved); aiter is not in scope as a regression cause.
• Commits in regression window f34222da..530b497a (Apr 27 → Apr 29, 16 commits) — none touch model_runner.init_torch_distributed, GPU memory accounting, or TP/dist init.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Runner-pool infrastructure issue: residual GPU memory occupied by leftover processes (or persistent cached allocations) on the linux-mi35x-gpu-8-4wrmw-runner-* pool, causing pre_model_load_memory < local_gpu_memory * 0.9 at TP init.
  • Supporting: identical error / memory values across 3 different runners in the same pool; identical error in 4 sibling mi35x-8gpu jobs in this run; only ~3% of MI355X HBM is free at process start (8 GB out of ~288 GB); deterministic numbers (8.44140625 GB) suggest the same leftover state on each runner.
  • Disconfirming: (none found) — sister jobs with smaller models (e.g. nightly-test-1-gpu-mi35x-rocm720, nightly-8-gpu-mi35x-grok1-int4-rocm720) passed in this run, but they may have used different runner pools (1-gpu vs 8-gpu) or models that fit in the residual memory.
  • How to verify: SSH to the runner pool when idle and run rocm-smi --showmeminfo vram — if non-zero usage with no sglang process, leftover allocation is confirmed. Or add a rocm-smi dump as an early step in this workflow.
• [LOW] Code regression in sglang between f34222da1b22..530b497a48e3 (Apr 27→29) introducing a memory leak in a previous test step.
  • Supporting: timing aligns with regression onset.
  • Disconfirming: this job has only ONE test step (the failing one), so prior tests in this job cannot leak; the failure happens before any sglang allocation. Sister jobs on different runners hitting the same deterministic numbers is hard to explain via code regression. None of the 16 commits in the window touch GPU memory init or TP code (verified via git log --oneline ... -- model_runner.py scheduler.py tp_worker.py returns only bd448e51 which is spec-decoding telemetry, unrelated).
  • How to verify: pick any commit in window, force a re-run on a freshly-rebooted runner; if the failure disappears with reboot, code is exonerated.
• [SPECULATION] Docker image change (Dockerfile / base image) reduced cleanup between runs.
  • Supporting: timing aligns; container is rocm/sgl-dev:v0.5.10.post1-rocm720-mi30x-....
  • Disconfirming: no Dockerfile change in regression window for AMD images visible in the commit list.

No hypothesis above HIGH for code-side cause; the HIGH is for an infra/runner-state cause, not a code change.

In-flight Fix Check

Searched open PRs:

• q=memory+capacity+unbalanced → 0 results
• q=mi35x+occupied+process → 0 results
• q=SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK → 1 result (#18037, only adds a warning — does not address runner state)
• q=runner+cleanup+gpu → 22 results, none specifically for mi35x leftover memory

In-flight fix: ❌ none found in open PRs — needs runner-pool investigation by AMD CI infra team.

Suggested Triage Steps

• Have AMD CI infra check the linux-mi35x-gpu-8-4wrmw-runner-* pool: run rocm-smi --showmeminfo vram --showpids on idle runners; identify the leftover process / cached allocation and add a teardown/reboot step between jobs.
• Add an early diagnostic step to nightly-test-amd-rocm720.yml mi35x jobs: rocm-smi --showmeminfo vram before the test step, so future failures show the residual occupant directly.
• Cross-check the 3 other mi35x-8gpu jobs failing with the same fingerprint in run 25225581444 (nightly-accuracy-8-gpu-mi35x-rocm720, nightly-8-gpu-mi35x-glm51-rocm720, nightly-8-gpu-mi35x-qwen35-rocm720) — fixing the runner state should resolve all four together.
• As a temporary safety valve, consider unsetting SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK on AMD nightly so the warning does not abort startup; the model would still likely OOM during weight load with only 6.74 GB free, but it gives a clearer signal of the actual capacity gap.
• If runner cleanup does not resolve it, bisect f34222da..530b497a (16 commits) on a clean MI35x runner to rule out a memory-retention regression; expected to be unproductive given the disconfirming evidence.

Status

• Persistence: Failing for 3 days (since 2026-04-29), every completed scheduled run since (Apr 28 was cancelled).
• Scope: At minimum 4 jobs in this single run share the identical error fingerprint on different runners of the linux-mi35x-gpu-8-4wrmw-runner-* pool (-kv-fp8, -accuracy, -glm51, -qwen35); all are 8-GPU mi35x jobs in nightly-test-amd-rocm720.yml.
• Blocked work: Blocks DeepSeek-R1-MXFP4 KV-FP8 accuracy signal on MI35x; together with sibling failures, blocks much of the mi35x-8gpu nightly accuracy signal.
• In-flight fix: ❌ none found — needs runner-pool investigation.

nightly-8-gpu-minimax-m27-rocm720 — failed step(s): Accuracy Test ROCm 7.2 (8-GPU MiniMax-M2.7)

Commit Info

• sglang: ece8a1a7
• aiter: a6bb49937 (source: dev/patched image — v0.1.12.post2.dev150+ga6bb49937, matches Dockerfile default head)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_minimax_m27_eval_amd.py	TestMiniMaxM27EvalAMD.test_minimax_m27_accuracy	AssertionError: Failed models: ['MiniMaxAI/MiniMax-M2.7 (TP8+EP8)'] — accuracy=0.925 threshold=0.93	Log

Failure Summary

The MiniMax-M2.7 GSM8K accuracy gate (200 questions, TP8+EP8, attention=aiter) measured 0.925 versus a threshold of 0.93, a 0.005 shortfall. The server completed all 200 requests without errors; only the post-run accuracy assertion failed.

Regression Status

Borderline / likely flaky. New failure today, but the threshold (0.93) sits at the lower edge of historically observed accuracy.

Recent history of test/registered/amd/accuracy/mi30x/test_minimax_m27_eval_amd.py (test_minimax_m27_accuracy) in job nightly-8-gpu-minimax-m27-rocm720 (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Accuracy	Notes
May 1	25225581444	job	❌ Failed	0.925	seed=888426668
Apr 30	25180829975	job	✅ Passed	0.935	seed=528452795
Apr 29	25125219864	job	✅ Passed	0.955
Apr 28	25069314464	—	⚪ Cancelled	—	(excluded)
Apr 27	25010858144	job	✅ Passed	0.930	exact threshold (margin = 0)
Apr 26	24962968926	job	✅ Passed	0.940
Apr 25	24936773213	job	✅ Passed	0.945
Apr 24	24903597060	job	✅ Passed	0.945

First failing date: 2026-05-01. Last passing: 2026-04-30. Observed accuracy range over 7 completed runs: 0.925 – 0.955 (sample mean ≈ 0.939, sample std ≈ 0.010). Per-seed binomial noise on 200 GSM8K examples at p≈0.94 is ≈ √(0.94·0.06/200) ≈ 0.017, so a 0.925 outcome is within one sigma of the historical mean.

Failure Cluster

MiniMax-M2.7 GSM8K accuracy gate borderline (0.925 vs 0.93 threshold)

Facts (verified observations, no interpretation)

• Verbatim assertion: accuracy=0.925 threshold=0.93 FAIL (log line 6137) → AssertionError: Failed models: ['MiniMaxAI/MiniMax-M2.7 (TP8+EP8)'] (log line 6149).
• Server completed all 200 GSM8K requests with HTTP 200; no model-level crash, no kernel error, no [CI-AITER-CHECK] mismatch.
• AITER inside the CI image was the dev build matching the Dockerfile default (a6bb49937); AITER_COMMIT_OVERRIDE was empty (this is not a scout run).
• Test config (test_minimax_m27_eval_amd.py:67): accuracy_threshold=0.93, num_questions=200, tp_size=8, --ep-size 8, --attention-backend aiter, --mem-fraction-static 0.85.
• Per-run random_seed differs (888426668 today vs 528452795 Apr 30) — sampling is non-deterministic across runs.
• Two stray Pyspy errors in the log (Pyspy failed (py-spy dump --native --pid 1139) at 18:02:43, 18:07:44) are monitoring artifacts and unrelated to the assertion.
• 39 sglang commits merged between last passing SHA 7bb7f604 and this SHA ece8a1a7.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Sampling noise on a tightly-set threshold (test design issue, not a regression).
  • Supporting: 7-run accuracy range 0.925–0.955; threshold 0.93 sits at the 5th–10th percentile of observed values; Apr 27 already passed by 0.000 margin (0.930 == 0.93). Binomial sampling sigma at N=200, p≈0.94 ≈ 0.017 — a 0.925 reading is well within 1σ of the empirical mean (0.939). Different random seeds across runs explain the variance. Server reports no functional errors.
  • Disconfirming: 6/6 prior completed runs in the window passed; today is the first observed failure. Cannot strictly rule out a small accuracy regression hiding inside the noise floor (a true ~1-pt drop is statistically indistinguishable from sampling jitter at this N).
  • How to verify: re-run the same job 3-5 times on the same SHA; if pass rate is ~70-90%, hypothesis is confirmed. Or raise N (e.g. GSM8K_NUM_QUESTIONS=500) and re-test the threshold.
• [LOW] MoE / EP runner change in the regression window may have nudged accuracy down by ~1 pt. Candidates that touch MoE or EP code paths exercised by MiniMax-M2.7 (TP8+EP8):
  • 108bfd8b "[MoE] Add Aiter MoE runner backend and purge aiter.fused_moe from quant methods" (#23597)
  • 8975479f "[LoRA][MOE] Fix EP correctness in MoE LoRA slicing and virtual-experts kernels" (#24171)
  • 7e17a7a1 "Upd: AITER->a6bb499" (#24218)
  • Supporting: all three land in the regression window and touch MoE/EP/aiter code paths exercised by the failing config (EP=8, attention=aiter). The MoE runner refactor in particular changes how aiter MoE is dispatched.
  • Disconfirming: the magnitude of drift required (~1 pt) is below the 1σ sampling noise, so temporal correlation alone is weak evidence. None of these PRs are directly traceable in the failing run's log (no error or warning references them). The MoE LoRA fix only impacts code when LoRA is enabled (it isn't here).
  • How to verify: A/B by running the GSM8K eval on parent SHA 7bb7f604 and HEAD ece8a1a7 with fixed seed and N≥500 and compare means.

No hypothesis above MEDIUM confidence.

In-flight Fix Check

Searched repo:sgl-project/sglang+is:pr+is:open+minimax+accuracy (30 hits — none target this MiniMax-M2.7 GSM8K threshold or the MI300X accuracy harness).

• In-flight fix: ❌ none found — needs new PR or investigation

Suggested Triage Steps

• First, verify flake hypothesis (cheap): re-run nightly-test-amd-rocm720.yml on the same SHA ece8a1a7 3 times via workflow_dispatch. If 2/3 pass → flake; close out and consider a config fix below.
• Config fix (if flake confirmed): lower accuracy_threshold from 0.93 to ~0.91 in test_minimax_m27_eval_amd.py:67, OR raise GSM8K_NUM_QUESTIONS to 500 (current 1σ ≈ 0.017 → would drop to ≈ 0.011 and let 0.93 stay safely above the noise floor).
• If reruns also fail (true regression): bisect 7bb7f604..ece8a1a7 — narrow on MoE/EP/aiter commits first (108bfd8b, 8975479f, 7e17a7a1). Use a fixed seed and N≥500 questions per probe to keep noise below the bisect signal.

Status

• Persistence: First failure 2026-05-01 17:45 UTC; 1 run affected. Apr 24–30 (6 completed runs) all passed.
• Scope: One job, one workflow, one test (nightly-8-gpu-minimax-m27-rocm720 in nightly-test-amd-rocm720.yml).
• Blocked work: AMD MI300X MiniMax-M2.7 nightly accuracy signal.
• In-flight fix: ❌ none found — needs new PR or investigation.

nightly-8-gpu-glm51-rocm720 — failed step(s): N/A

Commit Info

• sglang: ece8a1a7
• aiter: a6bb4993 (source: Dockerfile default; CI image was a dev/patched build v0.1.12.post2.dev150+ga6bb49937, rebuild was skipped)

Failed Tests

Test File	Test Function	Error	Log
N/A (infrastructure failure during Step 6 "Performance Test")	N/A — nightly-perf-8-gpu-glm51 suite was running but never produced a test verdict	The self-hosted runner lost communication with the server (GitHub Actions annotation) → outer Process completed with exit code 255	Job log

Note: Step 5 "Accuracy Test ROCm 7.2 (8-GPU GLM-5.1 NSA)" (test/registered/amd/accuracy/mi30x/test_glm51_eval_amd.py) finished with conclusion=success at 18:33:19 UTC. Step 6 "Performance Test ROCm 7.2 (8-GPU GLM-5.1)" (suite nightly-perf-8-gpu-glm51) started at 18:33:19 UTC and was marked status=in_progress / conclusion=null when the runner died at 18:55:19 UTC.

Failure Summary

The job ran on self-hosted runner linux-mi325-8gpu-sglang-bjvd5-runner-wlrc9. The accuracy step passed in ~36 min. The performance step (with continue-on-error: true, timeout 120 min) was running for ~22 min when the runner stopped heart-beating to GitHub. Because runner-loss bypasses continue-on-error, the job is marked failed even though no test produced a failure verdict.

Regression Status

Infrastructure issue — first seen today for this job (nightly-8-gpu-glm51-rocm720).

Recent history of step 5 + 6 of nightly-8-gpu-glm51-rocm720 on nightly-test-amd-rocm720.yml (event=schedule, branch=main, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
May 01	25225581444	nightly-8-gpu-glm51-rocm720	❌ Job failed (runner died mid-step 6; step 5 ✅ success)	N/A	runner lost communication
Apr 30	25180829975	nightly-8-gpu-glm51-rocm720	✅ Both steps passed	—	—
Apr 29	25125219864	nightly-8-gpu-glm51-rocm720	✅ Both steps passed	—	—
Apr 27	25010858144	nightly-8-gpu-glm51-rocm720	✅ Both steps passed	—	—
Apr 26	24962968926	nightly-8-gpu-glm51-rocm720	✅ Both steps passed	—	—

(First observed failure: 2026-05-01 18:55 UTC; last known passing run: 2026-04-30. Apr 28 run was cancelled and excluded.)

Failure Cluster

Self-hosted runner lost communication with GitHub Actions server (mid–GLM-5.1 performance test).

Facts (verified observations, no interpretation)

• GitHub check-run annotation on this job: failure: The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error. (verbatim from /check-runs/73968206663/annotations).
• Outer step exit: Process completed with exit code 255 (warning + failure annotations).
• Step 5 (Accuracy GLM-5.1 NSA) finished success at 18:33:19 UTC after ~36 min on the same runner.
• Step 6 (Performance GLM-5.1) started 18:33:19 UTC, last log line in available capture is at 18:32:53 UTC (truncated by GitHub's ~2 MB log cap; whole log volume points to verbose [aiter] shape ... and [fused_moe] chatter saturating the buffer).
• Runner: linux-mi325-8gpu-sglang-bjvd5-runner-wlrc9 (MI325 8-GPU pool).
• Job conclusion: failure; step 6 status=in_progress conclusion=null; step 12 (Post Checkout) never started (status=pending).
• AITER version that ran: dev image v0.1.12.post2.dev150+ga6bb49937 (matches Dockerfile default a6bb4993; no AITER_COMMIT_OVERRIDE).
• Sister nightly-8-gpu-minimax-m27-rocm720 job in the same run (different runner pod …-bjtrl of the same bjvd5 runnerset) also failed with Process completed with exit code 255.
• The previous 4 completed scheduled runs of this job (Apr 26, 27, 29, 30) all had both steps succeed.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Runner infrastructure loss — the GH-emitted annotation explicitly describes this category; combined with (a) step 5 succeeding 22 min earlier, (b) no test traceback / no exception in the captured log, (c) sister job on the same runnerset also dying with exit 255, and (d) job ending well before the 120-min step timeout. Most likely cause is the runner pod being OOM-killed, network-partitioned, or evicted while the perf workload was active.
  • Supporting: GH annotation text is the standard signature for runner-process death; multiple jobs on the linux-mi325-8gpu-sglang-bjvd5-… runnerset failed in the same run window.
  • Disconfirming: only this job and minimax-m27 failed with this signature; many other jobs on linux-mi325-… and linux-mi35x-… runnersets in the same run completed normally, so it is not a wholesale pool outage.
  • How to verify: pull runner-side dmesg / kubelet eviction events for pod linux-mi325-8gpu-sglang-bjvd5-runner-wlrc9 between 18:30–18:55 UTC; check ROCm SMI / host memory pressure during GLM-5.1 perf step.
• [LOW] GLM-5.1 perf workload starves runner of memory or wedges the GPU — the perf step passes SGLANG_USE_AITER=1 and uses nightly-perf-8-gpu-glm51 with --timeout-per-file 5400; if it hits a tight host-memory or HIP-driver edge, the watchdog may kill the runner agent.
  • Supporting: runner died only after step 6 began (step 5 same model on same runner was healthy); minimax-m27 is similarly heavy on the same runner pool.
  • Disconfirming: this job has passed cleanly for 4 consecutive scheduled runs prior; commit delta 7bb7f604..ece8a1a7 is 39 commits and none touch GLM-5.1 perf/server lifecycle obviously (closest hit is d48095ba "Bypass torch.cuda.use_mem_pool generator-CM in SymmetricMemoryContext", but symmetric-memory is not exercised by this perf suite).
  • How to verify: rerun the failed job on a fresh runner; if it passes, treat as transient infra. If it reliably reproduces, bisect the 39-commit window with the nightly-perf-8-gpu-glm51 suite alone.
• [LOW] Recent commit interaction — 39 sglang commits landed between Apr 30's passing run (7bb7f604) and today (ece8a1a7). Candidate-by-keyword: 108bfd8b "[MoE] Add Aiter MoE runner backend and purge aiter.fused_moe from quant methods" (#23597) touches the aiter MoE path that GLM-5.1's perf workload heavily exercises (visible [aiter] [fused_moe] chatter in the log).
  • Supporting: GLM-5.1 perf invokes aiter fused_moe; the commit reworks this path.
  • Disconfirming: the same commit was already in the codebase for the accuracy step that succeeded on the same runner 22 minutes earlier; an aiter MoE bug would be expected to surface in either the accuracy run or the perf run with a Python/HIP error, not with a runner-agent disconnect.
  • How to verify: revert 108bfd8b on a branch, re-run nightly-perf-8-gpu-glm51 only — only useful if hypothesis 1 is ruled out.

No hypothesis above MEDIUM confidence. The strongest evidence (annotation message + lack of any Python traceback + sister-job exit-255 on the same runnerset) all points to runner-side infra, not code.

In-flight Fix Check

Searched: runner+communication, runner+stability, ci+runner+stability in open sglang PRs.

• In-flight fix: ❌ none found directly addressing self-hosted-runner disconnect for the MI325 8-GPU pool. The closest active CI-stability PR is #24164 "[CI] Enable venv in CI install dep script" — unrelated mechanism. A new branch fix/ci-runner-stability-fixes exists but has no open PR yet.

Suggested Triage Steps

• First, retry the nightly-8-gpu-glm51-rocm720 job (re-run failed jobs from run 25225581444) on a fresh runner. If it passes, this was a transient infra event — close out.
• If it reproduces on retry, instrument the runner: pull container/host dmesg, kubelet pod-eviction events, and ROCm SMI logs for linux-mi325-8gpu-sglang-bjvd5-runner-wlrc9 between 18:30–18:55 UTC, and the analogous window for the sister minimax-m27 runner (…-bjtrl).
• Cross-check whether other jobs scheduled onto the linux-mi325-8gpu-sglang-bjvd5-… runnerset over the next 24 h hit the same exit-255 pattern; if yes, escalate to runner-pool/IT.
• If the perf step is implicated specifically (passes on retry only when step 6 is skipped), bisect 7bb7f604..ece8a1a7 using just python3 run_suite.py --hw amd --suite nightly-perf-8-gpu-glm51; start with 108bfd8b (aiter MoE refactor).

Status

• Persistence: First observed today (2026-05-01 18:55 UTC). Previous 4 completed scheduled runs (Apr 26 → Apr 30) all passed.
• Scope: 1 job in this workflow. Sister job nightly-8-gpu-minimax-m27-rocm720 on a peer runner pod of the same runnerset had the same exit code 255 failure — possibly correlated infra event, but other jobs on the same MI325 runnerset (nightly-8-gpu-deepseek-v32-rocm720, nightly-8-gpu-kimi-k25-rocm720, etc.) completed successfully.
• Blocked work: Tonight's GLM-5.1 perf signal on AMD ROCm 7.2 (no perf data captured); accuracy signal is intact (step 5 ✅).
• In-flight fix: ❌ none found in open PRs — needs new PR or investigation.

nightly-accuracy-8-gpu-mi35x-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU GPT-OSS)

Commit Info

• sglang: ece8a1a7
• aiter: a6bb49937 (source: dev image — v0.1.12.post2.dev150+ga6bb49937 detected, rebuild skipped; matches Dockerfile default a6bb499375849eec45d68c5ccaebc8865fd422c0)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_gpt_oss_eval_mi35x.py	test_gpt_oss_accuracy	AssertionError: Failed models: ['openai/gpt-oss-20b', 'openai/gpt-oss-120b'] (server failed to launch — RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes. pre_model_load_memory=6.73828125, local_gpu_memory=8.44140625)	Log

test_qwen3_coder_next_eval_mi35x.py was scheduled but never ran — the job exited after the first test failure.

Failure Summary

The single test in test_gpt_oss_eval_mi35x.py (test_gpt_oss_accuracy) failed because both server launches (openai/gpt-oss-20b and openai/gpt-oss-120b, both --tp 8) crashed during init_torch_distributed. On both attempts, TP rank 1 raised RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes. pre_model_load_memory=6.73828125, local_gpu_memory=8.44140625, local_gpu_memory * 0.9=7.597265625 while the other 7 ranks completed init normally with mem usage=10.3–10.4 GB. The whole test ran in 84s vs ~439s for yesterday's passing run.

Regression Status

New regression (1-day) — most likely runner-state issue rather than a code change.

Recent history of test_gpt_oss_eval_mi35x.py (test_gpt_oss_accuracy) in job nightly-accuracy-8-gpu-mi35x-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
May 1	25225581444	nightly-accuracy-8-gpu-mi35x-rocm720	❌ Failed	test_gpt_oss_accuracy	TP1 RuntimeError: memory capacity is unbalanced
Apr 30	25180829975	nightly-accuracy-8-gpu-mi35x-rocm720	✅ Passed	—	(439s, 2/2 passed)
Apr 29	25125219864	nightly-accuracy-8-gpu-mi35x-rocm720	✅ Passed	—	—
Apr 27	25010858144	nightly-accuracy-8-gpu-mi35x-rocm720	✅ Passed	—	—
Apr 26	24962968926	nightly-accuracy-8-gpu-mi35x-rocm720	❌ Failed	(different — prior failure mode, not this error)	—

First observed failure (this error pattern) for this test file: 2026-05-01 19:09 UTC. Last known passing date: 2026-04-30. Apr 28 run was cancelled.

Failure Cluster

TP rank memory imbalance during init_torch_distributed on MI35X 8-GPU runner

Facts (verified observations, no interpretation)

• TP1 is the only rank that fails. Logs show TP0/TP2-TP7 each completed Init torch distributed ends. elapsed=17.xx s, mem usage=10.3-10.4 GB, then TP1's scheduler raises the imbalance check (log:1999-2016, log:2229-2246).
• Stack trace top: model_runner.py:1231 raising the msg after the check if pre_model_load_memory < local_gpu_memory * 0.9 (verified at python/sglang/srt/model_executor/model_runner.py:1227-1232).
• Numbers: pre_model_load_memory=6.74 GB, local_gpu_memory=8.44 GB on TP1. Other 7 ranks reported ~10.3-10.4 GB mem usage. The check fires because rank 1 has materially MORE free memory than the other ranks consumed → asymmetric init across ranks.
• Same exact failure on both gpt-oss-20b and gpt-oss-120b launches in the same job → the per-rank state persisted across both server attempts.
• aiter version unchanged (a6bb49937 dev build) between the Apr 30 pass and May 1 fail (verified from [CI-AITER-CHECK] markers in both logs).
• Job ran in 84s vs ~439s for the Apr 30 passing run.
• SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK env-flag controls hard-fail vs warn (model_runner.py:1231); job is configured with the check ON since the error is fatal.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Runner infrastructure — leftover process holding ~2 GB on one of the 8 GPUs assigned to this job
  • Supporting: only TP1 is unhealthy; numbers show one GPU (the one rank 1 was assigned) has ~2 GB more free memory than the others used at this point; the env check message itself names this as the prototypical cause; identical symptom on both gpt-oss-20b and gpt-oss-120b launches in the same container indicates persistent runner state, not model-specific; aiter and code are essentially unchanged from the Apr 30 pass for the failing call path.
  • Disconfirming: the asymmetry is "more free on TP1, less on others" — the typical "leftover process" pattern would be the opposite (TP1 has less free). Possible alternate reading: TP1 booted faster and other ranks raced ahead in allocation; but the ~2 GB delta is too large for normal init noise.
  • How to verify: rocm-smi on the runner before/after job; check for orphaned python / sglang processes on the runner host; rerun the same SHA on the same runner pool to see if it reproduces.
• [LOW] Code change in regression window altering early init memory accounting on AMD
  • Supporting: window is 7bb7f604..ece8a1a7; ece8a1a7 (#24197) refactors device timer and metrics; d48095ba (#24190) modifies torch.cuda.use_mem_pool / SymmetricMemoryContext.
  • Disconfirming: neither commit touches init_torch_distributed, get_available_gpu_memory, or the pre_model_load_memory accounting in model_runner.py:1220-1235 (verified git log on file in window). The imbalance check itself is unchanged. Failure is a race between ranks, not a deterministic accounting drift, so a code-side regression would also reproduce on subsequent reruns of the same SHA.
  • How to verify: rerun this job on ece8a1a7. If it passes on rerun without code changes, hypothesis 1 is favored over this one.
• [LOW] aiter dev-build accounting drift
  • Supporting: aiter is a .dev150+ga6bb49937 build (not pinned to Dockerfile-default-rebuilt clean state).
  • Disconfirming: the same dev-aiter image was used Apr 30 with this test passing (same [CI-AITER-CHECK] output).
  • How to verify: not actionable until other hypotheses ruled out.

In-flight Fix Check

• Searched repo:sgl-project/sglang is:pr is:open for memory capacity unbalanced, gpt_oss_eval_mi35x, mi35x nightly — no PR matches this specific failure.
• In-flight fix: ❌ none found — likely runner state; if it reproduces, needs investigation.

Suggested Triage Steps

• Re-run this job on the same SHA (ece8a1a7) and observe whether TP1 still hits the imbalance check. If it passes on rerun → confirms runner-state hypothesis; coordinate with runner ops to drain/clean the MI35X-8GPU pool.
• If it reproduces deterministically: bisect 7bb7f604..ece8a1a7 (38 commits) — start with git revert d48095ba (#24190 — torch.cuda.use_mem_pool / SymmetricMemoryContext change is the only commit touching CUDA mem-pool init in the window) and re-run.
• On the runner host, capture rocm-smi --showpids and lsof /dev/kfd before next nightly cron to check for orphaned processes holding GPU memory.
• If runner-side cause confirmed and recurrent, consider tightening the cleanup step in nightly-test-amd-rocm720.yml between jobs (kill stray python processes, drop pagecache).

Status

• Persistence: 1 day (since 2026-05-01 19:09 UTC) — first occurrence in this error pattern; previous 3 completed runs (Apr 27/29/30) passed for this test file.
• Scope: 1 job in 1 workflow (nightly-test-amd-rocm720.yml / nightly-accuracy-8-gpu-mi35x-rocm720); also blocked the second test (test_qwen3_coder_next_eval_mi35x.py) which never ran.
• Blocked work: nightly MI35X-ROCm7.2 GPT-OSS + Qwen3-Coder-Next accuracy signal.
• In-flight fix: ❌ none found — likely runner state; if it reproduces, needs investigation.

nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2)

Commit Info

• sglang: ece8a1a7
• aiter: a6bb4993 (v0.1.12.post2.dev150+ga6bb49937, source: Dockerfile default — preserved as dev/patched build)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	AssertionError: Failed models: ['deepseek-ai/DeepSeek-V3.2 (basic)'] (server crashed during CUDA-graph capture, accuracy step never ran)	Log

Failure Summary

The DeepSeek-V3.2 (basic) server (--nsa-prefill-backend tilelang --nsa-decode-backend tilelang --tp 8) crashed during CUDA-graph capture. All 8 ROCm GPUs reported Memory access fault by GPU node-N simultaneously while capturing batches at bs=136 (CUDA graph batch ladder progressing 512→136). The scheduler aborted (exit code -6), _wait_for_scheduler_ready raised EOFError, and the test recorded the model as failed.

Regression Status

New regression with this specific signature (GPU memory access fault during graph capture).

Recent history of test_deepseek_v32_eval_mi35x.py::test_deepseek_v32_accuracy in nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 (same branch=main, event=schedule, completed runs only):

Date	Run	Test File Status	Failed Function	Error
May 1	25225581444	❌ Failed	test_deepseek_v32_accuracy	GPU memory access fault during CUDA-graph capture (bs=136); scheduler exit -6
Apr 30	25180829975	✅ Passed	—	—
Apr 29	25125219864	✅ Passed	—	—
Apr 28	25069314464	⚠ run cancelled	—	not comparable
Apr 27	25010858144	❌ Failed	test_deepseek_v32_accuracy	Different symptom: server reached eval; accuracy=0.925 threshold=0.93 ❌ FAIL (no GPU fault, no graph-capture crash)
Apr 26	24962968926	✅ Passed	—	—

Apr 27's failure was an accuracy-threshold miss, not the graph-capture crash; it is not the same regression and was already cleared by Apr 29/30 passes. The crash signature is first observed today (May 1). Last known passing date with this same matrix: Apr 30, sglang 7bb7f604.

Failure Cluster

GPU memory access fault during CUDA-graph capture for DeepSeek-V3.2 NSA tilelang backend (multi-rank simultaneous fault).

Facts (verified observations, no interpretation)

• All 8 ranks faulted simultaneously: Memory access fault by GPU node-2/3/4/5/6/7/8/9 (Agent handle: 0x...) on address 0x... Reason: Unknown. (log lines 20781–20788)
• The faults occurred exactly at the Capturing batches (bs=136 avail_mem=36.05 GB) step (52 capture sizes total, faulted at index 32/52 — i.e. graph capture had succeeded for bs ∈ {512,496,…,144} and faulted at bs=136).
• Immediately preceding log lines (20773–20780) are [aiter] shape is M:136, N:64, K:7168 dtype='torch.bfloat16' otype='torch.bfloat16' bias=False, scaleAB=False, bpreshuffle=False, not found tuned config in /tmp/aiter_configs/bf16_tuned_gemm.csv, will use default config! using torch solution:0 (one per rank). Note: log proximity is not causal proof.
• Crash propagation: Fatal Python error: Aborted → RuntimeError: Rank 0 scheduler died during initialization (exit code: -6) (lines 20791, 21469).
• Capture stack (unrelated to fault address but identifies the code path): init_device_graphs (model_runner.py:2792) → __init__ (cuda_graph_runner.py:714) → capture (cuda_graph_runner.py:896) → _capture_one_stream (884) → capture_one_batch_size (1165) → run_once (1143) → _forward_tilelang (nsa_backend.py:1862) → tilelang_sparse_fwd (tilelang_kernel.py:1363) (log lines 18608–18613, 18707–18708).
• aiter version verified from [CI-AITER-CHECK] markers: Dockerfile expects AITER_COMMIT=a6bb499375849eec45d68c5ccaebc8865fd422c0; AITER version inside CI image: v0.1.12.post2.dev150+ga6bb49937; Dev/patched version detected … skipping rebuild (log lines 1745–1747).
• Apr 30 passing run had Dockerfile expects AITER_COMMIT=v0.1.12.post1 (verified from commit history — sglang 7e17a7a1 is the bump landing May 1 01:26 PT).
• Regression window has 39 sglang commits in 7bb7f6049a..ece8a1a78. Of those, only 3 touch attention/aiter/MoE: 7e17a7a1 (aiter Dockerfile bump), 108bfd8b (Aiter MoE runner refactor), da7f8907 (Intel-XPU only, irrelevant for AMD).
• aiter delta v0.1.12.post1...a6bb499 = 150 aiter commits, including several gfx950-targeted kernel changes (Add native qh32 qseqlen2 MLA PS kernel for gfx950 fp8 decode, rm cu=80 configs and update dsv3 bpreshuffle configs, [OPUS] mma step_k …, Introduce 16x128 fused moe kernel with weights prefetch, etc.).
• Many other AMD jobs failed in the same nightly run (nightly-perf-8-gpu-mi35x-deepseek-v32-basic-rocm720, nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8-rocm720, nightly-8-gpu-mi35x-glm51-rocm720, nightly-8-gpu-mi35x-qwen35-rocm720, nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720, nightly-accuracy-8-gpu-mi35x-rocm720, etc.) — same-day blast radius is large but per-symptom comparison was not performed for each.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] aiter bump from v0.1.12.post1 → a6bb499 via sglang commit 7e17a7a1 (#24218)

  • Supporting: This is the only commit in the regression window that changes runtime kernel binaries on AMD; the new aiter delta has 150 commits including gfx950 (MI35X) kernel changes (e.g. ROCm/aiter #2676 "Add native qh32 qseqlen2 MLA PS kernel for gfx950 fp8 decode", #2683 "rm cu=80 configs and update dsv3 bpreshuffle configs"); aiter GEMM is the immediate predecessor log line before the fault on every rank; the same aiter version is loaded by many other today-failing AMD jobs (large blast radius consistent with a shared dependency change).
  • Disconfirming: Per CLAUDE.md two-variable trap, log-line proximity is not causation — the fault could equally be in the next sglang call (NSA _forward_tilelang at nsa_backend.py:1862 or graph-capture machinery in cuda_graph_runner.py). The aiter call shown is using torch solution:0 (i.e. it falls back to the torch reference path, not an aiter custom kernel), which weakens "aiter kernel did the bad write" — though aiter import-time hooks/registrations could still affect the GPU context. No Baseline A/B is available because this is the regular nightly, not an aiter-scout run.
  • How to verify: Re-run the same job with AITER_COMMIT_OVERRIDE=v0.1.12.post1 (rebuild aiter to old version) on the same sglang SHA. If the crash disappears, aiter is the cause and bisect within the 150-commit aiter delta. If the crash persists, sglang is the cause and bisect the 39-commit sglang delta.

• [LOW] sglang 108bfd8b "[MoE] Add Aiter MoE runner backend and purge aiter.fused_moe from quant methods" (#23597)

  • Supporting: Refactors how MoE quantization paths invoke aiter, lands in the regression window; DeepSeek-V3.2 is a MoE model and would exercise these code paths.
  • Disconfirming: Crash is during CUDA-graph capture inside the NSA tilelang attention path, not a MoE forward; no MoE call in the captured stack trace immediately preceding the fault; the failing graph batch sizes ≤136 are decode-shaped (attention-heavy), not the larger prefill batches that exercised MoE successfully earlier in the capture loop.
  • How to verify: Check git revert 108bfd8b on a branch and re-run the test on AMD MI35X with the same aiter override to isolate.

• [LOW] Other 36 sglang commits (HiCache, LoRA, PD-disagg, spec-decoding, CI plumbing, XPU, MUSA, refactors)

  • Supporting: They land in the regression window.
  • Disconfirming: None touch NSA/tilelang/CUDA-graph/aiter paths reachable from this test; many are pure CI / non-AMD device changes.
  • How to verify: Default fallback only if both hypotheses above are disconfirmed; bisect the 39-commit window.

No hypothesis reaches HIGH confidence without the aiter override A/B run above.

In-flight Fix Check

Searched repo:sgl-project/sglang is:pr is:open aiter a6bb499, … deepseek v32 memory fault, … aiter revert, … nsa tilelang:

• In-flight fix: ❌ none found in open PRs targeting the aiter a6bb499 bump or the DeepSeek-V3.2 NSA tilelang graph-capture crash — needs new investigation/PR.
• Adjacent open PR (not a fix for this): #23581 "[AMD] Default SGLANG_USE_AITER_AR to false to avoid HIP graph capture invalidation" — same area (graph capture + aiter on AMD) but different symptom (allreduce invalidation, not memory fault); worth flagging to the same reviewers.

Suggested Triage Steps

• Re-run nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720 once with AITER_COMMIT_OVERRIDE=v0.1.12.post1 on the same sglang SHA ece8a1a7 — this is the cheapest aiter A/B and immediately discriminates the MEDIUM hypothesis.
• If aiter is exonerated by the A/B, bisect the sglang 39-commit window 7bb7f604..ece8a1a7, starting with 108bfd8b (Aiter MoE runner refactor) since it's the only other AMD-touching commit.
• If aiter is implicated, bisect the 150-commit aiter delta v0.1.12.post1..a6bb499, prioritising gfx950-targeted kernel commits (#2676, #2683, #2696, #2700).
• Cross-check by inspecting whether the other today-failing MI35X jobs (e.g. nightly-perf-8-gpu-mi35x-deepseek-v32-basic-rocm720, nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8-rocm720, nightly-8-gpu-mi35x-glm51-rocm720) show the same Memory access fault … Capturing batches signature; broad cross-test reproduction would strengthen the aiter-bump hypothesis (shared dependency), narrow per-test failures would point at sglang.
• If the issue can be reproduced locally on an MI35X box, set HSA_ENABLE_SDMA=0 and AMD_LOG_LEVEL=3 to capture which kernel/op is the source of the bad address 0x...e00000.

Status

• Persistence: 1 completed run (today, 2026-05-01); previous nightly (Apr 30) passed this exact test.
• Scope: This job (nightly-accuracy-8-gpu-mi35x-deepseek-v32-rocm720) confirmed; other MI35X / MI300X jobs in the same nightly run also failed but per-symptom verification was not performed.
• Blocked work: Blocks DeepSeek-V3.2 accuracy signal on MI35X with NSA tilelang backend; potentially blocks broader AMD nightly signal if the aiter bump is the cause.
• In-flight fix: ❌ none found — needs new PR or investigation; recommend the aiter A/B re-run above first.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-05-02 20:25 UTC)

[amd-bot]

nightly-test-amd-rocm720.yml — Per-Job Analysis (continued 3/6)

nightly-perf-8-gpu-mi35x-deepseek-v32-basic-rocm720 — failed step(s): Performance Test MI35x ROCm 7.2 (8-GPU DeepSeek-V3.2 Basic)

Commit Info

• sglang: ece8a1a7
• aiter: a6bb499 (v0.1.12.post2.dev150+ga6bb49937, source: Dockerfile default — bumped today by #24218)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/perf/mi35x/test_deepseek_v32_basic_perf_mi35x.py	test_bench_one_batch (TestNightlyDeepseekV32BasicPerformance)	Exception: Server process exited with code 1 (root: 8× Memory access fault by GPU node-{2..9} during cuda-graph capture → Fatal Python error: Aborted → RuntimeError: Rank 0 scheduler died during initialization (exit code: -6))	Log L21556 · fault L20862

Failure Summary

DeepSeek-V3.2 server with TP=8, NSA backend (tilelang) crashed with simultaneous Memory access fault by GPU node-{2..9} on all 8 ranks during cuda-graph capture at Capturing batches (bs=136 avail_mem=36.05 GB) 31/52. All ranks aborted with exit code -6 (SIGABRT), causing popen_launch_server to fail with Server process exited with code 1. Total elapsed before crash: ~36 min (8 × 5-min "soft watchdog" notices fired during warm-up; the crash itself happened at the very end after capture progressed normally up to bs=136).

Regression Status

New regression — first failing run today, last 5 completed scheduled runs of the same job all passed.

Recent history of test_deepseek_v32_basic_perf_mi35x.py::test_bench_one_batch in job nightly-perf-8-gpu-mi35x-deepseek-v32-basic-rocm720 (workflow nightly-test-amd-rocm720.yml, branch main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
May 01	25225581444	nightly-perf-…-deepseek-v32-basic-rocm720	❌ Failed	test_bench_one_batch	GPU mem fault during graph capture
Apr 30	25180829975	nightly-perf-…-deepseek-v32-basic-rocm720	✅ Passed	—	—
Apr 29	25125219864	nightly-perf-…-deepseek-v32-basic-rocm720	✅ Passed	—	—
Apr 27	25010858144	nightly-perf-…-deepseek-v32-basic-rocm720	✅ Passed	—	—
Apr 26	24962968926	nightly-perf-…-deepseek-v32-basic-rocm720	✅ Passed	—	—
Apr 25	24936773213	nightly-perf-…-deepseek-v32-basic-rocm720	✅ Passed	—	—

First observed failure: 2026-05-01 18:58 UTC. Last passing: 2026-04-30 18:02 UTC.

Failure Cluster

GPU memory access fault during DeepSeek-V3.2 NSA cuda-graph capture (a8w8_blockscale_bpreshuffle GEMM call site).

Facts (verified observations, no interpretation)

• 8 simultaneous Memory access fault by GPU node-{2..9} (Agent handle: 0x...) on address 0x... Reason: Unknown. lines at log L20862-20871, then Fatal Python error: Aborted on every rank, then L21556: RuntimeError: Rank 0 scheduler died during initialization (exit code: -6).
• Crash happened during cuda-graph capture: progress bar reached Capturing batches (bs=136 avail_mem=36.05 GB) 60% 31/52 immediately before the fault.
• Top of the faulthandler stack on the live thread (deinterleaved from L20862-21000):
  • cuda_graph_runner.py:884 _capture_one_stream → cuda_graph_runner.py:896 capture (init_device_graphs)
  • → deepseek_v2.py:2312 forward_hip → forward_mla.py:215 forward_absorb_prepare / 2102 forward_absorb_core
  • → nsa_indexer.py:390 forward_cuda → nsa_indexer.py:884 forward → nsa_indexer.py:1289 _get_q_k_bf16
  • → linear.py:741 forward → fp8.py:432 apply → fp8_utils.py:763 aiter_w8a8_block_fp8_linear
  • → aiter/ops/gemm_op_a8w8.py:307 gemm_a8w8_blockscale → aiter/ops/triton/gemm/basic/gemm_a8w8_blockscale.py:154 gemm_a8w8_blockscale_bpreshuffle
• aiter version changed across the regression: v0.1.12.post1 (last passing, Apr 30 log L1741) → a6bb499 (failing). Confirmed via [CI-AITER-CHECK] markers in both logs.
• sglang delta 7bb7f604..ece8a1a7 = 39 commits.
• aiter delta v0.1.12.post1..a6bb499 = 150 commits (status: ahead, verified via GitHub compare API). Includes multiple commits touching the exact failing call site gemm_a8w8_blockscale_bpreshuffle: aiter #2683 "rm cu=80 configs and update dsv3 bpreshuffle configs", aiter #2811 "Fix bpreshuffle compile failure due to stale CShuffleEpilogueProblem arg", aiter #2896 "retune a8w8_blockscale_bpreshuffle_gemm".
• Soft watchdog timeouts (self.soft=True) every ~5 min during model load are expected for this test (server takes >30 min to initialize fully); they are not the failure trigger — capture progressed normally up to bs=136 after them.
• Same mem_fraction_static=0.85, same TP=8, same NSA backend across passing and failing runs (verified from server_args= line in both logs); no ServerArgs change.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] aiter bump in #24218 (7e17a7a1) introduced a regression in gemm_a8w8_blockscale_bpreshuffle for the NSA-indexer FP8 shapes used by DeepSeek-V3.2 on gfx950 / MI35x.

  • Supporting:
    • One-line code-level change: AITER_COMMIT_DEFAULT="v0.1.12.post1" → "a6bb499..." in docker/rocm.Dockerfile (4 stanzas, no other code change).
    • Temporal correlation is exact: failure starts the first scheduled run after this commit lands; previous 5 scheduled runs (using post1) all passed.
    • The faulting kernel gemm_a8w8_blockscale_bpreshuffle is exactly what the aiter delta retuned/refactored (#2683, #2811, #2896).
    • sglang side has no commits in the 39-commit delta touching nsa_indexer.py, forward_mla.py, fp8_utils.py, cuda_graph_runner.py, or related call sites (verified via git log --oneline 7bb7f604..ece8a1a7).
  • Disconfirming:
    • 150 aiter commits is a huge surface — the bump may also pull in MLA/PA/MoE kernel rewrites that interact with NSA paths in non-obvious ways; cannot pin to a single aiter commit without bisecting aiter.
    • The aiter delta is shared across ALL AMD targets (MI300x, MI325x, MI35x, gfx942/gfx950) — if gemm_a8w8_blockscale_bpreshuffle were broken in general we would expect failures in many other AMD jobs today; sister jobs need to be checked to confirm this is MI35x-specific.
  • How to verify:
    1. Revert #24218 on a branch (one-line Dockerfile change), rebuild MI35x image, re-run nightly-perf-8-gpu-mi35x-deepseek-v32-basic-rocm720. Expected: passes.
    2. If revert passes, bisect the aiter range v0.1.12.post1..a6bb499 focusing on commits touching gemm_a8w8_blockscale_bpreshuffle* (e.g. start with #2811, #2683, #2896).
    3. Cross-check today's other MI35x / gfx950 nightly jobs for the same gemm_a8w8_blockscale_bpreshuffle fault — broad presence supports an aiter-side root cause; isolation to NSA/DSv32 suggests an interaction with specific tensor shapes.

• [LOW] Resource issue (memory leak / fragmentation) hitting at bs=136.

  • Supporting: avail_mem=36.05 GB reported just before crash; cuda-graph captures multiple batch sizes which can fragment memory.
  • Disconfirming: 8 simultaneous faults across all GPUs at the same address-range pattern is not a typical OOM signature; faults say Reason: Unknown (not out of memory); same mem_fraction_static=0.85 worked yesterday on the same hardware. No dmesg/OOM-killer evidence in log.
  • How to verify: only investigate if hypothesis above is disproven.

In-flight Fix Check

Searched: aiter+a6bb499 (0 results), revert+aiter (5 results, none match the bump), mi35x+deepseek+v32+regression (3 results, all closed/old, none address this bump), deepseek+v32+mi35 open PRs (10 results, none are revert/pin of #24218).

• In-flight fix: ❌ none found in open PRs — needs new PR (revert [#24218](https://github.com/sgl-project/sglang/pull/24218)) or a forward-fix in aiter.

Suggested Triage Steps

• Revert sglang #24218 on a branch (one-line AITER_COMMIT_DEFAULT="v0.1.12.post1" restore), trigger a one-off nightly-test-amd-rocm720.yml schedule replay, and confirm nightly-perf-8-gpu-mi35x-deepseek-v32-basic-rocm720 recovers.
• Concurrently scan today's other AMD nightly/PR jobs for the same "Memory access fault" + gemm_a8w8_blockscale_bpreshuffle stack trace (suggested grep keys: Memory access fault, gemm_a8w8_blockscale_bpreshuffle, nsa_indexer). Broad presence ⇒ aiter-side bug; MI35x-only ⇒ likely a gfx950-specific tuning regression in the bump.
• If the regression is confirmed by the revert: open an issue against ROCm/aiter referencing the failing shapes (NSA _get_q_k_bf16, FP8 W8A8 blockscale bpreshuffle) and bisect the 150-commit window starting with the bpreshuffle-touching PRs (#2683, #2811, #2896).
• Until a fix lands, consider pinning the MI35x Dockerfile aiter back to v0.1.12.post1 to keep nightly perf signal green (decision for maintainers).

Status

• Persistence: 1 completed run (today, 2026-05-01); previous 5 completed scheduled runs of this job passed. First observed failure 2026-05-01 18:58 UTC.
• Scope: Confirmed in 1 job today (nightly-perf-8-gpu-mi35x-deepseek-v32-basic-rocm720); broader AMD MI35x / gfx950 fan-out unverified — see Suggested Triage Steps.
• Blocked work: DeepSeek-V3.2 MI35x ROCm 7.2 nightly perf signal.
• In-flight fix: ❌ none found in open PRs — needs new PR (revert #24218) or a forward-fix in aiter.

nightly-8-gpu-mi35x-glm51-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU GLM-5.1 NSA)

Commit Info

• sglang: ece8a1a7
• aiter: a6bb499375849eec45d68c5ccaebc8865fd422c0 (source: Dockerfile default; dev image v0.1.12.post2.dev150+ga6bb49937 preserved per [CI-AITER-CHECK])

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_glm51_eval_mi35x.py	TestGLM51EvalMI35x.test_glm51_accuracy	AssertionError: Failed models: ['zai-org/GLM-5.1-FP8 (nsa)'] (server failed to start: RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes. pre_model_load_memory=6.73828125, local_gpu_memory=8.44140625, local_gpu_memory * 0.9=7.597265625)	Log

Failure Summary

The single test in test_glm51_eval_mi35x.py failed because the SGLang server for zai-org/GLM-5.1-FP8 (nsa) could not initialise: TP1's scheduler raised RuntimeError: The memory capacity is unbalanced from model_runner.py:1231 during init_torch_distributed(). The driver test then reported Failed models: ['zai-org/GLM-5.1-FP8 (nsa)'] and exited with code 1.

Regression Status

New regression (first observed in this run; 4 prior consecutive scheduled runs of the same job passed).

Recent history of test/registered/amd/accuracy/mi35x/test_glm51_eval_mi35x.py (test_glm51_accuracy) in job nightly-8-gpu-mi35x-glm51-rocm720 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
May 1	25225581444	nightly-8-gpu-mi35x-glm51-rocm720	❌ Failed	test_glm51_accuracy	RuntimeError: memory capacity is unbalanced → AssertionError
Apr 30	25180829975	nightly-8-gpu-mi35x-glm51-rocm720	✅ Passed	—	—
Apr 29	25125219864	nightly-8-gpu-mi35x-glm51-rocm720	✅ Passed	—	—
Apr 27	25010858144	nightly-8-gpu-mi35x-glm51-rocm720	✅ Passed	—	—
Apr 26	24962968926	nightly-8-gpu-mi35x-glm51-rocm720	✅ Passed	—	—

(Last known passing date for this test file: 2026-04-30. First observed failure: 2026-05-01. Apr 28 and Apr 23 runs are excluded — they were cancelled. Workflow is not amd-aiter-scout.yml, so Failure Origin section is omitted.)

Failure Cluster

TP memory imbalance check raises during init_torch_distributed (server warmup OOM-class)

Facts (verified observations, no interpretation)

• Verbatim error from log line 2053: RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes. pre_model_load_memory=6.73828125, local_gpu_memory=8.44140625, local_gpu_memory * 0.9=7.597265625. The 6.74 GB value is the min across TP ranks (call site uses distributed=True); the local rank reported 8.44 GB free → ~1.71 GB skew between ranks (~20%).
• Stack: scheduler.py:3918 → scheduler.py:432 → scheduler.py:693 → scheduler.py:648 → tp_worker.py:260 → tp_worker.py:345 → model_runner.py:481 → model_runner.py:1231 (raised by the SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK = EnvBool(True) branch at python/sglang/srt/model_executor/model_runner.py:1227-1231).
• Failure happens in TP1 only; the other 7 TP ranks reach Init torch distributed ends successfully (log lines 2029-2035). Then sigquit propagates and the parent test asserts.
• Free GPU memory of ~8 GB on a linux-mi35x-gpu-8 runner (MI355x = 256+ GB HBM3e per GPU) is anomalously low — strongly suggests memory was held by another process before sglang launched. ([CI-AITER-CHECK] markers do not show any pre-launch sglang activity in this job.)
• AITER version is unchanged across pass/fail runs (Dockerfile default a6bb499375849eec45d68c5ccaebc8865fd422c0, dev image v0.1.12.post2.dev150+ga6bb49937) — no AITER_COMMIT_OVERRIDE set.
• Test script CLI: tp 8, mem-fraction-static 0.80, attention_backend=nsa, nsa_prefill_backend=tilelang, nsa_decode_backend=tilelang — same as previous passing runs (no PR in window changes these defaults for GLM-5.1).
• 39 sglang commits in the regression window 7bb7f604..ece8a1a7.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Runner-side resource contention / leftover GPU process held memory on at least one of the 8 MI355x GPUs at warmup time, tripping the 10%-skew memory-balance check.

  • Supporting: free memory ~8 GB on a 256 GB GPU is two orders of magnitude below expected → a non-sglang process (or a stale prior-job process) is holding memory; the check raised is exactly the one designed to catch this scenario; only TP1 was below the threshold, consistent with one specific GPU (not a global config change) being contended.
  • Disconfirming: PR #23314 (ci: limit nightly test parallelism to 1 job per hardware type) is in the regression window — if effective, it should reduce contention, not cause it; the bot did not verify whether concurrency was actually limited at the time of this run.
  • How to verify: re-run the job and check whether free memory at warmup is again low; SSH to the runner and rocm-smi --showpids / rocm-smi --showmeminfo to identify holders; review GitHub runner concurrency logs around 2026-05-01 19:31-19:42 UTC for overlapping jobs on linux-mi35x-gpu-8.

• [LOW] A code change in window 7bb7f604..ece8a1a7 introduced an early allocation that skews per-rank free memory. Candidates touching scheduling/memory/distributed init: ece8a1a7 (device timer + metrics refactor #24197), d48095ba (bypass torch.cuda.use_mem_pool CM in SymmetricMemoryContext #24190), 108bfd8b (Aiter MoE runner backend #23597).

  • Supporting: temporal correlation only.
  • Disconfirming: the symptom is imbalance between ranks, not high overall consumption — a sglang code change would normally consume similarly across all 8 ranks; the failed rank (TP1) actually reports more free memory than the min, ruling out an extra allocation by sglang on TP1 itself; check defaults to True in both passing and failing runs (verified at environ.py:212); TP-7 also reach Init torch distributed ends cleanly.
  • How to verify: bisect 7bb7f604..ece8a1a7 only after re-runs reproduce the failure; if the next nightly is green, this hypothesis is effectively falsified.

• [LOW] AITER-related allocation skew.

  • Supporting: aiter is loaded (module_aiter_core.so at log line 1837) before the imbalance check.
  • Disconfirming: aiter SHA is unchanged from previous (passing) runs; no override set.
  • How to verify: not actionable.

No hypothesis above MEDIUM confidence; environmental cause is the most plausible single explanation but cannot be confirmed without runner-side evidence.

In-flight Fix Check

Searched open sglang PRs for keywords memory capacity unbalanced, SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK, glm51 nightly:

• In-flight fix: ❌ none found — no open PR addresses the TP memory-imbalance check or the nightly-8-gpu-mi35x-glm51-rocm720 job. (#24195 "Fix flashinfer autotune oom glm51" is unrelated — it targets flashinfer autotune OOM, not the pre-load TP memory-balance check; the failing test uses attention_backend=nsa, not flashinfer.)

Suggested Triage Steps

• Re-run the job once to disambiguate flake vs. persistent regression; if green, the failure was almost certainly runner contention.
• If the next nightly fails again with the same symptom: SSH to the runner shortly after a failure and run rocm-smi --showpids --showmeminfo vram to identify the process holding memory before sglang launches.
• If the imbalance recurs, consider whether cleanup_processes_amd_ci.sh-style runner pre-job cleanup is being invoked for this job, and whether PR #23314's parallelism limit is actually in effect on the linux-mi35x-gpu-8 pool.
• As a temporary mitigation only (not a fix): setting SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 would downgrade the error to a warning and let the run proceed, but masks the underlying contention — this is a maintainer call, not a recommendation.
• If symptom persists across reruns and runner cleanup is verified clean: bisect 7bb7f604..ece8a1a7 starting with d48095ba and ece8a1a7.

Status

• Persistence: Failing for 1 run only (first observed 2026-05-01 19:41 UTC); 4 prior consecutive scheduled runs (Apr 26 → Apr 30) passed.
• Scope: 1 job (nightly-8-gpu-mi35x-glm51-rocm720), 1 workflow (nightly-test-amd-rocm720.yml), 1 test file. Other ROCm 7.2 nightly jobs in the same run are not in scope of this analysis.
• Blocked work: Blocks GLM-5.1-FP8 + NSA accuracy signal on AMD MI355x ROCm 7.2 nightly.
• In-flight fix: ❌ none found in open PRs — needs new PR or investigation.

nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 — failed step(s): Accuracy Test + Performance Test MI35x ROCm 7.2 (8-GPU Qwen3-235B-MXFP4)

Commit Info

• sglang: ece8a1a7
• aiter: a6bb499375849eec45d68c5ccaebc8865fd422c0 (source: dev image — v0.1.12.post2.dev150+ga6bb49937 preserved from CI image, no override)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	Server failed to launch: RuntimeError: Rank 0 scheduler died during initialization (exit code: -6); underlying cause is HIP error: operation not permitted on an event last recorded in a capturing stream (hipErrorCapturedEvent) raised in c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() during CUDA graph capture	Log

Failure Summary

The Qwen3-235B-A22B-Instruct-2507-mxfp4 server (TP=4, EP=2, attention-backend=aiter) aborts during CUDA graph capture in setUpClass. All 4 ranks of both PG ID 5 and PG ID 6 emit [E ProcessGroupNCCL.cpp:2057] Process group watchdog thread terminated with exception: HIP error: operation not permitted on an event last recorded in a capturing stream (hipErrorCapturedEvent) from WorkNCCL::finishedGPUExecutionInternal → WorkNCCL::isCompleted → Watchdog::runLoop, then terminate called after throwing 'c10::DistBackendError' and SIGABRT (-6) propagates to the test runner as Server process exited with code 1.

Regression Status

Known recurring failure — failing every nightly-test-amd-rocm720.yml schedule run since at least 2026-04-11.

Recent history of test/registered/amd/test_qwen3_instruct_mxfp4.py in job nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
May 1	25225581444	nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720	❌ Failed	setUpClass	hipErrorCapturedEvent → exit -6
Apr 30	25180829975	...qwen3-235b-mxfp4-rocm720	❌ Failed	setUpClass	same
Apr 29	25125219864	...qwen3-235b-mxfp4-rocm720	❌ Failed	setUpClass	same
Apr 27	25010858144	...qwen3-235b-mxfp4-rocm720	❌ Failed	setUpClass	same
Apr 26	24962968926	...qwen3-235b-mxfp4-rocm720	❌ Failed	setUpClass	same
Apr 25	24936773213	...qwen3-235b-mxfp4-rocm720	❌ Failed	setUpClass	same
Apr 24	24903597060	...qwen3-235b-mxfp4-rocm720	❌ Failed	setUpClass	same
Apr 22	24793595557	...qwen3-235b-mxfp4-rocm720	❌ Failed	setUpClass	same
Apr 14	24414394983	...qwen3-235b-mxfp4-rocm720	❌ Failed	setUpClass	same
Apr 12	24287967867	...qwen3-235b-mxfp4-rocm720	❌ Failed	setUpClass	same
Apr 11	24256125686	...qwen3-235b-mxfp4-rocm720	❌ Failed	setUpClass	first observed hipErrorCapturedEvent
Apr 9	24204867080	...qwen3-235b-mxfp4-rocm720	✅ Passed	—	—
Apr 8	24150105138	...qwen3-235b-mxfp4-rocm720	✅ Passed	—	—
Apr 7	24095886332	...qwen3-235b-mxfp4-rocm720	✅ Passed	—	—

First observed failure: 2026-04-11 (run 24256125686, head 5cb4ea1d). Last known passing run: 2026-04-09 (run 24204867080, head 021bd77c). 21 consecutive scheduled-run failures since.

Failure Cluster

HIP graph capture invalidation: hipErrorCapturedEvent in NCCL watchdog during multi-GPU AMD CUDA-graph warmup

Facts (verified observations, no interpretation)

• Verbatim error: [rank3]:[E501 18:53:14.092525532 ProcessGroupNCCL.cpp:2057] [PG ID 5 PG GUID 23 Rank 1] Process group watchdog thread terminated with exception: HIP error: operation not permitted on an event last recorded in a capturing stream (job log, line ~2762).
• Search hint emitted: Search for `hipErrorCapturedEvent' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__HIPRT__TYPES.html.
• Stack trace top frames (verbatim): frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal(), frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted(), frame #5: c10d::ProcessGroupNCCL::Watchdog::runLoop(). The watchdog calls hipEventQuery on an event that was recorded in a stream that is currently in capture mode → HIP rejects with hipErrorCapturedEvent.
• All 4 TP ranks emit identical exception on both PG ID 5 and PG ID 6 within ~30 ms; final user-visible error: RuntimeError: Rank 0 scheduler died during initialization (exit code: -6) raised in engine.py:1243 _wait_for_scheduler_ready.
• Failure occurs during the warmup loop Capturing batches (bs=512 avail_mem=25.03 GB): 0%| | 0/52 (line 2705), well after weight loading.
• Server command (line 3239): sglang serve --model-path amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 --tp 4 --ep 2 --trust-remote-code --attention-backend aiter --device cuda --host 127.0.0.1 --port 11000.
• Environment: ROCm 7.2.0 (image rocm/sgl-dev:v0.5.10.post1-rocm720-mi35x-20260501), MI35x runner, gfx950, aiter v0.1.12.post2.dev150+ga6bb49937 (no override). AITER is unchanged across the Apr 9 → Apr 11 transition (CI uses the same [CI-AITER-CHECK] Dev/patched version detected path consistently).
• 41 sglang commits in regression window git log --oneline 021bd77c..5cb4ea1d (verified locally). Closest-by-keyword candidates: f652135d [sgl] fix using symmetric memory issues for attention_tp (#22286) (touches parallel_state.py, linear.py, model_runner.py); e77bfba2 Fix NCCL AllGather hanging issue for Qwen3 Next MTP (#22458) (touches speculative eagle paths, not exercised by this dense MoE test); 5cb4ea1d perf: enable inductor combo_kernels for horizontal fusion (#21977) (only active when compiler == "inductor", this test launches with default backend so probably inactive).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] ROCm 7.2.0 hipEventQuery race: cross-thread calls from the NCCL watchdog ignore THREAD_LOCAL capture mode, so a watchdog poll on an event recorded in a stream that another thread is currently capturing returns hipErrorCapturedEvent. The AITER custom all-reduce in v0.1.12.post1+ widens the race window because IPCBufferPool launches helper kernels on an internal stream during capture.
  • Supporting: PR #24151 ("[Docker][ROCm] Bump ROCm720 base image from 7.2.0 to 7.2.2 to fix hipEventQuery race") body says verbatim "ROCm 7.2.0 has a known hipEventQuery runtime bug — cross-thread calls ignore THREAD_LOCAL capture mode, so the NCCL watchdog can invalidate in-flight HIP graph captures"; this is exactly our stack (WorkNCCL::isCompleted → hipEventQuery). The companion PR #23581 describes the AITER custom all-reduce path that creates the captured-stream events. Symptom (hipErrorCapturedEvent), code path (ProcessGroupNCCL::WorkNCCL::isCompleted), workflow (multi-GPU AMD CUDA-graph capture with aiter backend), and platform (ROCm 7.2.0) all match.
  • Disconfirming: ROCm 7.2.0 is unchanged across the Apr 9 (passing) → Apr 11 (failing) transition (same image base), so the underlying race must have been latent and one of the 41 sglang commits in 021bd77c..5cb4ea1d widened the exposure window. Cannot pinpoint which commit without bisect.
  • How to verify: rebuild the rocm720 image with PR #24151 (ROCm 7.2.2) and rerun test_qwen3_instruct_mxfp4.py; OR set SGLANG_USE_AITER_AR=false (PR #23581) and rerun; either should clear hipErrorCapturedEvent.
• [LOW] sglang commit f652135d ("fix using symmetric memory issues for attention_tp", #22286)
  • Supporting: lands in regression window (Apr 10), touches parallel_state.py and model_runner.py which are on the all-reduce / TP path that this 4-GPU test exercises.
  • Disconfirming: diff is 12 lines and only changes attention_tp init/finalize call sequencing; the failing watchdog poll happens deep inside torch's NCCL backend, not in sglang code. Without ROCm 7.2.0's hipEventQuery race this commit alone almost certainly would not produce hipErrorCapturedEvent.
  • How to verify: git revert f652135d on a branch, rerun the test in container rocm/sgl-dev:v0.5.10.post1-rocm720-mi35x-20260501. If the failure persists, this commit is exonerated.
• [SPECULATION] sglang commit 5cb4ea1d ("perf: enable inductor combo_kernels for horizontal fusion", #21977)
  • Supporting: it is the boundary commit (head of first failing run), and combo-kernel fusion can change kernel-launch ordering inside graph capture.
  • Disconfirming: the change is gated on self.compiler == "inductor"; this test does not pass --enable-torch-compile or any inductor compiler flag (verified from server command in log), so the new inductor_config.combo_kernels branch should not execute. Boundary-commit position is coincidence.
  • How to verify: log inductor_config.combo_kernels value inside configure_inductor() during the Qwen3-235B startup; if False/unset, this hypothesis is disproven.

No commit-level hypothesis ≥ MEDIUM. The HIGH-confidence hypothesis is environmental (ROCm 7.2.0 + AITER custom AR), not a single sglang commit.

In-flight Fix Check

Search performed via repo:sgl-project/sglang+is:pr+is:open with keywords rocm+amd+hip+graph+capture and symmetric+memory+attention_tp.

• In-flight fix: ✅ #24151 "[Docker][ROCm] Bump ROCm720 base image from 7.2.0 to 7.2.2 to fix hipEventQuery race" (open since 2026-04-30) — directly addresses our WorkNCCL::isCompleted → hipEventQuery race; merge and rebuild rocm720 image.
• In-flight fix (companion): ✅ #23581 "[AMD] Default SGLANG_USE_AITER_AR to false to avoid HIP graph capture invalidation" (open since 2026-04-23) — short-term mitigation by disabling the AITER custom all-reduce path that exercises the race.
• In-flight fix (related, broader): 🟡 #24245 "[AMD] upgrade to ROCm 7.2.2" (open since 2026-04-30, last updated 2026-05-01) — overlapping scope with #24151.

Do NOT open a duplicate PR; chase reviewers on #24151 + #23581.

Suggested Triage Steps

• Verify #24151 rebuilds the rocm720 image cleanly; once merged, retrigger this nightly job and confirm test_qwen3_instruct_mxfp4.py returns to green (resolves the HIGH-confidence hypothesis end-to-end).
• As a same-day mitigation while #24151 is in review, apply #23581 (SGLANG_USE_AITER_AR=false default) and rerun this job; if the test passes, that confirms the AITER custom-AR path is the trigger inside ROCm 7.2.0.
• If both fixes land and the test still fails, bisect sglang in 021bd77c..5cb4ea1d (41 commits) using a Mi35x runner with the ROCm 7.2.2 image — start with f652135d (#22286) and e77bfba2 (#22458) since they touch NCCL/TP paths.

Status

• Persistence: Failing for ~21 consecutive scheduled runs (since 2026-04-11 05:31 UTC, ~21 days); last known passing run 2026-04-09.
• Scope: 1 job in 1 workflow (nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 in nightly-test-amd-rocm720.yml); the same hipErrorCapturedEvent cluster is documented in #24151 / #23581 as affecting other multi-GPU AMD CUDA-graph capture tests.
• Blocked work: Qwen3-235B-A22B-Instruct-2507 mxfp4 nightly accuracy/perf signal on MI35x ROCm 7.2.0 since 2026-04-11.
• In-flight fix: ✅ #24151 (open since 2026-04-30) ROCm 7.2.0 → 7.2.2 image bump; ✅ #23581 (open since 2026-04-23) SGLANG_USE_AITER_AR=false mitigation — chase reviewers, do NOT duplicate.

nightly-8-gpu-mi35x-qwen35-rocm720 — failed step(s): Accuracy Test MI35x ROCm 7.2 (8-GPU Qwen 3.5)

Commit Info

• sglang: ece8a1a7
• aiter: N/A (workflow nightly-test-amd-rocm720.yml does not emit [CI-AITER-CHECK] markers; AITER_COMMIT_OVERRIDE= empty in env, so the Dockerfile-baked aiter version in image rocm/sgl-dev:v0.5.10.post1-rocm720-mi35x-20260501 was used)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_qwen35_eval_mi35x.py	TestQwen35EvalMI35x.test_lm_eval	Exception: Server process exited with code -9. Check server logs for errors. (root: RuntimeError: The memory capacity is unbalanced … pre_model_load_memory=6.738, local_gpu_memory=8.441, local_gpu_memory*0.9=7.597, raised on TP1 in model_runner.py:1231)	Log

Failure Summary

The Qwen3.5-397B-A17B server fails to start on a TP=8 MI35x runner: rank 1's scheduler raises a TP memory-imbalance RuntimeError because its free GPU memory (6.74 GB) is more than 10 % below the other ranks' (8.44 GB), triggering the SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK guard in python/sglang/srt/model_executor/model_runner.py:1227-1231. The server emits SIGQUIT, exits with code -9, the test retries once, the same imbalance recurs, and the test errors out after ~92 s.

Regression Status

New regression — first observed 2026-04-30; both completed runs since have failed with the same TP1 imbalance pattern.

Recent history of test/registered/amd/accuracy/mi35x/test_qwen35_eval_mi35x.py (test_lm_eval) in job nightly-8-gpu-mi35x-qwen35-rocm720 (branch main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
May 1	25225581444	nightly-8-gpu-mi35x-qwen35-rocm720	❌ Failed	test_lm_eval	TP1 imbalance, server exit -9
Apr 30	25180829975	nightly-8-gpu-mi35x-qwen35-rocm720	❌ Failed	test_lm_eval	TP1 imbalance, server exit -9 (pre_model_load_memory=11.46, local_gpu_memory=13.17)
Apr 29	25125219864	nightly-8-gpu-mi35x-qwen35-rocm720	✅ Passed	—	—
Apr 28	25069314464	(cancelled)	—	—	run cancelled, excluded
Apr 27	25010858144	nightly-8-gpu-mi35x-qwen35-rocm720	✅ Passed	—	—
Apr 26	24962968926	nightly-8-gpu-mi35x-qwen35-rocm720	✅ Passed	—	—

Last passing sglang SHA: 530b497a (Apr 29). First failing sglang SHA: 7bb7f604 (Apr 30). Regression window contains 38 commits.

Failure Cluster

TP memory imbalance pre-model-load on MI35x rank 1 (~1.7 GB short)

Facts (verified observations, no interpretation)

• Runner today: linux-mi35x-gpu-8-4wrmw-runner-57sbm. Yesterday's failing run used a different runner host linux-mi35x-gpu-8-4wrmw-runner-n6l2l (verified from log line 2 of each run). Same failure mode reproduces on two distinct physical hosts.
• The imbalance is reported on TP1 only in both runs (TP0,2-7 successfully reach Init torch distributed ends. with mem usage ≈10.3-10.5 GB; TP1's "Init torch distributed ends" line is missing because it raised before reaching it).
• Magnitude of imbalance is ~1.7 GB below peers in BOTH runs (today: 6.74 vs 8.44; yesterday: 11.46 vs 13.17). Different absolute baselines, identical delta — points to a steady ~1.7 GB pre-existing allocation on rank 1.
• The check that fails is at model_runner.py:1227; it runs after the optional pre_warm_nccl all_reduce warmup at lines 1196-1212 and before model load.
• Process retried once via retry() wrapper at python/sglang/srt/utils/common.py:2672 — both attempts hit the same imbalance, so it is not a transient first-call effect.
• Other MI35x rocm720 jobs in the same run also failed (e.g. nightly-8-gpu-mi35x-deepseek-r1-mxfp4-kv-fp8-rocm720 73968206660, nightly-8-gpu-mi35x-glm51-rocm720 73968206822, nightly-8-gpu-mi35x-qwen3-235b-mxfp4-rocm720 73968206819 — at least 11 jobs ❌ in this run). Whether they share this exact symptom needs separate per-job analysis.
• AITER override is empty (AITER_COMMIT_OVERRIDE: blank at log lines 33, 114, 963, 1917). Image is rocm/sgl-dev:v0.5.10.post1-rocm720-mi35x-20260501.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] Pre-existing leaked GPU 1 allocation on the MI35x runner pool (~1.7 GB), unrelated to sglang code change

  • Supporting: identical 1.7 GB delta reproduces on TWO different runners (57sbm today, n6l2l yesterday); always rank 1; runner pool is linux-mi35x-gpu-8-4wrmw shared infra
  • Disconfirming: the same runner pool was passing on Apr 29 (530b497a) and earlier — if the leak were purely a runner-host artifact, we'd expect intermittent (not 2-day persistent) failures and not perfectly aligned with a sglang SHA boundary
  • How to verify: ssh into a fresh linux-mi35x-gpu-8-4wrmw runner and rocm-smi --showmemuse immediately after image pull, before any sglang process; check whether GPU 1 reports baseline usage > GPU 0

• [LOW] A commit in window 530b497a..7bb7f604 regressed pre-load memory accounting on rank 1

  • Supporting: temporal correlation — last-pass/first-fail boundary aligns exactly with this 38-commit window; candidates touching distributed init / NCCL warmup / per-rank state include 7389743d (modelexpress p2p RDMA), 79dbfe45 (Use spec v2 by default), 9e68c852 (AMD Nightly Test checkout mechanism), c8c1c926 (LoRA support for qwen3.5)
  • Disconfirming: I have NOT verified that any of these commits modify rank-asymmetric init paths; the symptom (rank-1-specific allocation) is hard to attribute to broadly-applied changes; the ~1.7 GB delta is reproducible across two different host instances which is unusual for a code-only regression
  • How to verify: bisect 530b497a..7bb7f604 on a freshly-rebooted MI35x 8-GPU runner

• [SPECULATION] Image change in rocm/sgl-dev:v0.5.10.post1-rocm720-mi35x-20260501 (dated 20260501) shipped a ROCm/RCCL build that allocates extra workspace on rank 1

  • Supporting: image tag includes a date suffix; refresh cadence not verified
  • Disconfirming: image tag is identical between today's failing run and yesterday's failing run, so this would not explain the Apr 29 → Apr 30 transition unless the image was rebuilt with same tag (need to verify image digest history)
  • How to verify: pull rocm/sgl-dev:v0.5.10.post1-rocm720-mi35x-20260430 (if it exists) and re-run; or compare image SHA digest between Apr 29 and Apr 30 runs from log line 805/795

• No hypothesis above LOW confidence; needs git bisect across 530b497a..7bb7f604 AND a runner-host baseline check (rocm-smi on a clean MI35x runner) to disentangle code-vs-infra causation.

In-flight Fix Check

Searched open sglang PRs for memory capacity unbalanced, qwen35 mi35x, and test_qwen35_eval_mi35x keywords:

• In-flight fix: ❌ none found in open PRs — needs new PR or investigation

(API call: GET /search/issues?q=repo:sgl-project/sglang+is:pr+is:open+memory+capacity+unbalanced → 0 results; same for the other two queries.)

Suggested Triage Steps

• Run rocm-smi --showmemuse on a freshly-allocated linux-mi35x-gpu-8-4wrmw runner BEFORE any sglang process starts, to determine whether GPU 1 carries a baseline ~1.7 GB allocation independent of sglang. If yes → infra/runner-pool issue, escalate to runner ops.
• If runner GPUs are clean: bisect git log --oneline 530b497a..7bb7f604 on an MI35x 8-GPU runner running python3 /sglang-checkout/test/registered/amd/accuracy/mi35x/test_qwen35_eval_mi35x.py. Start by halving the window at 79dbfe45 (Use spec v2 by default).
• Check whether other failing MI35x jobs in run 25225581444 (deepseek-r1, glm51, qwen3-235b-mxfp4) share the TP1 imbalance symptom. If yes → confirms broader MI35x infra/runtime issue; if no → narrows the scope to qwen3.5 model-specific path.
• As a temporary unblocker (NOT a fix): the maintainer may set SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 on this job to convert the RuntimeError to a warning and let the test run further — useful only to gather more downstream evidence about whether model load itself would also fail.

Status

• Persistence: Failing for 2 days (since 2026-04-30 18:23 UTC), every completed scheduled run since.
• Scope: This job (nightly-8-gpu-mi35x-qwen35-rocm720) on nightly-test-amd-rocm720.yml. 11 other MI35x-rocm720 jobs in the same run also failed; relationship to this exact symptom is unverified and warrants the third triage step above.
• Blocked work: Qwen3.5-397B-A17B accuracy eval signal on MI35x ROCm 7.2.
• In-flight fix: ❌ none found in open PRs — needs new PR or investigation.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-05-02 20:25 UTC)

[amd-bot]

nightly-test-amd.yml — 23 failure(s)

Run started (UTC): 2026-05-01 17:55 UTC
Last scanned (UTC): 2026-05-02 22:27 UTC
Commits: sglang e0474fd

Summary

Counts: 23 failures · 7 clusters · 13 carrying over (≥2 days) · 10 new today

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	TP memory-imbalance at server init (MI35x runner GPU pre-occupied)	nightly-8-gpu-mi35x-glm5-mxfp4	test/registered/amd/accuracy/mi35x/test_glm5_mxfp4_eval_mi35x.py	test_glm5_mxfp4_accuracy	3 consecutive (since Apr 30)	[MEDIUM] Dirty MI35x runner — prior container leaks HBM (0.65 GB free of ≥192 GB)
2	TP memory-imbalance at server init	nightly-perf-8-gpu-mi35x-deepseek-v32-mtp	test/registered/amd/perf/mi35x/test_deepseek_v32_mtp_perf_mi35x.py	TestNightlyDeepseekV32MTPPerformance.test_bench_one_batch	2 consecutive (since Apr 30)	[MEDIUM] 79dbfe4 (#21062 "Use spec v2 by default") — possible spec-v2 init memory regression
3	TP memory-imbalance at server init	nightly-accuracy-8-gpu-mi35x	test/registered/amd/accuracy/mi35x/test_gpt_oss_eval_mi35x.py	test_gpt_oss_accuracy	2 consecutive (since Apr 30)	[MEDIUM] Dirty MI35x runner — leftover HIP IPC handles / zombie processes
4	TP memory-imbalance at server init	nightly-8-gpu-mi35x-glm5-mxfp4	test/registered/amd/accuracy/mi35x/test_glm5_mxfp4_eval_mi35x.py	test_glm5_mxfp4_accuracy	2 consecutive (since Apr 30)	[HIGH] MI35x rank-1 GPU shared with another process holding ~52 GB
5	TP memory-imbalance at server init	nightly-8-gpu-mi35x-glm51	test/registered/amd/accuracy/mi35x/test_glm51_eval_mi35x.py	test_glm51_accuracy	2 consecutive (since May 01)	[MEDIUM] Runner-state contamination on shared MI35X pool
6	TP memory-imbalance at server init	nightly-8-gpu-mi35x-qwen35	test/registered/amd/accuracy/mi35x/test_qwen35_eval_mi35x.py	TestQwen35EvalMI35x.test_lm_eval	Intermittent 2/7 runs	[MEDIUM] Stale GPU memory from prior process — runner-cleanup issue
7	TP memory-imbalance at server init	nightly-accuracy-8-gpu-mi35x-deepseek-v32	test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	1 run (May 01, new)	[HIGH] Runner host contention — 13.4 GB free of ≥192 GB
8	TP memory-imbalance at server init	nightly-8-gpu-mi35x-glm51	test/registered/amd/accuracy/mi35x/test_glm51_eval_mi35x.py	TestGLM51EvalMI35x.test_glm51_accuracy	1 run (May 01, new)	[MEDIUM] Stale GPU state on shared MI35x pool
9	TP memory-imbalance at server init	nightly-8-gpu-mi35x-grok2	test/registered/amd/accuracy/mi35x/test_grok2_eval_mi35x.py	test_grok2_accuracy	1 run (May 01, new)	[MEDIUM] Stray process held ~1.7 GB on TP rank-1 GPU
10	TP memory-imbalance at server init	nightly-8-gpu-mi35x-kimi-k25	test/registered/amd/accuracy/mi35x/test_kimi_k25_eval_mi35x.py	test_kimi_k25_gsm8k_accuracy	1 run (May 01, new)	[LOW] Stale process on physical GPU index 1
11	GPU memory access fault / hang during CUDA graph capture (DeepSeek-V3.2 NSA + FP8)	nightly-perf-8-gpu-mi35x-deepseek-v32-basic	test/registered/amd/perf/mi35x/test_deepseek_v32_basic_perf_mi35x.py	test_bench_one_batch	2 consecutive (since May 01)	[LOW] aiter bump 7e17a7a (#24218) v0.1.12.post1 → a6bb4993
12	GPU memory access fault during CUDA graph capture	nightly-perf-8-gpu-mi35x-deepseek-v32-basic	test/registered/amd/perf/mi35x/test_deepseek_v32_basic_perf_mi35x.py	test_bench_one_batch	1 run with this signature (May 01, new)	[MEDIUM] aiter bump 7e17a7a (#24218)
13	GPU memory access fault / hang during CUDA graph capture	nightly-4-gpu	test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	TestDeepSeekV32EvalMI35x.test_deepseek_v32_accuracy	1 run (May 02, new)	[MEDIUM] cb8fbd5 (#24243 "Reserve slot 0 as padding in all req pools")
14	GPU memory access fault / hang during CUDA graph capture	nightly-accuracy-8-gpu-mi35x-deepseek-v32	test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	1 run (May 02, new)	[MEDIUM] aiter bump #24218 introduces graph-capture-incompatible kernel
15	GPU memory access fault / hang during CUDA graph capture	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	(server-launch crash, no test frame)	1 run (May 02, new — log-pairing artifact noted)	[LOW] Job-name / log pairing artifact at GHA layer
16	Mistral / Mixtral GSM8K accuracy below threshold (AMD MI30x)	nightly-accuracy-2-gpu	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	TestNightlyGsm8KEval.test_gsm8k_all_models	≥9 days, never-passed in window	[MEDIUM] Thresholds calibrated on NVIDIA H100, never re-tuned for AMD MI30x
17	Mistral / Mixtral GSM8K accuracy below threshold	nightly-accuracy-2-gpu	test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models	≥20 days, never-passed in window	[MEDIUM] Same NVIDIA-derived thresholds untuned for aiter MoE on AMD
18	Qwen2.5-VL MMMU eval hang / SIGABRT / timeout (AMD MI325)	nightly-accuracy-2-gpu-vlm	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	(timed out before any subTest)	5 consecutive (since Apr 28; symptom morphing)	[LOW] aiter bump #24218 destabilises Qwen2-VL-7B execution
19	Qwen2.5-VL MMMU eval hang / SIGABRT / timeout	nightly-accuracy-2-gpu-vlm	test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models	5 consecutive; new SIGABRT signature May 01	[MEDIUM] aiter bump #24218 — Qwen2.5-VL vision-tower conv crash
20	DeepSeek-V3.2 + MTP GSM8K accuracy collapse on MI35x (0.04 vs 0.94)	nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp	test/registered/amd/accuracy/mi35x/test_deepseek_v32_mtp_eval_mi35x.py	TestDeepseekV32TPMTP.test_a_gsm8k	3 consecutive (since Apr 30)	[MEDIUM] 79dbfe45 (#21062 "Use spec v2 by default") flipped SGLANG_ENABLE_SPEC_V2
21	DeepSeek-V3.2 + MTP GSM8K accuracy collapse	nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp	test/registered/amd/accuracy/mi35x/test_deepseek_v32_mtp_eval_mi35x.py	test_a_gsm8k	3 consecutive (since Apr 30)	[LOW] Regression in ad785a22..530b497a (16 commits)
22	HIP error inside aiter custom_all_reduce.capture() (Qwen3-235B MXFP4)	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	≥17 days, never-passed in window	[MEDIUM] aiter custom_all_reduce.flush_graph_buffers calls dist.all_gather_object during HIP stream capture → hipErrorCapturedEvent
23	MiniMax-M2.7 GSM8K below 0.93 threshold (TP8+EP8 MI30x)	nightly-8-gpu-minimax-m27	test/registered/amd/accuracy/mi30x/test_minimax_m27_eval_amd.py	TestMiniMaxM27EvalAMD.test_minimax_m27_accuracy	1 run (May 01, new — score 0.915)	[MEDIUM] aiter bump #24218 v0.1.12.post1 → a6bb4993

Cluster: TP memory-imbalance at server init (MI35x runner GPU pre-occupied)

• Affected: 10 jobs across the May 01 + May 02 nightly runs, all on the MI35x runner pool: rows 1-10 above.
• Shared facts: identical RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes. raised at model_runner.py:1227 inside init_torch_distributed. Reported local_gpu_memory values range from 0.65 GB to 13.4 GB on devices that should have ≥192 GB free. Failure happens before any model weight is loaded — only NCCL/Gloo + custom-allreduce setup runs. aiter version is unchanged across passing and failing runs ([CI-AITER-CHECK] markers).
• Top hypothesis: [HIGH] Dirty MI35x runner pool — leftover HBM allocations from prior containers / zombie processes / leaked HIP IPC handles. The magnitude (single-digit GB free of 192 GB) is impossible to explain by sglang code change. Disconfirming: cannot directly observe other GPU tenants from the job log; one rank-1 hypothesis (row 9) shows only 1.7 GB occupancy which is small. Also possible interaction with 357e378 (#23314 "limit nightly test parallelism to 1 job per hardware type") which may not have taken effect for the mi35x lane.
• In-flight fix: ❌ none found in any open sglang PR — needs runner-side investigation, not a code PR.
• Suggested triage: SSH to the MI35x runner pool and capture rocm-smi --showmeminfo + fuser /dev/kfd + ps -ef while idle to identify leftover HBM holders; verify #23314 parallelism cap is actually applied to the mi35x labels; add a pre-job runner-cleanup hook (kill leftover ROCm processes, recycle the container).

Cluster: GPU memory access fault / hang during CUDA graph capture (DeepSeek-V3.2 NSA + FP8 + Qwen3-235B MXFP4)

• Affected: rows 11-15 above — 4 distinct DeepSeek-V3.2 jobs and 1 Qwen3-235B-MXFP4 (which crashed in the deepseek-v32 test file due to a job-name/log-pairing artifact). All on MI35x (gfx950).
• Shared facts: Memory access fault by GPU node-{2..9} or torch.cuda.synchronize blocked in HSA runtime — both occur mid-capture at large batch sizes (bs=136, bs=144). All 8 TP ranks then hang in c10::hip::device_synchronize until the watchdog or runner timeout fires. New today (May 02), with the perf-basic job's symptom going back to May 01.
• Top hypothesis: [MEDIUM] aiter bump #24218 (7e17a7a1) v0.1.12.post1 → a6bb4993 introduced a graph-capture-incompatible kernel/path on gfx950. Disconfirming: bump is no-op for this dev image (already shipped that aiter SHA). cb8fbd5 (#24243 "Reserve slot 0 as padding") also lands in the window and changes req-pool indexing reachable from the failing path.
• In-flight fix: ❌ none found in open PRs targeting either the aiter bump or #24243 — needs new investigation.
• Suggested triage: revert #24218 (aiter bump) on a branch and re-run test_deepseek_v32_eval_mi35x.py on MI35x; if still failing, revert #24243 next; bisect 7bb7f6049a..ece8a1a (39 commits) on the perf-basic test as a fallback.

Cluster: Mistral / Mixtral GSM8K accuracy below threshold (AMD MI30x)

• Affected: rows 16-17 — nightly-accuracy-2-gpu × 2 runs.
• Shared facts: mistralai/Mistral-7B-Instruct-v0.3, mistralai/Mixtral-8x7B-Instruct-v0.1, and FP8 quantized variants score 0.17–0.37 against thresholds 0.46–0.69. ≥20 consecutive days failing; no green run in queryable window.
• Top hypothesis: [MEDIUM] Thresholds calibrated against NVIDIA H100 numbers, never re-tuned for AMD MI30x aiter MoE / quantization stack. Disconfirming: scores are very low (0.17 for Mixtral-FP8) — that magnitude could also indicate a broken MoE routing or chat-template mismatch on the AMD path, not just a tight threshold.
• In-flight fix: ❌ none found in open PRs targeting Mistral/Mixtral accuracy on AMD.
• Suggested triage: ask model-team owner whether the AMD scores are expected; if yes, lower thresholds in test_gsm8k_eval_amd.py; if no, profile the AMD MoE path and chat template against an NVIDIA reference.

Cluster: Qwen2.5-VL MMMU eval hang / SIGABRT / timeout (AMD MI325)

• Affected: rows 18-19 — nightly-accuracy-2-gpu-vlm × 2 runs.
• Shared facts: 5 consecutive days red; symptom morphed accuracy-threshold (Apr 28-30) → Fatal Python error: Aborted in qwen2_5_vl forward (May 01) → silent 7200 s timeout (May 02).
• Top hypothesis: [MEDIUM] aiter bump #24218 destabilises Qwen2.5-VL vision-tower conv forward. Disconfirming: the test has been red for 5 days with morphing symptoms, suggesting multiple causes layered on top of an underlying Qwen2.5-VL issue — aiter is the most recent change but is unlikely to explain the full 5-day window.
• In-flight fix: ❌ none found.
• Suggested triage: bisect aiter bump #24218 on the test_vlms_mmmu_eval_amd.py test on MI325; capture py-spy dump on the hung server in the latest run for a stack trace.

Cluster: DeepSeek-V3.2 + MTP GSM8K accuracy collapse on MI35x (0.04 vs 0.94)

• Affected: rows 20-21 — nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp × 2 runs.
• Shared facts: GSM8K accuracy 0.04, invalid 0.96 — model generates parseable tokens but 96% of completions fail answer extraction. 3 consecutive days; identical numbers across runs.
• Top hypothesis: [MEDIUM] 79dbfe45 (#21062 "Use spec v2 by default") flipped SGLANG_ENABLE_SPEC_V2 default — drafts are accepted at near-max rate with broken target outputs. Disconfirming: the per-job analysis hedges as [LOW] for the alternative window-only hypothesis; no direct A/B with SGLANG_ENABLE_SPEC_V2=0 yet.
• In-flight fix: ❌ none found targeting the DSV3.2 MTP GSM8K accuracy collapse. Open spec-v2 PRs (#23000, #21895, #20916) are feature work, not fixes.
• Suggested triage: re-run the failing test with SGLANG_ENABLE_SPEC_V2=0; if accuracy returns, revert / gate #21062 for DSV3.2 MTP path on AMD.

Cluster: HIP error inside aiter custom_all_reduce.capture() (Qwen3-235B MXFP4)

• Affected: row 22 — nightly-8-gpu-mi35x-qwen3-235b-mxfp4 (73968531128).
• Shared facts: HIP error: operation not permitted on an event last recorded in a capturing stream (hipErrorCapturedEvent) inside aiter custom_all_reduce.capture() during sglang's CUDA-graph capture. ≥17 days persistent, never-passed in queryable window.
• Top hypothesis: [MEDIUM] aiter custom_all_reduce.flush_graph_buffers → _gather_ipc_meta calls dist.all_gather_object collective on __exit__ while the HIP stream is still in capture mode.
• In-flight fix: ❌ none found — likely needs aiter-side fix.
• Suggested triage: file aiter issue with the stack trace; meanwhile, gate Qwen3-235B-MXFP4 on disable_custom_all_reduce if a workaround is needed.

Cluster: MiniMax-M2.7 GSM8K below 0.93 threshold

• Affected: row 23 — nightly-8-gpu-minimax-m27 (73968531032).
• Shared facts: GSM8K score 0.915 vs threshold 0.93, TP8+EP8 on MI30x. Previous 5 nightlies green, one borderline 0.925 on Apr 25.
• Top hypothesis: [MEDIUM] aiter bump #24218 shifted MoE numerics enough to dip below the borderline threshold.
• In-flight fix: ❌ none found.
• Suggested triage: re-run once to rule out flake (score is 0.015 below threshold); if reproducible, bisect the 39-commit window or revert #24218.

Suggested triage order

Rank	Triage step	Targets	Why this first
1	SSH MI35x runner pool, snapshot rocm-smi + fuser /dev/kfd + ps -ef while idle; verify #23314 parallelism cap; add pre-job cleanup hook	Cluster A (rows 1-10)	Largest cluster (10 jobs), pure runner-side, cheap to confirm, blocks 9 distinct test files
2	Revert aiter bump #24218 (7e17a7a1) on a branch, re-run rows 11-15 + 18-19 + 23	Cluster B (5 rows) + Cluster F (2) + Cluster E (1) — 8 rows total	One revert tests the leading hypothesis for 3 clusters at once
3	Re-run row 20 with SGLANG_ENABLE_SPEC_V2=0 to A/B #21062	Cluster G (rows 20-21)	Cheap env-flag toggle; if accuracy returns, root cause is confirmed
4	Decide AMD threshold policy with model owner; either re-tune test_gsm8k_eval_amd.py thresholds or open AMD MoE numerics investigation	Cluster D (rows 16-17)	Persistent ≥20 days but stable; needs human decision, not bisect
5	File aiter issue for custom_all_reduce.flush_graph_buffers doing all_gather_object during HIP stream capture	Cluster C (row 22)	Single-row, well-understood, requires aiter-side fix

---

Job	Failed Steps	Started
nightly-accuracy-8-gpu-mi35x-deepseek-v32	Accuracy Test MI35x (8-GPU DeepSeek-V3.2)	2026-05-01T19:47
nightly-accuracy-2-gpu	Nightly Test (2-GPU)	2026-05-01T18:17
nightly-8-gpu-minimax-m27	Accuracy Test (8-GPU MiniMax-M2.7)	2026-05-01T20:12
nightly-8-gpu-mi35x-glm5-mxfp4	Accuracy Test MI35x (8-GPU GLM-5-MXFP4)	2026-05-01T19:56
nightly-8-gpu-mi35x-grok2	Accuracy Test MI35x (8-GPU Grok2)	2026-05-01T19:45
nightly-perf-8-gpu-mi35x-deepseek-v32-mtp	Performance Test MI35x (8-GPU DeepSeek-V3.2 MTP)	2026-05-01T19:20
nightly-accuracy-2-gpu-vlm	Nightly Accuracy Test (2-GPU VLM MMMU)	2026-05-01T17:55
nightly-accuracy-8-gpu-mi35x	Accuracy Test MI35x (8-GPU GPT-OSS)	2026-05-01T20:26
nightly-8-gpu-mi35x-glm51	Accuracy Test MI35x (8-GPU GLM-5.1 NSA)	2026-05-01T18:47
nightly-perf-8-gpu-mi35x-deepseek-v32-basic	Performance Test MI35x (8-GPU DeepSeek-V3.2 Basic)	2026-05-01T20:06
nightly-8-gpu-mi35x-qwen3-235b-mxfp4	Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)	2026-05-01T20:23
nightly-8-gpu-mi35x-kimi-k25	Accuracy Test MI35x (8-GPU Kimi-K2.5)	2026-05-01T20:13
nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp	Accuracy Test MI35x (8-GPU DeepSeek-V3.2 TP+MTP)	2026-05-01T20:33
nightly-8-gpu-mi35x-glm51	Accuracy Test MI35x (8-GPU GLM-5.1 NSA)	2026-05-02T18:16
nightly-8-gpu-mi35x-qwen35	Accuracy Test MI35x (8-GPU Qwen 3.5)	2026-05-02T18:36
nightly-accuracy-8-gpu-mi35x-deepseek-v32	Accuracy Test MI35x (8-GPU DeepSeek-V3.2)	2026-05-02T18:05
nightly-4-gpu	Nightly Test (4-GPU)	2026-05-02T17:45
nightly-8-gpu-mi35x-qwen3-235b-mxfp4	Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)	2026-05-02T18:45
nightly-8-gpu-mi35x-glm5-mxfp4	Accuracy Test MI35x (8-GPU GLM-5-MXFP4)	2026-05-02T18:58
nightly-accuracy-2-gpu	Nightly Test (2-GPU)	2026-05-02T18:03
nightly-accuracy-2-gpu-vlm	Nightly Accuracy Test (2-GPU VLM MMMU)	2026-05-02T18:06
nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp	Accuracy Test MI35x (8-GPU DeepSeek-V3.2 TP+MTP)	2026-05-02T20:11
nightly-perf-8-gpu-mi35x-deepseek-v32-basic	Performance Test MI35x (8-GPU DeepSeek-V3.2 Basic)	2026-05-02T20:15

Per-Job Analysis

nightly-accuracy-8-gpu-mi35x-deepseek-v32 — failed step(s): Accuracy Test MI35x (8-GPU DeepSeek-V3.2)

Commit Info

• sglang: ece8a1a
• aiter: a6bb499 (source: dev/patched image preserved — v0.1.12.post2.dev150+ga6bb49937, matches Dockerfile default AITER_COMMIT=a6bb499375849eec45d68c5ccaebc8865fd422c0)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py	test_deepseek_v32_accuracy	AssertionError: Failed models: ['deepseek-ai/DeepSeek-V3.2 (basic)'] (server failed to start — see Facts)	Log

Failure Summary

The single test in this nightly suite (test_deepseek_v32_accuracy) failed because the SGLang server never reached ready state. Each TP worker raised RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes. during init_torch_distributed (model_runner.py:1227) — the available-memory probe returned only ~13.4 GB on an MI35x GPU (which carries ≥192 GB HBM3), indicating the device was almost entirely occupied by another process before the test began.

Regression Status

New regression — first failure of test_deepseek_v32_accuracy in nightly-accuracy-8-gpu-mi35x-deepseek-v32 for at least the past 4 scheduled completed runs.

Recent history of test/registered/amd/accuracy/mi35x/test_deepseek_v32_eval_mi35x.py (test_deepseek_v32_accuracy) in job nightly-accuracy-8-gpu-mi35x-deepseek-v32 (branch=main, event=schedule, completed only):

Date	Run	Job	Test File Status	Failed Function	Error
May 01	25225678502	nightly-accuracy-8-gpu-mi35x-deepseek-v32	❌ Failed	test_deepseek_v32_accuracy	AssertionError: Failed models: [...] (server init OOM)
Apr 30	25180906638	nightly-accuracy-8-gpu-mi35x-deepseek-v32	✅ Passed	—	—
Apr 29	25125309566	nightly-accuracy-8-gpu-mi35x-deepseek-v32	✅ Passed	—	—
Apr 28	25069425979	nightly-accuracy-8-gpu-mi35x-deepseek-v32	✅ Passed	—	—
Apr 27	25010934840	nightly-accuracy-8-gpu-mi35x-deepseek-v32	✅ Passed	—	—

First observed failure: 2026-05-01 19:56 UTC. Last known passing: 2026-04-30 (run 25180906638).

Failure Cluster

GPU memory pre-occupied at server init — RuntimeError: memory capacity is unbalanced on MI35x

Facts (verified observations, no interpretation)

• Verbatim error from the log (line 2570):

  RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes.
  pre_model_load_memory=11.705078125, local_gpu_memory=13.408203125, local_gpu_memory * 0.9=12.0673828125
  

• Stack trace (verbatim, lines 2553-2570):

  scheduler.py:3918 → scheduler.py:432 (__init__) → scheduler.py:693 (init_model_worker)
  → scheduler.py:648 (init_tp_model_worker) → tp_worker.py:260 (__init__)
  → tp_worker.py:345 (_init_model_runner) → model_runner.py:481 (__init__)
  → model_runner.py:1231 (init_torch_distributed)  raise RuntimeError(msg)
  

• The check fires in init_torch_distributed at model_runner.py:1227 — pre_model_load_memory < local_gpu_memory * 0.9 ⇒ raises when SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK is enabled. Reported numbers local_gpu_memory=13.4 GB are two orders of magnitude smaller than MI35x normal free memory (≥192 GB HBM3), meaning ~94 % of HBM was already in use before TP init.
• Per-rank Init torch distributed ends lines for TP0/2/3/4/5/6/7 report mem usage=10.07-10.22 GB (NCCL allocations only). TP1 raised before logging mem usage. There is no model load yet — the memory was occupied by something external to this job.
• aiter version v0.1.12.post2.dev150+ga6bb49937 matches the Dockerfile default (a6bb499) and was unchanged across passing (Apr 27-30) and failing (May 01) runs ([CI-AITER-CHECK] markers, lines 2273-2279). Dockerfile commit was bumped to a6bb499 in 7e17a7a (#24218) which is in the regression window — but the bump is no-op for this job because the dev image already shipped that exact aiter SHA.
• Job ran for ~40 s before exit; the test was never able to perform any inference. First-attempt failure exit code 255 → no retry.
• Sister job nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp (same runner family, same image) has been intermittently failing for several runs (Apr 30, Apr 29) but -deepseek-v32 itself was passing through Apr 30.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] Runner / hardware contention — another process on the same MI35x host occupied the HBM before nightly-accuracy-8-gpu-mi35x-deepseek-v32 started its TP workers.

  • Supporting: local_gpu_memory=13.4 GB is wildly below MI35x's ≥192 GB free baseline; the imbalance check is precisely designed to detect "GPUs may be occupied by other processes". TP workers report mem usage ≈10 GB just from NCCL allocations on top of an already-near-full GPU. The 39 sglang commits in window 7bb7f6049a..ece8a1a contain no obvious GPU-memory regression touching init_torch_distributed or its callers (no edits to the imbalance-check site itself or to NCCL/Gloo init paths).
  • Disconfirming: cannot directly observe other GPU tenants from the job log; this is inferred from the magnitude of the free-memory number. No prior log evidence that this job has been affected by host contention, so this is the first time this exact symptom appears.
  • How to verify: re-run the same job; if it passes immediately, host contention is confirmed. Inspect the runner host (rocm-smi snapshot, fuser /dev/kfd) at scheduling time; check whether 357e378 ("ci: limit nightly test parallelism to 1 job per hardware type", #23314, in window) actually took effect for the mi35x pool.

• [LOW] Concurrent-job leak introduced by the new nightly parallelism limiter — 357e378 (#23314) lands in the regression window and changes nightly job scheduling.

  • Supporting: temporal correlation — first failure is the first nightly schedule run after this commit landed; the commit modifies parallelism control which is exactly the lever that prevents two GPU jobs from co-locating.
  • Disconfirming: I have not read the commit's diff to confirm it actually applies to mi35x or this workflow; the change is in CI config, not in the runtime imbalance-check code; only one of the two MI35x deepseek-v32 sub-jobs failed (mtp passed Apr 30 too historically), so not a uniform parallelism leak.
  • How to verify: read 357e378 diff; check whether mi35x runner pool concurrency is 1 after this commit; correlate with runner-host metrics for the failing window.

• [SPECULATION] Real model-side memory regression in DSV3.2 init pulled in by the 39-commit window (e.g. d48095b "Bypass torch.cuda.use_mem_pool generator-CM in SymmetricMemoryContext", 108bfd8 "Aiter MoE runner backend").

  • Supporting: these are the only commits in the window that touch GPU memory pool / MoE allocation paths.
  • Disconfirming: failure occurs before any model weight is loaded — only NCCL/Gloo + custom-allreduce setup runs. None of these commits touch that path. The 13.4 GB free figure is impossible to explain by a 39-commit code change alone — that magnitude implies an external occupant, not a code-side allocation increase.
  • How to verify: not actionable until the contention hypothesis is ruled out.

If hypothesis 1 is disproven (re-run still fails on a known-clean host), escalate to bisect 7bb7f6049a..ece8a1a on the failing test.

In-flight Fix Check

Searched repo:sgl-project/sglang is:pr is:open "memory capacity unbalanced" → 0 hits.

• In-flight fix: ❌ none found in open PRs — needs investigation (likely runner-side, not a code PR)

Suggested Triage Steps

• Re-run job 73968531007 on the same MI35x runner pool. If it passes on retry, treat as transient host contention and confirm whether #23314 (parallelism=1 per hardware) is in effect for the mi35x lane.
• SSH into the MI35x runner host that picked up this job (run page → runner labels) and capture rocm-smi --showmeminfo, fuser /dev/kfd, and ps -ef to identify any leftover process holding HBM.
• Inspect 357e378 (#23314) diff to verify mi35x runner pool concurrency is actually capped at 1; if not, follow up with a CI fix.
• Check whether sister jobs running on the same runner at ~19:55 UTC on May 01 (e.g. nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp from same run 25225678502) overlapped temporally — overlap on the same host would explain the imbalance.
• If the failure repeats on a clean host (contention disproven), bisect commits in 7bb7f6049a..ece8a1a (39 commits) using test_deepseek_v32_accuracy; start with d48095b and 108bfd8 as the only memory-adjacent suspects.

Status

• Persistence: 1 completed run (2026-05-01 19:56 UTC). Previous 4 scheduled completed runs (Apr 27 → Apr 30) all passed for this test file.
• Scope: 1 job in 1 workflow (nightly-test-amd.yml / nightly-accuracy-8-gpu-mi35x-deepseek-v32). The sister -mtp job has its own intermittent failures pre-dating this regression and is out of scope.
• Blocked work: Blocks DSV3.2 accuracy signal on MI35x; no known production impact.
• In-flight fix: ❌ none found in open PRs — needs investigation (likely runner-side, not a code PR).

nightly-accuracy-2-gpu — failed step(s): Nightly Test (2-GPU)

Commit Info

• sglang: ece8a1a7
• aiter: a6bb4993 (source: dev/patched image — v0.1.12.post2.dev150+ga6bb49937, matches Dockerfile-expected base SHA, rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py	test_gsm8k_all_models	AssertionError: The following models failed: (4 Mistral/Mixtral models below threshold)	Log

Per-model results (all 4 well below threshold; other 13 models in the suite pass):

Model	TP	Score	Threshold	Status
mistralai/Mistral-7B-Instruct-v0.3	1	0.373	0.47	❌ FAIL
mistralai/Mixtral-8x7B-Instruct-v0.1	2	0.313	0.69	❌ FAIL
neuralmagic/Mistral-7B-Instruct-v0.3-FP8	1	0.320	0.46	❌ FAIL
neuralmagic/Mixtral-8x7B-Instruct-v0.1-FP8	2	0.170	0.69	❌ FAIL

Failure Summary

test_gsm8k_all_models in test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py failed because four Mistral/Mixtral variants (BF16 + FP8) scored well under their gsm8k accuracy thresholds. Mixtral-8x7B-FP8 is the worst offender (0.17 vs 0.69 threshold — roughly random). All 13 non-Mistral models in the suite (Llama, Qwen, Gemma, DeepSeek-Coder-V2-Lite) pass.

Regression Status

Known recurring failure (long-standing, ≥20 days) — NOT a fresh regression introduced by ece8a1a7.

Recent history of test_gsm8k_all_models in job nightly-accuracy-2-gpu (branch main, event schedule, completed runs only; only runs whose shard actually executed test_gsm8k_eval_amd.py are listed — other dates ran test_dumper.py or test_vlms_mmmu_eval_amd.py and are skipped):

Date	Run	Job	Test File Status	Failed Function	Error
May 1	25225678502	nightly-accuracy-2-gpu	❌ Failed	test_gsm8k_all_models	4 Mistral/Mixtral models below threshold
Apr 30	25180906638	nightly-accuracy-2-gpu	❌ Failed	test_gsm8k_all_models	same 4 Mistral/Mixtral models
Apr 29	25125309566	nightly-accuracy-2-gpu	❌ Failed	test_gsm8k_all_models	same 4
Apr 27	25010934840	nightly-accuracy-2-gpu	❌ Failed	test_gsm8k_all_models	same 4
Apr 25	24936820562	nightly-accuracy-2-gpu	❌ Failed	test_gsm8k_all_models	same 4
Apr 24	24903695701	nightly-accuracy-2-gpu	❌ Failed	test_gsm8k_all_models	same 4
Apr 22	24793680817	nightly-accuracy-2-gpu	❌ Failed	test_gsm8k_all_models	same 4
Apr 21	24737667212	nightly-accuracy-2-gpu	❌ Failed	test_gsm8k_all_models	same 4
Apr 13	24358226838	nightly-accuracy-2-gpu	❌ Failed	test_gsm8k_all_models	same 4
Apr 12	24312584276	nightly-accuracy-2-gpu	❌ Failed	test_gsm8k_all_models	same 4

No passing run found for these 4 models within the queryable window (Apr 12 → May 1, 2026). Mark as Never-passed in queryable history. First observed failure date: ≥ Apr 12 (lookback limit reached without finding green). Last known passing date: unknown — may pre-date the queryable window.

Failure Cluster

Mistral / Mixtral gsm8k accuracy regression on AMD MI30x (BF16 + FP8 both affected)

Facts (verified observations, no interpretation)

• 4 specific models fail every run; all other 13 models in the same suite pass on the same job/runner/aiter combination (log line 35628).
• Failure mode is accuracy degradation, not a crash: server starts, eval runs to completion, scores are simply low.
• Both BF16 (mistralai/*) and FP8 (neuralmagic/*) variants of Mistral-7B-Instruct-v0.3 and Mixtral-8x7B-Instruct-v0.1 fail; FP8 Mixtral is worst (0.17 ≈ random).
• Test config at test/registered/amd/accuracy/mi30x/test_gsm8k_eval_amd.py:111-116 shows TRITON_MOE_MODELS entries for Mixtral-8x7B-Instruct-v0.1, Mixtral-8x7B-Instruct-v0.1-FP8, and Mistral-7B-Instruct-v0.3 are commented out, so they all run with SGLANG_USE_AITER=1 (the default branch in test_gsm8k_eval_amd.py:250-252). NO_MOE_PADDING_MODELS includes only neuralmagic/Mixtral-8x7B-Instruct-v0.1-FP8, so the other 3 run with SGLANG_MOE_PADDING=1.
• Same failure pattern with same scores (within ±0.03) reproduces across ≥10 distinct sglang HEAD SHAs spanning 20 days, ruling out any single recent commit as the cause.
• aiter commit running on May 1 is a dev image (v0.1.12.post2.dev150+ga6bb49937); the underlying SHA matches the Dockerfile default (a6bb4993) — i.e. no scout override.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Pre-existing Mistral/Mixtral accuracy issue on AMD MI30x — the test thresholds (0.47 / 0.69 / 0.46 / 0.69) appear to have been set to NVIDIA-derived expectations, but the AMD path (aiter MoE / aiter attention) returns scores ~0.17–0.37 reliably. This is consistent with either a real kernel-level numerical bug specific to Mistral architecture, or thresholds that were never validated end-to-end on AMD before being committed.
  • Supporting: 100% reproducibility across 10+ HEAD SHAs and 20 days; all other 13 models in the same harness pass; the comments at lines 112-115 (TRITON_MOE_MODELS entries for the 3 failing Mistral/Mixtral models commented out) suggest someone previously toggled aiter on/off for these specific models, hinting at a known aiter-vs-Mistral interaction.
  • Disconfirming: BF16 Mistral-7B (no MoE, no FP8) also fails (0.37 vs 0.47), so the issue is NOT exclusively a MoE-padding or FP8 problem — a pure-attention Mistral path is also degraded. This weakens an "aiter MoE bug" framing.
  • How to verify: locally re-run mistralai/Mistral-7B-Instruct-v0.3 with SGLANG_USE_AITER=0 on MI30x; if score recovers to ≥0.47, the regression is in aiter (attention path). If not, look at sglang's Mistral model definition / RoPE / sliding-window handling on ROCm.
• [LOW] Threshold drift / model-output-distribution drift unrelated to sglang code (e.g. tokenizer or HF model file change upstream).
  • Supporting: scores are stable across 10+ runs but stable-low rather than fluctuating — consistent with a deterministic codepath, not flaky HF downloads.
  • Disconfirming: thresholds in test_gsm8k_eval_amd.py:45-65 are commented as " - 5%" margins on observed reference scores, so they should already be tolerant of distribution drift.
  • How to verify: run the same gsm8k eval on the same Mistral checkpoint with stock vllm or HF transformers on the same MI30x — if scores match sglang's ~0.37, the issue is upstream/model-side; if they match ~0.5, the issue is in sglang or aiter.
• [LOW] aiter SHA a6bb4993 introduced a Mistral-specific kernel regression at some unknown earlier date.
  • Supporting: aiter version is unchanged across the entire 20-day failure window (Dockerfile default), and failures coincide with that period.
  • Disconfirming: aiter SHA is the same across all 10+ failing runs, so it cannot explain the lack of a baseline pass either way without a longer history before the Dockerfile bump. No A/B baseline exists in the queryable window where aiter is different AND Mistral passes.
  • How to verify: bisect aiter on MI30x using mistralai/Mistral-7B-Instruct-v0.3 + SGLANG_USE_AITER=1 as the repro; check whether any earlier aiter SHA scores ≥ 0.47.

No hypothesis above MEDIUM. Full causal attribution requires either (a) an SGLANG_USE_AITER=0 A/B run, or (b) git bisect against the last sglang/aiter combination where Mistral passed (which is outside the queryable history).

In-flight Fix Check

• In-flight fix: ❌ none found — searched open sglang PRs with keywords mistral+gsm8k, mistral+amd, mixtral+accuracy, nightly+accuracy+amd (15 hits, all unrelated — adding new model coverage, not fixing Mistral). No PR is targeting Mistral/Mixtral accuracy on AMD.

Suggested Triage Steps

• A/B disconfirmation: in container lmsysorg/sglang:v0.5.5.post1-rocm700 (or whatever the nightly uses), launch mistralai/Mistral-7B-Instruct-v0.3 on MI30x once with SGLANG_USE_AITER=0 and once with SGLANG_USE_AITER=1; run the gsm8k harness from test_gsm8k_eval_amd.py standalone. If aiter=0 passes (≥0.47) and aiter=1 fails (~0.37), the bug is in aiter's attention path for non-MoE Mistral; file an aiter issue.
• Repeat the A/B for mistralai/Mixtral-8x7B-Instruct-v0.1 to separate the MoE-path regression from the attention-path regression.
• If aiter-off doesn't fix it, bisect sglang back to before Apr 12 — if a passing SHA exists, run git log --oneline <pass>..ece8a1a7 -- python/sglang/srt/models/mixtral.py python/sglang/srt/models/mistral.py python/sglang/srt/layers/attention python/sglang/srt/layers/moe to enumerate suspect commits in Mistral/MoE/attention codepaths.
• Consider re-enabling the commented-out TRITON_MOE_MODELS entries at test_gsm8k_eval_amd.py:112-115 as an experiment — the comments suggest someone previously found Triton-MoE worked better than aiter for these specific models.
• If no fix is forthcoming and accuracy can't be restored, owners should decide whether to (a) remove the 4 Mistral/Mixtral models from the AMD nightly suite to unblock CI signal, (b) lower the thresholds to match observed scores (with an explicit comment noting AMD-path quality gap), or (c) keep the failure to keep visibility — but the current state is silent (the job has been red for 20+ days with no apparent ticket).

Status

• Persistence: Failing in every completed nightly-accuracy-2-gpu run that exercised test_gsm8k_eval_amd.py for ≥20 days (since at least 2026-04-12 06:24 UTC; queryable history exhausted without a green run).
• Scope: Single job nightly-accuracy-2-gpu in workflow nightly-test-amd.yml. Same 4 Mistral/Mixtral models fail; 13 other models pass on the identical runner/container/aiter.
• Blocked work: Blocks AMD nightly accuracy signal — the job is permanently red, masking any genuinely new regressions in the other 13 models.
• In-flight fix: ❌ none found in open PRs — needs new investigation.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-05-02 22:27 UTC)

[amd-bot]

nightly-test-amd.yml — Per-Job Analysis (continued 2/6)

nightly-8-gpu-minimax-m27 — failed step(s): Accuracy Test (8-GPU MiniMax-M2.7)

Commit Info

• sglang: ece8a1a7
• aiter: a6bb4993 (source: Dockerfile default; image dev build v0.1.12.post2.dev150+ga6bb49937 matched & preserved)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_minimax_m27_eval_amd.py	TestMiniMaxM27EvalAMD.test_minimax_m27_accuracy	AssertionError: Failed models: ['MiniMaxAI/MiniMax-M2.7 (TP8+EP8)'] — accuracy=0.915 threshold=0.93 FAIL (GSM8K, 200 samples)	Log line 6560

Failure Summary

The MiniMax-M2.7 GSM8K accuracy gate failed: measured accuracy was 0.915 versus required threshold 0.93 on the TP8+EP8 8-GPU MI30x configuration. The server warmed, ran all 200 prompts to completion, and the assertion fired at the post-eval threshold check (line 241). No crash, no kernel error — pure numeric drop.

Regression Status

Likely regression (with prior borderline flake)

Recent history of test/registered/amd/accuracy/mi30x/test_minimax_m27_eval_amd.py (test_minimax_m27_accuracy) in job nightly-8-gpu-minimax-m27 (workflow nightly-test-amd.yml, branch main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Accuracy	AITER
May 01	25225678502	nightly-8-gpu-minimax-m27	❌ Failed	test_minimax_m27_accuracy	0.915	a6bb4993
Apr 30	25180906638	nightly-8-gpu-minimax-m27	✅ Passed	—	0.950	v0.1.12.post1
Apr 29	25125309566	nightly-8-gpu-minimax-m27	✅ Passed	—	—	v0.1.12.post1
Apr 28	25069425979	nightly-8-gpu-minimax-m27	✅ Passed	—	—	v0.1.12.post1
Apr 27	25010934840	nightly-8-gpu-minimax-m27	✅ Passed	—	—	v0.1.12.post1
Apr 26	24963018193	nightly-8-gpu-minimax-m27	✅ Passed	—	—	v0.1.12.post1
Apr 25	24936820562	nightly-8-gpu-minimax-m27	❌ Failed	test_minimax_m27_accuracy	0.925	v0.1.12.post1

First failure today: 2026-05-01 20:43 UTC. Last green: 2026-04-30 (accuracy 0.950 — comfortably above threshold). One earlier borderline failure on Apr 25 (0.925) preceded a 5-run green streak. Lookback window: 7 days.

Failure Cluster

MiniMax-M2.7 GSM8K accuracy below 0.93 threshold (TP8+EP8 on AMD MI30x).

Facts (verified observations, no interpretation)

• Verbatim log: accuracy=0.915 threshold=0.93 FAIL (log:6560), AssertionError: Failed models: ['MiniMaxAI/MiniMax-M2.7 (TP8+EP8)'] (log:6573).
• All 200 prompts completed (100%|██████████| 200/200); no server crash, no kernel error, no OOM in surrounding context.
• AITER version flipped between the last passing run and this failing run: Apr 30 ran v0.1.12.post1 (log:2202); May 1 ran the dev build at sglang's new pinned commit a6bb4993 (image already had the matching dev tag, so no rebuild — [CI-AITER-CHECK] Dev/patched version detected … skipping rebuild, log:2213).
• The aiter delta v0.1.12.post1...a6bb4993 is 150 commits, and many touch MoE / fp8 blockscale / quant / MLA paths exercised by MiniMax-M2.7 (e.g. 9cc2b8ae "ck fp8 blockscale splitk in moe tunner", 988511f7 "fix config for moe silu_64x25", f8c1d76b "16x128 fused moe kernel", 5282d715 "fix fused_dynamic_mxfp4_quant_moe_sort dispatch", 710a4294 "fix … asm a8w8 splitk").
• The sglang delta 7bb7f604..ece8a1a7 is 39 commits. Only one is functionally aiter-related: 7e17a7a "Upd: AITER->(#2879)a6bb499 (#24218)" — a one-file Dockerfile bump of AITER_COMMIT_DEFAULT from v0.1.12.post1 → a6bb499375. No MiniMax-M2.7 / MoE-runner / quant code under python/sglang/ is touched in the window.
• Earlier borderline failure on Apr 25 (accuracy=0.925) used the same v0.1.12.post1 aiter — confirms the test sits near the noise floor even before the aiter bump.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] AITER bump from v0.1.12.post1 to a6bb4993 (sglang #24218 / commit 7e17a7a) regressed MiniMax-M2.7 numerical quality on AMD MI30x.

  • Supporting: this is the only sglang commit in the regression window that affects what runs on AMD; the aiter version is the one variable that changed between Apr 30 (acc 0.950, PASS) and May 1 (acc 0.915, FAIL); the 150-commit aiter delta includes many MoE/fp8-blockscale/quant changes (see Facts) that match MiniMax-M2.7's numerical-sensitive path (attention_backend='aiter', fp8 blockscale fused MoE, EP8); accuracy dropped 3.5 percentage points in one run, larger than the typical inter-run variance observed across Apr 26–30.
  • Disconfirming: no direct A/B disconfirmation (no nightly run today with AITER_COMMIT_OVERRIDE=v0.1.12.post1 to repro the prior aiter); Apr 25 also failed (0.925) on the old aiter, so this test was already brittle / sat near the threshold floor — the May 1 failure could partly be sample-variance compounded with a small aiter shift; no specific aiter commit has been pinpointed.
  • How to verify: re-run nightly-8-gpu-minimax-m27 against the current sglang HEAD with AITER_COMMIT_OVERRIDE=v0.1.12.post1 (or pre-bump sglang 7bb7f604 HEAD); if accuracy returns ≥ 0.93 across 2 reruns, confidence promotes to HIGH and aiter bisect within v0.1.12.post1...a6bb4993 (focus on the MoE/fp8-blockscale commits listed in Facts). If accuracy stays < 0.93 with old aiter too, this hypothesis is falsified — see next.

• [LOW] Test is borderline-flaky: GSM8K 200-sample accuracy on MiniMax-M2.7 sits close to 0.93, so any small numerical drift (kernel selection, dispatch, RNG) tips it below threshold even without a code change.

  • Supporting: Apr 25 failed at 0.925 on the same aiter as the Apr 26–30 passing runs — shows the test fires false negatives on stable code; sample size 200 ⇒ standard error around the threshold is ~0.018 (binomial), so a 0.93 model can fail ≈25–30% of the time by chance.
  • Disconfirming: today's drop to 0.915 is below where Apr 25's 0.925 sat, and only one of seven recent runs was a non-borderline pass at exactly 0.95 — but that's not strong evidence on a single sample.
  • How to verify: rerun the same job at the same SHA twice; if both reruns ≥ 0.93, treat as flake and consider raising sample size or lowering threshold; if both < 0.93, this is a real regression.

• [LOW] One of the non-aiter sglang commits in the 39-commit window introduced a numerical regression on MiniMax-M2.7's path (e.g. 108bfd8 "Aiter MoE runner backend and purge aiter.fused_moe from quant methods", 8975479 "[LoRA][MOE] Fix EP correctness in MoE LoRA slicing").

  • Supporting: 108bfd8 refactors how MoE quant methods invoke aiter, which is precisely the code path MiniMax-M2.7 uses (fp8-blockscale fused MoE).
  • Disconfirming: the test does not enable LoRA or virtual experts (enable_lora=None, lora_use_virtual_experts=False in server_args), so 8975479 is unlikely; for 108bfd8, this change landed a few hours before the aiter bump but no other AMD MoE workloads in the same nightly run regressed (would need cross-job verification to be sure).
  • How to verify: only worth investigating if the AITER-revert A/B in the top hypothesis fails to restore accuracy.

No hypothesis above MEDIUM confidence; needs the AITER-revert A/B above (or a 2x rerun) to promote/falsify.

In-flight Fix Check

Searched open PRs with keywords minimax accuracy, minimax m2.7, aiter a6bb499:

• In-flight fix: ❌ none found — needs new PR or investigation. Open PRs #22934 (eplb bugfix), #23946 (WIP attention kernel for m2.5), #22300 (NVIDIA fp8 m2.5) are unrelated to this M2.7 accuracy gate.

Suggested Triage Steps

• First, A/B against the AITER bump. Re-run nightly-8-gpu-minimax-m27 at sglang HEAD ece8a1a7 with AITER_COMMIT_OVERRIDE=v0.1.12.post1. Compare accuracy: if ≥ 0.93, the bump is implicated.
• Second, rerun-for-flake. Trigger the same job twice on the current SHA (no overrides). If one or both reach ≥ 0.93, the threshold is too tight for this model and the AITER hypothesis weakens.
• If AITER is implicated, bisect aiter commits in v0.1.12.post1...a6bb499375 — start with MoE/fp8-blockscale changes (9cc2b8ae, 988511f7, f8c1d76b, 5282d715) since MiniMax-M2.7 uses fp8 blockscale fused MoE on EP8.
• If both AITER-revert and rerun fail to restore accuracy, then bisect sglang 7bb7f604..ece8a1a7 — start with 108bfd8 (Aiter MoE runner backend refactor).
• Long-term: increase GSM8K sample count from 200 → 500+ and/or document the expected accuracy band for this model so a borderline drop doesn't masquerade as a regression.

Status

• Persistence: First fail today; previous 5 nightly runs (Apr 26–30) green; one borderline fail on Apr 25 (0.925).
• Scope: 1 test file × 1 test function × 1 job (nightly-8-gpu-minimax-m27) in 1 workflow (nightly-test-amd.yml).
• Blocked work: MiniMax-M2.7 nightly accuracy signal on AMD MI30x — single quality gate for this model on this hardware.
• In-flight fix: ❌ none found — needs new PR or investigation.

nightly-8-gpu-mi35x-glm5-mxfp4 — failed step(s): Accuracy Test MI35x (8-GPU GLM-5-MXFP4)

Commit Info

• sglang: ece8a1a78813f6f0bd8394b03589d144125d08fb
• aiter: a6bb499375849eec45d68c5ccaebc8865fd422c0 (source: dev image, v0.1.12.post2.dev150+ga6bb49937 — preserved, no rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_glm5_mxfp4_eval_mi35x.py	test_glm5_mxfp4_accuracy	AssertionError: Failed models: ['amd/GLM-5-MXFP4 (mxfp4)'] (server failed to launch — TP1 raised RuntimeError: The memory capacity is unbalanced)	Log

Failure Summary

The 8-GPU MI35x GLM-5-MXFP4 accuracy test failed because the SGLang server crashed at startup. TP rank 1 raised RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes. pre_model_load_memory=6.978515625, local_gpu_memory=8.681640625, local_gpu_memory * 0.9=7.8134765625 from model_runner.py:1231, aborting the multi-GPU init before any inference begins. The other TP ranks reported ~10 GB used after init, but TP1's local card had only ~8.68 GB free — i.e. another process was holding ~52 GB on GPU 1 of the runner.

Regression Status

Known recurring failure (2 consecutive failures, same test file, same root error).

Recent history of test_glm5_mxfp4_eval_mi35x.py (test_glm5_mxfp4_accuracy) in job nightly-8-gpu-mi35x-glm5-mxfp4 (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
May 1	25225678502	nightly-8-gpu-mi35x-glm5-mxfp4	❌ Failed	test_glm5_mxfp4_accuracy	RuntimeError: memory capacity is unbalanced (TP1 free=8.68 GB)
Apr 30	25180906638	nightly-8-gpu-mi35x-glm5-mxfp4	❌ Failed	test_glm5_mxfp4_accuracy	RuntimeError: memory capacity is unbalanced (TP1 free=0.65 GB)
Apr 29	25125309566	nightly-8-gpu-mi35x-glm5-mxfp4	✅ Passed	—	—
Apr 28	25069425979	nightly-8-gpu-mi35x-glm5-mxfp4	✅ Passed	—	—
Apr 27	25010934840	nightly-8-gpu-mi35x-glm5-mxfp4	✅ Passed	—	—

First observed failure: 2026-04-30 19:46 UTC. Last passing: 2026-04-29.

Failure Cluster

GPU memory unbalanced on MI35x runner — TP rank 1 sees insufficient free VRAM at distributed init (other process holding the card)

Facts (verified observations, no interpretation)

• TP1 raised RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes. pre_model_load_memory=6.978515625, local_gpu_memory=8.681640625, local_gpu_memory * 0.9=7.8134765625 (log line 2604).
• All other TP ranks (TP0, TP2-TP7) completed Init torch distributed ends with mem usage ≈ 10.07-10.22 GB (log lines 2580-2586). TP1 did not.
• Error originates from python/sglang/srt/model_executor/model_runner.py:1227-1231: the check pre_model_load_memory < local_gpu_memory * 0.9 (6.98 < 7.81) fires when one rank's GPU has notably more free memory than the others, indicating the other ranks already lost memory to a foreign process.
• Yesterday's failure on the same test had pre_model_load_memory=0.00390625, local_gpu_memory=0.65234375 — i.e. the runner was even more contended (only 0.65 GB free). Same root error, same code path.
• aiter version was a preserved dev build (v0.1.12.post2.dev150+ga6bb49937) — different from yesterday's v0.1.12.post1, but the failure happens before any aiter call, so aiter version is irrelevant.
• After the RuntimeError, parent process received sigquit from child, test got AssertionError: Failed models: ['amd/GLM-5-MXFP4 (mxfp4)'] (line 2615) and exited 255.
• Total elapsed before failure: ~39 seconds (server never started serving).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] MI35x runner GPU 1 is being shared with another process (or stale process) holding ~52 GB of VRAM, causing the TP-memory-balance check to abort init.
  • Supporting: error message text literally says "Some GPUs may be occupied by other processes"; only TP1 sees a discrepancy while TP0/2-7 are healthy; same fault recurred on a different runner state yesterday with even more contention (only 0.65 GB free); fault occurs before model load and before any sglang/aiter inference code runs.
  • Disconfirming: cannot directly verify the other process from the log (no rocm-smi output captured); two consecutive failures could in principle indicate a leak in a sglang teardown path that runs earlier in the same container, though no such co-tenant job is visible in the early log.
  • How to verify: SSH into the MI35x runner and run rocm-smi --showpids while idle to identify the foreign process; or add rocm-smi capture as a CI pre-step on this runner.
• [LOW] sglang code regression in TP init memory accounting between Apr 29 (passing) and Apr 30 (first failing).
  • Supporting: Apr 29 was the last green run, Apr 30 was the first red — temporal alignment.
  • Disconfirming: the check itself (model_runner.py:1227) is unchanged; the failure is a real GPU-memory imbalance reported by the AMD device driver, not a numerical calculation drift; the imbalance magnitude differs wildly between Apr 30 (0.65 GB) and May 1 (8.68 GB) — a code regression would produce stable numbers, runner contention would not; only this single runner/job shows it.
  • How to verify: bisect f34222da..7bb7f604 only if hypothesis 1 is conclusively ruled out by rocm-smi.

In-flight Fix Check

Searched is:pr is:open for memory capacity unbalanced (0 hits) and glm5-mxfp4 (1 unrelated hit — #20894 "Skip unnecessary testcases for CPU").

• In-flight fix: ❌ none found — needs runner-side investigation, not a code PR

Suggested Triage Steps

• Run rocm-smi --showpids and rocm-smi --showmeminfo vram on the MI35x runner that hosted job 73968531052 to identify which process holds ~52 GB on GPU 1.
• Check whether a previous job on the same runner (or a hung server from a prior nightly run) leaked GPU memory; if so, add a teardown / pkill -9 sglang / GPU-reset step to the AMD nightly job's pre-cleanup.
• If runner contention is confirmed, add rocm-smi --showpids as a fail-fast diagnostic at the top of nightly-test-amd.yml 8-GPU jobs so future occurrences are immediately visible in the log.
• If no foreign process is found, bisect sglang f34222da..7bb7f604 (Apr 29 → Apr 30 commit window) — but only after the runner-side check.
• Consider toggling SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 for this specific test to convert the hard error into a warning and let the model-load attempt proceed (diagnostic only — does not fix root cause).

Status

• Persistence: Failing for 2 consecutive scheduled nightly runs (since 2026-04-30 19:46 UTC). Apr 29 and earlier passed.
• Scope: 1 job in 1 workflow (nightly-8-gpu-mi35x-glm5-mxfp4 in nightly-test-amd.yml). Other MI35x 8-GPU jobs in the same nightly run (e.g. glm51, grok2, kimi-k25, deepseek-r1-mxfp4) had a mix of pass/fail with different errors — this specific failure pattern does not appear cross-job.
• Blocked work: GLM-5-MXFP4 accuracy CI signal on AMD MI35x.
• In-flight fix: ❌ none found — needs runner-side investigation, not a code PR

nightly-8-gpu-mi35x-grok2 — failed step(s): Accuracy Test MI35x (8-GPU Grok2)

Commit Info

• sglang: ece8a1a
• aiter: N/A (no [CI-AITER-CHECK] markers in this nightly job log)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_grok2_eval_mi35x.py	test_grok2_accuracy	Exception: Server process exited with code -9 (server-side: RuntimeError: The memory capacity is unbalanced …)	Log

Failure Summary

The Grok-2 server-launch step in test_grok2_accuracy aborted ~7 s after Init torch distributed ended. TP rank 1 reported pre_model_load_memory=7.51 GB while sister ranks reported local_gpu_memory=9.23 GB, breaking the pre_model_load_memory >= 0.9 * local_gpu_memory invariant in model_runner.py:1227, which raised RuntimeError("The memory capacity is unbalanced …"). The scheduler exception caused sglang to send SIGQUIT to the parent, and the server exited with code -9.

Regression Status

New regression — first failure of this signature on this test in the queryable window.

Recent history of test/registered/amd/accuracy/mi35x/test_grok2_eval_mi35x.py::test_grok2_accuracy in job nightly-8-gpu-mi35x-grok2 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
May 01	25225678502	nightly-8-gpu-mi35x-grok2	❌ Failed	test_grok2_accuracy	Server exit -9 (TP-memory unbalanced)
Apr 30	25180906638	nightly-8-gpu-mi35x-grok2	✅ Passed	—	—
Apr 29	25125309566	nightly-8-gpu-mi35x-grok2	✅ Passed	—	—
Apr 28	25069425979	nightly-8-gpu-mi35x-grok2	✅ Passed	—	—
Apr 27	25010934840	nightly-8-gpu-mi35x-grok2	✅ Passed	—	—
Apr 26	24963018193	nightly-8-gpu-mi35x-grok2	❌ Failed	test_grok2_accuracy	ValueError: Unrecognized model in alvarobartt/grok-2-tokenizer (different signature, not related)

First observed failure with this signature: May 1 ~19:54 UTC. Last passing: Apr 30 ~17:55 UTC.

Failure Cluster

GPU memory unbalanced at TP init (rank-1 GPU pre-occupied by stray process)

Facts (verified observations, no interpretation)

• Verbatim error from log line 2536: RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes. pre_model_load_memory=7.509765625, local_gpu_memory=9.228515625, local_gpu_memory * 0.9=8.3056640625. (Per-rank Init torch distributed ends lines show TP0/TP2/TP3 mem=9.69 GB, TP4/TP6/TP7 mem=9.56 GB, TP5 mem=9.62 GB — only TP1 reports the lower 7.5 GB available, indicating ~1.7 GB pre-occupied on TP1's GPU.)
• Server command: sglang serve --model-path xai-org/grok-2 --tp 8 --quantization fp8 --attention-backend aiter --mem-fraction-static 0.85 … (log line 2555).
• Failure happens during init_torch_distributed, before model weights are loaded — strict TP memory check (SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=1) at python/sglang/srt/model_executor/model_runner.py:1227-1231.
• Earliest log line 807: Local docker registry pull failed; falling back to public registry: rocm/sgl-dev:v0.5.10.post1-rocm700-mi35x-20260501 — the local registry was unavailable; the public image was used. (Indicates the runner had network/infra hiccups around launch time, but does not by itself explain GPU memory contention.)
• The previous 4 scheduled runs (Apr 27–30) on the same job/branch passed; Apr 26 failure was a separate tokenizer-config issue (different stack, see history table).
• Regression window contains 39 sglang commits (7bb7f60..ece8a1a). Of these, only ece8a1a and da7f890 touch model_executor/model_runner.py (verified via git log --oneline 7bb7f6049a..ece8a1a -- python/sglang/srt/model_executor/model_runner.py).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Runner-environment leak: a stray process held ~1.7 GB on the GPU bound to TP rank 1. The unbalanced-memory error is the model_runner.py:1227 check working as designed: TP0/2-7 all see ~9.6 GB free while only TP1 sees 7.5 GB free.

  • Supporting: Per-rank memory readout shows a clean delta isolated to one of the 8 GPUs; the symptom is intrinsically environmental (stale CUDA context / lingering process), not in any code path that the test exercises. Dockerized server launch happened seconds after a (failed) local-registry pull — possibly indicating infrastructure churn on this runner just before launch.
  • Disconfirming: The check is unconditional whenever SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=1; an environment leak should also surface in other 8-GPU MI35x nightly jobs but I did not enumerate sister-job failures in the same nightly run.
  • How to verify: On the next scheduled run (or a manual rerun), inspect rocm-smi --showpids / process list before launching the server on the runner; if a leftover process is found on one card, the runner's pre-test cleanup hook is leaking.

• [LOW] Regression introduced by one of the 39 sglang commits in 7bb7f60..ece8a1a. Only two of these touch model_runner.py (ece8a1a device-timer refactor, da7f890 Intel XPU attention).

  • Supporting: Temporal correlation — Apr 30 passed, May 1 failed.
  • Disconfirming: Neither commit modifies init_torch_distributed, get_available_gpu_memory, the pre_model_load_memory check, or any AMD device init path. The error is "memory pre-occupied", which is environmental rather than a sglang init regression. The other AMD commit in the window — a578bf8 "[AMD] fix moriep unittest failure" — touches MoE export plumbing, not memory init.
  • How to verify: Wait for the next nightly schedule. If the failure recurs with the same TP rank reporting low memory, environmental leak is confirmed (this hypothesis ruled out). If it recurs with random ranks failing, code regression becomes a stronger candidate and the window is small enough to bisect at PR granularity (git bisect 7bb7f6049a..ece8a1a78).

• [SPECULATION] AITER bump 7e17a7a "Upd: AITER->#2879 a6bb499" pulled new aiter ROCm/aiter.git@a6bb499 with a kernel that allocates persistent memory before sglang's TP check.

  • Supporting: AITER bumps land in this 39-commit window; the test enables --attention-backend aiter.
  • Disconfirming: The crash happens during init_torch_distributed, before any aiter forward path is touched. The asymmetry is on a single rank, not all ranks (an aiter persistent allocator would affect all ranks).
  • How to verify: Not actionable until the environmental hypothesis is ruled out.

No hypothesis above MEDIUM confidence; the most likely explanation is a single-runner environmental leak.

In-flight Fix Check

Searched open sglang PRs with keywords memory unbalanced, grok2 mi35x, nightly mi35x, SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK:

• In-flight fix: ❌ none found in open PRs — needs runner-side investigation or new PR.
• API call: GET /search/issues?q=repo:sgl-project/sglang+is:pr+is:open+memory+unbalanced → no relevant matches; ... +nightly+mi35x → no matches; ... +SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK → only #18037 (adds a warning, not a fix for this scenario).
• Note: #23314 ("ci: limit nightly test parallelism to 1 job per hardware type") landed inside this regression window as commit 357e378 and is the closest existing mitigation for cross-job GPU contention; it is already merged, so any leftover process must come from within the same job's earlier processes or from a runner that wasn't fully drained before the job started.

Suggested Triage Steps

• Re-run the same nightly job manually (workflow_dispatch) on the same MI35x runner. If it passes, the failure is the classic "stale GPU process" flake; mark this as flaky-on-infra and continue to monitor.
• If it fails again with the same per-rank memory asymmetry, SSH into the runner and run rocm-smi --showpids before the test step; identify and kill the leftover process. Inspect the runner's pre-job cleanup hook to ensure leftover sglang/aiter processes from prior jobs are reaped.
• If the failure persists across multiple reruns AND across runners, bisect the 39-commit window 7bb7f6049a..ece8a1a78; start with ece8a1a (device-timer refactor) and 7e17a7a (aiter bump) since they are the most plausible code-level vectors for an init-time GPU footprint change.
• As a temporary unblock (only if needed), the test could be re-run with SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 to convert the hard error to a warning — this is a diagnostic, not a fix, and would mask future genuine regressions.

Status

• Persistence: 1 failed run so far (~2 hours since first failure at May 01 19:54 UTC); previous 4 scheduled runs passed.
• Scope: Single job (nightly-8-gpu-mi35x-grok2) in nightly-test-amd.yml; cross-job MI35x impact not assessed in this report.
• Blocked work: Grok-2 nightly accuracy signal on MI35x.
• In-flight fix: ❌ none found — needs runner-side investigation or new PR.

nightly-perf-8-gpu-mi35x-deepseek-v32-mtp — failed step(s): Performance Test MI35x (8-GPU DeepSeek-V3.2 MTP)

Commit Info

• sglang: ece8a1a
• aiter: a6bb499 (source: dev image — v0.1.12.post2.dev150+ga6bb49937 detected, rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/perf/mi35x/test_deepseek_v32_mtp_perf_mi35x.py	test_bench_one_batch (TestNightlyDeepseekV32MTPPerformance)	Exception: Server process exited with code -9 (root: RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes. pre_model_load_memory=6.978..., local_gpu_memory=8.681..., local_gpu_memory * 0.9=7.813...)	Log

Failure Summary

The sglang serve launch for DeepSeek-V3.2 + EAGLE/MTP on TP=8 MI35x failed during init_torch_distributed because the per-rank available-GPU-memory check tripped: pre_model_load_memory (min over ranks) = 6.978 GB vs this rank's local_gpu_memory = 8.681 GB → ~1.7 GB asymmetry across ranks. TP1 raised, the server died with exit code -9, and popen_launch_server failed the test.

Regression Status

Recurring failure (2-day regression). Started Apr 30, failed again May 1 with the identical error message and stack frame.

Recent history of test_deepseek_v32_mtp_perf_mi35x.py (test_bench_one_batch) in job nightly-perf-8-gpu-mi35x-deepseek-v32-mtp (branch=main, event=schedule, completed only):

Date	Run	Job	Test File Status	Failed Function	Error
May 01	25225678502	perf-mi35x-dsv32-mtp	❌ Failed	test_bench_one_batch	Server exit -9 / memory unbalanced
Apr 30	25180906638	perf-mi35x-dsv32-mtp	❌ Failed	test_bench_one_batch	Server exit -9 / memory unbalanced (pre=11.705, local=13.408)
Apr 29	25125309566	perf-mi35x-dsv32-mtp	✅ Passed	—	—
Apr 28	25069425979	perf-mi35x-dsv32-mtp	✅ Passed	—	—
Apr 27	25010934840	perf-mi35x-dsv32-mtp	✅ Passed	—	—

First observed failure: 2026-04-30 20:23 UTC. Last passing: 2026-04-29 (head 530b497a). Failure window: 530b497a..7bb7f604.

Failure Cluster

TP memory-imbalance check trips at server launch (DeepSeek-V3.2 EAGLE/MTP on 8× MI35x).

Facts (verified observations, no interpretation)

• Stack: model_runner.py:1231 → init_torch_distributed → raise RuntimeError(msg) (log line). The check is if pre_model_load_memory < local_gpu_memory * 0.9: raise (gated by SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK, default True).
• TP1 was the rank that raised. All ranks reported Init torch distributed ends. elapsed=~7.5 s, mem usage≈10.1 GB — meaning ~10 GB was already used in distributed setup before the model loaded.
• Server cmd: sglang serve --model-path deepseek-ai/DeepSeek-V3.2 --tp 8 --nsa-prefill-backend tilelang --nsa-decode-backend tilelang --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --mem-fraction-static 0.7 ... (log line).
• aiter unchanged across passing & failing runs (dev image preserved both nights — Dockerfile pin a6bb499).
• 2 of 2 completed runs since Apr 30 reproduce; identical stack, identical asymmetry direction (~1.7 GB on May 1, ~1.7 GB on Apr 30).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Commit 79dbfe4 ("Use spec v2 by default", #21062) — flips SGLANG_ENABLE_SPEC_V2 from False → True in python/sglang/srt/environ.py, exactly matching the regression boundary.
  • Supporting: lands Apr 29 20:40 UTC, before the Apr 30 17:55 UTC nightly trigger; the failing test is the only AMD MI35x EAGLE/MTP perf job; this is the only commit in the window that directly toggles spec-decoding behavior; the failing test exercises exactly that code path (--speculative-algorithm EAGLE).
  • Disconfirming: the imbalance is reported at init_torch_distributed before model load, so a spec-v2 induced asymmetry must originate during torch/distributed setup (e.g., draft-worker context init) rather than later allocation. Also, the sister nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp has been failing since Apr 29 (one day earlier than this perf job), so spec-v2-default cannot explain the accuracy regression by itself.
  • How to verify: re-run with SGLANG_ENABLE_SPEC_V2=0 on AMD MI35x against deepseek-ai/DeepSeek-V3.2 + EAGLE/MTP; if launch succeeds, confirms hypothesis. Also: git revert 79dbfe4 on a branch and re-run the test.
• [LOW] Runner-side residual GPU memory (stale process leaks ~1.7 GB on a subset of ranks).
  • Supporting: error message itself reads "Some GPUs may be occupied by other processes"; same-runner job-to-job leakage is plausible on shared MI35x hosts.
  • Disconfirming: same job passed Apr 27/28/29 on the same runner pool with no cleanup change in the window; failure reproduces on two different days/runners (Apr 30 and May 1) with the same direction of asymmetry — too consistent for random environmental leak.
  • How to verify: rocm-smi --showpids in the prologue of the workflow; confirm no orphan PIDs hold memory at launch.
• [LOW] Commit 7389743 ("modelexpress p2p RDMA transfer", #23105) — touches distributed init paths.
  • Supporting: in window, modifies code reachable from init_torch_distributed.
  • Disconfirming: feature is opt-in (RDMA transfer flags not set in the failing command line).
  • How to verify: not actionable until the spec-v2 hypothesis is checked first.

No hypothesis above MEDIUM confidence; needs an A/B test of SGLANG_ENABLE_SPEC_V2=0 or a git bisect between 530b497a..7bb7f604 (38 commits).

In-flight Fix Check

Searched open sglang PRs for "memory capacity unbalanced", "spec v2 memory", "deepseek_v32_mtp":

• In-flight fix: ❌ none found in open PRs — needs new PR or investigation. Existing open spec-v2 PRs (#23000, #21895, #20916) are feature work, not regression fixes for this symptom.

Suggested Triage Steps

• Reproduce on MI35x with the failing command line and SGLANG_ENABLE_SPEC_V2=0 — if it launches, the spec-v2 default flip is implicated and the maintainer can decide whether to revert 79dbfe4, guard MTP perf jobs with the env, or fix spec-v2's per-rank memory accounting.
• If spec-v2 is implicated, A/B the imbalance threshold by setting SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 to confirm the launch otherwise succeeds and the rest of the test would have proceeded (validates that this is purely a startup gate rather than a downstream OOM).
• Bisect commits in 530b497a..7bb7f604 (~38 commits) on the MI35x runner if the spec-v2 A/B does not exonerate or implicate cleanly.
• Check rocm-smi --showpids at job start to rule out runner-side stale-process leakage on the MI35x pool (confirms or eliminates the LOW hypothesis).
• Investigate the sister nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp separately — it regressed Apr 29 (one day earlier) with a likely different cause; do not assume a single root.

Status

• Persistence: Failing for 2 consecutive completed scheduled runs (since 2026-04-30 20:23 UTC); 100% reproduction rate (2/2).
• Scope: 1 job (nightly-perf-8-gpu-mi35x-deepseek-v32-mtp) in nightly-test-amd.yml. Sister nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp is also failing but on a different timeline — likely a different root cause and out of scope for this report.
• Blocked work: AMD MI35x DeepSeek-V3.2 + MTP perf signal lost for 2 days; speculative-decoding regressions on the largest model are invisible until fixed.
• In-flight fix: ❌ none found in open PRs — needs new PR or investigation.

nightly-accuracy-2-gpu-vlm — failed step(s): Nightly Accuracy Test (2-GPU VLM MMMU)

Commit Info

• sglang: ece8a1a78813f6f0bd8394b03589d144125d08fb
• aiter: a6bb499375849eec45d68c5ccaebc8865fd422c0 (source: Dockerfile default — bumped from v0.1.12.post1 by sglang #24218 which landed in this run window)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi30x/test_vlms_mmmu_eval_amd.py	test_mmmu_vlm_models (process aborted before unittest finished)	HW Exception by GPU node-3 ... reason: GPU Hang → Fatal Python error: Aborted (exit 134) during Qwen/Qwen2.5-VL-7B-Instruct MMMU eval	Log

Failure Summary

The single VLM-MMMU test step crashed twice with exit 134 (SIGABRT). Both attempts hung on Qwen/Qwen2.5-VL-7B-Instruct: the GPU emitted "HW Exception … reason: GPU Hang" and the scheduler aborted while still inside the vision tower's Conv forward (transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py:113 → sglang/srt/models/qwen2_5_vl.py:415). No model-by-model accuracy summary was produced.

Regression Status

New regression — today's failure mode (GPU Hang / SIGABRT during Qwen2.5-VL vision forward) has never been observed in this job before. Past 4 completed runs all failed with exit 255 (assertion: VLM accuracy below threshold) on different VLM models, and never reached a GPU-hang signature.

Recent history of test_vlms_mmmu_eval_amd.py (test_mmmu_vlm_models) in job nightly-accuracy-2-gpu-vlm (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
May 01	25225678502	vlm	❌ Failed (exit 134)	aborted in Qwen2.5-VL-7B-Instruct forward	GPU Hang / Fatal Python error: Aborted
Apr 30	25180906638	vlm	❌ Failed (exit 255)	test_mmmu_vlm_models	accuracy below threshold for 5 VLM models (Qwen2-VL, Qwen2.5-VL, Qwen3-VL-30B-A3B, MiniCPM-o-2_6, deepseek-vl2-small)
Apr 29	25125309566	vlm	❌ Failed (exit 255)	test_mmmu_vlm_models	accuracy below threshold
Apr 28	25069425979	vlm	❌ Failed (exit 255)	test_mmmu_vlm_models	accuracy below threshold
Apr 27	25010934840	vlm	❌ Failed (exit 255)	test_mmmu_vlm_models	accuracy below threshold

First observed (today's signature): 2026-05-01 18:17 UTC. The pre-existing accuracy-threshold failure has not had a known passing run within the queryable lookback (≥5 days); it's a Never-passed parent issue, but today's GPU hang is a fresh, distinct failure stacked on top of it.

Failure Cluster

GPU Hang during Qwen2.5-VL vision tower conv forward (AMD MI325)

Facts (verified observations, no interpretation)

• Direct log evidence (verbatim, lines 5290–5300 of the job log):
  • HW Exception by GPU node-3 (Agent handle: 0x5602d6224a90) reason :GPU Hang (5 occurrences, all node-3)
  • Fatal Python error: Aborted
  • Top of crashing thread stack: torch/nn/modules/conv.py:712 _conv_forward → transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py:113 forward → sglang/srt/models/qwen2_5_vl.py:415 forward → qwen2_5_vl.py:682 get_image_feature → mm_utils.py:563/628/728/822/1052 → qwen2_5_vl.py:765 forward
• Process exit: ##[error]Process completed with exit code 134. at 2026-05-01 18:17:13 UTC
• Two attempts (per amd_ci_exec.sh retry logic): attempt 1 normal mode, attempt 2 with HF_HUB_OFFLINE=1. Both attempts crashed at the same point — Qwen2.5-VL forward → GPU Hang.
• Attempt 1 also showed earlier [2026-05-01 18:04:39] Health check failed. Server couldn't get a response from detokenizer for last 20 seconds on Qwen/Qwen2-VL-7B-Instruct (the previous model in the matrix); attempt 2 made it past Qwen2-VL-7B (passed at score 0.300/threshold 0.30) before hanging on Qwen2.5-VL.
• Attempt 2 also surfaced a transient huggingface_hub.errors.LocalEntryNotFoundError → OSError: We couldn't connect to 'https://huggingface.co' for Qwen2.5-VL config under HF_HUB_OFFLINE=1; the per-model CI_OFFLINE retry then succeeded with HF_HUB_OFFLINE=0 before the GPU hang.
• Aiter version delta vs the last completed run:
  • Apr 30 run 25180906638 — [CI-AITER-CHECK] Dockerfile expects AITER_COMMIT=v0.1.12.post1 ; image had v0.1.12.post1.
  • May 01 run 25225678502 — [CI-AITER-CHECK] Dockerfile expects AITER_COMMIT=a6bb499375849eec45d68c5ccaebc8865fd422c0 ; image had v0.1.12.post2.dev150+ga6bb49937.
• Code paths in stack: only sglang multimodal (mm_utils, qwen2_5_vl) and HF transformers conv (modeling_qwen2_5_vl.py:113). No direct aiter call in the crashing frames; aiter is invoked elsewhere in the same forward (the prior [aiter] shape is M:25116, ... log lines fire shortly before the hang).
• Runner: linux-mi325-2gpu-sglang-455ss-runner-nsxc5 (MI325, 2 GPU). GPU "node-3" hung.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] AITER version bump in this run window destabilises Qwen2.5-VL execution — sglang #24218 (7e17a7a) bumps AITER_COMMIT_DEFAULT from v0.1.12.post1 → a6bb499 (i.e. a delta of all of ROCm/aiter v0.1.12.post1..a6bb499).
  • Supporting: lands in regression window (Apr 30 → May 1); the [CI-AITER-CHECK] markers show the image actually picked it up; the new failure is an AMD-specific GPU Hang affecting the same job that previously only failed on accuracy; aiter is invoked all over Qwen2.5-VL's transformer blocks (visible from [aiter] shape is M:25116/... log lines firing seconds before the hang).
  • Disconfirming: the topmost crashing frame is a torch Conv2d._conv_forward inside HF transformers' qwen2_5_vl, not an aiter call site; GPU Hang at the kernel level is a generic AMD-side symptom that can be triggered by many things (driver/MIOpen state, memory pressure, prior bad kernel state); the prior model Qwen/Qwen2-VL-7B-Instruct did pass on attempt 2 (score 0.300, threshold 0.30), so aiter is not breaking all VLMs.
  • How to verify: re-run nightly-accuracy-2-gpu-vlm with AITER_COMMIT_OVERRIDE=v0.1.12.post1 (i.e. revert just the aiter image) and compare — if Qwen2.5-VL stops hanging, aiter is implicated. If it still hangs, look at the sglang delta below.
• [LOW] MoE/aiter integration refactor — sglang #23597 (108bfd8) "[MoE] Add Aiter MoE runner backend and purge aiter.fused_moe from quant methods".
  • Supporting: lands in window; touches aiter call paths.
  • Disconfirming: Qwen2.5-VL is a dense (non-MoE) model; the crashing frame is a vision-tower Conv2d, not a MoE op; the [aiter] shape ... log lines just before the hang are GEMM shapes, not MoE shapes.
  • How to verify: only worth investigating if the AITER override A/B is inconclusive.
• [LOW] MUSA/qwen series support — sglang #23654 (71e89e9) "[MUSA][19/N] Support qwen series models".
  • Supporting: title mentions qwen.
  • Disconfirming: this is a MUSA (Moore Threads) backend addition; MUSA paths are not exercised on AMD ROCm CI; unlikely to affect MI325 forward pass.
  • How to verify: read the diff to confirm it only adds MUSA-gated code; if so, dismiss.

If none of the above reaches MEDIUM after the AITER A/B test, this is a generic AMD GPU-Hang and warrants git bisect across 7bb7f6049a..ece8a1a788 (39 sglang commits) plus the aiter range above.

In-flight Fix Check

• Search 1: repo:sgl-project/sglang is:pr is:open aiter vlm → only stale PRs (#21720, #18263); no relevant in-flight fix.
• Search 2: repo:sgl-project/sglang is:pr is:open qwen2_5_vl amd → no relevant in-flight fix.
• Search 3: repo:sgl-project/sglang is:pr is:open GPU hang aiter → no relevant in-flight fix.
• In-flight fix: ❌ none found — needs new investigation

Suggested Triage Steps

• Re-run this job with AITER_COMMIT_OVERRIDE=v0.1.12.post1 (the previous default) to A/B-test whether the AITER bump in #24218 is the trigger. If the GPU Hang disappears, narrow within v0.1.12.post1..a6bb499 on the aiter side.
• If aiter A/B is inconclusive, bisect sglang in 7bb7f6049a..ece8a1a788 running only the failing matrix point (Qwen/Qwen2.5-VL-7B-Instruct MMMU 100 samples) on a 2-GPU MI325 to keep iteration cost down.
• Capture a rocm-smi snapshot and dmesg | grep -i 'gpu\|hang\|amdgpu' from the runner immediately after a reproduction; the bare "HW Exception … reason: GPU Hang" log is symptom-only and does not name the offending kernel.
• Independently, the pre-existing accuracy-threshold failure for the other VLM models (Qwen2-VL, Qwen3-VL-30B-A3B, MiniCPM-o-2_6, deepseek-vl2-small) needs its own investigation — this run did not get far enough to surface it, but it has been failing every day for ≥5 days and is a separate "Never-passed" issue that should be tracked outside today's GPU-Hang regression.

Status

• Persistence: GPU-Hang signature is new today (2026-05-01 18:17 UTC, both attempts in one run). Underlying job has been failing daily for ≥5 days, but with a different signature (accuracy threshold), so today is genuinely a new regression on top of an existing red job.
• Scope: 1 job (nightly-accuracy-2-gpu-vlm) on 1 workflow (nightly-test-amd.yml); only confirmed on AMD MI325 2-GPU runner. Whether other AMD nightly jobs also hang on Qwen2.5-VL today was not verified in this analysis (out of scope for per-job).
• Blocked work: Blocks AMD nightly VLM accuracy signal. Pre-existing accuracy issue continues to mask the underlying VLM correctness signal.
• In-flight fix: ❌ none found — needs new PR or investigation.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-05-02 22:27 UTC)

[amd-bot]

nightly-test-amd.yml — Per-Job Analysis (continued 3/6)

nightly-accuracy-8-gpu-mi35x — failed step(s): Accuracy Test MI35x (8-GPU GPT-OSS)

Commit Info

• sglang: ece8a1a7 — "Refactor device timer, clean up metrics collector, and add fwd occupancy metric (#24197)"
• aiter: a6bb4993 (source: dev image — [CI-AITER-CHECK] AITER version inside CI image: v0.1.12.post2.dev150+ga6bb49937 → skipping rebuild; Dockerfile expects same SHA, no AITER_COMMIT_OVERRIDE)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_gpt_oss_eval_mi35x.py	test_gpt_oss_accuracy	AssertionError: Failed models: ['openai/gpt-oss-20b', 'openai/gpt-oss-120b'] (both server launches aborted by RuntimeError: The memory capacity is unbalanced)	Log

The companion test test_qwen3_coder_next_eval_mi35x.py was scheduled but never executed because the runner exited with code 255 after the first-test failure (Test Summary: 0/2 passed).

Failure Summary

The gpt-oss-20b and gpt-oss-120b server launches both aborted during distributed init. On TP rank 1 the per-rank check at python/sglang/srt/model_executor/model_runner.py:1227 (if pre_model_load_memory < local_gpu_memory * 0.9) raised RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes. pre_model_load_memory=6.978515625, local_gpu_memory=8.681640625, local_gpu_memory * 0.9=7.8134765625. The other 7 ranks completed Init torch distributed reporting mem usage=10.07–10.22 GB for the second (120B) attempt; the same pattern repeated for the 20B attempt that ran first.

Regression Status

Recurring failure (2 consecutive days) — first failed 2026-04-30, last passing run 2026-04-29.

Recent history of test/registered/amd/accuracy/mi35x/test_gpt_oss_eval_mi35x.py::test_gpt_oss_accuracy in job nightly-accuracy-8-gpu-mi35x (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
May 01	25225678502	nightly-accuracy-8-gpu-mi35x	❌ Failed	test_gpt_oss_accuracy	RuntimeError: memory capacity is unbalanced (free=8.68 GB)
Apr 30	25180906638	nightly-accuracy-8-gpu-mi35x	❌ Failed	test_gpt_oss_accuracy	RuntimeError: memory capacity is unbalanced (free=0.65 GB)
Apr 29	25125309566	nightly-accuracy-8-gpu-mi35x	✅ Passed	—	—
Apr 28	25069425979	nightly-accuracy-8-gpu-mi35x	✅ Passed	—	—
Apr 27	25010934840	nightly-accuracy-8-gpu-mi35x	✅ Passed	—	—
Apr 26	24963018193	nightly-accuracy-8-gpu-mi35x	✅ Passed	—	—

First-observed failure: 2026-04-30 18:13 UTC. Last green: 2026-04-29.

Failure Cluster

"GPU memory imbalance at TP init on mi35x runner (tiny free-memory readings)"

Facts (verified observations, no interpretation)

• Verbatim error: RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes. pre_model_load_memory=6.978515625, local_gpu_memory=8.681640625, local_gpu_memory * 0.9=7.8134765625 (job log line 2807).
• Same exception repeated for both server launches in this run: lines 2567 (gpt-oss-20b attempt) and 2807 (gpt-oss-120b attempt) — same numbers (6.97 / 8.68).
• Yesterday's failure (Apr 30 log) showed the same exception with pre_model_load_memory=0.00390625, local_gpu_memory=0.65234375 — both readings far below the ~192 GB physical capacity of MI35x (gfx950).
• Other 7 ranks completed Init torch distributed successfully reporting mem usage of 10.07–10.22 GB (lines 2543-2549, 2783-2789), so it's not a model-load OOM — it's the pre-load TP imbalance check that fires.
• The triggering check is model_runner.py:1227 (raise gated on SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK). This file/check has not been modified inside the regression window (git log 530b497a..7bb7f604 -- python/sglang/srt/model_executor/: 0 commits).
• Same workflow, same branch, same event triggered → 2 consecutive failures; 4 prior consecutive passes.
• aiter version is unchanged across passing & failing runs (v0.1.12.post2.dev150+ga6bb49937 dev image = Dockerfile pin a6bb4993...); no override.
• Sister job nightly-accuracy-8-gpu-mi35x-deepseek-v32 passed on 2026-04-30 (same run 25180906638) — meaning the runner pool is not uniformly broken; the failing job ran on a contaminated host while the deepseek-v32 job ran on a clean one.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Dirty mi35x CI runner — leftover GPU memory from a previous job (zombie processes / leaked HIP IPC handles).

  • Supporting: Free GPU memory at startup is ~6.98 / 8.68 GB on a 192 GB MI35x — physically impossible without another process holding ~183 GB. Two days in a row, only the gpt-oss accuracy job is affected; sister job nightly-accuracy-8-gpu-mi35x-deepseek-v32 on the same workflow run passed (different runner instance). The codepath that raises (init_torch_distributed) and the env knob (SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK) are unchanged in the 38-commit regression window. There are pre-existing dev branches on the repo named fix/gpu-cleanup-between-tests, fix/ci-cleanup-stale-ipc, fix/ci-env-source-cleanup (visible in the git fetch output at log line 391-411), suggesting maintainers are already aware of mi35x runner cleanup gaps.
  • Disconfirming: I did not find a corresponding rocm-smi/process-list dump in the log to prove a residual process exists; the imbalance check could in principle be tripped by an HSA reporting glitch even when memory is not actually allocated. Also, no open PR explicitly titled around "memory imbalance" / "mi35x cleanup" was returned by the keyword search.
  • How to verify: ssh to the mi35x runner and run rocm-smi --showmeminfo vram --showpids between jobs; or add rocm-smi dump as a pre-test step in nightly-test-amd.yml for nightly-accuracy-8-gpu-mi35x. If a stale PID is shown, kill + reset the GPU (rocm-smi --gpureset) and rerun.

• [LOW] A code change in the 04-29..04-30 window leaks GPU memory in a prior nightly test, contaminating later jobs on the same runner.

  • Supporting: The regression starts cleanly on 04-30; some commit in git log 530b497a..7bb7f604 (38 commits) could have introduced an unreleased CUDA/HIP context. Candidate code areas worth scanning: anything touching cuda graphs, torch.cuda.empty_cache, multi-process scheduler shutdown, pynvml/rocml resource cleanup.
  • Disconfirming: 0 commits in the window modify python/sglang/srt/model_executor/ or python/sglang/srt/managers/. The leak (if any) would be in another job/test that runs before this one on the same runner — not in code this job exercises. The 04-30 numbers are dramatically worse (0.65 GB free) than the 05-01 numbers (8.68 GB free), which is more consistent with non-deterministic infra cleanup than with a deterministic code-level leak.
  • How to verify: bisect 530b497a..7bb7f604 by re-running nightly-accuracy-8-gpu-mi35x against intermediate SHAs on a freshly-rebooted mi35x host.

• [LOW] Spurious imbalance check trip from a transient HSA/HIP free-memory reading right after RCCL warmup.

  • Supporting: The check fires on TP1 only; other ranks complete Init torch distributed normally and the absolute free memory (~8 GB) is suspiciously round.
  • Disconfirming: The reading is consistent across a 30-second retry window (both 20B and 120B attempts get 6.97/8.68 in this run), so it's not a transient blip. And on Apr 30 the readings are radically different (0.65 GB), arguing against a fixed reporting artifact.
  • How to verify: instrument get_available_gpu_memory on the runner before/after dist.barrier() and compare with rocm-smi.

No hypothesis above MEDIUM confidence; needs runner-level inspection (rocm-smi --showmeminfo vram --showpids) before code-level bisect.

In-flight Fix Check

Searched: repo:sgl-project/sglang is:pr is:open with keywords gpu cleanup, memory unbalanced, memory capacity unbalanced, gpt-oss mi35x. None of the returned PRs match the symptom (top hits — #24222, #23848, #24246 — are unrelated). However, branches named fix/gpu-cleanup-between-tests, fix/ci-cleanup-stale-ipc, fix/ci-env-source-cleanup exist on origin (seen in this run's git fetch output) — they may be in-flight work that has not yet been pushed as a PR.

• In-flight fix: ❌ none found in open PRs — needs new PR or runner-side investigation. (Worth pinging whoever owns the fix/ci-cleanup-stale-ipc / fix/gpu-cleanup-between-tests branches.)

Suggested Triage Steps

• Add rocm-smi --showmeminfo vram --showpids (and pgrep -af python) as a pre-test step for nightly-accuracy-8-gpu-mi35x in .github/workflows/nightly-test-amd.yml to confirm the runner is contaminated and identify the leftover process.
• Identify which job runs on the mi35x runner immediately before nightly-accuracy-8-gpu-mi35x (check the runner labels); inspect its teardown for unkilled python / sglang processes or unreleased HIP IPC handles.
• If contamination is confirmed, contact the runner-pool owner (or the author of the fix/gpu-cleanup-between-tests branch) to land a per-job rocm-smi --gpureset + pkill -9 -f sglang cleanup hook.
• If hypothesis 1 is disproven (runner is clean and local_gpu_memory truly is 8 GB), bisect 530b497a..7bb7f604 (38 commits) on a freshly-rebooted host to find a sglang-side memory leak in a co-resident test.
• Re-run the failing job once on a known-clean mi35x runner to confirm the underlying gpt-oss accuracy code path is intact (separating runner contamination from any latent test bug).

Status

• Persistence: Failing for 2 consecutive scheduled runs (since 2026-04-30 18:13 UTC). Last green: 2026-04-29 (25125309566).
• Scope: 1 job in 1 workflow (nightly-accuracy-8-gpu-mi35x in nightly-test-amd.yml). Sister jobs nightly-accuracy-8-gpu-mi35x-deepseek-v32 and …-deepseek-v32-mtp on the same workflow run passed today, narrowing scope to a single runner host or a specific co-resident test sequence.
• Blocked work: Blocks the gpt-oss-20b + gpt-oss-120b accuracy signal on AMD MI35x; no other test files in this job were reached.
• In-flight fix: ❌ none found in open PRs — needs new PR or runner-side investigation.

nightly-8-gpu-mi35x-glm51 — failed step(s): Accuracy Test MI35x (8-GPU GLM-5.1 NSA)

Commit Info

• sglang: ece8a1a78813f6f0bd8394b03589d144125d08fb
• aiter: a6bb499375849eec45d68c5ccaebc8865fd422c0 (source: dev image preserved — [CI-AITER-CHECK] Dev/patched version detected: v0.1.12.post2.dev150+ga6bb49937 → skipping rebuild)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_glm51_eval_mi35x.py	TestGLM51EvalMI35x.test_glm51_accuracy	AssertionError: Failed models: ['zai-org/GLM-5.1-FP8 (nsa)'] (server failed to start: RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes. pre_model_load_memory=6.978515625, local_gpu_memory=8.681640625, local_gpu_memory * 0.9=7.8134765625)	Log

Failure Summary

The zai-org/GLM-5.1-FP8 server (TP=8, nsa attention backend, tilelang prefill+decode) aborted during init_torch_distributed on all 8 ranks. Each TP worker observed only ~8.68 GB free GPU memory (vs. expected ~190 GB on MI355x), and the per-rank symmetry check at python/sglang/srt/model_executor/model_runner.py:1227-1231 raised because pre_model_load_memory (6.98 GB) was below local_gpu_memory * 0.9 (7.81 GB), tripping the SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK guard. Total test wall time was only 39 s — failure is entirely in distributed init, no model weights were loaded.

Regression Status

New regression — first observed in today's nightly run.

Recent history of test_glm51_eval_mi35x.py (test_glm51_accuracy) in job nightly-8-gpu-mi35x-glm51 on nightly-test-amd.yml (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
May 01	25225678502	nightly-8-gpu-mi35x-glm51	❌ Failed	test_glm51_accuracy	Memory capacity is unbalanced
Apr 30	25180906638	nightly-8-gpu-mi35x-glm51	✅ Passed	—	—
Apr 29	25125309566	nightly-8-gpu-mi35x-glm51	✅ Passed	—	—
Apr 28	25069425979	nightly-8-gpu-mi35x-glm51	✅ Passed	—	—
Apr 27	25010934840	nightly-8-gpu-mi35x-glm51	✅ Passed	—	—

First observed failure: 2026-05-01 18:57 UTC. Last known passing: 2026-04-30 (run 25180906638).

Failure Cluster

Pre-model-load TP memory imbalance check (GPUs already occupied at server startup)

Facts (verified observations)

• Verbatim error at log line 2604 (verified): RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes. pre_model_load_memory=6.978515625, local_gpu_memory=8.681640625, local_gpu_memory * 0.9=7.8134765625.
• Stack: scheduler.py:3918 → scheduler.py:432 → scheduler.py:693 → scheduler.py:648 → tp_worker.py:260 → tp_worker.py:345 → model_runner.py:481 → model_runner.py:1231 (raised inside init_torch_distributed).
• The check is gated by envs.SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK.get() (model_runner.py:1230); when disabled this becomes a warning, not a fatal error.
• AITER unchanged across passing & failing runs: dev image v0.1.12.post2.dev150+ga6bb49937 preserved (Dockerfile expects a6bb499).
• Local registry pull failed once at job start (line 805–806: https://10.245.143.50:5000/v2/: context deadline exceeded), fell back to public rocm/sgl-dev:v0.5.10.post1-rocm700-mi35x-20260501 — image acquired successfully, not the cause of the GPU-memory error.
• Server command: --tp 8 --device cuda --nsa-prefill-backend tilelang --nsa-decode-backend tilelang --mem-fraction-static 0.80 --chunked-prefill-size 131072 (verified at log line 2604+1 trailer).
• Multiple sibling MI35x 8-GPU jobs in the same run also failed (e.g. nightly-8-gpu-mi35x-glm5-mxfp4 73968531052, nightly-8-gpu-mi35x-grok2 73968531061, nightly-8-gpu-mi35x-kimi-k25 73968531130, nightly-perf-8-gpu-mi35x-deepseek-v32-basic 73968531126, several others). All MI35x 8-GPU jobs ran on the linux-mi35x-gpu-8-4wrmw-runner-* pool — same GPU pool prefix.
• The 8.68 GB free figure is ~5% of MI355x's ~192 GB HBM3e — consistent with another process holding ~180 GB at server startup (NOT a normal "fresh container" state).
• HEAD commit ece8a1a modifies model_runner.py to add device_timer wrappers around forward_decode/extend/idle and adds self.device_timer = None + self.model_is_mrope attributes; it does NOT modify init_torch_distributed, the memory check, or any pre-init GPU allocation path (verified via git show ece8a1a -- python/sglang/srt/model_executor/model_runner.py).

Hypothesised Causes

• [MEDIUM] Stale GPU state on the shared linux-mi35x-gpu-8-4wrmw-runner-* pool — orphan processes from a prior job on the same physical host left ~180 GB of HBM allocated, and the per-rank symmetry guard fires before model load.

  • Supporting: local_gpu_memory=8.68 GB on an MI355x (~190 GB total) is impossible with a clean container; multiple sibling MI35x 8-GPU jobs in the same run also failed on the same pool prefix; HEAD sglang commit doesn't touch the failing code path; this same job passed on the same pool in the previous 4 nightly runs.
  • Disconfirming: cannot directly observe rocm-smi/amd-smi output in the log to confirm which process holds the memory; the docker registry fallback at job start could (very weakly) suggest container restart was unusual, though images downloaded successfully.
  • How to verify: rerun the failed job on a freshly-cycled MI35x 8-GPU runner (or after docker system prune + rocm-smi --gpureset) — if it passes, infra-side cleanup gap is confirmed; alternatively add rocm-smi dump immediately before init_torch_distributed in CI to capture the offending PID next time.

• [LOW] Sglang-side regression in import-path / pre-init GPU allocation in commit window 7bb7f604..ece8a1a.

  • Supporting: temporal correlation — last passing nightly was 7bb7f604, today's HEAD is ece8a1a; 38 commits in between including d48095b ("Bypass torch.cuda.use_mem_pool generator-CM in SymmetricMemoryContext") which touches CUDA mem pool semantics.
  • Disconfirming: an in-process pre-init allocation would show all TP ranks consuming ~equal memory, NOT 6.98 GB used + 8.68 GB free totalling only ~16 GB on a 190 GB device — the math implies an external process holds the rest. Multiple unrelated mi35x jobs failing on the same pool argues against a code regression (a code regression would also have hit the prior NVIDIA / mi325 jobs in the same run, which mostly passed).
  • How to verify: bisect 7bb7f604..ece8a1a against test_glm51_eval_mi35x.py on a clean runner — but only worth doing after the infra hypothesis is ruled out by a clean-runner rerun.

No hypothesis above MEDIUM confidence; primary candidate is infra contamination, requires a clean-runner rerun to confirm before any code investigation.

In-flight Fix Check

Searched open PRs for memory capacity unbalanced, SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK, GLM-5.1 nsa, mi35x cleanup, orphan gpu process.

• In-flight fix: ❌ none found in open PRs — nothing matches the symptom (TP memory imbalance check on MI35x). Closest PR is #18037 ("Add warning for SGLANG_DISABLE_TP_MEMORY_INBALANCE_CHECK") which adjusts wording but does not address the underlying GPU contamination.

Suggested Triage Steps

• Re-run job 73968531090 on the same linux-mi35x-gpu-8-* pool. If it passes on retry → confirms transient infra/contamination, file an infra issue against runner cleanup hooks for the MI35x 8-GPU pool.
• If it fails again on retry, capture rocm-smi / amd-smi --process output before init_torch_distributed (e.g. by adding a temporary debug step) to identify the offending PID/container.
• Cross-check the other failed sibling MI35x 8-GPU jobs in run 25225678502 (nightly-8-gpu-mi35x-glm5-mxfp4, nightly-8-gpu-mi35x-grok2, nightly-perf-8-gpu-mi35x-deepseek-v32-basic, nightly-accuracy-8-gpu-mi35x, nightly-8-gpu-mi35x-kimi-k25) — if they share the same pre_model_load_memory/local_gpu_memory symptom, that strengthens the infra hypothesis and would justify a single bulk infra fix rather than per-job investigation.
• Only if clean-runner reruns still reproduce the failure: bisect commits 7bb7f604..ece8a1a — start with d48095b (mem-pool context-manager change) and 108bfd8 (Aiter MoE runner), since these touch GPU memory paths.

Status

• Persistence: Failing for first time (1 run); previous 4 consecutive nightlies (Apr 27–30) on the same job all passed.
• Scope: Job nightly-8-gpu-mi35x-glm51 only at the test-file level; however ≥7 sibling MI35x 8-GPU jobs in the same nightly run 25225678502 (all on linux-mi35x-gpu-8-4wrmw-runner-* pool) failed alongside it — symptom may be shared at the runner-pool level, separate per-test-file analyses required to confirm.
• Blocked work: GLM-5.1-FP8 + tilelang NSA accuracy signal on MI35x lost for tonight's nightly.
• In-flight fix: ❌ none found in open PRs — needs clean-runner re-run first to triage infra vs. sglang regression.

nightly-perf-8-gpu-mi35x-deepseek-v32-basic — failed step(s): Performance Test MI35x (8-GPU DeepSeek-V3.2 Basic)

Commit Info

• sglang: ece8a1a7
• aiter: a6bb4993 (source: Dockerfile default — bumped today via 7e17a7a from v0.1.12.post1)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/perf/mi35x/test_deepseek_v32_basic_perf_mi35x.py	test_bench_one_batch (TestNightlyDeepseekV32BasicPerformance)	Memory access fault by GPU node-{2..9} during CUDA graph capture, then Scheduler watchdog timeout on all 8 TP workers, runner killed at TIMEOUT after 5400 seconds	Log line 19342 (fault) · Log line 27487 (timeout)

Failure Summary

The DeepSeek-V3.2 perf test loaded weights successfully (TP0 took ~28 min, all 8 ranks loaded), began CUDA graph capture, and at bs=136 (capture batch 31/52) all 8 GPUs simultaneously raised Memory access fault by GPU node-N ... Reason: Unknown. Schedulers then hung in c10::hip::device_synchronize (still inside cuda_graph_runner.py:1163 capture_one_batch_size); the soft watchdog (300s) printed py-spy dumps for all 8 TP workers; the runner finally killed the test at the 5400s hard timeout.

Regression Status

New regression (different symptom from yesterday's infra-like failure).

Recent history of test/registered/amd/perf/mi35x/test_deepseek_v32_basic_perf_mi35x.py (test_bench_one_batch) in job nightly-perf-8-gpu-mi35x-deepseek-v32-basic (branch=main, event=schedule, completed runs only):

Date	Run	Test File Status	Failed Function	Error
May 01	25225678502	❌ Failed	test_bench_one_batch	Memory access fault during CUDA graph capture @ bs=136 → 5400s timeout
Apr 30	25180906638	❌ Failed	test_bench_one_batch	Different: RuntimeError: The memory capacity is unbalanced in init_torch_distributed → server exit -9 at 40s (looks like dirty-GPU / leftover-process)
Apr 29	25125309566	✅ Passed	—	—
Apr 28	25069425979	✅ Passed	—	—
Apr 27	25010934840	✅ Passed	—	—

First observed failure of THIS symptom (memory access fault during graph capture): 2026-05-01. Last clean pass for this test file: 2026-04-29 (530b497a).

Failure Cluster

GPU memory access fault during CUDA graph capture (DeepSeek-V3.2 NSA + FP8 on gfx950/MI35x)

Facts (verified observations, no interpretation)

• Verbatim fault (8 ranks, ~same instant; log line 19342): Memory access fault by GPU node-5 (Agent handle: 0x592a29d66530) on address 0x7ca35c600000. Reason: Unknown. (and node-{2,3,4,6,7,8,9} with similar messages within 30 ms).
• Last successful capture iteration: Capturing batches (bs=144 avail_mem=36.80 GB): 60%|█████▉ | 31/52 [01:55<00:10, 1.98it/s] at 2026-05-01T20:50:33.9365930Z. Fault begins at 2026-05-01T20:50:33.9617100Z, immediately after starting bs=136 (capture batch 32/52).
• All 8 TP scheduler py-spy dumps show MainThread blocked in libhsa-runtime64.so.1.18.70000 → c10::hip::device_synchronize → THCPModule_cudaSynchronize → torch.cuda.synchronize → capture_one_batch_size (cuda_graph_runner.py:1163) → _capture_one_stream (cuda_graph_runner.py:884) → capture (cuda_graph_runner.py:896) → CudaGraphRunner.__init__ (cuda_graph_runner.py:714) → init_device_graphs (model_runner.py:2792).
• Aiter version markers in failing run: [CI-AITER-CHECK] Dockerfile expects AITER_COMMIT=a6bb499375849eec45d68c5ccaebc8865fd422c0 / AITER version inside CI image: v0.1.12.post2.dev150+ga6bb49937 (Dev/patched). The yesterday-failing run (Apr 30, 7bb7f604) ran with the OLD default v0.1.12.post1; the aiter bump landed today via 7e17a7a ("Upd: AITER->(#2879)a6bb499", merged 2026-05-01 01:26 PT, file docker/rocm.Dockerfile).
• Server config (verified from log): tp=8, model=deepseek-ai/DeepSeek-V3.2, attention_backend=nsa, nsa_prefill_backend=tilelang, nsa_decode_backend=tilelang, quant=fp8 (e4m3), cuda_graph_max_bs=512, page_size=1, mem_fraction_static=0.85. avail_mem=36.80 GB at the fault — not OOM-adjacent.
• Many [aiter] shape is M:..., N:64, K:7168 ... not found tuned config in /tmp/aiter_configs/bf16_tuned_gemm.csv, will use default config! using torch solution:0 lines preceded the fault (this is the aiter bf16 GEMM falling back to torch — i.e. the failing call sequence touches aiter call sites). This is log-proximity, not causation.
• sglang commit window since last green run: 77 commits across 530b497a..ece8a1a7. Subset for May 1 alone (since yesterday's failed sha 7bb7f604): 39 commits — see Hypothesised Causes.
• aiter delta v0.1.12.post1 (7b570738)..a6bb4993: 150 commits, 300 files changed.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Aiter version bump in 7e17a7a (v0.1.12.post1 → a6bb4993) — 150-commit aiter delta, several entries match the failing kernel surface:

  • Supporting: HEAD aiter commit a6bb4993 "Support preshuffled layout in indexer_k_quant_and_cache / cp_gather_indexer_k_quant_cache" — DeepSeek-V3.2 uses NSA (Native Sparse Attention) with an indexer path; this exact call site is touched by the bump. Aiter da67a099 "Re-enable native qh128 fp8 kernel mla_a8w8_qh128_m32x4_n16x2_msk0_ps on gfx950" lands in the same window — gfx950 is MI35x, the failing GPU. Aiter c71075ce "Fix 2-stage fused_allreduce_rmsnorm memory ordering" and 4331ad7a "fused AR + RMSNorm + per-group FP8 quant" are also relevant to the AR-heavy path being exercised during graph capture. The bump is the single largest delta in the 24h regression window.
  • Disconfirming: this is a perf-nightly job (no Scout sister-baseline run exists, so no clean A/B). The fault is "Reason: Unknown" with no kernel symbol; we cannot directly attribute it to the indexer or mla_a8w8_qh128 paths from the log alone. The "using torch solution:0" aiter log lines immediately preceding the fault are the bf16 GEMM (M:144, N:64, K:7168), not fp8/MLA — log proximity is not causation (per CLAUDE.md two-variable trap). Aiter delta touches 300 files; HIGH confidence is unwarranted without a revert experiment.
  • How to verify: re-run the perf job with AITER_COMMIT_OVERRIDE=v0.1.12.post1 (the previous default) on the same sglang sha ece8a1a7. If the fault disappears, the aiter bump is confirmed and the maintainer can bisect aiter 7b570738..a6bb4993 (150 commits) starting with the indexer/fp8/AR commits cited above.

• [LOW] sglang 108bfd8 "[MoE] Add Aiter MoE runner backend and purge aiter.fused_moe from quant methods (#23597)" — large refactor that swaps how aiter MoE kernels are invoked.

  • Supporting: lands in the regression window; touches the MoE forward path which DeepSeek-V3.2 exercises during graph capture; could interact badly with the aiter bump above (both change aiter call surfaces in the same nightly).
  • Disconfirming: no direct evidence from the stack trace that the fault is in MoE code; the captured frame is cuda_graph_runner.py:1163 → torch.cuda.synchronize (downstream from any MoE kernel). The Apr 30 sglang sha already contained much of the recent MoE work and didn't reach this code path (it died earlier in init_torch_distributed).
  • How to verify: only meaningful as a secondary bisect step after the aiter override A/B above.

• [LOW] sglang 583929c "fix the compatibility between --moe-dense-tp-size 1 and piecewise cuda graph (#23972)"

  • Supporting: directly touches piecewise cuda graph paths.
  • Disconfirming: server args show disable_piecewise_cuda_graph=True, so the piecewise-CG codepath is not engaged in this test (only the regular full cuda graph capture). Likely irrelevant.
  • How to verify: not actionable here.

• [LOW] sglang d48095b "Bypass torch.cuda.use_mem_pool generator-CM in SymmetricMemoryContext (#24190)"

  • Supporting: touches memory-pool wrapping during graph capture; could plausibly affect cuda graph stream/memory allocator behaviour.
  • Disconfirming: server has enable_symm_mem=False → the SymmetricMemoryContext path is not entered. Likely irrelevant.
  • How to verify: not actionable here.

• [LOW] Apr 30 failure was a different cause — 7bb7f604 showed RuntimeError: The memory capacity is unbalanced. ... pre_model_load_memory=0.00390625, local_gpu_memory=0.65234375 (a leftover process / dirty-GPU runner state) → server exit -9 in 40s. This is not the same failure mode as today and may indicate the runner image has had latent state issues for >24h. Mention only as a confounder for "Test File Status" — does not affect the May-1 hypothesis ranking.

No hypothesis above MEDIUM confidence; the aiter bump is the strongest candidate but needs an A/B verify to promote.

In-flight Fix Check

Searched: aiter+memory+access+fault, aiter+revert, deepseek+v32+capture against open PRs in sgl-project/sglang.

• Closest matches found, but none target this regression: #23581 "[AMD] Default SGLANG_USE_AITER_AR to false to avoid HIP graph capture invalidation" (related symptom area — HIP graph capture + aiter AR — but does not revert today's aiter bump and predates #24218); #11154 "[Bugfix][AMD] Fix memory access fault" (very stale, unrelated context).
• In-flight fix: ❌ none found that targets the aiter bump in [#24218](https://github.com/sgl-project/sglang/pull/24218) — needs new investigation.

Suggested Triage Steps

• Highest ROI A/B: re-trigger nightly-test-amd.yml on sglang ece8a1a7 with AITER_COMMIT_OVERRIDE=v0.1.12.post1 (or use amd-aiter-scout.yml which exposes that knob) to A/B today's aiter bump. Run only the nightly-perf-8-gpu-mi35x-deepseek-v32-basic matrix shard.
• If the override A/B passes → bisect aiter 7b570738..a6bb4993 starting with the NSA/fp8/AR-touching commits (a6bb4993, da67a099, c71075ce, 4331ad7a). If a single aiter commit is identified, file an issue against ROCm/aiter and consider reverting #24218 on main until aiter ships a fix.
• If the override A/B still fails → bisect sglang 530b497a..ece8a1a7 (77 commits) with focus on 108bfd8 (MoE Aiter runner refactor) and the other commits in the May-1 39-commit subset.
• Independent: the Apr 30 "memory capacity unbalanced" symptom suggests a leftover-process / dirty-GPU runner — ask infra to verify the MI35x runner image cleans GPU memory between jobs (orthogonal to today's regression but blocks this same test file).
• Local repro command: python test/registered/amd/perf/mi35x/test_deepseek_v32_basic_perf_mi35x.py inside container rocm/sgl-dev:v0.5.10.post1-rocm700-mi35x-20260501 on an 8×MI35x host; expect fault around capture iteration 32/52 (bs=136).

Status

• Persistence: This symptom (memory access fault during graph capture) first observed today (2026-05-01 20:50 UTC), 1 completed run with this exact failure mode. The same test file also failed yesterday but with a different (infra-like) cause, so the test file has been red for 2 consecutive nightly runs.
• Scope: 1 job (nightly-perf-8-gpu-mi35x-deepseek-v32-basic) in nightly-test-amd.yml so far. Other DeepSeek-V3.2 / NSA / MI35x jobs in tonight's nightly should be checked for the same symptom (cluster could expand).
• Blocked work: Blocks DeepSeek-V3.2 basic perf signal on MI35x. No accuracy / functional jobs known to share the symptom yet.
• In-flight fix: ❌ none found that targets aiter bump #24218 — needs new investigation.

nightly-8-gpu-mi35x-qwen3-235b-mxfp4 — failed step(s): Accuracy Test + Performance Test MI35x (8-GPU Qwen3-235B-MXFP4)

Commit Info

• sglang: ece8a1a78813f6f0bd8394b03589d144125d08fb
• aiter: a6bb499375849eec45d68c5ccaebc8865fd422c0 (source: Dockerfile default; dev image v0.1.12.post2.dev150+ga6bb49937 preserved, no override)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_qwen3_instruct_mxfp4.py	TestQwen3Instruct2507MXFP4.setUpClass	Server process exited with code 1 (scheduler died, exit -6) — root error: HIP error: operation not permitted on an event last recorded in a capturing stream (hipErrorCapturedEvent) inside aiter custom_all_reduce.capture() during sglang's CUDA-graph capture	Log

Failure Summary

TestQwen3Instruct2507MXFP4.setUpClass could not launch the server (amd/Qwen3-235B-A22B-Instruct-2507-mxfp4, --tp 4 --ep 2 --attention-backend aiter). All ranks crashed with HIP hipErrorCapturedEvent / hipErrorStreamCaptureUnsupported reported from the NCCL/RCCL ProcessGroup watchdog while inside aiter's custom_all_reduce.capture() context manager, leading to terminate called after throwing an instance of 'c10::DistBackendError' and Rank 0 scheduler died during initialization (exit code: -6).

Regression Status

Never-passed (in queryable window) — persistent failure for ≥17 days.

Recent history of test/registered/amd/test_qwen3_instruct_mxfp4.py (TestQwen3Instruct2507MXFP4.setUpClass) in job nightly-8-gpu-mi35x-qwen3-235b-mxfp4 (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
2026-05-01	25225678502	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	❌ Failed	setUpClass	HIP hipErrorCapturedEvent in custom_all_reduce.capture
2026-04-30	25180906638	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	❌ Failed	setUpClass	same
2026-04-29	25125309566	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	❌ Failed	setUpClass	same HIP hipErrorCapturedEvent in graph capture
2026-04-28	25069425979	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	❌ Failed	setUpClass	same
2026-04-27	25010934840	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	❌ Failed	setUpClass	same
2026-04-26	24963018193	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	❌ Failed	setUpClass	same HIP hipErrorCapturedEvent (verified at parallel_state.py:515 graph_capture)
2026-04-25	24936820562	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	❌ Failed	setUpClass	same
2026-04-24	24903695701	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	❌ Failed	setUpClass	same
2026-04-22	24793680817	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	❌ Failed	setUpClass	same
2026-04-15	24469668959	nightly-8-gpu-mi35x-qwen3-235b-mxfp4	❌ Failed	setUpClass	same Rank 1 scheduler died during initialization (exit code: -6)

Lookback window: every completed event=schedule run on main for nightly-test-amd.yml from 2026-04-15 through 2026-05-01 in which the job ran shows ❌ Failed for this test file. No green run found for test_qwen3_instruct_mxfp4.py in the queryable window — earliest sampled failure with the identical HIP capture symptom is 2026-04-15. Job 2026-04-23 was cancelled, not a real signal. Note: the file at test/registered/amd/test_qwen3_instruct_mxfp4.py was likely renamed/moved into its current path (current git log only shows commits touching it from 2026-04-27 onward), so a true "first-appearance / first-fail" date earlier than 2026-04-15 cannot be excluded from this snapshot.

Failure Cluster

HIP error during CUDA-graph capture inside aiter custom_all_reduce.capture() (RCCL watchdog reports hipErrorCapturedEvent / hipErrorStreamCaptureUnsupported → terminate → scheduler exit -6).

Facts (verified observations, no interpretation)

• [CI-AITER-CHECK] shows Dockerfile expects AITER_COMMIT=a6bb499375849eec45d68c5ccaebc8865fd422c0, image has v0.1.12.post2.dev150+ga6bb49937, AITER_COMMIT_OVERRIDE= empty (verbatim from job log lines 2271-2289).
• Workload: sglang serve --model-path amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 --tp 4 --ep 2 --trust-remote-code --attention-backend aiter --device cuda (line 3742 of log).
• All 4 ranks emit [E501 ...] ProcessGroupNCCL.cpp:2055 ... Process group watchdog thread terminated with exception: HIP error: operation not permitted on an event last recorded in a capturing stream followed by terminate called recursively and Fatal Python error: Aborted (lines 3282-3658).
• Faulting Python frames (line 3344-3350):

  File "/sgl-workspace/aiter/aiter/dist/device_communicators/custom_all_reduce.py", line 303 in _gather_ipc_meta
  File "/sgl-workspace/aiter/aiter/dist/device_communicators/custom_all_reduce.py", line 270 in flush_graph_buffers
  File "/sgl-workspace/aiter/aiter/dist/device_communicators/custom_all_reduce.py", line 470 in capture
  File "/usr/lib/python3.10/contextlib.py", line 153 in __exit__
  File "/sglang-checkout/python/sglang/srt/distributed/parallel_state.py", line 517 in graph_capture
  File "/sglang-checkout/python/sglang/srt/distributed/parallel_state.py", line 1548 in graph_capture
  

  i.e. the throw happens on __exit__ of ca_comm.capture() (sglang parallel_state.py:517 enters maybe_ca_context = ca_comm.capture()), where aiter's flush_graph_buffers calls _gather_ipc_meta which performs a dist.all_gather_object collective — a non-stream-captureable operation.
• Engine raises RuntimeError: Rank 0 scheduler died during initialization (exit code: -6) (line 3763), popen_launch_server then raises Exception: Server process exited with code 1 at test_qwen3_instruct_mxfp4.py:44 setUpClass (lines 3768-3775); FAILED (errors=1) (line 3780); job exits 255.
• Scope: same job has shown the same HIP graph-capture crash on every completed run from 2026-04-15 → 2026-05-01 sampled (8/8 sampled jobs verified by log inspection); aiter SHA was the same Dockerfile default in this and the 2026-04-30/2026-04-29/2026-04-26 runs sampled, so neither the sglang HEAD nor the aiter SHA changed within the last few days while the failure persisted.
• Sister 1-GPU MXFP4 jobs in the same workflow (nightly-8-gpu-mi35x-deepseek-r1-mxfp4, glm5-mxfp4, kv-fp8, ar-fusion) passed in run 25125309566 (2026-04-29) — so the failure is specific to the Qwen3-235B-A22B-Instruct-2507-mxfp4 configuration (TP=4 EP=2 + --attention-backend aiter), not a generic AMD/aiter graph-capture regression.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] aiter custom_all_reduce.capture() context manager calls flush_graph_buffers → _gather_ipc_meta on __exit__, which executes a dist.all_gather_object collective while the HIP stream-capture context is still active. RCCL is reporting hipErrorCapturedEvent / hipErrorStreamCaptureUnsupported because event waits / collectives outside the captured stream are not allowed during HIP graph capture.

  • Supporting: Stack trace on every rank pinpoints custom_all_reduce.py:303 _gather_ipc_meta → :270 flush_graph_buffers → :470 capture → parallel_state.py:517 graph_capture. The HIP error string hipErrorCapturedEvent directly names the stream-capture violation. Sister 1-GPU MXFP4 jobs that don't exercise the TP=4 multi-GPU all-reduce capture path pass cleanly.
  • Disconfirming: Other 8-GPU MXFP4 nightly jobs (deepseek-r1-mxfp4, glm5-mxfp4) presumably also enter this same ca_comm.capture() path during their TP capture but do not crash here — so either Qwen3-235B's specific graph (size, expert-parallel sharding 4×2, 256-expert MoE) triggers a different ipc-meta exchange, or those sister jobs run a different TP/EP layout that bypasses the path. (Need to inspect their server-launch flags to confirm.)
  • How to verify: Run the same model with SGLANG_DISABLE_CUSTOM_ALL_REDUCE=1 (or whatever flag bypasses ca_comm) and confirm whether the HIP capture error disappears. Alternatively, run with --cuda-graph-max-bs 0 / --disable-cuda-graph to bypass graph capture entirely; if that succeeds, capture path is implicated.

• [LOW] aiter SHA a6bb499375849eec45d68c5ccaebc8865fd422c0 (or its predecessor that landed before 2026-04-15) introduced the flush_graph_buffers collective inside capture. Persistent failure across SHAs of both repos suggests the bug is in the aiter capture protocol, not a sglang regression.

  • Supporting: sglang HEAD SHA changes daily but the failure pattern is identical from 2026-04-15 onward. The faulting frames are entirely inside aiter, not inside sglang.
  • Disconfirming: The Dockerfile-default aiter has been unchanged across multiple sampled runs (same a6bb499 head); we don't have evidence that an earlier aiter SHA passed this exact test. Could be a long-standing bug rather than a recent regression. Without a known-passing aiter SHA for this job, this hypothesis is just "aiter is in the stack" — [LOW].
  • How to verify: Try a previous aiter commit via AITER_COMMIT_OVERRIDE=<older> and rerun nightly-8-gpu-mi35x-qwen3-235b-mxfp4.

• [LOW] Test was added/registered into nightly without a previously-green baseline — i.e. the model+config never worked on MI35x in this CI context (Never-passed AMD port).

  • Supporting: No green run found for this test file in the entire queryable window (≥17 days). Job entry exists in workflow run lists from 2026-04-15 onward only.
  • Disconfirming: The model is published under amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 — if AMD ships it, presumably someone exercised it locally; the bug may be CI-environment-specific (RCCL version, ROCm version, HSA flags) rather than fundamentally broken.
  • How to verify: Ask the AMD owner of the amd/Qwen3-235B-A22B-Instruct-2507-mxfp4 test for any local pass log; if none, treat this as Never-passed and disable nightly registration until aiter capture path is fixed.

No hypothesis above MEDIUM confidence; needs A/B against an older aiter SHA and/or a --disable-cuda-graph rerun to reach HIGH.

In-flight Fix Check

Searched open sglang PRs:

• is:pr is:open qwen3 mxfp4 → 12 hits, none target this MI35x graph-capture failure (top: #23146 "[AMD] Enable EAGLE speculative decoding for Qwen3.5 FP8 and MXFP4 models with aiter's unified attention" — different model line).
• is:pr is:open custom_all_reduce graph capture → 2 hits, neither addresses the aiter capture protocol.
• is:pr is:open hipErrorCapturedEvent → 0 hits.
• is:pr is:open test_qwen3_instruct_mxfp4 → 0 hits.

In-flight fix: ❌ none found in open PRs — needs new investigation (likely a fix in aiter custom_all_reduce.flush_graph_buffersto avoidall_gather_object during stream capture).

Suggested Triage Steps

• Re-launch the failing config locally with --disable-cuda-graph (or set the sglang env that disables custom all-reduce) and confirm whether setUpClass succeeds — narrows the bug to the graph-capture path vs. a model-loading issue.
• Bisect aiter: run with AITER_COMMIT_OVERRIDE=<sha> for a few aiter commits prior to a6bb499 until the test passes, then read the diff to identify the offending change to custom_all_reduce.flush_graph_buffers / _gather_ipc_meta.
• File an aiter issue/PR to defer the IPC-meta all_gather_object to after graph capture exits, or move it into the warm-up phase before torch.cuda.graph is entered.
• If aiter fix is non-trivial, gate the Qwen3-235B MXFP4 nightly job (or skip the test on AMD) with a clear # TODO: re-enable after aiter custom_all_reduce capture fix until the upstream fix lands, to recover the green signal for the rest of the workflow.
• Confirm the symptom is config-specific by diffing this job's launch command against nightly-8-gpu-mi35x-deepseek-r1-mxfp4 (which passes); EP=2 / 256-expert MoE is a notable difference and may be what triggers the IPC-meta collective.

Status

• Persistence: Failing for ≥17 days — first sampled failure 2026-04-15 job 71506238357, every completed run since shows ❌ same HIP capture error. No green run for this test file in the queryable window.
• Scope: 1 job (nightly-8-gpu-mi35x-qwen3-235b-mxfp4) in 1 workflow (nightly-test-amd.yml). Sister MXFP4 nightly jobs (deepseek-r1-mxfp4, glm5-mxfp4, kv-fp8, ar-fusion) currently pass.
• Blocked work: Blocks AMD MI35x nightly signal for the Qwen3-235B-A22B-Instruct-2507 MXFP4 model; blocks accuracy/perf regression detection on this configuration.
• In-flight fix: ❌ none found in open PRs — needs new investigation (likely an aiter custom_all_reduce.flush_graph_buffers fix).

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-05-02 22:27 UTC)

[amd-bot]

nightly-test-amd.yml — Per-Job Analysis (continued 4/6)

nightly-8-gpu-mi35x-kimi-k25 — failed step(s): Accuracy Test MI35x (8-GPU Kimi-K2.5)

Commit Info

• sglang: ece8a1a7
• aiter: a6bb4993 (v0.1.12.post2.dev150+ga6bb49937, source: dev image preserved per [CI-AITER-CHECK] Dev/patched version detected; bumped from v0.1.12.post1 by sglang #24218 / 7e17a7a1)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_kimi_k25_eval_mi35x.py	test_kimi_k25_gsm8k_accuracy	RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes. pre_model_load_memory=6.978515625, local_gpu_memory=8.681640625, local_gpu_memory * 0.9=7.8134765625 raised on TP1 → child sigquit → Server process exited with code -9 (both retries failed)	Log

Failure Summary

The Kimi-K2.5 GSM8K accuracy test on MI35x failed because the SGLang server never came up: TP rank 1 raised RuntimeError: The memory capacity is unbalanced inside init_torch_distributed (memory imbalance check at model_runner.py:1231). Both popen_launch_server retries failed identically, with the imbalance check hitting on TP1 each time.

Regression Status

New regression — first failure today.

Recent history of test/registered/amd/accuracy/mi35x/test_kimi_k25_eval_mi35x.py (test_kimi_k25_gsm8k_accuracy) in job nightly-8-gpu-mi35x-kimi-k25 (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
May 01	25225678502	nightly-8-gpu-mi35x-kimi-k25	❌ Failed	test_kimi_k25_gsm8k_accuracy	memory imbalance, exit -9
Apr 30	25180906638	nightly-8-gpu-mi35x-kimi-k25	✅ Passed	—	—
Apr 29	25125309566	nightly-8-gpu-mi35x-kimi-k25	✅ Passed	—	—
Apr 28	25069425979	nightly-8-gpu-mi35x-kimi-k25	✅ Passed	—	—
Apr 27	25010934840	nightly-8-gpu-mi35x-kimi-k25	✅ Passed	—	—
Apr 26	24963018193	nightly-8-gpu-mi35x-kimi-k25	✅ Passed	—	—

Last passing SHA: 7bb7f604 · First failing SHA: ece8a1a7 (39 sglang commits in window).

Failure Cluster

GPU memory imbalance check failure during init_torch_distributed (TP1 only) on 8-GPU MI35x

Facts (verified observations, no interpretation)

• Verbatim error (both retries, same TP rank): [2026-05-01 20:22:27 TP1] Scheduler hit an exception ... RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes. pre_model_load_memory=6.978515625, local_gpu_memory=8.681640625, local_gpu_memory * 0.9=7.8134765625 (raised at python/sglang/srt/model_executor/model_runner.py:1231 from init_torch_distributed).
• Other ranks (TP0, TP2-TP7) reach Init torch distributed ends with mem usage=10.07–10.22 GB; TP1 raises the imbalance check before reaching that line.
• Test failure path (top of stack): test_kimi_k25_eval_mi35x.py:62 → popen_launch_server (test_utils.py:979) → Exception: Server process exited with code -9. After 2 retries: retry() exceed maximum number of retries.
• Aiter version diff vs last passing run (verified from [CI-AITER-CHECK] markers in both logs): passing run 73825579535 used aiter v0.1.12.post1; failing run uses aiter v0.1.12.post2.dev150+ga6bb49937 (a6bb4993). Bumped only by 7e17a7a1 (#24218) — docker/rocm.Dockerfile only.
• In passing run, TP1 reported mem usage=9.80 GB — already noticeably lower than the other 7 ranks' ~10.2 GB, suggesting GPU 1 on this MI35x runner pool tends to start with less free memory than its siblings.
• Local docker registry pull failed; falling back to public registry: rocm/sgl-dev:v0.5.10.post1-rocm700-mi35x-20260501 (CI image successfully pulled from public registry).
• Reproducibility within this run: 2/2 retries failed identically on TP1.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] Stale process / memory occupancy on physical GPU index 1 of the MI35x runner. The check fires when local_gpu_memory * 0.9 > pre_model_load_memory (cross-rank min); only TP1 reports the imbalance, and TP1 already had the lowest free memory in the previous (passing) run.
  • Supporting: Identical failure on both retries on the same TP rank; passing run also showed TP1 low (9.80 GB vs ~10.2 GB elsewhere); error message itself ("Some GPUs may be occupied by other processes") is the runtime's own infra-leak hypothesis.
  • Disconfirming: TP1 was below threshold in passing runs too; if it were purely a leftover process it should have been intermittent prior to today, not Never-failed-then-suddenly-failed.
  • How to verify: rerun the workflow on the same runner; if it passes, this is a flaky environment leak. If it fails again, suspect (a) below.
• [LOW] Aiter bump in #24218 (7e17a7a1) increased early-init memory footprint (e.g., changed JIT module sizes for module_aiter_core/module_custom_all_reduce or new tuned-config preloads) just enough to push TP1 — historically the slimmest rank — past the imbalance check.
  • Supporting: Strong temporal correlation (1st failure on the very first nightly run after the bump); the diff that landed in this window is purely a Dockerfile aiter version bump (no sglang source change touches init_torch_distributed); aiter delta v0.1.12.post1..a6bb4993 is ~150 commits including new tuned configs and JIT modules visible in the log (e.g., 11×[aiter] import [module_custom_all_reduce]).
  • Disconfirming: Other 7 ranks all report mem usage=10.07–10.22 GB (essentially identical to passing run), so the absolute footprint did not visibly grow; TP1's free memory was already the outlier before the bump; symptom is "imbalance", not "OOM", which is more consistent with environmental skew than with a uniform footprint increase.
  • How to verify: re-run with AITER_COMMIT_OVERRIDE=v0.1.12.post1 (revert aiter to last-passing version) on the same runner and observe whether TP1 still trips the imbalance check.
• [LOW] Unbalanced HIP/Triton JIT cache build during the aiter] import flurry on TP1 leaving a transient memory spike on GPU 1.
  • Supporting: The 8 ranks each emit their own [aiter] import [module_aiter_core] and re-merge tuned configs at staggered times, with TP1 being the last to reach Init torch distributed begin (20:22:59.4) in retry 2.
  • Disconfirming: Timing skew alone does not explain a ~3 GB free-memory delta; this is speculation.
  • How to verify: enable per-rank rocm-smi snapshot at the moment init_torch_distributed runs (or run with --log-level debug).

No hypothesis above LOW confidence; needs rerun + AITER_COMMIT_OVERRIDE=v0.1.12.post1 A/B and/or rocm-smi snapshot on the runner. Disconfirming for all "code change" hypotheses: none of the 39 sglang commits in 7bb7f604..ece8a1a7 modify init_torch_distributed, the imbalance check, or device-init memory accounting (verified by scanning subjects; only #24218 touches docker/rocm.Dockerfile, which is the aiter-bump path).

In-flight Fix Check

Searched is:pr is:open repo:sgl-project/sglang for keywords memory capacity unbalanced (0 matches), kimi-k25 nightly (1 match: #23848 "Add Kimi-K2.6 in nightly tests for MI30x and MI35x" — unrelated to this failure), aiter a6bb499 (0 matches).

• In-flight fix: ❌ none found in open PRs — needs new PR or investigation.

Suggested Triage Steps

• Re-run the nightly-8-gpu-mi35x-kimi-k25 job on the same runner pool to disambiguate "flaky GPU 1 occupancy" (passes on retry) vs "deterministic regression" (fails again).
• If failure recurs, run a one-shot manual A/B with AITER_COMMIT_OVERRIDE=v0.1.12.post1 (the last-passing aiter) on the same MI35x runner; if it passes, the aiter bump in #24218 is implicated and warrants either a revert or an mem_fraction_static/pre_model_load_memory tolerance bump in model_runner.py:1231.
• Capture rocm-smi and lsof /dev/dri/renderD129 (or equivalent) at the start of the job to confirm whether GPU 1 has a leaked process from a previous run.
• If the imbalance check is too strict for this hardware (TP1 chronically ~0.4 GB lower than peers), consider relaxing the 0.9 threshold or excluding aiter-side preallocations from the comparison — but only after bisect confirms the trigger.

Status

• Persistence: 1 run / 1 day (first failure on 2026-05-01 20:22 UTC; preceded by 5 consecutive green nightly runs Apr 26–30).
• Scope: 1 job in 1 workflow (nightly-8-gpu-mi35x-kimi-k25 in nightly-test-amd.yml); the sister job nightly-8-gpu-kimi-k25 (MI30x) was success in the same run, so the failure is MI35x-specific.
• Blocked work: Kimi-K2.5 accuracy signal on MI35x is now red.
• In-flight fix: ❌ none found in open PRs — needs new PR or investigation.

nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp — failed step(s): Accuracy Test MI35x (8-GPU DeepSeek-V3.2 TP+MTP)

Commit Info

• sglang: ece8a1a78813f6f0bd8394b03589d144125d08fb
• aiter: a6bb499375849eec45d68c5ccaebc8865fd422c0 (source: dev image — [CI-AITER-CHECK] reports Dev/patched version detected: v0.1.12.post2.dev150+ga6bb49937 → skipping rebuild; this is the gfx950 Dockerfile default expanded to a dev wheel)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_deepseek_v32_mtp_eval_mi35x.py	test_a_gsm8k	AssertionError: 0.04 not greater than 0.94 (GSM8K accuracy 0.04, invalid 0.96)	Log

Failure Summary

The DeepSeek-V3.2 + EAGLE MTP (TP=8, NSA tilelang prefill+decode backend) GSM8K accuracy test ran to completion twice (initial + 1 retry) but produced accuracy=0.04 against a threshold of 0.94, with invalid=0.96 (96% of generations could not be parsed as a GSM8K answer). Spec accept length is healthy (avg_spec_accept_length=3.969), so the EAGLE draft path is producing tokens — the failure is in the content itself.

Regression Status

Recurring failure with shifting symptoms (test file has failed every completed nightly since Apr 29 with three different surface errors).

Recent history of test/registered/amd/accuracy/mi35x/test_deepseek_v32_mtp_eval_mi35x.py (test_a_gsm8k) in job nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp (workflow nightly-test-amd.yml, branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
May 1	25225678502	nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp	❌ Failed	test_a_gsm8k	AssertionError: 0.04 not greater than 0.94
Apr 30	25180906638	nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp	❌ Failed	test_a_gsm8k	TIMEOUT after 3600s (Scheduler watchdog timeout self.watchdog_timeout=300)
Apr 29	25125309566	nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp	❌ Failed	setUpClass (server launch)	Server process exited with code -9 after RuntimeError: The memory capacity is unbalanced. pre_model_load_memory=11.70, local_gpu_memory=13.41
Apr 28	25069425979	nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp	✅ Passed	—	—
Apr 27	25010934840	nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp	✅ Passed	—	—

First failing date for this test file: 2026-04-29. Last known passing date: 2026-04-28. Note: the three failure days have different surface symptoms — that complicates the regression hypothesis (it may be one root cause manifesting differently, three independent issues, or partial infra noise). Today is the first day the test reaches the assertion stage and yields a real accuracy=0.04 reading.

Failure Cluster

DeepSeek-V3.2 + MTP GSM8K invalid-output collapse on MI35x (model generates parseable tokens but 96% of completions fail GSM8K answer extraction)

Facts (verified observations, no interpretation)

• Verbatim assertion (May 1, line 23561): AssertionError: 0.04 not greater than 0.94
• Per-completion metrics (line 23961): Accuracy: 0.040 / Invalid: 0.960 / Latency: 41.462 s / Output throughput: 2389.275 token/s
• Spec-decoding telemetry on the failing run: avg_spec_accept_length=3.9690414764834294 (out of --speculative-num-draft-tokens 4) — EAGLE MTP draft is being accepted at a normal-to-high rate, so the model is not just spitting random tokens.
• Both retry attempts produced the same accuracy=0.04, invalid=0.96 — not a stochastic flake.
• Server command (line 23987 area): sglang serve --model-path deepseek-ai/DeepSeek-V3.2 --trust-remote-code --tp 8 --nsa-prefill-backend tilelang --nsa-decode-backend tilelang --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --mem-fraction-static 0.7 --watchdog-timeout 1200 --device cuda.
• AITER version is unchanged across the regression window — [CI-AITER-CHECK] shows the gfx950 Dockerfile commit is a6bb499375849eec45d68c5ccaebc8865fd422c0 and the dev image carries v0.1.12.post2.dev150+ga6bb49937 (same SHA). Neither April-28 (passing) nor May-1 (failing) used AITER_COMMIT_OVERRIDE.
• Three runs in a row earlier in the day (TP3 only) emit [2026-05-01 20:47:38 TP3] Pyspy failed (py-spy dump --native --pid 1475). Error: Failed to get stack traces (lines 3052/3601/4150) — these correspond to the model-loading watchdog probes and stop once load completes; not directly tied to the accuracy failure.
• Regression window is 16 sglang commits, ad785a22..530b497a. Of those 16, only one mentions spec at all: bd448e5 [Spec] Split accept_length into num_accepted_drafts and num_accepted_tokens (#23962) (#23962). None of the 16 touch DeepSeek-V3.2, NSA, or tilelang.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] Regression introduced somewhere in the ad785a22..530b497a window (16 commits)
  • Supporting: test passed Apr 28 head, failed every nightly since (3 consecutive failing runs). No commits to the deepseek_v32 model file itself, but bd448e5 (#23962) modified spec-decode bookkeeping; 73e93be ([1/4] NVFP4 KV cache: quantization strategy abstraction and kernel) changed KV-cache abstractions; either could plausibly perturb the EAGLE+NSA path on AMD.
  • Disconfirming: (a) the three failure days have different symptoms (server-launch -9, watchdog timeout, accuracy=0.04). A single regression usually produces one symptom, not three. The Apr 29 GPU memory imbalance and Apr 30 watchdog look more like infra/runner issues than a code regression. (b) None of the 16 commits modify python/sglang/srt/models/deepseek_v32*.py, python/sglang/srt/layers/attention/nsa*, or any tilelang/MTP-specific code path; the linkage is indirect.
  • How to verify: git bisect ad785a22..530b497a running only test_a_gsm8k from test_deepseek_v32_mtp_eval_mi35x.py on an MI35x runner with the same launch command.
• [LOW] Spec-decode change bd448e5 (#23962) interacts badly with EAGLE+NSA on DeepSeek-V3.2
  • Supporting: only spec-touching commit in the window; the failing config uses EAGLE with --speculative-num-steps 3 --speculative-num-draft-tokens 4 and the MTP eval is the only configuration showing this collapse (the sister nightly-perf-8-gpu-mi35x-deepseek-v32-mtp job passes today, but it doesn't run gsm8k accuracy).
  • Disconfirming: the commit description claims it splits a single statistic into two; not a behavioral change. avg_spec_accept_length=3.97/4 is healthy, suggesting spec acceptance logic still works; the failure is in the content, not in draft acceptance counts.
  • How to verify: revert bd448e5 on a branch and rerun the test; or rerun the test with --speculative-algorithm removed to isolate whether MTP itself is implicated.
• [LOW] Sticky GPU/runner state from prior jobs causes degraded numerics on this specific MI35x box
  • Supporting: Apr 29 explicit log line RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes.; PR #24065 and #23314 recently reorganized AMD CI concurrency/parallelism, suggesting host contention has been a known concern.
  • Disconfirming: today's run actually completed model load and emitted real accuracy numbers (no -9, no watchdog), so this hypothesis explains Apr 29/30 better than May 1.
  • How to verify: rerun the May 1 job on a freshly rebooted MI35x runner; if accuracy returns to ≥0.94, the hypothesis is supported.
• No hypothesis above LOW confidence; needs git bisect between [ad785a22](https://github.com/sgl-project/sglang/commit/ad785a22)..[530b497a](https://github.com/sgl-project/sglang/commit/530b497a) (16 commits) plus a runner-isolation rerun.

In-flight Fix Check

Searched open PRs with: deepseek_v32+mtp, deepseek+v32+accuracy, nsa+accuracy, mi35x+accuracy, gsm8k+0.04, deepseek-v3.2+amd+accuracy. Closest hits (#22851 --nsa-topk-backend, #23877 Use TileLang NSA backend policy on HIP, #24148 SGLANG_AITER_FUSED_QK_ROPE_CAT_AND_CACHE_MLA) are NSA/AMD plumbing PRs, not fixes targeting this regression.

• In-flight fix: ❌ none found in open PRs that explicitly targets the DeepSeek-V3.2 MTP GSM8K accuracy collapse on MI35x — needs new PR or investigation

Suggested Triage Steps

• Bisect commits in ad785a22..530b497a (16 commits) running python3 test/registered/amd/accuracy/mi35x/test_deepseek_v32_mtp_eval_mi35x.py TestDeepseekV32TPMTP.test_a_gsm8k on an MI35x box — single-test runtime is ~12 min so bisect cost is bounded (~4 iterations).
• Before bisecting, repro on a freshly rebooted MI35x runner with no concurrent jobs to rule out hypothesis 3 (sticky GPU state). If accuracy returns to ≥0.94, the failure is infra-related, not a sglang regression.
• Inspect a few sample completions from the failing run (the eval prints generations on invalid cases) to determine how outputs are wrong — repetition / EOS-too-early / wrong language / off-topic. Each pattern points at a different subsystem (sampler, tokenizer, EAGLE draft, NSA attention numerics).
• A/B: rerun once with --speculative-algorithm removed (no MTP) to isolate whether the failure is MTP-specific. If accuracy recovers without MTP, the EAGLE/MTP path is implicated; if it stays at 0.04, the base DeepSeek-V3.2 + NSA tilelang path on MI35x is the culprit.
• If bisect implicates bd448e5, revert on a branch and rerun; if it implicates 73e93be, examine its KV-cache strategy abstraction for AMD/HIP code-path changes.

Status

• Persistence: Test file has failed in every completed nightly since 2026-04-29 (3 consecutive runs, 3 days). Today (2026-05-01) is the first run that reached the assertion with real numbers; Apr 29/30 failures stopped earlier in the pipeline with different symptoms.
• Scope: 1 job (nightly-accuracy-8-gpu-mi35x-deepseek-v32-mtp) in 1 workflow (nightly-test-amd.yml). Sister nightly-perf-8-gpu-mi35x-deepseek-v32-mtp job is currently green (it does not run a GSM8K accuracy gate, so it would not catch this).
• Blocked work: Blocks AMD MI35x DeepSeek-V3.2 + EAGLE MTP nightly accuracy signal.
• In-flight fix: ❌ none found in open PRs — needs new PR or investigation

nightly-8-gpu-mi35x-glm51 — failed step(s): Accuracy Test MI35x (8-GPU GLM-5.1 NSA)

Commit Info

• sglang: e0474fd
• aiter: N/A (no [CI-AITER-CHECK] markers in this job's log; install step did not log aiter version)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_glm51_eval_mi35x.py	test_glm51_accuracy	AssertionError: Failed models: ['zai-org/GLM-5.1-FP8 (nsa)'] (root: RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes. pre_model_load_memory=0.00390625, local_gpu_memory=0.65234375)	Log

Failure Summary

test_glm51_accuracy failed because the zai-org/GLM-5.1-FP8 (nsa) server launch crashed during init_torch_distributed. The TP memory imbalance guard (python/sglang/srt/model_executor/model_runner.py:1227) raised RuntimeError because the distributed-min free GPU memory across the TP=8 group was effectively 0 GB (pre_model_load_memory=0.0039 GB) while the local rank reported ~0.65 GB free — i.e. at least one of the 8 MI35X GPUs was already nearly fully occupied before the model attempted to load.

Regression Status

Recurring failure — second consecutive scheduled run failing with the same symptom on the same test.

Recent history of test/registered/amd/accuracy/mi35x/test_glm51_eval_mi35x.py (test_glm51_accuracy) in job nightly-8-gpu-mi35x-glm51 (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
May 02	25257970669	nightly-8-gpu-mi35x-glm51	❌ Failed	test_glm51_accuracy	TP memory imbalance (free=0.004/0.65 GB)
May 01	25225678502	nightly-8-gpu-mi35x-glm51	❌ Failed	test_glm51_accuracy	TP memory imbalance (free=6.98/8.68 GB)
Apr 30	25180906638	nightly-8-gpu-mi35x-glm51	✅ Passed	—	—
Apr 29	25125309566	nightly-8-gpu-mi35x-glm51	✅ Passed	—	—
Apr 28	25069425979	nightly-8-gpu-mi35x-glm51	✅ Passed	—	—

First observed failure: 2026-05-01 18:57 UTC. Last passing run: 2026-04-30. Lookback window: 5 most recent completed schedule runs.

Failure Cluster

TP memory imbalance at server startup — at least one MI35X GPU pre-occupied before nightly job starts

Facts (verified observations, no interpretation)

• Verbatim error from log line 2597: RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes. pre_model_load_memory=0.00390625, local_gpu_memory=0.65234375, local_gpu_memory * 0.9=0.587109375.
• Crash happens in python/sglang/srt/model_executor/model_runner.py:1231 inside init_torch_distributed, raised by the guard at line 1227 (if pre_model_load_memory < local_gpu_memory * 0.9).
• The error is raised by TP1 (one of the 8 ranks). Other TP ranks logged Init torch distributed ends. mem usage=9.16–9.58 GB, indicating the model never reached load.
• TP=8, server cmd line 2613: sglang serve --model-path zai-org/GLM-5.1-FP8 ... --nsa-prefill-backend tilelang --nsa-decode-backend tilelang --tp 8 --device cuda --host 127.0.0.1 --port 11000 --mem-fraction-static 0.80.
• Same job was ✅ passing on Apr 28-30 and ❌ failing on May 01 (pre_model_load_memory=6.98 GB, local_gpu_memory=8.68 GB) and May 02 (pre_model_load_memory=0.0039 GB, local_gpu_memory=0.65 GB). The free-memory delta has WORSENED between consecutive failures (from ~7 GB free yesterday to essentially 0 GB free today), inconsistent with a deterministic code regression.
• No [CI-AITER-CHECK] markers appear in this log; aiter version cannot be verified from this log alone.
• Sister job nightly-8-gpu-glm51 (NVIDIA) on the May 01 run 25225678502 ✅ passed (success), so the failure is AMD-MI35X-specific.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Runner-state contamination: a prior process on the MI35X runner is leaving GPU memory pinned before the nightly job starts, and the TP guard (SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK) correctly aborts the launch.

  • Supporting: free-memory levels differ wildly between the two failing runs (6.98 GB → 0.0039 GB) — a code regression would yield deterministic numbers, while shared-runner contamination is naturally noisy. Sister NVIDIA job nightly-8-gpu-glm51 passed on May 01, so the issue is not in the model or the test, but in the AMD runner environment. The crash site is a sanity guard, not the model code.
  • Disconfirming: failure is now reproducing two days in a row, which is more persistent than typical transient runner contamination. We cannot confirm "other processes are on the GPU" from inside this log — we only see the symptom.
  • How to verify: SSH into the MI35X runner before the next scheduled run and inspect rocm-smi/amd-smi for resident processes on each of the 8 GPUs; or have the job dump rocm-smi --showpidgpus before invoking sglang serve.

• [LOW] Some commit landed in main between Apr 30 (last pass) and May 01 (first fail) regressed memory accounting / leaked GPU memory in a previous test phase or in installer scripts.

  • Supporting: the temporal correlation: regression starts on the May 01 nightly (head_sha ece8a1a7) which includes commit window 530b497a..ece8a1a7.
  • Disconfirming: the test is the first/only file run by this job (line 2360) — there is no preceding test in this job that could leak. The Sister NVIDIA nightly-8-gpu-glm51 job at the same May 01 head SHA passed. The differing free-memory numbers across the two failures argue against deterministic code-driven leak. model_runner.py was last touched by the in-window commit ece8a1a (#24197 device-timer refactor) but the modified region does not affect init_torch_distributed memory accounting.
  • How to verify: bisect 530b497a..ece8a1a7 on the same MI35X runner with a clean GPU state.

• [LOW] GLM-5.1 NSA TileLang backend is leaking GPU memory in a co-tenant process on the same physical node.

  • Supporting: failure is GLM-5.1 + NSA-TileLang specific; multiple NSA-related PRs (#23877, #23963, #24125) are in flight.
  • Disconfirming: the leak would have to be from an external prior job — this job has not loaded the model yet. Hard to attribute without knowing what other jobs share the runner.
  • How to verify: query the runner's job history before the May 01/02 nightlies; check whether a prior job on the same physical host left a process behind.

No hypothesis above MEDIUM confidence; needs runner-state inspection to confirm.

In-flight Fix Check

Searched open sglang PRs with keywords {memory capacity unbalanced, mi35x glm51, GLM-5.1 nsa}.

• In-flight fix: ❌ none found in open PRs — no PR specifically targets the TP memory-imbalance crash for the MI35X GLM5.1 nightly. Related but non-fix PRs: #23877, #23963, #24125, #23550 all touch GLM-5.1 / NSA but none address pre-launch GPU contamination on the AMD nightly runner.

Suggested Triage Steps

• Inspect the MI35X runner state immediately before the next scheduled nightly: run rocm-smi --showpidgpus and amd-smi process on the host to identify which PID(s) are holding GPU memory; correlate to prior CI jobs on the same runner.
• Add a pre-test diagnostic step to nightly-8-gpu-mi35x-glm51 that prints rocm-smi --showmeminfo vram and rocm-smi --showpidgpus for all 8 GPUs before invoking sglang serve; this surfaces "which GPU and which PID" in future failures instead of just "imbalance detected".
• Compare the runner allocation history: was the same physical host used for a different job (e.g. another LLM) immediately before the nightly on May 01 and May 02 but not on Apr 28-30? If so, that prior job is the candidate culprit.
• If runner contamination is confirmed, add a forced GPU reset / pkill -9 cleanup step to the runner's pre-job hook; do NOT relax the TP imbalance guard, as that would mask real failures.
• If runner is clean, bisect 530b497a..ece8a1a7 on the MI35X runner.

Status

• Persistence: 2 consecutive scheduled runs failing (since 2026-05-01 18:57 UTC, ~24 h).
• Scope: 1 job (nightly-8-gpu-mi35x-glm51) in nightly-test-amd.yml. Sister NVIDIA job nightly-8-gpu-glm51 ✅ passes.
• Blocked work: GLM-5.1-FP8 NSA accuracy signal on AMD MI35X.
• In-flight fix: ❌ none found — needs runner-side investigation or new PR.

nightly-8-gpu-mi35x-qwen35 — failed step(s): Accuracy Test MI35x (8-GPU Qwen 3.5)

Commit Info

• sglang: e0474fdd
• aiter: a6bb4993 (source: dev image — v0.1.12.post2.dev150+ga6bb49937 preserved, no rebuild; matches Dockerfile default)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/accuracy/mi35x/test_qwen35_eval_mi35x.py	TestQwen35EvalMI35x.test_lm_eval	RuntimeError: The memory capacity is unbalanced. Some GPUs may be occupied by other processes. pre_model_load_memory=11.705078125, local_gpu_memory=13.408203125, local_gpu_memory * 0.9=12.0673828125 → server exits with code -9, retry exhausted	Log

Failure Summary

The single test in this nightly suite, TestQwen35EvalMI35x.test_lm_eval (Qwen3.5-397B-A17B, TP=8 on MI35x), failed because the sglang server's pre-model-load GPU memory probe (init_torch_distributed) found pre_model_load_memory=11.705078125 already occupied vs. a local_gpu_memory=13.408203125 ceiling, exceeding the 90% threshold and triggering an early RuntimeError. The check ran twice (initial + 1 retry); both attempts failed with the identical numbers, the scheduler emitted SIGQUIT, and the test reported Server process exited with code -9.

Regression Status

Flaky test / Infrastructure issue — intermittent failure with an identical numeric signature on the same runner family.

Recent history of test/registered/amd/accuracy/mi35x/test_qwen35_eval_mi35x.py (test_lm_eval) in job nightly-8-gpu-mi35x-qwen35 (branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
May 02	25257970669	nightly-8-gpu-mi35x-qwen35	❌ Failed	test_lm_eval	memory unbalanced 11.71/13.41
May 01	25225678502	nightly-8-gpu-mi35x-qwen35	✅ Passed	—	—
Apr 30	25180906638	nightly-8-gpu-mi35x-qwen35	❌ Failed	test_lm_eval	memory unbalanced 11.705078125 / 13.408203125 (numerically identical)
Apr 29	25125309566	nightly-8-gpu-mi35x-qwen35	✅ Passed	—	—
Apr 28	25069425979	nightly-8-gpu-mi35x-qwen35	✅ Passed	—	—
Apr 27	25010934840	nightly-8-gpu-mi35x-qwen35	✅ Passed	—	—
Apr 26	24963018193	nightly-8-gpu-mi35x-qwen35	✅ Passed	—	—

First observed failure (this signature): 2026-04-30. Last passing run: 2026-05-01. Failure ratio in last 7 completed runs: 2/7.

Failure Cluster

GPU memory pre-occupied at server startup on MI35x runner (sglang init_torch_distributed 90% capacity guard tripped)

Facts (verified observations, no interpretation)

• The error is raised by python/sglang/srt/model_executor/model_runner.py:1231 inside init_torch_distributed, before any model weights are loaded; the message is pre_model_load_memory=11.705078125, local_gpu_memory=13.408203125, local_gpu_memory * 0.9=12.0673828125 (log line 2666 / 2896).
• Both retry attempts (18:43:44 and 18:44:18 UTC) hit the exact same numeric values (11.705078125 and 13.408203125), and the Apr 30 failure (run 25180906638) reports the bit-identical numbers.
• Job duration was only 8 m 27 s (started 18:36:28, completed 18:44:55 UTC) — the failure occurs at server warmup, before lm-eval ever runs.
• aiter version inside the CI image was preserved (Dev/patched version detected: v0.1.12.post2.dev150+ga6bb49937 → skipping rebuild, line 2279). It matches the Dockerfile default commit a6bb499375849eec45d68c5ccaebc8865fd422c0 so no override was applied.
• AITER_COMMIT_OVERRIDE is empty in env (line 33, 114, 2417). This is a regular nightly, not an AITER Scout run, so no Baseline A/B is required.
• Only one test is registered for this suite (nightly-amd-accuracy-8-gpu-mi35x-qwen35), so the job's pass/fail equals the test file's pass/fail.
• ROCm-visible memory ceiling reads as 13.408… (likely GiB on whatever device subset the container exposes — significantly below an MI355X's nominal 288 GiB HBM, suggesting the visibility/partitioning of the device is itself constrained).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Stale GPU memory left over from a prior process on the same MI35x runner — i.e., a containerization / runner-cleanup issue, not an sglang code change.
  • Supporting: Identical numeric signature (11.705078125, 13.408203125) across two failing runs (May 02 e0474fdd and Apr 30 7bb7f604) two days apart with different code — strongly indicative of a fixed environmental footprint rather than a code path. 5 of 7 recent completed runs pass on the same job/runner family. Failure happens before model load, ruling out model-side regressions.
  • Disconfirming: We do not have direct rocm-smi evidence in the log of which PIDs hold the 11.7 GB; the inference is from the deterministic numeric signature. Could in principle be a reduced HIP_VISIBLE_DEVICES partition where 11.7 GB is "always" baseline (but then every run would fail, which is not observed).
  • How to verify: Add a rocm-smi --showpids / --showmeminfo capture to amd_ci_exec.sh before launching the server in the qwen35 mi35x suite; correlate the 11.7 GB resident with a leftover container/PID. Alternatively, force a runner reboot or docker system prune -af && rocm-smi --gpureset before this job.
• [LOW] sglang init_torch_distributed 0.9 capacity threshold is too strict for the MI35x partitioned-device CI image.
  • Supporting: With local_gpu_memory=13.4 GiB exposed to the container, baseline driver/runtime overhead is a large fraction of the budget; 11.7 GiB pre-allocated leaves only ~1.7 GiB headroom, which is < 10%.
  • Disconfirming: 5/7 passing runs go through the same code path on the same runner family without tripping; the threshold itself is therefore not the proximate cause when the runner is clean.
  • How to verify: Diff pre_model_load_memory values across the 5 passing runs vs. the 2 failing ones; if passing runs report values well below 12 GiB, the runner-cleanup hypothesis holds.
• [LOW] Some sglang change in the May 1 → May 2 window introduced an ordering bug that leaks GPU memory across processes within the same CI job.
  • Supporting: A new local branch origin/fix_mamba_leak is referenced in the fetch step (line 441), suggesting active work on memory leaks elsewhere.
  • Disconfirming: Apr 30 failure (commit 7bb7f604) used a much earlier sglang HEAD with the same numeric signature as today, so the failure is not specific to commits that landed in the May 1 → May 2 window. This makes a fresh sglang regression unlikely to fully explain the observation.
  • How to verify: git log --oneline 7bb7f604..e0474fdd python/sglang/srt/model_executor/ python/sglang/srt/managers/scheduler.py to enumerate scheduler/model-runner changes; would only matter if the next failing run drops back to a passing SHA.

No hypothesis above MEDIUM confidence; needs rocm-smi/runner-side evidence to promote to HIGH.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs for memory capacity unbalanced (0 hits) and qwen35 mi35x (1 hit, #18005 "AMD MXFP4 Support online MXFP4 quantization in quark_w4a4_mxfp4.py" — unrelated to this failure).

Suggested Triage Steps

• Re-run the failed shard once to confirm the flake hypothesis (passed/failed in 5/7 prior runs) — if it passes, the May 2 failure is consistent with the Apr 30 flake pattern and is runner-side.
• On the MI35x runner where this job lands, run rocm-smi --showpids / --showmeminfo immediately after the previous job exits and before this job starts, to identify the process holding 11.7 GiB. Add this dump to scripts/ci/amd/amd_ci_exec.sh for the next run so the next failure carries diagnostic data inline.
• Compare pre_model_load_memory values from the 5 recent passing runs of this same job vs. the 2 failing ones (search log for pre_model_load_memory= in each). If passing runs show values well under 12 GiB, the 0.9 ceiling is fine and the issue is purely leftover memory.
• If the runner-cleanup root cause is confirmed, add a pre-job docker system prune + GPU reset (or kill any leftover sglang.launch_server processes) to the runner setup; do not loosen the * 0.9 guard in model_runner.py:1231, as it correctly protects against OOMs at warmup.
• Out of scope here, but: investigate whether the very low local_gpu_memory=13.4 GiB visibility on the MI35x container is intentional partitioning; if not, a misconfigured HIP_VISIBLE_DEVICES / cgroup may be amplifying the impact of any leftover allocation.

Status

• Persistence: Intermittent — 2 failures in last 7 completed runs (May 02 and Apr 30), with identical numeric signature; passing on May 01, Apr 29, Apr 28, Apr 27, Apr 26.
• Scope: 1 job (nightly-8-gpu-mi35x-qwen35) in 1 workflow (nightly-test-amd.yml); the sister nightly-8-gpu-qwen35 job on a different (non-mi35x) runner passed in the same May 1 run, so failure is specific to the MI35x runner family.
• Blocked work: Loses Qwen3.5-397B-A17B accuracy signal on MI35x for this nightly when it triggers; does not block PR CI.
• In-flight fix: ❌ none found in open PRs — needs runner-side investigation or a new PR.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-05-02 22:27 UTC)

[amd-bot]

pr-test-amd.yml — 31 failure(s)

Run started (UTC): 2026-05-01 00:48 UTC
Last scanned (UTC): 2026-05-02 20:30 UTC
Commits: sglang e0474fd

Summary

Counts: 31 failures · 10 clusters

#	Cluster	Job	Test File	Test Function	Status	Hypothesis (confidence)
1	GPT-OSS 20B mxfp4 streaming returns empty content on MI35x	stage-b-mi35x	test_gpt_oss_1gpu.py	test_mxfp4_20b	≥18 days persistent (Never-passed)	[MEDIUM] mxfp4 MoE on gfx950 emits only reasoning-channel tokens; no output_text.delta events fire
2	GPT-OSS 20B mxfp4 streaming returns empty content on MI35x	stage-b-mi35x	test_gpt_oss_1gpu.py	test_mxfp4_20b	≥18 days persistent	same
3	GPT-OSS 20B mxfp4 streaming returns empty content on MI35x	stage-b-mi35x	test_gpt_oss_1gpu.py	test_mxfp4_20b	≥18 days persistent	same
4	GPT-OSS 20B mxfp4 streaming returns empty content on MI35x	stage-b-mi35x	test_gpt_oss_1gpu.py	test_mxfp4_20b	≥18 days persistent	same
5	GPT-OSS 20B mxfp4 streaming returns empty content on MI35x	stage-b-mi35x	test_gpt_oss_1gpu.py	test_mxfp4_20b	≥18 days persistent	same
6	GPT-OSS 20B mxfp4 streaming returns empty content on MI35x	stage-b-mi35x	test_gpt_oss_1gpu.py	test_mxfp4_20b	≥18 days persistent	same
7	GPT-OSS 20B mxfp4 streaming returns empty content on MI35x	stage-b-mi35x	test_gpt_oss_1gpu.py	test_mxfp4_20b	≥18 days persistent	same
8	Aiter fused AR+RMSNorm residual 1-ULP bf16 divergence	stage-c shard 2	test_aiter_allreduce_fusion_amd.py	test_fused_ar_rms_residual_accuracy	≥6 days persistent (Never-passed)	[MEDIUM] aiter a6bb499 still has ROCm/aiter#2586 residual-rounding bug
9	Aiter fused AR+RMSNorm residual 1-ULP bf16 divergence	stage-c shard 2	test_aiter_allreduce_fusion_amd.py	test_fused_ar_rms_residual_accuracy	≥6 days persistent	same
10	Aiter fused AR+RMSNorm residual 1-ULP bf16 divergence	stage-c shard 2	test_aiter_allreduce_fusion_amd.py	test_fused_ar_rms_residual_accuracy	≥6 days persistent	same
11	Aiter fused AR+RMSNorm residual 1-ULP bf16 divergence	stage-c shard 2	test_aiter_allreduce_fusion_amd.py	test_fused_ar_rms_residual_accuracy	≥6 days persistent	same
12	Aiter fused AR+RMSNorm residual 1-ULP bf16 divergence	stage-c shard 2	test_aiter_allreduce_fusion_amd.py	test_fused_ar_rms_residual_accuracy	≥6 days persistent	same
13	Aiter fused AR+RMSNorm residual 1-ULP bf16 divergence	stage-c shard 2	test_aiter_allreduce_fusion_amd.py	test_fused_ar_rms_residual_accuracy	≥6 days persistent	same
14	Aiter fused AR+RMSNorm residual 1-ULP bf16 divergence	stage-c shard 2	test_aiter_allreduce_fusion_amd.py	test_fused_ar_rms_benchmark	≥6 days persistent	same (different sub-test due to unittest ordering)
15	Aiter fused AR+RMSNorm residual 1-ULP bf16 divergence	stage-c shard 2	test_aiter_allreduce_fusion_amd.py	test_fused_ar_rms_benchmark	≥6 days persistent	same
16	MoriEP server crash / hang / timeout on MI325 8-GPU	stage-c shard 2	test_moriep_small.py	TestMTPwithTBONormal.test_gsm8k	≥11 days persistent (intermittent symptom)	[LOW] TBO+EAGLE+DP-attn interaction; watchdog timeout / HSA OOR / split_spec_info crash
17	MoriEP server crash

---

Job	Failed Steps	Started
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2)	Run test	2026-05-01T19:37
stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1)	Ensure VRAM is clear	2026-05-01T12:25
stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1)	Run test	2026-05-01T18:28
stage-b-test-large-8-gpu-35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Run test_disaggregation	2026-05-01T12:25
stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 0)	Run test	2026-05-01T07:01
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2)	Run test	2026-05-01T12:25
stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1)	Run test	2026-05-01T06:59
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2)	Run test	2026-05-01T07:00
stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1)	Run test	2026-05-01T00:48
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2)	Run test	2026-05-01T02:02
stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1)	Run test	2026-05-02T00:44
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 0)	N/A	2026-05-02T00:44
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2)	Run test	2026-05-02T00:44
stage-b-test-large-8-gpu-35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)	Ensure VRAM is clear	2026-05-02T06:44
stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1)	Run test	2026-05-02T06:44
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 13)	Run test	2026-05-02T06:44
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 0)	Run test	2026-05-02T06:44
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 11)	Run test	2026-05-02T06:44
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 7)	Run test	2026-05-02T06:44
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 4)	Run test	2026-05-02T06:44
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2)	Run test	2026-05-02T06:44
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 2)	Run test	2026-05-02T12:23
stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1)	Run test	2026-05-02T12:23
stage-a-test-1-gpu-small-amd (linux-mi325-1gpu-sglang)	Run test	2026-05-02T12:23
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 13)	Run test	2026-05-02T12:23
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 1)	Run test	2026-05-02T12:23
stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 7)	Run test	2026-05-02T12:23
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2)	N/A	2026-05-02T12:23
sgl-kernel-unit-test-amd (linux-mi325-1gpu-sglang)	Run test	2026-05-02T18:22
stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1)	Run test	2026-05-02T18:22
stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2)	Run test	2026-05-02T18:32

Per-Job Analysis

stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2) — failed step(s): Run test

Commit Info

• sglang: ece8a1a78
• aiter: a6bb49937 (source: dev image; [CI-AITER-CHECK] reports v0.1.12.post2.dev150+ga6bb49937, no override, dev/patched build preserved — Dockerfile expects the same SHA)

Failed Tests

Test File	Test Function	Error	Log
test/registered/ops/test_aiter_allreduce_fusion_amd.py	TestAiterAllreduceFusionAmd.test_fused_ar_rms_residual_accuracy	AssertionError: Residual accuracy check failed. — fused-AR+RMSNorm path's residual diverges from fused_ar + residual (max_diff=6.250000e-02, frac_nonzero≈0.28, AR_exact=yes) for every (m, hidden_dim) combination across hidden_dim ∈ {2880, 4096, 5120, 6144, 7168, 8192} on both 1-stage and 2-stage paths	Log
test/registered/amd/test_moriep_small.py	TestMTPwithTBONormal.test_gsm8k	Exception: retry() exceed maximum number of retries. — server died mid-eval; root cause is Scheduler watchdog timeout (self.watchdog_timeout=300) on DP7 TP7 EP7 followed by Subprocess scheduler_0 (pid=28611) crashed with exit code -3; py-spy of the hung scheduler sits in prepare_mlp_sync_batch_raw → all_gather → _get_fallback_tensor ⇒ c10::hip::memcpy_and_sync inside libamdhip64.so.7.0.70000	Log

Failure Summary

Two unrelated failures in stage-c shard 2 of run 25227171827: (1) the dedicated regression test for ROCm/aiter#2586 (1-stage fused all-reduce + RMSNorm residual accuracy) reports a constant max_diff=6.25e-2 across ALL hidden-dim/batch combinations even though the all-reduce result itself is bit-exact (AR_exact=yes); (2) the test_moriep_small GSM8K eval crashed because a DP-attention scheduler hung in _get_fallback_tensor for >300 s and was killed by the watchdog (exit code -3). Both have been failing in this shard for days.

Regression Status

Known recurring failure (both tests). This shard has been red on every completed scheduled pr-test-amd.yml run going back at least to 25109261379 (2026-04-29 12:38 UTC). No green run found in the queryable window (last ~10 schedule runs over ≥3 days). No older passing baseline located within the window.

Recent history of test/registered/ops/test_aiter_allreduce_fusion_amd.py and test/registered/amd/test_moriep_small.py in job stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2) (branch=main, event=schedule, completed runs only):

Date (UTC)	Run	Job	aiter-allreduce file status	aiter failed function	moriep file status	moriep failed function
2026-05-01 18:27	25227171827	stage-c shard 2	❌ Failed	test_fused_ar_rms_residual_accuracy	❌ Failed	test_gsm8k (TestMTPwithTBONormal)
2026-05-01 12:25	25214178926	stage-c shard 2	❌ Failed	test_fused_ar_rms_residual_accuracy	❌ Timeout (>3600 s)	(whole-file timeout)
2026-05-01 06:59	25205835327	stage-c shard 2	❌ Failed	test_fused_ar_rms_benchmark (line 253)	❌ Timeout (>3600 s)	(whole-file timeout)
2026-05-01 00:48	25196776069	stage-c shard 2	❌ Failed	test_fused_ar_rms_benchmark	❌ Failed	setUpClass (line 465)
2026-04-30 12:37	25165743184	stage-c shard 2	❌ Failed	test_fused_ar_rms_benchmark	❌ Failed	setUpClass (line 465)

Lookback range searched: 2026-04-29 12:38 UTC → 2026-05-01 18:27 UTC (10 completed schedule runs of pr-test-amd.yml). The aiter-allreduce file fails every run; which sub-function trips the assertion alternates (benchmark vs residual_accuracy) — that's an artifact of unittest method ordering (alphabetical: benchmark runs first; once a fix lands for the benchmark assertion, the next test exposes residual_accuracy). The moriep file fails every recent run with a different symptom (server crash / DP-attn watchdog / setUpClass failure), suggesting infrastructure-level flakiness rather than a single regression in the test itself.

Failure Cluster

Two distinct clusters in this shard:

• aiter fused-AR+RMSNorm residual diverges (max_diff=6.25e-2, AR_exact=yes)
• moriep server crash — DP-attn scheduler watchdog timeout in prepare_mlp_sync_batch_raw → all_gather

Facts (verified observations, no interpretation)

• aiter version on the runner: v0.1.12.post2.dev150+ga6bb49937 (a6bb49937); AITER_COMMIT_OVERRIDE empty; dev/patched build preserved ([CI-AITER-CHECK] markers at lines 2345-2351 of the log).
• The aiter-residual test prints AR_exact=yes for every shape — the all-reduce intermediate matches bit-for-bit between fused and unfused paths. The mismatch is only in the residual-add output (fused_res ≠ fused_ar + residual).
• max_diff=6.25e-2 = 1/16 = 2^-4, identical for every shape; frac_nonzero ≈ 0.28. This is the pattern the test was specifically written to catch (per its docstring referencing Fix: Numerical Accuracy in allreduce_fusion_kernel_1stage ROCm/aiter#2586: 1-stage kernel accumulates in f32 and adds residual before rounding to bf16, while the unfused path rounds AR to bf16 first; 1-ULP divergence at bf16 = 2^-7 × max-magnitude, which gives 6.25e-2 for values near 8).
• The test docstring: "Regression test for the 1-stage kernel accuracy bug (Fix: Numerical Accuracy in allreduce_fusion_kernel_1stage ROCm/aiter#2586) ... The 1-ULP divergence compounded across layers and caused a -2.6pp GSM8K regression." (test/registered/ops/test_aiter_allreduce_fusion_amd.py:18-28).
• moriep py-spy stack of the hung process (Process 28958: sglang::scheduler_DP7_TP7_EP7) at log line 16100-16175: event_loop_normal → get_next_batch_to_run → maybe_prepare_mlp_sync_batch → prepare_mlp_sync_batch → prepare_mlp_sync_batch_raw → all_gather → _get_fallback_tensor → … → c10::hip::memcpy_and_sync → libamdhip64.so.7.0.70000 → libhsa-runtime64.so.1.18.70000.
• Other shard-2 tests in this run passed: test_deepseek_v3_basic.py, test_deepseek_v3_mtp.py. So the shard-level infra (driver, NCCL setup) is at least partially functional.

Hypothesised Causes (with confidence, with disconfirming evidence)

Cluster A — aiter residual mismatch:

• [HIGH] The aiter allreduce_fusion_kernel_1stage (and the same defect path inherited by 2-stage in this build) still adds the residual before rounding the f32 AR accumulator to bf16, exactly the bug described in ROCm/aiter#2586. The test was designed to catch this and the symptom matches the docstring's prediction (max_diff = 1 ULP at bf16, AR_exact=yes).
  • Supporting evidence: identical max_diff=6.25e-2 for ALL shapes; AR_exact=yes (so it is NOT an AR communication bug); the test author's docstring describes precisely this signature; aiter pin in the dev image (a6bb49937) does not contain the upstream fix.
  • Disconfirming evidence: I have not located an aiter PR that explicitly fixes #2586 and pins it past a6bb49937 to confirm the fix is missing rather than reverted. Cannot confirm directly without diffing aiter's allreduce_fusion_kernel*.{cu,hip}.
  • How to verify: in the same container, run only _run_residual_accuracy_check() after rebuilding aiter from a SHA known to contain the #2586 fix (or against main of ROCm/aiter); confirm ALL PASSED.

Cluster B — moriep DP-attn scheduler hang:

• [LOW] DP-attention all-gather of fallback tensors hangs on shard 2 only. The py-spy stack shows the DP7 scheduler stuck in _get_fallback_tensor → memcpy_and_sync while other DP ranks (DP1) had Pyspy failed (py-spy dump --native --pid 1969). Error: Failed to get stack traces earlier — possibly indicating cross-rank state divergence, not a pure HIP runtime hang.
  • Supporting evidence: hang location is deterministic (scheduler_dp_attn_mixin.py:202 prepare_mlp_sync_batch_raw → all_gather); this same job/shard has reported moriep failures (server crash / setUpClass / timeout) on every completed run since at least 2026-04-30; this is an MTP+TBO+EP test (likely the most communication-heavy moriep config in the file).
  • Disconfirming evidence: failure symptoms vary per run (setUpClass failure → 3600 s timeout → 300 s watchdog crash). A single root cause should reproduce a consistent symptom; the variability suggests either (a) flaky multi-GPU comm or (b) a test bug that gets caught at different points depending on timing.
  • How to verify: re-run only TestMTPwithTBONormal.test_gsm8k 5× in isolation on the same runner; if symptom changes per run, treat as flaky/comm-instability rather than a single regression. Also check whether the same DP-attn hang appears in nightly-test-amd MoE jobs (sister-job comparison).
• [SPECULATION] Could be downstream of the same aiter version (same [CI-AITER-CHECK] build), since moriep MoE also calls aiter custom_all_reduce — but no concrete evidence connects the residual-rounding bug to a hang in DP-attn fallback all-gather.

No hypothesis above MEDIUM for Cluster B; it needs a clean repro / bisect on the moriep test specifically.

In-flight Fix Check

Searched open sglang PRs with keywords aiter+allreduce+fusion+residual, moriep, test_moriep_small:

• For Cluster A: ❌ no fix PR in sglang for the aiter residual-rounding bug — the fix has to land in ROCm/aiter (reference issue Fix: Numerical Accuracy in allreduce_fusion_kernel_1stage ROCm/aiter#2586). #17262 ("Fuse aiter allreduce with residual_rmsnorm and quantization (fp8)") is the original feature PR but does not address this rounding regression.
• For Cluster B: ❌ no in-flight fix found for test_moriep_small / DP-attn fallback-tensor all-gather hang. Open moriep-related PRs (#22985, #23760) are feature work, not bug fixes for this hang.

Suggested Triage Steps

• Cluster A (aiter residual): rebuild the dev image with an aiter SHA that includes the upstream fix for ROCm/aiter#2586 (or set AITER_COMMIT_OVERRIDE to such a SHA) and rerun this single test:
  python3 -m torch.distributed.run --standalone --nproc_per_node=4 test/registered/ops/test_aiter_allreduce_fusion_amd.py --residual-accuracy. Expect ALL PASSED. If the fix has not yet been merged in aiter, file/track it there.
• Cluster A: in parallel, audit ROCm/aiter's allreduce_fusion_kernel_1stage (and 2-stage) source to confirm whether the residual-add now happens after the bf16 cast on the AR accumulator. Diff against the version pinned in a6bb49937.
• Cluster B (moriep hang): re-run TestMTPwithTBONormal.test_gsm8k 3-5× on the same MI325 8-GPU runner in isolation to determine flakiness vs deterministic regression. If deterministic, run py-spy dump --native --pid <DP*> on every DP rank during the hang to see whether ranks diverge before all_gather (use the debug-distributed-hang playbook).
• Cluster B: check whether the same prepare_mlp_sync_batch_raw → _get_fallback_tensor hang reproduces in any other moriep or DP-attention test (nightly-test-amd.yml MoE shards) — if yes, the bug is in scheduler_dp_attn_mixin.py:60-93, not in moriep itself.
• Both: bisect the sglang commit window between this run's HEAD (ece8a1a78) and the last known-green stage-c shard 2 (NOT FOUND in the queryable 10-run window — needs an extended search of older completed runs to find a green baseline; if none exists in 30 days, treat as Never-passed for this shard configuration).

Status

• Persistence: Both tests have failed on every completed scheduled pr-test-amd.yml run for stage-c shard 2 in the lookback window (≥10 runs across ≥3 days, since at least 2026-04-29 12:38 UTC). The aiter-allreduce file has been failing in this same shard at least that long; the specific function (benchmark vs residual_accuracy) varies by run because of unittest's alphabetical ordering. The moriep file has been failing with varying symptoms (timeout / setUpClass / scheduler watchdog crash) the whole time.
• Scope: Stage-c shard 2 of pr-test-amd.yml. Need cross-job summary to determine whether the aiter residual bug also affects other shards / workflows. The aiter version is the Dockerfile default (no override), so any other AMD job pulling the same dev image is affected.
• Blocked work: Stage-c-test-large-8-gpu-amd shard 2 has been a persistent red signal for pr-test-amd.yml for the entire lookback window.
• In-flight fix: ❌ none in open sglang PRs for either cluster. Cluster A fix must come from ROCm/aiter (issue #2586). Cluster B needs investigation before a fix can be proposed.

stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: 4197c55968
• aiter: N/A (job aborted before container start; no [CI-AITER-CHECK] markers emitted in this job's log)

Failed Tests

Test File	Test Function	Error	Log
N/A (infrastructure failure)	N/A	Ensure VRAM is clear step exited with code 1: GPU[0] VRAM 93% allocated, 3 cleanup attempts failed	Log

No tests executed. The job failed at step 3 ("Ensure VRAM is clear") of the workflow before any test could start. Steps 4-6 (Start CI container, Install dependencies, Run test) were skipped.

Failure Summary

The job aborted at the pre-flight scripts/ci/amd/ensure_vram_clear.sh step on runner linux-mi35x-gpu-1-jljvn-runner-c7bhp. GPU[0] reported 93% VRAM already allocated when the job started, and after 3 cleanup attempts (with 30 s waits between attempts 2 and 3) VRAM was still at 93%. The cleanup script could not reclaim the memory because No processes found accessing /dev/kfd — the allocation is held by something not visible to the runner's process listing.

Regression Status

Infrastructure issue (intermittent). Same VRAM-not-clear failure has been observed on a different runner pod within the lookback window; surrounding runs were unrelated test failures.

Recent history of the VRAM-clear pre-flight in job stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) (branch main, event schedule, completed runs only):

Date (UTC)	Run	Job	VRAM-clear Status	Runner pod	Notes
May 1 18:28	25227171827	job	✅ Passed (clean, 0%)	runner-8jclv	Reached tests; test_gpt_oss_1gpu.py timed out separately
May 1 12:25	25214178926	job	❌ Failed (93%)	runner-c7bhp	This run — never reached tests
May 1 06:59	25205835327	job	✅ Passed (clean, 0%)	runner-cvbd6	Reached tests; test_gpt_oss_1gpu.py failed
May 1 00:48	25196776069	job	✅ Passed (clean, 0%)	runner-xtnzn	Reached tests; test_gpt_oss_1gpu.py timed out
Apr 30 18:39	25182702322	job	✅ Passed (clean, 0%)	runner-xvfnd	Reached tests; test_gpt_oss_1gpu.py timed out
Apr 30 12:37	25165743184	job	❌ Failed (94%)	runner-r2cmp	Same pre-flight failure pattern

Two failures of the VRAM-clear pre-flight in the last 6 runs (~24 h), both with ~93–94% VRAM at start and zero processes visible holding it. Different runner pod each time — points to a host-level / shared-GPU issue rather than a single sticky pod.

Failure Cluster

"VRAM-clear pre-flight failure on linux-mi35x-gpu-1 runner (93% VRAM, no visible processes)"

Facts (verified observations, no interpretation)

• Step 3 (Ensure VRAM is clear) exited with code 1; steps 4-6 were skipped (per job API metadata).
• Initial GPU memory snapshot at job start: GPU[0] : GPU Memory Allocated (VRAM%): 93 (line 132 of job log).
• Cleanup script attempted 3 times with Killing SGLang processes + Performing aggressive cleanup + 30 s wait; final reading was still 93% (lines 155, 174, 193).
• No processes found using /dev/kfd and No processes found accessing /dev/kfd (lines 207, 209) — the allocation is not held by any visible process.
• Memory Read/Write Activity (%): 0 throughout, so it is allocated-but-idle memory.
• Runner pod was linux-mi35x-gpu-1-jljvn-runner-c7bhp; the prior failure with the same symptom (Apr 30 12:37, run 25165743184) was on a different pod runner-r2cmp.
• All 5 historical runs that I downloaded successfully had Complete job name: stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1); in 4 of them the VRAM check passed at 0%, in 1 it failed identically.
• The job log is only 227 lines and contains no [CI-AITER-CHECK] markers — the container was never started, so no aiter version info is available.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Host-level GPU memory leak / shared-GPU contention on the linux-mi35x-gpu-1 runner pool — a process from a previous (unrelated) workload on the same physical GPU left VRAM allocated, and the new ephemeral pod sees it but cannot kill the holder because it has no visibility into other pods' processes.
  • Supporting: identical 93–94% VRAM at job start across two pods (c7bhp, r2cmp) on the same runner label; No processes found accessing /dev/kfd from inside the pod (cleanup is a no-op); 4 other runs on the same label started clean at 0%; all six runs share the same workflow/script versions.
  • Disconfirming: I cannot directly observe the host's other tenants from CI logs alone, and 4 of 6 recent runs were clean — so whatever holds the VRAM is not constant.
  • How to verify: from a runner host, rocm-smi --showpids (host-level, not pod-level) and lsof /dev/kfd outside the container at the moment the pod starts; or have the cleanup script log host-side process info via privileged access.
• [LOW] ensure_vram_clear.sh cleanup logic doesn't handle this case — it only kills processes inside the pod, so when the holder is on another pod / host process it is unrecoverable from inside the runner.
  • Supporting: script is a pod-local kill; the log shows 3 cleanup loops all reading 93%.
  • Disconfirming: this is a design limitation, not a regression — the script works fine when the leak is in-pod.
  • How to verify: read scripts/ci/amd/ensure_vram_clear.sh (103 lines, last touched in unrelated PR #23962) — confirm cleanup is process-kill-only.
• [SPECULATION] Specific sglang commit at SHA 4197c559 caused the leak.
  • Supporting: temporal correlation with this run.
  • Disconfirming: the cleanup happens BEFORE any sglang code is loaded into a container (step 3 of 13, before "Start CI container"); the SHA only affects what would have run inside the container; same symptom occurred at unrelated SHA 692979a8d9 (Apr 30 12:37 run).
  • How to verify: not actionable — the job aborts before any sglang code from the head SHA is exercised.

No hypothesis above MEDIUM confidence; no test-level regression to bisect — the failure is in CI infrastructure setup before any sglang code runs.

In-flight Fix Check

• Searched repo:sgl-project/sglang is:pr is:open VRAM mi35x → 1 hit (#23957 "[AMD] Add CDNA4 Gluon Extend Attention…") — unrelated (attention backend feature, not infra).
• Searched repo:sgl-project/sglang is:pr is:open ensure_vram_clear → 0 hits.
• In-flight fix: ❌ none found — this is a runner-pool / cluster operations issue, not a code change

Suggested Triage Steps

• Re-run the failing job once — if a fresh runner pod starts with clean VRAM, the failure is confirmed as an infra flake and the green re-run unblocks signal.
• Inspect host-level state on the linux-mi35x-gpu-1-jljvn runner pool: rocm-smi --showpids from the host (not the pod), nvidia-smi/rocm-smi history, and check whether GPU isolation (vGPU / time-slice / MIG-equivalent) is misconfigured so two workloads can share GPU[0].
• Consider extending scripts/ci/amd/ensure_vram_clear.sh to (a) emit the runner pod ID + host name on every failure (for trend tracking), and (b) attempt a rocm-smi --gpureset 0 (or equivalent host-privileged reset) before giving up.
• File or update an existing infra ticket against the AMD CI runner provider — provide the two affected pods (runner-c7bhp 2026-05-01 12:25 UTC, runner-r2cmp 2026-04-30 12:37 UTC) and ask whether other workloads share GPU[0].
• Do not investigate sglang commits at 4197c559 for this specific failure — the failing step runs before any sglang code is loaded.

Status

• Persistence: 2 of last 6 completed scheduled runs (~33%) of stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) failed at the VRAM pre-flight in the last 24 h (this run and Apr 30 12:37). Other 4 runs in the window started with clean VRAM.
• Scope: Only linux-mi35x-gpu-1 runner pool observed so far in this job. Each failure was on a different ephemeral pod, so it's not a single sticky pod — likely host-level.
• Blocked work: When this happens, the entire stage-b-test-1-gpu-small-amd-mi35x job (including the only test it runs, test_gpt_oss_1gpu.py) is skipped, blanking out one slice of AMD CI signal for that scheduled run.
• In-flight fix: ❌ none found — needs runner-pool ops investigation, not a sglang code PR.

stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) — failed step(s): Run test

Commit Info

• sglang: ece8a1a78
• aiter: a6bb49937 (source: dev/patched image, version v0.1.12.post2.dev150+ga6bb49937 — matches Dockerfile default; rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_moriep_small.py	TestMTPwithTBONormal.test_gsm8k	Server scheduler aborted: HSA_STATUS_ERROR_OUT_OF_RESOURCES … Available Free mem : 0 MB → Subprocess scheduler_0 (pid=28611) crashed with exit code -3 → ConnectionRefusedError [Errno 111] on /get_model_info; FAILED (errors=1)	Log
test/registered/ops/test_aiter_allreduce_fusion_amd.py	test_fused_ar_rms_residual_accuracy	AssertionError: Residual accuracy check failed. Fused vs unfused residual diverges for every shape (hidden_dim ∈ {2880,4096,5120,6144,7168,8192} × tokens ∈ {1,4,8,16,32,64,128}, both 1-stage and 2-stage). max_diff=6.25e-2, frac_nonzero≈0.27, but AR_exact=yes (i.e. raw allreduce is correct, only the fused-residual output differs).	Log

Other tests in the shard passed: test_gpt_oss_1gpu.py, TestMTPwithTBOLowLatency.test_gsm8k (in moriep_small) returned accuracy=0.945, test_deepseek_v3_basic.py, test_deepseek_v3_mtp.py. The moriep failure is in the SECOND server launch in the file (TestMTPwithTBONormal, DeepSeek-V3-0324 TP=8 EP=8 DP=8).

Failure Summary

Two unrelated failures in shard 1 of stage-b-test-1-gpu-small-amd-mi35x on MI35x:

1. TestMTPwithTBONormal.test_gsm8k aborts at the very first prefill batch of its DeepSeek-V3-0324 server with an HSA OOM (Available Free mem : 0 MB); the GSM8k client then retries and fails with connection-refused (FAILED (errors=1)).
2. test_fused_ar_rms_residual_accuracy fails accuracy on every shape it checks; the unfused all-reduce path is bit-exact (AR_exact=yes) but the fused all-reduce + RMSNorm + residual write produces max_diff=6.25e-2 with ~27.8% of elements differing — a structural divergence, not a numerical-tolerance issue.

Regression Status

test/registered/ops/test_aiter_allreduce_fusion_amd.py — Never-passed in queryable history of this shard.
Recent runs of the same job on event=schedule, branch=main (completed only):

Date	Run	Job	Test File Status	Failed Function	Error
2026-05-01 18:27	25227171827	stage-b-test-1-gpu-small-amd-mi35x	❌ Failed	test_fused_ar_rms_residual_accuracy	residual accuracy max_diff=6.25e-2
2026-05-01 12:25	25214178926	stage-b-test-1-gpu-small-amd-mi35x	⚠️ Not run (job failed at VRAM cleanup infra step before tests)	—	—
2026-05-01 06:59	25205835327	stage-b-test-1-gpu-small-amd-mi35x	⚠️ Not in shard (only test_gpt_oss_1gpu.py ran, timed out)	—	—
2026-05-01 00:48	25196776069	stage-b-test-1-gpu-small-amd-mi35x	⚠️ Not in shard (gpt_oss timeout)	—	—
2026-04-30 18:34	25182702322	stage-b-test-1-gpu-small-amd-mi35x	⚠️ Not in shard (gpt_oss timeout)	—	—

Lookback: last 5 completed scheduled runs. Today's run is the first in the queryable window where the matrix actually reached test_aiter_allreduce_fusion_amd.py and test_moriep_small.py:TestMTPwithTBONormal — earlier runs were monopolized by a test_gpt_oss_1gpu.py timeout (1200 s) that consumed the shard budget. So regression history for these two specific tests cannot be established from the last 5 runs; mark Never-passed (no green run found in queryable window).

test/registered/amd/test_moriep_small.py — same situation. PR #24205 (a578bf81, "[AMD] fix moriep unittest failure", merged 2026-05-01) added SGLANG_ENABLE_SPEC_V2=false to both TestMTPwithTBONormal and TestMTPwithTBOLowLatency and IS included in this run. TestMTPwithTBOLowLatency.test_gsm8k passed; TestMTPwithTBONormal.test_gsm8k still aborts with HSA OOM, so #24205 only partially fixed the file.

Failure Cluster

Two distinct clusters in this single job:

• Cluster A — MoriEP DeepSeek-V3 server OOM during first prefill (TestMTPwithTBONormal)
• Cluster B — Aiter fused allreduce+RMSNorm+residual produces wrong residual on MI35x (gfx950)

Facts (verified observations, no interpretation)

Cluster A (moriep TestMTPwithTBONormal):

• Server cmd: --model-path deepseek-ai/DeepSeek-V3-0324 --tp-size 8 --ep-size 8 --dp-size 8 --enable-dp-attention --moe-a2a-backend mori --enable-dp-lm-head --mem-fraction-static 0.72 --chunked-prefill-size 32768 --max-running-requests 128 --context-length 12288 --attention-backend aiter --speculative-algo EAGLE --deepep-mode normal --enable-two-batch-overlap (log line 14759).
• The previous test in the same file (TestMTPwithTBOLowLatency.test_gsm8k) PASSED with accuracy=0.945 at line 14756, then tearDownClass ran kill_process_tree and wait_all_ports_release before launching the next server.
• New server launches; first request hits Prefill batch, #new-seq: 1, #new-token: 667 (line 14768) — at the same instant rocdevice.cpp:3671 reports HSA_STATUS_ERROR_OUT_OF_RESOURCES … Available Free mem : 0 MB (line 14767).
• Fatal Python error: Aborted follows; Subprocess scheduler_0 (pid=28611) crashed with exit code -3 at line 16192.
• Pyspy stacks (lines 14852–14870) show the crashing thread blocked in c10::hip::memcpy_and_sync → libamdhip64.so → libhsa-runtime64.so — i.e. an HIP H2D/D2D copy hits the OOM allocator.
• PR #24205 (SGLANG_ENABLE_SPEC_V2=false) is present in this commit; the env var is propagated.

Cluster B (aiter fused allreduce residual):

• Test launches itself via torchrun --nproc_per_node=4 … --residual-accuracy (line 23893).
• Every shape reports the same pattern: max_diff=6.25e-2 frac_nonzero≈0.27–0.28 AR_exact=yes [FAIL] (see lines 23963–24014).
• The summary line is FAILED: fused residual output diverges from unfused path for some shapes. (line 24016).
• AR_exact=yes means the underlying all-reduce result agrees bit-exactly; the divergence is in the fused-path residual write only.
• Test file's docstring: Regression test for ROCm/aiter#2586 (test_aiter_allreduce_fusion_amd.py:289).
• Runner = linux-mi35x-gpu-1 (gfx950, MI35x); aiter image is dev build v0.1.12.post2.dev150+ga6bb49937.

Hypothesised Causes (with confidence, with disconfirming evidence)

Cluster A — TestMTPwithTBONormal HSA OOM:

• [MEDIUM] GPU memory not fully released between consecutive server launches in test_moriep_small.py (TBOLowLatency → TBONormal). The "Normal" mode has --enable-two-batch-overlap + EAGLE spec + larger activation buffers vs LowLatency, and at --mem-fraction-static 0.72 on top of leftover allocations from the previous server, the allocator hits 0 MB free at first prefill.
  • Supporting: tearDownClass only calls kill_process_tree(cls.process.pid) + wait_all_ports_release — neither call guarantees ROCm device memory is reclaimed before setUpClass of the next class spawns 8 new TP workers. The OOM occurs at the very first prefill batch (token usage 0.00, no kv cache populated yet), strongly suggesting fragmented/leftover device memory.
  • Disconfirming: TBOLowLatency ran first and passed; if memory was simply insufficient, TBOLowLatency itself should have failed first. The OOM only manifests on the second server with --enable-two-batch-overlap (Normal mode), so the trigger is mode-specific extra memory pressure rather than pure leak.
  • How to verify: rerun the file with only TestMTPwithTBONormal enabled (skip TBOLowLatency) — if Normal alone passes, the issue is residual GPU memory from the first launch. Or add an explicit torch.cuda.empty_cache() + sleep between classes.
• [LOW] Recent commit ece8a1a78 (this run's HEAD; "Refactor device timer, clean up metrics collector, and add fwd occupancy metric" #24197) touches metrics collection on the forward path.
  • Supporting: lands at this run's HEAD.
  • Disconfirming: changes are in metrics/timer code, not memory allocation; OOM happens before forward pass even completes one step.
  • How to verify: revert and rerun.
• [LOW] Commit d48095ba5 ("Bypass torch.cuda.use_mem_pool generator-CM in SymmetricMemoryContext" #24190) modifies allocator-pool behaviour — could change residency of symmetric-memory buffers.
  • Supporting: in the recent delta, touches memory pool semantics, and moriep uses MORI symmetric heap.
  • Disconfirming: comment in test (line 402) sets MORI_SHMEM_MODE=ISOLATION # avoid out of symmetric heap memory — already addresses the symmetric heap issue. The OOM is on the regular HIP allocator (Available Free mem), not symmetric heap.
  • How to verify: bisect across a578bf81..ece8a1a78.

Cluster B — fused allreduce residual accuracy:

• [MEDIUM] Fused all-reduce + RMSNorm + residual kernel in aiter (v0.1.12.post2.dev150+ga6bb49937) writes the residual incorrectly on gfx950 / MI35x for ALL shapes tested. The fact that AR_exact=yes (raw allreduce matches) but the fused output diverges with a uniform max_diff=6.25e-2 (≈ 1/16, suggestive of a bf16/fp8 quantization or rounding step) and frac_nonzero≈0.27 (≈1/4, hint of element-stride or warp-level off-by-N) points to a kernel-level bug in the fused residual write path on gfx950 specifically.
  • Supporting: test docstring cites Regression test for ROCm/aiter#2586. The aiter image is a dev/patched build (.dev150+ga6bb49937) ahead of any tagged release, indicating local kernel work in flight; uniform max_diff across vastly different shapes implies a structural (per-element) bug, not a tolerance issue.
  • Disconfirming: cannot establish whether this test ever passed on MI35x in the queryable history (the matrix only just started running it on this shard). Possible the test has been failing since it was added; cannot rule out "Never-passed on MI35x" rather than a fresh regression.
  • How to verify: (a) run git log --all --oneline -- test/registered/ops/test_aiter_allreduce_fusion_amd.py on a full clone to find the introducing PR and check whether MI35x was excluded; (b) re-run with AITER_COMMIT_OVERRIDE pinned to a known-good aiter SHA; (c) inspect ROCm/aiter#2586 for status.
• [LOW] sglang call site recently changed the inputs given to the fused kernel.
  • Supporting: large delta (~93 commits in the last 7 days touching MoE/quant/communicator paths).
  • Disconfirming: the test launches its own torchrun and exercises the aiter fused op directly — it does not go through sglang's model-side communicator. So sglang-side call-site changes are unlikely to affect this test.
  • How to verify: read the test's _run_residual_accuracy_check body to confirm it calls aiter.fused_allreduce_rmsnorm_* directly without sglang wrapping.

No hypothesis above MEDIUM in either cluster; both need bisection / direct kernel inspection.

In-flight Fix Check

• In-flight fix (Cluster A — moriep TBONormal OOM): ❌ none found in open PRs. Searched: moriep, HSA_STATUS_ERROR_OUT_OF_RESOURCES. The only open AMD-moriep PRs (#22985, #23760, #11150) are feature work, not OOM/cleanup fixes. PR #24205 merged today is partial — fixes TBOLowLatency only.
• In-flight fix (Cluster B — aiter fused allreduce residual): ❌ none found in open sglang PRs matching aiter+allreduce+fusion, test_aiter_allreduce_fusion_amd, residual+accuracy. Closest open PRs are #17262 (the fusion feature itself) and #23146 (Qwen3.5 EAGLE) — neither is a kernel-correctness fix. Likely needs a fix in ROCm/aiter (issue #2586) rather than sglang.

Suggested Triage Steps

Cluster A (TestMTPwithTBONormal OOM):

• Re-run the failing class in isolation: python3 -m unittest test.registered.amd.test_moriep_small.TestMTPwithTBONormal.test_gsm8k on the same MI35x runner. If it passes alone, the root cause is leftover GPU memory from TestMTPwithTBOLowLatency — the fix direction would be a stronger teardown (explicit torch.cuda.empty_cache() + a wait loop on rocm-smi --showmeminfo).
• If it still OOMs in isolation, lower --mem-fraction-static from 0.72 and rerun, to confirm it is purely a memory-budget issue vs an actual leak.
• Bisect a578bf81..ece8a1a78 (~10 commits) on a single AMD MI35x for TestMTPwithTBONormal.test_gsm8k.

Cluster B (fused allreduce residual):

• Inspect ROCm/aiter#2586 for status; the test is the regression coverage for that exact bug.
• Re-run with AITER_COMMIT_OVERRIDE=<previous_known_good_sha> to A/B the dev-build kernel against the prior aiter build.
• If aiter HEAD is the suspect, file an issue on ROCm/aiter with the divergence table (uniform max_diff=6.25e-2, frac_nonzero≈0.27 across all shapes, AR_exact=yes) — the symptom is highly diagnostic.
• If verification confirms the test has been failing since first execution on MI35x, consider quarantining (e.g. @unittest.skipIf(gfx950)) until the aiter-side fix lands, to unblock the rest of the shard.

Status

• Persistence (Cluster A): 1 run observed (this one). Cannot establish historical persistence on this shard because earlier runs of stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) did not reach TestMTPwithTBONormal — they were monopolized by a test_gpt_oss_1gpu.py 1200s timeout. Treat as Never-passed in queryable window until a deeper history check.
• Persistence (Cluster B): same — 1 run observed. Test docstring suggests it was added specifically as a regression guard for Fix: Numerical Accuracy in allreduce_fusion_kernel_1stage ROCm/aiter#2586, so it is plausible the test has never produced a green run on MI35x with the current dev aiter build.
• Scope: 1 job (stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1)), MI35x / gfx950 only. NVIDIA / MI300 / MI325 shards unaffected (different runners, different aiter binaries).
• Blocked work: blocks pr-test-amd.yml MI35x signal for moriep/MoE-A2A and for the aiter fused-residual regression coverage.
• In-flight fix: ❌ none found for either cluster. PR #24205 (merged) only fixed TBOLowLatency portion of moriep_small.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-05-02 20:30 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 2/8)

stage-b-test-large-8-gpu-35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — failed step(s): Run test_disaggregation

Commit Info

• sglang: 4197c559681afe4978bb77a012428378f17f4263
• aiter: a6bb499375849eec45d68c5ccaebc8865fd422c0 (source: Dockerfile default — AITER_COMMIT_OVERRIDE empty, image had v0.1.12.post1 and was rebuilt to the Dockerfile-pinned SHA per [CI-AITER-CHECK] markers)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/disaggregation/test_disaggregation_pp.py	TestDisaggregationDecodePPAccuracy.test_gsm8k	Exception: retry() exceed maximum number of retries. (root: Sync batch data transfer timeout after 30000001648ns → KVTransferError: Failed to get kvcache from prefill instance, it might be dead → scheduler watchdog timeout 300s → aiohttp.ClientConnectorError: Cannot connect to host 127.0.0.1:11200)	Log L18554

The sister file test_disaggregation_basic.py PASSED in the same job (Test Summary: 1/2 passed).

Failure Summary

The PD-disaggregation PP decode scheduler stalled mid-run during the test_gsm8k evaluation: at 12:43:26 (193/200 prompts ≈ 96 % done, ~20 s into the eval) the Mooncake KV transfer engine logged Sync batch data transfer timeout after 30000001648ns (30 s) in transfer_engine_py.cpp:414, which surfaced as KVTransferError: Failed to get kvcache from prefill instance, it might be dead on the decode side. The decode PP scheduler then hung in _pp_commit_comm_work waiting on a Gloo SendWork; the 300 s watchdog (scheduler_pp_mixin.py:859, _pp_commit_comm_work) tripped soft at 12:49:41 and hard at 12:50:15, killing the prefill server, after which the test client got Connect call failed ('127.0.0.1', 11200) and retry() exhausted its single retry.

Regression Status

Flaky test (1-shot transient) — recent history of test_disaggregation_pp.py::TestDisaggregationDecodePPAccuracy::test_gsm8k in job stage-b-test-large-8-gpu-35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) on main, event=schedule, completed runs only:

Date (UTC)	Run	Job	Test File Status	Failed Function	Error
2026-05-01 18:27	25227171827	job	✅ Passed	—	—
2026-05-01 12:25	25214178926	job	❌ Failed	test_gsm8k	KV transfer 30 s timeout → 300 s watchdog
2026-05-01 06:59	25205835327	job	✅ Passed	—	—
2026-05-01 00:48	25196776069	job	✅ Passed	—	—
2026-04-30 18:34	25182702322	job	✅ Passed	—	—
2026-04-30 12:37	25165743184	job	✅ Passed	—	—

First (and so far only) observed failure: 2026-05-01 12:25 UTC. The very next scheduled run 6 h later passed on a downstream commit (ece8a1a7). Lookback window: 6 most-recent completed scheduled runs of pr-test-amd.yml covering 2026-04-30 12:37 → 2026-05-01 18:27 UTC (≈ 30 h, 5 of 6 green for this test file, 1 red).

Failure Cluster

PP-disaggregation Mooncake KV transfer timeout → decode scheduler hang → prefill server killed

Facts (verified observations, no interpretation)

• 193/200 GSM8K prompts had already completed successfully before the stall (Log L17574). The first 96 % of the eval was healthy on the same server; failure occurred deep into the run, not at startup.
• First fault: I0501 12:43:26.042579 56289 transfer_engine_py.cpp:414] Sync batch data transfer timeout after 30000001648ns — exactly 30.000001648 s, i.e. the Mooncake engine's hard-coded 30 s sync timeout fired (Log L17574).
• Cascade on PP1 ranks: KVTransferError(bootstrap_room=993941937353998277): Failed due to an unknown reason from another rank (PP1 TP0 + TP1 prefill side, Log L17576-17577) and on the decode side Failed to get kvcache from prefill instance, it might be dead (Log L17585).
• After the Mooncake timeout, the decode PP scheduler made no further progress; py-spy dump at 12:49:41 (≈ 6 min later) shows main thread of scheduler_PP0_TP0 and scheduler_PP0_TP1 stuck at _pp_commit_comm_work (scheduler_pp_mixin.py:859) → c10d::ProcessGroupGloo::SendWork::wait → pthread_cond_clockwait (Log L17703-17709). Mooncake heartbeat thread (mooncake/conn.py:1504 heartbeat_checker) also visible.
• Soft watchdog tripped at 12:49:41 (watchdog_timeout=300, soft=True), hard at 12:50:15 (soft=False) on PP0 TP0+TP1 (Log L17688/17891/17988).
• After scheduler death the load-balancer router got Server disconnected then Cannot connect to host 127.0.0.1:11200 [Errno 111] from the prefill side (Log L18497). Test treated this as accuracy-eval failure (KeyError: 'answer'), retry() ran one retry which also failed because the server was already dead, then exit code 1.
• Job-level retry was suppressed: First attempt failed with exit code 255 — Exit code 255 indicates test failure (not network issue), not retrying (Log L18583-18584).
• Sibling test test_disaggregation_basic.py in the same job PASSED (Log L18576-18577) — bringup, RDMA/Mooncake init, basic transfers all work; the failure is specifically in the longer-running PP decode flow.
• AITER version is unchanged across the passing and failing scheduled runs (Dockerfile default a6bb499375849eec45d68c5ccaebc8865fd422c0) — AITER_COMMIT_OVERRIDE empty in all 6 runs analyzed.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Transient Mooncake / RDMA fabric stall on linux-mi35x-gpu-8.fabric runner — a single batch of KV-transfer descriptors took > 30 s, the Mooncake engine declared the peer dead, and PP comms then deadlocked because one side advanced past the failed micro-batch while the other did not, producing the Gloo SendWork hang.
  • Supporting: 96 % of the eval ran cleanly on the same server with the same code; the failing line is Mooncake's own 30 s timeout (transfer_engine_py.cpp:414); same test file passed on adjacent commits 6 h before and 6 h after on the same workflow/runner pool; AITER unchanged across pass/fail.
  • Disconfirming: only one run is affected — could equally be a runner-environment issue (RDMA congestion, a noisy neighbour on the fabric, or a stuck NIC) rather than a real product bug. Unable to A/B disconfirm without re-running the same SHA on the same runner.
  • How to verify: re-run 25214178926 on commit 4197c559 (single job re-run) — if it passes, runner / fabric flake is the most likely explanation.
• [LOW] Latent product bug: when Mooncake declares one rank's transfer failed mid-batch, the PP-disagg decode scheduler does not abort the in-flight Gloo PP send/recv pair, so all PP ranks deadlock on _pp_commit_comm_work until the 300 s watchdog fires. The 30 s Mooncake timeout + 300 s scheduler watchdog gap is a real failure-mode amplifier even if the trigger is environmental.
  • Supporting: py-spy clearly shows main thread blocked in _pp_commit_comm_work → Gloo SendWork::wait, not in Mooncake; Mooncake error path appears to have surfaced cleanly but PP comm work was already submitted and never cancelled.
  • Disconfirming: this would still need a triggering Mooncake timeout — does not by itself explain why a transfer timed out today and not yesterday. And the test file has only one observed failure, so cannot establish a recurring pattern.
  • How to verify: review python/sglang/srt/managers/scheduler_pp_mixin.py:402 (event_loop_pp_disagg_decode) and :859 (_pp_commit_comm_work) to see whether KVTransferError on one micro-batch aborts the corresponding Gloo PP send/recv pair on all ranks; if not, that is a known-deadlock cleanup gap.

No hypothesis above MEDIUM confidence; the failure is most consistent with a one-off Mooncake/RDMA transfer timeout on this MI35x runner, possibly amplified by a PP-disagg cleanup gap.

In-flight Fix Check

• Searched: repo:sgl-project/sglang is:pr is:open disaggregation transfer timeout — no matching open PR addresses the Mooncake Sync batch data transfer timeout / PP-disagg watchdog cascade.
• In-flight fix: ❌ none found in open PRs — failure is currently a one-shot flake; no PR needed unless it recurs.

Suggested Triage Steps

• Re-run the failed job (25214178926 / 73930864524) on the same SHA. If it passes, label as runner-flake and watch for recurrence — do NOT open a PR yet.
• Check linux-mi35x-gpu-8.fabric runner-pool health around 12:43 UTC 2026-05-01 (RDMA NIC counters, dmesg, fabric congestion) — the Mooncake 30 s sync timeout firing on an otherwise-healthy server typically means a fabric-side stall.
• If this cluster recurs in the next 24-48 h, audit python/sglang/srt/managers/scheduler_pp_mixin.py (event_loop_pp_disagg_decode ~ line 402 and _pp_commit_comm_work ~ line 859) for missing Gloo work cancellation when a KVTransferError is raised mid PP micro-batch — the gap between Mooncake's 30 s timeout and the scheduler's 300 s soft watchdog suggests cleanup is not happening.
• Consider lowering watchdog_timeout for PP-disagg decode (currently 300 s) or wiring KVTransferError to a fast-fail path so jobs surface failures in seconds instead of minutes.

Status

• Persistence: 1/6 completed scheduled runs of pr-test-amd.yml for this test file in the last ≈ 30 h (1 fail + 5 pass). Adjacent runs (6 h before, 6 h after) both pass on main.
• Scope: single job (stage-b-test-large-8-gpu-35x-disaggregation-amd (linux-mi35x-gpu-8.fabric)) in a single workflow (pr-test-amd.yml); only test_disaggregation_pp.py::test_gsm8k failed — the sister test_disaggregation_basic.py in the same job passed.
• Blocked work: none directly — does not block PR signal as of next scheduled run; if the pattern recurs it would degrade PD-disaggregation PP CI signal on MI35x.
• In-flight fix: ❌ none found — flake-grade incident, no PR required yet.

stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 0) — failed step(s): Run test

Commit Info

• sglang: 87dad74b3
• aiter: v0.1.12.post1 (source: Dockerfile default; [CI-AITER-CHECK] AITER version matches)

Failed Tests

Test File	Test Function	Error	Log
test/registered/perf/test_bench_serving_1gpu_part1.py	test_online_lora_latency_with_concurrent_adapter_updates	AssertionError: False is not true : Failed to load LoRA adapter: {"success":false,"error_message":"Server disconnected without sending a response.","loaded_adapters":{"nvidia/llama-3.1-nemoguard-8b-topic-control":...}}	Log

Failure Summary

The single failing test starts a server with --enable-lora --max-loras-per-batch 1 --max-lora-rank 256 pinned to nvidia/llama-3.1-nemoguard-8b-topic-control, then a background lora_loader_unloader_task (test/registered/perf/test_bench_serving_1gpu_part1.py:181) cycles through three other adapters (pbevan11/llama-3.1-8b-ocr-correction, faridlazuarda/valadapt-llama-3.1-8B-it-chinese, philschmid/code-llama-3-1-8b-text-to-sql-lora) calling /load_lora_adapter and /unload_lora_adapter. One /load_lora_adapter call returned HTTP 400 with body success:false, error_message:"Server disconnected without sending a response.", tripping the assertion on test_bench_serving_1gpu_part1.py:209. Both the initial attempt (07:31:46 UTC) and the single retry (07:33:33 UTC) hit the same failure within the same job.

Regression Status

Flaky test (single occurrence in 5 of 5 surrounding runs sampled).

Recent history of test_bench_serving_1gpu_part1.py (test_online_lora_latency_with_concurrent_adapter_updates) in job stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 0) (branch=main, event=schedule, completed runs only):

Date (UTC)	Run	Job	Test File Status	Failed Function	Error
May 01 18:54	25227171827	shard 0	✅ Passed (1/1)	—	—
May 01 12:52	25214178926	shard 0	✅ Passed (1/1)	—	—
May 01 06:59	25205835327	shard 0	❌ Failed	test_online_lora_latency_with_concurrent_adapter_updates	Server disconnected ...
May 01 01:13	25196776069	shard 0	✅ Passed (1/1)	—	—
Apr 30 19:01	25182702322	shard 0	✅ Passed (1/1)	—	—
Apr 30 13:11	25165743184	shard 0	✅ Passed (1/1, with internal retry on latency assertion line 169)	—	—

First observed failure for this test on shard 0: 2026-05-01 06:59 UTC (this run). Last passing: 2026-05-01 01:13 UTC. Next passing: 2026-05-01 12:52 UTC. The failure did NOT reproduce in either neighbouring run — strongly intermittent.

Failure Cluster

LoRA adapter load returns HTTP 400 "Server disconnected without sending a response" under concurrent load/unload background task

Facts (verified observations)

• HTTP response captured by the test (test_bench_serving_1gpu_part1.py:209 assertion message): {"rid":null,"http_worker_ipc":null,"success":false,"error_message":"Server disconnected without sending a response.","loaded_adapters":{"nvidia/llama-3.1-nemoguard-8b-topic-control":"nvidia/llama-3.1-nemoguard-8b-topic-control"}} — the response wraps the error so the load was rolled back; only the originally pinned nemoguard adapter remains loaded.
• Server log shows the corresponding HTTP-level outcome: INFO: 127.0.0.1:50822 - "POST /load_lora_adapter HTTP/1.1" 400 Bad Request at 07:33:26 UTC (job log line 17279), one second after LoRA adapter loading completes: LoRARef(... lora_name=faridlazuarda/valadapt-llama-3.1-8B-it-chinese ...) at 07:33:26 UTC (line 17278). Earlier first-attempt failure at 07:31:01 UTC also showed POST /load_lora_adapter HTTP/1.1 400 Bad Request (line 14281).
• Bench-serving traffic completed normally (Successful requests: 400 of 400; Concurrency: 89.51; line 17566) — the 400 came from the /load_lora_adapter administrative path while concurrent generation was healthy.
• No Killed, core dumped, hipError, memory access fault, or SIGSEGV markers anywhere in the 17 638-line log; the server process did NOT crash, only the internal load_lora_adapter coroutine returned an error.
• Server config: --enable-lora --max-loras-per-batch 1 --disable-radix-cache --mem-fraction-static 0.8 --max-lora-rank 256 on Llama-3.1-8B-Instruct.
• retry() shows WARNING: retry() failed once (0th try, maximum 1 retries). — only one in-test retry was performed; both attempts hit the same failure.
• Aiter version v0.1.12.post1 matches Dockerfile default; no override was used.
• ResourceWarning subprocess 9676 is still running at end of the failed retry — leaked sub-server process detected during teardown but not the cause of the assertion (consequence of the test's failure path leaving the spawned sglang serve running).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] Race in the LoRA load/unload code path between an in-flight admin request and a tokenizer/scheduler IPC pipe under high concurrent generation load (#running-req was 19–85 with request_rate=8 when the failing load fired).
  • Supporting: error string "Server disconnected without sending a response." is the canonical message emitted when a downstream IPC peer drops the response stream; the LoRA load path sends a request to the scheduler over a Python IPC channel and awaits a reply. The log shows the scheduler logged LoRA adapter loading completes for valadapt-llama-3.1-8B-it-chinese at 07:33:26.047 (line 17278), yet the HTTP layer returned 400 at 07:33:26.047 (line 17279) — the success message and the failure response coincide within ~1 ms, suggesting the response message arrived after the tokenizer-side coroutine already gave up.
  • Disconfirming: the same test passed in 5/5 immediately surrounding scheduled runs on the same shard, same job, same head SHAs only hours apart. No code in the regression window touches the LoRA admin / IPC path in a way that explains a sudden new bug — and even within this single failed run the surrounding bench-serving produced 400 successful generations and no scheduler crash. This is more consistent with a pre-existing intermittent race than a fresh regression.
  • How to verify: re-run shard 0 on same SHA; if pass-on-rerun, confirms flake. Optionally run test_online_lora_latency_with_concurrent_adapter_updates in a tight loop locally with --enable-lora --max-loras-per-batch 1 --lora-paths nvidia/llama-3.1-nemoguard-8b-topic-control --max-lora-rank 256 to gauge reproducibility rate.
• [SPECULATION] LoRA admin handler timeout when prefill/decode is saturated — a /load_lora_adapter awaiting scheduler quiescence may exceed an internal timeout.
  • Supporting: the failing load happened mid-burst (#running-req: 84-85, Prefill batch, #new-seq: 64, #new-token: 7450 at line 17291).
  • Disconfirming: many other load/unload pairs in the same run completed with 200 OK even at similar #running-req levels (e.g. lines 16345, 16678, 16873, 17094); only one out of dozens of admin calls hit this error.
  • How to verify: search server-side for any TimeoutError around the LoRA admin coroutine (not visible in this log) and add a debug log line on the IPC reply path.

No hypothesis above LOW confidence; evidence is consistent with a low-probability race, not a code regression. The recommended path is rerun + accumulate occurrences before deeper investigation.

In-flight Fix Check

• In-flight fix: ❌ none found — searched open PRs with test_online_lora_latency_with_concurrent and lora adapter disconnect (/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+...) — both returned total_count: 0.

Suggested Triage Steps

• Re-run only shard 0 of run 25205835327 on the same SHA to confirm flakiness; if it passes, file as a tracked flake rather than blocker.
• If failure recurs within the next ~3 scheduled runs of pr-test-amd.yml, capture server-side stderr around the LoRA load (TokenizerManager log lines surrounding the /load_lora_adapter 400 response) — currently the log only shows the FastAPI access line, not the inner traceback that produced the rolled-back response.
• Consider adding scheduler-side logging on the LoRA admin IPC reply path (lora_manager) to make this race observable when it next happens.
• Do NOT attribute to commits in 1742bfb610..87dad74b3 without a second occurrence — the flake-pattern across 6 sampled runs makes a regression hypothesis weak.

Status

• Persistence: 1 occurrence (this run only); test passed in the 5 of 5 surrounding scheduled runs sampled (4 future + 4 historical, sampled 5 most recent).
• Scope: 1 test, 1 job, 1 workflow (pr-test-amd.yml shard 0 on linux-mi325-1gpu-sglang).
• Blocked work: none — surrounding scheduled runs passing on shard 0; only this run's shard 0 failed.
• In-flight fix: ❌ none found.

stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2) — failed step(s): Run test

Commit Info

• sglang: 4197c559
• aiter: a6bb4993 (source: Dockerfile default; image carried v0.1.12.post1 → rebuilt to a6bb4993 per [CI-AITER-CHECK] markers)

Failed Tests

Test File	Test Function	Error	Log
test/registered/ops/test_aiter_allreduce_fusion_amd.py	test_fused_ar_rms_residual_accuracy	AssertionError: Residual accuracy check failed. (max_diff=6.250000e-02, frac_nonzero≈0.28, AR_exact=yes — 49 of 49 (m,n) shape rows fail across hidden_dim ∈ {2880,4096,5120,6144,7168,8192} and both 1-stage / 2-stage paths)	Log

The other two tests in the same file (test_fused_ar_rms_benchmark, test_fused_ar_rms_multi_hidden_dim) report ..F ⇒ both PASS in this run.

Failure Summary

test_fused_ar_rms_residual_accuracy invokes itself via torchrun (TP=4) and verifies that the fused tensor_model_parallel_fused_allreduce_rmsnorm produces a bit-exact residual match (max_diff == 0) versus the unfused path on bf16 inputs. The fused kernel returns results that differ from the explicit reference by exactly one bf16 ULP on ~28% of elements for every shape tested. The allreduce result itself is bit-exact (AR_exact=yes); divergence is in the residual-add step.

Regression Status

Never-passed for the failing test function in stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2) (same branch=main, event=schedule, completed runs only). The whole file has failed every run since it was first added on 2026-04-29 by bd448e51b; the AITER bump on 2026-05-01 (7e17a7a1 → #24218, v0.1.12.post1 → a6bb4993) shifted the failure from test_fused_ar_rms_benchmark (line 253) to test_fused_ar_rms_residual_accuracy (line 317), but the file has never been green on this shard.

Recent history of test_aiter_allreduce_fusion_amd.py in stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2) (same branch, event=schedule, completed only):

Date (UTC)	Run	Job	Test File Status	Failed Function	AITER
2026-05-01 18:27	25227171827	shard 2	❌ Failed	test_fused_ar_rms_residual_accuracy	a6bb4993
2026-05-01 12:25	25214178926	shard 2	❌ Failed	test_fused_ar_rms_residual_accuracy	a6bb4993
2026-05-01 06:59	25205835327	shard 2	❌ Failed	test_fused_ar_rms_benchmark	v0.1.12.post1
2026-05-01 00:48	25196776069	shard 2	❌ Failed	test_fused_ar_rms_benchmark	v0.1.12.post1
2026-04-30 12:37	25165743184	shard 2	❌ Failed	test_fused_ar_rms_benchmark	v0.1.12.post1
2026-04-29 03:32	25085263856	shard 2	❌ Failed	test_fused_ar_rms_benchmark	v0.1.12.post1

No green run found within the queryable lookback window (back to 2026-04-27). Lookback window searched: ≥120 hours of event=schedule branch=main runs.

Failure Cluster

bf16 residual 1-ULP divergence in aiter fused_allreduce_rmsnorm

Facts (verified observations, no interpretation)

• Verbatim error (Rank 0 stdout): FAILED: fused residual output diverges from unfused path for some shapes. followed by AssertionError: Residual accuracy check failed. (log).
• Per-shape pattern (verbatim from log lines 28119-28122): 1x2880 (1-stage): max_diff=6.250000e-02 frac_nonzero=0.2774 AR_exact=yes [FAIL] … all 49 shape rows show max_diff=6.250000e-02, frac_nonzero≈0.27-0.28, AR_exact=yes. 0.0625 = 2^-4 = 1 bf16 ULP for values in [1,2); frac_nonzero≈0.28 is consistent with random bf16 rounding boundaries.
• Failure spans both transport paths: 1-stage (m·n·2 ≤ 128 KiB) AND 2-stage (> 128 KiB) at every hidden_dim 2880-8192. So the divergence is NOT specific to the 1-stage kernel that aiter#2586 was supposed to fix.
• AITER version transition: [CI-AITER-CHECK] Dockerfile expects AITER_COMMIT=a6bb499375849eec45d68c5ccaebc8865fd422c0 and [CI-AITER-CHECK] Version mismatch: image=v0.1.12.post1, repo=a6bb499... ⇒ image was rebuilt to a6bb4993. Source: log lines 2275-2510.
• Aiter delta v0.1.12.post1..a6bb4993 = 150 commits, 0 behind. Allreduce/RMSNorm-relevant commits in that delta:
  • c71075ce #2890 "Fix 2-stage fused_allreduce_rmsnorm memory ordering"
  • 4331ad7a #2823 "Add fused AR + RMSNorm + per-group FP8 quant: optional bf16 side-output"
  • d098ae5a #2735 "Revert fix(car): graph capture err"
  • fab6ef31 #2745 "fix(car): sglang prefill launch error kernel"
  • d72c1f98 #2853 "make rmsnorm quant fusion support gemma"
  • d0f5f76a #2703 "fea(car): support custom group device"
• Aiter PR #2586 (the regression the test was authored against — round-allreduce-to-bf16-before-residual) was merged 2026-04-02 (43b7379b) and IS included in a6bb4993 (210 commits ahead of #2586 merge).
• Stack trace top frame: test_aiter_allreduce_fusion_amd.py:317 → self.fail("Residual accuracy check failed.\n…"). The actual divergence is computed at _run_residual_accuracy_check lines 94-110 (fused_random vs fused_ar + residual). Source verified by reading the test file in this checkout.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] Aiter delta v0.1.12.post1 → a6bb4993 (introduced via sglang 7e17a7a1 #24218) regressed the residual rounding behavior the test was authored to protect.

  • Supporting: the failure mode shifts exactly at the AITER bump (06:59 run = old aiter, line-253 benchmark failure; 12:25 run = new aiter, line-317 residual failure). 1 bf16 ULP divergence (0.0625) is the canonical "round before vs after add" symptom that aiter#2586 was supposed to fix.
  • Disconfirming: test_fused_ar_rms_residual_accuracy was never observed to pass in any prior run — under old aiter only test_fused_ar_rms_benchmark ran (the file reports "Ran 1 test"), so we have no green baseline for the residual test. The divergence affects BOTH 1-stage AND 2-stage paths equally; aiter#2586's fix was 1-stage-specific, so a clean regression of #2586 would only affect 1-stage. The current pattern looks like a shared code-path issue (residual add semantics) rather than a kernel-specific revert.
  • How to verify: revert 7e17a7a1 on a branch (force AITER_COMMIT_DEFAULT=v0.1.12.post1), re-run only test_fused_ar_rms_residual_accuracy with unittest's -k filter (so the line-253 benchmark assertion doesn't pre-empt). If it passes, blame aiter delta. If it still fails, blame the test design.

• [LOW] Aiter c71075ce #2890 "Fix 2-stage fused_allreduce_rmsnorm memory ordering" altered the kernel under test.

  • Supporting: directly modifies the 2-stage fused_allreduce_rmsnorm path; lands in the AITER delta window.
  • Disconfirming: the failure also appears for 1-stage shapes, which #2890 should not touch. So #2890 alone cannot explain the symptom.
  • How to verify: bisect within the aiter v0.1.12.post1..a6bb4993 delta against this single test.

• [LOW] Aiter 4331ad7a #2823 "Add fused AR + RMSNorm + per-group FP8 quant: optional bf16 side-output" may have altered residual rounding when the new bf16 side-output path is wired up.

  • Supporting: explicitly touches the bf16 side-output code path that the test reads.
  • Disconfirming: the test is on a non-quant code path; the new optional bf16 side-output should be off when no FP8 quant is requested. No code-level confirmation available without reading the diff.
  • How to verify: read the diff of #2823 and inspect whether the default bf16-side-output toggle changed.

• [LOW] Test design issue — max_diff == 0.0 may be too strict for the new aiter implementation, which could legitimately compute the residual add in a different (but still numerically valid) order than the unfused reference.

  • Supporting: 0.0625 == 1 bf16 ULP is the smallest possible non-zero difference; uniformly affects 28% of elements which is what you would expect from rounding-tie-direction changes.
  • Disconfirming: the test docstring explicitly cites a real downstream impact ("-2.6pp GSM8K regression") tied to this exact rounding order, and the assertion was designed by someone who understood the kernel semantics — relaxing it would silently re-open the original bug.
  • How to verify: check whether GSM8K accuracy on AMD has regressed since the 2026-05-01 AITER bump (compare with last green nightly accuracy run on a DSV3-class model).

No hypothesis above LOW confidence; needs aiter-side bisection within v0.1.12.post1..a6bb4993 (150 commits) against test_fused_ar_rms_residual_accuracy alone.

In-flight Fix Check

Search performed (is:pr is:open in sgl-project/sglang, keywords: aiter+allreduce, test_aiter_allreduce_fusion, residual+allreduce+rmsnorm, a6bb499 OR aiter revert):

• In-flight fix: ❌ none found in open PRs — no PR matches the failing test name, the AITER bump revert, or the residual-rounding symptom. The closest open PR (#23581 "Default SGLANG_USE_AITER_AR to false to avoid HIP graph capture invalidation") is unrelated to numerical accuracy; #17262 is the original feature PR (pre-existing).

Suggested Triage Steps

• Re-run test_fused_ar_rms_residual_accuracy in isolation against the previous AITER (v0.1.12.post1) by temporarily reverting 7e17a7a1 on a branch and invoking the test directly with python3 -m unittest test.registered.ops.test_aiter_allreduce_fusion_amd.TestAiterAllreduceFusionAmd.test_fused_ar_rms_residual_accuracy — this distinguishes "aiter delta caused it" from "test never passed".
• If aiter delta is implicated, bisect within the 150-commit v0.1.12.post1..a6bb4993 aiter range using python3 /sglang-checkout/test/registered/ops/test_aiter_allreduce_fusion_amd.py --residual-accuracy (the already-existing torchrun entry point) on a mi325-8gpu runner. Suspicious commits to check first: c71075ce (#2890), 4331ad7a (#2823), d72c1f98 (#2853).
• Independently, check whether GSM8K / MMLU nightly accuracy on AMD-DSV3 has regressed since the 2026-05-01 AITER bump — confirms whether this is a real numerical regression (must-fix in aiter) or a too-strict test (relax assertion).
• If the test is intentionally a Never-passed regression guard (waiting on an aiter fix that hasn't landed), file an aiter issue referencing this run and consider an @unittest.expectedFailure annotation to unblock the shard's signal until the aiter-side fix lands.
• Open an aiter-side issue at ROCm/aiter with the per-shape table from this log (49 rows of max_diff=0.0625, AR_exact=yes) to help the aiter team triage whether the residual-add ordering changed inside the kernel.

Status

• Persistence: test_aiter_allreduce_fusion_amd.py has failed in every completed pr-test-amd schedule run of stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2) since the file was added on 2026-04-29 by bd448e51b (≈3 days, 6+ consecutive runs verified). The specific failing function shifted from test_fused_ar_rms_benchmark (line 253) to test_fused_ar_rms_residual_accuracy (line 317) at the 2026-05-01 12:25 UTC run, coincident with the AITER bump in 7e17a7a1 #24218.
• Scope: 1 job, 1 test file. Confirmed in pr-test-amd shard 2 only; sister shards 0 and 1 of the same job pass. Not yet checked against pr-test-amd-rocm720, nightly-test-amd, or amd-aiter-scout for cross-workflow spread.
• Blocked work: stage-c-test-large-8-gpu-amd (mi325, 2) AMD signal — this single shard is permanently red, masking any further regressions added to the same shard.
• In-flight fix: ❌ none found in open PRs — needs new PR or investigation.

stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) — failed step(s): Run test

Commit Info

• sglang: 87dad74b3
• aiter: v0.1.12.post1 (source: Dockerfile default — [CI-AITER-CHECK] reports AITER version matches, AITER_COMMIT_OVERRIDE empty)

Failed Tests

Test File	Test Function	Error	Log
test/registered/core/test_gpt_oss_1gpu.py	test_mxfp4_20b	AssertionError: False is not true (raised by self.assertTrue(len(content) > 0) in gpt_oss_common.py:121)	Log line 6953

test_bf16_20b (the sister test in the same file, model lmsys/gpt-oss-20b-bf16) passed in the same run (Streaming check response: 1 + 1 = 2. at log line 2686). Only the mxfp4 variant of openai/gpt-oss-20b failed.

Failure Summary

The /v1/responses streaming-API smoke check inside _check_streaming_responses_api_request returned HTTP 200 but the captured text content was empty — the test concatenates response.output_text.delta events from the SSE stream and then asserts len(content) > 0. The assertion failed on both retry attempts, so the test errored out (FAILED (errors=1) after Ran 2 tests in 1177.449s).

Regression Status

Never-passed (no green run found in the queryable window). The bot scanned every pr-test-amd.yml event=schedule branch=main run from 2026-05-01 back through 2026-04-14 and the same test fails (or its job times out before reaching it) in every run where test_gpt_oss_1gpu.py actually ran.

Recent history of test/registered/core/test_gpt_oss_1gpu.py (test_mxfp4_20b) in job stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) (same branch, event=schedule, completed runs only):

Date (UTC)	Run	Job	Test File Status	Failed Function	Error
May 01 18:27	25227171827	job 73973512786	❌ Timeout (test_mxfp4_20b started, test never completed within 1200 s)	test_mxfp4_20b	timeout
May 01 12:25	25214178926	job 73930864499	⚠️ N/A (job aborted at runner setup: VRAM cleanup unsuccessful after 3 attempts; test did not run)	—	—
May 01 06:59	25205835327	job 73906083570	❌ Failed (current run)	test_mxfp4_20b	AssertionError: False is not true
May 01 00:48	25196776069	job 73879096560	❌ Failed	test_mxfp4_20b	exit 1
Apr 30 18:34	25182702322	job 73831818646	❌ Timeout / Failed	test_mxfp4_20b	timeout after AssertionError retry
Apr 30 12:37	25165743184	job 73771336090	⚠️ N/A (VRAM cleanup unsuccessful; test did not run)	—	—
Apr 30 06:57	25151900211	job 73724231747	❌ Timeout	test_mxfp4_20b	timeout after AssertionError retry

Sampled deeper history (same test file, same job): Apr 29 (25085263856, 25070972860), Apr 28 (25053383235, 25027540200), Apr 27 (25012616648, 24995433920, 24981041153, 24971260460), Apr 26 (24963679028, 24956447494), Apr 21 (24739407779), Apr 20 (24683238072, 24643155780), Apr 18 (24604476234), Apr 17 (24551739451), Apr 16 (24527146590), Apr 15 (24430282646), Apr 14 (24398811675) — all failed on test_mxfp4_20b (mix of AssertionError: False is not true and 1200 s timeout). First observed failure: ≥ 2026-04-14 (lookback boundary). Last known passing date: NONE in the lookback window.

Failure Cluster

gpt-oss-20b mxfp4 /v1/responses streaming returns empty output_text on AMD MI35x

Facts (verified observations, no interpretation)

• Verbatim error from log line 6953:

  File "/sglang-checkout/python/sglang/test/gpt_oss_common.py", line 121, in _check_streaming_responses_api_request
      self.assertTrue(len(content) > 0)
  AssertionError: False is not true
  

• Log line 3831 and 4730: POST /v1/responses HTTP/1.1 200 OK — the server returned HTTP 200 on both attempts, so it is not a server-error path; the SSE stream simply contained zero response.output_text.delta events with non-empty delta.
• Log line 6932 and 6936 print Streaming check response: (empty) on both retry attempts — confirms content == "".
• test_bf16_20b (same test file, model lmsys/gpt-oss-20b-bf16, identical streaming check) passed in this same job run at log line 2686 (Streaming check response: 1 + 1 = 2.). bf16 ≠ mxfp4 is the only meaningful difference between the passing and failing cases in this run.
• Server ServerArgs at log line 4406 for the failing run shows: model_path='openai/gpt-oss-20b', attention_backend='aiter', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', reasoning_parser=None, server log mentions Detected ROCm and MXFP4 quantization format for GPT-OSS model, enabling aiter MXFP4 MOE kernel. and Enable Aiter AllReduce Fusion for GptOssForCausalLM.
• Decode loop in the failing run (log lines 4250–4375) shows the model IS generating tokens at ~210 tok/s — the model is not hung; tokens are produced but they don't surface as output_text.delta SSE events.
• Aiter version: [CI-AITER-CHECK] block (log lines 2310–2316) reports Dockerfile expects AITER_COMMIT=v0.1.12.post1, AITER version inside CI image: v0.1.12.post1, AITER version matches. AITER_COMMIT_OVERRIDE: empty — this is NOT an AITER scout run, sglang is exercising the Dockerfile-default aiter.
• Reproducibility: same test failed in every queryable scheduled run since 2026-04-14 (≥18 days back) where the test got past runner setup. No green run found in the lookback window.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] The mxfp4 inference path for openai/gpt-oss-20b on AMD MI35x produces token IDs that the /v1/responses streaming pipeline classifies entirely as non-text-output content (e.g. all reasoning / harmony control tokens) so no response.output_text.delta events with non-empty delta ever fire, even though decoding is healthy.
  • Supporting: HTTP 200 + tokens being decoded at >200 tok/s + Streaming check response: empty → the SSE channel is alive but the text-delta event is never emitted. Same code path passes for lmsys/gpt-oss-20b-bf16 in the same run, so the responses-API serializer itself is fine; the difference is the model's token sequence under mxfp4 quant. Aiter MXFP4 MoE kernel and Aiter AllReduce Fusion are both enabled for GptOss only on this code path.
  • Disconfirming: bot did NOT verify by capturing the raw SSE stream contents (it only sees the test-side print), so it is possible the responses-API server-side serializer regressed for mxfp4 specifically rather than the model output. Cannot distinguish "model outputs garbage tokens" from "responses-API drops mxfp4 deltas" from log alone.
  • How to verify: in container rocm/sgl-dev:v0.5.10.post1-rocm700-mi35x-20260430, launch the failing server (sglang serve --model-path openai/gpt-oss-20b --cuda-graph-max-bs 600 --device cuda --host 127.0.0.1 --port 11000) and curl -N -d '{"model":"openai/gpt-oss-20b","input":"What is 1 + 1?","stream":true,"temperature":0}' http://127.0.0.1:11000/v1/responses to inspect the raw SSE event types. If only reasoning.* events appear, the model output is the issue; if output_text.delta events appear with non-empty delta, the test parser disagrees with the server.
• [LOW] Aiter MXFP4 MoE kernel (v0.1.12.post1) on gfx950 (MI35x) produces incorrect outputs that lead the model to never emit an end-of-reasoning / start-of-output token, so the model only ever produces "thinking" tokens and never visible text.
  • Supporting: aiter MXFP4 MoE kernel is explicitly enabled on this path (server log: enabling aiter MXFP4 MOE kernel); bf16 path doesn't use it and passes.
  • Disconfirming: aiter version is unchanged across the entire 18-day failing window (always v0.1.12.post1), and the test was never passing in the queryable window — so even if aiter is implicated, it cannot explain a regression that has no observed pre-regression baseline. This is steady-state "never-passed", not an aiter delta.
  • How to verify: bisect by overriding the aiter commit to an earlier known-good aiter (if one exists) and re-running; or disable the aiter MXFP4 MOE kernel and the aiter AllReduce fusion and re-run.
• [LOW] Test expectation is incompatible with how openai/gpt-oss-20b formats its responses-API stream (the model may always emit a reasoning prefix, and _check_streaming_responses_api_request only counts output_text.delta deltas, ignoring reasoning.* events).
  • Supporting: the prompt is "What is 1 + 1?"; the bf16 variant emits 1 + 1 = 2. directly, but mxfp4 may take longer to reach the answer-text phase given quant-induced perplexity. The test cuts the streaming check off after a single response.
  • Disconfirming: the test reads the stream via for line in response.iter_lines() until [DONE] — there is no early cutoff, so a successful-but-slow generation should still capture deltas. Also, decode log shows ~88 k tokens generated before the request returned, so the model definitely produced a long sequence.
  • How to verify: same as the first hypothesis — capture the raw SSE stream and check whether output_text.delta events ever fire for mxfp4 within the request.

No hypothesis above MEDIUM confidence. Needs raw-SSE capture + AMD MI35x reproduction to disambiguate model-output vs server-serializer vs aiter-kernel root cause.

In-flight Fix Check

Searched open sglang PRs for keywords matching this failure:

• test_mxfp4_20b → 0 open PRs (API call)
• test_gpt_oss_1gpu → 0 open PRs (API call)
• gpt-oss + mxfp4 → 11 open PRs but none target the streaming-empty-content issue (API call) — closest matches are #23218 [WIP] feat: add triton_quant_info for mxfp4 gpt-oss moe layer and #17190 [bugfix] Add gfx942 (MI300X) support for MXFP4 quantization, but neither is gfx950/MI35x or the responses-API.
• gpt-oss + amd → 7 open PRs, none match this symptom.
• In-flight fix: ❌ none found in open PRs — needs new PR or investigation.

Suggested Triage Steps

• Reproduce locally on an MI35x runner using the container rocm/sgl-dev:v0.5.10.post1-rocm700-mi35x-20260430 and the exact server invocation from the log: sglang serve --model-path openai/gpt-oss-20b --cuda-graph-max-bs 600 --device cuda --host 127.0.0.1 --port 11000. Hit /v1/responses with the test payload and dump the raw SSE bytes to disambiguate the three hypotheses above (model output vs serializer vs aiter kernel).
• If the raw stream shows no output_text.delta events: investigate whether openai/gpt-oss-20b (mxfp4) needs --reasoning-parser (it currently runs with reasoning_parser=None); the test passes for lmsys/gpt-oss-20b-bf16 which may incidentally produce text-channel output without a reasoning parser.
• If the raw stream DOES have output_text.delta events: regression is in the test/parser _check_streaming_responses_api_request — but note this is identical to the bf16 code path, so the test is unlikely to be the bug.
• A/B test by toggling enable_aiter_allreduce_fusion=False and --moe-runner-backend triton (avoiding the aiter MXFP4 MoE kernel) to isolate whether the empty content originates in the aiter quant kernel vs the model logic.
• If reproduction on MI35x is not possible for this engineer: consider quarantining test_mxfp4_20b on AMD MI35x (skip on gfx950) until a maintainer can investigate, since the test has been red continuously for ~18 days and may be blocking CI signal noise rather than catching new regressions.

Status (factual, no priority assignment)

• Persistence: Failing in every completed scheduled run of pr-test-amd.yml since at least 2026-04-14 (~18 days, lookback boundary; no green run found). Same AssertionError: len(content) > 0 symptom and/or 1200 s timeout in every failing instance.
• Scope: 1 test file × 1 test function × 1 job (stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1)) on pr-test-amd.yml. The sister test test_bf16_20b in the same file passes consistently. AMD MI35x / gfx950 only — no evidence this affects MI300X or NVIDIA stage-b jobs (would need a separate audit to confirm).
• Blocked work: keeps stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) red on every scheduled pr-test-amd.yml run, masking any regression in the other tests scheduled to run after test_gpt_oss_1gpu.py in this job.
• In-flight fix: ❌ none found — needs new PR or investigation.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-05-02 20:30 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 3/8)

stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2) — failed step(s): Run test

Commit Info

• sglang: 87dad74b3
• aiter: v0.1.12.post1 (source: Dockerfile default — [CI-AITER-CHECK] AITER version matches)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_moriep_small.py	(timeout — last running TestMTPwithTBOLowLatency/gsm8k benchmark)	TIMEOUT after 3600s; preceded by HSA_STATUS_ERROR_OUT_OF_RESOURCES (Code: 0x1008) on rocdevice queue 0x7ee0a7200000 → Fatal Python error: Aborted	Log
test/registered/ops/test_aiter_allreduce_fusion_amd.py	test_fused_ar_rms_benchmark	AssertionError: False is not true : Expected fused fallback for oversized eager shape(s) under default gate. (line 253)	Log

Failure Summary

Two test files failed in shard (linux-mi325-8gpu-sglang, 2) of stage-c-test-large-8-gpu-amd. (1) test_moriep_small.py ran 3600s and was killed by the harness; the underlying log shows an HSA_STATUS_ERROR_OUT_OF_RESOURCES on a rocdevice queue (Available Free mem : 17592186044276 MB, the standard "uninitialized / corrupted memory accounting" sentinel) followed by a Fatal Python error: Aborted, after which the parent process never reaped subworkers and the watchdog hung. (2) test_fused_ar_rms_benchmark asserts that for eager shapes >64 MiB/rank at least one row reports fused_available=False (i.e. the gate falls back). No row matched, so the assertion failed at test_aiter_allreduce_fusion_amd.py:253.

Regression Status

Both failures are Never-passed in queryable history for this job/shard.

Recent history of test/registered/ops/test_aiter_allreduce_fusion_amd.py (test_fused_ar_rms_benchmark) in stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2) (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
2026-05-01 18:27	25227171827	job	❌ Failed	test_fused_ar_rms_residual_accuracy	bit-exact residual mismatch (different sub-test)
2026-05-01 12:25	25214178926	job	❌ Failed	test_fused_ar_rms_residual_accuracy	bit-exact residual mismatch
2026-05-01 06:59	25205835327	job	❌ Failed	test_fused_ar_rms_benchmark	fused-fallback assertion (line 253)
2026-05-01 00:48	25196776069	job	❌ Failed	test_fused_ar_rms_benchmark	fused-fallback assertion
2026-04-30 12:37	25165743184	job	❌ Failed	test_fused_ar_rms_benchmark	fused-fallback assertion
2026-04-29 18:34	25126941077	job	❌ Failed	test_fused_ar_rms_benchmark	fused-fallback assertion
2026-04-26 12:19	24956447494	job	❌ Failed	test_fused_ar_rms_benchmark	fused-fallback assertion

The test file was added 2026-04-26 by c7878dbb6 (#23707 "[MoE] Deprecate act_and_mul_triton …") and has failed in every completed scheduled run since (≥ 6 days, lookback window 2026-04-22 → 2026-05-01, all runs were failure). No green run found.

Recent history of test/registered/amd/test_moriep_small.py in the same job (same filters):

Date	Run	Test File Status	Failure Mode
2026-05-01 18:27	25227171827	❌ Failed (errors=1)	test_gsm8k line 430 error
2026-05-01 12:25	25214178926	❌ TIMEOUT 3600s	(post-fix #24205)
2026-05-01 06:59	25205835327	❌ TIMEOUT 3600s	HSA_STATUS_ERROR_OUT_OF_RESOURCES + watchdog hang (post-fix #24205)
2026-05-01 00:48	25196776069	❌ Failed	setUpClass line 465 error (pre-fix)
2026-04-30 12:37	25165743184	❌ Failed	setUpClass line 465 error (pre-fix)
2026-04-29 18:34	25126941077	❌ Failed	exit 1
2026-04-26 12:19	24956447494	❌ Failed	exit 1

Both are persistent, never-passed-in-window failures. Today's run is the first scheduled run after fix #24205 (a578bf81) landed, so the symptom changed (setUpClass error → timeout/HSA OOR) but the test still does not complete.

Failure Cluster

• test_moriep_small.py → "AMD MoriEP MTP+TBO test 3600s timeout (HSA out-of-resources during gsm8k benchmark)"
• test_fused_ar_rms_benchmark → "AITER fused-allreduce-rmsnorm fallback gate not engaging on oversized eager shapes"

Facts (verified observations, no interpretation)

• Runner: linux-mi325-8gpu-sglang (MI325, gfx942), GPU_ARCH=mi30x, aiter v0.1.12.post1 matched Dockerfile ([CI-AITER-CHECK] markers at log line 2281–2287).
• Other 2 tests in this shard PASSED: test_deepseek_v3_basic.py (283s) and test_deepseek_v3_mtp.py — only the moriep + aiter-allreduce-fusion tests failed.
• moriep crash log line: :0:rocdevice.cpp :3671: 3679979337902 us: Callback: Queue 0x7ee0a7200000 Aborting with error : HSA_STATUS_ERROR_OUT_OF_RESOURCES … Available Free mem : 17592186044276 MB (the bogus 17.5 PB free-mem number is the standard "amd_smi failed to read" sentinel — actual cause is a dispatch-queue-resource exhaustion, not host OOM).
• moriep stack at crash: scheduler_dp_attn_mixin.py:60 _get_fallback_tensor → :93 all_gather → :202 prepare_mlp_sync_batch_raw while gsm8k benchmark was issuing Prefill batch, #new-seq:6, #new-token:4096 across 8 DP ranks.
• moriep timeline: HSA error fires at 08:04:35 (≈ 53 min into the 60 min budget, mid-test_gsm8k), the parent never exits, watchdog hangs, harness kills the test at 08:11:31 (3600s budget).
• aiter-allreduce-fusion assertion: line 253 expects at least one row in large_eager_rows (rows with bytes_per_rank > 64*1024*1024 = 64 MiB) where fused_available=="False". Test ran in 30s (no GPU crash); benchmark CSV was produced, the gate just never reported a fallback for any oversized eager row.
• test_fused_ar_rms_benchmark requires ≥8 GPUs and runs nproc=8, so this assertion is exclusive to the 8-GPU shard 2.
• The test_aiter_allreduce_fusion_amd.py file was introduced in the same PR (#23707, c7878dbb6, 2026-04-26) and has never produced a green scheduled run in the queryable window (2026-04-22 onward).
• moriep fix a578bf814 (#24205, merged 2026-05-01) added env["SGLANG_ENABLE_SPEC_V2"] = "false" to both TestMTPwithTBONormal and TestMTPwithTBOLowLatency. The test still fails after the fix.

Hypothesised Causes (with confidence, with disconfirming evidence)

For test_fused_ar_rms_benchmark:

• [MEDIUM] The fused-fallback gate in the aiter-allreduce-rmsnorm wrapper does not currently disable fusion for eager shapes >64 MiB/rank on gfx942 + aiter v0.1.12.post1, so the test's hard expectation at line 253–256 is not met by the production code as shipped.

  • Supporting: every scheduled run since the test landed (≥6 days, every shard-2 of stage-c on MI325) hits the same assertion at the same line; aiter version is unchanged across the entire window; the test was added together with #23707 which deprecates act_and_mul_triton (MoE / activation refactor, no allreduce-fusion gating change). The author of #23707 appears to have shipped the test before the matching gate logic existed in production.
  • Disconfirming: I did not git-blame the actual fused-allreduce gate code path to confirm no 64 MiB threshold exists; the assertion may be intentional and waiting for a separate kernel/feature PR.
  • How to verify: read python/sglang/srt/layers/aiter_allreduce_fusion*.py (or wherever fused_available is computed) and search for a bytes_per_rank > 64*1024*1024 style cap; run the test locally on MI325 with --correctness mode to capture the produced CSV and inspect fused_available per row.

• [LOW] aiter v0.1.12.post1 simply has a higher fused-mode upper limit than the test expects (the test's threshold is wrong, not the code).

  • Supporting: the test is brand new and uncalibrated; aiter version unchanged.
  • Disconfirming: same author shipped both — a bad threshold should have been caught in PR CI.
  • How to verify: ask the #23707 author what fused_available=False was supposed to indicate.

For test_moriep_small.py timeout:

• [LOW] The MoriEP MTP+TBO setup leaks dispatch-queue resources during the gsm8k workload, and once HSA exhausts them the parent never returns control, so the harness times out.

  • Supporting: HSA OOR on a rocdevice queue (not host OOM) ⇒ AMD runtime ran out of dispatch-queue slots; the watchdog stack shows _get_fallback_tensor/prepare_mlp_sync_batch_raw in mid-flight when it died; symptom only appears with SGLANG_USE_AITER=1 + MORI_SHMEM_MODE=ISOLATION.
  • Disconfirming: the #24205 fix author believed SGLANG_ENABLE_SPEC_V2=false would resolve it — they have more context than this analysis. The fix may have masked the original error and exposed a different latent issue.
  • How to verify: temporarily extend the per-test budget past 3600s, or unset SGLANG_USE_AITER, and inspect whether the gsm8k benchmark completes; collect a py-spy dump --native against the wedged scheduler ranks (the in-log py-spy already failed with "Failed to get stack traces", suggesting the worker is in an unrecoverable state).

• [SPECULATION] Aiter v0.1.12.post1 interacts poorly with MoriEP shmem-isolation mode at this DP/EP=8 scale.

  • Supporting: only 8-GPU MTP+TBO shard hits this; smaller-scale tests pass.
  • Disconfirming: aiter version is unchanged for the entire window, so this cannot be a fresh regression.
  • How to verify: not actionable until other hypotheses are ruled out.

No hypothesis above MEDIUM confidence for the moriep timeout — author of [#24205](https://github.com/sgl-project/sglang/pull/24205) (billishyahao) is the right person to confirm.

In-flight Fix Check

• test_fused_ar_rms_benchmark — In-flight fix: ❌ none found (searched is:pr is:open test_fused_ar_rms and is:pr is:open test_aiter_allreduce_fusion_amd — zero results; the test has been failing every run for 6+ days with no fix PR open).
• test_moriep_small.py — In-flight fix: ⚠️ partial — [#24205](https://github.com/sgl-project/sglang/pull/24205) merged 2026-05-01 (commit [a578bf81](https://github.com/sgl-project/sglang/commit/a578bf814cd4b8decd1ee0f1190217bf1b76ef72)) was intended to fix this but the next 3 scheduled runs (including this one) still time out / fail with new symptoms — needs a follow-up. No new open PR found via is:pr is:open moriep (the 6 open moriep PRs are unrelated features: eplb, mori all2all, MoeRunner unification, etc.).

Suggested Triage Steps

• For test_fused_ar_rms_benchmark: Read the fused-allreduce-rmsnorm gating code (grep -rn fused_available python/sglang/srt) and confirm whether the 64 MiB/rank fallback exists. If the gate is not implemented yet, either (a) ping the author of #23707 to ship the matching gate or (b) skip this assertion on AMD until the gate lands. If the gate is implemented but never triggers in this aiter version, ping for a threshold tune.
• For test_moriep_small.py: Reach out to the author of #24205 (billishyahao@amd.com) — the spec-v2-disable fix moved the failure but did not eliminate it. Capture a fresh py-spy --native dump of the wedged DP5 scheduler PID 1988 (in-log attempts failed; may need a manual repro). Investigate whether the HSA queue exhaustion is reproducible by re-running just TestMTPwithTBOLowLatency in isolation. Consider raising the per-test budget temporarily to see whether the benchmark eventually completes or stays wedged.
• General: Both failures are AMD-MI325-shard-2 specific and have no NVIDIA equivalent; they are not blocking PR signal on the main sglang code paths but they are gating AMD CI.

Status

• Persistence: Both tests have been failing for ≥6 days (test_fused_ar_rms_benchmark since the test was added 2026-04-26; test_moriep_small.py failing in every run in the queryable window). Today (2026-05-01) is the first run after the moriep fix attempt landed; the fix did not resolve the failure.
• Scope: Limited to stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2) — the only 8-GPU MI325 shard that runs test_fused_ar_rms_benchmark (which requires gpu_count >= 8). The moriep test also runs only here.
• Blocked work: Stage-C 8-GPU AMD CI signal cannot go green; both failures are in test/registered/amd/ and test/registered/ops/, not in the user-facing model serving paths.
• In-flight fix: test_fused_ar_rms_benchmark: ❌ none found (needs new PR or test-skip). test_moriep_small.py: ⚠️ #24205 merged but failure persists with new symptom — needs follow-up from the original fix author.

stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) — failed step(s): Run test

Commit Info

• sglang: 1742bfb6 — [CI] release-whl-kernel: skip musa wheels in update_kernel_whl_index (#24191)
• aiter: v0.1.12.post1 (Dockerfile default; no [CI-AITER-CHECK] markers emitted in this job's stage prior to timeout — value taken from sister jobs in the same workflow)

Failed Tests

Test File	Test Function	Error	Log
test/registered/core/test_gpt_oss_1gpu.py	TestGptOss1Gpu.test_mxfp4_20b	✗ TIMEOUT after 1200 seconds (estimated_time=750s)	Log

test_bf16_20b passed in the same run (~413 s wall-clock). test_mxfp4_20b started at 01:03:37 UTC and was still in the decode loop at 01:16:41 UTC when the per-file 1200 s budget was hit.

Failure Summary

The test_mxfp4_20b case in test/registered/core/test_gpt_oss_1gpu.py exceeded the 1200 s per-file timeout while serving openai/gpt-oss-20b with mxfp4 quantization. The server was producing only ~89 tok/s and the decode loop had accumulated ≥100 151 generated tokens for a single in-flight request before being killed.

Regression Status

Never-passed (no green run found in queryable window).

Recent history of test/registered/core/test_gpt_oss_1gpu.py (test_mxfp4_20b) in job stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) (branch main, event=schedule, completed runs only):

Date (UTC)	Run	Job	Test File Status	Failed Function	Error
2026-05-01 18:27	25227171827	stage-b-test-1-gpu-small-amd-mi35x	❌ Failed	test_mxfp4_20b	TIMEOUT after 1200s
2026-05-01 12:25	25214178926	stage-b-test-1-gpu-small-amd-mi35x	❌ Failed (env)	n/a	VRAM cleanup unsuccessful (infra)
2026-05-01 06:59	25205835327	stage-b-test-1-gpu-small-amd-mi35x	❌ Failed	test_mxfp4_20b	exit code 1 (1182s)
2026-05-01 00:48	25196776069	stage-b-test-1-gpu-small-amd-mi35x	❌ Failed	test_mxfp4_20b	TIMEOUT after 1200s (this run)
2026-04-30 18:34	25182702322	stage-b-test-1-gpu-small-amd-mi35x	❌ Failed	test_mxfp4_20b	TIMEOUT after 1200s
2026-04-30 12:37	25165743184	stage-b-test-1-gpu-small-amd-mi35x	❌ Failed (env)	n/a	VRAM cleanup unsuccessful (infra)
2026-04-30 06:57	25151900211	stage-b-test-1-gpu-small-amd-mi35x	❌ Failed	test_mxfp4_20b	TIMEOUT after 1200s
2026-04-30 00:45	25141482864	stage-b-test-1-gpu-small-amd-mi35x	❌ Failed	test_mxfp4_20b	TIMEOUT after 1200s
2026-04-29 18:34	25126941077	stage-b-test-1-gpu-small-amd-mi35x	❌ Failed	test_bf16_20b (also test_mxfp4_20b)	exit code 1 (709s)
2026-04-29 12:38	25109261379	stage-b-test-1-gpu-small-amd-mi35x	❌ Failed	test_mxfp4_20b	exit code 1 (963s)
2026-04-29 00:46	25085263856	stage-b-test-1-gpu-small-amd-mi35x	❌ Failed	test_mxfp4_20b	exit code 1 (808s)
2026-04-28 18:36	25070972860	stage-b-test-1-gpu-small-amd-mi35x	❌ Failed	test_mxfp4_20b	exit code 1 (809s)
2026-04-28 12:41	25053383235	stage-b-test-1-gpu-small-amd-mi35x	❌ Failed	test_mxfp4_20b	TIMEOUT after 1200s
2026-04-28 06:59	25038683025	stage-b-test-1-gpu-small-amd-mi35x	❌ Failed	test_mxfp4_20b	TIMEOUT after 1200s
2026-04-28 00:44	25027540200	stage-b-test-1-gpu-small-amd-mi35x	❌ Failed	test_mxfp4_20b	TIMEOUT after 1200s
2026-04-27 18:32 → 2026-04-26 00:41	(8 more runs)	various	❌ Failed	test_mxfp4_20b	TIMEOUT or exit code 1 (AssertionError: False is not true from _check_streaming_responses_api_request)

Lookback window searched: 2026-04-26 00:41 UTC → 2026-05-01 18:27 UTC (≥30 consecutive event=schedule runs on main). Zero passing runs of this test file were observed in the queryable window — this is a Never-passed condition for test_mxfp4_20b on the linux-mi35x-gpu-1 shard. (The Job column is for human reference; the regression verdict is based on Test File Status.)

Failure Cluster

GPT-OSS 20B mxfp4 wallclock-budget timeout on AMD MI35x (single-GPU, 1200 s test-runner cap)

Facts (verified observations, no interpretation)

• Log line filename='/sglang-checkout/test/registered/core/test_gpt_oss_1gpu.py', elapsed=1200, estimated_time=750.0 at 01:16:41 UTC, followed by ✗ TIMEOUT: ... after 1200 seconds at 01:16:46 UTC (Log).
• test_bf16_20b finished cleanly at 01:03:37 UTC after reporting all three GPQA reasoning-effort scores (low: 0.399, medium: 0.389, high: 0.328 — all above the configured expected_scores of 0.34/0.34/0.27).
• test_mxfp4_20b started at 01:03:37 UTC (Log) and only ~787 s of the 1200 s budget remained for it.
• Server log emitted: [2026-05-01 00:58:54] Disable piecewise CUDA graph because --disable-piecewise-cuda-graph is set and [2026-05-01 01:03:45] mxfp4 quantization is not fully optimized yet. The speed can be slower than non-quantized models. — the latter is printed by sglang's GPT-OSS model code itself.
• Decode loop telemetry at the timeout boundary: Decode batch, #running-req: 1, #full token: 100151, full token usage: 0.02, #swa token: 330, ..., cuda graph: True, gen throughput (token/s): 89.53, #queue-req: 0 — i.e. only 1 in-flight request had been generating continuously and had already produced ~100 151 tokens at the time of the kill.
• Test workload (python/sglang/test/gpt_oss_common.py:_run_one_eval): 198 GPQA questions × max_tokens=4096, run for 3 reasoning efforts (low/medium/high) in a ThreadPoolExecutor(max_workers=4). At ~89 tok/s sustained throughput this is structurally larger than a 1200 s budget.
• Two distinct symptom variants of the same cluster have been observed in the lookback window: (a) TIMEOUT after 1200 s (today + ~10 other runs) and (b) AssertionError: False is not true raised from python/sglang/test/gpt_oss_common.py:121 in _check_streaming_responses_api_request — self.assertTrue(len(content) > 0) — i.e. the streaming /v1/responses request returned an empty response.output_text.delta stream (e.g. job 73094628808).
• Test file test/registered/core/test_gpt_oss_1gpu.py has only been touched by 3 sglang commits since it was created on Jan 9 2026: 0c474273c (creation), 4e999404c (rename to mi35x suite), bd448e51b (spec accept_length rename). None of these changed the workload size or est_time=750.
• est_time=750 declared at test/registered/core/test_gpt_oss_1gpu.py:7; runner kills after 1200 s. The 750 s estimate is below all observed exit-code-1 elapsed times (709/744/808/963/1182 s) and well below the actual generation cost on this hardware.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [HIGH] AMD MI35x mxfp4 throughput on openai/gpt-oss-20b is structurally too slow for the configured 1200 s wallclock budget — i.e. the test budget / workload is mis-sized for this hardware, not a fresh code regression.

  • Supporting: server itself prints mxfp4 quantization is not fully optimized yet at startup; sustained ~89 tok/s decode rate; 100 k+ tokens generated for a single request before the kill (no termination); identical symptom across 30 consecutive scheduled runs over 5+ days on multiple sglang head SHAs and with no aiter delta (v0.1.12.post1 Dockerfile default both then and now); est_time=750 s declared in the test is already below every observed wallclock; test_bf16_20b on the same model size & runner finished in 413 s, confirming the 20 B model itself fits — only the mxfp4 path is over-budget.
  • Disconfirming: variants of the failure include an AssertionError: len(content) > 0 from _check_streaming_responses_api_request (a separate request before the GPQA evals), which suggests a second failure mode: the streaming /v1/responses call sometimes returns no text-delta events at all, which is not pure throughput. That sub-failure could indicate a real correctness/streaming-shape issue on mxfp4 + responses API; however it has produced exit-code-1 (not TIMEOUT) and is thus a distinct sub-cluster.
  • How to verify: rerun python3 test/registered/core/test_gpt_oss_1gpu.py TestGptOss1Gpu.test_mxfp4_20b on a linux-mi35x-gpu-1 runner with --cuda-graph-max-bs 600 exactly as in the test, time the full execution end-to-end, and compare the tok/s to the gpt-oss-20b mxfp4 reference numbers; or temporarily reduce num_examples from 198 → 32 in _run_one_eval to confirm whether the test passes within budget.

• [LOW] A specific recent commit broke mxfp4 throughput on MI35x.

  • Supporting: the failure is consistent and reproducible.
  • Disconfirming (strong): the test file is Never-passed in the queryable window — there is no pass_sha..fail_sha regression window to bisect. The cluster spans 30+ runs and many distinct sglang head SHAs (e.g. 14b4e6fa → 1742bfb6 → ece8a1a7) without ever passing.
  • How to verify: if a linux-mi35x-gpu-1 smoke-pass of this test ever existed pre-Apr 26, find that SHA and bisect against the first failing scheduled run; otherwise the hypothesis is unprovable from CI history.

• [LOW] _check_streaming_responses_api_request empty-delta AssertionError indicates a real bug in mxfp4 + /v1/responses streaming.

  • Supporting: variant failures on Apr 26 / 27 / 28 / 29 reach assertTrue(len(content) > 0) with content == "", meaning the streaming endpoint completed with zero response.output_text.delta events.
  • Disconfirming: the dominant failure mode in the most recent runs (including this one) is TIMEOUT, not the streaming AssertionError, so streaming is not the sole or primary blocker. Empty content could itself be a downstream artefact of the same mxfp4 slowness (the streaming check uses requests.post(..., stream=True) with no client-side timeout — slow first-token + a transport hiccup could close the stream before any delta arrives).
  • How to verify: add a requests.post(..., timeout=...) and dump the raw SSE bytes when len(content) == 0; reproduce locally with curl against the running server.

No hypothesis above HIGH confidence promotes to FACT without a controlled rerun: the throughput hypothesis is HIGH, the regression hypothesis is LOW (refuted by Never-passed history), and the streaming-bug hypothesis is LOW.

In-flight Fix Check

Searched open sglang PRs with keywords gpt_oss_1gpu, test_mxfp4_20b, gpt-oss mi35x, mxfp4 timeout:

curl -s -H "Authorization: token $GH_PAT" "https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+gpt_oss_1gpu"   # 0 results
curl -s -H "Authorization: token $GH_PAT" "https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+test_mxfp4_20b" # 0 results
curl -s -H "Authorization: token $GH_PAT" "https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+gpt-oss+mi35x"  # 4 unrelated results

• In-flight fix: ❌ none found in open PRs targeting this test or mxfp4-on-MI35x throughput — needs new PR or investigation. (#23146 "Enable EAGLE speculative decoding for Qwen3.5 FP8 and MXFP4" is unrelated; it does not touch gpt-oss or this test.)

Suggested Triage Steps

• Reproduce locally on a linux-mi35x-gpu-1 container by running python3 test/registered/core/test_gpt_oss_1gpu.py TestGptOss1Gpu.test_mxfp4_20b and measuring (a) end-to-end wall time, (b) sustained decode tok/s, (c) whether _check_streaming_responses_api_request returns non-empty content. This directly tests the HIGH hypothesis.
• If wall time exceeds 1200 s but the test would otherwise pass on accuracy, decide between:
  • widening the runner budget (raise est_time and the 1200 s per-file cap) — temporary unblocker;
  • shrinking the workload (lower num_examples in python/sglang/test/gpt_oss_common.py:_run_one_eval, or restrict expected_score_of_reasoning_effort to a single effort) — preserves CI signal at lower coverage;
  • quarantining/skipping test_mxfp4_20b on the AMD MI35x suite until mxfp4 throughput on this hardware is improved (gates other tests on this shard from getting blocked by a known-slow case).
• If the streaming sub-failure recurs after throughput is fixed, instrument _check_streaming_responses_api_request to dump the raw SSE response body when len(content) == 0 and add a client-side request timeout, then file a separate ticket targeting the /v1/responses mxfp4 path.
• Bot did not find a regression window, so a git bisect is not applicable here. Do not revert any specific commit on this signal.

Status

• Persistence: Failing on every completed scheduled run since at least 2026-04-26 00:41 UTC (5 days, ~30 consecutive runs of pr-test-amd.yml examined; Never-passed in queryable window).
• Scope: Confined to the single shard stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) of workflow pr-test-amd.yml. Same test on the CUDA suite stage-b-test-1-gpu-large is registered separately with est_time=408 and is not in scope of this AMD shard's failure. No other tests in the same job are affected by this cluster (the job's only test entry is test_gpt_oss_1gpu.py).
• Blocked work: Continuous red signal for stage-b-test-1-gpu-small-amd-mi35x shard on pr-test-amd.yml; masks any other regression that might land on this shard.
• In-flight fix: ❌ none found in open PRs.

stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2) — failed step(s): Run test

Commit Info

• sglang: 1742bfb6 ([CI] release-whl-kernel: skip musa wheels…)
• aiter: v0.1.12.post1 (source: Dockerfile default — [CI-AITER-CHECK] AITER version matches)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_moriep_small.py	TestMTPwithTBOLowLatency.setUpClass	Exception: Server process exited with code -9 (root cause inside scheduler: AttributeError: 'NoneType' object has no attribute 'shape' at two_batch_overlap.py:208)	Log
test/registered/ops/test_aiter_allreduce_fusion_amd.py	TestAiterAllreduceFusionAmd.test_fused_ar_rms_benchmark	AssertionError: False is not true : Expected fused fallback for oversized eager shape(s) under default gate.	Log

Failure Summary

Two unrelated test files failed in the stage-c-large-8gpu shard. test_moriep_small.py::TestMTPwithTBOLowLatency aborted in setUpClass because every DP scheduler crashed during cuda-graph replay-prepare with AttributeError: 'NoneType' object has no attribute 'shape' inside two_batch_overlap.split_spec_info. test_aiter_allreduce_fusion_amd.py::test_fused_ar_rms_benchmark hit a benchmark assertion that expects at least one oversized eager row to fall back to non-fused; none did.

Regression Status

Known recurring failure for both tests — failing in every completed pr-test-amd.yml schedule run sampled (back to 2026-04-29 at least). No green run found in the queryable window for either test on this job/shard.

Recent history of test/registered/amd/test_moriep_small.py (TestMTPwithTBOLowLatency.setUpClass) and test/registered/ops/test_aiter_allreduce_fusion_amd.py (test_fused_ar_rms_benchmark) in job stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2) (branch main, event=schedule, completed runs only):

Date (UTC)	Run	moriep status	allreduce-fusion status
2026-05-01 00:48	25196776069	❌ setUpClass error (mem fault, AttributeError)	❌ AssertionError "fused fallback"
2026-04-30 18:34	25182702322 (job 73831818937)	log truncated mid-run; job concluded failure	log truncated; concluded failure
2026-04-30 12:37	25165743184 (job 73771336332)	❌ setUpClass error	❌ AssertionError "fused fallback"
2026-04-30 06:57	25151900211 (job 73724231953)	log truncated; concluded failure	log truncated; concluded failure
2026-04-30 00:45	25141482864 (job 73692023562)	❌ setUpClass error	❌ AssertionError "fused fallback"
2026-04-29 18:34	25126941077 (job 73642472409)	❌ failed in test_gsm8k (different symptom in this run)	❌ AssertionError "fused fallback"

Lookback range searched: pr-test-amd.yml event=schedule branch=main, last ~30 runs (2026-04-17 → 2026-05-01). No completed run with both tests passing was found. First-observed failure date for the moriep AttributeError/setUpClass cluster is at least 2026-04-30 00:45 UTC (earlier runs fail with different symptoms on the same file). The aiter-allreduce assertion is the same exact line 253 error in every visible failing run since 2026-04-29.

Failure Cluster

1. TBO replay-prepare crash: spec_info.seq_lens_cpu is None — affects TestMTPwithTBOLowLatency setUpClass.
2. aiter allreduce fusion benchmark gate: no oversized eager fallback — affects test_fused_ar_rms_benchmark.

Facts (verified observations, no interpretation)

• All 8 DP/TP/EP ranks emit the same stack trace ending in:

  File ".../batch_overlap/two_batch_overlap.py", line 208, in split_spec_info
      if end_seq_index == spec_info.seq_lens_cpu.shape[0]:
  AttributeError: 'NoneType' object has no attribute 'shape'
  

  Verified at log line 26060+ (verbatim from the job log).
• Stack trace path: scheduler.event_loop_overlap → model_worker.forward_batch_generation → eagle_worker_v2.verify → prepare_for_v2_verify → cuda_graph_runner.replay_prepare → tbo_backend.init_forward_metadata_replay_cuda_graph → _init_forward_metadata_cuda_graph_split → two_batch_overlap.split_spec_info.
• eagle_worker_v2.py:390-404 constructs EagleVerifyInput(..., seq_lens_cpu=None) after the draft step (verified in source). two_batch_overlap.py:208 (and :185 in _compute_mask_offset) dereferences spec_info.seq_lens_cpu.shape[0] with no None check (verified in source).
• After all schedulers crash, Subprocess scheduler_0 (pid=22469) crashed with exit code -3. Triggering SIGQUIT for cleanup... is followed by SIGKILL of the server process; popen_launch_server raises Server process exited with code -9 and setUpClass errors out.
• Server command (verbatim from log) uses --enable-two-batch-overlap --speculative-algo EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --deepep-mode low_latency --moe-a2a-backend mori --enable-dp-attention on deepseek-ai/DeepSeek-V3-0324, tp=8 ep=8 dp=8.
• test_fused_ar_rms_benchmark failed at test_aiter_allreduce_fusion_amd.py:253:

  self.assertTrue(
      any(r["fused_available"] == "False" for r in large_eager_rows),
      "Expected fused fallback for oversized eager shape(s) under default gate.",
  )

  i.e. for all eager rows with bytes_per_rank > 64 MiB, fused_available came back as True. Test ran 24.9 s and only this single sub-test was executed.
• aiter version is v0.1.12.post1 (Dockerfile default, no override) — same as in prior failing runs, so aiter version is unchanged across the failing window.
• Two other tests in the same shard PASSED in this run: test_deepseek_v3_mtp.py, test_deepseek_v3_basic.py (Test Summary: 2/4 passed).

Hypothesised Causes (with confidence, with disconfirming evidence)

Cluster 1 — TBO + EAGLE-v2 verify hits seq_lens_cpu=None

• [HIGH] EagleVerifyInput produced by eagle_worker_v2.draft() has seq_lens_cpu=None (set explicitly at python/sglang/srt/speculative/eagle_worker_v2.py:403), but the TBO split path at python/sglang/srt/batch_overlap/two_batch_overlap.py:208 and :185 dereferences spec_info.seq_lens_cpu.shape[0] unconditionally.
  • Supporting: stack trace lands exactly on two_batch_overlap.py:208; the matching constructor in eagle_worker_v2.py:390-404 literally passes seq_lens_cpu=None while EagleVerifyInput.create_idle_input (eagle_info.py:108) passes torch.empty((0,), dtype=torch.int32) — confirming the v2 path forgets to populate this field. The TBO custom_mask is not None and draft_token is not None precondition (two_batch_overlap.py:206) is satisfied here (both are populated by the draft step), so the dereference is reached.
  • Disconfirming: this code shape is visible in the current main snapshot for bd448e51b ("[Spec] Split accept_length…" #23962, 2026-04-29) which is when these files first appear in the shallow clone, so I can't prove the bug was introduced by that commit vs predates it. The repo is shallow; older history was not queryable to bisect.
  • How to verify: revert/no-op-test by setting seq_lens_cpu=batch.seq_lens_cpu in eagle_worker_v2.py:403 (or guarding split_spec_info against seq_lens_cpu is None) on a branch and re-run python3 test/registered/amd/test_moriep_small.py on AMD MI325 (mori EP, EAGLE, TBO). Could also reproduce the fault locally without AMD by enabling the same flags on NVIDIA — failure is in pure Python, not aiter — to A/B against this single change.

Cluster 2 — aiter allreduce-fusion gate keeps fused_available=True for >64 MiB eager rows

• [LOW] Default gate threshold or bytes_per_rank accounting changed so that eager rows above 64 MiB still report fused_available=True, violating the test expectation of a fallback.
  • Supporting: Test asserts on bytes_per_rank > 64 * 1024 * 1024. Failure has been stable across many runs (same line 253 error since at least 2026-04-29), suggesting a deterministic gating threshold mismatch between code and test, not a flake.
  • Disconfirming: I cannot inspect the gate code (file paths for aiter_allreduce_fusion were not located in the sglang source — the gate may live in sgl-kernel or aiter itself). No log evidence isolates which side moved.
  • How to verify: read the _run_benchmark CSV emitted by the test (path printed earlier in the log), confirm bytes_per_rank and fused_available per row; then git log -S "fused_available" -- sgl-kernel/ and the aiter repo around the v0.1.12.post1 tag for changes to the eager allreduce-fusion gate threshold.

No hypothesis above LOW for cluster 2. Cluster 1 reaches HIGH on code-level evidence.

In-flight Fix Check

Searched open PRs for test_moriep_small, aiter_allreduce_fusion, two_batch_overlap, seq_lens_cpu, TestMTPwithTBOLowLatency, moriep, split_spec_info, TBO+spec.

• In-flight fix: ❌ none found in open PRs — needs new PR or investigation. (Closest matches like #21228 "Make seq_lens datatype consistent" and #20331 "fix len(rids) != num_seqs in tbo filter_batch()" are unrelated to the EAGLE-v2 seq_lens_cpu=None path.)

Suggested Triage Steps

• Cluster 1: read python/sglang/srt/speculative/eagle_worker_v2.py:390-404 and python/sglang/srt/batch_overlap/two_batch_overlap.py:180-217. Confirm EagleVerifyInput.seq_lens_cpu should always be populated when custom_mask and draft_token are populated; either populate it in draft() (set seq_lens_cpu=model_worker_batch.seq_lens_cpu, seq_lens_sum=model_worker_batch.seq_lens_sum) or guard the TBO split path. Apply the smaller change on a branch and re-run test/registered/amd/test_moriep_small.py::TestMTPwithTBOLowLatency on AMD MI325 in the AMD-bot container. If the fault still reproduces, fall back to disabling EAGLE-v2 + TBO combination for this test until a deeper fix lands.
• Cluster 1 (alt): bisect pr-test-amd.yml schedule runs older than 2026-04-29 to find the last green moriep run; if none exists, treat the test as Never-passed on the EAGLE-v2 codepath and consider gating the test off until the codepath is fixed.
• Cluster 2: pull the per-shape CSV produced by _run_benchmark, inspect fused_available for eager rows with bytes_per_rank > 64 MiB. If the gate is the regression, identify the aiter_allreduce_fusion gate code (likely in sgl-kernel or aiter); if the test is stale (threshold semantics moved), update the assertion. Cross-check against the prior aiter version that was used when this test last passed.
• For both clusters: confirm whether the pr-test-amd-rocm720.yml sister workflow shows the same failures on the same shard — if yes, both clusters are non-runner-specific and reproduce on any AMD MI3xx; if no, runner/image-specific.

Status

• Persistence: failing in every queryable completed pr-test-amd.yml schedule run on this job/shard since at least 2026-04-29 (~2 days). No green run identified in the queryable window. Cluster 2 (aiter-allreduce gate) is identical line 253 across all sampled runs.
• Scope: 2 failing test files in 1 shard (stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2)) of pr-test-amd.yml. The TBO/EAGLE-v2 stack would also be exercised by any other AMD job that runs DeepSeek-V3 with --enable-two-batch-overlap + EAGLE — needs cross-shard check to confirm scope (not done here).
• Blocked work: TBO + EAGLE-v2 + mori EP CI signal on AMD; aiter allreduce-fusion benchmark CI signal on AMD.
• In-flight fix: ❌ none found — needs new PR or investigation.

stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) — failed step(s): Run test

Commit Info

• sglang: cb8fbd53f
• aiter: a6bb499375849eec45d68c5ccaebc8865fd422c0 (source: dev image — v0.1.12.post2.dev150+ga6bb49937 detected, rebuild skipped; Dockerfile default unchanged)

Failed Tests

Test File	Test Function	Error	Log
test/registered/core/test_gpt_oss_1gpu.py	TestGptOss1Gpu.test_mxfp4_20b	AssertionError: False is not true at gpt_oss_common.py:121 (assertTrue(len(content) > 0) in _check_streaming_responses_api_request); retry exhausts → file-level TIMEOUT after 1200s	Log

test_bf16_20b ran first and passed. Only test_mxfp4_20b failed.

Failure Summary

GPT-OSS 20B mxfp4 streaming Responses API returns empty content, so assertTrue(len(content) > 0) fails in _check_streaming_responses_api_request. The retry path re-launches the server and pushes the test file past its 1200s budget, producing a TIMEOUT rather than a clean assertion failure exit.

Regression Status

Never-passed (no green run found in queryable window) — this test has been failing on stage-b-test-1-gpu-small-amd-mi35x in every completed scheduled run we've inspected.

Recent history of test/registered/core/test_gpt_oss_1gpu.py (test_mxfp4_20b) in job stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) (workflow pr-test-amd.yml, branch main, event schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
May 02	25239403651	stage-b-test-1-gpu-small-amd-mi35x	❌ Failed	test_mxfp4_20b	AssertionError: False is not true → TIMEOUT 1200s
May 01 18:27	25227171827	stage-b-test-1-gpu-small-amd-mi35x	❌ Failed	test_mxfp4_20b	TIMEOUT 1200s
May 01 12:25	25214178926	stage-b-test-1-gpu-small-amd-mi35x	❌ Failed (runner)	—	VRAM cleanup unsuccessful (test never ran)
May 01 06:59	25205835327	stage-b-test-1-gpu-small-amd-mi35x	❌ Failed	test_mxfp4_20b	exit 1

First observed failure: ≥ 2026-04-14 (per prior bot memory). No green run found in the queryable lookback. Lookback window covers ≥ 2026-04-30 → 2026-05-02 plus prior bot history.

Failure Cluster

GPT-OSS 20B mxfp4 streaming Responses API returns empty content (AMD MI35X)

Facts (verified observations, no interpretation)

• Verbatim error (log line 5397): AssertionError: False is not true raised inside _check_streaming_responses_api_request at python/sglang/test/gpt_oss_common.py:121 — self.assertTrue(len(content) > 0). Streaming Responses API returned zero content chunks for the mxfp4_20b model.
• The same call path retries once (retry() from python/sglang/srt/utils/common.py:2672); retry re-spawns the server and consumes the remaining file budget, producing the file-level TIMEOUT after 1200s (log line 7096).
• test_bf16_20b PASSED before this case (ran 00:52→00:59), confirming the failure is mxfp4-specific, not a general 20B bring-up issue.
• Server log shows mxfp4 GPT-OSS path active: Detected ROCm and MXFP4 quantization format for GPT-OSS model, enabling aiter MXFP4 MOE kernel. (log line 5407) and attention_backend='aiter' (server_args, line 5410).
• [CI-AITER-CHECK] confirms the CI image's aiter version (v0.1.12.post2.dev150+ga6bb49937) is preserved — no override; same aiter as Dockerfile expects (a6bb499).
• Failure reproduces in the immediately prior completed scheduled run 25227171827 (May 01 18:27, sglang ece8a1a78) and in 25205835327 (May 01 06:59, sglang 87dad74b3). Three different sglang HEADs, same failure → not tied to today's HEAD.
• PR #23829 ([AMD] Update test_gpt_oss_1gpu.py — wrap mxfp4_20b call with SGLANG_USE_AITER=1) was closed unmerged on 2026-04-28 (verified via API: state=closed merged=False). No replacement PR is open today (search for test_mxfp4_20b OR test_gpt_oss_1gpu open PRs returns 0 hits; broader gpt_oss + mxfp4 + amd returns 0 relevant hits).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] AMD-specific gating issue in the GPT-OSS mxfp4 streaming path — likely the same shape that the closed PR #23829 intended to fix (env-var gate SGLANG_USE_AITER=1 around the mxfp4_20b call when is_hip()).
  • Supporting: failure is MXFP4-specific (BF16 sibling test passes on the same job); the modified server_args confirm aiter MXFP4 MOE kernel is the active path; PR #23829 explicitly targeted this exact test on AMD.
  • Disconfirming: PR #23829 was closed unmerged with no public reason captured; we cannot confirm the env-var workaround actually produces non-empty streaming content on mi35x without a re-run. The PR may have been closed because the workaround was insufficient, not because the bug was fixed.
  • How to verify: cherry-pick the diff from #23829 on a branch and re-run test_gpt_oss_1gpu.py::test_mxfp4_20b on linux-mi35x-gpu-1; capture the streaming response payload to confirm content is non-empty.
• [LOW] Server-side mxfp4 generation produces zero output tokens (model crash silenced into empty stream).
  • Supporting: assertTrue(len(content) > 0) is the only assertion that fires; if generation aborted server-side, streaming chunks would be empty.
  • Disconfirming: no server crash / GPU fault visible in the log around 01:06–01:07; decode batches are running with gen throughput ≈ 124 token/s immediately before the assertion. So generation IS happening — it's the Responses API streaming path (or the test's content extraction) returning empty.
  • How to verify: instrument _check_streaming_responses_api_request to print the raw streaming events when len(content) == 0; distinguishes "server returned nothing" vs. "test failed to extract content".
• [LOW] Aiter MXFP4 MOE kernel regression on gfx950.
  • Supporting: Enable Aiter AllReduce Fusion for GptOssForCausalLM and aiter MXFP4 MOE kernel log lines indicate aiter is on the hot path.
  • Disconfirming: aiter version is unchanged across passing-of-bf16 vs failing-of-mxfp4 within the same job; the bf16 sibling test using the same aiter passes; failure is reproducible across multiple sglang HEADs but aiter has been stable on a6bb499 per [CI-AITER-CHECK]. No aiter-side regression window correlates.
  • How to verify: not actionable until the AMD-gating hypothesis is ruled out.

In-flight Fix Check

• In-flight fix: ❌ none found in open PRs. Searched repo:sgl-project/sglang is:pr is:open test_mxfp4_20b OR test_gpt_oss_1gpu → 0 hits. The previously intended fix #23829 is closed unmerged (verified state=closed merged=False closed_at=2026-04-28T15:29:31Z). A new PR is needed.

Suggested Triage Steps

• Reach out to @yctseng0211 (author of #23829) to ask why the PR was closed and whether the SGLANG_USE_AITER=1 workaround is still believed to be the right shape — needs a fresh PR if so.
• Locally cherry-pick the #23829 diff onto current main and re-run test/registered/core/test_gpt_oss_1gpu.py::test_mxfp4_20b on a linux-mi35x-gpu-1 runner; if it passes, open a replacement PR and link to #23829.
• If #23829 alone does not pass: capture the raw streaming events from the Responses API in _check_streaming_responses_api_request to determine whether the issue is server-side empty generation or a parser/content-extraction bug specific to the streaming Responses API path on AMD.
• Do NOT bisect sglang main — the failure spans multiple HEADs (cb8fbd53f, ece8a1a78, 87dad74b3) and is Never-passed; bisection has no green endpoint to anchor on.

Status

• Persistence: ≥ 18 days persistent (since at least 2026-04-14 per prior bot memory; verified failing in 3 of the last 3 completed scheduled runs where the test actually executed; the 4th-most-recent run aborted earlier with a runner-side VRAM cleanup unsuccessful so the test did not run).
• Scope: 1 job in pr-test-amd.yml (stage-b-test-1-gpu-small-amd-mi35x). Per prior bot memory, the same test also fails on stage-b-test-1-gpu-small-amd-mi35x-rocm720 in pr-test-amd-rocm720.yml — out of scope for this single-job report.
• Blocked work: blocks GPT-OSS 20B mxfp4 CI signal on AMD MI35X.
• In-flight fix: ❌ none found — PR #23829 closed unmerged 2026-04-28; a new PR is required.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-05-02 20:30 UTC)

[amd-bot]

pr-test-amd.yml — Per-Job Analysis (continued 4/8)

stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 0) — failed step(s): N/A

Commit Info

• sglang: cb8fbd53f
• aiter: a6bb49937 (source: dev image preserved — v0.1.12.post2.dev150+ga6bb49937)

Failed Tests

Test File	Test Function	Error	Log
test/registered/amd/test_deepseek_v32_basic.py	server-launch hang during weight load (test never reached assertions)	TP1 holds HF download lock for ~17 min, then 8× TP schedulers stall in MoE weight copy; only TP5 reaches Load weight end (1367.20 s) before the job's wall-clock budget expires. Repeated Scheduler watchdog timeout (self.watchdog_timeout=300, self.soft=True) every 5 min from 01:01:05 to 01:21:19 UTC; log truncates mid-pyspy dump.	Log

Failure Summary

The test_deepseek_v32_basic.py server (DeepSeek-V3.2 fp8, TP=8, attention_backend=nsa, NSA tilelang, enable_multithread_load: true) hung during model load. TP1 (PID 1637) acquired the HF Hub filelock at 00:56:44 and stayed inside snapshot_download → hf_hub_download → httpx until 01:13:48 (~17 min for download). All TPs then bottlenecked in _load_w13/_load_w2 MoE copies stuck on pthread_mutex_lock inside libamdhip64.so during c10::hip::memcpy_and_sync. Only TP5 reached Load weight end at 01:19:30 (elapsed=1367.20 s); the job was killed by the job-level timeout at 01:38:28 (54-min total) — partition 0 of stage-c-test-large-8-gpu-amd failed with no test assertions executed.

Regression Status

New regression (first observed today 2026-05-02; previous 5 scheduled runs all green for this exact partition).

Recent history of test/registered/amd/test_deepseek_v32_basic.py in job stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 0) (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
May 2	25239403651	stage-c …(0)	❌ Failed	server-launch hang	weight-load stall, watchdog timeouts, job killed
May 1 18:27	25227171827	stage-c …(0)	✅ Passed (1909 s)	—	—
May 1 12:25	25214178926	stage-c …(0)	✅ Passed	—	—
May 1 06:59	25205835327	stage-c …(0)	✅ Passed	—	—
May 1 00:48	25196776069	stage-c …(0)	✅ Passed	—	—
Apr 30 18:34	25182702322	stage-c …(0)	✅ Passed	—	—

First observed failure: 2026-05-02 00:55 UTC. Last known passing: 2026-05-01 18:27 UTC (head ece8a1a78).

Failure Cluster

DeepSeek-V3.2 server-launch hang during MoE weight load on MI325 (HIP memcpy_and_sync mutex contention; only 1/8 TPs finishes before job timeout)

Facts (verified observations, no interpretation)

• Test selected: log line 5557 python3 /sglang-checkout/test/registered/amd/test_deepseek_v32_basic.py (est_time=3600.0).
• Server args (line 5593): model_path='deepseek-ai/DeepSeek-V3.2', tp_size=8, attention_backend='nsa', nsa_prefill_backend='tilelang', nsa_decode_backend='tilelang', model_loader_extra_config='{"enable_multithread_load": true}', mem_fraction_static=0.862.
• Download timing: TP1 ACQUIRED lock at 00:56:44 (line 5831); TP0 reports Found local HF snapshot ... skipping download at 01:13:48 (line 7514) → ~17 min download (vs ~14 min in last passing run).
• Weight-load completion: only [2026-05-02 01:19:30 TP5] Load weight end. elapsed=1367.20 s (line 15380). TP0/1/2/3/4/6/7 never reach Load weight end in the log.
• Watchdog timeouts: 8× Scheduler watchdog timeout (self.watchdog_timeout=300, self.soft=True) events from 01:01:15 through 01:21:19 (lines 5938-20301), all schedulers’ pyspy dumps show identical stack _load_w13/_load_w2 (moe/fused_moe_triton/layer.py:474|542) → THPVariable_copy_ → at::native::copy_kernel_cuda → c10::hip::memcpy_and_sync → pthread_mutex_lock → libamdhip64.so.7.0.70000.
• Pyspy of lock-holder PID 1637 / TP1 at 01:01:15 (line 6135): _inner_hf_hub_download (huggingface_hub/_snapshot_download.py:435) → httpx ssl recv — TP1 was still inside HF Hub HTTPS GET when other TPs were already trying to load weights from the partial cache.
• Job timing: started 00:44:26, completed 01:38:28 (54 min); previous 4 runs of same job were 44/56/44/47 min and all success.
• Aiter version unchanged between yesterday’s pass and today’s fail (v0.1.12.post2.dev150+ga6bb49937 dev image preserved both times — see [CI-AITER-CHECK] markers, line 2282-2288).
• Commit window (ece8a1a78..cb8fbd53f, 7 commits): cb8fbd53f (#24243 req-pool slot 0 padding), b47fab6f5 (#24241 TBO splitter for DP), 05de73efd (#24232 Llama4 attention auto-select), 8a530468f (#24244 mamba mappings size), 79bc2505a (#23037 EAGLE GLM-5.1 IMA), b58fa60a1 (#24165 FlashInfer paged-wrapper env), c7566a860 (#24238 CI permissions).

Hypothesised Causes (with confidence, with disconfirming evidence)

• [LOW] Slow HF Hub download cascaded into a wall-clock blow-up that put the rest of the launch over the job-level timeout.
  • Supporting: TP1 spent ~17 min in httpx HTTPS download of DeepSeek-V3.2 weights vs ~14 min in the last passing run; total job runtime (54 min) is the longest of the recent 5 runs (44–56 min); job log truncates exactly when the 5-min-watchdog dumps grow huge, consistent with the runner being killed.
  • Disconfirming: a 3-min HF slowdown alone shouldn't tip a 47-min job into failure; the prior 12:25 run took 56 min and still passed, so the wall-clock budget appears to allow ≥56 min.
  • How to verify: re-run the job once (or trigger via workflow_dispatch) on the same SHA and check whether load timing returns to normal; if it does, treat as a network/runner flake.
• [LOW] HIP runtime contention from enable_multithread_load=true MoE copies intermittently deadlocks on pthread_mutex_lock inside libamdhip64.so 7.0.70000.
  • Supporting: every TP's pyspy stack at every 5-min watchdog tick is stuck in the same c10::hip::memcpy_and_sync → pthread_mutex_lock → libamdhip64.so frame; only TP5 ever advances past Load weight end; the multi-threaded loader (enable_multithread_load: true) and 8 concurrent TPs share the HIP runtime.
  • Disconfirming: identical settings + identical aiter + identical AMD container ran successfully on the prior 5 scheduled runs of this job; pyspy showing a thread blocked in pthread_mutex_lock during HIP memcpy is also the normal blocking-stream behavior, not necessarily a deadlock.
  • How to verify: re-run on the same SHA; if the hang reproduces, repro locally with python3 test/registered/amd/test_deepseek_v32_basic.py on linux-mi325-8gpu-sglang and try toggling enable_multithread_load=false in the test's server args.
• [LOW] One of the 7 commits in ece8a1a78..cb8fbd53f regressed model startup.
  • Supporting: temporal correlation only — first failure on the first scheduled run that includes these commits.
  • Disconfirming: none of the 7 commits touches the DeepSeek-V3.2 / NSA / MoE weight-loader / HIP runtime call paths in the failing stack trace; 8a530468f and cb8fbd53f touch req/Mamba pools but DeepSeek-V3.2 is not Mamba-based; HF Hub time (the dominant slowdown) is untouchable from sglang.
  • How to verify: a single re-run on cb8fbd53f distinguishes flake from regression cheaply before bisecting.

No hypothesis above LOW confidence; if the failure does not reproduce on a re-run, treat as flake; otherwise bisect [ece8a1a78..cb8fbd53f].

In-flight Fix Check

Searched is:pr is:open for deepseek_v32 OR enable_multithread_load OR deepseek-v3.2 and weight loading hang amd:

• In-flight fix: ❌ none found — no open PR matches the failure signature; needs investigation (recommended: re-run first).

Suggested Triage Steps

• Re-run the workflow once on the same SHA (cb8fbd53f) to distinguish a network/runner flake (HF Hub slow + slow MoE copy) from a real regression. Cost: 1 hour of one MI325 runner.
• If it reproduces, bisect ece8a1a78..cb8fbd53f — only 7 commits — by re-running only partition 0 of stage-c-test-large-8-gpu-amd on each SHA.
• If it reproduces and bisect points away from the 7 commits, disable enable_multithread_load for this test (model_loader_extra_config='{"enable_multithread_load": false}') on a branch and re-run, to test the HIP-contention hypothesis.
• Independently, persist the local HF snapshot in the runner image for deepseek-ai/DeepSeek-V3.2 so that the 14–17 min HF Hub fetch is removed from the critical path (this also makes future failures more diagnostic — they can no longer be attributed to network jitter).
• Capture the missing tail of the log: the log download truncates at 01:21:19 while the job ran until 01:38:28. Re-fetch via the run's archive (gh run download <run_id> --log) to see the actual exit reason.

Status

• Persistence: First observed today (2026-05-02 ~01:00 UTC); 1 of 6 most-recent completed runs of this job (1/6).
• Scope: Single job: stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 0) in pr-test-amd.yml. Sister stage-c-test-large-8-gpu-amd-mi35x (…, 0) in the same run 25239403651 should be checked separately to confirm scope (not requested in this prompt).
• Blocked work: blocks DeepSeek-V3.2 fp8 / NSA basic-correctness CI signal on MI325 in pr-test-amd.yml.
• In-flight fix: ❌ none found — needs investigation (recommended: re-run first, then bisect if it reproduces).

stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2) — failed step(s): Run test

Commit Info

• sglang: cb8fbd53
• aiter: a6bb49937 (source: Dockerfile default; image carried dev build v0.1.12.post2.dev150+ga6bb49937, rebuild skipped)

Failed Tests

Test File	Test Function	Error	Log
test/registered/ops/test_aiter_allreduce_fusion_amd.py	test_fused_ar_rms_residual_accuracy	AssertionError: Residual accuracy check failed. — fused 1-stage/2-stage residual diverges from unfused path; max_diff=6.25e-02 (≈1 bf16 ULP), frac_nonzero≈0.28 across all shapes (hidden_dim ∈ {2880,4096,5120,6144,7168,8192} × bs ∈ {1,4,8,16,32,64,128})	Log
test/registered/amd/test_moriep_small.py	(suite timeout, no specific test)	✗ TIMEOUT after 3600 s (est_time 1200 s); preceded by HSA_STATUS_ERROR_OUT_OF_RESOURCES — Available Free mem: 2 MB and Fatal Python error: Aborted in scheduler run_event_loop	Log

Failure Summary

Two suites in stage-c shard (linux-mi325-8gpu-sglang, 2) failed: a numerical-accuracy regression test for the aiter fused allreduce+RMSNorm 1-stage kernel reports a 1-ULP bf16 mismatch on every shape, and test_moriep_small.py ran past its 3600 s wall-clock budget after the GPU runtime aborted with HSA out-of-resources. Both have failed in every recent completed run of this shard.

Regression Status

Never-passed (no green run found in queryable window). Both test files were introduced together on 2026-04-29 in commit d9270b8c (fix(moe): relocate orphan tuned configs after #23019 — squash of #24004). Every shard‑2 run since shows the same two failures.

Recent history of (linux-mi325-8gpu-sglang, 2) on pr-test-amd.yml (event=schedule, branch main, completed runs only):

Date (UTC)	Run	Job	Test File Status	Failed Function	Error
2026-05-02 00:43	25239403651	shard 2	❌ aiter_allreduce_fusion_amd Failed; moriep_small Timeout	test_fused_ar_rms_residual_accuracy, suite-timeout	accuracy + 3600 s timeout
2026-05-01 18:27	25227171827	shard 2	❌ both Failed	test_fused_ar_rms_residual_accuracy, moriep_small errors=1	accuracy + setUp/test errors
2026-05-01 12:25	25214178926	shard 2	❌ both Failed	test_fused_ar_rms_residual_accuracy, suite-timeout	accuracy + 3600 s timeout
2026-05-01 06:59	25205835327	shard 2	❌ both Failed	test_fused_ar_rms_residual_accuracy, suite-timeout	accuracy + 3600 s timeout
2026-05-01 00:48	25196776069	shard 2	❌ both Failed	test_fused_ar_rms_residual_accuracy, moriep_small errors=1	accuracy + setUp/test errors
2026-04-30 18:34	25182702322	shard 2	❌ both Failed	similar	similar
2026-04-30 12:37	25165743184	shard 2	❌ both Failed	test_fused_ar_rms_benchmark (line 253), moriep_small.setUpClass (line 465)	benchmark + setUp
2026-04-30 06:57	25151900211	shard 2	❌ both Failed	similar	similar
2026-04-30 00:45	25141482864	shard 2	❌ both Failed	similar	similar
2026-04-29 18:34	25126941077	shard 2	❌ both Failed	test_gsm8k (line 429), test_fused_ar_rms_benchmark	first run after files added

(Both test files were introduced 2026-04-29 in the same commit. Shards 0 and 1 of this matrix pass — only shard 2 ever runs these two files. Lookback covers 10 consecutive completed scheduled runs from 2026-04-29 18:34 UTC onward; no green run for either test file in this window.)

Failure Cluster

• Cluster A — aiter fused allreduce 1-stage/2-stage residual: 1-ULP bf16 mismatch (regression test never green)
• Cluster B — test_moriep_small.py wall-clock timeout (3600 s) — HSA out-of-resources mid-run

Facts (verified observations, no interpretation)

• [CI-AITER-CHECK] reports: Runner GPU_ARCH=mi30x; Dockerfile expects AITER_COMMIT=a6bb499375849eec45d68c5ccaebc8865fd422c0; AITER version inside CI image: v0.1.12.post2.dev150+ga6bb49937; Dev/patched version detected → skipping rebuild. No AITER_COMMIT_OVERRIDE. (verbatim from log lines 2279-2285).
• test_fused_ar_rms_residual_accuracy output (verbatim):

  hidden_dim=2880:
        1x2880 (1-stage): max_diff=6.250000e-02  frac_nonzero=0.2774  AR_exact=yes  [FAIL]
        ... (every shape across hidden_dims 2880/4096/5120/6144/7168/8192 marked [FAIL])
  FAILED: fused residual output diverges from unfused path for some shapes.
  

  AR_exact=yes indicates the bare allreduce is bit-exact between the two paths; the divergence is only in the residual after the fused RMSNorm.
• max_diff=6.25e-02 is exactly 2^-4 — one bf16 ULP for values in [1,2). This matches the pre-fix symptom described in ROCm/aiter#2586 (the test's own docstring cites this as the regression it guards against).
• Aiter compare 43b7379...a6bb49937: ahead_by: 210, behind_by: 0 — the Dockerfile-pinned commit IS a descendant of the #2586 fix that landed 2026-04-02. Yet the test (which is the regression test for that exact bug) fails with the exact pre-fix signature.
• test_moriep_small.py: HSA crash at 01:49:46.8527 — HSA_STATUS_ERROR_OUT_OF_RESOURCES … Available Free mem: 2 MB, Fatal Python error: Aborted, scheduler stack at scheduler.py:3934 run_scheduler_process → scheduler.py:1505 run_event_loop → scheduler_dp_attn_mixin.py:60 _get_fallback_tensor. Job then sits for ~6 minutes more before the 3600 s outer timeout fires.
• The same shard 2 also failed test_fused_ar_rms_benchmark (line 253) in earlier runs (2026-04-29/30) — i.e., even before the residual-accuracy code path was added/exercised, the suite was already failing.

Hypothesised Causes (with confidence, with disconfirming evidence)

Cluster A — test_fused_ar_rms_residual_accuracy:

• [MEDIUM] The aiter dev/patched build present in the CI base image does NOT actually contain the ROCm/aiter#2586 fix on the gfx942 (MI325) code path, despite the SHA-stamped version being a descendant of the fix commit.

  • Supporting: failure signature is identical to the pre-fix bug (1 bf16 ULP, ~28 % nonzero, residual-only); aiter source SHA is "ahead by 210" of the fix; image is labelled Dev/patched version detected: v0.1.12.post2.dev150+ga6bb49937 → skipping rebuild, which implies the runtime aiter does not match the SHA's pristine source — the dev build could be a reverted or mid-development snapshot.
  • Disconfirming: aiter #2586 was merged 2026-04-02, well before this image was built; no in-flight aiter revert visible in the queryable issue list.
  • How to verify: inside the CI container run python -c "import aiter, inspect; print(inspect.getsourcefile(aiter)); from aiter import allreduce_fusion_kernel_1stage as k; print(k)" and inspect whether the bf16-roundtrip from PR #2586 is present in the actually-loaded kernel binary; or rebuild the image with AITER_COMMIT_OVERRIDE=43b7379... (just past the fix) and rerun the test.

• [LOW] MI325 (gfx942) hardware path of the fused kernel was never covered by #2586's fix (which may have targeted MI300 only).

  • Supporting: fault is uniform across all hidden_dims; runner is linux-mi325-8gpu-sglang.
  • Disconfirming: aiter PR #2586 description says the fix is in allreduce_fusion_kernel_1stage itself, not arch-gated.
  • How to verify: read the diff of 43b7379b and confirm whether the change is arch-conditional.

• [LOW] Test-side bug: the harness compares fused-vs-unfused incorrectly (e.g., wrong rounding mode in the reference path).

  • Supporting: test was authored very recently (2026-04-29) and has never been observed green in CI.
  • Disconfirming: AR_exact=yes shows the harness's own allreduce reference matches the kernel's allreduce; the divergence is exactly localized to the residual path the test was designed to validate. max_diff matching one bf16 ULP precisely is consistent with a real kernel bug, not test arithmetic noise.
  • How to verify: run the same script with the upstream-pristine aiter 43b7379 rebuild and check whether the test passes.

Cluster B — test_moriep_small.py timeout:

• [LOW] Resource leak / OOM accumulation across the multiple sub-tests this suite runs (it sequentially launches many MoE EP server fixtures — see :172, :276, :430, :490 deprecation lines in the log).

  • Supporting: HSA reports Available Free mem: 2 MB at the time of the abort; the abort happens partway through one of the later sub-tests after several earlier server lifecycles; suite has variants that completed differently across runs (sometimes setUp-class crash, sometimes timeout, sometimes test_gsm8k assertion).
  • Disconfirming: the same shard's test_aiter_allreduce_fusion_amd.py runs immediately after and successfully launches 4-GPU torchrun jobs, so cards aren't permanently wedged.
  • How to verify: run test_moriep_small.py standalone in the same image and capture rocm-smi --showmemuse between sub-tests; check whether per-test cleanup (tearDownClass) actually frees server processes.

• [LOW] 3600 s outer wall-clock budget is too tight for this suite (est_time was set to 1200 s in register_amd_ci).

  • Supporting: the run had not crashed earlier — multiple sub-tests printed deprecation lines past the 1200 s mark before the HSA abort hit; even the sub-tests that successfully ran took >25 minutes (line 12697 at 01:27 → start was ~00:55).
  • Disconfirming: the abort is a hard runtime fatal, not just slow progress — increasing the budget alone will not help.
  • How to verify: re-run with --timeout 7200 and observe whether the HSA abort still fires.

No hypothesis above MEDIUM confidence; needs in-container inspection of the actually-loaded aiter allreduce_fusion_kernel_1stage binary and a clean-rebuild A/B against 43b7379b.

In-flight Fix Check

Searched open sglang PRs for aiter+allreduce+fusion, allreduce+fusion+residual, aiter+allreduce+1stage, test_moriep_small, and moriep (curl https://api.github.com/search/issues?q=repo:sgl-project/sglang+is:pr+is:open+<keywords>).

• Cluster A — In-flight fix: ❌ none found in open sglang PRs. Open PRs that mention "aiter allreduce fusion" (#17262, #23765, #23402) are unrelated feature work, not regressions on this kernel. The aiter-side fix ROCm/aiter#2586 is already merged but the symptom recurs — needs aiter-side investigation, not a new sglang PR.
• Cluster B — In-flight fix: ❌ none found targeting this timeout. Open PRs mentioning "moriep" (#22985, #23760, #11150) are feature work, not test-time fixes.

Suggested Triage Steps

• Cluster A — confirm whether the runtime aiter actually has the #2586 fix. Inside rocm/sgl-test:* container, dump the loaded allreduce_fusion_kernel_1stage source/.so and diff against 43b7379b; rebuild the CI image with AITER_COMMIT_OVERRIDE=43b7379b and re-run python3 test/registered/ops/test_aiter_allreduce_fusion_amd.py --residual-accuracy to A/B.
• Cluster A — escalate to AMD aiter team with the test repro and the observation that max_diff = 2^-4 = 1 bf16 ULP with AR_exact=yes ; this is the exact pre-fix signature and argues the dev image has a regressed or incomplete patchset.
• Cluster A — until the aiter side is verified, consider gating the test with unittest.skipIf only on AMD MI325 + Dev/patched aiter image so the regression-detection signal is preserved on environments that actually contain the fix; do NOT delete the test.
• Cluster B — bisect across the moriep sub-tests. Run them individually (-k test_gsm8k, -k test_offline_throughput, …) inside the container and check rocm-smi --showmemuse between sub-tests to find the leaking variant; if reproducible, file an sglang issue against the MoE-EP server fixture teardown path.
• Cluster B — meanwhile increase shard‑2 wall-clock budget (currently 3600 s for an est_time=1200 test) only after the OOM root cause is understood; budget alone will not mask HSA_STATUS_ERROR_OUT_OF_RESOURCES.
• Once Cluster A is fixed, rerun shard 2 in isolation (workflow_dispatch of pr-test-amd.yml) and verify both tests pass before declaring the shard healthy again.

Status

• Persistence: Failing for 3 days across 10 consecutive completed scheduled runs (since the test files were merged in d9270b8c on 2026-04-29 18:34 UTC). Never observed green in the queryable window.
• Scope: 1 shard (stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2)) on pr-test-amd.yml; shards 0 and 1 of the same matrix pass (they don't run these two files). No spillover to other workflows observed.
• Blocked work: Permanent red on shard 2 of MI325 stage-c on every scheduled pr-test-amd.yml run; masks any genuine regression that would land on this shard.
• In-flight fix: ❌ none found in open sglang PRs for either cluster (see In-flight Fix Check above).

stage-b-test-large-8-gpu-35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — failed step(s): Ensure VRAM is clear

Commit Info

• sglang: b939d541
• aiter: N/A (job failed in pre-test VRAM cleanup before any container/aiter check ran)

Failed Tests

Test File	Test Function	Error	Log
N/A — pre-test infrastructure failure	N/A	ensure_vram_clear.sh exited 1 after 3 cleanup attempts; runner GPUs were already 88–89% VRAM-allocated at job start with no owning processes found	Log

Failure Summary

The job failed during the very first runner step bash scripts/ci/amd/ensure_vram_clear.sh rocm — before the docker container, sglang install, or any test was launched. All 8 GPUs on runner linux-mi35x-gpu-8.fabric-kbhnc-runner-z2fck reported 88–89% VRAM allocated at job start, and three cleanup passes (kill SGLang procs → 30s wait → re-check) could not free it. The cleanup script then reported No processes found using /dev/kfd and No processes found accessing /dev/kfd, indicating the memory was held by zombie/leaked allocations not attributable to any live process.

Regression Status

Flaky test / Infrastructure issue — same job slot has alternated pass/fail recently (lookback: 5 most recent completed event=schedule runs of pr-test-amd.yml on main).

Recent history of stage-b-test-large-8-gpu-35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) (same branch, same event, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
May 02 06:44	25246077188	job	❌ Failed	N/A (pre-test)	ensure_vram_clear.sh exit 1 — VRAM 88–89% with no owning procs
May 02 00:43	25239403651	job	✅ Passed	—	—
May 01 18:27	25227171827	job	✅ Passed	—	—
May 01 12:25	25214178926	job	❌ Failed	KVTransfer disagg test (different failure mode — VRAM cleanup passed in 1 attempt; failure was in the actual disaggregation test execution)	KVTransferError ... Decode transfer failed
May 01 06:59	25205835327	job	✅ Passed	—	—

Today's failure is distinct from the May 01 12:25 disaggregation-test failure: today's failure is purely the runner-side VRAM-cleanup gate, before the test even starts.

Failure Cluster

Pre-test VRAM-cleanup gate failure on AMD runner (ensure_vram_clear.sh exit 1, no owning processes found)

Facts (verified observations, no interpretation)

• Runner: linux-mi35x-gpu-8.fabric-kbhnc-runner-z2fck (single runner; only this job in the entire run failed — 1/43 jobs).
• Initial VRAM at job start (06:44:08 UTC): GPU0–7 at 88/89/89/89/89/88/88/88 %.
• Cleanup attempts: 3 passes, each killed sglang processes + aggressive cleanup + 30s wait. VRAM unchanged across all 3 attempts.
• Process audit after attempt 3: No processes found using /dev/kfd, No processes found accessing /dev/kfd. Memory is allocated but no live process owns it (typical of leaked HIP allocations or kernel-driver-held buffers from a previous container/job that was killed without graceful release).
• Failing step: Run bash scripts/ci/amd/ensure_vram_clear.sh rocm exited code 1 (log line :372).
• No prior [CI-AITER-CHECK] markers — failure occurred before container start.
• No tests executed in this run; this is a setup-step failure, not a test failure.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] Leaked GPU memory / orphaned KFD allocation from a previous job on the same runner that did not release VRAM cleanly before terminating.
  • Supporting: VRAM at 88–89% on every GPU at job start; no owning processes found; cleanup script's process-killing strategy cannot reclaim memory that has no owning PID (the killed processes are already gone). This pattern is consistent with a previous job's container being force-stopped while ROCm allocations were still pinned.
  • Disconfirming: We do not have direct evidence of which prior job on the runner left state behind. The runner's prior-job log was not inspected here.
  • How to verify: Use the GitHub Actions API to list jobs that ran on linux-mi35x-gpu-8.fabric-kbhnc-runner-z2fck in the few hours preceding 06:44 UTC and inspect their cleanup logs / exit modes.
• [LOW] Driver/kernel-level VRAM accounting bug where rocm-smi reports stale allocations.
  • Supporting: 88–89% with no live PID is unusual.
  • Disconfirming: This pattern has not been reported across other concurrent mi35x runners in this run; only this single runner is affected.
  • How to verify: Reboot the runner / reset GPUs (rocm-smi --gpureset) and re-check baseline VRAM.
• [LOW] ensure_vram_clear.sh is too aggressive (5% threshold) or its cleanup path is incomplete (does not call rocm-smi --gpureset or hipMemFlush equivalent).
  • Supporting: Cleanup tries kill+wait but never resets GPU state.
  • Disconfirming: The same script has been passing on other runs of this job within the last 24h (last passing run was 6h ago), so the script itself is not the trigger — runner state is.
  • How to verify: Add a GPU-reset fallback to attempt 3 and observe whether subsequent runs recover.

No hypothesis above MEDIUM confidence; this is most likely a transient runner-state issue rather than a code regression.

In-flight Fix Check

• Searched open PRs for keywords VRAM cleanup mi35x (0 hits) and ensure_vram_clear (0 hits).
• In-flight fix: ❌ none found in open PRs — no active work on ensure_vram_clear.sh or runner VRAM-leak handling.

Suggested Triage Steps

• Re-run the failed job once (cheapest test for transience) — if it passes on a different runner instance, this confirms an isolated runner-state issue.
• If it fails again on linux-mi35x-gpu-8.fabric-kbhnc-runner-z2fck, drain that runner and either reboot it or run rocm-smi --gpureset manually before returning it to the pool.
• Inspect the GitHub Actions API for the immediately-preceding job(s) on runner linux-mi35x-gpu-8.fabric-kbhnc-runner-z2fck (use https://api.github.com/repos/sgl-project/sglang/actions/runners + recent run timing) to identify which workload left ~89% VRAM behind. Look for jobs that timed out, were cancelled, or hit OOM/segfault rather than completing cleanly.
• Consider hardening scripts/ci/amd/ensure_vram_clear.sh to add a rocm-smi --gpureset (or equivalent) fallback on attempt 3 instead of failing — but only after confirming the runner-state hypothesis above; bot does NOT prescribe this change as the answer.

Status

• Persistence: 1 occurrence in the last 5 completed scheduled runs of this job (most recent run; previous run 6h earlier passed). Treat as flaky / runner-state, not a persistent regression.
• Scope: 1 job in 1 workflow run (1/43 jobs in this run failed). Affects only the linux-mi35x-gpu-8.fabric runner pool's kbhnc-runner-z2fck instance for this run.
• Blocked work: Loses the 8-GPU MI35x disaggregation signal for this run only; next scheduled run (~6h) will restore signal if the runner-state hypothesis is correct.
• In-flight fix: ❌ none found in open PRs — no active work on ensure_vram_clear.sh or runner VRAM-leak handling.

stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) — failed step(s): Run test

Commit Info

• sglang: b939d5410
• aiter: a6bb49937 (source: dev/patched image — v0.1.12.post2.dev150+ga6bb49937, rebuild skipped per [CI-AITER-CHECK] markers; matches Dockerfile AITER_COMMIT for gfx950 block)

Failed Tests

Test File	Test Function	Error	Log
test/registered/core/test_gpt_oss_1gpu.py	test_mxfp4_20b	AssertionError: False is not true (streaming response content is empty at gpt_oss_common.py:121 — self.assertTrue(len(content) > 0))	Log line

test_bf16_20b in the same file passed (gpqa scores 0.41/0.45/0.33 vs expected 0.34/0.34/0.27). Only the mxfp4 variant fails.

Failure Summary

test_mxfp4_20b (openai/gpt-oss-20b with quantization=mxfp4, served on the AMD MI35x / gfx950 1-GPU runner) fails the streaming-response sanity check inside _check_streaming_responses_api_request: the server returns HTTP 200 and decode-batch logs show ~180 tok/s with cuda graph enabled, but no response.output_text.delta events arrive, so the accumulated content string is empty. The retry once more yields the same empty response and the test errors out.

Regression Status

Never-passed on this job/runner within the queryable schedule history (lookback ≥ 17 days back to 2026-04-15 00:37 UTC, the oldest queryable schedule run 24430282646 — earlier scheduled runs aren't returned by the API in this query).

Recent history of test_gpt_oss_1gpu.py → test_mxfp4_20b in stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) (branch=main, event=schedule, completed runs only):

Date	Run	Job	Test File Status	Failed Function	Error
May 2 06:44	25246077188	stage-b…mi35x	❌ Failed	test_mxfp4_20b	assertTrue(len(content)>0) empty stream
May 2 00:43	25239403651	stage-b…mi35x	❌ Failed	test_mxfp4_20b	TIMEOUT 1200 s (same empty-stream + retry)
May 1 18:27	25227171827	stage-b…mi35x	❌ Failed	test_mxfp4_20b	TIMEOUT 1200 s
May 1 12:25	25214178926	stage-b…mi35x	❌ Failed	(job-level VRAM cleanup before test)	unrelated runner env failure
May 1 06:59	25205835327	stage-b…mi35x	❌ Failed	test_mxfp4_20b	assertTrue(len(content)>0) empty stream
May 1 00:48	25196776069	stage-b…mi35x	❌ Failed	test_mxfp4_20b	TIMEOUT 1200 s
Apr 15 00:37	24430282646	stage-b…mi35x	❌ Failed	test_mxfp4_20b	TIMEOUT 1200 s

No green run for test_mxfp4_20b on this runner has been observed in the queryable window. Symptom toggles between "empty streaming response → assertion fail → retry → still empty → ERROR" and "first attempt hangs long enough that the 1200 s wrapper TIMEOUT fires before the assertion is reached" — same root failure mode, different exit path.

Failure Cluster

gpt-oss-20b mxfp4 streaming response empty on MI35x (gfx950)

Facts (verified observations, no interpretation)

• Test file: test/registered/core/test_gpt_oss_1gpu.py:12 → calls BaseTestGptOss.run_test(model_variant="20b", quantization="mxfp4", …).
• Failure point: python/sglang/test/gpt_oss_common.py:121 — self.assertTrue(len(content) > 0). The preceding log line is Streaming check response: (i.e. empty after the colon — verified verbatim at log row 6953).
• Server is healthy at the time of failure: decode-batch traces show cuda graph: True, gen throughput (token/s): 179.97 … 180.00 immediately before the assertion (rows 6822-6823). No GPU fault, no exception, no kernel error in the surrounding ~50 log lines.
• Sister test in the same suite: test_bf16_20b (lmsys/gpt-oss-20b-bf16, no quant) passes on the same runner in the same job — Streaming check response: 1 + 1 = 2. is logged (row 2677-ish) and gpqa eval completes with score 0.41/0.45/0.33. So the harness, runner, and aiter image are functional for gpt-oss in bf16; only the mxfp4 path fails.
• aiter is a dev-patched image preserved without rebuild: [CI-AITER-CHECK] AITER version inside CI image: v0.1.12.post2.dev150+ga6bb49937 → skipping rebuild (rows 2311-2312). Underlying SHA matches Dockerfile expectation a6bb49937.
• Server config relevant to the failure: Detected ROCm and MXFP4 quantization format for GPT-OSS model, enabling aiter MXFP4 MOE kernel. (row immediately after model load).
• Reproducibility: every queryable scheduled run of this job since 2026-04-15 fails the same test_mxfp4_20b. No green observed in ≥ 17 days.

Hypothesised Causes (with confidence, with disconfirming evidence)

• [MEDIUM] The mxfp4 path on gfx950 produces output tokens that do not surface as response.output_text.delta events on the Responses API — possibly all generated tokens are routed to a reasoning/internal channel (the harmony chat template separates analysis / final channels for gpt-oss), or the mxfp4 MoE produces garbled/non-EOS tokens that the responses-api streaming layer filters out before emitting output_text.delta.
  • Supporting: server runs cleanly (no errors, ~180 tok/s decode), only len(content)==0 fails; the Responses-API streaming code only accumulates content from response.output_text.delta events; test enables aiter MXFP4 MOE kernel per log; bf16 sister test passes with the same harness and same model class, so the failure is mxfp4-specific.
  • Disconfirming: cannot confirm from log alone which event types were emitted (the test only logs accumulated content, not the raw SSE events). A deterministic-quality issue (mxfp4 tokens are nonsense) would more typically produce a _run_one_eval low-score failure rather than an empty-stream failure, so this hypothesis is not airtight.
  • How to verify: in a local repro, instrument _check_streaming_responses_api_request to also print data.get('type') for each event; or curl /v1/responses with the same payload manually against the running server and inspect the SSE stream.
• [LOW] aiter MXFP4 MOE kernel emits incorrect/non-text tokens on gfx950 (MI35x).
  • Supporting: failure is exclusive to mxfp4 quantization which routes through the aiter MXFP4 MOE kernel per server log; gfx950 (MI35x) is the only architecture this job runs on.
  • Disconfirming: aiter version is unchanged across the entire 17-day failure window (same dev image SHA every run), so aiter cannot have introduced a regression — at most it has never worked here. No log/stack-trace evidence pointing into the kernel; the assertion fires in user-space Python.
  • How to verify: run the same test with --quantization overridden to a non-mxfp4 path (e.g. fp8) on MI35x; or run on MI300x (gfx942) with mxfp4 to see whether the failure is gfx950-specific.
• [LOW] expected_score_of_reasoning_effort thresholds (0.34/0.34/0.27) for mxfp4-20b are mis-configured for AMD/MI35x and the _check_streaming_responses_api_request empty-content assertion is a spurious symptom of the run_test ordering.
  • Supporting: 20b/mxfp4 quality on AMD may differ from CUDA-tuned thresholds; the test file already carries a # TODO investigate comment on the high threshold.
  • Disconfirming: streaming-content check runs before any gpqa eval (gpt_oss_common.py:67 → _check_streaming_responses_api_request → returns; eval only follows). Empty content is a hard failure independent of score thresholds. So this hypothesis cannot explain the observed assertion at line 121.
  • How to verify: not actionable for this specific failure.

No hypothesis above MEDIUM confidence; root cause needs an interactive repro on an MI35x runner.

In-flight Fix Check

• Searched: repo:sgl-project/sglang is:pr is:open gpt-oss mxfp4 → returned #23218 [WIP] feat: add triton_quant_info for mxfp4 gpt-oss moe layer (potentially related — touches mxfp4 gpt-oss MoE path, but it's a WIP feature add, not an explicit fix for this empty-stream failure).
• Searched: repo:sgl-project/sglang is:pr is:open test_gpt_oss_1gpu → 0 results.
• Searched: repo:sgl-project/sglang is:pr is:open gpt-oss mi35x → 4 results, none target this test (closest: #23957 AMD CDNA4 Gluon Extend Attention — different code path).
• In-flight fix: ❌ none clearly targeting this failure. #23218 may incidentally fix it if it lands and changes the mxfp4 MoE invocation; worth tracking but not a direct fix.

Suggested Triage Steps

• Reproduce locally on an MI35x (gfx950) runner: python3 test/registered/core/test_gpt_oss_1gpu.py TestGptOss1Gpu.test_mxfp4_20b against the same dev-patched aiter image (v0.1.12.post2.dev150+ga6bb49937); add a print(data) inside the SSE loop in python/sglang/test/gpt_oss_common.py:104-118 to capture which event types the server is actually emitting.
• A/B against the bf16 path: run a manual curl -N to /v1/responses with the same payload after starting sglang serve --model-path openai/gpt-oss-20b --quantization mxfp4 vs --model-path lmsys/gpt-oss-20b-bf16 and diff the SSE event streams to isolate whether the mxfp4 server emits only response.reasoning.delta (or similar) events without response.output_text.delta.
• Cross-architecture A/B: run the same test_mxfp4_20b on MI300x (gfx942) — if it passes, the issue is gfx950-specific and points at the aiter MXFP4 MOE kernel for CDNA4; if it also fails, the issue is in the sglang Responses-API harmony channel routing for mxfp4.
• Track #23218 — if/when it merges, immediately re-run this job to see whether it incidentally restores the streaming output.
• If no fix lands and the failure must be unblocked from CI signal: consider gating test_mxfp4_20b on stage-b-test-1-gpu-small-amd-mi35x until the gfx950 mxfp4 path is supported (decision for maintainers; bot does not prescribe).

Status

• Persistence: Failing for ≥ 17 days (since at least 2026-04-15 00:37 UTC, the oldest queryable scheduled run; every completed stage-b-test-1-gpu-small-amd-mi35x schedule run in that window has the same test_mxfp4_20b failure, alternating between empty-stream-assertion and 1200 s wrapper timeout exit paths).
• Scope: 1 test (test_mxfp4_20b) in 1 job (stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1)) in 1 workflow (pr-test-amd.yml). Sister test test_bf16_20b in the same file passes. No CUDA equivalent runs on this code path (the registered CUDA suite for test_gpt_oss_1gpu.py is stage-b-test-1-gpu-large, not the mi35x suite).
• Blocked work: gpt-oss-20b mxfp4 quality signal on AMD MI35x (gfx950).
• In-flight fix: ❌ none clearly targeting this failure; #23218 (WIP, mxfp4 gpt-oss MoE triton_quant_info) may be related but is not labelled as a fix.

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-05-02 20:30 UTC)

[amd-bot]

pr-test-amd.yml — Cross-Run Pattern Summary (8 runs in lookback window)

Last scanned (UTC): 2026-05-02 20:32 UTC

Agent assessment

Persistent clusters (every completed run)

Cluster A · aiter fused AR+RMSNorm residual 1-ULP bf16 mismatch — 8/8 runs · since ≥ 2026-04-29

• Affected: stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2) — every run; same job on shard 0 reproduced once on 25239403651. Also stage-b-test-1-gpu-small-amd-mi35x on 25214178926.
• Tests: test/registered/ops/test_aiter_allreduce_fusion_amd.py::TestAiterAllreduceFusionAmd::test_fused_ar_rms_residual_accuracy (and test_fused_ar_rms_benchmark — alternates because unittest method ordering gates which one trips first). max_diff=6.25e-2 (= 1 bf16 ULP), frac_nonzero≈0.28, AR_exact=yes for every (m, hidden_dim) shape across both 1-stage and 2-stage paths.
• Top hypothesis: [HIGH] aiter allreduce_fusion_kernel_* adds residual before rounding the f32 AR accumulator to bf16 — the exact regression ROCm/aiter#2586 that this test was written to catch (per its docstring at test_aiter_allreduce_fusion_amd.py:18-28). Disconfirming: no diff against current aiter pin a6bb49937 verifying the fix is missing rather than reverted.
• In-flight fix: ❌ none in sglang (fix must land in ROCm/aiter).

Cluster B · moriep DeepSeek-V3 / TBO scheduler hang or HSA OOM — 8/8 runs of stage-c shard 2 (varying symptom)

• Affected: stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2) — every run (latest stage-c shard 2). Also fired on shard 0 once at 25239403651, and on stage-b-test-1-gpu-small-amd-mi35x at 25227171827.
• Tests: test/registered/amd/test_moriep_small.py — TestMTPwithTBONormal.test_gsm8k, TestMTPwithTBOLowLatency.setUpClass, or whole-file 3600 s timeout. Symptoms vary across runs: scheduler watchdog (300 s) during prepare_mlp_sync_batch_raw → all_gather → _get_fallback_tensor → c10::hip::memcpy_and_sync; or HSA_STATUS_ERROR_OUT_OF_RESOURCES — Available Free mem: 0–2 MB; or setUpClass AttributeError: 'NoneType' has no attribute 'shape' at two_batch_overlap.py:208 once.
• Top hypothesis: [LOW] DP-attention all-gather of fallback tensors hangs deterministically in this shard; symptom variability suggests multi-GPU comm instability rather than one regression. Disconfirming: setUpClass NoneType .shape is a code path bug, not a runtime hang — these may be ≥ 2 distinct issues co-clustered. [SPECULATION] shared aiter custom_all_reduce dependency with Cluster A.
• In-flight fix: ❌ none.

Cluster C · gpt-oss-20b mxfp4 streaming Responses API returns empty content on MI35x (gfx950) — 6/8 runs of stage-b-mi35x

• Affected: stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) on runs 25196776069, 25205835327, 25239403651, 25246077188, 25251804407, 25258688414.
• Test: test/registered/core/test_gpt_oss_1gpu.py::TestGptOss1Gpu::test_mxfp4_20b. assertTrue(len(content) > 0) fires at gpt_oss_common.py:121 because output_text.delta events never arrive; retry exhausts at the 1200 s file budget. Sister test_bf16_20b consistently passes — only the mxfp4 quant path fails on MI35x.
• Top hypothesis: [LOW] mxfp4 quantization decode emits no streaming text on gfx950; isolated to one quant variant on one arch. Disconfirming: the underlying decode runs (server log shows progress); the symptom is in the streaming-event emission, not the model — could be a fast-path skip in the Responses API plumbing.
• In-flight fix: ❌ none found.

Regression candidates (latest completed run only)

• sgl-kernel-unit-test-amd (linux-mi325-1gpu-sglang) — sgl-kernel/tests/test_moe_topk_sigmoid.py::test_topk_sigmoid[2048-128-4] — AssertionError: Indices mismatch (torch=tensor([[44,112,116,102],…]), SGLang=tensor([[44,112,116,102],…]) — visible head/tail print identically, divergence is in unprinted middle rows). 1 of 774 parametrizations failed. Per the per-job analysis this is a tied-score topk edge case (atol=0, rtol=0 comparison) and likely flaky rather than a deterministic regression. Suggested triage: rerun pytest sgl-kernel/tests/test_moe_topk_sigmoid.py::test_topk_sigmoid -k '2048-128-4' -p no:randomly --count=20 to confirm flakiness; if reproducible, switch to index-set comparison or relax tied-score handling. No commit-window bisect needed yet.

Flakes / intermittent

• HF Hub RemoteProtocolError / xet_get hang during setUpClass — burst on May 2 runs 25246077188 (4 affected: shards 0/4/11/13) and 25251804407 (5 affected: stage-a + shards 1/2/7/13). Tests: test_transformers_models.py, test_lora_eviction.py, test_torch_compile.py, test_srt_backend.py, test_llada2_mini_amd.py::setUpClass, test_generation_models.py (gemma2). Subsided in latest run 25258688414. Action: do not investigate as code regressions — investigate runner network / HF Hub status during 06:44–13:00 UTC May 2. Quarantine candidates: none; just rerun.
• Silent test_mmlu / test_latency 1800 s timeouts on MI325 — 2/8 runs. test_retract_decode.py::test_mmlu (74044904944), test_triton_attention_backend.py::test_latency shards 0 and 11 (74030480444, 74030480448). All silent — no pyspy dump emitted before kill. Action: enable pyspy --idle dump 60 s before harness timeout to capture state.
• stage-b-test-large-8-gpu-35x-disaggregation-amd — test_disaggregation_pp.py::test_gsm8k Mooncake KV transfer 30 s timeout 2/8 runs (25214178926, 25246077188 — note the latter is a pre-test VRAM-clear infra failure, not the same symptom). 1/8 deterministic.
• stage-b-test-1-gpu-large-amd shard 0 (73906083586) — test_bench_serving_1gpu_part1.py::test_online_lora_latency_with_concurrent_adapter_updates HTTP 400 "Server disconnected". 1/8 — likely flaky concurrent-load LoRA path.
• VRAM-clear pre-flight failure on linux-mi35x-gpu-1 — 2/8 runs (25214178926, 25246077188). Runner-state issue; reboot or kfd reset between scheduled runs.
• DeepSeek-V3.2 server-launch hang on stage-c shard 0 (74012273708) — 1/8. HF download lock + 8× TP MoE weight-load stall. Could be HF-burst-related.

Suggested next actions

1. Cluster A (aiter residual) — verify whether the upstream fix for ROCm/aiter#2586 has merged and is included in the dev image's pinned aiter SHA a6bb49937. If not, set AITER_COMMIT_OVERRIDE to a SHA containing the fix and rerun stage-c shard 2. Highest ROI: would clear the stage-c shard 2 floor.
2. HF Hub burst — confirm with the May 2 06:00–13:00 UTC HF Hub status page whether the 9 distinct setUpClass failures across runs 25246077188 and 25251804407 correlate with an HF outage; if yes, do not open per-test fix PRs. Consider making model loaders retry on RemoteProtocolError with backoff.
3. Cluster B (moriep) — rerun test_moriep_small.py::TestMTPwithTBONormal::test_gsm8k 5× in isolation on the same MI325 8-GPU runner to separate the deterministic DP-attn _get_fallback_tensor hang from the OOM and setUpClass NoneType .shape symptoms. Use the debug-distributed-hang playbook to dump py-spy on every DP rank.
4. Regression candidate (test_topk_sigmoid) — rerun the parametrization 20× before opening a fix PR; if flake-confirmed, relax tied-score comparison instead of hunting a regressor.
5. Silent-hang flakes (test_latency, test_mmlu, test_lora_eviction) — instrument the harness to py-spy 60 s before kill so the next occurrence yields actionable data; then revisit clustering.

---

Runs analysed

Run	Started (UTC)	Failed jobs
25196776069	2026-05-01 00:48	2
25205835327	2026-05-01 06:59	3
25214178926	2026-05-01 12:25	3
25227171827	2026-05-01 18:28	2
25239403651	2026-05-02 00:44	3
25246077188	2026-05-02 06:44	8
25251804407	2026-05-02 12:23	7
25258688414	2026-05-02 18:22	3

Persistent failures (in every run)

• stage-b-test-1-gpu-small-amd-mi35x (linux-mi35x-gpu-1) — in 8/8 runs (6776069, 5835327, 4178926, 7171827, 9403651, 6077188, 1804407, 8688414)
• stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 2) — in 8/8 runs (6776069, 5835327, 4178926, 7171827, 9403651, 6077188, 1804407, 8688414)

Latest-only failures (potential regressions)

• sgl-kernel-unit-test-amd (linux-mi325-1gpu-sglang) — in 1/8 runs (8688414)

Flaky / intermittent failures

• stage-a-test-1-gpu-small-amd (linux-mi325-1gpu-sglang) — in 1/8 runs (1804407)
• stage-b-test-1-gpu-large-amd (linux-mi325-1gpu-sglang, 0) — in 1/8 runs (5835327)
• stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 0) — in 1/8 runs (6077188)
• stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 1) — in 1/8 runs (1804407)
• stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 11) — in 1/8 runs (6077188)
• stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 13) — in 2/8 runs (6077188, 1804407)
• stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 2) — in 1/8 runs (1804407)
• stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 4) — in 1/8 runs (6077188)
• stage-b-test-1-gpu-small-amd (linux-mi325-1gpu-sglang, 7) — in 2/8 runs (6077188, 1804407)
• stage-b-test-large-8-gpu-35x-disaggregation-amd (linux-mi35x-gpu-8.fabric) — in 2/8 runs (4178926, 6077188)
• stage-c-test-large-8-gpu-amd (linux-mi325-8gpu-sglang, 0) — in 1/8 runs (9403651)

---

Generated by amd-bot using Claude Code CLI (last updated: 2026-05-02 20:32 UTC)