[AMD] Register 5 server-style 1-GPU tests for AMD PR CI by michaelzhang-ai · Pull Request #24572 · sgl-project/sglang

michaelzhang-ai · 2026-05-07T02:13:04Z

Summary

Register 5 server-style 1-GPU PR-CI tests on AMD's stage-b-test-1-gpu-small-amd suite. All 5 tests are already running on NVIDIA per-commit CI; they use only Triton attention / CPU torch / mocks / default small models — none of FA3, FlashInfer, ModelOpt, Marlin, EAGLE, GDN/KDA kernels, or NSA-FP8 paths that already have AMD exclusions.

This PR adds 5 tests to stage-b-test-1-gpu-small-amd.

Tests added (5) — all confirmed PASSED on AMD MI300 in run 25480011932

File	est_time	AMD CI runtime	Partition	Why it's safe on AMD
`distributed/test_parallel_state.py`	8s	9s ✅	12	Pure mocks, no GPU
`prefill_only/test_embed_overrides.py`	9s	10s ✅	5	CPU torch + mocks
`input_embedding/test_input_embeds_chunked.py`	43s	55s ✅	13	Default small-model server
`openai_server/basic/test_http2_server.py`	52s	5s ✅	10	Auto-skips when `granian` is not installed
`radix_cache/test_radix_cache_hit.py`	55s	66s ✅	13	Default small-model server

Total added est_time: ~167s spread across 14 partitions.

Verification

$ python3 ast-collector
OK: 837 registries with sanity_check=True

All 5 tests appear in --hw amd --suite stage-b-test-1-gpu-small-amd.
Lints clean.

CI status note: remaining failures are pre-existing infra issues, NOT caused by this PR

The latest AMD CI run (25480011932) had failures on 3 partitions, none from this PR's files:

stage-b-test-1-gpu-small-amd (partition 2) → test_torch_compile.py failed (pre-existing)
stage-b-test-1-gpu-small-amd (partition 7) → test_retract_decode.py failed (pre-existing)
stage-b-test-1-gpu-small-amd (partition 9) → 45-min timeout on test_update_weights_from_tensor.py (pre-existing)

Other failed checks (build-and-test XPU, stage-b-test-1-gpu-small-amd-mi35x, stage-b-test-1-gpu-large-amd (1), stage-b-test-2-gpu-large-amd (0/1), stage-b-test-1-gpu-small-amd-nondeterministic, large-8-gpu-disaggregation, stage-b-test-1-gpu-small/large/2-gpu-large NV partitions) are pre-existing flakes seen on multiple unrelated PRs to main.

Test plan

AMD CI passes on the 5 new files (5/5 PASSED in run 25480011932 on AMD MI300)
NVIDIA CI still green (no behavior change for register_cuda_ci)

Second batch of 1-GPU tests onboarded onto stage-b-test-1-gpu-small-amd, following PR sgl-project#24569 (8 unit tests). All tests are already running on NVIDIA per-commit CI; they use only Triton attention / CPU torch / mocks / default small models, none of FA3, FlashInfer, ModelOpt, Marlin, EAGLE, GDN/KDA kernels, or NSA-FP8 paths that already have AMD exclusions. Tests added (folder | est_time | scope): - distributed/test_parallel_state.py 8s pure mocks - prefill_only/test_embed_overrides.py 9s pure CPU mocks - input_embedding/test_input_embeds_chunked.py 43s small-model server - openai_server/basic/test_http2_server.py 52s auto-skips w/o granian - radix_cache/test_radix_cache_hit.py 55s small-model server - openai_server/basic/test_serving_transcription 60s Whisper server - prefill_only/test_score_api.py 71s small-model server - prefill_only/test_score_engine.py 85s Qwen3-0.6B engine - sessions/test_session_control.py 87s Triton attn backend - prefill_only/test_pooled_hidden_states.py 100s Qwen3-0.6B engine - rl/test_weight_checker_e2e.py 150s Qwen3-0.6B server - prefill_only/test_multi_item_scoring.py 211s small-model engine Total added est_time: 931s spread across 14 partitions (~67s/partition). After this change `stage-b-test-1-gpu-small-amd` grows by 12 tests. Combined with PR sgl-project#24569 (8 unit tests) the suite goes 88 -> 108. Skipped from this batch (will follow up): - attention/{chunk_gated_delta_rule,gdn_noncontiguous_stride,kda_kernels, normal_decode_set_metadata}.py -- GDN/KDA kernels need ROCm review - breakable_cuda_graph/test_breakable_cuda_graph.py -- requires cuda-python - kernels/test_nsa_indexer.py -- NSA + FP8 KV cache, ROCm-specific - model_loading/test_utils_update_weights.py -- needs distributed/NCCL setup - observability/test_tracing.py -- depends on opentelemetry stack - radix_cache/test_swa_radix_cache_kl.py -- gpt-oss-20b is large - sessions/test_session_{latency,streaming_session,streaming_session_swa}.py -- performance assertions / large est_time - hicache/test_hicache_spec_file_storage.py (600s) -- defer due to size

gemini-code-assist · 2026-05-07T02:13:09Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

The AMD CI run on the previous commit revealed that 6 of the originally proposed 12 tests have legitimate ROCm incompatibilities that need upstream test code changes (out of scope for a registration-only PR): - test_score_api.py, test_multi_item_scoring.py: Both use --enable-mis with attention_backend="flashinfer" which is NVIDIA-only. ROCm AITER backend rejects the combination with a hard AssertionError ("Multi-item scoring requires flashinfer attention backend"). - test_pooled_hidden_states.py: The TestPooledHiddenStatesMISEngine subclass uses enable_mis=True which triggers the same FlashInfer-only path. - test_score_engine.py: TestSeqClsMISScoring + TestSeqClsMISAdvancedScoring use the same FlashInfer+MIS path, AND TestCausalLMScoring.test_score_deterministic asserts assertAlmostEqual(places=5) on consecutive engine.score() calls -- AMD MI300 produces an ~9e-6 difference between two runs of the same input that exceeds the 1e-5 tolerance. - test_serving_transcription.py: Whisper model loader pulls in flashinfer_python at import time (PackageNotFoundError on ROCm). - test_session_control.py: TestSessionControlVision uses image inputs that go through torio / libavutil / libavfilter (FFmpeg). The AMD CI container does not have libavutil.so.58 / libavfilter.so.7 installed. These deserve their own follow-up PRs (either adding ROCm-aware skip conditions, splitting MIS tests out, loosening determinism tolerances, or installing the missing system libs in the AMD CI image). Tests still in this PR after the revert (6 of 12): - distributed/test_parallel_state.py 8s pure mocks - prefill_only/test_embed_overrides.py 9s pure CPU mocks - input_embedding/test_input_embeds_chunked.py 43s small-model server - openai_server/basic/test_http2_server.py 52s granian-skipped - radix_cache/test_radix_cache_hit.py 55s small-model server - rl/test_weight_checker_e2e.py 150s Qwen3-0.6B server After this revert stage-b-test-1-gpu-small-amd grows by 6 (88 -> 94). The two tests already verified PASSING on AMD MI300 in the previous run: - test_http2_server.py (5s, partition 5) - test_weight_checker_e2e.py (106s, partition 10)

…ests # Conflicts: # test/registered/rl/test_weight_checker_e2e.py

michaelzhang-ai · 2026-05-11T02:20:26Z

@amd-bot ci-status

amd-bot · 2026-05-11T05:21:53Z

@michaelzhang-ai

CI Status for PR #24572

PR: [AMD] Register 5 server-style 1-GPU tests for AMD PR CI
Head SHA: 946f235
Changed files (all test-only, +2/-1 each — only adds register_amd_ci(...) next to existing register_cuda_ci(...)):

test/registered/distributed/test_parallel_state.py
test/registered/input_embedding/test_input_embeds_chunked.py
test/registered/openai_server/basic/test_http2_server.py
test/registered/prefill_only/test_embed_overrides.py
test/registered/radix_cache/test_radix_cache_hit.py

AMD: 2 real failures (0 likely related) | Others: 1 real failure + 13 cascaded fast-fail (0 related)

The PR is purely additive: it registers 5 already-existing tests on the AMD stage-b-test-1-gpu-small-amd suite and does not touch any production code, CUDA registration, or the test bodies themselves. None of the failing tests below are among the 5 newly-registered files; partitions running the new tests are 5/10/12/13 and they passed.

AMD CI Failures

Job	Test File	Test Function	Error	Related?	Explanation	Log
stage-b-test-1-gpu-small-amd (mi300, 7)	`test_bench_serving.py` (Qwen2.5-VL-3B-Instruct perf)	perf-threshold test	`AssertionError: 808.82 not greater than 2000` (throughput); 0/8 passed; preceded by detokenizer health-check stalls	🟢 Unlikely	Pre-existing perf-threshold flake on AMD MI300 with `Qwen/Qwen2.5-VL-3B-Instruct`. PR does not touch this file or any perf threshold; PR-added files target partitions 5/10/12/13, not 7.	Log
stage-b-test-2-gpu-large-amd (mi300, 0)	`test/registered/perf/test_bench_serving_2gpu.py`	`test_moe_offline_throughput_default`	`RuntimeError: NCCL error: unhandled cuda error` → `Server process exited with code -9` during `ncclCommInitRank` (Mixtral-8x7B TP=2 startup)	🟢 Unlikely	NCCL/init infra failure on a 2-GPU MoE bench. PR adds only 1-GPU AMD tests; this 2-GPU test isn't among the 5 PR-added files.	Log
wait-for-stage-b-amd	N/A	N/A	Meta-gate job — fails because the two real AMD failures above failed	🟢 Unlikely	Pure cascade from the two rows above	Log
pr-test-amd-finish	N/A	N/A	Aggregator job for the AMD pipeline	🟢 Unlikely	Pure cascade	Log

Other CI Failures

Job	Test File	Test Function	Error	Related?	Explanation	Log
stage-b-test-1-gpu-small (0)	N/A (setup step)	N/A	`##[error]The action 'Install dependencies' has timed out after 20 minutes`	🟢 Unlikely	Pure runner/network infrastructure timeout during pip install, before any test ran. Cannot be caused by a test-registration-only PR.	Log
stage-b-test-1-gpu-small (5), (6), (7)	N/A	N/A	`Fast-fail: skipping — root cause job(s): wait-for-stage-b, stage-b-test-1-gpu-small (0)`	🟢 Unlikely	Cascaded fast-fail from partition 0 timeout above	logs
stage-b-test-1-gpu-large (5–13) (9 partitions)	N/A	N/A	Same `Fast-fail` cascade message	🟢 Unlikely	Cascade from partition-0 install-deps timeout	logs
stage-b-test-2-gpu-large (2), (3)	N/A	N/A	Same `Fast-fail` cascade	🟢 Unlikely	Cascade	logs
wait-for-stage-b, pr-test-finish	N/A	N/A	Meta-gate jobs	🟢 Unlikely	Pure cascade	logs

Details

PR scope check: git diff shows the only edits are 5 files each adding from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci and a single register_amd_ci(est_time=…, suite="stage-b-test-1-gpu-small-amd") line next to the existing register_cuda_ci(...) call. No production code, no CUDA registration, no test-body changes. Therefore any failing test that is not one of the 5 added files cannot be caused by this PR.
AMD partition 7 fail (Qwen2.5-VL perf): The five PR-added files are routed (per the PR description) to partitions 5, 10, 12, 13 of stage-b-test-1-gpu-small-amd. Partition 7 contains pre-existing tests; the failing test here is a Qwen2.5-VL-3B benchmark hitting AssertionError: 808.82 not greater than 2000 after detokenizer heartbeat stalls — a known perf/health-check flake unrelated to the PR.
AMD 2-GPU fail (test_bench_serving_2gpu.py): NCCL unhandled cuda error during ncclCommInitRank while bringing up Mixtral-8x7B with tp_size=2; server exits with -9. PR adds only 1-GPU tests, so this 2-GPU job is out of scope.
All NV failures: Originate from stage-b-test-1-gpu-small (0) whose Install dependencies step hit the 20-minute timeout — a pip/network infra issue completely independent of test registration. Every other NV partition logs Fast-fail: skipping — root cause job(s): wait-for-stage-b, stage-b-test-1-gpu-small (0).

Verdict for the developer: Nothing to fix here. None of the failures are caused by PR #24572. A re-run of the AMD nightly + NVIDIA pipelines is the recommended next step; if `stage-b-test-1-gpu-small-amd (7)` and `stage-b-test-2-gpu-large-amd (0)` keep failing on main with the same signatures, they belong to the maintainer's pre-existing-flake list, not this PR.

Generated by amd-bot using Claude Code CLI

…-amd test_vlm_perf_5090.py was hitting the 1800s per-file budget on MI300 PR CI (e.g. job 75240741847 on PR #24572) and aborting the whole 14-shard partition with exit 255 ("Test Summary: 0/8 passed" even though only one file failed). On MI325 the same file finishes in ~1000s (job 75850659317 on run 25817749454), so MI300 is roughly 1.8x slower per unit of work; aiter has no tuned bf16 GEMM configs for the shapes Qwen2.5-VL-3B exercises and falls back to a generic torch GEMM path on MI300, which is what eats the extra time. Bump --timeout-per-file from 1800 to 3000 (50min) on the AMD PR-CI suite invocation in pr-test-amd.yml, and bump the corresponding step-level timeout-minutes from 45 to 75 so the GitHub Actions step itself won't kill the job before the new per-file budget applies. Other suites in the same workflow (stage-b-test-1-gpu-small-amd-nondeterministic, stage-b-test-1-gpu-large-amd, stage-b-test-2-gpu-large-amd, etc.) and the rocm720 variant keep --timeout-per-file 1800 unchanged. No test code change.

…-amd Across 13 recent MI300 PR-CI runs of stage-b-test-1-gpu-small-amd shard 7, test_vlm_perf_5090.py elapsed time was: min=1183 max=1801 mean=1308 median=1258 stdev=172 The max=1801 entry is the run that motivated this PR (job 75240741847 on PR #24572) and is exactly at the 1800s rail; the other 12 runs all land in 1183-1471s, but mean+3sigma = ~1824s lines up with the May 10 outlier so the file genuinely has a long upper tail. At 1800s, the empirical fail rate is 1/14 (~7%); at 2400s it is 0/14 with 33% headroom over max observed (599s). Bump --timeout-per-file 1800->2400 (40min) on this one suite invocation. Bump the corresponding step-level timeout-minutes 45->60 so the GitHub Actions step itself doesn't kill the job before the new per-file budget applies (50min worst-case file + ~7min for the other files in the partition + setup overhead, vs 60min step budget). Other suites in this workflow (stage-b-test-1-gpu-small-amd-nondeterministic, large/disagg) keep 1800s/45min unchanged. The MI325-hardcoded pr-test-amd-rocm720.yml is also untouched (MI325 actual elapsed is ~1000s, well under 1800s). No test code change.

…24572)

michaelzhang-ai requested a review from sundar24295s as a code owner May 7, 2026 02:13

github-actions Bot added the npu label May 7, 2026

michaelzhang-ai added the run-ci CI: run the baseline test suite on this PR label May 7, 2026

michaelzhang-ai added 2 commits May 7, 2026 11:32

Merge branch 'main' into amd/add-1gpu-server-tests

0091800

michaelzhang-ai added run-ci CI: run the baseline test suite on this PR and removed run-ci CI: run the baseline test suite on this PR labels May 7, 2026

Merge remote-tracking branch 'origin/main' into amd/add-1gpu-server-t…

567b889

…ests # Conflicts: # test/registered/rl/test_weight_checker_e2e.py

michaelzhang-ai changed the title ~~[AMD] Register 12 server-style 1-GPU tests for AMD PR CI~~ [AMD] Register 5 server-style 1-GPU tests for AMD PR CI May 8, 2026

Merge branch 'main' into amd/add-1gpu-server-tests

946f235

amd-bot mentioned this pull request May 10, 2026

[CI Monitor] Daily Report - 2026-05-10 bingxche/sglang-ci-bot#66

Open

This was referenced May 11, 2026

[CI Monitor] Daily Report - 2026-05-11 bingxche/sglang-ci-bot#67

Open

[CI Monitor] Daily Report - 2026-05-12 bingxche/sglang-ci-bot#68

Open

[CI Monitor] Daily Report - 2026-05-13 bingxche/sglang-ci-bot#70

Open

michaelzhang-ai requested a review from HaiShaw May 13, 2026 00:51

michaelzhang-ai mentioned this pull request May 13, 2026

[AMD] Bump --timeout-per-file 1800->2400 for stage-b-test-1-gpu-small-amd #25112

Merged

2 of 3 tasks

HaiShaw approved these changes May 13, 2026

View reviewed changes

HaiShaw merged commit 245f7d8 into sgl-project:main May 13, 2026
99 of 120 checks passed

michaelzhang-ai mentioned this pull request May 14, 2026

[Bug] CI flake: stage-b-test-2-gpu-large-amd on linux-mi300-2gpu-sglang — ncclCommInitRank "unhandled cuda error" reproduces across unrelated PRs #25216

Closed

5 tasks done

amd-bot mentioned this pull request May 15, 2026

[CI Monitor] Daily Report - 2026-05-15 bingxche/sglang-ci-bot#72

Open

Chronostasys pushed a commit to MindLab-Research/sglang that referenced this pull request Aug 24, 2026

[AMD] Register 5 server-style 1-GPU tests for AMD PR CI (sgl-project#…

66b0cd7

…24572)

michaelzhang-ai mentioned this pull request Sep 18, 2026

[AMD][CI] Restore AMD registrations dropped by #37436 and fix the 3FS hicache suite #40060

Closed

5 tasks done

michaelzhang-ai deleted the amd/add-1gpu-server-tests branch October 20, 2026 18:17

Navigation Menu

Repository navigation

[AMD] Register 5 server-style 1-GPU tests for AMD PR CI - #24572

[AMD] Register 5 server-style 1-GPU tests for AMD PR CI#24572
HaiShaw merged 5 commits into
sgl-project:mainfrom
michaelzhang-ai:amd/add-1gpu-server-tests

michaelzhang-ai commented May 7, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented May 7, 2026

Uh oh!

michaelzhang-ai commented May 11, 2026

Uh oh!

amd-bot commented May 11, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

michaelzhang-ai commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tests added (5) — all confirmed PASSED on AMD MI300 in run 25480011932

Verification

CI status note: remaining failures are pre-existing infra issues, NOT caused by this PR

Test plan

Uh oh!

gemini-code-assist Bot commented May 7, 2026

Uh oh!

michaelzhang-ai commented May 11, 2026

Uh oh!

amd-bot commented May 11, 2026

CI Status for PR #24572

AMD CI Failures

Other CI Failures

Details

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

michaelzhang-ai commented May 7, 2026 •

edited

Loading