Repository navigation
Conversation
Run every base-b, base-b-kernel-unit and extra-a file registered on 1-gpu-large (H100) on an RTX 5090 (SM120, 32 GB) with the CI install and environment, and move the ones that pass without losing coverage. A file with tests skipped on the 5090 only moves when those tests also skip on H100 (ROCm-, SM100- or missing-op-gated). - base-b-kernel-unit: 154 of 176 files move. The kernel lane now runs 3 shards on 1-gpu-small and 1 job on 1-gpu-large (was 1 and 2). - base-b: 84 files move; extra-a: 15 files move. Both are sized from est_time by compute_partitions.py (base-b: 5090 8 -> 13, H100 4 -> 2). - test_nvfp4_marlin was already registered on 1-gpu-small; drop its 1-gpu-large duplicate. - Raise est_time to the measured 5090 duration where the H100 estimate was more than 1.5x low (24 files). - Point new kernel correctness tests at 1-gpu-small in test/README.md and the write-sglang-test / add-jit-kernel skills; 1-gpu-large stays for SM90-only kernels. About 189 min of per-commit H100 est_time moves to the 5090 pool. Kept on 1-gpu-large: FA3 / FlashMLA / DeepGEMM / HPC-Ops / SM90-only kernels; weights over ~26 GB or two resident models; H100-tuned perf thresholds; int8_scaled_mm (no SM120 kernel); lmms_eval (not installed on Blackwell runners); and files that fail on SM120 (Triton shared- memory limits, FlashInfer top-k, dsv3_fused_a_gemm / tiny_gemm mismatches, EAGLE3 deterministic parity).
github-actions
Bot
added
documentation
mmangkad
marked this pull request as ready for review
September 25, 2026 07:19
mmangkad
requested review from
Fridge003,
HaiShaw,
Kangyan-Zhou,
bingxche,
ispobock,
merrymercy and
sundar24295s
as code owners
September 25, 2026 07:19
Contributor
Author
|
/tag-and-rerun-ci extra |
github-actions
Bot
added
run-ci
…rge-tests-to-5090 # Conflicts: # python/sglang/test/kernels/fp8.py # test/registered/attention/test_gdn_fused_split_head_ratios.py # test/registered/kernels/ops/attention/test_dsv4_q_rope_store.py # test/registered/kernels/ops/attention/test_fused_kda_conv_recurrent_verify.py # test/registered/kernels/ops/diffusion/test_complex_rope.py # test/registered/kernels/ops/dllm/test_gemma4_sampling_cuda.py # test/registered/kernels/ops/elementwise/test_row_scale.py # test/registered/kernels/ops/kimi_k3/test_kimi_k3_prerequisite_ops.py # test/registered/kernels/ops/kvcache/test_hicache.py # test/registered/unit/model_loader/test_modelopt_export.py
…u-small Re-run the 48 base-b, base-b-kernel-unit and extra-a files registered on 1-gpu-large since the last sweep on an RTX 5090 with the CI environment. - base-b-kernel-unit: 41 files move; test_convrot_int8 was already registered on 1-gpu-small, so drop its 1-gpu-large duplicate. - base-b: 4 files move (both test_fp8_kernel splits and the two rust_server test_mm_transport_1node cells); extra-a: test_lilicorr_cuda. - Raise est_time to the measured 5090 duration where the estimate was more than 1.5x low (9 files). - Kernel lane: 4 shards on 1-gpu-small (was 3) to keep each shard near 30 min of est_time. Kept on 1-gpu-large: test_deep_select (SM90/SM100/SM103 only, skips on SM120) and test_flashattention_return_lse (sgl-kernel FA3 is SM90-only).
Run all 67 base-b-kernel-benchmark files on an RTX 5090 with the pr-test-jit-kernel.yml env, and check each log for empty result tables as well as exit codes (marker.skip leaves a table with no rows). - 62 files move to a new base-b-kernel-benchmark-test-1-gpu-small job (25.8 min est_time); 1-gpu-large keeps 5 (4.8 min). - Raise est_time to the measured 5090 duration where the estimate was more than 1.5x low (24 files). - Point new benchmarks at 1-gpu-small in test/README.md and the write-sglang-test / add-jit-kernel / ci-workflow-guide skills. Kept on 1-gpu-large: bench_sparse_mla_q8kv8_prefill_sm90 (SM90a/SM100f only), bench_topk (TopKPageTableTransform not supported on SM120), bench_hicache (TMA path: CUDA invalid argument on SM120), bench_fp4_index_logits and bench_bf16xfp32_router_gemm (Hopper only, produce no rows elsewhere).
Follow-up to the benchmark move: write-sglang-test, add-jit-kernel and ci-workflow-guide live under .agents/skills (.claude/skills is a symlink).
Both pass on an RTX 5090 with the pr-test-multimodal-gen.yml env and compare against no GPU-specific ground truth or perf baseline. - bcg-diffusion job: 1-gpu-h100 -> 1-gpu-5090, suite renamed base-b-test-diffusion-bcg-1-gpu-5090 (2 passed, 2m21s). - test_sana_video2: diffusion-1-gpu-h100 -> diffusion-1-gpu-5090, so it runs in the existing 5090 canary job (2 passed, 2m44s; est_time 60 -> 165).
…n H100 The full multimodal_gen/test/unit suite passes on an RTX 5090 (4807 passed, 146 skipped, 0 failed, 30m). Eight cases only run on H100, so a new unit-h100 file suite keeps the three files that hold them on a short H100 job: - test_kandinsky6_bf16_dtype.py: FA backend cases (SM12.x resolves FA to SDPA) - test_mxfp8_online_gemm.py: pre-Blackwell channelwise fallback - test_subblock_sparse_attention.py: SM90 kernel cases, >=48 GiB free case multimodal-gen-unit-test now runs on 1-gpu-5090 (suite base-b-test-diffusion-unit-1-gpu-5090); multimodal-gen-unit-test-h100 runs base-b-test-diffusion-unit-1-gpu-h100 (unit-h100, ~25s of tests). The 5090 run also covers the SM120 SubBlock kernel and Blackwell MXFP8 cases.
…s on 1-gpu-small Move the portable classes of 9 base-b / extra-a files to 1-gpu-small and keep the classes that need an H100 in sibling files on 1-gpu-large: - FA3 (SM80/SM90 only): test_deterministic_fa3, test_spec_standalone_extra_fa3, test_page_major_gpt_oss_fa3, test_page_major_qwen_hybrid_fa3 - Weights over 32 GB: test_hicache_mla (DeepSeek-Coder-V2-Lite), test_spec_mixed_chunk_dspark (Qwen3-14B, FA3 off SM100), test_vision_openai_server_a_large (Qwen3-VL-30B-A3B, Kimi-VL-A3B), test_vlm_input_format_kimi (Kimi-VL-A3B) - H100-tuned throughput floor: test_srt_engine_throughput (the 5090 reaches ~2970 of 3000 tok/s) The 1-gpu-small halves pass on an RTX 5090 with the CI env; their only skips are static (XPU-only, temporarily disabled MiniCPM, GLM-4.1V nightly-only). est_time of those halves is the measured 5090 time. 1-gpu-large est_time: base-b 89.2 -> 65.5 min, extra-a 136.1 -> 128.5 min.
…-to-5090 # Conflicts: # test/registered/chunked_prefill/test_scripted_swa_1gpu.py # test/registered/kernels/ops/attention/test_flash_attention_4.py # test/registered/kernels/ops/attention/test_tilelang_dsa_zero_rope.py # test/registered/kernels/ops/diffusion/test_activation.py # test/registered/kernels/ops/diffusion/test_norm.py # test/registered/kernels/ops/elementwise/test_launch_carveout.py # test/registered/kernels/ops/speculative/test_ngram_utils.py # test/registered/spec/test_spec_ngram_extra.py
Re-run the 16 base-b, base-b-kernel-unit and extra-a files registered on 1-gpu-large since the previous sweep on an RTX 5090 with the CI environment. - base-b-kernel-unit: 13 files move; base-b: test_mamba_extra_buffer_spec_grammar; extra-a: test_mamba_cache_kl. - Raise est_time to the measured 5090 duration where the estimate was more than 1.5x low: test_qknorm_rope_out_of_place 60->136, test_watermark_selector 15->43, test_block_topk_triton 10->17. - test_compressed_tensors_fp4_numeric keeps est_time 30: 14s with a warm JIT cache (407s cold). Its 4 skipped FlashInfer cases need SM100 and also skip on H100. Kept on 1-gpu-large: test_q8kv8_dsa_nope_bf16_prefill (its kpool-width test runs the SM90-only WGMMA sparse prefill kernel).
This comment has been minimized.
This comment has been minimized.
mmangkad
force-pushed
the
mmangkad/move-1gpu-large-tests-to-5090
branch
from
October 17, 2026 09:10
4a2d2e2 to
ffe5f43
Compare
mmangkad
requested review from
AgainstEntropy,
BBuf,
kevin-mii,
mickqian,
niehen6174,
ping1jing2 and
yichiche
as code owners
October 17, 2026 09:10
mmangkad
changed the title
…ensors_fp4_numeric Main already registered it on both 1-gpu-small (est_time 15) and 1-gpu-large; the previous commit switched the 1-gpu-large line to 1-gpu-small, so the file was registered twice in base-b-kernel-unit. Keep main's 1-gpu-small line (14s measured on the 5090 with a warm JIT cache).
…nels on 1-gpu-large; set est_time from the 5090 reruns
This was referenced Nov 1, 2026
4 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Run every base-b, base-b-kernel-unit, base-b-kernel-benchmark and extra-a file registered on 1-gpu-large (H100) on an RTX 5090 (SM120, 32 GB) with the CI install and environment, and move the ones that pass without losing coverage. A file with tests skipped on the 5090 only moves when those tests also skip on H100 (ROCm-, SM100- or missing-op-gated). The diffusion bcg, SANA-Video 2 and unit lanes move to the 5090 under the same rule.
Per-commit 1-gpu-large est_time against main (
f75232cb42):About 289 min of per-commit H100 est_time moves to the 5090 pool, plus most of the diffusion unit lane (~30 min).
benchmark-test-1-gpu-smalljob.test_deterministic,test_hicache_variants,test_spec_mixed_chunk,test_spec_standalone_extra,test_page_major_gpt_oss,test_page_major_qwen_hybrid,test_vision_openai_server_a,test_vlm_input_format,test_srt_engine.test_sana_video2move to 1-gpu-5090. The unit suite runs on 1-gpu-5090; a newmultimodal-gen-unit-test-h100job keeps the three files with H100-only cases (test_kandinsky6_bf16_dtype,test_mxfp8_online_gemm,test_subblock_sparse_attention).test_nvfp4_marlin,test_convrot_int8andtest_compressed_tensors_fp4_numericwere already registered on 1-gpu-small; drop their 1-gpu-large duplicates.Kept on 1-gpu-large: FA3 / FlashMLA / DeepGEMM / HPC-Ops / SM90-only kernels; weights over ~26 GB or two resident models; H100-tuned perf thresholds; int8_scaled_mm (no SM120 kernel); lmms_eval (not installed on Blackwell runners); and files that fail on SM120 (Triton shared-memory limits, FlashInfer and deep-select top-k, dsv3_fused_a_gemm / tiny_gemm mismatches, EAGLE3 deterministic parity, the hicache TMA benchmark).
The H100-side est_time of the split-off sibling files is estimated, not measured.
CI States
Latest PR Test (Base): ❌ Run #39028310123
Latest PR Test (Extra): ❌ Run #39028309924
Latest PR Test (AMD ROCm 10): ❌ Run #39028309935