Skip to content

[CI] base-b: Move kernel and unit tests that pass on the 5090 to 1-gpu-small - #44880

Open
mmangkad wants to merge 3 commits into
mainfrom
mmangkad/ci-5090-2-base-b-light
Open

mmangkad wants to merge 3 commits into
mainfrom
mmangkad/ci-5090-2-base-b-light

Conversation

@mmangkad

@mmangkad mmangkad commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Second of the series that splits #41231 into reviewable PRs, one CI lane each (first: #44876). This one moves 27 light base-b test files (kernels/ops, moe, unit, layers, mem_cache, model_loading, prefill_only) from 1-gpu-large (H100) to 1-gpu-small (RTX 5090). Every change is the runner_config of one register_cuda_ci line, with est_time set to the measured 5090 duration where the estimate was more than 1.5x low. test_nvfp4_marlin was already registered on 1-gpu-small (est_time 12, 11s measured); its 1-gpu-large duplicate is dropped.

Rule for a file to move: it passes on the 5090 (SM120, 32 GB) with the CI install and env, every test it skips on the 5090 also skips on H100, and where it launches a server, any architecture-dependent default it picks up (e.g. FA3 on H100 vs FlashInfer on SM120) is not what the test is about. No coverage is lost.

base-b per-commit est_time main this PR
1-gpu-large 129 files / 175.4 min 101 / 168.1 min
1-gpu-small 150 files / 191.9 min 177 / 205.8 min

Partitions are sized from est_time by compute_partitions.py, so no workflow change is needed. Independent of #44876 (no shared files).

Validation

All 30 candidate files passed on the 1-gpu-5090 runners at 8d58a33e16 with /rerun-test -c (run 39021620314), then compared with the same files on H100 in main's scheduled run:

  • test_mhc_kernels stays on 1-gpu-large: 33 passed, 16 skipped on the 5090 vs 49 passed on H100. The skipped cases are its test_hopper_* functions, which exercise the SM90-only mHC dispatch.
  • Same skips on both: test_topk_padded_region (3), test_fp8_utils (1). test_flash_mla_backends skips all 15 cases on H100 and runs 14 on the 5090.
  • test_post_capture_kv_sizing is the only file that starts a server; it gets FlashInfer on the 5090 instead of FA3. Post-capture KV sizing has no attention-backend-specific path, and the run confirms the sizing path ran.

Merged main 5e630fa571 into the branch: main deleted kernels/ops/gemm/test_fp8_kernel.py (#44690), so it is no longer part of this PR. Main changed 9 of the other files since the run above, so all 28 are being rerun on the 5090 with /rerun-test -c.

Follow-ups

extra-a, the round-4 class splits, the jit-kernel unit / benchmark lanes and diffusion follow as separate PRs; #41231 tracks them.


CI States

Latest PR Test (Base): ❌ Run #39360950121
Latest PR Test (Extra): ❌ Run #39360950155
Latest PR Test (AMD ROCm 10): ❌ Run #39360950180

@github-actions github-actions Bot added quant LLM Quantization hicache Hierarchical Caching for SGLang blackwell SM100/SM120 memory-pool labels Oct 17, 2026
@mmangkad
mmangkad marked this pull request as ready for review October 17, 2026 11:19
…se-b-light

# Conflicts:
#	test/registered/kernels/ops/gemm/test_fp8_kernel.py
@mmangkad

Copy link
Copy Markdown
Contributor Author

/rerun-test -c

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

🚀 1-gpu-5090 (28 tests): ✅ View workflow run

cd test/ && python3 registered/kernels/ops/attention/test_dcp_lse_combine.py
cd test/ && python3 registered/kernels/ops/attention/test_dsv32_indexer_fusion.py
cd test/ && python3 registered/kernels/ops/attention/test_dsv4_indexer_quant.py
cd test/ && python3 registered/kernels/ops/attention/test_flash_mla_backends.py
cd test/ && python3 registered/kernels/ops/gemm/test_nvfp4_marlin.py
cd test/ && python3 registered/kernels/ops/kvcache/test_hicache.py
cd test/ && python3 registered/kernels/ops/kvcache/test_hicache_page_first_write_back.py
cd test/ && python3 registered/kernels/ops/quantization/test_fp8_kernel.py
cd test/ && python3 registered/layers/test_layernorm_fusion.py
cd test/ && python3 registered/mem_cache/test_post_capture_kv_sizing.py
cd test/ && python3 registered/model_loading/test_utils_update_weights.py
cd test/ && python3 registered/moe/test_fused_append_remap_per_rank_shared_slots.py
cd test/ && python3 registered/moe/test_fused_append_shared_experts.py
cd test/ && python3 registered/moe/test_fused_append_shared_experts_top6.py
cd test/ && python3 registered/moe/test_fused_moe.py
cd test/ && python3 registered/moe/test_topk_padded_region.py
cd test/ && python3 registered/moe/test_topk_renormalize_degenerate.py
cd test/ && python3 registered/moe/test_triton_fused_moe.py
cd test/ && python3 registered/moe/test_triton_moe_channel_fp8_kernel.py
cd test/ && python3 registered/prefill_only/test_serving_rerank.py
cd test/ && python3 registered/unit/layers/moe/test_deepep_v2_masked_slab.py
cd test/ && python3 registered/unit/layers/quantization/test_fp8_utils.py
cd test/ && python3 registered/unit/mem_cache/pool_host/test_kvcacheio_asymmetric.py
cd test/ && python3 registered/unit/mem_cache/test_swa_unittest.py
cd test/ && python3 registered/unit/models/test_kimi_k3_bfa_overlap.py
cd test/ && python3 registered/unit/multimodal/test_cuda_ipc_transport.py
cd test/ && python3 registered/unit/multimodal/test_vit_cuda_graph_metadata_cuda.py
cd test/ && python3 registered/unit/spec/test_resolve_swa_kv_pool.py

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

blackwell SM100/SM120 hicache Hierarchical Caching for SGLang memory-pool quant LLM Quantization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant