Skip to content

Fix SM120 NVFP4 KV cache test OOM - #31653

Merged
Fridge003 merged 3 commits into
sgl-project:mainfrom
Fridge003:codex/fix-nvfp4-memory-headroom
Jul 18, 2026
Merged

Fridge003 merged 3 commits into
sgl-project:mainfrom
Fridge003:codex/fix-nvfp4-memory-headroom

Conversation

@Fridge003

@Fridge003 Fridge003 commented Jul 18, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • move the SM120 Llama NVFP4 KV-cache test from models_e2e to quant
  • use GSM8KMixin with DefaultServerBase for one server configuration instead of sweeping through run_combined_tests
  • keep the test registered in extra-a on the 1-gpu-small runner
  • lower mem_fraction_static from the default 0.885 to 0.87, preserving the test's 200-request concurrency while leaving about 0.47 GiB more runtime headroom on a 32 GiB RTX 5090
  • temporarily disable the registration because the test is currently failing its accuracy threshold

The test only covers the NVFP4 GEMM + NVFP4 KV-cache configuration; it does not add an MTP variant.

Root cause

The test maximizes its NVFP4 KV-cache allocation under the default static-memory fraction. During the first ModelOpt FP4 GEMM in FlashInfer autotuning, CUDA then has only 28.44 MiB free when FlashInfer requests another 32 MiB, so server startup fails and the test harness later reports exit code -9.

This is a narrow CI containment for the newly added test; it does not change production allocator behavior. The test remains disabled until its separate accuracy failure is resolved.

Failing job: https://github.com/sgl-project/sglang/actions/runs/29624032862/job/88035809117

Validation

  • pre-commit run --files test/registered/quant/test_llama8b_nvfp4_kv_cache_sm120.py
  • git diff --check
  • verified the CI registration checker accepts the Extra registration and disabled reason

The full end-to-end test requires an SM120 GPU and is intentionally not scheduled while disabled.


CI States

Latest PR Test (Base): ❌ Run #29636016544
Latest PR Test (Extra): ❌ Run #29636016460

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@github-actions github-actions Bot added the blackwell SM100/SM120 label Jul 18, 2026
@Fridge003

Copy link
Copy Markdown
Collaborator Author

/rerun-test test/registered/models_e2e/test_llama8b_nvfp4_kv_cache_sm120.py

@github-actions

github-actions Bot commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Results for /rerun-test test/registered/models_e2e/test_llama8b_nvfp4_kv_cache_sm120.py:

🚀 1-gpu-5090 (1 test): ❌ View workflow run

cd test/ && python3 registered/models_e2e/test_llama8b_nvfp4_kv_cache_sm120.py

@Fridge003

Copy link
Copy Markdown
Collaborator Author

/rerun-test test/registered/quant/test_llama8b_nvfp4_kv_cache_sm120.py

@github-actions

github-actions Bot commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Results for /rerun-test test/registered/quant/test_llama8b_nvfp4_kv_cache_sm120.py:

🚀 1-gpu-5090 (1 test): ❌ View workflow run

cd test/ && python3 registered/quant/test_llama8b_nvfp4_kv_cache_sm120.py

@Fridge003
Fridge003 marked this pull request as ready for review July 18, 2026 07:48
@Fridge003
Fridge003 merged commit 38b29dc into sgl-project:main Jul 18, 2026
100 of 113 checks passed
Zhylkaaa pushed a commit to Zhylkaaa/sglang that referenced this pull request Jul 29, 2026
jinzhenfan pushed a commit to jinzhenfan/sglang that referenced this pull request Jul 29, 2026
Chronostasys pushed a commit to MindLab-Research/sglang that referenced this pull request Aug 24, 2026
jakki-amd pushed a commit to jakki-amd/sglang that referenced this pull request Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

blackwell SM100/SM120

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant