[Fridge003]

MLA currently defaults prefill CUDA graph capture to 2048 tokens even when chunked prefill is larger. Default MLA capture to min(8192, cfg.chunked_prefill_size) to expand coverage while limiting capture memory growth.

Non-MLA models retain the existing default of cfg.chunked_prefill_size. In particular, Muse MegaMoE + LayerNorm SP with a 16384-token chunk keeps a 16384-token capture ceiling, satisfying its requirement that captured workspaces cover the largest prefill chunk.

For MLA, chunked prefill sizes of 4096 and 16384 default to capture maxima of 4096 and 8192, respectively. This applies when prefill max_bs is unset; explicit settings and existing downstream caps retain their behavior.

Validation:

• Executed the actual default-selection blocks and bucket generator in a standalone CPU harness: 16 cases passed across MLA/non-MLA, chunk sizes 4096/8192/16384/32768, explicit max_bs, and existing PP/max-total-tokens/Llama-2 caps. The non-MLA 16384 case failed on the previous revision and passes after this change. This is a focused configuration check, not a full server startup test.
• Python AST parsing, git diff --check, and all applicable pre-commit hooks passed.
• A prior GLM-5.3-Flash TP4/DFlash2/no-CP sweep on 4 x GB300 completed explicit 8192-token capture on all four ranks (12.16 GiB graph pool reserved per GPU). That experiment used fixed chunked prefill 16384, mem_fraction_static 0.60, and FlashInfer 0.7.1; it did not test this latest default-selection change or reproduce the B200 CI environment.
• GPU runtime tests were not rerun for this default-selection update.

---

CI States

Latest PR Test (Base): 🚫 Run #39735173853
Latest PR Test (Extra): ❌ Run #39735173847
Latest PR Test (AMD ROCm 10): ⏳ Run #39735173846

[Fridge003]

Extra test pass https://github.com/sgl-project/sglang/actions/runs/39573398566

[QNan0513]

@Fridge003 here is a B300 data point on the MLA prefill-graph cap from this PR.

On GPUs with at least 160 GB, chunked_prefill_size defaults to 16384, so min(8192, chunked_prefill_size) keeps every prefill forward of 8,193–16,384 tokens on the eager path. With GLM-5.3-Flash and the cookbook B300 FP8 Low Latency command (TP8, MTP 5/1/6, breakable prefill graphs), that covers every full chunk of a long prompt and every batch of short prompts that packs past 8,192 tokens. In two default-config runs, all 570 such batches logged cuda graph: False.

I compared the default against --cuda-graph-max-bs-prefill 16384 on 8x B300 (main 797af717, four servers in the order default, 16384, 16384, default):

• Median TTFT: 16k x 1 -19.7%, 32k x 1 -14.4%, 2k x 8 -28.7%, 4k x 8 -34.8%, 8k x 8 -39.7%. The 4k x 1 and 8k x 1 controls did not move.
• Short-request batches improved by about the same amount: 1k x 16 -39.7%, 512 x 32 -35.7%. In eager mode these forwards qualify for DSA's dense MHA one-shot path (every kv_len is at most 2048), while inside the graph DSA forces use_mha=False. The graph still came out ahead.
• Mixed prefill + decode: output throughput +13.9% at 16k x 8 and +9.0% at 4k x 16. Decode TPOT stayed within restart noise, and the MTP accept length was about 3.4 in both.
• Accuracy: GSM8K 0.931 vs 0.928; passkey retrieval at 16k and 30k tokens was 20/20 in both.
• Cost: prefill graph memory per GPU went from 2.81 to 4.72 GB (2.99 to 5.16 GB with EP8), capture time from 51 s to 65–75 s, and the KV pool stayed at 23,208,960 tokens.

Since that is the only setup I measured, #48214 sets the flag in the B300 FP8 Low Latency recipe only. Two questions about the default:

1. The description says 8192 limits capture memory growth. Would you accept letting MLA follow chunked_prefill_size on GPUs that default to 16384-token chunks, or would you rather keep 8192 and set the cap per recipe?
2. The 4x GB300 TP4/DFlash2 run in the description reserved a 12.16 GiB graph pool at 8192, much more than the 2.81 GB above, so it is probably where 16384 costs the most. Is that the setup you would want measured before any default change? I don't have GB300 access.

Not tested: B200, GB200, GB300, NVFP4, the High Throughput cells, other MLA models, prefill CP, PD and HiCache.