Repository navigation
feat: fp8 KV cache recipe for SM121, expert parallelism in vLLM, reference benchmarks - #2
Merged
Merged
Conversation
…rence benchmarks
- recipes/flashnext-fp8kv-1m8.yaml: opt-in SGLang recipe with --kv-cache-dtype
fp8_e4m3 on the 2026-09-11 nightly. KV pool 900k to 1.8M tokens, context
110k to 262k, decode within control drift of bf16.
- mods/sglang-sm121-qsa-fp8kv: the SM121 QSA decode kernel is BF16-only and
dies on fp8 keys ("unsupported SM121 QSA call: expected BF16 D=256 ...").
The mod allocates the gather scratch in the query dtype so selected keys
convert on store; the cache stays fp8. Idempotent, fail-closed.
- mods/vllm-qsa-fp8kv-pr55557: vllm-project/vllm#55557 as a mod for current
vLLM nightlies. Not referenced by a shipped recipe until tool-eval repeats
clean.
- recipes/flashnext-vllm-cached.yaml: promote --enable-expert-parallel with
allgather_reducescatter, measured with no regression. recipe_version 2.
- scripts/gate_37111.py: corruption gate for sgl-project/sglang#37111 against
a live server; does not reproduce on the shipped recipe.
- scripts/validate_recipes.py: registry-scoped mods warn instead of error,
directory mods allowed, EP rule.
- results/RESULTS.md, README.md, mods/README.md: 2026-09-11 tables on the
tonyd2wild and MiaAI-Lab harnesses, drift band, fp8 caveats, new recipe.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
flashnext-fp8kv-1m8(new, opt-in, SGLang):--kv-cache-dtype fp8_e4m3on the 2026-09-11 nightly (51a0563d, carries fix(qsa): dequantize FP8 cached prefixes in the sparse prefill kernels sgl-project/sglang#38855) plus the mod below. KV pool 899,968 to 1,800,000 tokens, context 110k to 262k. Decode within the 5% control band of bf16 on every cell; tool-eval 97, same as bf16.mods/sglang-sm121-qsa-fp8kv: the only QSA decode kernel qualified for SM121 is BF16-only and kills the scheduler on the first decode with fp8 keys. The mod allocates the packed gather scratch in the query dtype at both call sites so the extraction converts on store. At most 2055 rows per sequence; cache stays fp8. Valid while KV scales are 1.0.mods/vllm-qsa-fp8kv-pr55557: [Model] Qwen4Exp: fp8_e4m3 main KV cache on the QSA path vllm-project/vllm#55557 applied as a mod so current vLLM nightlies accept fp8 main KV without the six whole-file overlays. 2,069,156 KV tokens, decode parity, but one tool-eval run at 90/100. Not wired into a recipe until it repeats clean.flashnext-vllm-cached:--enable-expert-parallel --all2all-backend allgather_reducescatterpromoted. +3-6% prefill, +4-19% decode at depth on 2026-09-09, no regression on 2026-09-11.recipe_version2.scripts/gate_37111.py: corruption gate for [Bug] Qwen3.8-Flash-Next QSA + NEXTN decode graph silently corrupts output on GB10 TP2 sgl-project/sglang#37111 (essay + 4-marker retrieval). Passes on the shipped digest.Numbers (tonyd2wild's 40-prompt harness, fresh boots)
Reference TP2 profile on the same harness: 53.7 / 97.9 at 1 / 6 streams.
Not done here
fp8 quality is checked only by the 15-scenario tool-eval and needle retrieval to 88k. bf16 stays the default until a 76-scenario run and a full-window needle ladder are measured on both. Details in
results/RESULTS.md.