Repository navigation
GLM-5.3-Flash NVFP4: DFlash2 drafter + CUDA graphs, verified (prose 28.2/21.1/17.1, structured 51.4/41.5/35.7 tok/s, 327k ctx) - #1
Merged
Conversation
…witch in run.sh Co-authored-by: Sufyan <sfxnz@users.noreply.github.com>
Co-authored-by: Sufyan <sfxnz@users.noreply.github.com>
Co-authored-by: Sufyan <sfxnz@users.noreply.github.com>
… stack v10: Glm5Next lacked the EAGLE3 interface DFlash taps target hidden states through; capture the completed layer output with the SGLang-verified mHC hc_post + stream-average contraction (sgl-project/sglang#36708). v11: the bespoke GLM-5-Next KV grouping rejected the draft's sliding-window specs, falling back to a generic path that cannot unify the indexer page. Give the draft a dedicated KV group at a quarter of the attention block size. run.sh grows an EXTRA_ARGS passthrough (e.g. --load-format dummy for fast boot iteration). Co-authored-by: Cursor <cursoragent@cursor.com>
Measured on the cluster: 26.8 / 21.6 / 17.0 tok/s per stream at c=1/2/4 (aggregate 26.8 / 41.3 / 67.6), beating MTP-4 at every concurrency with 25% more context. 5 speculative tokens instead of the block's 7: draft positions 5-6 accept <15% and each slot costs a per-sequence KDA state copy, which starved c=4 admission at 7. KV pin stays 4.14 GiB; raising it slows decode ~20% (UMA) or crashes under load. A 318k-token prompt (97% of the window) prefilled in 4m05s and answered correctly. bench_decode: keep a wave alive when individual streams fail. Co-authored-by: Cursor <cursoragent@cursor.com>
sfxnz
changed the title
sfxnz
marked this pull request as ready for review
August 28, 2026 18:35
The EXL3 recipe benches the high-acceptance regime; ours only measured prose. Same measurement code per phase, so prose numbers stay comparable. Default runs both phases; --phase selects one. Offline-tested against a fake SSE + /metrics server (16 checks). Eager baseline on the cluster: prose 27.6/20.0/17.1 per stream at c=1/2/4, structured 49.6/41.8/35.0 with acceptance_len 5.45 of 6 (draft accept 0.89 vs 0.40 prose). Co-authored-by: Cursor <cursoragent@cursor.com>
sm_121 caution held graphs off since v8. Measured on the cluster with DFlash2-5: FULL_AND_PIECEWISE resolves with 4 uniform-decode graphs at the 6/12/18/24-token verify shapes, capture takes 27 s and no measurable memory, KV pool stays 400,497 tokens. Two full benches: prose 28.2/21.1/17.1 vs eager 27.6/20.0/17.1, structured 51.4/41.5/35.7 vs 49.6/41.8/35.0, acceptance unchanged, no stability regressions. Greedy stayed lossless (c=4 counting probe exact for 100+ tokens; the one text divergence is a 0.125-nat near-tie that flips run-to-run in eager too). Co-authored-by: Cursor <cursoragent@cursor.com>
sfxnz
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Makes DFlash2 (incoai/GLM-5.3-Flash-DFlash2) the recipe's default drafter, replacing MTP-4, and turns on CUDA graphs, both with independently verified benchmarks.
docker/dflash2_backport.diff+Dockerfile.sm121-v9: vLLM PR #52816 (DFlash2) backported as an exact-context patch onto the sm_121 image (two upstream hunks misplace under fuzzy application; corrected here).docker/patch_v10_dflash_glm5.py+Dockerfile.sm121-v10: the fork-only Glm5Next model lacked the EAGLE3 aux-hidden-state interface DFlash taps target features through. Implements the capture with mHC hyper-connection semantics verified against the SGLang reference integration ([DFLASH] Support GLM-5.3-Flash hidden-state capture sgl-project/sglang#36708): materialize the deferredhc_post, average the streams.docker/patch_v11_dflash_kv_groups.py+Dockerfile.sm121-v11: the bespoke GLM-5-Next KV grouping rejected the draft's sliding-window specs and fell into a generic path that cannot unify the DSA indexer's 33 B/token page. Adds a dedicated draft KV group at a quarter of the attention block size (block-id burn vs tensor-bytes tradeoff).run.sh: DFlash2-5 default (SPEC=mtpreverts), 327680 context, CUDA graphs default (ENFORCE_EAGER=1reverts),EXTRA_ARGSpassthrough, draft-weight handling, and a fix for bash truncating brace-containing${VAR:-...}defaults.bench_decode.py: prose + structured phases (structured is the count-1-to-200 high-acceptance protocol the EXL3 recipe benches), spec-decode acceptance length per concurrency block from/metrics; survives individual stream failures. Parsing/stats offline-tested against a fake SSE server before use.README.md: measured numbers for both phases and both execution modes, tuning rationale, and the draft model's CC BY-NC-ND 4.0 license note.Verified numbers (2x DGX Spark, TP=2, 3-run medians, greedy, 200 tok, 327680 ctx)
CUDA graphs is the new default; eager measured the same day on the same boot pair.
Draft acceptance rate ~0.40 prose, ~0.89 structured. Graphs numbers held across two full benches (second: prose 28.3/21.8/17.1, structured 51.2/40.8/35.5).
CUDA graphs finding
Graphs were held off on sm_121 since the v8 image (PDL is gated off there because it races KDA state kernels; graphs themselves were unvalidated). Measured now: with capture sizes [1,2,4,6,12,18,24] the engine resolves FULL_AND_PIECEWISE, captures 4 uniform-decode graphs at the 6/12/18/24-token verify shapes (num_spec+1 x seqs) in 27 s with no measurable memory cost; KV pool stays 400,497 tokens. Greedy stayed lossless: a c=4 counting probe decoded 100+ exact sequential tokens per stream through graph replay, and the only eager-vs-graphs text divergence was a near-tie token (physics/science, 0.125-nat margin) that flips run-to-run within a single boot in either mode. Net +2-5% at c=1/2, flat at c=4, no stability regressions.
Tried and rejected: nothing this round; k=7 under graphs was considered and declined without a boot because positions 5-6 accept <15% (a drafter/target property graphs cannot change) and the per-sequence KDA state copies that starve c=4 admission at k=7 are likewise execution-mode independent.
Load-bearing constraints (all measured)
num_speculative_tokens=5beats 7 (positions 5-6 accept <15% and each slot costs a per-sequence KDA state copy); the 4.14 GiB KV pin must stay (5.0 GiB slows decode 20-30% under UMA pressure and wedged the host under benchmark load; 5.14 GiB crashes); KV pool 400,497 tokens (1.22x at 327680); a 318k-token prompt prefills in 4m05s and answers correctly.Future work (documented, not implemented)
Draft-KV slot-sharing of MLA pages, the EXL3 recipe's approach: their drafter SWA slot-shares the target's MLA pages, freeing the draft's pool cost entirely and enabling much larger context. Our v11 layer instead gives the draft a dedicated KV group with its own tensors (
docker/patch_v11_dflash_kv_groups.py). Porting slot-sharing onto the GLM5 bespoke KV layout would reclaim that pool share and is the next capacity lever.