perf(glm5next): accelerate C4 prefill - #496
Open
voipmonitor wants to merge 4 commits into
Open
Conversation
Gather the packed FP8 page-tail cache into caller-owned workspace for DeepGEMM prefill scoring while retaining the B12X paged kernel for decode. Preserve exact host-side prefill sequence lengths so each request gathers only its visible pool prefix. Compatibility: the GLM cache layout and decode route are unchanged. The prefill selector uses the same DeepGEMM and vLLM top-k contract as the qualified functional runtime. Validation: 51 GLM model tests passed on SM120; packed-cache DeepGEMM top-k matched the B12X oracle; 4096-row measurements were 6.5-24.1% faster than B12X prefill for 4096-32768 pool contexts. Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>
Assign each completed C4 pool to an independent Triton program and update each request tail separately. Decode and speculative-decode requests retain ordered row processing, while prefill requires packed main-cache slots and consecutive positions. Validated with GLM-5.3-Flash-NVFP4 on four RTX PRO 6000 Blackwell GPUs using a 32k-token standalone-prefill benchmark and parity tests for boundary tails, state-slot isolation, and invalid dummy slots. Signed-off-by: Martin Vit <martin@voipmonitor.org>
Assert the per-token QSA tail independently from fixed page padding and provide the decode-only metadata field required by selector-state fixtures. The tests now preserve the runtime-sized page ABI shared with B12X instead of requiring the superseded padded allocation. Validation: 80 targeted GLM, sparse-attention, and workspace tests passed; 14 accelerator-dependent tests skipped. Assisted-by: OpenAI Codex <codex@openai.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>
Pass the decode-request count as an append-only argument to the C4 prefill pool kernel so positional launch parameters cannot shift when the common argument tuple changes. Exercise the DeepGEMM and B12X ranking oracle with 576 visible candidates and top-k 512, including multiple packed cache pages. Validation: the GLM pooled-indexer suite and the SM120 DeepGEMM/B12X oracle cover the supported page-tail and top-k contracts. Assisted-by: OpenAI Codex <codex@openai.com> Signed-off-by: Martin Vit <martin@voipmonitor.org>
|
Warning Review limit reachedNext included review available in 59 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (6)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This was referenced Aug 28, 2026
Author
|
@coderabbitai review |
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resulting behavior
Status: implemented and qualified for GLM-5.3 C4 prefill at tensor parallel size 4 and decode-context parallel size 1.
GLM C4 decode retains the B12X paged selector. C4 prefill gathers the visible packed FP8 page-tail cache into caller-owned workspace and uses DeepGEMM to rank candidates. Host-side sequence lengths bound each request to its visible pool prefix.
Completed C4 pools are updated by independent Triton programs. Each request updates its own partial tail, while decode and speculative-decode rows retain ordered processing. The prefill path requires packed main-cache slots and consecutive positions.
Source contract
local-inference-lab/vllm:dev/jovian-judgementatc79f35ca00e8e93e0943a0d79b85b22b18aac939.70bdc4d51e571b75d7e1a7e1721a488b35aadb83.Validation
git diff --check: passed.Duplicate-work check
Related pull requests implement different execution modes:
local-inference-lab/vllm#488gathers full compressed KV state for GLM decode-context-parallel prefill and overlaps DCP communication. This pull request qualifies DCP1 packed-cache gathering, DeepGEMM ranking, and parallel C4 pool updates.vllm-project/vllm#41834contains DeepSeek V4 SM120 sparse-indexer and prefill work, but it does not implement the GLM C4 page-tail gather and pool-update contract in this pull request.Review disclosure
OpenAI Codex assisted with implementation, tests, GPU oracle validation, profiling, benchmarking, and pull-request preparation. Human review of every changed line and the packed-cache ABI is required before merge.