perf(glm5next): dispatch mHC by token batch size - #498
Conversation
Use B12X mHC for decode-sized rank-local batches, including CUDA graph padding, and TileLang mHC for larger prefill batches. The dispatch threshold derives from scheduler capacity, speculative depth, CUDA graph capacity, tensor parallelism, and sequence parallelism. Every mHC operation in a decoder layer uses the same selected backend. Validated with GLM-5.3-Flash-NVFP4 in W4A4 mode on four RTX PRO 6000 Blackwell GPUs. A 32k-token standalone-prefill benchmark sustained 13,280 prompt tokens/s for 30 seconds; concurrency-one decode without speculative tokens sustained 117.2 tokens/s. Signed-off-by: Martin Vit <martin@voipmonitor.org>
|
Warning Review limit reachedNext included review available in 59 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@coderabbitai review |
|
1 participant
Resulting behavior
Status: implemented and qualified with
local-inference-lab/vllm#493for GLM-5.3 tensor parallel size 4.Each GLM decoder layer selects B12X multipath hyperconnection for decode-sized rank-local token batches and TileLang multipath hyperconnection for larger prefill batches. The threshold accounts for scheduler sequence capacity, speculative depth, CUDA graph capture capacity, tensor parallelism, and sequence parallelism.
Every multipath-hyperconnection operation in one decoder-layer invocation uses the same backend, including deferred post-processing outside the decoder-layer forward call. Systems without a supported B12X multipath-hyperconnection implementation retain TileLang execution.
Source contract
local-inference-lab/vllm:dev/jovian-judgementatc79f35ca00e8e93e0943a0d79b85b22b18aac939.535f4def142f819a73597cd09bf357f7376d9b15.local-inference-lab/vllm#493, which retains B12X binding owners for the captured graph lifetime.Validation
tests/models/test_glm5next_model.py -k 'mhc or hyperconnection': 7 passed, 27 deselected.git diff --check: passed.local-inference-lab/vllm#493captured and replayed target-plus-DFlash2 graph families without B12X binding-storage reuse failures.Duplicate-work check
Open upstream and
local-inference-lab/vllmpull requests were searched for GLM multipath hyperconnection, B12X, TileLang, and token-batch dispatch. No open pull request implements this per-layer size dispatch.local-inference-lab/vllm#493is complementary: it provides graph resource lifetime and does not choose the GLM multipath-hyperconnection backend.Review disclosure
OpenAI Codex assisted with implementation, tests, profiling, benchmarking, and pull-request preparation. Human review of every changed line and the backend-selection threshold is required before merge.