Optimize GLM 5.3 prefill and add backend-neutral metrics - #266
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the current prefill/tok/s tuning pass from #263 while keeping GLM B outside Ramjet routing and Qwen A unchanged.
What changed
chunked-prefill=6144,max-prefill-tokens=6144, andmax-total-tokens=500000context_frontier.pyreconcile backend-neutral request, prompt, cached-prompt, and generation counters for SGLang and vLLMLive result on node06
8K was rejected twice during built-in serving warmup: sparse attention needed a 256MiB output with 124.94MiB free at the full 524,288-token pool and 244.94MiB free at 500,000. Both failures occurred before readiness or benchmark traffic and stopped only GLM B.
6K/500K became ready in 1,036s and passed its smoke. Initial text shapes that produced late Triton loads were treated only as warmup. Subsequent clean, guarded, exactly reconciled cells measured:
The accepted c4 cell completed 20/20 requests and reconciled all 5,120 generated tokens. Intake was 42C; peak GLM GPU temperature was 62C in c4 and 73C in the long-context frontier. The accepted setting gives up 24,288 shared token slots (4.6%), so the recipe and journal state that trade explicitly.
The promoted Compose was staged without recreating the already-matching live engine. GLM B remains loopback/LB-isolated on GPUs 4-5; Qwen A remains healthy and unchanged; GPUs 6-7 remain free.
Engine research
Validation
cargo fmt --checkcargo clippy --locked --all-targets --all-features -- -D warningscargo test --lockedcargo build --release --lockedpython3 bench/agentbench.py validatePYTHONPATH=bench python3 -m unittest discover -s bench -p 'test_*.py'(723 tests)python3 deploy/glm53_flash_sm120/validate-compose.pybash -n deploy/glm53_flash_sm120/node06-prefill-6144-rollout.shgit diff --check