Skip to content

Optimize GLM 5.3 prefill and add backend-neutral metrics - #266

Merged
nessie993 merged 1 commit into
mainfrom
ops/glm53-prefill-opt
Sep 12, 2026
Merged

nessie993 merged 1 commit into
mainfrom
ops/glm53-prefill-opt

Conversation

@nessie993

Copy link
Copy Markdown
Contributor

Closes the current prefill/tok/s tuning pass from #263 while keeping GLM B outside Ramjet routing and Qwen A unchanged.

What changed

  • promote the measured GLM SM120 defaults to chunked-prefill=6144, max-prefill-tokens=6144, and max-total-tokens=500000
  • add a guarded B-only transition owner with deployment locking, 600W inference-ceiling admission, Qwen A identity checks, deferred inference budget, and immediate dead-engine detection
  • make context_frontier.py reconcile backend-neutral request, prompt, cached-prompt, and generation counters for SGLang and vLLM
  • add forced-length decode support so per-stream decode samples cannot end early at EOS
  • record the power A/B, 8K rejection, clean 6K frontier/c4 results, and current upstream engine research

Live result on node06

8K was rejected twice during built-in serving warmup: sparse attention needed a 256MiB output with 124.94MiB free at the full 524,288-token pool and 244.94MiB free at 500,000. Both failures occurred before readiness or benchmark traffic and stopped only GLM B.

6K/500K became ready in 1,036s and passed its smoke. Initial text shapes that produced late Triton loads were treated only as warmup. Subsequent clean, guarded, exactly reconciled cells measured:

cell 4K 6K change
2K cold prefill 5,174 tok/s 5,174 tok/s 0.0%
8K cold prefill 5,415 tok/s 5,817 tok/s +7.4%
32K cold prefill 5,897 tok/s 5,961 tok/s +1.1%
64K cold prefill 5,884 tok/s 5,951 tok/s +1.1%
c4 aggregate output 370.7 tok/s 388.2 tok/s +4.7%
c4 per-stream decode 106.3 tok/s 109.4 tok/s +2.9%

The accepted c4 cell completed 20/20 requests and reconciled all 5,120 generated tokens. Intake was 42C; peak GLM GPU temperature was 62C in c4 and 73C in the long-context frontier. The accepted setting gives up 24,288 shared token slots (4.6%), so the recipe and journal state that trade explicitly.

The promoted Compose was staged without recreating the already-matching live engine. GLM B remains loopback/LB-isolated on GPUs 4-5; Qwen A remains healthy and unchanged; GPUs 6-7 remain free.

Engine research

Validation

  • cargo fmt --check
  • cargo clippy --locked --all-targets --all-features -- -D warnings
  • cargo test --locked
  • cargo build --release --locked
  • python3 bench/agentbench.py validate
  • PYTHONPATH=bench python3 -m unittest discover -s bench -p 'test_*.py' (723 tests)
  • python3 deploy/glm53_flash_sm120/validate-compose.py
  • bash -n deploy/glm53_flash_sm120/node06-prefill-6144-rollout.sh
  • git diff --check

@nessie993
nessie993 merged commit 811f797 into main Sep 12, 2026
1 check passed
@nessie993
nessie993 deleted the ops/glm53-prefill-opt branch September 12, 2026 13:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant