Repository navigation
Add GLM-5.2-NVFP4 functional-test recipe for GKE g4 (RTX PRO 6000 / SM120) - #25
Merged
Merged
Conversation
…M120) SGLang standalone + NVIDIA Dynamo aggregated serving for nvidia/GLM-5.2-NVFP4 on g4-standard nodes. Includes a self-contained Dockerfile (FlashInfer 0.6.13 + DeepGEMM nv_dev + sglang PR #26928 applied at build), k8s yamls, a functional smoke-test script, and warm functional benchmark results. Companion to ds-v4-pro-nvfp4. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Re-benched standalone and Dynamo without --disable-radix-cache: smoke checks pass, decode metrics unchanged (Dynamo TPOT identical), TTFT gains are cache hits on repeated bench prompts. Radix cache is now on by default; the flag is documented as benchmark-only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… result Standalone and Dynamo at parity: 121 vs 127 tok/s out, median TTFT 127 vs 129 ms. The cache-disabled A/B is summarized in one line as validation of the default. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Re-measured both paths with complete bench_serving output: per-run tables (duration, throughput, concurrency, TTFT/TPOT/ITL/E2E median+P99, GPU memory) plus a fresh-prompt check — standalone 112.88 vs Dynamo 112.91 tok/s, parity to 0.03%. Summary tables updated to the re-measured run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- README: add Prerequisites (per ds-v4-pro pattern); drop companion line, config.json note, and the unverified cutedsl claim; keep trtllm_routed SM100-only (evidenced in ds-v4-pro). - Dense-MHA short-prefill branch: SGLang's use_mha gate already restricts it to SM90/SM100 at the pinned commit, so it cannot fire on SM120 — removed from the failure table and demoted the threshold env to a defensive no-op (kept in the image). - RESULTS.md: drop date, one topic per line in the header. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- FlashInfer: 15a2459 is a post-0.6.13 main commit containing flashinfer#3395 (SM120 sparse-MLA kernels, merged after the 0.6.13 release cut) — release wheels up to 0.6.13 lack the kernels; docs now say source-built at the commit instead of calling it the 0.6.13 release. - trtllm/flashmla row: SM90/SM100-only (FlashMLA supports SM90/SM100, not just SM100). - DeepGEMM row re-verified against the upstream README (SM90/SM100 only). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… empty, not disabled) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
inference/gke/dynamo/g4/glm-5.2-nvfp4/— a functional-test recipe for servingnvidia/GLM-5.2-NVFP4on GKEg4-standardnodes(8× RTX PRO 6000 Blackwell, SM120) with SGLang standalone and NVIDIA Dynamo (aggregated).
Stock SGLang cannot serve GLM-5.2's DeepSeek Sparse Attention on SM120: DeepGEMM releases have no
SM120 build, the
trtllm/flashmlaDSA kernels are SM90/SM100-only, and FlashInfer release wheelsup to 0.6.13 lack the SM120 sparse-MLA kernels (see sgl-project/sglang#26087). This recipe combines
the three public fixes:
flashinfer_sparse_mlaDSA backend) — applied from theincluded diff at image build until it merges
15a2459(post-0.6.13main, which includesfeat(attention): add SM120 sparse MLA kernels flashinfer-ai/flashinfer#3395 — the SM120 sparse-MLA kernels) +
flashinfer-cubin/jit-cache0.6.13nv_dev@a6b593d(SM120-capable indexer)plus
--disable-shared-experts-fusion(explained in the README). Earlier SM120 enablement work insgl-project/sglang#29586 was the starting point for this investigation.
Contents
README.md— root cause, fix, prerequisites, build/deploy/validate walkthroughDockerfile+pr26928.diff— self-contained image build with off-GPU preflightstandalone-sglang-glm52-nvfp4.yaml/dgd-sglang-glm52-nvfp4.yaml— prefix caching on by defaultsmoke-test.sh— 4 deterministic functional checks (factual, arithmetic, ~3600-token needle,instruction following)
bench-results/RESULTS.md— warm benchmark with per-run detail, standalone vs Dynamo (parity)Validation
Deployed and validated on GKE
g4-standard(8× RTX PRO 6000): all smoke checks pass on both servingpaths; warm benchmark 16/16 requests on each — standalone 123.6 tok/s out / median TTFT 121 ms vs
Dynamo 127.0 tok/s / 123 ms (parity; identical token counts). Fresh-prompt check: 112.88 vs
112.91 tok/s — parity to 0.03%. Prefix caching A/B-validated on both paths (correctness and
per-token decode latency unchanged vs
--disable-radix-cache).🤖 Generated with Claude Code