Repository navigation
feat(bench): add GLM-5.3-Flash FP8 and NVFP4 AgentX on GB300 / feat(bench): 新增 GB300 上的 GLM-5.3-Flash FP8 与 NVFP4 AgentX - #4715
functionstackx wants to merge 17 commits into
Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
functionstackx
added
the
full-sweep-fail-fast
functionstackx
changed the title
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| dp-attn: false | ||
| decode: { num-worker: 1, tp: 16, dcp-size: 16, ep: 1, dp-attn: false } | ||
|
|
||
| glm5.3flash-fp8-gb300-sglang-agentic-mtp: |
There was a problem hiding this comment.
🔴 The new GLM-5.3-Flash GB300 lane will never have results collected after merge, because the PR adds no perf-changelog.yaml entry. The new benchmarked key glm5.3flash-fp8-gb300-sglang-agentic-mtp changes what runs. AGENTS.md:87 requires a changelog entry for each affected config key, and that entry is what triggers result collection. Fix: append one block at the tail of inferencex-e2e/perf-changelog.yaml with config-keys: [glm5.3flash-fp8-gb300-sglang-agentic-mtp]. Keep every existing byte unchanged, since the file is append-only (AGENTS.md:88).
Why this was flagged
The diff adds a new master-config key glm5.3flash-fp8-gb300-sglang-agentic-mtp at nvidia-master.yaml:8823. It also adds the recipe glm5.3flash/sglang/gb300-fp8-mtp/agentic.yaml, which python -m infx.matrix.changed would list as affected. AGENTS.md:87 says a change that alters what any benchmark runs needs a perf-changelog.yaml entry for the affected config keys, and that "the entry triggers result collection". grep glm5.3flash inferencex-e2e/perf-changelog.yaml finds nothing, and the diff does not touch that file. After merge, no result collection is triggered for the new lane, so no GLM-5.3-Flash GB300 AgentX results are produced or published. The base branch has no such lane, so the PR's stated goal is not delivered. No other mechanism replaces the changelog trigger.
Verification: The trigger is merging the PR as it stands. The diff adds the new benchmarked key glm5.3flash-fp8-gb300-sglang-agentic-mtp at inferencex-e2e/configs/nvidia-master.yaml:8823. It does not touch inferencex-e2e/perf-changelog.yaml. So this new lane gets no PR sweep and no result collection after merge.
There was a problem hiding this comment.
Addressed in commit 505ac3a after the reviewed commit: one append-only block now lists glm5.3flash-fp8-gb300-sglang-agentic-mtp and this PR URL. Changelog validation passed. GPU jobs remain queued; this is not a GPU or canary pass.
已在审查所基于的提交之后，通过提交 505ac3a 追加唯一的 changelog 区块，包含该配置键和本 PR URL。Changelog 校验已通过。GPU 任务仍在排队，尚不能声称 GPU 或 canary 已通过。
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=42026173061 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=42026173061 |
|
/use 41308011021 |
|
@Oseltamivir staging run 41308011021. Existing staged runs will be preserved; a completion link will be posted here. |
Single-node SGLang v0.5.22 TP4 MTP 5/1/6 lanes for FP8 and NVFP4 using the SGLang cookbook GB300 recipe: GPU-resident KV at low concurrency and an auto-sized HiCache host tier (host memory fraction 0.90) at high concurrency.
Oseltamivir
changed the title
Oseltamivir
added
full-sweep-fail-fast
Pin lmsysorg/sglang:v0.5.23-cu130 for the FP8 and NVFP4 GB300 lanes and declare the expected SGLang version and FP8 checkpoint revision in the recipe identities.
Oseltamivir
added
full-sweep-fail-fast
Return both GB300 lanes to lmsysorg/sglang:v0.5.22-cu130 and declare SGLang 0.5.22 in the recipe identities.
Oseltamivir
added
full-sweep-fail-fast
|
/use 42026173061 |
|
@Oseltamivir staged run 42026173061.
This run remains available across future |
# Conflicts: # inferencex-e2e/perf-changelog.yaml
Set dram-utilization to 0.7495 (723 GB per node) on both GB300 lanes, covering the 708.7-722.9 GB per node the auto-sized HiCache pools allocated in run 42026173061.
Summary
This PR adds both GLM-5.3-Flash AgentX MTP lanes for one GB300 tray (TP4/EP1,
lmsysorg/sglang:v0.5.22-cu130@sha256:e3a4f06001a29542fc4a8638536d9ffc5eb9c9942b08603c09b19a56eae4a1ae):glm5.3flash-fp8-gb300-sglang-agentic-mtpzai-org/GLM-5.3-Flashglm5.3flash-fp4-gb300-sglang-agentic-mtpnvidia/GLM-5.3-Flash-NVFP4It replaces four experimental GB300 PRs: FP8 base (this PR), FP8 HiCache #4732, NVFP4 base #4716 and NVFP4 HiCache #4728.
Each precision forms one dashboard curve:
These concurrency points cover the throughput-versus-p90-E2EL Pareto frontier seen in those earlier runs. Points dropped as marginal or dominated: c2, c12, c56, c64 and c80. FP8 c2 and c12 sit on the interactivity frontier but add little next to c1, c4, c8 and c16. NVFP4 c1 is kept as the low-concurrency endpoint even though the earlier NVFP4 c2 point dominated it. MTP uses the GLM-5.3-Flash golden curve on main,
glm-5.3-flash__mtp__native__default-default(MTP5 = 3.60).Hidden admission cap: points stop at c48. Neither recipe sets
--max-running-requests. When the flag is unset under EAGLE, SGLang setsmax_running_requests=48(_handle_eagle_familyin v0.5.22speculative_hook.py). The server log of run 41308011021 showsmax_running_requests=48with no flag set. Requests above 48 queue instead of running. In the earlier runs, c56–c80 lost throughput and E2EL grew quickly, so no point above c48 is submitted.Recipe
Both precisions follow the SGLang cookbook GLM-5.3-Flash GB300 cells:
The served flags are
--tp-size 4,--dsa-prefill-backend trtllm,--dsa-decode-backend trtllm,--kv-cache-dtype fp8_e4m3,--moe-runner-backend flashinfer_trtllm, and EAGLE with--speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6. Parsers areauto(the cookbook default overlay). NVFP4 adds--quantization modelopt_fp4.expert-parallel-size 1andcontext-length 1048576equal the defaults; the model'smax_position_embeddingsis 1048576.The tuning flags in the earlier NVFP4 recipe are removed:
mem-fraction-static,max-running-requests,cuda-graph-max-bs-decode, chunked/max prefill sizes, loader options andenable-cache-report. The explicitSGLANG_NVFP4_CKPT_FP8_NEXTN_MOE=0andSGLANG_GLM_NEXTN_MOE_PTPC=0are also removed; both default toFalsein v0.5.22environ.py.trust-remote-codeis removed because it is a no-op for the nativeGlm5NextForConditionalGeneration.Deviation from the cookbook: HiCache sizing. The cookbook's HiCache option is
--enable-hierarchical-cache --hicache-size 32. This PR uses--enable-hierarchical-cache --hicache-host-memory-fraction 0.90instead, which is SGLang's auto-sizer at the fraction measured in #4732 and #4728.--hicache-size 32caps all host pools (KV, Mamba and DSA indexer) at 32 GB per rank in total (_split_hicache_size). The FP8 device pool is 103.94 GB KV plus 9.45 GB indexer per rank (18.6M tokens). v0.5.22 warns when a host pool is smaller than the device pool ("L2 cache effectiveness is reduced"), so a 32 GB tier adds almost no capacity on GB300.write_through,kernelI/O andpage_firstlayout equal the image defaults, so they are not set.CPU DRAM.
dram-utilization: 0.7495records 723 GB per node (int(920,000 MiB × 1.048576 × 0.7495) of 964.7 GB), the maximum the auto-sized HiCache pools allocated in run 42026173061. The auto-sizer follows each node's free host memory, so allocation varies by node. TP4 uses all four GPUs on the tray, so the MODELS.md proportional allowance is the whole node.The run was ingested with the earlier 0.7386 setting (712 GB per node). That stored value will be corrected to 723 GB through an InferenceX-app changelog backfill; the run is not repeated. Under the cookbook NVFP4 recipe the device KV pool is larger than in #4728, so the NVFP4 host/device ratio is 0.95 even at fraction 0.90.
Draft heads are served as shipped.
exclude_modules, so it ships BF16. It loads at 3.93 GB per rank through the image default (speculative_draft_model_quantizationauto, not set explicitly).Expected curves
Some earlier measurements are only estimates because the recipes changed.
The expected frontier is 7 points on throughput vs p90 E2EL and 9 on interactivity, per precision. The fresh sweep below replaces these estimates.
Chosen points and their sources (total tok/s per GPU, p90 E2EL)
The earlier runs record empty-stream
InvalidInferenceResultErrorrequests at every point (about 4–8% of records). Reported throughput excludes them.Harness-only settings (not serving optimizations)
These settings are identical in both recipes:
served-model-name,watchdog-timeout: 1800,enable-metrics.PYTHONNOUSERSITE,PYTHONUNBUFFERED,HF_HUB_OFFLINE,SGLANG_TIMEOUT_KEEP_ALIVE=900.time_limit,health_check.AIPERF_HTTP_TCP_USER_TIMEOUT,AIPERF_REQUIRED_SERVER_METRIC_PREFIX,KV_OFFLOADING.Both recipes declare
identityfor the runtime provenance check: the checkpoint revision (FP846a050be…, the revision the GB300 runtime fingerprint of run 41352868017 reported; NVFP41d007386…) andframeworks.sglang: 0.5.22.AIPERF_APPLY_CHAT_TEMPLATEis not set.runners.yamladds the stagedGLM-5.3-FlashandGLM-5.3-Flash-NVFP4checkpoints to thegb300-nvcluster.AI model disclosure
claude-opus-5-5(Claude Opus 5.5) consolidated the four GB300 PRs into this one: point selection, cookbook alignment, recipes, config, changelog, this description, and sweep monitoring.中文
本 PR 在单个 GB300 托盘（TP4/EP1，
lmsysorg/sglang:v0.5.22-cu130）上新增 GLM-5.3-Flash 的两条 AgentX MTP 配置：FP8（zai-org/GLM-5.3-Flash）和 NVFP4（nvidia/GLM-5.3-Flash-NVFP4）。它取代四个实验性 GB300 PR：FP8 基础（本 PR）、FP8 HiCache #4732、NVFP4 基础 #4716 和 NVFP4 HiCache #4728。每种精度构成一条曲线：并发 1、4、8、16、24、32 使用 GPU 内 KV，并发 40、44、48 使用 HiCache（CPU DRAM KV 卸载）。边际或被支配的 c2、c12、c56、c64、c80 已删除（FP8 的 c2、c12 位于交互性前沿上但增益很小；保留的 NVFP4 c1 是被支配点，作为低并发端点保留）。MTP 使用 main 上的 golden 曲线glm-5.3-flash__mtp__native__default-default（MTP5 = 3.60）。隐藏的准入上限：配方未设置
--max-running-requests，SGLang 在 EAGLE 下未设置该参数时通用地默认取max_running_requests=48（v0.5.22speculative_hook.py中的_handle_eagle_family，并非针对该模型或 GB300），超过 48 的请求只会排队，因此测试点止于 c48。配方遵循 SGLang cookbook 的 GLM-5.3-Flash GB300 单元（FP8 Low Latency；NVFP4 使用 NVDA 单元，额外设置
--quantization modelopt_fp4）。删除了旧 NVFP4 配方中的调优参数、两个 NextN 环境变量（v0.5.22 默认均为 False）以及无作用的trust-remote-code。与 cookbook 的偏差：cookbook 的 HiCache 为
--hicache-size 32，在 v0.5.22 中这是每个 rank 所有主机池合计 32 GB，远小于约 113 GB/rank 的设备 KV 池，v0.5.22 会警告这会降低 L2 效果。本 PR 改用 SGLang 自动分配，--hicache-host-memory-fraction 0.90（#4732/#4728 实测值）：#4732 的 c48 作业每 rank 分配 178.13 GB（KV 128.08 + Mamba 41.79 + 索引器 8.26），比例 1.12。FP8 在 c48 从 59.4k tok/s/GPU、p90 E2EL 33.7 s 提升到 63.8k、28.6 s。记录的 CPU DRAM 为每节点 723 GB（dram-utilization: 0.7495，节点 964.7 GB），即 run 42026173061 中自动分配的最大值；六个 HiCache 作业实测每节点 708.68–722.88 GB（FP8 708.68/712.00/722.88，NVFP4 720.00/722.16/722.24），随节点空闲内存变化。该 run 入库时记录为 712 GB，将通过 InferenceX-app 的 changelog backfill 更正为 723 GB，不重新运行。NVFP4 在 cookbook 配方下设备 KV 池更大，fraction 0.90 时主机/设备比例为 0.95。推测头按原样服务：FP8 的 NextN 为 FP8；NVFP4 的 MTP 层（第 45 层）在
exclude_modules中，以 BF16 发布并按镜像默认加载。未设置任何推测头精度覆盖。预期曲线中 FP8 无卸载点为精确值；HiCache 点（fraction 0.90）、NVFP4 全部点与插值点均为估计值（配方已改变）；新的完整 sweep 结果将取代这些估计。