[Klaud Cold] glm5.2-fp4-rtx6000pro-sglang-agentic: add GLM-5.2 SGLang AgentX recipe for RTX PRO 6000 / 新增 GLM-5.2 RTX PRO 6000 SGLang AgentX 基准配置

[Oseltamivir]

Summary

Adds the SGLang counterpart to the RTX PRO 6000 vLLM recipes (#2312 Qwen3.5, #2332 GLM-5.2): a GLM-5.2 NVFP4 AgentX trace-replay sweep on the 8× RTX PRO 6000 Blackwell (SM120) Latitude node, TP8/EP1 with GPU-resident FP8 KV cache at concurrency 1, 2 and 4.

New config key: glm5.2-fp4-rtx6000pro-sglang-agentic, image lmsysorg/sglang:v0.5.15.post1-cu130 (the same image as the merged glm5.2-fp4-b300-sglang-agentic recipe).

What SM120 forces this recipe to do differently

Every item below was established on the node, not inferred:

1. TP8 only. The checkpoint loads to 56 GB/GPU (433 GB total), so pure TP over all eight 96 GB GPUs is the only layout with room for a KV pool. TP4 cannot hold the weights and attention-DP would replicate the pool per rank; both are rejected by the script.
2. --disable-shared-experts-fusion is mandatory. SGLang enables shared-experts fusion for this config (n_routed_experts=256, n_shared_experts=1, EP off), but nvidia/GLM-5.2-NVFP4 stores the shared expert loose and unquantized (BF16 [2048, 6144]) while the routed experts are packed NVFP4 ([2048, 3072]). Weight load aborts with RuntimeError: The size of tensor a (3072) must match the size of tensor b (6144) at non-singleton dimension 1 — the same trap the in-tree comment records for the compressed-tensors Kimi-K2.5 checkpoint. Disabling fusion loads cleanly.
3. The sparse DSA attention backend cannot run on SM120, so this recipe uses the Triton MLA fallback. SGLang builds the DSA indexer's paged-MQA-logits metadata exclusively through DeepGEMM (dsa_paged_mqa_logits_backend accepts only deepgemm on CUDA; cutedsl is gated to SM100), and DeepGEMM aborts during warmup with tvm.error.InternalError: Assertion error (attention.hpp:227): Unsupported architecture. Verified still unfixed in v0.5.16-cu130. Consequence: GLM-5.2's sparse-attention advantage is not realized on this GPU family — prefill throughput decays with context as dense attention would (measured on-node, single stream: 872 tok/s at 4.4k → 580 tok/s at 100k → 418 tok/s at 244k).
4. SGLang's DeepseekV4-only SM120 kernel fixups are deliberately not carried. SGLang switches off FP8 weight-only GEMM, topk_v2, tilelang MHC prenorm and DeepGEMM HC prenorm (and forces torch paged-MQA logits) inside its DeepseekV4ForCausalLM branch only, which GlmMoeDsaForCausalLM misses. Both variants were booted on the node: startup and generation are identical, so the recipe stays minimal and the script header records where to look if the DSA path is ever revisited.
5. Context is capped at 256k, not the model's 1M, and the recipe replays the 256k-capped trace corpus. The FP8 KV pool is 464,768 tokens per rank at mem-fraction-static 0.85 (~51 KB/token across 78 layers), so 1M-token sessions cannot be resident and concurrency above 4 re-prefills continuously.

On-node validation

• TP8 boot on lmsysorg/sglang:v0.5.15.post1-cu130: weights loaded (56.04 GB/GPU), KV pool 464,768 tokens/rank (23.91 GB), CUDA graphs captured, /health 200.
• Short-prompt correctness: 17 * 24 → 408 with clean reasoning_content/content split through the glm45/glm47 parsers — no NaN or garbage from the SM120 flashinfer_cutlass NVFP4 MoE path (the failure mode of [Bug] NVFP4 models produce NaN outputs on RTX PRO 6000 Blackwell (SM120) sgl-project/sglang#18954).
• Long-context retrieval: a needle placed in engineering-log filler was returned exactly at 45,394 tokens (depth 0.5, answer also cited the correct log line), 155,552 tokens (depth 0.75) and 220,029 tokens (depth 0.9) with the server at 256k context length.
• Decode ran under CUDA graphs at ~28 tok/s per request with two ~110k-token sessions resident (56 tok/s aggregate), and the KV pool reached 0.79 occupancy with the radix cache holding prior turns — the basis for stopping the sweep at concurrency 4.

Can the sparse path be enabled instead? Attempted on-node — no

Because the caveat above decides how these numbers should be read, the sparse path was actually tried on the node rather than assumed dead. Patching SGLang so the indexer's paged-MQA logits go through its TileLang or torch implementations (both already used by the dsv4 indexer) and so the DeepGEMM schedule plan is no longer built unconditionally does clear the abort, and the server boots and serves. Four further walls follow:

1. TileLang's CUDA sparse-MLA kernel accepts bf16 KV only — its fp8 variants are ROCm-only — which halves the pool to 256,256 tokens/rank.
2. That kernel requests 170,048 B of dynamic shared memory against SM120's ~100 KB. Its tile size is not a knob: the barrier arrive_counts (384/256/128) encode the block_I=64 thread mapping, so block_I=32 compiles and then reads out of bounds.
3. The plain (v1) kernel does fit at block_I=32 / num_stages=1 / 128 threads, but CUDA-graph capture fails — TileLang JIT-compiles inside the capture (cudaErrorStreamCaptureUnsupported).
4. With graphs disabled it runs and is numerically wrong: degenerate repetition on a 26-token prompt, empty answer on the 45k-token needle — identically with the TileLang and the torch reference logits kernels, which localises the fault to the sparse-MLA kernel rather than the indexer.

Closing this needs an SM120-shaped sparse-MLA kernel (split the d_v=512 accumulation so KV tiles stay under ~32 KB, re-derive the barrier counts, validate against the dense path) — kernel work, not configuration. The recipe header records this so the next reader does not re-burn a sweep on the flag.

Reading these numbers against vLLM: PR #2332 serves the same checkpoint on the same node with --attention-backend B12X_MLA_SPARSE, i.e. vLLM does get sparse MLA on SM120 through B12X. A lower SGLang result here is engine kernel coverage, not a hardware or recipe regression.

Also included (shared with #2312 / #2332)

runners/launch_rtx6000pro-lat.sh, the rtx6000pro* / cluster:rtx6000pro-lat runner labels and hardware metadata, and the benchmark-tmpl.yml workspace-ownership repair for this direct-host runner are duplicated from those PRs because neither has merged yet. The launcher here additionally resolves framework-tagged script names (<model>_<precision>_rtx6000pro_<framework>[_mtp].sh, falling back to the untagged name) so SGLang and vLLM recipes can coexist on this runner. Whichever of the three PRs merges first, the others should drop the duplicated files on rebase.

中文说明

本 PR 为 RTX PRO 6000 增加 SGLang 侧的配方，与 #2312（Qwen3.5, vLLM）、#2332（GLM-5.2, vLLM）互补：在 8× RTX PRO 6000 Blackwell（SM120）Latitude 节点上运行 GLM-5.2 NVFP4 的智能体（AgentX）轨迹回放扫描，TP8/EP1，KV 缓存全部驻留 GPU（FP8），并发 1、2、4。

新增配置项 glm5.2-fp4-rtx6000pro-sglang-agentic，镜像 lmsysorg/sglang:v0.5.15.post1-cu130（与已合并的 glm5.2-fp4-b300-sglang-agentic 相同）。

SM120 带来的差异（均在节点上实测得出）

1. 仅支持 TP8。 权重加载后每卡占用 56 GB（共 433 GB），只有跨全部八张 96 GB GPU 的纯 TP 才留得下 KV 池；TP4 装不下权重，注意力 DP 会在每个 rank 复制 KV 池——脚本对两者都会直接报错退出。
2. 必须加 --disable-shared-experts-fusion。 SGLang 在该配置下（n_routed_experts=256、n_shared_experts=1、EP 关闭）会启用共享专家融合，但 nvidia/GLM-5.2-NVFP4 的共享专家是松散存放且未量化的 BF16 [2048, 6144]，而路由专家是打包后的 NVFP4 [2048, 3072]，加载权重时会报 3072 与 6144 的形状不匹配。这与代码注释中记录的 compressed-tensors 版 Kimi-K2.5 是同一个坑。关闭融合后加载正常。
3. 稀疏 DSA 注意力后端在 SM120 上跑不起来，本配方改用 Triton MLA 回退路径。 SGLang 的 DSA indexer 只能通过 DeepGEMM 构建 paged-MQA-logits 元数据（CUDA 上 dsa_paged_mqa_logits_backend 仅接受 deepgemm，cutedsl 限定 SM100），而 DeepGEMM 在预热阶段直接断言 Unsupported architecture；已确认最新的 v0.5.16-cu130 仍未修复。因此 GLM-5.2 的稀疏注意力优势在该 GPU 系列上无法体现：单流预填充吞吐随上下文增长而下降（实测 4.4k 时 872 tok/s，100k 时 580 tok/s，244k 时 418 tok/s），与稠密注意力的表现一致。
4. 有意不沿用 SGLang 中仅针对 DeepseekV4 的 SM120 内核修正开关。 SGLang 只在 DeepseekV4ForCausalLM 分支内关闭 FP8 weight-only GEMM、topk_v2、tilelang MHC prenorm、DeepGEMM HC prenorm 并强制走 torch paged-MQA logits，而 GlmMoeDsaForCausalLM 拿不到这些设置。两种配置都在节点上实际启动过：启动与生成结果完全一致，因此配方保持精简，并在脚本头部记录了后续若重新尝试 DSA 路径时该查看哪些开关。
5. 上下文上限设为 256k（而非模型的 1M），并改用 256k 截断版轨迹语料。在 mem-fraction-static 0.85 下 FP8 KV 池为每 rank 464,768 个 token（78 层合计约 51 KB/token），1M token 的会话无法常驻，并发超过 4 就会持续重新预填充。

节点验证

• 在 lmsysorg/sglang:v0.5.15.post1-cu130 上完成 TP8 启动：权重每卡 56.04 GB，KV 池每 rank 464,768 token（23.91 GB），CUDA graph 捕获完成，/health 返回 200。
• 短提示正确性：17 * 24 → 408，glm45/glm47 解析器正确区分 reasoning_content 与 content，SM120 上的 flashinfer_cutlass NVFP4 MoE 路径未出现 NaN 或乱码（即 [Bug] NVFP4 models produce NaN outputs on RTX PRO 6000 Blackwell (SM120) sgl-project/sglang#18954 的故障模式）。
• 长上下文检索：在工程日志式填充文本中埋入 needle，在 45,394 token（深度 0.5，回答还准确引用了对应日志行号）、155,552 token（深度 0.75）与 220,029 token（深度 0.9）下均被准确取回，服务端上下文上限为 256k。
• 两个约 110k token 的会话常驻时，解码在 CUDA graph 下约 28 tok/s/请求（合计 56 tok/s）；radix 缓存保留历史轮次后 KV 池占用达到 0.79——这也是扫描并发止步于 4 的依据。

能否改用稀疏路径？已在节点上尝试——不行

由于上述限制直接决定了这些数据该如何解读，我们并没有想当然地认为稀疏路径不可用，而是实际在节点上做了尝试。给 SGLang 打补丁，让 indexer 的 paged-MQA logits 走它自带的 TileLang 或 torch 实现（这两者 dsv4 的 indexer 已在使用），并且不再无条件构建 DeepGEMM 调度计划，确实可以消除那个断言，服务也能正常启动并对外提供推理。但后面还有四道坎：

1. TileLang 的 CUDA 稀疏 MLA 内核只接受 bf16 KV（fp8 变体仅 ROCm 有），这会把 KV 池减半到每 rank 256,256 个 token。
2. 该内核申请 170,048 B 动态共享内存，而 SM120 上限约 100 KB。其分块大小并不是一个可自由调节的参数：屏障的 arrive_count（384/256/128）已经把 block_I=64 的线程映射写死，所以改成 block_I=32 虽然能编译，但会越界访问。
3. 朴素版（v1）内核在 block_I=32 / num_stages=1 / 128 线程下确实能放下，但 CUDA graph 捕获失败——TileLang 会在捕获过程中做 JIT 编译（cudaErrorStreamCaptureUnsupported）。
4. 关闭 CUDA graph 后可以跑起来，但数值结果是错的：26 token 的短提示出现退化重复，45k token 的 needle 测试返回空答案；使用 TileLang 与 torch 参考 logits 内核的表现完全一致，说明问题出在稀疏 MLA 内核本身，而不是 indexer。

要彻底解决，需要一个适配 SM120 的稀疏 MLA 内核（把 d_v=512 的累加拆分，使 KV 分块保持在约 32 KB 以内，重新推导屏障计数，并与稠密路径做数值对齐验证）——这属于内核开发工作，而非配置调整。脚本头部已记录这一结论，避免后续读者再为这个开关浪费一次扫描。

与 vLLM 结果的对比方式： PR #2332 在同一节点、同一权重上使用 --attention-backend B12X_MLA_SPARSE，即 vLLM 通过 B12X 在 SM120 上确实跑的是稀疏 MLA。因此这里 SGLang 数据偏低反映的是推理引擎的内核覆盖差异，而不是硬件问题或配置退化。

与 #2312 / #2332 重复的文件

runners/launch_rtx6000pro-lat.sh、rtx6000pro* 与 cluster:rtx6000pro-lat 运行器标签及硬件元数据、以及针对该直连主机运行器的 benchmark-tmpl.yml 工作区属主修复，都因为那两个 PR 尚未合并而在此重复提交。本 PR 的启动器额外支持带框架后缀的脚本名（<model>_<precision>_rtx6000pro_<framework>[_mtp].sh，找不到时回退到不带后缀的名字），以便 SGLang 与 vLLM 配方在同一运行器上共存。三个 PR 中任一先合并后，其余应在 rebase 时删除重复文件。

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[claude]

🔴 When MODEL_PATH is unset — which it always is on this runner, since neither benchmark-tmpl.yml nor runners/launch_rtx6000pro-lat.sh set it (unlike launch_b300-nv.sh) — the else branch pins --revision $MODEL_REVISION only for the hf download step, then sets MODEL_PATH to the bare repo id nvidia/GLM-5.2-NVFP4. SGLang is later launched with --model-path $MODEL_PATH and no --revision flag anywhere in SGLANG_CMD, so it resolves main at server start rather than the pinned aec724e8... commit, silently defeating the reproducibility pin in the only code path CI actually exercises. Fix: add --revision "$MODEL_REVISION" to SGLANG_CMD (SGLang's launch_server supports it) or point MODEL_PATH at the resolved local snapshot directory.

Extended reasoning...

The bug. glm5.2_fp4_rtx6000pro_sglang.sh introduces a deliberate reproducibility pin: MODEL_REVISION="${GLM52_MODEL_REVISION:-aec724e8c7b8ee9db3b48c01c320f63f9cdaf8aa}". This pin is honored for the download step in both branches of the if [[ -n "${MODEL_PATH:-}" ]] conditional (lines 69-78), but only the if branch keeps MODEL_PATH pointed at a concrete local directory that was populated at that pinned revision. The else branch — taken when MODEL_PATH is unset — runs hf download \"$MODEL\" --revision \"$MODEL_REVISION\" (correctly fetching the pinned snapshot into the shared HF cache) and then does export MODEL_PATH=\"$MODEL\", i.e. sets MODEL_PATH to the bare HF repo id nvidia/GLM-5.2-NVFP4, with no revision attached to it at all.\n\nFurther down, SGLANG_CMD launches the server with --model-path \"$MODEL_PATH\" and, scanning the full argument list, there is no --revision flag anywhere. When --model-path is a bare repo id rather than a local directory, SGLang/huggingface_hub resolves the revision to the default main ref at load time — a live hub lookup, since HF_HUB_OFFLINE is not set anywhere in this recipe. So the pin governs what gets downloaded, but not what gets served: the server loads whatever main currently points to, which only coincidentally matches the pinned commit today.\n\nWhy this is not a theoretical edge case. The else branch is not a fallback for unusual manual runs — it is the only branch ever exercised in CI on this runner. benchmark-tmpl.yml's env block never sets MODEL_PATH. runners/launch_rtx6000pro-lat.sh only forwards --env MODEL_PATH to the container (i.e. passes through whatever is in the parent environment, which is nothing) — it never assigns a value, unlike the sibling launch_b300-nv.sh, which explicitly exports MODEL_PATH to a staged local directory under /data/models. So on the RTX PRO 6000 runner, MODEL_PATH is always unset, the else branch always runs, and the server is always launched against the unpinned bare repo id.\n\nStep-by-step proof:\n1. CI invokes this script with no MODEL_PATH in the environment (confirmed absent from both benchmark-tmpl.yml's env block and launch_rtx6000pro-lat.sh).\n2. [[ -n \"${MODEL_PATH:-}\" ]] is false → the else branch runs: hf download nvidia/GLM-5.2-NVFP4 --revision aec724e8c7b8ee9db3b48c01c320f63f9cdaf8aa, correctly fetching the pinned snapshot into $HF_HUB_CACHE.\n3. export MODEL_PATH=\"nvidia/GLM-5.2-NVFP4\" — the bare repo id, no revision suffix.\n4. SGLANG_CMD is built with --model-path \"nvidia/GLM-5.2-NVFP4\" and, checked against the full flag list in the script, no --revision argument exists to pin the load.\n5. python3 -m sglang.launch_server ... starts; since --model-path isn't a local directory, SGLang/transformers resolve revision "main" against the live hub (no HF_HUB_OFFLINE), and loads whatever commit main currently points to.\n6. Today, main happens to equal aec724e8..., so results are currently correct and nothing crashes — the bug is latent. But the moment upstream pushes a new commit to main for nvidia/GLM-5.2-NVFP4, step 2's download (still pinned) and step 5's load (unpinned) diverge: the server would silently serve a different checkpoint than the one benchmarked and validated in this PR's own on-node testing, potentially re-downloading a second 433 GB snapshot on top of the one already cached.\n\nWhy nothing else in the script catches this. There's no assertion after the else branch that the resolved MODEL_PATH/revision pair matches what was downloaded, and no --revision is threaded through to SGLANG_CMD from either branch — the pin is a write-only variable as far as the server launch is concerned.\n\nFix. The simplest correction is to add --revision \"$MODEL_REVISION\" to SGLANG_CMD (SGLang's launch_server accepts this flag) so the load path is pinned the same way the download path is. Alternatively, resolve MODEL_PATH to the actual local snapshot directory that hf download populates (e.g. via hf download --local-dir in the else branch too, mirroring the if branch) so --model-path always points at pinned, already-materialized content rather than a bare repo id resolved at load time.\n\nThis undermines the explicit reproducibility intent of the recipe (the B300 sibling recipe doesn't pin a revision at all, so this pin was clearly added deliberately for this config) in the exact code path that every real CI run on this runner takes, which is why I'm marking it normal rather than nit despite the fact that it's currently harmless while upstream main hasn't moved.

[claude]

🔴 SGLANG_CMD in this recipe never passes --kv-cache-dtype fp8_e4m3, so SGLang falls back to its default bf16 KV cache instead of the FP8 cache the recipe's header comments, nvidia-master.yaml, and the PR description all assume (464,768 tokens/rank in 23.91 GB, ~51 KB/token). The sibling glm5.2_fp4_b300_sglang.sh:118 recipe this was copied from sets the flag explicitly in the same low-latency branch — add --kv-cache-dtype fp8_e4m3 to SGLANG_CMD here too.

Extended reasoning...

The SGLANG_CMD array built at benchmarks/single_node/agentic/glm5.2_fp4_rtx6000pro_sglang.sh:172-194 never sets --kv-cache-dtype. SGLang has no auto-fp8 default for KV — omitting the flag makes it use the model dtype (bf16) for the MLA latent KV cache. This is confirmed by grepping the rest of the repo: every other recipe that relies on an FP8 KV pool sets the flag explicitly (glm5.2_fp4_b300_sglang.sh:118, dsv4_fp4_mi355x_sglang.sh:140, the srt-slurm and AMD models.yaml recipes, etc.). I verified directly that the B300 sibling this script was copied from does set --kv-cache-dtype fp8_e4m3 inside its non-DP ("STP") low-latency branch — the exact branch this RTX PRO 6000 recipe descends from — so the flag was dropped in the copy rather than being an intentional change.

The rest of the recipe is built entirely around the assumption that the KV cache is FP8. The script's own header comment, the new nvidia-master.yaml entry, and the PR description all state the "FP8 KV pool" holds 464,768 tokens per rank (23.91 GB) at mem-fraction-static 0.85, i.e. ~51 KB/token across GLM-5.2's 78 layers. That number is only consistent with an FP8 cache: GLM-5.2's MLA state is kv_lora_rank 512 + rope 64 = 576 elements/token/layer, which comes to roughly 44 KB/token in FP8 across 78 layers (close to the ~51 KB/token quoted, with indexer/page overhead) — 23.91 GB / 464,768 tokens = ~54 KB/token, again matching FP8. In bf16 the same state would be roughly double, ~88-90 KB/token, giving a pool of only ~230k-285k tokens/rank in the same 23.91 GB, not 464,768. So the on-node validation that produced 464,768 tokens/rank and the "needle retrieval at 220,029 tokens" / "two ~110k-token sessions resident" results in the PR description must have been run with --kv-cache-dtype fp8_e4m3 set — the committed script does not reproduce that configuration.

Concretely, if this ships as-is: the CI sweep runs a materially different (bf16-KV) server than the one that was validated and documented. The actual pool would be roughly half the size the sizing math assumes, which invalidates both the 256k context cap (--context-length 262144) and the concurrency-4 cutoff — a bf16 pool of ~230k-285k tokens/rank is close to or below the 262144-token context length, so a single full-length 256k session may not even fit resident, let alone the two ~110k sessions the recipe uses to justify stopping at concurrency 4. That risks continuous re-prefilling or startup/capacity failures that the documented validation did not encounter, undermining the primary purpose of this PR (recording believable on-node capacity numbers for this hardware).

To fix: add --kv-cache-dtype fp8_e4m3 to the SGLANG_CMD array, matching glm5.2_fp4_b300_sglang.sh:118, and re-validate that the pool size, context cap, and concurrency cutoff in the header/nvidia-master.yaml/PR description still hold under the corrected server config.

Step-by-step proof:

1. Read glm5.2_fp4_rtx6000pro_sglang.sh:172-194 — SGLANG_CMD contains --quantization modelopt_fp4, --attention-backend triton, --mem-fraction-static, etc., but no --kv-cache-dtype anywhere in the array or in any exported env var.
2. Grep the codebase for --kv-cache-dtype in other FP8-KV recipes — it is always passed explicitly (e.g. glm5.2_fp4_b300_sglang.sh:118), confirming SGLang requires an explicit flag rather than auto-detecting FP8 KV from the weight quantization.
3. Compute expected FP8 KV bytes/token for GLM-5.2 MLA: (512 + 64) elements * 78 layers * 1 byte (fp8) ≈ 44 KB/token, and cross-check against the documented 23.91 GB / 464,768 tokens ≈ 51.4 KB/token — both are in the FP8 regime.
4. Compute the bf16 equivalent: same element count in 2 bytes ≈ 88 KB/token, which in 23.91 GB yields ≈ 278k tokens/rank — roughly half of 464,768, and dangerously close to the 262144-token --context-length setting used by this recipe.
5. Conclude that the shipped script (bf16, missing flag) cannot reproduce the documented/validated 464,768-token pool, invalidating the capacity claims that justify the 256k cap and conc-4 cutoff.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30205556138
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30205556138

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=30205556138
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=30205556138

[cquil11]

@Oseltamivir the AgentX/AIPerf harness has been updated, please merge origin/main into your branch and refresh your submission. Additional tuning may be necessary depending on the config. I apologize for any inconvenience.

This is an automated message.