Skip to content

[Klaud Cold] glm5.2-fp4-rtx6000pro-sglang-agentic: add GLM-5.2 SGLang AgentX recipe for RTX PRO 6000 / 新增 GLM-5.2 RTX PRO 6000 SGLang AgentX 基准配置 #2335

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Oseltamivir wants to merge 1 commit into from
Closed

[Klaud Cold] glm5.2-fp4-rtx6000pro-sglang-agentic: add GLM-5.2 SGLang AgentX recipe for RTX PRO 6000 / 新增 GLM-5.2 RTX PRO 6000 SGLang AgentX 基准配置 #2335

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .github/workflows/benchmark-tmpl.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -235,6 +235,13 @@ jobs:
done
fi

# The direct-host RTX runner mounts this checkout into a rootful
# container. Repair files left by an interrupted job before
# actions/checkout attempts its clean reset.
if [ "${{ runner.name }}" = "rtx6000pro-lat_00" ]; then
sudo -n chown -R "$(id -u):$(id -g)" "$GITHUB_WORKSPACE"
fi

# Cleanup SLURM resources
if command -v squeue >/dev/null 2>&1; then
echo "[Slurm] Cleaning up jobs with name: ${{ runner.name }} ..."
Expand Down
236 changes: 236 additions & 0 deletions benchmarks/single_node/agentic/glm5.2_fp4_rtx6000pro_sglang.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,236 @@
#!/usr/bin/env bash
set -euo pipefail
set -x

# AgentX trace replay for GLM-5.2 NVFP4 on the 8x RTX PRO 6000 Blackwell
# (SM120) node using SGLang.
#
# Flags start from the merged B300 NVFP4 cookbook recipe
# (benchmarks/single_node/agentic/glm5.2_fp4_b300_sglang.sh, STP only) and
# then apply the SM120 deltas this GPU family needs:
#
# * TP8 only. The checkpoint is 433 GB, so pure TP across all eight 96 GB
# GPUs is the only layout that leaves room for the KV pool; TP4 does not
# fit the weights at all.
# * Shared-experts fusion is force-disabled. SGLang enables it for this
# config (n_routed_experts=256, n_shared_experts=1, EP off), but
# nvidia/GLM-5.2-NVFP4 stores the shared expert loose and unquantized
# (BF16 [2048, 6144]) while the routed experts are packed NVFP4
# ([2048, 3072]), so the fused loader aborts during weight load with
# "The size of tensor a (3072) must match the size of tensor b (6144)".
# This is the same trap the in-tree comment records for the
# compressed-tensors Kimi-K2.5 checkpoint.
# * Attention falls back to Triton MLA. SGLang would default this DSA model
# to --attention-backend dsa, whose indexer metadata comes only from
# DeepGEMM, and DeepGEMM has no SM120 kernel. Sparse attention is
# therefore not exercised on this GPU family, and prefill cost grows
# superlinearly with context.
# * MoE runner backend is left at SGLang's own SM120 choice for
# modelopt_fp4 (flashinfer_cutlass; trtllm-gen MoE is SM100-only) and is
# overridable through MOE_RUNNER_BACKEND.
# * HiCache is not supported on RTX PRO 6000, so this recipe is
# GPU-resident KV only.
#
# Required env vars:
# MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR, DURATION,
# EP_SIZE, DP_ATTENTION

source "$(dirname "$0")/../../benchmark_lib.sh"

check_env_vars \
MODEL \
TP \
CONC \
KV_OFFLOADING \
TOTAL_CPU_DRAM_GB \
RESULT_DIR \
DURATION \
EP_SIZE \
DP_ATTENTION

if [[ "$TP" != "8" ]]; then
echo "GLM-5.2 SGLang on RTX PRO 6000 requires TP8: the 433 GB NVFP4 checkpoint does not fit in fewer than eight 96 GB GPUs" >&2
exit 1
fi
if [[ "$KV_OFFLOADING" != "none" ]]; then
echo "GLM-5.2 SGLang on RTX PRO 6000 supports GPU-resident KV cache only (HiCache is unsupported on this GPU family)" >&2
exit 1
fi
if [[ "$DP_ATTENTION" == "true" ]]; then
echo "GLM-5.2 SGLang on RTX PRO 6000 does not support DP attention: attention-DP replicates the KV pool per rank, which the 96 GB GPUs cannot hold alongside 54 GB of weights" >&2
exit 1
fi
if [[ "$EP_SIZE" != "1" ]]; then
echo "GLM-5.2 SGLang on RTX PRO 6000 supports EP1 only" >&2
exit 1
fi

# `hf download` creates the target dir if missing and is itself idempotent.
# When MODEL_PATH is unset (stand-alone runs), fall back to the HF_HUB_CACHE.
# Either way, MODEL_PATH is what the server is launched with.
MODEL_REVISION="${GLM52_MODEL_REVISION:-aec724e8c7b8ee9db3b48c01c320f63f9cdaf8aa}"
if [[ -n "${MODEL_PATH:-}" ]]; then
if [[ ! -d "$MODEL_PATH" || -z "$(ls -A "$MODEL_PATH" 2>/dev/null)" ]]; then
hf download "$MODEL" --revision "$MODEL_REVISION" --local-dir "$MODEL_PATH"
fi
else
hf download "$MODEL" --revision "$MODEL_REVISION"
export MODEL_PATH="$MODEL"

Check failure on line 78 in benchmarks/single_node/agentic/glm5.2_fp4_rtx6000pro_sglang.sh

View check run for this annotation

Claude / Claude Code Review

Pinned MODEL_REVISION is not honored at server launch (MODEL_PATH unset in production) 

When `MODEL_PATH` is unset — which it always is on this runner, since neither `benchmark-tmpl.yml` nor `runners/launch_rtx6000pro-lat.sh` set it (unlike `launch_b300-nv.sh`) — the else branch pins `--revision $MODEL_REVISION` only for the `hf download` step, then sets `MODEL_PATH` to the bare repo id `nvidia/GLM-5.2-NVFP4`. SGLang is later launched with `--model-path $MODEL_PATH` and no `--revision` flag anywhere in `SGLANG_CMD`, so it resolves `main` at server start rather than the pinned `aec7
Comment on lines +69 to +78

@claude claude Bot Jul 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 When MODEL_PATH is unset — which it always is on this runner, since neither benchmark-tmpl.yml nor runners/launch_rtx6000pro-lat.sh set it (unlike launch_b300-nv.sh) — the else branch pins --revision $MODEL_REVISION only for the hf download step, then sets MODEL_PATH to the bare repo id nvidia/GLM-5.2-NVFP4. SGLang is later launched with --model-path $MODEL_PATH and no --revision flag anywhere in SGLANG_CMD, so it resolves main at server start rather than the pinned aec724e8... commit, silently defeating the reproducibility pin in the only code path CI actually exercises. Fix: add --revision "$MODEL_REVISION" to SGLANG_CMD (SGLang's launch_server supports it) or point MODEL_PATH at the resolved local snapshot directory.

Extended reasoning...

The bug. glm5.2_fp4_rtx6000pro_sglang.sh introduces a deliberate reproducibility pin: MODEL_REVISION="${GLM52_MODEL_REVISION:-aec724e8c7b8ee9db3b48c01c320f63f9cdaf8aa}". This pin is honored for the download step in both branches of the if [[ -n "${MODEL_PATH:-}" ]] conditional (lines 69-78), but only the if branch keeps MODEL_PATH pointed at a concrete local directory that was populated at that pinned revision. The else branch — taken when MODEL_PATH is unset — runs hf download \"$MODEL\" --revision \"$MODEL_REVISION\" (correctly fetching the pinned snapshot into the shared HF cache) and then does export MODEL_PATH=\"$MODEL\", i.e. sets MODEL_PATH to the bare HF repo id nvidia/GLM-5.2-NVFP4, with no revision attached to it at all.\n\nFurther down, SGLANG_CMD launches the server with --model-path \"$MODEL_PATH\" and, scanning the full argument list, there is no --revision flag anywhere. When --model-path is a bare repo id rather than a local directory, SGLang/huggingface_hub resolves the revision to the default main ref at load time — a live hub lookup, since HF_HUB_OFFLINE is not set anywhere in this recipe. So the pin governs what gets downloaded, but not what gets served: the server loads whatever main currently points to, which only coincidentally matches the pinned commit today.\n\nWhy this is not a theoretical edge case. The else branch is not a fallback for unusual manual runs — it is the only branch ever exercised in CI on this runner. benchmark-tmpl.yml's env block never sets MODEL_PATH. runners/launch_rtx6000pro-lat.sh only forwards --env MODEL_PATH to the container (i.e. passes through whatever is in the parent environment, which is nothing) — it never assigns a value, unlike the sibling launch_b300-nv.sh, which explicitly exports MODEL_PATH to a staged local directory under /data/models. So on the RTX PRO 6000 runner, MODEL_PATH is always unset, the else branch always runs, and the server is always launched against the unpinned bare repo id.\n\nStep-by-step proof:\n1. CI invokes this script with no MODEL_PATH in the environment (confirmed absent from both benchmark-tmpl.yml's env block and launch_rtx6000pro-lat.sh).\n2. [[ -n \"${MODEL_PATH:-}\" ]] is false → the else branch runs: hf download nvidia/GLM-5.2-NVFP4 --revision aec724e8c7b8ee9db3b48c01c320f63f9cdaf8aa, correctly fetching the pinned snapshot into $HF_HUB_CACHE.\n3. export MODEL_PATH=\"nvidia/GLM-5.2-NVFP4\" — the bare repo id, no revision suffix.\n4. SGLANG_CMD is built with --model-path \"nvidia/GLM-5.2-NVFP4\" and, checked against the full flag list in the script, no --revision argument exists to pin the load.\n5. python3 -m sglang.launch_server ... starts; since --model-path isn't a local directory, SGLang/transformers resolve revision "main" against the live hub (no HF_HUB_OFFLINE), and loads whatever commit main currently points to.\n6. Today, main happens to equal aec724e8..., so results are currently correct and nothing crashes — the bug is latent. But the moment upstream pushes a new commit to main for nvidia/GLM-5.2-NVFP4, step 2's download (still pinned) and step 5's load (unpinned) diverge: the server would silently serve a different checkpoint than the one benchmarked and validated in this PR's own on-node testing, potentially re-downloading a second 433 GB snapshot on top of the one already cached.\n\nWhy nothing else in the script catches this. There's no assertion after the else branch that the resolved MODEL_PATH/revision pair matches what was downloaded, and no --revision is threaded through to SGLANG_CMD from either branch — the pin is a write-only variable as far as the server launch is concerned.\n\nFix. The simplest correction is to add --revision \"$MODEL_REVISION\" to SGLANG_CMD (SGLang's launch_server accepts this flag) so the load path is pinned the same way the download path is. Alternatively, resolve MODEL_PATH to the actual local snapshot directory that hf download populates (e.g. via hf download --local-dir in the else branch too, mirroring the if branch) so --model-path always points at pinned, already-materialized content rather than a bare repo id resolved at load time.\n\nThis undermines the explicit reproducibility intent of the recipe (the B300 sibling recipe doesn't pin a revision at all, so this pin was clearly added deliberately for this config) in the exact code path that every real CI run on this runner takes, which is why I'm marking it normal rather than nit despite the fact that it's currently harmless while upstream main hasn't moved.

fi

if [[ -n "${SLURM_JOB_ID:-}" ]]; then
echo "JOB $SLURM_JOB_ID running on ${SLURMD_NODENAME:-unknown}"
fi
nvidia-smi
nvidia-smi topo -m || true

# The FP8 KV pool holds 464,768 tokens per rank (~51 KB/token across 78
# layers) at mem-fraction-static 0.85, so the model's 1M context does not fit
# and the server is capped at 256k. Replay the matching 256k-capped corpus
# instead of the 1M default this model prefix would otherwise select.
export WEKA_LOADER_OVERRIDE="${WEKA_LOADER_OVERRIDE:-semianalysis_cc_traces_weka_062126_256k}"
resolve_trace_source
install_agentic_deps

SERVER_LOG="$RESULT_DIR/server.log"
mkdir -p "$RESULT_DIR"

export PYTHONNOUSERSITE=1
export TORCH_CUDA_ARCH_LIST=12.0a
# All eight GPUs are SYS-connected (PCIe only, no NVLink) on this node, and
# NCCL 2.28.9 segfaults probing its bnxt_re devices; the runner already sets
# NCCL_IB_DISABLE=1. Keep collectives on the local PCIe/SHM transports.
export NCCL_P2P_LEVEL="${NCCL_P2P_LEVEL:-SYS}"
export NCCL_PROTO="${NCCL_PROTO:-LL,LL128,Simple}"
export GLOO_SOCKET_IFNAME="${GLOO_SOCKET_IFNAME:-lo}"
export NCCL_SOCKET_IFNAME="${NCCL_SOCKET_IFNAME:-lo}"
export OMP_NUM_THREADS="${OMP_NUM_THREADS:-16}"

# NOTE for whoever revisits the DSA path: SGLang carries a set of SM120
# kernel fixups (SGLANG_OPT_FP8_WO_A_GEMM / SGLANG_OPT_USE_TOPK_V2 /
# SGLANG_OPT_USE_TILELANG_MHC_PRE / SGLANG_OPT_DEEPGEMM_HC_PRENORM off,
# SGLANG_FP8_PAGED_MQA_LOGITS_TORCH on) that it applies only inside its
# DeepseekV4ForCausalLM branch, and GLM-5.2 (GlmMoeDsaForCausalLM) misses
# them. They were tested on-node and are NOT needed on this Triton MLA
# fallback path — boot and generation are identical with and without them —
# so they are deliberately not exported here. They become relevant again if
# the sparse DSA backend ever works on SM120.
#
# Do not simply switch --attention-backend back to dsa: enabling the sparse
# path on SM120 was attempted on-node (2026-07-26) and needs kernel work, not
# a flag. Patching SGLang to route the indexer's paged-MQA logits through its
# TileLang/torch implementations (both already used by the dsv4 indexer) and
# to stop building the DeepGEMM schedule plan does clear the "Unsupported
# architecture" abort, and the server then boots and serves — but four more
# walls follow:
# 1. TileLang's CUDA sparse-MLA kernel takes bf16 KV only (its fp8 variants
# are ROCm-only), which halves the KV pool to 256,256 tokens/rank.
# 2. That kernel asks for 170,048 B of dynamic shared memory; SM120 allows
# ~100 KB. Its tile size is not a knob — the barrier arrive_counts
# (384/256/128) encode the BI=64 thread mapping, so block_I=32 compiles
# and then reads out of bounds.
# 3. The plain (v1) kernel does fit at block_I=32 / num_stages=1 / 128
# threads, but CUDA-graph capture fails because TileLang JIT-compiles
# inside the capture (cudaErrorStreamCaptureUnsupported).
# 4. With graphs disabled it runs and is still numerically wrong: degenerate
# repetition on a 26-token prompt and an empty answer on the 45k-token
# needle, identically with the TileLang and the torch reference logits
# kernels — so the fault is the sparse-MLA kernel itself, not the indexer.
# Fixing this means an SM120-shaped sparse-MLA kernel (split the d_v=512
# accumulation so KV tiles stay under ~32 KB, re-derive the barrier counts,
# validate against the dense path), not a config change. Full logs from the
# attempt: rtx6000pro-lat:/home/ubuntu/glm52-sglang-patch/out/server.dsa*.log.

# Agentic warmup dispatches hundreds of large prompts at once; allow up to
# 15 minutes of TCP progress before AIPerf declares a connection dead.
export AIPERF_HTTP_TCP_USER_TIMEOUT=900000
# AIPerf pins one pooled keep-alive connection per session (client-side
# keep-alive 300s) while uvicorn's default SGLANG_TIMEOUT_KEEP_ALIVE is 5s;
# inter-turn idle gaps can reuse a socket exactly as the server closes it ->
# ECONNRESET -> terminal warmup failure. Outlast the client pool.
export SGLANG_TIMEOUT_KEEP_ALIVE=900
# Dense-attention prefill on this node measured 872 tok/s at 4.4k context
# falling to 418 tok/s at 244k (single stream), so a warmup snapshot of
# 100k-token histories needs several minutes per trajectory. Double the
# shared 1800s warmup grace so warmup drains instead of being declared
# failed; grace is a maximum wait, not a fixed sleep.
export AGENTIC_WARMUP_GRACE_PERIOD="${AGENTIC_WARMUP_GRACE_PERIOD:-3600}"

# AgentX concurrency counts live session trees, not individual requests.
# Allow subagent fan-out to exceed CONC without clipping request bursts.
MAX_RUNNING_REQUESTS=$((2 * CONC))
CUDA_GRAPH_MAX_BS=$MAX_RUNNING_REQUESTS
[ "$CUDA_GRAPH_MAX_BS" -gt 64 ] && CUDA_GRAPH_MAX_BS=64

MEM_FRACTION_STATIC="${MEM_FRACTION_STATIC:-0.85}"
CONTEXT_LENGTH="${CONTEXT_LENGTH:-262144}"
ATTENTION_BACKEND="${ATTENTION_BACKEND:-triton}"

MOE_ARGS=()
if [[ -n "${MOE_RUNNER_BACKEND:-}" ]]; then
MOE_ARGS=(--moe-runner-backend "$MOE_RUNNER_BACKEND")
fi

SGLANG_CMD=(
python3 -m sglang.launch_server
--model-path "$MODEL_PATH"
--served-model-name "$MODEL"
--host 0.0.0.0
--port "$PORT"
--trust-remote-code
--tp "$TP"
--ep-size "$EP_SIZE"
--quantization modelopt_fp4
# nvidia/GLM-5.2-NVFP4 keeps the shared expert loose and in BF16, so the
# fused-shared-expert loader cannot consume it (see the header note).
--disable-shared-experts-fusion
# SGLang would default this DSA model to --attention-backend dsa, whose
# indexer metadata is built by deep_gemm.get_paged_mqa_logits_metadata
# (the only CUDA option; 'cutedsl' is gated to SM100). DeepGEMM aborts
# with "Assertion error (attention.hpp:227): Unsupported architecture" on
# SM120 during warmup, so fall back to Triton MLA. Sparse attention is
# consequently not exercised on this GPU family.
--attention-backend "$ATTENTION_BACKEND"
"${MOE_ARGS[@]}"

Check failure on line 194 in benchmarks/single_node/agentic/glm5.2_fp4_rtx6000pro_sglang.sh

View check run for this annotation

Claude / Claude Code Review

Missing --kv-cache-dtype fp8_e4m3 flag defeats the recipe's entire FP8-KV sizing 

SGLANG_CMD in this recipe never passes `--kv-cache-dtype fp8_e4m3`, so SGLang falls back to its default bf16 KV cache instead of the FP8 cache the recipe's header comments, `nvidia-master.yaml`, and the PR description all assume (464,768 tokens/rank in 23.91 GB, ~51 KB/token). The sibling `glm5.2_fp4_b300_sglang.sh:118` recipe this was copied from sets the flag explicitly in the same low-latency branch — add `--kv-cache-dtype fp8_e4m3` to `SGLANG_CMD` here too.
Comment on lines +172 to +194

@claude claude Bot Jul 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 SGLANG_CMD in this recipe never passes --kv-cache-dtype fp8_e4m3, so SGLang falls back to its default bf16 KV cache instead of the FP8 cache the recipe's header comments, nvidia-master.yaml, and the PR description all assume (464,768 tokens/rank in 23.91 GB, ~51 KB/token). The sibling glm5.2_fp4_b300_sglang.sh:118 recipe this was copied from sets the flag explicitly in the same low-latency branch — add --kv-cache-dtype fp8_e4m3 to SGLANG_CMD here too.

Extended reasoning...

The SGLANG_CMD array built at benchmarks/single_node/agentic/glm5.2_fp4_rtx6000pro_sglang.sh:172-194 never sets --kv-cache-dtype. SGLang has no auto-fp8 default for KV — omitting the flag makes it use the model dtype (bf16) for the MLA latent KV cache. This is confirmed by grepping the rest of the repo: every other recipe that relies on an FP8 KV pool sets the flag explicitly (glm5.2_fp4_b300_sglang.sh:118, dsv4_fp4_mi355x_sglang.sh:140, the srt-slurm and AMD models.yaml recipes, etc.). I verified directly that the B300 sibling this script was copied from does set --kv-cache-dtype fp8_e4m3 inside its non-DP ("STP") low-latency branch — the exact branch this RTX PRO 6000 recipe descends from — so the flag was dropped in the copy rather than being an intentional change.

The rest of the recipe is built entirely around the assumption that the KV cache is FP8. The script's own header comment, the new nvidia-master.yaml entry, and the PR description all state the "FP8 KV pool" holds 464,768 tokens per rank (23.91 GB) at mem-fraction-static 0.85, i.e. ~51 KB/token across GLM-5.2's 78 layers. That number is only consistent with an FP8 cache: GLM-5.2's MLA state is kv_lora_rank 512 + rope 64 = 576 elements/token/layer, which comes to roughly 44 KB/token in FP8 across 78 layers (close to the ~51 KB/token quoted, with indexer/page overhead) — 23.91 GB / 464,768 tokens = ~54 KB/token, again matching FP8. In bf16 the same state would be roughly double, ~88-90 KB/token, giving a pool of only ~230k-285k tokens/rank in the same 23.91 GB, not 464,768. So the on-node validation that produced 464,768 tokens/rank and the "needle retrieval at 220,029 tokens" / "two ~110k-token sessions resident" results in the PR description must have been run with --kv-cache-dtype fp8_e4m3 set — the committed script does not reproduce that configuration.

Concretely, if this ships as-is: the CI sweep runs a materially different (bf16-KV) server than the one that was validated and documented. The actual pool would be roughly half the size the sizing math assumes, which invalidates both the 256k context cap (--context-length 262144) and the concurrency-4 cutoff — a bf16 pool of ~230k-285k tokens/rank is close to or below the 262144-token context length, so a single full-length 256k session may not even fit resident, let alone the two ~110k sessions the recipe uses to justify stopping at concurrency 4. That risks continuous re-prefilling or startup/capacity failures that the documented validation did not encounter, undermining the primary purpose of this PR (recording believable on-node capacity numbers for this hardware).

To fix: add --kv-cache-dtype fp8_e4m3 to the SGLANG_CMD array, matching glm5.2_fp4_b300_sglang.sh:118, and re-validate that the pool size, context cap, and concurrency cutoff in the header/nvidia-master.yaml/PR description still hold under the corrected server config.

Step-by-step proof:

  1. Read glm5.2_fp4_rtx6000pro_sglang.sh:172-194 — SGLANG_CMD contains --quantization modelopt_fp4, --attention-backend triton, --mem-fraction-static, etc., but no --kv-cache-dtype anywhere in the array or in any exported env var.
  2. Grep the codebase for --kv-cache-dtype in other FP8-KV recipes — it is always passed explicitly (e.g. glm5.2_fp4_b300_sglang.sh:118), confirming SGLang requires an explicit flag rather than auto-detecting FP8 KV from the weight quantization.
  3. Compute expected FP8 KV bytes/token for GLM-5.2 MLA: (512 + 64) elements * 78 layers * 1 byte (fp8) ≈ 44 KB/token, and cross-check against the documented 23.91 GB / 464,768 tokens ≈ 51.4 KB/token — both are in the FP8 regime.
  4. Compute the bf16 equivalent: same element count in 2 bytes ≈ 88 KB/token, which in 23.91 GB yields ≈ 278k tokens/rank — roughly half of 464,768, and dangerously close to the 262144-token --context-length setting used by this recipe.
  5. Conclude that the shipped script (bf16, missing flag) cannot reproduce the documented/validated 464,768-token pool, invalidating the capacity claims that justify the 256k cap and conc-4 cutoff.

# GLM-5.2 emits the GLM-4.7-style <tool_call>/<arg_key>/<arg_value>
# format; glm47 is required for structured message.tool_calls, and the
# reasoning parser keeps hybrid-thinking output in reasoning_content.
--tool-call-parser glm47
--reasoning-parser glm45
--context-length "$CONTEXT_LENGTH"
--chunked-prefill-size 8192
--mem-fraction-static "$MEM_FRACTION_STATIC"
--max-running-requests "$MAX_RUNNING_REQUESTS"
--cuda-graph-max-bs "$CUDA_GRAPH_MAX_BS"
--watchdog-timeout 1800
--enable-metrics
)

printf '%q ' "${SGLANG_CMD[@]}" | tee "$RESULT_DIR/sglang_command.txt"
printf '\n' | tee -a "$RESULT_DIR/sglang_command.txt"

echo "Starting SGLang server for RTX PRO 6000..."
"${SGLANG_CMD[@]}" > "$SERVER_LOG" 2>&1 &
SERVER_PID=$!
echo "Server PID: $SERVER_PID"

cleanup_agentic_server() {
local exit_code=$?
trap - EXIT INT TERM
set +e
stop_background_process_tree "$SERVER_PID" "SGLang server" 60
exit "$exit_code"
}
trap cleanup_agentic_server EXIT
trap 'exit 130' INT
trap 'exit 143' TERM

wait_for_server_ready --port "$PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID"

if [ "${EVAL_ONLY:-false}" = "true" ]; then
run_eval --port "$PORT"
else
build_replay_cmd "$RESULT_DIR"
REPLAY_CMD+=" --server-metrics http://localhost:$PORT/metrics"
run_agentic_replay_and_write_outputs "$RESULT_DIR"
fi
32 changes: 32 additions & 0 deletions configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8097,3 +8097,35 @@ glm5.2-fp4-b300-sglang-agentic:
# radix cache (GPU hit 0.93->0.57 at 48->64) and thrashes on re-prefill, so it is
# strictly dominated by conc 48 on both throughput and interactivity.
- { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: { name: hicache }, conc-list: [48], router: { name: sglang-router, version: "0.3.2" } }

# GLM-5.2 NVFP4 AgentX on the 8x RTX PRO 6000 Blackwell (SM120) Latitude node
# with SGLang. Three SM120 facts shape this entry:
#
# * TP8 only. The checkpoint is 433 GB (56 GB/GPU loaded), so pure TP across
# all eight 96 GB GPUs is the only layout that leaves room for a KV pool;
# TP4 cannot hold the weights, and attention-DP would replicate the pool
# per rank.
# * Sparse DSA attention is unavailable. SGLang's DSA backend builds its
# indexer paged-MQA-logits metadata through DeepGEMM (the only CUDA
# option; 'cutedsl' is gated to SM100), and DeepGEMM asserts "Unsupported
# architecture" on SM120, so the recipe runs the Triton MLA fallback with
# the sparse indexer bypassed. Prefill therefore scales superlinearly with
# context (measured on-node: 17.6k tokens 5.2 s -> 70.5k tokens 38.9 s).
# * Context is capped at 256k, not the model's 1M. The FP8 KV pool is
# 464,768 tokens per rank at mem-fraction-static 0.85 (~51 KB/token across
# 78 layers), so the recipe replays the 256k-capped trace corpus and stops
# at conc 4, where four full-length sessions already exceed the pool and
# start re-prefilling.
glm5.2-fp4-rtx6000pro-sglang-agentic:
image: lmsysorg/sglang:v0.5.15.post1-cu130
model: nvidia/GLM-5.2-NVFP4
model-prefix: glm5.2
runner: cluster:rtx6000pro-lat
precision: fp4
framework: sglang
multinode: false
scenarios:
agentic-coding:
- dram-utilization: 0.80
search-space:
- { tp: 8, ep: 1, dp-attn: false, kv-offloading: none, conc-list: [1, 2, 4] }
6 changes: 6 additions & 0 deletions configs/runners.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,10 @@ labels:
- gb300-nv_0
- gb300-nv_1
- gb300-nv_2
rtx6000pro:
- rtx6000pro-lat_00
rtx6000pro-lat:
- rtx6000pro-lat_00
cluster:h100-cw:
- h100-cw_00
- h100-cw_01
Expand Down Expand Up @@ -251,6 +255,8 @@ labels:
- gb300-nv_0
- gb300-nv_1
- gb300-nv_2
cluster:rtx6000pro-lat:
- rtx6000pro-lat_00
cluster:mi300x-amds:
- mi300x-amds_00
- mi300x-amds_01
Expand Down
16 changes: 16 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5066,3 +5066,19 @@
description:
- "Bump SGLang container image from lmsysorg/sglang:v0.5.12-cu130 to lmsysorg/sglang:v0.5.15.post1-cu130 (https://github.com/sgl-project/sglang/releases/tag/v0.5.15.post1)"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2313

- config-keys:
- glm5.2-fp4-rtx6000pro-sglang-agentic
scenario-type:
- agentic-coding
description:
- "Add a GLM-5.2 NVFP4 SGLang AgentX sweep on the 8x RTX PRO 6000 Blackwell (SM120) Latitude runner, TP8/EP1 with GPU-resident FP8 KV cache at concurrency 1, 2, and 4"
- "Pin lmsysorg/sglang:v0.5.15.post1-cu130, the same image as the merged B300 GLM-5.2 SGLang recipe and the first release whose GlmMoeDsaForCausalLM support is present"
- "Force --disable-shared-experts-fusion: SGLang enables fusion for this config, but nvidia/GLM-5.2-NVFP4 stores the shared expert loose in BF16 while the routed experts are packed NVFP4, so the fused loader aborts weight load with a 3072-vs-6144 shape mismatch"
- "Run the Triton MLA attention fallback instead of the sparse DSA backend: SGLang builds DSA indexer paged-MQA-logits metadata through DeepGEMM only ('cutedsl' is gated to SM100), and DeepGEMM asserts 'Unsupported architecture' on SM120, so GLM-5.2's sparse-attention advantage is not realized on this GPU family"
- "Do not carry SGLang's DeepseekV4-only SM120 kernel fixups (FP8 weight-only GEMM, topk_v2, tilelang/DeepGEMM hidden-compression prenorm, torch paged-MQA logits): tested on-node, boot and generation are identical with and without them on this Triton fallback path"
- "Cap context at 256k and replay the 256k-capped trace corpus: the FP8 KV pool holds 464,768 tokens per rank at mem-fraction-static 0.85 (~51 KB/token across 78 layers), so the model's 1M context does not fit and conc >4 re-prefills continuously"
- "Raise the agentic warmup grace period to 3600s: dense-attention prefill measured 872 tok/s at 4.4k context falling to 418 tok/s at 244k, so long-history warmup snapshots need more than the shared 1800s default"
- "Validated over SSH: TP8 boot on the node (56.04 GB weights/GPU, 464,768-token KV pool/rank, CUDA graphs captured, /health 200), short-prompt correctness through the glm45/glm47 parsers, and exact needle retrieval at 45,394 / 155,552 / 220,029-token prompts with no NaN from the SM120 flashinfer_cutlass NVFP4 MoE path"
- "Record in the recipe why the sparse DSA path cannot be re-enabled by flag: patching SGLang to use its TileLang/torch paged-MQA-logits kernels clears the DeepGEMM abort and boots, but the TileLang sparse-MLA kernel needs bf16 KV, asks for 170,048 B of shared memory against SM120's ~100 KB, cannot be CUDA-graph captured, and is numerically wrong at the only tile size that fits"
pr-link: XXX
Loading