fix(recipes): glm-5.2: move SGLang B200/H200 recipes to stock Dynamo 1.5.1 (SGLang 0.5.18, Mooncake PD, H200 decode KV fix)

[shugandhi]

Overview

Moves the GLM-5.3/5.2 SGLang recipes from the recipe-specific sglang-runtime:1.3.0-glm-5.2-dev.1 image to the stock sglang-runtime:1.5.1 image (Dynamo 1.5.1, SGLang 0.5.18). Recipe-level changes only; none of the dev-image SGLang patches are needed. All four targets are re-validated on 1.5.1 with GLM-5.3 (throughput and accuracy).

Details

Four profiles under recipes/glm-5.2/sglang/: agg-b200-agentic, disagg-b200-agentic, agg-h200-agentic, disagg-h200-agentic. Changes to every deploy.yaml (config only, no comments):

• image sglang-runtime:1.5.1 on frontend and workers
• dsa-* / cuda-graph-backend-prefill keys moved from the engine YAML to the command line (the 0.5.18 --config parser rejects them)
• disaggregated: disaggregation-transfer-backend: mooncake; per-pod UCX_NET_DEVICES derivation removed (the NIXL/UCX path segfaults and wedges under load on 1.5.1)
• B200 mem-fraction-static lowered (agg 0.88, prefill 0.86, decode 0.84); workers set DYN_FORWARDPASS_METRIC_PORT="" (the 0.5.18 metrics reporter crashes the scheduler on the first EAGLE decode batch)
• H200 disaggregated decode mem-fraction-static 0.88 -> 0.95: at 0.88 the per-rank KV pool holds ~237K tokens, smaller than one 250K-token request
• explicit liveness/readiness probes 60 s / 60 s / 10 on every worker

Results (GLM-5.3, recipe perf.yaml, AIPerf 0.11.0, 15% agentic Mooncake trace, 3,541 requests, synthetic MTP acceptance 2.69 on decode workers):

Target	GPUs	Conc.	tok/s/GPU (vs published)	User tok/s P50 (vs published)	TTFT P50 (vs published)	Completed
B200 aggregated (4 workers)	16	64	190.0 (+7.7%, was 176.4)	61.9 (+7.7%, was 57.5)	229 ms (-36%, was 356)	3,537 / 3,541
B200 disaggregated (3P1D)	20	128	323.8 (+0.9%, was 320.9)	63.1 (-3.0%, was 65.1)	1,280 ms (-34%, was 1,938)	3,541 / 3,541
H200 aggregated (3 workers)	24	32	60.9 (+11.6%, was 54.6)	57.3 (+9.5%, was 52.4)	1,158 ms (-35%, was 1,790)	3,535 / 3,541
H200 disaggregated (1P1D)	16	24	84.3 (+22.5%, was 68.9)	61.5 (+14.1%, was 53.9)	1,310 ms (-30%, was 1,874)	3,535 / 3,541

"Published" = the numbers in the current glm-5.2 README/docs, measured with the 1.3.0-glm-5.2-dev.1 image.

Accuracy (lm-eval 0.4.12, max_gen_toks 32,768, reasoning on, no SGLANG_SIMULATE_*), IFEval / GPQA-Diamond: B200 agg 0.8956 / 0.8106 (mean of 2); B200 disagg 0.8946 / 0.7879; H200 agg 0.9002 / 0.7879; H200 disagg 0.9057 / 0.8030 (1 rep each). All four agree within the per-run standard error. Same-day baseline on the 1.3.0-glm-5.2-dev.1 image (B200): agg 0.9011 / 0.7980 (mean of 2), disagg GPQA 0.8131; stock 1.5.1 is within 0.4 sigma of it on both tasks.

Docs:

• recipes/glm-5.2/README.md: configuration, deploy steps, one-row-per-target results, known issues; accuracy and benchmark metadata moved to the docs page
• docs/fern/pages/recipes/model-recipes/glm-5-2.mdx: Expected Performance and Accuracy for all four targets on sglang-runtime:1.5.1, Mooncake KV transfer, recipe parameter notes
• docs/fern/pages/recipes/_catalog/recipes/glm-5-2.yaml: runtime image, techniques, expected-performance summaries (all four available: true)

Where should the reviewer start?

docs/fern/pages/recipes/model-recipes/glm-5-2.mdx, then the four recipes/glm-5.2/sglang/*/deploy.yaml.

Related Issues

• Confirmed — no related issue.

Caveats

• On both H200 targets, 6 of 3,541 trace requests exceed the 250K context limit and are rejected by the frontend (HTTP 400); they count as incomplete in the tables.
• The checked-in manifests do not set the SGLANG_SIMULATE_* variables; the README and docs page state how to add them for throughput benchmarking only.
• The catalog artifacts block for 1.3.0-glm-5.2-dev.1 is unchanged (historical image provenance).

Test plan

All four targets deployed and benchmarked on stock sglang-runtime:1.5.1 with the recipe perf.yaml (full 3,541-request trace, 0 worker restarts) and evaluated with lm-eval (IFEval, GPQA-Diamond) against the endpoint. docs/fern/pages/recipes/_catalog/validate.py, docs/fern/scripts/docs_lint.py --scan docs, docs/fern/scripts/gen_llms_tables.py --check, and docs/tests/ pass.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-01a11c27-ef2b-758d-b067-89241a9ac7ea.docs.buildwithfern.com/dynamo/dev

[devin-ai-integration]

✅ Dynamo PR CI passed — run 37801318966 (attempt 1) on 322c6156f6

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

_{Posted automatically by Devin for run 37801318966. Updated on every full-CI run of this PR.}

[dmitry-tokarev-nv]

Approved at ae485126c8de6cd94de3b7bb3d29db4e068fca82: no open finding. The force-push fixes the 3 P3 findings of my review at 2013f51d04. Two fixes apply my own asks (MC_FORCE_HCA=1 and "3 failures"), so this review partly checks my own advice.

1. [P3, fixed] Both disaggregated recipes now set MC_FORCE_HCA: "1" on the prefill and the decode worker. No other SGLang worker in the four recipes starts Mooncake. Without an RDMA device, the Mooncake start of these workers now raises RuntimeError instead of a fallback to TCP.
2. [P3, fixed] The description no longer names the releases.data.ts and recipes/README.md edits.
3. [P3, fixed] All 6 probe comments now say "3 failures". This matches the liveness default of the v1.5.0 operator for a 1.5.0 runtime (component_worker.go:31-33).

Without an RDMA device, the old workers start Mooncake on TCP, and the new workers raise RuntimeError.

I used sglang-runtime:1.5.0@sha256:5d481e74... (Mooncake 0.3.13.post1, SGLang 0.5.18) with no /dev/infiniband. Dynamo 1.5.0 parse_args read the argv and the engine YAML of each worker. Then SGLang's maybe_init_shared_mooncake_transfer_engine ran with the env of the manifest. SGLang calls this function in the ModelRunner constructor, before the weight load.

Manifest	Workers	MC_FORCE_HCA	Result
2013f51d04, both disaggregated recipes	prefill, decode	unset	Mooncake starts: Using TCP transport (no HCA detected)
ae485126c8, both disaggregated recipes	prefill, decode	1	RuntimeError: Mooncake Transfer Engine initialization failed.
ae485126c8, both aggregated recipes (control)	agg	1, added	No Mooncake engine starts

The Mooncake source shows that the flag changes nothing on a node with a usable HCA.

The source at Mooncake tag v0.3.13.post1 matches the file and line of all 21 engine log lines from my run. Tags v0.3.14-rc1 and v0.3.12.post1 miss several of them.

• If !no_hca || force_hca is true, the engine installs RDMA (transfer_engine_impl.cpp:381). If discovery finds a usable HCA, !no_hca is true, and the flag has no effect.
• Discovery keeps an HCA only with an open /dev/infiniband/<dev> and one active port.
• If the HCA list is empty, the flag turns the TCP fallback (line 396) into a failed start (line 385).
• Lines 298 and 410 read the flag only in other builds. GetForceHca() has no caller.
• Control: with a custom topology that lists one HCA, the engine logs the same 16 lines with and without the flag.

No CI run failed on the head, but no deploy job ran a recipe.

• CI on ae485126c8: Pre Merge, PR, PR-XPU, Fern Docs, Docs link check, DCO, Copyright Checks, codeowners and Lint PR pass. Recipe Check passed 327 tests on the merge with a4f4f71c96.
• No deploy or GPU job ran a recipe. I used no node with HCAs, so the HCA answer comes from the source. I did not run a full worker to its exit after the RuntimeError.

All 11 review threads are now resolved.

Open item (not graded): recipes/templates/sglang/disagg/deploy-v1beta1.template.yaml still names the dev image and keeps dsa-* keys in YAML. It is outside this PR.

[ynpandey-nv]

/ok to test 210fd5a

[dmitry-tokarev-nv]

Approved at 210fd5a9c9e02c7acccf8746eccccdac2c6f8db4: one P3 and one open item that I do not grade. The two new commits change only 1.5.0 to 1.5.1, in 22 lines of 7 files. On sglang-runtime:1.5.1, the MC_FORCE_HCA and "3 failures" facts from my review at ae485126c8 still hold. Both came from my own asks, so part of this review covers my own advice.

1. [P3] The description says "All four targets are re-validated on 1.5.1 with GLM-5.3 (throughput and accuracy)". But every result number is identical to the 1.5.0 numbers at ae485126c8. The docs page now names 1.5.1 for the same targets (glm-5-2.mdx:242). Please say which image produced these numbers, or add a run on 1.5.1.

Open item (not graded): recipes/templates/sglang/disagg/deploy-v1beta1.template.yaml, outside this PR, still names the dev image and keeps dsa-* keys in YAML.

Their image changed, but the numbers stayed the same.

• At ae485126c8, committed at 2026-10-05T17:50:25Z, the README, the docs page and the catalog gave these numbers for sglang-runtime:1.5.0.
• If I map 1.5.1 and its digest back to 1.5.0 and its digest, the 7 changed files at the head match ae485126c8 byte for byte. No number changed.
• The description edit at 2026-10-07T04:14:16Z changed 1.5.0 to 1.5.1 in 6 lines. It changed no number.
• The 1.5.1 image was built at 2026-10-06T00:31:42Z, after those numbers were committed.
• I cannot see whether anyone ran the recipes on 1.5.1 after that. The engine parts of the two images are the same, but the Dynamo request path is not. The image block and the commit block below show the details.

The 1.5.1 image exists without a login, and only its Dynamo part differs from 1.5.0.

	sglang-runtime:1.5.0	sglang-runtime:1.5.1
Index digest	sha256:5d481e74...dc17	sha256:37badffd...d514d
amd64 image built	2026-09-18T00:29:46Z	2026-10-06T00:31:42Z
DYNAMO_COMMIT_SHA	32b8b2f8c6	2093662a4d (tag v1.5.1)
ai-dynamo, ai-dynamo-runtime	1.5.0	1.5.1
sglang	0.5.18	0.5.18
mooncake-transfer-engine-cuda13	0.3.13.post1	0.3.13.post1
CUDA_VERSION, torch	13.0.3, 2.13.0+cu130	13.0.3, 2.13.0+cu130

• Anonymous GET requests to nvcr.io return 200 for :1.5.1 and 404 for a bogus tag. The 1.5.1 index digest is sha256:37badffd83eb334aee9d06fb442e36fc3c963314336c92622c78ac5c037d514d, the digest in all 10 image lines of the head.
• The SGLang package (4,362 files), the Mooncake package (70 files) and the Mooncake engine.so have the same hashes in both images.
• In dynamo.sglang, only _version.py, handler_base.py, decode_handler.py, the image and video handlers, and tests differ. dynamo/_core.abi3.so differs.
• So the README line "(Dynamo 1.5.1, SGLang 0.5.18)" is true. The 6 comments that say that sglang-runtime:1.5.1 crashes the scheduler rest on SGLang code that did not change. I did not run an EAGLE decode batch.

On 1.5.1, MC_FORCE_HCA still stops a worker that has no RDMA device.

I ran the r1002 probe in both images on the GPU box, with no /dev/infiniband. Dynamo parse_args read the argv and the engine YAML of each head manifest. Then SGLang's maybe_init_shared_mooncake_transfer_engine ran with the env of the manifest.

Case	Workers	Result, the same in 1.5.0 and 1.5.1
Head manifests as written	4 disaggregated prefill and decode workers	RuntimeError: Mooncake Transfer Engine initialization failed.
Control: the same workers without MC_FORCE_HCA	4	Mooncake starts: Using TCP transport (no HCA detected)
Control: aggregated workers with the flag added	2	No Mooncake engine starts

I used no node with an HCA, and I did not run a full worker to its exit.

The v1.5.1 operator still gives these workers a default liveness probe of 5 s / 4 s / 3 failures.

If the operator reads a runtime version of 1.5.0 or later from the image tag of the main container, it gives 3 failures. Otherwise it gives 1. It ignores the @sha256 digest. I ran generateComponentContext and WorkerDefaults.GetBaseContainer of the v1.5.1 operator in a Go test, on the 6 workers of the 4 head manifests.

Image of the main container	Version	Liveness	Readiness
Head manifests (1.5.1@sha256:37badffd...)	1.5.1	5 s / 4 s / 3	10 s / 4 s / 3
Control: ae485126c8 manifests (1.5.0@sha256:5d481e74...)	1.5.0	5 s / 4 s / 3	10 s / 4 s / 3
Control: :latest	none	5 s / 4 s / 1	10 s / 4 s / 3
Control: :1.4.0	1.4.0	5 s / 4 s / 1	10 s / 4 s / 3

From v1.5.0 to v1.5.1, the operator changes only in tests, test data and a docs footer.

No commit from v1.5.0 to v1.5.1 changes these recipes, but some change Dynamo code that the recipes run.

The 18 commits change no file of this PR. No changed line names a flag or an environment variable that the recipes set.

• SGLang: 796fafb6af (#14951) changes handler_base.py and decode_handler.py. Its new min_tokens path needs skip_tokenizer_init, which these recipes do not set. 60e15ec7d2 (#14435) changes the image and video handlers. 71d679fb91 (#15669) pins PyNvVideoCodec 2.2.3 in the image.
• Router and frontend: 588a735aab (#15310) puts a time limit on the overload marks that worker responses set. 7fd8ec86f0 (#14806) moves dynamo-renderer to 5.1.2 and dynamo-protocols to 5.4.1. 01f30f0ef8 (#14962) changes model discovery.
• Operator: 796fafb6af, a327f1ac11 (#15370) and bd16d9e98c (#15308) change only test data, tests, chart versions and a docs footer.
• Mooncake: no commit.

The head merges with the main branch without a conflict, and all 7 jobs that the main branch requires pass. No CI job deployed a recipe.

• git merge-tree --write-tree of the head with main at 548a010548 gives no conflict. A planted change on a line that this PR changes gives a conflict.
• On 210fd5a9c9, Pre Merge passed. Its Recipe Check job passed 327 tests on the merge with acf7b8379f. The 6 later commits on main, up to 548a010548, change no recipe and no catalog file.
• PR, PR-XPU and Fern Docs passed on pull-request/15590. In PR and PR-XPU, every build, test and deploy job skipped.

[ynpandey-nv]

/ok to test 401b7f4

[ynpandey-nv]

/ok to test 322c615