GB200 DSv4 vLLM agentic full sweep (3p2d TEP8/TP8 + 2p1d DEP8/DEP8) - #2122
GB200 DSv4 vLLM agentic full sweep (3p2d TEP8/TP8 + 2p1d DEP8/DEP8)#2122cquil11 wants to merge 4 commits into
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
be6f299 to
e83df02
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28976572219 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - "Full agentic-coding sweep of the DSv4 FP4 GB200 dynamo-vllm disagg topologies on the June 21 AgentX corpus" | ||
| - "3p2d TEP8 prefill / TP8 decode (40 inference GPUs): c4-c80" | ||
| - "2p1d DEP8 prefill / DEP8 decode (24 inference GPUs): c32-c256 including exploratory tail past the c160 normalized-throughput peak" | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2122 |
There was a problem hiding this comment.
🔴 The new perf-changelog entry has pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/PLACEHOLDER — a forgotten template token. This isn't just a dead link: utils/validate_perf_changelog.py only accepts either the canonical .../pull/2122 URL or the literal placeholders XXX / .../pull/XXX, so the changelog validator will hard-fail CI until fixed. Replace PLACEHOLDER with 2122.
Extended reasoning...
The bug
The new entry at perf-changelog.yaml:4645 ends with:
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/PLACEHOLDERThe literal string PLACEHOLDER is a forgotten template token that no other entry in the file uses. The three immediately-preceding entries all use their real PR numbers (pull/2113, pull/2114, pull/2115), so the convention is clear — and per PR metadata this is PR #2122, so the value should be https://github.com/SemiAnalysisAI/InferenceX/pull/2122.
Why this fails CI (not just a dead link)
utils/validate_perf_changelog.py defines the set of accepted placeholder tokens for PR-authored runs:
# utils/validate_perf_changelog.py:24-27
PR_LINK_PLACEHOLDERS = {
"XXX",
"https://github.com/SemiAnalysisAI/InferenceX/pull/XXX",
}and validate_added_pr_link (lines 144-160) requires the link to match either the canonical .../pull/<pr_number> URL or one of those two placeholder tokens:
if link not in PR_LINK_PLACEHOLDERS and link != expected:
raise ChangelogValidationError(
f"new PR entry must use {expected!r} or an XXX placeholder; "
f"found {link!r}"
)Step-by-step proof
- On this PR, the validator runs with
pr_number=2122. - It reads the new entry's
pr-link="https://github.com/SemiAnalysisAI/InferenceX/pull/PLACEHOLDER". - It computes
expected = "https://github.com/SemiAnalysisAI/InferenceX/pull/2122"— does not match. - It checks
link in PR_LINK_PLACEHOLDERS— the set only contains"XXX"and".../pull/XXX", so"PLACEHOLDER"/".../pull/PLACEHOLDER"is not a member. - Both branches fail →
ChangelogValidationErroris raised → the changelog-validation CI job fails.
Impact
- CI blocks the merge on the changelog-validation check until fixed.
- Even if bypassed, the URL 404s and permanently breaks the changelog's provenance/traceability convention.
Fix
One-token change on line 4645:
- pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/PLACEHOLDER
+ pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2122| - config-keys: | ||
| - dsv4-fp4-gb200-dynamo-vllm-agentic-3p2d-tep8-tp8 | ||
| - dsv4-fp4-gb200-dynamo-vllm-agentic-2p1d-dep8-dep8 | ||
| description: | ||
| - "Full agentic-coding sweep of the DSv4 FP4 GB200 dynamo-vllm disagg topologies on the June 21 AgentX corpus" | ||
| - "3p2d TEP8 prefill / TP8 decode (40 inference GPUs): c4-c80" | ||
| - "2p1d DEP8 prefill / DEP8 decode (24 inference GPUs): c32-c256 including exploratory tail past the c160 normalized-throughput peak" | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2122 |
There was a problem hiding this comment.
🟡 Nit (pre-existing style rule): per AGENTS.md line 7, PR titles and descriptions must be bilingual — the title needs a / <中文标题> half and the body needs a ## 中文说明 section mirroring the English summary. Recent sibling PRs #2114/#2115 follow this format; please update the PR title and description on GitHub. No code changes needed.
Extended reasoning...
What the rule is. AGENTS.md line 7 states: "PR and GitHub-issue titles & descriptions must be bilingual — include a Simplified Chinese version in addition to English. Title format: <English title> / <中文标题>. In the PR/issue body, follow the English content with its Chinese translation (e.g. a ## 中文说明 section mirroring the summary). ... This applies to every PR and every issue."
What this PR has. The title is "GB200 DSv4 vLLM agentic full sweep (3p2d TEP8/TP8 + 2p1d DEP8/DEP8)" — no / <中文标题> half. The description ends after the English bullet list about the two topologies and their concurrency chunks — there is no ## 中文说明 (or equivalent) section mirroring that content.
Sibling PRs confirm the convention. Recently-merged neighbors in perf-changelog.yaml follow the bilingual format, e.g. commit d527fc8 — "Add GLM-5.1 NVFP4 GB300 disaggregated dynamo-sglang MTP config / 添加 GLM-5.1 NVFP4 GB300 分离式 dynamo-sglang MTP 配置 (#2114)" — and bed23ea (#2115) has the analogous GB200 Chinese half. #1780 (c57f1e6) does too.
Impact. Purely a policy/metadata issue. No code path is affected, the sweep runs regardless, and the changelog YAML itself has no bilingual requirement. But the rule is documented as mandatory-for-every-PR, so it is worth flagging.
Step-by-step proof.
- Open
AGENTS.mdand read line 7 — the bilingual requirement is stated verbatim with "This applies to every PR and every issue." - Look at this PR's title — "GB200 DSv4 vLLM agentic full sweep (3p2d TEP8/TP8 + 2p1d DEP8/DEP8)". There is no
/separator followed by Chinese characters. - Look at this PR's description above — it contains only English prose and no
## 中文说明heading. - Compare with
git log --oneline -5output, which shows Add GLM-5.1 NVFP4 GB300 disaggregated dynamo-sglang MTP config / 添加 GLM-5.1 NVFP4 GB300 分离式 dynamo-sglang MTP 配置 #2114/Add GLM-5.1 NVFP4 GB200 disaggregated dynamo-sglang MTP config / 添加 GLM-5.1 NVFP4 GB200 分离式 dynamo-sglang MTP 配置 #2115/[NV] add glm5-fp4-gb200-dynamo-sglang / 新增 GLM-5 NVFP4 GB200 Dynamo SGLang 分离式基准测试 #1780 all include the/ <中文标题>half.
How to fix. In the GitHub PR UI, edit the title to append / <中文标题> (e.g. / GB200 DSv4 vLLM 智能体编码全量 sweep (3p2d TEP8/TP8 + 2p1d DEP8/DEP8)) and append a ## 中文说明 section to the body mirroring the English bullets. No file changes required.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=28977119907 |
sweep-multi-node-agentic rendered the conc array directly into string
inputs ('[${{ matrix.config.conc }}]' and conc), which fails template
expansion for multinode agentic entries where conc is a list. Mirror
e2e-tests.yml: toJson() for conc-list and conc[0] for the first point.
Both first-pass sweep attempts of the 3p2d TEP8/TP8 topology died to prefill CUDA OOM on serving-time transients (sparse-indexer logits alloc during c64, DeepGEMM JIT module load during c4) with <1 GiB free per GPU at gpu-memory-utilization 0.9. MLA KV is compact, so 0.85 trades unneeded KV space for allocator headroom. Decode stays at 0.9; all 2p1d DEP8/DEP8 chunks passed untouched.
964b9d8 to
5fc7c84
Compare
cquil11
added
the
agentx
ChatGPT Codex Connector
matrix.config.conc for multi-node agentic entries is a JSON array
(chunked concurrencies per allocation), but sweep-multi-node-agentic
passed it directly into benchmark-multinode-tmpl.yml's `conc` input,
which is declared `type: string` ("First concurrency for
agentic-coding scenarios; CONC_LIST carries the full batch"). GitHub
Actions' reusable-workflow input validator rejects a sequence value
for a string-typed input at evaluation time, so the whole job failed
to even load:
evaluate reusable workflow inputs: .github/workflows/run-sweep.yml
(Line: 554, Col: 19): A sequence was not expected
Since the job never materializes when this happens, sweep-multi-node-
agentic silently disappeared from run summaries entirely instead of
showing as failed. Slice to the first element (matching the intended
"first concurrency" semantics and the same fix already applied
elsewhere, e.g. PR #2122) to restore a scalar value.
Co-authored-by: Cursor <cursoragent@cursor.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29028294481 |
3 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29028294481 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29028294481 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29028294481 |
# Conflicts: # .github/workflows/run-sweep.yml # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29309350340 |
matrix.config.conc for multi-node agentic entries is a JSON array
(chunked concurrencies per allocation), but sweep-multi-node-agentic
passed it directly into benchmark-multinode-tmpl.yml's `conc` input,
which is declared `type: string` ("First concurrency for
agentic-coding scenarios; CONC_LIST carries the full batch"). GitHub
Actions' reusable-workflow input validator rejects a sequence value
for a string-typed input at evaluation time, so the whole job failed
to even load:
evaluate reusable workflow inputs: .github/workflows/run-sweep.yml
(Line: 554, Col: 19): A sequence was not expected
Since the job never materializes when this happens, sweep-multi-node-
agentic silently disappeared from run summaries entirely instead of
showing as failed. Slice to the first element (matching the intended
"first concurrency" semantics and the same fix already applied
elsewhere, e.g. PR #2122) to restore a scalar value.
Co-authored-by: Cursor <cursoragent@cursor.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29310002441 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29310002441 |
…2170) * [AMD][AgentX] Add DeepSeek V4 MI355X agentic disagg * run-sweep: fix broken conc-list interpolation for multi-node agentic matrix.config.conc for multi-node agentic is already a JSON array (e.g. [16,32,64] from generate_sweep_configs.py). Wrapping it in a string literal as '[${{ matrix.config.conc }}]' can't correctly interpolate an array into a string, producing a malformed conc-list so fromJson(inputs.conc-list) in the reusable template ended up with an empty/wrong CONC_LIST. Use toJson(matrix.config.conc) instead, matching the pattern already used for the other two conc-list inputs in this file. Co-authored-by: Cursor <cursoragent@cursor.com> * bump image to rocm/sgl-dev:sglang-0.5.14-rocm720-mi35x-mori-0706 Signed-off-by: Theresa Shan <theresa.shan@amd.com> * run-sweep: fix sequence-typed conc input for multi-node agentic matrix.config.conc for multi-node agentic entries is a JSON array (chunked concurrencies per allocation), but sweep-multi-node-agentic passed it directly into benchmark-multinode-tmpl.yml's `conc` input, which is declared `type: string` ("First concurrency for agentic-coding scenarios; CONC_LIST carries the full batch"). GitHub Actions' reusable-workflow input validator rejects a sequence value for a string-typed input at evaluation time, so the whole job failed to even load: evaluate reusable workflow inputs: .github/workflows/run-sweep.yml (Line: 554, Col: 19): A sequence was not expected Since the job never materializes when this happens, sweep-multi-node- agentic silently disappeared from run summaries entirely instead of showing as failed. Slice to the first element (matching the intended "first concurrency" semantics and the same fix already applied elsewhere, e.g. PR #2122) to restore a scalar value. Co-authored-by: Cursor <cursoragent@cursor.com> * agentic: run one task per concurrency for multi-node sweeps Multi-node agentic sweeps batched up to 4 concurrencies per SLURM allocation, running them sequentially against one shared server session. A slow/hung conc could block the rest of the batch from ever producing results, which is why run #6719 only reported c16 despite a 16/32/48/64 conc-list. Drop the batch size to 1 so each concurrency gets its own task/allocation, matching the granularity already used for single-node agentic sweeps. Cherry-picked from backup/agentx-v1.0-rebase-pre-upstream-rewrite-20260710 (082a59d), adapted for the current test suite. Co-authored-by: Cursor <cursoragent@cursor.com> * Bump image to latest upstream image with 4 PR fixes included. * update con-list Signed-off-by: Theresa Shan <theresa.shan@amd.com> * setup_deps: remove disabled decode_tp_queue_agree patch patch_decode_tp_queue_agree()'s invocation was already commented out (dead code with no runtime effect), and the reference-only patches/decode_tp_queue_agree.patch it mirrored had no other callers. Drop both plus the README bullet pointing at it. Co-authored-by: Cursor <cursoragent@cursor.com> * setup_deps: broaden GLM transformers gate; disable prefill bootstrap-desync patch install_transformers_glm5() was gated on an exact MODEL_NAME == "GLM-5-FP8" match; broaden to any model name containing "GLM" so other GLM variants pick up the same glm_moe_dsa transformers fix. Also disable patch_disagg_prefill_bootstrap_desync's invocation (commented out, matching the already-disabled decode_tp_queue_agree pattern removed earlier). Co-authored-by: Cursor <cursoragent@cursor.com> * agentic: node-0 sibling benchmark-client container for DSv4 sweeps (#2147) * agentic: add node-0 sibling benchmark-client container for DSv4 sweeps Port the "same-node sibling container" client mode from ROCm/InferenceY: when CLIENT_IMAGE is set (and no CLIENT_NODES), node 0 launches the aiperf trace replay in its own pre-baked container via the host docker socket, instead of rebuilding the aiperf venv inside the server container and running it co-located. This keeps the client's CPU-heavy tokenize/aggregate work off the sglang scheduler + router, which inflates TTFT/E2E and lowers throughput under agentic concurrency. - server_sglang.sh: add IS_AGENTIC_RUN and a CLIENT_IMAGE sibling-container branch that writes client.env and docker-runs the client against the local router (--network host). - job.slurm: define CLIENT_CONT_NAME; when CLIENT_IMAGE is set, mount the host docker socket + CLI into the server container, forward HOST_REPO_DIR/HOST_MODEL_DIR/HOST_BENCH_LOGS/CLIENT_CONT_NAME, pre-pull the client image, and clean up the client container on teardown. - amd-master.yaml: enable the sibling client on dsv4-fp4-mi355x-sglang-disagg-agentic-hicache via CLIENT_IMAGE. The separate-client-NODE mode is intentionally not ported. * agentic: use server image for sibling client so upstream CI can pull it The pre-baked rocm/pytorch-private aiperf client image is not pullable by upstream CI runners. Reuse the (public) server image as CLIENT_IMAGE and build aiperf on the fly from /workspace/utils/aiperf, matching the co-located path. Gate the pre-baked-venv env (AIPERF_USE_PREBUILT / AIPERF_VENV) behind an optional CLIENT_AIPERF_VENV so a real pre-baked client image can still opt in. * enable log info and extend timeout Signed-off-by: Theresa Shan <theresa.shan@amd.com> * fix: address PR 2170 review comments Update the DSv4 MI355X agentic perf-changelog entry to point at PR 2170 with an accurate description, and move hf_dir into the DeepSeek-V4-Pro YAML block so the SGLang model path extractor can read it. 中文：修复 PR 2170 的 review 反馈：将 DSv4 MI355X agentic 的 perf-changelog 条目改为指向 PR 2170 并更新为准确描述，同时把 hf_dir 移入 DeepSeek-V4-Pro YAML 配置块，确保 SGLang 模型路径提取逻辑可以读取。 * setup_deps: remove prefill bootstrap-desync patch entirely The disaggregation-prefill bootstrap-desync patch was already disabled (commented out); drop the function, its comment block, and the stale invocation now that the fix is carried upstream in the pinned image. Co-authored-by: Cursor <cursoragent@cursor.com> * fix perf changelog * consolidate sglang agentic envs Signed-off-by: Theresa Shan <theresa.shan@amd.com> * select models.yaml recipe by IS_AGENTIC (AgentX/DI variants) Resolve the models.yaml entry per run type: agentic runs (IS_AGENTIC) use the '<model>-AgentX' recipe, non-agentic disaggregated runs use '<model>-DI', falling back to the bare model name when no variant key exists. Apply the same logic in both server_sglang.sh (lookup) and job.slurm (validation guard) so a bare MODEL_NAME (e.g. DeepSeek-V4-Pro) correctly maps to the suffixed key. Add DeepSeek-V4-Pro-DI and rename the agentic recipe to DeepSeek-V4-Pro-AgentX. Co-authored-by: Cursor <cursoragent@cursor.com> * DPTP enabling. Signed-off-by: Theresa Shan <theresa.shan@amd.com> * tune hicache ratio, router barrier timeout, and agentic conc sweep - server_sglang.sh: lower default HICACHE_RATIO 24->5 (cut host-pool pin/registration startup cost) and raise router health barrier timeout 1800->3000 so a slow server bring-up no longer trips the cross-node router wait first. - amd-master.yaml: restore conc 32 on the non-DP-attn (TP8) point and comment out the DP-attn c64/128 sweep block. - models.yaml: drop explicit max_total_tokens override for the AgentX dp variant (let it derive from context_length). Co-authored-by: Cursor <cursoragent@cursor.com> * raise hicache ratio to 16 Signed-off-by: Theresa Shan <theresa.shan@amd.com> * fix: restore qwen3.5 changelog entry removed during rebase perf-changelog.yaml must only gain entries; re-adding the upstream qwen3.5-fp8-gb300-dynamo-sglang block fixes process_changelog.py. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: validate agentic hicache master config for changelog gate Use kv-offload-backend object form and add kv-p2p-transfer: mori for the disaggregated dsv4-fp4-mi355x-sglang-disagg-agentic-hicache entry. Co-authored-by: Cursor <cursoragent@cursor.com> * feat: wire KV_P2P_TRANSFER env into SGLang launch flags Override models.yaml --disaggregation-transfer-backend when kv-p2p-transfer from amd-master.yaml is set via CI, and forward KV_P2P_TRANSFER to the agentic client env for result metadata. Co-authored-by: Cursor <cursoragent@cursor.com> * chore: drop max_total_tokens from DSv4 AgentX prefill no_dp recipe Let SGLang auto-size the prefill KV pool instead of pinning 1M tokens. Co-authored-by: Cursor <cursoragent@cursor.com> * tune: raise router barrier timeout to 3000s for worker nodes Give prefill/decode workers more time to wait for the node-0 proxy during long agentic server startup. Co-authored-by: Cursor <cursoragent@cursor.com> * test: fix sweep config fixtures for kv-p2p-transfer validation Add kv-p2p-transfer to disagg multinode test configs and use KVOffloadBackendMetadata dict form for kv-offload-backend fields. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor: centralize sync barrier timeouts for DSV4 startup Extract SYNC_BARRIER_TIMEOUT (3000s for DeepSeek-V4, 1800s otherwise) and apply it to server-up, router readiness, and prefill/decode proxy barriers. Also lower the default HICACHE_RATIO from 16 to 5. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: forward KV offload backend metadata to agentic client Pass KV_OFFLOAD_BACKEND_METADATA into server containers via job.slurm and forward it to sibling client containers in server_sglang client.env so process_agentic_result can write aggregate JSON. Derive KV_OFFLOAD_BACKEND from metadata in the DSv4 agentic recipe when unset. Co-authored-by: Cursor <cursoragent@cursor.com> * revert: drop metadata-derived KV_OFFLOAD_BACKEND from DSv4 recipe CI already sets KV_OFFLOAD_BACKEND from the workflow; keep the job.slurm and server_sglang metadata forwarding fixes without the redundant recipe logic. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: quote KV_OFFLOAD_BACKEND_METADATA in docker run env forwarding toJson() pretty-prints with embedded newlines, so the unquoted expansion in job.slurm's DOCKER_ENV_COMMON was word-split across the docker run command, corrupting its argument list ("docker: invalid reference format", exit 125). Quote the value so the whole JSON stays one token. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix client.env generation to flatten multi-line env values docker run --env-file requires strict single-line KEY=VALUE entries. KV_OFFLOAD_BACKEND_METADATA carries pretty-printed multi-line JSON, which split into invalid env-file lines (e.g. '"name": "hicache",') and aborted the sibling client container launch with "docker: invalid env file ... contains whitespaces". Strip embedded newlines before writing each value so the JSON stays valid but single-line. Co-authored-by: Cursor <cursoragent@cursor.com> * Re-serialize KV_OFFLOAD_BACKEND_METADATA as compact JSON instead of stripping newlines Naive newline-stripping could silently corrupt a value if a JSON string ever contained a literal embedded newline (merging tokens across the line break). Round-tripping through json.loads/json.dumps guarantees a correct compact single-line representation for client.env, and fails loudly with a clear error if the value isn't valid JSON. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix regression: skip JSON round-trip for empty KV_OFFLOAD_BACKEND_METADATA job.slurm always forwards KV_OFFLOAD_BACKEND_METADATA into the server container, even as "" when KV offloading is disabled (the common default case). The prior commit unconditionally ran it through json.loads/json.dumps, which raised JSONDecodeError on that empty string and hard-failed every agentic client-container launch without KV offloading. Only attempt the JSON round-trip when the value is non-empty and not "null", matching how optional_kv_offload_backend_metadata() already treats those as "no metadata" rather than invalid JSON. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(changelog): append PR entry after main * fix(changelog): omit null optional metadata * fix(agentx): set DeepSeek reasoning effort high * Remove stale patches/README.md documenting an already-retired mori_conn.py overlay mori_conn.py and the job.slurm auto-apply block it documented were already deleted upstream (retire MoRI conn.py PD-disagg overlay, aca9cee) once the real sglang fix landed in published images. A later unrelated commit (183a987) accidentally recreated this README from scratch while only intending to drop a bullet about a different, already-removed patch (decode_tp_queue_agree.patch), leaving stale docs pointing at a file and job.slurm block that no longer exist. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(agentx): enforce agentic DRAM offload budget for multinode disagg Multinode agentic entries hardcoded total-cpu-dram-gb=0, so the DRAM KV-offload pool was never bounded at runtime (HiCache fell back to a relative --hicache-ratio). Compute a real per-node budget from the prefill worker (the only KV offloader today) and wire it through to a --hicache-size cap. - generate_sweep_configs: fold multinode into agentic_dram_offload_gb; budget the prefill worker's per-node GPU footprint via worker_gpus_per_node, which rejects node-misaligned topologies instead of silently truncating. - validation: add total-cpu-dram-gb to MultiNodeAgenticMatrixEntry. - run-sweep: forward total-cpu-dram-gb to the multi-node agentic dispatch. - server_sglang.sh: size prefill HiCache via absolute --hicache-size (budget / ranks-per-node / host-pools), falling back to --hicache-ratio when no budget is provided; drop the dead HICACHE_TOTAL_CPU_DRAM_GB default. * Fix indentation of dsv4-fp4-b200-vllm-agentic entry in perf-changelog.yaml The config-keys entry was missing its leading indentation, breaking YAML list alignment with sibling entries. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agentx): skip --hicache-size for DeepSeek V4 in multinode HiCache DeepSeek V4's hybrid HiCache pool assembler rejects --hicache-size and requires --hicache-ratio, so the absolute per-node DRAM budget cannot be applied to it. Restrict the --hicache-size enforcement to non-V4 models; V4 falls back to --hicache-ratio. Budget stays recorded in the config. 中文：多节点 HiCache 对 DeepSeek V4 跳过 --hicache-size。DeepSeek V4 的 混合 HiCache 池不支持 --hicache-size、必须用 --hicache-ratio,因此绝对 的每节点 DRAM 预算无法作用于它。将 --hicache-size 限制为非 V4 模型; V4 回退到 --hicache-ratio。预算仍记录在 config 中。 --------- Signed-off-by: Theresa Shan <theresa.shan@amd.com> Co-authored-by: Cam Quilici <cjquilici@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: AMD-yanfeiwang <yanfei.wang@amd.com> Co-authored-by: Duyi-Wang <duyi.wang@amd.com> Co-authored-by: billishyahao <bill.he@amd.com>
1 participant
Perf-changelog entry to run the full agentic-coding sweep of the DSv4 FP4 GB200 dynamo-vllm disagg topologies on the June 21 AgentX corpus:
dsv4-fp4-gb200-dynamo-vllm-agentic-3p2d-tep8-tp8— 3 TEP8 prefill + 2 TP8 decode workers (40 inference GPUs), concurrencies 4-80 in three chunks.dsv4-fp4-gb200-dynamo-vllm-agentic-2p1d-dep8-dep8— 2 DEP8 prefill + 1 DEP8 decode workers (24 inference GPUs), concurrencies 32-256 in three chunks, including the exploratory tail past the measured c160 normalized-throughput peak.No config or recipe changes; changelog-only to trigger the sweep of the existing search space.