Repository navigation
Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| PYTHONUNBUFFERED: "1" | ||
| SGLANG_TIMEOUT_KEEP_ALIVE: "900" | ||
| SGLANG_OPT_UNIFIED_CACHE_FREE_OUT_OF_WINDOW_SLOTS: "1" | ||
| SGLANG_TRTLLM_MHA_DECODE_SEQ_LEN_SPLITS: "1" | ||
| SGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION: "0" | ||
| SGLANG_TRTLLM_MHA_FULL_ATTN_KV_SPLITS: "16" | ||
| SGLANG_TRTLLM_MHA_FULL_ATTN_KV_SPLITS_MIN_BS: "4" |
There was a problem hiding this comment.
🔴 Muse Spark agentic runs could get turns rejected with HTTP 529 once a worker fills up. The base branch's Dynamo 1.6 did not have this backend admission check. The image's Dynamo 1.7 admits only a limited number of requests per engine, and almost every new point runs above max-running-requests. Fix: set DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT, DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT and DYN_BACKEND_ADMISSION_REQUEST_QUEUE_LIMIT to "10000" on every engine role, which covers the 3 sites listed. Same pattern at 3 sites (agg-variants.yaml:54, disagg-variants.yaml:55, disagg-variants.yaml:113).
Why this was flagged
The trigger is load above what one engine has room for, under the Dynamo 1.7.0.dev20261102 that the image ships. The base branch installed the 1.6.0.dev20261016 wheel, but dynamo.install: false at agg-variants.yaml:9 and disagg-variants.yaml:9 now uses the image's 1.7 build. Every other dynamo-sglang 1.7 key in the repo raises these three limits to "10000", in dsv41flash agg-variants.yaml:69-73 and minimaxm31flash sglang agg-variants.yaml:63-65. Their comment says "Dynamo 1.7 backend admission rejects turns with 529 once the engine is full". The new points routinely run beyond that capacity. TP4 c46/c50/c56 cap max-running-requests at 32. Disagg decode capacity is 4x56=224 at c672. Rejected turns would fail the AgentX run or skew its throughput and interactivity results compared with the 1.6 runs on the base branch. Neither recipe sets any DYN_BACKEND_ADMISSION_* variable.
Verification: The diff changes the base's wheel: "1.6.0.dev20261016" to dynamo.install: false (agg-variants.yaml:9; also disagg-variants.yaml:9). Neither musespark recipe sets DYN_BACKEND_ADMISSION_*. Agg c46/c50/c56 run against max-running-requests: 32 on a single TP4 engine.
There was a problem hiding this comment.
Not applicable to this image. Dynamo 1.7.0.dev20261102 includes ai-dynamo/dynamo#17481 (merged 2026-10-31), which made backend admission opt-in: with no engine limit set, the gate is bypassed (no queueing, no 529). Every run of these recipes logs Backend admission configuration engine_request_limit=unlimited engine_request_waiting_limit=unlimited request_queue_limit=0, and no 529s occurred. The dsv41flash / minimaxm31flash recipes set 10000 because they targeted earlier 1.7 builds, where admission was on by default. Keeping the recipes as measured.
shyeh25
added
the
full-sweep-fail-fast
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=41854172505 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=41854172505 |
|
[by Claude Code] The SGLang cookbook integration of this PR's AgentX configs ( This PR has no authorized |
…ted points and new TP8 disaggregated points Move the GB300 Muse Spark Open Dynamo + SGLang AgentX lane to lmsysorg/sglang:nightly-dev-cu13-20261103-797af717 (digest pinned) with its bundled ai-dynamo 1.7.0.dev20261102. Rename the aggregated key to musespark-fp4-gb300-dynamo-sglang-agentic-dflash-agg with TP4 c46/c50/c56 (HiCache 140 GB per rank) and TP8 c1/c36, add musespark-fp4-gb300-dynamo-sglang-agentic-dflash-disagg with nine TP8 prefill / TP8 decode points (c32 to c672), and remove musespark-fp4-gb300-dynamo-sglang-agentic-dflash. 将 GB300 Muse Spark Open Dynamo + SGLang AgentX 迁移到 lmsysorg/sglang:nightly-dev-cu13-20261103-797af717（固定 digest），使用镜像自带的 ai-dynamo 1.7.0.dev20261102。聚合 key 改名为 -agg（TP4 c46/c50/c56，每 rank HiCache 140 GB；TP8 c1/c36），新增 -disagg（九个 TP8 prefill / TP8 decode 点， c32 至 c672），并移除原 key。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…G_TRTLLM_MHA_FULL_ATTN_KV_SPLITS Drop SGLANG_TRTLLM_MHA_DECODE_SEQ_LEN_SPLITS and SGLANG_TRTLLM_MHA_FULL_ATTN_KV_SPLITS_MIN_BS from the aggregated and disaggregated recipes; both fall back to the SGLang defaults (1 and 12). 聚合与分离式配方移除 SGLANG_TRTLLM_MHA_DECODE_SEQ_LEN_SPLITS 与 SGLANG_TRTLLM_MHA_FULL_ATTN_KV_SPLITS_MIN_BS，改用 SGLang 默认值（1 与 12）。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
b37254d to
704ebb2
Compare
|
[by Claude Code] sgl-project/sglang#48392 has been merged. This PR's AgentX configs ( This PR has no authorized |
…X client to load its trace dataset The recipe-level HF_HUB_OFFLINE=1 is propagated to the benchmark step, so AIPerf cannot load semianalysisai/cc-traces-weka-062126 when its dataset cache misses (OfflineModeIsEnabled). Set HF_HUB_OFFLINE=0 in benchmark.env for the aggregated and disaggregated recipes; workers stay offline. 基准测试步骤会继承配方顶层的 HF_HUB_OFFLINE=1，数据集缓存未命中时 AIPerf 无法加载 semianalysisai/cc-traces-weka-062126（OfflineModeIsEnabled）。 聚合与分离式配方在 benchmark.env 设置 HF_HUB_OFFLINE=0，推理 worker 仍保持离线。 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
[by Claude Code] The SGLang cookbook update of this PR's AgentX configs ( This PR has no authorized |
|
[by Claude Code] sgl-project/sglang#48634 has been merged. This PR's AgentX configs ( This PR has no authorized |
Summary
Moves the GB300 Muse Spark Open Dynamo + SGLang AgentX lane to
lmsysorg/sglang:nightly-dev-cu13-20261103-797af717(multiarch digest pinned) and runs the ai-dynamo1.7.0.dev20261102that the image ships (dynamo.install: false). It replaces the aggregated points under a new-aggkey, adds TP8 disaggregated points under a new-disaggkey, and removesmusespark-fp4-gb300-dynamo-sglang-agentic-dflash(#4344).musespark-fp4-gb300-dynamo-sglang-agentic-dflash-agg(new)musespark-fp4-gb300-dynamo-sglang-agentic-dflash-disagg(new)SGLANG_TRTLLM_MHA_FULL_ATTN_KV_SPLITS=16).SGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION=0on every role.AI model disclosure
PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
将 GB300 Muse Spark Open Dynamo + SGLang AgentX 配置迁移到
lmsysorg/sglang:nightly-dev-cu13-20261103-797af717（固定 multiarch digest），并直接使用镜像自带的 ai-dynamo1.7.0.dev20261102（dynamo.install: false）。本 PR 在新的-aggconfig key 下替换聚合配置点，在新的-disaggconfig key 下新增 TP8 分离式配置点，并移除musespark-fp4-gb300-dynamo-sglang-agentic-dflash（#4344）。musespark-fp4-gb300-dynamo-sglang-agentic-dflash-agg（新增）：musespark-fp4-gb300-dynamo-sglang-agentic-dflash-disagg（新增），prefill 与 decode worker 均为 TP8：SGLANG_TRTLLM_MHA_FULL_ATTN_KV_SPLITS=16）。SGLANG_FLASHINFER_NVFP4_PER_TOKEN_ACTIVATION=0。AI 模型使用说明：Claude Opus 5.5；负责调优与配置修改。
🤖 Generated with Claude Code