[AMD][DSV4][Disagg] Enable UMBP, DP-attention tuning, FP4 sparse-attention indexer optimizing - #2823
Conversation
…perts fusion and chunked-prefill sizing - models.yaml: remove the leftover --disable-shared-experts-fusion that was silently overriding --enforce-shared-experts-fusion in base_flags (the loader gives disable unconditional priority), and make prefill.no_dp.chunked_prefill_size TP-conditional (16384 @ TP8, 8192 @ TP4) to fix the conc>=16 queue-saturation/decode-stall failure mode, mirroring the single-node dsv4-fp4-mi355x-sglang-agentic-mtp fix (#2784). - env.sh: comment out DSv4 kernel-routing env vars that now match v0.5.18 defaults, add TORCH_BLAS_PREFER_HIPBLASLT and SGLANG_OPT_USE_AITER_BATCHED_GEMM, lower GPU_MAX_HW_QUEUES from 5 to 2. - amd-master.yaml: bump dsv4-fp4-mi355x-sglang-disagg-agentic-hicache-mtp to image v0.5.18-rocm720-mi35x-20260829 and add a TP4/EP1 no-offload arm at conc-list [1,2,4,8,16]. - perf-changelog.yaml: record the above. Co-authored-by: Cursor <cursoragent@cursor.com>
… bump image - env.sh: only export the DP-attention-only SGLang knobs (SGLANG_SHARED_EXPERT_TP1, SGLANG_DP_SHARED_EXPERT_LOCAL, SGLANG_DP_USE_GATHERV, SGLANG_DP_USE_REDUCE_SCATTER, wider GPU_MAX_HW_QUEUES) when PREFILL_ENABLE_DP or DECODE_ENABLE_DP is true, so TP-only arms keep GPU_MAX_HW_QUEUES=2 instead of inheriting the DP tuning unconditionally. Also add SGLANG_MORI_RECV_BOUND=1 for the disagg MoRI path. - models.yaml: round out dp_flags with --tokenizer-worker-num 8, --enable-dp-attention-local-control-broadcast, --stream-interval 20, and --prefill-decode-interval 10 to mirror the single-node recipe's DP-attention block; add enable_two_batch_overlap under prefill.dp; raise mem_fraction_static 0.85 -> 0.90 for both prefill and decode. - amd-master.yaml: bump the image to v0.5.18-rocm720-mi35x-20260831, switch the active search-space arm's decode side from ep:8 to ep:1 (DP-attention over TP-sharded MoE instead of EP), and widen its conc-list to [64, 96, 128, 192, 256]. Keep the TP4/TP8 no-DP arms as commented-out reference. Co-authored-by: Cursor <cursoragent@cursor.com>
…ags/no_dp_flags buckets sglang documents --enforce-shared-experts-fusion and --disable-shared-experts-fusion as mutually exclusive, but DeepSeek-V4-Pro-AgentX's base_flags carried the enforce flag unconditionally while dp_flags added the disable flag on top -- both landed on the same command line for the DP arm, only resolving correctly because DeepseekV4ForCausalLM.determine_num_fused_shared_experts() happens to check disable_shared_experts_fusion first. That's an internal implementation detail, not a guaranteed contract, and reads as a contradiction to anyone inspecting the launch command. - models.yaml: add a new no_dp_flags field (mirrors dp_flags, applied when DP attention is off) and move --enforce-shared-experts-fusion out of base_flags into it, so the two conflicting flags are never emitted together for any arm. - server_sglang.sh: parse MODEL_NO_DP_FLAGS and apply it in build_server_config() whenever enable_dp is false, alongside the existing dp_config branch. Co-authored-by: Cursor <cursoragent@cursor.com>
DeepSeek-V4-Pro-AgentX's decode.dp arm now serves concurrency up to 256 (configs/amd-master.yaml conc-list), so capture CUDA graphs up to bs=256 instead of 128 to keep the graph replay path covering the full sweep. Co-authored-by: Cursor <cursoragent@cursor.com>
Scope the sweep to just the highest concurrency point while iterating on the DP-attention arm, instead of re-running the full [64, 96, 128, 192, 256] sweep each time. Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: Theresa Shan <theresa.shan@amd.com>
ratio=6 (eef805c) eliminated host-pool saturation but regressed throughput -11% and TTFT +65%/+107% (mean/p90) vs ratio=3 at conc=256: avg host-pool occupancy barely moved (~15.5M tokens either way), so the extra capacity was mostly unused while lookup/restore cost scaled with the larger pool, hurting prefill-side TTFT specifically. ratio=4 targets the ~25M-token peak observed on the busiest rank (above ratio=3's ~19.5M cap) without doubling pool size like ratio=6. Co-authored-by: Cursor <cursoragent@cursor.com>
- DP-attention gating, EP1 tuning, and image bump to ...20260831 - Mutually-exclusive no_dp_flags/dp_flags split for shared-experts fusion - Widened decode cuda-graph bs range and conc-list narrowing - HICACHE_RATIO 3->6->4 investigation and rationale - FP4 sparse-attention indexer + image bump to ...20260903 (sglang PR #37353) Co-authored-by: Cursor <cursoragent@cursor.com>
…full search space - models.yaml: add --enable-deepseek-v4-fp4-indexer to DeepSeek-V4-Pro-AgentX base_flags now that sglang gfx95 support has landed upstream (PR #37353); lower prefill/decode mem_fraction_static 0.90 -> 0.86 to make room for the larger FP4 indexer working buffers within the same GPU memory budget. - amd-master.yaml: bump the disagg AgentX HiCache image from v0.5.18-rocm720-mi35x-20260831 to ...20260903 (first tag carrying the FP4 indexer gfx95 kernels). Restore the previously commented-out TP4/TP8 no-DP offload arms (conc-list [1,4,8,10] and [16]) and the TP8 HiCache no-DP arm (conc-list [32,48,64]) as active search-space entries, and widen the DP-attention arm's conc-list back from [256] to the full [64, 96, 128, 192, 256] sweep now that FP4-indexer-driven KV headroom changes the capacity picture. Drop the HICACHE_RATIO=4 override (and its explanatory comment) so the arm falls back to server_sglang.sh's default ratio, since the FP4 indexer's smaller per-token KV footprint (68B vs 132B) changes the sizing math behind that ratio=3->6->4 investigation. Co-authored-by: Cursor <cursoragent@cursor.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
ichbinblau
added
AMD
agentx
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.
Beyond the inline finding, I also checked the DP arm's new chunked_prefill_size formula for TP8 (16384 * PREFILL_TP_SIZE, i.e. 131072) against the "compressor kernel uint16 token cap (255*256)" comment retained on that line — the actual per-rank dispatch cap enforced by the mori kernel is governed separately by MORI_MAX_DISPATCH_TOKENS_PREFILL in env.sh (unchanged at 8192), so chunked_prefill_size doesn't feed that limit and the doubling isn't a functional regression, just a stale comment.
Extended reasoning...
This is the brief findings-present ruled-out note: one inline finding (perf-changelog.yaml missing a mem_fraction_static entry) is already reported. I additionally traced the DP arm's chunked_prefill_size formula change (models.yaml prefill.dp, ~line 392) that doubles to 131072 for TP8 versus the old MORI_MAX_DISPATCH_TOKENS_PREFILL-derived value, and confirmed via server_sglang.sh/env.sh that the actual per-rank mori dispatch cap referenced by the adjacent comment is controlled by the separate, unchanged env var MORI_MAX_DISPATCH_TOKENS_PREFILL=8192, not by chunked_prefill_size itself — so the doubling does not violate the documented compressor-kernel constraint, it's just a stale comment left over from when the two were tied together.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33837253408 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34074758948 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34926284365 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=34926284365 |
|
/stage-results 34926284365 |
|
/reuse-sweep-run 34926284365 |
|
@cquil11 staged run 34926284365: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-09-15~r34926284365 This run remains available across future |
4 participants
Summary
Tunes the
dsv4-fp4-mi355x-sglang-disagg-agentic-hicache-mtp(DeepSeek-V4-Pro-AgentX, MI355X, disaggregated, HiCache DRAM offload) recipe:DP-attention rework
SGLANG_SHARED_EXPERT_TP1,SGLANG_DP_SHARED_EXPERT_LOCAL,SGLANG_DP_USE_GATHERV,SGLANG_DP_USE_REDUCE_SCATTER, widerGPU_MAX_HW_QUEUES) behindPREFILL_ENABLE_DP/DECODE_ENABLE_DPinenv.sh, instead of exporting them unconditionally, so TP-only arms keepGPU_MAX_HW_QUEUES=2. AddSGLANG_MORI_RECV_BOUND=1for the disagg MoRI path.dp_flagsinmodels.yaml(--tokenizer-worker-num 8,--enable-dp-attention-local-control-broadcast,--stream-interval 20, ,enable_two_batch_overlap) to mirror the single-node DP-attention recipe.ep:8toep:1(DP-attention over TP-sharded MoE instead of expert-parallel dispatch).decode.dp.cuda_graph_bs_rangefrom1-128to1-256and the DP-attention arm'sconc-listback to the full[64, 96, 128, 192, 256]sweep.FP4 sparse-attention indexer
lmsysorg/sglang-rocm:v0.5.18-rocm720-mi35x-20260831to...202609011, the first tag carrying gfx95 (MI350X/MI355X) support for DeepSeek-V4's FP4 sparse-attention indexer via AITER kernels (sglang PR #37353, merged 2026-09-02). Previously the flag was gated to NVIDIA SM100/SM120 only and hard-failed on AMD.--enable-deepseek-v4-fp4-indexertoDeepSeek-V4-Pro-AgentXbase_flags.mem_fraction_static0.90 -> 0.86 in DP case.Use umbp-linker as KV offload backend for high concurrencies (128, 192, 256)
Umbp linker helps to improve the tput and reduce TTFT at high concurrency.
Note
Medium Risk
Touches disaggregated SGLang launch, KV offload paths, and benchmark concurrency limits; failures would show up as job timeouts or wrong throughput numbers rather than production user impact.
Overview
Updates the MI355X disaggregated DeepSeek-V4-Pro AgentX recipe to Pro-0813 on sglang v0.5.19, with
--enable-deepseek-v4-fp4-indexerand refreshed DSv4 env/kernel routing (DP-attention knobs gated on role,SGLANG_MORI_RECV_BOUND, wildcardDeepSeek-V4-Pro*matching).KV offload: Documents HiCache vs a new
umbp-linkerpath (separate from HiCache; prefill-only, DP-attention required). Adds recipe/Slurm env for UMBP,umbp_standalone_serverstartup with multi-stage readiness waits, linker flag auto-detection, and configurableCONTAINER_BARRIER_TIMEOUT. HiCache defaultHICACHE_RATIOis 2.5 (was 3). High-concurrency sweep arms use UMBP linker instead of HiCache.Harness / models:
models.yamlAgentX entry scalesmax_running_requestsand CUDA-graph ranges withBENCH_MAX_CONC_VALUE, per-DP/no-DPmem_fraction_static, updated flags (shared-expert fusion, DP broadcast), andDeepSeek-V4-Pro-0813-AgentXalias.server_sglang.shevaluates YAML formulas for graph ranges, caps decode concurrency by graph×TP, uses role-specific mem fractions, drops--prefill-round-robin-balance, and wires Pro-0813 synthetic MTP golden AL (3.01 for 3-step).amd-master.yamlexpands the sweep (TP4/TP8, HiCache at 32/48, UMBP+DP at 128/192/256).Reviewed by Cursor Bugbot for commit ebace4f. Bugbot is set up for automated code reviews on this repo. Configure here.