Fix for https://github.com/sgl-project/sglang/issues/22072 / 修复 SGLang MoRI 分离式推理在 MI325X 上的挂起问题 - #1806
Fix for https://github.com/sgl-project/sglang/issues/22072 / 修复 SGLang MoRI 分离式推理在 MI325X 上的挂起问题#1806davzhuAMD wants to merge 12 commits into
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
… mentioned in #22072.
| export REBUILD_LIBBNXT_IN_CONTAINER=1 | ||
| export PATH_TO_BNXT_TAR_PACKAGE=/workspace/driver/libbnxt_re-237.1.137.0.tar.gz | ||
|
|
||
| export PREFILL_NODE="45.63.71.103" |
There was a problem hiding this comment.
The start_test_*.sh scripts hardcode many cluster-specific values (real node IPs like 45.63.71.103 / 137.220.60.12 and Docker image tags) directly in a public repo. Maybe we could parameterize these (e.g. via env vars) instead of committing concrete cluster details.
There was a problem hiding this comment.
Thanks for your comments. These test scripts themselves (including the new start_test_, run_1p, and start_sglang*) don't necessarily need to make it into the final repo, but I added them to illustrate how we investigated and reproduced the original issue on our side. If it makes things clearer, I can remove the helper scripts from the commit and attach them somewhere else instead, leaving just the actual fixes in this PR.
| # IPADDRS order: prefill_ip,decode1_ip,decode2_ip. NNODES=3, xP=1, yD=1. | ||
| # | ||
| # Prerequisites: | ||
| # - Passwordless SSH from this machine to both nodes |
There was a problem hiding this comment.
Should this say 3 nodes? This script launches 1 prefill + 2 decode nodes (NNODES=3), but the comments/header still say "both nodes", which were copy-pasted from the single-decode launcher and read as if there are only 2 nodes total.
…er concurrencies. - Increase the decode dispatch buffer size from 512 to 4096 and correctly pass it along for the DP and EP cases, avoiding a stall under load. - Correctly set MORI parameters when prioritizing traffic as lossless, which otherwise interferes with the data transfers that the tests generate.
…e final commit. They'll remain availabel in the sglang-pd-mi325x branch.
There was a problem hiding this comment.
Guidance would be appreciated for how to handle a library file like this. During investigation and debug, we requested an update of our test nodes' NIC firmware and driver. However, the Docker image we used didn't have the updated driver itself, so our test scripts had to pass export REBUILD_LIBBNXT_IN_CONTAINER=1 to patch in this updated driver. In theory, using an updated image with a later compatible NIC driver should make this archive unnecessary to commit.
There was a problem hiding this comment.
Regarding my comment on libbnxt_re-237.1.137.0.tar.gz https://github.com/SemiAnalysisAI/InferenceX/pull/1806/changes#r3432816594, if anyone validates this fix using a Docker image that comes with an updated NIC driver compatible with the host's firmware, then this rebuild_bnxt.sh script becomes unnecessary.
functionstackx
changed the title
…fic both request the same lossless class.
…he scope the actual InferenceX-side change.
|
I removed the added scripts and driver package, as I considered them out of the scope of the InferenceX-side fix for the original issue. Those removed scripts would've mainly covered environment setup, which anyone testing or validating this change would likely need to tailor to their own setup anyway. The scripts still exist at my https://github.com/davzhuAMD/InferenceX/tree/sglang-pd-mi325x as I use them for my own local testing. |
chunfangamd
left a comment
chunfangamd
left a comment
There was a problem hiding this comment.
Thanks @davzhuAMD! The fixes are for MI325X, right? It seems the files are common files that can affect MI300X and MI355 too. Could you please check?
Meanwhile, please trigger an e2e test using your branch for verification.
|
My pleasure, @chunfangamd, and thanks for your comment. The changes to server_sglang.sh, and to the env.sh's NCCL and MORI QoS, are meant to be common, as they resolve what I believe is a genuine defect in the original code. Regardless of what node setup we have, the original code inconsistently set NCCL and MORI RDMA environment variables as one contributor to the original #22072 symptom, and my changes try to resolve that. As for triggering an e2e test, I don't seem to have permission to do so via GitHub Actions. How do I get access? |
…stead of hardcoding them.
… 0.9 now only get set under DSR + MI325X.
|
I updated my branch to set config settings only if it's DeepSeek-R1-0528 and RUNNER_TYPE mi325x, including for the prefill mem_fraction_static 0.9 that I formerly set in models.yaml (now models.yaml is unmodified). On that topic, what would be the best way to introduce MI325X-specific logic within env.sh and server_sglang.sh? Using RUNNER_TYPE like I have here, or getting the device ID somehow? I'm currently retesting. |
|
Thanks @davzhuAMD! The MI325X-specific scoping looks good. Could you please address or explain the following two concerns?
|
|
Thank you for your comments, @chunfangamd.
Good point, I'll remove this setting. The correct value could change depending on the exact cluster setup and layout.
Please see my previous comment regarding E2E. #1806 (comment) I thought I'd normally click a "Run workflow" button to set up an E2E test, but I don't have see one here. How do I get access, or does "E2E" refer to something else? |
…h indices can change per cluster setup or layout. Thanks to Chun for pointing it out.
|
Hi @chunfangamd and all, here are Terminal logs from successful runs of Tests 3 to 5 using this PR. |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 25e466a. Configure here.
…ch GPU family like in generate.py's _hardware_family.
|
I updated my fork to a later SemiAnalysisAI/InferenceX baseline, and the formerly-failing tests still pass. |
3 participants
Fix SGLang MORI disagg tests hanging on MI325X. The test matrix appeared as follows:
Investigation employed helper scripts from https://github.com/davzhuAMD/InferenceX/tree/sglang-pd-mi325x, derived from https://github.com/JohnQinAMD/InferenceX/tree/sglang-pd-mi300-mi325x, to aid investigation and repro. These are the run_* and start_* scripts that essentially act as config instructions to set up the repro.
To validate, copy the run_* and start_* scripts from the sglang-pd-mi325 branch mentioned above into the main InferenceX folder, update the PREFILL_MODEL_HOST_DIR, DECODE_MODEL_HOST_DIR, PREFILL_NODE, and DECODE_NODE IPs per test in these scripts, then run
Thanks to Akash Dhaka who investigated this issue with me.
Note
Medium Risk
Changes benchmark harness defaults for a narrow model+SKU and alters NCCL/MoRI QoS env propagation cluster-wide when QoS is detected; wrong gates could affect non-MI325X runs or mask fabric misconfiguration.
Overview
Fixes SGLang MoRI prefill/decode disaggregated benchmarks that used to hang or stall under load on MI325X, especially DeepSeek-R1-0528 with decode DP+EP and cross-node MoE all-to-all.
env.shintroducesRUNNER_FAMILY(normalized fromRUNNER_TYPE) so SKU-specific tuning can targetmi325xwithout matching fleet labels likecluster:mi325x-amds. When RDMA QoS is detected, traffic class and service level are propagated consistently to MoRI IO and NCCL (NCCL_IB_TC/NCCL_IB_SL), including deriving SL from TC when missing—avoiding bnxt_re DSCP/SL mismatches that downgrade to best-effort and stall KV/NCCL under load. For DeepSeek-R1-0528 on mi325x, decodeMORI_MAX_DISPATCH_TOKENS_DECODEis raised from 512 → 4096 (scoped to that model+SKU).server_sglang.shapplies the same model+SKU gate to setPREFILL_MEM_FRACTION_STATIC=0.9after the models.yaml loader (so it is not clobbered). The DP+EP decode path no longer tiesmax-running-requeststo bench concurrency alone or shrinks MoRI dispatch buffers toconc/dp_ranks; it scalesmax-running-requeststoconc × TP(floored atdp_ranks) and keeps env.sh MoRI dispatch/MOE limits instead of overriding them downward.Reviewed by Cursor Bugbot for commit e9863fc. Bugbot is set up for automated code reviews on this repo. Configure here.
中文说明
修复 SGLang MoRI 分离式基准测试在 MI325X 上的挂起问题。通过调优 RoCE/NCCL/MoRI 环境变量默认值和修复解码端 DP+EP 尺寸设置来解决（之前在负载下会导致跨节点 EP 资源不足）。主要变更包括：
env.sh新增默认NCCL_IB_GID_INDEX、NCCL_IB_TC、NCCL_IB_SL用于 bnxt_re RoCEv2 多节点通信；解码 MoRI dispatch 上限从 512 提升至 4096 tokens；server_sglang.sh修正max-running-requests缩放逻辑；新增启动辅助脚本_disagg_ssh_remote_inner.sh、detect_ibdevices_bnxt.sh、rebuild_bnxt.sh等。