[AMD] Document Mori XGMI for Single-Node PD Disaggregation

[clintg6]

Document Mori XGMI for Single-Node PD Disaggregation

Summary

This PR has been rebased on top of main and minimized to documentation only.

It documents Mori's native environment variable for enabling same-node XGMI fallback:

MORI_DISABLE_AUTO_XGMI=0

No new SGLang-specific environment variable is introduced. In particular, this PR no longer adds or documents SGLANG_MORI_USE_XGMI.

Mori PDD still initializes the Mori IO backend through the existing RDMA backend path. When MORI_DISABLE_AUTO_XGMI=0 is set and no active RDMA device is available, Mori can internally fall back to its XGMI path for same-node GPU-to-GPU KV cache transfer, as supported by ROCm/mori#316.

Motivation

Single-node prefill/decode disaggregation can run with prefill and decode GPUs connected locally through XGMI/Infinity Fabric. In that deployment, requiring users to discover Mori's XGMI fallback environment variable from outside the SGLang docs is confusing.

Since Mori already owns the fallback behavior, this PR documents the Mori-native control directly instead of adding a duplicate SGLang wrapper env var.

Changes

Documentation

• Adds MORI_DISABLE_AUTO_XGMI to docs_new/docs/references/environment_variables.mdx.
• Documents that setting MORI_DISABLE_AUTO_XGMI=0 allows Mori to automatically use XGMI for same-node PDD when no active RDMA device is available.

Removed From Earlier Revision

The earlier revision of this PR added a SGLang-specific env var and implementation hooks. Those changes have been removed.

This PR no longer changes:

• python/sglang/srt/environ.py
• python/sglang/srt/disaggregation/mori/conn.py
• test/manual/test_mori_transfer_engine_e2e.py

Behavior

Env var	Behavior
MORI_DISABLE_AUTO_XGMI unset	Existing Mori behavior.
MORI_DISABLE_AUTO_XGMI=0	Allows Mori to fall back to same-node XGMI when no active RDMA device is available.

SGLang does not create a separate XGMI backend and does not translate a SGLang-specific flag. Users configure Mori directly with the Mori env var.

Usage

Single-node Mori PDD with XGMI fallback enabled:

# Prefill
MORI_DISABLE_AUTO_XGMI=0 python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --disaggregation-mode prefill \
  --disaggregation-transfer-backend mori \
  --port 30000

# Decode
MORI_DISABLE_AUTO_XGMI=0 python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --disaggregation-mode decode \
  --disaggregation-transfer-backend mori \
  --base-gpu-id 1 \
  --port 30001

In the common single-host launch case, no extra Mori node ID setting is needed. Mori uses the host identity in its engine descriptors to decide whether peers are on the same node. If prefill and decode run in separate containers on the same host and those containers report different hostnames, set the same MORI_IO_NODE_ID in both containers so Mori still classifies the engines as same-node.

How It Works

Mori PDD transfers GPU KV cache through MoriKVManager.send_kvcache(), which ultimately calls Mori IO batch_write(). The transfer calls are transport-agnostic; the selected Mori IO backend determines whether the copy uses RDMA or XGMI.

With XGMI fallback enabled:

SGLang server args + env
  -> disaggregation_transfer_backend = mori
  -> MORI_DISABLE_AUTO_XGMI=0
  -> SGLang creates the existing Mori RDMA backend path
  -> Mori internally falls back to XGMI when no active RDMA device is present
  -> register GPU KV buffers
  -> batch_write() transfers KV cache over same-node XGMI/P2P

Auxiliary CPU metadata remains on the existing SGLang ZMQ/TCP path and does not require Mori IO.

[gemini-code-assist]

Code Review

This pull request introduces support for the Mori transfer engine in PD disaggregation, specifically adding a same-node XGMI IO backend. The changes include updated documentation, new server arguments with associated validation, and a manual end-to-end test for XGMI transfers. A critical syntax error was identified in the CLI argument parsing logic where a redundant function call was introduced.

[HaiShaw]

/tag-and-rerun-ci

[Lzy17]

Add @maning00 to the review

[maning00]

A mori-io fix proposed in ROCm/mori#316, which should work using MORI_DISABLE_AUTO_XGMI=0

[clintg6]

Thanks for the feedback. I’ve updated the PR to remove the Mori XGMI server argument and related docs changes. SGLang now exposes this as SGLANG_MORI_USE_XGMI in environ.py. When enabled, SGLang sets MORI_DISABLE_AUTO_XGMI=0 and lets Mori handle backend selection/fallback. I tested and confirmed that these changes work with the mori-io fix proposed in ROCm/mori#316. @maning00 @ShangmingCai @Lzy17

[ShangmingCai]

Others LGTM

[clintg6]

The PR is still blocked on CODEOWNERS approval. Please take a look when you get a chance @wisclmy0611 @zijiexia

[zijiexia]

Hi @clintg6 , we've migrated our documentations to docs_new/ and will deprecated docs soon, can you please make the changes to corresponding position under docs_new? Thanks!

[clintg6]

@zijiexia Thanks for the heads up. The current docs/ diff was unrelated to this PR and came from syncing with upstream main. I’ve cleaned that out after rebasing. If you’d like, I can document SGLANG_MORI_USE_XGMI under docs_new/.

[zijiexia]

@zijiexia Thanks for the heads up. The current docs/ diff was unrelated to this PR and came from syncing with upstream main. I’ve cleaned that out after rebasing. If you’d like, I can document SGLANG_MORI_USE_XGMI under docs_new/.

Thanks for cleaning that up! Yes, please go ahead and document SGLANG_MORI_USE_XGMI, that would be very helpful!

[clintg6]

@zijiexia docs_new has been updated to document SGLANG_MORI_USE_XGMI, including its behavior and default value. Could you please take another look when you have a chance?

[kkHuang-amd]

@amd-bot ci-status

[amd-bot]

@kkHuang-amd

CI Status for PR #25094

Merge verdict: No failure is caused by this PR — every red job is either an NVIDIA infra GPU-dirty cascade or a pre-existing NPU diffusion-latency miss. But do not read green as "verified": the actual new feature (the Mori XGMI branch) is never exercised by CI — it sits behind a default-off env var and its only test lives in test/manual/. AMD jobs are also still running, so the run is not yet fully green.

Caution

The PR's core change — the MORI_DISABLE_AUTO_XGMI / XGMI branch in conn.py, gated on SGLANG_MORI_USE_XGMI (default False) — is not exercised by any PR-CI test. The registered suite stage-b-test-large-8-gpu-mi35x-disaggregation-amd ran and passed, so it confirms the default RDMA path still works, but no test sets SGLANG_MORI_USE_XGMI=1, and the new test/manual/test_mori_transfer_engine_e2e.py is manual-only (SGLANG_MORI_MANUAL_E2E=1). Before merge, run the XGMI path on a single-node PDD setup (--disaggregation-mori-io-backend xgmi / SGLANG_MORI_USE_XGMI=1) to actually verify it.

Changed files: disaggregation/mori/conn.py (+5/-1), environ.py (+2/-1), docs_new/.../environment_variables.mdx (+5/-0), test/manual/test_mori_transfer_engine_e2e.py (+71/-0, manual-only, not in CI)

AMD: 0 completed failures (still running — 13 in-progress, 2 queued) · Others: 23 failures (0 related)

AMD CI

No AMD failures. The most relevant suite — stage-b-test-large-8-gpu-mi35x-disaggregation-amd (which runs the registered test_mori_transfer_engine_e2e.py) — completed ✅ success, confirming the default RDMA path is unbroken. Remaining stage-b-test-1-gpu-*-amd / 2-gpu-amd shards are still in progress; not yet green, do not conclude AMD passed until they finish.

Other CI Failures

Job	Test File	Test Function	Error	Related?	Why
base-b-test-4-gpu-b200 (0) (root; ~16 base-b shards + wait-for-base-b cascaded)	N/A	N/A	Pre-test health check: ERROR: memory >=10%: GPU 4 (83%), GPU 5 (83%)... → exit 1	🟢	Dirty/occupied runner GPU memory before the test even starts — infra, not code. Touches no Mori/env path.
multimodal-gen-test-1-npu-a3	test/server/ascend/test_server_1_npu.py	test_diffusion_generation[flux_image_t2i_npu]	assert 46159 <= 25838 (E2E Latency), failed all 7 retries	🟢	Flux diffusion NPU perf threshold; ~2× over limit consistently. No relation to Mori KV transfer or the env var.
pr-test-finish · pr-test-extra-finish · pr-test-npu-finish	N/A	N/A	The above jobs failed (aggregators)	🟢	Roll-up gates that fail because the cascades/NPU above failed.

Details / what to do before merge

• Coverage (the real action item): the XGMI feature is unverified by CI. Run it manually on a single-node Mori PDD setup with --disaggregation-transfer-backend mori --disaggregation-mori-io-backend xgmi (or SGLANG_MORI_USE_XGMI=1 via the manual test) and paste the result. The environ.py change (new EnvBool + an f-string lint fix) is trivially safe.
• NVIDIA base-b cascade: not actionable for the author — re-run base-b once a clean b200 runner is available; the root base-b-test-4-gpu-b200 (0) failed its GPU-occupancy pre-check, not the tests.
• NPU diffusion latency: pre-existing perf miss on the Ascend Flux test, unrelated to this PR.
• Wait for AMD stage-b 1-/2-GPU shards to finish before merging; only the disaggregation suite has reported so far.

---

Generated by amd-bot using Claude Code CLI

[kkHuang-amd]

LGTM, but I have one question. What mori version is depend by this environ variable "MORI_DISABLE_AUTO_XGMI"?

Has the current sglang image included this mori version?

Or we need to upgrade mori version?

[kkHuang-amd]

Could you modify your command in the Usage section?

It does not set "SGLANG_MORI_USE_XGMI" in launch command.

[clintg6]

@kkHuang-amd Good catch the Usage section was carried over from the initial version of this PR, before XGMI was refactored to be env-var-driven. XGMI is now controlled only by SGLANG_MORI_USE_XGMI (which sets MORI_DISABLE_AUTO_XGMI=0), not the old --disaggregation-mori-io-backend xgmi flag. I've updated both launch commands in the Usage section to set SGLANG_MORI_USE_XGMI=true and removed the stale flag.

The MoRI build dependency is already handled #26922 (merged into main) bumped MORI_COMMIT to d87651c in docker/rocm.Dockerfile, which includes the mori-io fix from ROCm/mori#316 this feature needs. The PR branch already pins this commit, so no further upgrade is needed.

[Duyi-Wang]

Since MORI already provides equivalent environment variables for this behavior, I don't think we need to introduce duplicated SGLang-level env vars for the same control. That would add another layer of configuration without much benefit.

I think the documentation update is useful. We can document the existing MORI environment variables and explain how users should set them when running SGLang.

[clintg6]

@kkHuang-amd @Duyi-Wang I rebased this PR on top of main and minimized the scope to only document the Mori-native env var for enabling same-node XGMI fallback. The PR now documents setting MORI_DISABLE_AUTO_XGMI=0 to allow Mori to use XGMI when no active RDMA device is available. I removed the duplicate SGLang-specific env var path so the XGMI behavior stays controlled by Mori directly.