[jhcdlg]

Summary

MultimemAllGatherer decides whether to use the multimem path from node topology only:

if (
    tp_group.world_size > 1
    and get_parallel().nnodes > 1
    and not all(in_the_same_node_as(tp_group.cpu_group, source_rank=0))
):
    self._state = None

A host can be single-node and still have no multicast fabric: two consumer GPUs on PCIe
with no NVLink and no P2P are co-located, so nnodes == 1 holds and the multimem path stays
enabled. On our box that ends in SIGFPE (exit code -8) inside
symm_mem.rendezvous() - called from create_state(), i.e. before the existing
multicast_ptr == 0 fallback in _build() ever runs. The server starts normally and the
first real inference request kills the scheduler.

This PR keeps the existing condition untouched and adds the capability condition it is
missing.

What changed

elif tp_group.world_size > 1 and not _local_gpus_have_peer_access():
    logger.warning(
        "multimem all-gather disabled because the local GPUs have no "
        "peer-to-peer access, so no multicast fabric is available "
        "between the TP ranks."
    )
    self._state = None

_local_gpus_have_peer_access() asks the driver whether the local GPUs can reach each other
(torch.cuda.can_device_access_peer, all ordered pairs). Notes on the design:

• It is used as a necessary condition only: it can turn multimem off, never on. A
  negative answer falls back to NCCL, which is always correct; a positive answer leaves the
  existing decision unchanged.
• It deliberately does not use gpu_p2p_access_check() / can_p2p(): those spawn helper
  processes, which is too expensive to run whenever a gatherer is constructed, and they are
  built around a rank->device mapping that a single rank process cannot read directly.
• If the capability cannot be established at all, it returns True - i.e. the previous
  behaviour is preserved rather than silently changing which path is taken.

How it was verified

CPU only - no GPU, no NCCL, no rendezvous involved. __init__ reads the parallel state
(get_tp_group(), get_parallel().nnodes, in_the_same_node_as()) and the driver's topology
(torch.cuda.device_count, torch.cuda.can_device_access_peer), so all five are faked and
_state is asserted directly (None = NCCL, _UNINIT = multimem stays enabled).

Five cases, each run against (a) unpatched code and (b) this patch:

Case	unpatched	patched
enabled=False -> NCCL	PASS	PASS
world_size == 1 -> stays lazy	PASS	PASS
cross-node -> disabled	PASS	PASS
single node with peer access -> stays enabled	PASS	PASS
single node without peer access -> disabled	FAIL	PASS

The single failing case on unpatched code is the bug this PR fixes; it doubles as proof that
the test is not vacuous. The test file is included in this PR as
test/registered/unit/distributed/test_triton_symm_mem_ag.py (it mirrors
python/sglang/srt/distributed/device_communicators/triton_symm_mem_ag.py), registers itself
for CPU CI via register_cpu_ci(est_time=5, suite="base-a-test-cpu") and subclasses
CustomTestCase, following test/registered/unit/README.md.

The A/B output below was produced by running this exact test file - first against the
unpatched source file, then against the patched one. Baselines, so the reader can check them:
unpatched 552 lines / md5 c89b4103acabdc9c88119cecbf575af0; patched 588 lines /
md5 4fe0be55fd15291646e1e8040969391f, which is the unpatched file plus the patch in this PR
(+36 / -0 lines). The revision these two files were taken from is upstream main @ 80da4432.
No GPU was involved: the driver's capability query is mocked, which is
precisely the point of the test.

Scope / limits we want to be upfront about

• Reproduced on one host: RTX 3080 (sm_86) + RTX 4080 SUPER (sm_89), PCIe x8 each, no
  NVLink, no P2P, WSL2 - same machine, same driver, SGLang revision 4cb5aeb runs fine, which
  is how we know it is not a hardware fault.
• We have no NVLink host, so we could not verify that machines which legitimately use
  multimem are unaffected. By construction they should be: the new condition only ever adds a
  reason to fall back. Worst case on an exotic host is a wrong "no P2P" verdict and a fallback
  to NCCL - slower, still correct. If you would prefer a different capability signal, we are
  happy to switch.
• torch.cuda.can_device_access_peer is the driver's own report and can be wrong in the
  optimistic direction (that is why gpu_p2p_access_check exists). This patch only trusts a
  negative answer, so an optimistic false report just leaves today's behaviour.
• This does not touch the multicast_ptr == 0 fallback in _build(). That check is on the
  wrong side of rendezvous to help here; fixing it is a larger, separate change.

Related

• PR [Fix] Skip cross-node probe in MultimemAllGatherer on single-node runs (fixes mooncake EP segfault) #30139 intentionally keeps multimem enabled on a single node (to avoid an
  in_the_same_node_as() segfault under EP/mooncake setups). Our case is compatible with that
  intent - the point is narrower: "a single node" does not imply "multicast-capable", so
  the guard lacks a capability probe. This PR adds one without undoing [Fix] Skip cross-node probe in MultimemAllGatherer on single-node runs (fixes mooncake EP segfault) #30139.
• PR [sglang-miles] Add SGLANG_DISABLE_MULTIMEM_AG to force the NCCL all-gather path #36110 (SGLANG_DISABLE_MULTIMEM_AG, merged) added a manual opt-out that forces the
  NCCL all-gather path, i.e. upstream has already accepted that some hosts need a way out of
  multimem. This PR makes that decision automatic for the case the capability probe can
  detect, instead of requiring every affected user to find and set an environment variable.
  (While reading today's main I could not find SGLANG_DISABLE_MULTIMEM_AG in environ.py
  or triton_symm_mem_ag.py any more - was it moved, or was that change based on a different
  branch? Not trying to make a point, just checking whether the env var is still the intended
  escape hatch before we rely on it.)
• The faulthandler stack and the exact reproduction steps are available - we can paste them in
  this thread, or open a separate issue if you would rather keep reports out of the PR.

A/B verification output (CPU only: unpatched 4/5 -> patched 5/5)

00:48:59 ===
python  : 3.11.16
sglang  : <sglang>/sglang

==================================================================
A) UNPATCHED upstream code  -- expect exactly 1 FAIL (the gap)
==================================================================
module  : sglang.srt.distributed.device_communicators.triton_symm_mem_ag (loaded from <abdir>/base.py)

[CI Test Method] TestMultimemAllGathererGuard.test_cross_node_is_disabled
multimem all-gather disabled because the TP group spans across nodes.
PASS  test_cross_node_is_disabled
[CI Test Method] TestMultimemAllGathererGuard.test_disabled_flag_always_uses_nccl
PASS  test_disabled_flag_always_uses_nccl
[CI Test Method] TestMultimemAllGathererGuard.test_single_node_with_peer_access_stays_enabled
PASS  test_single_node_with_peer_access_stays_enabled
[CI Test Method] TestMultimemAllGathererGuard.test_single_node_without_peer_access_is_disabled
Traceback (most recent call last):
  File "<sglang>/sglang/srt/utils/common.py", line 3414, in retry
    return fn()
           ^^^^
  File "<sglang>/sglang/test/test_utils.py", line 2292, in <lambda>
    lambda: super(CustomTestCase, self)._callTestMethod(method),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<venv>/lib/python3.11/unittest/case.py", line 579, in _callTestMethod
    if method() is not None:
       ^^^^^^^^
  File "<abdir>/test_triton_symm_mem_ag.py", line 103, in test_single_node_without_peer_access_is_disabled
    self.assertIs(gatherer._state, None)
  File "<venv>/lib/python3.11/unittest/case.py", line 1154, in assertIs
    self.fail(self._formatMessage(msg, standardMsg))
  File "<venv>/lib/python3.11/unittest/case.py", line 703, in fail
    raise self.failureException(msg)
AssertionError: <object object at 0x75f5ab60f880> is not None
FAIL  test_single_node_without_peer_access_is_disabled
            retry(
          File "<sglang>/sglang/srt/utils/common.py", line 3429, in retry
            raise Exception(f"retry() exceed maximum number of retries.")
        Exception: retry() exceed maximum number of retries.
[CI Test Method] TestMultimemAllGathererGuard.test_single_rank_is_left_alone
PASS  test_single_rank_is_left_alone

4 passed, 1 failed, 5 total

==================================================================
B) PATCHED module  -- expect 5/5 PASS
==================================================================
module  : sglang.srt.distributed.device_communicators.triton_symm_mem_ag (loaded from <abdir>/fixed.py)

[CI Test Method] TestMultimemAllGathererGuard.test_cross_node_is_disabled
multimem all-gather disabled because the TP group spans across nodes.
PASS  test_cross_node_is_disabled
[CI Test Method] TestMultimemAllGathererGuard.test_disabled_flag_always_uses_nccl
PASS  test_disabled_flag_always_uses_nccl
[CI Test Method] TestMultimemAllGathererGuard.test_single_node_with_peer_access_stays_enabled
PASS  test_single_node_with_peer_access_stays_enabled
[CI Test Method] TestMultimemAllGathererGuard.test_single_node_without_peer_access_is_disabled
multimem all-gather disabled because the local GPUs have no peer-to-peer access, so no multicast fabric is available between the TP ranks.
PASS  test_single_node_without_peer_access_is_disabled
[CI Test Method] TestMultimemAllGathererGuard.test_single_rank_is_left_alone
PASS  test_single_rank_is_left_alone

5 passed, 0 failed, 5 total

SUMMARY: unpatched 4/5  |  patched 5/5

---

CI States

Latest PR Test (Base): ❌ Run #35527701759
Latest PR Test (Extra): ❌ Run #35527701553
Latest PR Test (AMD ROCm 10): ❌ Run #35527701721