[SM120] Add optional FlashInfer PCIe-IPC all-reduce for switch-free hosts - #34528
AliceChenyy wants to merge 19 commits into
Conversation
On switch-free intra-node hosts -- no NVLink, no multicast, every peer transfer crossing the CPU root complex -- the existing custom all-reduce backends do not apply, so every per-layer reduction falls back to NCCL. FlashInfer's pcie_ipc kernels target exactly this fabric. Add an opt-in communicator behind SGLANG_ENABLE_PCIE_IPC_ALLREDUCE. It is a thin adapter: shape coverage is delegated to FlashInfer's own tuning table via workspace.supports(), so an unsupported shape is reported as such and the caller keeps its NCCL path. No size heuristic lives here. The workspace cannot grow after construction, so it is sized for the largest reduction the model issues -- one prefill chunk. The hidden size is unknown when the group is built, so construction is deferred to the first eligible tensor; ranks issue the same reduction sequence and so agree on the shape without extra exchange. Measured on one 8x SM120 host (GPUs 0-3 and 4-7 on separate NUMA nodes), GLM-5.2-NVFP4, TP8, 8k context, against NCCL: TPOT 40.16 -> 13.70 ms, output throughput 27.31 -> 77.27 tok/s. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The PCIe-IPC path was resolved ahead of ca, so on a host that does have NVLink -- where CustomAllreduce applies and is the faster of the two -- enabling the flag would have handed those reductions to the slower backend. FlashInfer's topology probe only looks for switch-local peers; it does not detect NVLink and so cannot decline on its own. Order is the fix: ca answers first on the fabrics it supports, and the PCIe-IPC kernels pick up only what it leaves, which is exactly the switch-free case they were written for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The workspace was sized for one prefill chunk so that prefill reductions would also run on these kernels. Measured on 8x SM120, TP8, 8k context, that is the wrong trade: against NCCL as the baseline, TTFT 1910 ms and TPOT 21.14 ms, a prefill-sized workspace gives 3176 ms / 13.64 ms while a decode-sized one gives 1849 ms / 13.62 ms. Handing prefill to the kernels costs 66% on TTFT and buys nothing on TPOT -- these kernels win by latency at small messages, and a prefill chunk is three orders of magnitude larger than a decode reduction, which is NCCL ring territory. The decode-sized workspace is also ~250x smaller, which matters at long context: at 128k with 4 concurrent requests the prefill-sized workspace regressed TPOT by 45% (122.33 ms against NCCL's 84.50 ms), and that regression disappears once it is sized for decode (84.60 ms). Derive the bound from the decode phase's captured batch. Verified end to end: the default builds max_numel = 393216 on this config and reproduces an explicit cap within 0.3% on every cell. GSM8K over the four runs spans 0.945-0.960, within binomial noise at 200 questions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every way this adapter declines a reduction is silent by design: an unsupported world size, a missing flashinfer build, a workspace that cannot be allocated, and a shape outside the tuning table all end in the caller keeping its NCCL path. That is the right behaviour and the reason it needs tests -- a regression there does not raise, it just quietly stops using the kernels, which reads as "the kernels are slow". CPU-only: the constructor is bypassed and the workspace stubbed, since none of the logic under test needs a process group or a device. Also pins the decode-sized default, so re-sizing the workspace for a prefill chunk cannot come back unnoticed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
They had landed after the Mori RDMA transfer timeout, which is an unrelated block; the repo's env-var conventions ask for entries to sit under the section they belong to. These two belong with CustomAllReduceV2 and the deterministic all-reduce knobs. Also drops "so there is no size knob here" from the enable flag's comment, which read oddly directly above a size knob. The point it was making -- that shape coverage is FlashInfer's decision and not something this env var controls -- is now said plainly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drops an unused import and reflows one assertion. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
TP2 integration result for DeepSeek-V4-Flash-0731 on 2× RTX PRO 6000 Blackwell Max-Q (SM120), connected over PCIe Gen4 x16. The measured consumer head was With DSPARK block size 5, target verification uses six token rows per running request. The derived workspace was:
The C32 target-verification bound is:
The 131,072-element workspace admitted at most Setting
The FlashInfer backend was #4393 at A later integration using current SGLang head |
…eed policy The workspace was built without ever calling tune(), so supports() answered from FlashInfer's seed policy and the kernels ran untuned tactics. FlashInfer needs a host-side group to rendezvous the autotuner on every rank, so pass the coordinator's cpu_group down and tune once at workspace build. Results persist to FlashInfer's cache, so only the first server on a host pays for it. Autotuning replays kernels and cannot run under graph capture; if the first eligible reduction arrives there, keep the seed policy and say so.
The workspace was built without a tune_cache, so the measured tactics lived only in that process: every server start re-measured them, contradicting the claim that only the first one pays. Name FlashInfer's default path explicitly so the file is written and the next start reuses it.
Building the workspace on the first eligible reduction puts the tuning call inside SGLang's own FlashInfer autotune pass, and FlashInfer declines to profile a collective from an autotune context it did not open -- so tune() returned having measured nothing, and the kernels kept running the seed policy. Verified: zero 'Tuning flashinfer::pcie_ipc_all_reduce' lines and no cache written. Add prepare(hidden) and call it from the runner's warmup just before the autotune context opens, which is where vLLM's port does the same thing. _tune() now refuses and says so when it finds itself nested, rather than reporting a tuning that did not happen.
tune() declines shapes silently, so a log line that only proves no exception was raised is how an autotune that measured nothing reads as a success -- which is exactly how the previous two revisions here were misread.
One conflict, in environ.py's all-reduce block: upstream deleted SGLANG_ENABLE_CUSTOM_ALL_REDUCE_V2_MULTINODE (no references remain anywhere in the tree) and moved the flashinfer/triton split-tile vars elsewhere in the file. Kept only this PR's two vars; everything else follows upstream. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…o feat/flashinfer-pcie-ipc-ar
| from sglang.test.ci.ci_register import register_cpu_ci | ||
| from sglang.test.test_utils import CustomTestCase | ||
|
|
||
| register_cpu_ci(est_time=5, suite="base-c-test-cpu") |
There was a problem hiding this comment.
base-c-test-cpu is no longer a registered suite. Could we move this unit test to base-a-test-cpu?
There was a problem hiding this comment.
Moved to base-a-test-cpu. Confirmed with collect_tests that the file is now picked up there, and that nothing in the tree still references base-c-test-cpu.
| if ( | ||
| envs.SGLANG_ENABLE_PCIE_IPC_ALLREDUCE.get() | ||
| and self.world_size > 1 | ||
| and "tp" in self.unique_name |
There was a problem hiding this comment.
This also matches attention_tp and moe_tp. Some of those groups have no pynccl or custom all-reduce communicator, so they hit the assertion at line 1032 before reaching the PCIe IPC code
This causes using the SGLANG_ENABLE_PCIE_IPC_ALLREDUCE with a model like Qwen3-8B TP4/DP2 with DP attention and Qwen3-30B-A3B TP4/EP2 to fail. I am not too certain on the scope of this PR, but I think this is definitely worht bringing up/addressing. If they’re outside its scope, please keep this backend disabled for them.
There was a problem hiding this comment.
Good catch, and thank you -- reproduced both failures on 8x SM120. eligible_group() now matches the group name exactly, so attention_tp / moe_tp / pdmux_prefill_tp keep NCCL; Qwen3-32B TP4/DP2 and Qwen3-30B-A3B TP4/EP2 both start with the flag set, where before they hit the assertion during graph capture.
| try: | ||
| from sglang.srt.server_args import get_global_server_args | ||
|
|
||
| config = get_global_server_args().cuda_graph_config |
There was a problem hiding this comment.
Could we read the resolved configuration here, or pass the workspace size from the runner into prepare()?
After upstream’s configuration changes, this field stays None when I pass --cuda-graph-max-bs 128. The resolved value in get_exec().graph.cuda_graph_config.decode.max_bs is 128, but this function returns None, so the workspace falls back to 64 rows.
A test using prepare_server_args() would catch this.
There was a problem hiding this comment.
It was worse than a None: ServerArgs has no cuda_graph_config at all, so the read raised AttributeError on every call and the bare except made the 64-row fallback permanent. It now reads get_exec().graph.cuda_graph_config and warns when it falls back; measured 16 and 32 rows for --cuda-graph-max-bs 16 and 32, against a constant 64 before.
|
|
||
| Decode runs inside a captured graph, so the largest captured batch bounds the | ||
| reduction. Speculative decoding verifies several tokens per sequence in one | ||
| forward, and the decode phase's ``max_bs`` is the batch the runner captures |
There was a problem hiding this comment.
max_bs counts requests, so it doesn’t include speculative token width. With four draft tokens, 64 requests need 256 rows for verification. Please account for that when sizing the workspace and update this comment.
There was a problem hiding this comment.
You are right, and the docstring was wrong to claim otherwise -- both are fixed. The width is now max_bs * draft_tokens, verified on hardware: 32 requests with 4 draft tokens size the workspace for 128 rows.
| comm._device = torch.device("cpu") | ||
| comm._world_size = world_size | ||
| comm._workspace_cls = workspace_cls or MagicMock() | ||
| comm._build_failed = False |
There was a problem hiding this comment.
Could we update this helper and mock _tune() for these sizing and shape tests? The later tuning change added _cpu_group, but this helper never sets it. _ensure_workspace() now calls _tune(), which reads that field, so six of the nine tests fail.
Adding _cpu_group = None alone still fails when FlashInfer is unavailable, because _tune() imports it before checking the field. We could mock tuning here and test its behavior separately.
There was a problem hiding this comment.
Done -- the helper sets _cpu_group and stubs tuning, and tuning is covered separately in TestTuning. As you noted, _cpu_group = None alone was not enough, since _tune imports FlashInfer before it reads the field.
| def should_pcie_ipc_ar(self, inp: torch.Tensor) -> bool: | ||
| """Whether FlashInfer has a tuned configuration for this exact shape. | ||
|
|
||
| ``supports`` consults the tuning table, so a shape the kernels lose on is |
There was a problem hiding this comment.
Could we update this description of supports()? The earlier version used a performance-based table to reject some shapes, but that changed upstream it seems. supports() now checks whether the kernel can run the tensor, so it can accept shapes even when NCCL would be faster.
There was a problem hiding this comment.
Rewritten, here and in the module docstring -- supports() reports whether the kernels can run a shape, not whether they beat NCCL, so the size bound is what keeps prefill-sized reductions on NCCL.
| # After ``ca``: the PCIe-IPC kernels are for hosts where no fabric-specific | ||
| # backend applies. They do not probe for NVLink, so on a host that has it | ||
| # this ordering is what keeps the faster backend in front of them. | ||
| if ( |
There was a problem hiding this comment.
In eager execution, the symmetric-memory branch at lines 757–767 returns through NCCL before reaching this code. Compiled execution can reach this code and select IPC instead. Could we clarify which backend should take priority when both this feature and --enable-symm-mem are enabled? If IPC should take priority in both cases, the earlier branch needs updating.
There was a problem hiding this comment.
Made explicit: the two flags are independent opt-ins, and with both set the backend now keeps symmetric memory and logs once. Deciding it in eligible_group() rather than in the dispatch means eager and compiled behave the same, so the earlier branch needs no change.
| """ | ||
| from flashinfer.autotuner import AutoTuner | ||
|
|
||
| if AutoTuner.get().is_tuning_mode: |
There was a problem hiding this comment.
The comment says FlashInfer won’t allow that call, is this correct?
Is there another reason we need this check? If so, could we explain that reason here? Otherwise, we could remove the check and update the comments.
There was a problem hiding this comment.
You were right to question it -- the comment was wrong. autotune() is reference-counted and workspace.tune() installs its own process group, so FlashInfer does permit the call; the real reason to decline is that tune() mutates shared autotuner state for the length of the call and would time the enclosing pass with our settings. Comment now says that.
| self._bound_stream = stream | ||
| return self._workspace.all_reduce(inp) | ||
|
|
||
| def destroy(self) -> None: |
There was a problem hiding this comment.
The workspace needs to be released before the process groups are destroyed, because FlashInfer uses those groups during cleanup.
Could we call this method from GroupCoordinator.destroy()?
There was a problem hiding this comment.
Now called from GroupCoordinator.destroy(), ahead of destroy_process_group(). One caveat worth flagging: the server shutdown path does not currently call destroy_model_parallel() at all, so today this is reached from benchmark/one_batch.py and tests -- verified clean there.
| return | ||
| try: | ||
| tuned = self._workspace.tune( | ||
| [hidden], dtype=torch.bfloat16, tune_group=self._cpu_group |
There was a problem hiding this comment.
supports() accepts fp16 tensors, but this call only tunes bf16. FlashInfer stores separate tuning results for each dtype, so an fp16 model won’t use the results from this call.
We should either tune fp16 when the model uses it, or explicitly limit this adapter to bf16.
There was a problem hiding this comment.
Limited to bf16, with a one-off warning for anything else. Tuning the model dtype would need measurements we do not have, so declining seemed the honest option until then.
base-c-test-cpu is no longer a registered suite, so this file was collected by nothing. The mock helper also never set _cpu_group, which _ensure_workspace now reads through _tune, so six of the nine cases errored once that call was added. Register on base-a-test-cpu, give the helper the field, and stub tuning there so the sizing and shape cases exercise what they claim to. Tuning gets its own class: every way it declines returns normally, which is indistinguishable from a successful tune unless asserted.
…eardown "tp" in unique_name also matches attention_tp, moe_tp and pdmux_prefill_tp. Those groups carry no pynccl or custom all-reduce communicator, so a reduction dispatched to them trips the assertion in _all_reduce_out_place instead of falling back: with the flag set, Qwen3-32B TP4/DP2 and Qwen3-30B-A3B TP4/EP2 fail during CUDA graph capture. Reproduced on 8x SM120, fixed by matching the group name exactly; the rule now lives next to the backend with tests. Separately, GroupCoordinator.destroy() never released the workspace. Release it there, ahead of destroy_process_group(), because FlashInfer collectives on that group while tearing the workspace down.
ServerArgs has no cuda_graph_config: the resolved config moved onto the exec bag, so the old read raised AttributeError on every call and a bare except turned that into a permanent, silent fallback to a fixed 64 rows. Whatever --cuda-graph-max-bs an operator passed, the workspace ignored it. Read the exec bag, and multiply by the speculative draft width -- max_bs counts requests, not rows, so a verify pass needs max_bs * draft_tokens of them. The fallback now says so instead of staying quiet. Measured on 8x SM120: max-bs 16 and 32 give 16 and 32 rows where both previously gave 64, and 32 requests with 4 draft tokens give 128. Also decline anything but bf16. FlashInfer keys tuning results by dtype and only bf16 is tuned here, so an fp16 model would have run the seed policy while looking tuned.
…ld to symm-mem prepare() sat inside _flashinfer_autotune, which only runs for models that autotune; every other model built the workspace lazily on its first reduction, inside another autotune context, where tuning declines. Move it to the general warmup path next to the other pre-initialised workspaces. That lazy path also ignored --disable-flashinfer-autotune: on 8x SM120 the adapter tuned seven shapes and wrote a cache with the flag set. Honour it. Finally, --enable-symm-mem and SGLANG_ENABLE_PCIE_IPC_ALLREDUCE are independent opt-ins, and with both set the result depended on the execution mode -- eager returned through symmetric memory's NCCL branch, compiled selected IPC. Keep symmetric memory, the pre-existing feature, and say so once.
|
Thanks for the thorough review -- all eleven points are addressed in four commits on top of Two of your comments turned out to hide larger problems, so flagging them explicitly:
I also re-measured on 8x SM120 (GPUs 0-3 and 4-7 on separate NUMA nodes), three reps per cell with the first discarded, rep spread reported:
GSM8K (200 examples, run at both ends of each arm): OFF 0.990 / 0.985, ON 0.990 / 0.990. TTFT is unchanged within noise, which is expected -- the workspace is decode-sized by design, so prefill stays on NCCL. One caveat that seems worth documenting rather than hiding: on TP4 within a single NUMA node the same measurement shows -8.5% at concurrency 1 and nothing measurable at 8 or 32. The all-reduce never crosses the socket link there, so there is little for these kernels to remove. Anyone evaluating this on a 4-GPU host should expect roughly nothing, and that is the topology, not the backend. |
3 participants
Motivation
On switch-free intra-node hosts — no NVLink, no multicast, every peer transfer crossing the CPU root complex — none of SGLang's custom all-reduce backends apply.
CustomAllreduceandQuickAllReduceassume NVLink/multicast,pymscclppassumes its own fabric, so on these machines every per-layer reduction falls back to NCCL. NCCL's ring is bandwidth-optimal but latency-poor at decode message sizes, and it shows up directly in TPOT.FlashInfer's
pcie_ipckernels (flashinfer-ai/flashinfer#4393) target exactly this fabric: naive all-to-all peer writes collapse here, so the kernels stage their pushes to keep one outbound and one inbound stream per rank, and the 8-rank path uses a 4+4 island decomposition so the scarce cross-socket link carries the minimum.This affects the whole RTX PRO 6000 / RTX 6000D class, where SGLang is otherwise well supported.
Scope. This is a decode-latency path, not a general NCCL replacement. It is sized so
prefill chunks stay on NCCL (see Workspace sizing), because these kernels win by latency
at small messages and lose to a ring at 200 MB. On a host that also wants prefill improved,
this composes with a bandwidth-optimal backend rather than replacing it.
Modifications
Opt-in behind
SGLANG_ENABLE_PCIE_IPC_ALLREDUCE(default off). +527/−0 across five files:distributed/device_communicators/pcie_ipc_ar.py(new) — adapter betweenGroupCoordinatorandPcieIpcAllReduceWorkspace.distributed/parallel_state.py(+29) — construction, one"pcie_ipc"branch in_resolve_outplace_all_reduce_method, one dispatch arm. Built only on the TP group; other groups would just pin IPC buffers without issuing the reductions these kernels target.environ.py(+11) — the gate plusSGLANG_PCIE_IPC_MAX_NUMEL.model_executor/runner/base_runner.py(+9) — oneprepare()call from warmup, before the FlashInfer autotune context opens. See Autotuning for why the timing is load-bearing.Two design points worth review:
No size heuristic lives in SGLang.
should_pcie_ipc_ar()delegates toworkspace.supports(), which consults FlashInfer's own tuning table. A shape the kernels do not beat is reported unsupported and the caller keeps its NCCL path. This avoids a threshold knob that would need per-machine tuning and would drift from the kernel side. SGLang's job is to make sure that table is populated for this host and these shapes rather than left on the seed policy -- see Autotuning.The workspace is built on the first eligible tensor, not at group construction. It cannot grow afterwards and costs ~
2 * world_size * max_numel * itemsizeper rank, so it must be sized for the largest reduction the model issues — one prefill chunk,chunked_prefill_size * hidden.chunked_prefill_sizeis in the server args buthiddenis not known when the group is built, so construction is deferred to the first reduction, whose trailing dimension is exactly that. Ranks run the same reduction sequence, so they reach that call with the same shape and build the same workspace without extra exchange.Sizing it for a prefill chunk instead is the wrong trade, which is worth
spelling out because an earlier revision of this PR did exactly that. These
kernels win by latency at small messages; a prefill chunk is three orders of
magnitude larger than a decode reduction, which is NCCL ring territory. Measured
at 8 ranks, TP8, 8k context:
chunked_prefill_size * hidden)Routing prefill through the kernels costs 66% on TTFT and buys nothing on TPOT.
The decode-sized workspace is also ~250x smaller, which matters at long context:
at 128k with 4 concurrent requests the prefill-sized workspace regressed TPOT by
45% (122.33 ms against NCCL's 84.50 ms), and that regression disappears once it
is sized for decode (84.60 ms).
Autotuning
supports()answers from FlashInfer's tuning table, so what is in that table decides both whichshapes reach the kernels and which launch config they get. Left alone the table holds FlashInfer's
seed policy — a default, not a measurement of this host — so this wires the autotuner in: the
workspace is constructed with
tune_batchesbounded by what it will actually accept and with anexplicit
tune_cachepath, the coordinator'scpu_groupis handed down as the autotuner'srendezvous group, and
tune()runs once when the workspace is built.The timing is the load-bearing part. The workspace is built on the first eligible reduction,
and that reduction arrives inside SGLang's own FlashInfer autotune pass. FlashInfer declines to
profile a collective from an autotune context it did not open, so
tune()returned having measurednothing while still returning cleanly — zero
Tuning flashinfer::pcie_ipc_all_reducelines and nocache written.
prepare(hidden)is therefore called from the runner's warmup, just before thatcontext opens, which is where vLLM's port of these kernels does the same thing.
_tune()refusesand says so if it still finds itself nested, and it logs how many shapes
tune()actually coveredrather than only that it returned: an autotune that measures nothing otherwise reads as a success,
which is exactly how two revisions of this change were misread.
Measured on 4x RTX PRO 6000 Blackwell SE (SM120, PCIe, no NVLink), TP4, DeepSeek-V4-Flash,
decode-only TPOT via a warm-prefix method, 2 repeats per cell. Baseline is this PR's own tree with
the backend off, so both columns share one NCCL path:
As first submitted this backend regressed decode at concurrency >= 8 (+2% to +4%); with the
table populated it is -9% to -14% across every cell. GSM8K 100q: 0.970.
Where the win actually comes from, since it is not where I first assumed. Running the same
build with the tuning genuinely executed and with it silently skipped lands within 1% on all eight
cells. The measured tactics are worth almost nothing here; the gain is the bucket alignment that
tune_batchesbrings, which changes which entry each shape resolves to. FlashInfer's seed tacticsare close to optimal on this fabric — what was wrong was asking the table about buckets the
workspace would never serve. The tuning call is still worth keeping: it is what makes the table
this host's own rather than a default, and on a fabric whose seed policy fits less well it is the
part that adapts.
Two consequences worth knowing before deploying:
pcieswitch-pairson this host,rootcplx-noswitchonanother 8x SM120 box). Tuning tables do not transfer between machine types; a new host pays
the measurement once, about 2 s for six shapes, and reuses it after that.
arrives there, the seed policy stays in force and a warning says so.
Accuracy
GSM8K, 200 questions, same server and args as the perf runs:
The 0.015 spread is 3 questions out of 200, within binomial noise at this sample
size (SE ~= 0.015). At the kernel level, 10 shapes from 1 to 16384 tokens at
world 8 / hidden 6144 / bf16 were compared elementwise against NCCL with 0
mismatches; the maximum absolute difference is a constant 2.5e-1, one bf16
quantization step at that magnitude, i.e. accumulation order rather than
corruption.
Benchmarking Results
One 8x RTX PRO 6000 Blackwell Server Edition host, GPUs 0-3 and 4-7 on separate
NUMA nodes, no NVLink. Same tree, same client, same server args; the arms differ
by one environment variable.
Mean values. KV pool was 342912 tokens in every arm;
#cached-tokenwas 0 onevery prefill, so no TTFT is a prefix-cache artifact.
TTFT (ms)
TPOT (ms)
Output throughput (tok/s)
TPOT is where these kernels pay: -34% to -36% at bs1, and the ratio does not
depend on context length, since a decode reduction's size is set by the batch.
TTFT and the 128k bs4 cells are level with NCCL by construction -- those
reductions are above the workspace bound and stay on NCCL.
Before any of this was measured, all 56 ordered GPU pairs were verified to carry
data intact. A sister host with the same driver returned zeroed buffers on every
cross-device copy while
nvidia-smi topo -p2p rreported all-OK; a bandwidthtest does not catch that.
Known limitations
Requires a recent FlashInfer. [feat]custom all reduce kernel flashinfer-ai/flashinfer#4393 merged on 2026-08-20, so
PcieIpcAllReduceWorkspaceis available upstream and its signature is settled. TheImportErrorpath degrades to NCCL with a warning, so builds without the module are unaffected.TP4 coverage arrived mid-review. Earlier revisions of this description said the
kernels had no effect at
world_size == 4: FlashInfer's policy table gated that branch onhidden == 4096exactly, so GLM's 6144 was rejected on every shape. Upstream has sinceremoved that constraint (
remove hidden size constraintin flashinfer#4393, closingflashinfer#4463), and TP4×PP2 now
works. Measured on the same host, against NCCL:
The gain is smaller than at 8 ranks (-10% against -36%), which is what the fabric predicts:
at 4 ranks NCCL has half the peers to reach and a less degraded baseline to beat. Kernel
numerics at world 4 / hidden 6144: 6 shapes, 0 skipped, 0 mismatches.
Not the best option at every point. On the same host a bandwidth-optimal FP8-compressing ring beats these kernels on prefill TTFT (1381 ms vs 2239 at 8k bs1) while losing on decode. There is no single best all-reduce on this fabric, which is part of why this is opt-in rather than autodetected.
Single host, single model, bf16 reductions only. No world-size-2 coverage, no unit test yet.
Checklist
pre-commit run --all-filestest/registered/unit/distributed/test_pcie_ipc_ar.py, CPU-only)CI States
Latest PR Test (Base): ❌ Run #34474387380
Latest PR Test (Extra): ❌ Run #34474387092
Latest PR Test (AMD ROCm 10): ❌ Run #34474387492