Repository navigation
feat(comm): add Kimi K3 H7168/K16 MNNVL CuTe DSL profiles for TP4 and TP8 - #5701
Merged
Vinnie6167 merged 3 commits intoOct 15, 2026
Merged
Vinnie6167 merged 3 commits into
Vinnie6167 merged 3 commits into
Conversation
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
benchislett
requested review from
Anerudhan,
Aneureka,
Jingfan-Sun,
YangXu1990uiuc,
aleozlx,
bkryu,
cyx-6,
dhiraj113,
feih-nv,
jiahanc,
jimmyzho,
kahyunnam,
nv-yunzheq,
qiching,
qsang-nv,
saltyminty,
samuellees,
sricketts,
yongwww,
yyihuang and
yzh119
as code owners
September 29, 2026 21:02
Contributor
|
Important Review skippedWe couldn't safely recover the incremental review. No full review was started, and the last reviewed checkpoint was preserved. Retry later, or explicitly request a full review by commenting You can disable this status message by setting the Use the checkbox below for a quick retry:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to
|
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Docstring Coverage | Docstring coverage is 4.55% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 22 functions across 11 files. | Write docstrings for the functions missing them to satisfy the coverage threshold. |
✅ Passed checks (4 passed)
-
Check name Status Explanation Title check ✅ Passed The title clearly and concisely identifies the addition of Kimi K3 H7168/K16 MNNVL CuTe DSL profiles for TP4 and TP8, matching the main changes. Description check ✅ Passed The description is complete and on topic. It explains the scope, routing, presets, performance results, reproduction steps, tests, checklist status, and reviewer focus areas. The experimental section … Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request. Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
-
🛠️ Fix failing CI checks 💡
-
- Commit to this branch
- Create a new PR
🧪 Generate unit tests (beta)
-
- Create a new PR
-
Comment @coderabbitai help to get the list of available commands.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
/bot run tests/comm |
Contributor
|
@flashinfer-bot run |
Collaborator
Anerudhan
approved these changes
Oct 15, 2026
5 tasks done
1 task done
triple-mu
added a commit
that referenced
this pull request
Oct 24, 2026
<!-- .github/pull_request_template.md --> ## 📌 Description Adds `UlyssesCommunicator(backend="pcie")`: an experimental single-node all-to-all transport for PCIe-connected GPU groups without NVLink (e.g. 8x RTX PRO 5000), targeting Ulysses sequence parallelism in video/image diffusion serving. Being experimental, it is reachable from `backend="auto"` only where NVLink is unavailable and `FLASHINFER_ALLOW_EXPERIMENTAL_AUTO_BACKENDS=1` is set identically on every rank; naming it explicitly never needs that variable. Either way it warns once. **Why.** The existing NVLink backend lets SM threads store into peer-visible staging buffers; across a PCIe host bridge that degenerates into small transactions (measured 5-13x slower than copy engines during bring-up, see `docs/design_docs/ulysses_pcie.md`). NCCL works but leaves bandwidth and stability on the table on this fabric (see numbers below). **What.** - Routes: world size 1 is an identity path; 2 uses all-pairs CUDA P2P copy engines (`cudaMemcpy2DAsync` writes land directly in the peer's final layout, no pack/unpack); 4/8 prefer an all-RDMA route (per-rank mlx5 RC QP, interleaved-UMR MKeys, GPUDirect RDMA) and fall back to all-P2P with a `RuntimeWarning`; an eight-rank 4+4 NUMA hybrid (same-NUMA P2P, cross-NUMA mlx5) is available via `FLASHINFER_ULYSSES_PCIE_ROUTE`. - Transforms: `scatter_heads` / `gather_heads` interleave the head axis; `exchange_chunks` takes `[1, 1, world_size, chunk]` to the same shape (`all_to_all_single` over a payload a preceding pack already laid out destination-major) and interleaves nothing. On the RDMA routes it configures no MKey, which is what frees it from the 16-bit `bytes_count` that caps the head-axis transforms at 65535 B per peer. - Synchronization: full-group epoch barriers in device memory (release/acquire, sticky abort half for the RDMA routes); the all-P2P route is CUDA-graph capturable, the RDMA routes refuse capture and bound NIC stalls with a 10 s deadline plus a group-wide abort/quiesce protocol. - Lifecycle: explicit `allocate_output` registration (collective, lives until `close()`); `input_buffer()` exposes the RDMA landing buffer so producers can write operands where the NIC reads them, removing the largest device-to-device copy on the path (measured ~6% per call at layer scale). The backend has no staging workspaces: `workspace=`, `create_workspace` and the head-chunk primitives from #5027 are refused at world size above one with a message that names `allocate_output()`. - Failure envelope: a rejection before native code runs (bad operand, refused capture, second stream) is decided from the call's own arguments, so under the SPMD contract every rank rejects it -- the RDMA routes raise the ordinary error and stay usable, the all-P2P route (no abort protocol) poisons teardown and enters BROKEN. A failure inside the transport is fail-stop on every route: on the RDMA routes the failing rank publishes the sticky abort, the peers leave the barrier and `close()` completes; on all-P2P `close()` refuses to synchronize and the process must exit. The design note has the table. - The element type is per-call (`dtype=` on the collectives), decoupled from the communicator: the transport moves opaque bytes, so FP8/INT8 payloads ride a BF16-sized communicator within the same byte budget. The constructor's capacity is therefore `max_bytes`, replacing main's element-count `max_elems`: the other communication workspaces in `flashinfer.comm` are sized in bytes as well (`buffer_size_in_bytes`, `workspace_size`), and an element count has no single meaning once the element type varies per call. #4924 already targets `max_bytes`. The staging-buffer API from #5027 (`UlyssesWorkspace(max_elems=...)`, `create_workspace(max_elems=...)`) keeps its element count, since those buffers hold one dtype. - Docs: design note (`docs/design_docs/ulysses_pcie.md`), `docs/api/comm.rst`, env-var reference; fi_trace templates for all three transforms (`scatter_heads` / `gather_heads` gain `trace=` decorators). - Housekeeping: six `flashinfer.comm` re-exports of JIT module helpers with no users in the repository (`gen_trtllm_comm_module`, `gen_vllm_comm_module`, `gen_pcie_ipc_comm_module`, `get_pcie_ipc_comm_module`, `gen_ulysses_a2a_module`, `get_ulysses_a2a_module`) are dropped; the helpers stay importable from their own modules and from `flashinfer.jit`. **Changes since the approved revision (a481f1d).** The branch was rebased onto main (#5027 workspace API, #5263 generated NVLink kernels) and the following landed on top, in this order: 1. `fix(comm): finish the max_bytes rename in the head-chunk tests, benchmark and export example` -- four `max_elems=` constructor calls that arrived with #5027 / #5263 (`_destination_passing_body`, `_head_chunk_body`, `bench_ulysses_head_chunk.py`, `examples/pytorch/ulysses_a2a_export/interface.py`) raised `TypeError` on any 2+ GPU host. 2. `format: format code style by pre-commit` -- the two hooks CI flagged. 3. `test(comm): share the Ulysses test helpers and gloo fixture` -- one FileStore `gloo_pg` fixture in `tests/comm/conftest.py`, `full_mesh` / `forbid_ipc_and_jit` / rendezvous helpers in `tests/test_helpers/ulysses.py`. 4. `fix(comm): check the group's CUDA all-to-all support before arming NVLink` -- the check ran only for `backend="nccl"`, after the NVLink IPC exports were armed. 5. `fix(comm): reject cross-rank FLASHINFER_ULYSSES_PCIE_NICS disagreement` -- the NIC override is now compared across ranks like the route and GID overrides. 6. `feat(comm): warn at garbage collection when a Ulysses communicator was not closed` -- `ResourceWarning` from `__del__` for an armed communicator; no teardown from the finalizer. 7. `fix(comm): refuse workspaces and the head-chunk primitives on the multi-rank PCIe backend` -- see Lifecycle above. 8. `fix(comm): publish the abort when the RDMA routes reject an operand natively` -- the RDMA routes' per-call operand checks run inside the native abort envelope, so a rank whose operand native rejects releases its peers instead of leaving them in the opening barrier (the only native change: the checks moved into `ValidateExchangeOperands`, protocol and wire format untouched); the caller's stream is bound only after a call passes validation; the Python-side envelope is unchanged and documented. 9. `refactor(comm): route Ulysses output validation through the shared helpers` -- `_output_geometry` is the single source of the output shape; the overlap test reuses `_storage_ranges_overlap`. 10. `docs(comm): update Ulysses PCIe prose for auto selection, the failure envelope and follow-ups`. 11. Rebased onto main 28ae778 (2026-09-24). The only conflict was `flashinfer/trace/templates/comm.py`, where #4929 and this PR both append templates at the end of the file; both blocks are kept, in main's order, and no other file changed in the rebase. 12. Merged main `7bcf91831` (2026-09-29): three additive conflicts (`flashinfer/aot.py`, `flashinfer/comm/__init__.py`, `tests/trace/template_registry.py`), both sides kept. Followed by `Find Conda rdma-core build inputs for PCIe Ulysses JIT` (72c958a): `gen_ulysses_pcie_module` adds `<sys.prefix>/include` and `-L<sys.prefix>/lib` when `infiniband/mlx5dv.h` / `libmlx5.so` live there (the CI Conda environment), and `missing_ulysses_pcie_dependencies()` now also probes for the `infiniband/verbs.h` / `infiniband/mlx5dv.h` headers (one host-mock test added; design note updated). 13. Merged main `7aa42e2dc` (2026-10-04). Two additive conflicts: `scripts/task_test_multi_gpu_comm_kernels.sh` (both sides extend `SPAWN_MANAGED_TEST_FILES`; the union is kept) and `tests/trace/example.py` (main sorted the `flashinfer.*` imports; `import flashinfer.comm` takes its place in that order). `CLAUDE.md`, `docs/api/comm.rst`, `flashinfer/aot.py` and `tests/trace/test_fi_trace_template_consistency.py` merged automatically. No file under `flashinfer/comm/ulysses*`, `flashinfer/jit/comm.py`, `csrc/ulysses*`, `include/flashinfer/comm/ulysses_pcie.cuh` or `tests/comm/test_ulysses*` changed on main between the two bases. 14. Merged main `188bdd769` (2026-10-05). Two modify/delete conflicts: #5953 removed `examples/pytorch/ulysses_a2a_export/` and moved its fixture to `tests/comm/cake_ulysses_a2a_fixture.py`, so the deletion is taken and the `max_bytes` rename from item 1 is re-applied to the moved fixture in `fix(comm): finish the max_bytes rename in the moved cake Ulysses fixture` (57ed509); `docs/api/comm.rst` and `flashinfer/aot.py` merged automatically. No file under `flashinfer/comm/ulysses*`, `flashinfer/jit/comm.py`, `csrc/ulysses*`, `include/flashinfer/comm/ulysses_pcie.cuh` or `tests/comm/test_ulysses*` changed on main between the two bases. 15. Rebased onto main `9c6797eda` (2026-10-05) at the maintainers' convenience for a linear history: the 28 non-merge commits above replay in order and the five merge commits (1f82831, 25bb39c, f7ce2ba, 5e8919a and their resolutions) are gone; `8668d921` is the new head. The replay re-resolved the same conflicts (`flashinfer/comm/__init__.py`, `flashinfer/aot.py`, `tests/trace/example.py`, `tests/trace/template_registry.py`, `scripts/task_test_multi_gpu_comm_kernels.sh`, the removed example files), and the result differs from the merged head `57ed5097` in exactly one place: `flashinfer/comm/__init__.py` no longer re-exports `gen_pcie_ipc_comm_module` / `get_pcie_ipc_comm_module`, which the merge in item 12 had brought back by accident; this is the drop described under Housekeeping, and the repository has no caller of either name through `flashinfer.comm`. 16. Rebased onto main `67c190f7d` (2026-10-06); `0f60c090` is the new head, still 28 linear commits. One conflict: the expected-output list in `tests/trace/example.py`'s docstring, where #6035 appended the `moe_layer_cudnn_frost_*` entries at the spot this PR appends the `ulysses_*` entries; both lists are kept, in main's order. `CLAUDE.md`, `flashinfer/aot.py` and `tests/trace/template_registry.py` merged automatically. No file under `flashinfer/comm/ulysses*`, `flashinfer/jit/comm.py`, `csrc/ulysses*`, `include/flashinfer/comm/ulysses_pcie.cuh` or `tests/comm/test_ulysses*` changed on main between the two bases. 17. Rebased onto main `87ef73934` (2026-10-14); `e7c7bbd50` is the new head, 29 linear commits. One conflict: the tail of `tests/trace/example.py`, where #5304 appended its `top_k_varlen` example calls at the spot this PR appends the single-rank Ulysses block; both are kept, in main's order (the docstring's expected-output list merged automatically this time). `CLAUDE.md`, `docs/api/comm.rst`, `flashinfer/aot.py`, `flashinfer/comm/__init__.py`, `scripts/task_test_multi_gpu_comm_kernels.sh` and `tests/trace/template_registry.py` merged automatically. #6332 (`flashinfer/comm/qkv_norm_rope_exchange_overlap.py` and `benchmarks/comm/bench_qkv_norm_rope_exchange_overlap.py`) arrived on main with three `UlyssesCommunicator(..., max_elems=...)` calls, so the 29th commit, `fix(comm): finish the max_bytes rename in the QKV norm/RoPE exchange overlap`, passes `max_bytes` (the same bf16 element counts times `torch.bfloat16.itemsize`) exactly as item 14 did for the moved fixture; nothing else on main constructs a communicator with the old keyword. No file under `flashinfer/comm/ulysses*`, `flashinfer/jit/comm.py`, `csrc/ulysses*`, `include/flashinfer/comm/ulysses_pcie.cuh`, `tests/comm/test_ulysses*` or `flashinfer/trace/templates/comm.py` changed on main between the two bases. 18. Rebased onto main `3637083a3` (2026-10-15); `f0ff368c1` is the new head, still 29 linear commits, no conflicts: #5701 extends `TORCHRUN_TEST_FILES` in `scripts/task_test_multi_gpu_comm_kernels.sh` next to the `SPAWN_MANAGED_TEST_FILES` line this PR extends, and both land. The base now includes #6509, the fix for the `tests/moe_ep/test_sm107_torch_staging.py` failure that was the only failing lane in this PR's 2026-10-15 CI run (unrelated to this PR, which touches no `moe_ep` file). No file under `flashinfer/comm/ulysses*`, `flashinfer/jit/comm.py`, `csrc/ulysses*`, `include/flashinfer/comm/ulysses_pcie.cuh`, `tests/comm/test_ulysses*` or `flashinfer/trace/templates/comm.py` changed on main between the two bases. **Performance** (8x RTX PRO 5000, dual-NUMA 4+4, PCIe Gen5 x16, one 400G mlx5 per GPU; `benchmarks/comm/bench_ulysses_pcie.py`, seq 37888 / heads 56 / head_dim 128 / bf16; median of host-clock loops, max-reduced across ranks, vs the `all_to_all_single`+permute reference; PCIe-side sd ≤ 8%, mostly < 1%, while the NCCL reference jitters up to 50-104% at eight ranks): | world | route (auto's pick in bold) | scatter | gather | qkv layer | per-rank remote BW | |---|---|---|---|---|---| | 2 | **all-P2P** | 2.02x | 2.00x | 2.01x | 50.5 GB/s | | 4 | **all-RDMA** | 1.97x | 1.97x | 1.97x | 42.3 GB/s | | 8 | **all-RDMA** | 1.49x | 1.43x | 1.48x | 42.6 GB/s | | 8 | 4+4 hybrid | — | — | 1.27x | 36.4 GB/s | | 8 | all-P2P (correctness fallback) | 0.81x | 0.43x | 0.30x | 8.8 GB/s | The transport moves opaque bytes, so the element type should change how long a call takes but not how fast the link runs. Same node, ws8 all-RDMA, seq 37888 / 56 heads / head_dim 128, only the dtype varies: | dtype | bytes/elem | scatter ms | qkv layer ms | remote BW (scatter) | |---|---|---|---|---| | bf16 | 2 | 1.389 | 5.583 | 42.8 GB/s | | fp8_e4m3fn | 1 | 0.695 | 2.871 | 42.7 GB/s | | int8 | 1 | 0.696 | 2.870 | 42.7 GB/s | Halving the element size halves the time (1.389 -> 0.695 ms) and leaves bandwidth put; the two 1-byte types are within noise of each other. ws2 P2P saturates ~95% of the effective Gen5 x16 link; the RDMA route reaches ~85% of the 400G NIC. Re-measured at this head on the same node (ws8 all-RDMA, same shapes, 5 trials x 200 iterations, other tenants idle) to confirm the round changed nothing on the hot path: scatter 1.427 ms at 41.6 GB/s remote bandwidth (table above: 1.389 ms / 42.8 GB/s), gather 1.554 ms at 38.2 GB/s, both with trial-to-trial sd below 2%; the q/k/v layer loop read 6.93 ms with sd 9.2%, which is inside its own noise band on a shared node and is not re-plotted here. The `rnr_nak_retry_err` counters on all eight NICs did not move during the run. **Limitations.** Experimental; single host; world sizes 1/2/4/8 (the copy scheduler's `rank ^ step` order needs powers of two); the RDMA routes require batch 1 and refuse CUDA-graph capture, and bound the head-axis transforms at head-row pitch `H*D*elem <= 65535` B (`exchange_chunks` registers no MKey and is not bounded); the JIT module links libibverbs/libmlx5 even when topology picks all-P2P, so it is JIT-only and deliberately absent from AOT; a transport failure is fail-stop as described above, and an all-P2P failure leaves the process to be terminated by the launcher. Validated on one topology (4+4 NUMA, one mlx5 per GPU); no work-request-level mlx5 unit tests yet — the RDMA data path is covered end to end. No CI lane exercises the PCIe wire path (the CI images carry no rdma-core and the GPU lanes are single-GPU); the host-mock and single-rank halves of the suite run there, the multi-rank half runs on the reference node. Planned follow-ups, prepared on a branch and deliberately kept out of this PR to leave the approved protocol untouched: publish the abort from Python on every route (the all-P2P route already imports every peer's signal buffer), so a Python-side rejection on a rank that broke the SPMD contract releases the peers too; publish a geometry word with the opening epoch so ranks that call with different shapes at the same position break the group instead of writing a misdescribed layout; post the RDMA receive before the opening barrier (closes the window in which a peer's write can arrive before the receive is posted; no RNR retries were counted during this round's benchmark, so this is hygiene rather than a measured stall); build the all-P2P route without rdma-core; `out=None` with lazy registration; per-output `capacity_bytes`. ## 🔍 Related Issues N/A ## 🚀 Pull Request Checklist ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. ## 🧪 Tests - [x] Tests have been added or updated as needed. - [x] All tests are passing (`unittest`, etc.). Ran on the 8-GPU reference node (8x RTX PRO 5000, dual-NUMA 4+4, one mlx5 per GPU, rdma-core 63, CUDA 13.4, torch 2.13) with all eight GPUs at `ef9a91a22`, the head before the rebase (2026-09-20): `tests/comm/test_ulysses_communicator.py` 132 passed / 22 skipped / 0 failed, `tests/comm/test_ulysses_topology.py` 112 passed / 0 skipped (244 passed / 22 skipped in one run), `tests/trace/test_ulysses_trace.py` + `tests/trace/test_template_registry.py` 10 passed. The 22 skips are the forced-NVLink family on this NVLink-less machine; every PCIe case ran (p2p, all-RDMA and 4+4 hybrid routes at world sizes 2/4/8, the chunk exchange, the landing-buffer producer path, the p2p fail-stop pair, and the two-rank RDMA case in which rank 0's native dtype rejection publishes the abort, rank 1 leaves the barrier with "peer 0 aborted the exchange", and both ranks close). Correctness is bit-exact against the `all_gather`-based reference for fp16/bf16/fp32 and byte-exact for int8/uint8/fp8 across all three routes, including non-power-of-two head dims on the UMR path, CUDA-graph capture+replay of a q/k/v/out layer on the P2P route, and `backend="auto"` with the experimental opt-in selecting PCIe on this node. Re-run after the rebase at the pushed head `4fec510aa` (2026-09-24) on the same node, which was shared with another tenant on GPUs 0-3, so the suite ran on GPUs 4-7 (`CUDA_VISIBLE_DEVICES=4,5,6,7`): `tests/comm/test_ulysses_communicator.py` 122 passed / 32 skipped / 0 failed, `tests/comm/test_ulysses_topology.py` 112 passed, `tests/trace/test_ulysses_trace.py` + `tests/trace/test_template_registry.py` 10 passed. The skips are the forced-NVLink family plus the world-size-8 cases (all-RDMA and the 4+4 hybrid at eight ranks), which need eight visible GPUs; every world-size 2/4 case ran (p2p and all-RDMA routes, the chunk exchange, the landing-buffer producer path, the fail-stop pairs). The eight-rank paths stand on the run above: no file under `flashinfer/comm/ulysses*`, `csrc/ulysses*` or `tests/comm/test_ulysses*` changed on main between the two bases, and the rebase touched only the trace template file. Re-run after the 2026-10-04 merge at `25bb39c4` on the same node, first on GPUs 4-7 while GPUs 0-3 were held by another tenant (`tests/comm/test_ulysses_communicator.py` 123 passed / 32 skipped), then on all eight GPUs once they freed up: `tests/comm/test_ulysses_communicator.py` 133 passed / 22 skipped / 0 failed, `tests/comm/test_ulysses_topology.py` 112 passed, `tests/trace/test_ulysses_trace.py` + `tests/trace/test_template_registry.py` 10 passed; `pre-commit run --from-ref origin/main --to-ref HEAD` is green, `scripts/check_pr_document.py` reports no findings and `scripts/check_pr_api_diff.py` only the `max_bytes` advisory. The 22 skips are the forced-NVLink family; every PCIe case ran, including the eight-rank all-RDMA and 4+4 hybrid routes. The head `f7ce2baf` on top merges main `4b6d64be` (#6040, `cake_selective_state_update` only) and touches no Ulysses file. Re-run after the 2026-10-05 merge at the pushed head `57ed5097` on the same node with all eight GPUs: `tests/comm/test_ulysses_communicator.py` 133 passed / 22 skipped / 0 failed, `tests/comm/test_ulysses_topology.py` 112 passed, `tests/trace/test_ulysses_trace.py` + `tests/trace/test_template_registry.py` 10 passed, and `tests/comm/test_cake_ulysses_a2a.py` + `tests/comm/test_ulysses_a2a.py` + `tests/comm/test_ulysses_head_chunk.py` 16 passed / 4 skipped (the generated NVLink portfolio skips on this SM120 node; the moved fixture was additionally exercised by constructing its communicator with `max_bytes` in a one-rank process). The skips are the forced-NVLink family. `pre-commit run --from-ref <main> --to-ref HEAD` is green. Re-run at the rebased head `8668d921` on the same node with all eight GPUs: `tests/comm/test_ulysses_communicator.py` 133 passed / 22 skipped / 0 failed, `tests/comm/test_ulysses_topology.py` 112 passed, `tests/trace/test_ulysses_trace.py` + `tests/trace/test_template_registry.py` 10 passed, `tests/comm/test_cake_ulysses_a2a.py` + `tests/comm/test_ulysses_a2a.py` + `tests/comm/test_ulysses_head_chunk.py` 16 passed / 4 skipped, and the pcie_ipc module tests that import the JIT helpers from `flashinfer.comm.pcie_ipc_ar` still pass; `pre-commit run --from-ref <main> --to-ref HEAD` is green. Re-run at the rebased head `0f60c090` on the same node (all eight GPUs for the communicator suite, GPUs 0-3 for the rest while another tenant held GPU 6): `tests/comm/test_ulysses_communicator.py` 133 passed / 22 skipped / 0 failed, `tests/comm/test_ulysses_topology.py` 112 passed, `tests/trace/test_ulysses_trace.py` + `tests/trace/test_template_registry.py` 10 passed, `tests/comm/test_cake_ulysses_a2a.py` + `tests/comm/test_ulysses_a2a.py` + `tests/comm/test_ulysses_head_chunk.py` 15 passed / 5 skipped, pcie_ipc module tests pass; `pre-commit run --from-ref <main> --to-ref HEAD` is green. On a single-GPU host the same files give 186 passed / 90 skipped (host-mock and single-rank halves), `pre-commit run --all-files` is green, `scripts/check_pr_document.py` reports no findings, and `scripts/check_pr_api_diff.py` reports only the `max_elems` -> `max_bytes` constructor advisory explained above. Re-run at the rebased head `f0ff368c1` (2026-10-15) on the same node with all eight GPUs, from a fresh checkout in a new directory (torch 2.14.1+cu130, nvcc 13.0, cold JIT cache): `tests/comm/test_ulysses_communicator.py` 133 passed / 22 skipped / 0 failed, `tests/comm/test_ulysses_topology.py` 112 passed, `tests/trace/test_ulysses_trace.py` + `tests/trace/test_template_registry.py` 10 passed, `tests/comm/test_cake_ulysses_a2a.py` + `tests/comm/test_ulysses_a2a.py` + `tests/comm/test_ulysses_head_chunk.py` 16 passed / 4 skipped, and the six `tests/comm/test_pcie_ipc_*.py` files that import `flashinfer.comm.pcie_ipc_ar` 73 passed / 12 skipped / 1 xfailed. The 22 skips are the forced-NVLink family; every PCIe case ran, including the eight-rank all-RDMA and 4+4 hybrid routes. On the single-GPU host the four core files give 187 passed / 90 skipped / 0 failed, `pre-commit run --all-files` is green, `scripts/check_pr_document.py` reports no findings and `scripts/check_pr_api_diff.py` only the `max_bytes` advisory. `git range-diff` against the previous head `313ac10b8` shows the 29 commits replay with context-only changes. ## Reviewer Notes - The riskiest pieces are the epoch-barrier protocol (`include/flashinfer/comm/ulysses_pcie.cuh`) and the mlx5 transport (`csrc/ulysses_pcie_transport.cuh`): QP bring-up, interleaved-MKey geometry (pitch ≤ 65535), completion accounting with per-exchange immediate tags, and the abort/quiesce teardown ledger. - `_validate`'s byte-budget capacity math after the per-call dtype change (`flashinfer/comm/ulysses.py`) is worth a close read; the rescale rationale is in the commit message of the dtype commit. - The failure envelope is route-dependent on purpose, and the design note's "Failure envelope" table is the contract to review against. Python-side rejections are rank-uniform under the SPMD contract, so the RDMA routes keep the communicator usable (the multi-rank tests rely on refusing capture and a batch > 1 operand recoverably); the all-P2P route poisons teardown because its barrier has no abort protocol at all. Native failures on the RDMA routes publish the abort (now including the per-call operand checks) and `close()` completes. Publishing the abort from Python on every route is the first follow-up above. - Things that are correct but will draw a question: (1) full-group CUDA IPC is required even on the RDMA routes, because the barrier signals are always IPC-mapped -- RDMA removes the P2P *data* requirement, not the P2P requirement; (2) `batch == 1` on the RDMA routes comes from mode 0's flat `rank * payload` remote offset, not from the NIC; (3) `input_ready` is deliberately reused between `RunHybridBarrier` and `EnqueueCopies` so the side streams inherit the post-barrier state; (4) the copy-engine writes are ordered against the closing barrier's release store through the `copy_done` event join, i.e. through `cudaMemcpyAsync` completion semantics; (5) the per-call `dtype` is really per-`allocate_output`: the registered buffer pins it and native re-checks every call against it. - Placement: the backend uses the experimental machinery (warn-once, gated `auto`, no AOT) but lives in `flashinfer/comm/` with tests in `tests/comm/`, because it extends the stable `UlyssesCommunicator` class and the routing shares its decision layer; there is no precedent for an `@experimental_backend` in core. If the maintainers prefer the Experimental Track layout, the code splits at the `_pcie_*` seam (the methods touch only `rank`, `world_size`, `device`, `dtype`, `max_bytes`, `backend`, `transport`, `decision`, `_state` and the `_gather` / `_require_open` / `_validate` / `_output_geometry` helpers) and we will move it. --------- Co-authored-by: Alex Yang <aleyang@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📌 Description
Disclaimer: this PR was generated by agent. I reviewed the code and generated description, as well as supervised+executed+verified the reported speedups from the performance tables below. Tests and pre-commit were confirmed to have been run and passing.
This PR extends the Blackwell MNNVL CuTe DSL all-reduce fusion backend to the
Kimi K3 communication shape:
hidden_size=7168, bf16,top_k=16, for TP4and TP8.
No kernel algorithm changes are included. The existing LL, BT, and HT
protocols are reused as-is; this change adds tuning presets and measured
M-range routing through the existing tuning dataclasses. It does not add newtuning knobs.
The profile covers both existing fusion patterns:
The Kimi K3 semantics used for tuning and validation are:
7168bfloat16top_k161e-50.01.0Profiles
The new presets use only the launch knobs already exposed by the LL, BT, and
HT tuning dataclasses.
cluster_size=7with 128 collective threads gives a fragment stride of7 * 128 = 896, so the collective covers one token in one fully utilizedtrip. TP8 finalize uses a separately measured decode preset with two rank
lanes, 256 collective threads, and an eight-route prefetch group.
throughput-oriented P1. The two presets are selected with one measured split
per path.
448 x 2consumer shape and TP4all-reduce uses
128 x 7; both tile all 896 packs exactly.HT is structurally unavailable for H7168 at TP8. A TP8 reduction shard has
896 / 8 = 112packs, which is not divisible across any legal 32-threadreduction-warp count. TP8 therefore remains on BT above the LL range. Adding a
TP8 HT route would require a predicated reduction tail, which is a kernel
change rather than a tuning-preset change.
Routing
The routing below comes from a 30-repeat CUDA-graph finalist sweep over
M=1..8192on NVIDIA GB200. TP4 used one exclusive node; TP8 used twoexclusive nodes over multi-node NVLink.
M <= 2425-512513-768M >= 769M <= 2829-512513-2048M >= 2049M <= 1213-512M >= 513M <= 1213-256M >= 257The dispatcher deliberately stays compact: one LL region and at most two BT
regions per path, followed by HT only where it is structurally valid. Compared
with selecting the best finalist independently at every measured
M, thecompact routes give up:
0.09%0.94%0.27%2.36%0.19%1.76%0.05%0.93%🔍 Related Issues
Performance
NVIDIA GB200, bf16,
hidden_size=7168,top_k=16. TP4 is one node; TP8spans two nodes over multi-node NVLink. Nodes were exclusive.
Each point uses 10 warmup iterations followed by 30 measured CUDA-graph
replays. Each graph contains 20 back-to-back calls. Reported latency is the
median of the per-sample maximum across ranks; this avoids reporting one rank
as complete while a collective peer is still running. Cold-L2 rotation is
disabled to measure the intended steady-state path. All times below are
microseconds.
2.14.0+cu13013.02.30.7Baselines:
fused_add_rmsnorm, preceded by atorch MoE finalize and shared-expert add for the finalize pattern.
TRT-LLM is shown only at TP4 because this benchmark path is single-node. Its
MoE-finalize one-shot path supports
M <= 128; larger finalize points aremarked
unsupportedrather than treated as failures. “Fused” is the exactcandidate selected by the new
DEFAULT_CONFIGroute.AR + residual + RMSNorm — TP4
MAR + residual + RMSNorm — TP8
MMoE finalize + shared expert + AR + residual + RMSNorm — TP4
MMoE finalize + shared expert + AR + residual + RMSNorm — TP8
MSummary over the full measured range
1.012-1.466x31/311.020-1.360x31/311.234-2.090x31/311.349-5.555x31/311.154-7.558x31/31M <= 128)1.922-3.630x18/1815.022-59.337x31/3115.452-51.992x31/31LL leads the decode range, BT covers the middle, and TP4 HT takes over at
larger
M. TP8 remains on BT because HT cannot legally tile the H7168 TP8reduction shard. The routed CuTe DSL profile beats MNNVL AUTO at every measured
all-reduce point, including the boundary regions where the margin is smallest.
Reproduction
The benchmark added by this PR can measure the shipped dispatcher directly
with
--protocols auto. It also acceptsll,bt, andht, plusbt0andbt1to pin one BT preset while checking a crossover.TP4, one node:
TP8, two nodes:
To inspect the measured protocol and BT-preset crossovers, replace
--protocols autowith--protocols ll,bt0,bt1,htat TP4 or--protocols ll,bt0,bt1at TP8. The included benchmark reports both fusedlatency and the unfused reference. MNNVL AUTO and TRT-LLM AUTO in the tables
above were collected with the same inputs, timing method, and numerical
validation in the final tuning campaign.
🧪 Tests
protocol and preset identity at every transition.
fusion patterns.
outputs during the tuning sweep.
pre-commit run --all-filessuccessfully.Local validation results:
tests/comm/test_mnnvl_cutedsl_config.pyDEFAULT_CONFIGTP4 benchmark smoke testThe GPU test covers:
rms_eps=1e-5,weight_bias=0.0semantics.
🚀 Pull Request Checklist
✅ Pre-commit Checks
pre-commit.pre-commit run --all-filesand fixed any reported issues.✅ Tests
from the benchmark added by this PR.
Reviewer Notes
The routing tables and presets are the substance of this PR; the LL, BT, and HT
kernel algorithms are unchanged. In particular, it is worth reviewing:
cluster_size=7;Moracle losses shownabove.
The benchmark also adds an explicit
automode so the shipped cross-protocolDEFAULT_CONFIGroute can be measured directly. Optionalbt0andbt1modes pin either BT preset for crossover analysis.
Summary by CodeRabbit