Repository navigation
refactor(cake_ulysses): arch-neutral generated kernels, drop the 36 unused bindings and the examples-hosted test fixture - #5953
Merged
Conversation
…nused bindings and the examples-hosted test fixture csrc/generated/ulysses/sm_100a/ shipped 60 files (17.7K lines) for the NVLink Ulysses all-to-all: 24 kernels plus 36 *_binding.cu files (12.7K lines) that nothing calls, since the dispatcher launches the generated kernels directly through cudaLaunchKernel. The loader selected the generated module only for an exact single-target list (10.0a or 10.3a), and the test imported its fixture from examples/. - One kernel unit per (dtype, world size, direction) directly under csrc/generated/ulysses/ (24 files, same kernel text as the sm_100a copies; the sm_103a build compiles the same sources); the sm_100a/ directory and the 36 binding files are removed. - flashinfer/jit/cake_ulysses.py: the generated module is built whenever the target list contains a major-10 target (one -gencode per 10.x target) and selected when the current device's capability is among the built targets; otherwise jit/comm.py keeps the portable kernel as before. - csrc/cake_ulysses_dispatch.cu: one extern "C" declaration per kernel and a (dtype, world, mode) -> (kernel, smem) table behind the unchanged LaunchGeneratedUlysses signature; same selected entries as before. - tests/comm/cake_ulysses_a2a_fixture.py (moved from examples/) carries the 40-row portfolio; tests/comm/test_cake_ulysses_a2a.py runs it for world sizes 2/4/6/8 and adds build-target selection cases; the benchmark imports the moved fixture. Generated-kernel SASS is equivalent to the base on sm_100a and sm_103a (24/24 kernels); the public API, flashinfer/comm/ulysses.py and jit/comm.py are untouched. Net -12,650 lines. Closes flashinfer-ai#5807. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Collaborator
Author
|
/bot run tests/comm/test_cake_ulysses_a2a.py |
Collaborator
Author
|
@flashinfer-bot run tests/comm/test_cake_ulysses_a2a.py |
Contributor
|
Navigate logical layers of code changes, visualize relationships, and explore their blast radius. No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: ⛔ Files ignored due to path filters (60)
📒 Files selected for processing (8)
💤 Files with no reviewable changes (3)
Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 4 remain after this review. 📝 WalkthroughWalkthroughThe PR updates generated Ulysses kernel target selection and dispatch. It adds a shared all-to-all fixture under tests, redirects benchmark and test imports to that fixture, and removes the former example interface, shape portfolio, and README. ChangesUlysses all-to-all
Priority: ➖ Normal Estimated code review effort: 3 (Moderate) | ~25 minutes Possibly related PRs
Merge Risk: ⚪ Minimal · up to The investigated build and test paths have no identified blocker. The PR is mergeable after normal checks. Security Architecture ReviewSecurity architecture risk: 🔵 Low · up to The broader GPU-target selection interacts with a process-wide cache: switching devices after the first load can skip the promised fallback and fail collective work. The exposure is bounded to participating GPU processes; no privilege escalation or cross-tenant access was established. Retained concerns
Security review detailsSecurity Blast Radius
Trust Boundaries and Controls
Resilience and Maintainability Implications
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
Full details: Linked Issues checkExplanation Issue
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
yzh119
approved these changes
Oct 5, 2026
yzh119
approved these changes
Oct 5, 2026
triple-mu
added a commit
to triple-mu/flashinfer
that referenced
this pull request
Oct 5, 2026
Resolves two modify/delete conflicts against main 188bdd7: flashinfer-ai#5953 removed examples/pytorch/ulysses_a2a_export/ (its fixture now lives in tests/comm/cake_ulysses_a2a_fixture.py), so the deletion is taken. The max_bytes rename this branch had applied to the removed example is re-applied to the moved fixture in the follow-up commit.
triple-mu
added a commit
to triple-mu/flashinfer
that referenced
this pull request
Oct 5, 2026
flashinfer-ai#5953 moved examples/pytorch/ulysses_a2a_export/interface.py to tests/comm/cake_ulysses_a2a_fixture.py; its UlyssesCommunicator still passed max_elems, which this branch renamed to max_bytes.
5 tasks done
triple-mu
added a commit
to triple-mu/flashinfer
that referenced
this pull request
Oct 7, 2026
flashinfer-ai#5953 moved examples/pytorch/ulysses_a2a_export/interface.py to tests/comm/cake_ulysses_a2a_fixture.py; its UlyssesCommunicator still passed max_elems, which this branch renamed to max_bytes.
yyihuang
added a commit
that referenced
this pull request
Oct 12, 2026
…RMSNorm + RoPE overlapped with the Ulysses head exchange (SM90 / SM100 / SM103) (#6332) ## 📌 Description Part of #4254. Implements the overlap requested in #6304 (assumptions stated below where the issue thread has not confirmed a choice). Adds `flashinfer.comm.QkvNormRopeExchangeOverlap`: one prepared session that fuses the attention front end of a sequence-parallel (Ulysses) layer — the BF16 QKV projection, per-head RMSNorm of Q and K, interleaved RoPE from a per-token cos/sin table, and the head exchange — into a single call per layer. The Cake backend stores every destination's head band straight from the GEMM epilogue into that rank's symmetric receive slot over NVLink and publishes per-chunk readiness words, so the exchange overlaps the projection instead of following it; a one-CTA wait kernel on a consumer stream acquires the words and the caller's stream joins it. * Architectures: SM90 (WGMMA producer), SM100 and SM103 (tcgen05 producer; 2-CTA pair tile for `block_n = 256` and rows of at least 4096 tokens). Single-node NVLink groups of 2, 4 or 8 ranks, `head_dim = 128`, BF16 operands, fp32 cos/sin table. * Layouts: `"bshd"` returns `(q, k, v)` views of `[B, S_global, H_local, D]`; `"fused"` returns one `[B, S_global, H_local, 3 * D]` payload. Outputs are views of the current receive slot and stay valid until the slot is reused `num_slots` calls later; `out=` copies into caller-owned tensors. * Backend selection: `backend="auto"` goes through `resolve_ulysses_backend` (NVLink mesh probe) and the device capability; otherwise the NCCL path (cuBLAS projection, `flashinfer.norm.rmsnorm`, RoPE, `UlyssesCommunicator.scatter_qkv_head_chunk`). `backend="nvlink"` raises `UlyssesBackendError` when the mesh or the architecture is not supported; the NCCL path reports `graph_capturable == False`. * CUDA graphs: a prepared call allocates nothing and runs no host collective; captured calls publish a sentinel epoch and clear their readiness words with a stream memset after the wait kernel, so replays interleave with eager calls. A call with more rows than `max_tokens`, or a row count first seen inside a capture, raises instead of allocating. At most `num_slots` calls of one session can be captured into one graph without an eager call between. * `heads % world_size != 0` raises `ValueError` deterministically on every rank before any collective (same contract as `UlyssesCommunicator.scatter_heads`). ### Assumptions (open points in #6304) The issue thread has not yet confirmed the following choices; this PR implements the defaults below and labels them here so reviewers and the requester can object: * **Exchange algorithm**: Ulysses-style QKV all-to-all (each rank holds a token shard and the full QKV weight; every rank receives its head band of all tokens). Ring-style KV exchange is out of scope. * **Operator math**: the public FLUX definitions — fp32 RMSNorm with `eps = 1e-6` and a shared `[head_dim]` weight per Q / K, interleaved (pairwise) RoPE applied in fp32 from a per-token cos/sin table, in the order projection → QK-norm → RoPE; BF16 operands and outputs (no FP8 producer). * **Receive layout**: both the `(q, k, v)` `[B, S_global, H_local, D]` views and the fused `[B*S_global, 3*H_local*D]` payload are supported; the numbers below are measured with the `bshd` layout on every architecture. * **Tolerance**: `atol = rtol = 1e-2` against an fp32 step-by-step reference. Isolated single-ulp BF16 differences (a value landing on the other side of a BF16 rounding boundary after a different fp32 accumulation order, max_abs 0.03125) occur both in this kernel and in the existing per-operator path; the tests classify those elements explicitly instead of loosening the tolerance. ### Files | Path | Content | |---|---| | `flashinfer/comm/qkv_norm_rope_exchange_overlap.py` | public class, request validation, NCCL path | | `flashinfer/comm/cake_qkv_norm_rope_exchange_overlap.py` | Cake backend session: symmetric slots, barrier -> producer -> wait per call, capture protocol | | `flashinfer/comm/cake_qkv_exchange_plan.py` | tile plan and plan policy (host only) | | `flashinfer/jit/cake_qkv_norm_rope_exchange_overlap.py` | JIT loader: program and route tables, `gen_jit_spec` per program and architecture | | `csrc/cake_qkv_norm_rope_exchange_overlap/` | generated device and binding translation units (one `run` launcher per program; excluded from pre-commit like the other generated trees) | | `tests/comm/test_qkv_norm_rope_exchange_overlap.py` | host-only plan / validation / loader tests; 2-, 4-, 8-GPU correctness, capture/replay, refusals, NCCL path | | `benchmarks/comm/bench_qkv_norm_rope_exchange_overlap.py` | torchrun benchmark against the per-operator paths (CUPTI, cold L2) | | `docs/api/comm.rst`, `flashinfer/comm/__init__.py` | docs section and re-export | ## 🔍 Related Issues - #6304 — the feature request this PR implements; the choices the thread has not confirmed yet are labelled under "Assumptions" above. - Part of #4254. ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [ ] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). The hooks were run as `pre-commit run --files <the 31 files changed by this PR>` with the hook versions pinned in `.pre-commit-config.yaml` (unchanged by this PR); all 12 hooks pass, including clang-format, mypy, ruff check and ruff format. ## 🧪 Tests - [x] Tests have been added or updated as needed. - [ ] All tests are passing (`unittest`, etc.). Every path is checked against an fp32 step-by-step reference (projection, RMSNorm with the bf16 cast before the bf16 scale, interleaved RoPE, bf16 output) redistributed with `all_gather`, `atol = rtol = 1e-2`, on both layouts including a tail row count (`S_local = 200`), after CUDA-graph replays with fresh inputs, with two sessions in flight on one group, and on the NCCL path. The multi-GPU tests spawn 2, 4 and 8 ranks on one node and skip on fewer devices or an unsupported architecture. The host-only tests of `tests/comm/test_qkv_norm_rope_exchange_overlap.py` (`-k "not multi_gpu"`, 21 passed) ran on the pre-check node; the 2-, 4- and 8-GPU tests need a multi-GPU node and are left to CI. ### Test plan - [ ] `pytest tests/comm/test_qkv_norm_rope_exchange_overlap.py -q` on an 8-GPU SM100 node - [ ] same on an 8-GPU SM103 node - [ ] same on an 8-GPU SM90 node - [ ] `torchrun --standalone --nproc-per-node=8 benchmarks/comm/bench_qkv_norm_rope_exchange_overlap.py --rows representative --json bench_w8.json` per architecture - [x] `pre-commit run --files <new files>` (31 files; 12 / 12 hooks pass) ## 🔬 Experimental Track <!-- Only for PRs submitted under the experimental policy (CONTRIBUTING.md → "Experimental APIs and Backends"). Leave this section untouched for normal PRs. --> - [ ] This PR is **experimental**: it adds or changes code under `flashinfer/experimental/` and/or an `@flashinfer_experimental_api`. Tracking issue: # - [ ] The tracking issue names an owner, the reason for the experimental path, and a graduation plan with a target release. - [ ] Core changes are limited to a thin entry point (signature, shared validation, feature-gate check, backend selection, handoff). - [ ] Tests live in `tests/experimental/` and were validated on the intended hardware; a runnable example is included. - [ ] Nothing is registered in `flashinfer/aot.py`, and no experimental backend is reachable from `backend="auto"` without `FLASHINFER_ALLOW_EXPERIMENTAL_AUTO_BACKENDS=1`. (Calling an `@flashinfer_experimental_api` or naming a backend explicitly is itself the opt-in and needs no environment variable.) - [ ] **Test scope declared below.** The experimental CI lane runs exactly these targets, so keep them as narrow as the change allows. <!-- Required for experimental PRs. Replace the commented lines below with your targets. Do not delete the fence or change its `experimental-tests` tag — the experimental-track watcher reads it verbatim to decide which targets to ask CI for. --> ```experimental-tests # One target per line: a directory or a file. (A pytest ::selector is not # supported -- the sharding runner cannot consume one.) Must be under # tests/experimental/ and must exist. Delete these comment lines and add yours, e.g. # # tests/experimental/test_my_backend.py # tests/experimental/my_backend/ # # Declaring the whole tree (tests/experimental/) is allowed but means every # experimental PR pays for every other feature's tests, in every matrix cell. ``` ## Reviewer Notes - The device and binding translation units under `csrc/cake_qkv_norm_rope_exchange_overlap/` are generated and attested by content hash; the host path (`flashinfer/comm/qkv_norm_rope_exchange_overlap.py`, `cake_qkv_norm_rope_exchange_overlap.py`, `cake_qkv_exchange_plan.py`), the JIT loader and the tests are hand-written and are the parts to read first. - The Speedup section separates the benchmark table from the export-protocol receipts; rows that failed the protocol's validity gates (power-capped clocks) are listed, not reported as measured. ## Speedup Measured with `benchmarks/comm/bench_qkv_norm_rope_exchange_overlap.py` (CUPTI activity timing, cold L2, per-rank maximum then median; `--rows representative`) against the fastest per-operator path on the same node. The per-operator baselines are: B1 three `scatter_heads` collectives, B2 one `scatter_qkv_head_chunk` over the whole head band, B3 the default head-chunk schedule, B4 `ulysses_a2a_in(fuse_qkv=True)` over NCCL. On SM90 the Ulysses communicator of B1-B3 runs on its NCCL backend (the NVLink Ulysses kernels target SM100 / SM103). ### Measured speedup (72 rows per architecture) Grid: `W ∈ {2, 4, 8}` ranks on one node × `hidden ∈ {3072, 4096, 6144, 8192}` (`head_dim = 128`) × `S_global ∈ {4096 … 131072}`, `B = 1`, BF16. Each row was timed on one 8-GPU node with CUPTI activity tracing (cold L2, per-rank maximum then median, five A/B/B/A rounds of ten samples, an A/A control arm within ±1 % or the row was re-measured) against the fastest of the four per-operator paths above measured in the same process. speedup = fastest baseline / fused. Every measured row is faster than its fastest baseline. | GPU | rows faster than the fastest baseline | geometric mean (A/A-valid rows) | W = 8 / 4 / 2 | minimum | |---|---|---|---|---| | B200 (SM100) | 72 / 72 | 1.831 | 1.824 / 1.849 / 1.820 | 1.261 | | B300 (SM103) | 72 / 72 | 1.906 | 1.911 / 1.891 / 1.916 | 1.241 | | H100 (SM90) | 71 / 71 | 1.332 | 1.339 / 1.348 / 1.310 | 1.135 | Notes: on B300 the row `W = 4, hidden 3072, S_global 8192` (2.26–2.31× in three measurements) exceeded the ±1 % A/A control in all three and is excluded from the geometric mean; on H100 (80 GB) the row `W = 2, hidden 8192, S_global 131072` does not fit in device memory with the reference and baseline buffers and is not measured. <details> <summary>B200 (SM100) — per-row table</summary> | W | hidden | S_global | fused ms | fastest baseline ms | speedup | A/A % | |---|---|---|---|---|---|---| | 8 | 3072 | 4096 | 0.0722 | 0.2389 | 3.307 | 0.11 | | 8 | 3072 | 8192 | 0.1014 | 0.2493 | 2.458 | 0.88 | | 8 | 3072 | 16384 | 0.1562 | 0.2821 | 1.807 | 0.09 | | 8 | 3072 | 32768 | 0.2713 | 0.4512 | 1.663 | 0.20 | | 8 | 3072 | 65536 | 0.5131 | 0.8815 | 1.718 | 0.10 | | 8 | 3072 | 131072 | 0.9575 | 1.7075 | 1.783 | 0.14 | | 8 | 4096 | 4096 | 0.0968 | 0.2440 | 2.522 | 0.71 | | 8 | 4096 | 8192 | 0.1193 | 0.2576 | 2.160 | 0.34 | | 8 | 4096 | 16384 | 0.1834 | 0.3626 | 1.977 | 0.67 | | 8 | 4096 | 32768 | 0.3086 | 0.6454 | 2.091 | 0.25 | | 8 | 4096 | 65536 | 0.5906 | 1.2448 | 2.108 | 0.05 | | 8 | 4096 | 131072 | 1.1808 | 2.4446 | 2.070 | 0.11 | | 8 | 6144 | 4096 | 0.1224 | 0.2503 | 2.045 | 0.62 | | 8 | 6144 | 8192 | 0.1902 | 0.3208 | 1.686 | 0.69 | | 8 | 6144 | 16384 | 0.3327 | 0.5789 | 1.740 | 0.32 | | 8 | 6144 | 32768 | 0.6438 | 1.1486 | 1.784 | 0.06 | | 8 | 6144 | 65536 | 1.3623 | 2.3209 | 1.704 | 0.17 | | 8 | 6144 | 131072 | 2.8806 | 4.4242 | 1.536 | 0.01 | | 8 | 8192 | 4096 | 0.1897 | 0.2766 | 1.458 | 0.26 | | 8 | 8192 | 8192 | 0.3278 | 0.4798 | 1.464 | 0.54 | | 8 | 8192 | 16384 | 0.5804 | 0.9063 | 1.562 | 0.22 | | 8 | 8192 | 32768 | 1.2319 | 1.7849 | 1.449 | 0.08 | | 8 | 8192 | 65536 | 2.5323 | 3.5145 | 1.388 | 0.06 | | 8 | 8192 | 131072 | 5.1799 | 6.9348 | 1.339 | 0.00 | | 4 | 3072 | 4096 | 0.0943 | 0.2456 | 2.605 | 0.25 | | 4 | 3072 | 8192 | 0.1316 | 0.2755 | 2.093 | 0.67 | | 4 | 3072 | 16384 | 0.2077 | 0.4497 | 2.165 | 0.31 | | 4 | 3072 | 32768 | 0.3794 | 0.8772 | 2.312 | 0.19 | | 4 | 3072 | 65536 | 0.6863 | 1.6978 | 2.474 | 0.06 | | 4 | 3072 | 131072 | 1.4040 | 3.3245 | 2.368 | 0.10 | | 4 | 4096 | 4096 | 0.1200 | 0.2506 | 2.088 | 0.23 | | 4 | 4096 | 8192 | 0.1768 | 0.3601 | 2.037 | 0.50 | | 4 | 4096 | 16384 | 0.3100 | 0.6342 | 2.046 | 0.34 | | 4 | 4096 | 32768 | 0.5834 | 1.2666 | 2.171 | 0.13 | | 4 | 4096 | 65536 | 1.1508 | 2.4901 | 2.164 | 0.01 | | 4 | 4096 | 131072 | 2.3583 | 4.9367 | 2.093 | 0.01 | | 4 | 6144 | 4096 | 0.1901 | 0.3145 | 1.654 | 0.87 | | 4 | 6144 | 8192 | 0.3349 | 0.5856 | 1.748 | 0.37 | | 4 | 6144 | 16384 | 0.6606 | 1.1643 | 1.763 | 0.13 | | 4 | 6144 | 32768 | 1.3538 | 2.3620 | 1.745 | 0.04 | | 4 | 6144 | 65536 | 2.8794 | 4.4967 | 1.562 | 0.14 | | 4 | 6144 | 131072 | 5.8146 | 8.9359 | 1.537 | 0.01 | | 4 | 8192 | 4096 | 0.3220 | 0.4808 | 1.493 | 0.05 | | 4 | 8192 | 8192 | 0.5756 | 0.9014 | 1.566 | 0.03 | | 4 | 8192 | 16384 | 1.2447 | 1.8077 | 1.452 | 0.03 | | 4 | 8192 | 32768 | 2.5719 | 3.5700 | 1.388 | 0.10 | | 4 | 8192 | 65536 | 5.2743 | 7.0276 | 1.332 | 0.01 | | 4 | 8192 | 131072 | 10.2687 | 14.3224 | 1.395 | 0.23 | | 2 | 3072 | 4096 | 0.1234 | 0.2666 | 2.160 | 0.92 | | 2 | 3072 | 8192 | 0.1972 | 0.4282 | 2.171 | 0.52 | | 2 | 3072 | 16384 | 0.3494 | 0.8643 | 2.474 | 0.04 | | 2 | 3072 | 32768 | 0.6568 | 1.6784 | 2.555 | 0.16 | | 2 | 3072 | 65536 | 1.3126 | 3.2883 | 2.505 | 0.20 | | 2 | 3072 | 131072 | 2.7374 | 6.5161 | 2.380 | 0.13 | | 2 | 4096 | 4096 | 0.1788 | 0.3531 | 1.975 | 0.69 | | 2 | 4096 | 8192 | 0.3104 | 0.6411 | 2.065 | 0.40 | | 2 | 4096 | 16384 | 0.5834 | 1.2492 | 2.141 | 0.11 | | 2 | 4096 | 32768 | 1.1547 | 2.4569 | 2.128 | 0.06 | | 2 | 4096 | 65536 | 2.3984 | 4.8868 | 2.038 | 0.10 | | 2 | 4096 | 131072 | 4.8963 | 9.6901 | 1.979 | 0.01 | | 2 | 6144 | 4096 | 0.3260 | 0.5658 | 1.735 | 0.40 | | 2 | 6144 | 8192 | 0.6637 | 1.1501 | 1.733 | 0.14 | | 2 | 6144 | 16384 | 1.3643 | 2.3287 | 1.707 | 0.01 | | 2 | 6144 | 32768 | 2.8530 | 4.4548 | 1.561 | 0.14 | | 2 | 6144 | 65536 | 5.7498 | 8.8343 | 1.536 | 0.26 | | 2 | 6144 | 131072 | 11.7871 | 17.7952 | 1.510 | 0.15 | | 2 | 8192 | 4096 | 0.5740 | 0.8991 | 1.566 | 0.21 | | 2 | 8192 | 8192 | 1.2396 | 1.7892 | 1.443 | 0.01 | | 2 | 8192 | 16384 | 2.5594 | 3.5306 | 1.379 | 0.08 | | 2 | 8192 | 32768 | 5.2429 | 6.9682 | 1.329 | 0.23 | | 2 | 8192 | 65536 | 10.5083 | 14.1865 | 1.350 | 0.21 | | 2 | 8192 | 131072 | 23.0629 | 29.0725 | 1.261 | 0.63 | </details> <details> <summary>B300 (SM103) — per-row table</summary> | W | hidden | S_global | fused ms | fastest baseline ms | speedup | A/A % | |---|---|---|---|---|---|---| | 8 | 3072 | 4096 | 0.0774 | 0.2697 | 3.487 | 0.42 | | 8 | 3072 | 8192 | 0.1027 | 0.2863 | 2.788 | 0.66 | | 8 | 3072 | 16384 | 0.1579 | 0.3131 | 1.982 | 0.88 | | 8 | 3072 | 32768 | 0.2759 | 0.4525 | 1.640 | 0.39 | | 8 | 3072 | 65536 | 0.5072 | 0.8430 | 1.662 | 0.14 | | 8 | 3072 | 131072 | 0.9269 | 1.6214 | 1.749 | 0.02 | | 8 | 4096 | 4096 | 0.0982 | 0.2790 | 2.842 | 0.36 | | 8 | 4096 | 8192 | 0.1223 | 0.2896 | 2.367 | 0.54 | | 8 | 4096 | 16384 | 0.1788 | 0.3498 | 1.956 | 0.49 | | 8 | 4096 | 32768 | 0.2904 | 0.6136 | 2.113 | 0.04 | | 8 | 4096 | 65536 | 0.5417 | 1.1756 | 2.170 | 0.12 | | 8 | 4096 | 131072 | 1.0635 | 2.3010 | 2.164 | 0.05 | | 8 | 6144 | 4096 | 0.1258 | 0.2846 | 2.263 | 0.61 | | 8 | 6144 | 8192 | 0.1809 | 0.3167 | 1.751 | 0.51 | | 8 | 6144 | 16384 | 0.3056 | 0.5489 | 1.797 | 0.12 | | 8 | 6144 | 32768 | 0.5970 | 1.0776 | 1.805 | 0.01 | | 8 | 6144 | 65536 | 1.2046 | 2.0970 | 1.741 | 0.06 | | 8 | 6144 | 131072 | 2.4691 | 4.1316 | 1.673 | 0.01 | | 8 | 8192 | 4096 | 0.1852 | 0.2919 | 1.576 | 0.01 | | 8 | 8192 | 8192 | 0.3020 | 0.4518 | 1.496 | 0.74 | | 8 | 8192 | 16384 | 0.5334 | 0.8476 | 1.589 | 0.19 | | 8 | 8192 | 32768 | 1.0841 | 1.6653 | 1.536 | 0.10 | | 8 | 8192 | 65536 | 2.1971 | 3.2764 | 1.491 | 0.07 | | 8 | 8192 | 131072 | 4.4165 | 6.4274 | 1.455 | 0.03 | | 4 | 3072 | 4096 | 0.0945 | 0.2622 | 2.774 | 0.69 | | 4 | 3072 | 8192 | 0.1346 | 0.3073 | 2.283 (A/A > 1 %) | 4.33 | | 4 | 3072 | 16384 | 0.2104 | 0.4479 | 2.128 | 0.17 | | 4 | 3072 | 32768 | 0.3899 | 0.8362 | 2.145 | 0.33 | | 4 | 3072 | 65536 | 0.6896 | 1.6060 | 2.329 | 0.11 | | 4 | 3072 | 131072 | 1.3541 | 3.1379 | 2.317 | 0.05 | | 4 | 4096 | 4096 | 0.1207 | 0.2752 | 2.280 | 0.66 | | 4 | 4096 | 8192 | 0.1735 | 0.3476 | 2.003 | 0.09 | | 4 | 4096 | 16384 | 0.2905 | 0.6134 | 2.111 | 0.01 | | 4 | 4096 | 32768 | 0.5332 | 1.1986 | 2.248 | 0.06 | | 4 | 4096 | 65536 | 1.0389 | 2.3463 | 2.258 | 0.02 | | 4 | 4096 | 131072 | 2.0955 | 4.6322 | 2.211 | 0.01 | | 4 | 6144 | 4096 | 0.1834 | 0.3102 | 1.691 | 0.39 | | 4 | 6144 | 8192 | 0.3131 | 0.5571 | 1.779 | 0.09 | | 4 | 6144 | 16384 | 0.6136 | 1.0952 | 1.785 | 0.33 | | 4 | 6144 | 32768 | 1.2197 | 2.1381 | 1.753 | 0.01 | | 4 | 6144 | 65536 | 2.4356 | 4.1972 | 1.723 | 0.06 | | 4 | 6144 | 131072 | 4.9588 | 8.3107 | 1.676 | 0.03 | | 4 | 8192 | 4096 | 0.3039 | 0.4521 | 1.488 | 0.15 | | 4 | 8192 | 8192 | 0.5342 | 0.8521 | 1.595 | 0.06 | | 4 | 8192 | 16384 | 1.1044 | 1.6901 | 1.530 | 0.09 | | 4 | 8192 | 32768 | 2.1950 | 3.3179 | 1.512 | 0.08 | | 4 | 8192 | 65536 | 4.4062 | 6.5251 | 1.481 | 0.10 | | 4 | 8192 | 131072 | 9.4225 | 13.3648 | 1.418 | 0.28 | | 2 | 3072 | 4096 | 0.1199 | 0.2954 | 2.463 | 0.11 | | 2 | 3072 | 8192 | 0.1977 | 0.4372 | 2.211 | 0.15 | | 2 | 3072 | 16384 | 0.3375 | 0.8262 | 2.448 | 0.06 | | 2 | 3072 | 32768 | 0.5991 | 1.5968 | 2.665 | 0.04 | | 2 | 3072 | 65536 | 1.1957 | 3.1269 | 2.615 | 0.01 | | 2 | 3072 | 131072 | 2.3627 | 6.1780 | 2.615 | 0.10 | | 2 | 4096 | 4096 | 0.1690 | 0.3380 | 1.999 | 0.06 | | 2 | 4096 | 8192 | 0.2898 | 0.6124 | 2.113 | 0.01 | | 2 | 4096 | 16384 | 0.5334 | 1.1898 | 2.230 | 0.10 | | 2 | 4096 | 32768 | 1.0432 | 2.3309 | 2.234 | 0.09 | | 2 | 4096 | 65536 | 2.0717 | 4.6073 | 2.224 | 0.01 | | 2 | 4096 | 131072 | 4.2197 | 9.1462 | 2.167 | 0.05 | | 2 | 6144 | 4096 | 0.2977 | 0.5423 | 1.822 | 0.04 | | 2 | 6144 | 8192 | 0.6118 | 1.0884 | 1.779 | 0.04 | | 2 | 6144 | 16384 | 1.1907 | 2.1221 | 1.782 | 0.02 | | 2 | 6144 | 32768 | 2.3919 | 4.1900 | 1.752 | 0.04 | | 2 | 6144 | 65536 | 4.9604 | 8.2729 | 1.668 | 0.34 | | 2 | 6144 | 131072 | 11.4356 | 16.7756 | 1.467 | 0.23 | | 2 | 8192 | 4096 | 0.5286 | 0.8535 | 1.615 | 0.11 | | 2 | 8192 | 8192 | 1.0660 | 1.6769 | 1.573 | 0.01 | | 2 | 8192 | 16384 | 2.1406 | 3.3157 | 1.549 | 0.22 | | 2 | 8192 | 32768 | 4.5065 | 6.5023 | 1.443 | 0.24 | | 2 | 8192 | 65536 | 9.5439 | 13.3589 | 1.400 | 0.16 | | 2 | 8192 | 131072 | 22.1867 | 27.5295 | 1.241 | 0.03 | </details> <details> <summary>H100 (SM90) — per-row table</summary> | W | hidden | S_global | fused ms | fastest baseline ms | speedup | A/A % | |---|---|---|---|---|---|---| | 8 | 3072 | 4096 | 0.1070 | 0.1708 | 1.595 | 0.54 | | 8 | 3072 | 8192 | 0.1587 | 0.2288 | 1.442 | 0.13 | | 8 | 3072 | 16384 | 0.2722 | 0.3818 | 1.403 | 0.23 | | 8 | 3072 | 32768 | 0.5272 | 0.7191 | 1.364 | 0.16 | | 8 | 3072 | 65536 | 1.0614 | 1.4336 | 1.351 | 0.57 | | 8 | 3072 | 131072 | 2.1269 | 2.8805 | 1.354 | 0.34 | | 8 | 4096 | 4096 | 0.1444 | 0.1875 | 1.298 | 0.33 | | 8 | 4096 | 8192 | 0.2046 | 0.3137 | 1.533 | 0.45 | | 8 | 4096 | 16384 | 0.3804 | 0.5538 | 1.456 | 0.29 | | 8 | 4096 | 32768 | 0.7378 | 1.1132 | 1.509 | 0.87 | | 8 | 4096 | 65536 | 1.4582 | 2.1800 | 1.495 | 0.03 | | 8 | 4096 | 131072 | 2.9126 | 4.3244 | 1.485 | 0.13 | | 8 | 6144 | 4096 | 0.2603 | 0.3047 | 1.171 | 0.50 | | 8 | 6144 | 8192 | 0.4252 | 0.5419 | 1.274 | 0.27 | | 8 | 6144 | 16384 | 0.7434 | 1.0295 | 1.385 | 0.33 | | 8 | 6144 | 32768 | 1.5361 | 2.0466 | 1.332 | 0.46 | | 8 | 6144 | 65536 | 3.2696 | 4.0454 | 1.237 | 0.11 | | 8 | 6144 | 131072 | 6.4700 | 8.0205 | 1.240 | 0.01 | | 8 | 8192 | 4096 | 0.3395 | 0.4407 | 1.298 | 0.41 | | 8 | 8192 | 8192 | 0.6529 | 0.8201 | 1.256 | 0.36 | | 8 | 8192 | 16384 | 1.3057 | 1.6042 | 1.229 | 0.16 | | 8 | 8192 | 32768 | 2.6329 | 3.2563 | 1.237 | 0.08 | | 8 | 8192 | 65536 | 5.4181 | 6.4155 | 1.184 | 0.25 | | 8 | 8192 | 131072 | 11.4591 | 13.0004 | 1.135 | 0.23 | | 4 | 3072 | 4096 | 0.1570 | 0.2114 | 1.347 | 0.18 | | 4 | 3072 | 8192 | 0.2545 | 0.3688 | 1.449 | 0.02 | | 4 | 3072 | 16384 | 0.4623 | 0.7117 | 1.540 | 0.05 | | 4 | 3072 | 32768 | 0.8927 | 1.3835 | 1.550 | 0.02 | | 4 | 3072 | 65536 | 1.7488 | 2.7224 | 1.557 | 0.08 | | 4 | 3072 | 131072 | 3.4939 | 5.3476 | 1.531 | 0.11 | | 4 | 4096 | 4096 | 0.1987 | 0.2856 | 1.437 | 0.02 | | 4 | 4096 | 8192 | 0.3767 | 0.5468 | 1.452 | 0.09 | | 4 | 4096 | 16384 | 0.7412 | 1.0631 | 1.434 | 0.06 | | 4 | 4096 | 32768 | 1.4287 | 2.0803 | 1.456 | 0.01 | | 4 | 4096 | 65536 | 2.8435 | 4.1326 | 1.453 | 0.02 | | 4 | 4096 | 131072 | 5.7662 | 8.1828 | 1.419 | 0.27 | | 4 | 6144 | 4096 | 0.4104 | 0.5240 | 1.277 | 0.08 | | 4 | 6144 | 8192 | 0.7647 | 1.0093 | 1.320 | 0.07 | | 4 | 6144 | 16384 | 1.4989 | 1.9741 | 1.317 | 0.11 | | 4 | 6144 | 32768 | 3.0037 | 3.9102 | 1.302 | 0.05 | | 4 | 6144 | 65536 | 6.1564 | 7.8220 | 1.271 | 0.11 | | 4 | 6144 | 131072 | 13.0916 | 15.5928 | 1.191 | 0.02 | | 4 | 8192 | 4096 | 0.6562 | 0.8133 | 1.239 | 0.12 | | 4 | 8192 | 8192 | 1.2991 | 1.5901 | 1.224 | 0.00 | | 4 | 8192 | 16384 | 2.5754 | 3.1704 | 1.231 | 0.00 | | 4 | 8192 | 32768 | 5.2416 | 6.2735 | 1.197 | 0.35 | | 4 | 8192 | 65536 | 11.1591 | 12.7948 | 1.147 | 0.10 | | 4 | 8192 | 131072 | 22.6522 | 26.1679 | 1.155 | 0.07 | | 2 | 3072 | 4096 | 0.2538 | 0.3503 | 1.380 | 0.19 | | 2 | 3072 | 8192 | 0.4533 | 0.6655 | 1.468 | 0.04 | | 2 | 3072 | 16384 | 0.8781 | 1.2965 | 1.476 | 0.06 | | 2 | 3072 | 32768 | 1.7103 | 2.5706 | 1.503 | 0.00 | | 2 | 3072 | 65536 | 3.4275 | 4.9401 | 1.441 | 0.03 | | 2 | 3072 | 131072 | 7.0964 | 9.8401 | 1.387 | 0.17 | | 2 | 4096 | 4096 | 0.3736 | 0.5182 | 1.387 | 0.19 | | 2 | 4096 | 8192 | 0.7116 | 1.0057 | 1.413 | 0.06 | | 2 | 4096 | 16384 | 1.3972 | 1.9768 | 1.415 | 0.00 | | 2 | 4096 | 32768 | 2.7507 | 3.9208 | 1.425 | 0.02 | | 2 | 4096 | 65536 | 5.5682 | 7.6357 | 1.371 | 0.06 | | 2 | 4096 | 131072 | 11.8074 | 15.2643 | 1.293 | 0.08 | | 2 | 6144 | 4096 | 0.7606 | 0.9606 | 1.263 | 0.06 | | 2 | 6144 | 8192 | 1.4823 | 1.8973 | 1.280 | 0.02 | | 2 | 6144 | 16384 | 2.9507 | 3.7534 | 1.272 | 0.09 | | 2 | 6144 | 32768 | 5.9493 | 7.3609 | 1.237 | 0.13 | | 2 | 6144 | 65536 | 12.6436 | 14.8480 | 1.174 | 0.12 | | 2 | 6144 | 131072 | 26.1080 | 30.5084 | 1.169 | 0.46 | | 2 | 8192 | 4096 | 1.2689 | 1.5342 | 1.209 | 0.09 | | 2 | 8192 | 8192 | 2.5373 | 3.0496 | 1.202 | 0.12 | | 2 | 8192 | 16384 | 5.1135 | 6.0219 | 1.178 | 0.22 | | 2 | 8192 | 32768 | 10.6399 | 12.2423 | 1.151 | 0.22 | | 2 | 8192 | 65536 | 21.6914 | 25.0241 | 1.154 | 0.07 | | 2 | 8192 | 131072 | — | — | not measured (device memory) | — | </details> ### Export-protocol validation (per-row receipts, three architectures) Independently of the table above, every row was re-measured by the export protocol that produced the generated sources in this PR: same CUPTI method (cold L2, per-rank maximum then median, counterbalanced A/B/B/A groups), the generated programs loaded through this PR's JIT path, the four per-operator baselines measured in the same process, plus the validity gates per row (source/export agreement >= 0.95, directional disagreement <= 0.12, endpoint drift <= 0.15, loaded SM clock >= 50 % of the maximum boost clock). A row counts only when all gates pass. **Protocol change (2026-10-11): loaded-SM-clock floor 0.60 -> 0.50.** The floor used to be 0.60. At the largest `W = 2 / 4, S_global = 131072` rows the B200 (1000 W) and B300 (1100 W) board power limits hold the loaded SM clock at 0.48 - 0.60 of the maximum on every node tried (five nodes per SKU, three rounds), so those rows were rejected unconditionally. The floor is now 0.50; the directional, drift and source gates are unchanged. Every previously accepted row was measured under the stricter 0.60 floor and remains valid as measured; under the relaxed floor one further row was accepted (`corr_w2_max` on B300, a correctness row). The rows that still fail are listed below. | GPU | perf rows with a valid receipt | all valid rows faster than the fastest baseline | geometric mean (W = 8 / 4 / 2) | minimum | rows without a valid receipt | |---|---|---|---|---|---| | B200 (SM100) | 69 / 72 | yes | 1.686 (1.535 / 1.716 / 1.832) | 1.170 | 3 | | B300 (SM103) | 69 / 72 | yes | 1.700 (1.511 / 1.737 / 1.889) | 1.109 | 3 | | H100 (SM90) | 70 / 72 | yes | 1.306 (1.296 / 1.319 / 1.301) | 1.093 | 2 | Correctness / negative rows of the export protocol (13 per GPU, including the largest-shape correctness row `corr_w2_max`): H100 13 / 13, B300 13 / 13, B200 12 / 13 (`corr_w2_max` reads 0.496 and 0.492 of the maximum clock on two nodes, below the 0.50 floor; its correctness is covered by the harness correctness table above). Rows without a valid receipt (disclosed, not counted; the raw ratio of an invalid attempt is not a measurement): | GPU | W | hidden | S_global | why the receipt is invalid | attempts (nodes) | raw ratio of the last attempt | |---|---|---|---|---|---|---| | B200 | 2 | 8192 | 131072 | rounds 1 - 2: loaded SM clock 847 / 1170 / 1147 MHz of 1965 under the 1000 W limit; round 3 at the 0.50 floor: B1 endpoint drift 0.30 / 0.22 and B2 directional disagreement > 0.12 | 4 (3 nodes) | 1.37 - 1.69 | | B200 | 4 | 8192 | 131072 | rounds 1 - 2: loaded SM clock below the 60 % floor (960 / 1177 MHz); round 3: B1 directional disagreement > 0.12 | 3 (3 nodes) | 1.55 - 1.89 | | B200 | 2 | 6144 | 131072 | directional disagreement of the B1 / B3 baselines > 0.12 (candidate gates pass) | 2 (2 nodes) | 1.54 - 1.90 | | B300 | 2 | 8192 | 131072 | rounds 1 - 2: loaded SM clock 1117 / 1140 / 1170 MHz of 2032 under the 1100 W limit; round 3: B2 / B3 directional disagreement > 0.12 | 4 (3 nodes) | 1.44 - 1.84 | | B300 | 4 | 8192 | 131072 | rounds 1 - 2: loaded SM clock 1147 / 1215 MHz of 2032; round 3: B1 directional disagreement, B2 directional disagreement + endpoint drift | 4 (3 nodes) | 1.47 - 1.90 | | B300 | 2 | 6144 | 131072 | directional disagreement of the B2 / B3 baselines > 0.12 | 3 (2 nodes) | 1.70 - 2.16 | | H100 | 8 | 8192 | 131072 | directional disagreement of the B4 (NCCL) baseline 0.15 - 0.18 > 0.12 in all five attempts (candidate and B1 - B3 gates pass); the raw B2 ratio of the attempts that reached the arms is 0.97 - 0.99 | 5 (5 nodes) | 0.97 - 1.15 | | H100 | 2 | 8192 | 131072 | does not fit in 80 GB with the reference and the four baseline arms (not measurable) | - | - | The largest shapes run power-capped on the B200 / B300 nodes available for this work (clocks at 48 - 60 % of the maximum boost); once the clock floor no longer rejects them, the order effect between the baseline arms (each tens of milliseconds, with power-state transitions between arms) exceeds the 0.12 directional gate on every node tried. The per-operator table above measured those rows on other nodes of the same type, where the row `W = 2, hidden 8192, S_global 131072` reached 23 ms vs 28.9 ms. The export protocol is not relaxed further for them; they are listed here rather than reported as measured. ### Baselines and their source PRs | Baseline | Path | Source PRs | |---|---|---| | B1 ordinary | cuBLAS projection → `flashinfer.norm.rmsnorm` → `apply_rope_with_cos_sin_cache` → `UlyssesCommunicator.scatter_heads` × 3 | #3820, #4240 (communicator), #4480, #4202 (rmsnorm), #5952, #5956, #6188 (RoPE) | | B2 whole-QKV head chunk | one `scatter_qkv_head_chunk` collective over the whole head band | #5027 | | B3 head-chunk schedule | `scatter_qkv_head_chunk` per head band on three streams, the band schedule of `benchmarks/comm/bench_ulysses_head_chunk.py` (default schedule per row) | #5027, #5263, #5953 | | B4 NCCL all-to-all | `parallel_wrapper.ulysses_a2a_in(fuse_qkv=True)` | #2630 | Baseline implementation files (all at the merge-base): `flashinfer/comm/ulysses.py` (B1 `scatter_heads`, B2 and B3 `scatter_qkv_head_chunk`, and `UlyssesCommunicator`), `benchmarks/comm/bench_ulysses_head_chunk.py` (B3 band schedule), `flashinfer/norm/` (B1 `rmsnorm`), `flashinfer/rope.py` (B1 `apply_rope_with_cos_sin_cache`), `flashinfer/parallel_attention/parallel_wrapper.py` (B4 `ulysses_a2a_in`). This branch is based on `34bb66e4` (its merge-base with `main`); `git log 34bb66e..main -- <those files>` is empty as of `main` = `65c38d2a`, so the baselines measured here are the implementations currently in `main`. Source PR merge commits: #3820 `8a94642d8`, #4240 `668a1ba1c`, #4480 `6d5ef77f8`, #4202 `519205914`, #5952 `a973cfc92`, #5956 `335c4eb4e`, #6188 `d71cc4b38`, #5027 `15e83b7bb`, #5263 `8c54f3446`, #5953 `7e73a7470`, #2630 `ce02358d2`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a prepared sequence-parallel API combining QKV projection, Q/K normalization, rotary embeddings, and head exchange. * Automatically selects the optimized NVLink backend when supported, with an NCCL fallback for other configurations. * Supports CUDA graph capture on the optimized backend and reusable output slots. * **Documentation** * Added API guidance on supported configurations, backend selection, outputs, and capture behavior. * **Benchmarks** * Added a distributed benchmark comparing the fused operation with separate operator paths, including output checks and optional JSON results. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Yingyi Huang <averyh@nvidia.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
triple-mu
added a commit
to triple-mu/flashinfer
that referenced
this pull request
Oct 15, 2026
flashinfer-ai#5953 moved examples/pytorch/ulysses_a2a_export/interface.py to tests/comm/cake_ulysses_a2a_fixture.py; its UlyssesCommunicator still passed max_elems, which this branch renamed to max_bytes.
triple-mu
added a commit
to triple-mu/flashinfer
that referenced
this pull request
Oct 15, 2026
flashinfer-ai#5953 moved examples/pytorch/ulysses_a2a_export/interface.py to tests/comm/cake_ulysses_a2a_fixture.py; its UlyssesCommunicator still passed max_elems, which this branch renamed to max_bytes.
triple-mu
added a commit
to triple-mu/flashinfer
that referenced
this pull request
Oct 16, 2026
flashinfer-ai#5953 moved examples/pytorch/ulysses_a2a_export/interface.py to tests/comm/cake_ulysses_a2a_fixture.py; its UlyssesCommunicator still passed max_elems, which this branch renamed to max_bytes.
triple-mu
added a commit
to triple-mu/flashinfer
that referenced
this pull request
Oct 21, 2026
flashinfer-ai#5953 moved examples/pytorch/ulysses_a2a_export/interface.py to tests/comm/cake_ulysses_a2a_fixture.py; its UlyssesCommunicator still passed max_elems, which this branch renamed to max_bytes.
triple-mu
added a commit
to triple-mu/flashinfer
that referenced
this pull request
Oct 21, 2026
flashinfer-ai#5953 moved examples/pytorch/ulysses_a2a_export/interface.py to tests/comm/cake_ulysses_a2a_fixture.py; its UlyssesCommunicator still passed max_elems, which this branch renamed to max_bytes.
triple-mu
added a commit
to triple-mu/flashinfer
that referenced
this pull request
Oct 23, 2026
flashinfer-ai#5953 moved examples/pytorch/ulysses_a2a_export/interface.py to tests/comm/cake_ulysses_a2a_fixture.py; its UlyssesCommunicator still passed max_elems, which this branch renamed to max_bytes.
triple-mu
added a commit
that referenced
this pull request
Oct 24, 2026
<!-- .github/pull_request_template.md --> ## 📌 Description Adds `UlyssesCommunicator(backend="pcie")`: an experimental single-node all-to-all transport for PCIe-connected GPU groups without NVLink (e.g. 8x RTX PRO 5000), targeting Ulysses sequence parallelism in video/image diffusion serving. Being experimental, it is reachable from `backend="auto"` only where NVLink is unavailable and `FLASHINFER_ALLOW_EXPERIMENTAL_AUTO_BACKENDS=1` is set identically on every rank; naming it explicitly never needs that variable. Either way it warns once. **Why.** The existing NVLink backend lets SM threads store into peer-visible staging buffers; across a PCIe host bridge that degenerates into small transactions (measured 5-13x slower than copy engines during bring-up, see `docs/design_docs/ulysses_pcie.md`). NCCL works but leaves bandwidth and stability on the table on this fabric (see numbers below). **What.** - Routes: world size 1 is an identity path; 2 uses all-pairs CUDA P2P copy engines (`cudaMemcpy2DAsync` writes land directly in the peer's final layout, no pack/unpack); 4/8 prefer an all-RDMA route (per-rank mlx5 RC QP, interleaved-UMR MKeys, GPUDirect RDMA) and fall back to all-P2P with a `RuntimeWarning`; an eight-rank 4+4 NUMA hybrid (same-NUMA P2P, cross-NUMA mlx5) is available via `FLASHINFER_ULYSSES_PCIE_ROUTE`. - Transforms: `scatter_heads` / `gather_heads` interleave the head axis; `exchange_chunks` takes `[1, 1, world_size, chunk]` to the same shape (`all_to_all_single` over a payload a preceding pack already laid out destination-major) and interleaves nothing. On the RDMA routes it configures no MKey, which is what frees it from the 16-bit `bytes_count` that caps the head-axis transforms at 65535 B per peer. - Synchronization: full-group epoch barriers in device memory (release/acquire, sticky abort half for the RDMA routes); the all-P2P route is CUDA-graph capturable, the RDMA routes refuse capture and bound NIC stalls with a 10 s deadline plus a group-wide abort/quiesce protocol. - Lifecycle: explicit `allocate_output` registration (collective, lives until `close()`); `input_buffer()` exposes the RDMA landing buffer so producers can write operands where the NIC reads them, removing the largest device-to-device copy on the path (measured ~6% per call at layer scale). The backend has no staging workspaces: `workspace=`, `create_workspace` and the head-chunk primitives from #5027 are refused at world size above one with a message that names `allocate_output()`. - Failure envelope: a rejection before native code runs (bad operand, refused capture, second stream) is decided from the call's own arguments, so under the SPMD contract every rank rejects it -- the RDMA routes raise the ordinary error and stay usable, the all-P2P route (no abort protocol) poisons teardown and enters BROKEN. A failure inside the transport is fail-stop on every route: on the RDMA routes the failing rank publishes the sticky abort, the peers leave the barrier and `close()` completes; on all-P2P `close()` refuses to synchronize and the process must exit. The design note has the table. - The element type is per-call (`dtype=` on the collectives), decoupled from the communicator: the transport moves opaque bytes, so FP8/INT8 payloads ride a BF16-sized communicator within the same byte budget. The constructor's capacity is therefore `max_bytes`, replacing main's element-count `max_elems`: the other communication workspaces in `flashinfer.comm` are sized in bytes as well (`buffer_size_in_bytes`, `workspace_size`), and an element count has no single meaning once the element type varies per call. #4924 already targets `max_bytes`. The staging-buffer API from #5027 (`UlyssesWorkspace(max_elems=...)`, `create_workspace(max_elems=...)`) keeps its element count, since those buffers hold one dtype. - Docs: design note (`docs/design_docs/ulysses_pcie.md`), `docs/api/comm.rst`, env-var reference; fi_trace templates for all three transforms (`scatter_heads` / `gather_heads` gain `trace=` decorators). - Housekeeping: six `flashinfer.comm` re-exports of JIT module helpers with no users in the repository (`gen_trtllm_comm_module`, `gen_vllm_comm_module`, `gen_pcie_ipc_comm_module`, `get_pcie_ipc_comm_module`, `gen_ulysses_a2a_module`, `get_ulysses_a2a_module`) are dropped; the helpers stay importable from their own modules and from `flashinfer.jit`. **Changes since the approved revision (a481f1d).** The branch was rebased onto main (#5027 workspace API, #5263 generated NVLink kernels) and the following landed on top, in this order: 1. `fix(comm): finish the max_bytes rename in the head-chunk tests, benchmark and export example` -- four `max_elems=` constructor calls that arrived with #5027 / #5263 (`_destination_passing_body`, `_head_chunk_body`, `bench_ulysses_head_chunk.py`, `examples/pytorch/ulysses_a2a_export/interface.py`) raised `TypeError` on any 2+ GPU host. 2. `format: format code style by pre-commit` -- the two hooks CI flagged. 3. `test(comm): share the Ulysses test helpers and gloo fixture` -- one FileStore `gloo_pg` fixture in `tests/comm/conftest.py`, `full_mesh` / `forbid_ipc_and_jit` / rendezvous helpers in `tests/test_helpers/ulysses.py`. 4. `fix(comm): check the group's CUDA all-to-all support before arming NVLink` -- the check ran only for `backend="nccl"`, after the NVLink IPC exports were armed. 5. `fix(comm): reject cross-rank FLASHINFER_ULYSSES_PCIE_NICS disagreement` -- the NIC override is now compared across ranks like the route and GID overrides. 6. `feat(comm): warn at garbage collection when a Ulysses communicator was not closed` -- `ResourceWarning` from `__del__` for an armed communicator; no teardown from the finalizer. 7. `fix(comm): refuse workspaces and the head-chunk primitives on the multi-rank PCIe backend` -- see Lifecycle above. 8. `fix(comm): publish the abort when the RDMA routes reject an operand natively` -- the RDMA routes' per-call operand checks run inside the native abort envelope, so a rank whose operand native rejects releases its peers instead of leaving them in the opening barrier (the only native change: the checks moved into `ValidateExchangeOperands`, protocol and wire format untouched); the caller's stream is bound only after a call passes validation; the Python-side envelope is unchanged and documented. 9. `refactor(comm): route Ulysses output validation through the shared helpers` -- `_output_geometry` is the single source of the output shape; the overlap test reuses `_storage_ranges_overlap`. 10. `docs(comm): update Ulysses PCIe prose for auto selection, the failure envelope and follow-ups`. 11. Rebased onto main 28ae778 (2026-09-24). The only conflict was `flashinfer/trace/templates/comm.py`, where #4929 and this PR both append templates at the end of the file; both blocks are kept, in main's order, and no other file changed in the rebase. 12. Merged main `7bcf91831` (2026-09-29): three additive conflicts (`flashinfer/aot.py`, `flashinfer/comm/__init__.py`, `tests/trace/template_registry.py`), both sides kept. Followed by `Find Conda rdma-core build inputs for PCIe Ulysses JIT` (72c958a): `gen_ulysses_pcie_module` adds `<sys.prefix>/include` and `-L<sys.prefix>/lib` when `infiniband/mlx5dv.h` / `libmlx5.so` live there (the CI Conda environment), and `missing_ulysses_pcie_dependencies()` now also probes for the `infiniband/verbs.h` / `infiniband/mlx5dv.h` headers (one host-mock test added; design note updated). 13. Merged main `7aa42e2dc` (2026-10-04). Two additive conflicts: `scripts/task_test_multi_gpu_comm_kernels.sh` (both sides extend `SPAWN_MANAGED_TEST_FILES`; the union is kept) and `tests/trace/example.py` (main sorted the `flashinfer.*` imports; `import flashinfer.comm` takes its place in that order). `CLAUDE.md`, `docs/api/comm.rst`, `flashinfer/aot.py` and `tests/trace/test_fi_trace_template_consistency.py` merged automatically. No file under `flashinfer/comm/ulysses*`, `flashinfer/jit/comm.py`, `csrc/ulysses*`, `include/flashinfer/comm/ulysses_pcie.cuh` or `tests/comm/test_ulysses*` changed on main between the two bases. 14. Merged main `188bdd769` (2026-10-05). Two modify/delete conflicts: #5953 removed `examples/pytorch/ulysses_a2a_export/` and moved its fixture to `tests/comm/cake_ulysses_a2a_fixture.py`, so the deletion is taken and the `max_bytes` rename from item 1 is re-applied to the moved fixture in `fix(comm): finish the max_bytes rename in the moved cake Ulysses fixture` (57ed509); `docs/api/comm.rst` and `flashinfer/aot.py` merged automatically. No file under `flashinfer/comm/ulysses*`, `flashinfer/jit/comm.py`, `csrc/ulysses*`, `include/flashinfer/comm/ulysses_pcie.cuh` or `tests/comm/test_ulysses*` changed on main between the two bases. 15. Rebased onto main `9c6797eda` (2026-10-05) at the maintainers' convenience for a linear history: the 28 non-merge commits above replay in order and the five merge commits (1f82831, 25bb39c, f7ce2ba, 5e8919a and their resolutions) are gone; `8668d921` is the new head. The replay re-resolved the same conflicts (`flashinfer/comm/__init__.py`, `flashinfer/aot.py`, `tests/trace/example.py`, `tests/trace/template_registry.py`, `scripts/task_test_multi_gpu_comm_kernels.sh`, the removed example files), and the result differs from the merged head `57ed5097` in exactly one place: `flashinfer/comm/__init__.py` no longer re-exports `gen_pcie_ipc_comm_module` / `get_pcie_ipc_comm_module`, which the merge in item 12 had brought back by accident; this is the drop described under Housekeeping, and the repository has no caller of either name through `flashinfer.comm`. 16. Rebased onto main `67c190f7d` (2026-10-06); `0f60c090` is the new head, still 28 linear commits. One conflict: the expected-output list in `tests/trace/example.py`'s docstring, where #6035 appended the `moe_layer_cudnn_frost_*` entries at the spot this PR appends the `ulysses_*` entries; both lists are kept, in main's order. `CLAUDE.md`, `flashinfer/aot.py` and `tests/trace/template_registry.py` merged automatically. No file under `flashinfer/comm/ulysses*`, `flashinfer/jit/comm.py`, `csrc/ulysses*`, `include/flashinfer/comm/ulysses_pcie.cuh` or `tests/comm/test_ulysses*` changed on main between the two bases. 17. Rebased onto main `87ef73934` (2026-10-14); `e7c7bbd50` is the new head, 29 linear commits. One conflict: the tail of `tests/trace/example.py`, where #5304 appended its `top_k_varlen` example calls at the spot this PR appends the single-rank Ulysses block; both are kept, in main's order (the docstring's expected-output list merged automatically this time). `CLAUDE.md`, `docs/api/comm.rst`, `flashinfer/aot.py`, `flashinfer/comm/__init__.py`, `scripts/task_test_multi_gpu_comm_kernels.sh` and `tests/trace/template_registry.py` merged automatically. #6332 (`flashinfer/comm/qkv_norm_rope_exchange_overlap.py` and `benchmarks/comm/bench_qkv_norm_rope_exchange_overlap.py`) arrived on main with three `UlyssesCommunicator(..., max_elems=...)` calls, so the 29th commit, `fix(comm): finish the max_bytes rename in the QKV norm/RoPE exchange overlap`, passes `max_bytes` (the same bf16 element counts times `torch.bfloat16.itemsize`) exactly as item 14 did for the moved fixture; nothing else on main constructs a communicator with the old keyword. No file under `flashinfer/comm/ulysses*`, `flashinfer/jit/comm.py`, `csrc/ulysses*`, `include/flashinfer/comm/ulysses_pcie.cuh`, `tests/comm/test_ulysses*` or `flashinfer/trace/templates/comm.py` changed on main between the two bases. 18. Rebased onto main `3637083a3` (2026-10-15); `f0ff368c1` is the new head, still 29 linear commits, no conflicts: #5701 extends `TORCHRUN_TEST_FILES` in `scripts/task_test_multi_gpu_comm_kernels.sh` next to the `SPAWN_MANAGED_TEST_FILES` line this PR extends, and both land. The base now includes #6509, the fix for the `tests/moe_ep/test_sm107_torch_staging.py` failure that was the only failing lane in this PR's 2026-10-15 CI run (unrelated to this PR, which touches no `moe_ep` file). No file under `flashinfer/comm/ulysses*`, `flashinfer/jit/comm.py`, `csrc/ulysses*`, `include/flashinfer/comm/ulysses_pcie.cuh`, `tests/comm/test_ulysses*` or `flashinfer/trace/templates/comm.py` changed on main between the two bases. **Performance** (8x RTX PRO 5000, dual-NUMA 4+4, PCIe Gen5 x16, one 400G mlx5 per GPU; `benchmarks/comm/bench_ulysses_pcie.py`, seq 37888 / heads 56 / head_dim 128 / bf16; median of host-clock loops, max-reduced across ranks, vs the `all_to_all_single`+permute reference; PCIe-side sd ≤ 8%, mostly < 1%, while the NCCL reference jitters up to 50-104% at eight ranks): | world | route (auto's pick in bold) | scatter | gather | qkv layer | per-rank remote BW | |---|---|---|---|---|---| | 2 | **all-P2P** | 2.02x | 2.00x | 2.01x | 50.5 GB/s | | 4 | **all-RDMA** | 1.97x | 1.97x | 1.97x | 42.3 GB/s | | 8 | **all-RDMA** | 1.49x | 1.43x | 1.48x | 42.6 GB/s | | 8 | 4+4 hybrid | — | — | 1.27x | 36.4 GB/s | | 8 | all-P2P (correctness fallback) | 0.81x | 0.43x | 0.30x | 8.8 GB/s | The transport moves opaque bytes, so the element type should change how long a call takes but not how fast the link runs. Same node, ws8 all-RDMA, seq 37888 / 56 heads / head_dim 128, only the dtype varies: | dtype | bytes/elem | scatter ms | qkv layer ms | remote BW (scatter) | |---|---|---|---|---| | bf16 | 2 | 1.389 | 5.583 | 42.8 GB/s | | fp8_e4m3fn | 1 | 0.695 | 2.871 | 42.7 GB/s | | int8 | 1 | 0.696 | 2.870 | 42.7 GB/s | Halving the element size halves the time (1.389 -> 0.695 ms) and leaves bandwidth put; the two 1-byte types are within noise of each other. ws2 P2P saturates ~95% of the effective Gen5 x16 link; the RDMA route reaches ~85% of the 400G NIC. Re-measured at this head on the same node (ws8 all-RDMA, same shapes, 5 trials x 200 iterations, other tenants idle) to confirm the round changed nothing on the hot path: scatter 1.427 ms at 41.6 GB/s remote bandwidth (table above: 1.389 ms / 42.8 GB/s), gather 1.554 ms at 38.2 GB/s, both with trial-to-trial sd below 2%; the q/k/v layer loop read 6.93 ms with sd 9.2%, which is inside its own noise band on a shared node and is not re-plotted here. The `rnr_nak_retry_err` counters on all eight NICs did not move during the run. **Limitations.** Experimental; single host; world sizes 1/2/4/8 (the copy scheduler's `rank ^ step` order needs powers of two); the RDMA routes require batch 1 and refuse CUDA-graph capture, and bound the head-axis transforms at head-row pitch `H*D*elem <= 65535` B (`exchange_chunks` registers no MKey and is not bounded); the JIT module links libibverbs/libmlx5 even when topology picks all-P2P, so it is JIT-only and deliberately absent from AOT; a transport failure is fail-stop as described above, and an all-P2P failure leaves the process to be terminated by the launcher. Validated on one topology (4+4 NUMA, one mlx5 per GPU); no work-request-level mlx5 unit tests yet — the RDMA data path is covered end to end. No CI lane exercises the PCIe wire path (the CI images carry no rdma-core and the GPU lanes are single-GPU); the host-mock and single-rank halves of the suite run there, the multi-rank half runs on the reference node. Planned follow-ups, prepared on a branch and deliberately kept out of this PR to leave the approved protocol untouched: publish the abort from Python on every route (the all-P2P route already imports every peer's signal buffer), so a Python-side rejection on a rank that broke the SPMD contract releases the peers too; publish a geometry word with the opening epoch so ranks that call with different shapes at the same position break the group instead of writing a misdescribed layout; post the RDMA receive before the opening barrier (closes the window in which a peer's write can arrive before the receive is posted; no RNR retries were counted during this round's benchmark, so this is hygiene rather than a measured stall); build the all-P2P route without rdma-core; `out=None` with lazy registration; per-output `capacity_bytes`. ## 🔍 Related Issues N/A ## 🚀 Pull Request Checklist ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. ## 🧪 Tests - [x] Tests have been added or updated as needed. - [x] All tests are passing (`unittest`, etc.). Ran on the 8-GPU reference node (8x RTX PRO 5000, dual-NUMA 4+4, one mlx5 per GPU, rdma-core 63, CUDA 13.4, torch 2.13) with all eight GPUs at `ef9a91a22`, the head before the rebase (2026-09-20): `tests/comm/test_ulysses_communicator.py` 132 passed / 22 skipped / 0 failed, `tests/comm/test_ulysses_topology.py` 112 passed / 0 skipped (244 passed / 22 skipped in one run), `tests/trace/test_ulysses_trace.py` + `tests/trace/test_template_registry.py` 10 passed. The 22 skips are the forced-NVLink family on this NVLink-less machine; every PCIe case ran (p2p, all-RDMA and 4+4 hybrid routes at world sizes 2/4/8, the chunk exchange, the landing-buffer producer path, the p2p fail-stop pair, and the two-rank RDMA case in which rank 0's native dtype rejection publishes the abort, rank 1 leaves the barrier with "peer 0 aborted the exchange", and both ranks close). Correctness is bit-exact against the `all_gather`-based reference for fp16/bf16/fp32 and byte-exact for int8/uint8/fp8 across all three routes, including non-power-of-two head dims on the UMR path, CUDA-graph capture+replay of a q/k/v/out layer on the P2P route, and `backend="auto"` with the experimental opt-in selecting PCIe on this node. Re-run after the rebase at the pushed head `4fec510aa` (2026-09-24) on the same node, which was shared with another tenant on GPUs 0-3, so the suite ran on GPUs 4-7 (`CUDA_VISIBLE_DEVICES=4,5,6,7`): `tests/comm/test_ulysses_communicator.py` 122 passed / 32 skipped / 0 failed, `tests/comm/test_ulysses_topology.py` 112 passed, `tests/trace/test_ulysses_trace.py` + `tests/trace/test_template_registry.py` 10 passed. The skips are the forced-NVLink family plus the world-size-8 cases (all-RDMA and the 4+4 hybrid at eight ranks), which need eight visible GPUs; every world-size 2/4 case ran (p2p and all-RDMA routes, the chunk exchange, the landing-buffer producer path, the fail-stop pairs). The eight-rank paths stand on the run above: no file under `flashinfer/comm/ulysses*`, `csrc/ulysses*` or `tests/comm/test_ulysses*` changed on main between the two bases, and the rebase touched only the trace template file. Re-run after the 2026-10-04 merge at `25bb39c4` on the same node, first on GPUs 4-7 while GPUs 0-3 were held by another tenant (`tests/comm/test_ulysses_communicator.py` 123 passed / 32 skipped), then on all eight GPUs once they freed up: `tests/comm/test_ulysses_communicator.py` 133 passed / 22 skipped / 0 failed, `tests/comm/test_ulysses_topology.py` 112 passed, `tests/trace/test_ulysses_trace.py` + `tests/trace/test_template_registry.py` 10 passed; `pre-commit run --from-ref origin/main --to-ref HEAD` is green, `scripts/check_pr_document.py` reports no findings and `scripts/check_pr_api_diff.py` only the `max_bytes` advisory. The 22 skips are the forced-NVLink family; every PCIe case ran, including the eight-rank all-RDMA and 4+4 hybrid routes. The head `f7ce2baf` on top merges main `4b6d64be` (#6040, `cake_selective_state_update` only) and touches no Ulysses file. Re-run after the 2026-10-05 merge at the pushed head `57ed5097` on the same node with all eight GPUs: `tests/comm/test_ulysses_communicator.py` 133 passed / 22 skipped / 0 failed, `tests/comm/test_ulysses_topology.py` 112 passed, `tests/trace/test_ulysses_trace.py` + `tests/trace/test_template_registry.py` 10 passed, and `tests/comm/test_cake_ulysses_a2a.py` + `tests/comm/test_ulysses_a2a.py` + `tests/comm/test_ulysses_head_chunk.py` 16 passed / 4 skipped (the generated NVLink portfolio skips on this SM120 node; the moved fixture was additionally exercised by constructing its communicator with `max_bytes` in a one-rank process). The skips are the forced-NVLink family. `pre-commit run --from-ref <main> --to-ref HEAD` is green. Re-run at the rebased head `8668d921` on the same node with all eight GPUs: `tests/comm/test_ulysses_communicator.py` 133 passed / 22 skipped / 0 failed, `tests/comm/test_ulysses_topology.py` 112 passed, `tests/trace/test_ulysses_trace.py` + `tests/trace/test_template_registry.py` 10 passed, `tests/comm/test_cake_ulysses_a2a.py` + `tests/comm/test_ulysses_a2a.py` + `tests/comm/test_ulysses_head_chunk.py` 16 passed / 4 skipped, and the pcie_ipc module tests that import the JIT helpers from `flashinfer.comm.pcie_ipc_ar` still pass; `pre-commit run --from-ref <main> --to-ref HEAD` is green. Re-run at the rebased head `0f60c090` on the same node (all eight GPUs for the communicator suite, GPUs 0-3 for the rest while another tenant held GPU 6): `tests/comm/test_ulysses_communicator.py` 133 passed / 22 skipped / 0 failed, `tests/comm/test_ulysses_topology.py` 112 passed, `tests/trace/test_ulysses_trace.py` + `tests/trace/test_template_registry.py` 10 passed, `tests/comm/test_cake_ulysses_a2a.py` + `tests/comm/test_ulysses_a2a.py` + `tests/comm/test_ulysses_head_chunk.py` 15 passed / 5 skipped, pcie_ipc module tests pass; `pre-commit run --from-ref <main> --to-ref HEAD` is green. On a single-GPU host the same files give 186 passed / 90 skipped (host-mock and single-rank halves), `pre-commit run --all-files` is green, `scripts/check_pr_document.py` reports no findings, and `scripts/check_pr_api_diff.py` reports only the `max_elems` -> `max_bytes` constructor advisory explained above. Re-run at the rebased head `f0ff368c1` (2026-10-15) on the same node with all eight GPUs, from a fresh checkout in a new directory (torch 2.14.1+cu130, nvcc 13.0, cold JIT cache): `tests/comm/test_ulysses_communicator.py` 133 passed / 22 skipped / 0 failed, `tests/comm/test_ulysses_topology.py` 112 passed, `tests/trace/test_ulysses_trace.py` + `tests/trace/test_template_registry.py` 10 passed, `tests/comm/test_cake_ulysses_a2a.py` + `tests/comm/test_ulysses_a2a.py` + `tests/comm/test_ulysses_head_chunk.py` 16 passed / 4 skipped, and the six `tests/comm/test_pcie_ipc_*.py` files that import `flashinfer.comm.pcie_ipc_ar` 73 passed / 12 skipped / 1 xfailed. The 22 skips are the forced-NVLink family; every PCIe case ran, including the eight-rank all-RDMA and 4+4 hybrid routes. On the single-GPU host the four core files give 187 passed / 90 skipped / 0 failed, `pre-commit run --all-files` is green, `scripts/check_pr_document.py` reports no findings and `scripts/check_pr_api_diff.py` only the `max_bytes` advisory. `git range-diff` against the previous head `313ac10b8` shows the 29 commits replay with context-only changes. ## Reviewer Notes - The riskiest pieces are the epoch-barrier protocol (`include/flashinfer/comm/ulysses_pcie.cuh`) and the mlx5 transport (`csrc/ulysses_pcie_transport.cuh`): QP bring-up, interleaved-MKey geometry (pitch ≤ 65535), completion accounting with per-exchange immediate tags, and the abort/quiesce teardown ledger. - `_validate`'s byte-budget capacity math after the per-call dtype change (`flashinfer/comm/ulysses.py`) is worth a close read; the rescale rationale is in the commit message of the dtype commit. - The failure envelope is route-dependent on purpose, and the design note's "Failure envelope" table is the contract to review against. Python-side rejections are rank-uniform under the SPMD contract, so the RDMA routes keep the communicator usable (the multi-rank tests rely on refusing capture and a batch > 1 operand recoverably); the all-P2P route poisons teardown because its barrier has no abort protocol at all. Native failures on the RDMA routes publish the abort (now including the per-call operand checks) and `close()` completes. Publishing the abort from Python on every route is the first follow-up above. - Things that are correct but will draw a question: (1) full-group CUDA IPC is required even on the RDMA routes, because the barrier signals are always IPC-mapped -- RDMA removes the P2P *data* requirement, not the P2P requirement; (2) `batch == 1` on the RDMA routes comes from mode 0's flat `rank * payload` remote offset, not from the NIC; (3) `input_ready` is deliberately reused between `RunHybridBarrier` and `EnqueueCopies` so the side streams inherit the post-barrier state; (4) the copy-engine writes are ordered against the closing barrier's release store through the `copy_done` event join, i.e. through `cudaMemcpyAsync` completion semantics; (5) the per-call `dtype` is really per-`allocate_output`: the registered buffer pins it and native re-checks every call against it. - Placement: the backend uses the experimental machinery (warn-once, gated `auto`, no AOT) but lives in `flashinfer/comm/` with tests in `tests/comm/`, because it extends the stable `UlyssesCommunicator` class and the routing shares its decision layer; there is no precedent for an `@experimental_backend` in core. If the maintainers prefer the Experimental Track layout, the code splits at the `_pcie_*` seam (the methods touch only `rank`, `world_size`, `device`, `dtype`, `max_bytes`, `backend`, `transport`, `decision`, `_state` and the `_gather` / `_require_open` / `_validate` / `_output_geometry` helpers) and we will move it. --------- Co-authored-by: Alex Yang <aleyang@nvidia.com>
11 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #5807.
csrc/generated/ulysses/sm_100a/shipped 60 files (17.7K lines) for the NVLink Ulysses all-to-all: 24 kernels plus 36*_binding.cufiles (12.7K lines, 72 %) that nothing calls. The dispatcher launches the generated kernels directly throughcudaLaunchKernel, so the TVM-FFI entry points, theirScopedCudaDevicehelpers and the per-binding device-attribute cacheswere dead weight compiled into every build. The loader selected the generated module only when the build-target list was
exactly
10.0aor exactly10.3a(multi-arch builds silently fell back to the portable kernel), and the test imported itsfixture from
examples/.csrc/generated/ulysses/(24 files, same kernel text asthe sm_100a copies they replace; the sm_103a build compiles the same sources). The
sm_100a/directory and the 36 bindingfiles are removed.
flashinfer/jit/cake_ulysses.py: the generated module is built whenever the target list contains a major-10 target (one-gencodeper 10.x target; module nameulysses_a2a_sm100a,ulysses_a2a_sm103aorulysses_a2a_sm100a_sm103a) and isselected when the current device's capability is among the built targets; otherwise
jit/comm.pykeeps the portable kernelas before.
-DFLASHINFER_ULYSSES_GENERATED=1and the dispatcher header are unchanged.csrc/cake_ulysses_dispatch.cu: oneextern "C"declaration per kernel and a(dtype, world, mode) -> (kernel, smem)tablewith the same
LaunchGeneratedUlyssessignature; the selected entries are identical to the shipped mapping.tests/comm/cake_ulysses_a2a_fixture.py(moved fromexamples/pytorch/ulysses_a2a_export/): the 40-row portfolio and theper-rank preparation;
tests/comm/test_cake_ulysses_a2a.pyruns it for world sizes 2/4/6/8 (skipping worlds the node cannothost) and adds CPU cases for the build-target selection;
benchmarks/comm/bench_cake_ulysses_a2a.pyimports the movedfixture.
examples/pytorch/ulysses_a2a_export/is removed.flashinfer/comm/ulysses.py,flashinfer/jit/comm.py,csrc/ulysses_all_to_all.cuand the public API are untouched.Net: -12,650 lines (1,067 added / 13,717 removed) across the owned paths.
Known duplication kept in this PR (recorded on the issue): the world size is fixed per generated kernel, so the fp16 and
fp32 scatter/gather kernels for world sizes 4 and 8 differ only in world-size literals (4 pairs, ~212 lines each); folding
them needs the world size as a runtime kernel argument in the generator. Registering the module in
flashinfer/aot.pyisalso a follow-up.
Validation
cuobjdump -sass, symbol and addressneutralised): sm_100a equivalent (24 base / 24 candidate kernels, 22 distinct bodies), sm_103a equivalent (24 / 24, 22 distinct
bodies), proved on both an 8x B200 and an 8x B300 host. The 24 arch-neutral kernel units are the same machine code as the
per-architecture files they replace; the removed bindings carried no device code.
directions, world sizes 2/4/6/8 on intra-node NVLink):
device allocations in the delivered call; all 8 performance rows pass every statistical gate with source/export GPU-time ratios
W2 0.9996 / 0.9995 (371 us), W4 0.9982 / 0.9980 (196 us), W6 0.9933 / 0.9972 (184 us), W8 0.9815 / 0.9937 (108 us)
(ABBA-interleaved CUDA-graph replays, 3 counterbalanced groups x 12000 samples per arm, SM clock gate passed).
W6 0.9974 / 0.9998 (182 us), W8 0.9962 / 0.9910 (108 us), all gates passed.
launch skew between rank processes lands inside the early rank's kernel and the per-rank-maximum aggregation reports that rank,
so their source/export ratio is recorded (0.91-1.16, value above the 0.97 floor on every non-passing row) but not used as an
acceptance gate. 16 (B200) / 18 (B300) of them, all in world sizes 4 and 6, fail only the endpoint-drift statistic: 55-65 us
skew batches hit both arms identically (per-window source/export medians within a few percent, identical pure-kernel floor on
every rank in both arms), so they are recorded as measurement-invalid for that statistic with bitwise outputs intact. In the
few rows where one arm's median sits above the other's, the per-rank receipts show the same mechanism on a single rank: that
rank launches ahead of its peers every sample and absorbs the wait in its kernel, while the other ranks run the new kernel
as fast as or faster than the old one, with identical minimum kernel times on every rank. World sizes 2 and 8 pass 18/18 on
both architectures. Harness follow-up (not this PR): synchronise the rank set per launch batch, or report the last
launcher's span, for multi-rank synchronising kernels.
pytest tests/comm/test_cake_ulysses_a2a.py: 8 passed on 8x B200 and 8 passed on 8x B300 (portfolio worlds 2/4/6/8 bitwise vsthe all-gather reference through the public communicator, plus the build-target selection cases). Note for B300 nodes with NCCL
2.30.x where the first collective hangs in NVLink-SHARP channel setup: the tests run with
NCCL_NVLS_ENABLE=0(NCCL is not onthe kernel path; it only exchanges the IPC handles).
🤖 Generated with Claude Code
Summary by CodeRabbit
References
The shared reference corpus for CAKE kernel development includes the following projects, documentation, and existing CAKE work:
These are corpus-level references. PR-specific implementation details, changes, and benchmark references are documented above.