fix(moe_ep): let the host framework pin the CUDA device via Bootstrap by mhoqueanik · Pull Request #4348 · flashinfer-ai/flashinfer

mhoqueanik · 2026-08-04T22:11:38Z

The moe_ep runtime derives its CUDA device from the environment (LOCAL_RANK, falling back to bootstrap.rank) and rebinds unconditionally in _ensure_cuda_device, _ensure_torch_dist, and _init_nvshmem_after_dist. Host frameworks like vLLM bind each worker's device themselves, often under a remapped CUDA_VISIBLE_DEVICES where the derived ordinal points at the wrong or a nonexistent GPU; the rebind then launches weight transforms against another device's pointers
(CUDA_ERROR_ILLEGAL_ADDRESS in deep_gemm transform_sf during load).

Add BootstrapConfig.device: when set, all three bind sites use it verbatim; when None, behavior is unchanged (LOCAL_RANK then rank, torchrun convention). Hosts pass torch.cuda.current_device() and no longer need to mutate LOCAL_RANK to steer the library.

Summary by CodeRabbit

New Features
- Added support for explicitly selecting the CUDA device used during bootstrap.
- Device selection now follows a predictable priority: configured device, local environment setting, then process rank.
- Invalid negative device values are rejected with a clear validation error.
Bug Fixes
- Improved consistency between CUDA, NCCL, and NVSHMEM device selection during initialization.
Tests
- Added coverage for device-selection precedence and invalid device configuration.

coderabbitai · 2026-08-04T22:12:13Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 5713edd2-110a-47c9-9e35-98d8bdc4a104

📥 Commits

Reviewing files that changed from the base of the PR and between 968fa04 and e4d7c1b.

📒 Files selected for processing (3)

flashinfer/moe_ep/config.py
flashinfer/moe_ep/core/runtime/bootstrap.py
tests/moe_ep/test_constraints.py

🚧 Files skipped from review as they are similar to previous changes (3)

tests/moe_ep/test_constraints.py
flashinfer/moe_ep/core/runtime/bootstrap.py
flashinfer/moe_ep/config.py

📝 Walkthrough

Walkthrough

BootstrapConfig now supports an optional CUDA device ordinal. Runtime initialization resolves the device from explicit configuration, LOCAL_RANK, or rank, and uses it for CUDA, NCCL, and NVSHMEM setup. Tests cover precedence, fallback, and validation.

Changes

Bootstrap device resolution

Layer / File(s)	Summary
Bootstrap device contract `flashinfer/moe_ep/config.py`	`BootstrapConfig` adds an optional `device` field and raises `ValueError` for negative values.
Runtime device resolution and validation `flashinfer/moe_ep/core/runtime/bootstrap.py`, `tests/moe_ep/test_constraints.py`	`_resolve_local_device` prioritizes the explicit device, then `LOCAL_RANK`, then rank. CUDA, NCCL, and NVSHMEM initialization use the resolved device. Tests cover precedence, fallback, and validation.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested reviewers: aleozlx, anerudhan, aneureka

Sequence Diagram(s)

sequenceDiagram
  participant BootstrapConfig
  participant _resolve_local_device
  participant CUDA
  participant NCCL
  participant NVSHMEM
  BootstrapConfig->>_resolve_local_device: provide explicit device, rank, and environment
  _resolve_local_device->>CUDA: select resolved device
  _resolve_local_device->>NCCL: select resolved device
  _resolve_local_device->>NVSHMEM: select resolved device

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description explains the problem and solution but omits the required template sections for related issues, checklist, tests, and reviewer notes.	Use the repository template and complete the related issues, pre-commit checklist, test status, and reviewer notes sections.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: allowing host frameworks to pin the CUDA device through BootstrapConfig.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@flashinfer/moe_ep/config.py`:
- Line 97: Update BootstrapConfig so the new device field cannot alter existing
positional argument binding: make device keyword-only (prefer the dataclass
kw_only mechanism) or move it after all existing fields, while preserving
nccl_comm’s current positional position.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 53460d0a-cadf-4c50-8cbe-810569c867c1

📥 Commits

Reviewing files that changed from the base of the PR and between 968fa04 and 1503f8918b49f9d294e3f26543d267c442a24fc9.

📒 Files selected for processing (3)

flashinfer/moe_ep/config.py
flashinfer/moe_ep/core/runtime/bootstrap.py
tests/moe_ep/test_constraints.py

coderabbitai · 2026-08-04T22:14:35Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

…Config The moe_ep runtime derives its CUDA device from the environment (LOCAL_RANK, falling back to bootstrap.rank) and rebinds unconditionally in _ensure_cuda_device, _ensure_torch_dist, and _init_nvshmem_after_dist. Host frameworks like vLLM bind each worker's device themselves, often under a remapped CUDA_VISIBLE_DEVICES where the derived ordinal points at the wrong or a nonexistent GPU; the rebind then launches weight transforms against another device's pointers (CUDA_ERROR_ILLEGAL_ADDRESS in deep_gemm transform_sf during load). Add BootstrapConfig.device: when set, all three bind sites use it verbatim; when None, behavior is unchanged (LOCAL_RANK then rank, torchrun convention). Hosts pass torch.cuda.current_device() and no longer need to mutate LOCAL_RANK to steer the library. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mohammad Hoque <mhoqueanik@nvidia.com>

coderabbitai · 2026-08-04T22:17:43Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

…#4348) The moe_ep runtime derives its CUDA device from the environment (LOCAL_RANK, falling back to bootstrap.rank) and rebinds unconditionally in _ensure_cuda_device, _ensure_torch_dist, and _init_nvshmem_after_dist. Host frameworks like vLLM bind each worker's device themselves, often under a remapped CUDA_VISIBLE_DEVICES where the derived ordinal points at the wrong or a nonexistent GPU; the rebind then launches weight transforms against another device's pointers (CUDA_ERROR_ILLEGAL_ADDRESS in deep_gemm transform_sf during load). Add BootstrapConfig.device: when set, all three bind sites use it verbatim; when None, behavior is unchanged (LOCAL_RANK then rank, torchrun convention). Hosts pass torch.cuda.current_device() and no longer need to mutate LOCAL_RANK to steer the library.  ## Summary by CodeRabbit - **New Features** - Added support for explicitly selecting the CUDA device used during bootstrap. - Device selection now follows a predictable priority: configured device, local environment setting, then process rank. - Invalid negative device values are rejected with a clear validation error. - **Bug Fixes** - Improved consistency between CUDA, NCCL, and NVSHMEM device selection during initialization. - **Tests** - Added coverage for device-selection precedence and invalid device configuration.  Signed-off-by: Mohammad Hoque <mhoqueanik@nvidia.com> Co-authored-by: Md Saidul Hoque Anik <mhoqueanik@login-preos01.a51.clusters.nvidia.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> (cherry picked from commit 3c57ef1)

Aligns the restructure branch with flashinfer TOT (95 commits, incl. the v0.6.17 release line). moe_ep conflict resolution: - All taxonomy-vs-flat naming/import conflicts (backends, tests, docs, configs, run_tests.sh, pre-commit/pyproject excludes) resolve to the restructure spellings; upstream's non-conflicting deltas auto-merge. - kernel_src/sm100/cutedsl_megamoe (old path, re-added by upstream's rename detection) is dissolved into kernel_src/cutedsl_megamoe; the one real upstream advance there — the 4fbac49 singleton-expert TMA-modes fix (PR flashinfer-ai#4296: shim/nvfp4.py + src/moe_nvfp4_swapab/kernel_fc12.py) — is ported onto the renamed paths and recorded in VENDOR.md. - kernel_src/sm90/pull_style_cutedsl_megakernel add/add conflicts take our tree verbatim (upstream b1d9585 is an older squash of the same drop; no upstream advances since). - tune.py keeps the restructure layout (upstream's _restage_skewed_routing hunk is our own pre-restructure code, now in backends/mega/kernel/tuning.py). - De-duplicated the mxfp8 multirank torch-oracle block the auto-merge re-added from upstream's copy (ours carries the try/finally symm-buffer cleanup from 640b75f). Notable upstream picks now in-tree: BootstrapConfig.device (3c57ef1, PR flashinfer-ai#4348 — resolves the vLLM smoke blocker recorded in TODO_sm100_next_resync.md) and the E_local=1 nvfp4 oracle regression test. AI-assisted (Claude Code). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ush-style FP8 backend; sync CuTe-DSL 4.7 quant-staging fix (#4449) ## Summary Three things: a layout/naming refactor of `flashinfer.moe_ep`'s mega-kernel layer, the incorporation of the SM90 push-style FP8 backend (#4069, since merged upstream) as the first new backend added in the restructured shape, and one vendored-kernel sync that fixes the fused activation-quant staging crash on CuTe-DSL 4.7 — un-blocking 4.7.x and lifting the temporary `==4.6.1` pin. The branch is merged up to upstream/main (2febce5, past the v0.6.17 line and the #4069 squash). The refactor organizes the layer around two orthogonal views: 1. **Taxonomy (user view)** — backends move to `backends/mega/kernel/sm<arch>/<act_dtype>_<weight_dtype>_<out_dtype>_<kernel_style>/`, and registry `kernel_name` strings plus config classes carry the same fully-qualified names. One glance at a name now tells you the architecture, the activation/weight/output dtypes, and the kernel style: | old kernel_name | new kernel_name | new config class | |---|---|---| | `deep_gemm_mega` | `sm100_fp8_fp4_bf16_deepgemm` | `Sm100_Fp8_Fp4_Bf16_Deepgemm_MegaMoeConfig` | | `nvfp4_cutedsl` | `sm100_nvfp4_nvfp4_bf16_cutedsl` | `Sm100_Nvfp4_Nvfp4_Bf16_Cutedsl_MegaMoeConfig` | | `mxfp8_cutedsl` | `sm100_mxfp8_mxfp8_bf16_cutedsl` | `Sm100_Mxfp8_Mxfp8_Bf16_Cutedsl_MegaMoeConfig` | | `sm90_pull_fp8` | `sm90_fp8_fp8_bf16_pull_cutedsl` | `Sm90_Fp8_Fp8_Bf16_PullCutedsl_MegaMoeConfig` | | `sm90_push_fp8` (new, from #4069) | `sm90_fp8_fp8_bf16_push_cuda` | `Sm90_Fp8_Fp8_Bf16_PushCuda_MegaMoeConfig` | Naming conventions: deep_gemm dtypes are plain `fp8`/`fp4`, matching upstream `deep_gemm.fp8_fp4_mega_moe`; the mx/nv prefixes are reserved for the cutedsl kernels' block-scaled formats. Output dtype is always bf16 — nvfp4's `combine_dtype` is comm-wire compression, not an output format. 2. **Provenance (kernel-dev view)** — vendored kernel sources in `kernel_src/` are keyed by upstream repo snapshot, not by architecture: `kernel_src/sm100/cutedsl_megamoe` moves to `kernel_src/cutedsl_megamoe` (the mother repo ships kernels for multiple arches, so an smXX level misrepresents it). Each drop mirrors the vendor repo layout — `src/` byte-for-byte upstream, all adaptation in `shim/` — and gains a `VENDOR.md` recording upstream repo/commit/sync state and pending local diffs. A new `kernel_src/README.md` states the contract explicitly: **no edits to `src/` of any kind — including docstrings, comments, and lint fixes**; tool warnings against vendored files (docstring-coverage gates, review bots) are handled by excluding the path, never by editing the file. The sm90 fork trees (`kernel_src/sm90/pull_style_cutedsl_megakernel` from #4113, `kernel_src/sm90/push_style_megamoe` from #4069) intentionally stay separate snapshots — one kernel_src dir = one upstream commit — and fold into the mother drop if/when upstream merges them. **Why:** verbatim snapshots must stay diffable against one upstream commit, and splitting vendored trees per-dtype or per-arch breaks re-sync; meanwhile users navigate by architecture and dtype, not by which vendor repo a kernel came from. Putting each concern where its audience looks resolves the tension. The layout rule is documented in `docs/design_docs/moe_ep_architecture.md`, and it is what makes new backend families routine — demonstrated in this very PR by the SM90 push-style incorporation below, and next by the follow-up backend-family PRs (SM100 BF16 #4386, SM120 MXFP8). ## Directories affected All changes live under `flashinfer/moe_ep/` plus its tests and docs: - `backends/mega/kernel/sm100/{fp8_fp4_bf16_deepgemm,nvfp4_nvfp4_bf16_cutedsl,mxfp8_mxfp8_bf16_cutedsl}/` and `backends/mega/kernel/sm90/{fp8_fp8_bf16_pull_cutedsl,fp8_fp8_bf16_push_cuda}/` — taxonomy backend wrappers (moved/renamed; push_cuda is new). - `kernel_src/cutedsl_megamoe/` (moved from `kernel_src/sm100/cutedsl_megamoe/`), `kernel_src/sm90/pull_style_cutedsl_megakernel/`, `kernel_src/sm90/push_style_megamoe/` (new) — provenance-keyed vendored drops, each with `VENDOR.md`; new `kernel_src/README.md` states the no-edits contract. - `backends/mega/kernel/tuning.py` + per-backend `tuner.py` files — tuning machinery moved out of `tune.py` (now a CLI shim). - `core/kernel/registry.py`, `moe_ep/__init__.py` — deprecated-alias resolution and re-exports. - `tests/moe_ep/`, `docs/design_docs/moe_ep_{architecture,runbook}.md`, `pyproject.toml`/`.pre-commit-config.yaml` excludes, `run_tests.sh` (new 2-GPU `sm90_push` target). ## Test results - **Full `run_tests.sh` matrix — all 12 targets green** on 4xH100 (job 2389821, 2026-08-13), including the new `sm90_push` Hopper target and the fault-tolerance suites after the deadlock fixes. - **B200** (jobs 2388315/2388326): registry/alias smoke, deprecated aliases, unit x3 green — 396 passed / 72 skipped (push cpu/packaging/contract tests run; Hopper-marked kernel tests skip). - **Unit target re-validated green** after the second upstream merge (job 2389880) and again after the round-2 CodeRabbit fixes (job 2389916), same 396/72 counts, B200. - **8x B200** (jobs 2384640/2384641/2384650): quant-staging sync matrix fully green on both dsl 4.6.1 and 4.7.0 (details in the vendored-sync section below). - **GB200 + B200**: mxfp8/nvfp4 multirank oracle suites with the per-cell tolerance band. - Microbenchmark re-run: no regressions vs pre-restructure reference numbers (deep_gemm parity; cutedsl kernels at or above their previous points). - `pre-commit run -a` fully green at the branch head (e9f791a). ## SM90 push-style FP8 backend (incorporates #4069) Ports #4069 (head 301f8ce; since merged to main as f9b13ef — re-diffed, byte-identical, no post-review deltas) onto the taxonomy/provenance layout, serving as the first proof of the "one taxonomy backend dir + one provenance-keyed kernel drop" recipe: - **`kernel_src/sm90/push_style_megamoe/`** — verbatim byte-for-byte drop from the PR head (`src/{a2a,fp8_gemm}` CUDA sources, `shim/`, ACKNOWLEDGEMENT.md) plus a `VENDOR.md` provenance record. - **`backends/mega/kernel/sm90/fp8_fp8_bf16_push_cuda/`** — the five wrapper files relocated from upstream's flat `kernel/sm90_push_fp8/`, config renamed to `Sm90_Fp8_Fp8_Bf16_PushCuda_MegaMoeConfig`, registered with `deprecated_aliases=("sm90_push_fp8",)`. - **Core deltas carried from the PR:** `mega_layer.py` allocates the output before `stage_inputs`; pyproject package-data ships the drop's `.cu`/`.cuh` for non-editable installs; the `isolated_deep_gemm_cache` conftest fixture; the mega-layer allocation-order regression test. - **Tests:** the nine sm90_push_fp8 test files (names kept to minimize re-sync friction) rewritten to the taxonomy. Deviation from upstream: `run_tests.sh` exposes `sm90_push` as its own 2-GPU Hopper target instead of folding it into multirank — on non-Hopper nodes the arch-marked files collect 0 tests and torchrun turns pytest exit 5 into a failure. ## CuTe-DSL 4.7 quant-staging fix (vendored sync) The `CUDA_ERROR_MISALIGNED_ADDRESS` crash on cutlass-dsl 4.7.0 — which presented as a deep_gemm mega multirank failure — was root-caused to the **fused bf16→quantized activation staging** (`DataPreprocess` in the vendored cutedsl_megamoe tree), which every mega staging path shares, deep_gemm's included. The kernel team's fix is synced in as a single-file partial re-sync per the vendoring policy: - `kernel_src/cutedsl_megamoe/src/src/inputs_process.py` + `src/common/host_utils.py` taken **verbatim** from upstream `bangyus/cutedsl_megamoe @ 50117315d`, recorded in `VENDOR.md` under pending-diffs (resolves at the next full re-sync). The mxfp8 quant kernel is reworked so each lane owns one contiguous 16-byte fp8 store (adjacent lanes reduce the 32-element block amax via `shuffle_sync_bfly`, even lane writes the E8M0 scale), and `__init__` gains a hidden-size row-alignment guard. - Also fixes a stale pre-commit exclude left by the directory move (`kernel_src/sm100/cutedsl_megamoe` → `kernel_src/cutedsl_megamoe`) so hooks stop reformatting the verbatim `src/` tree. Validated on 8x B200 (jobs 2384640/2384641/2384650), full matrix green on **both** DSL versions: | section | dsl 4.6.1 | dsl 4.7.0 | |---|---|---| | drop's own harness (`python -m src.inputs_process`: bit-exact scales + SNR vs reference, nvfp4 offline/online + mxfp8) | 3/3 | 3/3 | | `test_fused_quant_stage.py` | 11/11 | 11/11 | | mega multirank x4 ranks (deep_gemm + nvfp4 + mxfp8) | 20/rank | 20/rank | | single-rank kernel-vs-reference oracles | 6/6 | 6/6 | The deep_gemm multirank suite previously crashed deterministically on 4.7.0; it now passes there. On the strength of this, the runbook's temporary `==4.6.1` pin is lifted (see the DSL guidance bullet below). ## Also in this PR - **Per-backend tuners.** `flashinfer/moe_ep/tune.py` becomes a pure CLI shim (surface unchanged: `python -m flashinfer.moe_ep.tune`); dtype-specific tuning moves into the backends (`sm100/{nvfp4,mxfp8}.../tuner.py`), shared sweep machinery (dist lifecycle, skewed restage, schedule grid, timed sweep tail) into `backends/mega/kernel/tuning.py`. - **CUTLASS DSL guidance updated (pin lifted).** The test-container recipe briefly carried a hard `nvidia-cutlass-dsl==4.6.1` pin because 4.7.0 crashed the mega multirank path; with the crash root-caused and fixed above, the runbook now allows `-U` installs again. 4.6.1 remains the perf-validated reference (pin it when producing numbers meant to compare against the TUNING.md tables); 4.7.0 is correctness-validated. The library's supported floor remains 4.5.2 (the MR!27 WAR already in main). - **Per-cell bf16 term-magnitude tolerance band** for the mxfp8 multirank oracle compares — a principled per-cell bound derived from the bf16 accumulation term magnitudes, replacing the global rtol that produced rare single-cell false failures. Validated on GB200 and B200. - **One-direction import layering rules** codified in the architecture doc, with all `cutedsl_megamoe` access routed through the drop's `__init__` rather than deep-path imports. ## Merge with upstream/main and follow-up fixes The branch is merged up to upstream/main in two steps. First to aaf97df (95 commits, incl. the v0.6.17 release line): conflict resolution keeps the restructure spellings everywhere; upstream's one real kernel advance in the moved tree — the 4fbac49 singleton-expert TMA-modes fix (#4296) — is ported onto the renamed paths and recorded in `VENDOR.md`. Notable upstream picks now in-tree: `BootstrapConfig.device` (#4348) and the E_local=1 nvfp4 oracle regression test. Second merge to 2febce5 (13 commits), resolving the conflicts created when #4069 itself squash-merged upstream (f9b13ef) with the same moe_ep files in the pre-restructure flat layout. Every conflict resolves to the taxonomy spellings (upstream's side is the flat spelling of content this branch already carries); upstream's flat `backends/mega/kernel/sm90_push_fp8/` wrapper and its re-folding of `sm90_push` into the multirank target are dropped in favor of this branch's layout. The vendored push drop was re-diffed against the merged SHA: byte-for-byte identical, no post-review deltas (recorded in `VENDOR.md`). Post-merge hardening found and fixed by full-suite runs: - **Merge fallout:** auto-merged regions had re-introduced pre-restructure `kernel_src.sm100.cutedsl_megamoe` spellings in 12 files, silently skipping entire GPU test files via `importorskip`; restored, and upstream's re-added flat `sm90_pull_fp8/` wrapper removed. - **FT test deadlocks (4xH100):** the fault-tolerance multirank test's evicted victim ran a collective `destroy()` against the survivors' barrier sequence, deadlocking until the NCCL watchdog — the victim tail now mirrors the survivors' barrier→destroy→barrier shape. The FT smoke's survivors now keep forwarding past the kill window so they actually observe the fault, and `run_tests.sh` judges the smoke by counting `SMOKE_RESULT` markers (torchrun interleaves lines). - **Unit-suite crasher isolation:** the long-known in-suite-only interpreter abort (heap corruption accumulating over the ~200-test single-process run, firing during a plain module import or in CPython teardown) is worked around by running the trigger test in its own pytest process and exiting the unit invocations via `os._exit(pytest_rc)`; rationale in the runbook, root cause tracked (needs ASAN). All tests pass — this is process-teardown hygiene, not a kernel bug. **CodeRabbit review responses.** Two rounds of actionable findings are fixed in-branch (640b75f, 57926a9) — highlights from round 2: the push packaging test's import-boundary gate was building the pre-taxonomy flat backend path and passing vacuously (fixed, now validates all 5 wrapper files); the test baseline's weight cache gains weakref eviction; `cutedsl_megamoe/shim/__main__.py` added so the documented `python -m ...shim` commands resolve; the cutedsl_megamoe `VENDOR.md` provenance TODOs are filled. Findings inside verbatim-vendored `kernel_src/**/src/` trees are deliberately not patched locally — they route upstream per the vendoring policy in `kernel_src/README.md`. **Lint.** `pre-commit run -a` is fully green (clang-format, mypy, ruff check/format, whitespace hooks). The final e9f791a is a pure ruff-format pass over 13 moe_ep files — line wraps where the longer taxonomy class names pushed calls past the limit. The vendored `src/` trees are untouched by hooks (the exclude set holds). ## Backward compatibility External callers keep working unchanged — both the old config-class names and the old kernel_name strings remain as deprecated aliases: - **Config classes**: `DeepGemmMegaMoeConfig`, `Nvfp4CutedslMegaMoeConfig`, `Mxfp8CutedslMegaMoeConfig`, `Sm90PullFp8MegaMoeConfig`, and `Sm90PushFp8MegaMoeConfig` are plain aliases of the new `Sm<arch>_..._MegaMoeConfig` classes, defined (with a removal note) in `flashinfer/moe_ep/__init__.py` right below the taxonomy imports, and still exported via `__all__`. - **Registry kernel_name strings**: `deep_gemm_mega`, `nvfp4_cutedsl`, `mxfp8_cutedsl`, `sm90_pull_fp8`, and `sm90_push_fp8` resolve to the taxonomy backends through the `deprecated_aliases=` parameter of each backend's `@register_mega_kernel(...)` decoration; the resolution machinery lives in `flashinfer/moe_ep/core/kernel/registry.py`. Using one emits a `DeprecationWarning`, and aliases are excluded from the available-kernels listing. - Both alias families WILL BE REMOVED in a future release (noted at both locations above). ## Testing - Directory moves and renames are behavior-preserving by construction; registry tests exercise both the taxonomy names and the deprecated aliases (alias use warns; the kernel listing shows taxonomy names only). - Full `run_tests.sh` matrix (all 12 targets) green on 4xH100 (job 2389821); B200 unit/registry/alias validation (jobs 2388315/2388326) — see Test results above. - The quant-staging sync validated on both dsl 4.6.1 and 4.7.0 (matrix above); mxfp8/nvfp4 multirank oracle suites validated on GB200 and B200. - The standalone MoE-EP microbenchmark was re-run against this branch with no regressions vs the pre-restructure reference numbers (deep_gemm parity; cutedsl kernels at or above their previous points). ## Relation to other PRs Re-layering on top of #4113 (SM90 pull-style FP8 backend, merged) and incorporating #4069 (SM90 push-style FP8 backend, merged upstream 2026-08-12; the vendored drop was re-diffed against the merged SHA f9b13ef and is byte-identical). This is the base branch for the upcoming backend-family PRs — SM100 BF16 (#4386) and SM120 MXFP8 — each of which adds one taxonomy backend directory plus one provenance-keyed kernel drop in the shape this restructure establishes. Both follow-up branches are already rebased onto this branch's head (unit target green on each), so they apply as exactly their backend-specific commits once this merges. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: Md Anik <mhoqueanik@cw-dfw-cs-001-login-01.cm.cluster> Co-authored-by: Md Saidul Hoque Anik <mhoqueanik@login-preos01.a51.clusters.nvidia.com>

…ush-style FP8 backend; sync CuTe-DSL 4.7 quant-staging fix (flashinfer-ai#4449) ## Summary Three things: a layout/naming refactor of `flashinfer.moe_ep`'s mega-kernel layer, the incorporation of the SM90 push-style FP8 backend (flashinfer-ai#4069, since merged upstream) as the first new backend added in the restructured shape, and one vendored-kernel sync that fixes the fused activation-quant staging crash on CuTe-DSL 4.7 — un-blocking 4.7.x and lifting the temporary `==4.6.1` pin. The branch is merged up to upstream/main (2febce5, past the v0.6.17 line and the flashinfer-ai#4069 squash). The refactor organizes the layer around two orthogonal views: 1. **Taxonomy (user view)** — backends move to `backends/mega/kernel/sm<arch>/<act_dtype>_<weight_dtype>_<out_dtype>_<kernel_style>/`, and registry `kernel_name` strings plus config classes carry the same fully-qualified names. One glance at a name now tells you the architecture, the activation/weight/output dtypes, and the kernel style: | old kernel_name | new kernel_name | new config class | |---|---|---| | `deep_gemm_mega` | `sm100_fp8_fp4_bf16_deepgemm` | `Sm100_Fp8_Fp4_Bf16_Deepgemm_MegaMoeConfig` | | `nvfp4_cutedsl` | `sm100_nvfp4_nvfp4_bf16_cutedsl` | `Sm100_Nvfp4_Nvfp4_Bf16_Cutedsl_MegaMoeConfig` | | `mxfp8_cutedsl` | `sm100_mxfp8_mxfp8_bf16_cutedsl` | `Sm100_Mxfp8_Mxfp8_Bf16_Cutedsl_MegaMoeConfig` | | `sm90_pull_fp8` | `sm90_fp8_fp8_bf16_pull_cutedsl` | `Sm90_Fp8_Fp8_Bf16_PullCutedsl_MegaMoeConfig` | | `sm90_push_fp8` (new, from flashinfer-ai#4069) | `sm90_fp8_fp8_bf16_push_cuda` | `Sm90_Fp8_Fp8_Bf16_PushCuda_MegaMoeConfig` | Naming conventions: deep_gemm dtypes are plain `fp8`/`fp4`, matching upstream `deep_gemm.fp8_fp4_mega_moe`; the mx/nv prefixes are reserved for the cutedsl kernels' block-scaled formats. Output dtype is always bf16 — nvfp4's `combine_dtype` is comm-wire compression, not an output format. 2. **Provenance (kernel-dev view)** — vendored kernel sources in `kernel_src/` are keyed by upstream repo snapshot, not by architecture: `kernel_src/sm100/cutedsl_megamoe` moves to `kernel_src/cutedsl_megamoe` (the mother repo ships kernels for multiple arches, so an smXX level misrepresents it). Each drop mirrors the vendor repo layout — `src/` byte-for-byte upstream, all adaptation in `shim/` — and gains a `VENDOR.md` recording upstream repo/commit/sync state and pending local diffs. A new `kernel_src/README.md` states the contract explicitly: **no edits to `src/` of any kind — including docstrings, comments, and lint fixes**; tool warnings against vendored files (docstring-coverage gates, review bots) are handled by excluding the path, never by editing the file. The sm90 fork trees (`kernel_src/sm90/pull_style_cutedsl_megakernel` from flashinfer-ai#4113, `kernel_src/sm90/push_style_megamoe` from flashinfer-ai#4069) intentionally stay separate snapshots — one kernel_src dir = one upstream commit — and fold into the mother drop if/when upstream merges them. **Why:** verbatim snapshots must stay diffable against one upstream commit, and splitting vendored trees per-dtype or per-arch breaks re-sync; meanwhile users navigate by architecture and dtype, not by which vendor repo a kernel came from. Putting each concern where its audience looks resolves the tension. The layout rule is documented in `docs/design_docs/moe_ep_architecture.md`, and it is what makes new backend families routine — demonstrated in this very PR by the SM90 push-style incorporation below, and next by the follow-up backend-family PRs (SM100 BF16 flashinfer-ai#4386, SM120 MXFP8). ## Directories affected All changes live under `flashinfer/moe_ep/` plus its tests and docs: - `backends/mega/kernel/sm100/{fp8_fp4_bf16_deepgemm,nvfp4_nvfp4_bf16_cutedsl,mxfp8_mxfp8_bf16_cutedsl}/` and `backends/mega/kernel/sm90/{fp8_fp8_bf16_pull_cutedsl,fp8_fp8_bf16_push_cuda}/` — taxonomy backend wrappers (moved/renamed; push_cuda is new). - `kernel_src/cutedsl_megamoe/` (moved from `kernel_src/sm100/cutedsl_megamoe/`), `kernel_src/sm90/pull_style_cutedsl_megakernel/`, `kernel_src/sm90/push_style_megamoe/` (new) — provenance-keyed vendored drops, each with `VENDOR.md`; new `kernel_src/README.md` states the no-edits contract. - `backends/mega/kernel/tuning.py` + per-backend `tuner.py` files — tuning machinery moved out of `tune.py` (now a CLI shim). - `core/kernel/registry.py`, `moe_ep/__init__.py` — deprecated-alias resolution and re-exports. - `tests/moe_ep/`, `docs/design_docs/moe_ep_{architecture,runbook}.md`, `pyproject.toml`/`.pre-commit-config.yaml` excludes, `run_tests.sh` (new 2-GPU `sm90_push` target). ## Test results - **Full `run_tests.sh` matrix — all 12 targets green** on 4xH100 (job 2389821, 2026-08-13), including the new `sm90_push` Hopper target and the fault-tolerance suites after the deadlock fixes. - **B200** (jobs 2388315/2388326): registry/alias smoke, deprecated aliases, unit x3 green — 396 passed / 72 skipped (push cpu/packaging/contract tests run; Hopper-marked kernel tests skip). - **Unit target re-validated green** after the second upstream merge (job 2389880) and again after the round-2 CodeRabbit fixes (job 2389916), same 396/72 counts, B200. - **8x B200** (jobs 2384640/2384641/2384650): quant-staging sync matrix fully green on both dsl 4.6.1 and 4.7.0 (details in the vendored-sync section below). - **GB200 + B200**: mxfp8/nvfp4 multirank oracle suites with the per-cell tolerance band. - Microbenchmark re-run: no regressions vs pre-restructure reference numbers (deep_gemm parity; cutedsl kernels at or above their previous points). - `pre-commit run -a` fully green at the branch head (e9f791a). ## SM90 push-style FP8 backend (incorporates flashinfer-ai#4069) Ports flashinfer-ai#4069 (head 301f8ce; since merged to main as f9b13ef — re-diffed, byte-identical, no post-review deltas) onto the taxonomy/provenance layout, serving as the first proof of the "one taxonomy backend dir + one provenance-keyed kernel drop" recipe: - **`kernel_src/sm90/push_style_megamoe/`** — verbatim byte-for-byte drop from the PR head (`src/{a2a,fp8_gemm}` CUDA sources, `shim/`, ACKNOWLEDGEMENT.md) plus a `VENDOR.md` provenance record. - **`backends/mega/kernel/sm90/fp8_fp8_bf16_push_cuda/`** — the five wrapper files relocated from upstream's flat `kernel/sm90_push_fp8/`, config renamed to `Sm90_Fp8_Fp8_Bf16_PushCuda_MegaMoeConfig`, registered with `deprecated_aliases=("sm90_push_fp8",)`. - **Core deltas carried from the PR:** `mega_layer.py` allocates the output before `stage_inputs`; pyproject package-data ships the drop's `.cu`/`.cuh` for non-editable installs; the `isolated_deep_gemm_cache` conftest fixture; the mega-layer allocation-order regression test. - **Tests:** the nine sm90_push_fp8 test files (names kept to minimize re-sync friction) rewritten to the taxonomy. Deviation from upstream: `run_tests.sh` exposes `sm90_push` as its own 2-GPU Hopper target instead of folding it into multirank — on non-Hopper nodes the arch-marked files collect 0 tests and torchrun turns pytest exit 5 into a failure. ## CuTe-DSL 4.7 quant-staging fix (vendored sync) The `CUDA_ERROR_MISALIGNED_ADDRESS` crash on cutlass-dsl 4.7.0 — which presented as a deep_gemm mega multirank failure — was root-caused to the **fused bf16→quantized activation staging** (`DataPreprocess` in the vendored cutedsl_megamoe tree), which every mega staging path shares, deep_gemm's included. The kernel team's fix is synced in as a single-file partial re-sync per the vendoring policy: - `kernel_src/cutedsl_megamoe/src/src/inputs_process.py` + `src/common/host_utils.py` taken **verbatim** from upstream `bangyus/cutedsl_megamoe @ 50117315d`, recorded in `VENDOR.md` under pending-diffs (resolves at the next full re-sync). The mxfp8 quant kernel is reworked so each lane owns one contiguous 16-byte fp8 store (adjacent lanes reduce the 32-element block amax via `shuffle_sync_bfly`, even lane writes the E8M0 scale), and `__init__` gains a hidden-size row-alignment guard. - Also fixes a stale pre-commit exclude left by the directory move (`kernel_src/sm100/cutedsl_megamoe` → `kernel_src/cutedsl_megamoe`) so hooks stop reformatting the verbatim `src/` tree. Validated on 8x B200 (jobs 2384640/2384641/2384650), full matrix green on **both** DSL versions: | section | dsl 4.6.1 | dsl 4.7.0 | |---|---|---| | drop's own harness (`python -m src.inputs_process`: bit-exact scales + SNR vs reference, nvfp4 offline/online + mxfp8) | 3/3 | 3/3 | | `test_fused_quant_stage.py` | 11/11 | 11/11 | | mega multirank x4 ranks (deep_gemm + nvfp4 + mxfp8) | 20/rank | 20/rank | | single-rank kernel-vs-reference oracles | 6/6 | 6/6 | The deep_gemm multirank suite previously crashed deterministically on 4.7.0; it now passes there. On the strength of this, the runbook's temporary `==4.6.1` pin is lifted (see the DSL guidance bullet below). ## Also in this PR - **Per-backend tuners.** `flashinfer/moe_ep/tune.py` becomes a pure CLI shim (surface unchanged: `python -m flashinfer.moe_ep.tune`); dtype-specific tuning moves into the backends (`sm100/{nvfp4,mxfp8}.../tuner.py`), shared sweep machinery (dist lifecycle, skewed restage, schedule grid, timed sweep tail) into `backends/mega/kernel/tuning.py`. - **CUTLASS DSL guidance updated (pin lifted).** The test-container recipe briefly carried a hard `nvidia-cutlass-dsl==4.6.1` pin because 4.7.0 crashed the mega multirank path; with the crash root-caused and fixed above, the runbook now allows `-U` installs again. 4.6.1 remains the perf-validated reference (pin it when producing numbers meant to compare against the TUNING.md tables); 4.7.0 is correctness-validated. The library's supported floor remains 4.5.2 (the MR!27 WAR already in main). - **Per-cell bf16 term-magnitude tolerance band** for the mxfp8 multirank oracle compares — a principled per-cell bound derived from the bf16 accumulation term magnitudes, replacing the global rtol that produced rare single-cell false failures. Validated on GB200 and B200. - **One-direction import layering rules** codified in the architecture doc, with all `cutedsl_megamoe` access routed through the drop's `__init__` rather than deep-path imports. ## Merge with upstream/main and follow-up fixes The branch is merged up to upstream/main in two steps. First to aaf97df (95 commits, incl. the v0.6.17 release line): conflict resolution keeps the restructure spellings everywhere; upstream's one real kernel advance in the moved tree — the 4fbac49 singleton-expert TMA-modes fix (flashinfer-ai#4296) — is ported onto the renamed paths and recorded in `VENDOR.md`. Notable upstream picks now in-tree: `BootstrapConfig.device` (flashinfer-ai#4348) and the E_local=1 nvfp4 oracle regression test. Second merge to 2febce5 (13 commits), resolving the conflicts created when flashinfer-ai#4069 itself squash-merged upstream (f9b13ef) with the same moe_ep files in the pre-restructure flat layout. Every conflict resolves to the taxonomy spellings (upstream's side is the flat spelling of content this branch already carries); upstream's flat `backends/mega/kernel/sm90_push_fp8/` wrapper and its re-folding of `sm90_push` into the multirank target are dropped in favor of this branch's layout. The vendored push drop was re-diffed against the merged SHA: byte-for-byte identical, no post-review deltas (recorded in `VENDOR.md`). Post-merge hardening found and fixed by full-suite runs: - **Merge fallout:** auto-merged regions had re-introduced pre-restructure `kernel_src.sm100.cutedsl_megamoe` spellings in 12 files, silently skipping entire GPU test files via `importorskip`; restored, and upstream's re-added flat `sm90_pull_fp8/` wrapper removed. - **FT test deadlocks (4xH100):** the fault-tolerance multirank test's evicted victim ran a collective `destroy()` against the survivors' barrier sequence, deadlocking until the NCCL watchdog — the victim tail now mirrors the survivors' barrier→destroy→barrier shape. The FT smoke's survivors now keep forwarding past the kill window so they actually observe the fault, and `run_tests.sh` judges the smoke by counting `SMOKE_RESULT` markers (torchrun interleaves lines). - **Unit-suite crasher isolation:** the long-known in-suite-only interpreter abort (heap corruption accumulating over the ~200-test single-process run, firing during a plain module import or in CPython teardown) is worked around by running the trigger test in its own pytest process and exiting the unit invocations via `os._exit(pytest_rc)`; rationale in the runbook, root cause tracked (needs ASAN). All tests pass — this is process-teardown hygiene, not a kernel bug. **CodeRabbit review responses.** Two rounds of actionable findings are fixed in-branch (640b75f, 57926a9) — highlights from round 2: the push packaging test's import-boundary gate was building the pre-taxonomy flat backend path and passing vacuously (fixed, now validates all 5 wrapper files); the test baseline's weight cache gains weakref eviction; `cutedsl_megamoe/shim/__main__.py` added so the documented `python -m ...shim` commands resolve; the cutedsl_megamoe `VENDOR.md` provenance TODOs are filled. Findings inside verbatim-vendored `kernel_src/**/src/` trees are deliberately not patched locally — they route upstream per the vendoring policy in `kernel_src/README.md`. **Lint.** `pre-commit run -a` is fully green (clang-format, mypy, ruff check/format, whitespace hooks). The final e9f791a is a pure ruff-format pass over 13 moe_ep files — line wraps where the longer taxonomy class names pushed calls past the limit. The vendored `src/` trees are untouched by hooks (the exclude set holds). ## Backward compatibility External callers keep working unchanged — both the old config-class names and the old kernel_name strings remain as deprecated aliases: - **Config classes**: `DeepGemmMegaMoeConfig`, `Nvfp4CutedslMegaMoeConfig`, `Mxfp8CutedslMegaMoeConfig`, `Sm90PullFp8MegaMoeConfig`, and `Sm90PushFp8MegaMoeConfig` are plain aliases of the new `Sm<arch>_..._MegaMoeConfig` classes, defined (with a removal note) in `flashinfer/moe_ep/__init__.py` right below the taxonomy imports, and still exported via `__all__`. - **Registry kernel_name strings**: `deep_gemm_mega`, `nvfp4_cutedsl`, `mxfp8_cutedsl`, `sm90_pull_fp8`, and `sm90_push_fp8` resolve to the taxonomy backends through the `deprecated_aliases=` parameter of each backend's `@register_mega_kernel(...)` decoration; the resolution machinery lives in `flashinfer/moe_ep/core/kernel/registry.py`. Using one emits a `DeprecationWarning`, and aliases are excluded from the available-kernels listing. - Both alias families WILL BE REMOVED in a future release (noted at both locations above). ## Testing - Directory moves and renames are behavior-preserving by construction; registry tests exercise both the taxonomy names and the deprecated aliases (alias use warns; the kernel listing shows taxonomy names only). - Full `run_tests.sh` matrix (all 12 targets) green on 4xH100 (job 2389821); B200 unit/registry/alias validation (jobs 2388315/2388326) — see Test results above. - The quant-staging sync validated on both dsl 4.6.1 and 4.7.0 (matrix above); mxfp8/nvfp4 multirank oracle suites validated on GB200 and B200. - The standalone MoE-EP microbenchmark was re-run against this branch with no regressions vs the pre-restructure reference numbers (deep_gemm parity; cutedsl kernels at or above their previous points). ## Relation to other PRs Re-layering on top of flashinfer-ai#4113 (SM90 pull-style FP8 backend, merged) and incorporating flashinfer-ai#4069 (SM90 push-style FP8 backend, merged upstream 2026-08-12; the vendored drop was re-diffed against the merged SHA f9b13ef and is byte-identical). This is the base branch for the upcoming backend-family PRs — SM100 BF16 (flashinfer-ai#4386) and SM120 MXFP8 — each of which adds one taxonomy backend directory plus one provenance-keyed kernel drop in the shape this restructure establishes. Both follow-up branches are already rebased onto this branch's head (unit target green on each), so they apply as exactly their backend-specific commits once this merges. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: Md Anik <mhoqueanik@cw-dfw-cs-001-login-01.cm.cluster> Co-authored-by: Md Saidul Hoque Anik <mhoqueanik@login-preos01.a51.clusters.nvidia.com>

…L_RANK Adopt flashinfer-ai/flashinfer#4348: BootstrapConfig grew an optional kw-only device field that all three of the runtime's bind sites use verbatim, so make_fi_moe_ep_bootstrap now passes the device vLLM already bound (torch.cuda.current_device()) and the os.environ["LOCAL_RANK"] mutation in ensure_fi_moe_ep_runtime is gone. Without this, a remapped CUDA_VISIBLE_DEVICES made the runtime rebind to the wrong ordinal and launch the weight transforms against another device's pointers (CUDA_ERROR_ILLEGAL_ADDRESS in deep_gemm transform_sf during load). Requires a flashinfer build that includes vllm-project#4348; older builds raise TypeError at bootstrap construction. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mohammad Hoque <mhoqueanik@nvidia.com> Signed-off-by: Md Saidul Hoque Anik <mhoqueanik@nvidia.com>

The flashinfer_moe_ep backends pass BootstrapConfig(device=...) to pin the CUDA device (flashinfer-ai/flashinfer#4348), which first ships in the 0.6.18 release. Bump the pin in requirements/cuda.txt and the Dockerfile/versions.json together, per the comment in cuda.txt. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mohammad Hoque <mhoqueanik@nvidia.com> Signed-off-by: Md Saidul Hoque Anik <mhoqueanik@nvidia.com>

Selecting a flashinfer_moe_ep backend against a flashinfer older than 0.6.18 (no BootstrapConfig.device, flashinfer-ai/flashinfer#4348) now fails at config time with an actionable ImportError instead of rebinding workers to the wrong GPU under a remapped CUDA_VISIBLE_DEVICES. Probe the dataclass field rather than the package version so source builds with the fix pass regardless of version metadata. The requirements pin stays at 0.6.15.post1: the backends are opt-in and the gate only fires on use. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mohammad Hoque <mhoqueanik@nvidia.com> Signed-off-by: Md Saidul Hoque Anik <mhoqueanik@nvidia.com>

BootstrapConfig.device (flashinfer-ai/flashinfer#4348) ships in the 0.6.17 release, not 0.6.18; update the gate's error message, docstring, and test to name the right version. The check itself is unchanged (field probe, fires only on fi backend use). Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mohammad Hoque <mhoqueanik@nvidia.com> Signed-off-by: Md Saidul Hoque Anik <mhoqueanik@nvidia.com>

The FlashInfer requirements pin will carry the BootstrapConfig.device dependency (flashinfer-ai/flashinfer#4348), so the config-time field probe is redundant; the plain ImportError from _has_fi_moe_ep_runtime remains the fail-fast for builds without the moe_ep runtime. This reverts commits 706316543 and 0c5454829. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mohammad Hoque <mhoqueanik@nvidia.com> Signed-off-by: Md Saidul Hoque Anik <mhoqueanik@nvidia.com>

Upstream flashinfer-ai#4348 asserts _resolve_local_device returns the raw LOCAL_RANK; the unconditional local_rank % device_count fold broke that test on 1-GPU CI (5 % 1 == 0). Fold only under the sm_12x rank-sharing flow (MEGA_SINGLE_GPU_GLOO=1), where it is required, and add a unit test for the gated fold. AI-assisted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

mhoqueanik requested review from Anerudhan, Aneureka, IwakuraRein, aleozlx, feih-nv, jiahanc, nv-yunzheq, samuellees and yzh119 as code owners August 4, 2026 22:11

mhoqueanik force-pushed the moe_ep-respect-caller-device branch from 1503f89 to 089472b Compare August 4, 2026 22:13

coderabbitai Bot reviewed Aug 4, 2026

View reviewed changes

Comment thread flashinfer/moe_ep/config.py Outdated

mhoqueanik enabled auto-merge (squash) August 4, 2026 22:14

mhoqueanik disabled auto-merge August 4, 2026 22:15

mhoqueanik force-pushed the moe_ep-respect-caller-device branch from 089472b to e4d7c1b Compare August 4, 2026 22:17

Anerudhan approved these changes Aug 5, 2026

View reviewed changes

mhoqueanik enabled auto-merge (squash) August 5, 2026 20:23

mhoqueanik merged commit 3c57ef1 into flashinfer-ai:main Aug 5, 2026
119 of 158 checks passed

aleozlx added the v0.6.17 label Aug 6, 2026

mhoqueanik mentioned this pull request Aug 7, 2026

[Model][MoE] DeepSeek-V4: add opt-in FlashInfer moe_ep expert backend vllm-project/vllm#49636

Merged

coderabbitai Bot mentioned this pull request Aug 7, 2026

feat(moe_ep): SM120 MXFP8 swap-AB CuTeDSL MegaMoE kernel #4387

Merged

mhoqueanik self-assigned this Aug 11, 2026

aleozlx mentioned this pull request Aug 11, 2026

v0.6.17 release highlights draft #4471

Closed

mhoqueanik mentioned this pull request Aug 13, 2026

refactor(moe_ep): taxonomy/provenance restructure; incorporate SM90 push-style FP8 backend; sync CuTe-DSL 4.7 quant-staging fix #4449

Merged

Navigation Menu

Uh oh!

fix(moe_ep): let the host framework pin the CUDA device via Bootstrap - #4348

fix(moe_ep): let the host framework pin the CUDA device via Bootstrap#4348
mhoqueanik merged 1 commit into
flashinfer-ai:mainfrom
mhoqueanik:moe_ep-respect-caller-device

mhoqueanik commented Aug 4, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Aug 4, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

coderabbitai Bot commented Aug 4, 2026

Uh oh!

coderabbitai Bot commented Aug 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

mhoqueanik commented Aug 4, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai Bot commented Aug 4, 2026

Uh oh!

coderabbitai Bot commented Aug 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mhoqueanik commented Aug 4, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Aug 4, 2026 •

edited

Loading