Skip to content

fix(moe): name-filter output-scale-incompatible cubins until pinned packages carry mDtypeSfC - #4235

Merged
Vinnie6167 merged 1 commit into
flashinfer-ai:release-v0.6.16from
Vinnie6167:fix-moe-outsf-name-filter
Jul 29, 2026
Merged

Vinnie6167 merged 1 commit into
flashinfer-ai:release-v0.6.16from
Vinnie6167:fix-moe-outsf-name-filter

Conversation

@Vinnie6167

Copy link
Copy Markdown
Contributor

📌 Description

Fixes two coupled problems on release-v0.6.16:

1. The branch tip cannot compile the trtllm-gen fused-MoE module — on any architecture.
#4213 checks BatchedGemmOptions::mDtypeSfC, but that field does not exist in either pinned cubin package's exported headers (Blackwell b368d003, Rubin 46d3f356 — verified 0 occurrences in both published BatchedGemmOptions.h, and in the internal 81a53cff package as well). Any JIT or AOT rebuild of fused_moe_trtllm_* fails with:

trtllm_batched_gemm_runner.cu(128): error: class "BatchedGemmOptions" has no member "mDtypeSfC"

(The original PR #4168 shows the same failure in its CI on main.)

2. The FP4-MoE illegal-memory-access #4213 targets (#4164 / NVBug 6517914) is still live on Rubin.
The offending bmm_E2m1xFp32_* kernels emit linear FP32 output scale-factors into the E4M3-sized (1 B/block) buffer the MoE pipeline allocates → device-side overflow → async IMA, near-deterministic under autotune. Verified against the published catalogs: 132 such kernels, all in the sm_107a package (46d3f356); zero in the Blackwell package — which is why the per-arch pin split (#4191) healed Blackwell while Rubin still crashes.

What changed

Replace the un-compilable typed check with a kernel-name filter carrying the same intent: reject configs whose mFunctionName contains the E2m1xFp32 token. This compiles against the currently published packages and removes the bad candidates before autotune can run them. #4213's other changes (per-token-scaling restructure, launcher tactic-enumeration alignment) are inherited unchanged.

Scope note: the name filter is deliberately narrower than the typed check — it rejects only the known-bad family. Against the published catalogs this is behaviorally equivalent (the E2m1xFp32 kernels are the only offenders present). The TODO in the code ties restoring the typed mDtypeSfC check to publishing + pinning packages that carry the field.

Validation

  • Build break reproduced (sbsa AOT wheel build, CTK 13.4) at the branch tip; gone with this patch.
  • VR200 (Rubin) re-run of the affected MoE test files in progress; results will be posted here before marking ready for review.

Refs: #4164, #4213, #4168 · NVBug 6517914

🤖 Generated with Claude Code

…ages carry mDtypeSfC

The mDtypeSfC check from flashinfer-ai#4213 cannot work against the currently pinned
cubin packages: BatchedGemmOptions in both published packages (Blackwell
b368d003, Rubin 46d3f356) predates the field, so the check is a compile
error (nvcc: class BatchedGemmOptions has no member mDtypeSfC) for any
JIT/AOT rebuild of the trtllm-gen MoE module, on every architecture.

Replace it with a name filter carrying the same intent: the offending
kernels (linear-FP32 output scale factors overflowing the E4M3-sized
buffer -> illegal memory access, flashinfer-ai#4164 / NVBug 6517914) encode the
non-standard SF dtype as the "E2m1xFp32" token in their function names.
Verified against the published catalogs: 132 such kernels, all in the
sm_107a package (46d3f356); zero in the Blackwell package, so the filter
is a no-op there. The typed mDtypeSfC check should be restored once
packages carrying the field are published and pinned.

The flashinfer-ai#4213 per-token-scaling restructure and launcher tactic-enumeration
alignment are unchanged; they use existing fields.

AI-assisted (Claude Code).
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 5c676bce-d81b-47d4-8c74-272436f57b06

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@kahyunnam kahyunnam left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm to merge for rc4 pending validation on rubin

@Vinnie6167
Vinnie6167 marked this pull request as ready for review July 29, 2026 03:49
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@Vinnie6167
Vinnie6167 merged commit b9bb507 into flashinfer-ai:release-v0.6.16 Jul 29, 2026
5 checks passed
aleozlx added a commit that referenced this pull request Jul 30, 2026
<!-- .github/pull_request_template.md -->

## 📌 Description

This PR relands SM 107 support to main branch (reverted in #4171) as
well as some other release fixes.

#### Cherry Picks
- #4191
- #4189
- #4200
- #4215
- #4225
- #4230
- #4235
- #4226
- #4257
- #4258 
- #4261

#### Other Changes
- Rubin guards from #4252's conflict resolution (`TLLM_RUBIN_FEATURES`:
SiTuGlu
static_asserts + tile-192 advertisement, compiled out for the Rubin BMM
pin)
- Test-contract update: `test_unified_moe.py` arch assertions written
post-revert
(#4159) flipped to the restored contract (FP4/BF16 claim 107; FP8 stays
100/103)

<!-- What does this PR do? Briefly describe the changes and why they’re
needed. -->

## 🔍 Related Issues

<!-- Link any related issues here -->

#4107, #4164, reverts #4171

## 🚀 Pull Request Checklist

Thank you for contributing to FlashInfer! Before we review your pull
request, please make sure the following items are complete.

### ✅ Pre-commit Checks

- [x] I have installed `pre-commit` by running `pip install pre-commit`
(or used your preferred method).
- [x] I have installed the hooks with `pre-commit install`.
- [x] I have run the hooks manually with `pre-commit run --all-files`
and fixed any reported issues.

> If you are unsure about how to set up `pre-commit`, see [the
pre-commit documentation](https://pre-commit.com/).

## 🧪 Tests

- [ ] Tests have been added or updated as needed.
- [ ] All tests are passing (`unittest`, etc.).

## Reviewer Notes

<!-- Optional: anything you'd like reviewers to focus on, concerns, etc.
-->


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added support for Rubin/SM107 GPUs across GEMM, MoE, attention,
quantization, sampling, and DeepGEMM workflows.
* Added architecture-aware kernel selection, memory sizing, compilation,
and artifact handling.
* **Bug Fixes**
* Improved validation and error messages for incompatible GPU
architectures and invalid kernel configurations.
  * Clearly rejects unsupported NVFP4 KV-cache operations on SM107.
* **Documentation**
  * Updated installation guidance with the SM107 architecture target.
* **Tests**
* Expanded architecture coverage and compatibility checks across GPU
test suites.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Vinnie6167 <Vinnie6167@users.noreply.github.com>
Co-authored-by: Ka-Hyun Nam <knam@nvidia.com>
Co-authored-by: Alex Yang <aleyang@nvidia.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Jimmy Zhou <79552142+jimmyzho@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants