fix(moe): name-filter output-scale-incompatible cubins until pinned packages carry mDtypeSfC - #4235
Conversation
…ages carry mDtypeSfC The mDtypeSfC check from flashinfer-ai#4213 cannot work against the currently pinned cubin packages: BatchedGemmOptions in both published packages (Blackwell b368d003, Rubin 46d3f356) predates the field, so the check is a compile error (nvcc: class BatchedGemmOptions has no member mDtypeSfC) for any JIT/AOT rebuild of the trtllm-gen MoE module, on every architecture. Replace it with a name filter carrying the same intent: the offending kernels (linear-FP32 output scale factors overflowing the E4M3-sized buffer -> illegal memory access, flashinfer-ai#4164 / NVBug 6517914) encode the non-standard SF dtype as the "E2m1xFp32" token in their function names. Verified against the published catalogs: 132 such kernels, all in the sm_107a package (46d3f356); zero in the Blackwell package, so the filter is a no-op there. The typed mDtypeSfC check should be restored once packages carrying the field are published and pinned. The flashinfer-ai#4213 per-token-scaling restructure and launcher tactic-enumeration alignment are unchanged; they use existing fields. AI-assisted (Claude Code).
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
kahyunnam
left a comment
There was a problem hiding this comment.
lgtm to merge for rc4 pending validation on rubin
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
<!-- .github/pull_request_template.md --> ## 📌 Description This PR relands SM 107 support to main branch (reverted in #4171) as well as some other release fixes. #### Cherry Picks - #4191 - #4189 - #4200 - #4215 - #4225 - #4230 - #4235 - #4226 - #4257 - #4258 - #4261 #### Other Changes - Rubin guards from #4252's conflict resolution (`TLLM_RUBIN_FEATURES`: SiTuGlu static_asserts + tile-192 advertisement, compiled out for the Rubin BMM pin) - Test-contract update: `test_unified_moe.py` arch assertions written post-revert (#4159) flipped to the restored contract (FP4/BF16 claim 107; FP8 stays 100/103) <!-- What does this PR do? Briefly describe the changes and why they’re needed. --> ## 🔍 Related Issues <!-- Link any related issues here --> #4107, #4164, reverts #4171 ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [ ] Tests have been added or updated as needed. - [ ] All tests are passing (`unittest`, etc.). ## Reviewer Notes <!-- Optional: anything you'd like reviewers to focus on, concerns, etc. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added support for Rubin/SM107 GPUs across GEMM, MoE, attention, quantization, sampling, and DeepGEMM workflows. * Added architecture-aware kernel selection, memory sizing, compilation, and artifact handling. * **Bug Fixes** * Improved validation and error messages for incompatible GPU architectures and invalid kernel configurations. * Clearly rejects unsupported NVFP4 KV-cache operations on SM107. * **Documentation** * Updated installation guidance with the SM107 architecture target. * **Tests** * Expanded architecture coverage and compatibility checks across GPU test suites. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Vinnie6167 <Vinnie6167@users.noreply.github.com> Co-authored-by: Ka-Hyun Nam <knam@nvidia.com> Co-authored-by: Alex Yang <aleyang@nvidia.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Jimmy Zhou <79552142+jimmyzho@users.noreply.github.com>
📌 Description
Fixes two coupled problems on
release-v0.6.16:1. The branch tip cannot compile the trtllm-gen fused-MoE module — on any architecture.
#4213 checks
BatchedGemmOptions::mDtypeSfC, but that field does not exist in either pinned cubin package's exported headers (Blackwellb368d003, Rubin46d3f356— verified 0 occurrences in both publishedBatchedGemmOptions.h, and in the internal81a53cffpackage as well). Any JIT or AOT rebuild offused_moe_trtllm_*fails with:(The original PR #4168 shows the same failure in its CI on main.)
2. The FP4-MoE illegal-memory-access #4213 targets (#4164 / NVBug 6517914) is still live on Rubin.
The offending
bmm_E2m1xFp32_*kernels emit linear FP32 output scale-factors into the E4M3-sized (1 B/block) buffer the MoE pipeline allocates → device-side overflow → async IMA, near-deterministic under autotune. Verified against the published catalogs: 132 such kernels, all in the sm_107a package (46d3f356); zero in the Blackwell package — which is why the per-arch pin split (#4191) healed Blackwell while Rubin still crashes.What changed
Replace the un-compilable typed check with a kernel-name filter carrying the same intent: reject configs whose
mFunctionNamecontains theE2m1xFp32token. This compiles against the currently published packages and removes the bad candidates before autotune can run them. #4213's other changes (per-token-scaling restructure, launcher tactic-enumeration alignment) are inherited unchanged.Scope note: the name filter is deliberately narrower than the typed check — it rejects only the known-bad family. Against the published catalogs this is behaviorally equivalent (the
E2m1xFp32kernels are the only offenders present). The TODO in the code ties restoring the typedmDtypeSfCcheck to publishing + pinning packages that carry the field.Validation
Refs: #4164, #4213, #4168 · NVBug 6517914
🤖 Generated with Claude Code