fix(moe): don't mark trtllm-gen FP4 MoE backend supported on SM120/SM121 - #3670
waynehacking8 wants to merge 1 commit into
Conversation
📝 WalkthroughWalkthrough
ChangesTrtllmFp4Config SM100-family gating
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Possibly related issues
Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request restricts the supported architectures for TrtllmFp4Config to the SM100 family (specifically 100 <= arch < 120) to prevent dispatch failures on unsupported architectures. The reviewer suggests using an explicit set of supported architectures (100, 103, 110) instead of a range check for greater precision and consistency with other configurations.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| # makes MoELayer build a runner that then fails at dispatch | ||
| # (cubin-not-found) instead of raising the clean "no usable backend" | ||
| # error at construction. | ||
| return 100 <= arch < 120 |
There was a problem hiding this comment.
Since the trtllm-gen SM100 module cubins are specifically built for SM100, SM103, and SM110, using an explicit set of supported architectures is more precise and robust than a range check. This also aligns with the pattern used in CuteDslConfig.supported (which uses arch in (100, 103)).
| return 100 <= arch < 120 | |
| return arch in (100, 103, 110) |
TrtllmFp4Config.supported() returned True for all arch >= 100, but the routed
runner delegates to the trtllm-gen sm100 module whose cubins are SM100-family
only (SM100/SM103/SM110). On SM120/SM121 (consumer Blackwell) there are no
trtllm-gen MoE cubins, so MoELayer built a TrtllmFp4RoutedRunner that then
failed deep at dispatch (cubin-not-found) instead of raising the clean
"no usable backend" error at construction.
The existing comment already reasoned about the lower bound (don't mark the
backend available on SM90/H100 where it "fails at dispatch"). Apply the same
reasoning to the upper bound by gating on the explicit SM100-family set
`arch in (100, 103, 110)` (the trtllm-gen MoE cubin set; matches the idiom in
CuteDslConfig.supported). SM100/SM103/SM110 behavior is unchanged.
Verified on RTX PRO 6000 (SM120, CUDA 13.0):
before: supported(120)=True; MoELayer(NVFP4) selects the trtllm runner ->
SM100 cubins -> dispatch cubin-miss.
after: supported = {100:T, 103:T, 110:T, 120:F, 121:F}; MoELayer(NVFP4)
raises a clean RuntimeError at construction on SM120.
Note: the sibling trtllm-gen configs (TrtllmFp8Block/PerTensor/Bf16/MxInt4)
share the same `arch >= 100` pattern and latent upper-bound gap, but are not
reachable in the NVFP4-only MVP; left for a follow-up to keep this focused.
AI-assisted: investigated and authored with Claude Code.
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: WEI CHENG CHIU <waynehacking8@gmail.com>
448303f to
995afb7
Compare
|
Thanks @gemini-code-assist — agreed, applied |
|
Thank you for the verification. Using an explicit whitelist for the |
|
Closing as superseded by current main. |
2 participants
Description
TrtllmFp4Config.supported()returnedTruefor allarch >= 100, but the routed runner (TrtllmFp4RoutedRunner) delegates to the trtllm-gen sm100 module, whose cubins are SM100-family only (SM100/SM103/SM110). On SM120/SM121 (consumer/workstation Blackwell) there are no trtllm-gen MoE cubins, soMoELayerselected and built aTrtllmFp4RoutedRunnerthat then failed deep at dispatch (cubin-not-found) instead of raising the clean "no usable backend" error at construction.The existing comment already reasoned about the lower bound (don't mark the backend available on SM90/H100 where it "fails at dispatch"). This PR extends the same reasoning to the upper bound: gate on
100 <= arch < 120. SM100/SM103/SM110 behavior is unchanged.Related Issues
None — found while auditing SM120 MoE dispatch on an RTX PRO 6000.
Tests
Verified on an RTX PRO 6000 Blackwell (SM120, CUDA 13.0) with a minimal
MoEConfig(QuantVariant.NVFP4):supported(120)supported(100)/(110)TrueTrueTrtllmFp4RoutedRunner→ SM100 cubins → dispatch cubin-missFalseTrue(unchanged)RuntimeError: none of the configured backends are usable on sm120at constructionpre-commit run --filespasses (ruff check, ruff format, mypy, codespell).Reviewer Notes
b12x_fused_moepath; the unifiedMoELayerMVP remains SM100-family for now. Full unified-API SM120 enablement (a CUTLASS/b12x NVFP4 runner mapped into_BACKEND_RUNNERS) is a larger change that overlaps the NVFP4-scale plumbing in fix: make the cutlass MoE gemm profiler MXFP8-aware (autotune crash on MXFP8xMXFP8) #3614.TrtllmFp8Block/PerTensor/Bf16/MxInt4) share the identicalarch >= 100upper-bound gap but are unreachable in the NVFP4-only MVP; left for a focused follow-up.Summary by CodeRabbit