cherry-pick: #4180 ([feat] Add SITU trtllmgen MOE) + conflict resolution by jimmyzho · Pull Request #4252 · flashinfer-ai/flashinfer

jimmyzho · 2026-07-29T23:39:23Z

Add SiTU activation support for TRTLLM-Gen MoE for MXFP4 × MXFP8 and NVFP4 × NVFP4

Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete.

I have installed pre-commit by running pip install pre-commit (or used your preferred method).
I have installed the hooks with pre-commit install.
I have run the hooks manually with pre-commit run --all-files and fixed any reported issues.

If you are unsure about how to set up pre-commit, see the
pre-commit documentation.

Tests have been added or updated as needed.
All tests are passing (unittest, etc.).

New Features
Added SiTU gated activation support for FP4 MoE in routed and non-routed execution, with configurable per-expert gemm1_alpha, gemm1_beta, and gemm1_clamp_limit.
Extended FP4 MoE trace templates with an activation_type input to select SiTU.
Bug Fixes
Improved activation-type mapping/validation so SiTU uses the correct gated behavior.
Made MoE kernel/tile selection dtype-aware (activation + weight) and corrected per-token scaling eligibility during config generation.
Tests
Expanded MoE coverage for SiTU and optional per-expert parameters, including additional routed-logit and autotune regression checks.
Chores
- Updated the remote BMM artifact reference and checksum.

(cherry picked from commit a34a735)

📌 Description

🔍 Related Issues

🚀 Pull Request Checklist

Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete.

✅ Pre-commit Checks

I have installed pre-commit by running pip install pre-commit (or used your preferred method).
I have installed the hooks with pre-commit install.
I have run the hooks manually with pre-commit run --all-files and fixed any reported issues.

If you are unsure about how to set up pre-commit, see the pre-commit documentation.

🧪 Tests

Tests have been added or updated as needed.
All tests are passing (unittest, etc.).

Reviewer Notes

Add SiTU activation support for TRTLLM-Gen MoE for MXFP4 × MXFP8 and NVFP4 × NVFP4  Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). - [x] Tests have been added or updated as needed. - [x] All tests are passing (`unittest`, etc.).   * **New Features** * Added SiTU gated activation support for FP4 MoE in routed and non-routed execution, with configurable per-expert `gemm1_alpha`, `gemm1_beta`, and `gemm1_clamp_limit`. * Extended FP4 MoE trace templates with an `activation_type` input to select SiTU. * **Bug Fixes** * Improved activation-type mapping/validation so SiTU uses the correct gated behavior. * Made MoE kernel/tile selection dtype-aware (activation + weight) and corrected per-token scaling eligibility during config generation. * **Tests** * Expanded MoE coverage for SiTU and optional per-expert parameters, including additional routed-logit and autotune regression checks. * **Chores** * Updated the remote BMM artifact reference and checksum.  --------- Signed-off-by: Siyuan Fu <siyuanf@nvidia.com> Co-authored-by: Siyuan Fu <siyuanf@nvidia.com> (cherry picked from commit a34a735)

gemini-code-assist · 2026-07-29T23:39:26Z

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

coderabbitai · 2026-07-29T23:39:30Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 8ed7d894-c1dd-4d90-9cea-c9a440c6551b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

… package The Rubin fused-MoE module (fused_moe_trtllm_sm107, compiled with -DTLLM_RUBIN_FEATURES) builds against TRTLLM_GEN_BMM_RUBIN, which predates this feature: - Its gemmGatedAct::ActType is {SwiGlu, GeGlu, None} with None == 2, so the SiTuGlu/None static_asserts in KernelRunner.h fail name lookup and break every Rubin fused-MoE build, not only SITU. - It has no 192-tile FP4 kernels (its only 192-tile kernels are FP8). Launchers are built eagerly for every advertised tile, so advertising 192 makes FP4BlockScaleLauncher construction throw for every NVFP4xNVFP4 / MXFP4xMXFP8 MoE call on Rubin. Guard both behind TLLM_RUBIN_FEATURES; remove the guards together when the Rubin pin is bumped to a package that carries SiTuGlu and 192-tile FP4 kernels. SITU requests on Rubin fail with the existing clean no-valid-configs error.

## 📌 Description This PR relands SM 107 support to main branch (reverted in #4171) as well as some other release fixes. #### Cherry Picks - #4191 - #4189 - #4200 - #4215 - #4225 - #4230 - #4235 - #4226 - #4257 - #4258 - #4261 #### Other Changes - Rubin guards from #4252's conflict resolution (`TLLM_RUBIN_FEATURES`: SiTuGlu static_asserts + tile-192 advertisement, compiled out for the Rubin BMM pin) - Test-contract update: `test_unified_moe.py` arch assertions written post-revert (#4159) flipped to the restored contract (FP4/BF16 claim 107; FP8 stays 100/103)  ## 🔍 Related Issues  #4107, #4164, reverts #4171 ## 🚀 Pull Request Checklist Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete. ### ✅ Pre-commit Checks - [x] I have installed `pre-commit` by running `pip install pre-commit` (or used your preferred method). - [x] I have installed the hooks with `pre-commit install`. - [x] I have run the hooks manually with `pre-commit run --all-files` and fixed any reported issues. > If you are unsure about how to set up `pre-commit`, see [the pre-commit documentation](https://pre-commit.com/). ## 🧪 Tests - [ ] Tests have been added or updated as needed. - [ ] All tests are passing (`unittest`, etc.). ## Reviewer Notes   ## Summary by CodeRabbit * **New Features** * Added support for Rubin/SM107 GPUs across GEMM, MoE, attention, quantization, sampling, and DeepGEMM workflows. * Added architecture-aware kernel selection, memory sizing, compilation, and artifact handling. * **Bug Fixes** * Improved validation and error messages for incompatible GPU architectures and invalid kernel configurations. * Clearly rejects unsupported NVFP4 KV-cache operations on SM107. * **Documentation** * Updated installation guidance with the SM107 architecture target. * **Tests** * Expanded architecture coverage and compatibility checks across GPU test suites.  --------- Co-authored-by: Vinnie6167 <Vinnie6167@users.noreply.github.com> Co-authored-by: Ka-Hyun Nam <knam@nvidia.com> Co-authored-by: Alex Yang <aleyang@nvidia.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Jimmy Zhou <79552142+jimmyzho@users.noreply.github.com>

jimmyzho requested review from Aneureka, IwakuraRein, aleozlx, bkryu, cyx-6, dhiraj113, feih-nv, jiahanc, kahyunnam, nv-yunzheq, qiching, qsang-nv, saltyminty, samuellees, sricketts, yongwww, yyihuang and yzh119 as code owners July 29, 2026 23:39

flashinfer-bot added the op: moe label Jul 29, 2026

jimmyzho changed the title ~~cherry-pick: #4180 ([feat] Add SITU trtllmgen MOE)~~ cherry-pick: #4180 ([feat] Add SITU trtllmgen MOE) + conflict resolution Jul 30, 2026

jimmyzho merged commit c4053a2 into release-v0.6.16 Jul 30, 2026
5 checks passed

jimmyzho deleted the cherry-pick-4180-release-v0.6.16 branch July 30, 2026 03:54

Vinnie6167 mentioned this pull request Jul 30, 2026

SM 107 Reland + Merge Back from v0.6.16 Release Branch #4280

Merged

5 tasks

This was referenced Aug 12, 2026

[Regression][SM100/SM103] TRTLLM NVFP4 MoE emits NaNs in v0.6.16.post4/v0.6.17 (v0.6.16rc4 passes) #4486

Closed

[Regression][SM100/SM103] FlashInfer TRTLLM NVFP4 MoE tile-192 path produces NaNs sgl-project/sglang#34629

Closed

Navigation Menu

Uh oh!

cherry-pick: #4180 ([feat] Add SITU trtllmgen MOE) + conflict resolution - #4252

cherry-pick: #4180 ([feat] Add SITU trtllmgen MOE) + conflict resolution#4252
jimmyzho merged 2 commits into
release-v0.6.16from
cherry-pick-4180-release-v0.6.16

jimmyzho commented Jul 29, 2026

Uh oh!

gemini-code-assist Bot commented Jul 29, 2026

Uh oh!

coderabbitai Bot commented Jul 29, 2026 •

edited

Loading

Review skipped

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

jimmyzho commented Jul 29, 2026

📌 Description

🔍 Related Issues

🚀 Pull Request Checklist

✅ Pre-commit Checks

🧪 Tests

Reviewer Notes

Uh oh!

gemini-code-assist Bot commented Jul 29, 2026

Uh oh!

coderabbitai Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

coderabbitai Bot commented Jul 29, 2026 •

edited

Loading