feat(moe): support GeGLU and SiTU in W4A16

[ziang-and]

📌 Description

@HumansAnd

• Extend CuTe DSL W4A16 fused MoE with tanh-approximate GeGLU and SiTU.
• Reuse the activation math and validation introduced for W4A4 in feat(moe): add CuTe-DSL GeGLU-tanh and SiTU activations #4009:
  • GeGLU-tanh: up * GELU(gate, approximate="tanh").
  • SiTU: ActivationType.Swiglu plus situ_beta, with optional situ_linear_beta for the up-branch clamp.
• Apply each expert's W1 alpha before activation, matching the existing W4A4 contract.
• Include both SiTU parameters in autotuner and compiled-kernel cache identities.
• Keep activation in GEMM1; GEMM2 routing-weight reduction and fused finalize are unchanged.
• Extend existing Cartesian tests across W4A4 per-tensor, W4A4 per-token, and W4A16 modes; no new standalone test functions are added.

🔍 Related Issues

• feat(moe): add CuTe-DSL GeGLU-tanh and SiTU activations #4009
• feat(moe): support BF16 activations in CuTe DSL NVFP4 MoE #4048

🚀 Pull Request Checklist

Thank you for contributing to FlashInfer! Before we review your pull request, please make sure the following items are complete.

✅ Pre-commit Checks

• I have installed pre-commit by running pip install pre-commit (or used your preferred method).
• I have installed the hooks with pre-commit install.
• I have run the hooks manually with pre-commit run --all-files and fixed any reported issues.

If you are unsure about how to set up pre-commit, see the pre-commit documentation.

🧪 Tests

• Tests have been added or updated as needed.
• All tests are passing (unittest, etc.).

B200 source-only validation with lmsysorg/sglang:nightly-dev-cu13-20260806-ae5f8c94:

• pre-commit run --all-files — passed.
• GeGLU/SiTU functional and cache-key matrix — 11 passed.
• W4A16 GeGLU/SiTU wrapper-autotune rows — 2 passed, 6 deselected.
• moe_utils is_aot: False; editable-install data links resolved inside the synced source checkout.

Reviewer Notes

• SiTU intentionally remains ActivationType.Swiglu plus its beta parameters, matching feat(moe): add CuTe-DSL GeGLU-tanh and SiTU activations #4009.
• Scope follows feat(moe): add CuTe-DSL GeGLU-tanh and SiTU activations #4009's functional and wrapper APIs; the unified CuteDslNvfp4Runner activation configuration is unchanged.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: d0d9d7fd-aff4-4fb0-8226-ccad48e4bb6a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}