[AMD] Qwen3.5 MXFP4 breaks after shared expert fusion is enabled

[mqhc2020]

Motivation

After shared expert fusion is enabled for Qwen3.5 models (as in #20736 ), MXFP4 model hits an issue: the shared expert in the checkpoint is based on BF16 but current weight loading can only treat it as MXFP4 which is the dtype of routed experts. So before either online quantization is ready, or shared expert in MXFP4 models has been pre-quanted as MXFP4,
we have to skip shared expert fusion feature for MXFP4 model.

Modifications

In qwen2_moe.py, disable fusion when the shared experts are detected:

• "model.language_model.layers.xxx.mlp.shared_expert.down_proj"
• "model.language_model.layers.xxx.mlp.shared_expert.gate_proj"
• "model.language_model.layers.xxx.mlp.shared_expert.up_proj"

Note that the following layers are irrelevant:

• model.language_model.layers.xxx.mlp.shared_expert_gate"
• mtp.layers.xxx.mlp.shared_expert*"

Accuracy Tests

Speed Tests and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

[gemini-code-assist]

Code Review

This pull request updates the can_fuse_shared_expert function in qwen2_moe.py to prevent the fusion of shared experts when they are explicitly excluded from quantization. This logic ensures that FP32 shared experts are not incorrectly fused into quantized MoE weight tensors, which would require unsupported online quantization. Feedback was provided to correct the type hint for the quant_config parameter from None to Optional[QuantizationConfig] to accurately reflect its usage and maintain consistency with the rest of the codebase.

[hubertlu-tw]

@mqhc2020 Thanks for the fix.
@yctseng0211 do we have capacity to add a 4-GPU e2e test to our CI for this model?

[yctseng0211]

@mqhc2020 Thanks for the fix. @yctseng0211 do we have capacity to add a 4-GPU e2e test to our CI for this model?

@hubertlu-tw yeah I think we can add it in amd Nightly Test. If we get more capacity then we can move it to pr-test if needed.

[BowenBao]

can we add a method can_fuse_shared_expert to QuantConfig and implement this check inside QuarkConfig? That way we keep quark specific logic within quark.

and a more precise check is probably checking if shared_expert share the same quantization spec with moe layers.

[HaiShaw]

@mqhc2020 approve now, but look for refactor is feasible