Fix Gemma4 NVFP4 MoE default attention backend

[mmangkad]

Summary

I noticed that nvidia/Gemma-4-26B-A4B-NVFP4 is fully broken with the current SM10X default trtllm_mha backend. This is not just a small regression: MMLU drops from 0.622 with triton to 0.037 with the default, and the outputs look nonsensical. This PR changes only the affected Gemma4 NVFP4 MoE default to triton; users can still explicitly pass --attention-backend trtllm_mha.

Accuracy Comparison

Model	Quant	MoE	Attention backend	MMLU avg
nvidia/Gemma-4-26B-A4B-NVFP4	NVFP4	yes	trtllm_mha default	0.037
nvidia/Gemma-4-26B-A4B-NVFP4	NVFP4	yes	triton	0.622
google/gemma-4-26B-A4B-it	BF16	yes	trtllm_mha default	0.627
nvidia/Gemma-4-31B-IT-NVFP4	NVFP4	no	trtllm_mha default	0.681

NVFP4 MoE, Default TRTLLM MHA Attention

CUDA_VISIBLE_DEVICES=2 sglang serve --model-path nvidia/Gemma-4-26B-A4B-NVFP4 --trust-remote-code

python bench_sglang.py --parallel 512

100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:27<00:00, 514.40it/s]
subject: abstract_algebra, #q:100, acc: 0.010
subject: anatomy, #q:135, acc: 0.015
subject: astronomy, #q:152, acc: 0.013
subject: business_ethics, #q:100, acc: 0.160
subject: clinical_knowledge, #q:265, acc: 0.011
subject: college_biology, #q:144, acc: 0.062
subject: college_chemistry, #q:100, acc: 0.020
subject: college_computer_science, #q:100, acc: 0.010
subject: college_mathematics, #q:100, acc: 0.060
subject: college_medicine, #q:173, acc: 0.017
subject: college_physics, #q:102, acc: 0.029
subject: computer_security, #q:100, acc: 0.140
subject: conceptual_physics, #q:235, acc: 0.128
subject: econometrics, #q:114, acc: 0.105
subject: electrical_engineering, #q:145, acc: 0.014
subject: elementary_mathematics, #q:378, acc: 0.026
subject: formal_logic, #q:126, acc: 0.056
subject: global_facts, #q:100, acc: 0.000
subject: high_school_biology, #q:310, acc: 0.032
subject: high_school_chemistry, #q:203, acc: 0.025
subject: high_school_computer_science, #q:100, acc: 0.050
subject: high_school_european_history, #q:165, acc: 0.042
subject: high_school_geography, #q:198, acc: 0.051
subject: high_school_government_and_politics, #q:193, acc: 0.098
subject: high_school_macroeconomics, #q:390, acc: 0.023
subject: high_school_mathematics, #q:270, acc: 0.004
subject: high_school_microeconomics, #q:238, acc: 0.038
subject: high_school_physics, #q:151, acc: 0.033
subject: high_school_psychology, #q:545, acc: 0.031
subject: high_school_statistics, #q:216, acc: 0.042
subject: high_school_us_history, #q:204, acc: 0.020
subject: high_school_world_history, #q:237, acc: 0.059
subject: human_aging, #q:223, acc: 0.126
subject: human_sexuality, #q:131, acc: 0.023
subject: international_law, #q:121, acc: 0.190
subject: jurisprudence, #q:108, acc: 0.056
subject: logical_fallacies, #q:163, acc: 0.018
subject: machine_learning, #q:112, acc: 0.045
subject: management, #q:103, acc: 0.194
subject: marketing, #q:234, acc: 0.009
subject: medical_genetics, #q:100, acc: 0.030
subject: miscellaneous, #q:783, acc: 0.091
subject: moral_disputes, #q:346, acc: 0.066
subject: moral_scenarios, #q:895, acc: 0.011
subject: nutrition, #q:306, acc: 0.098
subject: philosophy, #q:311, acc: 0.013
subject: prehistory, #q:324, acc: 0.037
subject: professional_accounting, #q:282, acc: 0.082
subject: professional_law, #q:1534, acc: 0.003
subject: professional_medicine, #q:272, acc: 0.000
subject: professional_psychology, #q:612, acc: 0.000
subject: public_relations, #q:110, acc: 0.000
subject: security_studies, #q:245, acc: 0.000
subject: sociology, #q:201, acc: 0.000
subject: us_foreign_policy, #q:100, acc: 0.000
subject: virology, #q:166, acc: 0.000
subject: world_religions, #q:171, acc: 0.000
Total latency: 27.341
Average accuracy: 0.037

NVFP4 MoE, Triton Attention

CUDA_VISIBLE_DEVICES=2 sglang serve --model-path nvidia/Gemma-4-26B-A4B-NVFP4 --attention-backend triton --trust-remote-code

python bench_sglang.py --parallel 512

100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:27<00:00, 512.88it/s]
subject: abstract_algebra, #q:100, acc: 0.380
subject: anatomy, #q:135, acc: 0.719
subject: astronomy, #q:152, acc: 0.697
subject: business_ethics, #q:100, acc: 0.580
subject: clinical_knowledge, #q:265, acc: 0.585
subject: college_biology, #q:144, acc: 0.708
subject: college_chemistry, #q:100, acc: 0.490
subject: college_computer_science, #q:100, acc: 0.600
subject: college_mathematics, #q:100, acc: 0.510
subject: college_medicine, #q:173, acc: 0.590
subject: college_physics, #q:102, acc: 0.549
subject: computer_security, #q:100, acc: 0.640
subject: conceptual_physics, #q:235, acc: 0.379
subject: econometrics, #q:114, acc: 0.456
subject: electrical_engineering, #q:145, acc: 0.690
subject: elementary_mathematics, #q:378, acc: 0.519
subject: formal_logic, #q:126, acc: 0.659
subject: global_facts, #q:100, acc: 0.290
subject: high_school_biology, #q:310, acc: 0.832
subject: high_school_chemistry, #q:203, acc: 0.645
subject: high_school_computer_science, #q:100, acc: 0.810
subject: high_school_european_history, #q:165, acc: 0.739
subject: high_school_geography, #q:198, acc: 0.667
subject: high_school_government_and_politics, #q:193, acc: 0.788
subject: high_school_macroeconomics, #q:390, acc: 0.674
subject: high_school_mathematics, #q:270, acc: 0.448
subject: high_school_microeconomics, #q:238, acc: 0.811
subject: high_school_physics, #q:151, acc: 0.550
subject: high_school_psychology, #q:545, acc: 0.767
subject: high_school_statistics, #q:216, acc: 0.750
subject: high_school_us_history, #q:204, acc: 0.814
subject: high_school_world_history, #q:237, acc: 0.886
subject: human_aging, #q:223, acc: 0.511
subject: human_sexuality, #q:131, acc: 0.733
subject: international_law, #q:121, acc: 0.736
subject: jurisprudence, #q:108, acc: 0.704
subject: logical_fallacies, #q:163, acc: 0.669
subject: machine_learning, #q:112, acc: 0.536
subject: management, #q:103, acc: 0.650
subject: marketing, #q:234, acc: 0.654
subject: medical_genetics, #q:100, acc: 0.590
subject: miscellaneous, #q:783, acc: 0.710
subject: moral_disputes, #q:346, acc: 0.587
subject: moral_scenarios, #q:895, acc: 0.550
subject: nutrition, #q:306, acc: 0.627
subject: philosophy, #q:311, acc: 0.527
subject: prehistory, #q:324, acc: 0.753
subject: professional_accounting, #q:282, acc: 0.518
subject: professional_law, #q:1534, acc: 0.561
subject: professional_medicine, #q:272, acc: 0.596
subject: professional_psychology, #q:612, acc: 0.567
subject: public_relations, #q:110, acc: 0.609
subject: security_studies, #q:245, acc: 0.661
subject: sociology, #q:201, acc: 0.557
subject: us_foreign_policy, #q:100, acc: 0.780
subject: virology, #q:166, acc: 0.398
subject: world_religions, #q:171, acc: 0.480
Total latency: 27.425
Average accuracy: 0.622

BF16 MoE, Default TRTLLM MHA Attention

CUDA_VISIBLE_DEVICES=2 sglang serve --model-path google/gemma-4-26B-A4B-it --trust-remote-code

python bench_sglang.py --parallel 512

100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [00:38<00:00, 361.65it/s]
subject: abstract_algebra, #q:100, acc: 0.400
subject: anatomy, #q:135, acc: 0.719
subject: astronomy, #q:152, acc: 0.632
subject: business_ethics, #q:100, acc: 0.720
subject: clinical_knowledge, #q:265, acc: 0.577
subject: college_biology, #q:144, acc: 0.639
subject: college_chemistry, #q:100, acc: 0.490
subject: college_computer_science, #q:100, acc: 0.640
subject: college_mathematics, #q:100, acc: 0.500
subject: college_medicine, #q:173, acc: 0.572
subject: college_physics, #q:102, acc: 0.471
subject: computer_security, #q:100, acc: 0.630
subject: conceptual_physics, #q:235, acc: 0.549
subject: econometrics, #q:114, acc: 0.509
subject: electrical_engineering, #q:145, acc: 0.752
subject: elementary_mathematics, #q:378, acc: 0.606
subject: formal_logic, #q:126, acc: 0.643
subject: global_facts, #q:100, acc: 0.320
subject: high_school_biology, #q:310, acc: 0.845
subject: high_school_chemistry, #q:203, acc: 0.611
subject: high_school_computer_science, #q:100, acc: 0.810
subject: high_school_european_history, #q:165, acc: 0.794
subject: high_school_geography, #q:198, acc: 0.758
subject: high_school_government_and_politics, #q:193, acc: 0.772
subject: high_school_macroeconomics, #q:390, acc: 0.656
subject: high_school_mathematics, #q:270, acc: 0.385
subject: high_school_microeconomics, #q:238, acc: 0.836
subject: high_school_physics, #q:151, acc: 0.490
subject: high_school_psychology, #q:545, acc: 0.809
subject: high_school_statistics, #q:216, acc: 0.704
subject: high_school_us_history, #q:204, acc: 0.799
subject: high_school_world_history, #q:237, acc: 0.890
subject: human_aging, #q:223, acc: 0.596
subject: human_sexuality, #q:131, acc: 0.725
subject: international_law, #q:121, acc: 0.752
subject: jurisprudence, #q:108, acc: 0.657
subject: logical_fallacies, #q:163, acc: 0.693
subject: machine_learning, #q:112, acc: 0.580
subject: management, #q:103, acc: 0.670
subject: marketing, #q:234, acc: 0.701
subject: medical_genetics, #q:100, acc: 0.620
subject: miscellaneous, #q:783, acc: 0.713
subject: moral_disputes, #q:346, acc: 0.610
subject: moral_scenarios, #q:895, acc: 0.571
subject: nutrition, #q:306, acc: 0.660
subject: philosophy, #q:311, acc: 0.502
subject: prehistory, #q:324, acc: 0.769
subject: professional_accounting, #q:282, acc: 0.518
subject: professional_law, #q:1534, acc: 0.565
subject: professional_medicine, #q:272, acc: 0.621
subject: professional_psychology, #q:612, acc: 0.417
subject: public_relations, #q:110, acc: 0.536
subject: security_studies, #q:245, acc: 0.710
subject: sociology, #q:201, acc: 0.478
subject: us_foreign_policy, #q:100, acc: 0.670
subject: virology, #q:166, acc: 0.367
subject: world_religions, #q:171, acc: 0.602
Total latency: 38.874
Average accuracy: 0.627

NVFP4 Dense, Default TRTLLM MHA Attention

CUDA_VISIBLE_DEVICES=2 sglang serve --model-path nvidia/Gemma-4-31B-IT-NVFP4 --trust-remote-code

python bench_sglang.py --parallel 512

100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14042/14042 [01:23<00:00, 168.80it/s]
subject: abstract_algebra, #q:100, acc: 0.460
subject: anatomy, #q:135, acc: 0.667
subject: astronomy, #q:152, acc: 0.796
subject: business_ethics, #q:100, acc: 0.690
subject: clinical_knowledge, #q:265, acc: 0.675
subject: college_biology, #q:144, acc: 0.799
subject: college_chemistry, #q:100, acc: 0.550
subject: college_computer_science, #q:100, acc: 0.760
subject: college_mathematics, #q:100, acc: 0.560
subject: college_medicine, #q:173, acc: 0.607
subject: college_physics, #q:102, acc: 0.510
subject: computer_security, #q:100, acc: 0.700
subject: conceptual_physics, #q:235, acc: 0.596
subject: econometrics, #q:114, acc: 0.632
subject: electrical_engineering, #q:145, acc: 0.621
subject: elementary_mathematics, #q:378, acc: 0.619
subject: formal_logic, #q:126, acc: 0.651
subject: global_facts, #q:100, acc: 0.320
subject: high_school_biology, #q:310, acc: 0.839
subject: high_school_chemistry, #q:203, acc: 0.709
subject: high_school_computer_science, #q:100, acc: 0.880
subject: high_school_european_history, #q:165, acc: 0.855
subject: high_school_geography, #q:198, acc: 0.763
subject: high_school_government_and_politics, #q:193, acc: 0.912
subject: high_school_macroeconomics, #q:390, acc: 0.733
subject: high_school_mathematics, #q:270, acc: 0.404
subject: high_school_microeconomics, #q:238, acc: 0.853
subject: high_school_physics, #q:151, acc: 0.563
subject: high_school_psychology, #q:545, acc: 0.835
subject: high_school_statistics, #q:216, acc: 0.764
subject: high_school_us_history, #q:204, acc: 0.887
subject: high_school_world_history, #q:237, acc: 0.840
subject: human_aging, #q:223, acc: 0.502
subject: human_sexuality, #q:131, acc: 0.687
subject: international_law, #q:121, acc: 0.851
subject: jurisprudence, #q:108, acc: 0.593
subject: logical_fallacies, #q:163, acc: 0.785
subject: machine_learning, #q:112, acc: 0.652
subject: management, #q:103, acc: 0.796
subject: marketing, #q:234, acc: 0.842
subject: medical_genetics, #q:100, acc: 0.780
subject: miscellaneous, #q:783, acc: 0.679
subject: moral_disputes, #q:346, acc: 0.613
subject: moral_scenarios, #q:895, acc: 0.658
subject: nutrition, #q:306, acc: 0.745
subject: philosophy, #q:311, acc: 0.614
subject: prehistory, #q:324, acc: 0.762
subject: professional_accounting, #q:282, acc: 0.589
subject: professional_law, #q:1534, acc: 0.568
subject: professional_medicine, #q:272, acc: 0.798
subject: professional_psychology, #q:612, acc: 0.647
subject: public_relations, #q:110, acc: 0.582
subject: security_studies, #q:245, acc: 0.735
subject: sociology, #q:201, acc: 0.652
subject: us_foreign_policy, #q:100, acc: 0.860
subject: virology, #q:166, acc: 0.452
subject: world_religions, #q:171, acc: 0.696
Total latency: 83.233
Average accuracy: 0.681

---

CI States

Latest PR Test (Base): ✅ Run #26838018967
Latest PR Test (Extra): ❌ Run #26838018455

[gemini-code-assist]

Code Review

This pull request refactors _handle_model_specific_adjustments in python/sglang/srt/server_args.py to cache model_config and introduces logic to detect Gemma4 models with modelopt_fp4 quantization and MoE enabled. For these models, the default attention backend is temporarily set to "triton" instead of "trtllm_mha" on SM100-supported hardware to avoid a known accuracy issue. The reviewer suggested explicitly casting the retrieved enable_moe_block attribute to a boolean to ensure robust logical checks.

[mmangkad]

cc @pyc96 I saw you made this default in #25054

[pyc96]

Can you share your flashinfer version and the output of python3 -m sglang.check_env?

I also noticed some regression on quality for MoE model but not as much as yours. #26518

cc @nvpohanh @wenscarl

[mmangkad]

Can you share your flashinfer version and the output of python3 -m sglang.check_env?

I also noticed some regression on quality for MoE model but not as much as yours. #26518

cc @nvpohanh @wenscarl

I'm currently AFK, so I can't grab the check_env output right now.

For context on my run, I used the unmodified dependencies from main with FlashInfer 0.6.11.post1 (and upgrading to 0.6.12 didn't fix it). The main difference is that I tested this on a GB300 (SM103) instead of an SM100.

[pyc96]

/tag-and-rerun-ci

[pyc96]

I think it makes sense to switch back to Triton given multiple reports of quality degradation with nvfp4 moe + trtllm_mha

[nvpohanh]

Makes sense. But could you file a github issue for the accuracy issue so that I can ask @wenscarl to debug it? thanks!

[pyc96]

@nvpohanh Sure, we can reuse #26518 for tracking. Thank you!

[mmangkad]

Merging for now until we have a clearer understanding of the issue and a verified fix in place