fix: route Thor SM110 FP8 and ModelOpt NVFP4 auto backends

[janbernloehr]

Motivation

Fixes #41764.

On NVIDIA Thor (SM110), SGLang's automatic backend choices prevent Qwen3.6-35B-A3B server startup:

• FP8 enters DeepGEMM JIT warmup and fails with /deepgemm/csrc/apis/gemm.hpp:220: Unsupported architecture.
• ModelOpt NVFP4 MoE enters the TensorRT-LLM path and fails with AttributeError: 'FusedMoE' object has no attribute 'g1_scale_c'.

Both failures reproduced on SGLang g79cafec01 with the backend flags left at auto. These jobs already selected FlashInfer attention; this patch addresses the remaining GEMM and MoE routing failures, separate from the earlier attention restriction in #35080.

Modifications

• Add an SM110 platform probe requiring CUDA 12.8 or newer.
• Route automatic FP8 GEMM selection on SM110 to Triton instead of DeepGEMM. Preserve explicit backend choices and the existing SM120 CUTLASS route.
• Route automatic ModelOpt FP4 MoE selection on SM110 to flashinfer_cutlass, extending the existing SM120 rule. Preserve explicit backend choices and other quantization schemes.
• Add CPU regression tests for the platform probe and the automatic/explicit routing boundaries.

Implementation: janbernloehr:fix/sm110-auto-backend-routing, commit 8fdf1be4e8c706c849085f0085671f7a31efaac9.

Accuracy Tests

The Thor validation image applied the four production routing edits from the linked commit to installed SGLang g79cafec01, retaining its matching Rust TreeCore extension. All three single-GPU Thor accuracy workloads completed successfully with automatic backend selection. Recorded gsm8k_10perc scores (132 examples):

Model variant	Score	Correct
Qwen3.5-27B FP8	0.795455	105/132
Qwen3.6-35B-A3B FP8	0.454545	60/132
Qwen3.6-35B-A3B NVFP4	0.151515	20/132

An earlier prototype containing the same SM110 GEMM/MoE routing plus a subsequently retired attention override also completed its accuracy pipeline (4/4 jobs) at SGLang g06008c170. Its Thor scores were 0.795455 (105/132) for Qwen3.5-27B FP8, 0.515152 (68/132) for Qwen3.6-35B-A3B FP8, and 0.121212 (16/132) for Qwen3.6-35B-A3B NVFP4. These are separate validation runs on different SGLang revisions, not an accuracy A/B comparison.

Speed Tests and Profiling

Both Qwen3.6-35B-A3B throughput workloads completed on the Thor validation image with the standard auto backend selections. Recorded average output-token throughput at concurrency 1:

Model variant	ISL 2,000 / OSL 128	ISL 2,000 / OSL 1,000, 32,000-token prefix
FP8	48.974 tokens/s	43.575 tokens/s
NVFP4	60.789 tokens/s	54.146 tokens/s

For an earlier controlled A/B at SGLang g0f6761b54, the unpatched Thor pipeline failed at startup, while the prototype containing this routing change and the now-retired attention override passed 3/3 jobs. The patched prototype recorded 48.362 and 31.309 tokens/s for FP8, and 60.572 and 43.253 tokens/s for NVFP4, for the same two benchmark shapes above. A second patched pipeline at SGLang g582389cec also passed 3/3 jobs.

Checklist

• The linked commit includes CPU regression coverage for SM110 detection, FP8 and ModelOpt FP4 automatic routing, and preservation of explicit backend choices.
• The Thor image build passed the fork regression suite (3/3) and installed-package routing checks (3/3).
• Python 3.12 compileall passed for all five changed files; targeted Ruff checks (F401,F821,F823,UP037), the new test file's Ruff formatting check, and git diff --check passed.
• Thor accuracy and speed results are reported above.
• No documentation change is needed for this automatic backend-selection correction.

Review and Merge Process

Follow the repository's PR merge process: request Merge Oncall and CODEOWNERS review, trigger upstream CI, and merge after its required checks and approvals.

---

CI States

Latest PR Test (Base): ❌ Run #37276146329
Latest PR Test (Extra): ❌ Run #37276145974
Latest PR Test (AMD ROCm 10): ❌ Run #37276146143

[nvpohanh]

/rerun-test test/registered/unit/test_platform_context.py test/registered/unit/test_model_overrides.py test/registered/e2e/quantization/test_nvfp4_gemm_sm120.py test/registered/moe/test_glm4_moe_models.py test/registered/unit/layers/quantization/test_fp8_blockwise_linear_backends.py test/registered/unit/test_mxfp8_moe_runner_backend.py test/registered/e2e/quantization/test_modelopt_fp8.py test/registered/e2e/models/test_deepseek_v3_fp4.py

[github-actions]

⚠️ Rebase Required Before Re-run

A major update has landed on main. Your PR is diverged relative to required base commit 8854857a991e.

Re-run was not dispatched. What to do:

• Rebase your branch onto the latest main and push again
• Follow issue #21065 for context
• CI-fix PRs may request the bypass-maintenance label to skip this check

[nvpohanh]

/rerun-test test/registered/unit/test_platform_context.py test/registered/unit/test_model_overrides.py test/registered/e2e/quantization/test_nvfp4_gemm_sm120.py test/registered/moe/test_glm4_moe_models.py test/registered/unit/layers/quantization/test_fp8_blockwise_linear_backends.py test/registered/unit/test_mxfp8_moe_runner_backend.py test/registered/e2e/quantization/test_modelopt_fp8.py test/registered/e2e/models/test_deepseek_v3_fp4.py

[github-actions]

⚠️ Rebase Required Before Re-run

A major update has landed on main. Your PR is diverged relative to required base commit fea3c088a141.

Re-run was not dispatched. What to do:

• Rebase your branch onto the latest main and push again
• Follow issue #21065 for context
• CI-fix PRs may request the bypass-maintenance label to skip this check

[nvpohanh]

/rerun-test test/registered/unit/test_platform_context.py test/registered/unit/test_model_overrides.py test/registered/e2e/quantization/test_nvfp4_gemm_sm120.py test/registered/moe/test_glm4_moe_models.py test/registered/unit/layers/quantization/test_fp8_blockwise_linear_backends.py test/registered/unit/test_mxfp8_moe_runner_backend.py test/registered/e2e/quantization/test_modelopt_fp8.py test/registered/e2e/models/test_deepseek_v3_fp4.py

[github-actions]

🚀 ubuntu-latest (3 tests): ✅ View workflow run

cd test/ && python3 registered/unit/test_platform_context.py
cd test/ && python3 registered/unit/test_model_overrides.py
cd test/ && python3 registered/unit/test_mxfp8_moe_runner_backend.py

🚀 1-gpu-5090 (2 tests): ✅ View workflow run

cd test/ && python3 registered/e2e/quantization/test_nvfp4_gemm_sm120.py
cd test/ && python3 registered/unit/layers/quantization/test_fp8_blockwise_linear_backends.py

🚀 2-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/moe/test_glm4_moe_models.py

🚀 4-gpu-b200 (2 tests): ✅ View workflow run

cd test/ && python3 registered/unit/layers/quantization/test_fp8_blockwise_linear_backends.py
cd test/ && python3 registered/e2e/models/test_deepseek_v3_fp4.py

🚀 1-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/unit/layers/quantization/test_fp8_blockwise_linear_backends.py
cd test/ && python3 registered/e2e/quantization/test_modelopt_fp8.py