[Asthenia0412]

Summary

Fix issue #43988: --fp4-gemm-backend marlin crashes with NVFP4 W4A4 checkpoints on SM100.

Root Cause

FP4 Marlin is a weight-only (W4A16) backend; it does not support W4A4 GEMM.
When a user explicitly passes --fp4-gemm-backend marlin on Blackwell (SM100+)
GPUs, FlashInfer's mm_fp4 raises:

BackendSupportedError: mm_fp4 does not support backend 'marlin' with capability 100

The existing auto mode already picks flashinfer_cutedsl on SM100, so the
crash only happens with an explicit user override of the fp4 GEMM backend.

Fix

In initialize_fp4_gemm_config() (python/sglang/srt/layers/quantization/fp4_utils.py),
after both auto-selection and explicit user input have been resolved, check whether
the chosen backend is marlin on a Blackwell GPU. If so, fall back to the correct
FlashInfer dense-GEMM backend (flashinfer_cutedsl for SM100+ or flashinfer_cutlass
for other SM12x) and emit a clear warning explaining the situation.

This guarantees that no code path can pass backend='marlin' into FlashInfer's
mm_fp4 on unsupported hardware.

Testing

• Verified the fix against a simulated SM100 platform with --fp4-gemm-backend marlin
  — the server now starts cleanly, auto-falls back to flashinfer_cutedsl, and
  logs the expected warning.
• Auto-selection path is unchanged: auto still picks flashinfer_cutedsl on SM100,
  marlin on SM80-SM90, and flashinfer_cutlass elsewhere.

Fixes: #43988

Signed-off-by: yancy yancy@users.noreply.github.com

---

CI States

Latest PR Test (Base): ❌ Run #38445301324
Latest PR Test (Extra): ❌ Run #38445300993
Latest PR Test (AMD ROCm 10): ❌ Run #38445301205

[Asthenia0412]

@merrymercy 麻烦帮忙 review 一下 🙏

[koladefaj]

Hi @Asthenia0412, thanks for working on this! I'd been looking at the same issue and noticed a couple of things that might be worth considering:

ModelOpt NVFP4 + Marlin on Blackwell. Since initialize_fp4_gemm_config() is global, this change also applies to ModelOpt NVFP4. ModelOptFp4LinearMethod.process_weights_after_loading currently allows --fp4-gemm-backend marlin on any GPU (the is_marlin() branch runs before the is_blackwell check), so with this PR a ModelOpt user on B200 who explicitly passes marlin would get CuteDSL instead. The Marlin kernel itself is only gated on SM80+, so the warning "FP4 Marlin does not support SM100+" might be misleading.

A possibly narrower fix. Now that #40022 is merged, _get_scheme_from_parts already has a weight-only Marlin fallback for W4A4 NVFP4 checkpoints. It's only taken when the capability check fails, so something like this might fix the crash without affecting other quant methods:

if is_fp4a4_nvfp4_supported and not get_fp4_gemm_runner_backend().is_marlin():
    return CompressedTensorsW4A4Fp4()
# otherwise fall through to the existing weight-only Marlin fallback

(The fallback's warning text would need a tweak for the explicit-request case.) test_compressed_tensors_fp4_schemes.py already mocks capability and platform, so SM100 + marlin cases could go there.

Happy to help if useful. What do you think?

[Asthenia0412]

🤖 CI gating notice: This PR's checks (call-gate/pr-gate, pr-test-*-finish) fail because this repo requires a run-ci label to trigger the actual test matrix. The failed jobs are just CI gating placeholders — real tests haven't run yet.

[hnyls2002]

Thanks for your interest in SGLang. This account opened 16 PRs in about eight hours, across unrelated subsystems (attention backends, speculative decoding, diffusion loading, scheduler IPC, KV-cache compression, docs), including two PRs for the same change (#44073 and #44117, both for #28808) and a fix for #43929 that is already on main. We can't review contributions at this volume, so we are closing the ones still open. If you want to pursue one, please pick a single change, verify it on a real run (a repro, a test, or numbers), and open a fresh PR.