[Bugfix] DSv4 MXFP4 selector: stop narrowing explicit aliases to their BF16 variant - #53586
Conversation
The DSv4 selector routed explicit non-b12x aliases through _get_requested_backends(..., None), whose BF16-preference filter drops every non-BF16 variant when the model declares no activation. The FlashInfer CUTLASS BF16 variant is gated to SM90, so on SM100+/SM120 `--moe-backend flashinfer_cutlass` failed outright instead of falling through to the W4A8 (MXFP8) variant. Restore trying every variant of the alias in priority order; keep the narrowing scoped to b12x, where it encodes the W4A8-by-default backend policy. Adds a regression test that simulates the SM120 gate (BF16 variant unsupported) and requires selection to land on the MXFP8 variant. Verified end-to-end on RTX 6000 Pro (SM120), DeepSeek-V4-Flash-0731: `--moe-backend flashinfer_cutlass` now starts and serves. Signed-off-by: Zihua Wu <zihuaw@nvidia.com> Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
/ci run |
|
✅ Triggered Buildkite CI #85470 for commit |
github-actions
Bot
added
the
ready
|
/ci run |
|
✅ Triggered Buildkite CI #85471 for commit |
Head branch was pushed to by a user without write access
88d7194 to
8ca081d
Compare
|
Thanks for the review! The only failing CI job is |
…nferlab-integration
|
/ci run |
|
✅ Triggered Buildkite CI #86474 for commit |
|
/ci run |
|
✅ Triggered Buildkite CI #87489 for commit |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (2)
Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review. 📝 SummarySummary by CodeRabbit
WalkthroughThe DeepSeek v4 MXFP4 backend selector now preserves activation variants for explicit backends. A regression test verifies that ChangesDeepSeek v4 backend selection
Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk: ⚪ Minimal · up to Explicit non-b12x DeepSeek-V4 backend aliases can now fall through to supported MXFP4/MXFP8 variants on SM120-class hardware, while b12x behavior remains unchanged. Regression and reported end-to-end coverage support merge readiness. Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…r BF16 variant (vllm-project#53586) Signed-off-by: Zihua Wu <zihuaw@nvidia.com> Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Jee Jee Li <pandaleefree@gmail.com> Co-authored-by: Yongye Zhu <zyy1102000@gmail.com> Signed-off-by: Jyotirmoy Roy <jyotirmoyroy649@gmail.com>
3 participants
[Bugfix] DSv4 MXFP4 selector: stop narrowing explicit aliases to their BF16 variant
Purpose
On DeepSeek-V4-class MXFP4 models, passing an explicit non-b12x MoE backend
(e.g.
--moe-backend flashinfer_cutlass) can fail outright on SM100+/SM120even though a supported variant of that backend exists.
Root cause: the DSv4 selector routed explicit aliases through
_get_requested_backends(alias, None). With no declared model activation,the BF16-preference filter keeps only the BF16 variant of the alias. The
FlashInfer CUTLASS BF16 variant is gated to SM90, so on SM100+/SM120 the
selection dead-ends instead of falling through to the W4A8 (MXFP4 weights +
MXFP8 activations) variant, which is the variant that actually supports those
devices.
Fix: for explicit non-b12x aliases, try every variant of the alias in priority
order (
map_mxfp4_backend). The narrowing stays scoped tob12x, where itencodes the intended W4A8-by-default backend policy.
A regression test simulates the SM120 gate (BF16 variant unsupported, W4A8
variant supported) and requires
flashinfer_cutlassto selectFLASHINFER_CUTLASS_MXFP4_MXFP8; it fails on unfixed code and passes withthis change.
Test plan
pytest tests/kernels/moe/test_b12x.py -k "flashinfer_cutlass_falls_through or b12x_activation_selection"— 3 passed; the new test fails without thefix (fail-before verified).
pre-commit run --from-ref upstream/main --to-ref HEAD— all hooks pass.DSpark speculative decoding:
--moe-backend flashinfer_cutlasspreviouslyerrored at startup; with this fix the server starts and an 8K/1K random
benchmark completes with zero failed requests. Current
mainadditionallyneeds [Bugfix][SM120] DSv4: pass contiguous C128A decode topk indices on SM120 #53574 (one-line C128A contiguity fix) to boot DSv4 + spec decoding on
SM120 at all, so the E2E was run with that fix stacked.
Notes
AI assistance (Kimi Code / OpenAI Codex) was used in preparing this change;
every line was reviewed and tested by the submitting human.