MiMoV2: do not force flashinfer_trtllm when an EP a2a backend is active - #39079
Open
whn09 wants to merge 1 commit into
Open
MiMoV2: do not force flashinfer_trtllm when an EP a2a backend is active#39079whn09 wants to merge 1 commit into
whn09 wants to merge 1 commit into
Conversation
The sm100 override rewrites moe_runner_backend auto -> flashinfer_trtllm without
looking at moe_a2a_backend. FusedOpPool has no ("deepep", "flashinfer_trtllm") or
("deepep_v2", "flashinfer_trtllm") entry and flashinfer_trtllm's runner_core is
None, so every DeepEP arm on Blackwell dies at model init with
NotImplementedError: Runner backend MoeRunnerBackend.FLASHINFER_TRTLLM requires
a fused func for a2a backend deepep, but none is registered
Left alone, "auto" would have resolved to deep_gemm at runtime, which is the correct
and working choice for deepep; the config-time override just gets there first. The
glm4_moe, qwen3_moe, deepseek_v4 and nemotron_h overrides all gate the same sm100
trtllm rewrite on moe_a2a_backend == "none"; this adds the missing clause. Invisible
on Hopper, where the override does not fire.
This was referenced Sep 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The MiMoV2 config-time override forces
moe_runner_backend=flashinfer_trtllmon sm100 FP8without checking whether an EP all-to-all backend is active:
FusedOpPoolhas noflashinfer_trtllmfused func registered fordeepepordeepep_v2,and
flashinfer_trtllmhas norunner_core(fused path only), soMoeRunner.__init__hitsrunner_core is None and fused_func is None:The user asked for
auto.autoresolves todeep_gemm, which works. The override turns aworking default into a hard failure at model init, and because the substitution is silent the
traceback names a backend the command line never mentioned.
Sibling overrides already carry exactly this guard —
glm4_moe.py:39,qwen3_moe.py:43,deepseek_v4.py:62,nemotron_h.py:86all testcfg.moe_a2a_backend == "none"beforeforcing a runner.
mimo_v2.pyis the one that does not, so this brings it in line ratherthan introducing a new convention.
Modifications
One condition,
and cfg.moe_a2a_backend == "none", plus a comment explaining why (the nextperson to read this will otherwise wonder whether the guard is load-bearing). No behaviour
change without an a2a backend: the sm100 FP8 fast path is untouched for the plain-TP case
the override was written for.
Accuracy Tests
No numerical change. With
--moe-a2a-backend nonethe resolved runner isflashinfer_trtllmbefore and after — verified via
/get_server_infoon 8× B200 with an FP8 MiMoV2 checkpoint.With
--moe-a2a-backend deepep_v2 --moe-runner-backend autothe resolved runner goes from"init raises" to
deep_gemm, whose numerics against thedeepep(v1) path are 10/12 promptsbit-identical on a fixed logprob set (details in the companion PR).
Speed Tests and Profiling
Unchanged where the override still fires. Where it no longer fires, the comparison is
NotImplementedErrorversus a running server.Worth stating plainly for anyone weighing it:
flashinfer_trtllmreally is the faster runnerwhere it is usable — on decode at global bs=32 it is 24.58 ms/token versus 34.90 for
deep_gemm, ~30%. That is an argument for registering adeepep/deepep_v2fused func, notfor forcing a runner that cannot load. (The one existing
deepep+flashinfer_cutedslfusedfunc takes
CuteDslFp4MoeQuantInfo, i.e. NVFP4 only, so it does not help an FP8 checkpoint.)Checklist
_mimo_v2_overridesreturns nomoe_runner_backendwhenmoe_a2a_backend != "none"; it needs no GPU, say the wordReview and Merge Process
I cannot add the
run-cilabel myself, so this needs someone with write access to start CI.Companions, independent of each other: #39080 (enable
deepep_v2for MiMoV2 — the PR thatmakes this guard reachable), #39078 (ElasticBuffer vs CUDA graph capture), #37211
(
ep_scatter_from_psumkernel args).CI States
Latest PR Test (Base): ❌ Run #34590432083
Latest PR Test (Extra): ❌ Run #34590431824
Latest PR Test (AMD ROCm 10): ❌ Run #34590431968