[MRV2] Capture routed experts from supported monolithic MoE kernels

[aoshen02]

Stacked on #38163. Model Runner V2 currently binds only BaseRouter capture callbacks, so it misses monolithic MoE kernels that perform routing inside the expert kernel. This follow-up binds the monolithic replay callback only when the expert advertises supports_routing_replay_capture(). It depends on #38163 and the monolithic support introduced by #44214. Tests in the vllm/vllm-openai:nightly container: 2 passed, 4 deselected for the V2 binding tests; py_compile and git diff --check pass. AI assistance was used; please review every changed line.

[aoshen02]

Stacked on #38163; this PR is intentionally based on woosuk/mrv2-expert-indices because the helper it extends is not yet on main. It adds only monolithic expert replay binding and its V2 unit test. Once #38163 merges, this can be retargeted/rebased onto main.

[aoshen02]

Added unsupported-kernel guard regression test. Container verification now passes: 3 passed, 4 deselected for -k gpu_model_runner_v2; py_compile and git diff --check remain clean.

[aoshen02]

Additional nightly runtime evidence on local GB200: Qwen3-30B-A3B, TP=1, --enable-return-routed-experts, max-model-len=512. FlashInfer TRTLLM monolithic served successfully but returned (4, 48, 8) routed_experts with 0 nonzero entries; Triton with the same request returned 1,522 nonzero entries. This confirms the pre-fix monolithic capture gap. The 4-GPU attempt also reproduced an expert-kernel shape failure: args->intermediate_size % 128 == 0 (64 vs. 0).

[aoshen02]

More targeted runtime evidence: local gpt-oss-20b MXFP4 checkpoint selected TrtLlmMxfp4ExpertsMonolithic under nightly --moe-backend flashinfer_trtllm --enable-return-routed-experts. Request succeeded, response routed_experts shape (4, 24, 4) but had 0 nonzero entries. This reproduces the monolithic R3 capture gap on the MXFP4 backend shown in the support matrix.

[aoshen02]

Fixed CI pre-commit failure: mypy could not see the replay setter on the #38163 base type, so the callback setter is now capability-gated via callable(getattr(...)). Also applied the hook formatter changes. Local nightly container test remains 3 passed, 4 deselected; py_compile and diff-check pass.

[aoshen02]

Pre-commit failure was only ruff-format rewriting two files; applied the exact formatter output and pushed commit 31515ba. The prior mypy errors are resolved.

[aoshen02]

Corrected two-node runtime validation (PR #44214 actually applied inside the nightly containers):

• Nodes: gb200-rack1-03 + gb200-rack1-04, 4x GB200 each, Ray TP=8
• Image: vllm/vllm-openai:nightly, digest sha256:7f2bc168366c77fbd8329368f00310d208531c14ece6c2de31a6611ef99f6ec8
• PR application: copied the [RL Infra][FlashInfer] Enable router replay output from FlashInfer monolithic MoE kernel #44214 Python changes from the checked-out PR head e0b5bae into both containers' installed vLLM package; the runtime log shows Initializing routed experts capturer on all 8 workers.
• Model: local gpt-oss-20b (MXFP4), --moe-backend flashinfer_trtllm --enable-return-routed-experts
• Request completed successfully. Decoded routed_experts: shape=(4, 24, 4), dtype=uint8, nonzero=375, max=31.

Control/failure on the same two-node TP=8 setup with Qwen3-30B-A3B and FlashInfer TRTLLM: worker initialization failed in the expert weight conversion path with RuntimeError: shape '[2048, 1, 128]' is invalid for input of size 393216 (FlashInfer monolithic layout constraint). This is the requested unsupported-shape monolithic + R3 expert error.

The earlier TP=1 all-zero result was from an unmodified nightly container and is not evidence against #44214.

[aoshen02]

Exact Qwen3.5-122B-A10B two-node benchmark completed after applying #44214 in both nightly containers:

• Nodes: gb200-rack1-03 + gb200-rack1-04 (4x GB200 each), Ray TP=8
• Image digest: sha256:7f2bc168366c77fbd8329368f00310d208531c14ece6c2de31a6611ef99f6ec8
• Model: Qwen/Qwen3.5-122B-A10B BF16, 250.2 GB snapshot downloaded to shared Lustre
• Server: --tensor-parallel-size 8 --moe-backend flashinfer_trtllm --gpu-memory-utilization 0.90 --max-model-len 32768 --enable-return-routed-experts
• Benchmark: random dataset, input=1024, output=512, 500 prompts, request-rate=inf

Result: 500/500 successful, duration 26.6346 s, request throughput 18.7726 req/s, output throughput 9611.55 tok/s, total throughput 28834.66 tok/s, mean TTFT 6544.21 ms, median TTFT 6490.62 ms, P99 TTFT 12534.64 ms, mean TPOT 38.13 ms, P99 TPOT 47.83 ms.

The result file is /home/aoshen/bench-results/qwen35-fi-r3.json. This is the first run in this thread that both applied #44214 and used the exact Qwen3.5 benchmark model/configuration.

[aoshen02]

Follow-up direct payload verification: after starting the exact two-node TP=8 Qwen3.5-122B-A10B server with --enable-return-routed-experts, a real /v1/chat/completions response contained choices[0].routed_experts. Decoded base64 .npy: shape=(24, 48, 8), dtype=uint8, nonzero=9150, min=0, max=255. Response was saved locally as /tmp/qwen35-r3-response.json.

[aoshen02]

Superseded by a clean Model Runner V2-only implementation based directly on the latest main. The replacement removes the dependency on the prior non-main stack and will be submitted as a separate PR after validation.