[Refactor] Extract shared helpers from MXFP4 MoE backend selectors

[demian-overflow]

Summary

Addresses #41291 by extracting the duplicated machinery shared between select_gpt_oss_mxfp4_moe_backend and select_mxfp4_moe_backend into private module-level helpers in vllm/model_executor/layers/fused_moe/oracle/mxfp4.py.

What changes

Both selectors carried four nested closures each (_make_log_backend, _make_log_unsupported, _return_or_raise) plus near-identical copies of the priority-iteration loop and the explicit-runner-backend branch. This PR lifts those into six private module-level helpers:

• _activation_format_for_config(config) — pulls the BatchedExperts vs Standard decision out of both functions.
• _make_log_backend(backend) / _make_log_unsupported(backend, reason) — the two log-message templates, deduplicated.
• _try_kernel_classes(backend, config, ..., *, log_failures=False) — probes every kernel class registered for backend. The log_failures switch preserves the difference between the priority-loop path (which logs each unsupported kernel via debug_once) and the explicit-backend path (which only surfaces failures by raising).
• _return_or_raise(backend, config, ...) — resolves a single backend or raises ValueError, used by every direct-pick site (env vars, runner-backend, XPU fallback).
• _select_explicit_runner_backend(config, activation_format) — honors config.moe_backend when set, returns None to signal "fall through to priority list".
• _select_first_supported_backend(config, priority_backends, activation_format) — walks a priority list and returns the first backend whose kernel matches.

select_gpt_oss_mxfp4_moe_backend and select_mxfp4_moe_backend are now thin orchestration that compose those helpers. Both keep their public signatures and observable behavior, so the three callers in vllm/model_executor/layers/quantization/mxfp4.py and vllm/model_executor/layers/quantization/quark/quark_moe.py need no edit.

Behavioral preservation

The refactor preserves observable behavior 1:1, with two intentional no-op cleanups bundled in:

1. Removed the redundant logger.info_once(_make_log_backend(backend)) immediately before _return_or_raise(Mxfp4MoeBackend.XPU, ...) in the GPT-OSS XPU branch — _return_or_raise already emits the same info_once message on success, so the original would have logged twice and info_once deduplicated it.
2. Removed the explicit scope="local" kwargs in select_mxfp4_moe_backend's log calls. "local" is the default for info_once / debug_once per vllm/logger.py, so the behavior is identical and the calls match the rest of the file.

The GPT-OSS path's behavior matrix is fully preserved:

• LoRA branch (raises on non-CUDA, picks TRITON_UNFUSED or MARLIN).
• All four env-var override branches (VLLM_USE_FLASHINFER_MOE_MXFP4_BF16, VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8, VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8_CUTLASS, VLLM_MXFP4_USE_MARLIN) — kept in select_gpt_oss_mxfp4_moe_backend, since they don't apply to the generic MXFP4 path.
• The BATCHED_MARLIN coercion when the format is BatchedExperts.
• The XPU fallback after the priority loop.
• The CUDA/ROCm NotImplementedError and the final (NONE, None) return.
• The per-kernel-class debug_once for unsupported configs in the priority loop (preserved via log_failures=True).

Why this shape and not the issue's literal ask

The issue proposes "a single select_mxfp4_moe_backend() function … with activation_key parameter or quantization_config parameter". Reading both functions, the divergence is bigger than that:

Aspect	select_gpt_oss_mxfp4_moe_backend	select_mxfp4_moe_backend
LoRA branch	yes	no
Env-var overrides	yes (4 branches)	no
Priority list	BF16-focused (_get_priority_backends_for_gpt_oss)	MXFP8/DeepGEMM (_get_priority_backends)
XPU fallback	yes	no
Final fallback	returns (NONE, None)	raises NotImplementedError

A single function with a mode='gpt_oss' flag (or several boolean kwargs) would be harder to read than two named entry points and would lock in a shape that the larger class-hierarchy refactor in PR #37776 will rework anyway. This PR achieves the spirit of the dedup ask — eliminate the four copies of the helper closures and the duplicated priority loop — while keeping both entry points distinct.

Why it isn't a duplicate

Searched for existing PRs:

gh pr list --repo vllm-project/vllm --state all --search "41291 in:body"        → no PR
gh pr list --repo vllm-project/vllm --state open --search "select_mxfp4_moe_backend"  → no overlap
gh pr list --repo vllm-project/vllm --state open --search "merge mxfp4 backend"       → no overlap

The closest open PR is #37776 ([MoE] Unify MoE oracles with class structure, needs-rebase), which restructures all MoE oracles into a class hierarchy. That PR predates the GPT-OSS split that #39604 introduced and #40860 reshaped, so it does not touch select_gpt_oss_mxfp4_moe_backend at all. The two PRs don't conflict; whichever lands first, the other rebases cleanly.

Tracking issue: #41291.

Tests run

This is a Python-only refactor with no observable behavior change, so I ran the lint/typecheck commands instead of building the full vLLM wheel:

• ruff check vllm/model_executor/layers/fused_moe/oracle/mxfp4.py — clean.
• ruff format --check vllm/model_executor/layers/fused_moe/oracle/mxfp4.py — already formatted.
• mypy --python-version 3.10 --follow-imports=skip --ignore-missing-imports vllm/model_executor/layers/fused_moe/oracle/mxfp4.py — Success: no issues found in 1 source file.
• typos vllm/model_executor/layers/fused_moe/oracle/mxfp4.py — clean.
• python -c "import ast; ast.parse(open(...).read())" — OK.
• grep -rn "select_gpt_oss_mxfp4_moe_backend\|select_mxfp4_moe_backend" across the repo — three call sites in quantization/mxfp4.py and quantization/quark/quark_moe.py, none need editing because the public signatures are unchanged.

I did not run the full pytest suite or build the wheel — the PR makes no behavior changes and the helpers are private (no new public API surface). Happy to run any specific kernel/MoE test the reviewer thinks is relevant.

AI assistance disclosure

Claude (Anthropic) assisted with: reading both selector functions and the issue thread, comparing against the in-flight PR #37776, designing the helper extraction shape, applying the edit, and running the lint/mypy/typos commands. I (Demian Havdun) reviewed every changed line, checked that the helpers preserve the original behavior path-by-path (especially the per-kernel-class debug logging and the XPU branch double-log dedup), and signed off as committer per DCO. Co-author trailer added per AGENTS.md.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀

[gemini-code-assist]

Code Review

This pull request refactors the MXFP4 MoE backend selection logic by extracting common functionality into reusable helper functions, reducing code duplication in select_gpt_oss_mxfp4_moe_backend and select_mxfp4_moe_backend. A review comment highlights an unconventional call to a static method that explicitly passes the class as an argument, suggesting a potential design inconsistency in the underlying kernel interface.

[gemini-code-assist]

The is_supported_config method in vllm/model_executor/layers/fused_moe/modular_kernel.py is defined as a @staticmethod but explicitly takes cls as its first argument. While calling it as k_cls.is_supported_config(k_cls, ...) is technically correct for that specific definition, it is highly unconventional for a static method and suggests a potential design flaw in the base class or a misunderstanding of @staticmethod vs @classmethod. If is_supported_config is intended to be overridden by subclasses and needs access to the class, it should be a @classmethod. If it doesn't need access to the class, the cls argument should be removed from the definition. Given the current definition in modular_kernel.py, this call is correct, but it highlights a maintenance risk if the base class is ever cleaned up to follow standard Python idioms.