[Feature] Add DeepEPv2 (ElasticBuffer) MoE A2A backend

[MengYu10151]

Reland of #29525, reverted by #35568 because its e2e test used an
unregistered CI runner name. This reland restores the backend and fixes that
registration without changing unrelated CI configuration.

Motivation

Add DeepEP v2 ElasticBuffer as a standalone MoE A2A backend named
deepep_v2, alongside the existing deepep backend. Its fixed-capacity
communication shapes make decode CUDA-graph capturable with both single-node
and multi-node topologies.

Supported scope

Area	Supported in this PR
Models	DeepseekV3ForCausalLM, DeepseekV4ForCausalLM, Qwen3MoeForCausalLM
Expert precision	Dynamic blockwise FP8, 128x128 weight blocks
MoE runner	deep_gemm
Backend modes	`--deepep-v2-mode direct
DeepEP package	sgl-deep-ep==0.1.2 (deep_ep 2.1.0)

direct and hybrid select the ElasticBuffer topology; they are not aliases
for legacy DeepEP low_latency and normal. Decode uses the expanded masked
layout, while extend/prefill uses the contiguous layout in either mode.

This intentionally does not claim support for other MoE architectures or
quantization contracts. In particular, Qwen3.8
(Qwen3_5MoeForCausalLM), BF16 experts, MXFP8, non-DeepGEMM runners,
deterministic inference, and a speculative draft worker using deepep_v2 are
not supported by this landing. These configurations now fail before A2A
execution instead of reaching a model-specific forward path with incorrect
post-expert reduction or incompatible activation scales.

Implementation

• Adds an independent DeepEPv2Dispatcher using ElasticBuffer dispatch and
  combine. Legacy deepep objects and mode semantics are not reused.
• Adds masked-slab repack kernels and DeepGEMM adapters for contiguous prefill
  and masked decode, including Hopper FP32 scales and Blackwell packed UE8M0
  scales.
• Stores ElasticBuffer state in RuntimeContext resources so context reset and
  sequential engine construction release ownership consistently. The cache key
  includes the process group and a new buffer is published only after collective
  construction succeeds.
• Treats dispatch handles as single-use and preserves the dispatch-selected SM
  settings for combine. SGLANG_DEEPEP_V2_NUM_SMS=0 delegates automatic SM/QP
  selection to sgl-deep-ep==0.1.2.
• Enables the backend's minimum NCCL symmetric-memory requirement with
  os.environ.setdefault("NCCL_CUMEM_ENABLE", "1"); an explicit user setting
  still wins, and --enable-symm-mem remains optional.

Capacity and fail-fast behavior

SGLANG_DEEPEP_V2_NUM_MAX_DISPATCH_TOKENS_PER_RANK configures ElasticBuffer's
per-rank capacity; it is a memory reservation, not a semantic token limit.
Startup checks cover the canonical prefill budget, decode CUDA-graph batch,
per-DP-rank running-request bound, and speculative width. The dispatch adapter
also checks the actual runtime row count, which remains the authoritative guard
for dynamic batches.

Errors include actionable guidance for adjusting the capacity,
--chunked-prefill-size, or --cuda-graph-max-bs. Unsupported model, runner,
quantization, deterministic, and draft-backend combinations fail at startup or
MoE layer initialization.

Validation

• Relevant CPU suite: 204 tests and 15 subtests passed.
• Qwen3 FP8: direct and hybrid booted and returned correct capital,
  arithmetic, and translation answers.
• DeepSeek V3 FP8: both modes passed the same functional gate; GSM8K was 0.960
  (direct) and 0.965 (hybrid) versus 0.955 for legacy DeepEP.
• DeepSeek V4 Flash FP8, TP8/DP8/EP8 with DP attention and DeepGEMM:

Platform / metric	v2 direct	v2 hybrid	legacy DeepEP
H20 x8 MMLU (500)	0.888	0.890	0.888
H20 x8 GSM8K (100)	0.980	0.980	0.970
B200 x8 MMLU (500)	0.886	0.884	-
B200 x8 GSM8K (100)	0.950	0.970	-

Representative back-to-back serving results:

Workload	deepep_v2	legacy DeepEP	Delta
H20 x8 decode	4,339 tok/s	4,379 tok/s	-0.9%
H20 x8 prefill	22,353 in_tok/s	21,651 in_tok/s	+3.2%
B200 x8 decode	8,429 tok/s	8,675 tok/s	-2.8%
B200 x8 prefill	32,447 in_tok/s	31,533 in_tok/s	+2.9%

Multi-node hybrid was also validated on 2 x H20 x8 over RoCE at 4,610 tok/s
decode and 20,688 in_tok/s prefill. There is no same-fabric legacy DeepEP
baseline because its NVSHMEM data plane did not complete on that setup.

Checklist

• Format and static checks.
• Unit and boundary tests.
• Functional accuracy validation.
• Single-node and multi-node performance validation.
• User documentation in a follow-up PR.

Co-authored-by: @jianxiongzhang

---

CI States

Latest PR Test (Base): 🚫 Run #33033834791
Latest PR Test (Extra): ⏳ Run #33034831627
Latest PR Test (AMD ROCm 7.2): 🚫 Run #33033834696

[ch-wan]

/tag-and-rerun-ci

[ch-wan]

Summary

This reland adds a standalone deepep_v2 MoE A2A backend (ElasticBuffer dispatch/combine, masked-slab Triton kernels, DeepGEMM permute adapters, server-args/env knobs, and capturer classification). The DeepSeek path looks carefully built: layout is chosen by inference phase, the handle lifecycle is fail-closed, the boot-time chunk-vs-cap check is real, and the packed-UE8M0 / CUDA-graph kernel tests guard previously-broken cases. The dominant correctness gap is that deepep_v2 is still not a DeepEP-family backend in is_deepep_class_backend(), so non-DeepSeek MoE models that key A2A forward/setup on that helper (or a raw is_deepep() list) will run FusedMoE's new dispatcher and then extra-reduce. Remaining notes are DX (NCCL symmetric memory) and test-convention nits, not kernel bugs.

Issue counts by severity

• bugs: 1
• suggestions: 5
• nits: 0

[ch-wan]

[bug] is_deepep_class_backend() now includes deepep_v2, which is enough for Qwen3 (and Qwen2's extra-reduce scale) because those sites call the helper. It is not enough for models that still keep their own A2A-family lists. Those lists were the other half of the original "skip A2A / extra-reduce" bug, and they still omit v2:

• glm4_moe.py:533 _enable_a2a_moe (deepep/mooncake/nixl/mori/ascend_fuseep/flashinfer) — GLM-4.5 takes forward_normal.
• glm4_moe_lite.py:296 and mimo_v2.py:458 — same pattern.
• qwen2_moe.py:654 (is_deepep() or is_mori()) — Qwen2-MoE never enters _forward_deepep; shared-expert TP1 at qwen2_moe.py:369 is also missed.
  DeepSeek's parallel _enable_a2a_moe (deepseek_v2.py:845) was updated in this PR; the copies were not. FusedMoE still builds DeepEPv2Dispatcher, so forward_normal runs A2A dispatch/combine and then a post-experts TP all-reduce. should_skip_post_experts_all_reduce does not skip deepep_v2 (only flashinfer/pplx), so this is a silent double-count, not a clean fallback.

Suggestion: Add or get_moe_a2a_backend().is_deepep_v2() to every A2A-family list that already includes mooncake/nixl/mori/flashinfer (the DeepSeek _enable_a2a_moe / shared-expert TP1 shape). For Qwen2, either add is_deepep_v2() next to is_deepep() on the forward, ep_size, and shared-expert TP1 checks, or route those three sites through is_deepep_class_backend() — do not replace GLM4's list with the helper, because that list also admits flashinfer/nixl/ascend_fuseep.

[MengYu10151]

Thanks, this made it clear that treating v2 as a global A2A dispatcher would require a broader audit of model-specific forward contracts. I chose the narrower landing instead of adding is_deepep_v2() to every backend list in this PR. 20806a2aee now accepts only the three architectures exercised here: DeepseekV3ForCausalLM, DeepseekV4ForCausalLM, and Qwen3MoeForCausalLM. Every other architecture fails in _handle_a2a_moe() before weight loading, so Qwen2/GLM/MiMo cannot enter the dispatcher and then take an extra reduction. The boundary tests cover unsupported and missing architectures, including the official Qwen3.8 Qwen3_5MoeForCausalLM config. Supporting another model can now be a separate change with its model-specific A2A path and accuracy tests.

[ch-wan]

[suggestion] The boot-time capacity guard only compares per-rank chunked_prefill_size to SGLANG_DEEPEP_V2_NUM_MAX_DISPATCH_TOKENS_PER_RANK. Decode CUDA graph is left enabled, and _handle_gpu_memory_settings often sets cuda_graph_config.decode.max_bs to 256/512 while the default cap is 128. The error text tells users to lower --chunked-prefill-size; doing that (and leaving max_bs alone) still blows the per-rank dispatch assert on graph capture. Prefill was called out because smoke traffic may never hit a full chunk; decode graph warmup does hit max_bs, but the failure is the opaque dispatch ValueError rather than this handler's message.

Suggestion: Also reject when per-rank cuda_graph_config.decode.max_bs (and/or max_running_requests) exceeds the cap, and mention --cuda-graph-max-bs in the error alongside the chunked-prefill hint.

[MengYu10151]

Good catch on decode graph warmup. 20806a2aee extends the startup check beyond the prefill chunk: it now covers the resolved decode CUDA-graph max batch size, the per-DP-rank running-request bound, and speculative width, and the error points to both --chunked-prefill-size and --cuda-graph-max-bs. I kept the calculation limited to canonical startup bounds rather than duplicating scheduler batching logic; the dispatcher also validates its actual row count at runtime, which remains the authoritative guard for dynamic/adaptive batches. Tests cover the exact graph boundary, DP division, speculative width, adaptive graph, and disabled-graph cases.

[ch-wan]

[suggestion] DeepEPv2Buffer stores the process-wide ElasticBuffer on class attributes. DeepEP v1's DeepEPBuffer already parks the same kind of singleton on get_resources().buffers so reset_context() and sequential Engine rebuilds release it. destroy() only nulls the Python refs (deepep_v2.py:202); a key-mismatch recreate drops the old object without an explicit DeepEP teardown, so NCCL/CUDA state can leak across tests or a second init in-process.

Suggestion: Follow the DeepEP v1 facade: keep DeepEPv2Buffer.get_buffer / destroy as the API, put _buffer / _buffer_key in get_resources().buffers, and if ElasticBuffer exposes a close/destroy, call it from destroy().

[MengYu10151]

Updated in f550b88142. The facade now stores its ElasticBuffer state under get_resources().buffers, includes the ProcessGroup identity in the cache key, and publishes a replacement only after collective construction succeeds. destroy() removes the runtime-context entry and local references. I checked the 0.1.2 lifecycle before adding an explicit native teardown: communicator destruction is tied to DeepEP's explicit-destroy mode rather than a normal per-facade close API, so calling it here would be unsafe. The new tests cover reuse, key/group replacement, construction failure, facade destroy, and reset_context().

[ch-wan]

[suggestion] "expanded wins on decode and regresses prefill. Keeping it off the comm mode is what makes the masked-GEMM + CUDA-graph decode path available under hybrid too" narrates the layout/comm-mode split instead of stating the invariant. The next two lines already say use_masked = use_expand_layout.

Suggestion: Replace with the invariant only, e.g. that expand vs contiguous follows get_is_extend_in_batch() (decode expand + masked GEMM is graph-safe under both direct and hybrid).

[MengYu10151]

Agreed. f550b88142 reduces this to the invariant: layout follows the inference phase, so decode uses expand plus masked GEMM and extend uses contiguous, independently of direct versus hybrid. The surrounding implementation carries the rest.

[ch-wan]

[nit] The descriptor comment says 0 lets ElasticBuffer choose the SM count. _resolve_num_sms_qps must resolve 0 in SGLang because ElasticBuffer treats 0 as "zero SMs/QPs", not auto (deepep_v2.py:242).

Suggestion: Say that 0 means SGLang calls get_theoretical_num_sms / get_theoretical_num_qps (ElasticBuffer does not treat 0 as auto).

[MengYu10151]

I rechecked this against the pinned sgl-deep-ep==0.1.2, whose ElasticBuffer now handles num_sms=0 natively: dispatch resolves the theoretical SM/QP counts and combine reuses the handle's selected settings. f550b88142 therefore removes SGLang's duplicate _resolve_num_sms_qps, passes zero through, and updates the descriptor comment to match the current package behavior. Explicit positive values are still forwarded unchanged.

[ch-wan]

/tag-and-rerun-ci