[Bugfix] DeepEP-V2: expert_tokens_meta must be None on the decode/cudagraph path (empty recv_expert_num_tokens) by dmvevents · Pull Request #52632 · vllm-project/vllm

dmvevents · 2026-08-17T16:39:16Z

Purpose

Fix a deterministic startup crash in graph-enabled (default-compilation) serving with --all2all-backend deepep_v2.

In DeepEPV2PrepareAndFinalize._receiver, expert_tokens_meta was built unconditionally via mk.ExpertTokensMetadata.make_from_list(recv_expert_num_tokens, ...). On the decode/cudagraph path, do_cpu_sync = not self.use_cudagraph (this file) intentionally skips the CPU sync, leaving recv_expert_num_tokens empty — so the call builds a present-but-empty ExpertTokensMetadata. That violates the decode-mode contract documented at the top of this file (expert_tokens_meta must be None when counts are absent), and downstream it crashes DeepEP's combine during startup profile_run.

Measured, two independent reproductions of the same call path (profile_run → finalize_async → DeepEP elastic.py combine):

H200/B200, EP16 (DP16, 2 nodes over EFA, NCCL-GIN CPU proxy): Triton illegal-memory-access at deepep_v2.py combine, 12+/16 ranks, deterministic ~48 s into startup, 2/2 repro (Xid 43, ECC clean).
Independently reproduced by AWS on B300, EP16, DeepSeek-V2-Lite BF16: CUDA_ERROR_LAUNCH_FAILED (719) raised from DeepEP's JIT handle at the same phase — one root cause surfacing at different synchronization points.

--enforce-eager avoids the crash (the sync happens); this guard makes default compilation safe.

Changes

One guard: return expert_tokens_meta = None when recv_expert_num_tokens is empty (the decode/cudagraph path), matching the documented contract; the prefill path (do_cpu_sync=True) is unchanged.

Test Plan / Validation

With the guard applied at e2f993dc4-era pins: graph-enabled vllm serve (no --enforce-eager) on EP16/DP16 across 2 nodes over EFA completes CUDA-graph capture, reaches Application startup complete, and serves: 153/153 HTTP 200 across a 1/8/16/32/64 concurrency sweep (previously: deterministic crash in profile_run before startup completed).
Eager path unaffected (guard only changes the empty-list case).

Signed-off-by: Anton Alexander <dmvevents@gmail.com>

dmvevents · 2026-08-17T16:54:15Z

Cross-referencing for the maintainers: @tlrmchlsmth's draft #50511 contains a functionally equivalent guard at this same call site (plus padding-row zeroing before whole-tensor dynamic quantization, which this PR does not touch).

This PR is offered as the minimal, GPU-validated subset: just the expert_tokens_meta=None guard for the empty recv_expert_num_tokens (decode/cudagraph) case, validated on live EP16/DP16 2-node EFA serving — graph capture completes, Application startup complete reached, 153/153 HTTP 200 (details in the description; independently reproduced by AWS on B300).

Happy to defer to #50511 if the superset lands first — in that case this can be closed in its favor. Filing separately only because #50511 is marked draft/not-yet-GPU-validated and this subset is measured and independently landable.

mergify · 2026-08-17T21:39:46Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @dmvevents.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

…graph path On the graph-enabled path do_cpu_sync=False leaves recv_expert_num_tokens empty, so make_from_list built a present-but-EMPTY ExpertTokensMetadata, violating the decode-mode contract documented in this file (must be None) and crashing DeepEP combine during startup profile_run: reproduced as a Triton illegal-memory-access at deepep_v2.py (B200, EP16, 12+/16 ranks, 2/2 repro, Xid 43, ECC clean) and independently by AWS as CUDA_ERROR_LAUNCH_FAILED(719) from DeepEP's JIT handle (B300, EP16, DeepSeek-V2-Lite) - one root cause surfacing at different sync points. --enforce-eager avoids it; this guard makes default compilation safe. Signed-off-by: Anton Alexander <dmvevents@gmail.com>

jeffye-dev · 2026-08-19T06:49:29Z

I get the same issue, the fix is working

ywang96 · 2026-08-19T10:32:07Z

/ci run

github-actions · 2026-08-19T10:32:09Z

✅ @dmvevents, CI is now available for this PR.

/ci run starts upstream CI; /amd-ci run starts AMD CI only.
/ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
/amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
/ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

github-actions · 2026-08-19T10:32:24Z

✅ Triggered Buildkite CI #84584 for commit 40ade20c88a5.

mergify · 2026-08-19T10:44:35Z

Hi @dmvevents, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

…path Fixes mypy union-attr error: expert_tokens_meta can be None on the decode/cudagraph path, but the recv_topk_idx is None (prefill) branch always has recv_expert_num_tokens populated. Co-authored-by: Kimi Code Signed-off-by: Roger Wang <hey@rogerw.io>

ywang96 · 2026-08-19T20:08:09Z

/ci run

github-actions · 2026-08-19T20:08:30Z

✅ Triggered Buildkite CI #84675 for commit b67e5b478da0.

…agraph path (empty recv_expert_num_tokens) (vllm-project#52632) Signed-off-by: Anton Alexander <dmvevents@gmail.com> Signed-off-by: Roger Wang <hey@rogerw.io> Co-authored-by: Roger Wang <hey@rogerw.io> Signed-off-by: Zhu, Zufang <zufang.zhu@intel.com>

The kill switch only nulled _align_m_fn, reverting the dispatch-side align. But _receiver unconditionally ran the cudagraph block that carries psum_recv_per_rank and creates a non-None expert_tokens_meta, which drives the feature's always-on apply-side rewrite (valid_tokens bounding of SITU + moe_fused_mul_sum). So DISABLE=1 still ran the new apply path and was NOT the pre-feature reference -- both DISABLE=0 and DISABLE=1 measured ~0.65 gsm8k. Gate that carry under the same switch (self._fused_moe_align_disabled) so DISABLE=1 restores the pre-feature contract: expert_tokens_meta is None on the cudagraph decode path (PR vllm-project#52632), valid_tokens is None, and SITU + moe_fused_mul_sum revert to unbounded reduction. The globalize kernel already sanitizes the padding tail to -1, so this is safe. This isolates whether the regression lives in the feature's apply-side rewrite. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Elvir Crncevic <elvircrn@gmail.com>

The VLLM_DEEPEP_DISABLE_FUSED_MOE_ALIGN kill switch was gating the per-rank psum carrier (`psum_recv_per_rank`) on `_fused_moe_align_disabled`, so DISABLE=1 also suppressed it. That carrier is the pre-fusion known-good mechanism (PR vllm-project#52632): under cudagraph decode it always populated expert_tokens_meta so num_valid bounds SITU + moe_fused_mul_sum. Suppressing it forced the untested has_num_valid=False path, which left stale cudagraph output rows and cost ~0.05 gsm8k (0.94 -> 0.89) even after the earlier padding-row fix. Restore the baseline `if self.use_cudagraph:` gate so the carrier always runs; the fused MoE-align metadata attach stays correctly gated by `fused_moe_align is not None` (None under DISABLE=1), keeping the fusion itself off. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Elvir Crncevic <elvircrn@gmail.com>

…agraph path (empty recv_expert_num_tokens) (vllm-project#52632) Signed-off-by: Anton Alexander <dmvevents@gmail.com> Signed-off-by: Roger Wang <hey@rogerw.io> Co-authored-by: Roger Wang <hey@rogerw.io>

… removed OFI_NCCL_GIN_GDAKI; retire "guard pending" wording - serve.sh runs `set -u`; its launch banner still expanded `${OFI_NCCL_GIN_GDAKI}`, which the previous commit stopped exporting — that would have aborted the launch with "unbound variable". The banner now prints `gdaki=auto(--enable-gdaki build)`, which is what actually selects GDAKI at the pinned plugin. - README + serve.sh still described the empty-ExpertTokensMetadata guard (vllm-project/vllm#52632) as "filed upstream / pending" and referred to a "fix stack" that was deleted on 2026-08-17. #52632 merged 2026-08-20; it post-dates the pinned e2f993dc4. Both now say: not shipped at this pin, a VLLM_SHA bump past the merge commit + re-measure enables it, no patch step. Verification: bash -n on every script (no `${OFI_NCCL_GIN_GDAKI}` / `${EP_EFA_*}` / `${DEEP_EP_BACKEND}` expansion remains anywhere in the folder); markdownlint-cli2 0 issues. Not rebuilt or re-run on cluster. Signed-off-by: Anton Alexander <dmvevents@gmail.com>

dmvevents requested review from mgoin, pavanimajety and zyongye as code owners August 17, 2026 16:39

mergify Bot added nvidia bug Something isn't working labels Aug 17, 2026

github-project-automation Bot added this to NVIDIA Aug 17, 2026

dmvevents added a commit to dmvevents/awsome-distributed-ai that referenced this pull request Aug 17, 2026

docs(deepep-efa): link the upstream guard PR — vllm-project/vllm#52632

c33fedc

Signed-off-by: Anton Alexander <dmvevents@gmail.com>

dmvevents added a commit to dmvevents/awsome-distributed-ai that referenced this pull request Aug 17, 2026

docs(deepep-efa): link the upstream guard PR — vllm-project/vllm#52632

1d306e4

Signed-off-by: Anton Alexander <dmvevents@gmail.com>

This was referenced Aug 17, 2026

feat(deepep-efa): vLLM DeepEP-V2 MoE all-to-all over EFA via NCCL-GIN GDAKI (GPU-initiated) awslabs/awsome-distributed-ai#1231

Open

feat(deepep-efa): vLLM DeepEP-V2 MoE all-to-all over EFA (eager + non-eager) awslabs/awsome-distributed-ai#1230

Open

mergify Bot added the needs-rebase label Aug 17, 2026

dmvevents force-pushed the fix/deepep-v2-empty-expert-tokens-meta branch from 0228d99 to 5b294d9 Compare August 17, 2026 22:18

mergify Bot removed the needs-rebase label Aug 17, 2026

This was referenced Aug 19, 2026

[Bugfix] Do not fabricate empty expert counts on the DeepEP v2 decode path aoshen02/vllm#31

Closed

[WideEP] Remove DeepEP high-throughput backend, redirect to DeepEP v2 #46887

Draft

ywang96 added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 19, 2026

Merge branch 'main' into fix/deepep-v2-empty-expert-tokens-meta

40ade20

ywang96 added 3 commits August 19, 2026 11:32

Merge branch 'main' into fix/deepep-v2-empty-expert-tokens-meta

79bdbc7

Merge branch 'main' into fix/deepep-v2-empty-expert-tokens-meta

b67e5b4

ywang96 approved these changes Aug 20, 2026

View reviewed changes

github-project-automation Bot moved this to Ready in NVIDIA Aug 20, 2026

ywang96 merged commit 14617c2 into vllm-project:main Aug 20, 2026
112 checks passed

github-project-automation Bot moved this from Ready to Done in NVIDIA Aug 20, 2026

almogtavor mentioned this pull request Aug 26, 2026

[Bug]: deepep_v2 crash in cudagraph mode #52894

Open

1 task

Navigation Menu

Uh oh!

[Bugfix] DeepEP-V2: expert_tokens_meta must be None on the decode/cudagraph path (empty recv_expert_num_tokens) - #52632

[Bugfix] DeepEP-V2: expert_tokens_meta must be None on the decode/cudagraph path (empty recv_expert_num_tokens)#52632
ywang96 merged 5 commits into
vllm-project:mainfrom
dmvevents:fix/deepep-v2-empty-expert-tokens-meta

dmvevents commented Aug 17, 2026

Uh oh!

dmvevents commented Aug 17, 2026

Uh oh!

mergify Bot commented Aug 17, 2026

Uh oh!

jeffye-dev commented Aug 19, 2026

Uh oh!

ywang96 commented Aug 19, 2026

Uh oh!

github-actions Bot commented Aug 19, 2026

Uh oh!

github-actions Bot commented Aug 19, 2026

Uh oh!

mergify Bot commented Aug 19, 2026

Uh oh!

ywang96 commented Aug 19, 2026

Uh oh!

github-actions Bot commented Aug 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

dmvevents commented Aug 17, 2026

Purpose

Changes

Test Plan / Validation

Uh oh!

dmvevents commented Aug 17, 2026

Uh oh!

mergify Bot commented Aug 17, 2026

Uh oh!

jeffye-dev commented Aug 19, 2026

Uh oh!

ywang96 commented Aug 19, 2026

Uh oh!

github-actions Bot commented Aug 19, 2026

Uh oh!

github-actions Bot commented Aug 19, 2026

Uh oh!

mergify Bot commented Aug 19, 2026

Uh oh!

ywang96 commented Aug 19, 2026

Uh oh!

github-actions Bot commented Aug 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants