moe: add DeepEP V2 ElasticBuffer support to MoE token dispatcher - #24443
moe: add DeepEP V2 ElasticBuffer support to MoE token dispatcher#24443dmvevents wants to merge 11 commits into
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for DeepEP V2 by adding a probe for ElasticBuffer and an opt-in path gated by the SGLANG_DEEPEP_USE_V2 environment variable. It includes the implementation of _build_v2_buffer and a new unit test to verify the probe logic. Feedback includes suggestions to explicitly define ElasticBuffer as None on import failure for better static analysis, centralize the new environment variable within the Envs class, and remove an unused parameter in the _build_v2_buffer method.
| try: | ||
| from deep_ep import ElasticBuffer | ||
|
|
||
| have_deepep_v2 = True | ||
| except ImportError: | ||
| have_deepep_v2 = False |
There was a problem hiding this comment.
To improve code robustness and assist static analysis tools, it is recommended to explicitly define ElasticBuffer as None in the except block. This ensures the symbol is always present in the module namespace, even if the import fails.
| try: | |
| from deep_ep import ElasticBuffer | |
| have_deepep_v2 = True | |
| except ImportError: | |
| have_deepep_v2 = False | |
| try: | |
| from deep_ep import ElasticBuffer | |
| have_deepep_v2 = True | |
| except ImportError: | |
| ElasticBuffer = None | |
| have_deepep_v2 = False |
There was a problem hiding this comment.
Applied — the except ImportError block sets ElasticBuffer = None explicitly at the current head (9ca5af3). This thread can be resolved.
| if have_deepep_v2 and get_bool_env_var( | ||
| "SGLANG_DEEPEP_USE_V2", default="false" |
There was a problem hiding this comment.
The environment variable SGLANG_DEEPEP_USE_V2 should be added to the Envs class in python/sglang/srt/environ.py to maintain consistency with other DeepEP configurations and leverage the centralized environment management system. Using envs.SGLANG_DEEPEP_USE_V2.get() is preferred over direct calls to get_bool_env_var.
There was a problem hiding this comment.
Applied in c3ea3a6: SGLANG_DEEPEP_USE_V2 and SGLANG_DEEPEP_V2_TIMING are both registered in the Envs class (next to the existing SGLANG_DEEPEP_* entries in environ.py) and read via envs.<NAME>.get() at both call sites. This thread can be resolved.
| cls, | ||
| group: dist.ProcessGroup, | ||
| hidden_size: int, | ||
| param_bytes: int, |
There was a problem hiding this comment.
The parameter param_bytes is passed to _build_v2_buffer but is not utilized within the method body. It should be removed from the function signature to improve code clarity and maintainability.
There was a problem hiding this comment.
Closing the loop in-thread: this was applied — _build_v2_buffer no longer takes param_bytes (V2's MoE-shape ctor doesn't need it; the V1 path still uses it for its byte-size hints). This thread can be resolved.
* Add minimal SGLang overlay for Wave 29 smoke Scaffold-only Dockerfile lacked pip install; would produce base-only image. Adding minimal overlay: - pip install sglang[all]==0.5.6.post2 - torch/nccl/nvshmem pin guards matching v0.2.5 base Wave 29 will materialize full adapter (sgl-project/sglang#24443) + JIT cache. * Add CodeBuild buildspec + pins.env + minimal overlay Wave 29 prep: - Minimal SGLang 0.5.6.post2 overlay (full adapter TBD) - CodeBuild buildspec + pins.env matching sibling pattern - NCCL/NVSHMEM pin guards for v0.2.5 base Unblocks Plan A Wave 29 cross-node smoke. --------- Co-authored-by: Anton Alexander <dmvevents@users.noreply.github.com>
The pre-commit lint job flagged two files that exceeded black's default 88-character line limit on multi-line if/assignment continuations. Collapsed them back onto single lines to match what black reformats them to: - python/sglang/srt/layers/moe/token_dispatcher/deepep.py - test/srt/test_deepep_v2_probe.py No logic changes. Addresses lint check failure on PR sgl-project#24443. Signed-off-by: Anton Alexander <dmvevents@users.noreply.github.com>
|
Triage of the 9 red checks (from check-run logs on head
What actually needed a code change: 1 of 9 (lint). The other 8 are all symptoms of the PR being in Draft state — sglang's CI policy is to short-circuit every downstream check until the PR is marked ready. Keeping this in Draft until we get reviewer signal on the V2 opt-in design, but the lint fix is landed so the CI will be fully green the moment this flips to Ready. Diff for the lint fix is purely formatting (-9 / +3 lines, 2 files, no logic change):
|
|
Agree on both. Pushing:
Single follow-up commit on the same branch. |
|
Pushed
CI should pick it up automatically. |
| packed_recv_hidden, self.packed_recv_count, self.handle, event, hook = ( | ||
| buffer.low_latency_dispatch( | ||
| hidden_states, | ||
| topk_ids, | ||
| self.num_max_dispatch_tokens_per_rank, | ||
| self.num_experts, | ||
| use_fp8=use_fp8, | ||
| **(dict(use_nvfp4=True) if use_nvfp4 else dict()), | ||
| **( | ||
| dict(x_global_scale=input_global_scale) | ||
| if input_global_scale is not None | ||
| else dict() | ||
| ), | ||
| async_finish=not self.return_recv_hook, | ||
| return_recv_hook=self.return_recv_hook, | ||
| **fp8_deepgemm_scale_opts, |
There was a problem hiding this comment.
ElasticBuffer does not have a low_latency_dispatch method.
I don't think that simply changing the construction of a buffer will allow direct use of DeepEP V2.
V2 allows a hybrid mode, which can uniformly use dispatch method to handle normal and low-latency calls.
Are there any other updates?
There was a problem hiding this comment.
Correct — V2 only exposes dispatch(); low_latency_dispatch() does not exist on ElasticBuffer. This PR is the probe + opt-in ctor only — the dispatch call sites at line 565 and line 746 are still V1 and unchanged. The PR body explicitly defers full V2 dispatch/combine wiring as a follow-up ("not yet end-to-end validated in this PR"), and the PR is in Draft for that reason. Full V2 dispatch path (single unified dispatch() covering both normal and low-latency, new 5-tuple return, EPHandle lifetime for low-latency cleanup) lands in a follow-up once this probe is in. Thanks for catching it.
There was a problem hiding this comment.
Correcting my earlier reply here — it's stale on both points and your underlying objection was right. The PR is no longer draft (ready since Aug 4), and it no longer stops at the constructor: the V2 dispatch and combine call sites are now wired through ElasticBuffer.dispatch()/ElasticBuffer.combine() behind SGLANG_DEEPEP_USE_V2=1 (V2 infers layout internally, so there is no low_latency_dispatch on it by design — the dispatcher's V2 branches call the V2 API, not the V1 method names). Top-k sizing also now threads the model's real router_topk instead of 0. The branch is rebased onto current main (past the check-changes required base), which also postdates the new deepep_v2 A2A backend (#29525) — this PR is the narrower, opt-in path on the classic dispatcher and doesn't touch that backend; happy to reconcile the two if a maintainer prefers.
20af19d to
d2d9df4
Compare
The pre-commit lint job flagged two files that exceeded black's default 88-character line limit on multi-line if/assignment continuations. Collapsed them back onto single lines to match what black reformats them to: - python/sglang/srt/layers/moe/token_dispatcher/deepep.py - test/srt/test_deepep_v2_probe.py No logic changes. Addresses lint check failure on PR sgl-project#24443. Signed-off-by: Anton Alexander <dmvevents@users.noreply.github.com>
d2d9df4 to
84fd157
Compare
The pre-commit lint job flagged two files that exceeded black's default 88-character line limit on multi-line if/assignment continuations. Collapsed them back onto single lines to match what black reformats them to: - python/sglang/srt/layers/moe/token_dispatcher/deepep.py - test/srt/test_deepep_v2_probe.py No logic changes. Addresses lint check failure on PR sgl-project#24443. Signed-off-by: Anton Alexander <dmvevents@users.noreply.github.com>
The pre-commit lint job flagged two files that exceeded black's default 88-character line limit on multi-line if/assignment continuations. Collapsed them back onto single lines to match what black reformats them to: - python/sglang/srt/layers/moe/token_dispatcher/deepep.py - test/srt/test_deepep_v2_probe.py No logic changes. Addresses lint check failure on PR sgl-project#24443. Signed-off-by: Anton Alexander <dmvevents@users.noreply.github.com>
b2959be to
43d6310
Compare
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
|
This PR is now marked ready for review — the branch carries measured 2-node TP16/EP16 evidence (accuracy + AIPerf sweep in the description). pr-gate is currently blocking on the missing |
|
/rerun-failed-ci |
Adds an opt-in DeepEP V2 code path alongside the existing V1 Buffer, based on PR sgl-project#24443, plus the fixes required to make it run on H100 + RoCE. 1. Warm the NCCL communicator before construction. torch creates the ncclComm_t LAZILY, on first collective. ElasticBuffer takes the raw handle (elastic.py:301) straight into C++ (elastic.py:306 calculate_elastic_buffer_size); on a group that has never carried a collective the handle is null and C++ dereferences it -> SIGSEGV. DeepEP own tests/elastic/test_ep.py never hits this because init_dist runs collectives during setup, which is why identical buffer parameters work there and crash here. One dummy all_reduce fixes it. 2. Pass the real num_topk instead of 0. PR sgl-project#24443 passes num_topk=0, which V2 defaults to 32 (csrc/elastic/buffer.hpp:663), sizing the buffer for top-32 on a top-8 model -- about 4x (3.5 vs 0.9 GiB per rank). 3. Optional explicit SM count via SGLANG_DEEPEP_V2_NUM_SMS, default 0. 0 is DeepEP own default and is already correct: dispatch resolves it with get_theoretical_num_sms (elastic.py:928) and combine reuses the dispatch handle (elastic.py:1086). The knob exists only for SM sweeps. Requires --enable-symm-mem: SGLang derives NCCL_CUMEM_ENABLE from it (entrypoints/engine.py:1222) and NCCL gates symmetric memory on cuMem (init.cc:1674). Without it ncclDevCommCreate fails with 'Communicator does not support symmetric memory'. Validated: DeepSeek-V3, TP32/DP32 with DP attention, 2 and 4 nodes, H100 + 8x400G RoCE, NCCL 2.30.7, DeepEP epv2-release. Serving only -- this branch makes no throughput claim; see the note below. NOTE ON SCOPE: upstream already has three open PRs for V2 support (sgl-project#24443, sgl-project#29402, sgl-project#29525). This branch is a working integration for our cluster, not a competing proposal. The num_topk finding in (2) is the part worth upstreaming, as a review comment on sgl-project#24443.
|
In // NOTES: there are lots of `kNumTopk <= 32` restrictions, so we use 32 to calculate token size
num_topk = num_topk == 0 ? 32 : num_topk;So passing For a concrete anchor, with I have not measured the The dispatcher does know the value: the |
|
@markw14 You're right, and the comment in my code was wrong about what Fixed in I went with construction-time plumbing rather than On the red checks: the previous head's 12 failures were all aggregator jobs ( |
Measured update: 4-node EP32 (TP32/EP32) serve PASS on the V2 pathSince the top-k sizing fix (9ca5af3, thanks @markw14), we've run this PR's Setup (public substrate, reproducible)
Gates (all 4 nodes)
AIPerf (ISL 512 / OSL 128, streaming chat)
(c=1 ran first against a cold server so its TTFT is warmup-contaminated; omitted. 2-node EP16 reference on the same harness: ITL ~146 ms — the +20 ms/token at EP32 is the wider all-to-all fan-out, consistent with dispatch micro-bench p50 632 µs @ep32 vs ~265 µs @EP16.) Operational findings from the 32-rank bring-up (for reviewers / future users)
The V2 opt-in path in this PR is otherwise unchanged from what was reviewed — findings 2/3 are deployment notes, and finding 1 is the only candidate code delta. Still blocked on the |
DeepEP V2 (deepseek-ai/DeepEP#605, merged 2026-04-29) introduces `ElasticBuffer` alongside the legacy `Buffer`. Both classes are exported from `deep_ep.__init__`, so SGLang's existing V1 code path continues to work unchanged on a V2 install. This patch adds a second import probe (`have_deepep_v2`) next to the existing `use_deepep` flag, then lets users opt in to the V2 `ElasticBuffer` ctor behind `SGLANG_DEEPEP_USE_V2=1`. The V1 path is byte-identical when the env var is unset, so this is a backwards- compatible addition. V2 collapses the V1 NVL/RDMA byte-pool ctor into a single MoE-shape ctor and derives the internal buffer size from `num_max_tokens_per_rank`, `hidden`, and `num_topk`. `num_allocated_qps=0` asks V2 to auto-size and auto-cap the Queue-Pair budget on AWS EFA (the 128-slot GIN ring ceiling; see `_is_efa_fabric` in `deep_ep/buffers/elastic.py`). Why: - Removes the need for users to monkeypatch `deep_ep.Buffer` to reach V2 semantics, and mirrors the probe shape already used in NVIDIA/Megatron-LM's `fused_a2a.py` (`HAVE_DEEP_EP_V2`). Opt-in env-var gate keeps the change infra-bump-shaped, not a default switch flip. - vLLM is landing a parallel native-V2 path in vllm-project/vllm#41183 (16 commits, active review). SGLang users running mixed-framework MoE stacks benefit from having V2 reachable in SGLang without a runtime monkeypatch. V2 parity rules baked in: - `num_max_tokens_per_rank` sourced from `envs.SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK` when the caller did not pin it (matches the V1 low-latency path). - `num_topk=0` at ctor time: DeepEP V2 accepts this and derives a conservative group-size-based hint, so we don't need to know the router topk at buffer construction. - `clean_buffer()` skips the legacy `clean_low_latency_buffer` call when the underlying buffer is an ElasticBuffer (no equivalent on V2, per `deep_ep/buffers/elastic.py`). Related: - DeepEP V2 PR: deepseek-ai/DeepEP#605 (merged 2026-04-29) - vLLM DeepEP V2: vllm-project/vllm#41183 (open) - Megatron-LM DeepEP V2: NVIDIA/Megatron-LM#4632 (open)
Adds a lightweight guard for the `have_deepep_v2` / `use_deepep` probe pattern introduced in the previous commit. Exercises the three reachable states: - V1 `Buffer` only installed (legacy path) - V2 `ElasticBuffer` + `Buffer` both exported (typical V2 install) - `deep_ep` missing entirely The test uses a synthetic `deep_ep` module (via sys.modules monkey patch) so it runs on any CPU host without CUDA, NCCL, or EFA. The test is a `skipTest`-tolerant guard: when the SGLang module stack cannot be imported in the CI environment (for any upstream reason), it skips rather than fails, so this does not introduce a new dependency on a full SGLang install in the unit-test tier.
The pre-commit lint job flagged two files that exceeded black's default 88-character line limit on multi-line if/assignment continuations. Collapsed them back onto single lines to match what black reformats them to: - python/sglang/srt/layers/moe/token_dispatcher/deepep.py - test/srt/test_deepep_v2_probe.py No logic changes. Addresses lint check failure on PR sgl-project#24443. Signed-off-by: Anton Alexander <dmvevents@users.noreply.github.com>
- deepep.py:68 — set ElasticBuffer = None in the except ImportError branch so static-analysis tools always see the symbol in module namespace (matches the have_deepep_v2 = False fallback shape). - deepep.py:279 — drop unused param_bytes from _build_v2_buffer. DeepEP V2's ElasticBuffer ctor derives buffer size from num_max_tokens_per_rank * hidden * num_topk; the legacy V1 byte-pool sizing is not used. The V1 path in get_buffer keeps param_bytes unchanged.
test_neither_installed asserted use_deepep is False, but the 'absent' simulation only popped deep_ep from sys.modules. On any host where deep_ep is genuinely installed (e.g. a GPU serving pod), the probe's 'import deep_ep' re-imported the real package from site-packages, so use_deepep came back True and the test failed. It only passed on CPU-only CI where deep_ep is absent. Insert a sentinel sys.modules['deep_ep'] = None so 'import deep_ep' raises ImportError deterministically on every host, matching the other two probe states which already stub deep_ep explicitly. The finally-block pop + restore already cleans the sentinel up. Verified: unittest 3/3 OK on a cgk p5en pod with deep_ep V2 installed (was 2 pass / 1 fail before this change).
…atcher _dispatch_core: ElasticBuffer has no get_dispatch_layout — V2 dispatch infers layout from topk_idx and returns (recv_x, recv_topk_idx, recv_topk_weights, EPHandle, event) with num_recv_tokens_per_expert_list on the handle. Mirrors the proven Megatron fused_a2a V2 translation. _dispatch_core + _combine_core: V2 pybind requires deep_ep._C.EventHandle for previous_event — re-derive via the elastic buffer capture() instead of passing the V1 Buffer.capture() EventOverlap (TypeError otherwise). combine drops the V1 config kwarg; num_sms=0 reuses the dispatch handle SM count. Proven in-pod (2x p5en, proxy-Gin substrate): server up, ElasticBuffer on all 16 ranks, /v1/models 200. First-dispatch behavior still under live debug.
… timing The V2 branch of get_deepep_buffer stored the buffer on cls._buffer, but the cache guard reads state.buffer — so every _get_buffer() constructed a fresh ElasticBuffer (NCCL comm split + ring alloc + QP setup, ~0.35s), twice per MoE layer per forward. Measured on 2-node TP16/EP16 EFA: ~34s/token from ctor churn while the actual V2 dispatch/combine kernels are 7.2ms/6.6ms p50 (SGLANG_DEEPEP_ V2_TIMING=1 instrumentation, included, default-off). Cache to state.buffer.
ElasticBuffer hard-asserts num_tokens <= num_max_tokens_per_rank on every dispatch (csrc/elastic/buffer.hpp:684). The env default (128) targets the V1 low-latency decode path; in normal mode a chunked-prefill batch dispatches up to chunked_prefill_size tokens at once — measured: concurrency 8 x ISL 512 aborts all 16 ranks on the assert. Size the V2 buffer to cover the prefill chunk in normal mode.
pr-gate blocked all runs while the PR was draft; pull_request workflows do not fire on ready_for_review, so a synchronize event is needed.
Review (markw14): passing num_topk=0 is not a proportional fallback -- DeepEP V2 pins the token layout to its top-32 ceiling (csrc/elastic/buffer.hpp: `num_topk = num_topk == 0 ? 32 : num_topk`), which oversizes the top-k-dependent regions of the dispatch/combine buffers 4x for a top-8 model, and the absolute cost grows with num_ranks. The router's top-k is fixed per model and already known at dispatcher construction, so thread router_topk from _DeepEPDispatcherImplBase through get_deepep_buffer() into _build_v2_buffer() and pass it to the ElasticBuffer ctor. 0 remains the conservative ceiling only when a caller cannot provide it. The previous comment claiming V2 'falls back to a group-size-based conservative hint' described behaviour that does not exist and is corrected. V1 path untouched (the new parameter is unused there), so the default code path stays byte-identical. Verified: py_compile clean; black --check and isort --check-only (profile=black) clean. Signed-off-by: Anton Alexander <dmvevents@gmail.com>
Both env vars now live in the Envs registry next to the existing SGLANG_DEEPEP_* entries and are read via envs.<NAME>.get(), per repo convention, instead of raw get_bool_env_var calls. Signed-off-by: Anton Alexander <dmvevents@gmail.com>
9ca5af3 to
c3ea3a6
Compare
3 participants
Motivation
DeepEP V2 (deepseek-ai/DeepEP#605, merged 2026-04-29) introduces
ElasticBufferalongside the legacyBuffer. SGLang's MoE token dispatcher (python/sglang/srt/layers/moe/token_dispatcher/deepep.py) only knows how to build and drive the V1Buffer, so there is currently no way to reach the V2 path from SGLang without monkeypatchingDeepEPBuffer.get_deepep_buffer()at runtime.The V2 API differs from V1 in ways that touch the whole dispatch/combine contract, not just the constructor:
ElasticBuffer(group, num_max_tokens_per_rank, hidden, num_topk, ...)instead of V1's NVL/RDMA byte-pool ctor.get_dispatch_layout(). V2 infers the layout internally fromtopk_idx, so the V1 pre-pass is gone at the call site.(recv_x, recv_topk_idx, recv_topk_weights, EPHandle, event);num_recv_tokens_per_expert_listnow lives on theEPHandle, not in the return tuple.deep_ep._C.EventHandleforprevious_event; passing a V1Buffer.capture()EventOverlapraisesTypeError.combinesignature. V2combinedrops the V1configkwarg and reuses the dispatch handle's SM count.deep_ep.__init__on a V2 install exports bothBuffer(frombuffers/legacy.py) andElasticBuffer(frombuffers/elastic.py), so SGLang's existingfrom deep_ep import Buffer, Configsurface keeps working unchanged. This PR adds the missing consumer-side plumbing soElasticBufferis reachable and correct.Parallel upstream work adopts the same
HAVE_DEEP_EP_V2probe shape, and this PR mirrors it so the frameworks stay consistent:fused_a2a.pyV2 translation. This PR's dispatch/combine translation mirrors that one.Modifications
All changes are in
python/sglang/srt/layers/moe/token_dispatcher/deepep.pyplus its unit test. The V2 path is opt-in behindSGLANG_DEEPEP_USE_V2=1; with it unset (or on a pre-V2deep_epinstall) the V1Bufferpath runs byte-identical to before.Probe + opt-in buffer build. A
try: from deep_ep import ElasticBufferprobe setshave_deepep_v2.get_deepep_buffer()builds anElasticBufferwhen V2 is importable and the env var is set, otherwise the legacyBuffer.clean_buffer()skipsclean_low_latency_bufferwhen the live object is anElasticBuffer(V2 has no equivalent).Call-site translation in
_dispatch_core/_combine_core(b65c871). Drops the V1get_dispatch_layout()pre-pass; unpacks the 5-tuple and readsnum_recv_tokens_per_expert_listoff theEPHandle; re-derives the typedEventHandlevia the elastic buffer'scapture()forprevious_event; drops the V1configkwarg oncombine. Mirrors the Megatronfused_a2aV2 path.Cache-slot fix (
0bb2713). The V2 branch stored the buffer oncls._buffer, but the cache guard readsstate.buffer— so every_get_buffer()(twice per MoE layer per forward) constructed and tore down a freshElasticBuffer(NCCL comm split + GIN ring alloc + QP setup, ~0.35 s each). On a 2-node TP16/EP16 run this was ~34 s/token of pure constructor churn. Caching tostate.bufferfixes it (this bug ships inside this PR's earlier commit; the fix is in the same PR).num_max_tokens_per_ranksizing (b2959be).ElasticBufferhard-assertsnum_tokens <= num_max_tokens_per_rankon every dispatch (csrc/elastic/buffer.hpp:684). The env default (128) is tuned for the V1 low-latency decode batch; in normal mode a chunked-prefill batch dispatches up tochunked_prefill_sizetokens at once. Two layers had to be fixed together: bound the ctor capacity bychunked_prefill_size/max_prefill_tokens, and passnum_max_tokens_per_rank=Noneat the dispatch call — becauseelastic.pyresolvesvalue_or(passed, ctor), so a passed decode value (128) silently overrides the prefill-sized ctor and re-trips the assert.Env-gated timing instrumentation (
0bb2713, default-off).SGLANG_DEEPEP_V2_TIMING=1wraps dispatch/combine in a cuda-synced_V2Timerthat logs per-call and cumulative ms. Dead when unset.Hermetic unit test (
84fd157).test/srt/test_deepep_v2_probe.pyexercises the three probe states. The "neither installed" case now inserts asys.modules['deep_ep'] = Nonesentinel soimport deep_epraises deterministically even on a host wheredeep_epis genuinely installed (previously it only passed on CPU-only CI).V2 fall-through matrix:
have_deepep_v2=False(pre-V2 install) → legacyBuffer, byte-identical to pre-patch.have_deepep_v2=True, env unset → legacyBuffer, byte-identical. V2 latent.have_deepep_v2=True,SGLANG_DEEPEP_USE_V2=1→ newElasticBufferpath.use_deepep=False→ unchanged; module still imports.Accuracy Tests
Validated end-to-end on 2× p5en.48xlarge (16× H200), TP16/EP16,
--moe-a2a-backend deepep --deepep-mode normal, Qwen3-30B-A3B-FP8, cross-node dispatch/combine over the DeepEP V2ElasticBufferpath (SGLANG_DEEPEP_USE_V2=1confirmed on all 16 ranks in the server log:constructing deep_ep.ElasticBuffer).Coherent generation through the V2 path (greedy,
temperature=0):/v1/modelsreturns 200 on both nodes; the probe unit test passes 3/3 on a pod withdeep_epV2 installed (was 2 pass / 1 fail before the hermetic fix), and skips cleanly on CPU-only hosts.Speed Tests and Profiling
AIPerf sweep on the same 2-node TP16/EP16 serve, ISL 512 / OSL 128, streaming chat completions:
ITL is flat across concurrency (decode-bound per user; the all-to-all transport is not the bottleneck at these shapes), and the full sweep runs with 0 buffer asserts and the server alive throughout — the sizing fix (modification #4) is what makes c=8 survive.
Kernel-level profiling with the shipped
SGLANG_DEEPEP_V2_TIMING=1instrumentation (reproducible from this PR): the V2 dispatch/combine kernels measure 7.2 ms / 6.6 ms p50 at TP16/EP16 cross-node. This is what surfaced the cache-slot bug (modification #3) — before the fix, wall time was ~34 s/token of constructor churn around ~14 ms of actual kernels; after, ~150 ms/token.Checklist
pre-commit run --files python/sglang/srt/layers/moe/token_dispatcher/deepep.py— black-jupyter applied.)test/srt/test_deepep_v2_probe.py, hermetic across CPU-only and GPU hosts.)Related
antonai-work/vllm-deepep-v2-efa,antonai-work/nemo-rl-deepep-v2-efa/cc @zhyncs @merrymercy @Ying1123
CI States
Latest PR Test (Base): ❌ Run #32297147893
Latest PR Test (Extra): ❌ Run #32297146926