[Elastic EP] Reuse CUDA graphs across reconfiguration - #54985
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
bc6df59 to
7ce6dc8
Compare
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (2)
Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review. 📝 SummarySummary by CodeRabbit
WalkthroughElastic EP now accepts a configured maximum data-parallel size. NIXL EP derives capacity from this value, stores EPLB state in preallocated buffers, and reuses fused MoE kernels and CUDA graphs during compatible rank changes. ChangesElastic EP reconfiguration
Estimated code review effort: 4 (Complex) | ~60 minutes Merge Risk: ⚪ Minimal · up to The change improves Elastic EP reconfiguration by reusing NIXL CUDA execution state, with no substantiated merge-blocking correctness or availability issue remaining. Sequence Diagram(s)sequenceDiagram
participant DPLBAsyncMPClient
participant ElasticEPScalingExecutor
participant NixlEPAll2AllManager
participant EplbState
participant FlashInferAutotune
DPLBAsyncMPClient->>ElasticEPScalingExecutor: prepare_elastic_ep(new_data_parallel_size)
ElasticEPScalingExecutor->>NixlEPAll2AllManager: stage_ep_size()
ElasticEPScalingExecutor->>FlashInferAutotune: sync_flashinfer_autotune_cache()
ElasticEPScalingExecutor->>EplbState: reconfigure_physical_expert_slots()
ElasticEPScalingExecutor->>NixlEPAll2AllManager: commit_ep_size()
ElasticEPScalingExecutor->>ElasticEPScalingExecutor: warmup_new_worker()
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
This pull request has merge conflicts that must be resolved before it can be |
7ce6dc8 to
07cafd8
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
07cafd8 to
1bf454e
Compare
|
@itayalroy hi looks great, just one more thing when reading it through, I think this can go OOB with round-robin after scale-up. E.g. if before scaling If EPLB now returns physical expert ID |
Elastic EP requires EPLB, which does not support round-robin expert placement, so |
| tensor[4][dp_rank] = int(allow_ubatching) | ||
| tensor[5][dp_rank] = num_reqs | ||
| dist.all_reduce(tensor, group=group) | ||
| if is_uniform_dp_batch(): |
There was a problem hiding this comment.
Can you check this in coordinate_batch_across_dp and short-circuit there instead?
There was a problem hiding this comment.
For MRv1, we already short-circuit inside coordinate_batch_across_dp. MRv2 does not use that function, its equivalent is sync_cudagraph_and_dp_padding, which is the current function, so we need to short-circuit here as well.
MRV2 does not support EEP yet, but support will be added soon with #53934, so I have already tested CUDA graph reuse on top of that PR to ensure we will not need a follow-up PR to support MRv2 + CUDA graph reuse.
| if new_dp_size > old_dp_size and self._can_reuse_fused_moe_kernel(): | ||
| target_world_group = get_standby_world_group() | ||
| assert target_world_group is not None | ||
| sync_flashinfer_autotune_cache(self.worker.model_runner, target_world_group) |
There was a problem hiding this comment.
Do we need to call this on all ranks or just new ones?
There was a problem hiding this comment.
This must run on all ranks because it broadcasts over the target world group, which includes both existing and new ranks. We could create a separate group or p2p flow containing only the new ranks, but that would add quite a bit of complexity, and I don’t think avoiding a small CPU broadcast to existing ranks during preparation is worth it
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
|
Great work! |
|
/ci run |
|
✅ Triggered Buildkite CI #88396 for commit |
|
/ci retry |
|
✅ The previous CI build is still running: https://buildkite.com/vllm/ci/builds/88396 |
|
/ci retry |
|
✅ Triggered Buildkite CI #88494 for commit |
Existing NIXL EP ranks keep their CUDA graphs across a reconfigure after vllm-project#54985 and no longer re-warm at commit, so MRV2 only drains when the backend re-warms. warm_and_capture keeps the NIXL warmup contexts and the runner serving state context manager in one with statement. Signed-off-by: almogtavor <almogtavor@gmail.com>
4 participants
Summary
Async preparation and eager-mode optimizations moved most Elastic EP steps out of the blocking commit path, but CUDA graph mode still blocked serving during kernel warmups and CUDA graph capture on every reconfiguration. This PR removes that work from commit as follows:
CUDA graph reuse is currently only enabled for NIXL EP. Other backends keep the existing recapture path.
With this PR, we consider Elastic EP reconfiguration downtime solved.
Results
We tested full-size
deepseek-ai/DeepSeek-V3with native FP8 on 24 GB200 GPUs across six nodes, scaling DP/EP16 -> 24 -> 16under heavy traffic. The test used vLLM defaults for model length, maximum sequences, and CUDA graph capture. Each revision ran once to warm its torch compile caches and once for measurement on the same allocation. Accuracy passed before and after both reconfigurations.Downtime is the interval in which existing requests do not progress and new requests are rejected.
16 -> 2424 -> 16Total reconfiguration time covers background preparation and commit.
16 -> 2424 -> 16Effect of Max-Capacity Input on Expert Kernel Performance
To keep CUDA graphs stable, the dispatch output buffer is sized for the maximum EP capacity. Padding exists without Elastic EP, but reserving maximum capacity increases the amount of padding. DeepGEMM and FlashInfer CuteDSL avoid padded work and are effectively unaffected. Batched Triton and CUTLASS process more of the padded buffer (for example, Triton does not support masked activations), so Elastic EP amplifies their inefficiency. With active EP fixed at 16, the table compares capacity 64 against capacity 16:
cc @tlrmchlsmth @SageMoore @ilmarkov