Repository navigation
[Compile] Remove torch.compile-based sequence parallelism and AsyncTP - #60517
Merged
Merged
Conversation
Remove the SequenceParallelismPass and the AsyncTPPass built on top of it, along with the PassConfig flags (enable_sp, fuse_gemm_comms, sp_min_token_num) and the model-runner/worker/config plumbing that only existed to support them. The feature is not supported in Model Runner V2, and we are reducing reliance on torch.compile. Manual (model-level) sequence parallelism under vllm/models/ is unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
WoosukKwon
requested review from
AndreasKaratzas,
BoyuanFeng,
Harry-Chen,
ProExpertProg,
houseroad,
jikunshang,
khluu,
mgoin,
njhill,
robertgshaw2-redhat,
tjtanaa,
tlrmchlsmth,
xuechendi,
yewentao256,
youkaichao and
zou3519
as code owners
October 8, 2026 01:54
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
Contributor
|
Documentation preview: https://vllm--60517.org.readthedocs.build/en/60517/ |
mergify
Bot
added
documentation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Collaborator
Author
|
/ci run |
|
❌ This PR is 6 commits behind upstream |
Collaborator
Author
|
/ci run |
|
✅ Triggered Buildkite CI #93485 for commit |
Contributor
CI selector (shadow): 156 test steps (220 jobs) instead of 96 (122 jobs)Shadow mode: this changes nothing about what CI runs. It shows what the evidence-based selector would pick for this PR, next to today's rules. How it works. Feedback welcome: reply here if it would skip a step this change needs, or runs something unrelated.
Selector would run (156)
Would skip (today's rules run them) (13)
Would add (today's rules do not run them) (73)
AMD mirrors: would skip (8)
AMD mirrors: would add (59)
34 changed files · base |
6 tasks done
tlrmchlsmth
approved these changes
Oct 8, 2026
20 of 26 tasks
lesj0610
added a commit
to lesj0610/vllm
that referenced
this pull request
Oct 9, 2026
…bytes-profiling The only conflict is the `vllm.v1.worker.utils` import list in gpu_model_runner.py. vllm-project#60517 dropped `is_residual_scattered_for_sp`, and this branch adds `build_minimal_kv_cache_config` and `clear_layer_kv_caches`, so the resolution keeps the two additions without the removed helper. Signed-off-by: lesj0610 <lesj0610@gmail.com>
This was referenced Oct 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Remove the torch.compile-based sequence parallelism pass (
SequenceParallelismPass) and the AsyncTP pass (AsyncTPPass) that is built on top of it. This feature is not supported in Model Runner V2, and we are reducing our reliance on torch.compile. Manual (model-level) sequence parallelism undervllm/models/is unaffected.Supersedes open work that extends compile-based SP/AsyncTP: #36851, #37489, #38151.
Claims
vllm/compilation/passes/fusion/sequence_parallelism.pyandcollective_fusion.py(AsyncTP only ever ran after the SP rewrite, so without SP it matches nothing).PassConfig.enable_sp,fuse_gemm_comms, andsp_min_token_num, plus all SP-only plumbing:splitting_ops=[]/FULL cudagraph, TP-divisible cudagraph capture sizes, the batch-invariance override, and the MRV2 unsupported-feature entry;num_tokensto a multiple of TP,is_residual_scattered_for_sp, and the residual all-gather on PP send/recv;resolve_cudagraph_mode_and_sizes/adjust_cudagraph_sizes_for_spec_decodedrop their now-unusedtensor_parallel_sizeargument;*_mm_outhelpers that only AsyncTP used.fusions.md,cuda_graphs.md,vllm_ir.md,batch_invariance.md).Validation
pre-commit(ruff, mypy, etc.) passes.Qwen/Qwen3-0.6Bwith TP=2, PP=2 generates correctly on both MRV1 (VLLM_USE_V2_MODEL_RUNNER=0) and the default runner, covering the edited PP send/recv and intermediate-tensor paths.tests/test_config.py,tests/v1/worker/test_gpu_worker.py,tests/model_executor/test_routed_experts_capture.py,tests/distributed/test_comm_ops.py: all pass.tests/compile/test_config.py: all pass except 9 cudagraph capture-size tests, which fail the same way onmainon this machine (unrelated).tests/models/{kimi_k3,deepseek_v32,glm5next}/test_sequence_parallel.pyandtests/distributed/test_custom_all_gather_reduce_scatter.py(30 passed, 1 skipped).IS_DENSE = False), so the default execution path is unchanged.Details
PassConfigforbids extra fields, so passingenable_sp,fuse_gemm_comms, orsp_min_token_numnow raises a validation error instead of being silently ignored. These were opt-in, off by default, and unsupported on MRV2.all_gather_tensorsoption onGroupCoordinator.{send,isend,recv,irecv}_tensor_dict. Nothing passes it after this PR, but a future model that combines manual SP with PP may need it. Today, every manual-SP model gates SP onpipeline_parallel_size == 1(or has PP disabled), so no sharded tensors cross PP stages.vllm.reduce_scatter/vllm.all_gathercustom ops, compile ranges,use_inductor_graph_partition, and the attention-fusionsplitting_opslogic stay, because other passes and models still use them.Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.
🤖 Generated with Claude Code