[XPU] Enable sequence parallel support for XPU by chaojun-zhang · Pull Request #38608 · vllm-project/vllm

chaojun-zhang · 2026-03-31T03:45:32Z

Test Plan

Test Result

UT :
pytest -s -v tests/compile/correctness_e2e/test_sequence_parallel.py
pytest -s -v pytest -s -v tests/compile/correctness_e2e/test_sequence_parallel.py

Accuracy (GSM8K)

meta-llama/Llama-2-13b-chat-hf

Case	Strict	Flexible
`Enable SP`	0.360	0.364
`Disable SP`	0.372	0.376
`Eager`	0.368	0.372

Qwen/Qwen3-32B

Configuration	Strict	Flexible
`Enable SP`	0.740	0.652
`Disable SP`	0.752	0.644
`Eager`	0.764	0.648

Benchmark

meta-llama/Llama-2-13b-chat-hf

Case	Mean TTFT (ms)	Median TTFT (ms)	P99 TTFT (ms)	Mean TPOT (ms)	Median TPOT (ms)	P99 TPOT (ms)	Output token throughput (tok/s)
`Eager`	2246.43	2024.30	2964.17	52.33	52.01	59.83	196.31
`Disable SP`	2238.68 (+0.34%)	2018.57 (+0.28%)	2951.89 (+0.41%)	52.80 (-0.90%)	51.57 (+0.85%)	64.85 (-8.39%)	181.18 (-7.71%)
`Enable SP`	2216.08 (+1.35%)	1998.83 (+1.26%)	2920.51 (+1.47%)	52.92 (-1.13%)	51.73 (+0.54%)	64.65 (-8.06%)	181.94 (-7.32%)

Qwen/Qwen3-32B

Case	Mean TTFT (ms)	Median TTFT (ms)	P99 TTFT (ms)	Mean TPOT (ms)	Median TPOT (ms)	P99 TPOT (ms)	Output token throughput (tok/s)
`Eager`	6733.47	6860.12	10630.22	63.18	62.95	73.86	419.04
`Disable SP`	6675.10 (+0.87%)	6801.23 (+0.86%)	10533.09 (+0.91%)	63.12 (+0.09%)	62.88 (+0.11%)	73.69 (+0.23%)	420.04 (+0.24%)
`Enable SP`	6393.19 (+5.05%)	6511.97 (+5.07%)	10087.97 (+5.10%)	62.81 (+0.59%)	62.58 (+0.59%)	72.92 (+1.27%)	424.83 (+1.38%)

Summary

On meta-llama/Llama-2-13b-chat-hf, Enable SP gives the best TTFT, while Eager keeps the best throughput.
On Qwen/Qwen3-32B, Enable SP is the best case on TTFT, TPOT, and throughput.
Accuracy stays in the same range across these variants;

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

gemini-code-assist

Code Review

This pull request enables sequence parallelism tests on XPU platforms by updating pytest markers and generalizing device selection using current_platform.device_type and torch.accelerator. It also moves the SequenceParallelismPass import out of the CUDA-specific guard in the pass manager. Feedback indicates that moving this pass alone is insufficient, as RMSNormQuantFusionPass remains guarded but is required by the newly enabled XPU tests, which will likely result in a NameError.

gemini-code-assist · 2026-03-31T03:47:30Z

@@ -23,13 +23,14 @@
        RocmAiterTritonAddRMSNormPadFusionPass,
    )

+from .fusion.sequence_parallelism import SequenceParallelismPass


Moving SequenceParallelismPass out of the is_cuda_alike() guard is necessary to support it on XPU. However, RMSNormQuantFusionPass (currently at line 32) remains inside the guard. Since the XPU tests added in this PR (tests/compile/passes/distributed/test_sequence_parallelism.py) explicitly enable fuse_norm_quant, the PostGradPassManager.configure() method will raise a NameError on XPU platforms when it attempts to instantiate RMSNormQuantFusionPass.

Additionally, AsyncTPPass (line 39) is guarded by is_cuda(), which will cause a similar NameError if fuse_gemm_comms is enabled on XPU. You should move RMSNormQuantFusionPass out of the guard as well, and ensure AsyncTPPass is handled safely for XPU.

mergify · 2026-03-31T18:41:31Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @chaojun-zhang.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

yma11 · 2026-04-02T01:23:30Z

@chaojun-zhang Any latency difference w/ and w/o this feature enabled? and also the case with asyncTP enabled.

chaojun-zhang · 2026-04-02T02:07:03Z

@chaojun-zhang Any latency difference w/ and w/o this feature enabled? and also the case with asyncTP enabled.

For latency please refer to Test Result in PR description.
We will have a separate draft PR for asyncTP

ProExpertProg

2 nits

jikunshang · 2026-04-08T14:48:41Z

@@ -247,6 +247,10 @@ def is_data_center_gpu(cls) -> bool:
        device_name = cls.get_device_name().lower()
        return device_name.count("data center gpu") > 0

+    @classmethod
+    def use_custom_op_collectives(cls) -> bool:


I remember that all reduce op provided by xpu communicator is in place instead of out place, this may have accuracy issue or some other issue.

jikunshang · 2026-04-09T06:16:59Z

@@ -111,7 +129,9 @@ def __init__(
        self.tp_size = get_tensor_model_parallel_world_size()

    def _all_reduce(self, x: torch.Tensor) -> torch.Tensor:
-        return tensor_model_parallel_all_reduce(x)
+        return torch.ops.vllm.all_reduce.default(


is it safe to replace?

yes. safe to use custom op for torch.compile mode , otherwise it will throw below error when xpu set use_custom_op_collectives to False.

NotImplementedError: c10d::allreduce_: attempted to run this operator with Meta tensors, but there was no fake impl or Meta kernel registered.

my question may be :

will it break other platform/OOT device if it doesn't registere this but enabled this pass.

will it impact cuda graph capture behavior.

It won't break other platform. when enable SP pass, custom-op collectives must be use. To use the custom-op collective, either:

directly in the SP pass, or

indirectly by enabling use_custom_op_collectives(). (CUDA & ROCM follow this path)

I have reverted code and enabling use_custom_op_collectives For XPU.

mergify · 2026-04-15T03:12:14Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @chaojun-zhang.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mergify · 2026-05-06T11:50:47Z

Documentation preview: https://vllm--38608.org.readthedocs.build/en/38608/

mergify · 2026-05-06T11:51:30Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @chaojun-zhang.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mergify · 2026-05-06T11:57:01Z

Hi @chaojun-zhang, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

mergify · 2026-05-06T12:07:36Z

Hi @chaojun-zhang, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

Signed-off-by: chaojun-zhang <chaojun.zhang@intel.com> Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>

Signed-off-by: Chaojun,Zhang <chaojun.zhang@intel.com> Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>

- Fix mock_cuda_platform: is_xpu.return_value was set to 'not is_cuda', so mock_cuda_platform(is_cuda=False) accidentally triggered the XPU branch in get_sequence_parallelism_threshold. Set is_xpu=False always. - Add mock_xpu_platform fixture for XPU platform mocking. - Add TestGetSequenceParallelismThresholdXPU test class covering: - XPU small hidden_size returns None - XPU large model returns calculated threshold - XPU threshold calculation with various parameters - XPU hidden_size boundary behavior Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>

mergify Bot added the intel-gpu Related to Intel GPU label Mar 31, 2026

gemini-code-assist Bot reviewed Mar 31, 2026

View reviewed changes

chaojun-zhang force-pushed the seq_parallel branch from 8e50c2c to 0bcadcd Compare March 31, 2026 05:20

chaojun-zhang changed the title ~~[Tests] Update sequence parallelism tests to support XPU~~ [XPU] Enable sequence parallel support for XPU Mar 31, 2026

chaojun-zhang force-pushed the seq_parallel branch from 0bcadcd to 9b5336e Compare March 31, 2026 07:43

jikunshang reviewed Mar 31, 2026

View reviewed changes

Comment thread vllm/compilation/passes/pass_manager.py Outdated

chaojun-zhang force-pushed the seq_parallel branch from 9b5336e to aae4a27 Compare March 31, 2026 13:45

mergify Bot added the needs-rebase label Mar 31, 2026

chaojun-zhang force-pushed the seq_parallel branch 2 times, most recently from 751f05a to e31dea0 Compare April 2, 2026 01:33

mergify Bot removed the needs-rebase label Apr 2, 2026

chaojun-zhang marked this pull request as ready for review April 2, 2026 02:16

chaojun-zhang requested review from BoyuanFeng, ProExpertProg, vadiklyutiy, youkaichao and zou3519 as code owners April 2, 2026 02:16

ProExpertProg approved these changes Apr 3, 2026

View reviewed changes

Comment thread tests/compile/passes/distributed/test_sequence_parallelism.py Outdated

Comment thread tests/compile/passes/distributed/test_sequence_parallelism.py Outdated

yma11 reviewed Apr 8, 2026

View reviewed changes

Comment thread vllm/compilation/passes/fusion/sequence_parallelism.py Outdated

jikunshang reviewed Apr 8, 2026

View reviewed changes

chaojun-zhang force-pushed the seq_parallel branch from 6209eda to 4e23d12 Compare April 9, 2026 06:05

jikunshang reviewed Apr 9, 2026

View reviewed changes

mergify Bot added the needs-rebase label Apr 15, 2026

chaojun-zhang force-pushed the seq_parallel branch from 4e23d12 to b3ee390 Compare April 22, 2026 11:50

chaojun-zhang requested a review from xuechendi as a code owner April 22, 2026 11:50

chaojun-zhang force-pushed the seq_parallel branch from b3ee390 to cfa425e Compare April 24, 2026 01:34

chaojun-zhang requested review from 22quinn, LucasWilkinson, MatthewBonanni, benchislett, houseroad and luccafong as code owners May 6, 2026 11:49

github-project-automation Bot added this to gpt-oss Issues & Enhancements May 6, 2026

chaojun-zhang mentioned this pull request May 7, 2026

vLLM-xpu torch.compile integration track vllm-project/vllm-xpu-kernels#276

Open

jikunshang approved these changes Jun 4, 2026

View reviewed changes

chaojun-zhang added 6 commits June 9, 2026 13:05

[XPU] Enable sequence parallelism on XPU

161ff2b

Signed-off-by: chaojun-zhang <chaojun.zhang@intel.com> Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>

fix mypy error

c9145ad

Signed-off-by: Chaojun,Zhang <chaojun.zhang@intel.com> Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>

Merge branch 'main' into seq_parallel

d36760f

Merge branch 'main' into seq_parallel

eedc22e

Merge branch 'main' into seq_parallel

2039916

Uh oh!

Uh oh!

Conversation

chaojun-zhang commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Plan

Test Result

Accuracy (GSM8K)

meta-llama/Llama-2-13b-chat-hf

Qwen/Qwen3-32B

Benchmark

meta-llama/Llama-2-13b-chat-hf

Qwen/Qwen3-32B

Summary

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mergify Bot commented Mar 31, 2026

Uh oh!

yma11 commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chaojun-zhang commented Apr 2, 2026

Uh oh!

ProExpertProg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jikunshang Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jikunshang Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

chaojun-zhang Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jikunshang Apr 24, 2026

Choose a reason for hiding this comment

Uh oh!

chaojun-zhang Apr 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Apr 15, 2026

Uh oh!

mergify Bot commented May 6, 2026

Uh oh!

mergify Bot commented May 6, 2026

Uh oh!

mergify Bot commented May 6, 2026

Uh oh!

mergify Bot commented May 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

chaojun-zhang commented Mar 31, 2026 •

edited

Loading

yma11 commented Apr 2, 2026 •

edited

Loading

chaojun-zhang Apr 9, 2026 •

edited

Loading

chaojun-zhang Apr 26, 2026 •

edited

Loading