Skip to content
Merged
Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
ce98bcd
fix(compilation): improve Ascend SP compile flow and no-op cleanup
realliujiaxu Mar 5, 2026
9ee5c66
fix(compilation): add SP cleanup pass for all_gather+chunk no-op
realliujiaxu Mar 5, 2026
eb13e84
remove info log
realliujiaxu Mar 6, 2026
c94c119
feat(compilation): 把moe相关的pattern提取为SequenceParallelismAllgatherEpPass
realliujiaxu Mar 6, 2026
c308e34
fix lint
realliujiaxu Mar 6, 2026
a3fe5bb
rename pass and update doc
realliujiaxu Mar 6, 2026
6aec611
fix lint
realliujiaxu Mar 6, 2026
66dcd91
replace sp_threshold with sp_min_token_num from vllm
realliujiaxu Mar 12, 2026
ed4a4d7
fix ut
realliujiaxu Mar 12, 2026
97d245c
support alltoall EP
realliujiaxu Mar 12, 2026
73e781e
add FAQ
realliujiaxu Mar 13, 2026
1e76e76
fix alltoall bug
realliujiaxu Mar 13, 2026
b75c23d
move enable_sp_by_pass to ascend config
realliujiaxu Mar 13, 2026
12bfad8
fix None type bug
realliujiaxu Mar 15, 2026
bd3aa0f
Add test for SequenceParallelismMoePass and update config.yaml
realliujiaxu Mar 19, 2026
d90ced1
add debug log
realliujiaxu Mar 19, 2026
2a98a8b
update ut and add comment to patch
realliujiaxu Mar 20, 2026
cfb44a1
test: speed up sequence parallelism moe coverage
realliujiaxu Mar 20, 2026
f64cbb9
fix mypy error
realliujiaxu Mar 20, 2026
4f03ab1
fix lint
realliujiaxu Mar 20, 2026
c35285c
delete allreudce and allgather custom op patch
realliujiaxu Mar 22, 2026
175efd5
remove redudant compile_ranges_split_points
realliujiaxu Mar 23, 2026
d66eee2
fix debug log and remove TODO
realliujiaxu Mar 24, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/scripts/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,8 @@ e2e-multicard-2-cards:
estimated_time: 164
- name: tests/e2e/multicard/2-cards/test_sp_pass.py
estimated_time: 198
- name: tests/e2e/multicard/2-cards/test_sequence_parallelism_moe.py
estimated_time: 120
e2e-multicard-4-cards:
- name: tests/e2e/multicard/4-cards/test_qwen3_next.py
estimated_time: 1868
Expand Down
Binary file added docs/source/assets/sp_moe.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,6 @@ The following table lists additional configuration options available in vLLM Asc
| `pa_shape_list` | list | `[]` | The custom shape list of page attention ops. |
| `enable_kv_nz` | bool | `False` | Whether to enable KV cache NZ layout. This option only takes effects on models using MLA (e.g., DeepSeek). |
| `layer_sharding` | dict | `{}` | Configuration options for Layer Sharding Linear |
| `sp_threshold` | int | `1000` | For dense models, only num_tokens > threshold will enable sequence parallelism. |

The details of each configuration option are as follows:

Expand Down
64 changes: 58 additions & 6 deletions docs/source/user_guide/feature_guide/sequence_parallelism.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,13 @@ Currently, vllm-ascend has implemented Sequence Parallelism for VL-class models
```bash
vllm serve Qwen/Qwen3-VL-2B-Instruct \
--tensor-parallel-size 2 \
--compilation-config '{"pass_config": {"enable_sp": true}}' \
--additional_config={"sp_threshold": 1000}
--compilation-config '{"pass_config": {"enable_sp": true, , "sp_min_token_num": 1000}}'
```

- `"pass_config": {"enable_sp": true}`: This is the switch for SP. Since SP relies on graph mode, it must be enabled and is not supported in eager mode.
- `--additional_config={"sp_threshold": 1000}`: Based on our experiments, when the number of tokens is small (empirical value is less than 1000), SP can actually bring negative benefits. This is because when the communication volume is small, the fixed overhead of the communication operator becomes the dominant factor. Therefore, when one communication operator (Allreduce) is split into two communication operators (ReduceScatter+Allgather), the end-to-end latency often becomes longer. Thus, we have reserved the `sp_threshold`parameter; SP will only take effect when `num_tokens >= sp_threshold`. **The default value is 1000, which generally does not need to be modified.** `sp_threshold` will be appended into `compile_ranges_split_points`, which is a parameter provided by vllm that splits the graph compilation range `[1, max_num_batched_tokens]` into `{[1, split_points[0]], [split_points[0] + 1, split_points[1]], ..., [split_points[-1] + 1, max_num_batched_tokens]}`, and sequentially checks whether the `is_applicable_for_range` of the pass returns `True`.
- `"enable_sp"`: This is the switch for SP. Since SP relies on graph mode, it is not supported in eager mode.
- `sp_min_token_num` (from upstream vllm's `pass_config`): Based on our experiments, when the number of tokens is small (empirical value is less than 1000), SP can actually bring negative benefits. This is because when the communication volume is small, the fixed overhead of the communication operator becomes the dominant factor. SP will only take effect when `num_tokens >= sp_min_token_num`. **The default value is 1000 on Ascend, which generally does not need to be modified.** To customize, use `--compilation-config '{"pass_config": {"enable_sp": true, "sp_min_token_num": 512}}'`. The value will be appended into `compile_ranges_split_points`, which splits the graph compilation range and checks whether the pass is applicable per range.

Without modifying `sp_threshold`, the simplest way and recommended way to enable SP is:
Without modifying `sp_min_token_num`, the simplest way and recommended way to enable SP is:

```bash
vllm serve Qwen/Qwen3-VL-2B-Instruct \
Expand All @@ -44,7 +43,7 @@ FC1 is a unique optimization in vllm-ascend, currently implemented based on Cust

| | VL + Dense | VL + MoE | non-VL + Dense | non-VL + MoE |
| -------------------- | ---------- | -------- | -------------- | ------------ |
| Sequence Parallelism | graph | x | x | x |
| Sequence Parallelism | graph | graph | x | x |
| Flash Comm V1 | x | x | eager/graph | eager/graph |

### With Quantization
Expand All @@ -55,3 +54,56 @@ SP currently does not support quantization and is under adaptation.
| -------------------- | ---------- | -------- | -------------- | ------------ |
| Sequence Parallelism | x | x | x | x |
| Flash Comm V1 | x | x | eager/graph | eager/graph |

## Pass Design

When SP is enabled, the following passes run in order: `SequenceParallelismPass` then `SequenceParallelismMoePass`.

### SequenceParallelismPass

Runs `NoOpEliminationPass` first to eliminate redundant view-like operations, then applies AllReduce-based patterns:

| Pattern | Match | Replacement |
| -------------------------------------- | -------------------------------- | ------------------------------------------------------------------------------------- |
| `MiddleAllReduceRMSNormPattern` | `all_reduce` + `layernorm` | `reduce_scatter` + `layernorm` + `all_gather` |
| `LastAllReduceRMSNormPattern` | Same (last layer, no residual) | Same |
| `Qwen3VLMiddleAllReduceRMSNormPattern` | `all_reduce` + add + `layernorm` | `reduce_scatter` + chunk(`deepstack_input_embeds`) + add + `layernorm` + `all_gather` |

**Why Qwen3 VL needs special handling by Qwen3VLMiddleAllReduceRMSNormPattern**

Qwen3-VL middle layers insert an extra add between `all_reduce` and `layernorm`: `hidden_states=hidden_states + deepstack_input_embeds`. Under SP, `hidden_states` (i.e., `input`) is reduced-scattered to shape `[seq_len/tp, hidden]` per rank, while `deepstack_input_embeds` comes from the vision/deepstack path and stays full-sequence `[seq_len, hidden]` (typically replicated across TP ranks). Simply doing `reduce_scatter(input) + deepstack_input_embeds` would cause a shape mismatch.
The fix is to chunk `deepstack_input_embeds` by `tp_size` so each rank uses `add(reduce_scatter, chunk(deepstack_input_embeds)[tp_rank])`, keeping shapes consistent before `layernorm` and `all_gather`.

### SequenceParallelismMoePass

After `SequenceParallelismPass` applies, the MoE model computation graph looks like:

![AllGather EP computation graph](../../assets/sp_moe.png)

**Overview**

1. **Postponing allgather**: Under SP, `residual` is chunked by tensor parallelism. This causes a shape mismatch between hidden states and residual in the next layer's layernorm: hidden states are gathered (full sequence) while residual remains chunked. The fix is to move `all_gather` to *after* layernorm so that layernorm operates on consistent shapes per rank. `MiddleLayerAllgatherAddRMSNormPattern`, `LastLayerAllgatherRMSNormPattern`, and `Qwen3VLMiddleLayerAllgatherAddRMSNormPattern` are designed for this purpose, each handling different layer and structure variants (see the table below).

2. **AllGatherChunkNoOp cleanup**: When MoE SP is enabled, vllm introduces a `sequence_parallel_chunk` op (corresponding to `sp_chunk` in the diagram). Together with the preceding `all_gather`, the pair forms a redundant no-op (all_gather gathers, then chunk re-splits). `AllGatherChunkNoOpPattern` replaces this pair with identity to eliminate the redundant communication and computation.

**Pattern details:**

| Pattern | Match | Replacement |
| ---------------------------------- | ---------------------------------------- | --------------------------------------- |
| `MiddleLayerAllgatherAddRMSNormPattern` | `all_gather` + slice + `layernorm` | `layernorm` + `all_gather` |
| `LastLayerAllgatherRMSNormPattern` | Same (last layer, no residual) | Same |
| `Qwen3VLMiddleLayerAllgatherAddRMSNormPattern` | `all_gather` + slice + add + `layernorm` | add(chunk) + `layernorm` + `all_gather` |
| `AllGatherChunkNoOpPattern` | `all_gather` + `sequence_parallel_chunk_impl` | identity (no-op) |

### FAQ

#### Q1: Is SP enabled by default?

No, SP is not enabled by default. SP is currently in the experimental stage and will be enabled by default in the future.

The processing flow of `enable_sp` in the code is:

- In `pass_config`, `enable_sp` and `sp_min_token_num` default to `None`
- `NPUPlatform.apply_config_platform_defaults`: If `enable_sp` is `True` and `sp_min_token_num` is None, set default `sp_min_token_num` (1000 for Dense models, 1 for MoE models)
- `VllmConfig._apply_optimization_level_defaults`: `enable_sp` is set to `True` for dense models.
- `VllmConfig.__post_init__`: If `sp_min_token_num` is still `None`, then `enable_sp` is set to `False`
Loading
Loading