Conversation
Signed-off-by: qsunnyy <qybottle@163.com> Signed-off-by: q00832892 <qiaoyang19@huawei.com>
Signed-off-by: qsunnyy <qybottle@163.com> Signed-off-by: q00832892 <qiaoyang19@huawei.com>
Signed-off-by: q00832892 <qiaoyang19@huawei.com>
Signed-off-by: q00832892 <qiaoyang19@huawei.com>
Signed-off-by: q00832892 <qiaoyang19@huawei.com>
MengqingCao
reviewed
Mar 18, 2025
| from vllm.distributed.parallel_state import ( | ||
| destroy_distributed_environment, destroy_model_parallel) | ||
|
|
||
| import vllm_ascend # noqa |
Collaborator
There was a problem hiding this comment.
I think there is no need to import vllm-ascend
| from . import config # noqa | ||
| from . import forward_context # noqa | ||
| from .distributed import utils # noqa | ||
| from .engine import llm_engine # noqa |
Collaborator
There was a problem hiding this comment.
these patches seems no related to model registion, if we have to do so, let's do patch in vllm_ascend/patch
| backend_class._set_sequence_number_for_group() | ||
| pg._register_backend(device, backend_type, backend_class) | ||
|
|
||
| elif backend == "nccl": |
Collaborator
There was a problem hiding this comment.
Let's remove the cuda related code here
| LLMEngine.__init__ = new_init | ||
| LLMEngine.has_unfinished_requests = new_has_unfinished_requests | ||
| LLMEngine.has_unfinished_requests_dp = new_has_unfinished_requests_dp | ||
| print("[Success] Custom LLMEngine patch applied!") |
Collaborator
There was a problem hiding this comment.
why do we need to patch engine? I don't think it's a good idea to patch engine
| dtype=torch.int32) | ||
| from vllm.distributed.parallel_state import get_dp_group | ||
| dist.all_reduce(num_tokens_tensor, group=get_dp_group().device_group) | ||
| cu_tokens_across_dp_npu = torch.cumsum(num_tokens_tensor, dim=0) |
Collaborator
There was a problem hiding this comment.
why gpu could use cu_tokens_across_dp_cpu tensor, but we need to use a npu tensor? same question for process group.
| # we use synchronous scheduling right now, | ||
| # adding a sync point here should not affect | ||
| # scheduling of the next batch | ||
| torch.cuda.synchronize() |
Collaborator
There was a problem hiding this comment.
Suggested change
| torch.cuda.synchronize() | |
| torch.npu.synchronize() |
linfeng-yuan
pushed a commit
that referenced
this pull request
Aug 5, 2026
…n the max number of tokens across all DP ranks for collective communications. (#11213) ### What this PR does / why we need it? Fix `XliteWrapper` to consume the correct DP metadata and align the max number of tokens across all DP ranks for collective communications, and bump the pinned `xlite` dependency to `0.2.0rc0`. **Why**: Under data parallelism (DP), each DP rank serves a different number of tokens, but the xlite runtime issues collective communications that require every rank to pass an identical, padded token count. Previously `XliteWrapper` read `forward_context.batch_descriptor.num_tokens` (the per-rank token count) and sized the working `hidden_states` slice from it. That value diverges across DP ranks, so the collectives mismatched and produced wrong/unaligned results. The fix sources the token count from `forward_context.max_tokens_across_dp`, which the vLLM-Ascend forward context already computes as `max(num_tokens_across_dp_cpu)` and pads to a TP-aligned length — the same value the rest of the stack uses for MoE comm method selection. The detailed issue can be found in xlite repo's [issue #15](https://atomgit.com/openeuler/GVirt/issues/15). And xlite [PR #339](https://atomgit.com/openeuler/GVirt/pull/339) fixes this issue in xlite backend. **What changed** (`vllm_ascend/xlite/xlite.py`): - In the xlite forward path, `num_tokens` is now taken from `forward_context.max_tokens_across_dp` instead of `forward_context.batch_descriptor.num_tokens`. The hidden-states buffer slice `self.hidden_states[:num_tokens]` is therefore sized to the DP-aligned maximum, keeping all ranks in lock-step for the underlying collectives. The comment is updated to record that, under DP, `num_tokens` is the max number of tokens across all DP ranks for data alignment. - `XliteWrapper.__init__` no longer derives `local_rank` from `get_world_group().local_rank`; the runtime `devid` is now `device.index` and `rank` is `torch.distributed.get_rank()` directly. `get_world_group` is dropped from the imports, and the one-shot `rank`/`local_rank` locals are inlined at their use sites. - `StandardXliteModel`: `xlite_config.max_m` (the full-mode capacity) is now rounded up with `math.ceil(max_num_batched_tokens / tp_size) * tp_size` so it is TP-aligned and consistent with the downstream. - `MiniMaxM2XliteModel`: sets `xlite_config.gate_captured = False`. - `requirements-dev.txt`: `xlite==0.1.0rc12.dev210` → `xlite==0.2.0rc0` to pick up the upstream runtime fixes that the metadata alignment above relies on. DP test cases were also added to the e2e script at `tests/e2e/pull_request/two_card/test_xlite.py`. ### Does this PR introduce _any_ user-facing change? No public API change. This fixes an internal correctness bug in `XliteWrapper`'s DP metadata handling and bumps the dev-only `xlite` pin in `requirements-dev.txt`. Serving behavior for non-DP xlite configurations is unchanged; DP configurations now use the DP-aligned token count for collective communications. ### How was this patch tested? The e2e test passes locally with `pytest tests/e2e/pull_request/two_card/test_xlite.py`. We also benchmarked model accuracies with multiple models using `aisbench`. The [batch_aisbench.py](https://atomgit.com/openeuler/GVirt/blob/master/xlite/tests/e2e/batch_aisbench.py) script was used with the `ceval` dataset, comparing `xlite full`, `xlite decode-only`, and `aclgraph` backends across dense, MoE, and DP configurations: ```bash # on one Atlas A3 python batch_aisbench.py /root/benchmark \ -N 128 \ --models Qwen3-32B Qwen3-VL-32B-Instruct 2~Qwen3-30B-A3B 2~MiniMax-M2.7-w8a8-QuaRot 2~GLM-4.7-W8A8-floatmtp \ --tps 4 4 4 2 8 4 8 4 \ --dps 1 1 1 1 1 2 1 2 \ --eps 0 0 1 1 1 1 1 1 \ -MNS 128 \ -MML 8192 \ --xlite 2 1 0 \ --broadcast-xlite ``` **No accuracy regression is observed across the full matrix relative to the `aclgraph` baseline:** | Model | Dataset | TP | EP | DP | Backend | Metric | Accuracy | | --- | --- | --- | --- | --- | --- | --- | --- | | Qwen3-32B | ceval-weighted | 4 | N | 1 | xlite full | weighted_average | 88.26 | | Qwen3-32B | ceval-weighted | 4 | N | 1 | xlite decode-only | weighted_average | 88.19 | | Qwen3-32B | ceval-weighted | 4 | N | 1 | aclgraph | weighted_average | 86.70 | | Model | Dataset | TP | EP | DP | Backend | Metric | Accuracy | | --- | --- | --- | --- | --- | --- | --- | --- | | Qwen3-VL-32B-Instruct | ceval-weighted | 4 | N | 1 | xlite full | weighted_average | 86.92 | | Qwen3-VL-32B-Instruct | ceval-weighted | 4 | N | 1 | xlite decode-only | weighted_average | 86.85 | | Qwen3-VL-32B-Instruct | ceval-weighted | 4 | N | 1 | aclgraph | weighted_average | 87.44 | | Model | Dataset | TP | EP | DP | Backend | Metric | Accuracy | | --- | --- | --- | --- | --- | --- | --- | --- | | Qwen3-30B-A3B | ceval-weighted | 4 | Y | 1 | xlite full | weighted_average | 85.22 | | Qwen3-30B-A3B | ceval-weighted | 4 | Y | 1 | xlite decode-only | weighted_average | 85.74 | | Qwen3-30B-A3B | ceval-weighted | 4 | Y | 1 | aclgraph | weighted_average | 85.74 | | Qwen3-30B-A3B | ceval-weighted | 2 | Y | 1 | xlite full | weighted_average | 85.74 | | Qwen3-30B-A3B | ceval-weighted | 2 | Y | 1 | xlite decode-only | weighted_average | 86.11 | | Qwen3-30B-A3B | ceval-weighted | 2 | Y | 1 | aclgraph | weighted_average | 85.14 | | Model | Dataset | TP | EP | DP | Backend | Metric | Accuracy | | --- | --- | --- | --- | --- | --- | --- | --- | | MiniMax-M2.7-w8a8-QuaRot | ceval-weighted | 8 | Y | 1 | xlite full | weighted_average | 82.24 | | MiniMax-M2.7-w8a8-QuaRot | ceval-weighted | 8 | Y | 1 | xlite decode-only | weighted_average | 82.76 | | MiniMax-M2.7-w8a8-QuaRot | ceval-weighted | 8 | Y | 1 | aclgraph | weighted_average | 81.80 | | MiniMax-M2.7-w8a8-QuaRot | ceval-weighted | 4 | Y | 2 | xlite full | weighted_average | 83.43 | | MiniMax-M2.7-w8a8-QuaRot | ceval-weighted | 4 | Y | 2 | xlite decode-only | weighted_average | 84.03 | | MiniMax-M2.7-w8a8-QuaRot | ceval-weighted | 4 | Y | 2 | aclgraph | weighted_average | 82.47 | | Model | Dataset | TP | EP | DP | Backend | Metric | Accuracy | | --- | --- | --- | --- | --- | --- | --- | --- | | GLM-4.7-W8A8-floatmtp | ceval-weighted | 8 | Y | 1 | xlite full | weighted_average | 90.71 | | GLM-4.7-W8A8-floatmtp | ceval-weighted | 8 | Y | 1 | xlite decode-only | weighted_average | 91.01 | | GLM-4.7-W8A8-floatmtp | ceval-weighted | 8 | Y | 1 | aclgraph | weighted_average | 90.49 | | GLM-4.7-W8A8-floatmtp | ceval-weighted | 4 | Y | 2 | xlite full | weighted_average | 90.12 | | GLM-4.7-W8A8-floatmtp | ceval-weighted | 4 | Y | 2 | xlite decode-only | weighted_average | 90.56 | | GLM-4.7-W8A8-floatmtp | ceval-weighted | 4 | Y | 2 | aclgraph | weighted_average | 90.27 | - vLLM version: v0.26.0 - vLLM main: vllm-project/vllm@0351e9a Signed-off-by: Sijie Fu <fusijie@huawei.com>
HMCCMH
pushed a commit
to hotTea123/vllm-ascend
that referenced
this pull request
Aug 12, 2026
…n the max number of tokens across all DP ranks for collective communications. (vllm-project#11213) ### What this PR does / why we need it? Fix `XliteWrapper` to consume the correct DP metadata and align the max number of tokens across all DP ranks for collective communications, and bump the pinned `xlite` dependency to `0.2.0rc0`. **Why**: Under data parallelism (DP), each DP rank serves a different number of tokens, but the xlite runtime issues collective communications that require every rank to pass an identical, padded token count. Previously `XliteWrapper` read `forward_context.batch_descriptor.num_tokens` (the per-rank token count) and sized the working `hidden_states` slice from it. That value diverges across DP ranks, so the collectives mismatched and produced wrong/unaligned results. The fix sources the token count from `forward_context.max_tokens_across_dp`, which the vLLM-Ascend forward context already computes as `max(num_tokens_across_dp_cpu)` and pads to a TP-aligned length — the same value the rest of the stack uses for MoE comm method selection. The detailed issue can be found in xlite repo's [issue vllm-project#15](https://atomgit.com/openeuler/GVirt/issues/15). And xlite [PR vllm-project#339](https://atomgit.com/openeuler/GVirt/pull/339) fixes this issue in xlite backend. **What changed** (`vllm_ascend/xlite/xlite.py`): - In the xlite forward path, `num_tokens` is now taken from `forward_context.max_tokens_across_dp` instead of `forward_context.batch_descriptor.num_tokens`. The hidden-states buffer slice `self.hidden_states[:num_tokens]` is therefore sized to the DP-aligned maximum, keeping all ranks in lock-step for the underlying collectives. The comment is updated to record that, under DP, `num_tokens` is the max number of tokens across all DP ranks for data alignment. - `XliteWrapper.__init__` no longer derives `local_rank` from `get_world_group().local_rank`; the runtime `devid` is now `device.index` and `rank` is `torch.distributed.get_rank()` directly. `get_world_group` is dropped from the imports, and the one-shot `rank`/`local_rank` locals are inlined at their use sites. - `StandardXliteModel`: `xlite_config.max_m` (the full-mode capacity) is now rounded up with `math.ceil(max_num_batched_tokens / tp_size) * tp_size` so it is TP-aligned and consistent with the downstream. - `MiniMaxM2XliteModel`: sets `xlite_config.gate_captured = False`. - `requirements-dev.txt`: `xlite==0.1.0rc12.dev210` → `xlite==0.2.0rc0` to pick up the upstream runtime fixes that the metadata alignment above relies on. DP test cases were also added to the e2e script at `tests/e2e/pull_request/two_card/test_xlite.py`. ### Does this PR introduce _any_ user-facing change? No public API change. This fixes an internal correctness bug in `XliteWrapper`'s DP metadata handling and bumps the dev-only `xlite` pin in `requirements-dev.txt`. Serving behavior for non-DP xlite configurations is unchanged; DP configurations now use the DP-aligned token count for collective communications. ### How was this patch tested? The e2e test passes locally with `pytest tests/e2e/pull_request/two_card/test_xlite.py`. We also benchmarked model accuracies with multiple models using `aisbench`. The [batch_aisbench.py](https://atomgit.com/openeuler/GVirt/blob/master/xlite/tests/e2e/batch_aisbench.py) script was used with the `ceval` dataset, comparing `xlite full`, `xlite decode-only`, and `aclgraph` backends across dense, MoE, and DP configurations: ```bash # on one Atlas A3 python batch_aisbench.py /root/benchmark \ -N 128 \ --models Qwen3-32B Qwen3-VL-32B-Instruct 2~Qwen3-30B-A3B 2~MiniMax-M2.7-w8a8-QuaRot 2~GLM-4.7-W8A8-floatmtp \ --tps 4 4 4 2 8 4 8 4 \ --dps 1 1 1 1 1 2 1 2 \ --eps 0 0 1 1 1 1 1 1 \ -MNS 128 \ -MML 8192 \ --xlite 2 1 0 \ --broadcast-xlite ``` **No accuracy regression is observed across the full matrix relative to the `aclgraph` baseline:** | Model | Dataset | TP | EP | DP | Backend | Metric | Accuracy | | --- | --- | --- | --- | --- | --- | --- | --- | | Qwen3-32B | ceval-weighted | 4 | N | 1 | xlite full | weighted_average | 88.26 | | Qwen3-32B | ceval-weighted | 4 | N | 1 | xlite decode-only | weighted_average | 88.19 | | Qwen3-32B | ceval-weighted | 4 | N | 1 | aclgraph | weighted_average | 86.70 | | Model | Dataset | TP | EP | DP | Backend | Metric | Accuracy | | --- | --- | --- | --- | --- | --- | --- | --- | | Qwen3-VL-32B-Instruct | ceval-weighted | 4 | N | 1 | xlite full | weighted_average | 86.92 | | Qwen3-VL-32B-Instruct | ceval-weighted | 4 | N | 1 | xlite decode-only | weighted_average | 86.85 | | Qwen3-VL-32B-Instruct | ceval-weighted | 4 | N | 1 | aclgraph | weighted_average | 87.44 | | Model | Dataset | TP | EP | DP | Backend | Metric | Accuracy | | --- | --- | --- | --- | --- | --- | --- | --- | | Qwen3-30B-A3B | ceval-weighted | 4 | Y | 1 | xlite full | weighted_average | 85.22 | | Qwen3-30B-A3B | ceval-weighted | 4 | Y | 1 | xlite decode-only | weighted_average | 85.74 | | Qwen3-30B-A3B | ceval-weighted | 4 | Y | 1 | aclgraph | weighted_average | 85.74 | | Qwen3-30B-A3B | ceval-weighted | 2 | Y | 1 | xlite full | weighted_average | 85.74 | | Qwen3-30B-A3B | ceval-weighted | 2 | Y | 1 | xlite decode-only | weighted_average | 86.11 | | Qwen3-30B-A3B | ceval-weighted | 2 | Y | 1 | aclgraph | weighted_average | 85.14 | | Model | Dataset | TP | EP | DP | Backend | Metric | Accuracy | | --- | --- | --- | --- | --- | --- | --- | --- | | MiniMax-M2.7-w8a8-QuaRot | ceval-weighted | 8 | Y | 1 | xlite full | weighted_average | 82.24 | | MiniMax-M2.7-w8a8-QuaRot | ceval-weighted | 8 | Y | 1 | xlite decode-only | weighted_average | 82.76 | | MiniMax-M2.7-w8a8-QuaRot | ceval-weighted | 8 | Y | 1 | aclgraph | weighted_average | 81.80 | | MiniMax-M2.7-w8a8-QuaRot | ceval-weighted | 4 | Y | 2 | xlite full | weighted_average | 83.43 | | MiniMax-M2.7-w8a8-QuaRot | ceval-weighted | 4 | Y | 2 | xlite decode-only | weighted_average | 84.03 | | MiniMax-M2.7-w8a8-QuaRot | ceval-weighted | 4 | Y | 2 | aclgraph | weighted_average | 82.47 | | Model | Dataset | TP | EP | DP | Backend | Metric | Accuracy | | --- | --- | --- | --- | --- | --- | --- | --- | | GLM-4.7-W8A8-floatmtp | ceval-weighted | 8 | Y | 1 | xlite full | weighted_average | 90.71 | | GLM-4.7-W8A8-floatmtp | ceval-weighted | 8 | Y | 1 | xlite decode-only | weighted_average | 91.01 | | GLM-4.7-W8A8-floatmtp | ceval-weighted | 8 | Y | 1 | aclgraph | weighted_average | 90.49 | | GLM-4.7-W8A8-floatmtp | ceval-weighted | 4 | Y | 2 | xlite full | weighted_average | 90.12 | | GLM-4.7-W8A8-floatmtp | ceval-weighted | 4 | Y | 2 | xlite decode-only | weighted_average | 90.56 | | GLM-4.7-W8A8-floatmtp | ceval-weighted | 4 | Y | 2 | aclgraph | weighted_average | 90.27 | - vLLM version: v0.26.0 - vLLM main: vllm-project/vllm@0351e9a Signed-off-by: Sijie Fu <fusijie@huawei.com>
MmMmaru
pushed a commit
to jiaqi-lee/vllm-ascend
that referenced
this pull request
Aug 19, 2026
…n the max number of tokens across all DP ranks for collective communications. (vllm-project#11213) ### What this PR does / why we need it? Fix `XliteWrapper` to consume the correct DP metadata and align the max number of tokens across all DP ranks for collective communications, and bump the pinned `xlite` dependency to `0.2.0rc0`. **Why**: Under data parallelism (DP), each DP rank serves a different number of tokens, but the xlite runtime issues collective communications that require every rank to pass an identical, padded token count. Previously `XliteWrapper` read `forward_context.batch_descriptor.num_tokens` (the per-rank token count) and sized the working `hidden_states` slice from it. That value diverges across DP ranks, so the collectives mismatched and produced wrong/unaligned results. The fix sources the token count from `forward_context.max_tokens_across_dp`, which the vLLM-Ascend forward context already computes as `max(num_tokens_across_dp_cpu)` and pads to a TP-aligned length — the same value the rest of the stack uses for MoE comm method selection. The detailed issue can be found in xlite repo's [issue vllm-project#15](https://atomgit.com/openeuler/GVirt/issues/15). And xlite [PR vllm-project#339](https://atomgit.com/openeuler/GVirt/pull/339) fixes this issue in xlite backend. **What changed** (`vllm_ascend/xlite/xlite.py`): - In the xlite forward path, `num_tokens` is now taken from `forward_context.max_tokens_across_dp` instead of `forward_context.batch_descriptor.num_tokens`. The hidden-states buffer slice `self.hidden_states[:num_tokens]` is therefore sized to the DP-aligned maximum, keeping all ranks in lock-step for the underlying collectives. The comment is updated to record that, under DP, `num_tokens` is the max number of tokens across all DP ranks for data alignment. - `XliteWrapper.__init__` no longer derives `local_rank` from `get_world_group().local_rank`; the runtime `devid` is now `device.index` and `rank` is `torch.distributed.get_rank()` directly. `get_world_group` is dropped from the imports, and the one-shot `rank`/`local_rank` locals are inlined at their use sites. - `StandardXliteModel`: `xlite_config.max_m` (the full-mode capacity) is now rounded up with `math.ceil(max_num_batched_tokens / tp_size) * tp_size` so it is TP-aligned and consistent with the downstream. - `MiniMaxM2XliteModel`: sets `xlite_config.gate_captured = False`. - `requirements-dev.txt`: `xlite==0.1.0rc12.dev210` → `xlite==0.2.0rc0` to pick up the upstream runtime fixes that the metadata alignment above relies on. DP test cases were also added to the e2e script at `tests/e2e/pull_request/two_card/test_xlite.py`. ### Does this PR introduce _any_ user-facing change? No public API change. This fixes an internal correctness bug in `XliteWrapper`'s DP metadata handling and bumps the dev-only `xlite` pin in `requirements-dev.txt`. Serving behavior for non-DP xlite configurations is unchanged; DP configurations now use the DP-aligned token count for collective communications. ### How was this patch tested? The e2e test passes locally with `pytest tests/e2e/pull_request/two_card/test_xlite.py`. We also benchmarked model accuracies with multiple models using `aisbench`. The [batch_aisbench.py](https://atomgit.com/openeuler/GVirt/blob/master/xlite/tests/e2e/batch_aisbench.py) script was used with the `ceval` dataset, comparing `xlite full`, `xlite decode-only`, and `aclgraph` backends across dense, MoE, and DP configurations: ```bash # on one Atlas A3 python batch_aisbench.py /root/benchmark \ -N 128 \ --models Qwen3-32B Qwen3-VL-32B-Instruct 2~Qwen3-30B-A3B 2~MiniMax-M2.7-w8a8-QuaRot 2~GLM-4.7-W8A8-floatmtp \ --tps 4 4 4 2 8 4 8 4 \ --dps 1 1 1 1 1 2 1 2 \ --eps 0 0 1 1 1 1 1 1 \ -MNS 128 \ -MML 8192 \ --xlite 2 1 0 \ --broadcast-xlite ``` **No accuracy regression is observed across the full matrix relative to the `aclgraph` baseline:** | Model | Dataset | TP | EP | DP | Backend | Metric | Accuracy | | --- | --- | --- | --- | --- | --- | --- | --- | | Qwen3-32B | ceval-weighted | 4 | N | 1 | xlite full | weighted_average | 88.26 | | Qwen3-32B | ceval-weighted | 4 | N | 1 | xlite decode-only | weighted_average | 88.19 | | Qwen3-32B | ceval-weighted | 4 | N | 1 | aclgraph | weighted_average | 86.70 | | Model | Dataset | TP | EP | DP | Backend | Metric | Accuracy | | --- | --- | --- | --- | --- | --- | --- | --- | | Qwen3-VL-32B-Instruct | ceval-weighted | 4 | N | 1 | xlite full | weighted_average | 86.92 | | Qwen3-VL-32B-Instruct | ceval-weighted | 4 | N | 1 | xlite decode-only | weighted_average | 86.85 | | Qwen3-VL-32B-Instruct | ceval-weighted | 4 | N | 1 | aclgraph | weighted_average | 87.44 | | Model | Dataset | TP | EP | DP | Backend | Metric | Accuracy | | --- | --- | --- | --- | --- | --- | --- | --- | | Qwen3-30B-A3B | ceval-weighted | 4 | Y | 1 | xlite full | weighted_average | 85.22 | | Qwen3-30B-A3B | ceval-weighted | 4 | Y | 1 | xlite decode-only | weighted_average | 85.74 | | Qwen3-30B-A3B | ceval-weighted | 4 | Y | 1 | aclgraph | weighted_average | 85.74 | | Qwen3-30B-A3B | ceval-weighted | 2 | Y | 1 | xlite full | weighted_average | 85.74 | | Qwen3-30B-A3B | ceval-weighted | 2 | Y | 1 | xlite decode-only | weighted_average | 86.11 | | Qwen3-30B-A3B | ceval-weighted | 2 | Y | 1 | aclgraph | weighted_average | 85.14 | | Model | Dataset | TP | EP | DP | Backend | Metric | Accuracy | | --- | --- | --- | --- | --- | --- | --- | --- | | MiniMax-M2.7-w8a8-QuaRot | ceval-weighted | 8 | Y | 1 | xlite full | weighted_average | 82.24 | | MiniMax-M2.7-w8a8-QuaRot | ceval-weighted | 8 | Y | 1 | xlite decode-only | weighted_average | 82.76 | | MiniMax-M2.7-w8a8-QuaRot | ceval-weighted | 8 | Y | 1 | aclgraph | weighted_average | 81.80 | | MiniMax-M2.7-w8a8-QuaRot | ceval-weighted | 4 | Y | 2 | xlite full | weighted_average | 83.43 | | MiniMax-M2.7-w8a8-QuaRot | ceval-weighted | 4 | Y | 2 | xlite decode-only | weighted_average | 84.03 | | MiniMax-M2.7-w8a8-QuaRot | ceval-weighted | 4 | Y | 2 | aclgraph | weighted_average | 82.47 | | Model | Dataset | TP | EP | DP | Backend | Metric | Accuracy | | --- | --- | --- | --- | --- | --- | --- | --- | | GLM-4.7-W8A8-floatmtp | ceval-weighted | 8 | Y | 1 | xlite full | weighted_average | 90.71 | | GLM-4.7-W8A8-floatmtp | ceval-weighted | 8 | Y | 1 | xlite decode-only | weighted_average | 91.01 | | GLM-4.7-W8A8-floatmtp | ceval-weighted | 8 | Y | 1 | aclgraph | weighted_average | 90.49 | | GLM-4.7-W8A8-floatmtp | ceval-weighted | 4 | Y | 2 | xlite full | weighted_average | 90.12 | | GLM-4.7-W8A8-floatmtp | ceval-weighted | 4 | Y | 2 | xlite decode-only | weighted_average | 90.56 | | GLM-4.7-W8A8-floatmtp | ceval-weighted | 4 | Y | 2 | aclgraph | weighted_average | 90.27 | - vLLM version: v0.26.0 - vLLM main: vllm-project/vllm@0351e9a Signed-off-by: Sijie Fu <fusijie@huawei.com>
shiqiangA
pushed a commit
to shiqiangA/vllm-ascend
that referenced
this pull request
Aug 20, 2026
…n the max number of tokens across all DP ranks for collective communications. (vllm-project#11213) ### What this PR does / why we need it? Fix `XliteWrapper` to consume the correct DP metadata and align the max number of tokens across all DP ranks for collective communications, and bump the pinned `xlite` dependency to `0.2.0rc0`. **Why**: Under data parallelism (DP), each DP rank serves a different number of tokens, but the xlite runtime issues collective communications that require every rank to pass an identical, padded token count. Previously `XliteWrapper` read `forward_context.batch_descriptor.num_tokens` (the per-rank token count) and sized the working `hidden_states` slice from it. That value diverges across DP ranks, so the collectives mismatched and produced wrong/unaligned results. The fix sources the token count from `forward_context.max_tokens_across_dp`, which the vLLM-Ascend forward context already computes as `max(num_tokens_across_dp_cpu)` and pads to a TP-aligned length — the same value the rest of the stack uses for MoE comm method selection. The detailed issue can be found in xlite repo's [issue vllm-project#15](https://atomgit.com/openeuler/GVirt/issues/15). And xlite [PR vllm-project#339](https://atomgit.com/openeuler/GVirt/pull/339) fixes this issue in xlite backend. **What changed** (`vllm_ascend/xlite/xlite.py`): - In the xlite forward path, `num_tokens` is now taken from `forward_context.max_tokens_across_dp` instead of `forward_context.batch_descriptor.num_tokens`. The hidden-states buffer slice `self.hidden_states[:num_tokens]` is therefore sized to the DP-aligned maximum, keeping all ranks in lock-step for the underlying collectives. The comment is updated to record that, under DP, `num_tokens` is the max number of tokens across all DP ranks for data alignment. - `XliteWrapper.__init__` no longer derives `local_rank` from `get_world_group().local_rank`; the runtime `devid` is now `device.index` and `rank` is `torch.distributed.get_rank()` directly. `get_world_group` is dropped from the imports, and the one-shot `rank`/`local_rank` locals are inlined at their use sites. - `StandardXliteModel`: `xlite_config.max_m` (the full-mode capacity) is now rounded up with `math.ceil(max_num_batched_tokens / tp_size) * tp_size` so it is TP-aligned and consistent with the downstream. - `MiniMaxM2XliteModel`: sets `xlite_config.gate_captured = False`. - `requirements-dev.txt`: `xlite==0.1.0rc12.dev210` → `xlite==0.2.0rc0` to pick up the upstream runtime fixes that the metadata alignment above relies on. DP test cases were also added to the e2e script at `tests/e2e/pull_request/two_card/test_xlite.py`. ### Does this PR introduce _any_ user-facing change? No public API change. This fixes an internal correctness bug in `XliteWrapper`'s DP metadata handling and bumps the dev-only `xlite` pin in `requirements-dev.txt`. Serving behavior for non-DP xlite configurations is unchanged; DP configurations now use the DP-aligned token count for collective communications. ### How was this patch tested? The e2e test passes locally with `pytest tests/e2e/pull_request/two_card/test_xlite.py`. We also benchmarked model accuracies with multiple models using `aisbench`. The [batch_aisbench.py](https://atomgit.com/openeuler/GVirt/blob/master/xlite/tests/e2e/batch_aisbench.py) script was used with the `ceval` dataset, comparing `xlite full`, `xlite decode-only`, and `aclgraph` backends across dense, MoE, and DP configurations: ```bash # on one Atlas A3 python batch_aisbench.py /root/benchmark \ -N 128 \ --models Qwen3-32B Qwen3-VL-32B-Instruct 2~Qwen3-30B-A3B 2~MiniMax-M2.7-w8a8-QuaRot 2~GLM-4.7-W8A8-floatmtp \ --tps 4 4 4 2 8 4 8 4 \ --dps 1 1 1 1 1 2 1 2 \ --eps 0 0 1 1 1 1 1 1 \ -MNS 128 \ -MML 8192 \ --xlite 2 1 0 \ --broadcast-xlite ``` **No accuracy regression is observed across the full matrix relative to the `aclgraph` baseline:** | Model | Dataset | TP | EP | DP | Backend | Metric | Accuracy | | --- | --- | --- | --- | --- | --- | --- | --- | | Qwen3-32B | ceval-weighted | 4 | N | 1 | xlite full | weighted_average | 88.26 | | Qwen3-32B | ceval-weighted | 4 | N | 1 | xlite decode-only | weighted_average | 88.19 | | Qwen3-32B | ceval-weighted | 4 | N | 1 | aclgraph | weighted_average | 86.70 | | Model | Dataset | TP | EP | DP | Backend | Metric | Accuracy | | --- | --- | --- | --- | --- | --- | --- | --- | | Qwen3-VL-32B-Instruct | ceval-weighted | 4 | N | 1 | xlite full | weighted_average | 86.92 | | Qwen3-VL-32B-Instruct | ceval-weighted | 4 | N | 1 | xlite decode-only | weighted_average | 86.85 | | Qwen3-VL-32B-Instruct | ceval-weighted | 4 | N | 1 | aclgraph | weighted_average | 87.44 | | Model | Dataset | TP | EP | DP | Backend | Metric | Accuracy | | --- | --- | --- | --- | --- | --- | --- | --- | | Qwen3-30B-A3B | ceval-weighted | 4 | Y | 1 | xlite full | weighted_average | 85.22 | | Qwen3-30B-A3B | ceval-weighted | 4 | Y | 1 | xlite decode-only | weighted_average | 85.74 | | Qwen3-30B-A3B | ceval-weighted | 4 | Y | 1 | aclgraph | weighted_average | 85.74 | | Qwen3-30B-A3B | ceval-weighted | 2 | Y | 1 | xlite full | weighted_average | 85.74 | | Qwen3-30B-A3B | ceval-weighted | 2 | Y | 1 | xlite decode-only | weighted_average | 86.11 | | Qwen3-30B-A3B | ceval-weighted | 2 | Y | 1 | aclgraph | weighted_average | 85.14 | | Model | Dataset | TP | EP | DP | Backend | Metric | Accuracy | | --- | --- | --- | --- | --- | --- | --- | --- | | MiniMax-M2.7-w8a8-QuaRot | ceval-weighted | 8 | Y | 1 | xlite full | weighted_average | 82.24 | | MiniMax-M2.7-w8a8-QuaRot | ceval-weighted | 8 | Y | 1 | xlite decode-only | weighted_average | 82.76 | | MiniMax-M2.7-w8a8-QuaRot | ceval-weighted | 8 | Y | 1 | aclgraph | weighted_average | 81.80 | | MiniMax-M2.7-w8a8-QuaRot | ceval-weighted | 4 | Y | 2 | xlite full | weighted_average | 83.43 | | MiniMax-M2.7-w8a8-QuaRot | ceval-weighted | 4 | Y | 2 | xlite decode-only | weighted_average | 84.03 | | MiniMax-M2.7-w8a8-QuaRot | ceval-weighted | 4 | Y | 2 | aclgraph | weighted_average | 82.47 | | Model | Dataset | TP | EP | DP | Backend | Metric | Accuracy | | --- | --- | --- | --- | --- | --- | --- | --- | | GLM-4.7-W8A8-floatmtp | ceval-weighted | 8 | Y | 1 | xlite full | weighted_average | 90.71 | | GLM-4.7-W8A8-floatmtp | ceval-weighted | 8 | Y | 1 | xlite decode-only | weighted_average | 91.01 | | GLM-4.7-W8A8-floatmtp | ceval-weighted | 8 | Y | 1 | aclgraph | weighted_average | 90.49 | | GLM-4.7-W8A8-floatmtp | ceval-weighted | 4 | Y | 2 | xlite full | weighted_average | 90.12 | | GLM-4.7-W8A8-floatmtp | ceval-weighted | 4 | Y | 2 | xlite decode-only | weighted_average | 90.56 | | GLM-4.7-W8A8-floatmtp | ceval-weighted | 4 | Y | 2 | aclgraph | weighted_average | 90.27 | - vLLM version: v0.26.0 - vLLM main: vllm-project/vllm@0351e9a Signed-off-by: Sijie Fu <fusijie@huawei.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Set up data parallel communication, compatible with TP&EP.
Example commands to use data parallel: python examples/offline_inference_data_parallel.py