Skip to content

[Fusion] Add rmsnorm dynamic quant fusion pass#6274

Merged
wangxiyuan merged 1 commit intovllm-project:mainfrom
Zhang-Bryan:rmsnorm-quant
Feb 4, 2026
Merged

[Fusion] Add rmsnorm dynamic quant fusion pass#6274
wangxiyuan merged 1 commit intovllm-project:mainfrom
Zhang-Bryan:rmsnorm-quant

Conversation

@Zhang-Bryan
Copy link
Copy Markdown
Contributor

@Zhang-Bryan Zhang-Bryan commented Jan 26, 2026 •
edited
Loading

What this PR does / why we need it?

This PR introduces four new patterns to support the fusion of RMSNorm and DynamicQuant operators. After replacing the fusion operators, the execution time has been reduced from 22.8us to 16.9us.

Does this PR introduce any user-facing change?

N/A

How was this patch tested?

@Zhang-Bryan Zhang-Bryan requested a review from yiz-liu as a code owner January 26, 2026 09:53
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jan 26, 2026

👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:‌‌

  • A PR should do only one thing, smaller PRs enable faster reviews.
  • Every PR should include unit tests and end-to-end tests ‌to ensure it works and is not broken by other future PRs.
  • Write the commit message by fulfilling the PR description to help reviewer and future developers understand.

If CI fails, you can run linting and testing checks locally according Contributing and Testing.

gemini-code-assist[bot]
gemini-code-assist bot reviewed Jan 26, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces new fusion patterns for AddRMSNorm and dynamic quantization. The changes add four new pattern matching classes to handle cases with and without bias, and with and without sequence parallelism.

My review has identified two critical bugs related to mismatched arguments between get_inputs and the pattern function in two of the new classes. These will cause runtime errors and need to be fixed.

Additionally, there is a significant amount of code duplication across the four new classes. I've provided a suggestion to refactor this to improve maintainability. Addressing these points will make the code more robust and easier to manage in the future.

Comment thread vllm_ascend/compilation/passes/norm_quant_fusion_pass.py Outdated
Comment thread vllm_ascend/compilation/passes/norm_quant_fusion_pass.py Outdated
Comment thread vllm_ascend/compilation/passes/norm_quant_fusion_pass.py
@Zhang-Bryan Zhang-Bryan force-pushed the rmsnorm-quant branch 3 times, most recently from 75f50f1 to de50342 Compare January 28, 2026 04:26
@Zhang-Bryan Zhang-Bryan changed the title Add new pattern form rmsnorm_quant_fusion_pass [Fusion] Add new pattern form rmsnorm_quant_fusion_pass Jan 28, 2026
@Zhang-Bryan Zhang-Bryan changed the title [Fusion] Add new pattern form rmsnorm_quant_fusion_pass [Fusion] Add rmsnorm dynamic quant fusion pass Jan 28, 2026
@wangxiyuan wangxiyuan added ready read for review ready-for-test start test by label for PR labels Jan 28, 2026
@Zhang-Bryan
[Fusion] Add new pattern form norm_quant_fusion_pass
1f612fc 
Signed-off-by: Bryan <250470359+Zhang-Bryan@users.noreply.github.com>
@wangxiyuan wangxiyuan merged commit 804a9ec into vllm-project:main Feb 4, 2026
26 checks passed
845473182 pushed a commit to 845473182/vllm-ascend that referenced this pull request Feb 6, 2026
Merge branch 'main' of https://github.com/vllm-project/vllm-ascend in…
d612874 
…to qwen3next_rebase

* 'main' of https://github.com/vllm-project/vllm-ascend: (59 commits)
  [Feat.]: 310p support MOE models (vllm-project#6530)
  [Doc] backport 0.13.0 release note (vllm-project#6584)
  [CI] Update UT CANN version to 8.5.0 for main branch (vllm-project#6564)
  [CI] Change A2 runner (vllm-project#6557)
  [Bugfix] Fix the incorrect use of the output parameter in _forward_fia_slidingwindow (vllm-project#6469)
  [main2main] upgrade vllm main 0202 (vllm-project#6560)
  [CI][npugraph_ex]Fix npugraph ex e2e test (vllm-project#6553)
  [Feature]KV pool supports sparse attention (vllm-project#6339)
  [bugfix]Fix accuracy issue in PCP/DCP with speculative decoding (vllm-project#6491)
  perf: adaptive block size selection in linear_persistent kernel (vllm-project#6537)
  [ModelRunner][Fix] Pads query_start_loc to satisfy FIA/TND constraint (vllm-project#6475)
  [Bugfix]Fix of Pooling Code and Update of Pooling Usage Guide (vllm-project#6126)
  [Fusion] Add rmsnorm dynamic quant fusion pass (vllm-project#6274)
  [Bugfix] Synchronize only the current stream to avoid device sync (vllm-project#6432)
  [CI] Add long and short prompt tests for DeepSeek-V3.2 (vllm-project#6499)
  [Refactor] MLP weight prefetch to consistency with MoE Model's prefetching in terms of code and usage (vllm-project#6442)
  [bugfix][npugraph_ex]duplicate pattern issue (vllm-project#6513)
  [bugfix][npugraph_ex]add the extra check for allreduce rmsnorm fusion pass (vllm-project#6430)
  [Quant] GLM4.7-Flash Support W8A8 (vllm-project#6492)
  [Nightly][BugFix] Remove kv_cache nz test case for test_mla_preprocess_nq.py (vllm-project#6505)
  ...
chenchuw886 pushed a commit to chenchuw886/vllm-ascend that referenced this pull request Feb 12, 2026
@Zhang-Bryan
[Fusion] Add rmsnorm dynamic quant fusion pass (vllm-project#6274)
a9a8f3e 
### What this PR does / why we need it?

This PR introduces four new patterns to support the fusion of RMSNorm
and DynamicQuant operators. After replacing the fusion operators, the
execution time has been reduced from 22.8us to 16.9us.

### Does this PR introduce _any_ user-facing change?

N/A

### How was this patch tested?

- vLLM version: v0.14.1
- vLLM main:
vllm-project/vllm@d7de043

Signed-off-by: Bryan <250470359+Zhang-Bryan@users.noreply.github.com>
Signed-off-by: momochenchuw <chenchuw@huawei.com>
@wangxiyuan wangxiyuan mentioned this pull request Feb 24, 2026
Closed
ZRJ026 pushed a commit to ZRJ026/vllm-ascend that referenced this pull request Feb 28, 2026
@Zhang-Bryan
[Fusion] Add rmsnorm dynamic quant fusion pass (vllm-project#6274)
9441255 
### What this PR does / why we need it?

This PR introduces four new patterns to support the fusion of RMSNorm
and DynamicQuant operators. After replacing the fusion operators, the
execution time has been reduced from 22.8us to 16.9us.

### Does this PR introduce _any_ user-facing change?

N/A

### How was this patch tested?

- vLLM version: v0.14.1
- vLLM main:
vllm-project/vllm@d7de043

Signed-off-by: Bryan <250470359+Zhang-Bryan@users.noreply.github.com>
Signed-off-by: zrj026 <zhangrunjiang026@gmail.com>
maoxx241 pushed a commit to maoxx241/vllm-ascend that referenced this pull request Mar 2, 2026
@Zhang-Bryan @maoxx241
[Fusion] Add rmsnorm dynamic quant fusion pass (vllm-project#6274)
3d5f2d9 
### What this PR does / why we need it?

This PR introduces four new patterns to support the fusion of RMSNorm
and DynamicQuant operators. After replacing the fusion operators, the
execution time has been reduced from 22.8us to 16.9us.

### Does this PR introduce _any_ user-facing change?

N/A

### How was this patch tested?


- vLLM version: v0.14.1
- vLLM main:
vllm-project/vllm@d7de043

Signed-off-by: Bryan <250470359+Zhang-Bryan@users.noreply.github.com>
ZRJ026 pushed a commit to ZRJ026/vllm-ascend that referenced this pull request Mar 4, 2026
@Zhang-Bryan
[Fusion] Add rmsnorm dynamic quant fusion pass (vllm-project#6274)
3fa4590 
### What this PR does / why we need it?

This PR introduces four new patterns to support the fusion of RMSNorm
and DynamicQuant operators. After replacing the fusion operators, the
execution time has been reduced from 22.8us to 16.9us.

### Does this PR introduce _any_ user-facing change?

N/A

### How was this patch tested?

- vLLM version: v0.14.1
- vLLM main:
vllm-project/vllm@d7de043

Signed-off-by: Bryan <250470359+Zhang-Bryan@users.noreply.github.com>
Signed-off-by: zrj026 <zhangrunjiang026@gmail.com>
LCAIZJ pushed a commit to LCAIZJ/vllm-ascend that referenced this pull request Mar 7, 2026
@Zhang-Bryan @LCAIZJ
[Fusion] Add rmsnorm dynamic quant fusion pass (vllm-project#6274)
c93ca9c 
### What this PR does / why we need it?

This PR introduces four new patterns to support the fusion of RMSNorm
and DynamicQuant operators. After replacing the fusion operators, the
execution time has been reduced from 22.8us to 16.9us.

### Does this PR introduce _any_ user-facing change?

N/A

### How was this patch tested?


- vLLM version: v0.14.1
- vLLM main:
vllm-project/vllm@d7de043

Signed-off-by: Bryan <250470359+Zhang-Bryan@users.noreply.github.com>
jiangyunfan1 pushed a commit to jiangyunfan1/vllm-ascend that referenced this pull request Apr 9, 2026
@Zhang-Bryan @jiangyunfan1
[Fusion] Add rmsnorm dynamic quant fusion pass (vllm-project#6274)
c511aa6 
### What this PR does / why we need it?

This PR introduces four new patterns to support the fusion of RMSNorm
and DynamicQuant operators. After replacing the fusion operators, the
execution time has been reduced from 22.8us to 16.9us.

### Does this PR introduce _any_ user-facing change?

N/A

### How was this patch tested?


- vLLM version: v0.14.1
- vLLM main:
vllm-project/vllm@d7de043

Signed-off-by: Bryan <250470359+Zhang-Bryan@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wangxiyuan wangxiyuan wangxiyuan approved these changes

@yiz-liu yiz-liu Awaiting requested review from yiz-liu yiz-liu is a code owner

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ready read for review ready-for-test start test by label for PR

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Zhang-Bryan @wangxiyuan