[Feature][MRV1][MRV2] Refactor Host-Side Parameter Updates for ACL Graph Replay. by zhiyu-wa · Pull Request #15908 · vllm-project/vllm-ascend

zhiyu-wa · 2026-09-07T04:01:09Z

What this PR does / why we need it?
Implements the UpdatableGraph design discussed in #13058.
Refactors the FIA and Speculative Decoding to work with the new UpdatableGraph design.

Manually validated on A3 with the following scenarios:

FIA: Qwen3-0.6B with MRV1 and MRV2
MTP: Qwen3.5-27B with MRV1 and MRV2

Manually validated on A5 with the following scenarios:

FIA: Qwen3-0.6B with MRV1 and MRV2
MTP: Qwen3.5-9B with MRV1 and MRV2
PA: gemma4 with MRV1 and MRV2

Does this PR introduce any user-facing change?
No.

How was this patch tested?
Temporarily validated on some model.

vLLM main: vllm-project/vllm@b2f6858

gemini-code-assist · 2026-09-07T04:02:09Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces an UpdatableGraph architecture to replace static ACL graph parameter updates on Ascend NPUs. By decoupling parameter resolution from graph execution, this change enables more flexible and efficient handling of dynamic model parameters, specifically for Fused Infer Attention (FIA) and speculative decoding scenarios. The implementation includes a new task registration system and ensures compatibility with existing graph replay mechanisms across the Ascend backend.

Highlights

UpdatableGraph Architecture: Introduced a new UpdatableGraph class to manage dynamic parameter updates for ACL graph replay, decoupling parameter resolution from graph execution.
FIA and Speculative Decoding Refactor: Refactored AscendAttentionBackendImpl and speculative decoding workflows to utilize the UpdatableGraph design for improved flexibility.
Dynamic Parameter Resolution: Added FIAParamProvider to handle dynamic parameter resolution, ensuring correct metadata handling during graph replay.
Utility Enhancements: Updated weak_ref_tensors to support recursive weak referencing of tensors within nested containers.
Integration: Integrated UpdatableGraph into existing worker and graph manager workflows, including patching torch.cuda.CUDAGraph.

New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

github-actions · 2026-09-07T04:02:20Z

👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:‌‌

A PR should do only one thing, smaller PRs enable faster reviews.
Every PR should include unit tests and end-to-end tests ‌to ensure it works and is not broken by other future PRs.
Write the commit message by fulfilling the PR description to help reviewer and future developers understand.

If CI fails, you can run linting and testing checks locally according Contributing and Testing.

Tip

💡 Consider Linking a Related Issue or RFC

Your PR title contains the [Feature] tag, indicating a bug fix or new feature.

Linking a related issue or RFC in the PR description is strongly encouraged — it gives reviewers helpful context and speeds up the review. You can use any of these keywords:

Fixes #<issue_number>
Closes #<issue_number>
Resolves #<issue_number>
Refs #<rfc_or_issue_number> (for RFCs)

🙏 Thanks for helping us keep the project well-organized!

gemini-code-assist

Code Review

Suggested PR Title:

[Attention][Feature] Introduce UpdatableGraph for Fused Infer Attention on Ascend

Suggested PR Summary:

### What this PR does / why we need it?
This PR introduces `UpdatableGraph` to support updatable graphs for Fused Infer Attention (FIA) on Ascend NPUs, replacing the manual graph parameter updates with a task-registration and resolution system. This simplifies the graph replay path and improves maintainability.

Feedback has been provided to address several critical issues, including a signature mismatch in `use_updatable_graph`, potential `AttributeError`s when accessing uncompiled `_runnable` methods or `None` layers, unsafe `issubclass` checks, and the use of `assert` for runtime validation.

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Tested via existing CI and dummy runs for speculative decoding.

github-actions · 2026-09-08T16:41:43Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

github-actions · 2026-09-08T16:41:43Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

github-actions · 2026-09-09T06:25:14Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

github-actions · 2026-09-09T06:53:29Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

yiz-liu · 2026-09-11T03:53:47Z

/weekly GLM-5_1-W8A8_A3_weekly
weekly command triggered.

Weekly-A3: workflow run

yiz-liu · 2026-09-11T03:55:07Z

/weekly Minimax_m2.7_in198_prefix99
weekly command triggered.

Weekly-A3: workflow run

yiz-liu · 2026-09-11T03:56:27Z

/weekly Qwen3_5_397B_A17B_w8a8_mxfp8_A5
weekly command triggered.

github-actions · 2026-09-11T06:10:09Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

Signed-off-by: zhiyu-wa <1959864813@qq.com>

…r ACL Graph Replay. (vllm-project#15908)" This reverts commit daa6441. Signed-off-by: chenzeyu <2978509328@qq.com>

…aph Replay. (vllm-project#15908) **What this PR does / why we need it?** Implements the `UpdatableGraph` design discussed in vllm-project#13058. Refactors the `FIA` and `Speculative Decoding` to work with the new `UpdatableGraph` design. Manually validated on A3 with the following scenarios: - FIA: Qwen3-0.6B with MRV1 and MRV2 - MTP: Qwen3.5-27B with MRV1 and MRV2 Manually validated on A5 with the following scenarios: - FIA: Qwen3-0.6B with MRV1 and MRV2 - MTP: Qwen3.5-9B with MRV1 and MRV2 - PA: gemma4 with MRV1 and MRV2 **Does this PR introduce any user-facing change?** No. **How was this patch tested?** Temporarily validated on some model. - vLLM main: vllm-project/vllm@b2f6858 --------- Signed-off-by: zhiyu-wa <1959864813@qq.com>

yiz-liu · 2026-09-12T08:29:27Z

/revert
[Bot]: revert completed successfully. New PR created: #16409

czydyy · 2026-09-12T08:31:48Z

/revert
[Bot]: revert branch has been refreshed. Existing PR: #16409

…r ACL Graph Replay. (vllm-project#15908)" This reverts commit daa6441.

@zhiyu-wa

…pdates for ACL Graph Replay." (#15908) (#16409) Revert of PR #15908 (merged onto `main`). Original PR: #15908 Original author: @zhiyu-wa Merge commit: `daa644121e1142821777e3ba60a49ab6dd354920` --- **What this PR does / why we need it?** Implements the `UpdatableGraph` design discussed in #13058. Refactors the `FIA` and `Speculative Decoding` to work with the new `UpdatableGraph` design. Manually validated on A3 with the following scenarios: - FIA: Qwen3-0.6B with MRV1 and MRV2 - MTP: Qwen3.5-27B with MRV1 and MRV2 Manually validated on A5 with the following scenarios: - FIA: Qwen3-0.6B with MRV1 and MRV2 - MTP: Qwen3.5-9B with MRV1 and MRV2 - PA: gemma4 with MRV1 and MRV2 **Does this PR introduce any user-facing change?** No. **How was this patch tested?** Temporarily validated on some model. - vLLM main: vllm-project/vllm@a97dacb Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…hado/vllm-ascend into main_fix_mrv2_eagle3_mamba * 'main_fix_mrv2_eagle3_mamba' of https://github.com/windshado/vllm-ascend: (42 commits) Update vllm_ascend/worker/v2/model_states/mamba_hybrid.py [Feature][Kimi K3 DSPark] Enable TP for context_proj (vllm-project#16344) [BugFix][SpecDecode] Refresh replicated PCP draft graph cache mappings (vllm-project#16300) [Feature][Model] Integrate Triton KeyPool indexing for GLM-5.3-Flash (vllm-project#16253) [BugFix][Offloader] Re-bind params to NZ static buffers after npu_format_cast (vllm-project#15415) [Feature][Model] Integrate AscendC KDA and causal convolution for GLM-5.3-Flash (vllm-project#16251) [Performance][Communicator] Replace per-layer F.pad with cat of a persistent zero block in MoE prepare (vllm-project#16343) [Feature][Operator] Add DeepSeek V4.1 sparse attention operators (vllm-project#16422) [Doc][Misc] Document batch invariance scheduling limitations (vllm-project#16232) [CI][MRV2] Enable mrv2 dspark e2e test (vllm-project#16319) [BugFix] Precast MoE gate weight_fp32 to avoid aclop Cast (vllm-project#16189) [Feature][MRV2][310P] MRv2 adapting MTP on the 310P for Qwen3.5 (vllm-project#16043) [Revert] Revert "[Feature][MRV1][MRV2] Refactor Host-Side Parameter Updates for ACL Graph Replay." (vllm-project#15908) (vllm-project#16409) [Feature][Ops] Add Triton KeyPool compression and pooled indexing (vllm-project#16243) [Feature][Attention] Support NoPE in the shared SFA backend (vllm-project#16252) [Performance][Model] Reuse fused mHC operators for GLM-5.3-Flash (vllm-project#16321) [Feature][Model] Enable MiniMax-M3 FP8 MSA index score on A5 (vllm-project#15918) [Performance][KDA] Reduce preprocessing copies and redundant output masks (vllm-project#16067) [Feature][Model][MTP] Support speculative decoding for GLM-5.3-Flash (vllm-project#16214) [BugFix][Model] Skip unused hash-router bias when loading DeepSeek-V4 weights (vllm-project#16259) ...

…aph Replay. (vllm-project#15908) **What this PR does / why we need it?** Implements the `UpdatableGraph` design discussed in vllm-project#13058. Refactors the `FIA` and `Speculative Decoding` to work with the new `UpdatableGraph` design. Manually validated on A3 with the following scenarios: - FIA: Qwen3-0.6B with MRV1 and MRV2 - MTP: Qwen3.5-27B with MRV1 and MRV2 Manually validated on A5 with the following scenarios: - FIA: Qwen3-0.6B with MRV1 and MRV2 - MTP: Qwen3.5-9B with MRV1 and MRV2 - PA: gemma4 with MRV1 and MRV2 **Does this PR introduce any user-facing change?** No. **How was this patch tested?** Temporarily validated on some model. - vLLM main: vllm-project/vllm@b2f6858 --------- Signed-off-by: zhiyu-wa <1959864813@qq.com> Signed-off-by: tianming2009 <13246728590@163.com>

@zhiyu-wa

…pdates for ACL Graph Replay." (vllm-project#15908) (vllm-project#16409) Revert of PR vllm-project#15908 (merged onto `main`). Original PR: vllm-project#15908 Original author: @zhiyu-wa Merge commit: `daa644121e1142821777e3ba60a49ab6dd354920` --- **What this PR does / why we need it?** Implements the `UpdatableGraph` design discussed in vllm-project#13058. Refactors the `FIA` and `Speculative Decoding` to work with the new `UpdatableGraph` design. Manually validated on A3 with the following scenarios: - FIA: Qwen3-0.6B with MRV1 and MRV2 - MTP: Qwen3.5-27B with MRV1 and MRV2 Manually validated on A5 with the following scenarios: - FIA: Qwen3-0.6B with MRV1 and MRV2 - MTP: Qwen3.5-9B with MRV1 and MRV2 - PA: gemma4 with MRV1 and MRV2 **Does this PR introduce any user-facing change?** No. **How was this patch tested?** Temporarily validated on some model. - vLLM main: vllm-project/vllm@a97dacb Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: tianming2009 <13246728590@163.com>

…aph Replay. (vllm-project#15908) **What this PR does / why we need it?** Implements the `UpdatableGraph` design discussed in vllm-project#13058. Refactors the `FIA` and `Speculative Decoding` to work with the new `UpdatableGraph` design. Manually validated on A3 with the following scenarios: - FIA: Qwen3-0.6B with MRV1 and MRV2 - MTP: Qwen3.5-27B with MRV1 and MRV2 Manually validated on A5 with the following scenarios: - FIA: Qwen3-0.6B with MRV1 and MRV2 - MTP: Qwen3.5-9B with MRV1 and MRV2 - PA: gemma4 with MRV1 and MRV2 **Does this PR introduce any user-facing change?** No. **How was this patch tested?** Temporarily validated on some model. - vLLM main: vllm-project/vllm@b2f6858 --------- Signed-off-by: zhiyu-wa <1959864813@qq.com> Signed-off-by: like-0517 <ithwlike@126.com>

@zhiyu-wa

…pdates for ACL Graph Replay." (vllm-project#15908) (vllm-project#16409) Revert of PR vllm-project#15908 (merged onto `main`). Original PR: vllm-project#15908 Original author: @zhiyu-wa Merge commit: `daa644121e1142821777e3ba60a49ab6dd354920` --- **What this PR does / why we need it?** Implements the `UpdatableGraph` design discussed in vllm-project#13058. Refactors the `FIA` and `Speculative Decoding` to work with the new `UpdatableGraph` design. Manually validated on A3 with the following scenarios: - FIA: Qwen3-0.6B with MRV1 and MRV2 - MTP: Qwen3.5-27B with MRV1 and MRV2 Manually validated on A5 with the following scenarios: - FIA: Qwen3-0.6B with MRV1 and MRV2 - MTP: Qwen3.5-9B with MRV1 and MRV2 - PA: gemma4 with MRV1 and MRV2 **Does this PR introduce any user-facing change?** No. **How was this patch tested?** Temporarily validated on some model. - vLLM main: vllm-project/vllm@a97dacb Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: like-0517 <ithwlike@126.com>

### What this PR does / why we need it? **Depends on #15908 for Full Decode Only graph replay. Please merge after that dependency.** Two independent MRV2 Qwen3.5 PP fixes: - **Graph backend selection:** skip metadata-only attention backends whose `get_impl_cls()` raises `NotImplementedError` while collecting graph-update backends, so the PP stage can select an executable attention backend (GDN-style backends have no attention impl). - **Draft inputs:** disable external multimodal embedding collection on the speculator when PP is enabled, set right after draft loading and before profiling. The last PP stage has no encoder cache; ordinary token embedding remains in the draft model. Non-PP behavior is unchanged. Plus one four-card E2E (`test_qwen35_pp_mtp.py`) covering serial, concurrent, and chunked-prefill requests. ### Does this PR introduce _any_ user-facing change? Addresses PP + MTP backend-selection and draft-input failures for Qwen3.5 under MRV2. Full Decode Only support still relies on #15908; this PR is not a standalone replacement for that dependency. ### How was this patch tested? - **E2E (CI four-card):** `tests/e2e/pull_request/four_card/model_runner_v2/test_qwen35_pp_mtp.py` — serial answers, concurrency, and a prompt exceeding the prefill budget. - vLLM main: vllm-project/vllm@84030bb --------- Signed-off-by: LostFox11 <wangziyue17@huawei.com> Co-authored-by: LostFox11 <wangziyue17@huawei.com>

### What this PR does / why we need it? **Depends on vllm-project#15908 for Full Decode Only graph replay. Please merge after that dependency.** Two independent MRV2 Qwen3.5 PP fixes: - **Graph backend selection:** skip metadata-only attention backends whose `get_impl_cls()` raises `NotImplementedError` while collecting graph-update backends, so the PP stage can select an executable attention backend (GDN-style backends have no attention impl). - **Draft inputs:** disable external multimodal embedding collection on the speculator when PP is enabled, set right after draft loading and before profiling. The last PP stage has no encoder cache; ordinary token embedding remains in the draft model. Non-PP behavior is unchanged. Plus one four-card E2E (`test_qwen35_pp_mtp.py`) covering serial, concurrent, and chunked-prefill requests. ### Does this PR introduce _any_ user-facing change? Addresses PP + MTP backend-selection and draft-input failures for Qwen3.5 under MRV2. Full Decode Only support still relies on vllm-project#15908; this PR is not a standalone replacement for that dependency. ### How was this patch tested? - **E2E (CI four-card):** `tests/e2e/pull_request/four_card/model_runner_v2/test_qwen35_pp_mtp.py` — serial answers, concurrency, and a prompt exceeding the prefill budget. - vLLM main: vllm-project/vllm@84030bb --------- Signed-off-by: LostFox11 <wangziyue17@huawei.com> Co-authored-by: LostFox11 <wangziyue17@huawei.com>

zhiyu-wa requested review from weijinqian0 and yiz-liu as code owners September 7, 2026 04:01

github-actions Bot added the module:core label Sep 7, 2026

gemini-code-assist Bot reviewed Sep 7, 2026

View reviewed changes

zhiyu-wa force-pushed the new-refactor branch 2 times, most recently from 8e3d3ef to 9d9c169 Compare September 7, 2026 11:27

yiz-liu added the ready-precise run selected e2e test for pr label Sep 8, 2026

github-actions Bot added the module:tests label Sep 8, 2026

zhiyu-wa force-pushed the new-refactor branch 2 times, most recently from c7f3e4a to 6d59749 Compare September 8, 2026 03:32

github-actions Bot added the merge-conflicts label Sep 8, 2026

zhiyu-wa force-pushed the new-refactor branch 2 times, most recently from 8cf4361 to 50d2574 Compare September 9, 2026 01:19

github-actions Bot removed the merge-conflicts label Sep 9, 2026

yiz-liu added ready-a5 ready-all run all e2e test for pr labels Sep 9, 2026

yiz-liu reviewed Sep 9, 2026

View reviewed changes

github-actions Bot added the merge-conflicts label Sep 9, 2026

github-actions Bot removed the merge-conflicts label Sep 9, 2026

zhiyu-wa force-pushed the new-refactor branch from 8684f76 to 9c73783 Compare September 9, 2026 06:52

github-actions Bot added the merge-conflicts label Sep 9, 2026

github-actions Bot removed the merge-conflicts label Sep 9, 2026

zhiyu-wa force-pushed the new-refactor branch 2 times, most recently from 2c82adc to a897bdb Compare September 9, 2026 07:14

github-actions Bot added the merge-conflicts label Sep 11, 2026

github-actions Bot removed the merge-conflicts label Sep 11, 2026

zhiyu-wa added 4 commits September 11, 2026 15:31

refator

3929865

Signed-off-by: zhiyu-wa <1959864813@qq.com>

fix ut and e2e

d2b0a3c

Signed-off-by: zhiyu-wa <1959864813@qq.com>

optimized code

794caf2

Signed-off-by: zhiyu-wa <1959864813@qq.com>

fix pa

a82894f

Signed-off-by: zhiyu-wa <1959864813@qq.com>

zhiyu-wa force-pushed the new-refactor branch from 95d2be8 to a82894f Compare September 11, 2026 07:33

fix ut

47dbd65

Signed-off-by: zhiyu-wa <1959864813@qq.com>

yiz-liu enabled auto-merge (squash) September 11, 2026 09:51

yiz-liu approved these changes Sep 11, 2026

View reviewed changes

yiz-liu merged commit daa6441 into vllm-project:main Sep 11, 2026
31 checks passed

czydyy added a commit to czydyy/vllm-ascend that referenced this pull request Sep 12, 2026

Revert "[Feature][MRV1][MRV2] Refactor Host-Side Parameter Updates fo…

68e3956

…r ACL Graph Replay. (vllm-project#15908)" This reverts commit daa6441. Signed-off-by: chenzeyu <2978509328@qq.com>

vllm-ascend-ci mentioned this pull request Sep 12, 2026

[Revert] Revert "[Feature][MRV1][MRV2] Refactor Host-Side Parameter Updates for ACL Graph Replay." (#15908) #16409

Merged

vllm-ascend-ci pushed a commit to vllm-ascend-ci/vllm-ascend that referenced this pull request Sep 12, 2026

Revert "[Feature][MRV1][MRV2] Refactor Host-Side Parameter Updates fo…

84a13b0

…r ACL Graph Replay. (vllm-project#15908)" This reverts commit daa6441.

swimming2007-doge mentioned this pull request Sep 17, 2026

[BugFix][Spec Decode] Resolve draft ACL graph params per layer for Gemma4 MTP Problem #16737

Closed

Conversation

zhiyu-wa commented Sep 7, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot commented Sep 7, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

github-actions Bot commented Sep 7, 2026

💡 Consider Linking a Related Issue or RFC

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Sep 8, 2026

Uh oh!

github-actions Bot commented Sep 8, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

github-actions Bot commented Sep 9, 2026

Uh oh!

yiz-liu commented Sep 11, 2026 • edited by vllm-ascend-ci Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yiz-liu commented Sep 11, 2026 • edited by vllm-ascend-ci Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yiz-liu commented Sep 11, 2026 • edited by vllm-ascend-ci Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 11, 2026

Uh oh!

Uh oh!

yiz-liu commented Sep 12, 2026 • edited by vllm-ascend-ci Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

czydyy commented Sep 12, 2026 • edited by vllm-ascend-ci Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zhiyu-wa commented Sep 7, 2026 •

edited by github-actions Bot

Loading

yiz-liu commented Sep 11, 2026 •

edited by vllm-ascend-ci

Loading

yiz-liu commented Sep 11, 2026 •

edited by vllm-ascend-ci

Loading

yiz-liu commented Sep 11, 2026 •

edited by vllm-ascend-ci

Loading

yiz-liu commented Sep 12, 2026 •

edited by vllm-ascend-ci

Loading

czydyy commented Sep 12, 2026 •

edited by vllm-ascend-ci

Loading