[Perf] Optimize fused_experts quantization code to save npu memory #784
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ApsarasX
force-pushed
the
optimize-w8a8-memory
branch
from
4f4df17 to
70b8c9f
Compare
Signed-off-by: ApsarasX <[email protected]>
ApsarasX
force-pushed
the
optimize-w8a8-memory
branch
from
70b8c9f to
b95ca84
Compare
Collaborator
Author
|
@linfeng-yuan @ganyi1996ppo @zzzzwwjj @Yikun Please review this PR. |
wangxiyuan
approved these changes
May 9, 2025
Collaborator
wangxiyuan
left a comment
wangxiyuan
left a comment
There was a problem hiding this comment.
Please attach the improvement result later to make sure this PR works as expect. Thanks.
| dynamic_scale = None | ||
|
|
||
| down_out_list = apply_mlp(expand_x, | ||
| # place hidden_states in a list to transfer its ownership into the `apply_mlp` function |
Collaborator
There was a problem hiding this comment.
I'm fine with this quick fix. you have describe the case more clear later. For example, this copy to list action aims to deal with the lifecycle for the tensor so that the memory usage can be improved.
| down_out_list = apply_mlp(expand_x, | ||
| # place hidden_states in a list to transfer its ownership into the `apply_mlp` function | ||
| hidden_states_wrapper = [expand_x] | ||
| del expand_x |
Collaborator
There was a problem hiding this comment.
There still a lot of del action everywhere. We can create a new func like release_tenstor to make the del aciton more clear for developer
venus-taibai
pushed a commit
to venus-taibai/vllm-ascend
that referenced
this pull request
May 15, 2025
…e npu memory (vllm-project#784) Merge branch dev-v0.8.5.508-memory-optimization of [email protected]:Theta/vllm-ascend.git into dev-v0.8.5.508 https://code.alipay.com/Theta/vllm-ascend/pull_requests/9 Signed-off-by: 康安 <[email protected]> * [Perf] Optimize fused_experts quantization code to save npu memory (vllm-project#784)
ganyi1996ppo
pushed a commit
that referenced
this pull request
May 29, 2025
### What this PR does / why we need it? 1. In previous PRs #580 #784, I saved GPU memory by promptly deleting unnecessary tensors. For tensors passed from upper-layer functions, I used a list container to transfer the parameter and then popped the tensor from the list within the inner function to achieve deletion. Recently, I discovered a better implementation in sglang—the `dispose_tensor` function and I recommend adopting this approach. 2. Dispose `hidden_states` and `residual` from the previous layer once they're no longer used. 3. Avoid to generate `self.inputs_embeds` in `ModelRunnerV1` in non-multimodal scenarios. With the aforementioned optimizations, using the DeepSeek-R1-W8A8 model under the conditions of `TP=16` and `max-model-len=32768`, we can save 1.3GB of npu memory. **Reference**: sgl-project/sglang#6147 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? --------- Signed-off-by: ApsarasX <[email protected]>
zxdukki
pushed a commit
to zxdukki/vllm-ascend
that referenced
this pull request
Jun 3, 2025
…oject#966) ### What this PR does / why we need it? 1. In previous PRs vllm-project#580 vllm-project#784, I saved GPU memory by promptly deleting unnecessary tensors. For tensors passed from upper-layer functions, I used a list container to transfer the parameter and then popped the tensor from the list within the inner function to achieve deletion. Recently, I discovered a better implementation in sglang—the `dispose_tensor` function and I recommend adopting this approach. 2. Dispose `hidden_states` and `residual` from the previous layer once they're no longer used. 3. Avoid to generate `self.inputs_embeds` in `ModelRunnerV1` in non-multimodal scenarios. With the aforementioned optimizations, using the DeepSeek-R1-W8A8 model under the conditions of `TP=16` and `max-model-len=32768`, we can save 1.3GB of npu memory. **Reference**: sgl-project/sglang#6147 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? --------- Signed-off-by: ApsarasX <[email protected]>
David9857
pushed a commit
to David9857/vllm-ascend
that referenced
this pull request
Jun 3, 2025
…oject#966) ### What this PR does / why we need it? 1. In previous PRs vllm-project#580 vllm-project#784, I saved GPU memory by promptly deleting unnecessary tensors. For tensors passed from upper-layer functions, I used a list container to transfer the parameter and then popped the tensor from the list within the inner function to achieve deletion. Recently, I discovered a better implementation in sglang—the `dispose_tensor` function and I recommend adopting this approach. 2. Dispose `hidden_states` and `residual` from the previous layer once they're no longer used. 3. Avoid to generate `self.inputs_embeds` in `ModelRunnerV1` in non-multimodal scenarios. With the aforementioned optimizations, using the DeepSeek-R1-W8A8 model under the conditions of `TP=16` and `max-model-len=32768`, we can save 1.3GB of npu memory. **Reference**: sgl-project/sglang#6147 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? --------- Signed-off-by: ApsarasX <[email protected]>
chopper0126
pushed a commit
to chopper0126/vllm-ascend
that referenced
this pull request
Oct 16, 2025
…llm-project#784) ### What this PR does / why we need it? In the w8a8 quantization code of `fused_experts`, the output of almost every operator is assigned a new variable name. If we want to save NPU memory, we manually `del` these variables to end their lifecycle, which fills the code with `del` statements and looks inelegant. Therefore, I plan to names the output of most operators as `hidden_states`, thereby ending the lifecycle of the previous `hidden_states`. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Signed-off-by: ApsarasX <[email protected]>
chopper0126
pushed a commit
to chopper0126/vllm-ascend
that referenced
this pull request
Oct 16, 2025
…oject#966) ### What this PR does / why we need it? 1. In previous PRs vllm-project#580 vllm-project#784, I saved GPU memory by promptly deleting unnecessary tensors. For tensors passed from upper-layer functions, I used a list container to transfer the parameter and then popped the tensor from the list within the inner function to achieve deletion. Recently, I discovered a better implementation in sglang—the `dispose_tensor` function and I recommend adopting this approach. 2. Dispose `hidden_states` and `residual` from the previous layer once they're no longer used. 3. Avoid to generate `self.inputs_embeds` in `ModelRunnerV1` in non-multimodal scenarios. With the aforementioned optimizations, using the DeepSeek-R1-W8A8 model under the conditions of `TP=16` and `max-model-len=32768`, we can save 1.3GB of npu memory. **Reference**: sgl-project/sglang#6147 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? --------- Signed-off-by: ApsarasX <[email protected]>
Angazenn
pushed a commit
to Angazenn/vllm-ascend
that referenced
this pull request
Oct 21, 2025
…llm-project#784) ### What this PR does / why we need it? In the w8a8 quantization code of `fused_experts`, the output of almost every operator is assigned a new variable name. If we want to save NPU memory, we manually `del` these variables to end their lifecycle, which fills the code with `del` statements and looks inelegant. Therefore, I plan to names the output of most operators as `hidden_states`, thereby ending the lifecycle of the previous `hidden_states`. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Signed-off-by: ApsarasX <[email protected]>
Angazenn
pushed a commit
to Angazenn/vllm-ascend
that referenced
this pull request
Oct 21, 2025
…oject#966) ### What this PR does / why we need it? 1. In previous PRs vllm-project#580 vllm-project#784, I saved GPU memory by promptly deleting unnecessary tensors. For tensors passed from upper-layer functions, I used a list container to transfer the parameter and then popped the tensor from the list within the inner function to achieve deletion. Recently, I discovered a better implementation in sglang—the `dispose_tensor` function and I recommend adopting this approach. 2. Dispose `hidden_states` and `residual` from the previous layer once they're no longer used. 3. Avoid to generate `self.inputs_embeds` in `ModelRunnerV1` in non-multimodal scenarios. With the aforementioned optimizations, using the DeepSeek-R1-W8A8 model under the conditions of `TP=16` and `max-model-len=32768`, we can save 1.3GB of npu memory. **Reference**: sgl-project/sglang#6147 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? --------- Signed-off-by: ApsarasX <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it?
In the w8a8 quantization code of
fused_experts, the output of almost every operator is assigned a new variable name. If we want to save NPU memory, we manuallydelthese variables to end their lifecycle, which fills the code withdelstatements and looks inelegant.Therefore, I plan to names the output of most operators as
hidden_states, thereby ending the lifecycle of the previoushidden_states.Does this PR introduce any user-facing change?
No
How was this patch tested?