Skip to content

feat: separately wakeup vllm to reduce peak memory in refitting#190

Closed
zpqiu wants to merge 1 commit intomainfrom
alexq/vllm-separately-wakeup
Closed

feat: separately wakeup vllm to reduce peak memory in refitting#190
zpqiu wants to merge 1 commit intomainfrom
alexq/vllm-separately-wakeup

Conversation

@zpqiu
Copy link
Copy Markdown
Contributor

@zpqiu zpqiu commented Apr 15, 2025

During experiments, I encountered OOM (Out of Memory) issues when waking up VLLM. Additionally, I noticed that in the latest VLLM version 0.8.3, they updated the wakeup API by adding a 'tags' parameter.

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

As shown in the figure below, we can first load only the weights, then update the parameters, and finally load the KV cache to reduce peak memory usage during the refit_policy_generation phase. This logic has also been implemented in veRL.

wakeup

Issues

List issues that this PR closes (syntax):

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this 

Before your PR is "Ready for review"

Pre checks:

  • [x ] Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

  • ...

@zpqiu zpqiu linked an issue Apr 15, 2025 that may be closed by this pull request
@zpqiu zpqiu changed the title feat: separately wakeup vllm to reduce refitting peak memory feat: separately wakeup vllm to reduce peak memory in refitting Apr 15, 2025
@parthchadha
Copy link
Copy Markdown
Contributor

@zpqiu thanks for the suggestion! We have included these changes in #176.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Separately wakeup vllm to reduce peak memory in refitting

2 participants