[Feature] Option to save model weights to CPU when memory saver mode is enabled #10873
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mattnappo
requested review from
Ying1123,
hnyls2002,
ispobock,
merrymercy and
zhyncs
as code owners
mattnappo
requested review from
ByronHsu,
CatherineSue,
HaiShaw,
slin1237 and
xiezhq-hermann
as code owners
Bump torch_memory_saver version Only enable CPU backup for model weights Add flag Update test_release_memory_occupation.py
Collaborator
|
|
JustinTong0323
approved these changes
Sep 30, 2025
Collaborator
|
I think this might be causing some CI failures when trying to install for ARM. https://github.com/sgl-project/sglang/actions/runs/18236550332/job/51931286314 and https://github.com/sgl-project/sglang/actions/runs/18251221010/job/51965850200 |
0xtoward
pushed a commit
to 0xtoward/sglang
that referenced
this pull request
Oct 5, 2025
…is enabled (sgl-project#10873) Co-authored-by: molocule <[email protected]>
ch-tiger1
pushed a commit
to ch-tiger1/sglang
that referenced
this pull request
Oct 9, 2025
…is enabled (sgl-project#10873) Co-authored-by: molocule <[email protected]>
lpc0220
pushed a commit
to lpc0220/sglang
that referenced
this pull request
Oct 29, 2025
…is enabled (sgl-project#10873) Co-authored-by: molocule <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
This PR addresses confusion in
enable_memory_savermode, which is raised in several issues:release_memory_occupationandresume_memory_occupationsupposed to work #6367 (comment)Even though the initial use case for
torch_memory_saveris for RL, offloading weights from GPU to CPU memory is useful for other use cases, such as memory snapshot/restore. This PR adds a flag to setenable_weights_cpu_backupto enable offloading model weights from GPU to CPU memory so that the weights can be restored.Modifications
enable_weights_cpu_backuptoServerArgsto enable offloading model weights from GPU to CPU memory.enable_weights_cpu_backupduring model loading.torch-memory-saverversion from0.0.8to0.0.9rc1Benchmark and Profiling
enable_weights_cpu_backupinenable_memory_savermode.enable_weights_cpu_backupinenable_memory_savermode.enable_weights_cpu_backupinenable_memory_savermode.