Skip to content

Fix redundancy_buffer_memory not taken account in determine_available_memory() #37420

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
panpan0000 wants to merge 6 commits into
base: main
Choose a base branch
from
+8 −5
Open

Fix redundancy_buffer_memory not taken account in determine_available_memory() #37420

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
5be3c38
Fix redundancy_buffer_memory not taken account in available_kv_cache_…
panpan0000 Mar 18, 2026
b752351
Merge branch 'main' into fix_redundancy_buffer_memory
panpan0000 Mar 18, 2026
317ff2d
Merge branch 'main' into fix_redundancy_buffer_memory
panpan0000 Mar 19, 2026
ad19632
Merge branch 'main' into fix_redundancy_buffer_memory
panpan0000 Apr 9, 2026
fa63571
Merge branch 'main' into fix_redundancy_buffer_memory
panpan0000 Apr 10, 2026
1921f78
Merge branch 'main' into fix_redundancy_buffer_memory
panpan0000 Apr 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions vllm/v1/worker/gpu_worker.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,11 @@

logger = init_logger(__name__)

# empirically observed that the memory profiling may
# slightly underestimate the memory consumption.
# So leave a small buffer (=150MiB) to avoid OOM.
REDUNDANCY_BUFFER_MEMORY_BYTES = 150 * (1 << 20)
Comment thread
panpan0000 marked this conversation as resolved.

if TYPE_CHECKING:
from vllm.model_executor.model_loader.tensorizer import TensorizerConfig
from vllm.v1.worker.gpu_model_runner import GPUModelRunner
Expand Down Expand Up @@ -418,6 +423,7 @@ def determine_available_memory(self) -> int:
self.requested_memory
- profile_result.non_kv_cache_memory
- cudagraph_memory_estimate_applied
- REDUNDANCY_BUFFER_MEMORY_BYTES
)

unrequested_memory = self.init_snapshot.free_memory - self.requested_memory
Expand Down Expand Up @@ -614,11 +620,8 @@ def compile_or_warm_up_model(self) -> float:
# Users may want fine-grained control to specify kv cache
# memory size.

# empirically observed that the memory profiling may
# slightly underestimate the memory consumption.
# So leave a small buffer (=150MiB) to avoid OOM.
redundancy_buffer_memory = 150 * (1 << 20)

redundancy_buffer_memory = REDUNDANCY_BUFFER_MEMORY_BYTES

non_kv_cache_memory = (
self.model_runner.model_memory_usage
+ self.peak_activation_memory
Expand Down
Loading