Skip to content

[Bugfix] default set cuda_graph_sizes to max_num_seqs for v1 engine #20062

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aarnphm merged 5 commits into from
Jun 25, 2025
Merged

[Bugfix] default set cuda_graph_sizes to max_num_seqs for v1 engine #20062

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
b09f596
adjust cuda_graph_sizes shrink to max_num_seqs for v1 engine
izhuhaoran Jun 25, 2025
b62185a
add comment for cuda_graph_sizes set
izhuhaoran Jun 25, 2025
3e0089c
fix lint error
izhuhaoran Jun 25, 2025
774f789
fix ci lint error
izhuhaoran Jun 25, 2025
8118ceb
chore: use default constructor as list
aarnphm Jun 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions vllm/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2033,11 +2033,12 @@ class SchedulerConfig:
NOTE: This will be replaced by speculative config in the future; it is
present to enable correctness tests until then."""

cuda_graph_sizes: list[int] = field(default_factory=lambda: [512])
"""Cuda graph capture sizes, default is 512.
1. if one value is provided, then the capture list would follow the
cuda_graph_sizes: list[int] = field(default_factory=list)
"""Cuda graph capture sizes
1. if none provided, then default set to [max_num_seqs]
2. if one value is provided, then the capture list would follow the
pattern: [1, 2, 4] + [i for i in range(8, cuda_graph_sizes + 1, 8)]
2. more than one value (e.g. 1 2 128) is provided, then the capture list
3. more than one value (e.g. 1 2 128) is provided, then the capture list
will follow the provided list."""

delay_factor: float = 0.0
Expand Down Expand Up @@ -2202,6 +2203,10 @@ def __post_init__(self) -> None:
self.max_num_partial_prefills, self.max_long_partial_prefills,
self.long_prefill_token_threshold)

# If cuda_graph_sizes is not specified, default set to [max_num_seqs].
if not self.cuda_graph_sizes:
self.cuda_graph_sizes = [self.max_num_seqs]

@model_validator(mode='after')
def _verify_args(self) -> Self:
if (self.max_num_batched_tokens < self.max_model_len
Expand Down