Skip to content

[Performance] Cache loaded custom logitsprocs to avoid overheads #28462

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
njhill merged 2 commits into from
Nov 12, 2025
Merged

[Performance] Cache loaded custom logitsprocs to avoid overheads #28462

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
128723b
cache custom logitsprocs loading
Isotr0py Nov 11, 2025
4b63eef
gemini
Isotr0py Nov 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions vllm/v1/sample/logits_processor/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
import itertools
from abc import abstractmethod
from collections.abc import Sequence
from functools import partial
from functools import lru_cache, partial
from typing import TYPE_CHECKING

import torch
Expand Down Expand Up @@ -216,11 +216,17 @@ def build_logitsprocs(
)


cached_load_custom_logitsprocs = lru_cache(_load_custom_logitsprocs)
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Nov 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

While using lru_cache on _load_custom_logitsprocs is a good performance improvement, it can be made more efficient. _load_custom_logitsprocs internally calls _load_logitsprocs_plugins(), which does not depend on any arguments. With the current implementation, _load_logitsprocs_plugins() will be re-executed for every cache miss of cached_load_custom_logitsprocs (i.e., for each new logits_processors value).

To avoid this repeated work, _load_logitsprocs_plugins() should be cached independently. The ideal solution would be to apply @lru_cache directly to _load_logitsprocs_plugins and _load_logitsprocs_by_fqcns. This would require modifying those functions, which are outside the current diff.

For example:

@lru_cache(maxsize=None)
def _load_logitsprocs_plugins() -> list[type[LogitsProcessor]]:
    # ... function body

@lru_cache(maxsize=None)
def _load_logitsprocs_by_fqcns(
    logits_processors: tuple[str | type[LogitsProcessor], ...] | None,
) -> list[type[LogitsProcessor]]:
    # ... function body

Then _load_custom_logitsprocs can call these cached functions, and validate_logits_processors_parameters can call _load_custom_logitsprocs directly without an extra caching layer. This would be the most efficient implementation.



def validate_logits_processors_parameters(
logits_processors: Sequence[str | type[LogitsProcessor]] | None,
sampling_params: SamplingParams,
):
for logits_procs in _load_custom_logitsprocs(logits_processors):
logits_processors = (
tuple(logits_processors) if logits_processors is not None else None
)
for logits_procs in cached_load_custom_logitsprocs(logits_processors):
logits_procs.validate_params(sampling_params)


Expand Down