Skip to content

[Bugfix] fix logit processor excceed vocab size issue #6927

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
DarkLight1337 merged 4 commits into from
Jul 31, 2024
Merged

[Bugfix] fix logit processor excceed vocab size issue #6927

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d94c22d
fix logit processor excceed vocab size issue
FeiDeng Jul 30, 2024
26871dd
refactor according to comments
FeiDeng Jul 30, 2024
bfdfefb
reformat
FeiDeng Jul 31, 2024
caeadcd
fix failed gate
FeiDeng Jul 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions vllm/entrypoints/openai/logits_processors.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,12 @@ def get_logits_processors(
"Found token_id in logit_bias that is not "
"an integer or string representing an integer") from exc

# Check if token_id is within the vocab size
for token_id, bias in clamped_logit_bias.items():
if token_id < 0 or token_id >= tokenizer.vocab_size:
raise ValueError("token_id in logit_bias contains "
"out-of-vocab token id")

def logit_bias_logits_processor(token_ids: List[int],
logits: torch.Tensor) -> torch.Tensor:
for token_id, bias in clamped_logit_bias.items():
Expand Down