Skip to content

[Bugfix] Fix --enable-prompt-tokens-details omitting zero cached tokens #44383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vllm-bot merged 9 commits into from
Jun 12, 2026
Merged

[Bugfix] Fix --enable-prompt-tokens-details omitting zero cached tokens #44383

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
bdace41
[Bugfix] Fix --enable-prompt-tokens-details omitting zero cached tokens
Jun 3, 2026
047916b
[Bugfix] Fix pre-commit ruff formatting
Jun 3, 2026
ed40541
[Bugfix] Add regression test for zero cached tokens in prompt_tokens_…
Jun 3, 2026
b8fea3d
[Bugfix] Fix zero cached tokens omitted in disagg serving path and ad…
Jun 3, 2026
fc41840
Merge branch 'main' into fix/prompt-tokens-details-zero-cached
sasindharan Jun 3, 2026
bc2750f
[Bugfix] Fix ruff formatting in disagg serving path
Jun 3, 2026
b1cd899
[Bugfix] Update test_single_completion to expect cached_tokens=0 with…
Jun 4, 2026
8b632ad
[Bugfix] Fix test assertion to not use openai.types.PromptTokensDetails
Jun 4, 2026
b555f9e
Merge branch 'main' into fix/prompt-tokens-details-zero-cached
chaunceyjiang Jun 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions vllm/entrypoints/openai/chat_completion/serving.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -879,7 +879,7 @@
completion_tokens=completion_tokens,
total_tokens=num_prompt_tokens + completion_tokens,
)
if self.enable_prompt_tokens_details and num_cached_tokens:
if self.enable_prompt_tokens_details and num_cached_tokens is not None:
final_usage.prompt_tokens_details = PromptTokenUsageInfo(
cached_tokens=num_cached_tokens
)
Expand Down Expand Up @@ -1303,7 +1303,7 @@
completion_tokens=num_generated_tokens,
total_tokens=num_prompt_tokens + num_generated_tokens,
)
if self.enable_prompt_tokens_details and final_res.num_cached_tokens:
if self.enable_prompt_tokens_details and final_res.num_cached_tokens is not None:

Check failure on line 1306 in vllm/entrypoints/openai/chat_completion/serving.py

View workflow job for this annotation

GitHub Actions / pre-commit

Ruff (E501) 

vllm/entrypoints/openai/chat_completion/serving.py:1306:89: E501 Line too long (89 > 88)
usage.prompt_tokens_details = PromptTokenUsageInfo(
cached_tokens=final_res.num_cached_tokens
)
Expand Down
4 changes: 2 additions & 2 deletions vllm/entrypoints/openai/completion/serving.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -443,7 +443,7 @@ async def completion_stream_generator(
total_tokens=total_prompt_tokens + total_completion_tokens,
)

if self.enable_prompt_tokens_details and num_cached_tokens:
if self.enable_prompt_tokens_details and num_cached_tokens is not None:
final_usage_info.prompt_tokens_details = PromptTokenUsageInfo(
cached_tokens=num_cached_tokens
)
Expand Down Expand Up @@ -583,7 +583,7 @@ def request_output_to_completion_response(
if (
self.enable_prompt_tokens_details
and last_final_res
and last_final_res.num_cached_tokens
and last_final_res.num_cached_tokens is not None
):
usage.prompt_tokens_details = PromptTokenUsageInfo(
cached_tokens=last_final_res.num_cached_tokens
Expand Down
Loading