Skip to content

[Front-end] [Messages] Populate num_cache_creation_tokens - #48535

Merged
tlrmchlsmth merged 4 commits into
vllm-project:mainfrom
yzong-rh:yzong-rh/messages-cache-stats
Jul 18, 2026
Merged

tlrmchlsmth merged 4 commits into
vllm-project:mainfrom
yzong-rh:yzong-rh/messages-cache-stats

Conversation

@yzong-rh

@yzong-rh yzong-rh commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Purpose

Follow up to #40912.

When --enable-prompt-tokens-details is toggled, compute the number of prompt tokens written to prefix cache and pass it through to OpenAI Chat Completions and eventually to Anthropic Messages API. Ensures num_cache_creation_tokens is populated.

Test Plan

.venv/bin/python -m pytest \
  tests/entrypoints/anthropic/test_anthropic_messages_conversion.py \
  tests/entrypoints/anthropic/test_messages.py::test_anthropic_streaming_cache_usage \
  tests/entrypoints/openai/chat_completion/test_serving_chat.py::test_mm_prompt_tokens_details

Send and examine real requests to Qwen3.6-35B-A3B served via Messages API, both streaming and non-streaming

Test Result

49 passed, 16 warnings in 35.46s
First request:
{
  "cache_creation_input_tokens": 4224,
  "cache_read_input_tokens": 0,
  "input_tokens": 527,
  "output_tokens": 197
}

Same request sent again:
{
  "cache_creation_input_tokens": 0,
  "cache_read_input_tokens": 4224,
  "input_tokens": 527,
  "output_tokens": 310
}

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@yzong-rh
yzong-rh force-pushed the yzong-rh/messages-cache-stats branch from de97bac to 12f8c8b Compare July 13, 2026 22:54
@yzong-rh

yzong-rh commented Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

Comment thread vllm/v1/core/sched/scheduler.py Outdated
self.kv_cache_manager.coordinator.find_longest_cache_hit(
request.block_hashes,
request.num_prompt_tokens,
)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is going to be too expensive to do for every request since it hashes every block

One potential alternative is to add a high-watermark number of blocks cached by the request to PrefillStats and then update it inside cache_blocks() in the KVCache coordinator implementations.

Otherwise if that doesn't work out, we can guard this by only calling find_longest_cache_hit when --enable-prompt-tokens-details is set but that will require plumbing

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updating it inside cache_blocks() was quite messy. Found a reasonable way to estimate it given block.block_hash_num_tokens.

@yzong-rh
yzong-rh force-pushed the yzong-rh/messages-cache-stats branch from 3dcf93c to d4adbad Compare July 17, 2026 03:03
Comment on lines 1771 to 1773

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the current code, kv_transfer_params and ec_transfer_params will never be truthy without stopped being true.

Comment on lines +1745 to +1748
if should_emit_output:
prefill_stats = request.take_prefill_stats()
if prefill_stats is not None:
prefill_stats.finalize(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We take prefill_stats here to ensure we can traverse request's KVCacheBlocks before they are freed

yzong-rh added 2 commits July 17, 2026 17:17
Draft

Signed-off-by: Yifan Zong <yzong@redhat.com>

Fix rebase

Signed-off-by: Yifan Zong <yzong@redhat.com>
Signed-off-by: Yifan Zong <yzong@redhat.com>
Signed-off-by: Yifan Zong <yzong@redhat.com>
@yzong-rh
yzong-rh force-pushed the yzong-rh/messages-cache-stats branch from d4adbad to 0f63913 Compare July 17, 2026 21:52
@tlrmchlsmth tlrmchlsmth added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 17, 2026
@tlrmchlsmth
tlrmchlsmth enabled auto-merge (squash) July 17, 2026 22:05
Signed-off-by: Yifan Zong <yzong@redhat.com>
@yzong-rh

Copy link
Copy Markdown
Contributor Author

Two failures are v1/core/test_scheduler.py::test_abort_request_when_structured_output_fsm_cannot_advance and v1/core/test_async_scheduler.py::test_abort_request_when_structured_output_fsm_cannot_advance.

Caused by not mocking scheduler.kv_cache_manager.estimate_cached_tokens.return_value.

auto-merge was automatically disabled July 18, 2026 00:54

Head branch was pushed to by a user without write access

@tlrmchlsmth
tlrmchlsmth merged commit a287eb1 into vllm-project:main Jul 18, 2026
101 checks passed
ArjunPakhan pushed a commit to ArjunPakhan/vllm that referenced this pull request Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

frontend ready ONLY add when PR is ready to merge/full CI is needed rust v1

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants