[Front-end] [Messages] Populate num_cache_creation_tokens - #48535
Conversation
de97bac to
12f8c8b
Compare
| self.kv_cache_manager.coordinator.find_longest_cache_hit( | ||
| request.block_hashes, | ||
| request.num_prompt_tokens, | ||
| ) |
There was a problem hiding this comment.
This is going to be too expensive to do for every request since it hashes every block
One potential alternative is to add a high-watermark number of blocks cached by the request to PrefillStats and then update it inside cache_blocks() in the KVCache coordinator implementations.
Otherwise if that doesn't work out, we can guard this by only calling find_longest_cache_hit when --enable-prompt-tokens-details is set but that will require plumbing
There was a problem hiding this comment.
Updating it inside cache_blocks() was quite messy. Found a reasonable way to estimate it given block.block_hash_num_tokens.
3dcf93c to
d4adbad
Compare
There was a problem hiding this comment.
In the current code, kv_transfer_params and ec_transfer_params will never be truthy without stopped being true.
| if should_emit_output: | ||
| prefill_stats = request.take_prefill_stats() | ||
| if prefill_stats is not None: | ||
| prefill_stats.finalize( |
There was a problem hiding this comment.
We take prefill_stats here to ensure we can traverse request's KVCacheBlocks before they are freed
Draft Signed-off-by: Yifan Zong <yzong@redhat.com> Fix rebase Signed-off-by: Yifan Zong <yzong@redhat.com>
Signed-off-by: Yifan Zong <yzong@redhat.com>
Signed-off-by: Yifan Zong <yzong@redhat.com>
d4adbad to
0f63913
Compare
Signed-off-by: Yifan Zong <yzong@redhat.com>
|
Two failures are Caused by not mocking |
Head branch was pushed to by a user without write access
…ject#48535) Signed-off-by: Yifan Zong <yzong@redhat.com>
Purpose
Follow up to #40912.
When
--enable-prompt-tokens-detailsis toggled, compute the number of prompt tokens written to prefix cache and pass it through to OpenAI Chat Completions and eventually to Anthropic Messages API. Ensuresnum_cache_creation_tokensis populated.Test Plan
Send and examine real requests to Qwen3.6-35B-A3B served via Messages API, both streaming and non-streaming
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.