Conversation
cache_salt (explicit prefix-cache isolation) was available on the OpenAI-compatible Completions / Chat Completions routes but not on the Anthropic Messages API, forcing Anthropic-schema clients to switch formats to get deterministic cache isolation in multi-tenant setups. Add cache_salt to AnthropicMessagesRequest (alongside the existing vLLM-specific kv_transfer_params / chat_template_kwargs fields) and pass it through _build_base_request into the ChatCompletionRequest, which already plumbs cache_salt into the prefix-cache hash. A field validator rejects an empty cache_salt at request-validation time (HTTP 422) instead of letting it surface as a 500 from the Messages router, mirroring the OpenAI route's non-empty rule. Fixes vllm-project#46688 Signed-off-by: abinggo <107740309+abinggo@users.noreply.github.com>
|
Thank you! |
Contributor
|
This pull request has merge conflicts that must be resolved before it can be |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #46688.
cache_salt(explicit prefix-cache isolation) is available on theOpenAI-compatible Completions / Chat Completions routes but not on the Anthropic
Messages API. Anthropic-schema clients currently have to switch to the OpenAI
format just to get deterministic cache isolation in multi-tenant setups.
Change
cache_salt: str | NonetoAnthropicMessagesRequest, alongside theexisting vLLM-specific
kv_transfer_params/chat_template_kwargsfields._build_base_requestinto theChatCompletionRequest, whichalready plumbs
cache_saltinto the prefix-cache hash — so behavior matchesthe OpenAI route exactly (no engine-side changes needed).
cache_saltat request-validation time(HTTP 422) instead of letting it fall through to a 500 from the Messages
router, mirroring the OpenAI route's non-empty requirement.
count_tokensis intentionally left untouched (it only renders/counts).Tests
tests/entrypoints/anthropic/test_anthropic_messages_conversion.py(
TestCacheSaltPassthrough): field accepted / defaults toNone; passthroughinto the built
ChatCompletionRequest;NonestaysNone; empty string isrejected with a
ValidationError. All pass locally.