fix: set prompt cache key from anthropic integration - #4086
Merged
akshaydeo merged 1 commit intoJun 5, 2026
Conversation
|
tejas ghatte seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
Contributor
📝 WalkthroughWalkthroughThis PR derives Anthropic PromptCacheKey from ChangesPrompt Cache Key from User ID Metadata
Sequence Diagram(s)sequenceDiagram
participant TestClient as Test (integration)
participant Bifrost as Bifrost
participant OpenAI as OpenAI Provider
participant PromptCache as PromptCache
TestClient->>Bifrost: messages.create (Anthropic format, metadata.user_id)
Bifrost->>OpenAI: provider request (includes params.PromptCacheKey derived from metadata.user_id)
OpenAI->>PromptCache: cache write (store prompt keyed by PromptCacheKey)
Note right of PromptCache: async write may complete after response
TestClient->>Bifrost: second messages.create (same metadata.user_id)
Bifrost->>OpenAI: provider request (same PromptCacheKey)
OpenAI->>PromptCache: cache read (usage.cache_read_input_tokens > 0 on hit)
OpenAI->>Bifrost: response (includes usage.cache_read_input_tokens)
Bifrost->>TestClient: response returned
Estimated code review effort: Suggested reviewers:
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 golangci-lint (2.12.2)level=error msg="[linters_context] typechecking error: pattern ./...: directory prefix . does not contain main module or its selected dependencies" Comment |
TejasGhatte
marked this pull request as ready for review
June 5, 2026 12:30
Contributor
Confidence Score: 5/5Safe to merge — the change is confined to a single conversion helper, the core logic is mathematically correct, and non-OpenAI providers are already protected by the compat drop-params plugin. The key-derivation logic is correct: SHA-256 of any input always produces exactly 32 bytes → 64 hex chars, so the hashing branch can never produce a key that exceeds OpenAI's limit. The only gap is the absence of a unit test for the >64-char path, which is a coverage concern rather than a present defect. No files require special attention; the single changed Go function is small and self-contained. Important Files Changed
Reviews (3): Last reviewed commit: "fix: set prompt cache key from anthropic..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/integrations/python/tests/test_anthropic.py`:
- Line 145: Remove the leftover debug print by deleting the call to
print(base_url) in the test (the standalone print statement referencing the
variable base_url); ensure no other debug prints remain in the same test
function and run tests to confirm nothing else depends on console output.
- Around line 2905-2976: The test
test_53_openai_prompt_cache_key_from_metadata_user_id is brittle: add an early
provider availability check (e.g., call a helper like
get_provider_anthropic_client("openai") guarded by a skip or
is_provider_configured/is_provider_available and pytest.skip if absent) so the
test is skipped when OpenAI isn't configured; replace the invalid model string
assigned to model ("openai/gpt-5.5") with a valid model (e.g., use
get_model("openai", "chat") or "gpt-4-turbo") so requests succeed; remove the
unnecessary f-string prefixes on the two plain print messages (the prints around
the test headers) so they are normal strings; and make the hardcoded
time.sleep(10) configurable (use a TEST_PROMPT_CACHE_WAIT env var, a value from
test_config, or at minimum add a comment explaining the async cache write and
why the wait is needed) so the wait can be tuned in CI.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 5dfda566-a099-4da1-aa80-6b8c7f53dbdd
📒 Files selected for processing (2)
core/providers/anthropic/responses.gotests/integrations/python/tests/test_anthropic.py
TejasGhatte
force-pushed
the
06-05-fix_set_prompt_cache_key_from_anthropic_integration
branch
from
June 5, 2026 12:47
7f5f1c3 to
b3372fa
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (4)
tests/integrations/python/tests/test_anthropic.py (4)
2904-2919:⚠️ Potential issue | 🔴 Critical | ⚡ Quick winFix critical test issues: invalid model, missing provider check.
This test has multiple issues that were flagged in previous reviews but remain unaddressed:
- Critical: Line 2918 uses invalid model
"openai/gpt-5.5"— OpenAI doesn't offer this model. Useget_model("openai", "chat")or a valid model like"gpt-4-turbo"or"gpt-4o".- Major: Missing provider availability check. The test will fail ungracefully if OpenAI isn't configured. Add an early check and
pytest.skip()if the provider is unavailable.🔧 Proposed fix
def test_53_openai_prompt_cache_key_from_metadata_user_id(self, test_config): """Test Case 53: OpenAI prompt_cache_key derived from Anthropic metadata.user_id When an Anthropic-format request carries metadata.user_id, Bifrost sets prompt_cache_key on the outgoing OpenAI request so each user gets an isolated cache bucket. The second request with the same prefix and same user_id should hit OpenAI's automatic prompt cache, confirmed by cache_read_input_tokens > 0 in the response usage. OpenAI caches prompts automatically once the prefix exceeds 1024 tokens, so the system message here is intentionally large. """ _ = test_config + + # Check if OpenAI provider is configured + config = get_config() + if not config.provider_supports_scenario("openai", "simple_chat"): + pytest.skip("OpenAI provider not configured") + client = get_provider_anthropic_client("openai") - model = "openai/gpt-5.5" + model = get_model("openai", "chat") # Use configured model instead of hardcoded invalid model user_id = "bifrost-test-cache-user"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/integrations/python/tests/test_anthropic.py` around lines 2904 - 2919, In test_53_openai_prompt_cache_key_from_metadata_user_id update the invalid model and add a provider availability guard: replace the hardcoded model "openai/gpt-5.5" with a valid model (e.g. use get_model("openai", "chat") or "gpt-4-turbo"/"gpt-4o") and, after calling get_provider_anthropic_client("openai"), check provider availability and call pytest.skip() if OpenAI isn't configured so the test fails gracefully when the provider is absent.
2974-2974:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winRemove unnecessary f-string prefix.
Line 2974 uses an f-string without placeholders. Remove the
fprefix.🧹 Proposed fix
- print(f"✓ prompt_cache_key correctly derived from metadata.user_id") + print("✓ prompt_cache_key correctly derived from metadata.user_id")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/integrations/python/tests/test_anthropic.py` at line 2974, Remove the unnecessary f-string prefix from the print call that prints "✓ prompt_cache_key correctly derived from metadata.user_id" (the print statement with f"✓ prompt_cache_key correctly derived from metadata.user_id"); change it to a normal string literal without the leading f so the output remains the same but avoids using an f-string with no placeholders.
2924-2925:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winRemove unnecessary f-string prefix.
Line 2924 uses an f-string without placeholders. Remove the
fprefix.🧹 Proposed fix
- print(f"\n=== Testing OpenAI prompt_cache_key from metadata.user_id ===") + print("\n=== Testing OpenAI prompt_cache_key from metadata.user_id ===")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/integrations/python/tests/test_anthropic.py` around lines 2924 - 2925, The first print call that uses an unnecessary f-string prefix should be changed to a normal string; locate the print statement printing "\n=== Testing OpenAI prompt_cache_key from metadata.user_id ===" (in the test file with the print calls near the OpenAI prompt_cache_key checks) and remove the leading "f" so it becomes print("...") while leaving the subsequent print that uses {user_id} as an f-string unchanged.
2943-2945:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winDocument or make configurable the hardcoded cache-wait sleep.
The
time.sleep(10)is hardcoded without explanation of why 10 seconds is sufficient (or might be too long/short in CI). Either make this configurable via an environment variable ortest_config, or add a detailed comment explaining the async cache write delay and why this value was chosen.📝 Proposed fix (add explanatory comment)
- # OpenAI writes the prompt cache asynchronously after returning the response. - # A short wait ensures the cache entry is ready before the second request. + # OpenAI writes the prompt cache asynchronously. Wait for cache propagation. + # This duration (10s) is empirically determined; adjust if tests become flaky. time.sleep(10)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/integrations/python/tests/test_anthropic.py` around lines 2943 - 2945, Replace the hardcoded time.sleep(10) with a configurable value (e.g. read CACHE_WAIT_SECONDS from environment or from the existing test_config) and fall back to a sensible default (10) if not set, or alternatively add a detailed comment explaining the async cache write delay and why 10s was chosen; update the code around the time.sleep(10) call so it uses the named variable (e.g. cache_wait_seconds) and document the configuration option and rationale where the sleep occurs.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@core/providers/anthropic/responses.go`:
- Around line 2212-2219: The code sets params.PromptCacheKey whenever
req.Metadata.UserID exists, but does not guard against empty or whitespace-only
IDs; normalize and validate the user id first (e.g., trim whitespace), only
proceed if the resulting key is non-empty, then apply the length/hash logic
(sha256.Sum256) and set params.PromptCacheKey to a pointer to the final key;
ensure you do not assign an empty string pointer to params.PromptCacheKey and
keep all changes around the req.Metadata.UserID -> key -> params.PromptCacheKey
flow.
---
Duplicate comments:
In `@tests/integrations/python/tests/test_anthropic.py`:
- Around line 2904-2919: In
test_53_openai_prompt_cache_key_from_metadata_user_id update the invalid model
and add a provider availability guard: replace the hardcoded model
"openai/gpt-5.5" with a valid model (e.g. use get_model("openai", "chat") or
"gpt-4-turbo"/"gpt-4o") and, after calling
get_provider_anthropic_client("openai"), check provider availability and call
pytest.skip() if OpenAI isn't configured so the test fails gracefully when the
provider is absent.
- Line 2974: Remove the unnecessary f-string prefix from the print call that
prints "✓ prompt_cache_key correctly derived from metadata.user_id" (the print
statement with f"✓ prompt_cache_key correctly derived from metadata.user_id");
change it to a normal string literal without the leading f so the output remains
the same but avoids using an f-string with no placeholders.
- Around line 2924-2925: The first print call that uses an unnecessary f-string
prefix should be changed to a normal string; locate the print statement printing
"\n=== Testing OpenAI prompt_cache_key from metadata.user_id ===" (in the test
file with the print calls near the OpenAI prompt_cache_key checks) and remove
the leading "f" so it becomes print("...") while leaving the subsequent print
that uses {user_id} as an f-string unchanged.
- Around line 2943-2945: Replace the hardcoded time.sleep(10) with a
configurable value (e.g. read CACHE_WAIT_SECONDS from environment or from the
existing test_config) and fall back to a sensible default (10) if not set, or
alternatively add a detailed comment explaining the async cache write delay and
why 10s was chosen; update the code around the time.sleep(10) call so it uses
the named variable (e.g. cache_wait_seconds) and document the configuration
option and rationale where the sleep occurs.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 76d0f165-e8db-4b48-95f8-44bf75779bdd
📒 Files selected for processing (2)
core/providers/anthropic/responses.gotests/integrations/python/tests/test_anthropic.py
TejasGhatte
force-pushed
the
06-05-fix_set_prompt_cache_key_from_anthropic_integration
branch
from
June 5, 2026 13:15
b3372fa to
8b1b757
Compare
Contributor
There was a problem hiding this comment.
♻️ Duplicate comments (1)
tests/integrations/python/tests/test_anthropic.py (1)
2904-2975:⚠️ Potential issue | 🟠 Major | ⚡ Quick winPreviously flagged issues remain unaddressed.
The issues identified in the previous review have not been fixed:
- Invalid model name:
gpt-5.5doesn't exist in OpenAI's model lineup. Useget_model("openai", "chat")or a valid model likegpt-4-turboorgpt-4o.- Missing provider availability check: The test will fail if OpenAI is not configured. Add an early check with
pytest.skip().- Unnecessary f-string prefixes (lines 2924, 2974): Remove the
fprefix from strings without placeholders.- Hardcoded sleep duration (line 2945): The 10-second wait should be configurable via an environment variable or test_config, or at minimum add a comment explaining the async cache write timing.
🔧 Proposed fixes
def test_53_openai_prompt_cache_key_from_metadata_user_id(self, test_config): """Test Case 53: OpenAI prompt_cache_key derived from Anthropic metadata.user_id When an Anthropic-format request carries metadata.user_id, Bifrost sets prompt_cache_key on the outgoing OpenAI request so each user gets an isolated cache bucket. The second request with the same prefix and same user_id should hit OpenAI's automatic prompt cache, confirmed by cache_read_input_tokens > 0 in the response usage. OpenAI caches prompts automatically once the prefix exceeds 1024 tokens, so the system message here is intentionally large. """ _ = test_config + + # Check if OpenAI provider is configured + config = get_config() + if not config.provider_supports_scenario("openai", "simple_chat"): + pytest.skip("OpenAI provider not configured") + client = get_provider_anthropic_client("openai") - model = "openai/gpt-5.5" + model = get_model("openai", "chat") # Use configured model user_id = "bifrost-test-cache-user" # Must exceed OpenAI's 1024-token minimum for automatic prompt caching system = f"You are a legal document analysis assistant.\n\n{PROMPT_CACHING_LARGE_CONTEXT}" - print(f"\n=== Testing OpenAI prompt_cache_key from metadata.user_id ===") - print(f"First request: populating cache bucket for user '{user_id}'...") + print("\n=== Testing OpenAI prompt_cache_key from metadata.user_id ===") + print(f"First request: populating cache bucket for user '{user_id}'...") response1 = client.messages.create( model=model, system=system, messages=[{"role": "user", "content": "Summarize the indemnification clauses."}], max_tokens=256, metadata={"user_id": user_id}, ) assert response1 is not None, "First response should not be None" assert hasattr(response1, "usage"), "First response should have usage" print( f" input_tokens: {response1.usage.input_tokens}, " f"cache_read: {getattr(response1.usage, 'cache_read_input_tokens', 0)}, " f"cache_write: {getattr(response1.usage, 'cache_creation_input_tokens', 0)}" ) - # OpenAI writes the prompt cache asynchronously after returning the response. - # A short wait ensures the cache entry is ready before the second request. + # OpenAI writes the prompt cache asynchronously. Wait for cache propagation. + # This duration is empirically determined; adjust if tests become flaky. time.sleep(10) print(f"\nSecond request: should hit cache for user '{user_id}'...") response2 = client.messages.create( model=model, system=system, messages=[{"role": "user", "content": "What are the governing law provisions?"}], max_tokens=256, metadata={"user_id": user_id}, ) assert response2 is not None, "Second response should not be None" assert hasattr(response2, "usage"), "Second response should have usage" cache_read_tokens = getattr(response2.usage, "cache_read_input_tokens", 0) print( f" input_tokens: {response2.usage.input_tokens}, " f"cache_read: {cache_read_tokens}, " f"cache_write: {getattr(response2.usage, 'cache_creation_input_tokens', 0)}" ) assert cache_read_tokens > 0, ( f"Second request should hit OpenAI prompt cache (cache_read_input_tokens > 0), " f"got {cache_read_tokens}. Check that the system prompt exceeds 1024 tokens " f"and that prompt_cache_key is being set from metadata.user_id." ) print(f"✓ OpenAI prompt cache hit: {cache_read_tokens} cached tokens read") - print(f"✓ prompt_cache_key correctly derived from metadata.user_id") + print("✓ prompt_cache_key correctly derived from metadata.user_id")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/integrations/python/tests/test_anthropic.py` around lines 2904 - 2975, The test test_53_openai_prompt_cache_key_from_metadata_user_id uses an invalid model name, lacks a provider availability check, contains unnecessary f-string prefixes in plain strings, and hardcodes a 10s sleep; fix by (1) replacing model = "openai/gpt-5.5" with a valid model obtained via get_model("openai", "chat") or a concrete valid name like "gpt-4-turbo", (2) adding an early provider check/skip (use pytest.skip) if get_provider_anthropic_client("openai") is unavailable or test_config indicates OpenAI is not configured, (3) remove the superfluous f prefixes on the two print calls that have no interpolations (the prints around the test header and final success), and (4) replace time.sleep(10) with a configurable wait derived from test_config or an environment variable (e.g., prompt_cache_wait) and default to 10s if not provided, so the async cache write wait is configurable.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Duplicate comments:
In `@tests/integrations/python/tests/test_anthropic.py`:
- Around line 2904-2975: The test
test_53_openai_prompt_cache_key_from_metadata_user_id uses an invalid model
name, lacks a provider availability check, contains unnecessary f-string
prefixes in plain strings, and hardcodes a 10s sleep; fix by (1) replacing model
= "openai/gpt-5.5" with a valid model obtained via get_model("openai", "chat")
or a concrete valid name like "gpt-4-turbo", (2) adding an early provider
check/skip (use pytest.skip) if get_provider_anthropic_client("openai") is
unavailable or test_config indicates OpenAI is not configured, (3) remove the
superfluous f prefixes on the two print calls that have no interpolations (the
prints around the test header and final success), and (4) replace time.sleep(10)
with a configurable wait derived from test_config or an environment variable
(e.g., prompt_cache_wait) and default to 10s if not provided, so the async cache
write wait is configurable.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 67f9113b-ffaa-48e4-be68-bccf06a7feb5
📒 Files selected for processing (2)
core/providers/anthropic/responses.gotests/integrations/python/tests/test_anthropic.py
Contributor
Merge activity
|
akshaydeo
deleted the
06-05-fix_set_prompt_cache_key_from_anthropic_integration
branch
June 5, 2026 13:37
18 tasks
akshaydeo
added a commit
that referenced
this pull request
Jun 6, 2026
## Summary This PR bumps the Go toolchain version from `1.26.3` to `1.26.4` across all modules and CI workflows, and cuts a new release (`core` v1.5.17, `framework` v1.3.17, `transports` v1.5.9, `plugins/compat` v0.1.16, `plugins/governance` v1.5.17, and associated plugin versions) incorporating a large batch of features and fixes accumulated since the previous release. ## Changes - **Go 1.26.4** — Updated `go-version` in all GitHub Actions workflows (`e2e-tests`, `helm-release`, `pr-tests`, `release-cli`, `release-pipeline`, `snyk`) and all `go.mod` files (core, framework, transports, cli, all plugins, examples, and test modules). - **Core (v1.5.17)** — OpenAI compaction support, multi-customer logs and usage tracking, multiple team/business unit support, `request_headers` wildcard pattern capture for OTel and Maxim plugins, xAI `x_search` tool, fetch URL validation with SSRF hardening, `file://` pricing URL scheme, virtual key provider fan-out filtering, and a broad set of fixes including Anthropic prompt cache key, empty thinking block stripping, OpenAI stream usage event cleanup, Gemini numeric schema constraints, stale connection retries, Azure Claude diagnostic strip, and passthrough budget handling. - **Framework (v1.3.17)** — Scope-aware budgets and limits wired from model configs, provider-level governance, multiple customer budget support with `calendar_aligned` windows, paginated virtual key fetch, `config.json` source-of-truth flow, FTS index cap reduction, sync worker drift fix, cascade deletes for model configs, and high-scale virtual key flow improvements. - **Transports (v1.5.9)** — Full changelog covering all of the above plus UI improvements (log navigation, customer detail sheet, `BudgetDisplay` component, inline loading shell, materialized view alias), SCIM provisioning fields, Helm/config schema additions (`roles`, `per_user_oauth`), client IP resolution from forwarded headers, and dependency upgrades (`recharts` to 3.8.1, `golang.org/x` CVE remediation). - **Plugins** — `governance` v1.5.17 adds team budget/rate-limit exporters, ghost node reconciliation fix, and VK double usage counting fix; `logging` v1.5.17 adds wildcard header capture and file attachment rendering; `otel` v1.2.17 adds `disable_content_logging` and multiple collectors support; `maxim` v1.6.17 adds `request_headers` wildcard capture; `compat` v0.1.16 fixes `max_tokens` preservation during param filtering. ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [x] Chore/CI ## Affected areas - [x] Core (Go) - [x] Transports (HTTP) - [x] Providers/Integrations - [x] Plugins - [x] UI (React) - [ ] Docs ## How to test ```sh # Verify Go version go version # should report go1.26.4 # Run core tests cd core && go test ./... # Run framework tests cd framework && go test ./... # Run transports tests cd transports && go test ./... # Run plugin tests cd plugins/governance && go test ./... cd plugins/logging && go test ./... cd plugins/otel && go test ./... # UI cd ui pnpm i pnpm build pnpm test ``` ## Breaking changes - [ ] Yes - [x] No ## Related issues #4053, #4066, #4041, #4012, #3976, #3947, #3991, #4045, #3957, #3938, #3937, #3939, #3981, #3998, #3997, #4092, #4091, #4079, #4080, #4086, #3929, #3994, #4028, #3970, #3919, #3861, #3664, #3999, #4088, #4070, #4051, #4043, #4057, #4023, #3941, #3955, #4024, #3956, #3967, #3925, #3992, #3900 ## Security considerations - Fetch URL validation hardened against SSRF by tightening IP checks for private networks and link-local addresses (#4092, #3947, #3991). - Transitive `golang.org/x` dependencies (crypto, net, sys, text) bumped to address Docker Scout CVEs (#3900). ## Checklist - [x] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [x] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [x] I verified the CI pipeline passes locally if applicable <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * OpenAI compaction, multi-customer/team logstore support, request-header wildcard capture, enhanced governance (provider-level & scope-aware limits), disable-content-logging option, support for multiple OpenTelemetry collectors, SSRF hardening and URL validation. * **Chores** * Bumped Go toolchain across modules and updated component/plugin version releases. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Merged
akshaydeo
pushed a commit
that referenced
this pull request
Jun 7, 2026
## Summary When an Anthropic-format request includes `metadata.user_id`, Bifrost now automatically derives a `prompt_cache_key` for the outgoing OpenAI request. This ensures each user gets an isolated cache bucket, enabling OpenAI's automatic prompt caching to work correctly on a per-user basis. ## Changes - When `metadata.user_id` is present and `prompt_cache_key` has not already been set, the user ID is used directly as the cache key. If the user ID exceeds 64 characters (OpenAI's limit), it is SHA-256 hashed to produce a valid fixed-length key. - An integration test (`test_53`) validates the end-to-end behavior: two sequential requests with the same `metadata.user_id` and a large system prompt (exceeding OpenAI's 1024-token caching threshold) confirm that the second request hits the prompt cache (`cache_read_input_tokens > 0`). ## Type of change - [ ] Bug fix - [x] Feature - [ ] Refactor - [ ] Documentation - [ ] Chore/CI ## Affected areas - [x] Core (Go) - [ ] Transports (HTTP) - [x] Providers/Integrations - [ ] Plugins - [ ] UI (React) - [ ] Docs ## How to test ```sh # Core go test ./core/providers/anthropic/... # Integration test (requires valid API keys configured) cd tests/integrations/python pytest tests/test_anthropic.py::TestAnthropicMessages::test_53_openai_prompt_cache_key_from_metadata_user_id -v ``` The integration test sends two requests to an OpenAI model via the Anthropic-compatible client, both carrying the same `metadata.user_id` and a large system prompt. After a 10-second wait for OpenAI's async cache write, the second response is expected to return `cache_read_input_tokens > 0`. ## Breaking changes - [ ] Yes - [x] No ## Security considerations User IDs passed via `metadata.user_id` are used as cache keys and may be forwarded to the upstream provider. If user IDs are sensitive or long, they are SHA-256 hashed before being sent, preventing PII leakage in the raw key value. ## Checklist - [ ] I read `docs/contributing/README.md` and followed the guidelines - [x] I added/updated tests where appropriate - [ ] I updated documentation where needed - [x] I verified builds succeed (Go and UI) - [ ] I verified the CI pipeline passes locally if applicable <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Prompt caching now derives cache keys from metadata user IDs; very long IDs are normalized (hashed) to fit cache limits, improving cache reuse for repeated requests. * **Tests** * Added an integration test to verify prompt cache key behavior when metadata user IDs are used and to ensure cache reads occur on repeated requests. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
akshaydeo
added a commit
that referenced
this pull request
Jun 7, 2026
## ✨ Features - **OpenAI Compaction** — Added OpenAI conversation compaction support across core, framework, logging, and the API surface (#4053) - **Multi-Customer & Org Hierarchy** — Logs and usage tracking now support multiple customers, teams, and business units, including business unit CRUD, team assignment, and governance endpoints in the OpenAPI spec (#4066, #4041, #4082) - **Provider-Level Governance** — Budgets & limits are now scope-aware and can be applied at the virtual-key top level and per provider, wired from the model configs table, with UI filters for scope and providers (#3938, #3937, #3939, #3981, #3962) - **Customer Budgets** — Customers support multiple budgets and `calendar_aligned` budget windows (#3998, #3997) - **Virtual Key Attribution & Controls** — Added a `created_by` user attribution column and a `blacklisted_models` column for virtual key provider configs (#3672, #3653) - **Request Header Capture** — OTel and Maxim observability plugins capture `request_headers` by pattern, with wildcard support (e.g. `x-custom-*`); logging gained the same wildcard header capture (#4012, #3958) - **OTel Content Controls & Collectors** — New `disable_content_logging` option drops message/tool content from exported spans, plus support for multiple OTel collectors (#4064, #3894) - **xAI x_search** — Added xAI `x_search` tool support (#3976) - **URL Validation** — Added fetch URL validation with private-network configuration and link-local blocking (#3947, #3991) - **File Scheme Pricing URLs** — Pricing source URLs now accept the `file://` scheme for air-gapped and self-hosted deployments (#4045) - **Paginated Virtual Keys** — Virtual key fetching is paginated to handle deployments with very large numbers of keys (#3957) - **Client IP Resolution** — Resolve client IP from `X-Forwarded-For`/`X-Real-IP` headers - **SCIM Provisioning** — Added `attributeType`/`attributeValue` SCIM provisioning fields - **Helm/Config Schema** — Added `roles` RBAC governance config and `per_user_oauth` MCP auth to the Helm chart and config schema (#4004, #4009) - **Log Navigation UI** — Added a "View logs" menu item to customer, team, and virtual key tables, clickable links in log detail views, a customer detail sheet, and a reusable `BudgetDisplay` component (#4073, #4054, #4026, #4055) - **Faster First Paint** — Added an inline loading shell to `#root` before React mounts (#4063) - **Materialized View Alias** — Added an `alias` column to the materialized view with filter support (#4078) ## 🐞 Fixed - **Fetch URL IP Checks** — Hardened fetch URL IP checks against SSRF (#4092) - **Mantle Model Matching** — Broadened Mantle model matching to all `gpt` variants (#4091) - **Empty Thinking Blocks** — Strip thinking blocks when the signature is empty (#4079) - **OpenAI Stream Usage** — Removed usage from the `responses.created` event in the OpenAI stream (#4080) - **Prompt Cache Key** — Set the prompt cache key from the Anthropic integration (#4086) - **Upstream Failure Status** — Map upstream connection failures to 502 instead of 400 (#3929) (thanks [@chris-colinsky](https://github.com/chris-colinsky)!) - **Gemini Schema Constraints** — Accept numeric schema integer constraints for Gemini (#3994) (thanks [@yanhao98](https://github.com/yanhao98)!) - **Files Provider Param** — Accept the `?provider=` query param on `GET /v1/files` (#3971) (thanks [@alexef](https://github.com/alexef)!) - **Optional Batch Model** — Made the `model` field optional on `POST /v1/batches` (#3973) (thanks [@alexef](https://github.com/alexef)!) - **Helm Azure Config** — Added missing `azure_key_config` fields to the Helm schema (#3996) (thanks [@axelray-dev](https://github.com/axelray-dev)!) - **Text Completion Chunk Model** — Added the missing `Model` field to `TextCompletionChunkResponse` (#3970) (thanks [@kuishou68](https://github.com/kuishou68)!) - **MCP Inline stdio Env** — MCP stdio server configs accept inline environment variable assignments (#3861) (thanks [@Shushmitaaaa](https://github.com/Shushmitaaaa)!) - **Orphaned Tool Results** — Orphaned tool results in the OpenAI to Anthropic conversion flow are no longer rejected by the Anthropic API (#3919) - **Node Usage Reconciliation** — Added a monotonic `inc_number` log cursor so node usage reconciliation does not skip late async log writes (#3664) - **Bedrock Output Assessments** — Corrected the type of `outputAssessments` in Bedrock responses (#4028) - **Model Pool Pricing Reloads** — Preserve non-pricing model pool entries across pricing reloads (#3999) - **Ghost Node Reconciliation** — Replicate the VK hierarchy flow for ghost node reconciliation (#4088) - **VK Double Usage Counting** — Fixed double usage counting when creating a virtual key (#4070) - **Model Config Lifecycle** — Cascade deletes for model configs and removal of stale in-memory model configs (#4051, #4043) - **FTS Index Cap** — Reduced the FTS index `left()` cap from 800k to 250k chars to stay within the tsvector limit (#4057) - **Sync Worker Drift** — Reduced the sync worker ticker period to 5m to prevent threshold drift (#4023) - **Passthrough** — Fixed passthrough budgets, gated passthrough models per VK, model extraction for Azure passthrough, and restricted fallbacks/provider selection to the VK boundary (#3941, #3988, #3983, #3924) - **Provider Response Headers** — Strip provider response headers and add a content-type filter (#3955, #4024) - **Stream Handling** — Drain non-SSE stream readers and retry stale connections (#3956, #3967) - **Azure Claude** — Strip Azure diagnostic property for Claude models (#3925) - **Compat max_tokens** — Preserve chat `max_tokens` during param filtering (#3992) - **Raw Request Flag** — Removed the raw request flag from providers that don't support it (#4058) - **UI Fixes** — Standardized page container layout, virtual key model configs UI, and dashboard chart tooltips (#4046, #4052, #4044) ## 🔧 Maintenance - **Dependency Upgrades** — Bumped transitive `golang.org/x` dependencies (crypto, net, sys, text) for Docker Scout CVE remediation and `recharts` to 3.8.1; cascaded version bumps across all modules (#3900, #4003)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When an Anthropic-format request includes
metadata.user_id, Bifrost now automatically derives aprompt_cache_keyfor the outgoing OpenAI request. This ensures each user gets an isolated cache bucket, enabling OpenAI's automatic prompt caching to work correctly on a per-user basis.Changes
metadata.user_idis present andprompt_cache_keyhas not already been set, the user ID is used directly as the cache key. If the user ID exceeds 64 characters (OpenAI's limit), it is SHA-256 hashed to produce a valid fixed-length key.test_53) validates the end-to-end behavior: two sequential requests with the samemetadata.user_idand a large system prompt (exceeding OpenAI's 1024-token caching threshold) confirm that the second request hits the prompt cache (cache_read_input_tokens > 0).Type of change
Affected areas
How to test
The integration test sends two requests to an OpenAI model via the Anthropic-compatible client, both carrying the same
metadata.user_idand a large system prompt. After a 10-second wait for OpenAI's async cache write, the second response is expected to returncache_read_input_tokens > 0.Breaking changes
Security considerations
User IDs passed via
metadata.user_idare used as cache keys and may be forwarded to the upstream provider. If user IDs are sensitive or long, they are SHA-256 hashed before being sent, preventing PII leakage in the raw key value.Checklist
docs/contributing/README.mdand followed the guidelinesSummary by CodeRabbit
New Features
Tests