Repository navigation
fix(sse): stop thrashing the provider prompt cache for caching-aware clients - #8705
Merged
Merged
Conversation
…clients - shouldPreserveCacheControl: preserve client cache_control markers for every combo strategy. The deterministic-strategy gate forced marker rewrites whose per-request breakpoint positions are not stable turn-over-turn, thrashing the upstream prompt cache (observed in production as ~200k cache_write tokens per turn on quota-share combos). Preserving is never worse: on a stable target the client's breakpoints advance deterministically; on a target switch both approaches miss equally. - prepareClaudeRequest: translator-path opt-in fallback — when preserve-mode has nothing to preserve (client sent no cache_control anywhere), apply the standard heuristic so requests never ship with zero cache breakpoints. The claude-code-compatible relay path keeps its no-supplement contract. - anthropic-beta: forward the client-negotiated context-1m-2025-08-07 through the allowlist merge so a /model <id>[1m] client keeps its long-context negotiation behind the proxy (never forced when the client did not send it).
…ze pending on quota-share block - translateNonStreamingResponse (claude→openai): fold cache_read into prompt_tokens and expose prompt_tokens_details.cached_tokens / cache_creation_tokens, mirroring the streaming contract (#1426/#2215). Non-streaming OpenAI clients behind a cached Claude upstream previously saw prompt_tokens=<uncached remainder> (e.g. 23 for a ~9k request) with no cache visibility. - chatCore quota-share block: finalize the pending-request slot before returning the policy 429 — the path never reaches upstream and the orphaned pending lingered as a status-0 call-log row until the reaper swept it. Validated live on the staging box (repro: blocked key → orphan row; after: clean).
Merged
3 of 4 tasks
HouMinXi
pushed a commit
to HouMinXi/OmniRoute
that referenced
this pull request
Aug 2, 2026
…clients (cheaper-inference#8705) * fix(sse): stop thrashing the provider prompt cache for caching-aware clients - shouldPreserveCacheControl: preserve client cache_control markers for every combo strategy. The deterministic-strategy gate forced marker rewrites whose per-request breakpoint positions are not stable turn-over-turn, thrashing the upstream prompt cache (observed in production as ~200k cache_write tokens per turn on quota-share combos). Preserving is never worse: on a stable target the client's breakpoints advance deterministically; on a target switch both approaches miss equally. - prepareClaudeRequest: translator-path opt-in fallback — when preserve-mode has nothing to preserve (client sent no cache_control anywhere), apply the standard heuristic so requests never ship with zero cache breakpoints. The claude-code-compatible relay path keeps its no-supplement contract. - anthropic-beta: forward the client-negotiated context-1m-2025-08-07 through the allowlist merge so a /model <id>[1m] client keeps its long-context negotiation behind the proxy (never forced when the client did not send it). * fix(sse): surface cache tokens in non-streaming OpenAI usage + finalize pending on quota-share block - translateNonStreamingResponse (claude→openai): fold cache_read into prompt_tokens and expose prompt_tokens_details.cached_tokens / cache_creation_tokens, mirroring the streaming contract (cheaper-inference#1426/cheaper-inference#2215). Non-streaming OpenAI clients behind a cached Claude upstream previously saw prompt_tokens=<uncached remainder> (e.g. 23 for a ~9k request) with no cache visibility. - chatCore quota-share block: finalize the pending-request slot before returning the policy 429 — the path never reaches upstream and the orphaned pending lingered as a status-0 call-log row until the reaper swept it. Validated live on the staging box (repro: blocked key → orphan row; after: clean). --------- Co-authored-by: diegosouzapw <diegosouzapw24@gmail.com>
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…clients (cheaper-inference#8705) * fix(sse): stop thrashing the provider prompt cache for caching-aware clients - shouldPreserveCacheControl: preserve client cache_control markers for every combo strategy. The deterministic-strategy gate forced marker rewrites whose per-request breakpoint positions are not stable turn-over-turn, thrashing the upstream prompt cache (observed in production as ~200k cache_write tokens per turn on quota-share combos). Preserving is never worse: on a stable target the client's breakpoints advance deterministically; on a target switch both approaches miss equally. - prepareClaudeRequest: translator-path opt-in fallback — when preserve-mode has nothing to preserve (client sent no cache_control anywhere), apply the standard heuristic so requests never ship with zero cache breakpoints. The claude-code-compatible relay path keeps its no-supplement contract. - anthropic-beta: forward the client-negotiated context-1m-2025-08-07 through the allowlist merge so a /model <id>[1m] client keeps its long-context negotiation behind the proxy (never forced when the client did not send it). * fix(sse): surface cache tokens in non-streaming OpenAI usage + finalize pending on quota-share block - translateNonStreamingResponse (claude→openai): fold cache_read into prompt_tokens and expose prompt_tokens_details.cached_tokens / cache_creation_tokens, mirroring the streaming contract (cheaper-inference#1426/cheaper-inference#2215). Non-streaming OpenAI clients behind a cached Claude upstream previously saw prompt_tokens=<uncached remainder> (e.g. 23 for a ~9k request) with no cache visibility. - chatCore quota-share block: finalize the pending-request slot before returning the policy 429 — the path never reaches upstream and the orphaned pending lingered as a status-0 call-log row until the reaper swept it. Validated live on the staging box (repro: blocked key → orphan row; after: clean). --------- Co-authored-by: diegosouzapw <diegosouzapw24@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A Claude Code (or any caching-aware) client routed through a combo had its own
cache_controlmarkers stripped and rewritten unless the combo used one of thetwo "deterministic" strategies (
priority,cost-optimized). The rewrittenbreakpoint positions are recomputed per request and are not stable
turn-over-turn, which invalidates the provider prompt cache on every turn.
Observed in production on quota-share (
qtSd/) combos:cacheReadstuck at thetools+system prefix while
cacheWritere-wrote ~200k tokens per turn — theaccount quota drained at a multiple of the expected rate.
Fix (three coordinated changes)
shouldPreserveCacheControlnow preserves client markers for every combostrategy. Preserving is never worse than rewriting: on a stable target the
client's breakpoints advance deterministically (the per-request heuristic's
do not), and on a target switch both approaches miss equally. The
alwaysPreserveClientCache: "always" | "never"overrides are unchanged, asare the caching-aware-client and caching-provider gates.
prepareClaudeRequestgains a translator-path opt-in fallback: whenpreserve-mode has nothing to preserve (the client sent no
cache_controlanywhere), the standard heuristic still applies so a request never ships with
zero cache breakpoints. The claude-code-compatible relay path keeps its
long-standing no-supplement contract (
cc-compatible-provider.test.tsunchanged and green).
FORWARDABLE_CLIENT_BETASnow forwards a client-negotiatedcontext-1m-2025-08-07. Claude Code with/model <id>[1m]sends that betaitself; the allowlist merge silently dropped it for non-Opus models. It is
still never forced when the client did not send it (guarded by test).
Tests
tests/unit/cache-control-policy.test.ts— contract updated: preservation forall strategies incl.
quota-share; regressions kept (non-caching provider,non-caching-aware client, always/never overrides).
tests/unit/claude-prepare-request-preserve-fallback.test.ts(new) — thefallback + the no-supplement contract without the opt-in flag.
tests/unit/claude-context-1m-client-beta-forward.test.ts(new) — forwardwhen negotiated, never forced otherwise; [BUG] tool-search-tool-2025-10-19 beta dropped on Claude OAuth path → 400 "Tool reference not found" with ENABLE_TOOL_SEARCH #3974 tool-search regression kept.
typecheck:coreclean; lint clean on touched files;translator suite 688/688; touched suites 115/115; full
npm run test:unit(26k+ tests) with the only failures reproducing identically on the clean base
branch in this environment (env-doc-sync / i18n parity / monaco /
ServiceSupervisor port-adoption — pre-existing, unrelated files).