Repository navigation
fix(compression): preserve cacheable prefix for automatic-cache providers (#3955) - #4334
Merged
Merged
Conversation
…ntrol (#3955) OpenAI / Codex / Azure-OpenAI use automatic prefix caching: the upstream caches the longest matching prefix of a request (system prompt + earliest messages) WITHOUT any explicit cache_control markers. The cache-aware compression guard only protected that prefix when the body carried explicit cache_control, so for automatic-cache providers the guard was skipped — and with compression active + preserveSystemPrompt:false (or a prefix-compressing mode) it rewrote the prefix, guaranteeing a cache miss and higher token spend through OmniRoute than going direct. getCacheAwareStrategy now treats isCachingProvider alone as sufficient to skip the system prompt and downgrade aggressive/ultra (the explicit cache_control path is a subset). openai/codex/azure are added to CACHING_PROVIDERS so they are recognized as automatic-cache providers (this also activates the intended prompt_cache_key cache-routing hint for OpenAI in chatCore). Compression remains off by default — this only affects operators who enabled it with prefix preservation turned off. TDD: tests/unit/compression-cache-guard-3955.test.ts (RED 5/7 fail → GREEN 7/7). Aligned the existing cachingAware / strategySelector-cache-aware / cache-control-policy / cache-control-claude-providers tests that encoded the old (buggy) "openai is non-caching" behavior. Refs #3955
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
…ompression-cache-guard
tkgo11
pushed a commit
to tkgo11/OmniRoute
that referenced
this pull request
Sep 23, 2026
…ders (cheaper-inference#3955) (cheaper-inference#4334) OpenAI / Codex / Azure-OpenAI use automatic prefix caching: the upstream caches the longest matching prefix of a request (system prompt + earliest messages) WITHOUT any explicit cache_control markers. The cache-aware compression guard only protected that prefix when the body carried explicit cache_control, so for automatic-cache providers the guard was skipped — and with compression active + preserveSystemPrompt:false (or a prefix-compressing mode) it rewrote the prefix, guaranteeing a cache miss and higher token spend through OmniRoute than going direct. getCacheAwareStrategy now treats isCachingProvider alone as sufficient to skip the system prompt and downgrade aggressive/ultra (the explicit cache_control path is a subset). openai/codex/azure are added to CACHING_PROVIDERS so they are recognized as automatic-cache providers (this also activates the intended prompt_cache_key cache-routing hint for OpenAI in chatCore). Compression remains off by default — this only affects operators who enabled it with prefix preservation turned off. TDD: tests/unit/compression-cache-guard-3955.test.ts (RED 5/7 fail → GREEN 7/7). Aligned the existing cachingAware / strategySelector-cache-aware / cache-control-policy / cache-control-claude-providers tests that encoded the old (buggy) "openai is non-caching" behavior. Refs cheaper-inference#3955
tkgo11
pushed a commit
to tkgo11/OmniRoute
that referenced
this pull request
Sep 23, 2026
… lote drift) (cheaper-inference#4336) cheaper-inference#4327 (per-key USD usage quotas), cheaper-inference#4334 (cache-aware compression guard) and cheaper-inference#4326 (phaseComboSetup) added new conditional branches landing after cheaper-inference#4330 measured 1890; the release fast-path doesn't run check:complexity. Measured 1896 on the merged tip.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #3955 (the deterministic, reproducible sub-bug)
Problem
Higher prompt-cache miss rate through OmniRoute vs direct for OpenAI/Codex. The cache-aware compression guard (
getCacheAwareStrategy) only protected the cacheable prefix whenisCachingProvider && hasCacheControl. But OpenAI / Codex / Azure-OpenAI use automatic prefix caching — the upstream caches the longest matching prefix without anycache_controlmarkers. Those providers weren't even inCACHING_PROVIDERS, sogetCacheAwareStrategywas a no-op for them, and with compression active +preserveSystemPrompt:false(oraggressive/ultra) the system prompt/earliest messages got rewritten → guaranteed cache miss.Fix (twofold — both required)
open-sse/utils/cacheControlPolicy.ts— addopenai,codex,azuretoCACHING_PROVIDERS(they have automatic prefix caching). This also activates the already-present-but-dormantprompt_cache_keycache-routing hint for OpenAI inchatCore.ts(gated onproviderSupportsCaching; injects a prefix-stable key fromgeneratePromptCacheKey→analyzePrefix().prefixHash, only when the client didn't set one;codex/nvidia/xaistay excluded from that injection).open-sse/services/compression/cachingAware.ts—getCacheAwareStrategyprotects the prefix onisCachingProvideralone (the explicit-cache_controlpath is now a subset):skipSystemPrompt: true+ downgradeaggressive/ultra→standard.Validation (Hard Rule #18 — TDD)
tests/unit/compression-cache-guard-3955.test.ts(7 cases): caching provider with a long system prompt and nocache_control+ compression on +preserveSystemPrompt:false→ prefix preserved (RED 2/7 → GREEN 7/7); a genuinely non-caching provider (gemini/google) is unaffected. Four existing tests that encoded the old buggy "openai is non-caching" behavior were aligned to the corrected behavior (negative cases switched to a real non-caching provider — not weakened). Regression: fulltests/unit/compression/(87 files) 737/737 + cache-control/codex-cache-key suites; 82/82 on my own re-run. Lint 0-err,typecheck:coreclean, no file-size bump.