Skip to content

fix(compression): preserve upstream prompt cache for memory + compression (#3890) - #3936

Merged
diegosouzapw merged 1 commit into
release/v3.8.26from
fix/3890-compression-cache-preserve
Jun 15, 2026
Merged

diegosouzapw merged 1 commit into
release/v3.8.26from
fix/3890-compression-cache-preserve

Conversation

@diegosouzapw

Copy link
Copy Markdown
Collaborator

Closes #3890

Problem

With compression and/or memory enabled, requests to caching providers (Anthropic-family) miss the prompt cache on every turn, multiplying cost (the reporter saw ~5x). Disabling both restored a 90-99% hit rate.

Root causes (two, both addressed)

  1. Memory injection poisoned the cacheable prefix. injectMemory prepended the retrieved memories at index 0 of the message array. Retrieved memories vary per user query (retrieveMemories({ query: lastUserQuery })), so the very front of the prompt changed every turn → every cache_control breakpoint downstream missed.
  2. The cache-aware skipSystemPrompt flag was dropped. getCacheAwareStrategy() computes skipSystemPrompt/deterministicOnly, but selectCompressionStrategy() can only return a mode string, so those flags were discarded — the system prompt (part of the cacheable prefix) could still be compressed under caching.

Fix

  1. Memory (src/lib/memory/injection.ts + memorySkillsInjection.ts): when the request carries cache_control breakpoints (detectCachingContext(...).hasCacheControl), insert the memory message just before the last user message instead of at index 0 — the cacheable prefix (system prompt + prior turns) stays byte-stable while memory still contextualizes the current turn. Non-caching requests keep the existing leading-injection behavior.
  2. Compression (strategySelector.ts + chatCore.ts): new resolveCacheAwareConfig() applies the dropped skipSystemPrompt decision by forcing preserveSystemPrompt on for caching requests, even when the operator disabled it. Non-caching requests are untouched.

Both changes only ever preserve more of the prefix under caching — they cannot make caching worse.

Validation (TDD — Hard Rule #18)

  • tests/unit/memory-cache-safe-injection.test.ts (new): proves memory is inserted before the last user message under caching (prefix byte-stable across two turns), keeps legacy index-0 behavior when caching is off, and falls back safely when there is no user message.
  • tests/unit/compression/strategySelector-cache-aware.test.ts (new): proves resolveCacheAwareConfig forces preserveSystemPrompt for caching requests and leaves non-caching requests untouched.

All green; typecheck:core clean, lint 0 errors, no new cycles, regression suites green (memory-glm-injection 22/22, cachingAware 17/17). file-size re-baselined (chatCore 5815→5823).

@xenstar — the unit tests prove the request no longer injects per-query content into the cacheable prefix; please confirm the cache-hit rate recovers on v3.8.26 with your RTK/Caveman + memory setup, since upstream cache-hit rates can only be observed live.

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@diegosouzapw
diegosouzapw force-pushed the fix/3890-compression-cache-preserve branch from 71a4dcb to 5ebfea0 Compare June 15, 2026 23:05
@diegosouzapw
diegosouzapw merged commit f367901 into release/v3.8.26 Jun 15, 2026
2 checks passed
@diegosouzapw
diegosouzapw deleted the fix/3890-compression-cache-preserve branch June 15, 2026 23:05
@diegosouzapw diegosouzapw mentioned this pull request Jun 16, 2026
diegosouzapw added a commit that referenced this pull request Jun 16, 2026
OmniRoute v3.8.26 — see CHANGELOG.md [3.8.26] for the full notes.

Highlights: Vertex AI media generation (#3929), GLM-5.2 effort-tier routing (#3885),
sticky round-robin combos (#3846), OpenRouter connection presets (#3878), compression
prompt-cache fix (#3936/#3890), and a security pass (form-data/vite + workflow hardening, #3949).

Co-authored-by: artickc <artickc@users.noreply.github.com>
Co-authored-by: rdself <rdself@users.noreply.github.com>
Co-authored-by: herjarsa <herjarsa@users.noreply.github.com>
Co-authored-by: Jack Smith <16862258+YunyunZhai@users.noreply.github.com>
Co-authored-by: dhaern <dhaern@users.noreply.github.com>
Co-authored-by: adivekar-utexas <adivekar-utexas@users.noreply.github.com>
Co-authored-by: megamen32 <megamen32@users.noreply.github.com>
Co-authored-by: zhiru <zhiru@users.noreply.github.com>
Co-authored-by: insoln <insoln@users.noreply.github.com>
Co-authored-by: diego-anselmo <diego-anselmo@users.noreply.github.com>
HouMinXi pushed a commit to HouMinXi/OmniRoute that referenced this pull request Aug 2, 2026
OmniRoute v3.8.26 — see CHANGELOG.md [3.8.26] for the full notes.

Highlights: Vertex AI media generation (cheaper-inference#3929), GLM-5.2 effort-tier routing (cheaper-inference#3885),
sticky round-robin combos (cheaper-inference#3846), OpenRouter connection presets (cheaper-inference#3878), compression
prompt-cache fix (cheaper-inference#3936/cheaper-inference#3890), and a security pass (form-data/vite + workflow hardening, cheaper-inference#3949).

Co-authored-by: artickc <artickc@users.noreply.github.com>
Co-authored-by: rdself <rdself@users.noreply.github.com>
Co-authored-by: herjarsa <herjarsa@users.noreply.github.com>
Co-authored-by: Jack Smith <16862258+YunyunZhai@users.noreply.github.com>
Co-authored-by: dhaern <dhaern@users.noreply.github.com>
Co-authored-by: adivekar-utexas <adivekar-utexas@users.noreply.github.com>
Co-authored-by: megamen32 <megamen32@users.noreply.github.com>
Co-authored-by: zhiru <zhiru@users.noreply.github.com>
Co-authored-by: insoln <insoln@users.noreply.github.com>
Co-authored-by: diego-anselmo <diego-anselmo@users.noreply.github.com>
Poid-ZA pushed a commit to Poid-ZA/OmniRoute that referenced this pull request Aug 5, 2026
OmniRoute v3.8.26 — see CHANGELOG.md [3.8.26] for the full notes.

Highlights: Vertex AI media generation (cheaper-inference#3929), GLM-5.2 effort-tier routing (cheaper-inference#3885),
sticky round-robin combos (cheaper-inference#3846), OpenRouter connection presets (cheaper-inference#3878), compression
prompt-cache fix (cheaper-inference#3936/cheaper-inference#3890), and a security pass (form-data/vite + workflow hardening, cheaper-inference#3949).

Co-authored-by: artickc <artickc@users.noreply.github.com>
Co-authored-by: rdself <rdself@users.noreply.github.com>
Co-authored-by: herjarsa <herjarsa@users.noreply.github.com>
Co-authored-by: Jack Smith <16862258+YunyunZhai@users.noreply.github.com>
Co-authored-by: dhaern <dhaern@users.noreply.github.com>
Co-authored-by: adivekar-utexas <adivekar-utexas@users.noreply.github.com>
Co-authored-by: megamen32 <megamen32@users.noreply.github.com>
Co-authored-by: zhiru <zhiru@users.noreply.github.com>
Co-authored-by: insoln <insoln@users.noreply.github.com>
Co-authored-by: diego-anselmo <diego-anselmo@users.noreply.github.com>
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
OmniRoute v3.8.26 — see CHANGELOG.md [3.8.26] for the full notes.

Highlights: Vertex AI media generation (cheaper-inference#3929), GLM-5.2 effort-tier routing (cheaper-inference#3885),
sticky round-robin combos (cheaper-inference#3846), OpenRouter connection presets (cheaper-inference#3878), compression
prompt-cache fix (cheaper-inference#3936/cheaper-inference#3890), and a security pass (form-data/vite + workflow hardening, cheaper-inference#3949).

Co-authored-by: artickc <artickc@users.noreply.github.com>
Co-authored-by: rdself <rdself@users.noreply.github.com>
Co-authored-by: herjarsa <herjarsa@users.noreply.github.com>
Co-authored-by: Jack Smith <16862258+YunyunZhai@users.noreply.github.com>
Co-authored-by: dhaern <dhaern@users.noreply.github.com>
Co-authored-by: adivekar-utexas <adivekar-utexas@users.noreply.github.com>
Co-authored-by: megamen32 <megamen32@users.noreply.github.com>
Co-authored-by: zhiru <zhiru@users.noreply.github.com>
Co-authored-by: insoln <insoln@users.noreply.github.com>
Co-authored-by: diego-anselmo <diego-anselmo@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants