fix(cache): keep prompt_cache_key warm across compression session rotation - #86733
Merged
kshitijk4poor merged 3 commits intoAug 15, 2026
Merged
Conversation
…ation Legacy compaction mode (compression.in_place: false) rotates the physical session_id mid-conversation. The prompt-cache scope introduced in NousResearch#79161 was derived from that physical id, so every rotation moved the same conversation into a fresh cache bucket - the prompt cache went cold at every rotation boundary (NousResearch#79017). Fix: resolve a rotation-stable logical scope - the compression-lineage ROOT of the current session (SessionDB.get_compression_lineage, fork-aware post-NousResearch#79193) - once per turn, memoized per transcript segment, and prefer it over the physical session_id at every prompt_cache_key derivation site: - agent/prompt_cache_scope.py (new): resolve_prompt_cache_scope(agent) - lineage-root walk with per-segment memo; falls back to the physical id when no DB is attached or the walk fails, degrading to pre-fix behavior. - transports/codex.py: build_kwargs accepts cache_scope_id and prefers it for the body prompt_cache_key, the xAI x-grok-conv-id header, and the Codex x-client-request-id routing header. The Codex session_id header keeps the raw physical id (transcript identity, NousResearch#57012 contract). - transports/chat_completions.py: _add_prompt_cache_key accepts cache_scope_id with the same precedence. - chat_completion_helpers.py: build_api_kwargs threads the resolved scope into all three build_kwargs call sites (codex, profile, legacy). - auxiliary_client.py: set_runtime_main carries cache_scope; the aux Responses cache-key site prefers it over the physical session_id. - turn_context.py: resolves the scope once per turn and threads it through set_runtime_main (no DB walk on the per-API-call hot path). Scope semantics preserved from NousResearch#79161: /new starts a fresh scope (new lineage), /branch children, delegate subagents, and tool children stay isolated (explicit-fork exclusion in get_compression_lineage), unrelated sessions keep distinct buckets, and cron per-fire timestamps still normalize via _cache_scope_from_session_id. Default installs compact in place (session_id never rotates), so they hit the memo and produce byte-identical keys to before. Fixes NousResearch#79017
- prompt_cache_scope: memo key now includes DB presence (a lazily attached _session_db re-resolves instead of staying pinned to the physical id); _persist_disabled agents (background-review forks that never get a DB row) memoize the fallback instead of re-querying the lineage per API call; module docstring cross-references get_conversation_root and why the two lineage resolvers must not be deduplicated. - chat_completion_helpers: hoist the triplicated _prompt_cache_scope_for_agent(agent) call to a single local above the OpenAI-wire dispatch (after the anthropic/bedrock early returns, which don't use prompt_cache_key). - codex transport docstring: x-client-request-id mirrors the derived body key, not the raw scope id. - turn_context comment: acknowledge the first-turn pre-persist fallback. - tests: +2 (persist-disabled memoization; lazy DB attach re-resolution).
/simplify-code finding: turn_context evaluated resolve_prompt_cache_scope() inside set_runtime_main's argument list under the umbrella try/except — a resolution failure would silently skip the ENTIRE runtime binding (provider/model/base_url/api_key/session_id for all aux calls that turn), not just the cache scope. - prompt_cache_scope: add resolve_prompt_cache_scope_safe() (never raises, returns None on failure/empty). - turn_context: resolve the scope into a local via the safe variant BEFORE the set_runtime_main call, so a failure can only lose the scope. - chat_completion_helpers: _prompt_cache_scope_for_agent delegates to the shared safe variant (guarded import retained). - tests: +1 (hostile-property agent -> None; normal/empty passthrough).
alt-glitch
added
type/bug
1 task
Closed
6 tasks
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
prompt_cache_keynow stays warm across context-compression session rotation: the cache scope is derived from the rotation-invariant compression-lineage ROOT instead of the physicalsession_id, which rotates mid-conversation in legacy compaction mode (compression.in_place: false) and went cache-cold at every rotation boundary.Fixes #79017.
Root cause
#79161 scoped
prompt_cache_keyby the physical session id (correct for the cross-session bucket-sharing bug it fixed). But legacy-mode compaction mints a new physical id mid-conversation, so the scope — and therefore the key — changed at every rotation even though it's the same conversation continuing:Changes
agent/prompt_cache_scope.py(new):resolve_prompt_cache_scope(agent)— walksSessionDB.get_compression_lineage()(the fork-aware walk hardened in fix(state): stop delegate/tool children corrupting compression lineage #79193) to the rotation-invariant root. Memoized on the agent per(session_id, db-presence);_persist_disabledreview forks memoize the fallback; a pre-persist miss on a persisting agent deliberately re-resolves until the row lands. Explicitly NOTget_conversation_root— that walk collapses/branchand delegate trees, which would break fix(cache): scope prompt_cache_key by session to stop cross-session bucket sharing #79161 isolation.agent/transports/codex.py:build_kwargsacceptscache_scope_id, preferred oversession_idfor the bodyprompt_cache_keyhash and the xAIx-grok-conv-idheader. The Codexsession_idheader keeps the raw physical id (transcript identity, Align Codex cache affinity header #57012);x-client-request-idmirrors the final body key as before.agent/transports/chat_completions.py:_add_prompt_cache_keyacceptscache_scope_idwith the same precedence; both build paths (profile + legacy) forward it.agent/chat_completion_helpers.py:build_api_kwargsresolves the scope once (after the anthropic/bedrock early returns, which don't useprompt_cache_key) and passes it to all three OpenAI-wire branches.agent/auxiliary_client.py+agent/turn_context.py:set_runtime_maincarries acache_scopefield, bound once per turn; the auxiliary Responses cache-key site prefers it — compression/title/MoA aux calls stay rotation-stable too.Scope semantics (all preserved from #79161)
/new/branch(_branched_from)_delegate_from)source="tool")_cache_scope_from_session_idunchanged)Default installs compact in place (physical id never rotates) → keys byte-identical to before; the lineage walk resolves
[sid]once and memoizes.Design constraints follow the maintainer thread on #79017: no DB walk on the per-API-call hot path (resolved once per turn/segment, memoized), scope keyed off real production markers (
_delegate_from/_branched_from/source, notis_subagent), nogateway_session_key(outlives/new), main's anchored cron regex untouched, explicitprompt_cache_keyoverrides (top-level /request_overrides/extra_body) still win.Validation
tests/agent/test_prompt_cache_scope.py)/new/branch/delegate/tool isolation, memo invalidation on rotation, pre-persist un-pinning, lazy-DB-attach re-resolution, persist-disabled memoization, transport wiring, header contracts, cron normalization, aux threadingSessionDB, tempHERMES_HOME, fullbuild_api_kwargschain)pck_key; physical header preserved; 1 lineage walk across 3 API-call builds