fix(cache): scope prompt_cache_key by session to stop cross-session bucket sharing - #79161
Merged
Merged
Conversation
…ucket sharing Cherry-picked from PR NousResearch#78959 by @JoaoMarcos44 with authorship preserved. Follow-up: hoist _cache_scope_from_session_id(session_id) to a local in build_kwargs so it's computed once instead of 4 times per call. Closes NousResearch#78941. Closes NousResearch#79012. Closes NousResearch#79013. Closes NousResearch#79014. Closes NousResearch#79015. Co-authored-by: JoaoMarcos44 <joaomarcosdias444@gmail.com>
kshitijk4poor
enabled auto-merge (rebase)
August 5, 2026 07:09
5 tasks
This was referenced Aug 5, 2026
Closed
kshitijk4poor
added a commit
that referenced
this pull request
Aug 15, 2026
…ation Legacy compaction mode (compression.in_place: false) rotates the physical session_id mid-conversation. The prompt-cache scope introduced in #79161 was derived from that physical id, so every rotation moved the same conversation into a fresh cache bucket - the prompt cache went cold at every rotation boundary (#79017). Fix: resolve a rotation-stable logical scope - the compression-lineage ROOT of the current session (SessionDB.get_compression_lineage, fork-aware post-#79193) - once per turn, memoized per transcript segment, and prefer it over the physical session_id at every prompt_cache_key derivation site: - agent/prompt_cache_scope.py (new): resolve_prompt_cache_scope(agent) - lineage-root walk with per-segment memo; falls back to the physical id when no DB is attached or the walk fails, degrading to pre-fix behavior. - transports/codex.py: build_kwargs accepts cache_scope_id and prefers it for the body prompt_cache_key, the xAI x-grok-conv-id header, and the Codex x-client-request-id routing header. The Codex session_id header keeps the raw physical id (transcript identity, #57012 contract). - transports/chat_completions.py: _add_prompt_cache_key accepts cache_scope_id with the same precedence. - chat_completion_helpers.py: build_api_kwargs threads the resolved scope into all three build_kwargs call sites (codex, profile, legacy). - auxiliary_client.py: set_runtime_main carries cache_scope; the aux Responses cache-key site prefers it over the physical session_id. - turn_context.py: resolves the scope once per turn and threads it through set_runtime_main (no DB walk on the per-API-call hot path). Scope semantics preserved from #79161: /new starts a fresh scope (new lineage), /branch children, delegate subagents, and tool children stay isolated (explicit-fork exclusion in get_compression_lineage), unrelated sessions keep distinct buckets, and cron per-fire timestamps still normalize via _cache_scope_from_session_id. Default installs compact in place (session_id never rotates), so they hit the memo and produce byte-identical keys to before. Fixes #79017
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Scopes
prompt_cache_keyby session_id so unrelated sessions stop sharing a cache bucket, while cron re-fires of the same job still share a stable key.Root cause:
prompt_cache_keywas content-addressed only from the static request prefix (system instructions + tool schemas), with no session/tenant component. Any two unrelated sessions sharing the same system prompt and tool set collapsed onto the same cache routing bucket.Changes
agent/transports/codex.py: New_cache_scope_from_session_id()normalizes cron per-fire timestamps;_content_cache_key()gainsscope_idparam; Codex headers now separate identity (session_id= raw) from routing (x-client-request-id= body key); xAIextra_body.prompt_cache_keyhonors top-level overrideagent/transports/chat_completions.py: Threadssession_idthrough_add_prompt_cache_keyagent/auxiliary_client.py: Auxiliary Codex adapter now scopes by session viaset_runtime_main(session_id=...)agent/turn_context.py:build_turn_contextpassessession_idtoset_runtime_mainplugins/model-providers/nous/__init__.py+openrouter/__init__.py: Apply_cache_scope_from_session_idto sticky routing keys andx-grok-conv-id_cache_scopeto a local inbuild_kwargs(computed once instead of 4 times)Closes #78941. Closes #79012. Closes #79013. Closes #79014. Closes #79015.
Salvaged from #78959 by @JoaoMarcos44 — cherry-picked with authorship preserved.
Validation
set_runtime_main