fix(codex): persist app-server turns to session DB, exactly once (salvage #49225) - #56343
Merged
kshitijk4poor merged 2 commits intoJul 1, 2026
Merged
Conversation
…all)
The codex_app_server runtime path (run_codex_app_server_turn in
agent/codex_runtime.py) is an early-return that bypasses
conversation_loop and never calls _flush_messages_to_session_db().
Meanwhile, gateway/run.py sets:
agent_persisted = self._session_db is not None # always True
and passes skip_db=agent_persisted to every append_to_transcript call,
assuming the agent self-persisted (correct for the standard runtime,
wrong for codex). The result: codex turn messages are persisted nowhere.
state.db accumulates only session_meta rows; session_search (full-text
search over state.db) and conversation-distill are blind to real gateway
conversations, causing 'the agent has no memory of what we discussed'.
Fix (three-part, all backward-compatible):
1. agent/codex_runtime.py — run_codex_app_server_turn success return
now includes 'agent_persisted': False, signalling that the codex path
did NOT self-persist its turn.
2. gateway/run.py — the agent_persisted assignment now reads:
agent_result.get('agent_persisted', self._session_db is not None)
For the standard runtime (which does not set the key) the default
(self._session_db is not None) preserves the existing skip-db
behaviour so no duplicate-write regression (NousResearch#860 / NousResearch#42039) occurs.
For the codex runtime the flag is False, so the gateway writes the
new turn's messages to state.db and FTS index.
3. gateway/run.py — the rebuilt result dict (run_agent return, which
becomes agent_result upstream) now includes agent_persisted passed
through from result_holder[0], with a safe True default. Without
this passthrough the flag set in step 1 was discarded when the result
was reconstructed, causing agent_result.get('agent_persisted', ...)
to always see the default True and never write codex turns.
Follow-up correcting the salvaged fix's persistence approach to avoid a duplicate user-message write (verified via E2E — the NousResearch#860/NousResearch#42039 bug class the original diff aimed to avoid). Root cause: in gateway mode the AIAgent is built WITH a session_db, so the inbound user turn is already flushed at turn start (turn_context. _persist_session). The original fix returned agent_persisted=False, making the gateway re-write the whole new-message slice via append_to_transcript -> append_message (a raw INSERT with no dedup), duplicating the already-flushed user turn. Corrected approach (single writer): run_codex_app_server_turn now flushes its OWN projected assistant/tool messages via _flush_messages_to_session_db (which dedups the already-persisted user turn through _DB_PERSISTED_MARKER) and returns agent_persisted=True so the gateway skips its write. Net result: session_search/distill see the full codex conversation, each message persisted exactly once. Adds regression coverage asserting exactly-once persistence on a real SessionDB, agent_persisted=True, FTS visibility, and standard-runtime skip-db behaviour preserved. Co-authored-by: Lubos Buracinsky <lubos@komfi.health>
alt-glitch
added
type/bug
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Codex app-server gateway turns now reach the session DB, so
session_search(FTS) and conversation-distill can see real codex conversations. Previously the codex runtime early-returned pastconversation_loop's per-step flushes, so its projected assistant/tool messages were persisted nowhere (state.db held only session_meta rows).Salvage of #49225 by @lubosxyz, cherry-picked with authorship preserved, plus a maintainer follow-up correcting the persistence approach and adding tests.
Changes
agent/codex_runtime.py(contributor commit + follow-up):run_codex_app_server_turnflushes its own projected assistant/tool messages via_flush_messages_to_session_db()after splicing them, and returnsagent_persisted=True.gateway/run.py(contributor commit): readsagent_persistedfrom the runtime result at both persistence sites (default preserves standard-runtime behaviour).tests/agent/test_codex_app_server_persist.py(follow-up): regression coverage.Follow-up correction (why the approach changed)
The contributor's original diff returned
agent_persisted=Falseand had the gateway write the codex turn. During review I found — and confirmed by E2E — that this reintroduces the exact #860/#42039 duplicate-write bug: in gateway mode the AIAgent is built with asession_db, so the inbound user turn is already flushed at turn start (turn_context._persist_session). Having the gateway then re-write the new-message slice viaappend_to_transcript→append_message(a raw INSERT with no dedup) duplicates that user turn.E2E proof (before fix): turn-start flush + gateway write →
USER_TURNrow count = 2.Corrected single-writer approach: the codex runtime flushes its own projected messages (the marker-based
_flush_messages_to_session_dbdedups the already-persisted user turn) and reportsagent_persisted=Trueso the gateway skips its write. Lubos's diagnosis and gateway wiring are preserved; the correction lands as a follow-up commit withCo-authored-by.Validation
tests/agent/test_codex_app_server_persist.py(5) +tests/gateway/test_42039_duplicate_user_message.py(4) — 9 passed.SessionDB+ realAIAgent._flush_messages_to_session_db): user turn persisted exactly once, projected assistant message exactly once, FTS finds the codex turn,agent_persisted=True. No bug: SQLite session transcript accumulates duplicate messages (3-4x token inflation) #860/Bug: User messages stored twice in state.db when agent and gateway both write to SQLite #42039 regression.