fix(agent): resume codex app-server thread across turns to preserve session context by gmkseta · Pull Request #41905 · NousResearch/hermes-agent

gmkseta · 2026-06-08T07:36:43Z

What does this PR do?

The codex_app_server runtime is documented to keep "one Codex thread per Hermes session" (agent/transports/codex_app_server_session.py), but the gateway builds a fresh AIAgent per inbound message. Before this fix, each follow-up could call a new thread/start, handing Codex an empty working thread even though Hermes still had the message transcript.

A Codex thread's working context — files read, command output, plan state, and long-task runtime state — lives inside the Codex thread, not in Hermes' transcript replay. Losing the thread id makes Discord/Telegram follow-ups look like the agent forgot the task.

This PR records the Codex thread id per Hermes session and resumes it with thread/resume. The mapping is persisted in ~/.hermes/sessions/sessions.json, so a gateway restart does not lose the resume target. For durable thread/topic lanes, normal idle/daily Hermes transcript resets also keep the Codex working thread; explicit fresh-start boundaries still drop it.

Related Issue

Fixes #41904

Type of Change

Bug fix (non-breaking change that fixes an issue)

Changes Made

agent/transports/codex_app_server_session.py — CodexAppServerSession accepts resume_thread_id. ensure_started() issues thread/resume {threadId} when set, falling back to thread/start on resume failure so the user turn is not dropped.
agent/codex_runtime.py — run_codex_app_server_turn() passes the requested resume thread into the session and records the returned codex_thread_id on the agent after each turn.
gateway/session.py — persists SessionEntry.codex_thread_id to sessions.json with helpers for reading/writing/clearing it. Idle/daily reset carries the id only for durable thread/topic lanes; DM/non-thread sessions and suspended stuck sessions still start fresh.
gateway/run.py — restores the persisted thread id onto fresh per-message agents, persists successful turn ids, skips stale-generation writes after /new or /stop, keeps thread/topic ids through expiry finalization, and updates auto-reset system/user notices so transcript reset does not contradict Codex resume behavior.
Docs — updates the Codex app-server runtime docs with cross-turn persistence details, architecture notes, and an operational runbook for verifying gateway restart and daily-reset recovery.

How to Test

Validated on the rebased PR branch:

python -m py_compile gateway/run.py gateway/session.py agent/codex_runtime.py agent/transports/codex_app_server_session.py tests/gateway/test_session.py
pytest tests/agent/transports/test_codex_app_server_session.py tests/run_agent/test_codex_app_server_integration.py tests/agent/transports/test_codex_app_server_runtime.py tests/gateway/test_session.py::TestLastPromptTokens tests/gateway/test_session.py::TestCodexThreadPersistence tests/gateway/test_session_env.py tests/gateway/test_session_reset_notify.py tests/gateway/test_restart_resume_pending.py -q

Result: 226 passed.

New coverage includes:

TestThreadResume for thread/resume versus thread/start, resume fallback, and thread id extraction.
TestCodexThreadResumePersistence for runtime round-tripping of the resume id.
TestCodexThreadPersistence for SessionEntry JSON roundtrip, old sessions.json compatibility, SessionStore save/reload/clear behavior, thread idle/daily reset carry-over, DM reset fresh-start behavior, and suspended reset fresh-start behavior.

Operational Notes

For a gateway lane such as a Discord thread, a successful Codex app-server turn writes codex_thread_id under that lane's session_key in ~/.hermes/sessions/sessions.json. After a gateway restart, the next turn in the same platform thread loads that persisted id and asks Codex to thread/resume.

Idle/daily Hermes transcript resets rotate the Hermes session id but keep the Codex thread id for durable thread/topic lanes, so same-thread follow-ups can continue Codex working context without replaying Discord history into the prompt. /new, /reset, suspended stuck-session recovery, and explicit lane re-binding remain fresh-start boundaries.

Platforms tested

macOS (Apple Silicon), Python 3.11, Codex CLI app-server runtime

Checklist

Conventional commits
Rebased onto current origin/main
Relevant tests pass
Documentation and runbook updated
No new config keys; resume is automatic and transparent

gmkseta · 2026-06-08T09:21:06Z

Live verification against real Codex 0.133

Ran the actual codex app-server subprocess (no mocks) to confirm thread/resume preserves context across a session teardown — i.e. the gateway's fresh-AIAgent-per-message pattern:

Turn 1 — fresh thread 019ea688-7067…: "Remember this secret code: BANANA-42" → ok
session closed (simulating the gateway tearing down the per-message agent)
Control, no resume — brand-new thread 019ea688-7dfe…: "what was the secret code?" → I DO NOT KNOW ← the bug this PR fixes
Fix, resume — thread 019ea688-7067… (the same id as turn 1): "what was the secret code?" → BANANA-42 ← context preserved

  turn1 thread id             : 019ea688-7067-7461-a6a6-4237815eebc2
  CONTROL (no resume) recalls : False   <- expected (fresh thread has no record)
  FIX     (resume)    recalls : True    <- resumed thread reattaches + recalls
  PROVEN: True

The resumed session reattaches to the exact thread id from turn 1 and the model recalls context the fresh thread/start has no record of. Verified on macOS (Apple Silicon), Python 3.11, Codex CLI 0.133.0.

tonydwb

Code Review Summary

Verdict: Approved

Analysis

Correctness ✅

The fix solves a real problem: the gateway builds a fresh AIAgent per message, but codex app-server needs the same thread across turns to preserve working context (file reads, tool output, long-running task state).
The implementation is layered and correct: gateway persists codex_thread_id → runtime passes resume_thread_id to session → session calls thread/resume if set → falls back to thread/start on any failure.
The _session_codex_threads hasattr guard handles forward compatibility correctly.

Testing ✅

Excellent test coverage: thread/resume success, resume failure (method unsupported, timeout, RuntimeError), no resume id, session id alias cross-fill, integration test for thread id round-tripping.

Security ✅

No security concerns.

Documentation ✅

Added cross-turn continuity docs to the transport file.
Updated user guide docs.

Recommendation

Approve — well-engineered fix with thorough tests.

gmkseta · 2026-06-14T07:30:36Z

@teknium1 when you have a chance, could this get another quick look and be merged if it still looks good?

I rebased it onto current main and added the follow-up persistence path we discussed in the PR body: codex_thread_id now survives gateway restarts, and durable Discord/Telegram thread/topic lanes keep the Codex working thread across normal idle/daily Hermes transcript resets. Explicit fresh-start boundaries like /new, /reset, suspended recovery, and lane re-binding still clear it.

Relevant validation on the rebased branch is passing: 226 passed. Happy to adjust anything else you would prefer before merge.

…ession context The codex_app_server runtime is documented to keep "one Codex thread per Hermes session", but the gateway builds a fresh AIAgent per inbound message, so agent._codex_session was None at the start of every turn and ensure_started() issued a brand-new thread/start. A Codex thread's working context (files read, command output, plan state, long-running task state) lives inside the thread, not in Hermes' transcript (turn/start only sends the latest user message), so every follow-up message handed the model an empty thread and silently lost all prior work. Observed live: three distinct codex thread ids across three Discord messages in one session. Persist the codex thread id per session and resume it: - transport: CodexAppServerSession gains resume_thread_id; ensure_started() issues thread/resume {threadId} (reloads the on-disk rollout) instead of thread/start, falling back to thread/start on any failure (CodexAppServerError / TimeoutError / RuntimeError) so a turn is never lost. Thread-id extraction is shared by start + resume via _extract_thread_id(). - runtime: run_codex_app_server_turn passes the stored id into the session and records turn.thread_id after each turn (covers should_retire respawns within one process); a guard rebuilds a live session only if it is on a different thread than requested (never in normal operation, so codex is not respawned on every cache hit). - gateway: a per-session _session_codex_threads map (mirrors _session_model_overrides) keyed by the stable session_key so it survives compaction's session_id rotation; restored onto the fresh per-message agent, persisted from the turn result (skipped when the run generation is stale so a discarded turn can't repopulate the map after a reset), and cleared on /new, auto-reset, compression-exhaustion reset, and session expiry. Tests: TestThreadResume (resume vs start, graceful fallback incl. RuntimeError, id flows into turn/start + result), TestCodexThreadResumePersistence (id round-trips, cache-hit reuse vs mismatch rebuild). Docs: new "Cross-turn thread persistence" section. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

teknium1

Thanks for the well-scoped continuity fix. The underlying defect is still present on current main: CodexAppServerSession.ensure_started() unconditionally sends thread/start (agent/transports/codex_app_server_session.py:240-271), and the runtime constructs it with no resume target (agent/codex_runtime.py:392-400).

Problems

The PR's gateway/run.py:14220 and gateway/run.py:14696 use raw self.session_store inside async gateway code. Current main enforces the awaited self.async_session_store boundary to keep blocking SQLite/filesystem work off the event loop (tests/gateway/test_async_session_store.py:56-107).
The persistence layer has moved since this branch: state.db's gateway_routing table is primary and sessions.json is only an optional mirror (gateway/session.py:1043-1083, commit 94205a113). The runbook should not depend on sessions.json being present.

Suggested changes

Salvage the resume adapter/runtime logic, but port gateway persistence to await self.async_session_store and cover restart recovery with gateway.write_sessions_json: false.

Automated hermes-sweeper review.

teknium1 · 2026-07-14T06:30:20Z

+                )
+                if not _codex_resume_thread_id:
+                    try:
+                        _codex_resume_thread_id = self.session_store.get_codex_thread_id(


Current main requires async gateway code to use the awaited async_session_store facade; raw self.session_store calls are rejected by tests/gateway/test_async_session_store.py. Port this read to await self.async_session_store.get_codex_thread_id(session_key) during salvage.

teknium1 · 2026-07-14T06:30:20Z

+                if _codex_tid and hasattr(self, "_session_codex_threads"):
+                    self._session_codex_threads[session_key] = _codex_tid
+                    try:
+                        self.session_store.set_codex_thread_id(session_key, _codex_tid)


This synchronous routing-index write runs in an async gateway function. Use await self.async_session_store.set_codex_thread_id(...) on current main so the SQLite/filesystem write stays off the event loop.

Shaance · 2026-07-16T22:46:12Z

Confirmed the unresolved restart/cache-eviction path on current main (42bd436), including a gateway-hygiene trigger that evicts the live app-server session after native compaction is skipped/no-op. Full sanitized reproduction and source analysis: #41904 (comment)

kuki1029 · 2026-07-31T21:01:59Z

I independently reproduced and validated the deployment case this PR addresses on Ubuntu 24.04 with Codex CLI 0.146, Discord, and a systemd-managed gateway.

Persisting the Codex thread ID in session metadata, closing the original app-server process, then resuming that thread from a fresh process preserved an exact earlier-context recall. Without resume, a gateway restart or agent-cache eviction created a blank Codex thread and lost that context.

The maintainer request to use the async SessionStore matches what the deployment needs: the durable thread ID must be written before the in-memory agent disappears, then used to resume the next process. This would fix a real restart/eviction failure mode, not only an optimization.

TheAngryPit · 2026-08-10T19:02:20Z

I reviewed the restart/cache-eviction path against current main and Codex CLI 0.147.0. The July maintainer review still points to the smallest current fix: keep this PR as the ownership surface, move persistence to await self.async_session_store, and prove restart recovery with gateway.write_sessions_json: false.

Current main now gives us the native pieces needed for a narrow refresh:

SessionEntry.metadata already persists small routing values in state.db.gateway_routing;
AsyncSessionStore already offloads SessionStore calls from the event loop;
reset_session() creates a fresh entry without copying metadata, so /new and /reset naturally clear a stored Codex thread ID;
active_turn_token already owns gateway unclean-turn recovery.

That means this PR does not need a second lease, binding state machine, or thread/read recovery layer. I also confirmed from the Codex 0.147.0 source that a fresh app-server can normalize a persisted inProgress turn to interrupted when it has no live local handle. thread/read(includeTurns=true) therefore must not be used as a cross-process active-turn oracle. This does not block the clean restart/cache-eviction resume path in this PR.

I suggest shrinking the refresh to:

persist codex_thread_id through await self.async_session_store.set_session_metadata(...);
restore it through await self.async_session_store.get_session_metadata(...);
keep the adapter/runtime thread/resume plumbing;
cover restart recovery with gateway.write_sessions_json: false;
keep explicit reset fresh and leave automatic session-rotation continuity out of this patch.

@gmkseta, @teknium1: would you prefer that the existing PR branch be refreshed with this scope, or a small successor PR that preserves #41905's authorship and history? I can prepare the focused patch and tests after you choose the route.

alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery labels Jun 8, 2026

gmkseta force-pushed the fix/codex-app-server-thread-resume branch from 325aa3d to 6359c6e Compare June 8, 2026 11:41

tonydwb approved these changes Jun 8, 2026

View reviewed changes

gmkseta force-pushed the fix/codex-app-server-thread-resume branch 3 times, most recently from 42d8235 to e8b595e Compare June 14, 2026 07:18

gmkseta force-pushed the fix/codex-app-server-thread-resume branch from e8b595e to b2c19da Compare June 14, 2026 07:53

gmkseta and others added 2 commits June 14, 2026 16:56

fix(gateway): persist codex thread ids across restarts

7bb6a5c

gmkseta force-pushed the fix/codex-app-server-thread-resume branch from b2c19da to 7bb6a5c Compare June 14, 2026 07:57

This was referenced Jun 19, 2026

fix(codex): persist app-server turns to session DB (fixes starved recall) #49225

Closed

fix(gateway): resume codex app-server threads #49643

Closed

teknium1 mentioned this pull request Jul 14, 2026

Codex app-server runtime loses thread context across turns (gateway starts a fresh thread each message) #41904

Closed

teknium1 reviewed Jul 14, 2026

View reviewed changes

alt-glitch mentioned this pull request Jul 16, 2026

Persist Codex threads across Hermes sessions #65872

Closed

teknium1 added the area/sessions Session lifecycle, resume, persistence, history label Jul 19, 2026

GottZ mentioned this pull request Jul 26, 2026

[Bug]: codex_app_server Kanban workers cannot write artifacts to HERMES_KANBAN_WORKSPACE when it differs from cwd #27941

Open

1 task

kuki1029 mentioned this pull request Jul 31, 2026

fix(codex): normalize projected tool names #75658

Open

alt-glitch added the provider/openai OpenAI / Codex Responses API label Jul 31, 2026

alt-glitch mentioned this pull request Aug 2, 2026

fix(codex): resume app-server threads and stop double-sending the final reply #76798

Closed

GottZ mentioned this pull request Aug 3, 2026

[Bug]: codex_app_server Kanban worktree workers cannot install dependencies or write Git metadata headlessly #63351

Open

leonagente mentioned this pull request Aug 13, 2026

feat(agent): add model-agnostic session compiler and native lifecycle #85227

Closed

rodrigoscoelho mentioned this pull request Aug 31, 2026

fix(codex): harden app-server turn continuity #98995

Open

19 tasks

alt-glitch mentioned this pull request Sep 5, 2026

fix(codex): persist session bindings and refuse silent conversation resets #103352

Open

Navigation Menu

Uh oh!

fix(agent): resume codex app-server thread across turns to preserve session context - #41905

fix(agent): resume codex app-server thread across turns to preserve session context#41905
gmkseta wants to merge 2 commits into
NousResearch:mainfrom
gmkseta:fix/codex-app-server-thread-resume

gmkseta commented Jun 8, 2026 •

edited

Loading

Uh oh!

gmkseta commented Jun 8, 2026

Uh oh!

tonydwb left a comment

Uh oh!

gmkseta commented Jun 14, 2026

Uh oh!

teknium1 left a comment

Uh oh!

teknium1 Jul 14, 2026

Uh oh!

teknium1 Jul 14, 2026

Uh oh!

Shaance commented Jul 16, 2026

Uh oh!

kuki1029 commented Jul 31, 2026

Uh oh!

TheAngryPit commented Aug 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

gmkseta commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Related Issue

Type of Change

Changes Made

How to Test

Operational Notes

Platforms tested

Checklist

Uh oh!

gmkseta commented Jun 8, 2026

Live verification against real Codex 0.133

Uh oh!

tonydwb left a comment

Choose a reason for hiding this comment

Code Review Summary

Analysis

Recommendation

Uh oh!

gmkseta commented Jun 14, 2026

Uh oh!

teknium1 left a comment

Choose a reason for hiding this comment

Problems

Suggested changes

Uh oh!

teknium1 Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

teknium1 Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

Shaance commented Jul 16, 2026

Uh oh!

kuki1029 commented Jul 31, 2026

Uh oh!

TheAngryPit commented Aug 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

gmkseta commented Jun 8, 2026 •

edited

Loading