fix(gateway): rebuild cached agent when a session grows in another process

[aldoeliacim]

Fixes #45966

What & why

The gateway caches an AIAgent per session for prompt-prefix caching. The cached agent carries an in-memory replay of the transcript as it existed when the agent was built. When a second process that shares the same HERMES_HOME appends turns to the same session in the shared SessionDB, the gateway's cached agent never re-reads from disk, so the next platform turn replies with stale context and the on-disk transcript diverges from what the live agent reasons over (split-brain).

Concrete trigger: the desktop app spawns a hermes dashboard backend (a separate process from hermes gateway run). Both share the session DB, but each keeps its own _agent_cache. Reply to a platform-origin session from the desktop, then send another message from the platform — the gateway answers as if the desktop turn never happened, and the recorded transcript no longer matches the context the gateway used.

The existing eviction triggers (/reset, /model, context compression, auto-reset) all fire on same-process mutations via _evict_cached_agent. There was no signal for external (cross-process) transcript growth.

The fix

Snapshot the session's on-disk message count next to the cached agent, and on a cache-hit compare it against the live count:

• Cache entry widens (agent, sig) → (agent, sig, msg_count). All existing readers use index access (cached[0]/cached[1], isinstance(x, tuple)) and tolerate a legacy 2-tuple, so in-flight caches survive the rollout (snapshot=None → check skipped → reuse).
• On hit: if live_count > snapshot, evict (releasing the stale agent's client pool) and fall through to the existing rebuild path, which reads the current transcript from disk. Otherwise reuse exactly as before — the unchanged case is untouched, so prompt caching is preserved.
• The count comes from SessionDB.message_count(session_id) (an indexed COUNT). A new _session_transcript_len helper wraps it and fails safe: missing DB / missing session_id / any probe error returns None, which degrades to the current reuse-the-cached-agent behavior — it never crashes a turn and never falsely evicts on a transient DB hiccup.

How to test

Repro (no fix): run a gateway and a hermes dashboard against the same HERMES_HOME, open a platform session, reply to it through the dashboard's /api/ws, then send another platform message — the reply ignores the dashboard-added turn.

Automated:

python -m pytest tests/gateway/test_agent_cache.py -q

Adds TestAgentCacheCrossProcessCoherence (4 tests, real temp SessionDB):

• test_transcript_len_reads_live_count — probe reflects external appends immediately.
• test_transcript_len_fails_safe — no DB / falsy session_id / raising probe → None, never raises.
• test_external_growth_invalidates_cache_reuse — the reuse decision flips True→False after an external append; asserts the unchanged case still reuses (prompt-cache preserved).
• test_legacy_two_tuple_entry_reuses — a pre-existing 2-tuple entry skips the check and is never falsely evicted.

Verified the whole gateway cache/session/eviction surface stays green (test_agent_cache.py 69, plus test_fallback_eviction.py, test_session.py, test_session_boundary_hooks.py, test_session_reset_notify.py, test_session_model_reset.py, test_model_switch_persistence.py — 186 together), and the adjacent test_run_progress_topics.py / test_telegram_* files (which exercise the no-cache dispatch path) pass — that path is what caught an early version that referenced an unbound cached var outside the cache block, now fixed by an explicit _stale_cache_entry flag.

Platforms

Developed and tested on Linux (Python 3.11). No platform-specific code; the change is in gateway/run.py's session-agnostic cache path.

[liuhao1024]

Verification: LGTM ✅

Reviewed the full diff and test suite. The cross-process cache coherence mechanism is well-designed:

1. Deadlock avoidance: The stale entry is flagged inside _agent_cache_lock but _evict_cached_agent() is called OUTSIDE the lock — correct, since _evict_cached_agent re-acquires the same lock.

2. Fail-safe design: _session_transcript_len() returns None on any error (missing DB, exception, empty session_id), and the cache-hit guard treats None as "skip the check" (legacy behavior). This means a DB hiccup degrades to existing cache-reuse rather than crashing a turn.

3. Legacy 2-tuple compatibility: The len(cached) > 2 guard correctly handles pre-existing 2-tuple cache entries from before this change — they skip the coherence check and reuse as before, preventing false evictions during rollout.

4. Test coverage: 4 tests covering live count, fail-safe paths, external growth invalidation, and legacy tuple compatibility. The _BoomDB test specifically verifies that DB exceptions are swallowed (fail-safe, not fail-closed).

5. Performance: message_count() is described as an indexed COUNT — this is O(1) on SQLite with the right index, so the per-turn overhead is a single indexed lookup. Acceptable for a correctness fix.

[liuhao1024]

Verification: Reviewed the full diff — clean concurrency fix for split-brain agent cache. When another process sharing HERMES_HOME (e.g., the desktop dashboard backend) appends turns to the same session's transcript in the shared SessionDB, the gateway's cached agent would reply with stale context. The fix stores a message_count snapshot alongside the cached agent (as the 3rd tuple element) and compares it with a cheap indexed COUNT on each cache hit. Key design choices verified: (1) legacy 2-tuple entries gracefully skip the check — no false evictions during rollout; (2) eviction happens OUTSIDE _agent_cache_lock to avoid deadlock since _evict_cached_agent re-acquires the same lock; (3) _session_transcript_len returns None on any error (fail-safe, degrades to reuse); (4) _stale_cache_entry flag is set inside the lock but acted on outside it. Three tests cover the happy path, the legacy tuple path, and the error-swallowing path. No findings.

[kshitijk4poor]

Thanks for the sharp diagnosis here @aldoeliacim — the split-brain write-up in #45966 was spot-on and drove the fix.

We've landed the resolution in #46237 (merged as 7f245b0 + 3bc4a2f, on main via 6c34088), so I'm closing this as superseded. Two notes on why that PR rather than this one:

• Both approaches snapshot the session's on-disk message_count next to the cached agent and rebuild on a cache-hit mismatch — equivalent at the core.
• Both, however, share the same subtle flaw: the snapshot is taken at agent-build time (before the turn writes its own rows) and is never refreshed on reuse. So the gateway's own turn grows message_count, and the next turn sees a mismatch and rebuilds the agent — every turn, for every conversation — quietly destroying per-conversation prompt caching. (Your live_len > cached_len and the merged PR's != both trip on this, since the same-process count always grows.)

#46237 adds a post-turn re-baseline (_refresh_agent_cache_message_count) that resets the snapshot to the live count after each turn, so the guard fires only on a genuine cross-process write — preserving the cache for normal single-process operation. There's a regression test driving the real SessionDB that proves 5 consecutive same-process turns all reuse the cached agent (0 reuses without the re-baseline).

Really appreciate the clear repro and issue — it made this an easy fix to get right. 🙏