fix(gateway): set ended_at on expired gateway sessions to bound state.db growth - #56603
adityachaudhary99 wants to merge 1 commit into
Conversation
….db growth Expired gateway sessions were finalized in _session_expiry_watcher by closing the cached agent, but an idle session's agent is almost always soft-evicted (which preserves the row for resume, never calling end_session) before its reset policy expires. So the DB row kept ended_at IS NULL forever — the unbounded state.db growth in NousResearch#54189 (659MB / 938 never-ended sessions). Call the existing end_session(session_id, "session_expired") helper at the finalization point, independent of whether a cached agent still exists (first-reason-wins, no-op if already ended). cron/subagent/one-shot/interactive paths already set ended_at. A retention/prune policy for the now-eligible rows is left as a follow-up. Fixes NousResearch#54189 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
9fa9b6c to
6550d99
Compare
|
Current-main coordination update (no duplicate PR opened): I rebased the bounded expiry closeout onto current Verification on the clean current-main branch: 379 passed ( |
teknium1
left a comment
There was a problem hiding this comment.
Thanks for isolating a real lifecycle gap: current main's expiry watcher persists expiry_finalized at gateway/run.py:7843, but does not set the session row's ended_at.
Problems
- The proposed call at
gateway/run.py:7707invokes synchronoussession_store._db.end_session()on the async gateway loop. Current main routes loop-side database work throughAsyncSessionDB(hermes_state.py:7196-7210), andtests/gateway/test_async_session_db.py:315-344enforces that boundary. - The regression test targets the older synchronous watcher setup. Current main first awaits
async_session_storeatgateway/run.py:7753; the test needs the current async store/database facades to exercise the live path.
Suggested changes
- Salvage the closeout through awaited
self._session_db.end_session(entry.session_id, "session_expired"), preserving best-effort failure handling beforeset_expiry_finalized. - Update the test to wrap its temporary
SessionDBinAsyncSessionDBand configure the async session-store path.
Automated hermes-sweeper review.
| _sess_db = getattr(self.session_store, "_db", None) | ||
| if _sess_db is not None and entry.session_id: | ||
| try: | ||
| _sess_db.end_session( |
There was a problem hiding this comment.
This runs inside the async gateway watcher, but _sess_db is the synchronous SessionDB. Current main requires loop-side database calls to go through the awaited AsyncSessionDB facade (await self._session_db.end_session(...)) so SQLite writes are offloaded rather than blocking the gateway event loop.
|
Already fixed on main: |
What does this PR do?
Fixes unbounded
state.dbgrowth (659MB / 938 never-ended sessions in ~2 weeks). Expired gateway sessions were finalized in_session_expiry_watcherby closing the cached agent — but an idle session's agent is almost always soft-evicted first (cache eviction deliberately preserves the row for resume and never callsend_session). So when the expiry watcher runs there's no cached agent to close, and the DB row keepsended_at IS NULLforever, accumulating without bound.This calls the existing
SessionDB.end_session(session_id, "session_expired")at the finalization point in_session_expiry_watcher, independent of whether a cached agent still exists. It's first-reason-wins and no-ops on an already-ended row, so it's safe alongsideagent_close. I verified the cron / sync-subagent / async-subagent / one-shot / interactive-CLI paths already setended_at, so the expiry watcher was the one real gap.Scope is intentionally bounded to closing the lifecycle gap. A retention/prune policy for the now-eligible rows (there's already a
maybe_auto_prune_and_vacuumhook) is left as a follow-up to avoid a design debate here.Related Issue
Fixes #54189
Type of Change
Changes Made
gateway/run.py: in_session_expiry_watcher's per-entry finalization, callend_session(entry.session_id, "session_expired")(best-effort, logged on failure).How to Test
state.dbsession row keepsended_at IS NULL.ended_atset withend_reason = "session_expired".Added
test_idle_expiry_sets_ended_at_in_dbintests/gateway/test_session_boundary_hooks.py(realSessionDB, empty agent cache to reproduce the leak, assertsended_atis set after the watcher runs).