fix(state): purge gateway_routing entries when prune_sessions hard-deletes - #59512
fix(state): purge gateway_routing entries when prune_sessions hard-deletes#59512pierrenode wants to merge 1 commit into
Conversation
|
Thanks for tracing a real consistency gap: current Problems
Suggested changes
Automated hermes-sweeper review. |
245431a to
04903b9
Compare
…letes prune_sessions() (hermes sessions prune) deletes sessions/messages rows and on-disk transcript files directly in hermes_state.py, bypassing gateway/session.py's SessionStore entirely. It never touched the gateway_routing table (NousResearch#59203), so a routing entry (session_key -> full serialized SessionEntry) written before the prune stayed dangling, pointing at a session_id whose row no longer existed. gateway/session.py's own stale-entry self-heal (_is_session_ended_in_db) does not catch this: its own docstring says a missing row returns False ("keep") — that check exists for a different case (an entry not yet persisted), not "the row was deleted out from under it". The next message on that session_key would rehydrate the stale entry, load an empty transcript via load_transcript(), and ensure_session()'s INSERT OR IGNORE would silently recreate a blank sessions row — the user sees the conversation lose all memory instead of either continuing or starting cleanly fresh. Purging gateway_routing alone leaves two more ways for the stale mapping to resurrect itself: 1. The legacy sessions.json mirror (on by default) still has the entry. sessions.json fills any routing key the DB table doesn't have on the next SessionStore load, so a restarted gateway would silently re-import and re-persist exactly what was just purged. Now purged in the same pass, atomically written like the live SessionStore's own mirror writes. 2. A gateway process already running when the CLI prune executes has the stale entry cached in memory. Its per-request self-heal previously only checked whether the session was *ended* in state.db (_is_session_ended_in_db); a hard-deleted row reads identically to "not yet persisted" from that check alone, so it could never catch this case without also breaking the legitimate not-yet-persisted race. Added a second, narrower check (_is_routing_entry_purged) that asks the routing table directly whether it still carries the entry — a "no" is unambiguous once legacy sessions.json imports are persisted immediately on load (also fixed here) rather than left to the next incidental save. This is the same failure class the gateway/session.py self-heal was built for (NousResearch#54878, NousResearch#52804: "stale sessions.json entry silently drops messages until restart") applied to the new gateway_routing table, exposed further by NousResearch#59327's --newer-than filters, which make it realistic to prune sessions from just hours ago rather than only 90-day-old ones.
04903b9 to
6332226
Compare
|
Rebased onto current `upstream/main`. `gateway/session.py`'s stale-entry check was restructured into an explicit lock-read/no-lock-I/O/lock-write phase split since this PR's branch point — merged this PR's `_purged_by_prune` check into that structure (computed in the no-lock I/O phase alongside `_is_stale`, consistent with the refactor's own stated goal of keeping I/O out of the lock). `hermes_state.py`'s conflict was two independent, non-overlapping additions at the same insertion point — kept both. The test-file conflict included a fragment of a pre-existing test (`test_force_new_skips_stale_check`) that turned out to belong to a class upstream has since restructured/renamed elsewhere in the same file — verified upstream's current `TestRuntimeStaleGuard` no longer ends with that test, so it wasn't something this PR removed; kept only this PR's own new test classes. Mutation-verified: removing `_purged_by_prune` from the stale-entry condition breaks `test_purged_entry_creates_fresh_session`. Full `test_session_store_runtime_stale_guard.py` + `test_hermes_state.py` (245 tests) passes. Ruff clean. (A broader `tests/gateway/test_session*.py tests/state/` sweep showed one unrelated failure — confirmed pre-existing test-order pollution by reproducing the identical failure against clean pre-fix `upstream/main` code with the same broad file selection.) Squashed to a single commit on top of current `upstream/main`. |
What does this PR do?
prune_sessions()(hermes sessions prune) deletessessions/messagesrows and on-disk transcript files directly inhermes_state.py, bypassinggateway/session.py'sSessionStoreentirely. It never touched thegateway_routingtable (#59203), so a routing entry (session_key-> full serializedSessionEntry) written before the prune stayed dangling, pointing at asession_idwhose row no longer existed.gateway/session.py's own stale-entry self-heal (_is_session_ended_in_db) does not catch this: its own docstring says a missing row returnsFalse("keep") — that check exists for a different case (an entry not yet persisted), not "the row was deleted out from under it". The next message on thatsession_keywould rehydrate the stale entry, load an empty transcript viaload_transcript(), andensure_session()'sINSERT OR IGNOREwould silently recreate a blanksessionsrow — the user sees the conversation lose all memory instead of either continuing normally or starting cleanly fresh.This is the same failure class the
gateway/session.pyself-heal was originally built for (#54878, #52804: "stale sessions.json entry silently drops messages until restart") — now reappearing against the newgateway_routingtable, and made more realistic to hit by #59327's--newer-thanfilters, which let an operator prune sessions from just hours ago instead of only 90-day-old ones.Related Issue
No filed issue — found via cross-PR review of #59203 (gateway_routing table) and #59327 (expanded prune filters), both merged in the same window, neither cross-referencing the other's effect on this table.
Type of Change
Changes Made
hermes_state.py:prune_sessions()now calls a new_purge_gateway_routing_for_sessions()helper after deleting session rows, which scansgateway_routing(scoped tosessions_dir, mirroringSessionStore._routing_scope()) for entries whose embeddedsession_idwas just deleted, and drops them via the existing (previously unused)delete_gateway_routing_entries(). No-op whensessions_diris not passed (can't compute scope) or when nothing was pruned — same graceful-degradation convention_remove_session_filesalready uses.tests/test_hermes_state.py: two new regression tests —test_prune_sessions_drops_stale_gateway_routing_entries(verified: fails without the fix, passes with it) andtest_prune_sessions_without_sessions_dir_skips_routing_cleanup(no-sessions_dircase doesn't crash or touch other scopes)How to Test
Also re-ran the full file and the gateway session-routing suite to confirm no regressions:
tests/test_hermes_state.py(324 passed),tests/gateway/test_session.py(100 passed).Checklist