fix(gateway): notify user when stale-routing self-heal can't recover the old session - #59597
Open
hejuntt1014 wants to merge 1 commit into
Open
hejuntt1014 wants to merge 1 commit into
hejuntt1014 wants to merge 1 commit into
Conversation
alt-glitch
added
type/bug
jcjc81
added a commit
to jcjc81/hermes-agent
that referenced
this pull request
Jul 14, 2026
…ead sessions The NousResearch#54878 self-heal (SessionStore.get_or_create_session) drops a routing key pointing at a session already ended in state.db and recovers/recreates a fresh session_id under the same session_key. The NousResearch#54947 fix (agent-cache cache-hit guard in gateway/run.py) treats a cached agent whose snapshot session_id differs from the current session_id, under the same session_key, as an intentional /resume-/branch-style switch between two live sibling conversations, and reuses it unchanged to protect the prompt cache. These two fixes compose incorrectly: when the NousResearch#54878 self-heal just fired, the cached agent's session_id is not a live sibling — it's the dead session just routed away from. NousResearch#54947's "different session_id -> reuse freely" rule reuses it anyway. The stale agent runs the turn, and the post-run "session split" sync (agent.session_id != session_id) then writes the routing key straight back onto the dead session_id, undoing the self-heal. This repeats on every subsequent message until an interrupt (e.g. /stop) happens to race in before that post-run sync, silently discarding conversation context. Reproduced live on the engineering gateway (2026-07-12, routing key agent:main:telegram:dm:170829464:544520): 5 consecutive self-heal log lines over ~40 minutes, each followed by the dead session_id being reused and re-synced back, until an interrupted /stop finally let a fresh session stick — at which point all prior context was gone. No open upstream issue tracks this specific interaction as of 2026-07-12 (checked NousResearch#54878, NousResearch#54947, NousResearch#59580, NousResearch#59597, NousResearch#61220 — all cover adjacent but distinct edges of the self-heal / agent-cache system). Fix: before applying NousResearch#54947's reuse-on-mismatch rule, check (outside the cache lock, via SessionStore._is_session_ended_in_db) whether the cached snapshot's session_id is itself ended in state.db. If so, treat it as a stale self-heal artifact and evict/rebuild fresh -- same as a genuine cross-process write -- instead of reusing it. Re-validates the peeked verdict against the tuple actually held under the lock so a race can't apply a stale verdict to a different (possibly live) cache entry. Tests: tests/gateway/test_stale_self_heal_agent_cache_eviction.py (5 new cases: dead-session eviction, live-sibling reuse preserved [NousResearch#54947 intact], cross-process invalidation preserved [NousResearch#45966 intact], same-session_id dead edge case, lock-race re-validation). Full tests/gateway/ suite: 14 failed, 9040 passed, 11 skipped -- all 14 failures verified pre-existing on unpatched main (confirmed via git stash + re-run), unrelated to this change. (cherry picked from commit cb055f54850b30f1e956a87c4ddbfd2560d51f8b)
jcjc81
added a commit
to jcjc81/hermes-agent
that referenced
this pull request
Jul 15, 2026
…ead sessions The NousResearch#54878 self-heal (SessionStore.get_or_create_session) drops a routing key pointing at a session already ended in state.db and recovers/recreates a fresh session_id under the same session_key. The NousResearch#54947 fix (agent-cache cache-hit guard in gateway/run.py) treats a cached agent whose snapshot session_id differs from the current session_id, under the same session_key, as an intentional /resume-/branch-style switch between two live sibling conversations, and reuses it unchanged to protect the prompt cache. These two fixes compose incorrectly: when the NousResearch#54878 self-heal just fired, the cached agent's session_id is not a live sibling — it's the dead session just routed away from. NousResearch#54947's "different session_id -> reuse freely" rule reuses it anyway. The stale agent runs the turn, and the post-run "session split" sync (agent.session_id != session_id) then writes the routing key straight back onto the dead session_id, undoing the self-heal. This repeats on every subsequent message until an interrupt (e.g. /stop) happens to race in before that post-run sync, silently discarding conversation context. Reproduced live on the engineering gateway (2026-07-12, routing key agent:main:telegram:dm:170829464:544520): 5 consecutive self-heal log lines over ~40 minutes, each followed by the dead session_id being reused and re-synced back, until an interrupted /stop finally let a fresh session stick — at which point all prior context was gone. No open upstream issue tracks this specific interaction as of 2026-07-12 (checked NousResearch#54878, NousResearch#54947, NousResearch#59580, NousResearch#59597, NousResearch#61220 — all cover adjacent but distinct edges of the self-heal / agent-cache system). Fix: before applying NousResearch#54947's reuse-on-mismatch rule, check (outside the cache lock, via SessionStore._is_session_ended_in_db) whether the cached snapshot's session_id is itself ended in state.db. If so, treat it as a stale self-heal artifact and evict/rebuild fresh -- same as a genuine cross-process write -- instead of reusing it. Re-validates the peeked verdict against the tuple actually held under the lock so a race can't apply a stale verdict to a different (possibly live) cache entry. Tests: tests/gateway/test_stale_self_heal_agent_cache_eviction.py (5 new cases: dead-session eviction, live-sibling reuse preserved [NousResearch#54947 intact], cross-process invalidation preserved [NousResearch#45966 intact], same-session_id dead edge case, lock-race re-validation). Full tests/gateway/ suite: 14 failed, 9040 passed, 11 skipped -- all 14 failures verified pre-existing on unpatched main (confirmed via git stash + re-run), unrelated to this change.
4 tasks
kshitijk4poor
pushed a commit
that referenced
this pull request
Jul 15, 2026
…ead sessions The #54878 self-heal (SessionStore.get_or_create_session) drops a routing key pointing at a session already ended in state.db and recovers/recreates a fresh session_id under the same session_key. The #54947 fix (agent-cache cache-hit guard in gateway/run.py) treats a cached agent whose snapshot session_id differs from the current session_id, under the same session_key, as an intentional /resume-/branch-style switch between two live sibling conversations, and reuses it unchanged to protect the prompt cache. These two fixes compose incorrectly: when the #54878 self-heal just fired, the cached agent's session_id is not a live sibling — it's the dead session just routed away from. #54947's "different session_id -> reuse freely" rule reuses it anyway. The stale agent runs the turn, and the post-run "session split" sync (agent.session_id != session_id) then writes the routing key straight back onto the dead session_id, undoing the self-heal. This repeats on every subsequent message until an interrupt (e.g. /stop) happens to race in before that post-run sync, silently discarding conversation context. Reproduced live on the engineering gateway (2026-07-12, routing key agent:main:telegram:dm:170829464:544520): 5 consecutive self-heal log lines over ~40 minutes, each followed by the dead session_id being reused and re-synced back, until an interrupted /stop finally let a fresh session stick — at which point all prior context was gone. No open upstream issue tracks this specific interaction as of 2026-07-12 (checked #54878, #54947, #59580, #59597, #61220 — all cover adjacent but distinct edges of the self-heal / agent-cache system). Fix: before applying #54947's reuse-on-mismatch rule, check (outside the cache lock, via SessionStore._is_session_ended_in_db) whether the cached snapshot's session_id is itself ended in state.db. If so, treat it as a stale self-heal artifact and evict/rebuild fresh -- same as a genuine cross-process write -- instead of reusing it. Re-validates the peeked verdict against the tuple actually held under the lock so a race can't apply a stale verdict to a different (possibly live) cache entry. Tests: tests/gateway/test_stale_self_heal_agent_cache_eviction.py (5 new cases: dead-session eviction, live-sibling reuse preserved [#54947 intact], cross-process invalidation preserved [#45966 intact], same-session_id dead edge case, lock-race re-validation). Full tests/gateway/ suite: 14 failed, 9040 passed, 11 skipped -- all 14 failures verified pre-existing on unpatched main (confirmed via git stash + re-run), unrelated to this change.
Collaborator
|
Thanks for isolating the notification-only half of this failure mode. The premise remains valid on current main: Problems
Suggested changes
Automated hermes-sweeper review. |
teknium1
added
the
sweeper:blast-moderate
Port NousResearch#59597 onto current main's refactored SessionStore: when the NousResearch#54878 self-heal drops an ended routing entry and recovery declines to reopen it (e.g. tui_shutdown), mark the fresh session as stale_routing_recovered so GatewayRunner always notifies instead of silently replacing the thread. Co-authored-by: Cursor <cursoragent@cursor.com>
hejuntt1014
force-pushed
the
fix/stale-routing-notify
branch
from
July 19, 2026 14:37
078b059 to
497426e
Compare
Contributor
Author
|
Reimplemented on current Changes
|
Gravezzz
pushed a commit
to Gravezzz/hermes-agent
that referenced
this pull request
Jul 21, 2026
…ead sessions The NousResearch#54878 self-heal (SessionStore.get_or_create_session) drops a routing key pointing at a session already ended in state.db and recovers/recreates a fresh session_id under the same session_key. The NousResearch#54947 fix (agent-cache cache-hit guard in gateway/run.py) treats a cached agent whose snapshot session_id differs from the current session_id, under the same session_key, as an intentional /resume-/branch-style switch between two live sibling conversations, and reuses it unchanged to protect the prompt cache. These two fixes compose incorrectly: when the NousResearch#54878 self-heal just fired, the cached agent's session_id is not a live sibling — it's the dead session just routed away from. NousResearch#54947's "different session_id -> reuse freely" rule reuses it anyway. The stale agent runs the turn, and the post-run "session split" sync (agent.session_id != session_id) then writes the routing key straight back onto the dead session_id, undoing the self-heal. This repeats on every subsequent message until an interrupt (e.g. /stop) happens to race in before that post-run sync, silently discarding conversation context. Reproduced live on the engineering gateway (2026-07-12, routing key agent:main:telegram:dm:170829464:544520): 5 consecutive self-heal log lines over ~40 minutes, each followed by the dead session_id being reused and re-synced back, until an interrupted /stop finally let a fresh session stick — at which point all prior context was gone. No open upstream issue tracks this specific interaction as of 2026-07-12 (checked NousResearch#54878, NousResearch#54947, NousResearch#59580, NousResearch#59597, NousResearch#61220 — all cover adjacent but distinct edges of the self-heal / agent-cache system). Fix: before applying NousResearch#54947's reuse-on-mismatch rule, check (outside the cache lock, via SessionStore._is_session_ended_in_db) whether the cached snapshot's session_id is itself ended in state.db. If so, treat it as a stale self-heal artifact and evict/rebuild fresh -- same as a genuine cross-process write -- instead of reusing it. Re-validates the peeked verdict against the tuple actually held under the lock so a race can't apply a stale verdict to a different (possibly live) cache entry. Tests: tests/gateway/test_stale_self_heal_agent_cache_eviction.py (5 new cases: dead-session eviction, live-sibling reuse preserved [NousResearch#54947 intact], cross-process invalidation preserved [NousResearch#45966 intact], same-session_id dead edge case, lock-race re-validation). Full tests/gateway/ suite: 14 failed, 9040 passed, 11 skipped -- all 14 failures verified pre-existing on unpatched main (confirmed via git stash + re-run), unrelated to this change.
randlee
pushed a commit
to randlee/hermes-agent
that referenced
this pull request
Aug 11, 2026
…ead sessions The NousResearch#54878 self-heal (SessionStore.get_or_create_session) drops a routing key pointing at a session already ended in state.db and recovers/recreates a fresh session_id under the same session_key. The NousResearch#54947 fix (agent-cache cache-hit guard in gateway/run.py) treats a cached agent whose snapshot session_id differs from the current session_id, under the same session_key, as an intentional /resume-/branch-style switch between two live sibling conversations, and reuses it unchanged to protect the prompt cache. These two fixes compose incorrectly: when the NousResearch#54878 self-heal just fired, the cached agent's session_id is not a live sibling — it's the dead session just routed away from. NousResearch#54947's "different session_id -> reuse freely" rule reuses it anyway. The stale agent runs the turn, and the post-run "session split" sync (agent.session_id != session_id) then writes the routing key straight back onto the dead session_id, undoing the self-heal. This repeats on every subsequent message until an interrupt (e.g. /stop) happens to race in before that post-run sync, silently discarding conversation context. Reproduced live on the engineering gateway (2026-07-12, routing key agent:main:telegram:dm:170829464:544520): 5 consecutive self-heal log lines over ~40 minutes, each followed by the dead session_id being reused and re-synced back, until an interrupted /stop finally let a fresh session stick — at which point all prior context was gone. No open upstream issue tracks this specific interaction as of 2026-07-12 (checked NousResearch#54878, NousResearch#54947, NousResearch#59580, NousResearch#59597, NousResearch#61220 — all cover adjacent but distinct edges of the self-heal / agent-cache system). Fix: before applying NousResearch#54947's reuse-on-mismatch rule, check (outside the cache lock, via SessionStore._is_session_ended_in_db) whether the cached snapshot's session_id is itself ended in state.db. If so, treat it as a stale self-heal artifact and evict/rebuild fresh -- same as a genuine cross-process write -- instead of reusing it. Re-validates the peeked verdict against the tuple actually held under the lock so a race can't apply a stale verdict to a different (possibly live) cache entry. Tests: tests/gateway/test_stale_self_heal_agent_cache_eviction.py (5 new cases: dead-session eviction, live-sibling reuse preserved [NousResearch#54947 intact], cross-process invalidation preserved [NousResearch#45966 intact], same-session_id dead edge case, lock-race re-validation). Full tests/gateway/ suite: 14 failed, 9040 passed, 11 skipped -- all 14 failures verified pre-existing on unpatched main (confirmed via git stash + re-run), unrelated to this change.
13 tasks
prmartinow
pushed a commit
to prmartinow/hermes-agent
that referenced
this pull request
Aug 26, 2026
…ead sessions The NousResearch#54878 self-heal (SessionStore.get_or_create_session) drops a routing key pointing at a session already ended in state.db and recovers/recreates a fresh session_id under the same session_key. The NousResearch#54947 fix (agent-cache cache-hit guard in gateway/run.py) treats a cached agent whose snapshot session_id differs from the current session_id, under the same session_key, as an intentional /resume-/branch-style switch between two live sibling conversations, and reuses it unchanged to protect the prompt cache. These two fixes compose incorrectly: when the NousResearch#54878 self-heal just fired, the cached agent's session_id is not a live sibling — it's the dead session just routed away from. NousResearch#54947's "different session_id -> reuse freely" rule reuses it anyway. The stale agent runs the turn, and the post-run "session split" sync (agent.session_id != session_id) then writes the routing key straight back onto the dead session_id, undoing the self-heal. This repeats on every subsequent message until an interrupt (e.g. /stop) happens to race in before that post-run sync, silently discarding conversation context. Reproduced live on the engineering gateway (2026-07-12, routing key agent:main:telegram:dm:170829464:544520): 5 consecutive self-heal log lines over ~40 minutes, each followed by the dead session_id being reused and re-synced back, until an interrupted /stop finally let a fresh session stick — at which point all prior context was gone. No open upstream issue tracks this specific interaction as of 2026-07-12 (checked NousResearch#54878, NousResearch#54947, NousResearch#59580, NousResearch#59597, NousResearch#61220 — all cover adjacent but distinct edges of the self-heal / agent-cache system). Fix: before applying NousResearch#54947's reuse-on-mismatch rule, check (outside the cache lock, via SessionStore._is_session_ended_in_db) whether the cached snapshot's session_id is itself ended in state.db. If so, treat it as a stale self-heal artifact and evict/rebuild fresh -- same as a genuine cross-process write -- instead of reusing it. Re-validates the peeked verdict against the tuple actually held under the lock so a race can't apply a stale verdict to a different (possibly live) cache entry. Tests: tests/gateway/test_stale_self_heal_agent_cache_eviction.py (5 new cases: dead-session eviction, live-sibling reuse preserved [NousResearch#54947 intact], cross-process invalidation preserved [NousResearch#45966 intact], same-session_id dead edge case, lock-race re-validation). Full tests/gateway/ suite: 14 failed, 9040 passed, 11 skipped -- all 14 failures verified pre-existing on unpatched main (confirmed via git stash + re-run), unrelated to this change.
melon-xf
added a commit
to melon-xf/hermes-agent
that referenced
this pull request
Sep 3, 2026
…ead sessions The NousResearch#54878 self-heal (SessionStore.get_or_create_session) drops a routing key pointing at a session already ended in state.db and recovers/recreates a fresh session_id under the same session_key. The NousResearch#54947 fix (agent-cache cache-hit guard in gateway/run.py) treats a cached agent whose snapshot session_id differs from the current session_id, under the same session_key, as an intentional /resume-/branch-style switch between two live sibling conversations, and reuses it unchanged to protect the prompt cache. These two fixes compose incorrectly: when the NousResearch#54878 self-heal just fired, the cached agent's session_id is not a live sibling — it's the dead session just routed away from. NousResearch#54947's "different session_id -> reuse freely" rule reuses it anyway. The stale agent runs the turn, and the post-run "session split" sync (agent.session_id != session_id) then writes the routing key straight back onto the dead session_id, undoing the self-heal. This repeats on every subsequent message until an interrupt (e.g. /stop) happens to race in before that post-run sync, silently discarding conversation context. Reproduced live on the engineering gateway (2026-07-12, routing key agent:main:telegram:dm:170829464:544520): 5 consecutive self-heal log lines over ~40 minutes, each followed by the dead session_id being reused and re-synced back, until an interrupted /stop finally let a fresh session stick — at which point all prior context was gone. No open upstream issue tracks this specific interaction as of 2026-07-12 (checked NousResearch#54878, NousResearch#54947, NousResearch#59580, NousResearch#59597, NousResearch#61220 — all cover adjacent but distinct edges of the self-heal / agent-cache system). Fix: before applying NousResearch#54947's reuse-on-mismatch rule, check (outside the cache lock, via SessionStore._is_session_ended_in_db) whether the cached snapshot's session_id is itself ended in state.db. If so, treat it as a stale self-heal artifact and evict/rebuild fresh -- same as a genuine cross-process write -- instead of reusing it. Re-validates the peeked verdict against the tuple actually held under the lock so a race can't apply a stale verdict to a different (possibly live) cache entry. Tests: tests/gateway/test_stale_self_heal_agent_cache_eviction.py (5 new cases: dead-session eviction, live-sibling reuse preserved [NousResearch#54947 intact], cross-process invalidation preserved [NousResearch#45966 intact], same-session_id dead edge case, lock-race re-validation). Full tests/gateway/ suite: 14 failed, 9040 passed, 11 skipped -- all 14 failures verified pre-existing on unpatched main (confirmed via git stash + re-run), unrelated to this change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
The #54878/#55485 stale-routing self-heal in
get_or_create_sessioncorrectly detects when asessions.jsonrouting entry points at a session that state.db already marked ended (e.g. a TUI attached to the samesession_keyclosed and wroteend_reason="tui_shutdown", while the Gateway routing entry was left untouched). It drops the stale entry and falls through to_recover_session_from_db.But when that recovery attempt also fails — because
hermes_state.py::find_latest_gateway_session_for_peeronly treatsended_at IS NULLorend_reason='agent_close'as recoverable, andtui_shutdownisn't in that set — a brand-new, empty session is created in total silence. The self-heal branch hardcodeswas_auto_reset = False, so not even the generic idle/daily fallback notice (whichidle/daily/suspended/resume_pending_expiredresets already get) fires. The user has no way to learn their routing silently moved until they notice the missing context.This PR implements the "safe" half of the two-part fix proposed in #59580 — the notification gap, which is a pure additive fix with no behavior-changing side effects. The second half (whether
tui_shutdownshould joinagent_closein the recoverableend_reasonset, i.e. whether the session itself should be reopened instead of replaced) is a design question that needs maintainer input and is intentionally left untouched here.Related Issue
Addresses #59580 (notification gap only — see the issue for the open design question on recoverability)
Type of Change
Changes Made
gateway/session.py(get_or_create_session):stale_routing_dropped_session_id/stale_routing_had_activity._recover_session_from_dbfails to reopen it, setwas_auto_reset=Truewith a newauto_reset_reason="stale_routing_recovered"so the existing fix(gateway): notify user and write correct end_reason for resume_pending_expired resets #58935 notification plumbing fires for this path too.db_end_session_idis deliberately left unset for this path — the old session was already finalized elsewhere (with its real end_reason, e.g.tui_shutdown) and must not be overwritten.None), so this only fires for the actual self-heal case.gateway/run.py(_handle_message_with_agent,_was_auto_resetblock):elif reset_reason == "stale_routing_recovered":for the agent context note."stale_routing_recovered"to theshould_notifyalways-on set (alongside"suspended","resume_pending_expired").elif reset_reason == "stale_routing_recovered":for the user-facing noticereason_text.tests/gateway/test_session_reset_notify.py:TestStaleRoutingSelfHealNotifywith 5 new tests:test_unrecoverable_end_reason_notifies_and_creates_new_session—end_reason='tui_shutdown'→ new session withauto_reset_reason == "stale_routing_recovered"test_unrecoverable_end_reason_does_not_overwrite_db_end_reason—db.end_sessionis never called for the old session (its real reason is preserved)test_had_activity_reflects_dropped_session—reset_had_activityreflects the dropped session's token usagetest_recoverable_end_reason_reopens_without_notifying— non-regression:end_reason='agent_close'IS recoverable, silently reopens the SAMEsession_id, no notificationtest_brand_new_peer_does_not_trigger_stale_routing_reason— a peer with no prior routing entry is just a normal first session, not a self-healHow to Test
session_id = Avia any Gateway platform.Ain state.db withend_reason='tui_shutdown'(e.g. attach and cleanly close a TUI against the samesession_key) whilesessions.jsonstill points atA.gateway.session: routing key ... is ended in state.db ... (#54878)is logged, a new empty session is created, and the user receives no notice at all.◐ Session automatically reset (previous session ended and could not be auto-resumed). ..., andA'send_reasonin state.db is untouched.Or run the new unit tests:
Checklist
Code
fix(scope):,feat(scope):, etc.)pytest tests/gateway/test_session_reset_notify.py -vand all 26 tests passTestStaleRoutingSelfHealNotify)Documentation & Housekeeping
cli-config.yaml.exampleif I added/changed config keys — or N/A (no new config keys)Screenshots / Logs
Field incident from #59580 (profile
dingding, 2026-07-05 → 2026-07-06):After this fix, the user would have received:
Made with Cursor