fix(gateway): follow session rotation in the heartbeat poller - #80208
Open
0xGr1mm wants to merge 1 commit into
Open
fix(gateway): follow session rotation in the heartbeat poller#802080xGr1mm wants to merge 1 commit into
0xGr1mm wants to merge 1 commit into
Conversation
5 tasks
This was referenced Aug 6, 2026
`/heartbeat` stops firing permanently, and silently, the first time a
gateway session compresses.
`_register_heartbeat_watch` stores `quick_key -> (source, session_id)` and
the poller used that captured `session_id` on every tick. A context
compression rotates the route onto a fresh session and carries the
heartbeat across with `migrate_heartbeat_to_session`, which writes the
state to the child and marks the parent row `cleared`. The poller, still
holding the parent id, read the cleared row, took it for "user removed the
heartbeat", and dropped the watch:
mgr = HeartbeatManager(session_id=session_id) # parent id, now cleared
if not mgr.has_heartbeat():
watch.pop(quick_key, None) # gone for good
continue
Nothing surfaces it. `/heartbeat status` resolves the *live* session via
`_get_heartbeat_manager_for_event`, finds the migrated state, and keeps
reporting `♥ Heartbeat (every 10m, next in ~599s)` for a heartbeat that
will never fire again in that process. Reproduced against the real state
layer: firing before rotation, watch dropped after it, status still
counting down.
The CLI driver is unaffected — `_get_heartbeat_manager` rebinds whenever
`session_id` changes, so it follows the rotation already. This restores the
same property on the gateway.
Re-resolve the route's current session id each tick through the new
`_live_heartbeat_session_id`, and cache it back into the registry.
`SessionStore.peek_session_id` is the read-only, lock-held accessor for the
key→session_id mapping that `advance_compression_session` repairs on
rotation; `quick_key` is the same `session_key` that mapping is keyed by.
It never creates a session, so a route the user has since ended resolves to
nothing, falls back to the captured id, finds no heartbeat, and drops the
watch as before.
Unchanged: busy routes still coalesce their tick, routes whose heartbeat
was genuinely cleared still unregister, and a failing store lookup falls
back to the captured id instead of taking the poller down.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
0xGr1mm
force-pushed
the
fix/gateway-heartbeat-follows-session-rotation
branch
from
August 9, 2026 08:56
90b85c2 to
706f7fd
Compare
10 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
/heartbeatstops firing permanently the first time a gateway session compresses, and nothing tells the user._register_heartbeat_watchstoresquick_key -> (source, session_id), and the poller used that capturedsession_idon every tick. A context compression rotates the route onto a fresh session and carries the heartbeat across withmigrate_heartbeat_to_session, which writes the state to the child and marks the parent rowcleared. The poller is still holding the parent id, so it reads the cleared row, reads it as "the user removed the heartbeat", and unregisters the watch:The failure is silent in the worst way:
/heartbeat statusgoes through_get_heartbeat_manager_for_event, which resolves the live session, finds the migrated state, and keeps reporting a healthy heartbeat with a ticking countdown. So the one surface a user would check to diagnose it actively confirms everything is fine.Reproduced against the real state layer (
migrate_heartbeat_to_session+HeartbeatManager, with the poller's decision replayed verbatim):The CLI driver does not have this bug —
_get_heartbeat_managerrebinds wheneversession_idchanges, so it follows the rotation already. This PR restores the same property on the gateway.The fix
Re-resolve the route's current session id on every tick through a new
_live_heartbeat_session_id, and cache the result back into the registry so the next tick starts from the live id.SessionStore.peek_session_idis the right accessor here: it is the public, lock-held, read-only view of the same key→session_id mapping thatadvance_compression_sessionrepairs during rotation, andquick_keyis exactly thesession_keythat mapping is keyed by (_session_key_for_source→_generate_session_key, the same value passed toadvance_compression_session). It never creates a session, so a route the user has since ended resolves to nothing, falls back to the captured id, finds no heartbeat, and drops the watch exactly as before.Related Issue
No existing issue. Searched open and merged PRs and issues first, per CONTRIBUTING's search-first section:
Bug was introduced with
/heartbeatin #79681.Type of Change
Bug fix (non-breaking).
Changes Made
gateway/run.py— added_live_heartbeat_session_id(quick_key, fallback); the poll loop resolves through it and writes the resolved id back into_heartbeat_watch; updated the_register_heartbeat_watchdocstring so the stored id is documented as a starting point rather than the source of truth.tests/gateway/test_heartbeat_poller_session_rotation.py— new, 6 tests: the rotation regression, the two unchanged-behaviour paths (cleared heartbeat still unregisters, busy route still coalesces), and three_live_heartbeat_session_idcases (live mapping wins, unknown route falls back, raising store falls back).How to Test
6 passed.
The regression test drives the real
_start_heartbeat_pollerloop against a bareGatewayRunner, with the route mapping mutated mid-test the way a real rotation mutates the store. Reverting onlygateway/run.pyand rerunning:The two that still pass without the fix are the unchanged-behaviour tests, which is the point — they pin that this change does not alter the drop path or the busy-coalesce path.
End to end: set
/heartbeat every 60s …on a gateway platform, drive the session until it compresses, and confirm the heartbeat keeps arriving. Before this change it goes quiet at the rotation while/heartbeat statuskeeps counting down.Full gateway suite, this branch vs
main, same environment:The failing sets are identical — zero added, zero removed — and those 7 are pre-existing in a non-hermetic single-process run (
pytest tests/gateway/); CI's per-file isolation viarun_tests_parallel.pyis the supported path.Checklist
Code
fix(gateway): …)Documentation & Housekeeping
_register_heartbeat_watch; the user-facingheartbeat.mdneeds no change because it describes the behaviour this restorescli-config.yaml.example— N/A, no config keysCONTRIBUTING.md/AGENTS.md— N/A/heartbeatis a slash commandNotes for the reviewer
peek_session_idis a synchronous, lock-held call made from the poll task. That matches what the loop already does — it constructs aHeartbeatManageron the same tick, which reads SQLite synchronously — and it is the lighter of the two. If you would rather the whole tick move off the loop, say so and I will wrap both inasyncio.to_threadin this PR.One deliberate non-change: a route reset with
/new(aswitch_session, not a compression) does not migrate heartbeat state, so the new session has none and the watch is dropped. That reads as correct for a session-scoped feature —/newis a fresh conversation — so I left it alone rather than widen the scope here.Stacked on the same feature area as #80185 (
/heartbeatinterval parsing) but independent: different files, no overlap, either can merge first.Unrelated observation from verifying this, in case it is news:
gateway.status._get_lock_dir()resolves to a machine-global path ($XDG_STATE_HOME/hermes/locks, i.e.~/.local/state/hermes/locks) andtests/conftest.pydoes not sandboxHERMES_GATEWAY_LOCK_DIRthe way it sandboxesHERMES_HOME. So the gateway platform-lock tests write to the developer's real lock directory, and two concurrentpytest tests/gateway/runs on one machine collide —test_whatsapp_connect.py::TestDataInitialized::test_no_name_error_when_json_always_failsfails with "WhatsApp session already in use". I hit exactly that while running a branch and a baseline suite side by side; serially both are clean. Not touched here — flagging it in case it is worth a conftest sandbox.