fix(gateway): clear stale resume_pending markers in scheduler - #52637
Closed
crayfish-ai wants to merge 1 commit into
Closed
fix(gateway): clear stale resume_pending markers in scheduler#52637crayfish-ai wants to merge 1 commit into
crayfish-ai wants to merge 1 commit into
Conversation
When a session's resume_pending marker outlasts the freshness window, the scheduler skips auto-resuming it but leaves the stale flag in sessions.json. Over multiple restarts, these accumulate as zombie entries. On startup, the scheduler now calls clear_resume_pending() for each expired marker so it doesn't persist. This is a defense-in-depth complement to the freshness gate in get_or_create_session() (PR NousResearch#46963), which prevents stale sessions from being served to users at message time. Together they close the resume_pending residual accumulation loop. Fixes NousResearch#46934
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #46934
When the gateway restarts,
_schedule_resume_pending_sessions()skips auto-resuming sessions whoseresume_pendingmarker has expired (beyondHERMES_AUTO_CONTINUE_FRESHNESSwindow). However, it leaves the stale flag insessions.json. Over multiple restarts, these accumulate as zombie entries that are never cleaned up.Root Cause
In
_schedule_resume_pending_sessions()(gateway/run.py:5466), expired sessions are skipped with justcontinue— no call toclear_resume_pending().Fix
Add
self.session_store.clear_resume_pending(entry.session_key)beforecontinuein the freshness-expired path.Real-World Reproduction (Feishu, June 2026)
A user with active conversations in both a DM and a group chat restarted their gateway. The restart auto-resumed both sessions simultaneously, producing a DM response that referenced group-chat context. The user perceived this as "messages routed to the wrong chat."
Investigation (spanning two independent code audits and gateway log analysis of >10,000 lines) confirmed zero chat_id routing bugs. All inbound/response chat_id fields matched perfectly in logs.
The actual root cause was
resume_pendingaccumulation across restarts:Previous restart — a DM session was marked
resume_pending=True. The user sent a new message during the recovery turn, causinginterrupted=True._should_clear_resume_pending_after_turn()returnedFalsefor interrupted turns (correctly — soft interrupts need retry). The marker was left on the session.Second restart (days later) —
suspend_recently_active(120)skipped the already-marked session (line 1285:if entry.resume_pending: continue), logging "marked 1 session." But_schedule_resume_pending_sessions()picked up 2 sessions — the freshly-marked group session plus the residual DM session from the previous restart. Both were within the 3600s freshness window (based onlast_resume_marked_atfrom the recent restart). Both auto-resumed simultaneously.Dual auto-resume produced two recovery turns in parallel. The DM agent, resuming with group-chat context in its transcript, produced output that appeared "wrongly routed" to the user.
Log excerpt (redacted):
After this fix,
_schedule_resume_pending_sessions()clears stale markers on startup:Relationship to PR #46963
PR #46963 adds a freshness gate in
get_or_create_session()— when a user sends a message to a staleresume_pendingsession, it falls through to session reset. This PR complements it by cleaning up stale markers at the scheduler level, preventing them from accumulating in the session store across multiple restarts. Together they close theresume_pendingresidual accumulation loop from both sides (scheduler cleanup + inbound-message gate).Before
After