Skip to content

fix(gateway): clear stale resume_pending markers in scheduler - #52637

Closed
crayfish-ai wants to merge 1 commit into
NousResearch:mainfrom
crayfish-ai:fix/resume-pending-stale-cleanup
Closed

fix(gateway): clear stale resume_pending markers in scheduler#52637
crayfish-ai wants to merge 1 commit into
NousResearch:mainfrom
crayfish-ai:fix/resume-pending-stale-cleanup

Conversation

@crayfish-ai

@crayfish-ai crayfish-ai commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Fixes #46934

When the gateway restarts, _schedule_resume_pending_sessions() skips auto-resuming sessions whose resume_pending marker has expired (beyond HERMES_AUTO_CONTINUE_FRESHNESS window). However, it leaves the stale flag in sessions.json. Over multiple restarts, these accumulate as zombie entries that are never cleaned up.

Root Cause

In _schedule_resume_pending_sessions() (gateway/run.py:5466), expired sessions are skipped with just continue — no call to clear_resume_pending().

Fix

Add self.session_store.clear_resume_pending(entry.session_key) before continue in the freshness-expired path.

Real-World Reproduction (Feishu, June 2026)

A user with active conversations in both a DM and a group chat restarted their gateway. The restart auto-resumed both sessions simultaneously, producing a DM response that referenced group-chat context. The user perceived this as "messages routed to the wrong chat."

Investigation (spanning two independent code audits and gateway log analysis of >10,000 lines) confirmed zero chat_id routing bugs. All inbound/response chat_id fields matched perfectly in logs.

The actual root cause was resume_pending accumulation across restarts:

  1. Previous restart — a DM session was marked resume_pending=True. The user sent a new message during the recovery turn, causing interrupted=True. _should_clear_resume_pending_after_turn() returned False for interrupted turns (correctly — soft interrupts need retry). The marker was left on the session.

  2. Second restart (days later)suspend_recently_active(120) skipped the already-marked session (line 1285: if entry.resume_pending: continue), logging "marked 1 session." But _schedule_resume_pending_sessions() picked up 2 sessions — the freshly-marked group session plus the residual DM session from the previous restart. Both were within the 3600s freshness window (based on last_resume_marked_at from the recent restart). Both auto-resumed simultaneously.

  3. Dual auto-resume produced two recovery turns in parallel. The DM agent, resuming with group-chat context in its transcript, produced output that appeared "wrongly routed" to the user.

Log excerpt (redacted):

[TIME] gateway restart → Marked 1 in-flight session(s) as resumable
[TIME] Scheduled auto-resume for 2 restart-interrupted session(s)  ← 2, not 1!
[TIME] DM auto-resume    → response (28 chars) to DM_chat_id
[TIME] Group auto-resume → response (75 chars) to group_chat_id

After this fix, _schedule_resume_pending_sessions() clears stale markers on startup:

  • Session count: 1 marked, 1 auto-resumed (the residual is cleaned up)
  • No stale markers persist across restarts

Relationship to PR #46963

PR #46963 adds a freshness gate in get_or_create_session() — when a user sends a message to a stale resume_pending session, it falls through to session reset. This PR complements it by cleaning up stale markers at the scheduler level, preventing them from accumulating in the session store across multiple restarts. Together they close the resume_pending residual accumulation loop from both sides (scheduler cleanup + inbound-message gate).

Before

if marker is not None and (now - marker).total_seconds() > window:
    continue

After

if marker is not None and (now - marker).total_seconds() > window:
    # Stale marker: clear it so resume_pending flags don't
    # accumulate across restarts. The next user message will
    # still trigger normal reset-policy evaluation in
    # get_or_create_session().
    try:
        self.session_store.clear_resume_pending(entry.session_key)
    except Exception:
        pass
    continue

When a session's resume_pending marker outlasts the freshness window,
the scheduler skips auto-resuming it but leaves the stale flag in
sessions.json. Over multiple restarts, these accumulate as zombie
entries. On startup, the scheduler now calls clear_resume_pending()
for each expired marker so it doesn't persist.

This is a defense-in-depth complement to the freshness gate in
get_or_create_session() (PR NousResearch#46963), which prevents stale sessions
from being served to users at message time. Together they close the
resume_pending residual accumulation loop.

Fixes NousResearch#46934
@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state labels Jun 25, 2026
@crayfish-ai
crayfish-ai deleted the fix/resume-pending-stale-cleanup branch June 26, 2026 02:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: stale resume_pending sessions bypass idle reset, causing context bleed after gateway restart

2 participants