Skip to content

fix(gateway): clear session-level overrides and YOLO state on shutdown drain - #32835

Closed
waefrebeorn wants to merge 2 commits into
NousResearch:mainfrom
waefrebeorn:fix/gateway-drain-clear-session-state
Closed

waefrebeorn wants to merge 2 commits into
NousResearch:mainfrom
waefrebeorn:fix/gateway-drain-clear-session-state

Conversation

@waefrebeorn

Copy link
Copy Markdown
Contributor

What does this PR do?

When the Hermes gateway drains and restarts, session-level configuration state was leaking across the boundary:

  1. YOLO mode persisted/yolo toggled _session_yolo in tools/approval.py, but _finalize_shutdown_agents never cleared it. On restart, YOLO stayed active.
  2. Model/reasoning overrides leaked_session_model_overrides and _session_reasoning_overrides were in-memory dicts that survived restart.
  3. Stale pending approvals_pending_approvals carried old gateway approval requests that could block new sessions.

Changes

  • tools/approval.py: Added clear_all_sessions() — atomically clears _session_yolo, _session_approved, _gateway_queues, and _pending under the module lock. Denies any in-flight approval entries so they don't hang.
  • gateway/run.py (_finalize_shutdown_agents): Now clears _session_model_overrides, _session_reasoning_overrides, _pending_approvals, _pending_model_notes, and calls clear_all_sessions() before agent resource cleanup.

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Checklist

…n drain

Gateway shutdown/_finalize_shutdown_agents was cleaning per-agent resources
(memory provider, tool resources) but leaving session-scoped state intact:

- _session_model_overrides retained stale model switches across restarts
- _session_yolo (in tools/approval.py) kept YOLO approval bypass active
- _pending_approvals carried stale gateway approval requests

Add clear_all_sessions() to tools/approval.py that resets all per-session
approval and YOLO state atomically under lock. Call it from
_finalize_shutdown_agents alongside clearing model/reasoning overrides
and pending approval dicts.

This fixes: YOLO mode persisting through gateway restart, stale model
overrides after drain, and gateway drain not actually resetting session
configuration.
@rodriguez46p-ui

Copy link
Copy Markdown

CI is mostly green, but check-attribution is currently blocking this PR.

Failing line from the Actions log:

New contributor email(s) not in AUTHOR_MAP:
  wubu.bounty.hunter@users.noreply.github.com (WuBuBountyHunter)

The requested fix is in scripts/release.pyAUTHOR_MAP, e.g. add a mapping for:

"wubu.bounty.hunter@users.noreply.github.com": "<github-username>",

Use the intended GitHub username/credit string for the value (the log labels it WuBuBountyHunter). The focused local regression still passes on Windows: tests/cron/test_scheduler.py tests/gateway/test_status.py132 passed.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery labels May 26, 2026
@alt-glitch

Copy link
Copy Markdown
Contributor

Related: #14246 / #14266 (session-boundary approval/yolo state cleanup on /resume, /branch, /new). Also related to #32184 (model/reasoning override cleanup on resume/branch). This PR extends the pattern to the shutdown drain path.

waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request May 26, 2026
When _try_refresh_nous_client_credentials fails on every credential
path (Portal OAuth, refresh token, shared credentials store), write
~/.hermes/.auth_restart_requested so the GatewayRunner can trigger
a process restart as the last-resort recovery mechanism. This is the
final step in the fix cascade:
1. Session overrides cleared on drain → PR NousResearch#32835
2. Shared credentials fallback → this PR
3. Gateway restart (last resort) → this commit
… resort

When _try_refresh_nous_client_credentials exhausts every credential
path, it writes ~/.hermes/.auth_restart_requested. The gateway now
checks for this marker at the start of every message and triggers a
process restart if the marker is recent (< 5 min). This is the final
step in the fix cascade:
1. Session overrides cleared on drain → this PR
2. Shared credentials fallback → PR NousResearch#32826
3. Gateway restart (last resort) → this commit
@waefrebeorn

Copy link
Copy Markdown
Contributor Author

Last-resort auto-restart added in 64ccc29: when all credential paths fail (session drain → shared creds fallback), the gateway now auto-restarts on the next incoming message. The .auth_restart_requested marker is written by the agent when _try_refresh_nous_client_credentials returns False, and the GatewayRunner checks it at message handler entry.

Fix cascade now:

  1. ✅ Session overrides cleared on drain
  2. ✅ Shared credentials fallback (fix: fall back to shared credentials store on Nous 401 refresh failure #32826)
  3. ✅ Gateway auto-restart (this commit)

@waefrebeorn

Copy link
Copy Markdown
Contributor Author

AUTHOR_MAP fix landed in PR #33044 — adds wubu.bounty.hunter@users.noreply.github.com → WuBuBountyHunter. Once that merges, this PR's check-attribution should pass.

@alt-glitch alt-glitch added the sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state label Jul 1, 2026
@waefrebeorn waefrebeorn closed this Jul 2, 2026
@waefrebeorn
waefrebeorn deleted the fix/gateway-drain-clear-session-state branch July 2, 2026 07:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants