Skip to content

fix: /update now drains active agents before gateway exit - #60690

Closed
mahdiwafy wants to merge 1 commit into
NousResearch:mainfrom
mahdiwafy:fix/update-drain-before-exit
Closed

mahdiwafy wants to merge 1 commit into
NousResearch:mainfrom
mahdiwafy:fix/update-drain-before-exit

Conversation

@mahdiwafy

Copy link
Copy Markdown

Summary

/update spawned the update helper via setsid but did NOT call request_restart(), so the gateway exited without running the drain phase (active_at_start=0, drain took 0.00s). This killed active cron jobs and tool subprocesses mid-work.

A production incident with the Daily family briefing during /update showed the more severe failure mode: the cron job started, then never logged completion or delivery.

Fix

Call request_restart() after spawning the update helper, same as /restart does. This triggers stop() -> _drain_active_agents() with the configured restart_drain_timeout, giving in-flight work a graceful drain window before the gateway process exits.

Comparison: /restart vs /update (before fix)

/restart /update (before) /update (after)
Spawn detached helper yes yes yes
request_restart() yes no yes
stop() -> _drain_active_agents() yes no yes
Drain timeout restart_drain_timeout none restart_drain_timeout

Closes #60432

…ch#60432)

/update spawned the update helper via setsid but did NOT call
request_restart(), so the gateway exited without running the drain
phase. This killed active cron jobs and tool subprocesses mid-work.

Fix: call request_restart() after spawning the update helper, same
as /restart does. This triggers stop() -> _drain_active_agents()
with the configured restart_drain_timeout, giving in-flight work
a graceful drain window.

Closes NousResearch#60432
@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery comp/cron Cron scheduler and job management P2 Medium — degraded but workaround exists labels Jul 8, 2026
@teknium1

Copy link
Copy Markdown
Collaborator

Thanks for the focused investigation and the clear comparison with /restart.

Automated hermes-sweeper review found that the behavioral guarantee is already present on current main:

Closing as implemented on main.

@teknium1 teknium1 closed this Jul 10, 2026
@teknium1 teknium1 added the sweeper:implemented-on-main Sweeper: behavior already present on current main label Jul 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:implemented-on-main Sweeper: behavior already present on current main type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

/update bypasses drain and interrupts in-flight cron tool work

3 participants