Skip to content

fix(gateway): systemd KillMode=mixed + restart drain timeout + cron stop guard (#37454 #37453 #37858) - #39577

Open
ashishpatel26 wants to merge 1 commit into
NousResearch:mainfrom
ashishpatel26:fix/systemd-rebase-37454-37453-37858
Open

ashishpatel26 wants to merge 1 commit into
NousResearch:mainfrom
ashishpatel26:fix/systemd-rebase-37454-37453-37858

Conversation

@ashishpatel26

Copy link
Copy Markdown
Contributor

Summary

  • gateway/run.py: added
    unner=None param to _start_cron_ticker(); each tick checks getattr(runner, "_draining", False) before calling cron_tick(), preventing outbound LLM calls after hermes gateway stop
  • hermes_cli/gateway.py: manual restart path now derives timeout from _get_restart_drain_timeout() instead of hardcoded 10s, preventing race-to-port when drain takes longer
  • Tests: TestSystemdKillMode, TestManualRestartDrainWait, est_cron_drain_guard.py (4 tests)

Fixes #37454 #37453 #37858

🤖 Generated with Claude Code

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery comp/cron Cron scheduler and job management area/config Config system, migrations, profiles labels Jun 5, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing the shutdown/restart races. The manual-restart part still applies, but the cron hunk needs rebasing onto the current scheduler architecture.

Problems

  • gateway/run.py:19301 changes _start_cron_ticker, but current main marks that function as a deprecated compatibility shim at gateway/run.py:20324-20336. Production starts cron_provider.start(...) at gateway/run.py:20836-20846; InProcessCronScheduler.start() calls cron_tick() without a drain gate at cron/scheduler_provider.py:166-179. The proposed runner argument therefore is not on the production path.
  • The KillMode=mixed change is already on main for both generated systemd units (hermes_cli/gateway.py:2764, 2798), with existing coverage in tests/hermes_cli/test_gateway_service.py:426-449 and 537-561.

Suggested changes

  • Retain and rebase the manual fallback change: main still hardcodes timeout=10.0, force_after=5.0 at hermes_cli/gateway.py:7020.
  • Put the cron gate into the active provider startup/loop path and test that path; remove the superseded KillMode hunk.

Automated hermes-sweeper review.

Comment thread gateway/run.py
@@ -19301,10 +19301,10 @@ def _run_planned_stop_watcher(
stop_event.wait(poll_interval)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_start_cron_ticker is no longer the production gateway ticker on current main: it is now a deprecated shim, while start_gateway() invokes the resolved CronScheduler directly. Please move this drain gate into the active provider startup/InProcessCronScheduler.start() path; otherwise this new runner parameter is never used by the running gateway.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/cron Cron scheduler and job management comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gateway: orphan processes in cgroup block systemd restart for 6+ minutes

3 participants