fix(gateway): reap cgroup orphans via ExecStopPost to unblock systemd restart - #54090
Merged
Conversation
Long-lived helpers spawned indirectly by tool calls (adb, platform bridges) were left in the service cgroup after the gateway's main process exited. When the kernel rejected the deferred cgroup-wide kill with EINVAL, systemd blocked Restart=always for 6+ minutes, taking down all platforms and cron windows (#37454). Add a small ExecStopPost helper (gateway.cgroup_cleanup) that walks cgroup.procs and sends per-PID SIGKILLs — a different kernel code path than cgroup.kill, so it succeeds where the cgroup-wide write failed. KillMode=mixed is preserved so the gateway still reaps its own tool-call children before systemd intervenes (#8202). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
🔎 Lint report:
|
| Rule | Count |
|---|---|
unresolved-import |
1 |
First entries
tests/gateway/test_cgroup_cleanup.py:9: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
✅ Fixed issues: none
Unchanged: 6097 pre-existing issues carried over.
Diagnostics are surfaced as warnings — this check never fails the build.
- read_text(encoding='utf-8') (PLW1514) - # windows-footgun: ok on signal.SIGKILL — module is Linux-only (reads /proc, /sys/fs/cgroup; runs from a systemd unit) - test lambda accepts the new encoding kwarg
alt-glitch
added
type/bug
7 tasks
Code Review SummaryVerdict: LGTM Adds ExecStopPost cgroup reaper to systemd unit to SIGKILL orphaned processes after gateway shutdown. Fixes #37454 (long-lived helpers blocking Restart=always).
Reviewed by Hermes Agent |
This was referenced Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A long-lived helper left in the gateway's systemd cgroup after the main process exits no longer blocks
Restart=always— a newExecStopPost=reaper SIGKILLs whatever the gateway didn't clean up itself, so the unit can restart immediately instead of stalling for minutes.Root cause (#37454):
KillMode=mixedonly signals the main PID. On shutdown an untracked helper (e.g.adb) survived in the cgroup; systemd's cgroup-wide kill returnedEINVAL, leaving the process parked in the cgroup andRestart=alwaysunable to fire for 6+ minutes — a full outage across every platform and all cron jobs.Changes
gateway/cgroup_cleanup.py(new): reads the unit's owncgroup.procsand SIGKILLs each remaining PID individually (skipping itself), tolerating already-exited / unkillable PIDs. Deliberately per-PID rather than writingcgroup.kill, because the reported failure wasEINVALon the cgroup-wide kill — per-PIDos.killis the delivery path that still works.hermes_cli/gateway.py: addExecStopPost=-{python} -m gateway.cgroup_cleanupto both systemd unit templates (user + system).KillMode=mixedis preserved so the gateway still reaps its own tool-call children during drain (Gateway process not cleaned up on unclean shutdown after provider timeout — bash/sleep children SIGKILL'd by systemd #8202); the reaper is a safety net, not a replacement.scripts/release.py: AUTHOR_MAP entry for the salvaged author.tests/gateway/test_cgroup_cleanup.py,tests/hermes_cli/test_gateway_service.py.Validation
KillMode=mixedpreserved/proc/self/cgroupSalvaged from #37550 by @PRATHAMESH75 (cherry-picked, authorship preserved). Closes #37454.
Infographic