fix: write update exit code before gateway restart (cgroup kill race) - #8288
Merged
Conversation
When /update runs via Telegram, hermes update --gateway is spawned inside
the gateway's systemd cgroup. The update process itself calls
systemctl restart hermes-gateway, which tears down the cgroup with
KillMode=mixed — SIGKILL to all remaining processes. The wrapping bash
shell is killed before it can execute the exit-code epilogue, so
.update_exit_code is never created. The new gateway's update watcher
then polls for 30 minutes and sends a spurious timeout message.
Fix: write .update_exit_code from Python inside cmd_update() immediately
after the git pull + pip install succeed ("Update complete!"), before
attempting the gateway restart. The shell epilogue still writes it too
(idempotent overwrite), but now the marker exists even when the process
is killed mid-restart.
Tommyeds
pushed a commit
to Tommyeds/hermes-agent
that referenced
this pull request
Apr 12, 2026
…NousResearch#8288) When /update runs via Telegram, hermes update --gateway is spawned inside the gateway's systemd cgroup. The update process itself calls systemctl restart hermes-gateway, which tears down the cgroup with KillMode=mixed — SIGKILL to all remaining processes. The wrapping bash shell is killed before it can execute the exit-code epilogue, so .update_exit_code is never created. The new gateway's update watcher then polls for 30 minutes and sends a spurious timeout message. Fix: write .update_exit_code from Python inside cmd_update() immediately after the git pull + pip install succeed ("Update complete!"), before attempting the gateway restart. The shell epilogue still writes it too (idempotent overwrite), but now the marker exists even when the process is killed mid-restart.
negaterium
pushed a commit
to negaterium/hermes-agent
that referenced
this pull request
Apr 13, 2026
…NousResearch#8288) When /update runs via Telegram, hermes update --gateway is spawned inside the gateway's systemd cgroup. The update process itself calls systemctl restart hermes-gateway, which tears down the cgroup with KillMode=mixed — SIGKILL to all remaining processes. The wrapping bash shell is killed before it can execute the exit-code epilogue, so .update_exit_code is never created. The new gateway's update watcher then polls for 30 minutes and sends a spurious timeout message. Fix: write .update_exit_code from Python inside cmd_update() immediately after the git pull + pip install succeed ("Update complete!"), before attempting the gateway restart. The shell epilogue still writes it too (idempotent overwrite), but now the marker exists even when the process is killed mid-restart.
aj-nt
pushed a commit
to aj-nt/hermes-agent
that referenced
this pull request
May 1, 2026
…NousResearch#8288) When /update runs via Telegram, hermes update --gateway is spawned inside the gateway's systemd cgroup. The update process itself calls systemctl restart hermes-gateway, which tears down the cgroup with KillMode=mixed — SIGKILL to all remaining processes. The wrapping bash shell is killed before it can execute the exit-code epilogue, so .update_exit_code is never created. The new gateway's update watcher then polls for 30 minutes and sends a spurious timeout message. Fix: write .update_exit_code from Python inside cmd_update() immediately after the git pull + pip install succeed ("Update complete!"), before attempting the gateway restart. The shell epilogue still writes it too (idempotent overwrite), but now the marker exists even when the process is killed mid-restart.
1 task
02356abc
pushed a commit
to 02356abc/hermes-agent
that referenced
this pull request
May 14, 2026
…NousResearch#8288) When /update runs via Telegram, hermes update --gateway is spawned inside the gateway's systemd cgroup. The update process itself calls systemctl restart hermes-gateway, which tears down the cgroup with KillMode=mixed — SIGKILL to all remaining processes. The wrapping bash shell is killed before it can execute the exit-code epilogue, so .update_exit_code is never created. The new gateway's update watcher then polls for 30 minutes and sends a spurious timeout message. Fix: write .update_exit_code from Python inside cmd_update() immediately after the git pull + pip install succeed ("Update complete!"), before attempting the gateway restart. The shell epilogue still writes it too (idempotent overwrite), but now the marker exists even when the process is killed mid-restart.
waefrebeorn
pushed a commit
to waefrebeorn/slermes
that referenced
this pull request
Jul 2, 2026
…NousResearch#8288) When /update runs via Telegram, hermes update --gateway is spawned inside the gateway's systemd cgroup. The update process itself calls systemctl restart hermes-gateway, which tears down the cgroup with KillMode=mixed — SIGKILL to all remaining processes. The wrapping bash shell is killed before it can execute the exit-code epilogue, so .update_exit_code is never created. The new gateway's update watcher then polls for 30 minutes and sends a spurious timeout message. Fix: write .update_exit_code from Python inside cmd_update() immediately after the git pull + pip install succeed ("Update complete!"), before attempting the gateway restart. The shell epilogue still writes it too (idempotent overwrite), but now the marker exists even when the process is killed mid-restart.
19 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the spurious "❌ Hermes update timed out after 30 minutes" message that appears after a successful
/updatevia Telegram.Root Cause
When
/updateis triggered from the gateway,hermes update --gatewayis spawned as a child of the gateway process. Despite usingsetsidfor session isolation, the child process inherits the gateway's systemd cgroup. The service template usesKillMode=mixed, so when the update process callssystemctl restart hermes-gateway:hermes update --gatewayand its wrapping bash shellprintf $status > .update_exit_code) never executesThe user sees the streamed "✓ Update complete!" text (printed before the restart attempt) and then the spurious timeout 30 minutes later.
Fix
Write
.update_exit_codefrom Python insidecmd_update()immediately after git pull + pip install succeed, before attempting the gateway restart. Only done in--gatewaymode (normal CLI updates still rely on the shell epilogue).The shell epilogue still writes the file too (idempotent overwrite with the same value), but now the marker exists even when the process is killed mid-restart.
Tests
3 new tests in
test_update_gateway_restart.py:test_exit_code_written_in_gateway_mode— verifies marker is createdtest_exit_code_not_written_in_normal_mode— no side effects for CLI usagetest_exit_code_written_before_restart_call— verifies ordering: exit code exists beforesystemctl restartis calledAll 34 tests in the file pass.
Files Changed
hermes_cli/main.py— +20 lines (write exit code before restart block)tests/hermes_cli/test_update_gateway_restart.py— +117 lines (3 new tests)