feat(gateway): restart manual profile gateways after update - #18178
Merged
Conversation
The PR wired in a detached watcher that respawns manual profile gateways after they exit. Pair that with a SIGUSR1 graceful drain (same path systemd/launchd use) so in-flight agent runs finish instead of getting SIGTERM'd. Fall back to SIGTERM if SIGUSR1 isn't wired or the gateway doesn't exit within the drain budget — the watcher sees the exit and relaunches either way. Tested end-to-end against an orphaned gateway: graceful drain exits in 0.5s and the watcher fires the relaunch command.
alt-glitch
added
comp/cli
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Salvages #17982 by @quocanh261997 onto current main and adds a graceful-drain follow-up.
What
gateway.pidfile.hermes update, relaunch profile-mapped manual gateways after terminating the old process via a detached Python watcher that waits for the old PID to exit, then spawnshermes [-p <profile>] gateway run --replace._graceful_restart_via_sigusr1()path as systemd/launchd services, so in-flight agent runs finish instead of being SIGTERM'd. Watcher detects the clean exit and relaunches.Why
hermes updatealready restarts service-managed gateways. Manual profile gateways were only stopped, leaving users to restart each one by hand — painful with many profiles. The PR #17982 approach (PID-file mapping + detached watcher) is clean. Adding SIGUSR1 drain brings manual profile gateways to parity with service-managed ones for in-flight work.Validation
→ 43 passed (original 41 + 1 updated for graceful-drain path + 1 new for SIGTERM-fallback path).
E2E tested end-to-end against a real orphaned gateway (init-reparented, realistic production topology):
Credit
Original PR #17982 by @quocanh261997. Their commit is preserved as the middle commit in this branch. Please rebase-merge to keep authorship attribution.
Closes #17982.