hermes gateway stop no longer exits 1 (and gets revived) when the stop watcher beats the CLI's SIGTERM - #120152
Merged
Merged
Conversation
… CLI's SIGTERM `hermes gateway stop` writes the planned-stop marker, then sends SIGTERM. The gateway's planned-stop watcher (0.5 s poll) can fire in between: its shutdown call consumes the marker as planned, and the CLI's SIGTERM that follows finds no marker, is classified as an external kill and the gateway exits 1 — which Restart=on-failure supervisors answer by reviving a gateway the operator just stopped. Remember an accepted planned stop in the handler so the trailing SIGTERM of the same stop is treated as planned. Found by the core parity E2E matrix (gateway/api_server graceful_exit cell), reproduced deterministically by signalling after the watcher has consumed the marker.
…marker stays planned Invariant for the planned-stop race: the watcher consumes the marker, the CLI trailing SIGTERM must not read as an external kill; a bare SIGTERM with no planned stop before it still does.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
hermes gateway stopnow always exits the gateway 0, so aRestart=on-failure(or any exit-code-driven) supervisor no longer revives a gateway the operator just stopped.gateway/run.py::_start_gateway_make_shutdown_signal_handler, +8 lines): the handler remembers that it already accepted a planned stop, and the trailing SIGTERM of that same stop stays planned. A--replacetakeover still takes its own branch, and a bare SIGTERM with no planned stop before it is still an external kill (exit 1, supervisor revives).hermes gateway stop, the orphan reaper, the launchd fallback, and the systemdExecStop=marker from fix(gateway): direct systemctl restart/stop exits 0 instead of logging 'Failed with result exit-code' (#116551, salvage #116589) #116730, which also writes the marker before systemd delivers SIGTERM and so hit the same race.tests/gateway/test_planned_stop_watcher.pydrives the real handler with a real self-targeting marker file. The watcher's call consumes the marker, then the CLI's SIGTERM arrives. Control: a bare SIGTERM on a fresh handler is still flagged. Red on base (assert True is False), green with the fix.graceful_exitcell ongateway (fake adapter)+api_server, forces the worst-case interleaving) lands separately in the E2E-suite PR.Live repro (real gateway processes, driven by the parity E2E driver: marker written for the gateway PID, SIGTERM sent only after the watcher consumed it):
gateway (fake adapter)api_serverorigin/main07646a7)parity cells red: ['graceful_exit'],host exit code: 1parity cells red: ['graceful_exit'],host exit code: 1Under load the natural interleaving hit this 1 in 24 stops. The driver now forces it every time.
Related, not superseded: #107230 (@lawcheck) and earlier #41690 / #41642 / #24351 target a different root cause: a systemd-initiated stop with no marker at all, which main already handles via the
ExecStop=marker (#116730). None of them touches the watcher-consumes-marker race. #107230's parent-is-systemd heuristic would mask this race only under systemd, not for launchd, s6, a barehermes gateway rununder another supervisor, or the orphan reaper. No open or closed PR fixes this race (swept:planned stop marker,planned-stop watcher,gateway stop exit 1,consume_planned_stop_marker_for_self,shutdown_signal_handler,planned stop SIGTERM race).Validation:
scripts/run_tests.sh tests/gateway/(8660 passed, 1 failed, 31 skipped. The one failure,test_session_hygiene_turnhold_adoption.py::test_turn_hold_keeps_admission_and_adopts_watermark_fenced_summary('denied' == 'timeout'), is a load-dependent timing flake that also fails 2 of 4 standalone runs on unmodifiedorigin/mainat host load ~90. It doesn't touch the signal handler.);ruff checkclean;check_no_tmp_literals.py gateway tests/gatewayclean;git diff --checkclean.Infographic