fix(update): crashed updates no longer strand stopped gateways — a fresh process settles the restart debt (#92145, salvage #94392) - #95930
Merged
Conversation
…restart recovery Salvage adjustments to PR #94392 per review: - Narrow the supervisor claim to the systemd-VERIFIED path only. The fresh recovery child now probes 'systemctl --user is-active' after each relaunch; only an observed-active systemd unit is reported 'verified'. A relaunch that merely exited 0 is labelled 'relaunch_attempted', never counts as supervisor coverage, and never clears gateway_fleet_restart_incomplete. - Serve-owned runtimes (serve/dashboard entries from the spawn ledger, per the update_inventory serve collector) are no longer silently skipped: the recovery pass records them (and manual gateways) as skipped-with-reason in the recovery result and the persisted update receipt. - Receipt fresh_recovery persists the conservative vocabulary (requested/verified/relaunch_attempted/failed/skipped); 'succeeded' is gone. - Added an end-to-end test that drives the real recovery module in a genuinely fresh interpreter (sitecustomize shim intercepts the grandchild 'gateway restart' and systemctl probes).
Contributor
૮ >ﻌ< ა ci reviewran on 7fa2d4f — fix(update): conservative outcomes + serve-ledger coverage f
|
alt-glitch
added
type/bug
alt-glitch
added
comp/gateway
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An update that crashes AFTER stopping gateways but BEFORE relaunching them no longer strands them dead — the next
hermes update(a fresh process) discovers the unpaid restart debt and settles it (#92145; salvage of #94392 by @JoaoMarcos44, all four commits cherry-picked with authorship preserved, plus one adjustment commit from review). Fleet relevance (#91277): an interrupted update leaving a whole fleet's gateways down with no operator signal is the worst version of the stale-gateway class.Changes
gateway restartis not proof of a running gateway. Onlysystemctl --user is-active-confirmed units reportverified; everything else reportsrelaunch_attempted, never clearsgateway_fleet_restart_incomplete, stays out ofrelaunched_profiles, and prints a "not supervisor-verified" operator warning. Parent fails closed on unaccounted profiles.Validation
Live repro: A/B above — the fresh-process path and
fresh_recoveryreceipt key don't exist on merge-base (KeyError).Honest limits: launchd/s6 have no verification probe by design (they report
relaunch_attempted; a launchctl verifier is a natural follow-up), and no real production systemd fleet was harmed in testing — the systemd observation path is proven via shim + units. Refs #92145 (the reviewer's RED→GREEN systemd harness standard for a full "Fixes" claim stands).Credit: @JoaoMarcos44.
Infographic