Skip to content

fix(cli): post-update fleet check fails closed when a running gateway yields zero rows (#93406, salvage #93410) - #93654

Merged
teknium1 merged 5 commits into
mainfrom
salv/93406-fleet-check-fail-closed
Aug 24, 2026
Merged

teknium1 merged 5 commits into
mainfrom
salv/93406-fleet-check-fail-closed

Conversation

@teknium1

Copy link
Copy Markdown
Collaborator

Summary

hermes update can no longer report success while its post-update fleet version check silently produced zero rows for a gateway that was demonstrably running. The success path swallowed probe failures and treated an empty snapshot as "nothing to check" — the fail-closed guard only existed on the restart-failure path, and on Windows the pause/resume phase populates none of the signals the naive guard would key on. Fixes #93406.

Changes

  • hermes_cli/update_cmd.py: empty fleet snapshot is treated as incomplete when ANY pre-update liveness signal fired — plan runtimes, pre-restart gateway pids, the Windows resume token's profiles/unmapped entries, or restarted_services/killed_pids (salvaged guard from fix(cli): treat empty fleet probe as incomplete when gateways were restarted (#93406) #93410 widened cross-platform)
  • The 2.0s settle window is gated on the same condition, so freshly resumed Windows gateways get their settle before the probe
  • Zero-rows-despite-expected-runtimes keying preserves 'unknown'-state rows passing

Validation

Scenario Before After
Empty snapshot, gateway was running exit 0, no rows INCOMPLETE (fail-closed)
Empty snapshot, nothing ran pre-update success success (unchanged)
Windows resume path guard never fired guard + settle window fire
Tests 11 new regressions (sabotage-proven), 107 passed

Salvaged from #93410 (@RelaxJonh), authorship preserved; guard gap identified live by @Halldrix on the PR.

Infographic

Post-update fleet check fails closed on empty results

RelaxJonh and others added 3 commits August 24, 2026 00:06
…started (#93406)

collect_fleet_versions() swallows every probe exception via
logger.debug() and returns whatever accumulated — which can be an
empty list.  print_fleet_version_matrix([]) returns False (no rows
to report), so the update exits 0 with "success" even though no
gateway was actually verified.

After the restart phase touches live gateways (restarted_services or
killed_pids is truthy), an empty fleet snapshot means verification
failed, not that everything is healthy.  Treat it as incomplete so
the receipt records "partial" and the exit code is 1.

Fixes #93406
…ness signals (#93406)

The #93410 guard keyed on (restarted_services or killed_pids), which never
fires on Windows: _pause_windows_gateways_for_update /
_resume_windows_gateways_after_update populate neither list, so a healthy
resumed Windows gateway still yielded zero fleet rows and exit 0.

Hoist the decision into _fleet_probe_expected_runtimes(), keyed on every
pre-update liveness signal:
- restarted_services / killed_pids (POSIX restart bookkeeping)
- _pre_restart_gateway_pids non-empty or None (unreadable pre-state,
  same fail-closed contract as _restart_phase_failure_is_incomplete, #78574)
- pre-update plan inventoried >=1 runtime
- Windows pause/resume token carries profiles or unmapped entries

Gate the 2.0s settle sleep on the same condition so a resumed Windows
gateway gets its settle window before the probe. The guard keys only on
zero-rows-despite-expected-runtimes; non-empty snapshots (including
'unknown'-state rows) are still judged solely by print_fleet_version_matrix.

Regression tests cover: empty snapshot + plan runtimes -> incomplete;
empty snapshot + genuinely idle -> success; Windows-resume token path ->
fail-closed + settle sleep wiring.

Builds on RelaxJonh's #93410. Fixes #93406
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/cli CLI entry point, hermes_cli/, setup wizard platform/windows Native Windows-specific behavior or breakage area/install-update Installer, updater, packaging, wheels, doctor sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Aug 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/install-update Installer, updater, packaging, wheels, doctor comp/cli CLI entry point, hermes_cli/, setup wizard P2 Medium — degraded but workaround exists platform/windows Native Windows-specific behavior or breakage sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows type/bug Something isn't working

Projects

None yet

3 participants