Skip to content

fix(gateway): wait for launchd restart readiness in-band (#56524) - #735

Open
hashbender wants to merge 1 commit into
mainfrom
mirror/pr-56908
Open

fix(gateway): wait for launchd restart readiness in-band (#56524)#735
hashbender wants to merge 1 commit into
mainfrom
mirror/pr-56908

Conversation

@hashbender

Copy link
Copy Markdown
Owner

What does this PR do?

Fixes the macOS launchd in-band restart/update path so Hermes does not report success before a fresh gateway process is actually runtime-ready.

The old self-restart path returned immediately after sending SIGUSR1, which let hermes update / hermes gateway restart complete while the old gateway was still draining and before launchd had confirmed a healthy replacement. In the gateway-hosted terminal path, that also left fallback launchctl kickstart -k vulnerable to being killed with the old gateway process group.

This PR waits for the old PID to exit, waits for a fresh launchd-managed PID to reach gateway_state=running, and uses a detached watcher plus a shared launchd recovery helper so forced fallback survives in-band teardown.

Related Issue

Fixes NousResearch#56524

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • hermes_cli/gateway.py
    • add _wait_for_pid_exit() and _wait_for_launchd_service_restart() so launchd restart success means a fresh runtime-ready gateway PID, not just a signal request
    • add a detached launchd restart watcher for in-band self-restart / update flows so fallback restart survives gateway teardown
    • unify launchd kickstart / unloaded-job recovery / detached fallback behind _kickstart_launchd_service_and_wait()
    • route launchd_restart() through the shared recovery path and only clear the unsupported marker after runtime-ready recovery
  • tests/hermes_cli/test_gateway_service.py
    • cover graceful self-restart watcher flow
    • cover drain-timeout fallback to launchctl kickstart -k
    • cover fast relaunch with no None PID gap
    • cover readiness timeout log guidance
    • cover unloaded-job recovery waiting for runtime-ready after bootout -> bootstrap -> kickstart

How to Test

  1. Reproduce the old behavior on the clean baseline for #56524:
    • self-restart requests return ✓ Service restart requested immediately after SIGUSR1, before launchd runtime-ready confirmation
  2. Verify the targeted launchd restart coverage:
    • ./scripts/run_tests.sh tests/hermes_cli/test_gateway_service.py -k 'launchd_restart_drains_running_gateway_before_kickstart or launchd_restart_self_request or wait_for_launchd_service_restart_timeout_reports_logs or launchd_restart_boots_out_stale_registration_before_bootstrap or launchd_restart_falls_back_to_detached_on_error_5'
    • .venv/bin/python -m pytest -q tests/hermes_cli/test_gateway_service.py -k 'launchd_restart_self_request or wait_for_launchd_service_restart_timeout_reports_logs'
  3. Verify static checks:
    • ruff check hermes_cli/gateway.py tests/hermes_cli/test_gateway_service.py
    • git diff --check
  4. Optional broader check:
    • .venv/bin/python -m pytest -q tests/hermes_cli/test_gateway_service.py
    • On this macOS host it still ends at 182 passed, 6 failed; the 6 failures are the pre-existing user-systemd baseline failures, unrelated to this patch

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: macOS 26.3 / launchd

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Screenshots / Logs

N/A


Mirror-of: NousResearch#56908
NousResearch#56908

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Telegram in-band hermes update can leave launchd gateway offline after drain timeout

1 participant