Harden macOS gateway service recovery

[developer1-ckkccom]

Summary

• harden macOS launchd_start() recovery by selecting the correct launchd domain for background sessions and re-bootstrapping unloaded jobs before kickstart
• add hermes doctor guidance for macOS LaunchAgent installs under non-console users, which is the failure mode behind headless/service-account deployments
• document the difference between macOS LaunchAgents and LaunchDaemons, including the recommended fallback for headless installs
• make the Termux doctor path consistently treat Docker as unavailable instead of respecting ambient TERMINAL_ENV=docker

Testing

• venv/bin/python -m pytest tests/hermes_cli/test_gateway_service.py tests/hermes_cli/test_update_gateway_restart.py tests/hermes_cli/test_doctor.py -q

Context

This packages the macOS gateway outage/recovery work after reproducing a broken launchd topology on a non-console account. The code fix handles unloaded jobs and background launchd domains; the doctor/docs changes make the unsupported service topology explicit instead of silently failing at runtime.

[alt-glitch]

Supersedes closed #13553 (same title/author). Related to #3318 (macOS gateway start fails with missing plist).

[mu0111]

I tested this exact failure mode on a macOS Background launchd session. The gui/<uid> -> user/<uid> fallback is necessary, but it was not sufficient on this machine: bootstrapping the plist into user/<uid> failed until the plist also declared it can load in the Background session type.

Observed locally:

launchctl managername -> Background
launchctl print gui/501 -> fails / no GUI bootstrap domain
launchctl bootstrap user/501 ~/Library/LaunchAgents/ai.hermes.gateway.plist -> exit 5 without LimitLoadToSessionType
launchctl kickstart user/501/ai.hermes.gateway -> exit 125: Domain does not support specified action

Adding this to the generated plist fixed it:

<key>LimitLoadToSessionType</key>
<string>Background</string>

Because maintainerCanModify is false on this PR, I cannot push directly to the branch. Suggested patch below. It also moves a few _pin_launchd_manager() calls before plist generation; otherwise those tests can fail when run from a real Background session because the fixture plist is generated for the host session, then compared against an Aqua-pinned generated plist.

Validation on /tmp/hermes-pr13560:

python -m pytest tests/hermes_cli/test_gateway_service.py -q
# 101 passed in 2.10s

python -m pytest tests/hermes_cli/test_update_gateway_restart.py tests/hermes_cli/test_doctor.py -q
# 62 passed, 6 warnings in 3.98s

Patch:

diff --git a/hermes_cli/gateway.py b/hermes_cli/gateway.py
index b648358f8..65cf83f76 100644
--- a/hermes_cli/gateway.py
+++ b/hermes_cli/gateway.py
@@ -1689,6 +1689,16 @@ def generate_launchd_plist() -> str:
     log_dir.mkdir(parents=True, exist_ok=True)
     label = get_launchd_label()
     profile_arg = _profile_arg(hermes_home)
+    limit_load_to_session_type = ""
+    if (_launchd_managername() or "").strip().lower() == "background":
+        # A plist bootstrapped into user/<uid> from a Background launchd
+        # session must be explicitly loadable in that session type. Without
+        # this, `launchctl bootstrap user/<uid> ...` can fail with exit 5 and
+        # `launchctl kickstart user/<uid>/<label>` follows with exit 125.
+        limit_load_to_session_type = """
+    <key>LimitLoadToSessionType</key>
+    <string>Background</string>
+"""
     # Build a sane PATH for the launchd plist.  launchd provides only a
     # minimal default (/usr/bin:/bin:/usr/sbin:/sbin) which misses Homebrew,
     # nvm, cargo, etc.  We prepend venv/bin and node_modules/.bin (matching
@@ -1732,7 +1742,7 @@ def generate_launchd_plist() -> str:
 <dict>
     <key>Label</key>
     <string>{label}</string>
-
+{limit_load_to_session_type}
     <key>ProgramArguments</key>
     <array>
         {prog_args_xml}
diff --git a/tests/hermes_cli/test_gateway_service.py b/tests/hermes_cli/test_gateway_service.py
index 711f61f05..7bfeef394 100644
--- a/tests/hermes_cli/test_gateway_service.py
+++ b/tests/hermes_cli/test_gateway_service.py
@@ -221,11 +221,11 @@ class TestLaunchdServiceRecovery:
         ]
 
     def test_launchd_start_reloads_unloaded_job_and_retries(self, tmp_path, monkeypatch):
+        _pin_launchd_manager(monkeypatch)
         plist_path = tmp_path / "ai.hermes.gateway.plist"
         plist_path.write_text(gateway_cli.generate_launchd_plist(), encoding="utf-8")
         label = gateway_cli.get_launchd_label()
 
-        _pin_launchd_manager(monkeypatch)
         calls = []
         domain = gateway_cli._launchd_domain()
         target = f"{domain}/{label}"
@@ -250,11 +250,11 @@ class TestLaunchdServiceRecovery:
 
     def test_launchd_start_reloads_on_kickstart_exit_code_113(self, tmp_path, monkeypatch):
         """Exit code 113 (\"Could not find service\") should also trigger bootstrap recovery."""
+        _pin_launchd_manager(monkeypatch)
         plist_path = tmp_path / "ai.hermes.gateway.plist"
         plist_path.write_text(gateway_cli.generate_launchd_plist(), encoding="utf-8")
         label = gateway_cli.get_launchd_label()
 
-        _pin_launchd_manager(monkeypatch)
         calls = []
         domain = gateway_cli._launchd_domain()
         target = f"{domain}/{label}"
@@ -278,11 +278,11 @@ class TestLaunchdServiceRecovery:
         ]
 
     def test_launchd_start_bootstraps_before_kickstart_when_label_is_unloaded(self, tmp_path, monkeypatch):
+        _pin_launchd_manager(monkeypatch)
         plist_path = tmp_path / "ai.hermes.gateway.plist"
         plist_path.write_text(gateway_cli.generate_launchd_plist(), encoding="utf-8")
         label = gateway_cli.get_launchd_label()
 
-        _pin_launchd_manager(monkeypatch)
         calls = []
         domain = gateway_cli._launchd_domain()
         target = f"{domain}/{label}"
@@ -425,6 +425,21 @@ class TestLaunchdServiceRecovery:
 
         assert gateway_cli._launchd_domain() == f"gui/{os.getuid()}"
 
+    def test_launchd_plist_limits_load_to_background_session_when_needed(self, monkeypatch):
+        _pin_launchd_manager(monkeypatch, "Background")
+
+        plist = gateway_cli.generate_launchd_plist()
+
+        assert "<key>LimitLoadToSessionType</key>" in plist
+        assert "<string>Background</string>" in plist
+
+    def test_launchd_plist_does_not_limit_load_session_for_aqua(self, monkeypatch):
+        _pin_launchd_manager(monkeypatch, "Aqua")
+
+        plist = gateway_cli.generate_launchd_plist()
+
+        assert "<key>LimitLoadToSessionType</key>" not in plist
+
     def test_launchd_status_reports_local_stale_plist_when_unloaded(self, tmp_path, monkeypatch, capsys):
         plist_path = tmp_path / "ai.hermes.gateway.plist"
         plist_path.write_text("<plist>old content</plist>", encoding="utf-8")

[teknium1]

Thanks for the macOS recovery work. An automated hermes-sweeper review found that the requested behavior has since shipped on main, including the Background-session plist detail raised in the later discussion.

• 3606307339ee2003d59eb9fcf04b95ea80170b3a (fix(gateway): use user launchd domain + Background session, detached fallback (macOS 26)) probes the actual launchd domain and supports recovery from unloaded jobs.
• hermes_cli/gateway.py:3953 emits LimitLoadToSessionType for both Aqua and Background; hermes_cli/gateway.py:4178 regenerates/re-bootstraps and retries the service start path.
• tests/hermes_cli/test_gateway_service.py:1399 and :2593 cover Background-domain selection and the plist session types.
• The implementation is contained in release v2026.6.19.

This is an automated hermes-sweeper review.