fix(gateway): report launchd supervision accurately

[americodias]

What does this PR do?

Fixes two misleading macOS gateway diagnostics:

1. hermes gateway status reported a live gateway as “Running manually” when it was actually owned by a system LaunchDaemon using an operator-defined label such as com.example.hermes-gateway.
2. Shutdown forensics labelled any PID-1 parent as under_systemd=yes, even though PID 1 is launchd on macOS and may be another init/supervisor on Linux.

The status path now correlates discovered gateway PIDs with launchctl list, so externally managed launchd jobs are reported truthfully without suggesting a duplicate service install. Shutdown logs now emit supervisor=systemd|launchd|pid1|none; the legacy structured under_systemd field remains but is true only when explicit systemd markers are present.

Related Issue

No issue filed. I searched open issues and PRs for launchd status manual system service, under_systemd launchd, and the exact "Running manually" launchd symptom; no duplicate was found.

Related but non-duplicative: #65177 changes restart lifecycle ownership. It does not address external launchd labels or shutdown supervisor labelling.

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• hermes_cli/gateway.py
  • add optional service-name metadata to the runtime snapshot
  • correlate gateway PIDs with all launchd jobs, not only Hermes' generated label
  • report operator-managed launchd services as supervised and suppress duplicate-install guidance
• gateway/shutdown_forensics.py
  • distinguish systemd, launchd, generic PID 1, and unsupervised processes
  • emit supervisor=... instead of the misleading under_systemd=yes|no log token
• Add regression coverage for custom launchd labels, status rendering, and macOS PID-1 classification.

How to Test

1. Run the focused regression set:

   pytest tests/hermes_cli/test_gateway_service.py::TestLaunchdServiceRecovery \
     tests/hermes_cli/test_gateway_service.py::TestGatewaySystemServiceRouting::test_gateway_status_reports_external_launchd_supervisor \
     tests/gateway/test_shutdown_forensics.py -o 'addopts=' -q

2. On macOS, run a gateway from a system LaunchDaemon with a non-generated label, then run hermes gateway status from a shell.
3. Confirm status names the launchd service and does not say “Running manually” or suggest hermes gateway install.
4. Trigger/format a shutdown context and confirm the log contains supervisor=launchd, not under_systemd=yes.

Verification performed after rebasing onto current upstream main:

• Focused regression set: 68 passed
• Broader affected files: 220 passed, 4 deselected (four systemd restart tests invoke Linux user-D-Bus preflight and are incompatible with the macOS host)
• Ruff focused check: passed
• Python compile check: passed
• git diff --check: passed
• Direct behavior contract probe: passed

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass — focused and affected-file suites were run; full repository suite was not run locally
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: macOS

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — docstrings updated; no user-facing docs change required
• I've updated cli-config.yaml.example if I added/changed config keys — N/A, no config changes
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A, no architecture/workflow change
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide
• I've updated tool descriptions/schemas if I changed tool behavior — N/A, no tool schema changes

Screenshots / Logs

Observed before:

✓ Gateway is running (PID: 4242)
  (Running manually, not as a system service)

Observed after:

✓ Gateway is running, supervised by launchd (PID: 4242)
  Service: com.example.hermes-gateway
  (Externally managed launchd service; no duplicate install needed)

Shutdown formatter after:

signal=SIGTERM supervisor=launchd parent_pid=1 ...

[teknium1]

Thanks for the focused gateway diagnostics fix. The two reported premises remain on current main: hermes_cli/gateway.py:7263-7266 reports a discovered PID as manual when no generated plist exists, and gateway/shutdown_forensics.py:134-143 treats any PPID-1 process as systemd.

Problems

• hermes_cli/gateway.py:1314 adds subprocess.run(..., text=True) without an explicit encoding. scripts/check-windows-footguns.py:328-348 flags that pattern, and commit c89481db5e standardized encoding='utf-8', errors='replace' for production text-mode subprocess calls.

Suggested changes

• Add encoding="utf-8", errors="replace" to the new launchctl probe, matching the adjacent launchctl call at hermes_cli/gateway.py:1335-1340 on the PR head.

Automated hermes-sweeper review.

[teknium1]

Please add encoding="utf-8", errors="replace" here. scripts/check-windows-footguns.py:328-348 flags subprocess calls using text=True without encoding=, and this would regress the production-wide UTF-8 policy from c89481db5e.

[americodias]

Addressed in 9e0a5ca — the new launchctl list probe now passes encoding="utf-8", errors="replace", matching the c89481d convention. Ran tests/hermes_cli/test_gateway_service.py + tests/gateway/test_shutdown_forensics.py: no new failures (the 4 pre-existing TestGatewaySystemServiceRouting systemd failures reproduce identically on the untouched PR head on macOS).

[GottZ]

This was generated by AI during triage.

Summary

Twenty-three PRs address or reference four shutdown-forensics clusters: macOS supervisor detection, systemd manager-scope timeout false positives, Windows planned-stop labels, and portable shutdown diagnostics; several also bundle unrelated voice, installer, auxiliary-client, or WhatsApp changes. The focused diffs either constrain or replace the PPID-1 heuristic, select or validate the owning systemd manager, relabel marker-driven stops, or remove the GNU timeout dependency.

Related pull requests

• fix: align auxiliary defaults and timeout handling #23605 related — (+156/-41) — n/a: Replaces GNU timeout with a Python wrapper, but also bundles title-generation and superseded Codex-stream changes; the keep_open review identifies the portable diagnostic wrapper and provider-default temperature as the concrete salvage path.
• fix: under_systemd incorrectly True on macOS (launchd PID 1, not systemd) #25511 [closed] fixes — (+49/-19) — n/a: Restricts PPID-1 systemd detection to Linux but tests part of the behavior through literal expressions rather than the runtime path; closed and superseded by fix(gateway): restrict ppid==1 systemd detection to Linux only #25525.
• fix: suppress SyntaxWarning in _voice_processing by moving return out… #25518 [closed] related — (+23/-22) — n/a: Moves voice no-speech handling out of a finally block and does not address the shutdown-forensics causes in this complex; closed as a duplicate of fix(voice): remove return-in-finally to fix Python 3.14+ SyntaxWarning #21100 per the contributor discussion.
• fix(gateway): restrict ppid==1 systemd detection to Linux only #25525 best fix — (+25/-1) — n/a: Directly gates the PPID-1 heuristic on Linux and adds runtime monkeypatch tests for Darwin, Linux, and INVOCATION_ID; it is the recorded best existing fix for Bug: shutdown_forensics incorrectly sets under_systemd=True on macOS (ppid==1 is launchd, not systemd) #25508/shutdown_forensics incorrectly reports under_systemd=yes on macOS (launchd PID 1, not systemd) #25510 and its keep_open review reports high salvageability.
• fix(gateway): restrict ppid==1 systemd detection to Linux only #25566 [closed] fixes — (+25/-1) — n/a: Duplicates fix(gateway): restrict ppid==1 systemd detection to Linux only #25525's platform gate and tests while bundling an unrelated .env permission change; closed as duplicate of fix(gateway): restrict ppid==1 systemd detection to Linux only #25525.
• fix(cli): suppress SyntaxWarning in _voice_processing by moving return out of finally #25570 [closed] partial — (+39/-15) — n/a: Bundles the fix(gateway): restrict ppid==1 systemd detection to Linux only #25525-style platform gate with unrelated installer and voice changes, and the voice hunk retains incorrect indentation identified by a blocking contributor comment; closed and not a consolidation target.
• fix(shutdown_forensics): only treat ppid==1 as systemd on Linux (salvage #25511) #25687 [closed] fixes — (+31/-7) — n/a: Implements the same Linux-only PPID-1 gate with runtime tests but also changes release attribution metadata; closed by its contributor author in favor of fix(gateway): restrict ppid==1 systemd detection to Linux only #25525.
• fix(gateway): skip systemd timing check for non-loaded units #36766 fixes — (+18/-2) — n/a: Adds a LoadState check so a nonexistent user unit falls through to the system manager, but also restores the obsolete --replace command and lacks regression coverage; its keep_open review supports salvaging only the forensics hunk plus a two-manager test.
• fix(gateway): ignore unloaded systemd units in forensics #37324 best fix — (+123/-11) — n/a: Requires LoadState=loaded, falls through from a phantom user unit to the loaded system unit, and tests loaded, not-found, other non-loaded, and missing-state responses; it is the recorded best existing LoadState-gate fix for bug(diagnostics): check_systemd_timing_alignment false-positive — systemctl show returns default 90s for nonexistent --user unit #36755/shutdown_forensics: false-positive 'Stale systemd unit' warning when unit exists in system scope only #61003/False-positive "Stale systemd unit" warning: timing check reads synthesized defaults from systemctl --user for system-scope units #63158.
• fix(gateway): make shutdown diagnostics and 2 tests work on macOS #41958 [closed] related — (+28/-3) — n/a: Targets macOS shutdown-diagnostic portability plus two macOS test failures; closed by its author in favor of earlier portability PRs fix: allow active tempdir writes in file tools #13733, fix(file-tools): allow writes to platform temp directory on macOS #31021, and fix(file-tools): allow macOS var folders temp writes #32344.
• fix(gateway): prefer systemd scope for timeout check #54396 best fix — (+61/-4) — n/a: Derives manager preference from /proc/self/cgroup and tests that /system.slice selects the real system timeout before a stale user value; it is a recorded best existing fix for shutdown_forensics: false-positive 'Stale systemd unit' warning when unit exists in system scope only #61003 and remains relevant at rebased head 65e1745148f20cfc585ed2460e9a666a1f5ab355.
• Fix systemd timeout check for missing user unit #57901 [closed] fixes — (+46/-2) — n/a: Adds a LoadState gate and a user-not-found-to-system fallback regression test; closed as a duplicate of the LoadState cluster, with fix(gateway): ignore unloaded systemd units in forensics #37324 carrying broader loaded-state coverage.
• fix(gateway): don't capture full process argv in shutdown forensics #59929 related — (+505/-31) — n/a: Removes persisted argv from shutdown diagnostics and creates logs as 0600, but bundles extensive unrelated WhatsApp webhook work and does not tighten permissions on an existing 0644 log as required by its keep_open review.
• fix(gateway): honor systemd unit scope in shutdown check #61017 fixes — (+86/-5) — n/a: Selects only the manager indicated by /system.slice or /user.slice and tests both paths; it duplicates the earlier cgroup-scope mechanism in fix(gateway): prefer systemd scope for timeout check #54396 despite its favorable keep_open review.
• fix(gateway): query systemd timeout in owning scope #61532 [closed] fixes — (+68/-4) — n/a: Implements the same cgroup-owned single-manager lookup as fix(gateway): prefer systemd scope for timeout check #54396/fix(gateway): honor systemd unit scope in shutdown check #61017 with a system-scope regression test; closed in favor of fix(gateway): prefer systemd scope for timeout check #54396.
• fix(gateway): label Windows planned stops instead of reporting signal=UNKNOWN #61615 best fix — (+105/-0) — n/a: Relabels marker-driven signal=None shutdowns as PLANNED_STOP or PLANNED_TAKEOVER only after marker consumption while preserving real signals and unexplained UNKNOWN; it is the recorded best existing fix for [Windows] Gateway shutdown reports misleading 'signal=UNKNOWN' on normal stop #61596 and has a high-salvage keep_open review.
• fix(gateway): query owning systemd manager for drain timing #64029 best fix — (+100/-26) — n/a: Combines cgroup-owner manager selection with a strict LoadState=loaded requirement and tests system, user, and not-found cases; it is a recorded best existing fix for bug(diagnostics): check_systemd_timing_alignment false-positive — systemctl show returns default 90s for nonexistent --user unit #36755/shutdown_forensics: false-positive 'Stale systemd unit' warning when unit exists in system scope only #61003/False-positive "Stale systemd unit" warning: timing check reads synthesized defaults from systemctl --user for system-scope units #63158, with its keep_open review treating overlap with fix(gateway): prefer systemd scope for timeout check #54396 as a salvage choice rather than a correctness defect.
• fix: make shutdown diagnostics portable without coreutils timeout #64515 related — (+48/-1) — n/a: Replaces GNU timeout with a detached Python supervisor that kills the diagnostic process group at deadline and adds a launcher regression test; its keep_open review reports the focused repair as mechanically salvageable.
• Fix tests/gateway/test_shutdown_forensics.py on macOS (1 failing test) #67708 related — (+16/-1) — n/a: Rewrites only the test-side Popen command to emulate timeout, leaving production's GNU-only launcher unchanged; the contributor review says this masks rather than exercises the macOS failure, so salvage requires pairing the test with a production portability fix.
• fix(gateway): report launchd supervision accurately #67744 partial — (+213/-9) — n/a: Adds explicit systemd/launchd/generic-PID1 supervisor reporting and detects externally labelled launchd jobs by matching launchctl list PIDs, while overlapping fix(gateway): restrict ppid==1 systemd detection to Linux only #25525's exact macOS PPID false-positive fix; the visible encoding objection is addressed in the shown diff with encoding="utf-8", errors="replace".
• fix: report launchd supervision accurately in gateway status and shutdown forensics #71617 [closed] duplicate — (+211/-9) — n/a: Covers the same external launchd status and shutdown-supervisor behavior as fix(gateway): report launchd supervision accurately #67744; closed explicitly as a duplicate of fix(gateway): report launchd supervision accurately #67744, where the encoding feedback was addressed.
• fix(gateway): don't accept systemd's not-found default for TimeoutStopUSec #72943 [closed] fixes — (+140/-10) — n/a: Adds LoadState-aware timeout lookup and useful manager-fallback tests, but accepts states other than not-found; closed after contributor review required the stricter LoadState == "loaded" guard already present in fix(gateway): ignore unloaded systemd units in forensics #37324/fix(gateway): query owning systemd manager for drain timing #64029.
• fix(gateway): make shutdown diagnostics portable on macOS #74011 related — (+40/-12) — n/a: Replaces GNU timeout with a detached Python process-group supervisor and adds Darwin-native process/load commands; its keep_open review finds no blocking defect and distinguishes its Darwin snapshot scope from fix: make shutdown diagnostics portable without coreutils timeout #64515's narrower launcher repair.

Duplicates

#25511, #25566, and #25687 duplicate #25525; the shutdown-forensics portion of #25570 also overlaps #25525. #36766, #37324, #57901, and #72943 share the LoadState-gate mechanism, while #61017 and #61532 duplicate #54396's cgroup-scope mechanism; #64029 combines both mechanisms. #71617 duplicates #67744. #64515 and #74011 overlap on the Python timeout supervisor, while #74011 additionally adds Darwin-native diagnostics; #67708 is only a test-side shim for that same portability problem.

Suggested consolidation

Author action on #67744: rebase onto main or split out the uniquely salvageable external-label launchd status detection and supervisor taxonomy, retain the now-addressed UTF-8 subprocess fix, and avoid duplicating #25525, the recorded best existing fix for the exact macOS PPID-1 issue. Keep the distinct recorded best-fix paths visible—#37324 for the strict LoadState fallback, #54396 for cgroup-scope selection, #64029 for their combined hardening, and #61615 for Windows planned-stop labels—while closing or leaving closed their explicit duplicates; coordinate #64515/#74011 around one portable supervisor, preserving #74011's Darwin-native diagnostics.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I25508(["issue #25508 (closed)"])
    I25510(["issue #25510 (closed)"])
    subgraph Dup67744 ["PRs duplicating each other"]
        P67744["PR #67744 (open)"]
        P71617["PR #71617 (closed)"]
    end
    P67744 -.->|partial| I25508
    P67744 -.->|partial| I25510
    class I25508 closed
    class I25510 closed
    class P67744 open
    class P71617 closed
    class P67744 target
    click I25508 "https://github.com/NousResearch/hermes-agent/issues/25508"
    click I25510 "https://github.com/NousResearch/hermes-agent/issues/25510"
    click P67744 "https://github.com/NousResearch/hermes-agent/pull/67744"
    click P71617 "https://github.com/NousResearch/hermes-agent/pull/71617"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 23 pull requests and 6 issues in this complex. Diffs were read for 20 of 23 PRs (rest unavailable); Assessment working set: 132 kB of PR diffs, 62 kB of issue/PR text, 29 kB of discussion (45 comments), 65 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

[andrexibiza]

Rebase-on-behalf delivery — this PR's branch was rebased onto current main (9d6c5a920c7).

Rebased branch: andrexibiza:rebase-on-behalf/67744 (commits a0bcfc0d326 + 8664b896681, original authorship preserved)

Conflict resolution (3 files):

• hermes_cli/gateway.py — merged cleanly (the explicit-encoding launchctl probe + supervision reporting delta applied; 89 additions).
• tests/gateway/test_shutdown_forensics.py — 2 pure additions from this PR (parent-summary, formatter coverage); took this PR's side; main's 10 tests all preserved (merged file has 32).
• tests/hermes_cli/test_gateway_service.py — 1 positional conflict (HEAD empty side; this PR adds 8 status/routing tests incl. termux, planned-restart recovery, external-launchd supervisor); took this PR's side; verified 0 main tests lost (merged file 89 vs main's 79 — superset).

To adopt (author):

git fetch https://github.com/andrexibiza/hermes-agent.git rebase-on-behalf/67744
git checkout <your-branch-name>
git reset --hard FETCH_HEAD
git push --force-with-lease origin <your-branch-name>

py_compile verified; test-coverage superset verified (no main regressions). This unblocks the gateway R1 extraction window (hunk gate).

[andrexibiza]

Interlock binding — this PR addresses the gateway.py extraction window of EPIC #78647 (shard #78640). Rebase-on-behalf delivery: andrexibiza:rebase-on-behalf/67744.