Skip to content

fix(profiles): detect a separate-process gateway in profile status - #51709

Merged
benbarclay merged 1 commit into
mainfrom
fix/profiles-gateway-status-runtime-fallback
Jun 24, 2026
Merged

fix(profiles): detect a separate-process gateway in profile status#51709
benbarclay merged 1 commit into
mainfrom
fix/profiles-gateway-status-runtime-fallback

Conversation

@benbarclay

Copy link
Copy Markdown
Collaborator

Infographic

gateway-status-fixed

Summary

The dashboard Profiles view shows "Gateway stopped" for a gateway that is in fact running, while the sidebar status strip and hermes gateway status (CLI) both correctly report it running. Reported on v0.17.0 with the gateway + dashboard running in a single Docker container:

$ hermes gateway status
✓ Gateway is running (PID: 134)
  (Running manually, not as a system service)

Root cause

Three liveness surfaces, three detection strengths — all reading the same gateway.pid under $HERMES_HOME:

Surface Detector Result
hermes gateway status (CLI) find_gateway_pids() — process-table scan ✅ running
Sidebar /api/status get_running_pid() + gateway_state.json PID fallback + health-URL probe ✅ running
Profiles view _check_gateway_running() = get_running_pid() only, no fallback ❌ stopped

get_running_pid() (gateway/status.py) short-circuits to None the moment the runtime lock (gateway.lock) doesn't register as held by the calling process — before it inspects the PID record. That's always the case when the reader is a separate process from the gateway (in the container the dashboard is its own s6 service), and also for any launch-service-managed gateway that left a fresh gateway_state.json but no live PID file. So the Profiles view alone reported the live gateway as stopped.

Fix

Give _check_gateway_running() the same fallback the sidebar already has (web_server.py:1854): after the pid-file/lock check misses, validate the PID recorded in that profile's gateway_state.json against the live process table via the existing get_runtime_status_running_pid().

read_runtime_status() gains an optional path argument so a specific profile's state file can be read without mutating the process-global HERMES_HOME — preserving the contextvar-based profile isolation the dashboard relies on, and avoiding the env-mutation race an earlier approach (#20488) would have introduced. The fallback is also strictly stronger than #20488's service_running-only check, which is always False in Docker (no systemd/launchd) and would still miss PID 134.

Backward compatible: every existing caller of read_runtime_status() passes no argument.

Tests

  • test_gateway_running_check_falls_back_to_runtime_state — live gateway, pid-file/lock check returns None, must still report running. Verified failing on baseline, passing with the fix.
  • test_gateway_running_check_runtime_state_stopped — a gateway_state.json with state stopped is never reported running, even with a live recorded PID.

tests/hermes_cli/test_profiles.py (140) + the gateway runtime-status suites all pass.


🎨 Infographic is decorative — details are illustrative, not a spec.

The dashboard Profiles view showed "Gateway stopped" for a gateway that
is in fact running — while the sidebar status strip and `hermes gateway
status` (CLI) both correctly showed it running. Reported on v0.17.0
running the gateway + dashboard in one Docker container.

Root cause: three liveness surfaces with three detection strengths, all
reading the same `gateway.pid`:

  - `hermes gateway status` -> find_gateway_pids() (process-table scan)
  - sidebar /api/status     -> get_running_pid() + gateway_state.json PID
                               fallback + health-URL probe
  - Profiles view           -> _check_gateway_running() = get_running_pid()
                               ONLY, no fallback

`get_running_pid()` short-circuits to None the moment the runtime lock
(`gateway.lock`) doesn't register as held by the *calling* process —
which is always true when the reader is a separate process from the
gateway (the dashboard is its own s6 service in the container), and also
for any launch-service-managed gateway that left a fresh
`gateway_state.json` but no live PID file. So the Profiles view alone
reported the live gateway as stopped.

Fix: give _check_gateway_running the same fallback the sidebar already
has — after the pid-file/lock check misses, validate the PID recorded in
that profile's gateway_state.json against the live process table via the
existing get_runtime_status_running_pid(). read_runtime_status() gains an
optional path arg so a profile's state file can be read without mutating
the process-global HERMES_HOME (preserving the contextvar-based profile
isolation the dashboard relies on). Backward compatible: every existing
caller passes no argument.

Tests: a regression test that fails pre-fix (live gateway, lock check
returns None -> must still report running) and a guard test that a
'stopped' state file is never reported running even with a live PID.
@github-actions

Copy link
Copy Markdown
Contributor

🔎 Lint report: fix/profiles-gateway-status-runtime-fallback vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 11171 on HEAD, 11171 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 5900 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

@alt-glitch alt-glitch added type/bug Something isn't working comp/cli CLI entry point, hermes_cli/, setup wizard comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have labels Jun 24, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related: this addresses open issue #20254 (profile list/show reports Gateway stopped). Same profile-status detection gap; the dashboard-side surface was handled by merged #47843.

@benbarclay
benbarclay merged commit 31bced1 into main Jun 24, 2026
27 checks passed
@benbarclay
benbarclay deleted the fix/profiles-gateway-status-runtime-fallback branch June 24, 2026 06:36
justemu added a commit to justemu/hermes-agent that referenced this pull request Jul 10, 2026
Systemd-managed gateway processes show just "hermes" in /proc/PID/cmdline
rather than "hermes gateway run".  The _gateway_command_subcommand parser
failed to identify these processes because it only accepted an explicit
"gateway" token before the subcommand.

When has_gateway_entry is True but no "gateway" subcommand is found, and
the only remaining token is the bare "hermes"/"hermes.exe" entry-point
binary, treat it as a gateway run process.  The existing
_command_line_belongs_to_profile guard (which validates HERMES_HOME in
/proc/PID/environ) prevents false positives.

This fixes the L2 fallback in _check_gateway_running (introduced in
PR NousResearch#51709) that silently failed for systemd-managed profile gateways,
causing `hermes profile list/show` to report "Gateway: stopped" for
actually-running gateways that lacked a gateway.pid file.

Closes NousResearch#20254
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants