Skip to content

fix(gateway): never report a copied profile dir as running (stale gateway.pid / gateway_state.json) - #119772

Closed
b2089766906-droid wants to merge 2 commits into
NousResearch:mainfrom
b2089766906-droid:fix/gateway-liveness-copied-profile-home
Closed

b2089766906-droid wants to merge 2 commits into
NousResearch:mainfrom
b2089766906-droid:fix/gateway-liveness-copied-profile-home

Conversation

@b2089766906-droid

Copy link
Copy Markdown

What does this PR do?

A profile directory can be copied wholesale to another HERMES_HOME (sandbox injection, restore-from-backup, cloning a profile). The copy carries that home's gateway.pid and gateway_state.json along, and the liveness ladder believed them: rung 1 trusts the PID in profile_dir/gateway.pid, rung 3's get_runtime_status_running_pid() re-reads the process home's gateway_state.json when the scoped read finds nothing, and rung 4 matches the live multiplexer by profile name — which a copied directory keeps. Result: hermes -p X status / the dashboard report a phantom gateway as running (usually with the production gateway's PID) while hermes cron / gateway list say the opposite on the same machine.

The repo already has the right predicate for "this record belongs to another home" — recorded_gateway_home_conflicts() (line 245) — but it only served the cross-profile kill guard (its only call site, line 1918). This PR wires it into the liveness ladder and adds the layout premise rung 4 was implicitly assuming.

Symptom

A gateway appears "running" for a profile that has no gateway process, with a PID owned by a different HERMES_HOME. The dashboard and the CLI contradict each other on the same page load; gateway stop on that profile then refuses or targets a foreign PID.

Minimal reproduction (real output from this machine)

import json, shutil, tempfile
from pathlib import Path
from gateway import status
import hermes_constants

real_root = Path(hermes_constants.get_default_hermes_root())
sandbox = Path(tempfile.mkdtemp(prefix="sandbox-home-"))
prof = sandbox / "profiles" / "eagle"          # a "copied profile dir"
prof.mkdir(parents=True)
shutil.copyfile(real_root / "gateway.pid", prof / "gateway.pid")   # carries ANOTHER home's record

print(status.resolve_gateway_liveness(profile_dir=prof))

On main (false green — the PID belongs to the production gateway of another home):

copied record: pid=32068 hermes_home=C:\Users\20897\AppData\Local\hermes
SANDBOX HOME  ...\Temp\sandbox-home-zz3qwqsl\profiles\eagle -> running=True pid=32068 source=runtime_status

With this PR:

SANDBOX HOME  ...\Temp\sandbox-home-fw0qoodg\profiles\eagle -> running=False pid=None source=none

Reverse check (must not be wounded — a genuinely served profile under the real default root), unchanged before and after:

REAL SERVED   C:\Users\20897\AppData\Local\hermes\profiles\eagle -> running=True pid=32068 source=multiplexer

Root cause (line level), all in gateway/status.py

1328  def resolve_gateway_liveness(...):
1368      pid = guarded(_pid_probe, profile_dir / "gateway.pid") if scoped else guarded(_pid_probe)
1369      if pid is not None:
1370          return GatewayLiveness(running=True, pid=pid, source="pid")      # rung 1: trusts the file
...
1384      runtime_pid = guarded(_runtime_pid_probe, runtime, **probe_kwargs)   # rung 3
1395      served = guarded(multiplexer_liveness_for_profile, own_home)          # rung 4: matches by NAME
1406  def get_runtime_status_running_pid(runtime=None, *, expected_home=None):
1413      payload = runtime if runtime is not None else read_runtime_status()   # None -> the PROCESS home's file

and the name-based matching both rung 3 (_host_gateway_serves_home, line 643) and rung 4 (multiplexer_liveness_for_profile, line 1259) rely on: profile_name_for_home(profile_dir) returns "eagle" for a copied .../profiles/eagle just as it does for the real one.

Related Issue

No existing issue. Searched resolve_gateway_liveness / "gateway liveness profile copy": the neighbours are different shapes — #101487 (a multiplexed secondary profile showing as unreachable) and #117777 (a dead runtime PID must be marked stale) — neither covers "the record itself belongs to another home".

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

gateway/status.py:

  • new _recorded_pid_home_conflict(profile_dir) — reads gateway.pid and defers to the existing recorded_gateway_home_conflicts(..., expected_home=profile_dir) (legacy record without hermes_home proves nothing → False; comparison failure → fail closed → True);
  • new _profile_dir_is_default_root_profile(profile_dir) — is this really <default root>/profiles/<name> (the pooled layout rung 4 assumes)?
  • rung 1: a record naming another home no longer ends the ladder with a green; the ladder falls through;
  • rung 3: same guard for a copied gateway_state.json, and a scoped read that found no record is handed to the PID probe as an empty record instead of None, so get_runtime_status_running_pid() cannot silently fall back to the process home's gateway_state.json;
  • rung 4: only applies when the pooled layout premise holds; otherwise the ladder returns running=False rather than matching a profile name across homes.

tests/gateway/test_gateway_liveness_home_guard.py — new file, 4 tests (3 red on main).

Note for reviewers: the rung-3 "empty record" line came out of building the end-to-end repro above — without it the ladder still went green through the default home's record, so the fix would not have fixed the symptom. Happy to split it into its own PR if you prefer smaller commits.

How to Test

  1. Run the reproduction above: main = running=True with a foreign PID, this branch = running=False. Then check a genuinely served profile (<default root>/profiles/<name> under a live hermes --profile X serve) still reads running=True, source=multiplexer.
  2. New tests on pristine main (patch reverted) — 3 of 4 fail, each on a different rung:
$ python -m pytest -q tests/agent/test_pre_compress_checkpoint_visibility.py tests/gateway/test_gateway_liveness_home_guard.py
FAILED tests/gateway/test_gateway_liveness_home_guard.py::test_identity_files_recorded_for_another_home_never_report_running
FAILED tests/gateway/test_gateway_liveness_home_guard.py::test_a_profile_dir_outside_the_default_root_never_takes_the_multiplexer_rung
FAILED tests/gateway/test_gateway_liveness_home_guard.py::test_a_scoped_read_that_found_nothing_does_not_borrow_the_process_home_record
5 failed, 1 passed in 15.97s
  1. Same command with this PR applied:
$ python -m pytest -q tests/gateway/test_gateway_liveness_home_guard.py
4 passed in 12.59s
  1. Regression (liveness consumers: status, cron profile gate, multiplexer/served-profile records, process identity), identical result on pristine main and on this PR:
$ python -m pytest -q tests/gateway/test_status.py tests/gateway/test_cron_profile_gate.py \
    tests/hermes_cli/test_gateway_multiplex_served_record.py tests/hermes_cli/test_served_profile_mirror_platforms.py \
    tests/hermes_cli/test_pooled_served_profile_backend_unscoped.py tests/hermes_cli/test_process_identity_canonical_matchers.py \
    tests/agent/test_pre_compress_checkpoint_contract.py tests/agent/test_pre_compress_memory_context_handoff.py \
    tests/agent/test_memory_provider.py
1 failed, 233 passed, 1 skipped

The single failure is pre-existing and unrelated: tests/gateway/test_status.py::TestReadProcessCmdlinePsFallback::test_ps_fallback_when_proc_unavailable fails on pristine main too (Windows host, POSIX /proc-fallback test).

What I could not verify

  • pytest tests/ -q (the whole suite) was not run — this host has no development clone with the full environment. Executed: the two new files plus the nine suites listed above, on Windows 11 / Python 3.11.16, against the shipped tree.
  • The shipped venv has no pytest, so pytest ran from a throwaway uv environment with the tree's own site-packages on PYTHONPATH.
  • The end-to-end numbers above come from a machine with a live gateway multiplexing default|eagle|lite|monitor|render. Rungs 3/4 depend on live host-topology records, so the "no live multiplexer" and "multi-host" shapes are only covered by the injected seams in the unit tests.
  • Behaviour change flagged for review: a multiplexer that serves profiles whose directories are not under <default root>/profiles/ would now report running=False at rung 4 (the layout guard). That is the intent here, but if a supported deployment has that shape it needs an explicit premise instead of my assumption.
  • Not checked: whether hermes gateway stop / the dashboard's own call sites pass profile_dir the same way my probe does; the change is inside the shared ladder, so they inherit it.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass — not run (no full dev clone here); see "What I could not verify"
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: Windows 11, Python 3.11.16

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — docstrings on both new helpers
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A (no config keys)
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — the home comparison reuses the existing _canonical_hermes_home (host case semantics); paths compared with resolve(strict=False)
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Screenshots / Logs

Before/after resolve_gateway_liveness() output for a copied profile directory and for a genuinely served profile: see "Minimal reproduction" and "How to Test" above.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery area/profiles Multi-profile isolation, HERMES_HOME scoping sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Sep 23, 2026
teknium1 added a commit that referenced this pull request Sep 23, 2026
…the multiplexer roster

Slim redo of the copied-profile-dir guard (#119772) at the seams that actually leaked on main:

- get_runtime_status_running_pid(expected_home=...) now rejects a record whose hermes_home
  stamp names another home (rung 3 of resolve_gateway_liveness and every direct caller:
  live_gateway_pid_for_home, the dashboard messaging/status readers, the update inventory).
  Rung 1 already applied recorded_gateway_home_conflicts inside get_running_pid.
- A scoped read with no gateway_state.json hands rung 3 an empty record, never None -- None
  re-read the PROCESS home's record and lent its live PID to the copied directory.
- multiplexer_liveness_for_profile only answers for <default root>/profiles/<name>: the roster
  is matched by name and a same-named directory under another root is not the served home.

Two invariant tests replace the PR's four (same contract, real records instead of probe stubs).
@teknium1

Copy link
Copy Markdown
Collaborator

Landed on main via #120085 (merge e5131dc6b2), cherry-picked with your authorship preserved — thank you @b2089766906-droid. Closing this PR since its commits are now on main; the follow-up fixes and tests from review are in #120085's body.

@teknium1 teknium1 closed this Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/profiles Multi-profile isolation, HERMES_HOME scoping comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants