feat(gateway): auto-restart dispatcher on hermes_cli code changes

[sahilm-ti]

Summary

Fixes the recurring "stale-dispatcher-runs-old-code" class of bugs (kanban t_014159c0).

When a PR merges to main, any gateway process started before the merge keeps executing
old code from memory. This shows up as workers spawned after the merge still running
the buggy behavior the fix supposedly cured — because the dispatcher never reloaded.

Solution: gateway/code_watcher.py — a background daemon thread that polls the
mtime of every hermes-agent .py file imported into sys.modules. When any file's
mtime is newer than the gateway process start time, the watcher logs a warning and
calls os.execv to reload in-place, preserving the PID for launchd/systemd supervisors.

Changes

• gateway/code_watcher.py (new) — CodeWatcher class, start_code_watcher() factory
• gateway/run.py — hooks start_code_watcher() alongside the cron ticker in start_gateway()
• tests/gateway/test_code_watcher.py (new) — 21 unit tests, all passing

Opt-outs

Config (~/.hermes/config.yaml):

gateway:
  auto_restart_on_code_change: false

Env (wins over config):

HERMES_GATEWAY_NO_AUTO_RESTART=1

Trigger policy

• Poll interval: 60 seconds (one background daemon thread, ~100 stat() calls per tick)
• Restart delay: 5 seconds (lets any in-flight supervisor-loop work quiesce)
• os.execv preserves PID — launchd/systemd supervisors don't see a process replacement

Verification

Smoke test run locally with a fake process start time (5 minutes in the past) and
os.execv monkeypatched to record calls instead of actually re-exec'ing:

DEBUG gateway.code_watcher: gateway code-watcher: watching /Users/sahilmarwaha/.hermes/worktrees/t_014159c0
WARNING gateway.code_watcher: gateway: code change detected in .../hermes_cli/__init__.py, re-execing in 0s to pick up new code
INFO gateway.code_watcher: gateway: re-execing as /path/to/python -
SUCCESS: os.execv called with executable=/path/to/python
  argv[0]=/path/to/python, argv matches sys.argv: True

Manual smoke test recipe (for PR reviewer / Sahil):

# Start gateway in a terminal
hermes --gateway

# In a second terminal, touch any watched file
touch ~/.hermes/hermes-agent/hermes_cli/kanban_db.py

# Within 90s, observe in gateway.log:
# WARNING gateway.code_watcher: gateway: code change detected in .../kanban_db.py, re-execing in 5s to pick up new code
# INFO gateway.code_watcher: gateway: re-execing as /path/to/python ...
# Gateway PID should remain the same (os.execv replaces the process image in-place)

Screenshot N/A — log lines verified via stdout capture pasted above.

Tests

21 passed in 0.49s

[coderabbitai]

Warning

Review limit reached

@sahilm-ti, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 20 minutes and 45 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: f12b6a38-d874-429c-ad60-aa9dc6524062

📥 Commits

Reviewing files that changed from the base of the PR and between 2dd78f3 and f2b817a9127dae1e08c09a17a7c2e5f7b7a5e6ea.

📒 Files selected for processing (3)

• gateway/code_watcher.py
• gateway/run.py
• tests/gateway/test_code_watcher.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch kanban/t_014159c0

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

🔎 Lint report: kanban/t_014159c0 vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 9585 on HEAD, 9583 on base (🆕 +2)

🆕 New issues (2):

Rule	Count
unresolved-import	2

First entries

tests/gateway/test_code_watcher.py:23: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
gateway/code_watcher.py:298: [unresolved-import] unresolved-import: Cannot resolve imported module `psutil`

✅ Fixed issues: none

Unchanged: 5040 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

[sahilm-ti]

auto-review: changes requested.

Matrix checks (U1–U5, C1–C5)

• U1 in-scope-files: PASS — all 3 files match AC scope
• U2 out-of-scope-deletions: PASS — no unexpected deletions
• U3 no-secrets: PASS
• U4 ac-coverage: PASS — all 7 AC items have matching diff hunks
• U5 pr-mergeable: PASS (UNSTABLE — non-required checks pending; required checks checked below)
• C1 CI green: FAIL ❌ — 2 blocking checks failed:
  • ruff enforcement (blocking) — FAILURE
    • gateway/code_watcher.py:306: open(f"/proc/{pid}/stat") as f — PLW1514: open in text mode without explicit encoding argument
    • gateway/code_watcher.py:313: open("/proc/stat") as f — PLW1514: same issue
  • Windows footguns (blocking) — FAILURE (same two lines; flagged as platform-encoding footgun)
  • Fix: add encoding='utf-8' to both open() calls. They're inside a Linux-only /proc branch, so encoding='utf-8' is correct (Linux /proc files are ASCII/UTF-8). Add # windows-footgun: ok on each line if the file is intentionally Linux-only, or just add the encoding kwarg (fixes both ruff and the footgun checker in one).
• C2 type-discipline: FAIL ❌ — new # type: ignore introduced:
  • gateway/code_watcher.py (~line 288): import psutil # type: ignore[import-untyped]
  • Per project rules, new type: ignore comments are not permitted in source. Psutil has a stub package: add psutil to the project's type-stub dependencies (or use a typed alternative), or guard the import differently so the type: ignore isn't needed.
• C3 lint-clean: FAIL (blocked by C1/ruff)
• C4 tests-touched: PASS — tests/gateway/test_code_watcher.py added alongside new source
• C5 worker-identity: PASS — single commit 1dd9d4be authored by 266772320+sahilm-ai@users.noreply.github.com
• C6 ui-screenshot: N/A — no UI files in diff

Code-quality judgment (role-reviewer)

Overall the implementation is solid. The two issues above are the only blockers. Once fixed:

• The _get_process_start_time() multi-fallback chain is well-structured and the /proc Linux path is correctly wrapped in try/except.
• The _detect_checkout_root() sys.modules walk is safe and handles the editable-install case correctly.
• The CodeWatcher._run() loop correctly short-circuits on stop_event during the restart delay.
• 21 unit tests covering opt-out, scan, execv, collect — good coverage.

Required fixes before approval

1. gateway/code_watcher.py:306 — open(f"/proc/{pid}/stat", encoding='utf-8') as f
2. gateway/code_watcher.py:313 — open("/proc/stat", encoding='utf-8') as f
3. Remove or resolve # type: ignore[import-untyped] on the psutil import (install psutil-stubs or types-psutil as a dev dep, or annotate the guarded path differently).

Amend the commit, force-push, and CI should go green.

[sahilm-ti]

Fix push (run 3): Addressed the two sdlc-review rejections:

• C1: Added encoding='utf-8' to both open() calls in get_process_start_time() (lines 306, 313 → now 307, 314). Ruff PLW1514 passes.
• C2: Removed # type: ignore[import-untyped] from import psutil — ty reports it as an unused suppression (psutil has bundled stubs in this env). Matches how all other files in the codebase import psutil.
  21/21 tests still pass.

[sahilm-ti]

auto-review: approved, awaiting human merge + kanban_approve.

Matrix checks (U1-U5, C1-C5): All pass.

• U1: in-scope files only (gateway/code_watcher.py, gateway/run.py, tests/)
• U2: no deletions
• U3: no secrets
• U4: all 7 AC bullets addressed (module, daemon thread, opt-outs, execv test, smoke-test recipe, verification section, two-stage flow)
• U5: CLEAN + MERGEABLE
• C1: all CI checks green (ruff enforcement, Windows footguns, ruff+ty diff, nix linux+macos, tests 1-6, e2e, supply chain)
• C2: no new type: ignore or cast() in diff
• C3: lint clean (ruff enforcement blocking: pass)
• C4: 21 tests added in tests/gateway/test_code_watcher.py
• C5: commit f2b817a9 — author + committer both 266772320+sahilm-ai@users.noreply.github.com

Code-quality judgment (role-reviewer): APPROVED — no blockers. 4-level platform fallback in _get_process_start_time() is documented and intentional. CodeWatcher is well-scoped, daemon thread correctly stopped on shutdown. Tests are behavioral.

[coderabbitai]

Warning

Review limit reached

@sahilm-ti, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 2 minutes and 29 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 84f51f2d-1186-4dff-bdd3-57938acb107b

📥 Commits

Reviewing files that changed from the base of the PR and between 5b5e072 and a9b7540.

📒 Files selected for processing (3)

• gateway/code_watcher.py
• gateway/run.py
• tests/gateway/test_code_watcher.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch kanban/t_014159c0

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[sahilm-ti]

auto-review: approved. All U1-U5 and C1-C5 pass.

Matrix checks (U1–U5, C1–C5):

• U1 in-scope files: PASS (3 files, all match AC scope)
• U2 deletions: PASS (no unauthorized deletions)
• U3 secrets: PASS
• U4 AC coverage: PASS (all 7 AC items covered)
• U5 mergeable: PASS (UNSTABLE — nix non-required; MERGEABLE=true)
• C1 CI: PASS (ruff enforcement, Windows footguns, test 1-6, e2e, ruff+ty diff all green; nix non-required)
• C2 type-discipline: PASS (no new type:ignore)
• C3 lint: PASS
• C4 tests: PASS (314-line test file, 21 tests, 21/21 pass locally)
• C5 identity: PASS (a9b7540 authored by 266772320+sahilm-ai@users.noreply.github.com)

Code-quality judgment (role-reviewer): APPROVED, no Blockers or Majors.

• DI clean: CodeWatcher takes all deps as constructor params
• Fallbacks acceptable: _get_process_start_time() conservative fallback to time.time() is documented + logged
• Thread safety: threading.Event stop signal correct, daemon thread won't orphan
• run.py integration: _code_watcher local variable stays in scope through shutdown block correctly

Awaiting Sahil's merge + kanban_approve.