Skip to content

fix(dashboard): cache gateway PID status probes - #53511

Closed
0disoft wants to merge 1 commit into
NousResearch:mainfrom
0disoft:task/dashboard-status-lock-cache
Closed

fix(dashboard): cache gateway PID status probes#53511
0disoft wants to merge 1 commit into
NousResearch:mainfrom
0disoft:task/dashboard-status-lock-cache

Conversation

@0disoft

@0disoft 0disoft commented Jun 27, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add a short-lived cached wrapper around get_running_pid() for read-only dashboard/status polling.
  • Invalidate the cache when PID, runtime lock, or runtime status files change, and when the current process writes/removes PID or lock ownership.
  • Use the cached probe from /api/status so high-frequency dashboard polling does not repeatedly open and lock gateway.lock.

Root cause

/api/status called get_running_pid() on every request. That path checks is_gateway_runtime_lock_active(), which opens gateway.lock and probes the advisory lock each time. Under active dashboard polling this churns file descriptors and can hit OSError: [Errno 24] Too many open files in the dashboard process.

Validation

  • bash scripts/run_tests.sh tests/gateway/test_status.py tests/hermes_cli/test_web_server.py could not start on this Windows host due to the local Bash/CallMsi REGDB_E_CLASSNOTREG launcher error.
  • .\.venv\Scripts\python.exe -m pytest tests/gateway/test_status.py::TestGatewayPidState -q -> 15 passed
  • .\.venv\Scripts\python.exe -m pytest tests/hermes_cli/test_web_server.py::TestWebServerEndpoints tests/hermes_cli/test_web_server.py::TestStatusRemoteGateway tests/hermes_cli/test_web_server.py::TestGatewayBusyReadout -q -> 136 passed, 4 warnings
  • ruff check gateway/status.py hermes_cli/web_server.py tests/gateway/test_status.py tests/hermes_cli/test_web_server.py -> passed
  • git diff --check -> passed, with CRLF conversion warnings only

I also ran the full tests/gateway/test_status.py file directly on Windows. It now has 86 passed, 3 failed; the remaining failures are pre-existing Windows/local-environment failures unrelated to this PR (sleep executable missing for two process-start-time tests, plus an existing Windows path/cmdline assertion in test_runtime_status_running_pid_accepts_matching_profile_cmdline).

Fixes #53484

@alt-glitch alt-glitch added type/perf Performance improvement or optimization comp/gateway Gateway runner, session dispatch, delivery comp/dashboard Web dashboard / control panel UI (dashboard/, landing) P3 Low — cosmetic, nice to have labels Jun 27, 2026

@tonydwb tonydwb left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

Verdict: Approved

Adds file-change-based caching for get_running_pid() dashboard polling. Uses threading.Lock for thread safety, file stat signatures (mtime_ns, size) for cache invalidation, and a 1-second TTL. Properly clears cache on PID file writes, lock acquisition/release, and stale cleanup. Includes thorough tests.


Reviewed by Hermes Agent

liuhao1024 pushed a commit to liuhao1024/hermes-agent that referenced this pull request Jul 8, 2026
…ests

get_status now probes via get_running_pid_cached() (NousResearch#53511 salvage);
these tests were added on main after that PR was cut and still patched
web_server.get_running_pid, so their fakes were bypassed and CI slice
5/8 failed. Patch the name the handler actually calls.
@kshitijk4poor

Copy link
Copy Markdown
Collaborator

Merged via #60884 (rebase) — your commit landed on main as 7d0ddbb with your authorship preserved. The TTL + file-signature cache design held up well under review (the signature check even covers a legacy direct-unlink path in --replace). Thanks for the EMFILE fix!

santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026
…ests

get_status now probes via get_running_pid_cached() (NousResearch#53511 salvage);
these tests were added on main after that PR was cut and still patched
web_server.get_running_pid, so their fakes were bypassed and CI slice
5/8 failed. Patch the name the handler actually calls.
justemu pushed a commit to justemu/hermes-agent that referenced this pull request Jul 18, 2026
…ests

get_status now probes via get_running_pid_cached() (NousResearch#53511 salvage);
these tests were added on main after that PR was cut and still patched
web_server.get_running_pid, so their fakes were bypassed and CI slice
5/8 failed. Patch the name the handler actually calls.
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
…ests

get_status now probes via get_running_pid_cached() (NousResearch#53511 salvage);
these tests were added on main after that PR was cut and still patched
web_server.get_running_pid, so their fakes were bypassed and CI slice
5/8 failed. Patch the name the handler actually calls.
leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026
…ests

get_status now probes via get_running_pid_cached() (NousResearch#53511 salvage);
these tests were added on main after that PR was cut and still patched
web_server.get_running_pid, so their fakes were bypassed and CI slice
5/8 failed. Patch the name the handler actually calls.
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
…ests

get_status now probes via get_running_pid_cached() (NousResearch#53511 salvage);
these tests were added on main after that PR was cut and still patched
web_server.get_running_pid, so their fakes were bypassed and CI slice
5/8 failed. Patch the name the handler actually calls.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/dashboard Web dashboard / control panel UI (dashboard/, landing) comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have type/perf Performance improvement or optimization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dashboard handler leaks fds via repeated is_gateway_runtime_lock_active calls (v0.17.0)

4 participants