fix(serve): cache /api/status profile-gateway topology scan (GIL stalls starve desktop boot)

[lost9999]

What does this PR do?

/api/status is the liveness probe the desktop app polls (~1/s) while waiting for the backend to become ready (the dashboard badge polls it too). Since #60537, every call runs a full profile-gateway topology scan — for each profile home: yaml.safe_load of config.yaml (pure-Python loader), psutil process-table probes, and realpath walks — inside the default executor.

On multi-profile installs those scans pile up under the poll rate and hold the GIL for 14–16 s at a stretch, starving the event loop. The WS sidecar then cannot flush gateway.ready (ws ready frame send failed), the desktop client times out and re-dials into the next stall, and after enough attempts boot escalates to the "Hermes couldn't start" overlay.

This PR memoizes the scan behind a 10 s TTL cache with a collapse lock, so concurrent polls share one scan. Topology only changes when gateways start/stop, so a ≤10 s stale badge is an acceptable trade for not starving the loop.

Measured on the affected machine (Windows 11, 7 profiles, 3 MCP servers):

	before	after
event loop stalled … (GIL pressure suspected)	every 25–30 s, 14–16 s each, while desktop polls	exactly one on cold start (first scan)
desktop boot	fails with overlay (all WS dials land inside stalls)	boots; WS stable over 90 s observation

py-spy stack captures during a real failing boot land squarely in the scan on executor threads:

construct_document / compose_mapping_node / ...
load (yaml\__init__.py:81)
safe_load (yaml\__init__.py:125)
_profile_platform_ports (hermes_cli\web_server.py:2947)
_collect_profile_gateway_topology (hermes_cli\web_server.py:3015)
run (concurrent\futures\thread.py:58)

Related Issue

Fixes the dominant, measured stall source in #60800 on multi-profile installs (that issue's own hypothesis section had not located this path). Perf regression introduced by #60537.

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

• hermes_cli/web_server.py: add _collect_profile_gateway_topology_cached() (10 s TTL, threading.Lock collapse so concurrent executor calls share one scan); the /api/status handler now calls the cached wrapper. Scan logic itself unchanged.
• tests/test_web_server_status_topology_cache.py (new): cache hit within TTL, rescan after TTL expiry, and 8-thread concurrent-poll collapse to a single scan.

How to Test

1. pytest tests/test_web_server_status_topology_cache.py tests/test_web_server.py -q
2. Repro (without the patch): on a multi-profile install, launch the desktop app and watch logs/errors.log — repeating pairs of event loop stalled … GIL pressure suspected + ws ready frame send failed appear while the boot overlay shows.
3. With the patch: at most one cold-start stall; the desktop reaches the UI and the WS stays connected.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate (fix(dashboard): eliminate avoidable status-poll subprocess forks #61742 and fix(desktop): avoid rich status during local startup #68185 touch status-poll cost but not this scan)
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass — ran the targeted suites (test_web_server_status_topology_cache.py, test_web_server.py) on the affected production install; deferring the full suite to CI
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: Windows 11 Pro for Workstations (7-profile install where the bug reproduces deterministically)

Documentation & Housekeeping

• I've updated relevant documentation — N/A (internal perf fix, behavior documented in code comment)
• I've updated cli-config.yaml.example if I added/changed config keys — N/A (no config keys)
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A
• I've considered cross-platform impact — pure stdlib (threading/time), no platform-specific code
• I've updated tool descriptions/schemas if I changed tool behavior — N/A

Screenshots / Logs

Failing boot (before), logs/errors.log:

19:01:39,757 WARNING hermes_cli.web_server: event loop stalled 15.8s (GIL pressure suspected)
19:01:39,873 ERROR   tui_gateway.ws: ws ready frame send failed peer=127.0.0.1:52792
19:02:03,055 WARNING hermes_cli.web_server: event loop stalled 14.5s (GIL pressure suspected)
19:02:03,181 ERROR   tui_gateway.ws: ws ready frame send failed peer=127.0.0.1:57212
19:02:36,324 WARNING hermes_cli.web_server: event loop stalled 14.4s (GIL pressure suspected)
19:02:36,434 ERROR   tui_gateway.ws: ws ready frame send failed peer=127.0.0.1:57933

After the patch (same machine, 110 s observation): a single cold-start stall, zero recurrence, desktop boots to a working UI.

[lost9999]

CI slice 7 caught TestStatusEndpointTopology::test_profile_names_and_mode_public_when_auth_gated: that class monkeypatches _collect_profile_gateway_topology per test, and the warm cache from the fixture's collector leaked into the later case.

Fixed by keying the cache on the collector's identity as well as the TTL — a swapped collector is a cache miss, so monkeypatching tests stay hermetic with no reset hook, and production behavior is unchanged (the identity is constant there). Added test_topology_cache_misses_when_collector_is_swapped for that property; the full tests/hermes_cli/test_web_server_gateway_topology.py suite now passes locally alongside the new cache tests (25 passed).

[teknium1]

Thanks for isolating the expensive topology collector and covering TTL, expiry, concurrent collapse, and collector replacement.

Problems

• The new comments describe current Desktop boot as repeatedly polling /api/status. Current Desktop boot instead calls /api/health at apps/desktop/electron/backend-health.ts:131-135; it reaches /api/status only for a backend without the health route (apps/desktop/electron/backend-health.ts:151-160). Please reframe this as a dashboard/status-consumer optimization.
• tests/test_web_server_status_topology_cache.py tests the helper directly, but not the route integration. The production boundary remains hermes_cli/web_server.py:3239-3243; add two HTTP /api/status calls with a mocked collector and assert one scan within the TTL.

The underlying current-main issue remains: every status request directly schedules _collect_profile_gateway_topology at hermes_cli/web_server.py:3239-3243, while the dashboard sidebar polls that endpoint in web/src/hooks/useSidebarStatus.ts:14-22.

Automated hermes-sweeper review.

[teknium1]

Current Desktop boot now probes /api/health (apps/desktop/electron/backend-health.ts:131-135) and reaches /api/status only for a missing health route. Please reframe this comment around dashboard or other ongoing status consumers.

[kshitijk4poor]

Merged via #76969 — thank you @lost9999. Your commit was cherry-picked, so you remain the author in git history.

One honest note recorded in the salvage PR: since your branch, 23cb26c moved desktop boot readiness to /api/health, so the boot-time GIL-starvation symptom your PR described no longer occurs on current desktops. We merged it anyway because the value is real beyond boot — /api/status is still polled continuously (web sidebar 10s, statusbar 60s, Portal, multi-window), and your TTL cache + collapse lock deduplicates the genuinely expensive topology scan for all of them. The double-checked locking and the fn-identity test-hermeticity trick were both correct as written; nothing was changed.