fix: probe remote gateway via HERMES_API_URL before local fallback (#3281)

[Sanjays2402]

Root cause

api/agent_health.py determines gateway health by importing the local gateway.status module and reading gateway.pid / gateway_state.json from disk. In multi-container Docker deployments the WebUI image does not ship the gateway Python package, so importlib.import_module("gateway.status") raises ModuleNotFoundError. The payload then falls through to the terminal gateway_not_configured state and the Tasks/Cron panel shows a spurious amber "Gateway not configured" banner — even though HERMES_API_URL points at a perfectly reachable remote gateway.

Fix

When HERMES_API_URL is set, treat it as an explicit declaration that the gateway lives elsewhere and probe it over HTTP before touching any local module/pid/state-file signal:

• GET {HERMES_API_URL}/health, falling back to /status and /api/gateway/status.
• 2xx → alive: true, reason: "remote_gateway".
• All paths fail (timeout / 5xx / network error) → alive: false, reason: "remote_gateway_unreachable" with the last status_code / error name captured for diagnostics.
• Result cached for 5 seconds per base URL so banner rerenders don't hammer the gateway.
• 2-second timeout, stdlib urllib only (no new deps).

When HERMES_API_URL is unset the legacy local importlib/pid/state-file path runs unchanged, preserving single-container/local-install behaviour.

Tests

New tests/test_agent_health_remote.py covers:

1. HERMES_API_URL set + remote returns 200 → healthy (remote_gateway).
2. HERMES_API_URL set + remote unreachable → unhealthy (remote_gateway_unreachable).
3. HERMES_API_URL unset → falls back to existing local probe.
4. Second call within the 5s window reuses the cached result (no second network call).

$ python -m pytest tests/test_agent_health_remote.py -q -o addopts=
....                                                                     [100%]
4 passed

Closes #3281

[nesquena-hermes]

Reading the diff at api/agent_health.py against origin/master, plus the routes.py consumer (api/routes.py:5313-5342), the existing cross-container probe in api/updates.py:269-310, and the agent's API server (gateway/platforms/api_server.py), the mechanism this fixes is real and the routes.py integration is correct — but there are three concrete issues to resolve before this lands, two of which were already flagged by the maintainer on #3281.

The good part

The early-return before the importlib.import_module("gateway.status") path is the right shape, and the payload maps cleanly onto the routes.py tri-state: alive: True → configured=True, running=True (api/routes.py:5316-5318), alive: False → configured=True, running=False (5319-5321). So a reachable remote gateway clears the banner and an unreachable-but-declared one reads as "down, not unconfigured." Caching, 2s timeout, stdlib-only — all reasonable.

Issue 1 — two of the three probe paths don't exist on the agent

The PR probes ("/health", "/status", "/api/gateway/status"). But the agent's API server only registers:

# gateway/platforms/api_server.py:4092-4094
self._app.router.add_get("/health", self._handle_health)
self._app.router.add_get("/health/detailed", self._handle_health_detailed)
self._app.router.add_get("/v1/health", self._handle_health)

There is no /status and no /api/gateway/status on the agent — /api/gateway/status is a WebUI route (api/routes.py:5294), not an agent one. So in practice only /health ever matches; the other two are dead fallbacks. Worse, /health returns a static {"status": "ok"} (api_server.py:1023-1025) — it proves the aiohttp server is up but carries no gateway-state nuance. The endpoint you actually want is /health/detailed, which returns gateway_state, platforms, pid, updated_at (api_server.py:1027-1045). Probing that lets you populate details["gateway_state"], which api/routes.py:5338-5342 already keys on for the stale-running case. Suggest ("/health/detailed", "/health") and map gateway_state into the details dict.

Issue 2 — env var name collides with the WebUI's existing convention

The WebUI already has a cross-container gateway probe (_gateway_health_base_url, api/updates.py:269-280) and it standardizes on:

os.environ.get('GATEWAY_HEALTH_URL')
or os.environ.get('HERMES_GATEWAY_HEALTH_URL')
or 'http://hermes-agent:8642'

This PR introduces a third name, HERMES_API_URL, that isn't referenced anywhere else in the WebUI (api/, server.py) or in the agent (I grepped gateway/, hermes_cli/, agent/, mcp_server/ — zero hits). That creates a split-brain: agent-version detection reads GATEWAY_HEALTH_URL, but gateway-health reads HERMES_API_URL, so a deployment can have one working and the other silently falling through. Recommend reusing _gateway_health_base_url() from updates.py (or at minimum falling back to the same two vars + http://hermes-agent:8642 default) so there's one canonical "where is the gateway" resolver.

Issue 3 — in the reporter's actual deployment, HERMES_API_URL is on the wrong container

In the compose @szmania posted on #3281, HERMES_API_URL=http://hermes-agent-personal:8642 is set on the workspace container, not on hermes-webui-personal. This probe reads the WebUI process env, so as written it won't fire for their setup unless they also add the var to the webui service. Reusing the GATEWAY_HEALTH_URL resolver (Issue 2) sidesteps this since it defaults to http://hermes-agent:8642 even when unset — worth a line in the PR/docs either way.

Design note

The maintainer's #3281 comment leaned toward a tri-state at api/agent_health.py that distinguishes gateway_status_unavailable (module/volume genuinely absent → unknown, suppress the banner rather than assert "not configured") from a confirmed not-running gateway, and called a remote-poll bridge "a larger feature on top." This PR is that larger feature. It's a legitimate direction, but given the maintainer's stated lean it's worth getting explicit buy-in on remote-poll-vs-tri-state before investing more here — and the two could be complementary (tri-state as the safe default, remote poll when a gateway URL is configured).

Net: solid mechanism and correct routes.py wiring; fix the probe paths (/health/detailed), unify the env var with _gateway_health_base_url(), and confirm the direction with the maintainer.

[nesquena-hermes]

Shipped in v0.51.200 (stage-batch12, via #3354). Thanks @Sanjays2402 — the remote HERMES_API_URL probe fixes the spurious multi-container Docker banner. Rebased onto master (your CI failure was stale-base) and removed one unused io import. Codex confirmed it preserves single-container local-probe behavior + bounds the probe with a 2s timeout + maps gateway-down to running:false (no false-green). Filed #3355 for two non-blocking follow-ups Opus noted (probe /health/detailed first for gateway_state; unify on the GATEWAY_HEALTH_URL convention). Closes #3281. 🎉