[codex] stabilize Hermes Feishu gateway lifecycle

[foras910521-lab]

What does this PR do?

Stabilizes Hermes gateway behavior around Feishu websocket startup/reconnect, Hindsight local_external health checks, API server memory-provider cleanup, and macOS launchd restart handoff.

The fixes keep retryable Feishu startup failures inside the gateway lifecycle instead of letting service managers spin in restart loops. They also make Hindsight local_external availability reflect the actual local HTTP endpoint, and close memory providers after API server runs.

Root Cause

• Feishu websocket startup could be marked connected too early or fail during transient OpenAPI/WSS timeouts.
• When Feishu was the only configured messaging platform, retryable startup failures could make the gateway exit, causing launchd/service-manager churn instead of background reconnect.
• macOS launchd generated gateway run --replace, mixing service-manager restart semantics with Hermes replace handoff.
• Hindsight local_external availability was config-only and could report healthy while 127.0.0.1:9177 was unreachable.
• API server run paths did not consistently close the agent memory provider after completion.

Changes Made

• gateway/platforms/feishu.py: wait for websocket readiness, improve disconnect/CLOSE handling, reduce sensitive SDK URL logging, and drop stale replayed messages after reconnect.
• gateway/run.py: keep the gateway alive for retryable startup platform failures and let the reconnect watcher recover.
• plugins/memory/hindsight/__init__.py: require reachable /health or /version for local_external availability.
• gateway/platforms/api_server.py: shut down memory provider in finally after run handling.
• hermes_cli/gateway.py: stop adding --replace to macOS launchd plist generation.
• Tests updated/added for the changed gateway, Feishu, Hindsight, API server, and service behavior.

How to Test

1. venv/bin/python -m ruff check .
2. scripts/run_tests.sh tests/gateway/test_runner_startup_failures.py tests/gateway/test_platform_reconnect.py tests/gateway/test_feishu.py tests/hermes_cli/test_gateway_service.py tests/hermes_cli/test_update_gateway_restart.py tests/plugins/memory/test_hindsight_provider.py tests/gateway/test_api_server.py tests/gateway/test_api_server_runs.py
3. On macOS launchd, restart ai.hermes.gateway and confirm ProgramArguments are python -m hermes_cli.main gateway run without --replace.
4. With transient Feishu OpenAPI/WSS timeouts, confirm gateway remains running, starts the reconnect watcher, and Feishu later returns to connected.
5. Confirm Hindsight local API health returns {"status":"healthy","database":"connected"}.

Validation Performed

• Ruff passed.
• Targeted official wrapper tests passed: 670 passed.
• Rebased cleanly on origin/main at dd0923bb8.
• macOS runtime check: gateway PID refreshed, launchd has no --replace, Hindsight healthy, Feishu connected.

Platform Tested

• macOS launchd on local Hermes gateway.

Notes

This PR intentionally does not modify system networking, Wi-Fi, DNS, proxy services, or third-party proxy app configuration. The stability work stays inside Hermes gateway and provider lifecycle boundaries.

[foras910521-lab]

Draft upstream de-dup note after rechecking main on 2026-05-12:

• Current upstream main is still dd0923b; I did not find a merged change that fully supersedes this draft.
• This PR is intentionally still draft because it overlaps several open or closed PRs and should likely be split or narrowed before review.
• The launchd --replace part overlaps strongly with fix(gateway): stop default replace in service runs #23281 / fix(gateway): remove --replace from systemd/launchd service definitions (fixes #23272) #23330 / fix(gateway): avoid --replace in generated services #21907. fix(gateway): remove --replace from systemd/launchd service definitions (fixes #23272) #23330 appears more complete across service generators, so this draft should not push that as the primary contribution.
• Retryable startup failure handling overlaps fix(gateway): keep startup retry failures alive for reconnect recovery #17984. This draft preserves nonretryable startup failure semantics, but the status model should be reconciled with fix(gateway): keep startup retry failures alive for reconnect recovery #17984 before becoming ready.
• Hindsight local_external reachability is complementary to fix(memory): probe hindsight_client import in is_available (#18875) #22586: fix(memory): probe hindsight_client import in is_available (#18875) #22586 checks SDK importability, while this draft checks the configured local HTTP endpoint.
• API server memory-provider cleanup appears independent from the open PRs I found.
• Feishu lifecycle work overlaps older closed PRs fix(gateway): skip stale pre-connect messages in Feishu adapter #21712 / Fix/feishu reconnect and shutdown #5500 and open PR fix: improve platform gateway reliability — multi-process protection, hook chat_id, and Feishu event queueing #4789 only partially. Local runtime logs still show SDK event-loop lifecycle errors, so the Feishu part needs a separate audit before review.

Proposed next step: keep this PR as an umbrella draft, then split or narrow it into focused PRs rather than asking for review in its current shape.

[foras910521-lab]

Follow-up split: the low-overlap Hindsight local_external reachability and API server memory-provider cleanup changes have been separated into draft PR #24353. This keeps #24337 as an umbrella/Feishu-gateway draft rather than asking reviewers to evaluate unrelated lifecycle fixes together.

[foras910521-lab]

Additional Feishu lifecycle note from local re-diagnosis: this umbrella draft should not be promoted as-is. The local runtime still shows Lark SDK event-loop lifecycle errors (Future attached to a different loop and _select was never awaited). The risky part appears to be the draft Feishu shutdown path that proactively calls the SDK internal _disconnect() from Hermes shutdown via the SDK thread loop; that can overlap with the SDK receive-loop disconnect/reconnect lifecycle. I will treat the Feishu portion as needing a smaller redo from main: no cross-loop SDK internal _disconnect(), no broad lifecycle coupling, and ready/cleanup behavior tested in isolation.

[foras910521-lab]

Follow-up split: the narrower Feishu websocket readiness work has been separated into draft PR #24358. That PR starts from main, waits for the SDK websocket _connect() readiness signal before marking Feishu connected, drops stale pre-connect message replays, and intentionally does not include the broader shutdown experiment that called SDK _disconnect() across the SDK loop boundary.

[foras910521-lab]

Follow-up split: retryable gateway startup degradation has been separated into draft PR #24364. It aligns with #17984 by keeping the process alive in degraded state for all-retryable startup failures, while preserving current main semantics for non-retryable startup conflicts (startup_failed plus clean exit). This keeps #24337 as an umbrella draft rather than mixing gateway runner state policy with Feishu adapter lifecycle changes.