fix(gateway/discord): REST liveness probe to detect zombie clients (#26656) - #53924
Merged
Conversation
…26656) The Discord adapter could enter a silent zombie state after a network outage / proxy stall: the process is alive, _client looks open, but the underlying socket is dead. discord.py's WebSocket reconnect never sees a RST through a wedged proxy/NAT, so client.start() spins forever without exiting — which means the bot-task done callback (which only fires on task completion) never trips either. The bot stays "offline" in Discord until a manual `hermes gateway restart`. Reported offline for 13-17h. Adds an out-of-band REST liveness probe in DiscordAdapter. Every `discord.liveness_interval_seconds` (default 60s) the adapter issues a cheap fetch_user(bot_id) — the same REST path as message delivery, so it fails when the proxy/NAT is wedged. After `discord.liveness_failure_threshold` consecutive failures (default 3) the probe closes the wedged client and surfaces a retryable fatal error, which trips the gateway's existing _platform_reconnect_watcher and rebuilds the adapter. Operators disable it by setting either knob to 0. Config lives in config.yaml (discord.liveness_*) per the .env-is-secrets policy; _apply_yaml_config bridges it to internal env vars the adapter reads, matching the existing HERMES_DISCORD_TEXT_BATCH_* pattern. Co-authored-by: Hermes Agent <agent@nousresearch.com>
Contributor
🔎 Lint report:
|
| Rule | Count |
|---|---|
unresolved-attribute |
5 |
unresolved-import |
1 |
First entries
run_agent.py:3014: [unresolved-attribute] unresolved-attribute: Object of type `Self@get_credits_spent_micros` has no attribute `_credits_session_start_micros`
tests/gateway/test_discord_liveness.py:179: [unresolved-attribute] unresolved-attribute: Unresolved attribute `fetch_user` on type `_LiveBot`
tests/run_agent/test_credits_notices_toggle.py:76: [unresolved-attribute] unresolved-attribute: Unresolved attribute `_credits_session_start_micros` on type `AIAgent`
tests/gateway/test_discord_liveness.py:130: [unresolved-attribute] unresolved-attribute: Attribute `fetch_user` is not defined on `None` in union `Unknown | None`
tests/gateway/test_discord_liveness.py:162: [unresolved-attribute] unresolved-attribute: Attribute `is_closed` is not defined on `None` in union `Unknown | None`
tests/gateway/test_discord_liveness.py:17: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
✅ Fixed issues (1):
| Rule | Count |
|---|---|
invalid-assignment |
1 |
First entries
tests/run_agent/test_credits_notices_toggle.py:76: [invalid-assignment] invalid-assignment: Object of type `None` is not assignable to attribute `_credits_session_start_micros` of type `int`
Unchanged: 6066 pre-existing issues carried over.
Diagnostics are surfaced as warnings — this check never fails the build.
Collaborator
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Discord adapters self-heal after a network outage instead of going silently dark until a manual
hermes gateway restart. Salvages #26684 (@xxxigm) onto currentmain(the adapter moved toplugins/platforms/discord/adapter.pysince the PR was opened), with the config moved toconfig.yamlper the .env-is-secrets policy.Root cause: when a proxy/NAT wedges the socket without delivering a RST, discord.py's WebSocket reconnect never fires and
client.start()spins forever without exiting — so the_bot_taskdone-callback (which only triggers on task completion) never trips either. The bot stays "offline" in Discord for hours. The done-callback already onmaincovers the "discord.py gives up andstart()exits" case; it cannot see this wedged-but-spinning case.Changes
plugins/platforms/discord/adapter.py: out-of-band REST liveness probe. Everydiscord.liveness_interval_seconds(default 60s) the adapter issuesfetch_user(bot_id)— the same REST path as message delivery. Afterdiscord.liveness_failure_thresholdconsecutive failures (default 3) it closes the wedged client and raises a retryable fatal error, which trips the gateway's existing_platform_reconnect_watcherand rebuilds the adapter. Probe is cancelled cleanly ondisconnect(). Set either knob to0to disable.config.yamlunderdiscord.*;_apply_yaml_configbridges it to internal env vars the adapter reads, matching the existingHERMES_DISCORD_TEXT_BATCH_*pattern (env-is-secrets-only honored; env still wins for explicit overrides).tests/gateway/test_discord_liveness.py(new): probe disabled oninterval=0/threshold=0, healthy probe keeps running with no fatal error, threshold failures close client + retryable fatal + handler fired,disconnect()cancels the probe.website/docs/reference/environment-variables.md: documents the two internal bridge vars, pointing atconfig.yamlas the surface.Validation
start()spins forever, bot dark until manual restarthermes gateway restarttests/gateway/test_discord_liveness.py5/5 green; existingtest_discord_connect.py+test_discord_runtime_failure.pystill greenSalvages #26684. Closes #26656.
Infographic