Skip to content

fix(gateway/discord): REST liveness probe to detect zombie clients (#26656) - #53924

Merged
teknium1 merged 1 commit into
mainfrom
hermes/hermes-2fe48744
Jun 28, 2026
Merged

fix(gateway/discord): REST liveness probe to detect zombie clients (#26656)#53924
teknium1 merged 1 commit into
mainfrom
hermes/hermes-2fe48744

Conversation

@teknium1

Copy link
Copy Markdown
Contributor

Summary

Discord adapters self-heal after a network outage instead of going silently dark until a manual hermes gateway restart. Salvages #26684 (@xxxigm) onto current main (the adapter moved to plugins/platforms/discord/adapter.py since the PR was opened), with the config moved to config.yaml per the .env-is-secrets policy.

Root cause: when a proxy/NAT wedges the socket without delivering a RST, discord.py's WebSocket reconnect never fires and client.start() spins forever without exiting — so the _bot_task done-callback (which only triggers on task completion) never trips either. The bot stays "offline" in Discord for hours. The done-callback already on main covers the "discord.py gives up and start() exits" case; it cannot see this wedged-but-spinning case.

Changes

  • plugins/platforms/discord/adapter.py: out-of-band REST liveness probe. Every discord.liveness_interval_seconds (default 60s) the adapter issues fetch_user(bot_id) — the same REST path as message delivery. After discord.liveness_failure_threshold consecutive failures (default 3) it closes the wedged client and raises a retryable fatal error, which trips the gateway's existing _platform_reconnect_watcher and rebuilds the adapter. Probe is cancelled cleanly on disconnect(). Set either knob to 0 to disable.
  • Config lives in config.yaml under discord.*; _apply_yaml_config bridges it to internal env vars the adapter reads, matching the existing HERMES_DISCORD_TEXT_BATCH_* pattern (env-is-secrets-only honored; env still wins for explicit overrides).
  • tests/gateway/test_discord_liveness.py (new): probe disabled on interval=0/threshold=0, healthy probe keeps running with no fatal error, threshold failures close client + retryable fatal + handler fired, disconnect() cancels the probe.
  • website/docs/reference/environment-variables.md: documents the two internal bridge vars, pointing at config.yaml as the surface.

Validation

Before After
Socket wedged behind dead proxy start() spins forever, bot dark until manual restart probe detects in ~3×interval → reconnect
Recovery manual hermes gateway restart automatic via existing reconnect watcher
Targeted tests tests/gateway/test_discord_liveness.py 5/5 green; existing test_discord_connect.py + test_discord_runtime_failure.py still green
E2E (real imports) config.yaml→env bridge, env precedence, and wedged-socket→retryable-fatal→handler-fired-once all verified

Salvages #26684. Closes #26656.

Infographic

Discord liveness probe

…26656)

The Discord adapter could enter a silent zombie state after a network
outage / proxy stall: the process is alive, _client looks open, but the
underlying socket is dead. discord.py's WebSocket reconnect never sees a
RST through a wedged proxy/NAT, so client.start() spins forever without
exiting — which means the bot-task done callback (which only fires on
task completion) never trips either. The bot stays "offline" in Discord
until a manual `hermes gateway restart`. Reported offline for 13-17h.

Adds an out-of-band REST liveness probe in DiscordAdapter. Every
`discord.liveness_interval_seconds` (default 60s) the adapter issues a
cheap fetch_user(bot_id) — the same REST path as message delivery, so it
fails when the proxy/NAT is wedged. After
`discord.liveness_failure_threshold` consecutive failures (default 3) the
probe closes the wedged client and surfaces a retryable fatal error,
which trips the gateway's existing _platform_reconnect_watcher and
rebuilds the adapter. Operators disable it by setting either knob to 0.

Config lives in config.yaml (discord.liveness_*) per the .env-is-secrets
policy; _apply_yaml_config bridges it to internal env vars the adapter
reads, matching the existing HERMES_DISCORD_TEXT_BATCH_* pattern.

Co-authored-by: Hermes Agent <agent@nousresearch.com>
@github-actions

Copy link
Copy Markdown
Contributor

🔎 Lint report: hermes/hermes-2fe48744 vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 11531 on HEAD, 11521 on base (🆕 +10)

🆕 New issues (6):

Rule Count
unresolved-attribute 5
unresolved-import 1
First entries
run_agent.py:3014: [unresolved-attribute] unresolved-attribute: Object of type `Self@get_credits_spent_micros` has no attribute `_credits_session_start_micros`
tests/gateway/test_discord_liveness.py:179: [unresolved-attribute] unresolved-attribute: Unresolved attribute `fetch_user` on type `_LiveBot`
tests/run_agent/test_credits_notices_toggle.py:76: [unresolved-attribute] unresolved-attribute: Unresolved attribute `_credits_session_start_micros` on type `AIAgent`
tests/gateway/test_discord_liveness.py:130: [unresolved-attribute] unresolved-attribute: Attribute `fetch_user` is not defined on `None` in union `Unknown | None`
tests/gateway/test_discord_liveness.py:162: [unresolved-attribute] unresolved-attribute: Attribute `is_closed` is not defined on `None` in union `Unknown | None`
tests/gateway/test_discord_liveness.py:17: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`

✅ Fixed issues (1):

Rule Count
invalid-assignment 1
First entries
tests/run_agent/test_credits_notices_toggle.py:76: [invalid-assignment] invalid-assignment: Object of type `None` is not assignable to attribute `_credits_session_start_micros` of type `int`

Unchanged: 6066 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery platform/discord Discord bot adapter sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Jun 28, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related: salvage of #26684 (@xxxigm) onto current main (adapter relocated to plugins/platforms/discord/adapter.py); fixes #26656; part of the gateway-wide zombie-adapter watchdog effort tracked in #32574.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists platform/discord Discord bot adapter sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Discord adapter enters silent zombie state after network outage; never auto-reconnects

3 participants