Skip to content

fix(gateway): add adapter liveness watchdog - #46920

Closed
konsisumer wants to merge 1 commit into
NousResearch:mainfrom
konsisumer:fix/gateway-liveness-watchdog-discord
Closed

fix(gateway): add adapter liveness watchdog#46920
konsisumer wants to merge 1 commit into
NousResearch:mainfrom
konsisumer:fix/gateway-liveness-watchdog-discord

Conversation

@konsisumer

Copy link
Copy Markdown
Contributor

What changed and why

This lands a narrow phase-1 fix for the zombie-adapter class reported in #32574.

  • Added a new gateway.liveness_watchdog config block with enabled, interval, and failure_threshold so the gateway can periodically check live adapters without inventing a second reconnect system.
  • Added BasePlatformAdapter.is_alive() with a conservative default (_running), then wired a new GatewayRunner._platform_liveness_watchdog() loop that counts consecutive failed checks and routes recovery through the existing fatal-error/reconnect path.
  • Added a real Discord override: once the adapter is marked ready, a finished Bot.start() task or a closed ready client is treated as dead and gets requeued for the existing reconnect watcher.
  • Per the reporter's June 15, 2026 follow-up, this first pass does not claim to solve Signal's "transport healthy but no real inbound envelopes" case. The framework stays compatible with a stricter Signal-specific override, but this PR does not equate quiet traffic with a dead connection.

How to test

  • pytest tests/gateway/test_platform_reconnect.py -q
  • pytest tests/plugins/test_discord_runtime_failure.py -q
  • pytest tests/gateway/test_runner_fatal_adapter.py -q
  • pytest tests/gateway/test_runner_startup_failures.py -q

What platforms tested on

  • macOS (unit tests only)

Refs #32574

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery platform/discord Discord bot adapter labels Jun 15, 2026
@konsisumer

Copy link
Copy Markdown
Contributor Author

Rebased onto origin/main and resolved the gateway/config.py conflict by keeping both the upstream gateway.multiplex_profiles parsing and this PR's liveness_watchdog config loading. Local verification passed with ruff check, python scripts/check-windows-footguns.py --diff <base>..HEAD, git diff --check, and the targeted gateway/Discord watchdog test set in the shared Hermes venv (49 passed).

@konsisumer
konsisumer force-pushed the fix/gateway-liveness-watchdog-discord branch from 10145d1 to 1c5ba87 Compare June 19, 2026 17:54
@konsisumer

Copy link
Copy Markdown
Contributor Author

Rebased onto origin/main and resolved the conflict in tests/gateway/test_platform_reconnect.py by keeping both the upstream liveness stub state and this PR's reconnect-call assertions. ruff check, git diff --check, and the in-scope gateway/Discord tests passed locally in the shared Hermes venv. The bounded repo-wide pytest tests/ -q -x --timeout=60 run is still blocked locally by missing optional fastapi/uvicorn dependencies during collection outside this PR's scope.

@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the careful phase-1 framing here, @konsisumer — the routing-through-the-existing-reconnect-path design and the deliberate non-claim on Signal's quiet-traffic case were exactly right, and your clarification comment on #32574 was what scoped this whole cluster.

Closing without merge, though, because the two real-world failure modes this targets are both already fixed on main:

That leaves the generic is_alive() + _platform_liveness_watchdog + gateway.liveness_watchdog framework with a single real consumer (Discord) that's already solved another way, plus a conservative _running-only default that adds no detection for any other adapter today. We'd rather land the framework when a concrete second adapter needs an override — as @poisdahl flagged, a sound is_alive() has to key on last real inbound, not transport/daemon health, which is per-adapter work the generic default can't get right.

Not a quality judgment on the code — the design was sound. If you hit a zombie on a non-Discord platform on latest main, open a focused issue for that adapter and we'll revisit the generic watchdog with a real consumer behind it.

@teknium1 teknium1 closed this Jun 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists platform/discord Discord bot adapter type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants