Skip to content

fix: queue env-enabled platforms for background reconnect after fatal error - #71285

Closed
lowband wants to merge 1 commit into
NousResearch:mainfrom
lowband:fix/env-platform-reconnect-queue
Closed

fix: queue env-enabled platforms for background reconnect after fatal error#71285
lowband wants to merge 1 commit into
NousResearch:mainfrom
lowband:fix/env-platform-reconnect-queue

Conversation

@lowband

@lowband lowband commented Jul 25, 2026

Copy link
Copy Markdown

Problem

Platforms enabled via .env (e.g. TELEGRAM_BOT_TOKEN) are not present in config.yaml's gateway.platforms dict. When such a platform hits a retryable fatal error (e.g. 10 consecutive network failures during a proxy outage), _handle_adapter_fatal_error_impl calls self.config.platforms.get(adapter.platform) which returns None, so the platform is never queued into _failed_platforms. The background reconnect watcher therefore never tries to recover it.

Observed behavior: Gateway process stays alive (other platforms still connected), but Telegram adapter is deaf. Messages sent from Telegram are silently dropped until manual gateway restart. In the reported case, the outage lasted 1h47m before the user noticed and restarted manually.

Root Cause

# gateway/run.py, _handle_adapter_fatal_error_impl()
if adapter.fatal_error_retryable:
    platform_config = self.config.platforms.get(adapter.platform)  # ← None for .env platforms
    if platform_config and ...:  # ← False, skip queue
        self._failed_platforms[adapter.platform] = { ... }

The existing stranded guard (added in a prior commit to mitigate this) exits the gateway so the service manager restarts it. But this is a workaround, not a root-cause fix: it converts a transient outage into a restart loop, losing in-process state (active sessions, cron context, etc.) every time.

Fix

When platform_config is None, build a minimal PlatformConfig from the same env vars that were used to start the adapter originally, using the existing PLATFORM_TOKEN_ENV_NAMES mapping:

if platform_config is None:
    platform_config = self._build_env_fallback_platform_config(adapter.platform)

The fallback reads PLATFORM_TOKEN_ENV_NAMES[platform] (e.g. TELEGRAM_BOT_TOKEN) and {PLATFORM}_HOME_CHANNEL from os.environ, constructs a PlatformConfig(token=..., home_channel=..., enabled=True). This lets the reconnect watcher recover the platform without a full gateway restart.

Testing

Unit tests verify:

  1. _build_env_fallback_platform_config correctly constructs PlatformConfig from env vars
  2. Returns None when env vars are absent (no false queueing)
  3. Retryable fatal error on an env-enabled platform correctly enters _failed_platforms
  4. Non-retryable errors are not queued (unchanged behavior)
  5. Old bug scenario reproduced then fixed: config.platforms has no Telegram entry, retryable fatal error → platform correctly queued
✅ 5/5 tests passed

Scope

  • 1 file changed, 44 insertions, 0 deletions
  • No behavioral change for platforms already in config.yaml
  • No new env vars introduced (reuses existing PLATFORM_TOKEN_ENV_NAMES)
  • The stranded guard remains as a safety net for any edge case the fallback doesn't cover

… error

Platforms enabled via .env (e.g. TELEGRAM_BOT_TOKEN) are not present in
config.yaml's gateway.platforms dict. When such a platform hits a retryable
fatal error (e.g. 10 consecutive network failures), _handle_adapter_fatal_error
looked up self.config.platforms.get(platform) which returned None, so the
platform was never queued into _failed_platforms. The background reconnect
watcher therefore never tried to recover it, and the gateway stayed alive
but deaf until manual restart.

The existing 'stranded' guard (added in a prior commit) exits the gateway
so the service manager can restart it — but this is a workaround that
converts a transient outage into a restart loop, losing in-process state.

This fix addresses the root cause: when platform_config is None, build a
minimal PlatformConfig from the env vars that were used to start the adapter
originally (via PLATFORM_TOKEN_ENV_NAMES), so the reconnect watcher can
recover the platform without a full gateway restart.
@alt-glitch alt-glitch added type/bug Something isn't working P1 High — major feature broken, no workaround comp/gateway Gateway runner, session dispatch, delivery sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Jul 25, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Closing. The premise — that .env-enabled platforms are absent from config.platforms and never queued — doesn't hold on current main: load_gateway_config()_apply_env_overrides()_enable_from_env() inserts the platform for every env token, and the post-#70987 stranded guard exits-with-failure for any retryable-but-unqueued platform. There's also a type bug (home_channel=home_channel or "" passes a str into an Optional[HomeChannel] field). The real stranding class (#71758 — watcher death) is addressed by #72366. Thanks for the report.

@teknium1 teknium1 closed this Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P1 High — major feature broken, no workaround sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants