Skip to content

fix(gateway): stay alive on mixed retryable + non-retryable startup failures - #69880

Closed
shannonsands wants to merge 1 commit into
NousResearch:mainfrom
shannonsands:fix/ns-609-gateway-exit78-retryable
Closed

fix(gateway): stay alive on mixed retryable + non-retryable startup failures#69880
shannonsands wants to merge 1 commit into
NousResearch:mainfrom
shannonsands:fix/ns-609-gateway-exit78-retryable

Conversation

@shannonsands

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes a hosted-gateway crash-loop / permanent-down edge in gateway startup: when connected_count == 0 and at least one platform failed non-retryably, the runner exited with GATEWAY_FATAL_CONFIG_EXIT_CODE (78) even when other platforms failed for merely retryable reasons.

Real-world shape (Linear NS-609, hosted Fly instance): WhatsApp enabled but never paired (non-retryable whatsapp_not_paired) + Telegram TimedOut during polling startup (retryable) → exit 78. Depending on the supervisor this either crash-loops the gateway or takes it permanently down (s6 finish → 125 from #51228, systemd RestartPreventExitStatus=78). Either way Telegram never gets its retry and the dashboard drops with every exit — a single unpaired platform plus one network blip disconnected every channel on the instance.

After this change, exit 78 is reserved for the case where all startup failures are non-retryable (true config error, nothing to wait for). With mixed failures the gateway stays alive in degraded state: the reconnect watcher recovers the retryable platforms, and the misconfigured ones stay fatal-parked and visible in runtime status (gateway_state.json).

Related Issue

Linear NS-609 (hosted gateway disconnects after enabling WhatsApp alongside Telegram). Diagnosed on a live hosted instance: repeated asyncio.run.SystemExit code=78 in gateway-exit-diag.log correlating with Telegram startup timeouts while WhatsApp sat unpaired.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature
  • Security fix
  • Documentation update
  • Tests
  • Refactor
  • New skill

Changes Made

  • gateway/run.py: the connected_count == 0 exit-78 branch now requires startup_nonretryable_errors and not startup_retryable_errors. A new mixed-failure branch logs the fatally-parked platforms loudly and falls through to the existing degraded/retry path (reconnect watcher takes over).
  • tests/gateway/test_runner_startup_failures.py: new regression test test_runner_stays_alive_on_mixed_retryable_and_nonretryable_errors — one fatal platform + one retryable platform must leave the gateway alive (should_exit_cleanly is False, exit_code is None), queue the retryable platform, park the fatal one, and log the parked platform(s).

How to Test

  1. pytest tests/gateway/test_runner_startup_failures.py -q → 13 passed (includes the new mixed-failure regression test).
  2. pytest tests/gateway/test_whatsapp_connect.py -q → 29 passed (fatal-park path unchanged).
  3. Manual repro: enable WhatsApp without pairing (no creds.json) alongside a Telegram token that hits a transient startup failure → gateway must stay up in degraded, Telegram reconnects when the network recovers, WhatsApp shows fatal / whatsapp_not_paired in runtime status.

Behavior matrix after the fix:

Startup failures Before After
all non-retryable exit 78 exit 78 (unchanged)
all retryable degraded, retry degraded, retry (unchanged)
mixed exit 78 degraded; retryables retried, fatals parked

Checklist

Code

  • I have read the contributing guidelines
  • Commit messages follow conventional commits
  • This is not a duplicate of an existing PR
  • Tests pass locally
  • Tested on macOS (gateway test suite)

Documentation & Housekeeping

  • N/A — no user-facing docs affected (behavioral bug fix; comments in code explain the contract)
  • N/A — no config example changes
  • Cross-platform: pure Python control-flow change, no platform-specific code

Screenshots / Logs

Diagnostic signature from the affected hosted instance (profiles/<p>/logs/gateway-exit-diag.log):

{"tag":"asyncio.run.SystemExit","code":78}   # repeated, seconds apart, s6 restarting each time

with gateway.log showing ✗ whatsapp failed to connect (whatsapp_not_paired) and Telegram TimedOut / polling-degraded warnings in the same startup windows.

…ailures

When connected_count == 0 and at least one platform failed with a
non-retryable error, the runner exited with GATEWAY_FATAL_CONFIG_EXIT_CODE
(78) even if OTHER platforms failed for merely transient reasons.

Real-world shape (NS-609, hosted instance): WhatsApp enabled but never
paired (non-retryable whatsapp_not_paired) + Telegram TimedOut during
polling startup (retryable) => exit 78 => the gateway either goes
permanently down (supervisors honoring the exit-78 contract via
RestartPreventExitStatus / the s6 finish->125 translation from NousResearch#51228) or
crash-loops (anything else). Either way Telegram never gets its retry and
the dashboard drops with every exit, so a single unpaired platform plus
one network blip disconnected every channel on the instance.

Now exit 78 is reserved for the case where ALL startup failures are
non-retryable (true config error, nothing to wait for). With mixed
failures the gateway stays alive in degraded state: the reconnect watcher
recovers the retryable platforms and the misconfigured ones stay
fatal-parked and visible in runtime status.
@alt-glitch alt-glitch added type/bug Something isn't working P1 High — major feature broken, no workaround comp/gateway Gateway runner, session dispatch, delivery platform/telegram Telegram bot adapter platform/whatsapp WhatsApp Business adapter sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Jul 23, 2026
@egilewski

Copy link
Copy Markdown
Contributor

looks mergeable

The mixed startup-failure branch now preserves the retryable platform’s recovery path without weakening the existing exit-78 behavior when every startup failure is non-retryable. The focused startup and adjacent WhatsApp suites also remain green.

Security evidence:

  • trust boundary: platform adapter startup failures cross into the gateway process-liveness and supervisor-exit decision.
  • source/sink/invariant: a retryable startup failure must remain queued for reconnection, while a non-retryable peer remains fatal-parked and cannot force exit 78 unless no retryable failures exist.
  • current-main reproduction: the mixed Discord-fatal plus Telegram-retryable probe on 53bdcacf17ddb7483633013e76697672fb3106e0 exited cleanly with code 78 and marked startup failed.
  • PR-head or patch-replay validation: the same probe on a7667d6ec1c4b307bdf2f81c349d633a4d8247b9 and on its patch replay over current main stayed alive, retained Telegram in the retry queue, and kept Discord fatal.
  • positive/negative cases: the 13 startup-failure tests cover all-retryable, all-nonretryable, mixed, disabled, and missing-adapter cases; the 29 WhatsApp connection tests also pass.
  • residual bypass search: initial adapter failures, exception failures, fatal status parking, retry queue ownership, and reconnect-watcher handling were inspected.
  • reviewer validation: source inspection, exact mixed-failure probes, current-main patch replay, and 42 focused tests agree.

Signed: GPT-5.6-sol-xhigh in Codex

@teknium1

Copy link
Copy Markdown
Contributor

Merged via #70987 as a separate commit with your authorship — the mixed retryable/non-retryable startup failure fix was distinct from the reconnect-watcher wedge, so both landed together.

@teknium1 teknium1 closed this Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P1 High — major feature broken, no workaround platform/telegram Telegram bot adapter platform/whatsapp WhatsApp Business adapter sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants