fix(gateway): stay alive on mixed retryable + non-retryable startup failures - #69880
Closed
shannonsands wants to merge 1 commit into
Closed
fix(gateway): stay alive on mixed retryable + non-retryable startup failures#69880shannonsands wants to merge 1 commit into
shannonsands wants to merge 1 commit into
Conversation
…ailures When connected_count == 0 and at least one platform failed with a non-retryable error, the runner exited with GATEWAY_FATAL_CONFIG_EXIT_CODE (78) even if OTHER platforms failed for merely transient reasons. Real-world shape (NS-609, hosted instance): WhatsApp enabled but never paired (non-retryable whatsapp_not_paired) + Telegram TimedOut during polling startup (retryable) => exit 78 => the gateway either goes permanently down (supervisors honoring the exit-78 contract via RestartPreventExitStatus / the s6 finish->125 translation from NousResearch#51228) or crash-loops (anything else). Either way Telegram never gets its retry and the dashboard drops with every exit, so a single unpaired platform plus one network blip disconnected every channel on the instance. Now exit 78 is reserved for the case where ALL startup failures are non-retryable (true config error, nothing to wait for). With mixed failures the gateway stays alive in degraded state: the reconnect watcher recovers the retryable platforms and the misconfigured ones stay fatal-parked and visible in runtime status.
Contributor
|
looks mergeable The mixed startup-failure branch now preserves the retryable platform’s recovery path without weakening the existing exit-78 behavior when every startup failure is non-retryable. The focused startup and adjacent WhatsApp suites also remain green. Security evidence:
Signed: GPT-5.6-sol-xhigh in Codex |
This was referenced Jul 24, 2026
Contributor
|
Merged via #70987 as a separate commit with your authorship — the mixed retryable/non-retryable startup failure fix was distinct from the reconnect-watcher wedge, so both landed together. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes a hosted-gateway crash-loop / permanent-down edge in gateway startup: when
connected_count == 0and at least one platform failed non-retryably, the runner exited withGATEWAY_FATAL_CONFIG_EXIT_CODE(78) even when other platforms failed for merely retryable reasons.Real-world shape (Linear NS-609, hosted Fly instance): WhatsApp enabled but never paired (non-retryable
whatsapp_not_paired) + TelegramTimedOutduring polling startup (retryable) → exit 78. Depending on the supervisor this either crash-loops the gateway or takes it permanently down (s6finish→ 125 from #51228, systemdRestartPreventExitStatus=78). Either way Telegram never gets its retry and the dashboard drops with every exit — a single unpaired platform plus one network blip disconnected every channel on the instance.After this change, exit 78 is reserved for the case where all startup failures are non-retryable (true config error, nothing to wait for). With mixed failures the gateway stays alive in
degradedstate: the reconnect watcher recovers the retryable platforms, and the misconfigured ones stay fatal-parked and visible in runtime status (gateway_state.json).Related Issue
Linear NS-609 (hosted gateway disconnects after enabling WhatsApp alongside Telegram). Diagnosed on a live hosted instance: repeated
asyncio.run.SystemExit code=78ingateway-exit-diag.logcorrelating with Telegram startup timeouts while WhatsApp sat unpaired.Type of Change
Changes Made
gateway/run.py: theconnected_count == 0exit-78 branch now requiresstartup_nonretryable_errors and not startup_retryable_errors. A new mixed-failure branch logs the fatally-parked platforms loudly and falls through to the existing degraded/retry path (reconnect watcher takes over).tests/gateway/test_runner_startup_failures.py: new regression testtest_runner_stays_alive_on_mixed_retryable_and_nonretryable_errors— one fatal platform + one retryable platform must leave the gateway alive (should_exit_cleanly is False,exit_code is None), queue the retryable platform, park the fatal one, and log the parked platform(s).How to Test
pytest tests/gateway/test_runner_startup_failures.py -q→ 13 passed (includes the new mixed-failure regression test).pytest tests/gateway/test_whatsapp_connect.py -q→ 29 passed (fatal-park path unchanged).creds.json) alongside a Telegram token that hits a transient startup failure → gateway must stay up indegraded, Telegram reconnects when the network recovers, WhatsApp showsfatal/whatsapp_not_pairedin runtime status.Behavior matrix after the fix:
Checklist
Code
Documentation & Housekeeping
Screenshots / Logs
Diagnostic signature from the affected hosted instance (
profiles/<p>/logs/gateway-exit-diag.log):with
gateway.logshowing✗ whatsapp failed to connect(whatsapp_not_paired) and TelegramTimedOut/ polling-degraded warnings in the same startup windows.