Skip to content

fix(gateway): distinguish interrupted, unreachable and ambiguous model connections (#116323) - #116394

Closed
Halldrix wants to merge 2 commits into
NousResearch:mainfrom
Halldrix:fix/classify-reset-vs-unreachable-116323
Closed

Halldrix wants to merge 2 commits into
NousResearch:mainfrom
Halldrix:fix/classify-reset-vs-unreachable-116323

Conversation

@Halldrix

@Halldrix Halldrix commented Sep 19, 2026 •

Copy link
Copy Markdown

What does this PR do?

The gateway's connection-error reply previously collapsed three different failures into one
"endpoint is not running or is unreachable" message:

This PR splits the single connection row in _PROVIDER_ERROR_REPLIES into three: interrupted,
unreachable, and a catch-all ambiguous reply that explicitly names both possibilities and never
claims the endpoint is down. It also removes the optional api prefix from the generic
connection error|timeout marker so an SDK class name like APIConnectionError no longer
matches the generic "connection error" phrase and falls into the wrong category.

The actionable guidance is preserved: every reply still names a slash command (/retry) and
hermes doctor, and auth/policy/rate-limit precedence is unchanged (rate-limit > auth > policy

connection, first match wins).

Related Issue

Fixes #116323

Related: #109701 (open, CONFLICTING) covers the same classification split but does not retain
the #111600 reply contract and newer auth/quota tests; this PR keeps that contract.

Type of Change

  • 🐛 Bug fix (non-breaking change which fixes an issue)

Changes Made

  • gateway/run.py: add _CONNECTION_INTERRUPTED_MARKERS / _GATEWAY_CONNECTION_INTERRUPTED_RE
    and _ENDPOINT_UNREACHABLE_MARKERS / _GATEWAY_ENDPOINT_UNREACHABLE_RE; add two new rows to
    _PROVIDER_ERROR_REPLIES (interrupted, unreachable) and reword the existing catch-all row to
    an ambiguous reply that does not claim the endpoint is down; drop the (?:api\s*)? prefix
    from the generic connection error|timeout marker in _CONNECTION_ERROR_MARKERS and from the
    unreachable set so APIConnectionError is not classified as "unreachable"; anchor the
    errno 104 / errno 111 / winerror 10061 numeric markers with word boundaries so a longer
    error code (e.g. Errno 1043) does not false-positive into a connection category.
  • tests/gateway/test_local_model_connection_reply.py: add INTERRUPTED_ENVELOPES,
    UNREACHABLE_ENVELOPES, AMBIGUOUS_ENVELOPES; add test_three_connection_causes_are_three_ distinct_categories and test_interrupted_connection_delivery_keeps_precedence_and_redaction
    (rides _sanitize_gateway_final_response on telegram/slack/feishu, checks precedence with
    tainted 401/400/429 + reset envelopes and redaction); remove the now-invalid assertion that a
    bare openai.APIConnectionError (no provider-failure envelope prefix) is a provider error.

How to Test

scripts/run_tests.sh tests/gateway/test_local_model_connection_reply.py \
  tests/gateway/test_telegram_noise_filter.py -q

All 152 tests pass with the fix (including the word-boundary anchors), and the 4 new regression
tests fail on the pre-fix code (verified by reverting gateway/run.py to origin/main and
re-running).

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature
  • I've run scripts/run_tests.sh on the touched suites and all tests pass
  • I've added tests for my changes
  • I've tested on my platform: Debian 13 (trixie), Linux 6.12.107

Documentation & Housekeeping

  • I've updated relevant documentation — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Exclusions (left for follow-ups, issue stays open)

@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Sep 19, 2026
@teknium1

Copy link
Copy Markdown
Collaborator

Thanks @Halldrix — this landed via #116438 (merge 30de041), which supersedes this PR: your word-boundary anchors on the regex rows were taken into the landed fix and credited in the body. Closing with credit; reopen if you see a case it misses.

@teknium1 teknium1 closed this Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Gateway classifies reset and cause-free APIConnectionError as endpoint not running/unreachable

3 participants