fix(agent): rebuild connection pool for socket-layer transport errors (BrokenPipe, ConnectionReset, etc.) - #52226
Open
DavidMetcalfe wants to merge 1 commit into
Conversation
…nPipe, ConnectionReset, etc.)
The transport-recovery gate at `agent/agent_runtime_helpers.py:818`
consulted a stale frozenset (`_TRANSIENT_TRANSPORT_ERRORS`, only 7 type
names) that missed socket-layer errors the classifier at
`agent/error_classifier.py:_TRANSPORT_ERROR_TYPES` already treats as
retryable: `BrokenPipeError`, `ConnectionResetError`,
`ConnectionAbortedError`, `ServerDisconnectedError`, `ReadError`,
plus the SSL family.
Result: reasoning models on providers whose cloud gateway closes
sockets during long thinking phases (NVIDIA Nemotron 3 Ultra on hosted
NIM's ~120s upstream idle kill, also Anthropic Opus 4.7 thinking and
OpenAI o1/o3) burned all 3 outer retries against the same dead
connection pool, then surfaced `API call failed after 3 retries:
[Errno 32] Broken pipe`.
Fix: alias the recovery set to the canonical classifier set. Single
source of truth — any future addition to `_TRANSPORT_ERROR_TYPES`
automatically flows through, no drift possible.
```python
_TRANSIENT_TRANSPORT_ERRORS = _TRANSPORT_ERROR_TYPES
```
Verified:
- 238 tests passing across `test_error_classifier.py`,
`test_primary_runtime_restore.py`, `test_streaming.py`.
- 8 new regression tests in `test_primary_runtime_restore.py`:
- 7 parametrized over the socket-layer types (ConnectionError,
ConnectionResetError, ConnectionAbortedError, BrokenPipeError,
TimeoutError, ReadError, ServerDisconnectedError) — all fail
without the source fix, all pass with it.
- 1 identity check (`_TRANSIENT_TRANSPORT_ERRORS is _TRANSPORT_ERROR_TYPES`)
guards against future re-introduction of a duplicated frozenset.
- Ruff lint clean on both modified files.
- No import cycle (verified: `agent/error_classifier.py` has no reverse
import of `agent.agent_runtime_helpers`).
- Zero mutations on either set anywhere in the codebase.
Fixes NousResearch#52216.
This was referenced Jun 25, 2026
Contributor
|
Thanks for the focused recovery fix. The premise still holds on current The alias is a small, direct way to remove that drift. Automated hermes-sweeper review. |
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The transport-recovery gate at
agent/agent_runtime_helpers.py:818consulted a stale frozenset (_TRANSIENT_TRANSPORT_ERRORS, only 7 type names) that missed socket-layer errors the canonical classifier atagent/error_classifier.py:_TRANSPORT_ERROR_TYPESalready treats as retryable. Result: providers whose cloud gateway closes sockets during long thinking phases (NVIDIA Nemotron 3 Ultra on hosted NIM's ~120s upstream idle kill — first-party reproduction at NVIDIA/NemoClaw#4846 — also Anthropic Opus 4.7 thinking and OpenAI o1/o3) burned all 3 outer retries against the same dead connection pool, then surfacedAPI call failed after 3 retries: [Errno 32] Broken pipe.One source-file line change:
Single source of truth — any future addition to
_TRANSPORT_ERROR_TYPESautomatically flows through, no drift possible.Test Plan
python3 -m pytest tests/run_agent/test_primary_runtime_restore.py tests/agent/test_error_classifier.py tests/run_agent/test_streaming.py -q— 238 passed, zero failures, zero regressions.ruff check agent/agent_runtime_helpers.py tests/run_agent/test_primary_runtime_restore.py— clean._try_recover_primary_transport refused recovery for BrokenPipeError; the gate at agent_runtime_helpers.py:818 is out of sync with agent/error_classifier.py:_TRANSPORT_ERROR_TYPES). Then restored the fix and confirmed all 39 tests intest_primary_runtime_restore.pypass.agy -p:passed: true, zero blockers/should-fix/nits.passed: true, zero blockers/should-fix/nits.agent/error_classifier.pyhas no reverse import ofagent.agent_runtime_helpers), the parametrized test correctly mirrors the productiontype(api_error).__name__check at line 818, and the identity assertion (is, not==) catches future re-introduction of a duplicated frozenset literal.Notes
if agent._is_openrouter_url(): return False) and provider skip at lines 825-826 (Nous, Nous-Research) are preserved. Aggregators that manage their own retry infra still skip the rebuild._retry.primary_recovery_attemptedatagent/conversation_loop.py:3421). For truly persistent upstream issues that the rebuild cannot fix, the next outer retry surfaces the same error and the classifier's broader signals route to fallback or abort — no recovery loop._TRANSPORT_ERROR_TYPESis genuinely transient:ReadTimeout,ConnectTimeout,PoolTimeout— explicit timeouts, transient.ConnectError,RemoteProtocolError,ConnectionError,ConnectionResetError,ConnectionAbortedError,BrokenPipeError,ServerDisconnectedError,ReadError— socket-level disconnects, transient (a stale-connection-pool rebuild is exactly the right recovery).TimeoutError— generic timeout, transient.SSLError,SSLZeroReturnError,SSLWantReadError,SSLWantWriteError,SSLEOFError,SSLSyscallError— TLS handshake/record failures, transient.APIConnectionError,APITimeoutError— OpenAI SDK wrappers for the above, transient.agent/reasoning_timeouts.py) and integration into two stale-detector scaling blocks, and is shipped separately.Fixes #52216.