fix(run_agent): pool socket walk descends into HTTPConnection wrapper - #6216
Closed
bkadish wants to merge 1 commit into
Closed
fix(run_agent): pool socket walk descends into HTTPConnection wrapper#6216bkadish wants to merge 1 commit into
bkadish wants to merge 1 commit into
Conversation
The CLOSE-WAIT cleanup added in #4481 walks the httpx pool looking for `conn._network_stream` on each entry, but in httpcore >=1.0 the pool contains `HTTPConnection` wrapper objects whose `_network_stream` attribute does not exist — the real stream lives one level deeper at `conn._connection._network_stream` (inside the wrapped HTTP11/HTTP2 Connection). The walk silently yielded zero sockets, so both `_force_close_tcp_sockets` and `_cleanup_dead_connections` were no-ops against any httpcore-1.x environment. Symptom in production: CLOSE-WAIT sockets accumulate against long-lived upstream targets (in our case a localhost LLM proxy) until httpx pool hands a dead socket to a new request, write succeeds into the kernel buffer, read hangs until HERMES_STREAM_READ_TIMEOUT, and the user sees the "Connection to provider dropped (ReadTimeout)" warnings on a schedule. The original cleanup code was supposed to prevent this. This commit: - Adds `_iter_pool_sockets`, a single helper that descends through `conn._connection` if present, then unwraps the network stream's raw socket. Handles both the sync httpcore backend (`stream._sock`) and the anyio async backend (`stream._stream.extra(SocketAttribute .raw_socket)`). - Rewrites `_force_close_tcp_sockets` and `_cleanup_dead_connections` to consume the helper. Behavior is unchanged when the walk works; fixed when it didn't. Verified live against an OpenAI client pointed at a local proxy on httpcore 1.0.9: pre-fix walk yields 0 sockets, post-fix yields 1.
20 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The CLOSE-WAIT cleanup added in #4481 (
_force_close_tcp_sockets,_cleanup_dead_connections) walks the httpx pool looking forconn._network_streamon each pool entry. In httpcore >= 1.0 thepool contains
HTTPConnectionwrapper objects whose_network_streamattribute does not exist — the real stream lives one level deeper at
conn._connection._network_stream, inside the wrappedHTTP11Connection/HTTP2Connection(seehttpcore/_sync/connection.pyline ~64 and
http11.pyline ~54).So the walk silently yields zero sockets and both cleanup functions
become no-ops under any httpx >= 0.25 / httpcore >= 1.0 install. The
log line
tcp_force_closed=Nalways reports 0; pool rebuilds happenon retry but no surgical socket shutdown ever runs.
Symptom we hit in production
Long-lived gateway processes pointed at a local LLM proxy accumulated
⚠️ Connection to provider dropped (ReadTimeout). Reconnecting..."
CLOSE-WAIT sockets until httpx's pool handed a dead socket to a new
request. Write succeeded (into kernel send buffer), read hung until
HERMES_STREAM_READ_TIMEOUT, the agent surfaced the"
warning every few minutes, and recovery required restarting the
gateway. Exactly the failure #4481 was supposed to prevent.
Fix
_iter_pool_sockets, a single helper that descends throughconn._connectionwhen present, then unwraps the network stream'sraw socket. Handles both the sync httpcore backend
(
stream._sock) and the anyio async backend(
stream._stream.extra(SocketAttribute.raw_socket))._force_close_tcp_socketsand_cleanup_dead_connectionsto consume the helper. Behavior is otherwise unchanged.
Verification
Reproduced live against
openai.OpenAI(base_url="http://localhost:3456/v1")on
httpx 0.28.1/httpcore 1.0.9:The recovered socket is the actual
socket.socketobject backing theclient's keep-alive connection —
shutdown(SHUT_RDWR)+close()on it now does what the original commit message promised.
Notes
transports) — the helper falls back to the same attribute names the
original code tried.
tryblock, onlyattempted when the sync
_sockpath returns nothing.