Skip to content

fix(run_agent): pool socket walk descends into HTTPConnection wrapper - #6216

Closed
bkadish wants to merge 1 commit into
NousResearch:mainfrom
bkadish:fix/httpx-pool-walk-httpcore1
Closed

fix(run_agent): pool socket walk descends into HTTPConnection wrapper#6216
bkadish wants to merge 1 commit into
NousResearch:mainfrom
bkadish:fix/httpx-pool-walk-httpcore1

Conversation

@bkadish

@bkadish bkadish commented Apr 8, 2026

Copy link
Copy Markdown
Contributor

Problem

The CLOSE-WAIT cleanup added in #4481 (_force_close_tcp_sockets,
_cleanup_dead_connections) walks the httpx pool looking for
conn._network_stream on each pool entry. In httpcore >= 1.0 the
pool contains HTTPConnection wrapper objects whose _network_stream
attribute does not exist — the real stream lives one level deeper at
conn._connection._network_stream, inside the wrapped
HTTP11Connection / HTTP2Connection (see
httpcore/_sync/connection.py
line ~64 and http11.py line ~54).

So the walk silently yields zero sockets and both cleanup functions
become no-ops
under any httpx >= 0.25 / httpcore >= 1.0 install. The
log line tcp_force_closed=N always reports 0; pool rebuilds happen
on retry but no surgical socket shutdown ever runs.

Symptom we hit in production

Long-lived gateway processes pointed at a local LLM proxy accumulated
CLOSE-WAIT sockets until httpx's pool handed a dead socket to a new
request. Write succeeded (into kernel send buffer), read hung until
HERMES_STREAM_READ_TIMEOUT, the agent surfaced the
"⚠️ Connection to provider dropped (ReadTimeout). Reconnecting..."
warning every few minutes, and recovery required restarting the
gateway. Exactly the failure #4481 was supposed to prevent.

Fix

  • Add _iter_pool_sockets, a single helper that descends through
    conn._connection when present, then unwraps the network stream's
    raw socket. Handles both the sync httpcore backend
    (stream._sock) and the anyio async backend
    (stream._stream.extra(SocketAttribute.raw_socket)).
  • Rewrite _force_close_tcp_sockets and _cleanup_dead_connections
    to consume the helper. Behavior is otherwise unchanged.

Verification

Reproduced live against openai.OpenAI(base_url="http://localhost:3456/v1")
on httpx 0.28.1 / httpcore 1.0.9:

upstream walk found: 0    # pre-fix code from main
sockets found: 1          # this PR

The recovered socket is the actual socket.socket object backing the
client's keep-alive connection — shutdown(SHUT_RDWR) + close()
on it now does what the original commit message promised.

Notes

  • No behavior change when the walk did work (older httpcore, exotic
    transports) — the helper falls back to the same attribute names the
    original code tried.
  • No new dependency on anyio: the import is inside a try block, only
    attempted when the sync _sock path returns nothing.

The CLOSE-WAIT cleanup added in #4481 walks the httpx pool looking for
`conn._network_stream` on each entry, but in httpcore >=1.0 the pool
contains `HTTPConnection` wrapper objects whose `_network_stream`
attribute does not exist — the real stream lives one level deeper at
`conn._connection._network_stream` (inside the wrapped HTTP11/HTTP2
Connection). The walk silently yielded zero sockets, so both
`_force_close_tcp_sockets` and `_cleanup_dead_connections` were no-ops
against any httpcore-1.x environment.

Symptom in production: CLOSE-WAIT sockets accumulate against long-lived
upstream targets (in our case a localhost LLM proxy) until httpx pool
hands a dead socket to a new request, write succeeds into the kernel
buffer, read hangs until HERMES_STREAM_READ_TIMEOUT, and the user sees
the "Connection to provider dropped (ReadTimeout)" warnings on a
schedule. The original cleanup code was supposed to prevent this.

This commit:

- Adds `_iter_pool_sockets`, a single helper that descends through
  `conn._connection` if present, then unwraps the network stream's
  raw socket. Handles both the sync httpcore backend (`stream._sock`)
  and the anyio async backend (`stream._stream.extra(SocketAttribute
  .raw_socket)`).
- Rewrites `_force_close_tcp_sockets` and `_cleanup_dead_connections`
  to consume the helper. Behavior is unchanged when the walk works;
  fixed when it didn't.

Verified live against an OpenAI client pointed at a local proxy on
httpcore 1.0.9: pre-fix walk yields 0 sockets, post-fix yields 1.
@bkadish bkadish closed this by deleting the head repository Apr 28, 2026
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint labels Apr 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants