Skip to content

fix(dispatcher): stale keep-alive burst on local egress - #14315

Merged
diegosouzapw merged 2 commits into
diegosouzapw:release/v3.8.51from
VIPKaiser:fix/dispatcher-stale-keep-alive
Sep 29, 2026
Merged

diegosouzapw merged 2 commits into
diegosouzapw:release/v3.8.51from
VIPKaiser:fix/dispatcher-stale-keep-alive

Conversation

@VIPKaiser

@VIPKaiser VIPKaiser commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Problem

On macOS Docker Desktop, every *.internal / *.local hostname the
container reaches through host.docker.internal (every local LLM
provider wired through the proxy: oMLX, lightning-mlx, mtplx, Breeze TTS,
Parakeet STT, Kokoro TTS) can hit a 30-second ECONNREFUSED burst when
Docker Desktop's NAT silently drops idle keep-alive sockets inside the
direct round-robin pool. Triggers: a request lands on a stale pooled
socket, retries, gets a fresh one, hits the same dead path, then
cooldown/lockout kicks in.

Three concrete amplifiers:

  • proxyDispatcher.ts:73-76 sets autoSelectFamilyAttemptTimeout: 1000,
    but the IPv6 form of host.docker.internal
    (fdc4:f303:9324::254) is ENETUNREACH inside the container (the
    IPv4 form 192.168.65.254 is the working address). Every healthy
    request waits 1s on the dead family before getting ECONNREFUSED on v4.
  • getDefaultDispatcher() caches the round-robin pool in
    globalThis and never reaps on PROXY_UNREACHABLE — a single stale
    socket poisons the pool for all subsequent requests until process
    restart.
  • The default keepAliveMaxTimeout (4s) sits squarely inside the NAT's
    silent-drop window for local egress.

Fix

Three coordinated changes, all backward-compatible:

1. Hostname-aware dispatcher options —
getDispatcherOptions(hostname?) shortens
keepAliveMaxTimeout to 1000ms and autoSelectFamilyAttemptTimeout to
200ms when the hostname matches *.internal / *.local. New
exported isLocalEgressHostname() helper.

2. Parallel cache for local-egress dispatchers —
proxyDispatcherCache.ts adds LOCAL_DEFAULT_DISPATCHER_KEY /
LOCAL_RETRY_DISPATCHER_KEY with matching accessors.
getDefaultDispatcher(hostname?) / getRetryDispatcher(hostname?)
route to them when the hostname is local-egress. The cloud-upstream
pool keeps its wider keep-alive.

3. Cache invalidation on PROXY_UNREACHABLE for local egress —
proxyFetch.ts calls clearDispatcherCache() before _nativeFallback
when the error is PROXY_UNREACHABLE and the target hostname is
local-egress, forcing the next request to rebuild the pool with fresh
sockets.

Files

  • open-sse/utils/proxyDispatcher.ts (+63 / −8)
  • open-sse/utils/proxyDispatcherCache.ts (+29)
  • open-sse/utils/proxyFetch.ts (+22 / −1)

Backward compatibility

Every existing caller of getDefaultDispatcher() /
getRetryDispatcher() / __getDefaultDispatcherOptionsForTest() keeps
working — the hostname? parameter is optional and defaults to the
existing cloud behaviour. No config / env / DB changes.

Verification

Scope Pass Fail Skipped
Targeted proxy/dispatcher/executor suite (39 files) 742 0 1
proxyfetch-direct-response-start-timeout-10214 ✔
direct-dispatcher-pipelining-4580 ✔
proxy-egress-isolation-bdd ✔
proxy-fetch ✔
proxy-concurrency-keepalive-regression ✔
web-cookie-validation-proxy-7058 ✔
socks-connect-timeout-e2e ✔

Pre-existing failures on the cycle (uc-video persona-timeout,
models-catalog Jina/GLM-5.2, tunnel-routes sanitize, zai-stream
error code) reproduce on main unchanged — none introduced by this PR.

Notes

Cherry-picked onto release/v3.8.51 from a separate working branch.
One conflict in proxyFetch.ts resolved by taking upstream's rename of
the inline tlsProfileForProvider return type to a named
TlsProfileResult alias (no semantic change).

The companion AgentBridge MITM autostart and call_logs.id UNIQUE race
fixes are intentionally not included here — they're independent fixes
and will land separately.

Related work (not duplicates)

These recently merged neighbours touch nearby layers but do not
supersede this PR — included here to head off the "is this duplicate?"
review question:

None of the above touches the three properties this PR changes: (1) hostname-keyed getDispatcherOptions, (2) parallel LOCAL_*_DISPATCHER_KEY cache slots, (3) clearDispatcherCache() invocation on PROXY_UNREACHABLE for local-egress.

Docker Desktop NAT silently drops idle keep-alive sockets inside the
direct round-robin pool, and getDefaultDispatcher() never reaps them on
PROXY_UNREACHABLE — every .internal/.local hostname (host.docker.internal
plus all local LLM providers wired through it) can hit a 30s ECONNREFUSED
burst after the keepAliveMaxTimeout window expires.

Three coordinated changes:

1. Hostname-aware dispatcher options. getDispatcherOptions() now accepts
   the target hostname and shortens keepAliveMaxTimeout to 1000ms and
   autoSelectFamilyAttemptTimeout to 200ms when the hostname matches
   *.internal / *.local. The latter matters because the IPv6 form of
   host.docker.internal (fdc4:f303:9324::254) is ENETUNREACH inside the
   container; the default 1s Happy-Eyeballs wait was pure latency on
   every healthy request.

2. Parallel cache for local-egress dispatchers. Added LOCAL_DEFAULT /
   LOCAL_RETRY_DISPATCHER_KEY in proxyDispatcherCache and routed
   getDefaultDispatcher(hostname?) / getRetryDispatcher(hostname?) to
   them when isLocalEgressHostname(hostname). The cloud-upstream pool
   keeps its wider keep-alive; the local-egress pool gets the tighter
   settings without contaminating each other.

3. Cache invalidation on PROXY_UNREACHABLE for local egress. proxyFetch
   now calls clearDispatcherCache() before _nativeFallback when the
   target hostname is local-egress and the error is a proxy unreachable,
   forcing the next request to rebuild the pool with fresh sockets.

Backward compatible: every existing caller of getDefaultDispatcher() /
getRetryDispatcher() / __getDefaultDispatcherOptionsForTest() continues
to work — the hostname parameter is optional and defaults to the cloud
behaviour.

Cherry-picked onto release/v3.8.51; one conflict in proxyFetch.ts
resolved by taking upstream rename of TlsProfileResult to a named type
alias (no semantic change).
…apw#14315

isLocalEgressHostname(), the LOCAL_DEFAULT/LOCAL_RETRY dispatcher cache
routing, the shortened local-egress timeouts, and the
PROXY_UNREACHABLE-on-local-egress clearDispatcherCache() branch in
proxyFetch.ts had zero test coverage. Adds a focused suite covering the
4 cases flagged in review: hostname boundary matching, the
hostname-branched dispatcher options, LOCAL_* cache-slot routing (vs.
the shared DEFAULT/RETRY slots), clearDispatcherCache() clearing both
local slots, and that a local-egress PROXY_UNREACHABLE tears down the
local dispatcher pool while a cloud-upstream one does not.

Also prunes a stale eslint-suppressions.json entry for
proxyDispatcher.ts (an unused-vars violation the branch's own diff had
already fixed; the frozen count no longer matched reality and blocked
lint-staged).

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@diegosouzapw
diegosouzapw merged commit 3cbc9c9 into diegosouzapw:release/v3.8.51 Sep 29, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants