Repository navigation
fix(sse): propagate client abort to the upstream request after headers - #14342
Merged
diegosouzapw merged 2 commits intoSep 24, 2026
Conversation
Three breaks let an upstream keep working on an abandoned streamed request: - executeWithUpstreamStartTimeout removed its client-abort link as soon as headers arrived, so a later disconnect never reached the fetch signal. - createDisconnectAwareStream.pull() closed the client stream on disconnect without cancelling the upstream reader. - The combo streaming quality peek awaited reader.read() with no abort awareness, holding the attempt open until the first upstream byte.
Collaborator
|
Thanks @VIPKaiser — this closes a real gap: previously a client abort that arrived after |
diegosouzapw
merged commit Sep 24, 2026
5e5773a
into
cheaper-inference:release/v3.8.51
3 checks passed
SCys
pushed a commit
to SCys/OmniRoute
that referenced
this pull request
Sep 25, 2026
…no live body Root cause: cheaper-inference#14342 changed executeWithUpstreamStartTimeout to keep the client-signal `abortListener` (the link that aborts combinedController) whenever the start-timeout race settled successfully, so a client abort after headers could still reach the upstream fetch. It kept the link for every resolved result, including results that carry no streaming body, so one listener stayed on the client signal after the race settled. That broke the cheaper-inference#12406 guard ("every listener registered for the race must be removed once it settles", 1 !== 0). Fix: keep the link only when the settled result (a Response or the executor's `{ response }` wrapper) still has an unconsumed body that will stream on the combined signal. Otherwise remove it in the finally, as for failed attempts. The abortPromise listener is still always removed. The cheaper-inference#14342 post-headers propagation is unchanged for streaming responses. Tests: the cheaper-inference#12406 guard is green again; two new cases pin both sides (a body-less Response releases the link; a streaming body keeps exactly one link, propagates the abort, and it self-removes on abort). Refs cheaper-inference#14547
diegosouzapw
added a commit
that referenced
this pull request
Sep 29, 2026
…leak, MCP bundle deadlocks, sidebar keys, flush-empty-retry, proxy-status and pack-policy tests (#14820) Fixes the reds still present on the tip: client-abort listener leak (#14342 vs #12406), MCP bundle deadlock (quotaCache init cycle via the quotaCacheState leaf, plus the new #14892 proxyLogs -> upstreamStatusCapture -> usage/migrations cycle via the zero-import timingMs leaf), sidebar Model catalog keys, and the stale flush-empty-retry / proxy-status / pack-policy tests. Combo pre-content retry and zh-TW glossary were already fixed on the tip and dropped. Merged tree: 102/102 focused incl. mcp-bundle-startup (red on the tip), typecheck:core clean, open-sse typecheck 0, check:cycles OK, file-size OK.
thinh0704hcm
pushed a commit
to thinh0704hcm/OmniRoute
that referenced
this pull request
Oct 4, 2026
…per-inference#14342) Our port rewrote the resolve-settles test to expect an unconditional kept abort link, but upstream's rule is conditional: the link is kept only while the settled Response still has a live body. The rewritten expectation failed (0 listeners vs 1) for a plain string result. Upstream's own keep-link tests already cover post-resolve abort propagation, and all 9 pass against our null-safe implementation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a client aborts a streamed
/v1/chat/completions, OmniRoute never cancels the upstreamrequest. A local upstream (oMLX) keeps prefilling/decoding the abandoned request and blocks the
next one.
Observed (oMLX server log, single-slot local backend):
Three independent breaks, all fixed here:
open-sse/handlers/chatCore/upstreamTimeouts.tsexecuteWithUpstreamStartTimeout: links theclient signal to the fetch signal, but its
finallyremoved that listener as soon as thePromise.racesettled (headers received). The timeout is meant to bound time-to-headers only;a later client abort no longer reached the fetch. The abort link now stays for the fetch
lifetime on success; only the timer is cleared. It is still dropped when the attempt fails.
(The
oncelistener lives on the per-request signal, so it is bounded.)open-sse/utils/streamHandler.tscreateDisconnectAwareStream().pull(): on!isConnected()it closed the client-facing stream but never cancelled the upstream reader(only
cancel()did). It now cancels the reader/writer withclient_disconnectedfirst,except after a client terminal marker or while a completed tool handoff drain owns the reader.
open-sse/services/combo/validateQuality.tsvalidateResponseQualitypeek loop:await reader.read()was not abort-aware; a stalled prefill held the combo attempt until the firstbyte. It now takes an optional
signal(passed fromruntimeUnits.ts), races each read againstit, cancels the peek reader on abort and returns
{ valid: true }(a client abort is not aprovider failure, so no failover/cooldown; the combo loop's existing
signal.abortedcheck thenreturns 499).
Usage accounting
No change to conventions.
handleDisconnectalready finalizes the pending call as 499client_disconnectedbefore either abort propagates, and that is the recorded outcome. A cancelledupstream never delivers its final usage chunk, so an aborted stream records no upstream usage; I did
not add partial-token estimation. Non-aborted streams take the same code path as before (covered by
tests). Happy to add estimation from tokens seen if maintainers want it.
Tests Added Or Updated
tests/unit/client-abort-propagates-upstream.test.ts(new, 8 tests):socket closes early (fails without fix 1).
pipeWithDisconnect: disconnect cancels the upstream body; a normal stream is forwardedintact and not cancelled.
validateResponseQuality: exits promptly and cancels the reader on abort (stalled body), returnsimmediately on an already-aborted signal, unchanged for a normal stream with a signal.
stream-handler*,stream-disconnect*,codex-tool-handoff*,streaming-empty*,combo*(389 tests, 0 fail),npm run typecheck:core,npm run check:open-sse-typecheck, eslint + prettier on touched files.Reviewer Notes
streamController.signal),but covers executors that ignore the signal.
Response.clone()tee branch does not by itself abort the upstream; that is whyfix 1 is the load-bearing change. Peek cancellation intentionally is not awaited.
changelog.d/fixes/0000-propagate-client-abort-upstream.md(rename to the PR number).