Skip to content

fix(sse): surface OpenRouter mid-stream error chunks instead of a false empty success - #8210

Merged
diegosouzapw merged 3 commits into
cheaper-inference:release/v3.8.49from
hartmark:fix/openrouter-midstream-error-surfacing
Jul 23, 2026
Merged

diegosouzapw merged 3 commits into
cheaper-inference:release/v3.8.49from
hartmark:fix/openrouter-midstream-error-surfacing

Conversation

@hartmark

Copy link
Copy Markdown
Contributor

Summary

OpenRouter (and similar OpenAI-compatible aggregators) can send an HTTP 200 SSE stream whose body carries a chat.completion.chunk with empty choices and a top-level error object instead of any delta — e.g. the underlying provider hitting its own capacity limit mid-request. The Responses-API response translator's !chunk.choices?.length branch treated this exactly like a legitimate trailing-usage/no-op chunk, so the stream silently ended with response.completed / error: null / output: [] — a false "successful but empty" response masking a real 502 provider_unavailable failure.

Live repro: a real production request (OpenRouter, nvidia/nemotron-3-ultra-550b-a55b:free) got HTTP 200 with 0 tokens in/out after Nvidia returned "Worker local total request limit reached (33/32)" mid-stream; the client saw an empty completed response with no indication anything failed, which surfaced downstream as a confusing agent-side "couldn't generate a response" failure with zero diagnostic signal.

Mirrors the existing Gemini mid-stream error fix (#4177): sets state.upstreamError from the in-band error chunk so stream.ts's existing upstreamError handling takes over (sendCompleted emits status: "failed" with a real error object instead of a fabricated empty success).

Test plan

  • tests/unit/openrouter-midstream-error-chunk.test.ts (new, 2 tests) — TDD: confirmed RED before the fix (chunk silently dropped, state.upstreamError never set), GREEN after. Covers the 502 case and a 429 rate-limit-coded case.
  • Full existing translator regression suite (responses-translation-fixes, responses-additional-tools, responses-chat-translation-gaps, responses-handler, responses-usage-trailing-6906, gemini-midstream-error-4177, gemini-midstream-responses, combo-responses-sse-failure-fallback) — all pass, no regressions.

…se empty success

OpenRouter (and similar OpenAI-compatible aggregators) can send an HTTP 200
SSE stream whose body carries a chat.completion.chunk with empty choices and
a top-level error object instead of any delta -- e.g. the underlying
provider hitting its own capacity limit mid-request. The Responses-API
response translator's `!chunk.choices?.length` branch treated this exactly
like a legitimate trailing-usage/no-op chunk, so the stream silently ended
with response.completed / error: null / output: [] -- a false "successful
but empty" response masking a real 502 provider_unavailable failure.

Live repro: request 1784726796287-a45bb3 (OpenClaw via the default combo,
nvidia/nemotron-3-ultra-550b-a55b:free via OpenRouter) got HTTP 200 with 0
tokens in/out after Nvidia returned "Worker local total request limit
reached (33/32)" mid-stream; the client saw an empty completed response
with no indication anything failed.

Mirrors the Gemini mid-stream error fix (cheaper-inference#4177): set state.upstreamError
from the in-band error chunk so stream.ts's existing upstreamError handling
takes over (sendCompleted emits status: "failed" with a real error object).

Co-Authored-By: Markus Hartung <markus.hartream@gmail.com>
@hartmark
hartmark requested a review from diegosouzapw as a code owner July 22, 2026 21:41
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

hartmark and others added 2 commits July 22, 2026 23:42
…8210)

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
# Conflicts:
#	config/quality/file-size-baseline.json
@diegosouzapw

Copy link
Copy Markdown
Collaborator

Thanks, @hartmark! 🙏 Merged — surfaces OpenRouter-style mid-stream aggregator errors (HTTP 200 SSE with empty choices + top-level error) as a real upstream error instead of a false empty-success, so combo fallback kicks in. Rebased onto the current release tip and rebaselined openai-responses.ts for the +26 own-growth. Test 2/2 green.

@diegosouzapw
diegosouzapw merged commit ebe086e into cheaper-inference:release/v3.8.49 Jul 23, 2026
@diegosouzapw diegosouzapw mentioned this pull request Jul 23, 2026
HouMinXi pushed a commit to HouMinXi/OmniRoute that referenced this pull request Aug 2, 2026
…se empty success (cheaper-inference#8210)

* fix(sse): surface OpenRouter mid-stream error chunks instead of a false empty success

OpenRouter (and similar OpenAI-compatible aggregators) can send an HTTP 200
SSE stream whose body carries a chat.completion.chunk with empty choices and
a top-level error object instead of any delta -- e.g. the underlying
provider hitting its own capacity limit mid-request. The Responses-API
response translator's `!chunk.choices?.length` branch treated this exactly
like a legitimate trailing-usage/no-op chunk, so the stream silently ended
with response.completed / error: null / output: [] -- a false "successful
but empty" response masking a real 502 provider_unavailable failure.

Live repro: request 1784726796287-a45bb3 (OpenClaw via the default combo,
nvidia/nemotron-3-ultra-550b-a55b:free via OpenRouter) got HTTP 200 with 0
tokens in/out after Nvidia returned "Worker local total request limit
reached (33/32)" mid-stream; the client saw an empty completed response
with no indication anything failed.

Mirrors the Gemini mid-stream error fix (cheaper-inference#4177): set state.upstreamError
from the in-band error chunk so stream.ts's existing upstreamError handling
takes over (sendCompleted emits status: "failed" with a real error object).

Co-Authored-By: Markus Hartung <markus.hartream@gmail.com>

* chore(quality): file-size baseline for own-growth (cheaper-inference#8210)

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: Markus Hartung <markus.hartream@gmail.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…se empty success (cheaper-inference#8210)

* fix(sse): surface OpenRouter mid-stream error chunks instead of a false empty success

OpenRouter (and similar OpenAI-compatible aggregators) can send an HTTP 200
SSE stream whose body carries a chat.completion.chunk with empty choices and
a top-level error object instead of any delta -- e.g. the underlying
provider hitting its own capacity limit mid-request. The Responses-API
response translator's `!chunk.choices?.length` branch treated this exactly
like a legitimate trailing-usage/no-op chunk, so the stream silently ended
with response.completed / error: null / output: [] -- a false "successful
but empty" response masking a real 502 provider_unavailable failure.

Live repro: request 1784726796287-a45bb3 (OpenClaw via the default combo,
nvidia/nemotron-3-ultra-550b-a55b:free via OpenRouter) got HTTP 200 with 0
tokens in/out after Nvidia returned "Worker local total request limit
reached (33/32)" mid-stream; the client saw an empty completed response
with no indication anything failed.

Mirrors the Gemini mid-stream error fix (cheaper-inference#4177): set state.upstreamError
from the in-band error chunk so stream.ts's existing upstreamError handling
takes over (sendCompleted emits status: "failed" with a real error object).

Co-Authored-By: Markus Hartung <markus.hartream@gmail.com>

* chore(quality): file-size baseline for own-growth (cheaper-inference#8210)

Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

---------

Co-authored-by: Markus Hartung <markus.hartream@gmail.com>
Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants