fix(sse): estimate usage in passthrough stream even with include_usage by maxmad64bis · Pull Request #12151 · cheaper-inference/OmniRoute

maxmad64bis · 2026-08-30T19:21:28Z

⚠️ base-red inherited: #12109

Summary

Streams in passthrough mode could end with no usage even though the client asked for it, so some providers always showed 0 tokens. The fix estimates usage at the finish marker when the upstream stays silent, and drops any duplicate trailing usage chunk.

Scenario: client sends stream:true with stream_options: {include_usage:true}, upstream closes with finish_reason: stop but no usage, and the proxy still returns usage to the client (flagged estimated:true). If the upstream later sends a trailing choices:[] usage:{...} it's ignored so the client doesn't see two usages.
open-sse/utils/stream.ts:1982,1749 · open-sse/utils/usageTracking.ts:651,664

Related Issues

Related to silent passthrough usage on providers that don't meter.

Validation

Change type: provider
Focused tests and category gates from the golden path
npm run lint (exit 0; npm install es-abstract@1.23.9 --no-save --no-package-lock in worktree only — lockfile still pins 1.24.1, base reproduces the same load error)
Reconciled with the current active release base; focused checks rerun afterward (origin/release/v3.8.51 at 2acbfc6)
Production-code changes include a new or updated automated test in this PR

Tests Added Or Updated

tests/unit/stream-passthrough-usage-estimation.test.ts — 6 cases: predicate (finish without usage but with content, trailing valid, empty response, tool-only) + 2 SSE harness cases via createSSEStream passthrough (finish without usage emits estimated:true, trailing valid after estimated is dropped).

Coverage Notes

Touches open-sse/utils/stream.ts, open-sse/utils/usageTracking.ts, open-sse/handlers/chatCore/clientUsageBuffer.ts through the new passthrough estimation branch and the shared isEmptyUsage/hasValidUsage helpers. Covered by the new test above and existing tests/unit/chatcore-client-usage-buffer.test.ts (10 cases). hasValidUsage now handles Gemini totalTokenCount so a single-field Gemini usage isn't misclassified.

Reviewer Notes

Tradeoff: open-sse/utils/stream.ts:1749,1984 uses a stream-scoped passthroughForwardedUsage guard (per-line injectedUsage resets each line, so it can't prevent a trailing choices:[] usage from forwarding a second time). If a finish without usage is already estimated, a valid trailing usage is dropped in favor of the estimate — wins for muse-spark (never sends trailing), pragmatic for OpenAI routes that do. Long-term alternative is deferring the finish, similar to open-sse/translator/response/openai-to-claude.ts.
tool_calls only (no text) keeps totalContentLength at 0, so no estimate — same as before, callers can revisit if needed.

The helper was leftover from cheaper-inference#12151 and fails the absolute lint:json --max-warnings 0 gate on every PR that includes it. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

…erscore Pre-existing cheaper-inference#12151 test file tripped the strict no-unused-vars gate (all three identifiers never referenced). Tests still pass.

… + stream gates Docs cheaper-inference#12320 added radarPage.colLimits / trainsOnPrompts / trainsOnPromptsHelp as __MISSING__ in 42 locales but never added the EN source. That broke two gates: i18n-missing-placeholder-fallback (cheaper-inference#7258, "__MISSING__ after EN fallback") and settings-i18n-keys ("direct translation calls have English messages"). Add the three EN keys (values are the sentinels' English text). cheaper-inference#12151 changed passthrough to estimate usage on finish without usage when content exists and to drop the trailing choices:[] valid usage once estimated. tests/unit/stream-utils "forwards OpenAI usage-only empty choices chunks" still expected the old forward. Update it to assert estimated:true and no choices:[] (same tradeoff documented in stream-passthrough-usage-estimation). Also drops the dead collectSSE helper in stream-passthrough-usage- estimation.test.ts (3 unused identifiers) that reds No new ESLint warnings — same as already-merged fix/release-v3.8.51-eslint-unused- collectsse. Reproduced on pure tip 438db55: 4 RED (ESLint, i18n-missing-fallback, settings-i18n-keys, stream-utils ×2 via TIA + shard 4/4). After: 4 suites pass locally (i18n-missing, settings-i18n-keys, stream- passthrough, stream-utils 40/40).

…e regression + radarPage i18n keys (#12327) * fix(sse): forward the upstream's real trailing usage in passthrough; estimate only at flush #12151 injected estimated usage into the finish chunk and dropped the real trailing usage block that genuine OpenAI upstreams send afterwards — metered clients got estimates instead of real token counts. The estimate now leaves via a canonical usage-only chunk at flush, only when the upstream stayed silent; a real trailing block is forwarded verbatim and wins. The tool_calls finish_reason normalization now materializes its own rewrite (it piggybacked on the removed finish-time rewrite), and the dead collectSSE helper goes with it (subsumes #12324). * fix(i18n): seed the radarPage limits/training keys the #12320 UI already consumes RadarCatalogTable.tsx references radarPage.colLimits / trainsOnPrompts / trainsOnPromptsHelp but #12320 never added them to en.json, so the EN fallback could not resolve the __MISSING__ markers across 42 locales. Real translations for pt-BR and vi; the rest resolve via the EN fallback. * fix(sse): carry the chat stream id into flush-time synthetic chunks The estimated usage-only chunk emitted at flush used passthroughResponsesId, which is only ever set on the Responses path — on the chat path the synthetic chunk shipped id: null, breaking the string-id invariant pinned by stream-numeric-ids.test.ts. Track the upstream chat-completion id in the passthrough loop and reuse it (falling back to the Responses id, then a generated one). Sibling sweep: 74 files importing utils/stream — 603/603.

* feat(ui): enable React Compiler (#67) Enable reactCompiler: true in next.config.mjs (Next 16 + React 19.2.8). This automates memoization at build time, removing manual useCallback/useMemo debt (591 + 283 instances respectively) and preventing stale-closure bugs. Test results (pre-existing failures unchanged): vitest UI: 282/295 files pass (13 fail = missing router/ReactFlow mocks) vitest: 1805/1857 tests pass (52 fail = same pre-existing mock issues) node:test: api/services/db all pass (except platform-specific serviceSupervisorSpawnError — Windows spawn("ls") issue) No new failures introduced by the compiler transform. Optional cleanup: remove now-redundant useCallback/useMemo in hot components. * fix(build): add babel-plugin-react-compiler peer dependency (#67) React Compiler (reactCompiler: true in next.config.mjs) requires babel-plugin-react-compiler as an explicit peer dependency — Next.js declares it as optional ("*") and does not auto-install it. Installed babel-plugin-react-compiler@1.0.0 as a devDependency. Resolves correctly from both the project root and the next package context (Turbopack resolution path). * fix(ci): allowlist babel-plugin-react-compiler for React Compiler The React Compiler peer is a real npm package (facebook/react, MIT) required by Next 16 `reactCompiler: true`. Adding it to the anti-slopsquat allowlist unblocks check:deps and the 6A.8 unit-test gate. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * fix(ci): drop unused collectSSE helper that trips ESLint The helper was leftover from #12151 and fails the absolute lint:json --max-warnings 0 gate on every PR that includes it. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> --------- Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> Co-authored-by: WebPerson <jonlwheat2-gif@users.noreply.github.com>

cheaper-inference#12151 landed with a never-called collectSSE helper in stream-passthrough-usage-estimation.test.ts; its three unused identifiers (collectSSE / stream / writable) fail 'npm run lint:json -- --max-warnings 0', so the 'No new ESLint warnings' job reds every PR merge-ref cut from the current tip (first seen on PR cheaper-inference#11564's post-sync run). createSSEStream and parseSSEUsage are actually used and stay; only the dead closure goes. File suite still passes 6/6. (cherry picked from commit 3b1520b) Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com>

…ream stays silent (#12828) Validado numa worktree combinada com a onda de streaming desta leva sobre `release/v3.8.51`: typecheck:core limpo, check-api-typecheck OK (289), check-file-size OK após rebaseline, 88/88 nos testes focados. O #12151 cobriu só metade: passthrough emitia o chunk final de usage, translate calculava a estimativa **depois** de fechar o stream, então o número só chegava ao log do servidor e nunca ao cliente. Fechar essa metade é o que faz a feature existir de fato. Não emitir segundo chunk quando o upstream já mandou usage real é o detalhe que impede a correção de virar contagem dobrada.

@maxmad64bis

cheaper-inference#12151) A passthrough stream could end with no usage even though the client asked for it via stream_options: {include_usage: true}, so providers that do not meter always showed 0 tokens. The fix estimates usage at the finish marker when the upstream stays silent (flagged estimated: true) and drops any duplicate trailing usage chunk so the client never sees two. open-sse/utils/stream.ts:1982,1749 · open-sse/utils/usageTracking.ts:651,664 Six cases: the predicate (finish without usage but with content, trailing valid, empty response, tool-only) plus two SSE harness cases through createSSEStream passthrough. Note on base: this branch forked 442 commits back and carried a base-red marker for cheaper-inference#12109, which is now closed — the release tip has no open base-red issue. It merged cleanly against the current tip regardless. Verified in a combined batch worktree: 174/174 focused tests across all 11 PRs of this batch (this PR's stream-passthrough-usage-estimation suite included), typecheck:core clean, check:cycles and check:docs-counts green. Thanks @maxmad64bis.

…e regression + radarPage i18n keys (cheaper-inference#12327) * fix(sse): forward the upstream's real trailing usage in passthrough; estimate only at flush cheaper-inference#12151 injected estimated usage into the finish chunk and dropped the real trailing usage block that genuine OpenAI upstreams send afterwards — metered clients got estimates instead of real token counts. The estimate now leaves via a canonical usage-only chunk at flush, only when the upstream stayed silent; a real trailing block is forwarded verbatim and wins. The tool_calls finish_reason normalization now materializes its own rewrite (it piggybacked on the removed finish-time rewrite), and the dead collectSSE helper goes with it (subsumes cheaper-inference#12324). * fix(i18n): seed the radarPage limits/training keys the cheaper-inference#12320 UI already consumes RadarCatalogTable.tsx references radarPage.colLimits / trainsOnPrompts / trainsOnPromptsHelp but cheaper-inference#12320 never added them to en.json, so the EN fallback could not resolve the __MISSING__ markers across 42 locales. Real translations for pt-BR and vi; the rest resolve via the EN fallback. * fix(sse): carry the chat stream id into flush-time synthetic chunks The estimated usage-only chunk emitted at flush used passthroughResponsesId, which is only ever set on the Responses path — on the chat path the synthetic chunk shipped id: null, breaking the string-id invariant pinned by stream-numeric-ids.test.ts. Track the upstream chat-completion id in the passthrough loop and reuse it (falling back to the Responses id, then a generated one). Sibling sweep: 74 files importing utils/stream — 603/603.

…r-inference#11783) * feat(ui): enable React Compiler (cheaper-inference#67) Enable reactCompiler: true in next.config.mjs (Next 16 + React 19.2.8). This automates memoization at build time, removing manual useCallback/useMemo debt (591 + 283 instances respectively) and preventing stale-closure bugs. Test results (pre-existing failures unchanged): vitest UI: 282/295 files pass (13 fail = missing router/ReactFlow mocks) vitest: 1805/1857 tests pass (52 fail = same pre-existing mock issues) node:test: api/services/db all pass (except platform-specific serviceSupervisorSpawnError — Windows spawn("ls") issue) No new failures introduced by the compiler transform. Optional cleanup: remove now-redundant useCallback/useMemo in hot components. * fix(build): add babel-plugin-react-compiler peer dependency (cheaper-inference#67) React Compiler (reactCompiler: true in next.config.mjs) requires babel-plugin-react-compiler as an explicit peer dependency — Next.js declares it as optional ("*") and does not auto-install it. Installed babel-plugin-react-compiler@1.0.0 as a devDependency. Resolves correctly from both the project root and the next package context (Turbopack resolution path). * fix(ci): allowlist babel-plugin-react-compiler for React Compiler The React Compiler peer is a real npm package (facebook/react, MIT) required by Next 16 `reactCompiler: true`. Adding it to the anti-slopsquat allowlist unblocks check:deps and the 6A.8 unit-test gate. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> * fix(ci): drop unused collectSSE helper that trips ESLint The helper was leftover from cheaper-inference#12151 and fails the absolute lint:json --max-warnings 0 gate on every PR that includes it. Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> --------- Co-authored-by: diegosouzapw <8016841+diegosouzapw@users.noreply.github.com> Co-authored-by: WebPerson <jonlwheat2-gif@users.noreply.github.com>

…ream stays silent (cheaper-inference#12828) Validado numa worktree combinada com a onda de streaming desta leva sobre `release/v3.8.51`: typecheck:core limpo, check-api-typecheck OK (289), check-file-size OK após rebaseline, 88/88 nos testes focados. O cheaper-inference#12151 cobriu só metade: passthrough emitia o chunk final de usage, translate calculava a estimativa **depois** de fechar o stream, então o número só chegava ao log do servidor e nunca ao cliente. Fechar essa metade é o que faz a feature existir de fato. Não emitir segundo chunk quando o upstream já mandou usage real é o detalhe que impede a correção de virar contagem dobrada.

maxmad64bis requested a review from diegosouzapw as a code owner August 30, 2026 19:21

maxmad64bis force-pushed the fix/stream-passthrough-usage-estimation branch from 271f8f6 to 0430001 Compare August 30, 2026 19:22

fix(sse): estimate usage in passthrough stream even with include_usage

e29540b

maxmad64bis force-pushed the fix/stream-passthrough-usage-estimation branch from 0430001 to e29540b Compare August 30, 2026 19:50

diegosouzapw merged commit 5e6c9a9 into cheaper-inference:release/v3.8.51 Sep 1, 2026
8 of 16 checks passed

This was referenced Sep 1, 2026

feat(sse): support native max reasoning effort and per-model clamping #11875

Merged

fix(release): drain the 2026-09-01 base-red window — passthrough usage regression + radarPage i18n keys #12327

Merged

maxmad64bis mentioned this pull request Sep 1, 2026

fix(quality): clear base-reds on release/v3.8.51 — radar i18n EN keys + stream gates #12331

Closed

6 tasks done

maxmad64bis mentioned this pull request Sep 5, 2026

fix(sse): emit trailing usage estimate in translate streams when upstream stays silent #12828

Merged

4 of 5 tasks

patrykkopycinski mentioned this pull request Sep 6, 2026

test(sse): drop the dead collectSSE helper that reds the lint gate patrykkopycinski/OmniRoute#11

Merged

maxmad64bis deleted the fix/stream-passthrough-usage-estimation branch September 24, 2026 21:15

claw-io mentioned this pull request Oct 5, 2026

feat(upstream): import v3.8.51 Pass 2 core alignment (18 PRs) b3nw/omniroute-slim#19

Merged

huangyoudashi mentioned this pull request Oct 6, 2026

fix(backend): passthrough usage-dedup drops the real trailing usage-only summary - cache reads recorded as 0 (regression from #12151) #15619

Closed

diegosouzapw mentioned this pull request Oct 8, 2026

fix(stream): keep real trailing usage-only summary in passthrough (cache reads) (#15619) #15826

Merged

skygunner mentioned this pull request Oct 22, 2026

fix(api): relay the OpenAI usage-only chunk and token details on /v1beta (#17841) #17943

Open

Navigation Menu

Repository navigation

fix(sse): estimate usage in passthrough stream even with include_usage - #12151

fix(sse): estimate usage in passthrough stream even with include_usage#12151
diegosouzapw merged 1 commit into
cheaper-inference:release/v3.8.51from
maxmad64bis:fix/stream-passthrough-usage-estimation

maxmad64bis commented Aug 30, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

maxmad64bis commented Aug 30, 2026

Summary

Related Issues

Validation

Tests Added Or Updated

Coverage Notes

Reviewer Notes

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants