Repository navigation
fix(api): stop the 2000-token safety buffer from inflating usage.prompt_tokens in the client response (#8331) - #8356
Merged
Conversation
…pt_tokens in the client response (#8331)
…e providers The #8331 fix correctly stopped folding the 2000-token context-window safety margin into client-visible prompt_tokens/input_tokens/total_tokens for normal API metering clients. But it also silently changed the response shape for Claude-Code-compatible providers, whose own context accounting reads the buffered number straight out of usage — regressing tests/unit/cc-compatible-provider.test.ts (expected 2007, got 7). Fold the computed context_budget_* fields back into the visible usage fields for that one path only (applyClientUsageBuffer's new preserveContextBudgetInVisibleUsage option, gated on the existing isClaudeCodeCompatible flag in chatCore.ts). Every other caller keeps the real, unbuffered #8331 numbers.
diegosouzapw
pushed a commit
that referenced
this pull request
Jul 25, 2026
* test(sse): repair two base-red gates on release/v3.8.49 `release/v3.8.49` is red at its own HEAD (36f8fd1) — `Quality Gates` and `Release-Green (continuous)` both fail — so every PR opened against it inherits four failing checks regardless of content. Two of those causes had no owner: #8480/#8481/#8482 cover the compression ladder, the antigravity catalog, and the resilience/translator regressions respectively, none of them these. 1. `chatcore-client-usage-buffer.test.ts` — 5/5 failing. #8331/#8356 (e871978) inserted an `options` parameter between `clientResponseFormat` and `deps` on `applyClientUsageBuffer()`. The five call sites here still passed `deps` in the fourth position, so the injected spies landed in the `options` slot and the real implementations ran — every `calls.*.length` assertion saw 0. The `Parameters<typeof applyClientUsageBuffer>[3]` cast in `makeDeps()` masked the type error, which is why it reached the branch. Call sites updated to `(resp, body, format, {}, deps)` and the cast repointed to `[4]`. Two cases added for the parameter that caused this, since nothing at this layer exercised it: `preserveContextBudgetInVisibleUsage` re-folds `context_budget_*` into the visible fields for the Claude-Code path, and the default path keeps the real unbuffered #8331 numbers. 2. `claude-to-openai-think-close-5123.test.ts` — 4 unsuppressed `@typescript-eslint/no-explicit-any` errors, failing `npm run lint`. Its suppression entry allows `count: 2` but the file had grown to four `(chunk: any)` callbacks. Rather than raise the frozen count, the cause is fixed: `collectChunks()` returns `StreamChunk[]` instead of `unknown[]`, narrowing once at the boundary so all four callbacks need no annotation. The suppression entry is then stale and removed, which ratchets the file to zero. Test-only plus one allowlist deletion; no production code. Verified on a clean worktree of the base commit: `npm run lint` clean (was 4 errors), `chatcore-client-usage-buffer` 7/7 (was 0/5), `claude-to-openai-think-close-5123` 3/3. * chore(ci): rebaseline five inherited file-size overages on release/v3.8.49 `check:file-size` fails on release/v3.8.49 at its own HEAD (36f8fd1), which is what turns `Fast Quality Gates` red for every PR against this base: tokenHealthCheck.ts 832 -> 841 chat.ts 1865 -> 1866 auth.ts 2475 -> 2486 accountFallback.ts 1941 -> 1960 combo.ts 3630 -> 3642 None of these files is touched by this PR. The growth was inherited from already merged PRs that did not bump their entries, so there is no offending branch left to fix — the same situation the baseline already records under `_rebaseline_2026_07_02_5798_release_green`, and the procedure it documents is to raise the frozen values with a justification note. Raised to the current base values only. The files stay frozen and cannot grow further; an in-flight PR that adds lines to them bumps its own entry as usual (#8482 touches accountFallback.ts and combo.ts and will need that). * chore(ci): register 11 covering unit tests in stryker tap.testFiles `check:mutation-test-coverage --strict` fails on release/v3.8.49 at its own HEAD: 11 unit tests that cover mutated modules are absent from `stryker.conf.json` `tap.testFiles`, so their mutant kills do not count. The gate does not self-heal — it names the exact files to add. open-sse/services/accountFallback.ts + 8247-accountfallback-model-unhealthy, 8248-accountfallback-nvidia-degraded, model-lockout-exact-cooldown-cap, repro-antigravity-404-family-cooldown-hijack src/sse/services/auth.ts + 7993-noauth-proxy-routing, 8200-perplexity-web-401-cooldown, sse-auth-antigravity-credits src/server/authz/routeGuard.ts + authz/route-guard-vnc-session-local-only open-sse/utils/error.ts + error-sensitive-redaction open-sse/utils/publicCreds.ts + adobe-firefly src/shared/utils/circuitBreaker.ts + 8332-combo-vision-fallback None is a file this PR touches, and the gate reports the identical 11 on a worktree carrying none of these base-red fixes. Additions only (11 insertions, 0 deletions); the array stays sorted. This widens what the mutation run accounts for rather than relaxing anything. * chore(ci): re-measure accountFallback.ts file-size cap against the current base tip The entry frozen in this PR (1960) was the value at 36f8fd1; the base has since advanced to 1cafd32 and the file is 1966 there, so check:file-size would still have been red on the merge commit. Re-measured to 1966. Same inherited drift the note already documents: check:file-size does not run on the PR->release fast path, so growth accrues unmeasured between release rebaselines. The other four entries still match the current tip (tokenHealthCheck 841, chat 1866, auth 2486, combo frozen 3642 >= 3640). --------- Co-authored-by: backryun <busan011@ormbiz.co.kr>
HouMinXi
pushed a commit
to HouMinXi/OmniRoute
that referenced
this pull request
Aug 2, 2026
…pt_tokens in the client response (cheaper-inference#8331) (cheaper-inference#8356) * fix(api): stop the 2000-token safety buffer from inflating usage.prompt_tokens in the client response (cheaper-inference#8331) * fix(sse): scope cheaper-inference#8331's usage-buffer fix around Claude-Code-compatible providers The cheaper-inference#8331 fix correctly stopped folding the 2000-token context-window safety margin into client-visible prompt_tokens/input_tokens/total_tokens for normal API metering clients. But it also silently changed the response shape for Claude-Code-compatible providers, whose own context accounting reads the buffered number straight out of usage — regressing tests/unit/cc-compatible-provider.test.ts (expected 2007, got 7). Fold the computed context_budget_* fields back into the visible usage fields for that one path only (applyClientUsageBuffer's new preserveContextBudgetInVisibleUsage option, gated on the existing isClaudeCodeCompatible flag in chatCore.ts). Every other caller keeps the real, unbuffered cheaper-inference#8331 numbers.
HouMinXi
pushed a commit
to HouMinXi/OmniRoute
that referenced
this pull request
Aug 2, 2026
…rence#8490) * test(sse): repair two base-red gates on release/v3.8.49 `release/v3.8.49` is red at its own HEAD (d61504d) — `Quality Gates` and `Release-Green (continuous)` both fail — so every PR opened against it inherits four failing checks regardless of content. Two of those causes had no owner: cheaper-inference#8480/cheaper-inference#8481/cheaper-inference#8482 cover the compression ladder, the antigravity catalog, and the resilience/translator regressions respectively, none of them these. 1. `chatcore-client-usage-buffer.test.ts` — 5/5 failing. cheaper-inference#8331/cheaper-inference#8356 (f698fec) inserted an `options` parameter between `clientResponseFormat` and `deps` on `applyClientUsageBuffer()`. The five call sites here still passed `deps` in the fourth position, so the injected spies landed in the `options` slot and the real implementations ran — every `calls.*.length` assertion saw 0. The `Parameters<typeof applyClientUsageBuffer>[3]` cast in `makeDeps()` masked the type error, which is why it reached the branch. Call sites updated to `(resp, body, format, {}, deps)` and the cast repointed to `[4]`. Two cases added for the parameter that caused this, since nothing at this layer exercised it: `preserveContextBudgetInVisibleUsage` re-folds `context_budget_*` into the visible fields for the Claude-Code path, and the default path keeps the real unbuffered cheaper-inference#8331 numbers. 2. `claude-to-openai-think-close-5123.test.ts` — 4 unsuppressed `@typescript-eslint/no-explicit-any` errors, failing `npm run lint`. Its suppression entry allows `count: 2` but the file had grown to four `(chunk: any)` callbacks. Rather than raise the frozen count, the cause is fixed: `collectChunks()` returns `StreamChunk[]` instead of `unknown[]`, narrowing once at the boundary so all four callbacks need no annotation. The suppression entry is then stale and removed, which ratchets the file to zero. Test-only plus one allowlist deletion; no production code. Verified on a clean worktree of the base commit: `npm run lint` clean (was 4 errors), `chatcore-client-usage-buffer` 7/7 (was 0/5), `claude-to-openai-think-close-5123` 3/3. * chore(ci): rebaseline five inherited file-size overages on release/v3.8.49 `check:file-size` fails on release/v3.8.49 at its own HEAD (d61504d), which is what turns `Fast Quality Gates` red for every PR against this base: tokenHealthCheck.ts 832 -> 841 chat.ts 1865 -> 1866 auth.ts 2475 -> 2486 accountFallback.ts 1941 -> 1960 combo.ts 3630 -> 3642 None of these files is touched by this PR. The growth was inherited from already merged PRs that did not bump their entries, so there is no offending branch left to fix — the same situation the baseline already records under `_rebaseline_2026_07_02_5798_release_green`, and the procedure it documents is to raise the frozen values with a justification note. Raised to the current base values only. The files stay frozen and cannot grow further; an in-flight PR that adds lines to them bumps its own entry as usual (cheaper-inference#8482 touches accountFallback.ts and combo.ts and will need that). * chore(ci): register 11 covering unit tests in stryker tap.testFiles `check:mutation-test-coverage --strict` fails on release/v3.8.49 at its own HEAD: 11 unit tests that cover mutated modules are absent from `stryker.conf.json` `tap.testFiles`, so their mutant kills do not count. The gate does not self-heal — it names the exact files to add. open-sse/services/accountFallback.ts + 8247-accountfallback-model-unhealthy, 8248-accountfallback-nvidia-degraded, model-lockout-exact-cooldown-cap, repro-antigravity-404-family-cooldown-hijack src/sse/services/auth.ts + 7993-noauth-proxy-routing, 8200-perplexity-web-401-cooldown, sse-auth-antigravity-credits src/server/authz/routeGuard.ts + authz/route-guard-vnc-session-local-only open-sse/utils/error.ts + error-sensitive-redaction open-sse/utils/publicCreds.ts + adobe-firefly src/shared/utils/circuitBreaker.ts + 8332-combo-vision-fallback None is a file this PR touches, and the gate reports the identical 11 on a worktree carrying none of these base-red fixes. Additions only (11 insertions, 0 deletions); the array stays sorted. This widens what the mutation run accounts for rather than relaxing anything. * chore(ci): re-measure accountFallback.ts file-size cap against the current base tip The entry frozen in this PR (1960) was the value at d61504d; the base has since advanced to 59ef81b and the file is 1966 there, so check:file-size would still have been red on the merge commit. Re-measured to 1966. Same inherited drift the note already documents: check:file-size does not run on the PR->release fast path, so growth accrues unmeasured between release rebaselines. The other four entries still match the current tip (tokenHealthCheck 841, chat 1866, auth 2486, combo frozen 3642 >= 3640). --------- Co-authored-by: backryun <busan011@ormbiz.co.kr>
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…pt_tokens in the client response (cheaper-inference#8331) (cheaper-inference#8356) * fix(api): stop the 2000-token safety buffer from inflating usage.prompt_tokens in the client response (cheaper-inference#8331) * fix(sse): scope cheaper-inference#8331's usage-buffer fix around Claude-Code-compatible providers The cheaper-inference#8331 fix correctly stopped folding the 2000-token context-window safety margin into client-visible prompt_tokens/input_tokens/total_tokens for normal API metering clients. But it also silently changed the response shape for Claude-Code-compatible providers, whose own context accounting reads the buffered number straight out of usage — regressing tests/unit/cc-compatible-provider.test.ts (expected 2007, got 7). Fold the computed context_budget_* fields back into the visible usage fields for that one path only (applyClientUsageBuffer's new preserveContextBudgetInVisibleUsage option, gated on the existing isClaudeCodeCompatible flag in chatCore.ts). Every other caller keeps the real, unbuffered cheaper-inference#8331 numbers.
muhamadgalihsaputra
pushed a commit
to niyatna/NiyatnaRoute
that referenced
this pull request
Sep 27, 2026
…rence#8490) * test(sse): repair two base-red gates on release/v3.8.49 `release/v3.8.49` is red at its own HEAD (75eb952) — `Quality Gates` and `Release-Green (continuous)` both fail — so every PR opened against it inherits four failing checks regardless of content. Two of those causes had no owner: cheaper-inference#8480/cheaper-inference#8481/cheaper-inference#8482 cover the compression ladder, the antigravity catalog, and the resilience/translator regressions respectively, none of them these. 1. `chatcore-client-usage-buffer.test.ts` — 5/5 failing. cheaper-inference#8331/cheaper-inference#8356 (d4ab3af) inserted an `options` parameter between `clientResponseFormat` and `deps` on `applyClientUsageBuffer()`. The five call sites here still passed `deps` in the fourth position, so the injected spies landed in the `options` slot and the real implementations ran — every `calls.*.length` assertion saw 0. The `Parameters<typeof applyClientUsageBuffer>[3]` cast in `makeDeps()` masked the type error, which is why it reached the branch. Call sites updated to `(resp, body, format, {}, deps)` and the cast repointed to `[4]`. Two cases added for the parameter that caused this, since nothing at this layer exercised it: `preserveContextBudgetInVisibleUsage` re-folds `context_budget_*` into the visible fields for the Claude-Code path, and the default path keeps the real unbuffered cheaper-inference#8331 numbers. 2. `claude-to-openai-think-close-5123.test.ts` — 4 unsuppressed `@typescript-eslint/no-explicit-any` errors, failing `npm run lint`. Its suppression entry allows `count: 2` but the file had grown to four `(chunk: any)` callbacks. Rather than raise the frozen count, the cause is fixed: `collectChunks()` returns `StreamChunk[]` instead of `unknown[]`, narrowing once at the boundary so all four callbacks need no annotation. The suppression entry is then stale and removed, which ratchets the file to zero. Test-only plus one allowlist deletion; no production code. Verified on a clean worktree of the base commit: `npm run lint` clean (was 4 errors), `chatcore-client-usage-buffer` 7/7 (was 0/5), `claude-to-openai-think-close-5123` 3/3. * chore(ci): rebaseline five inherited file-size overages on release/v3.8.49 `check:file-size` fails on release/v3.8.49 at its own HEAD (75eb952), which is what turns `Fast Quality Gates` red for every PR against this base: tokenHealthCheck.ts 832 -> 841 chat.ts 1865 -> 1866 auth.ts 2475 -> 2486 accountFallback.ts 1941 -> 1960 combo.ts 3630 -> 3642 None of these files is touched by this PR. The growth was inherited from already merged PRs that did not bump their entries, so there is no offending branch left to fix — the same situation the baseline already records under `_rebaseline_2026_07_02_5798_release_green`, and the procedure it documents is to raise the frozen values with a justification note. Raised to the current base values only. The files stay frozen and cannot grow further; an in-flight PR that adds lines to them bumps its own entry as usual (cheaper-inference#8482 touches accountFallback.ts and combo.ts and will need that). * chore(ci): register 11 covering unit tests in stryker tap.testFiles `check:mutation-test-coverage --strict` fails on release/v3.8.49 at its own HEAD: 11 unit tests that cover mutated modules are absent from `stryker.conf.json` `tap.testFiles`, so their mutant kills do not count. The gate does not self-heal — it names the exact files to add. open-sse/services/accountFallback.ts + 8247-accountfallback-model-unhealthy, 8248-accountfallback-nvidia-degraded, model-lockout-exact-cooldown-cap, repro-antigravity-404-family-cooldown-hijack src/sse/services/auth.ts + 7993-noauth-proxy-routing, 8200-perplexity-web-401-cooldown, sse-auth-antigravity-credits src/server/authz/routeGuard.ts + authz/route-guard-vnc-session-local-only open-sse/utils/error.ts + error-sensitive-redaction open-sse/utils/publicCreds.ts + adobe-firefly src/shared/utils/circuitBreaker.ts + 8332-combo-vision-fallback None is a file this PR touches, and the gate reports the identical 11 on a worktree carrying none of these base-red fixes. Additions only (11 insertions, 0 deletions); the array stays sorted. This widens what the mutation run accounts for rather than relaxing anything. * chore(ci): re-measure accountFallback.ts file-size cap against the current base tip The entry frozen in this PR (1960) was the value at 75eb952; the base has since advanced to 1651617 and the file is 1966 there, so check:file-size would still have been red on the merge commit. Re-measured to 1966. Same inherited drift the note already documents: check:file-size does not run on the PR->release fast path, so growth accrues unmeasured between release rebaselines. The other four entries still match the current tip (tokenHealthCheck 841, chat 1866, auth 2486, combo frozen 3642 >= 3640). --------- Co-authored-by: backryun <busan011@ormbiz.co.kr>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #8331
Root cause
DEFAULT_BUFFER_TOKENS = 2000(open-sse/utils/usageTracking.ts) is a context-window safety margin, documented as headroom for CLI clients tracking remaining context.addBufferToUsage()added it directly into the client-visibleprompt_tokens/input_tokens/total_tokensfields, which are also the fields billing/metering tools treat as ground truth. Since the buffer is unset (usageTokenBuffer: null) by default, every non-estimated response was inflated by exactly 2000 — a real 69-token request was reported to the client as 2069, whilecall_logs.tokens_inand the raw upstream body (which read pre-buffer usage) correctly showed 69. Three call sites feed client-visible usage through this function: the non-streaming success path (chatCore.ts→clientUsageBuffer.ts), and two streaming SSE usage-frame builders instream.ts.Fix
addBufferToUsage()no longer mutates the metering fields. It still computes the safety margin (nothing about the buffer/settings/env-var mechanism was removed —usageTokenBuffer/USAGE_TOKEN_BUFFERstill work exactly as configured), but now attaches it under separate non-standard fields (context_budget_prompt_tokens/context_budget_input_tokens/context_budget_total_tokens) instead of the metering fields.filterUsageForFormat()does not allow-list those fields, so they're stripped before any response — streaming or non-streaming — reaches a client. This is the single choke point all three call sites share, so no other file needed changes.TDD evidence (Hard Rule #18)
New permanent regression file
tests/unit/8331-usage-buffer-inflation.test.ts, reusing the plan-file's proven repro shape ({prompt_tokens:69, completion_tokens:5, total_tokens:74}):AssertionError: SSE usage frame must not be buffer-inflated — 2069 !== 69(and same failure viaapplyClientUsageBuffer, the exact non-streaming call site chatCore.ts:4236 uses).applyClientUsageBuffer) and the streaming SSE usage-frame chain (addBufferToUsage+filterUsageForFormat, verbatim whatstream.ts:1018/:1897call), plus Claude-formatinput_tokens, confirming the buffer is still computed internally (not deleted), and confirmingestimated: trueusage stays untouched (unaffected by this fix, as before).Sibling tests aligned to the corrected contract (old assertions encoded the buggy inflated values — updated, never weakened):
tests/unit/usage-token-buffer.test.ts. Ran the full sibling sweep found by grepping forusageTracking|clientUsageBuffer|prompt_tokens|DEFAULT_BUFFER_TOKENSin the touched area (usage-token-buffer,chatcore-client-usage-buffer,cursor-streaming,chatcore-non-streaming-usage-stats,chatcore-streaming-usage-stats,stream-utils,executor-notion-web,stream-prompt-tokens-zero-upstream,usage-extractor) — 154/154 pass. (cursor.ts's usage is alwaysestimated: true, so it was already unaffected by the buffer and needed no test changes.)Gates run (from inside the worktree)
node --import tsx/esm --test tests/unit/8331-usage-buffer-inflation.test.ts— 5/5 passnode scripts/check/check-file-size.mjs— OKnode scripts/check/check-complexity.mjs/check-cognitive-complexity.mjs— both report the same pre-existing baseline drift (2167/956) on a disposable probe worktree pinned toorigin/release/v3.8.49— identical counts confirm base-red, not introduced by this diffnpm run typecheck:core— exit 0npx eslint --suppressions-location config/quality/eslint-suppressions.json <changed files>— exit 0node scripts/check/check-changelog-integrity.mjs— OK