Repository navigation
Reject invalid output token budgets #7379
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
diegosouzapw
merged 5 commits into
cheaper-inference:release/v3.8.49
from
JxnLexn:dev/context-overflow-output-budget
Jul 18, 2026
Merged
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
3cfb300
fix(context): reject invalid output token budgets
JxnLexn f5cf4cf
fix(context): enforce output budgets across request formats
JxnLexn b5bd75a
fix(context): enforce default Claude output budgets
JxnLexn 42dc61c
fix(context): include Responses API input in token budget
JxnLexn fbe528b
ci: rerun pull request checks
JxnLexn File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| export const OUTPUT_TOKEN_FIELDS = ["max_tokens", "max_completion_tokens"] as const; | ||
|
JxnLexn marked this conversation as resolved.
Resolved
|
||
|
|
||
| export type OutputTokenBudgetResult = | ||
| | { | ||
| ok: true; | ||
| body: Record<string, unknown>; | ||
| availableOutputTokens: number; | ||
| adjustedFields: string[]; | ||
| } | ||
| | { | ||
| ok: false; | ||
| estimatedInputTokens: number; | ||
| contextLimit: number; | ||
| }; | ||
|
|
||
| /** | ||
| * Enforce the target model's context budget immediately before translation. | ||
| * | ||
| * Compression and combo selection are best-effort: a request may still be too | ||
| * large for a concrete target, and some OpenAI-compatible gateways derive an | ||
| * internal max_tokens value by subtracting the prompt from the context window. | ||
| * Reject that target locally instead of allowing the derived value to become | ||
| * negative upstream. Positive client limits are capped to the remaining room; | ||
| * invalid numeric limits are removed. | ||
| */ | ||
| export function enforceOutputTokenBudget( | ||
| body: Record<string, unknown>, | ||
| estimatedInputTokens: number, | ||
| contextLimit: number | ||
| ): OutputTokenBudgetResult { | ||
| const normalizedInputTokens = Math.max(0, Math.ceil(estimatedInputTokens)); | ||
| const normalizedContextLimit = Math.max(1, Math.floor(contextLimit)); | ||
| const availableOutputTokens = normalizedContextLimit - normalizedInputTokens; | ||
|
|
||
| if (availableOutputTokens < 1) { | ||
| return { | ||
| ok: false, | ||
| estimatedInputTokens: normalizedInputTokens, | ||
| contextLimit: normalizedContextLimit, | ||
| }; | ||
| } | ||
|
JxnLexn marked this conversation as resolved.
Resolved
|
||
|
|
||
| let nextBody = body; | ||
| const adjustedFields: string[] = []; | ||
| for (const field of OUTPUT_TOKEN_FIELDS) { | ||
| const value = nextBody[field]; | ||
| if (typeof value !== "number") continue; | ||
|
|
||
| const normalized = Number.isFinite(value) && value > 0 ? Math.floor(value) : null; | ||
| if (normalized === null) { | ||
| if (nextBody === body) nextBody = { ...body }; | ||
| delete nextBody[field]; | ||
| adjustedFields.push(field); | ||
| continue; | ||
| } | ||
|
|
||
| const capped = Math.min(normalized, availableOutputTokens); | ||
| if (capped !== value) { | ||
| if (nextBody === body) nextBody = { ...body }; | ||
| nextBody[field] = capped; | ||
| adjustedFields.push(field); | ||
| } | ||
| } | ||
|
|
||
| return { ok: true, body: nextBody, availableOutputTokens, adjustedFields }; | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| import test from "node:test"; | ||
| import assert from "node:assert/strict"; | ||
|
|
||
| import { enforceOutputTokenBudget } from "../../open-sse/handlers/chatCore/outputTokenBudget.ts"; | ||
|
|
||
| test("rejects a prompt that cannot leave one output token", () => { | ||
| const result = enforceOutputTokenBudget({ max_tokens: 8192 }, 527_058, 128_000); | ||
|
|
||
| assert.deepEqual(result, { | ||
| ok: false, | ||
| estimatedInputTokens: 527_058, | ||
| contextLimit: 128_000, | ||
| }); | ||
| }); | ||
|
|
||
| test("caps a positive output budget to the target's remaining context", () => { | ||
| const input = { messages: [], max_tokens: 12_000 }; | ||
| const result = enforceOutputTokenBudget(input, 127_000, 128_000); | ||
|
|
||
| assert.equal(result.ok, true); | ||
| if (!result.ok) return; | ||
| assert.equal(result.body.max_tokens, 1_000); | ||
| assert.equal(input.max_tokens, 12_000, "must not mutate the shared combo request body"); | ||
| }); | ||
|
|
||
| test("removes non-positive numeric output limits before upstream dispatch", () => { | ||
| const result = enforceOutputTokenBudget( | ||
| { max_tokens: -398_464, max_completion_tokens: 0 }, | ||
| 1_000, | ||
| 128_000 | ||
| ); | ||
|
|
||
| assert.equal(result.ok, true); | ||
| if (!result.ok) return; | ||
| assert.equal("max_tokens" in result.body, false); | ||
| assert.equal("max_completion_tokens" in result.body, false); | ||
| }); |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.