fix(gateway): keep max_tokens above thinking budget

[steebchen]

Problem

Production support chat was failing with:

`max_tokens` must be greater than `thinking.budget_tokens`.

The support-chat route (apps/api/src/routes/public-chat-support.ts) calls the gateway with model: "auto" and maxOutputTokens: 1024. When auto-routing lands on a reasoning-capable Anthropic model it sets reasoning_effort = "low", which maps to a thinking budget of exactly 1024 — Anthropic's minimum. The direct anthropic / vertex-anthropic branch of prepareRequestBody then forwarded the caller's max_tokens verbatim, so the upstream request had max_tokens == thinking.budget_tokens and Anthropic 400s.

The aws-bedrock branch already guarded against this; the direct Anthropic branch did not, despite its comment claiming to ("Set max_tokens, ensuring it's higher than thinking budget"). It only enforced the floor on the fallback value used when the caller omitted max_tokens.

Approach

Mirror the bedrock logic: when non-adaptive thinking is enabled, raise max_tokens to thinking.budget_tokens + 1000 if the caller-supplied value doesn't clear it.

Raising max_tokens is the only possible fix — the budget can't be lowered to fit, since 1024 is Anthropic's floor for budget_tokens, so any max_tokens <= 1024 is unsatisfiable without bumping it.

Adaptive models (Opus 4.7+) are untouched: they use thinking: { type: "adaptive" } with no budget_tokens, so the constraint doesn't apply.

Verification

• New specs in prepare-request-body.spec.ts cover both the raise (1024 → 2024 at reasoning_effort: "low") and the no-op case (an already-sufficient 8000 stays 8000).
• packages/actions unit suite: 581 passed.
• pnpm format and full pnpm build clean.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes
  • Improved Anthropic reasoning requests by ensuring the response token limit is always higher than the configured thinking budget.
  • Preserved caller-provided limits when they already meet the required threshold.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[coderabbitai]

Walkthrough

The Anthropic request builder now raises insufficient max_tokens values above the thinking budget. Tests cover adjusted values, sufficient values, and the resulting request behavior.

Changes

Anthropic thinking budget handling

Layer / File(s)	Summary
Enforce and test response token limits packages/actions/src/prepare-request-body.ts, packages/actions/src/prepare-request-body.spec.ts	Anthropic thinking requests now use at least thinkingBudget + 1000 for max_tokens. Tests verify that equal values increase from 1024 to 2024 and that an existing value of 8000 remains unchanged.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• theopenco/llmgateway#3075: Both changes update Anthropic thinking-budget handling and verify that max_tokens exceeds the reasoning budget.
• theopenco/llmgateway#3108: Both changes update Anthropic thinking handling in prepareRequestBody, but this PR changes max_tokens while the other changes temperature and top_p.

Suggested reviewers: smakosh, ratchaw

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: keeping max_tokens above the Anthropic thinking budget.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix-chat-support-thinking-budget

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

packages/actions/src/prepare-request-body.spec.ts (1)

3839-3845: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Remove the new as any assertions.

Use toMatchObject to assert thinking and max_tokens without bypassing TypeScript.

• packages/actions/src/prepare-request-body.spec.ts#L3839-L3845: Remove as any and assert both fields with toMatchObject.
• packages/actions/src/prepare-request-body.spec.ts#L3867-L3869: Remove as any and assert max_tokens with toMatchObject.

Proposed fix

-		)) as any;
-
-		expect(requestBody.thinking).toEqual({
-			type: "enabled",
-			budget_tokens: 1024,
-		});
-		expect(requestBody.max_tokens).toBe(2024);
+		));
+
+		expect(requestBody).toMatchObject({
+			thinking: { type: "enabled", budget_tokens: 1024 },
+			max_tokens: 2024,
+		});
...
-		)) as any;
-
-		expect(requestBody.max_tokens).toBe(8000);
+		));
+
+		expect(requestBody).toMatchObject({ max_tokens: 8000 });

As per coding guidelines, “do not use any or as any unless absolutely necessary.”

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/actions/src/prepare-request-body.spec.ts` around lines 3839 - 3845,
Remove the `as any` assertions in
`packages/actions/src/prepare-request-body.spec.ts` at lines 3839-3845 and
3867-3869. Update the first expectation to use `toMatchObject` for both
`thinking` and `max_tokens`, and the second to use `toMatchObject` for
`max_tokens`, preserving the asserted values without bypassing TypeScript.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@packages/actions/src/prepare-request-body.spec.ts`:
- Around line 3839-3845: Remove the `as any` assertions in
`packages/actions/src/prepare-request-body.spec.ts` at lines 3839-3845 and
3867-3869. Update the first expectation to use `toMatchObject` for both
`thinking` and `max_tokens`, and the second to use `toMatchObject` for
`max_tokens`, preserving the asserted values without bypassing TypeScript.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 81fda677-7ef5-4ab2-87db-20d6b547d868

📥 Commits

Reviewing files that changed from the base of the PR and between 2c91a02 and d6016ec.

📒 Files selected for processing (2)

• packages/actions/src/prepare-request-body.spec.ts
• packages/actions/src/prepare-request-body.ts