test: add reasoning token and adjustGoogleCandidateTokens tests

[steebchen]

Summary

• Added tests to extract-token-usage.spec.ts covering reasoning token handling and adjustGoogleCandidateTokens
• Confirms that Anthropic's totalTokens is correctly calculated as promptTokens + completionTokens — Anthropic's output_tokens already includes reasoning_output_tokens, so no separate addition is needed
• Tests cover: reasoning present, absent, zero, combined with cache tokens
• Also tests the adjustGoogleCandidateTokens helper for deduplicating thoughts from candidates

Context

Investigation confirmed that Anthropic's API already includes reasoning tokens within output_tokens, so the existing totalTokens calculation is correct. These tests document and guard that behavior.

Test plan

• All 17 tests in extract-token-usage.spec.ts pass
• All 131 gateway chat/tools tests pass
• Tests use realistic Anthropic API response data (e.g. 51 input + 136 output = 187 total, with 31 reasoning included in the 136)

🤖 Generated with Claude Code

Summary by CodeRabbit

• Tests
  • Added comprehensive unit tests for token usage extraction across providers, covering complete usage data, missing fields, zero-values, and cache interactions.
  • Added tests for token-adjustment logic to validate candidate token corrections when reasoning/thought tokens are present, absent, or null.

[coderabbitai]

Walkthrough

Adds unit tests for token-usage logic and exports a new helper adjustGoogleCandidateTokens from the token extraction module; updates manifest and package metadata. Tests cover Anthropic/OpenAI token fields, Google candidate token adjustments, and multiple edge cases (missing/zero token fields).

Changes

Cohort / File(s)	Summary
Token Usage Tests apps/gateway/src/chat/tools/extract-token-usage.spec.ts	Adds ~130 lines of unit tests covering extractTokenUsage scenarios (Anthropic/OpenAI variations, missing/zero fields) and new adjustGoogleCandidateTokens behavior tests.
Token Usage Implementation apps/gateway/src/chat/tools/extract-token-usage.js	Exports adjustGoogleCandidateTokens (signature inferred: (promptTokens, thoughtsTokenCount, candidatesTokenCount, totalTokens)) and likely small logic related to Google candidate token adjustments.
Manifest & Package manifest_file, package.json	Updates to project manifest and package metadata (lines changed noted in summary).

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• feat(token-usage): enhance token estimation for images #1202: Modifies token-usage module; related to token-count behavior adjustments (mentions imageByteSize changes).
• fix(gateway): fix Google thinking token counting #1631: Directly relates to exposing/using adjustGoogleCandidateTokens and Google token-count handling.
• fix: Correct total token calculation in extractTokenUsage #867: Touches extractTokenUsage total token computation logic that these tests validate.

🚥 Pre-merge checks | ✅ 3 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Merge Conflict Detection	⚠️ Warning	❌ Merge conflicts detected (2 files): ⚔️ apps/gateway/src/anthropic/anthropic.ts (content) ⚔️ apps/gateway/src/chat/tools/extract-token-usage.spec.ts (content) These conflicts must be resolved before merging into main.	Resolve conflicts locally and push changes to this branch.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately reflects the main changes: adding test coverage for reasoning tokens and the adjustGoogleCandidateTokens function.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch fix/anthropic-reasoning-token-total

⚔️ Resolve merge conflicts (beta)

• Auto-commit resolved conflicts to branch fix/anthropic-reasoning-token-total
• Create stacked PR with resolved conflicts
• Post resolved changes as copyable diffs in a comment

---

No actionable comments were generated in the recent review. 🎉

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Copilot]

Pull request overview

Fixes undercounted totalTokens for the Anthropic provider by ensuring reasoning_output_tokens are included in the total token calculation, aligning billing/analytics/rate-limit inputs with Anthropic’s “extended thinking” usage reporting.

Changes:

• Update Anthropic extractTokenUsage to compute totalTokens as promptTokens + completionTokens + reasoningTokens.
• Add Vitest coverage for Anthropic token extraction (with/without reasoning tokens, cache tokens, missing usage, zero reasoning tokens).
• Add unit tests for adjustGoogleCandidateTokens and basic OpenAI usage parsing.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
apps/gateway/src/chat/tools/extract-token-usage.ts	Fixes Anthropic totalTokens calculation to include reasoning_output_tokens.
apps/gateway/src/chat/tools/extract-token-usage.spec.ts	Adds regression tests covering Anthropic reasoning tokens and related token parsing helpers.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.