Skip to content

Fix cost tracking accuracy and OpenRouter model pricing #3189

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
angiejones merged 9 commits into from
Jul 2, 2025
Merged

Fix cost tracking accuracy and OpenRouter model pricing #3189

angiejones merged 9 commits into from
Jul 2, 2025

Conversation

@jackjackbits
Copy link
Contributor

@jackjackbits jackjackbits commented Jul 1, 2025

Summary

  • Fix severe cost tracking overestimation for Anthropic models (was 4x too high)
  • Resolve "pricing not available" issue for OpenRouter models like anthropic/claude-sonnet-4
  • Fix frontend token counter reset when switching models

Key Changes

Cost Tracking Accuracy (Anthropic)

  • Properly implement Anthropic's cache pricing model:
    • Fresh tokens: 100% of regular price (1.0x)
    • Cache creation: 125% of regular price (1.25x)
    • Cache reads: 10% of regular price (0.1x)
  • Fix severe overestimation: improved from 400% error to ~5% variance

OpenRouter Model Pricing

  • Add proper model name parsing for models like anthropic/claude-sonnet-4
  • Extract provider/model components for accurate pricing API calls
  • Implement dual caching for both original and parsed model identifiers

Frontend Token Management

  • Fix token counters resetting to 0 when switching models
  • Properly restore token counts from session metadata

Code Quality

  • Remove debug logging from production
  • Reduce verbose pricing logs from INFO to DEBUG level
  • Add comprehensive test coverage
  • Fix compiler warnings

Validation Results

User reported excellent accuracy improvement:

  • Before: $0.93 actual vs $3.51 reported (4x overestimation)
  • After: $1.27 actual vs $1.21 reported (5% underestimation)

Test Plan

  • Rust compilation passes (cargo check)
  • TypeScript compilation passes (npm run typecheck)
  • Manual testing with real Anthropic API calls shows accurate pricing
  • OpenRouter models now display pricing correctly
  • Token counters persist when switching models

jack and others added 9 commits July 1, 2025 00:47
Fix cost tracking accuracy by excluding cache tokens
7f77b72 
- Fixed token counting to only use raw input tokens, not cache-related tokens
  * Anthropic API returns cache_creation_input_tokens and cache_read_input_tokens
  * These were being added to the base input_tokens, causing inflated counts
  * Now only using the raw input_tokens field for billing purposes

- Added debug logging for cache token breakdown when present
- Updated tests to reflect the corrected token counting behavior

The pricing is fetched dynamically from OpenRouter, so no hardcoded pricing
changes were needed. This should resolve the discrepancy between actual
Anthropic billing and Goose's cost tracking display.
Fix cost tracking accuracy and OpenRouter model pricing
1ddca4a 
- Fix Anthropic cache token pricing using correct rates (1.25x for cache creation, 0.1x for reads)
- Resolve OpenRouter model pricing lookup by parsing provider/model from names like 'anthropic/claude-sonnet-4'
- Fix token counter reset issue when switching models in ChatView
- Add comprehensive test coverage for pricing calculations
- Implement dual caching for OpenRouter models to optimize lookups
Merge remote-tracking branch 'origin/main' into fix-cost-tracking-acc…
08f779b 
…uracy
Remove debug logging from Anthropic provider
5af6f76 
- Remove tracing::info statement that was logging token breakdown details
- Clean up development debugging that was appearing in production logs
Clean up debug logging and temporary files
fe30ccd 
- Remove additional tracing::info statement from Anthropic provider
- Delete temporary test_pricing_debug.rs file
- Final cleanup of development artifacts
Reduce verbose pricing logs to debug level
bf2cd6f 
- Move frequent operational logs from INFO to DEBUG level
- Keep only essential API fetch notifications at INFO level
- Significantly reduces log noise in production
Fix unused variable warning
612bc2e 
- Prefix unused total_actual_tokens variable with underscore
- Resolves compiler warning in Anthropic provider
@angiejones
Fix Rust formatting issues
393afe9 
- Remove trailing whitespace
- Fix line breaks and indentation
- Organize imports properly
- Ensure consistent formatting across all modified files
@angiejones
Fix pricing integration tests
3601476 
- Update test to use actual model names available in OpenRouter API
- Replace non-existent 'claude-3-5-sonnet-20241022' with 'claude-3.5-sonnet'
- Replace non-existent 'claude-sonnet-4-latest' with 'claude-sonnet-4'
- Replace 'gemini-1.5-pro' with 'gemini-flash-1.5' (available model)
- Fix performance assertion to be more realistic for microsecond-level operations
- All pricing integration tests now pass successfully
@angiejones angiejones requested a review from baxen July 1, 2025 15:50
wendytang
wendytang approved these changes Jul 2, 2025
View reviewed changes
Copy link
Contributor

@wendytang wendytang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@angiejones angiejones merged commit 620474b into block:main Jul 2, 2025
6 checks passed
jsibbison-square added a commit that referenced this pull request Jul 2, 2025
@jsibbison-square
Merge branch 'main' into jsibbison-20250702-max-turns
44742af 
* main:
  feat: Structured output for recipes (#3188)
  Fix cost tracking accuracy and OpenRouter model pricing (#3189)
  docs: update cli install instructions for windows (#3205)
  Docs: Cost tracking on the desktop app (#3204)
  feat: Adding streamable-http transport support for backend, desktop and cli (#2942)
  fix: use the correct `contains` syntax on create-recipe-pr.yml (#3193)
wpfleger96 added a commit to wpfleger96/goose that referenced this pull request Jul 2, 2025
@wpfleger96
Merge branch 'main' into wpfleger/run-sub-recipes-from-github
c2f81a2 
* main:
  fix: Pass Google AI API key in HTTP header, not query param (block#3192)
  docs: add linter to CONTRIBUTING.md (block#3168)
  feat: Structured output for recipes (block#3188)
  Fix cost tracking accuracy and OpenRouter model pricing (block#3189)
  docs: update cli install instructions for windows (block#3205)
  Docs: Cost tracking on the desktop app (block#3204)
  feat: Adding streamable-http transport support for backend, desktop and cli (block#2942)
  fix: use the correct `contains` syntax on create-recipe-pr.yml (block#3193)
AaronGoldsmith added a commit to AaronGoldsmith/goose that referenced this pull request Jul 2, 2025
@AaronGoldsmith
Merge branch 'main' into ag/feat-desktop-recipe-params
04bf72c 
* main: (37 commits)
  fix: fix desktop recipe url generation (block#3209)
  feat: improve UX for saving recipes (block#3214)
  fix: Pass Google AI API key in HTTP header, not query param (block#3192)
  docs: add linter to CONTRIBUTING.md (block#3168)
  feat: Structured output for recipes (block#3188)
  Fix cost tracking accuracy and OpenRouter model pricing (block#3189)
  docs: update cli install instructions for windows (block#3205)
  Docs: Cost tracking on the desktop app (block#3204)
  feat: Adding streamable-http transport support for backend, desktop and cli (block#2942)
  fix: use the correct `contains` syntax on create-recipe-pr.yml (block#3193)
  Temporarily Remove GH Copilot Provider  (block#3199)
  docs: fix tab navigation (block#3201)
  feat: use tiktoken-rs instead of tokenizers, single global tokenizer (block#3115)
  add playwright-mcp server to extensions list (block#3010)
  Add `/extension` path for extension installation (block#3011)
  feat(desktop): Prioritize suffix when truncating path in header (block#3110)
  chore(release): release version 1.0.31 (block#3185)
  feat: additional sub recipes via command line (block#3163)
  Add Internal Recipes To Recipes Cookbook (block#3179)
  pipe the argument to storage (block#3184)
  ...
baxen added a commit to Developerayo/goose that referenced this pull request Jul 2, 2025
@baxen
Merge branch 'main' into feat/theme/persist
b2e8ddf 
* main: (150 commits)
  Defend against invalid sessions (block#3229)
  Clean up session file optionality for --no-session (block#3230)
  Feat: Support Recipe Parameters in Goose desktop app (block#3155)
  docs: update recipe example (block#3222)
  Add native OAuth 2.0 authentication support to MCP client (block#3213)
  build: Check in Cargo.lock changes (block#3220)
  fix: fix desktop recipe url generation (block#3209)
  feat: improve UX for saving recipes (block#3214)
  fix: Pass Google AI API key in HTTP header, not query param (block#3192)
  docs: add linter to CONTRIBUTING.md (block#3168)
  feat: Structured output for recipes (block#3188)
  Fix cost tracking accuracy and OpenRouter model pricing (block#3189)
  docs: update cli install instructions for windows (block#3205)
  Docs: Cost tracking on the desktop app (block#3204)
  feat: Adding streamable-http transport support for backend, desktop and cli (block#2942)
  fix: use the correct `contains` syntax on create-recipe-pr.yml (block#3193)
  Temporarily Remove GH Copilot Provider  (block#3199)
  docs: fix tab navigation (block#3201)
  feat: use tiktoken-rs instead of tokenizers, single global tokenizer (block#3115)
  add playwright-mcp server to extensions list (block#3010)
  ...
@chaitanyarahalkar chaitanyarahalkar mentioned this pull request Jul 5, 2025
Open
atarantino pushed a commit to atarantino/goose that referenced this pull request Jul 14, 2025
@jackjackbits @angiejones @atarantino
Fix cost tracking accuracy and OpenRouter model pricing (block#3189)
998d356 
Co-authored-by: jack <>
Co-authored-by: angiejones <[email protected]>
Signed-off-by: Adam Tarantino <[email protected]>
s-soroosh pushed a commit to s-soroosh/goose that referenced this pull request Jul 18, 2025
@jackjackbits @angiejones @s-soroosh
Fix cost tracking accuracy and OpenRouter model pricing (block#3189)
cffdb01 
Co-authored-by: jack <>
Co-authored-by: angiejones <[email protected]>
Signed-off-by: Soroosh <[email protected]>
kwsantiago pushed a commit to kwsantiago/goose that referenced this pull request Jul 19, 2025
@jackjackbits @angiejones @kwsantiago
Fix cost tracking accuracy and OpenRouter model pricing (block#3189)
3bff1b8 
Co-authored-by: jack <>
Co-authored-by: angiejones <[email protected]>
Signed-off-by: Kyle Santiago <[email protected]>
cbruyndoncx pushed a commit to cbruyndoncx/goose that referenced this pull request Jul 20, 2025
@jackjackbits @angiejones
Fix cost tracking accuracy and OpenRouter model pricing (block#3189)
250ab10 
Co-authored-by: jack <>
Co-authored-by: angiejones <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wendytang wendytang wendytang approved these changes

@baxen baxen Awaiting requested review from baxen

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jackjackbits @wendytang @angiejones