feat(billing): use inputPrice fallback

[steebchen]

Summary

• Bill input tokens for cached responses using inputPrice as fallback when cached price is unavailable
• Compute and record costs for both cached responses and upstream cached responses
• Aligns billing for cached paths with non-cached paths to ensure accurate cost accounting

Changes

Backend

• apps/gateway/src/chat/chat.ts

  • When serving cached responses, calculate costs via calculateCosts using promptTokens, completionTokens, cachedTokens, and reasoningTokens, then log input/output costs, total cost, estimatedCost, and discount accordingly.
  • When serving upstream cached responses, calculate cachedCosts from cached usage data (prompt_tokens, completion_tokens, cached_tokens, reasoning_tokens) and log the corresponding cost fields.

• apps/gateway/src/lib/costs.ts

  • Change cachedInputPrice default from using || 0 to ?? inputPrice so that undefined cachedInputPrice falls back to inputPrice.

Why this helps

• Prevents under-billing or zero-cost logs for cached inputs when a cached input price is missing
• Maintains consistent cost reporting across cached and non-cached flows
• Keeps cost calculations in sync with how costs are computed for non-cached responses

Examples of impact

• Logged costs for cached responses now reflect actual input/output/cached input costs instead of zeros when a cached price is missing
• The system will fallback to the standard input price for cached inputs, ensuring predictable billing

Testing

• Validate that cached response logs include inputCost, outputCost, cachedInputCost, requestCost, cost, and discount fields populated
• Validate upstream cached response logs similarly
• Confirm that when cachedInputPrice is undefined, costs reflect inputPrice as expected
• Ensure non-cached response costs remain unchanged

Notes

• No API surface changes; this is an internal billing/logging improvement to handle missing cached input prices more gracefully.

🌿 Generated by Terry

---

ℹ️ Tag @terragon-labs to ask questions and address PR feedback

📎 Task: https://www.terragonlabs.com/task/30d762a7-b081-414d-aeaa-f24ca887d05a

Summary by CodeRabbit

• Improvements
  • Enhanced cost tracking for cached responses with accurate calculation and logging of input costs, output costs, and cached input costs across different cache scenarios.
  • Improved fallback logic for cached input pricing to ensure more accurate cost calculations.

[coderabbitai]

Walkthrough

Added cost calculation logging for cached chat responses across three code paths (streaming from cache, non-streaming cache hits, and newly cached streaming responses) by computing costs via calculateCosts and populating cost-related log fields. Also modified cachedInputPrice fallback logic to use inputPrice instead of zero when undefined.

Changes

Cohort / File(s)	Summary
Cost calculation and logging for cached responses apps/gateway/src/chat/chat.ts	Added cost calculations for three cached response scenarios: (1) streaming from cache, (2) non-streaming cache hit, and (3) newly cached streaming response. Each path now computes costsObject via calculateCosts and populates detailed cost fields (inputCost, outputCost, cachedInputCost, requestCost, cost, estimatedCost, discount) in log entries, replacing hard-coded zeros/nulls. Also adjusts timeToFirstToken and related fields for cached/no-processing scenarios.
Cached input price fallback logic apps/gateway/src/lib/costs.ts	Changed cachedInputPrice default fallback from providerInfo.cachedInputPrice || 0 to providerInfo.cachedInputPrice ?? inputPrice, so undefined/null cachedInputPrice now defaults to inputPrice instead of zero.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

• Verify calculateCosts is correctly invoked across all three cached-response code paths in chat.ts
• Confirm cost field assignments align with actual token counts and pricing rules
• Validate that the cachedInputPrice fallback change in costs.ts does not unintentionally affect pricing calculations where zero was previously the intended default
• Check for consistency of field population (?? 0 fallbacks) across the three logging paths

Possibly related PRs

• fix(cost-estimation): fix zero output tokens #913: Modifies calculateCosts behavior and signature to handle toolResults and zero-output scenarios, which directly affects how costs are computed in this PR's logging additions.
• feat(gateway): add e2e test for cached input cost validation #904: Modifies the same cost/caching paths in apps/gateway/src/chat/chat.ts and apps/gateway/src/lib/costs.ts to surface cached-input token/cost data.
• feat(costs): include reasoning tokens in cost calculations #1076: Extends calculateCosts and chat.ts to include reasoningTokens in cost computations, directly impacting the cost calculation logic introduced in this PR.

Pre-merge checks and finishing touches

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'feat(billing): use inputPrice fallback' directly corresponds to the main change: modifying cachedInputPrice logic to use inputPrice as fallback instead of 0, which is the core billing fix in costs.ts.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch terragon/fix-billing-for-missing-cached-input-price-l6golb

---

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 2469dc1 and 86d5f86.

📒 Files selected for processing (2)

• apps/gateway/src/chat/chat.ts (4 hunks)
• apps/gateway/src/lib/costs.ts (1 hunks)

🧰 Additional context used

📓 Path-based instructions (4)

**/*.{ts,tsx,js,jsx}

📄 CodeRabbit inference engine (AGENTS.md)

Always use top-level import; never use require() or dynamic import()

Files:

• apps/gateway/src/lib/costs.ts
• apps/gateway/src/chat/chat.ts

apps/{gateway,api}/**/*.ts

📄 CodeRabbit inference engine (AGENTS.md)

apps/{gateway,api}/**/*.ts: Use Hono for HTTP routing in Gateway and API services
Use Zod schemas for request/response validation in server routes

Files:

• apps/gateway/src/lib/costs.ts
• apps/gateway/src/chat/chat.ts

**/*.{ts,tsx}

📄 CodeRabbit inference engine (CLAUDE.md)

**/*.{ts,tsx}: Never use any or as any in this TypeScript project unless absolutely necessary
Always use top-level import; do not use require or dynamic import()

Files:

• apps/gateway/src/lib/costs.ts
• apps/gateway/src/chat/chat.ts

{apps/{api,gateway}/**/*.ts,packages/db/**/*.ts}

📄 CodeRabbit inference engine (CLAUDE.md)

For read operations, use db().query.<table>.findMany() or db().query.<table>.findFirst()

Files:

• apps/gateway/src/lib/costs.ts
• apps/gateway/src/chat/chat.ts

🧬 Code graph analysis (1)

apps/gateway/src/chat/chat.ts (1)

apps/gateway/src/lib/costs.ts (1)

• calculateCosts (25-207)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (9)

• GitHub Check: test / run
• GitHub Check: build / run
• GitHub Check: lint / run
• GitHub Check: e2e-shards (3)
• GitHub Check: e2e-shards (5)
• GitHub Check: e2e-shards (4)
• GitHub Check: autofix
• GitHub Check: e2e-shards (2)
• GitHub Check: e2e-shards (1)

🔇 Additional comments (2)

apps/gateway/src/lib/costs.ts (1)

173-173: LGTM! Proper fallback for missing cached input price.

The change from || to ?? correctly addresses the billing issue. When cachedInputPrice is undefined or null, it now falls back to inputPrice instead of 0, preventing under-billing. The nullish coalescing operator also preserves explicit 0 values, which is the correct behavior.

apps/gateway/src/chat/chat.ts (1)

2914-3001: LGTM! Proper cost calculation and logging for streaming responses.

The cost calculation correctly includes:

• Prompt, completion, cached, and reasoning tokens
• Full output object for estimation when needed
• Tool results for accurate completion token estimation

The costs are logged directly without unnecessary null coalescing, allowing the database to store null when costs cannot be calculated. This is the expected behavior that should be consistent across all code paths.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}