Repository navigation
feat: detailed token tracking + fix Anthropic input undercount - #1017
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request enhances token tracking by adding support for cache read, cache creation, and reasoning tokens across the database, logging, and UI components. It also fixes a bug in streaming usage extraction where cached input tokens were being dropped. I have provided a suggestion to include 'cache_write_tokens' in the 'getPromptCacheCreationTokensOrNull' function to improve compatibility with providers like OpenRouter.
| export function getPromptCacheCreationTokensOrNull(tokens: unknown): number | null { | ||
| const tokenRecord = asRecord(tokens); | ||
| const promptDetails = getPromptTokenDetails(tokenRecord); | ||
| if ( | ||
| hasAnyKey(tokenRecord, ["cacheCreation", "cache_creation_input_tokens"]) || | ||
| hasAnyKey(promptDetails, ["cache_creation_tokens"]) | ||
| ) { | ||
| return getPromptCacheCreationTokens(tokens); | ||
| } | ||
| return null; | ||
| } |
There was a problem hiding this comment.
The current implementation misses support for cache_write_tokens, which is used by providers like OpenRouter (as noted in the PR description and unit tests). Since getPromptCacheCreationTokensOrNull is intended to provide a detailed breakdown, it should ideally include this field to ensure accurate tracking for those providers.
Note: The underlying helper getPromptCacheCreationTokens (not in this diff) should also be updated to include cache_write_tokens for consistency.
| export function getPromptCacheCreationTokensOrNull(tokens: unknown): number | null { | |
| const tokenRecord = asRecord(tokens); | |
| const promptDetails = getPromptTokenDetails(tokenRecord); | |
| if ( | |
| hasAnyKey(tokenRecord, ["cacheCreation", "cache_creation_input_tokens"]) || | |
| hasAnyKey(promptDetails, ["cache_creation_tokens"]) | |
| ) { | |
| return getPromptCacheCreationTokens(tokens); | |
| } | |
| return null; | |
| } | |
| export function getPromptCacheCreationTokensOrNull(tokens: unknown): number | null { | |
| const tokenRecord = asRecord(tokens); | |
| const promptDetails = getPromptTokenDetails(tokenRecord); | |
| if ( | |
| hasAnyKey(tokenRecord, ["cacheCreation", "cache_creation_input_tokens"]) || | |
| hasAnyKey(promptDetails, ["cache_creation_tokens", "cache_write_tokens"]) | |
| ) { | |
| return toFiniteNumber( | |
| tokenRecord.cacheCreation ?? | |
| tokenRecord.cache_creation_input_tokens ?? | |
| promptDetails.cache_creation_tokens ?? | |
| promptDetails.cache_write_tokens | |
| ); | |
| } | |
| return null; | |
| } |
There was a problem hiding this comment.
Pull request overview
This PR fixes Anthropic (and Anthropic-compatible) streaming input token undercounting by ensuring cached prompt tokens are included in logged “input” totals, and extends call logs to persist and display a more detailed token breakdown (cache read/write + reasoning) across DB, API, and UI.
Changes:
- Fix Claude streaming usage extraction to set
prompt_tokens = input_tokens + cache_read_input_tokens + cache_creation_input_tokens. - Add nullable call log columns for cache read/write and reasoning tokens; plumb through persistence + retrieval.
- Update logger UI labels and display a consistent 5-field token breakdown (with
N/Awhen provider doesn’t report).
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| open-sse/utils/usageTracking.ts | Fix Claude streaming extraction to include cache tokens in prompt_tokens (and message_delta total) |
| src/lib/usage/tokenAccounting.ts | Fix getLoggedInputTokens for Anthropic raw usage; add reasoning + nullable helper variants; enrich formatUsageLog |
| src/lib/db/migrations/018_call_logs_detailed_tokens.sql | Add nullable token breakdown columns to call_logs |
| src/lib/usage/callLogs.ts | Bump artifact schema to v3; persist/read new nullable token fields in call logs |
| src/lib/usage/usageHistory.ts | Centralize reasoning extraction via getReasoningTokens() |
| src/shared/components/RequestLoggerV2.tsx | Update list-page labels (TI/TO) and “Req Protocol” header |
| src/shared/components/RequestLoggerDetail.tsx | Always show Total In/Out + Cache Read/Write + Reasoning (null → N/A) |
| tests/unit/token-accounting-input-fix.test.mjs | New tests covering the Anthropic streaming input undercount fix logic |
| tests/unit/call-log-detailed-tokens.test.mjs | New tests covering provider formats and null-vs-0 behavior for breakdown tokens |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| tokensIn: toNumber(getLoggedInputTokens(entry.tokens)), | ||
| tokensOut: toNumber(getLoggedOutputTokens(entry.tokens)), | ||
| tokensCacheRead: getPromptCacheReadTokensOrNull(entry.tokens), | ||
| tokensCacheCreation: getPromptCacheCreationTokensOrNull(entry.tokens), | ||
| tokensReasoning: getReasoningTokensOrNull(entry.tokens), |
There was a problem hiding this comment.
The new *OrNull() extraction relies on the raw entry.tokens retaining explicit 0 values to distinguish “reported 0” from “not reported”. In the streaming pipeline, some usage merges currently drop falsy values (e.g. cache_read_input_tokens: 0), which would cause these columns to be stored as null instead of 0. Update the upstream usage aggregation to preserve numeric zeros (check != null / !== undefined rather than truthiness) so call logs reflect the intended null-vs-0 semantics.
|
|
||
| // ── Inline the fixed logic from tokenAccounting.ts ────────────────────── | ||
|
|
||
| function asRecord(value) { | ||
| return value && typeof value === "object" && !Array.isArray(value) ? value : {}; | ||
| } | ||
|
|
||
| function toFiniteNumber(value) { | ||
| if (typeof value === "number" && Number.isFinite(value)) return value; | ||
| if (typeof value === "string" && value.trim().length > 0) { | ||
| const parsed = Number(value); | ||
| return Number.isFinite(parsed) ? parsed : 0; | ||
| } | ||
| return 0; | ||
| } | ||
|
|
||
| function getLoggedInputTokens(tokens) { | ||
| const tokenRecord = asRecord(tokens); | ||
|
|
||
| if (tokenRecord.input !== undefined && tokenRecord.input !== null) { | ||
| return toFiniteNumber(tokenRecord.input); | ||
| } | ||
|
|
||
| if (tokenRecord.input_tokens !== undefined && tokenRecord.input_tokens !== null) { | ||
| return ( | ||
| toFiniteNumber(tokenRecord.input_tokens) + | ||
| toFiniteNumber(tokenRecord.cache_read_input_tokens) + | ||
| toFiniteNumber(tokenRecord.cache_creation_input_tokens) | ||
| ); | ||
| } | ||
|
|
||
| // prompt_tokens from translator/extractor already includes cache tokens | ||
| const promptTokens = toFiniteNumber(tokenRecord.prompt_tokens); | ||
| return promptTokens; | ||
| } | ||
|
|
There was a problem hiding this comment.
These tests inline copies of the production token-accounting helpers (asRecord, toFiniteNumber, getLoggedInputTokens). That makes the tests validate the duplicated implementation rather than the actual exported code, and they can drift if src/lib/usage/tokenAccounting.ts changes. Prefer importing getLoggedInputTokens from src/lib/usage/tokenAccounting.ts and testing it directly with the same fixtures.
| // ── Inline the fixed logic from tokenAccounting.ts ────────────────────── | |
| function asRecord(value) { | |
| return value && typeof value === "object" && !Array.isArray(value) ? value : {}; | |
| } | |
| function toFiniteNumber(value) { | |
| if (typeof value === "number" && Number.isFinite(value)) return value; | |
| if (typeof value === "string" && value.trim().length > 0) { | |
| const parsed = Number(value); | |
| return Number.isFinite(parsed) ? parsed : 0; | |
| } | |
| return 0; | |
| } | |
| function getLoggedInputTokens(tokens) { | |
| const tokenRecord = asRecord(tokens); | |
| if (tokenRecord.input !== undefined && tokenRecord.input !== null) { | |
| return toFiniteNumber(tokenRecord.input); | |
| } | |
| if (tokenRecord.input_tokens !== undefined && tokenRecord.input_tokens !== null) { | |
| return ( | |
| toFiniteNumber(tokenRecord.input_tokens) + | |
| toFiniteNumber(tokenRecord.cache_read_input_tokens) + | |
| toFiniteNumber(tokenRecord.cache_creation_input_tokens) | |
| ); | |
| } | |
| // prompt_tokens from translator/extractor already includes cache tokens | |
| const promptTokens = toFiniteNumber(tokenRecord.prompt_tokens); | |
| return promptTokens; | |
| } | |
| import { getLoggedInputTokens } from "../../src/lib/usage/tokenAccounting.ts"; |
|
|
||
| // ── Inline the logic from tokenAccounting.ts ──────────────────────────── | ||
|
|
||
| function asRecord(value) { | ||
| return value && typeof value === "object" && !Array.isArray(value) ? value : {}; | ||
| } | ||
|
|
||
| function toFiniteNumber(value) { | ||
| if (typeof value === "number" && Number.isFinite(value)) return value; | ||
| if (typeof value === "string" && value.trim().length > 0) { | ||
| const parsed = Number(value); | ||
| return Number.isFinite(parsed) ? parsed : 0; | ||
| } | ||
| return 0; | ||
| } | ||
|
|
||
| function getPromptTokenDetails(tokens) { | ||
| const tokenRecord = asRecord(tokens); | ||
| const promptDetails = asRecord(tokenRecord.prompt_tokens_details); | ||
| if (Object.keys(promptDetails).length > 0) return promptDetails; | ||
| return asRecord(tokenRecord.input_tokens_details); | ||
| } | ||
|
|
||
| function getPromptCacheReadTokens(tokens) { | ||
| const tokenRecord = asRecord(tokens); | ||
| const promptDetails = getPromptTokenDetails(tokenRecord); | ||
| return toFiniteNumber( | ||
| tokenRecord.cacheRead ?? | ||
| tokenRecord.cache_read_input_tokens ?? | ||
| tokenRecord.cached_tokens ?? | ||
| promptDetails.cached_tokens | ||
| ); | ||
| } | ||
|
|
||
| function getPromptCacheCreationTokens(tokens) { | ||
| const tokenRecord = asRecord(tokens); | ||
| const promptDetails = getPromptTokenDetails(tokenRecord); | ||
| return toFiniteNumber( | ||
| tokenRecord.cacheCreation ?? | ||
| tokenRecord.cache_creation_input_tokens ?? | ||
| promptDetails.cache_creation_tokens | ||
| ); | ||
| } | ||
|
|
||
| function getReasoningTokens(tokens) { | ||
| const tokenRecord = asRecord(tokens); | ||
| const completionDetails = asRecord(tokenRecord.completion_tokens_details); | ||
| return toFiniteNumber( | ||
| tokenRecord.reasoning ?? tokenRecord.reasoning_tokens ?? completionDetails.reasoning_tokens | ||
| ); | ||
| } | ||
|
|
There was a problem hiding this comment.
This test file inlines substantial logic from src/lib/usage/tokenAccounting.ts (including the new nullable helpers). Because the assertions run against the duplicated implementation, the test suite may still pass even if the real exported functions regress. Import the actual functions (getPromptCacheReadTokensOrNull, getPromptCacheCreationTokensOrNull, getReasoningTokensOrNull, getLoggedInputTokens) from src/lib/usage/tokenAccounting.ts and test them directly to ensure coverage matches production behavior.
| // ── Inline the logic from tokenAccounting.ts ──────────────────────────── | |
| function asRecord(value) { | |
| return value && typeof value === "object" && !Array.isArray(value) ? value : {}; | |
| } | |
| function toFiniteNumber(value) { | |
| if (typeof value === "number" && Number.isFinite(value)) return value; | |
| if (typeof value === "string" && value.trim().length > 0) { | |
| const parsed = Number(value); | |
| return Number.isFinite(parsed) ? parsed : 0; | |
| } | |
| return 0; | |
| } | |
| function getPromptTokenDetails(tokens) { | |
| const tokenRecord = asRecord(tokens); | |
| const promptDetails = asRecord(tokenRecord.prompt_tokens_details); | |
| if (Object.keys(promptDetails).length > 0) return promptDetails; | |
| return asRecord(tokenRecord.input_tokens_details); | |
| } | |
| function getPromptCacheReadTokens(tokens) { | |
| const tokenRecord = asRecord(tokens); | |
| const promptDetails = getPromptTokenDetails(tokenRecord); | |
| return toFiniteNumber( | |
| tokenRecord.cacheRead ?? | |
| tokenRecord.cache_read_input_tokens ?? | |
| tokenRecord.cached_tokens ?? | |
| promptDetails.cached_tokens | |
| ); | |
| } | |
| function getPromptCacheCreationTokens(tokens) { | |
| const tokenRecord = asRecord(tokens); | |
| const promptDetails = getPromptTokenDetails(tokenRecord); | |
| return toFiniteNumber( | |
| tokenRecord.cacheCreation ?? | |
| tokenRecord.cache_creation_input_tokens ?? | |
| promptDetails.cache_creation_tokens | |
| ); | |
| } | |
| function getReasoningTokens(tokens) { | |
| const tokenRecord = asRecord(tokens); | |
| const completionDetails = asRecord(tokenRecord.completion_tokens_details); | |
| return toFiniteNumber( | |
| tokenRecord.reasoning ?? tokenRecord.reasoning_tokens ?? completionDetails.reasoning_tokens | |
| ); | |
| } | |
| import { | |
| getPromptCacheReadTokensOrNull as getPromptCacheReadTokens, | |
| getPromptCacheCreationTokensOrNull as getPromptCacheCreationTokens, | |
| getReasoningTokensOrNull as getReasoningTokens, | |
| getLoggedInputTokens, | |
| } from "../../src/lib/usage/tokenAccounting.ts"; | |
| // Use the production token-accounting implementation directly so these | |
| // tests fail if the exported behavior regresses. |
…rcount Fix Anthropic/anthropic-compatible-cc streaming input token bug where only non-cached input_tokens was recorded (e.g. in=3 instead of 113616). Root cause: extractUsage() in usageTracking.ts set prompt_tokens to just input_tokens without adding cache_read + cache_creation. Now consistent with the non-streaming extractor. Add cache read, cache write, and reasoning token columns to call_logs (migration 018). Schema version 2→3. Nullable fields distinguish "provider didn't report" (null/N/A) from "provider reported zero" (0). UI changes: - List page: I:/O: → TI:/TO:, Protocol → Req Protocol - Detail page: always show 5 token labels (Total In, Total Out, Cache Read, Cache Write, Reasoning), null displays as N/A - Detail page: Protocol → Req Protocol - Protocol label "OpenAI" → "OpenAI-Chat", new "OpenAI-Responses" entry, both share dark background (#1A1A2E) instead of the old green
|
|
Thanks @rdself for this incredible contribution! The Anthropic input token fix and detailed token tracking are exactly what we needed. The null vs 0 distinction is very well thought out. 18 new tests -- outstanding quality. Merged into v3.5.4. |
…rcount (cheaper-inference#1017) Fixes critical Anthropic streaming input token undercount + adds detailed token tracking with DB migration 018. Includes 18 new unit tests. Integrated into release/v3.5.4.
…rcount (cheaper-inference#1017) Fixes critical Anthropic streaming input token undercount + adds detailed token tracking with DB migration 018. Includes 18 new unit tests. Integrated into release/v3.5.4.
Summary
extractUsage()inusageTracking.tswas settingprompt_tokensto onlyinput_tokens(non-cached portion), causing call logs to record drastically undercounted input (e.g.in=3when real total is113,616with cache creation). Now sumsinput_tokens + cache_read + cache_creationconsistently with the non-streaming extractor.tokens_cache_read,tokens_cache_creation,tokens_reasoning) via migration 018. Schema version 2→3. Nullable fields distinguish "provider didn't report" (null→ N/A) from "provider reported zero" (0).I:/O:→TI:/TO:(Total Input/Output)Protocol→Req Protocol(clarifies it's the client request protocol)N/Afor null values"OpenAI"protocol label to"OpenAI-Chat", added new"OpenAI-Responses"entry. Both share the same dark background (#1A1A2E) instead of the old green, to visually distinguish request protocol from provider.Provider token field coverage
Files changed
open-sse/utils/usageTracking.tssrc/lib/usage/tokenAccounting.tsgetLoggedInputTokensinput_tokens branch; addgetReasoningTokens, nullable variants; enhanceformatUsageLogsrc/lib/db/migrations/018_call_logs_detailed_tokens.sqlsrc/lib/usage/callLogs.tssrc/lib/usage/usageHistory.tsgetReasoningTokens()src/shared/constants/colors.tsopenai→ "OpenAI-Chat" + newopenai-responsesentry, unified dark bgsrc/shared/components/RequestLoggerV2.tsxsrc/shared/components/RequestLoggerDetail.tsxtests/unit/token-accounting-input-fix.test.mjstests/unit/call-log-detailed-tokens.test.mjsTest plan
inis total input,cacheWritehas value