fix(anthropic): prompt-cache breakpoints + accounting parity for the direct-Anthropic path - #1141
Conversation
…direct-Anthropic path
The native Vertex+Claude path has had full prompt caching since 9.79.1;
the direct anthropic provider was left behind in three ways:
1. No rolling history breakpoints: only the system prompt (MessageBuilder)
and last tool (GenerationHandler) carried cache_control, so the growing
conversation history fell after the last marker and was re-billed at
full input price every turn. doGenerate and executeStream now apply
applyAnthropicHistoryCacheBreakpoints, counting pre-existing markers
via countAnthropicCacheMarkers so a request can never exceed
Anthropic's four-marker cap.
2. generate()-path cache accounting silently dropped: the native V3 model
reports cache tokens only through ai@6's normalized usage shape
(cachedInputTokens / inputTokenDetails.{cacheReadTokens,cacheWriteTokens}),
which extractCacheReadTokens/extractCacheCreationTokens did not
recognize. Both extractors (and RawUsageObject) now do; legacy field
names keep precedence. Also unblocks Bedrock on the same path.
3. stream()-path cache accounting never attempted: executeStream now
accumulates cache_read_input_tokens/cache_creation_input_tokens off
message_start events and threads them through resolveUsage into the
OTel span and cost calculation.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
✅ Single Commit Policy - COMPLIANTStatus: Policy requirements met • 1 commit • Valid format • Ready for merge 📊 View validation details📝 Commit Details
✅ Validation Results
🤖 Automated validation by NeuroLink Single Commit Enforcement |
📝 WalkthroughWalkthroughAnthropic requests now apply rolling history cache breakpoints in non-streaming and multi-step streaming flows. Cache read/write tokens are extracted, included in cost and usage totals, and represented in shared usage types with expanded test coverage. ChangesAnthropic cache integration
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 2 | ❌ 3❌ Failed checks (3 warnings)
✅ Passed checks (2 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
🤖 AI Review & Build Compliance ✅Status: AI analysis complete • Build rules validated • Ready for review 📊 View detailed analysis results🛡️ Analysis Complete
📋 Ready for Merge When
🤖 AI analysis complete - check individual code comments for specific feedback |
There was a problem hiding this comment.
Pull request overview
This PR brings the direct Anthropic provider path up to parity with the existing Vertex+Claude prompt-caching behavior by adding rolling history cache breakpoints (under Anthropic’s 4-marker cap) and ensuring cache read/write tokens are correctly extracted and propagated through both generate and stream usage/cost accounting.
Changes:
- Added direct-Anthropic rolling history cache breakpoints and marker counting/budgeting utilities.
- Extended cache token extraction to recognize AI SDK v6 normalized usage shapes (cachedInputTokens + inputTokenDetails cacheRead/cacheWrite).
- Updated Anthropic streaming usage aggregation to include cache read/write tokens; added regression tests covering direct path behavior and extraction shapes.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| test/continuous-test-suite-cache-breakpoints.ts | Adds coverage for direct-Anthropic marker budgeting/idempotence and ai@6 cache token extraction. |
| src/lib/utils/tokenUtils.ts | Extends cache read/write extractors to recognize ai@6 normalized usage shapes. |
| src/lib/utils/anthropicCacheBreakpoints.ts | Introduces marker counting and history-only breakpoint application for direct Anthropic requests, respecting the 4-marker cap. |
| src/lib/types/common.ts | Adds ai@6 cache fields to RawUsageObject and introduces a DeferredUsage type supporting optional cache fields. |
| src/lib/providers/openaiChatCompletionsClient.ts | Updates deferred analytics typing to use DeferredUsage. |
| src/lib/providers/anthropic.ts | Applies direct-path history breakpoints in generate/stream paths and accumulates cache reads/writes for streaming usage/cost/OTel. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| const thinking = options.providerOptions?.anthropic?.thinking as | ||
| | { type: "enabled"; budget_tokens: number } | ||
| | undefined; | ||
|
|
||
| // Prompt-cache parity with the native Vertex+Claude path: upstream |
| const cacheMarkersUsed = countAnthropicCacheMarkers({ | ||
| system, | ||
| tools, | ||
| messages: messages as unknown as VertexAnthropicMessage[], | ||
| }); | ||
| const cachedMessages = applyAnthropicHistoryCacheBreakpoints( | ||
| messages as unknown as VertexAnthropicMessage[], | ||
| ANTHROPIC_MAX_CACHE_BREAKPOINTS - cacheMarkersUsed, | ||
| ) as unknown as Anthropic.Messages.MessageParam[]; |
| const cacheMarkersUsed = countAnthropicCacheMarkers({ | ||
| system: payload.system, | ||
| tools: anthropicTools, | ||
| messages: conversation as unknown as VertexAnthropicMessage[], | ||
| }); | ||
| const cachedConversation = applyAnthropicHistoryCacheBreakpoints( | ||
| conversation as unknown as VertexAnthropicMessage[], | ||
| ANTHROPIC_MAX_CACHE_BREAKPOINTS - cacheMarkersUsed, | ||
| ) as unknown as Anthropic.Messages.MessageParam[]; |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
test/continuous-test-suite-cache-breakpoints.ts (1)
495-534: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winTests only validate extraction in isolation — doesn't cover the
extractTokenUsage()cost-calculation flow.These assertions correctly validate
extractCacheReadTokens/extractCacheCreationTokensin isolation, but none of them combine an inclusiveinputTokensvalue withcachedInputTokens/inputTokenDetailsin a single usage object run throughextractTokenUsage()— the path where the double-counting concern flagged intokenUtils.ts(extractCacheCreationTokens/extractCacheReadTokens, lines 100-156) would actually surface. Once that concern is resolved, consider adding an end-to-endextractTokenUsage()case with both fields present to lock in the fix.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@test/continuous-test-suite-cache-breakpoints.ts` around lines 495 - 534, The cache extraction tests do not cover the full cost-calculation path through extractTokenUsage(). Extend testAi6CacheUsageExtraction with an end-to-end usage object containing inclusive inputTokens plus cachedInputTokens or inputTokenDetails cache fields, and assert the resulting token usage prevents double-counting while preserving cache read/creation values. Use extractTokenUsage() rather than testing only extractCacheReadTokens() and extractCacheCreationTokens().
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/lib/utils/tokenUtils.ts`:
- Around line 94-98: The token extraction and pricing logic must treat ai@6
cache token fields as subsets of total input, not additive tokens. Update the
relevant functions in tokenUtils.ts, including the cache extraction and
calculateCost paths, to subtract cachedInputTokens,
inputTokenDetails.cacheReadTokens, and inputTokenDetails.cacheWriteTokens from
inputTokens before pricing, while preserving existing provider-specific
handling.
---
Nitpick comments:
In `@test/continuous-test-suite-cache-breakpoints.ts`:
- Around line 495-534: The cache extraction tests do not cover the full
cost-calculation path through extractTokenUsage(). Extend
testAi6CacheUsageExtraction with an end-to-end usage object containing inclusive
inputTokens plus cachedInputTokens or inputTokenDetails cache fields, and assert
the resulting token usage prevents double-counting while preserving cache
read/creation values. Use extractTokenUsage() rather than testing only
extractCacheReadTokens() and extractCacheCreationTokens().
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 44e24e76-a94b-4b2b-ae8f-f6c66d93df63
📒 Files selected for processing (6)
src/lib/providers/anthropic.tssrc/lib/providers/openaiChatCompletionsClient.tssrc/lib/types/common.tssrc/lib/utils/anthropicCacheBreakpoints.tssrc/lib/utils/tokenUtils.tstest/continuous-test-suite-cache-breakpoints.ts
| /** | ||
| * Extract cache creation token count from various provider formats | ||
| * Supports: cacheCreationInputTokens, cacheCreationTokens | ||
| * Supports: cacheCreationInputTokens, cacheCreationTokens, and the ai@6 | ||
| * normalized `inputTokenDetails.cacheWriteTokens` (the only shape the native | ||
| * direct-Anthropic path reports through generateText). |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== tokenUtils.ts ==\n'
sed -n '1,240p' src/lib/utils/tokenUtils.ts
printf '\n== search extractInputTokens / calculateCost usages ==\n'
rg -n "extractInputTokens|calculateCost|extractCacheReadTokens|extractCacheCreationTokens|cachedInputTokens|inputTokenDetails|cacheReadTokens|cacheWriteTokens" src/lib -SRepository: juspay/neurolink
Length of output: 26702
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== ai sdk anthopic source/docs search terms ==\n'
rg -n "cacheReadTokens|cacheWriteTokens|cachedInputTokens|inputTokens.*cache|inputTokenDetails|LanguageModelUsage" . -SRepository: juspay/neurolink
Length of output: 21836
🌐 Web query:
AI SDK 6 Anthropic inputTokens inputTokenDetails cacheReadTokens cacheWriteTokens inclusive exclusive semantics
💡 Result:
In the Vercel AI SDK (version 6 and later), token usage reporting for Anthropic has been standardized to use the provider-agnostic usage object, moving away from provider-specific metadata fields [1]. Key components of the usage object related to Anthropic prompt caching are found within result.usage.inputTokenDetails [2][1][3][4]: - cacheReadTokens: Represents the number of input tokens served from the cache (cache hit) [1][5][3]. - cacheWriteTokens: Represents the number of input tokens written to the cache (cache creation/miss) [1][5][3]. - noCacheTokens: Represents the number of input tokens that were not cached [3]. These values are accessible directly from the usage object returned by generateText and streamText [1][4]. The AI SDK previously exposed cache creation tokens via providerMetadata.anthropic (as cacheCreationInputTokens), but this field was removed to unify the interface across all providers [1]. Regarding semantics: - The inputTokens total in the usage object generally reflects the total number of tokens processed [6]. - Cache read and cache write tokens are distinct; the first turn often involves cacheWriteTokens > 0 (and cacheReadTokens = 0), while subsequent turns with a byte-stable prefix will show cacheReadTokens > 0 [5]. - "Inclusive" or "exclusive" phrasing is not standard terminology for these fields in the SDK; rather, these fields are additive components that categorize the total inputTokens (i.e., inputTokens ≈ noCacheTokens + cacheReadTokens + cacheWriteTokens) [6][3]. For implementation, you set cache breakpoints using providerOptions in your message content [4]: { providerOptions: { anthropic: { cacheControl: { type: 'ephemeral' } } } } [4]
Citations:
- 1: vercel/ai@832f86f
- 2: https://ai-sdk.dev/providers/ai-sdk-providers/anthropic
- 3: https://cdn.jsdelivr.net/npm/ai@6.0.172/src/types/usage.ts
- 4: https://ai-sdk.dev/v7/providers/ai-sdk-providers/anthropic
- 5: https://trigger.dev/docs/ai-chat/prompt-caching
- 6: Amazon Bedrock and Anthropic report different usages vercel/ai#11532
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '730,770p' src/lib/utils/pricing.ts
printf '\n== relevant tokenUtils block ==\n'
sed -n '236,320p' src/lib/utils/tokenUtils.ts
printf '\n== tests around cache breakpoints ==\n'
sed -n '470,525p' test/continuous-test-suite-cache-breakpoints.tsRepository: juspay/neurolink
Length of output: 6499
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== src/lib/types/common.ts slice ==\n'
sed -n '330,390p' src/lib/types/common.ts
printf '\n== test/continuous-test-suite-cache-breakpoints.ts slice ==\n'
sed -n '490,535p' test/continuous-test-suite-cache-breakpoints.ts
printf '\n== GenerationHandler usage/cost slice ==\n'
sed -n '650,950p' src/lib/core/modules/GenerationHandler.tsRepository: juspay/neurolink
Length of output: 14236
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
p = Path('test/continuous-test-suite-cache-breakpoints.ts')
text = p.read_text()
for i, line in enumerate(text.splitlines(), 1):
if 525 <= i <= 575:
print(f"{i}: {line}")
PYRepository: juspay/neurolink
Length of output: 715
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
p = Path('test/continuous-test-suite-cache-breakpoints.ts')
for i, line in enumerate(p.read_text().splitlines(), 1):
if 525 <= i <= 575:
print(f"{i}: {line}")
PYRepository: juspay/neurolink
Length of output: 715
Treat ai@6 cache tokens as overlapping with inputTokens
inputTokens is the inclusive total for the direct-Anthropic generateText shape, so cachedInputTokens, inputTokenDetails.cacheReadTokens, and inputTokenDetails.cacheWriteTokens need to be removed from input before pricing. As written, they’re handled like additive buckets and calculateCost will bill the cached/write portion twice.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/lib/utils/tokenUtils.ts` around lines 94 - 98, The token extraction and
pricing logic must treat ai@6 cache token fields as subsets of total input, not
additive tokens. Update the relevant functions in tokenUtils.ts, including the
cache extraction and calculateCost paths, to subtract cachedInputTokens,
inputTokenDetails.cacheReadTokens, and inputTokenDetails.cacheWriteTokens from
inputTokens before pricing, while preserving existing provider-specific
handling.
Tara-ag
left a comment
Tara-ag
left a comment
There was a problem hiding this comment.
Review Summary
Files reviewed: 6 changed files, processed one at a time.
New issues raised: 2 (both MINOR/SUGGESTION, non-blocking).
Findings
-
src/lib/providers/anthropic.ts(line ~1812) — 💡 The inline comment claimsconversationitself is never mutated, but the loop laterpushes assistant/tool turns into it. The breakpoint helper is indeed pure (it clones), so re-counting stays stable; the wording just needs to credit the helper rather than the mutable array. -
src/lib/providers/openaiChatCompletionsClient.ts(line ~643) — 💡 Consolidating the deferred-usage shape into the newDeferredUsagetype is good. SinceopenaiChatCompletionsBase.tsconsumescreateDeferredAnalyticsand currently resolves only the three required fields, cache reads/writes can still be dropped for OpenAI-compatible providers. Suggest a follow-up to plumb cache fields throughrunStreamLoopwhen providers populate them.
Blocking assessment
- No hardcoded secrets or credentials.
- No security vulnerabilities (injection, SSRF, unsafe eval, etc.).
- No breaking changes to the public SDK API (
DeferredUsageand the new optionalRawUsageObjectfields are additive). - No violations of non-negotiable CLAUDE.md Critical Rules enforced outside CI (dynamic imports, Gemini/tools+schema mutual exclusion, CLI/SDK separation,
formatProviderErrorreturn contract, backward compatibility). - The Anthropic cache-marker budget logic correctly respects Anthropic’s four-marker cap and is idempotent across multi-step loops.
- Cache token accounting in both
doGenerateandexecuteStreammatches Anthropic’s separate cache-read/cache-write reporting and feedscalculateCostcorrectly.
Decision: Approve. The two comments are minor clarity/follow-up notes and do not block merge.
| const conversation = payload.messages.slice(); | ||
| let totalInput = 0; | ||
| let totalOutput = 0; | ||
| let totalCacheRead = 0; |
There was a problem hiding this comment.
💡 The comment claims conversation itself is never mutated, but the loop later calls conversation.push(...) twice per tool step. The breakpoint helper is pure (it clones the array), but the statement about conversation is misleading. Consider rephrasing to clarify that the helper’s purity is what keeps re-counting stable, e.g.:
// Pure: applyAnthropicHistoryCacheBreakpoints clones `conversation`,
// so re-counting the original array at the top of each step stays stable.| completionTokens: number; | ||
| totalTokens: number; | ||
| }>((r) => { | ||
| let resolveUsage: (u: DeferredUsage) => void = () => {}; |
There was a problem hiding this comment.
💡 The new DeferredUsage type is a good consolidation. Just confirming: createDeferredAnalytics is also consumed by openaiChatCompletionsBase.ts (via this module). The base class currently resolves usage with only promptTokens/completionTokens/totalTokens. Since DeferredUsage now carries optional cache fields, consider whether openaiChatCompletionsBase.runStreamLoop should forward cache values when providers populate them. Not blocking for this PR, but worth a follow-up so OpenAI-compatible cache reads aren't dropped at the resolve boundary.
|
🎉 This PR is included in version 9.86.2 🎉 The release is available on: Your semantic-release bot 📦🚀 |
Problem
The native Vertex+Claude path has had full prompt caching since 9.79.1 (
applyVertexAnthropicCacheBreakpoints: system + tools + rolling history markers, cache-token capture, discounted pricing). The directanthropicprovider was left behind in three ways:cache_control— the growing, tool-result-heavy conversation history falls after the last marker and is re-billed at full input price on every turn.cachedInputTokensand nestedinputTokenDetails.{cacheReadTokens,cacheWriteTokens}(asLanguageModelUsage()) — none of whichextractCacheReadTokens/extractCacheCreationTokensrecognized. Caching happened and Anthropic billed the discount, but NeuroLink reported the cache tokens as undefined. (Bedrock hits the same GenerationHandler path.)executeStream()accumulated onlyinput_tokens/output_tokensoffmessage_start;cache_read_input_tokens/cache_creation_input_tokenson the same event were never read.Fix
utils/anthropicCacheBreakpoints.ts— newapplyAnthropicHistoryCacheBreakpoints()(rolling tail markers for paths whose stable-prefix markers are managed upstream) +countAnthropicCacheMarkers(). The count feeds the budget so a request can never exceed Anthropic's four-marker cap — critical on the direct path, where the AI-SDK pipeline has already placed system/tool markers before the provider sees the request. Already-marked tail blocks are skipped without consuming budget, so per-step re-application stays idempotent.applyVertexAnthropicCacheBreakpointsis untouched (all 34 existing tests pass unchanged).providers/anthropic.ts—doGenerateand theexecuteStreamstep loop apply the history breakpoints (re-applied per step, mirroring the Vertex loops);executeStreamnow accumulates cache reads/writes and threads them throughresolveUsageinto the OTel span (gen_ai.usage.cached_input_tokens) andcalculateCost(which already pricescacheReadTokens/cacheCreationTokens).utils/tokenUtils.ts+types/common.ts— extractors recognize the ai@6 shapes; legacy field names keep precedence, so no behavior change for any existing provider.Tests
Extended
test/continuous-test-suite-cache-breakpoints.ts(pnpm run test:cache): 47/47 pass — 13 new assertions covering marker counting, budget clamping (incl. zero/negative), newest-first placement, purity, idempotent re-application, and the ai@6 extraction shapes.Origin
Found while investigating cached-token accounting for curator (Claude-on-Vertex works today; this brings the currently-dormant direct-Anthropic path to parity before anyone switches to it, e.g. for OAuth/Claude-subscription access).
Summary by CodeRabbit
New Features
Bug Fixes