Skip to content
105 changes: 105 additions & 0 deletions internal/cli/pricing.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
package cli

import (
"strings"

agentruntime "github.com/fullsend-ai/fullsend/internal/runtime"
)

// modelRates holds per-million-token prices (USD) for an Anthropic model.
type modelRates struct {
Input float64
Output float64
CacheWrite float64
CacheRead float64
}

// knownModelRates maps model-ID prefixes to their published rates.
// Source: https://docs.anthropic.com/en/docs/about-claude/pricing
var knownModelRates = map[string]modelRates{
"claude-fable-5-1": {10, 50, 12.50, 0.25},

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[low] idiomatic-go

The knownModelRates map uses unkeyed composite literals ({10, 50, 12.50, 0.25}) for modelRates, which has four consecutive float64 fields (Input, Output, CacheWrite, CacheRead). Positional literals of identical types make cache-write vs. cache-read swaps easy to introduce and hard to spot in review (e.g. claude-fable-5-1's CacheRead is 0.25 vs. claude-fable-5's CacheRead of 1).

Suggested fix: Key the fields in the struct literals, e.g. {Input: 10, Output: 50, CacheWrite: 12.50, CacheRead: 0.25}.

"claude-mythos-5-1": {10, 50, 12.50, 0.25},
"claude-fable-5": {10, 50, 12.50, 1},
"claude-mythos-5": {10, 50, 12.50, 1},
"claude-opus-5": {5, 25, 6.25, 0.50},
"claude-opus-4-8": {5, 25, 6.25, 0.50},
"claude-opus-4-7": {5, 25, 6.25, 0.50},
"claude-opus-4-6": {5, 25, 6.25, 0.50},
"claude-opus-4-5": {5, 25, 6.25, 0.50},
"claude-sonnet-5": {2, 10, 2.50, 0.20},
"claude-sonnet-4-6": {3, 15, 3.75, 0.30},
"claude-sonnet-4-5": {3, 15, 3.75, 0.30},
"claude-haiku-4-5": {1, 5, 1.25, 0.10},
"claude-haiku-3-5": {0.80, 4, 1, 0.08},
}

// lookupRates resolves a model ID to its pricing rates. It handles
// provider prefixes ("anthropic-vertex/claude-opus-4-6") and trailing
// date suffixes ("claude-haiku-4-5-20251001").
func lookupRates(model string) (modelRates, bool) {
if _, after, ok := strings.Cut(model, "/"); ok {
model = after
}
if r, ok := knownModelRates[model]; ok {
return r, true
}
for s := model; ; {
i := strings.LastIndex(s, "-")
if i <= 0 {
break
}
s = s[:i]
if r, ok := knownModelRates[s]; ok {
return r, true
}
}
return modelRates{}, false
}

func estimateCostFromTokens(model string, input, output, reasoning, cacheWrite, cacheRead int) float64 {
rates, ok := lookupRates(model)
if !ok {
return 0
}
return (float64(input)*rates.Input +
float64(output+reasoning)*rates.Output +
float64(cacheWrite)*rates.CacheWrite +
float64(cacheRead)*rates.CacheRead) / 1_000_000
}

// estimateRunMetricsCost fills in TotalCostUSD on per-iteration RunMetrics
// when it is zero but tokens are present.
func estimateRunMetricsCost(m *agentruntime.RunMetrics) {
if m.TotalCostUSD > 0 {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[high] backward-incompatible

estimateRunMetricsCost writes token-derived list-price USD into the same TotalCostUSD / PerModelUsage.CostUSD fields (then copied to fullsend.cost_usd, metrics.json total_cost_usd, and the status comment) whenever TotalCostUSD == 0 and tokens are present for a known Claude model prefix. This inverts the repository's own 'Cost data contract' (docs/guides/infrastructure/distributed-tracing.md, established via #6735/#6736 ~2 weeks before this PR): that section states fullsend does not maintain a model-price table and that a missing runtime cost propagates as $0.00 on all surfaces with no fallback calculation from token counts. docs/cli/run.md's total_cost_usd row repeats the same guarantee verbatim ('no fullsend-side pricing-table fallback'). There is no provenance flag distinguishing runtime-reported from estimated values anywhere in metrics.json, the OTel span attributes, or the status comment — per-model cost_usd (PerModelUsage) is blended the same way while still advertised as satisfying the documented sum(cost_usd) == total_cost_usd invariant. This PR's diff touches only internal/cli/pricing.go, pricing_test.go, and run.go; it does not revise the contract docs or add an ADR.

Suggested fix: Either (a) keep total_cost_usd / fullsend.cost_usd as strictly runtime-reported (zero when the runtime never supplied a cost) and surface the estimate through a distinct, clearly-labeled field/attribute (e.g. estimated_cost_usd, a cost_usd_source: reported|estimated marker), or (b) get explicit sign-off to revise the Cost data contract and then update docs/guides/infrastructure/distributed-tracing.md (No pricing-table fallback, token-counts-not-inputs, Distinction from backend-derived estimates) and docs/cli/run.md (total_cost_usd row and per-model usage) to describe the new behavior precisely. Do not silently reuse the fields the contract calls authoritative.

return
}
totalTokens := m.InputTokens + m.OutputTokens +
m.CacheCreationInputTokens + m.CacheReadInputTokens
if totalTokens == 0 {
return
}

if len(m.PerModelUsage) > 0 {
var total float64
for spec, u := range m.PerModelUsage {
if u.CostUSD > 0 {
total += u.CostUSD
continue
}
est := estimateCostFromTokens(spec,
u.InputTokens, u.OutputTokens, 0,
u.CacheCreationInputTokens, u.CacheReadInputTokens)
if est > 0 {
u.CostUSD = est
m.PerModelUsage[spec] = u
total += est
}
}
m.TotalCostUSD = total
return
}

m.TotalCostUSD = estimateCostFromTokens(m.Model,
m.InputTokens, m.OutputTokens, m.ReasoningTokens,
m.CacheCreationInputTokens, m.CacheReadInputTokens)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. Cancelled thinking runs understate spend 🐞 Bug ≡ Correctness

estimateRunMetricsCost prices m.ReasoningTokens, but Claude's cancellation path neither
preserves TokensEvent.ReasoningTokens in RunMetrics nor includes them in its deferred final
token snapshot. When a Claude run using extended thinking is cancelled before its result event, the
estimator receives zero reasoning tokens and every downstream cost sink records an estimate that
omits them.
Agent Prompt
## Issue description
Cancelled Claude runs discard reasoning-token usage before the new estimator reads `RunMetrics.ReasoningTokens`, causing their estimated cost and reasoning telemetry to be understated.

## Fix Focus Areas
- internal/runtime/claude_progress.go[135-155]
- internal/runtime/claude_progress.go[285-308]
- internal/runtime/claude.go[157-163]
- internal/cli/pricing.go[102-104]

## Recommended Fix
Make incremental and deferred `TokensEvent` values carry the cumulative reasoning-token total, and copy `TokensEvent.ReasoningTokens` into `RunMetrics.ReasoningTokens` alongside the other cumulative counters. Add a cancellation-style regression test with thinking tokens and no result event, asserting that the final metrics retain reasoning tokens and the estimated cost includes them.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

}
185 changes: 185 additions & 0 deletions internal/cli/pricing_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,185 @@
package cli

import (
"math"
"testing"

agentruntime "github.com/fullsend-ai/fullsend/internal/runtime"
)

func almostEqual(a, b, tol float64) bool {
return math.Abs(a-b) < tol
}

func TestLookupRates_ExactMatch(t *testing.T) {
r, ok := lookupRates("claude-sonnet-5")
if !ok {
t.Fatal("expected match for claude-sonnet-5")
}
if r.Input != 2 || r.Output != 10 {
t.Errorf("got Input=%v Output=%v, want 2/10", r.Input, r.Output)
}
}

func TestLookupRates_DateSuffix(t *testing.T) {
r, ok := lookupRates("claude-haiku-4-5-20251001")
if !ok {
t.Fatal("expected match for claude-haiku-4-5-20251001")
}
if r.Input != 1 || r.Output != 5 {
t.Errorf("got Input=%v Output=%v, want 1/5", r.Input, r.Output)
}
}

func TestLookupRates_ProviderPrefix(t *testing.T) {
r, ok := lookupRates("anthropic-vertex/claude-opus-4-6")
if !ok {
t.Fatal("expected match for anthropic-vertex/claude-opus-4-6")
}
if r.Input != 5 || r.Output != 25 {
t.Errorf("got Input=%v Output=%v, want 5/25", r.Input, r.Output)
}
}

func TestLookupRates_ProviderPrefixWithDate(t *testing.T) {
r, ok := lookupRates("anthropic/claude-sonnet-4-5-20241022")
if !ok {
t.Fatal("expected match for anthropic/claude-sonnet-4-5-20241022")
}
if r.Input != 3 || r.Output != 15 {
t.Errorf("got Input=%v Output=%v, want 3/15", r.Input, r.Output)
}
}

func TestLookupRates_CompoundSuffix(t *testing.T) {
r, ok := lookupRates("claude-opus-4-6-exp-20250826")
if !ok {
t.Fatal("expected match for claude-opus-4-6-exp-20250826")
}
if r.Input != 5 {
t.Errorf("got Input=%v, want 5", r.Input)
}
}

func TestLookupRates_Unknown(t *testing.T) {
_, ok := lookupRates("gpt-4o")
if ok {
t.Fatal("expected no match for gpt-4o")
}
}

func TestLookupRates_Empty(t *testing.T) {
_, ok := lookupRates("")
if ok {
t.Fatal("expected no match for empty string")
}
}

func TestEstimateCostFromTokens(t *testing.T) {
cost := estimateCostFromTokens("claude-sonnet-5", 10000, 5000, 0, 0, 0)
if !almostEqual(cost, 0.07, 0.0001) {
t.Errorf("got %v, want 0.07", cost)
}
}

func TestEstimateCostFromTokens_WithCache(t *testing.T) {
cost := estimateCostFromTokens("claude-opus-5", 5000, 2000, 0, 10000, 20000)
if !almostEqual(cost, 0.1475, 0.0001) {
t.Errorf("got %v, want 0.1475", cost)
}
}

func TestEstimateCostFromTokens_WithReasoning(t *testing.T) {
cost := estimateCostFromTokens("claude-sonnet-5", 1000, 500, 2000, 0, 0)
if !almostEqual(cost, 0.027, 0.0001) {
t.Errorf("got %v, want 0.027", cost)
}
}

func TestEstimateCostFromTokens_UnknownModel(t *testing.T) {
cost := estimateCostFromTokens("unknown-model", 10000, 5000, 0, 0, 0)
if cost != 0 {
t.Errorf("expected 0 for unknown model, got %v", cost)
}
}

func TestEstimateRunMetricsCost(t *testing.T) {
m := &agentruntime.RunMetrics{
Model: "claude-opus-4-6",
InputTokens: 20000,
OutputTokens: 8000,
CacheCreationInputTokens: 5000,
CacheReadInputTokens: 50000,
}
estimateRunMetricsCost(m)
if !almostEqual(m.TotalCostUSD, 0.35625, 0.001) {
t.Errorf("got %v, want 0.35625", m.TotalCostUSD)
}
}

func TestEstimateRunMetricsCost_PerModelUsage(t *testing.T) {
m := &agentruntime.RunMetrics{
InputTokens: 15000,
OutputTokens: 8000,
PerModelUsage: map[string]agentruntime.ModelUsage{
"anthropic-vertex/claude-sonnet-5": {
InputTokens: 10000,
OutputTokens: 5000,
},
"anthropic-vertex/claude-haiku-4-5": {
InputTokens: 5000,
OutputTokens: 3000,
},
},
}
estimateRunMetricsCost(m)
// sonnet: (10000*2 + 5000*10) / 1e6 = 0.07
// haiku: (5000*1 + 3000*5) / 1e6 = 0.02
if !almostEqual(m.TotalCostUSD, 0.09, 0.001) {
t.Errorf("got %v, want 0.09", m.TotalCostUSD)
}
if !almostEqual(m.PerModelUsage["anthropic-vertex/claude-sonnet-5"].CostUSD, 0.07, 0.001) {
t.Errorf("sonnet cost = %v, want 0.07", m.PerModelUsage["anthropic-vertex/claude-sonnet-5"].CostUSD)
}
if !almostEqual(m.PerModelUsage["anthropic-vertex/claude-haiku-4-5"].CostUSD, 0.02, 0.001) {
t.Errorf("haiku cost = %v, want 0.02", m.PerModelUsage["anthropic-vertex/claude-haiku-4-5"].CostUSD)
}
}

func TestEstimateRunMetricsCost_PerModelUsage_MixedCosts(t *testing.T) {
m := &agentruntime.RunMetrics{
InputTokens: 15000,
OutputTokens: 8000,
PerModelUsage: map[string]agentruntime.ModelUsage{
"anthropic/claude-sonnet-5": {
InputTokens: 10000,
OutputTokens: 5000,
CostUSD: 0.07,
},
"anthropic/claude-haiku-4-5": {
InputTokens: 5000,
OutputTokens: 3000,
},
},
}
estimateRunMetricsCost(m)
if !almostEqual(m.TotalCostUSD, 0.09, 0.001) {
t.Errorf("got %v, want 0.09", m.TotalCostUSD)
}
if m.PerModelUsage["anthropic/claude-sonnet-5"].CostUSD != 0.07 {
t.Error("existing CostUSD was overwritten")
}
}

func TestEstimateRunMetricsCost_Noop_WhenCostPresent(t *testing.T) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[medium] missing-test

The 15 new unit tests cover lookup/estimate arithmetic, per-model fill-in, mixed reported+estimated per-model costs, and the TotalCostUSD > 0 skip, but do not lock down the production contract this PR claims. There is no estimateRunMetricsCost-level test that a Codex-shaped RunMetrics (tokens present, openai/gpt model, cost 0) stays at 0 — only a lower-level estimateCostFromTokens_UnknownModel test exists. There is no test that estimateRunMetricsCost followed by aggregateRunMetrics (the actual run.go call order) persists a non-zero estimate end-to-end. Existing cancellation tests in internal/cli/run_test.go still assert total_cost_usd == 0 and never call the new estimator, so they would not catch a regression in the new wiring.

Suggested fix: Add tests for: (1) estimateRunMetricsCost on a Codex/unknown-model RunMetrics with tokens present stays at TotalCostUSD 0; (2) a cancelled-shaped RunMetrics run through estimateRunMetricsCost then aggregateRunMetrics yields the expected table estimate in TotalCostUSD; (3) if successful zero-cost Anthropic runs should stay at $0, an explicit test for that case beyond the existing Noop_WhenCostPresent test.

m := &agentruntime.RunMetrics{
TotalCostUSD: 0.50,
Model: "claude-sonnet-5",
InputTokens: 10000,
OutputTokens: 5000,
}
estimateRunMetricsCost(m)
if m.TotalCostUSD != 0.50 {
t.Errorf("cost changed from 0.50 to %v", m.TotalCostUSD)
}
}
15 changes: 10 additions & 5 deletions internal/cli/run.go
Original file line number Diff line number Diff line change
Expand Up @@ -2268,6 +2268,11 @@ func runAgent(ctx context.Context, agentName, fullsendDir, outputBase, targetRep
}
}

// Estimate USD cost from tokens when the authoritative cost is
// missing (zero) — e.g. cancelled runs that never receive the
// result event. On successful runs this is a no-op.
estimateRunMetricsCost(&metrics)
Comment thread
qodo-code-review[bot] marked this conversation as resolved.
Resolved

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[medium] logic-error

estimateRunMetricsCost(&metrics) runs on every iteration before handleRunCancellation, not only on the cancelled/incomplete-stream path. The surrounding comment claims 'On successful runs this is a no-op,' but the only skip is TotalCostUSD > 0 (pricing.go:73). Zero therefore means both 'runtime never reported cost' (Claude cancelled runs with no ResultEvent; Codex, per ADR 0099) and 'runtime reported a real $0.' Any completed Anthropic-model iteration whose stream omits or zeroes cost (pi usage.cost.total unset, a Claude result event missing total_cost_usd, OpenCode with no step costs) gets list-price USD written into TotalCostUSD. Codex currently stays at $0 only because gpt/openai model IDs miss knownModelRates, not because estimation is cancellation-scoped. The PR title and linked issue (#6936) frame this as cancelled-run cost recovery, but the call site applies unconditionally to any zero-cost iteration.

Suggested fix: Gate estimateRunMetricsCost to the cancellation/incomplete-stream path (e.g. call it from within handleRunCancellation, or skip unless the run was cancelled / no ResultEvent supplied a cost). Do not treat TotalCostUSD == 0 as 'cost missing' on successful iterations. Keep Codex/unknown models at 0.


// Accumulate behavioral metrics across iterations.
aggregateRunMetrics(&aggMetrics, &metrics, iteration)

Expand Down Expand Up @@ -3711,11 +3716,11 @@ func transcriptErrorMessage(te agentruntime.TranscriptError) string {
// artifact upload step (if: always()) captures the partial usage data
// (#6936).
//
// NOTE: TotalCostUSD will be zero in the persisted metrics because dollar
// cost is only available from the terminal ResultEvent, which a cancelled
// run never emits. Token counts (input, output, cache_read, cache_creation)
// are captured via the deferred TokensEvent and will be non-zero. See #6936
// for background.
// NOTE: TotalCostUSD is estimated from token counts and the published
// rate table when the authoritative cost is missing (cancelled runs never
// emit the terminal ResultEvent). The estimate uses 5-minute cache-write
// rates and is zero if the model is unrecognized. See estimateRunMetricsCost
// and #6936 for background.
//
// cancelled is false when ctx is still live, in which case the caller's
// normal control flow continues unchanged; the other return values are
Expand Down
Loading