Repository navigation
fix(#6936): send SIGINT into sandbox so cancelled runs report total_cost_usd #7208
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
4549445
43f5e6e
1ea267e
d708592
7262bad
e0399b7
d419f66
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,105 @@ | ||
| package cli | ||
|
|
||
| import ( | ||
| "strings" | ||
|
|
||
| agentruntime "github.com/fullsend-ai/fullsend/internal/runtime" | ||
| ) | ||
|
|
||
| // modelRates holds per-million-token prices (USD) for an Anthropic model. | ||
| type modelRates struct { | ||
| Input float64 | ||
| Output float64 | ||
| CacheWrite float64 | ||
| CacheRead float64 | ||
| } | ||
|
|
||
| // knownModelRates maps model-ID prefixes to their published rates. | ||
| // Source: https://docs.anthropic.com/en/docs/about-claude/pricing | ||
| var knownModelRates = map[string]modelRates{ | ||
| "claude-fable-5-1": {10, 50, 12.50, 0.25}, | ||
| "claude-mythos-5-1": {10, 50, 12.50, 0.25}, | ||
| "claude-fable-5": {10, 50, 12.50, 1}, | ||
| "claude-mythos-5": {10, 50, 12.50, 1}, | ||
| "claude-opus-5": {5, 25, 6.25, 0.50}, | ||
| "claude-opus-4-8": {5, 25, 6.25, 0.50}, | ||
| "claude-opus-4-7": {5, 25, 6.25, 0.50}, | ||
| "claude-opus-4-6": {5, 25, 6.25, 0.50}, | ||
| "claude-opus-4-5": {5, 25, 6.25, 0.50}, | ||
| "claude-sonnet-5": {2, 10, 2.50, 0.20}, | ||
| "claude-sonnet-4-6": {3, 15, 3.75, 0.30}, | ||
| "claude-sonnet-4-5": {3, 15, 3.75, 0.30}, | ||
| "claude-haiku-4-5": {1, 5, 1.25, 0.10}, | ||
| "claude-haiku-3-5": {0.80, 4, 1, 0.08}, | ||
| } | ||
|
|
||
| // lookupRates resolves a model ID to its pricing rates. It handles | ||
| // provider prefixes ("anthropic-vertex/claude-opus-4-6") and trailing | ||
| // date suffixes ("claude-haiku-4-5-20251001"). | ||
| func lookupRates(model string) (modelRates, bool) { | ||
| if _, after, ok := strings.Cut(model, "/"); ok { | ||
| model = after | ||
| } | ||
| if r, ok := knownModelRates[model]; ok { | ||
| return r, true | ||
| } | ||
| for s := model; ; { | ||
| i := strings.LastIndex(s, "-") | ||
| if i <= 0 { | ||
| break | ||
| } | ||
| s = s[:i] | ||
| if r, ok := knownModelRates[s]; ok { | ||
| return r, true | ||
| } | ||
| } | ||
| return modelRates{}, false | ||
| } | ||
|
|
||
| func estimateCostFromTokens(model string, input, output, reasoning, cacheWrite, cacheRead int) float64 { | ||
| rates, ok := lookupRates(model) | ||
| if !ok { | ||
| return 0 | ||
| } | ||
| return (float64(input)*rates.Input + | ||
| float64(output+reasoning)*rates.Output + | ||
| float64(cacheWrite)*rates.CacheWrite + | ||
| float64(cacheRead)*rates.CacheRead) / 1_000_000 | ||
| } | ||
|
|
||
| // estimateRunMetricsCost fills in TotalCostUSD on per-iteration RunMetrics | ||
| // when it is zero but tokens are present. | ||
| func estimateRunMetricsCost(m *agentruntime.RunMetrics) { | ||
| if m.TotalCostUSD > 0 { | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [high] backward-incompatible estimateRunMetricsCost writes token-derived list-price USD into the same TotalCostUSD / PerModelUsage.CostUSD fields (then copied to fullsend.cost_usd, metrics.json total_cost_usd, and the status comment) whenever TotalCostUSD == 0 and tokens are present for a known Claude model prefix. This inverts the repository's own 'Cost data contract' (docs/guides/infrastructure/distributed-tracing.md, established via #6735/#6736 ~2 weeks before this PR): that section states fullsend does not maintain a model-price table and that a missing runtime cost propagates as $0.00 on all surfaces with no fallback calculation from token counts. docs/cli/run.md's Suggested fix: Either (a) keep total_cost_usd / fullsend.cost_usd as strictly runtime-reported (zero when the runtime never supplied a cost) and surface the estimate through a distinct, clearly-labeled field/attribute (e.g. estimated_cost_usd, a cost_usd_source: reported|estimated marker), or (b) get explicit sign-off to revise the Cost data contract and then update docs/guides/infrastructure/distributed-tracing.md (No pricing-table fallback, token-counts-not-inputs, Distinction from backend-derived estimates) and docs/cli/run.md (total_cost_usd row and per-model usage) to describe the new behavior precisely. Do not silently reuse the fields the contract calls authoritative. |
||
| return | ||
| } | ||
| totalTokens := m.InputTokens + m.OutputTokens + | ||
| m.CacheCreationInputTokens + m.CacheReadInputTokens | ||
| if totalTokens == 0 { | ||
| return | ||
| } | ||
|
|
||
| if len(m.PerModelUsage) > 0 { | ||
| var total float64 | ||
| for spec, u := range m.PerModelUsage { | ||
| if u.CostUSD > 0 { | ||
| total += u.CostUSD | ||
| continue | ||
| } | ||
| est := estimateCostFromTokens(spec, | ||
| u.InputTokens, u.OutputTokens, 0, | ||
| u.CacheCreationInputTokens, u.CacheReadInputTokens) | ||
| if est > 0 { | ||
| u.CostUSD = est | ||
| m.PerModelUsage[spec] = u | ||
| total += est | ||
| } | ||
| } | ||
| m.TotalCostUSD = total | ||
| return | ||
| } | ||
|
|
||
| m.TotalCostUSD = estimateCostFromTokens(m.Model, | ||
| m.InputTokens, m.OutputTokens, m.ReasoningTokens, | ||
| m.CacheCreationInputTokens, m.CacheReadInputTokens) | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 1. Cancelled thinking runs understate spend estimateRunMetricsCost prices m.ReasoningTokens, but Claude's cancellation path neither preserves TokensEvent.ReasoningTokens in RunMetrics nor includes them in its deferred final token snapshot. When a Claude run using extended thinking is cancelled before its result event, the estimator receives zero reasoning tokens and every downstream cost sink records an estimate that omits them. Agent Prompt
|
||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,185 @@ | ||
| package cli | ||
|
|
||
| import ( | ||
| "math" | ||
| "testing" | ||
|
|
||
| agentruntime "github.com/fullsend-ai/fullsend/internal/runtime" | ||
| ) | ||
|
|
||
| func almostEqual(a, b, tol float64) bool { | ||
| return math.Abs(a-b) < tol | ||
| } | ||
|
|
||
| func TestLookupRates_ExactMatch(t *testing.T) { | ||
| r, ok := lookupRates("claude-sonnet-5") | ||
| if !ok { | ||
| t.Fatal("expected match for claude-sonnet-5") | ||
| } | ||
| if r.Input != 2 || r.Output != 10 { | ||
| t.Errorf("got Input=%v Output=%v, want 2/10", r.Input, r.Output) | ||
| } | ||
| } | ||
|
|
||
| func TestLookupRates_DateSuffix(t *testing.T) { | ||
| r, ok := lookupRates("claude-haiku-4-5-20251001") | ||
| if !ok { | ||
| t.Fatal("expected match for claude-haiku-4-5-20251001") | ||
| } | ||
| if r.Input != 1 || r.Output != 5 { | ||
| t.Errorf("got Input=%v Output=%v, want 1/5", r.Input, r.Output) | ||
| } | ||
| } | ||
|
|
||
| func TestLookupRates_ProviderPrefix(t *testing.T) { | ||
| r, ok := lookupRates("anthropic-vertex/claude-opus-4-6") | ||
| if !ok { | ||
| t.Fatal("expected match for anthropic-vertex/claude-opus-4-6") | ||
| } | ||
| if r.Input != 5 || r.Output != 25 { | ||
| t.Errorf("got Input=%v Output=%v, want 5/25", r.Input, r.Output) | ||
| } | ||
| } | ||
|
|
||
| func TestLookupRates_ProviderPrefixWithDate(t *testing.T) { | ||
| r, ok := lookupRates("anthropic/claude-sonnet-4-5-20241022") | ||
| if !ok { | ||
| t.Fatal("expected match for anthropic/claude-sonnet-4-5-20241022") | ||
| } | ||
| if r.Input != 3 || r.Output != 15 { | ||
| t.Errorf("got Input=%v Output=%v, want 3/15", r.Input, r.Output) | ||
| } | ||
| } | ||
|
|
||
| func TestLookupRates_CompoundSuffix(t *testing.T) { | ||
| r, ok := lookupRates("claude-opus-4-6-exp-20250826") | ||
| if !ok { | ||
| t.Fatal("expected match for claude-opus-4-6-exp-20250826") | ||
| } | ||
| if r.Input != 5 { | ||
| t.Errorf("got Input=%v, want 5", r.Input) | ||
| } | ||
| } | ||
|
|
||
| func TestLookupRates_Unknown(t *testing.T) { | ||
| _, ok := lookupRates("gpt-4o") | ||
| if ok { | ||
| t.Fatal("expected no match for gpt-4o") | ||
| } | ||
| } | ||
|
|
||
| func TestLookupRates_Empty(t *testing.T) { | ||
| _, ok := lookupRates("") | ||
| if ok { | ||
| t.Fatal("expected no match for empty string") | ||
| } | ||
| } | ||
|
|
||
| func TestEstimateCostFromTokens(t *testing.T) { | ||
| cost := estimateCostFromTokens("claude-sonnet-5", 10000, 5000, 0, 0, 0) | ||
| if !almostEqual(cost, 0.07, 0.0001) { | ||
| t.Errorf("got %v, want 0.07", cost) | ||
| } | ||
| } | ||
|
|
||
| func TestEstimateCostFromTokens_WithCache(t *testing.T) { | ||
| cost := estimateCostFromTokens("claude-opus-5", 5000, 2000, 0, 10000, 20000) | ||
| if !almostEqual(cost, 0.1475, 0.0001) { | ||
| t.Errorf("got %v, want 0.1475", cost) | ||
| } | ||
| } | ||
|
|
||
| func TestEstimateCostFromTokens_WithReasoning(t *testing.T) { | ||
| cost := estimateCostFromTokens("claude-sonnet-5", 1000, 500, 2000, 0, 0) | ||
| if !almostEqual(cost, 0.027, 0.0001) { | ||
| t.Errorf("got %v, want 0.027", cost) | ||
| } | ||
| } | ||
|
|
||
| func TestEstimateCostFromTokens_UnknownModel(t *testing.T) { | ||
| cost := estimateCostFromTokens("unknown-model", 10000, 5000, 0, 0, 0) | ||
| if cost != 0 { | ||
| t.Errorf("expected 0 for unknown model, got %v", cost) | ||
| } | ||
| } | ||
|
|
||
| func TestEstimateRunMetricsCost(t *testing.T) { | ||
| m := &agentruntime.RunMetrics{ | ||
| Model: "claude-opus-4-6", | ||
| InputTokens: 20000, | ||
| OutputTokens: 8000, | ||
| CacheCreationInputTokens: 5000, | ||
| CacheReadInputTokens: 50000, | ||
| } | ||
| estimateRunMetricsCost(m) | ||
| if !almostEqual(m.TotalCostUSD, 0.35625, 0.001) { | ||
| t.Errorf("got %v, want 0.35625", m.TotalCostUSD) | ||
| } | ||
| } | ||
|
|
||
| func TestEstimateRunMetricsCost_PerModelUsage(t *testing.T) { | ||
| m := &agentruntime.RunMetrics{ | ||
| InputTokens: 15000, | ||
| OutputTokens: 8000, | ||
| PerModelUsage: map[string]agentruntime.ModelUsage{ | ||
| "anthropic-vertex/claude-sonnet-5": { | ||
| InputTokens: 10000, | ||
| OutputTokens: 5000, | ||
| }, | ||
| "anthropic-vertex/claude-haiku-4-5": { | ||
| InputTokens: 5000, | ||
| OutputTokens: 3000, | ||
| }, | ||
| }, | ||
| } | ||
| estimateRunMetricsCost(m) | ||
| // sonnet: (10000*2 + 5000*10) / 1e6 = 0.07 | ||
| // haiku: (5000*1 + 3000*5) / 1e6 = 0.02 | ||
| if !almostEqual(m.TotalCostUSD, 0.09, 0.001) { | ||
| t.Errorf("got %v, want 0.09", m.TotalCostUSD) | ||
| } | ||
| if !almostEqual(m.PerModelUsage["anthropic-vertex/claude-sonnet-5"].CostUSD, 0.07, 0.001) { | ||
| t.Errorf("sonnet cost = %v, want 0.07", m.PerModelUsage["anthropic-vertex/claude-sonnet-5"].CostUSD) | ||
| } | ||
| if !almostEqual(m.PerModelUsage["anthropic-vertex/claude-haiku-4-5"].CostUSD, 0.02, 0.001) { | ||
| t.Errorf("haiku cost = %v, want 0.02", m.PerModelUsage["anthropic-vertex/claude-haiku-4-5"].CostUSD) | ||
| } | ||
| } | ||
|
|
||
| func TestEstimateRunMetricsCost_PerModelUsage_MixedCosts(t *testing.T) { | ||
| m := &agentruntime.RunMetrics{ | ||
| InputTokens: 15000, | ||
| OutputTokens: 8000, | ||
| PerModelUsage: map[string]agentruntime.ModelUsage{ | ||
| "anthropic/claude-sonnet-5": { | ||
| InputTokens: 10000, | ||
| OutputTokens: 5000, | ||
| CostUSD: 0.07, | ||
| }, | ||
| "anthropic/claude-haiku-4-5": { | ||
| InputTokens: 5000, | ||
| OutputTokens: 3000, | ||
| }, | ||
| }, | ||
| } | ||
| estimateRunMetricsCost(m) | ||
| if !almostEqual(m.TotalCostUSD, 0.09, 0.001) { | ||
| t.Errorf("got %v, want 0.09", m.TotalCostUSD) | ||
| } | ||
| if m.PerModelUsage["anthropic/claude-sonnet-5"].CostUSD != 0.07 { | ||
| t.Error("existing CostUSD was overwritten") | ||
| } | ||
| } | ||
|
|
||
| func TestEstimateRunMetricsCost_Noop_WhenCostPresent(t *testing.T) { | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [medium] missing-test The 15 new unit tests cover lookup/estimate arithmetic, per-model fill-in, mixed reported+estimated per-model costs, and the TotalCostUSD > 0 skip, but do not lock down the production contract this PR claims. There is no estimateRunMetricsCost-level test that a Codex-shaped RunMetrics (tokens present, openai/gpt model, cost 0) stays at 0 — only a lower-level estimateCostFromTokens_UnknownModel test exists. There is no test that estimateRunMetricsCost followed by aggregateRunMetrics (the actual run.go call order) persists a non-zero estimate end-to-end. Existing cancellation tests in internal/cli/run_test.go still assert total_cost_usd == 0 and never call the new estimator, so they would not catch a regression in the new wiring. Suggested fix: Add tests for: (1) estimateRunMetricsCost on a Codex/unknown-model RunMetrics with tokens present stays at TotalCostUSD 0; (2) a cancelled-shaped RunMetrics run through estimateRunMetricsCost then aggregateRunMetrics yields the expected table estimate in TotalCostUSD; (3) if successful zero-cost Anthropic runs should stay at $0, an explicit test for that case beyond the existing Noop_WhenCostPresent test. |
||
| m := &agentruntime.RunMetrics{ | ||
| TotalCostUSD: 0.50, | ||
| Model: "claude-sonnet-5", | ||
| InputTokens: 10000, | ||
| OutputTokens: 5000, | ||
| } | ||
| estimateRunMetricsCost(m) | ||
| if m.TotalCostUSD != 0.50 { | ||
| t.Errorf("cost changed from 0.50 to %v", m.TotalCostUSD) | ||
| } | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -2268,6 +2268,11 @@ func runAgent(ctx context.Context, agentName, fullsendDir, outputBase, targetRep | |
| } | ||
| } | ||
|
|
||
| // Estimate USD cost from tokens when the authoritative cost is | ||
| // missing (zero) — e.g. cancelled runs that never receive the | ||
| // result event. On successful runs this is a no-op. | ||
| estimateRunMetricsCost(&metrics) | ||
|
qodo-code-review[bot] marked this conversation as resolved.
Resolved
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [medium] logic-error estimateRunMetricsCost(&metrics) runs on every iteration before handleRunCancellation, not only on the cancelled/incomplete-stream path. The surrounding comment claims 'On successful runs this is a no-op,' but the only skip is TotalCostUSD > 0 (pricing.go:73). Zero therefore means both 'runtime never reported cost' (Claude cancelled runs with no ResultEvent; Codex, per ADR 0099) and 'runtime reported a real $0.' Any completed Anthropic-model iteration whose stream omits or zeroes cost (pi usage.cost.total unset, a Claude result event missing total_cost_usd, OpenCode with no step costs) gets list-price USD written into TotalCostUSD. Codex currently stays at $0 only because gpt/openai model IDs miss knownModelRates, not because estimation is cancellation-scoped. The PR title and linked issue (#6936) frame this as cancelled-run cost recovery, but the call site applies unconditionally to any zero-cost iteration. Suggested fix: Gate estimateRunMetricsCost to the cancellation/incomplete-stream path (e.g. call it from within handleRunCancellation, or skip unless the run was cancelled / no ResultEvent supplied a cost). Do not treat TotalCostUSD == 0 as 'cost missing' on successful iterations. Keep Codex/unknown models at 0. |
||
|
|
||
| // Accumulate behavioral metrics across iterations. | ||
| aggregateRunMetrics(&aggMetrics, &metrics, iteration) | ||
|
|
||
|
|
@@ -3711,11 +3716,11 @@ func transcriptErrorMessage(te agentruntime.TranscriptError) string { | |
| // artifact upload step (if: always()) captures the partial usage data | ||
| // (#6936). | ||
| // | ||
| // NOTE: TotalCostUSD will be zero in the persisted metrics because dollar | ||
| // cost is only available from the terminal ResultEvent, which a cancelled | ||
| // run never emits. Token counts (input, output, cache_read, cache_creation) | ||
| // are captured via the deferred TokensEvent and will be non-zero. See #6936 | ||
| // for background. | ||
| // NOTE: TotalCostUSD is estimated from token counts and the published | ||
| // rate table when the authoritative cost is missing (cancelled runs never | ||
| // emit the terminal ResultEvent). The estimate uses 5-minute cache-write | ||
| // rates and is zero if the model is unrecognized. See estimateRunMetricsCost | ||
| // and #6936 for background. | ||
| // | ||
| // cancelled is false when ctx is still live, in which case the caller's | ||
| // normal control flow continues unchanged; the other return values are | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[low] idiomatic-go
The knownModelRates map uses unkeyed composite literals ({10, 50, 12.50, 0.25}) for modelRates, which has four consecutive float64 fields (Input, Output, CacheWrite, CacheRead). Positional literals of identical types make cache-write vs. cache-read swaps easy to introduce and hard to spot in review (e.g. claude-fable-5-1's CacheRead is 0.25 vs. claude-fable-5's CacheRead of 1).
Suggested fix: Key the fields in the struct literals, e.g. {Input: 10, Output: 50, CacheWrite: 12.50, CacheRead: 0.25}.