fix(#6905): capture token telemetry on cancelled Claude runs - #6908
Conversation
The Claude runtime only populated RunMetrics token fields from the terminal ResultEvent, which is never emitted when the process is killed (SIGTERM on cancellation). This left InputTokens, OutputTokens, and cache token fields at zero for every cancelled run, corrupting budget tracking and usage analytics. The fix has three parts: 1. Cumulative token tracking in parseClaudeStream: per-message token counters are folded into cumulative counters on each message_start, so TokensEvent now carries run-wide totals instead of single-message values. 2. TokensEvent handler in the metrics handler (claude.go and progressParser): incrementally updates RunMetrics from the cumulative TokensEvent. ResultEvent still overwrites with authoritative totals on successful runs. 3. Deferred final TokensEvent at EOF: when the stream ends without a ResultEvent (cancelled run), a deferred function emits one last TokensEvent with the cumulative snapshot, ensuring the metrics handler captures all available data. Cost (TotalCostUSD) remains zero for cancelled runs because the Claude stream does not expose per-API-call cost; a separate strategy is needed (noted in #6905). Closes #6905
|
🤖 Finished Review · ✅ Success · Started 11:28 AM UTC · Completed 11:47 AM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $7.12 |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
Risk Assessment: moderate (2/5) DetailsModerate risk driven primarily by high Tier 2 git history signals: these runtime files are a churn hotspot with 9 distinct contributors and heavy fix-commit density. The change itself is low-complexity — a well-scoped bug fix capturing token telemetry on cancelled runs with clear acceptance criteria and good test coverage. |
ReviewFindingsLow
|
![fullsend-ai-review[bot]](https://avatars.githubusercontent.com/in/3478501?s=60&v=4){kind=small}
fullsend-ai-review
Bot
added
the
ready-for-merge
|
🤖 Finished Retro · ✅ Success · Started 9:26 PM UTC · Completed 9:47 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $6.26 |
Retro: PR #6908 — fix(#6905): capture token telemetry on cancelled Claude runsOverall assessment: A well-executed single-commit bug fix with strong test coverage and fast turnaround (~10 hours from issue to merge). The code agent produced a correct, well-tested implementation on the first attempt with no rework cycles. The review agent found two valid low-severity observations but missed one medium-severity correctness bug (see proposal below). Timeline
Total agent cost: $12.08 across triage ($0.90), code ($4.06), and review ($7.12). What went well
Evidence for existing issues
Proposals filed
|
GitHub Actions cancellation sends SIGINT followed by SIGTERM ~7.5 s later. signal.NotifyContext stops listening after the first signal, so the second signal hit the default "terminate" handler and killed the process before metrics.json and the agent telemetry span could be written — silently losing API credit usage data on every cancelled Review run. Two fixes: 1. Replace signal.NotifyContext with a persistent signal channel that absorbs all subsequent SIGINT/SIGTERM deliveries. The first signal still cancels the context; subsequent signals no longer terminate the process, giving the cleanup path time to flush metrics and telemetry before the process exits normally. 2. Add a cancellation short-circuit in the agent iteration loop: when ctx.Err() is non-nil after rt.Run returns, immediately aggregate metrics, finalize the agent span, and write metrics.json — before extraction and validation that would be pointless on a dead sandbox. This ensures the artifact upload step (if: always()) captures the partial usage data even if the process is subsequently killed. Dollar cost (TotalCostUSD) remains zero on cancellation because it is only available from the terminal ResultEvent, which a killed run never emits. Token counts (input, output, cache_read, cache_creation) are captured via the deferred TokensEvent from PR fullsend-ai#6908. Closes fullsend-ai#6936
2 participants
Summary
Cancelled Claude Code runs reported $0.00 cost and 0 tokens in telemetry and status comments despite consuming real API credits. The root cause:
RunMetricstoken fields were only populated from the terminalResultEvent, which is never emitted when the process is killed via SIGTERM.Changes
claude_progress.go): Per-message token counters are now folded into cumulative counters on eachmessage_start, soTokensEventcarries run-wide totals instead of single-message valuesclaude.go,progressParser): AddedTokensEventcase to both metrics handlers to incrementally updateRunMetrics.ResultEventstill overwrites with authoritative totals on successful runsclaude_progress.go): When the stream ends without aResultEvent(cancelled run), a deferred function emits one lastTokensEventwith the cumulative snapshotResultEventto the throttle test to isolate in-stream throttle behavior from the new deferred EOF emissionTesting
TestProgressParserCancelledRunCapturesTokens— core regression test: stream with tool calls and token usage but noResultEvent, verifies metrics are non-zeroTestProgressParserResultOverwritesIncrementalTokens— verifiesResultEventauthoritative totals overwrite incrementalTokensEventsnapshotTestParseClaudeStreamCumulativeTokensAcrossMessages— verifies cumulative token tracking across multiple API callsTestParseClaudeStreamNoFinalTokensEventAfterResult— verifies no duplicateTokensEventafterResultEventTestParseClaudeStreamFinalTokensEventOnCancel— verifies deferred emission when below throttle thresholdgo vet,gofmt, secret scan cleanNote:
TotalCostUSDremains zero for cancelled runs because the Claude stream does not expose per-API-call cost. A separate strategy is needed (documented in #6905).Closes #6905
Post-script verification
agent/6905-claude-cancelled-run-tokens)7f0c80c8f8b5de7e148b1065a7f3a75029d78fdc..HEAD)