fix(#6936): send SIGINT into sandbox so cancelled runs report total_cost_usd - #7208
fix(#6936): send SIGINT into sandbox so cancelled runs report total_cost_usd#7208ascerra wants to merge 7 commits into
Conversation
When a run is cancelled, Claude Code never emits its terminal result event (which carries total_cost_usd), so TotalCostUSD stays zero in telemetry. This adds a client-side pricing table that estimates cost from captured token counts and the model's published per-MTok rates. The estimation runs per-iteration before aggregation, so the estimated cost flows into the agent span, root span, metrics.json, and status comment — all four telemetry sinks. On successful runs where the authoritative cost is already present, the estimation is a no-op. lookupRates handles provider prefixes (anthropic-vertex/...) and iteratively strips trailing date/experiment suffixes to match model IDs against the rate table. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
🤖 Finished Review · ✅ Success · Started 11:29 AM UTC · Completed 11:47 AM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $5.92 |
Code Review by Qodo
1. Three agents miss cancellation signals
|
| m.TotalCostUSD = estimateCostFromTokens(m.Model, | ||
| m.InputTokens, m.OutputTokens, m.ReasoningTokens, | ||
| m.CacheCreationInputTokens, m.CacheReadInputTokens) |
There was a problem hiding this comment.
1. Cancelled thinking runs understate spend 🐞 Bug ≡ Correctness
estimateRunMetricsCost prices m.ReasoningTokens, but Claude's cancellation path neither preserves TokensEvent.ReasoningTokens in RunMetrics nor includes them in its deferred final token snapshot. When a Claude run using extended thinking is cancelled before its result event, the estimator receives zero reasoning tokens and every downstream cost sink records an estimate that omits them.
Agent Prompt
## Issue description
Cancelled Claude runs discard reasoning-token usage before the new estimator reads `RunMetrics.ReasoningTokens`, causing their estimated cost and reasoning telemetry to be understated.
## Fix Focus Areas
- internal/runtime/claude_progress.go[135-155]
- internal/runtime/claude_progress.go[285-308]
- internal/runtime/claude.go[157-163]
- internal/cli/pricing.go[102-104]
## Recommended Fix
Make incremental and deferred `TokensEvent` values carry the cumulative reasoning-token total, and copy `TokensEvent.ReasoningTokens` into `RunMetrics.ReasoningTokens` alongside the other cumulative counters. Add a cancellation-style regression test with thinking tokens and no result event, asserting that the final metrics retain reasoning tokens and the estimated cost includes them.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
Risk Assessment: moderate (2/5) DetailsTier 1 dropped sharply from the prior review's context (11 files/827 lines) to 5 files/151 lines with no protected, security, CI, or dependency signals, yielding a low Tier 1 composite; this is weighted against a high Tier 2 composite driven by very hot churn/author/fix-revert history on run.go, sandbox.go, and claude.go, and a Tier 3 composite reflecting good scope alignment offset by an unflagged change to existing signal-propagation behavior with no rollback flag. The weighted composite rounds to 2 (moderate), matching the prior score but for a materially smaller blast radius — file/line reduction lowered Tier 1's contribution while sustained hotspot churn on the same files kept Tier 2 elevated. Previous runRisk Assessment: moderate (2/5) DetailsScore holds at 2 (moderate), matching the prior assessment: fresh Tier 1 evaluation of the now-11-file/827-line diff still finds no protected/security/CI/dependency signals, keeping the heavily-weighted Tier 1 composite low despite the larger blast radius; this outweighs high Tier 2 churn/author/fix-revert history on the perennially hot run.go/sandbox.go/claude.go files and a rollback-safety concern from unflagged existing-behavior changes, offset by scope-aligned issue coverage. Previous run (2)Risk Assessment: moderate (2/5) DetailsComposite = 0.50(1.75) + 0.30(3.29) + 0.20(2.6) = 2.38 -> 2 (moderate), matching the prior assessment: re-evaluating Tier 1 fresh for the now-10-file, 761-line diff still finds no protected/security/CI/dependency signals, so the heavily-weighted Tier 1 composite stays low (1.75) despite the larger blast radius; this outweighs high Tier 2 churn/coupling/fix-revert history on the perennially hot run.go/sandbox.go/tracing.md files (3.29) and a rollback-safety concern from unflagged existing-behavior changes plus largely-covered issue-scope alignment (2.6), so the score is preserved at 2 rather than escalated. Previous run (3)Risk Assessment: moderate (2/5) DetailsTier 1 signals changed materially since the prior assessment (2 new files, larger blast radius) and were re-evaluated fresh rather than anchored; despite high Tier 2 churn/coupling on the touched run.go/run.md/tracing.md hotspots and a rollback-safety concern (no feature flag on a change to existing telemetry behavior), the heavily-weighted Tier 1 composite stays low (good test ratio, no protected/security/CI/dependency signals, experienced non-bot author), yielding a moderate composite score of 2. Previous run (4)Risk Assessment: moderate (2/5) DetailsSmall, well-tested change touching one high-churn, multi-author file with a recurring fix history, paired with a narrowly-scoped follow-up to an already-substantially-resolved closed issue that lacks a rollback flag — combining to an overall moderate risk score. |
ReviewFindingsMedium
Low
These findings should be addressed before merge — the medium-severity SIGINT-targeting gap means this PR's stated fix does not yet work for the Since the prior review: this PR pivoted away from the previously-reviewed "cost-estimation via historical rate cache" approach — Next steps:
Previous runReviewFindingsMedium
Low
These findings should be addressed before merge — the medium-severity items (rate-cache persistence, Codex cost fabrication, missing test coverage, the SIGINT-targeting gap for pi/Codex, and the openshell relay's weakened kill guarantees) are functional gaps in the cancellation/cost-estimation mechanism this PR is meant to fix, not stylistic nits. Next steps:
Previous run (2)ReviewFindingsMedium
Low
Next steps:
Previous run (3)ReviewFindingsMedium
Low
This PR follows up on closed issue #6936, replacing the earlier static-pricing-table approach (flagged in the prior review) with a self-calibrating per-model rate cache — a real improvement in design direction. However, the cache as wired has no persistence mechanism across the ephemeral GitHub Actions runs that motivated the original issue, so the estimator is a no-op in the primary target environment; the estimation call site is still not gated to the cancellation path the docs describe; and both doc files were only partially updated, leaving several passages that contradict the new behavior. These should be addressed before merge. Next steps:
Previous run (4)ReviewFindingsHigh
Medium
Low
This PR follows up on closed issue #6936 (already substantially resolved by the separately-merged #6938), using the issue's "add a separate cost-reconciliation mechanism" acceptance-criterion alternative as authorization. However, as implemented, the mechanism is not "separate" — it overwrites the same fields the repository's recently-established Cost data contract designates as authoritative and guarantees have no pricing-table fallback. That contract conflict, combined with the overly-broad (non-cancellation-gated) trigger condition, should be resolved before merge. Next steps:
|
Replace the hardcoded Anthropic pricing table with a self-calibrating rate cache that derives effective $/token rates from successful runs. Rates are stored in .fullsend-cache/pricing-rates.json and updated via exponential moving average. This handles any model (Anthropic, OpenAI, etc.) without going stale when prices change. Fixes from review: - Restrict suffix-stripping to date suffixes and known tags (exp, preview, etc.) to prevent false matches like gpt-4o-mini → gpt-4o - Exclude ReasoningTokens from rate denominator since the deferred TokensEvent on cancelled runs does not capture them - Surface saveRateCache errors via printer.StepWarn instead of silently discarding - Update cost data contract docs and run.md field description Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
🤖 Finished Review · ✅ Success · Started 2:32 PM UTC · Completed 2:55 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $5.25 |
Layer 1 — action.yml: use `exec fullsend run` so signals go directly to fullsend instead of bash. Without exec, bash exits on SIGINT, fullsend becomes an orphan, and GHA kills it before cleanup can flush telemetry. Layer 2 — ExecStreamReader: send SIGINT (not SIGKILL) to openshell on context cancellation with a 5s WaitDelay, giving Claude Code time to flush its result event through the pipe. Layer 3 — parseClaudeStream: extract token usage from assistant-event messages (stream-json format) as fallback when stream_event wrappers are absent. Without this, the deferred TokensEvent on cancelled runs reports zero tokens. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
🤖 Finished Review · ✅ Success · Started 4:20 PM UTC · Completed 4:46 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $7.74 |
The prior Cancel function only SIGINTs the openshell CLI, which does not propagate through gRPC to in-sandbox processes. Claude Code never receives SIGINT and never flushes the result event with total_cost_usd. Four changes fix this: 1. Setsid: isolate openshell from GHA's process-group SIGINT so the relay stays alive to pipe the result event back. 2. PID file: record the in-sandbox process PID at startup so the cancel function can target it directly. 3. Kill-exec: on context cancellation, spawn a separate openshell sandbox exec to send kill -INT to the PID inside the sandbox. 4. exec claude: replace the wrapper shell with claude so the PID file points to claude's process, not the shell. Includes diagnostic stderr output at cancel and stream-end points to verify the mechanism in e2e testing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
🤖 Review · Commit: |
cmd.Process.Signal(SIGINT) kills openshell immediately, closing the stdout pipe before the kill-exec goroutine can send SIGINT to claude inside the sandbox. Claude's result event (with total_cost_usd) has nowhere to go because the pipe is already dead. Remove the direct SIGINT to openshell. The kill-exec goroutine is the sole mechanism: it sends SIGINT to claude via a separate openshell sandbox exec. Openshell stays alive to relay claude's result event through the pipe. WaitDelay (8s) is the fallback if claude doesn't exit. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
🤖 Finished Review · ✅ Success · Started 6:23 PM UTC · Completed 6:49 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $8.03 |
…sandbox fix (#6936) Remove pricing.go, pricing_test.go, and rate-cache wiring in run.go. Remove assistant-message usage tracking from claude_progress.go (separate concern). Revert doc changes that described the rate-cache approach. Update handleRunCancellation docstring to reflect the SIGINT mechanism. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
) - Fix WaitDelay docstring: "5-second" → "8-second" to match code - Add empty-PID diagnostic in killScript so logs distinguish "pid-file missing" from "kill delivered" - Document stderrW thread-safety requirement and single-use invariant - Add comment on exec claude noting PID-file dependency Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
🤖 Review · Commit: |
PR Summary by QodoPropagate cancellation SIGINT to preserve Claude cost telemetry
AI Description
Diagram
High-Level Assessment
Files changed (5)
|
|
🤖 Review · ❌ Terminated · Started 7:14 PM UTC · Ended 7:37 PM UTC Commit: |
| // exec replaces the shell so ExecStreamReader's PID file points to | ||
| // claude, not sh — required for the SIGINT-into-sandbox cancel mechanism. | ||
| parts := []string{ | ||
| fmt.Sprintf("cd %s && . %s && claude", params.RepoDir, envFile), | ||
| fmt.Sprintf("cd %s && . %s && exec claude", params.RepoDir, envFile), |
There was a problem hiding this comment.
3. Signal routing can regress untested 📘 Rule violation ▣ Testability
buildRunCommand now relies on exec claude for correct signal delivery, but no modified Claude runtime test asserts that the generated command includes the shell replacement. If a later command refactor removes exec, existing tests still pass while the PID file targets the intermediate shell instead of the process that must flush final telemetry.
Agent Prompt
## Issue description
The new signal-delivery mechanism depends on `buildRunCommand` producing `exec claude`, but the corresponding runtime tests do not assert this behavior.
## Fix Focus Areas
- internal/runtime/claude_test.go[217-223]
## Recommended Fix
Update `TestBuildRunCommand_Basic` to assert that the generated command contains `exec claude`, ensuring removal of the required shell replacement causes a test failure.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| // exec replaces the shell so ExecStreamReader's PID file points to | ||
| // claude, not sh — required for the SIGINT-into-sandbox cancel mechanism. | ||
| parts := []string{ | ||
| fmt.Sprintf("cd %s && . %s && claude", params.RepoDir, envFile), | ||
| fmt.Sprintf("cd %s && . %s && exec claude", params.RepoDir, envFile), |
There was a problem hiding this comment.
4. Runtime change skips guide review 📘 Rule violation ⛨ Security
ClaudeRuntime.Run and buildRunCommand change an existing runtime backend's execution behavior, but the PR description does not state that docs/contributing/runtime-implementation.md was consulted. Because direct process execution participates in the documented egress-binary contract, reviewers cannot verify from the change record that the runtime checklist was evaluated.
Agent Prompt
## Issue description
The Claude runtime backend now directly replaces its shell process, but the PR does not record consultation of the runtime implementation guide as required for backend behavior changes.
## Fix Focus Areas
- internal/runtime/claude.go[355-358]
## Recommended Fix
Update the PR description to explicitly state that `docs/contributing/runtime-implementation.md` was consulted and that the existing Claude egress-binary mapping, wire protocol, and workspace layout remain unchanged.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| // in-sandbox process. Callers that use "exec" in their command (e.g. | ||
| // buildRunCommand) replace the shell, so the PID file points to the | ||
| // final process. | ||
| wrapped := fmt.Sprintf("echo $$ > %s; %s", execPIDFile, command) |
There was a problem hiding this comment.
1. Three agents miss cancellation signals 🔗 Cross-repo conflict ☼ Reliability
ExecStreamReader records the wrapper shell's $$ and later sends SIGINT only to that PID, but buildPiRunCommand launches Pi as a child without exec and the Codex builder uses a pipeline rather than replacing the shell as Claude does. Cancellation of Codex or the agents repository's Pi-based code, prioritize, and review agents can therefore leave the real agent running without flushing terminal output until the relay fallback timeout, WaitDelay, or sandbox teardown.
Agent Prompt
## Issue description
`ExecStreamReader` assumes the PID written from `$$` becomes the agent PID, but that is only true when the supplied command replaces the wrapper shell. Codex uses a pipeline and Pi launches without `exec`, so cancellation targets `sh` rather than the actual agent process.
## Fix Focus Areas
- internal/sandbox/sandbox.go[1390-1440]
- internal/runtime/codex_run.go[365-370]
- internal/runtime/pi_run.go[497-505]
- internal/sandbox/sandbox_test.go[306-363]
## Recommended Fix
Change the launch protocol so the PID file reliably identifies the actual agent process or its dedicated process group for every `ExecStreamReader` caller. Launch Pi with `exec` after completing its setup and guard commands, update Codex command construction as needed, and add cancellation tests covering all three runtime command shapes to verify that each agent receives SIGINT, exits gracefully, and flushes its final output rather than relying on `WaitDelay`.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
|
Code review by qodo was updated up to the latest commit d419f66 |
|
This bypasses OpenShell’s supported stop lifecycle: |
| "PID=$(cat %s 2>/dev/null); "+ | ||
| "if [ -z \"$PID\" ]; then echo \"sandbox-cancel: pid-file missing or empty\" >&2; exit 1; fi; "+ | ||
| "echo \"sandbox-cancel: pid=$PID\" >&2; "+ | ||
| "kill -INT $PID 2>&1; echo \"sandbox-cancel: kill=$?\" >&2", |
There was a problem hiding this comment.
[medium] runtime-mechanism
ExecStreamReader records echo $$ > /tmp/.fullsend_exec.pid and on cancel sends SIGINT only to that PID (kill -INT $PID). Claude's exec claude replaces the wrapping shell, so $$ is the runtime and SIGINT is delivered correctly. Codex still ends in a pipeline (printf ... | "$codex" exec in internal/runtime/codex_run.go) — a shell cannot last-exec a pipeline, so $$ is the wrapper shell and kill -INT $PID never reaches the Codex process. Pi launches "$FULLSEND_PI_BIN" --print without exec (internal/runtime/pi_run.go); it may happen to inherit $$ via shell last-command-exec, but that is implicit and untested. Both non-Claude callers also now pay the new 8s WaitDelay on every cancel/timeout before the openshell relay is SIGKILLed. TestExecStreamReader_SendsSIGINTOnCancel only covers a cooperative same-PID child, so this gap is untested. This mechanism is squarely this PR's stated purpose, and as implemented it verifiably works for Claude but not for two of the three supported runtimes.
Suggested fix: Either exec the payload in every ExecStreamReader caller (so $$ is always the runtime process, matching the Claude pattern) or deliver SIGINT to the recorded PID's process group (kill -INT -- -$PID) so wrapper-shell children receive it. Extend the cancellation test to a wrapping shell with a non-exec'd child and a pipeline child.
| // (SIGINT from GHA cancellation) don't kill it before cmd.Cancel | ||
| // has a chance to send SIGINT into the sandbox and relay the | ||
| // result event through stdout. | ||
| cmd.SysProcAttr = &syscall.SysProcAttr{Setsid: true} |
There was a problem hiding this comment.
[low] permission-expansion
Setsid: true moves the openshell relay into a new session, and cmd.Cancel returns nil without ever signaling that host process directly — the only host-side kill is cmd.WaitDelay = 8s, with no Pdeathsig. If the parent fullsend process is SIGKILLed before WaitDelay fires, the relay can be reparented to init and keep running. An 8s WaitDelay also sits at or past GitHub Actions' own ~7.5s SIGINT-to-SIGKILL step window, so the intended force-kill of the relay may never run on a cancelled GHA job. Blast radius is limited to an already-sandboxed relay process (bounded by the existing --timeout, and torn down with the GHA VM), so this is a lifecycle-hardening gap rather than a genuine privilege expansion.
Suggested fix: If Setsid is required, also set SysProcAttr.Pdeathsig = syscall.SIGKILL (Linux) so a killed parent cannot leave the relay orphaned. Shrink WaitDelay (or the internal 5s kill-exec timeout) so cleanup still fits inside GitHub Actions' cancellation budget.
| "PID=$(cat %s 2>/dev/null); "+ | ||
| "if [ -z \"$PID\" ]; then echo \"sandbox-cancel: pid-file missing or empty\" >&2; exit 1; fi; "+ | ||
| "echo \"sandbox-cancel: pid=$PID\" >&2; "+ | ||
| "kill -INT $PID 2>&1; echo \"sandbox-cancel: kill=$?\" >&2", |
There was a problem hiding this comment.
[low] fail-open
ExecStreamReader's cancel script still treats the in-sandbox PID file as the sole SIGINT target. The new [ -z "$PID" ] check (added since the prior review) closes the absent/empty-file case, but the script still doesn't validate the content is a bare positive integer, and kill -INT $PID remains unquoted — a malformed PID file could still pass multiple IFS-split operands or a negative value. Blast radius stays inside the sandbox, and a bad PID still degrades to the WaitDelay fallback rather than granting host access, so this is a partial fix of the prior finding rather than a new issue.
Suggested fix: Reject non-numeric PID file contents (e.g. case "$PID" in ''|[!0-9]) exit 1;; esac) before signaling, and use kill -INT -- "$PID".
|
|
||
| if parseErr := parseClaudeStream(r, handler); parseErr != nil { | ||
| parseErr := parseClaudeStream(r, handler) | ||
| fmt.Fprintf(os.Stderr, " stream ended: parseErr=%v ctxErr=%v cost=%.4f\n", parseErr, ctx.Err(), metrics.TotalCostUSD) |
There was a problem hiding this comment.
[low] logging-conventions
fmt.Fprintf(os.Stderr, " stream ended: parseErr=%v ctxErr=%v cost=%.4f\n", ...) (line 181) and fmt.Fprintf(os.Stderr, " wait done: exitCode=%d waitErr=%v\n", ...) (line 193) execute unconditionally on every ClaudeRuntime.Run, including normal successful runs, adding stderr noise beyond the existing error-only " progress parser: %v\n" pattern. parseErr is also printed with a raw %v here (unsanitized) before being sanitized again on the error-handling path a few lines later.
Suggested fix: Remove the unconditional diagnostic prints, or gate them behind an explicit debug/verbose flag if they are meant to persist as permanent diagnostics.
| } | ||
| assert.Contains(t, string(all), "flushed-after-sigint", | ||
| "in-sandbox process should receive SIGINT via the kill-exec mechanism and flush output") | ||
|
|
There was a problem hiding this comment.
[low] test-idioms
In TestExecStreamReader_SendsSIGINTOnCancel, waitErr := cmd.Wait() assigns the error to a named variable that is immediately discarded via _ = waitErr — dead code / unnecessary indirection that also misses an opportunity to assert the wait outcome.
Suggested fix: Replace with _ = cmd.Wait() // wait for subprocess cleanup after cancellation, or assert the expected outcome.
|
🤖 Finished Review · ✅ Success · Started 7:14 PM UTC · Completed 7:37 PM UTC Commit: Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $6.63 |
Problem
Cancelled agent runs report
total_cost_usd: 0in telemetry. The authoritative cost comes from Claude Code'sresultevent — the terminal NDJSON event in the stream. When GitHub Actions cancels a run, SIGINT reaches thefullsendprocess but never propagates through openshell's gRPC boundary to theclaudeprocess inside the sandbox. Claude never flushes its result event, so cost is lost.Root cause
openshell sandbox execdoes not propagate signals from the CLI process to the in-sandbox process. SIGINT to the openshell CLI just kills the relay — the sandboxedclaudeprocess keeps running (orphaned) until the sandbox is torn down, never emitting its result event.Fix
Three coordinated changes make SIGINT reach claude inside the sandbox:
1. PID file + kill-exec in
ExecStreamReader(internal/sandbox/sandbox.go)echo $$ > /tmp/.fullsend_exec.pidto the command so the in-sandbox process records its PID at startupSysProcAttr.Setsid = true) so GHA's process-group SIGINT doesn't kill the relay before we can use itopenshell sandbox execthat reads the PID file and sendskill -INTto the in-sandbox process2.
exec claudeinbuildRunCommand(internal/runtime/claude.go)Replace
claudewithexec claudeso the shell process is replaced by claude. This makes the PID file point directly to claude's PID rather than the parent shell.3.
exec fullsend runinaction.ymlReplace bash with fullsend so GHA's cancellation signals go directly to the fullsend process instead of bash, which would kill fullsend before the cancel mechanism can run.
Timeline on cancellation
E2E verification
Tested on
fullsend-playground/go-hello(run 34513766953):Before:
total_cost_usd: 0on every cancelled runAfter:
total_cost_usd: 0.7879— claude flushed its result event within ~25ms of receiving SIGINTFiles changed
internal/sandbox/sandbox.go— PID file, Setsid, kill-exec cancel mechanism, WaitDelayinternal/sandbox/sandbox_test.go— test with fake openshell verifying SIGINT→flush→outputinternal/runtime/claude.go—exec claude, diagnostic logging at stream end and waitinternal/cli/run.go— updatedhandleRunCancellationdocstring to reflect SIGINT mechanismaction.yml—exec fullsend runReview squad results (3 agents: Claude x2, Grok)
Test plan
TestExecStreamReader_SendsSIGINTOnCancel— fake openshell, trap SIGINT, verify flushed outputtotal_cost_usd: 0.7879(was 0)TestExecStreamReader_OpenshellNotInPathstill passes