fix(#6936): persist token telemetry when agents are cancelled

[fullsend-ai-coder]

GitHub Actions cancellation sends SIGINT followed by SIGTERM ~7.5 s later. signal.NotifyContext stops listening after the first signal, so the second signal hit the default "terminate" handler and killed the process before metrics.json and the agent telemetry span could be written — silently losing API credit usage data on every cancelled Review run.

Two fixes:

1. Replace signal.NotifyContext with a persistent signal channel that
   absorbs all subsequent SIGINT/SIGTERM deliveries. The first signal
   still cancels the context; subsequent signals no longer terminate
   the process, giving the cleanup path time to flush metrics and
   telemetry before the process exits normally.

2. Add a cancellation short-circuit in the agent iteration loop: when
   ctx.Err() is non-nil after rt.Run returns, immediately aggregate
   metrics, finalize the agent span, and write metrics.json — before
   extraction and validation that would be pointless on a dead sandbox.
   This ensures the artifact upload step (if: always()) captures the
   partial usage data even if the process is subsequently killed.

Dollar cost (TotalCostUSD) remains zero on cancellation because it is only available from the terminal ResultEvent, which a killed run never emits. Token counts (input, output, cache_read, cache_creation) are captured via the deferred TokensEvent from PR #6908.

---

Closes #6936

Post-script verification

• Branch is not main/master (agent/6936-persist-cancel-metrics)
• Secret scan passed (gitleaks — 318e04c56f6457aa88b368b9731305d1f9659adb..HEAD)
• PR body secret scan: N/A (commit body path)

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Ended 10:48 PM UTC

Commit: a41e109 · View workflow run →

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 10:50 PM UTC · Completed 11:06 PM UTC

Commit: a41e109 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $4.66

[codecov]

Codecov Report

❌ Patch coverage is 81.81818% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
cmd/fullsend/main.go	81.81%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[fullsend-ai-review]

Risk Assessment: moderate (2/5)

Details

Re-review anchoring: Tier 1 metadata unchanged from prior run (small-medium diff, no protected/security/CI/dependency changes, strong 0.67 test ratio, bot author) at ~1.25; Tier 2 confirms run.go/run_test.go/telemetry_run_test.go remain hot, multi-author, fix/revert-prone files, offset by very recent code age, at ~2.5; Tier 3 unchanged in character (no risk labels, small scoped follow-up) at ~2.25. Composite 1.825 rounds to 2, matching the prior moderate score with no signal providing a specific reason to diverge.

Previous run

Risk Assessment: moderate (2/5)

Details

Tier 1 metadata is low risk (small-medium diff, no protected/security paths, no CI/dependency changes, strong 0.60 test ratio, bot author), but Tier 2 git history shows run.go and run_test.go are hot, multi-author, fix/revert-prone files with coupling to files not touched in this PR; Tier 3 is moderate given 5/6 acceptance criteria addressed and no risk labels. The weighted composite (Tier1=1, Tier2=3, Tier3=2) rounds to 2 (moderate).

Previous run (2)

Risk Assessment: moderate (2/5)

Details

Re-review confirms prior score: Tier 1 signals are unchanged (1.25) — bot author, 0 protected paths, no dependency/CI changes, 60% test ratio; internal/cli/run.go remains a high-churn hotspot but this is characteristic of the file (Tier 2: 2.8); issue scope matches well with most acceptance criteria covered and no unresolved discussions, rollback safety slightly elevated due to signal-handling changes (Tier 3: 2.0). Composite 1.87 rounds to 2.

Previous run (3)

Risk Assessment: moderate (2/5)

Details

Re-review confirms prior score: Tier 1 signals are unchanged (1.25) — bot author, 0 protected paths, no dependency/CI changes, 60% test ratio; internal/cli/run.go remains a high-churn hotspot but this is characteristic of the file (Tier 2: 2.8); issue scope matches well with clear acceptance criteria covered and no unresolved discussions, rollback safety slightly elevated due to signal-handling changes (Tier 3: 2.0). Composite 1.87 rounds to 2.

Previous run (4)

Risk Assessment: moderate (2/5)

Details

Re-review confirms prior score: small bug fix (4 files, 206 lines, 50% test ratio) by bot author with no protected paths or dependency/CI changes (Tier 1: 1.25); internal/cli/run.go remains an extreme churn hotspot but this is characteristic of the file (Tier 2: 3.0); issue scope matches with clear acceptance criteria mostly covered, though rollback safety is elevated since the change modifies process signal handling (Tier 3: 2.0). Composite 1.93 rounds to 2.

Previous run (5)

Risk Assessment: moderate (2/5)

Details

Small bug fix (4 files, 206 lines, 50% test ratio) by bot author with no protected paths, no dependency or CI changes — Tier 1 is low (1.25). Tier 2 is elevated (3.25) due to high churn on internal/cli/run.go (100 commits in 30d, 18 authors). Tier 3 moderate (2.75) — issue scope matches but change modifies process lifecycle without a feature flag. Composite 2.15 rounds to 2.

[fullsend-ai-review]

Looks good to me

Previous run

Review

Findings

Medium

• [missing-test] internal/cli/run.go:2268 — The cancellation short-circuit in runAgent (ctx.Err() check after rt.Run returns, then finalizeAgentSpan + writeMetricsJSON + return) is the production path that actually persists partial metrics before extraction/validation — the core fix for Persist cost and token telemetry when Review agents are cancelled by GitHub Actions #6936. No test executes this branch. The new tests only cover already-existing helpers (aggregateRunMetrics, writeMetricsJSON) and parseClaudeStream/progressParser; deleting the short-circuit would still pass them. ClaudeRuntime.Run returns (exitCode, nil) when ProcessState is non-nil (including a killed openshell exec), so the pre-existing if runErr != nil metrics-write branch does not run for this case — this new branch is load-bearing and untested.
  Remediation: Add a test that drives the post-Run cancel path with non-zero RunMetrics (fake/stub Runtime or an extracted helper): cancelled ctx, rt.Run returning a nil error, then assert metrics.json was written with the partial token counts, the returned error wraps context.Canceled, and extraction/validation is not attempted.

Low

• [test-adequacy] cmd/fullsend/main_test.go:74 — TestSignalContext_CleanupStopsForwarding claims to verify that cleanup() disables signal forwarding, but it never delivers a signal after cleanup() — it only asserts that calling cleanup() itself doesn't cancel the context. It does not catch a registration leak (the underlying goroutine stays blocked on <-sigCh, which is never closed).
  Remediation: Tighten the test to assert the documented Stop behavior without killing the test process — e.g. install a second Notify on a probe channel, send SIGINT after cleanup(), and require the probe (not signalContext's own context) to receive it.

---

Other dimensions reviewed with no findings above the reporting threshold: style-conventions (the previously-flagged lastExitCode/StepWarn ordering nit at internal/cli/run.go remains present but is cosmetic and non-blocking), intent-coherence (scope traces cleanly to issue #6936; the dollar-cost documentation gap flagged by an earlier review iteration is adequately covered by the existing "No pricing-table fallback" section of docs/guides/infrastructure/distributed-tracing.md plus the new code comment/PR body), docs-currency (no stale docs found), risk-assessment (moderate).

---

Next steps:

• /fs-fix — agent addresses review findings automatically
• /fs-fix <your instruction> — agent fixes with your specific guidance
• Push commits directly — review re-runs automatically on push
• /fs-fix-stop — disable automatic fix runs for this PR

Previous run (2)

Review

Findings

Low

• [code-organization] internal/cli/run.go:2135 — In the new cancellation block, lastExitCode = exitCode is assigned on line 2135 before printer.StepWarn(...) on line 2136, while the adjacent runErr != nil block reverses this order (printer on line 2146, assignment on line 2150). The ordering difference is cosmetic — both blocks terminate with return immediately after, so lastExitCode is never read after assignment in either path, and the two statements have no data dependency on each other.
  Remediation: Reorder to match the adjacent block for visual consistency, or leave as-is since the difference is functionally immaterial.

Previous run (3)

Review

Findings

Medium

• [scope-gap] internal/cli/run.go — Issue Persist cost and token telemetry when Review agents are cancelled by GitHub Actions #6936 acceptance criterion 5 requires documenting the remaining dollar-cost limitation or adding a cost-reconciliation mechanism. The PR does neither — TotalCostUSD is zero on cancellation, acknowledged in test comments but not in user-facing documentation, and no follow-up issue was opened to track this gap.
  Remediation: Either add a brief doc note explaining that TotalCostUSD is not populated when a run is cancelled by GitHub Actions, or open a follow-up issue scoped to cost reconciliation and link it from the PR body so the gap is explicitly deferred rather than silently dropped.

Low

• [error-handling-idiom] internal/cli/run.go:2123 — ctx.Err() is called twice in succession — once as the if-guard (line 2123) and once to assign cancelErr (line 2124). Since context cancellation is monotonic, this is safe but mildly redundant. An inline assignment if cancelErr := ctx.Err(); cancelErr != nil would be marginally cleaner.
  Remediation: Collapse lines 2123–2124 into: if cancelErr := ctx.Err(); cancelErr != nil {

---

Next steps:

• /fs-fix — agent addresses review findings automatically
• /fs-fix <your instruction> — agent fixes with your specific guidance
• Push commits directly — review re-runs automatically on push
• /fs-fix-stop — disable automatic fix runs for this PR

Previous run (4)

Looks good to me

Previous run (5)

Review

Findings

Low

• [error-message-convention] internal/cli/run.go:2132 — The cancellation-path StepWarn uses a novel "Cancellation: failed to write metrics.json" prefix. The structurally identical failure path 14 lines below uses "Failed to write metrics.json: " — the established pattern throughout the file is a capitalized verb phrase without a topic prefix (e.g., "Failed to ...", "Could not ...").
  Remediation: Change to "Failed to write metrics.json: " to match the existing pattern used on the runErr path below and at line 2319.

---

Next steps:

• /fs-fix — agent addresses review findings automatically
• /fs-fix <your instruction> — agent fixes with your specific guidance
• Push commits directly — review re-runs automatically on push
• /fs-fix-stop — disable automatic fix runs for this PR

[fullsend-ai-coder]

🤖 Finished Fix · ✅ Success · Started 11:08 PM UTC · Completed 11:15 PM UTC

Commit: a41e109 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $0.93

[fullsend-ai-coder]

🔧 Fix agent — iteration 1 (bot-triggered)

Aligned the cancellation-path error message with the existing convention used elsewhere in runAgent. Single-line change from novel "Cancellation: failed to ..." prefix to the established "Failed to ..." pattern.

Fixed (1):

1. error-message-convention: cancellation-path StepWarn uses novel prefix (internal/cli/run.go): Changed "Cancellation: failed to write metrics.json: " to "Failed to write metrics.json: " to match the established error-message pattern used on the runErr path (line 2149) and at line 2340

Tests: passed

Next steps:

• /fs-review — request a re-review of the changes
• /fs-fix <your instruction> — run another fix pass with specific guidance
• Push commits directly — review re-runs automatically on push
  _{Updated by fullsend fix agent}

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 11:18 PM UTC · Completed 11:31 PM UTC

Commit: 37bc3fc · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $3.62

Superseded by updated review

[rh-hemartin]

/fs-fix increase the code coverage

[fullsend-ai-coder]

🤖 Finished Fix · ✅ Success · Started 3:06 PM UTC · Completed 3:18 PM UTC

Commit: 37bc3fc · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $2.23

[fullsend-ai-coder]

🔧 Fix agent — iteration 2 (human-triggered)

Increased code coverage by extracting signalContext() from main() (3 new tests, 100% function coverage) and adding 2 multi-iteration cancellation scenario tests for aggregateRunMetrics and writeMetricsJSON round-trip. cmd/fullsend coverage rose from 0% to 52.9%.

Fixed (2):

1. Increase code coverage for signal handling (cmd/fullsend/main.go): Extracted signalContext() from main() to make the signal-absorption logic testable. Added three tests: first-signal-cancels-context, subsequent-signals-absorbed, and cleanup-stops-forwarding. Coverage for cmd/fullsend rose from 0% to 52.9% (signalContext function: 100%).
2. Increase code coverage for cancellation metrics aggregation (internal/cli/run_test.go): Added TestAggregateRunMetrics_MultiIterationCancel and TestWriteMetricsJSON_MultiIterationCancelRoundTrip to cover the multi-iteration cancellation scenario where one iteration completes normally and the second is cancelled. These tests exercise the full data path the cancellation short-circuit relies on.

Tests: passed

Decision points

• Extract signalContext as a named function vs test main() directly (alternatives: Test main() via TestMain with exec.Command, Extract signal setup into a testable signalContext() function; rationale: Extracting signalContext() provides direct, reliable unit tests without subprocess overhead. main() itself calls os.Exit and cli.Execute which require integration-level testing.)

---

Next steps:

• /fs-review — request a re-review of the changes
• /fs-fix <your instruction> — run another fix pass with specific guidance
• Push commits directly — review re-runs automatically on push
  _{Updated by fullsend fix agent}

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 3:20 PM UTC · Completed 3:37 PM UTC

Commit: d2e0857 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $4.91

Superseded by updated review

[fullsend-ai-coder]

🤖 Finished Fix · ✅ Success · Started 3:39 PM UTC · Completed 3:47 PM UTC

Commit: d2e0857 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $1.21

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 3:48 PM UTC · Completed 4:05 PM UTC

Commit: 4a67379 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $5.14

Superseded by updated review

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 3:42 PM UTC · Completed 4:02 PM UTC

Commit: a7038c0 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $4.94

Superseded by updated review

[fullsend-ai-coder]

🤖 Finished Fix · ✅ Success · Started 4:04 PM UTC · Completed 4:20 PM UTC

Commit: a7038c0 · View workflow run →

Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $1.91

[fullsend-ai-coder]

🔧 Fix agent — iteration 4 (bot-triggered)

Addressed both review findings on PR #6938. Extracted the untested cancellation short-circuit in runAgent into handleRunCancellation and added TestHandleRunCancellation exercising the cancelled-context, non-cancelled-context, and pre-existing-runErr cases, verifying metrics.json persistence, error wrapping of context.Canceled, span finalization, and that downstream extraction/validation is skipped. Tightened TestSignalContext_CleanupStopsForwarding to actually send a signal after cleanup() and verify via an independent probe channel that signal.Stop disabled forwarding, rather than asserting on a state that was true regardless of whether cleanup worked. This is iteration 4, so per strategy escalation guidance I moved from the prior iterations' approach (adding tests only for already-existing helpers like aggregateRunMetrics/writeMetricsJSON) to extracting and directly testing the actual load-bearing branch.

Fixed (2):

1. missing-test: the cancellation short-circuit in runAgent (ctx.Err() check after rt.Run returns, then finalizeAgentSpan + writeMetricsJSON + return) is untested load-bearing code for Persist cost and token telemetry when Review agents are cancelled by GitHub Actions #6936 (internal/cli/run.go): Extracted the short-circuit block verbatim into a new function handleRunCancellation (internal/cli/run.go) so it can be driven directly in a unit test instead of only through a full sandbox run. Added TestHandleRunCancellation in internal/cli/telemetry_run_test.go covering: a cancelled context with non-zero RunMetrics writes metrics.json with the partial token counts, the returned error wraps context.Canceled via errors.Is, the agent span is finalized (ended) with an error status, attachIterationContent is called with finish_reason="error" before any extraction/validation, and a pre-existing runErr from rt.Run is preserved as the wrapped cause. A companion case confirms a live (non-cancelled) context does not fire the short-circuit, does not write metrics.json, does not end the span, and does not attach content -- verifying downstream extraction/validation is reached instead.
2. test-adequacy: TestSignalContext_CleanupStopsForwarding never delivers a signal after cleanup(), so it only asserts that calling cleanup() itself doesn't cancel the context and cannot catch a registration leak (cmd/fullsend/main_test.go): Rewrote the test to call cleanup() first, then register an independent probe channel via signal.Notify(probe, syscall.SIGINT), send SIGINT to the process, and assert both that the probe receives it (proving the signal was actually delivered, so the test isn't vacuously passing) and that signalContext's own ctx is still not Done (proving signal.Stop actually disabled forwarding to its channel). This matches the reviewer's suggested remediation exactly.

Tests: passed

Strategy change: Iterations 1-3 added tests only for pre-existing helper functions (aggregateRunMetrics, writeMetricsJSON) and made small wording/ordering tweaks, which the reviewer correctly flagged as not covering the actual cancellation branch. On this iteration (4), I extracted the branch itself into a testable function (handleRunCancellation) and wrote a test that drives that exact code path, rather than adding more tests around its dependencies.

Decision points

• Chose to extract the cancellation short-circuit into a standalone function (handleRunCancellation) rather than stub/fake the full agentruntime.Runtime and drive runAgent end-to-end (alternatives: Fake/stub the agentruntime.Runtime interface and construct enough of runAgent's sandbox/openshell/gateway setup to reach rt.Run and exercise the branch in-place, Extract the short-circuit body into a small, directly testable function and call it from runAgent unchanged; rationale: runAgent resolves its Runtime deep inside sandbox provisioning (openshell exec, gateway checks, provider/profile orchestration) with no existing seam for injecting a fake Runtime that reaches rt.Run; building that scaffolding would be a large, fragile addition far beyond the finding's scope. The reviewer's own remediation text offered 'an extracted helper' as an equally acceptable alternative to a fake Runtime. The extraction is a pure move (no logic changes -- verified via diff) that preserves runAgent's control flow exactly while making the branch unit-testable, consistent with existing patterns in this file (e.g. finalizeAgentSpan, aggregateRunMetrics, writeMetricsJSON are all already extracted and unit-tested the same way).)

---

Next steps:

• /fs-review — request a re-review of the changes
• /fs-fix <your instruction> — run another fix pass with specific guidance
• Push commits directly — review re-runs automatically on push
  _{Updated by fullsend fix agent}

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 4:22 PM UTC · Completed 4:35 PM UTC

Commit: 78d6425 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $3.06

Superseded by updated review

[rh-hemartin]

This is important enough and it looks good to me, merging.

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 7:32 AM UTC · Completed 7:44 AM UTC

Commit: 78d6425 · View workflow run →

Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $1.67

[fullsend-ai-retro]

PR #6938 (fullsend-ai/fullsend, merged 2026-09-10) fixed issue #6936: a double-signal (SIGINT then SIGTERM) killing cancelled Review runs before telemetry/metrics.json could be written. The rework loop itself was clean — 4 fix iterations, each mapping 1:1 to a real prior review finding, no thrashing or reverted work, and iteration 4 shows good self-correction (the code agent escalated from testing only helpers to extracting and directly unit-testing the actual handleRunCancellation-equivalent branch after review flagged it as untested). A mid-PR runtime/model switch (opus → pi/sonnet) between the 09-08 and 09-09 review runs showed no apparent quality regression.

However, this run reproduces three failure patterns that already have open tracking issues, and I'm treating them as corroborating evidence rather than new proposals per the duplicate-check policy:

1. Review agent CI/Codecov blindness (tracked in Review agent should check PR CI status before finalizing verdict agents#833 and Review agent should check CI status before approving PRs #369, Review bot applies ready-for-merge label without checking CI status #1424, Defer re-review dispatch when required CI checks are failing #5755): Codecov commented a failing patch-coverage check (81.82%, marked ✗) at 09-02 22:58. The review agent ran twice afterward (09-02 23:06 and 23:18) and never mentioned it, flagging only an unrelated low-severity naming nit. Confirmed via source read: agents/review.md, harness/review.yaml, and all pr-review/code-review skills in fullsend-ai/agents contain no logic that reads GitHub Checks/Statuses or Codecov state — review is scoped purely to diff-level code evaluation.

2. Stale PR after passing bot review (tracked in Surface agent-approved PRs awaiting human review after N days #966, Add stale-approval nudge for agent-authored PRs awaiting merge #3046): the PR then sat idle ~5.5 days (09-02 23:18 → 09-08 15:04) until a human manually commented /fs-fix increase the code coverage. This is the same staleness pattern as Surface agent-approved PRs awaiting human review after N days #966/Add stale-approval nudge for agent-authored PRs awaiting merge #3046, but with the failing check specifically as the stall cause rather than idle-with-green-CI as in Add stale-approval nudge for agent-authored PRs awaiting merge #3046.

3. Coverage-met-but-wrong-lines-covered (tracked in Review agent test-adequacy check should verify tests exercise code changed by the PR, not just that tests exist #5330, Review agent should detect untested branch behavior in new code #1504, Review agent should flag untested new conditional branches in controller logic at medium severity #5148, Review agent should flag new helper functions lacking dedicated unit tests #2066, Review agent should flag new exported Go functions that lack test coverage #4046, Counter-evidence for review autonomy: agent missed actionable test coverage gap caught by human on operator controller PR #8228 #5149 and Correctness sub-agent should estimate aggregate coverage for new-code PRs and escalate when below merge-gate threshold agents#397, Admin web: organization hub — navigation, account context, and org detail #512, plus tooling-side fix: harden entrypoint output handling in fullsend action #230/Code agent should validate patch coverage meets repo threshold before opening PR #2810): the human-triggered fix iteration (09-08) raised coverage 0%→52.9% by testing pre-existing helpers, but left the actual new load-bearing cancellation-short-circuit branch untested. Only the next day's review run (09-09) caught this and forced extraction of the branch into a directly-testable function. This is the same root cause as Counter-evidence for review autonomy: agent missed actionable test coverage gap caught by human on operator controller PR #8228 #5149's prior incident (human caught what review missed on a controller conditional-branch PR), now recurring with the review agent itself eventually catching it one cycle later.

No new proposals filed — all identified gaps are already covered by open issues in fullsend-ai/agents and fullsend-ai/fullsend. Recommend the humans triaging those issues use this PR's timeline as a fresh, concrete data point (#6938).

[ascerra]

Follow-up PR #7208 addresses the remaining USD cost gap: when a run is cancelled before the result event, TotalCostUSD stayed zero. The follow-up adds a client-side pricing table (internal/cli/pricing.go) that estimates cost from captured token counts using published Anthropic per-MTok rates. The estimate flows into all four telemetry sinks (agent span, root span, metrics.json, status comment) and is a no-op on successful runs where the authoritative cost is already present.