Skip to content

Show what each bot spends - #185

Merged
aivsomkar merged 3 commits into
mainfrom
feat/harness-1.5-cost-visibility
Aug 18, 2026
Merged

Show what each bot spends#185
aivsomkar merged 3 commits into
mainfrom
feat/harness-1.5-cost-visibility

Conversation

@aivsomkar

@aivsomkar aivsomkar commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

In plain language

Before: the engines reported tokens (and Claude a dollar cost) on every turn, and the app threw it away. It also turned out the per-driver token reports meant different things (a per-call delta on Claude, a running thread total on Codex), so summing them naively double-counted.

What changes in the app: a small chip in the chat header shows what the current task has spent — e.g. 12.4k tok · $0.06 (dollars only for Claude, the only engine that reports a price). Bot settings get a Usage card; App settings get a Usage tab ranking every bot by cost then tokens with a total — the "which of my bots is costing me money" answer. Cost is captioned honestly: on a subscription it's an equivalent, not a charge. Figures start from zero on upgrade.


In plain terms

An app that runs agents unattended should be able to say what that costs. Cost/token data was already on the wire from the drivers and the UI discarded it — but two things the plan assumed turned out to be wrong, and both shaped this:

  1. Only Claude reports a dollar cost. Codex, grok, ACP and Antigravity send cost: null. So this is token-first, dollars where known.
  2. thread.token-usage.updated means different things per driver — a per-API-call delta on Claude, the thread's running total on Codex, a per-step figure and a turn total on Antigravity. Summing it naively double-counts. The harness never had a consistent figure to keep.

Changes

  • Contract (additive): turn.completed.usage?: { input, output }this turn's total, the one figure the harness banks. Each driver computes it from what it already has: Claude result.usage (input + cache read + cache creation), Codex tokenUsage.last ?? total, Antigravity result.usage. The live indicator is unchanged and never summed. ProviderSnapshot.billing?: "metered" | "subscription" — Claude and Codex both deliberately strip API keys and run on the CLI's own login, so both report subscription; a Claude cost is an equivalent and the UI says so.
  • Harness: store.addTaskUsage() banks each settled turn onto TaskRecord.usage { input, output, costUsd, turns } (cost stays null until any turn reports one); flows to clients through the existing wireTask. Bot and fleet totals are client-side sums.
  • UI:
    • Chat header chip for the open task — 12.4k tok · $0.06 (cost only when known); hover for turns / in-out split / billing caption; click → bot settings.
    • Bot settings Usage card: this bot across all tasks, captioned "equivalent — on your subscription, not billed" / "billed to your API key" / "doesn't report a price; tokens are counted".
    • App settings Usage tab: bots ranked by cost then tokens, fleet total row — the "which of my bots is costing me money" answer.

Figures start from zero on upgrade (older turns weren't recorded per task). Out of scope: reset/period windows, per-model breakdown, charts.

Item 1.5 of docs/plans/agent-harness-upgrades-v2.md.

Test plan

  • Driver tests assert turn.completed.usage on the existing fakes: claude ({12, 5} incl. cache read), codex ({7, 3} from a thread total), antigravity ({105, 20} from result.usage)
  • server/store.test.ts (+4): banks and counts turns; cost null until reported then sums only reported; token-less turn still counted; unknown task ignored
  • src/lib/usage.test.ts (5): compact tokens, small dollars stay visible, chip text, cross-task sums, billing captions
  • pnpm typecheck clean; pnpm vitest run green (65 files, 536 passed)
  • Manual in the dev app: Claude turns show the chip with $; bot Usage card and app-settings Usage tab populate; /api/instances reports billing: subscription for claude and codex

🤖 Generated with Claude Code

Cost and token data was on the wire and the UI threw it away — but the
harness also never had a consistent number to keep: thread.token-usage
.updated means a per-call delta on Claude, a running thread total on
Codex, and a per-step figure plus a turn total on Antigravity, so summing
it double-counted. And only Claude reports a price at all.

- turn.completed gains usage {input, output}: THIS turn's total, from
  what each driver already has (Claude result.usage incl. cache reads,
  Codex tokenUsage.last ?? total, Antigravity result.usage). The live
  indicator is unchanged and never summed.
- ProviderSnapshot.billing: Claude and Codex both strip API keys and run
  on the CLI's own login, so both report "subscription" — a Claude cost
  is an equivalent, and the UI captions it that way.
- store.addTaskUsage banks each settled turn onto TaskRecord.usage
  {input, output, costUsd, turns}; flows to clients via wireTask.
- UI: header chip for the open task ("12.4k tok · $0.06", cost only
  when known); a Usage card in bot settings; a Usage tab in app
  settings ranking bots by cost then tokens with a fleet total.

Item 1.5 of docs/plans/agent-harness-upgrades-v2.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Aug 17, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@aivsomkar, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 44 minutes

Limit details: You’ve used all 3 included reviews currently available under your plan.

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 2ec2d584-f8ae-438a-afba-9b59301bb811

📥 Commits

Reviewing files that changed from the base of the PR and between ef784ac and 597b744.

📒 Files selected for processing (18)
  • server/contracts.ts
  • server/drivers/antigravity.test.ts
  • server/drivers/antigravity.ts
  • server/drivers/claude.test.ts
  • server/drivers/claude.ts
  • server/drivers/codex.test.ts
  • server/drivers/codex.ts
  • server/index.ts
  • server/store.test.ts
  • server/store.ts
  • server/testing/fake-claude-cli.ts
  • src/components/ChatView.tsx
  • src/components/SettingsModal.tsx
  • src/components/SettingsPanel.tsx
  • src/components/UsageSection.tsx
  • src/lib/usage.test.ts
  • src/lib/usage.ts
  • src/state/store.tsx

Comment @coderabbitai help to get the list of available commands.

@aivsomkar aivsomkar added enhancement New feature or request good first issue Good for newcomers labels Aug 17, 2026
aivsomkar and others added 2 commits August 18, 2026 00:59
main's #192 landed its own per-task usage tally (last token-usage event
per turn, no cost). Reconciled: TaskRecord.usage keeps the superset shape
{input, output, costUsd, turns}; one addTaskUsage (with main's NaN/negative
sanitizing); at turn.completed the driver's own per-turn figure
(turn.completed.usage) is authoritative and main's last-reported value is
the fallback for drivers that only stream the running indicator. Records
written before cost existed read costUsd as null.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
main gained #205 (token spend in the task picker, using the same
TaskRecord.usage), the store change stream, activity states, and typed
approvals. Reconciled: one addTaskUsage with the superset shape (costUsd
kept; emits a bot change like main's did), settle path banks the turn's
authoritative usage then moves the bot to idle via setActivity, the
Usage settings tab sits beside main's new Companion tab.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@aivsomkar
aivsomkar merged commit da7176e into main Aug 18, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request good first issue Good for newcomers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant