Skip to content

feat(server): measure provider turn token usage - #9132

Merged
t3dotgg merged 14 commits into
mainfrom
t3code/posthog-token-efficiency-data
Sep 4, 2026
Merged

feat(server): measure provider turn token usage#9132
t3dotgg merged 14 commits into
mainfrom
t3code/posthog-token-efficiency-data

Conversation

@t3dotgg

@t3dotgg t3dotgg commented Sep 1, 2026

Copy link
Copy Markdown
Member

T3 Code cannot compare token use across provider turns because each provider reports usage in a different format. Existing analytics record when a turn starts, but not its normalized result or token totals.

This adds a provider.turn.completed event with normalized main-agent input, cache, output, and reasoning token counts for Codex, Claude Code, and OpenCode. Cursor, Grok, and Antigravity report usage as unavailable until their token fields and scope are verified. The event marks usage as complete, partial, or unavailable and includes the turn metadata needed for valid comparisons. It uses bounded deduplication, respects the existing telemetry opt-out, and does not include prompts, responses, thread IDs, turn IDs, provider instance IDs, raw provider payloads, or child-agent output.

Takeover changes on top of the original work:

  • Merged main, which added context compaction and citations to ProviderService.
  • Codex turns after resume or rollback reported unavailable, and any counter reset marked the turn partial. Real event logs show Codex sends the newest response's usage as last next to the running total, so that is now the delta when no prior total exists or the total shrank. Every Codex turn reports counts.
  • Claude turns now report reasoningTokens from output_tokens_details.thinking_tokens, so Claude and Codex turns compare on the same fields.

Verification: 308 focused tests across the provider adapters, ProviderService, analytics, and contracts pass. Server and contracts typechecks, targeted lint, and formatting pass. Codex delta math was checked against 2,936 real thread/tokenUsage/updated notifications from local logs: within a turn the growth of total always equals last, and every decrease in total coincides with a fresh session where total equals last.

Original implementation by GPT-5.6 Sol through the Codex harness in T3 Code. Takeover audit and fixes by Claude Fable 5.1 through the Claude Code harness in T3 Code.


Note

Medium Risk
Changes turn-completion payloads and analytics timing across all major provider adapters; incorrect delta or association logic could misreport usage or duplicate/skip analytics events.

Overview
Adds normalized per-turn token usage on turn.completed and turn.aborted runtime events, then emits a single provider.turn.completed analytics event with terminal status, duration, model metadata, and main-agent token breakdown (input, cache read/write, output, reasoning, subagent flag).

Claude derives tokenUsage from result usage (including cache counters and thinking tokens) and sets hasSubagents when collaboration task_started events classify as agent work. Codex accumulates deltas from cumulative thread/tokenUsage/updated totals (using last when the baseline is missing or resets), clamps cache/reasoning subsets, ignores late updates for finished turns, and resets state on rollback. OpenCode sums owned step-finish parts per prompt, marks partial when ownership is unresolved, flags subagents via child sessions, and attaches usage on all terminal turn paths.

ProviderService correlates sendTurn requests with adapter turnIds before recording analytics (deferring completions that arrive early), deduplicates by provider-instance/thread/turn, flushes deferred completions on session stop and shutdown, and only counts model selection when it matches the routed instance. Tests and docs cover telemetry opt-out and the new usage/telemetry user doc link.

Reviewed by Cursor Bugbot for commit 2d3d95d. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Add provider.turn.completed analytics event with normalized token usage

  • Introduces a TurnTokenUsage schema in providerRuntime.ts and attaches an optional tokenUsage field to TurnCompletedPayload and TurnAbortedPayload.
  • Adds per-turn token accumulation to the Codex, OpenCode, and Claude adapters, each normalizing provider-specific counters into a main-agent usage record with complete, partial, or unavailable status.
  • ProviderService now tracks turn lifecycle across send responses and runtime events, emitting one deduplicated provider.turn.completed analytics record per provider-instance/thread/turn with terminal status, token usage, model metadata, and duration.
  • Session start, stop, and global shutdown now clear and finalize deferred turn analytics state.
  • Adds user-facing telemetry documentation and internal product-analytics docs for the new event.
  • Risk: OpenCodeAdapter turn.started payloads no longer include a variant-derived effort field; any consumer relying on that field will stop receiving it. sendTurn now wraps the adapter call in Effect.acquireUseRelease, which changes the cleanup path for pending analytics state.

Macroscope summarized 2d3d95d.

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XL 500-999 changed lines (additions + deletions). labels Sep 1, 2026
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 13.7 KiB 13.7 KiB −1 B (−0.0%) 15.1 KiB
Codex Thread snapshot wire 7.0 KiB 7.0 KiB −5 B (−0.1%) 7.3 KiB
Codex Live turn WebSocket wire 6.7 KiB 6.7 KiB +4 B (+0.1%) 7.8 KiB
Codex Live turn WebSocket decoded 58.5 KiB 58.5 KiB 0 B (0.0%) 66.4 KiB
Codex Live turn messages 10 10 0 (0.0%) 21
Claude Total thread wire 13.5 KiB 13.6 KiB +65 B (+0.5%) 15.1 KiB
Claude Thread snapshot wire 7.0 KiB 7.0 KiB +2 B (+0.0%) 7.3 KiB
Claude Live turn WebSocket wire 6.5 KiB 6.5 KiB +63 B (+0.9%) 7.8 KiB
Claude Live turn WebSocket decoded 57.8 KiB 57.9 KiB +88 B (+0.1%) 66.4 KiB
Claude Live turn messages 8 10 +2 (+25.0%) 21

Baseline: 2460649 · PR result: 2d3d95d · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 113.8 KiB
  • Claude decoded thread snapshot: 114.5 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

Comment thread apps/server/src/provider/Layers/ProviderService.ts Outdated
Comment thread apps/server/src/provider/Layers/OpenCodeAdapter.ts
Comment thread apps/server/src/provider/Layers/ClaudeAdapter.ts Outdated
@macroscopeapp

macroscopeapp Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR adds cross-provider token accounting and default-path product telemetry, with substantial lifecycle and concurrency state in production adapters and ProviderService. The new externally collected data and broad runtime changes require human review.

No code changes detected at 2d3d95d. Prior analysis still applies.

You can add or adjust custom eligibility rules. Learn more.

Comment thread apps/server/src/provider/Layers/OpenCodeAdapter.ts Outdated
Comment thread apps/server/src/provider/Layers/OpenCodeAdapter.ts
@t3dotgg

t3dotgg commented Sep 2, 2026

Copy link
Copy Markdown
Member Author

Note

🤖 GPT-5.6 Sol responding on behalf of Theo

@macroscope-app review this PR

Please run a fresh approvability assessment now that all review threads are resolved. The remaining unknown-owner behavior is intentional and is explained in its resolved thread.

@macroscopeapp

macroscopeapp Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Manual reviews triggered for commit e7d8b00:

All prior checks · these links stay valid even if you push more commits.

@macroscopeapp

macroscopeapp Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Fresh review and approvability assessment are in progress.

Comment thread apps/server/src/provider/Layers/OpenCodeAdapter.ts
Comment thread apps/server/src/provider/Layers/ProviderService.ts
@t3dotgg
t3dotgg force-pushed the t3code/posthog-token-efficiency-data branch from e7d8b00 to 12989d8 Compare September 2, 2026 00:39
Comment thread apps/server/src/provider/Layers/ProviderService.ts Outdated
Comment thread apps/server/src/provider/Layers/ProviderService.ts
Comment thread apps/server/src/provider/Layers/ProviderService.ts Outdated
Comment thread packages/contracts/src/providerRuntime.ts Outdated
@t3dotgg
t3dotgg force-pushed the t3code/posthog-token-efficiency-data branch from 12989d8 to 3c2b18c Compare September 2, 2026 00:56
Comment thread apps/server/src/provider/Layers/ProviderService.ts
Comment thread docs/user/telemetry.md Outdated
Comment thread apps/server/src/provider/Layers/ProviderService.ts Outdated
@github-actions github-actions Bot added size:XXL 1,000+ changed lines (additions + deletions). and removed size:XL 500-999 changed lines (additions + deletions). labels Sep 2, 2026
Comment thread apps/server/src/provider/Layers/ProviderService.ts
Resolves conflicts with the compaction command and citation changes in
ProviderService.
…ng tokens

Codex only counted a turn's usage when a prior cumulative total existed,
so the first turn after resume or rollback reported unavailable and any
counter reset marked the turn partial. Real logs show Codex sends the
newest response usage as `last` alongside `total`, so use `last` as the
delta whenever there is no prior total or the total shrank.

Claude results carry `output_tokens_details.thinking_tokens`. Report it
as reasoningTokens so Claude and Codex turns compare on the same fields.

Also lists Antigravity with the providers whose usage stays unavailable.
Comment thread packages/contracts/src/providerRuntime.ts
Comment thread apps/server/src/provider/Layers/OpenCodeAdapter.ts Outdated

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 23e23f1. Configure here.

Comment thread apps/server/src/provider/Layers/ProviderService.ts Outdated
A turn.started event used to claim the sole in-flight sendTurn's model,
effort, interaction mode, and start time. Claude can start and close a
synthetic turn for leftover agent output while sendTurn prepares the real
turn, so that synthetic turn took the user's request metadata and the
real turn lost it. Now only the adapter's sendTurn response links a
request to a turn. Completions that land earlier wait for that response,
which the existing deferred path already handles.

OpenCode now also marks hasSubagents when a child session is proven
related by ancestry lookup after a missed session.created event.
Codex accumulates cachedInputTokens and cacheCreationTokens separately
from inputTokens, so a drifting counter could report a cache subset
larger than the input total. Clamp both the same way reasoningTokens is
clamped to outputTokens.
Comment thread apps/server/src/provider/Layers/CodexAdapter.ts Outdated
Codex token totals are thread-wide. A late update for a finished turn
that arrived after the next turn started left the baseline stale, so the
next turn was charged for the finished turn's growth.
Comment thread apps/server/src/provider/Layers/ProviderService.ts
@t3dotgg
t3dotgg merged commit 1587f24 into main Sep 4, 2026
26 checks passed
@t3dotgg
t3dotgg deleted the t3code/posthog-token-efficiency-data branch September 4, 2026 14:54
Olivier-OG added a commit to Olivier-OG/t3code that referenced this pull request Sep 4, 2026
* upstream/main: (30 commits)
  perf(server): use one query for buffered provider events (pingdotgg#9706)
  perf(server): stop rebuilding terminal history per chunk (pingdotgg#9703)
  perf(marketing): serve website fonts locally (pingdotgg#9701)
  perf(web): defer diff workers until a code view opens (pingdotgg#9692)
  perf(server): avoid full patches for checkpoint summaries (pingdotgg#9694)
  perf(marketing): stop continuous homepage motion (pingdotgg#9697)
  feat(server): measure provider turn token usage (pingdotgg#9132)
  perf(web): defer composer draft serialization (pingdotgg#9695)
  perf(mobile): reuse diff rows during comment edits (pingdotgg#9693)
  perf(clients): avoid waiting to read cached relay tokens (pingdotgg#9691)
  perf(server): omit repeated OpenCode progress logs (pingdotgg#9689)
  perf(mobile): reuse chat feed rows during streaming (pingdotgg#9688)
  perf(server): stop retaining unused OpenCode tool history (pingdotgg#9684)
  perf(server): batch projector cursor writes (pingdotgg#9671)
  perf(server): cache and stream static web assets (pingdotgg#9669)
  perf(marketing): serve images at their display size (pingdotgg#9682)
  perf(mobile): skip unused legacy list work (pingdotgg#9679)
  perf(web): keep Markdown mounted during streaming (pingdotgg#9677)
  perf(mobile): bound diff syntax highlighting work (pingdotgg#9673)
  fix(web): show machine icons in the environment picker (pingdotgg#9668)
  ...
github-actions Bot added a commit to omarcresp/t3code-flake that referenced this pull request Sep 4, 2026
## What's Changed
* fix(desktop): quit immediately on a second shortcut press by @t3dotgg in pingdotgg/t3code#9657
* fix(server): update Claude Agent SDK to 0.3.260 by @t3dotgg in pingdotgg/t3code#9135
* perf(server): stop loading message bodies for thread summaries by @t3dotgg in pingdotgg/t3code#9662
* perf(web): speed up terminal snapshots by @t3dotgg in pingdotgg/t3code#9663
* fix(web): show machine icons in the environment picker by @t3dotgg in pingdotgg/t3code#9668
* perf(mobile): bound diff syntax highlighting work by @t3dotgg in pingdotgg/t3code#9673
* perf(web): keep Markdown mounted during streaming by @t3dotgg in pingdotgg/t3code#9677
* perf(mobile): skip unused legacy list work by @t3dotgg in pingdotgg/t3code#9679
* perf(marketing): serve images at their display size by @t3dotgg in pingdotgg/t3code#9682
* perf(server): cache and stream static web assets by @t3dotgg in pingdotgg/t3code#9669
* perf(server): batch projector cursor writes by @t3dotgg in pingdotgg/t3code#9671
* perf(server): stop retaining unused OpenCode tool history by @t3dotgg in pingdotgg/t3code#9684
* perf(mobile): reuse chat feed rows during streaming by @t3dotgg in pingdotgg/t3code#9688
* perf(server): omit repeated OpenCode progress logs by @t3dotgg in pingdotgg/t3code#9689
* perf(clients): avoid waiting to read cached relay tokens by @t3dotgg in pingdotgg/t3code#9691
* perf(mobile): reuse diff rows during comment edits by @t3dotgg in pingdotgg/t3code#9693
* perf(web): defer composer draft serialization by @t3dotgg in pingdotgg/t3code#9695
* feat(server): measure provider turn token usage by @t3dotgg in pingdotgg/t3code#9132
* perf(marketing): stop continuous homepage motion by @t3dotgg in pingdotgg/t3code#9697
* perf(server): avoid full patches for checkpoint summaries by @t3dotgg in pingdotgg/t3code#9694
* perf(web): defer diff workers until a code view opens by @t3dotgg in pingdotgg/t3code#9692
* perf(marketing): serve website fonts locally by @t3dotgg in pingdotgg/t3code#9701
* perf(server): stop rebuilding terminal history per chunk by @t3dotgg in pingdotgg/t3code#9703


**Full Changelog**: pingdotgg/t3code@v0.0.39-nightly.20260904.1278...v0.0.39-nightly.20260904.1279

Upstream release: https://github.com/pingdotgg/t3code/releases/tag/v0.0.39-nightly.20260904.1279
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL 1,000+ changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant