Report real Grok token usage and list-price cost, from the CLI logs and OpenCodex alike - #3135
Report real Grok token usage and list-price cost, from the CLI logs and OpenCodex alike#3135olddonkey wants to merge 6 commits into
Conversation
|
🦞👀 Pull request received. I will update this pull request when review starts. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 09cf7edb0f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if provider == .grok, | ||
| self.tokenSnapshotPublicationForCurrentProviderConfig(for: provider) == nil | ||
| { |
There was a problem hiding this comment.
When a Grok billing probe fails after a fallback scan has already published local tokens, this condition is false and control enters the tokenCostRequiresProviderSnapshot branch below, where clearTokenSnapshot deletes the only readable usage data. Consequently, a second offline/auth failure removes Grok's local cost row instead of preserving or refreshing it, contradicting the fallback behavior this block is intended to provide.
Useful? React with 👍 / 👎.
| guard let resolvedPricing = CostUsagePricing.resolvedCodexPricing( | ||
| model: model, | ||
| pricingDate: pricingDate, | ||
| modelsDevCatalog: pricing.modelsDevCatalog, | ||
| modelsDevCacheRoot: pricing.modelsDevCacheRoot) |
There was a problem hiding this comment.
On a fresh installation or Grok-only configuration with no models.dev cache, this lookup fails for every xAI SKU because there is no bundled xAI pricing. The repository's only ModelsDevPricingPipeline.refreshIfNeeded call is in CostUsageFetcher.refreshPricingIfAllowed, which explicitly permits only Codex and Claude, so the Grok path never obtains pricing itself and known Grok models remain unpriced until an unrelated provider happens to populate the shared cache.
Useful? React with 👍 / 👎.
clawsweeper
Bot
added
merge-risk: 🚨 compatibility
|
Codex review: found issues before merge. Reviewed August 22, 2026, 1:42 PM ET / 17:42 UTC. ClawSweeper reviewWhat this changesThe PR derives Grok token and list-price estimates from completed CLI logs, then merges OAuth-configured OpenCodex xAI usage into the Grok spend row. Merge readinessKeep open: the PR has credible real-world proof, but it still reclassifies all historical xAI records from the current OpenCodex auth setting and leaves a fresh price catalog unpublished until a later scan. Priority: P2 Review scores
Verification
Live VerificationCommand: Result: FAIL (failed) — execution before step 1 Assertions:
How this fits togetherCodexBar turns local Grok and OpenCodex logs into provider cost snapshots used by the menu-bar usage and spend views. This change affects attribution, daily aggregation, pricing, and the final Grok dashboard row. flowchart LR
A[Grok CLI logs] --> B[Local usage scan]
C[OpenCodex usage log] --> D[Current auth config check]
D --> E[Subscription attribution]
B --> F[Provider cost snapshot]
E --> F
G[Pricing catalog] --> F
F --> H[Usage and Spend views]
Decision needed
Why: The usage log has no credential-era field, so code cannot distinguish prior API-key records from OAuth records after a configuration change. Before merge
Findings
Agent review detailsSecurityNone. Review metrics
Merge-risk optionsMaintainer options:
Technical reviewBest possible solution: Preserve auth provenance with each OpenCodex usage record before subscription fan-out, or keep records token-only when that provenance is unavailable; then republish or invalidate the first Grok snapshot after catalog refresh. Do we have a high-confidence way to reproduce the issue? Yes for the source-level failures: retain an xAI API-key record, change the current config to OAuth, and reload the dashboard; separately start with no price catalog and observe that its completed refresh does not republish the snapshot. Is this the best way to solve the issue? No: current-config routing is not a durable authorization signal for historical records, and the detached catalog refresh needs publication invalidation or a completion-driven rescan. Full review comments:
Overall correctness: patch is incorrect AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 27c7f334e3c4. LabelsLabel changes:
Label justifications:
EvidenceWhat I checked:
Likely related people:
Rank-up movesOptional improvements that raise the rating; they are not merge blockers.
Rating scale
Overall follows the weaker of proof and patch quality. Workflow
HistoryReview history (4 earlier review cycles)
|
…on logs two ways and expensive in a third. Wrong tokens: the scanner summed `contextTokensUsed` from `signals.json`, which is the session's ENDING context-window occupancy, not what it consumed. On a real machine that reported 653K where actual consumption was 48.0M. Read the sibling `updates.jsonl` instead, where every `turn_completed` event carries the turn's real usage, and bucket by the per-line timestamp so a session crossing local midnight lands in both days. No cost: `toCostUsageTokenSnapshot` hardcoded nil dollars, and nothing could have priced a Grok model anyway because `codexModelsDevProviderIDs` had no `xai`. Add it, and resolve `grok-<version>-build` onto its base catalog model — the `-build` suffix is an artifact of the responses-API surface, not a separate SKU. `grok-build-0.1` is a real model and is never rewritten. Cost is the public xAI card via models.dev, provenance `.listPriceEstimate`, so Grok stays comparable with Claude and Codex. grok's own `costUsdTicks` is deliberately not used for display. A turn's `usage` is the aggregate of `modelCalls` API calls, so tiering on the turn total would push nearly every multi-call turn into the >=200k bracket. Price on the per-call average instead, in closed form over the two synthetic call groups. This under-tiers slightly when context grows within a turn (measured ~4% below the vendor's own accounting on a 27-turn sample, against ~+10% for aggregate tiering); the trade is documented at the call site and pinned by a test. Main-actor cost: the scan ran synchronously inside `@MainActor UsageStore` on every menu-card build, refresh and dashboard load. It now reads the projection the async probe already produced, and the remaining fallback scans on a detached task with one scan in flight at a time. The probe projects the maximum window and consumers narrow it, so `costUsageHistoryDays` and the dashboard's 365-day request are both honoured. Hardening: `modelCalls` comes from a file, so it is validated before it can size any work; parsing is cached per (path, size, mtime) with entries evicted when a file is no longer visited; the cache lock is not held across file reads. Note for upgraders: adding `xai` to `codexModelsDevProviderIDs` changes the Codex pricing-cache key, so the first launch after this re-prices existing Codex history once. Same one-time cost as when kimi and deepseek were added.
OpenCodex sends inference straight to api.x.ai using the Grok account's OAuth credentials, so it burns the same SuperGrok subscription the Grok provider reports on. It only spawns the `grok` binary to refresh tokens, so those requests never reach ~/.grok/sessions and the local session scanner cannot see them — 1,435 requests on one real machine that CodexBar attributed to nothing. Route the `xai` provider prefix to the Grok subscription, the same way `openai` already routes to Codex. Like that mapping, this routes on the prefix and does not distinguish OAuth from API-key traffic. The `-build` suffix seen in the data is a responses-API protocol artifact, not a separate billing pool, so traffic is not split by it. Routing alone would have produced tokens with no dollars. The aggregator priced the bare `entry.model`, and a name without a route prefix is resolved against the `openai` provider — which is why `gpt-5.6-sol` prices today and `grok-4.6` resolved to `openai/grok-4.6` and missed. Qualify an unprefixed model with its provider before pricing. Codex rows are unaffected (the qualified name resolves to the same target), and providers outside the supported set keep returning nil.
Grok resolved list prices straight out of the cached models.dev catalog, but nothing in its path ever fetched that catalog. The only fetch trigger is CostUsageFetcher.refreshPricingIfAllowed, which is gated to Codex and Claude — and Grok never reaches it at all, because its snapshot comes from the provider probe rather than the shared token-cost pipeline. On a machine where Codex or Claude is also enabled the cache is already there, so this is invisible. Enable only Grok and the file never appears: every price lookup returns nil and the Cost row shows tokens with no money, permanently. Request ModelsDevPricingPipeline.refreshIfNeeded from the Grok scan paths. It is safe to call repeatedly — it returns immediately unless the cache is stale and serialises through its own coordinator — and it is detached rather than awaited, matching how the Codex and Claude paths already treat it: pricing availability must never delay or fail a local scan, and the next refresh fills in the value. `summarize` stays synchronous and side-effect free; the refresh lives in a wrapper so the parse-cache behaviour and existing tests are untouched. Reported as P2 by the automated review on the pull request.
… proof The opt-in live proof scanned real sessions but printed tokens only, which cannot evidence the half of this change that is about money. It now also reports today's and the window's list-price cost, the provenance, the window actually used, and how many days carried a price versus tokens — so an all-unpriced result is visible in the output instead of reading as zero. Still skipped unless CODEXBAR_LIVE_GROK_CATALOG_PROOF=1.
08360b5 to
e3cd3b9
Compare
olddonkey
changed the title
clawsweeper
Bot
added
merge-risk: 🚨 auth-provider
The regression guard drove a single failing refresh after a local publication existed. The defect it covers is specifically about the *second* failure: the first one publishes through the fallback scan, and only the next one arrives with a publication already in place — which is what used to hit the generic clear branch. Drive the failure twice and assert the row and the scan count both hold.
|
Both automated findings are addressed, plus the review's other checklist items. The inline comments were left against P1 — Preserve the Grok fallback on repeated probe failuresFixed in if provider == .grok {
if self.tokenSnapshotPublicationForCurrentProviderConfig(for: provider) == nil {
Task { @MainActor [weak self] in
await self?.scanAndPublishGrokLocalTokenSnapshot(...)
}
}
} else if Self.tokenCostRequiresProviderSnapshot(provider) {
self.clearTokenSnapshot(for: provider)
}Regression coverage is in P2 — Refresh pricing before scanning Grok sessionsCorrect, and thank you — this was a genuine gap and not one the local tests would have surfaced. Fixed in Note the inline comment still points at Coverage: Real-session evidence
The same corpus on Those figures were cross-checked against an independent reimplementation of the pricing formula over the same logs; the two agree to the cent. Merge risk / branch stateRebased onto current One thing deliberately left undone: no |
clawsweeper
Bot
added
rating: 🧂 unranked krab
Routing every OpenCodex `xai` record to the Grok subscription is right for the case that motivated it — traffic authenticated with the user's Grok account, which is what makes it burn the SuperGrok quota. It is wrong for anyone using an xAI API key: their pay-as-you-go developer-platform spend gets folded into the subscription row, silently inflating it. CodexBar models that platform as its own xAI provider precisely to keep the two apart. The usage log carries no per-record credential evidence, so the decision has to come from the OpenCodex provider config, which records `authMode` per provider. Read it, and attribute to Grok only when that mode is OAuth; anything else is token-only spend that belongs to no tracked subscription. Fail closed: a missing or malformed config, no `xai` entry, or an absent `authMode` all count as no OAuth evidence and keep the records off the Grok row. The dispatcher stays a pure function — the set of OAuth-backed provider ids is threaded in from the caller rather than read at the routing site — and the gate applies only to `xai`, leaving the other routes exactly as they were. Also records why the Grok pricing refresh stays fire-and-forget: the parse cache holds parsed turns rather than prices, so the next scan reprices against the refreshed catalog, and plumbing completion back to republish was judged disproportionate to a delay Codex and Claude already share. Raised as P1 by the automated review; the owner chose verifiable attribution over prefix-only routing.
clawsweeper
Bot
added
rating: 🦐 gold shrimp
|
Both new findings addressed at P1 — Do not map every xAI log record to the Grok subscriptionAgreed, and taken as specified rather than argued down. Routing on the prefix alone is right for the case that motivated this — traffic authenticated with the user's Grok account, which is what makes it consume SuperGrok quota — but it silently folds an API-key user's pay-as-you-go xAI spend into the subscription row. CodexBar already models the developer platform as its own The usage log carries no per-record credential evidence; I checked every field emitted for So attribution now requires positive OAuth evidence:
Coverage:
P2 — Republish the Grok snapshot after a missing catalog refreshesI looked at this closely and am deliberately not adding a republish path. Reasoning, so you can overrule it if you disagree: The refresh is fire-and-forget, so the scan that requests it returns whatever the cache currently holds — that part is accurate. But the parse cache stores parsed turns, not prices, so aggregation and pricing re-run on every The alternative — plumbing a completion signal back across the actor boundary into the Happy to build it if you would rather have it. EvidenceThe attribution itself only becomes visible in the app: The two halves stay distinguishable in the UI, which makes the attribution legible rather than something you have to take on trust: the CLI goes through the responses API so its SKU is
Independently recomputing the same corpus agrees to the cent on both halves: 54,121,501 tokens / $50.52 for the CLI logs, and $166.25 across 1,520 OpenCodex The negative direction — API-key traffic staying off the Grok row — is covered by tests rather than a screenshot, since demonstrating it live would mean rewriting the machine's OpenCodex config. StateFull suite on 
|
clawsweeper
Bot
added
proof: 📸 screenshot
1 participant
Rebased onto current
main(27c7f334e) and extended to cover both places Grok usage comes from, plus the two findings from the automated review. Four commits, kept separate so each can be read on its own.Problem
#3085 turned on
supportsTokenCostfor Grok, which lit up the provider "Cost" row. The row it lit up was wrong in two ways, and the way it was computed was expensive in a third.Wrong tokens.
GrokLocalSessionScannersummedtotalTokensBeforeCompaction + contextTokensUsedfromsignals.json.contextTokensUsedis the session's ending context-window occupancy — it sits next tocontextWindowTokens: 500000andcontextWindowUsage: 78— not what the session consumed. On a real machine it reported 653K where actual consumption was 52.7M.No cost, ever.
toCostUsageTokenSnapshot(historyDays:)hardcoded nil dollars, so the money half rendered—permanently. Nothing could have priced a Grok model anyway:CostUsagePricing.codexModelsDevProviderIDshad noxai, socodexModelsDevPricingTargetsreturned[]for every grok id.Main-actor cost.
UsageStore.tokenSnapshot(fromProviderSnapshot:provider:historyDays:)called the scanner synchronously, andUsageStoreis@MainActor. Callers include every menu-card build, every provider refresh, and the spend dashboard. That was survivable while the scan read a few KB ofsignals.json; it would not have been once the source becameupdates.jsonl(29 MB on a real machine, individual files up to 6 MB, growing without bound).What the commits do
1.
fix(grok): report real token usage and list-price cost from CLI session logsReads the sibling
updates.jsonl, where every completed turn appends aturn_completedevent carrying that turn's actual usage. Matching is onparams.update.sessionUpdate, not onmethod, because the CLI emits bothsession/updateand_x.ai/session/update. Bucketing uses the per-linetimestamprather than file mtime, so a session crossing local midnight lands in both days.Prices it with the public xAI card: adds
xaitocodexModelsDevProviderIDsand resolvesgrok-<version>-buildonto its base catalog model — the-buildsuffix is an artifact of the responses-API surface, not a separate SKU.grok-build-0.1is a real, separately priced model and is never rewritten; an exact catalog entry always wins over the normalized one. Cost carries.listPriceEstimateprovenance so Grok stays comparable with Claude and Codex. grok's owncostUsdTicksis deliberately not used for display.A turn's
usageis the aggregate ofmodelCallsAPI calls (observed 1–28, aggregate inputs up to 5.4M), so tiering on the turn total would push nearly every multi-call turn into the ≥200k bracket. Cost is computed on the per-call average instead, in closed form over the at-most-two synthetic call groups — O(1), not a loop. This under-tiers slightly because context grows within a turn; measured against the vendor's own accounting on a 27-turn sample it lands about 4% low, versus roughly +10% for per-turn aggregate tiering. The trade is documented at the call site and pinned by a test so it does not get "fixed" later. Reported token totals are always the raw aggregates — the split feeds only the pricing math, and the partition is lossless.Keeps the scan off the main actor: the
.grokprojection consumes the snapshot the async probe already produced, and the remaining fallback paths scan on a detached utility task with a single scan in flight. The probe projects the maximum window and consumers narrow it through a newCostUsageTokenSnapshot.narrowed(toHistoryDays:calendar:), so bothcostUsageHistoryDaysand the dashboard's 365-day request get the window they asked for, bucketed with the configured calendar.Hardening:
modelCallscomes from a file, so it is validated before it can size any work; parsing is cached per(path, size, mtime)with entries evicted once a file is no longer visited; the cache lock is not held across file reads or JSON parsing; the scan checks for cancellation.2.
feat(grok): count OpenCodex xAI traffic toward the Grok spend rowOpenCodex sends inference straight to
api.x.aiwith the Grok account's OAuth credentials, so it burns the same subscription; it only spawns thegrokbinary to refresh tokens, so those requests never reach~/.grok/sessions. On one real machine that is 1,435 requests attributed to nothing.Routes the
xaiprovider prefix to the Grok subscription, the same wayopenaialready routes to Codex — and, like that mapping, on the prefix rather than distinguishing OAuth from API-key traffic. The-buildsuffix in the data is a protocol artifact, not a separate billing pool, so traffic is not split by it.Routing alone would have produced tokens with no dollars: the aggregator priced the bare
entry.model, and a name with no route prefix resolves against theopenaiprovider — which is whygpt-5.6-solprices today andgrok-4.6resolved toopenai/grok-4.6and missed. Unprefixed models are now qualified with their provider before pricing. Codex rows are unaffected (the qualified name resolves to the same target) and providers outside the supported set keep returning nil.3.
fix(grok): fetch the models.dev catalog on Grok-only installs— addresses P2 from the automated reviewGrok resolved prices out of the cached models.dev catalog, but nothing in its path ever fetched it: the only fetch trigger is
CostUsageFetcher.refreshPricingIfAllowed, gated to Codex and Claude, and Grok never reaches it because its snapshot comes from the probe. With Codex or Claude also enabled the cache is already there, so this is invisible — enable only Grok and the file never appears and the Cost row shows tokens with no money, permanently.The Grok scan paths now request
ModelsDevPricingPipeline.refreshIfNeeded. It is safe to call repeatedly (returns immediately unless the cache is stale, and serialises through its own coordinator) and is detached rather than awaited, matching how the Codex and Claude paths already treat it: pricing availability must never delay or fail a local scan.4.
test: show cost, provenance and priced-day coverage in the gated Grok proofThe opt-in live proof printed tokens only, which cannot evidence the half of this change that is about money.
On P1 from the automated review
The failed-probe fall-through that cleared an existing Grok publication was found and fixed before the review arrived; it is part of commit 1, with a regression test that publishes a snapshot, forces a refresh failure, and asserts the publication survives and no redundant rescan runs.
Real-session evidence
CODEXBAR_LIVE_GROK_CATALOG_PROOF=1 swift test --filter GrokXAISpendCatalogTests, against real local Grok CLI sessions, through the shipped code path:history_days=365shows the requested window is honoured (it was pinned to 30 before).priced_days == token_daysshows no day was silently left unpriced. The same corpus onmainreports 653K tokens and no cost at all.Those figures were cross-checked against an independent reimplementation of the pricing formula over the same logs, which agrees to the cent: 54,121,501 tokens / $50.52. (The totals grow between runs because the machine keeps using the Grok CLI; the app, the gated proof and the independent recomputation were re-run together and still agree exactly.)
Note for upgraders
Adding
xaitocodexModelsDevProviderIDschanges the Codex pricing-cache key, so the first launch after this re-prices existing Codex history once. The previous parser hash is registered incompatiblePredecessorParserHashes, so the store itself is adopted rather than rebuilt from the JSONL corpus.Testing
-buildnormalization order and bare-model rejection, malformed lines,signals.jsonmetadata-only fallback, cache decode counts and eviction, the unpriced-over-threshold path, the absurd-modelCallsguard, window narrowing, catalog-refresh requests, and the zero-additional-decode path when a snapshot is supplied. All fixtures use a temporaryGROK_HOMEand injected models.dev catalogs — no real~/.grok, no network, no Keychain.make test).swiftformat --lintclean (0/1984) andswiftlint --strictclean.No changelog entry: 0.54.1 was finalized and there is no open Unreleased section, so I have left that to the maintainer.