Skip to content

Add Google and xAI models.dev pricing lookup - #2377

Open
joeVenner wants to merge 9 commits into
steipete:mainfrom
joeVenner:codex/modelsdev-google-xai-models
Open

Add Google and xAI models.dev pricing lookup#2377
joeVenner wants to merge 9 commits into
steipete:mainfrom
joeVenner:codex/modelsdev-google-xai-models

Conversation

@joeVenner

@joeVenner joeVenner commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add an additive provider-aware models.dev lookup seam for Google/Gemini, Vertex AI, and xAI/Grok providers.
  • Map Gemini to google, Vertex AI to google-vertex with google fallback, and Grok to xai.
  • Wire those provider-scoped lookups into the Pi session runtime cost path and expose the supported providers through codexbar cost.

Proof

  • Previous-context check: PR Add GPT-5.6 Sol/Terra/Luna Codex pricing #2023 covered OpenAI GPT-5.6 pricing and Fix: Refresh pricing for unknown models #2018 covered unknown-model refresh behavior; neither mapped Google/xAI provider families through models.dev.
  • Absence check against current base returned no matches:
    git grep -n -E "gemini-3\.5-flash|gemini-3\.1-pro-preview|gemini-3\.1-flash-lite|grok-4\.5|grok-4\.3|grok-4\.20" origin/main -- Sources Tests docs
  • Source catalog check from https://models.dev/api.json showed:
    • google: gemini-3.5-flash / Gemini 3.5 Flash / input 1.5 / output 9 / cache_read 0.15 / context 1048576
    • google: gemini-3.1-pro-preview / Gemini 3.1 Pro Preview / input 2 / output 12 / cache_read 0.2 / context 1048576
    • google: gemini-3.1-flash-lite / Gemini 3.1 Flash Lite / input 0.25 / output 1.5 / cache_read 0.025 / context 1048576
    • xai: grok-4.5 / Grok 4.5 / input 2 / output 6 / cache_read 0.3 / context 500000
    • xai: grok-4.3 / Grok 4.3 / input 1.25 / output 2.5 / cache_read 0.2 / context 1000000
    • xai: grok-4.20-0309-reasoning / Grok 4.20 (Reasoning) / input 1.25 / output 2.5 / cache_read 0.2 / context 1000000
  • Runtime regression test writes cached models.dev fixture data with separate google, google-vertex, and xai rates, then verifies PiSessionCostScanner.loadDailyReport computes provider-scoped costs for Pi JSONL rows:
    • google / gemini-3.5-flash: 160 tokens, cost 0.0006015
    • google-vertex / gemini-3.1-pro-preview: 160 tokens, cost 0.0008171, proving Vertex uses the Vertex catalog before Google fallback
    • xai / grok-4.5: 160 tokens, cost 0.000503
  • Built CLI proof used an isolated temp home with a synthetic .pi/agent/sessions/2026-07-28T10-00-00-000Z_google-xai-proof.jsonl containing only assistant rows. No real account files, Keychain, or browser cookies were read. Command shape:
    HOME=[temp] CFFIXED_USER_HOME=[temp] XDG_CONFIG_HOME=[temp]/.config XDG_CACHE_HOME=[temp]/.cache .build/debug/CodexBarCLI cost --provider <provider> --format json --pretty --refresh
  • Built CLI output excerpts:
    { "provider" : "gemini", "source" : "local", "sessionTokens" : 160000, "sessionCostUSD" : 0.6015,
      "daily" : [{ "date" : "2026-07-28", "modelBreakdowns" : [{ "modelName" : "gemini-3.5-flash", "totalTokens" : 160000, "cost" : 0.6015 }] }] }
    { "provider" : "vertexai", "source" : "local", "sessionTokens" : 160000, "sessionCostUSD" : 0.802,
      "daily" : [{ "date" : "2026-07-28", "modelBreakdowns" : [{ "modelName" : "gemini-3.1-pro-preview", "totalTokens" : 160000, "cost" : 0.802 }] }] }
    { "provider" : "grok", "source" : "local", "sessionTokens" : 160000, "sessionCostUSD" : 0.503,
      "daily" : [{ "date" : "2026-07-28", "modelBreakdowns" : [{ "modelName" : "grok-4.5", "totalTokens" : 160000, "cost" : 0.503 }] }] }

Verification

  • CLANG_MODULE_CACHE_PATH=/private/tmp/codexbar-clang-cache swift test --disable-sandbox --scratch-path /Users/mosaab/Documents/Projects/CodexBar/.build --filter PiSessionCostScannerTests: 19 tests passed.
  • CLANG_MODULE_CACHE_PATH=/private/tmp/codexbar-clang-cache swift test --disable-sandbox --scratch-path /Users/mosaab/Documents/Projects/CodexBar/.build --filter 'provider lookup resolves current Google and xAI models': 1 test passed.
  • git diff --check: clean.
  • make check completed generated parser hash, package/docs/locales checks, SwiftFormat lint mode, and SwiftLint strict with 0 violations, then exited on the sandbox-only macOS plist cache write permission error after lint shutdown.

No live provider probes, browser-cookie imports, or Keychain reads were run.

@joeVenner

Copy link
Copy Markdown
Contributor Author

@clawsweeper review

@clawsweeper

clawsweeper Bot commented Jul 27, 2026

Copy link
Copy Markdown

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal priority bug or improvement with limited blast radius. labels Jul 27, 2026
@clawsweeper

clawsweeper Bot commented Jul 27, 2026

Copy link
Copy Markdown

Codex review: needs maintainer review before merge. Reviewed August 15, 2026, 12:31 PM ET / 16:31 UTC.

ClawSweeper review

What this changes

Adds models.dev-backed local Pi-session cost estimates and cost-CLI support for Gemini, Vertex AI, and Grok.

Merge readiness

⚠️ Ready for maintainer review - 1 item remains

Keep open: current main still lacks this provider-cost coverage, the prior gatekeeper blocker is addressed on the new head, and the PR has credible isolated runtime proof. A maintainer should audit the broad gatekeeper reclassification before merge.

Priority: P2
Reviewed head: 8dee931d9c8ab012aa7b4cb340d71337c76a7796

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) The feature has strong isolated runtime proof and focused cost-path tests; the broad guardrail-test refresh merits maintainer inspection.
Proof confidence 🦞 diamond lobster (5/6) Sufficient (terminal): The PR body shows after-fix built-CLI output for Gemini, Vertex AI, and Grok using isolated synthetic session files and no account credentials.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Verified Sufficient (terminal): The PR body shows after-fix built-CLI output for Gemini, Vertex AI, and Grok using isolated synthetic session files and no account credentials.
Evidence reviewed 5 items Current-main gap: Current main returns an empty Pi report for every provider except Codex and Claude, so the requested Google/xAI cost reports are not already implemented.
Provider-aware implementation: The PR maps Gemini to google, Vertex AI to google-vertex with a Google fallback, and Grok to xai before calculating Pi-session costs.
Focused regression coverage: The new scanner regression exercises Google, Vertex, and xAI JSONL rows with deliberately distinct catalog prices and verifies each resulting cost.
Findings None None.
Security None None.

How this fits together

CodexBar scans local Pi/OMP session JSONL, maps each provider and model to cached models.dev pricing, then emits token and cost snapshots for the menu bar and CLI. This PR expands that pipeline from Codex/Claude to Google, Vertex AI, and xAI sessions.

flowchart LR
A[Pi and OMP session JSONL] --> B[Provider and model mapping]
B --> C[models.dev pricing cache]
C --> D[Local cost scanner]
D --> E[Token cost snapshot]
E --> F[Cost CLI and dashboard]
Loading

Before merge

  • Resolve merge risk (P1) - The 662-line ProviderArchitectureGatekeeperTests rewrite reclassifies existing guardrail allowances; passing tests establish internal consistency but not that every new allowlist boundary is intentional.
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Production versus test delta production +180/-28, tests +583/-395 Most churn is regression and architecture-gate coverage; the gatekeeper update deserves a focused review alongside the cost-path additions.
Provider coverage 3 providers enabled Gemini, Vertex AI, and Grok gain local cost-reporting support through the same cached catalog path.

Merge-risk options

Maintainer options:

  1. Audit the gatekeeper refresh before merge (recommended)
    Confirm that the reclassified existing provider references remain checked by an equivalent precise allowance, then merge the provider-cost feature.

Technical review

Best possible solution:

Land the additive provider mappings after a maintainer confirms the gatekeeper refresh preserves the intended architecture checks.

Do we have a high-confidence way to reproduce the issue?

Not applicable as a bug report; the PR supplies an isolated after-fix CLI run using synthetic Pi sessions for all three provider paths.

Is this the best way to solve the issue?

Yes. It extends the existing models.dev cache and Pi cost-scanning path rather than adding a parallel pricing store, while preserving provider-scoped catalog selection.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against f15f142a7787.

Labels

Label justifications:

  • P2: This is a bounded local cost-reporting enhancement without evidence of an urgent existing-user regression.
  • merge-risk: 🚨 automation: The PR materially changes the provider-architecture test gate and its existing allowlist classifications.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body shows after-fix built-CLI output for Gemini, Vertex AI, and Grok using isolated synthetic session files and no account credentials.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body shows after-fix built-CLI output for Gemini, Vertex AI, and Grok using isolated synthetic session files and no account credentials.

Evidence

What I checked:

Likely related people:

  • steipete: The repository owner stated that this provider-mapping work remains valuable and requested the rebased series be reviewed; the available local history is shallow at the latest release boundary. (role: owner reviewer and current-main history boundary; confidence: medium; commits: 0e453c4a5b2a; files: Sources/CodexBarCore/PiSessionCostScanner.swift, Sources/CodexBarCore/Vendored/CostUsage/CostUsagePricing.swift, Tests/CodexBarTests/ProviderArchitectureGatekeeperTests.swift)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Have a maintainer audit the ProviderArchitectureGatekeeperTests reclassification before merge.

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (33 earlier review cycles; latest 8 shown)
  • reviewed 2026-08-09T04:55:31.346Z sha 1eac100 :: found issues before merge. :: [P1] Update the provider capability gate
  • reviewed 2026-08-09T14:32:28.743Z sha 46c997f :: needs changes before merge. :: [P1] Update the provider capability gate expectations
  • reviewed 2026-08-09T20:12:22.152Z sha 46c997f :: needs changes before merge. :: [P1] Update provider-capability gate expectations
  • reviewed 2026-08-09T22:02:34.671Z sha 46c997f :: needs changes before merge. :: [P1] Update provider capability-gate expectations
  • reviewed 2026-08-09T22:50:38.143Z sha b836225 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-09T23:13:17.775Z sha b836225 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-12T09:14:33.055Z sha b836225 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-15T15:26:49.465Z sha 034e534 :: needs changes before merge. :: [P1] Restore the current gatekeeper allowances

@joeVenner

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 28, 2026

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Command router queued. I will update this comment with the next step.

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. labels Jul 28, 2026
@joeVenner

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 28, 2026

Copy link
Copy Markdown

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@clawsweeper clawsweeper Bot added proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jul 28, 2026
@joeVenner
joeVenner force-pushed the codex/modelsdev-google-xai-models branch from 2d19620 to b99340f Compare July 31, 2026 17:30
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jul 31, 2026
@joeVenner
joeVenner force-pushed the codex/modelsdev-google-xai-models branch 2 times, most recently from 87ad75c to 1eac100 Compare August 6, 2026 20:36
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Aug 6, 2026
@joeVenner
joeVenner force-pushed the codex/modelsdev-google-xai-models branch from 1eac100 to 46c997f Compare August 9, 2026 14:28
@joeVenner

Copy link
Copy Markdown
Contributor Author

@steipete — this PR has been rebased onto latest main and is ready for review. Thanks!

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. labels Aug 9, 2026
@joeVenner
joeVenner force-pushed the codex/modelsdev-google-xai-models branch from b836225 to 034e534 Compare August 15, 2026 15:20
@clawsweeper clawsweeper Bot added merge-risk: 🚨 automation 🚨 Merging this PR could break CI, automerge, proof capture, label sync, or automation. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Aug 15, 2026
…DevCostUSD out of enum to satisfy lint limits
ClawSweeper flagged that a custom codexHomePath should continue to suppress merging default Pi-session spend. Add a test verifying ambient scans merge Pi while scoped scans do not.
After rebasing the Google/xAI models.dev pricing work, ProviderArchitectureGatekeeperTests failed because new provider-specific branches (Gemini, Grok, VertexAI) appeared in shared cost code and existing allowlist anchors shifted. Add '// Provider-specific by design:' markers at each provider-owned dispatch point, update the gatekeeper's hardcoded supportsTokenSnapshot set and shifted suppressed references, and remove obsolete CostUsageFetcher/PiSessionCostScanner allowlist entries whose anchors no longer match.

Closes provider-architecture gatekeeper failures on the Google/xAI rebase.
@joeVenner
joeVenner force-pushed the codex/modelsdev-google-xai-models branch from 034e534 to 8dee931 Compare August 15, 2026 16:27
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. labels Aug 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 automation 🚨 Merging this PR could break CI, automerge, proof capture, label sync, or automation. P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants