fix(pricing): block brand-token fuzzy matches and prefer canonical pricing sources - #707
Merged
Conversation
…urces A post-#634 audit of lookup.rs against the real Anthropic catalog found two mispricing bugs: 1. Retired `claude-2.1`/`claude-2.0` (in historical logs, absent from every dataset) eroded to bare `claude` in try_strip_unknown_suffix (the "2.1" segment failed the all-digits version check) and then fuzzy-matched `anthropic/claude-opus-4.7-fast` at $30/$150. Fixed by (a) generalizing the stripper guard's version-segment recognition to digits-with-optional-dot and refusing strips that erode a claude id to a bare brand token, and (b) blocklisting bare `claude` and `anthropic` from fuzzy matching outright. 2. `claude-opus-4-6-fast` resolved to Models.dev reseller `venice/claude-opus-4-6-fast` ($36/$180) instead of canonical `anthropic/claude-opus-4.6-fast` ($30/$150) because the models.dev model-part pass ran before the separator-normalized OpenRouter exact pass in lookup_auto. Fixed by reordering so canonical separator-normalized passes run first, with models.dev kept as the long-tail fallback; the models.dev model-part index now also picks a deterministic provider (anthropic/ first, then shorter key, then lexicographic) instead of HashMap-iteration-order roulette. Reorder safety diff over the real cached datasets (197 probed ids): only the intended resolutions changed (claude-2.x/claude/anthropic -> None; opus-4-6/4-7-fast -> canonical; identical-price namespace moves; one previously nondeterministic equal-length tie made deterministic). Constraint: ids absent from all datasets must resolve unpriced, never to another model's price Constraint: existing #634 lookup contract must hold (only the assertion pinning the bug itself changed: is_fuzzy_eligible("claude") is now false by design) Rejected: hardcoding claude-2.x ids | the guard must cover future family-less shapes Rejected: pure lexicographic provider choice for shared model parts | would silently move 161 model parts to different-priced providers; anthropic-then-shortest-then-lex preserves historical winners Confidence: high Scope-risk: moderate Directive: lookup_auto pass ordering is load-bearing — canonical (LiteLLM/OpenRouter) separator-normalized exact passes must stay ahead of the models.dev model-part pass Not-tested: providers whose models.dev keys differ only by case
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7146f5ad76
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
459
to
460
| if let Some(result) = self.exact_match_openrouter(&version_normalized) { | ||
| return Some(result); |
There was a problem hiding this comment.
When a caller passes a provider hint for a models.dev-only provider, this unscoped normalized fallback now runs before exact_match_models_dev_with_provider, so provider-specific pricing can be skipped. For example, with provider_id = Some("venice"), claude-opus-4-6-fast can return OpenRouter's anthropic/claude-opus-4.6-fast here before the later models.dev pass has a chance to select venice/claude-opus-4-6-fast, causing costs to be computed for the wrong provider in provider-aware lookups.
Useful? React with 👍 / 👎.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
2 issues found across 2 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
…kup fallbacks Addresses three #707 review findings: 1. Provider-hint bypass: the canonical-source reorder let the unscoped separator-normalized OpenRouter pass return anthropic/... before the provider-scoped models.dev pass could honor a hint like `venice`. Provider-scoped models.dev passes (raw and version-normalized) now run before the unscoped normalized fallback whenever a hint is present, so the reorder only preempts models.dev for unhinted lookups. 2. Bare-brand-token coverage: `anthropic` joins `claude`/`claude-2.x` in the must-resolve-to-None assertions (catalog invariant test and the in-module fuzzy-blocklist regression test). 3. Unpriced-key shadowing: the models.dev model-part index now skips entries without any usable pricing, so the anthropic-first preference cannot pick an unpriced anthropic/<model> row over a priced reseller row and bill usage at zero. The loader already guarantees priced entries (models_dev::cost_to_pricing), but new_with_models_dev is public, so the index guards itself. Constraint: unhinted resolutions must keep the canonical-first behavior the PR introduced (bug 2) Rejected: gating the unscoped raw exact litellm/openrouter passes on the hint too | pre-existing behavior outside this PR's regression, broader blast radius Rejected: filtering zero-cost (0.0) entries from the model-part index | 0.0 is deliberately valid pricing for free models (is_valid_price_value) Confidence: high Scope-risk: narrow Directive: provider-scoped passes must stay ahead of unscoped normalized fallbacks in lookup_auto; both new regression tests fail if either guard is removed Not-tested: hinted lookups where the hint matches multiple models.dev providers sharing a model part
Contributor
There was a problem hiding this comment.
1 issue found across 2 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
dbec203 pinned provider-hinted lookups ahead of the separator-normalized fallbacks, but the unscoped OpenRouter exact pass still ran first and its model-part fallback could leak: for a dotted id like claude-opus-4.6-fast, exact_match_openrouter matches anthropic/claude-opus-4.6-fast by model-part and returns before the provider-scoped models.dev pass — so a venice-hinted lookup got the canonical price instead of venice's own key. The hyphenated form fell through correctly (its model-part is dotted), which is why dbec203's test missed this. Split exact_match_openrouter into _full_key (the id's own canonical key, which still wins under a hint) and _model_part (matches any provider sharing the model-part, which a hint must override). In lookup_auto the provider- scoped models.dev pass now runs between them. Unhinted lookups are unchanged (full_key.or(model_part) == the old combined match). Constraint: an exact full-key match is the id's own key and stays first even under a hint Constraint: a hint for a provider with no matching key must fall through to the canonical resolution, not None Rejected: gating the unscoped litellm/openrouter full-key passes on the hint too | full-key is the id's own key; broader blast radius for no correctness gain Confidence: high Scope-risk: narrow Directive: provider-scoped models.dev must stay between OpenRouter full-key and model-part passes in lookup_auto; the dotted-id regression test fails if it moves
Harvesting the distinct model ids in real local session data surfaced three ids (`model-zero-usage-v1`, `model-nonzero-usage-v1`, `test-model`) that mispriced to `azure_ai/model_router` ($0.14/MTok). Root cause: after suffix/prefix stripping their only fuzzy-eligible remnant is the bare word `model`, which substring-matches the priced key `azure_ai/model_router` (and `router` matches `kilo/switchpoint/router`). These are generic English words carrying no model identity — the same defect class as the already- blocked bare brand tokens (`claude`, `anthropic`). Add `model` and `router` to FUZZY_BLOCKLIST so such ids stay unpriced (never-degrade) instead of billing at an unrelated key's rate. Exact-key matches (e.g. the real `azure/model-router` id) are unaffected. Constraint: must not change resolution of any real model id — before/after resolution over the full harvested local id set differs only on the three noise ids (now None). Rejected: raise MIN_FUZZY_MATCH_LEN | `model`/`router` are >=5 chars, would not be caught and longer generic words would still slip through. Confidence: high Scope-risk: narrow Not-tested: ids that strip to a generic word longer than `router`
The catalog invariant test used a hardcoded ~16-id list. Replace its core with the DISTINCT model ids actually harvested from local session data (committed as tests/fixtures/local_model_ids.txt — model-id strings plus family/version/price-band expectations only, NO usage counts or user data). - anthropic_catalog_resolves_to_correct_family_version_and_price now drives every harvested claude-* id (regional/suffix/provider forms incl. vertex_ai/llmgateway/github-copilot variants) through the real cached datasets, asserting family token, major-minor version token, and price within +/-25%. A few hardcoded provider/regional forms remain as guards. - Add real_local_models_all_resolve_sanely: iterates the FULL harvested set (gpt/gemini/grok/glm/kimi/claude) and asserts each id resolves with a matching vendor token or is a documented acceptable-None, and that NO id resolves to a cross-family key. Both stay #[ignore]d (skip-graceful when cache absent). The generic-token misprice fix is covered by a non-ignored unit test in lookup.rs. Constraint: fixture must contain only model-id strings + pricing metadata, no usage counts / session ids / paths / user-identifying content. Confidence: high Scope-risk: narrow Directive: when adding a new model id to the fixture, audit its real resolution first — the version column uses the dashed spelling the matched KEY carries (e.g. claude-3-5 keys need "3-5", not "3.5").
Contributor
There was a problem hiding this comment.
1 issue found across 3 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
…iced The real-data catalog test's acceptable-None gate keyed off input_cost_per_token.is_some(), so a documented-None id resolving to a key with input: None but output: Some(..) would have passed silently. Mirror the sibling catalog test: flag ANY resolution for an acceptable-None id, and surface both input and output prices in the failure message. Issue identified by cubic. Confidence: high Scope-risk: narrow
junhoyeo
added a commit
to haunchen/tokscale
that referenced
this pull request
Jun 17, 2026
…arser tests Map the raw junhoyeo#19 responseModel `gemini-3-flash-a` onto the priced `gemini-3-flash-preview` so Antigravity CLI cost no longer resolves to 0. Add alias-resolution, junhoyeo#9/junhoyeo#10==junhoyeo#3 field-mapping invariant, and malformed-protobuf bounds tests. Constraint: must not weaken the junhoyeo#707 brand-token fuzzy-match guard in lookup.rs Confidence: high Scope-risk: narrow
junhoyeo
added a commit
that referenced
this pull request
Jun 17, 2026
#713) * feat(sessions): read Antigravity CLI usage from local SQLite databases The Antigravity CLI (the terminal agent that stores its data under `~/.gemini/antigravity-cli/`) was never counted. tokscale only knew two Gemini-family sources: the Gemini CLI (scans `~/.gemini/tmp/*.{json,jsonl}`) and Antigravity (pulls usage from a running IDE language server over RPC and caches it under the config dir). The Antigravity CLI fell into neither bucket, so its on-disk usage was invisible — `tokscale antigravity sync` found the filesystem candidates but cached zero because its only artifact path still requires a live language-server RPC connection. This adds Antigravity CLI as a first-class local scan source so its usage updates automatically like every other file-based source — no RPC, no `antigravity sync`. A new `antigravity-cli` client globs `~/.gemini/antigravity-cli/conversations/*.db` (honoring `GEMINI_CLI_HOME`) and a new parser reads each conversation database directly. Each `gen_metadata` row is one generation encoded as the same `GeneratorMetadata` protobuf the IDE returns over `GetCascadeTrajectoryGeneratorMetadata`. The repository has no `.proto`/prost decoder (the IDE path receives JSON because the language server does the proto-to-JSON conversion), so the parser ships a tiny dependency-free wire-format reader and pulls only the fields it needs. The field numbers were reverse-engineered from real databases and cross-checked across 6 sessions / 140 turns: `chatModel.#19` is the response model, `usage.#5`/`#9`/`#10` are cacheRead/output/thinking (verified by the invariant `#9 + #10 == #3`, the stored total output), `#11` is the responseId used for dedup, and input combines the fixed system-prompt count `#1` with the newly-processed input `#2`. The session timestamp and workspace come from `trajectory_metadata_blob`. Adding the new `ClientId` variant fans out to the usual registration points: the scanner gains a `*.db` glob arm (which naturally rejects `.db-wal`/`.db-shm` sidecars), both local-parse dispatch paths gain a branch, and the CLI `ClientFilter`, client labels, TUI picker, and frontend source maps gain entries. The deprecated per-client boolean flags intentionally do not, since `antigravity-cli` is reachable only via the canonical `--client antigravity-cli`. Closes #712. * fix(sessions): handle file:// authority/UNC paths and test Antigravity CLI wiring Addresses the cubic review on #713. `file_uri_to_path` previously stripped `file://` and only special-cased the leading slash before a Windows drive letter, so a non-empty authority (`file://host/share/...`, the UNC form) lost its host and collapsed into a bare path. It now treats an empty-authority remainder as before (`/C:/x` → `C:/x`, `/home/x` kept) and reconstructs a non-empty authority as a UNC path (`host/share/x` → `//host/share/x`) so `normalize_workspace_key` preserves the `//` prefix. A unit test covers the Windows-drive, POSIX, UNC, and percent-encoded-CJK cases. The new `AntigravityCli` client wiring is now asserted in `test_client_as_str`, `test_client_key`, and `test_client_from_key` (display name "Antigravity CLI", hotkey `f`, and the reverse hotkey mapping). * style: rustfmt antigravity_cli.rs * fix(antigravity-cli): add gemini-3-flash-a pricing alias and harden parser tests Map the raw #19 responseModel `gemini-3-flash-a` onto the priced `gemini-3-flash-preview` so Antigravity CLI cost no longer resolves to 0. Add alias-resolution, #9/#10==#3 field-mapping invariant, and malformed-protobuf bounds tests. Constraint: must not weaken the #707 brand-token fuzzy-match guard in lookup.rs Confidence: high Scope-risk: narrow --------- Co-authored-by: Junho Yeo <i@junho.io>
pinion05
added a commit
to pinion05/tokscale
that referenced
this pull request
Jun 23, 2026
…icing sources (junhoyeo#707) * fix(pricing): block brand-token fuzzy matches and prefer canonical sources A post-junhoyeo#634 audit of lookup.rs against the real Anthropic catalog found two mispricing bugs: 1. Retired `claude-2.1`/`claude-2.0` (in historical logs, absent from every dataset) eroded to bare `claude` in try_strip_unknown_suffix (the "2.1" segment failed the all-digits version check) and then fuzzy-matched `anthropic/claude-opus-4.7-fast` at $30/$150. Fixed by (a) generalizing the stripper guard's version-segment recognition to digits-with-optional-dot and refusing strips that erode a claude id to a bare brand token, and (b) blocklisting bare `claude` and `anthropic` from fuzzy matching outright. 2. `claude-opus-4-6-fast` resolved to Models.dev reseller `venice/claude-opus-4-6-fast` ($36/$180) instead of canonical `anthropic/claude-opus-4.6-fast` ($30/$150) because the models.dev model-part pass ran before the separator-normalized OpenRouter exact pass in lookup_auto. Fixed by reordering so canonical separator-normalized passes run first, with models.dev kept as the long-tail fallback; the models.dev model-part index now also picks a deterministic provider (anthropic/ first, then shorter key, then lexicographic) instead of HashMap-iteration-order roulette. Reorder safety diff over the real cached datasets (197 probed ids): only the intended resolutions changed (claude-2.x/claude/anthropic -> None; opus-4-6/4-7-fast -> canonical; identical-price namespace moves; one previously nondeterministic equal-length tie made deterministic). Constraint: ids absent from all datasets must resolve unpriced, never to another model's price Constraint: existing junhoyeo#634 lookup contract must hold (only the assertion pinning the bug itself changed: is_fuzzy_eligible("claude") is now false by design) Rejected: hardcoding claude-2.x ids | the guard must cover future family-less shapes Rejected: pure lexicographic provider choice for shared model parts | would silently move 161 model parts to different-priced providers; anthropic-then-shortest-then-lex preserves historical winners Confidence: high Scope-risk: moderate Directive: lookup_auto pass ordering is load-bearing — canonical (LiteLLM/OpenRouter) separator-normalized exact passes must stay ahead of the models.dev model-part pass Not-tested: providers whose models.dev keys differ only by case * fix(pricing): respect provider hints and priced-key preference in lookup fallbacks Addresses three junhoyeo#707 review findings: 1. Provider-hint bypass: the canonical-source reorder let the unscoped separator-normalized OpenRouter pass return anthropic/... before the provider-scoped models.dev pass could honor a hint like `venice`. Provider-scoped models.dev passes (raw and version-normalized) now run before the unscoped normalized fallback whenever a hint is present, so the reorder only preempts models.dev for unhinted lookups. 2. Bare-brand-token coverage: `anthropic` joins `claude`/`claude-2.x` in the must-resolve-to-None assertions (catalog invariant test and the in-module fuzzy-blocklist regression test). 3. Unpriced-key shadowing: the models.dev model-part index now skips entries without any usable pricing, so the anthropic-first preference cannot pick an unpriced anthropic/<model> row over a priced reseller row and bill usage at zero. The loader already guarantees priced entries (models_dev::cost_to_pricing), but new_with_models_dev is public, so the index guards itself. Constraint: unhinted resolutions must keep the canonical-first behavior the PR introduced (bug 2) Rejected: gating the unscoped raw exact litellm/openrouter passes on the hint too | pre-existing behavior outside this PR's regression, broader blast radius Rejected: filtering zero-cost (0.0) entries from the model-part index | 0.0 is deliberately valid pricing for free models (is_valid_price_value) Confidence: high Scope-risk: narrow Directive: provider-scoped passes must stay ahead of unscoped normalized fallbacks in lookup_auto; both new regression tests fail if either guard is removed Not-tested: hinted lookups where the hint matches multiple models.dev providers sharing a model part * fix(pricing): pin provider hints ahead of unscoped OpenRouter model-part dbec203 pinned provider-hinted lookups ahead of the separator-normalized fallbacks, but the unscoped OpenRouter exact pass still ran first and its model-part fallback could leak: for a dotted id like claude-opus-4.6-fast, exact_match_openrouter matches anthropic/claude-opus-4.6-fast by model-part and returns before the provider-scoped models.dev pass — so a venice-hinted lookup got the canonical price instead of venice's own key. The hyphenated form fell through correctly (its model-part is dotted), which is why dbec203's test missed this. Split exact_match_openrouter into _full_key (the id's own canonical key, which still wins under a hint) and _model_part (matches any provider sharing the model-part, which a hint must override). In lookup_auto the provider- scoped models.dev pass now runs between them. Unhinted lookups are unchanged (full_key.or(model_part) == the old combined match). Constraint: an exact full-key match is the id's own key and stays first even under a hint Constraint: a hint for a provider with no matching key must fall through to the canonical resolution, not None Rejected: gating the unscoped litellm/openrouter full-key passes on the hint too | full-key is the id's own key; broader blast radius for no correctness gain Confidence: high Scope-risk: narrow Directive: provider-scoped models.dev must stay between OpenRouter full-key and model-part passes in lookup_auto; the dotted-id regression test fails if it moves * fix(pricing): block generic "model"/"router" tokens from fuzzy match Harvesting the distinct model ids in real local session data surfaced three ids (`model-zero-usage-v1`, `model-nonzero-usage-v1`, `test-model`) that mispriced to `azure_ai/model_router` ($0.14/MTok). Root cause: after suffix/prefix stripping their only fuzzy-eligible remnant is the bare word `model`, which substring-matches the priced key `azure_ai/model_router` (and `router` matches `kilo/switchpoint/router`). These are generic English words carrying no model identity — the same defect class as the already- blocked bare brand tokens (`claude`, `anthropic`). Add `model` and `router` to FUZZY_BLOCKLIST so such ids stay unpriced (never-degrade) instead of billing at an unrelated key's rate. Exact-key matches (e.g. the real `azure/model-router` id) are unaffected. Constraint: must not change resolution of any real model id — before/after resolution over the full harvested local id set differs only on the three noise ids (now None). Rejected: raise MIN_FUZZY_MATCH_LEN | `model`/`router` are >=5 chars, would not be caught and longer generic words would still slip through. Confidence: high Scope-risk: narrow Not-tested: ids that strip to a generic word longer than `router` * test(pricing): drive catalog regression from real local model ids The catalog invariant test used a hardcoded ~16-id list. Replace its core with the DISTINCT model ids actually harvested from local session data (committed as tests/fixtures/local_model_ids.txt — model-id strings plus family/version/price-band expectations only, NO usage counts or user data). - anthropic_catalog_resolves_to_correct_family_version_and_price now drives every harvested claude-* id (regional/suffix/provider forms incl. vertex_ai/llmgateway/github-copilot variants) through the real cached datasets, asserting family token, major-minor version token, and price within +/-25%. A few hardcoded provider/regional forms remain as guards. - Add real_local_models_all_resolve_sanely: iterates the FULL harvested set (gpt/gemini/grok/glm/kimi/claude) and asserts each id resolves with a matching vendor token or is a documented acceptable-None, and that NO id resolves to a cross-family key. Both stay #[ignore]d (skip-graceful when cache absent). The generic-token misprice fix is covered by a non-ignored unit test in lookup.rs. Constraint: fixture must contain only model-id strings + pricing metadata, no usage counts / session ids / paths / user-identifying content. Confidence: high Scope-risk: narrow Directive: when adding a new model id to the fixture, audit its real resolution first — the version column uses the dashed spelling the matched KEY carries (e.g. claude-3-5 keys need "3-5", not "3.5"). * test(pricing): flag any acceptable-None resolution, not just input-priced The real-data catalog test's acceptable-None gate keyed off input_cost_per_token.is_some(), so a documented-None id resolving to a key with input: None but output: Some(..) would have passed silently. Mirror the sibling catalog test: flag ANY resolution for an acceptable-None id, and surface both input and output prices in the failure message. Issue identified by cubic. Confidence: high Scope-risk: narrow
pinion05
added a commit
to pinion05/tokscale
that referenced
this pull request
Jun 23, 2026
junhoyeo#713) * feat(sessions): read Antigravity CLI usage from local SQLite databases The Antigravity CLI (the terminal agent that stores its data under `~/.gemini/antigravity-cli/`) was never counted. tokscale only knew two Gemini-family sources: the Gemini CLI (scans `~/.gemini/tmp/*.{json,jsonl}`) and Antigravity (pulls usage from a running IDE language server over RPC and caches it under the config dir). The Antigravity CLI fell into neither bucket, so its on-disk usage was invisible — `tokscale antigravity sync` found the filesystem candidates but cached zero because its only artifact path still requires a live language-server RPC connection. This adds Antigravity CLI as a first-class local scan source so its usage updates automatically like every other file-based source — no RPC, no `antigravity sync`. A new `antigravity-cli` client globs `~/.gemini/antigravity-cli/conversations/*.db` (honoring `GEMINI_CLI_HOME`) and a new parser reads each conversation database directly. Each `gen_metadata` row is one generation encoded as the same `GeneratorMetadata` protobuf the IDE returns over `GetCascadeTrajectoryGeneratorMetadata`. The repository has no `.proto`/prost decoder (the IDE path receives JSON because the language server does the proto-to-JSON conversion), so the parser ships a tiny dependency-free wire-format reader and pulls only the fields it needs. The field numbers were reverse-engineered from real databases and cross-checked across 6 sessions / 140 turns: `chatModel.junhoyeo#19` is the response model, `usage.junhoyeo#5`/`junhoyeo#9`/`junhoyeo#10` are cacheRead/output/thinking (verified by the invariant `junhoyeo#9 + junhoyeo#10 == junhoyeo#3`, the stored total output), `junhoyeo#11` is the responseId used for dedup, and input combines the fixed system-prompt count `#1` with the newly-processed input `junhoyeo#2`. The session timestamp and workspace come from `trajectory_metadata_blob`. Adding the new `ClientId` variant fans out to the usual registration points: the scanner gains a `*.db` glob arm (which naturally rejects `.db-wal`/`.db-shm` sidecars), both local-parse dispatch paths gain a branch, and the CLI `ClientFilter`, client labels, TUI picker, and frontend source maps gain entries. The deprecated per-client boolean flags intentionally do not, since `antigravity-cli` is reachable only via the canonical `--client antigravity-cli`. Closes junhoyeo#712. * fix(sessions): handle file:// authority/UNC paths and test Antigravity CLI wiring Addresses the cubic review on junhoyeo#713. `file_uri_to_path` previously stripped `file://` and only special-cased the leading slash before a Windows drive letter, so a non-empty authority (`file://host/share/...`, the UNC form) lost its host and collapsed into a bare path. It now treats an empty-authority remainder as before (`/C:/x` → `C:/x`, `/home/x` kept) and reconstructs a non-empty authority as a UNC path (`host/share/x` → `//host/share/x`) so `normalize_workspace_key` preserves the `//` prefix. A unit test covers the Windows-drive, POSIX, UNC, and percent-encoded-CJK cases. The new `AntigravityCli` client wiring is now asserted in `test_client_as_str`, `test_client_key`, and `test_client_from_key` (display name "Antigravity CLI", hotkey `f`, and the reverse hotkey mapping). * style: rustfmt antigravity_cli.rs * fix(antigravity-cli): add gemini-3-flash-a pricing alias and harden parser tests Map the raw junhoyeo#19 responseModel `gemini-3-flash-a` onto the priced `gemini-3-flash-preview` so Antigravity CLI cost no longer resolves to 0. Add alias-resolution, junhoyeo#9/junhoyeo#10==junhoyeo#3 field-mapping invariant, and malformed-protobuf bounds tests. Constraint: must not weaken the junhoyeo#707 brand-token fuzzy-match guard in lookup.rs Confidence: high Scope-risk: narrow --------- Co-authored-by: Junho Yeo <i@junho.io>
t1000040
pushed a commit
to tmobi-internal/tokscale
that referenced
this pull request
Jun 30, 2026
…icing sources (junhoyeo#707) * fix(pricing): block brand-token fuzzy matches and prefer canonical sources A post-junhoyeo#634 audit of lookup.rs against the real Anthropic catalog found two mispricing bugs: 1. Retired `claude-2.1`/`claude-2.0` (in historical logs, absent from every dataset) eroded to bare `claude` in try_strip_unknown_suffix (the "2.1" segment failed the all-digits version check) and then fuzzy-matched `anthropic/claude-opus-4.7-fast` at $30/$150. Fixed by (a) generalizing the stripper guard's version-segment recognition to digits-with-optional-dot and refusing strips that erode a claude id to a bare brand token, and (b) blocklisting bare `claude` and `anthropic` from fuzzy matching outright. 2. `claude-opus-4-6-fast` resolved to Models.dev reseller `venice/claude-opus-4-6-fast` ($36/$180) instead of canonical `anthropic/claude-opus-4.6-fast` ($30/$150) because the models.dev model-part pass ran before the separator-normalized OpenRouter exact pass in lookup_auto. Fixed by reordering so canonical separator-normalized passes run first, with models.dev kept as the long-tail fallback; the models.dev model-part index now also picks a deterministic provider (anthropic/ first, then shorter key, then lexicographic) instead of HashMap-iteration-order roulette. Reorder safety diff over the real cached datasets (197 probed ids): only the intended resolutions changed (claude-2.x/claude/anthropic -> None; opus-4-6/4-7-fast -> canonical; identical-price namespace moves; one previously nondeterministic equal-length tie made deterministic). Constraint: ids absent from all datasets must resolve unpriced, never to another model's price Constraint: existing junhoyeo#634 lookup contract must hold (only the assertion pinning the bug itself changed: is_fuzzy_eligible("claude") is now false by design) Rejected: hardcoding claude-2.x ids | the guard must cover future family-less shapes Rejected: pure lexicographic provider choice for shared model parts | would silently move 161 model parts to different-priced providers; anthropic-then-shortest-then-lex preserves historical winners Confidence: high Scope-risk: moderate Directive: lookup_auto pass ordering is load-bearing — canonical (LiteLLM/OpenRouter) separator-normalized exact passes must stay ahead of the models.dev model-part pass Not-tested: providers whose models.dev keys differ only by case * fix(pricing): respect provider hints and priced-key preference in lookup fallbacks Addresses three junhoyeo#707 review findings: 1. Provider-hint bypass: the canonical-source reorder let the unscoped separator-normalized OpenRouter pass return anthropic/... before the provider-scoped models.dev pass could honor a hint like `venice`. Provider-scoped models.dev passes (raw and version-normalized) now run before the unscoped normalized fallback whenever a hint is present, so the reorder only preempts models.dev for unhinted lookups. 2. Bare-brand-token coverage: `anthropic` joins `claude`/`claude-2.x` in the must-resolve-to-None assertions (catalog invariant test and the in-module fuzzy-blocklist regression test). 3. Unpriced-key shadowing: the models.dev model-part index now skips entries without any usable pricing, so the anthropic-first preference cannot pick an unpriced anthropic/<model> row over a priced reseller row and bill usage at zero. The loader already guarantees priced entries (models_dev::cost_to_pricing), but new_with_models_dev is public, so the index guards itself. Constraint: unhinted resolutions must keep the canonical-first behavior the PR introduced (bug 2) Rejected: gating the unscoped raw exact litellm/openrouter passes on the hint too | pre-existing behavior outside this PR's regression, broader blast radius Rejected: filtering zero-cost (0.0) entries from the model-part index | 0.0 is deliberately valid pricing for free models (is_valid_price_value) Confidence: high Scope-risk: narrow Directive: provider-scoped passes must stay ahead of unscoped normalized fallbacks in lookup_auto; both new regression tests fail if either guard is removed Not-tested: hinted lookups where the hint matches multiple models.dev providers sharing a model part * fix(pricing): pin provider hints ahead of unscoped OpenRouter model-part dbec203 pinned provider-hinted lookups ahead of the separator-normalized fallbacks, but the unscoped OpenRouter exact pass still ran first and its model-part fallback could leak: for a dotted id like claude-opus-4.6-fast, exact_match_openrouter matches anthropic/claude-opus-4.6-fast by model-part and returns before the provider-scoped models.dev pass — so a venice-hinted lookup got the canonical price instead of venice's own key. The hyphenated form fell through correctly (its model-part is dotted), which is why dbec203's test missed this. Split exact_match_openrouter into _full_key (the id's own canonical key, which still wins under a hint) and _model_part (matches any provider sharing the model-part, which a hint must override). In lookup_auto the provider- scoped models.dev pass now runs between them. Unhinted lookups are unchanged (full_key.or(model_part) == the old combined match). Constraint: an exact full-key match is the id's own key and stays first even under a hint Constraint: a hint for a provider with no matching key must fall through to the canonical resolution, not None Rejected: gating the unscoped litellm/openrouter full-key passes on the hint too | full-key is the id's own key; broader blast radius for no correctness gain Confidence: high Scope-risk: narrow Directive: provider-scoped models.dev must stay between OpenRouter full-key and model-part passes in lookup_auto; the dotted-id regression test fails if it moves * fix(pricing): block generic "model"/"router" tokens from fuzzy match Harvesting the distinct model ids in real local session data surfaced three ids (`model-zero-usage-v1`, `model-nonzero-usage-v1`, `test-model`) that mispriced to `azure_ai/model_router` ($0.14/MTok). Root cause: after suffix/prefix stripping their only fuzzy-eligible remnant is the bare word `model`, which substring-matches the priced key `azure_ai/model_router` (and `router` matches `kilo/switchpoint/router`). These are generic English words carrying no model identity — the same defect class as the already- blocked bare brand tokens (`claude`, `anthropic`). Add `model` and `router` to FUZZY_BLOCKLIST so such ids stay unpriced (never-degrade) instead of billing at an unrelated key's rate. Exact-key matches (e.g. the real `azure/model-router` id) are unaffected. Constraint: must not change resolution of any real model id — before/after resolution over the full harvested local id set differs only on the three noise ids (now None). Rejected: raise MIN_FUZZY_MATCH_LEN | `model`/`router` are >=5 chars, would not be caught and longer generic words would still slip through. Confidence: high Scope-risk: narrow Not-tested: ids that strip to a generic word longer than `router` * test(pricing): drive catalog regression from real local model ids The catalog invariant test used a hardcoded ~16-id list. Replace its core with the DISTINCT model ids actually harvested from local session data (committed as tests/fixtures/local_model_ids.txt — model-id strings plus family/version/price-band expectations only, NO usage counts or user data). - anthropic_catalog_resolves_to_correct_family_version_and_price now drives every harvested claude-* id (regional/suffix/provider forms incl. vertex_ai/llmgateway/github-copilot variants) through the real cached datasets, asserting family token, major-minor version token, and price within +/-25%. A few hardcoded provider/regional forms remain as guards. - Add real_local_models_all_resolve_sanely: iterates the FULL harvested set (gpt/gemini/grok/glm/kimi/claude) and asserts each id resolves with a matching vendor token or is a documented acceptable-None, and that NO id resolves to a cross-family key. Both stay #[ignore]d (skip-graceful when cache absent). The generic-token misprice fix is covered by a non-ignored unit test in lookup.rs. Constraint: fixture must contain only model-id strings + pricing metadata, no usage counts / session ids / paths / user-identifying content. Confidence: high Scope-risk: narrow Directive: when adding a new model id to the fixture, audit its real resolution first — the version column uses the dashed spelling the matched KEY carries (e.g. claude-3-5 keys need "3-5", not "3.5"). * test(pricing): flag any acceptable-None resolution, not just input-priced The real-data catalog test's acceptable-None gate keyed off input_cost_per_token.is_some(), so a documented-None id resolving to a key with input: None but output: Some(..) would have passed silently. Mirror the sibling catalog test: flag ANY resolution for an acceptable-None id, and surface both input and output prices in the failure message. Issue identified by cubic. Confidence: high Scope-risk: narrow
t1000040
pushed a commit
to tmobi-internal/tokscale
that referenced
this pull request
Jun 30, 2026
junhoyeo#713) * feat(sessions): read Antigravity CLI usage from local SQLite databases The Antigravity CLI (the terminal agent that stores its data under `~/.gemini/antigravity-cli/`) was never counted. tokscale only knew two Gemini-family sources: the Gemini CLI (scans `~/.gemini/tmp/*.{json,jsonl}`) and Antigravity (pulls usage from a running IDE language server over RPC and caches it under the config dir). The Antigravity CLI fell into neither bucket, so its on-disk usage was invisible — `tokscale antigravity sync` found the filesystem candidates but cached zero because its only artifact path still requires a live language-server RPC connection. This adds Antigravity CLI as a first-class local scan source so its usage updates automatically like every other file-based source — no RPC, no `antigravity sync`. A new `antigravity-cli` client globs `~/.gemini/antigravity-cli/conversations/*.db` (honoring `GEMINI_CLI_HOME`) and a new parser reads each conversation database directly. Each `gen_metadata` row is one generation encoded as the same `GeneratorMetadata` protobuf the IDE returns over `GetCascadeTrajectoryGeneratorMetadata`. The repository has no `.proto`/prost decoder (the IDE path receives JSON because the language server does the proto-to-JSON conversion), so the parser ships a tiny dependency-free wire-format reader and pulls only the fields it needs. The field numbers were reverse-engineered from real databases and cross-checked across 6 sessions / 140 turns: `chatModel.#19` is the response model, `usage.#5`/`#9`/`#10` are cacheRead/output/thinking (verified by the invariant `#9 + #10 == #3`, the stored total output), `#11` is the responseId used for dedup, and input combines the fixed system-prompt count `#1` with the newly-processed input `#2`. The session timestamp and workspace come from `trajectory_metadata_blob`. Adding the new `ClientId` variant fans out to the usual registration points: the scanner gains a `*.db` glob arm (which naturally rejects `.db-wal`/`.db-shm` sidecars), both local-parse dispatch paths gain a branch, and the CLI `ClientFilter`, client labels, TUI picker, and frontend source maps gain entries. The deprecated per-client boolean flags intentionally do not, since `antigravity-cli` is reachable only via the canonical `--client antigravity-cli`. Closes junhoyeo#712. * fix(sessions): handle file:// authority/UNC paths and test Antigravity CLI wiring Addresses the cubic review on junhoyeo#713. `file_uri_to_path` previously stripped `file://` and only special-cased the leading slash before a Windows drive letter, so a non-empty authority (`file://host/share/...`, the UNC form) lost its host and collapsed into a bare path. It now treats an empty-authority remainder as before (`/C:/x` → `C:/x`, `/home/x` kept) and reconstructs a non-empty authority as a UNC path (`host/share/x` → `//host/share/x`) so `normalize_workspace_key` preserves the `//` prefix. A unit test covers the Windows-drive, POSIX, UNC, and percent-encoded-CJK cases. The new `AntigravityCli` client wiring is now asserted in `test_client_as_str`, `test_client_key`, and `test_client_from_key` (display name "Antigravity CLI", hotkey `f`, and the reverse hotkey mapping). * style: rustfmt antigravity_cli.rs * fix(antigravity-cli): add gemini-3-flash-a pricing alias and harden parser tests Map the raw #19 responseModel `gemini-3-flash-a` onto the priced `gemini-3-flash-preview` so Antigravity CLI cost no longer resolves to 0. Add alias-resolution, #9/#10==#3 field-mapping invariant, and malformed-protobuf bounds tests. Constraint: must not weaken the junhoyeo#707 brand-token fuzzy-match guard in lookup.rs Confidence: high Scope-risk: narrow --------- Co-authored-by: Junho Yeo <i@junho.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by post-#634 catalog audit of
crates/tokscale-core/src/pricing/lookup.rsagainst the real Anthropic model catalog.Bug 1 (severe): retired
claude-2.xbilled atclaude-opus-4.7-fastratesWith the real cached datasets, before this PR:
claude-2.1/claude-2.0are retired models that still appear in historical usage logs and are absent from every pricing dataset — they must resolve unpriced, never to another model's price.Root cause chain:
try_strip_unknown_suffixerodesclaude-2.1to bareclaude— thestrips_claude_numeric_minorguard didn't fire because segment"2.1"contains a dot and failed the all-digits version check. Bareclaude(len 6 ≥MIN_FUZZY_MATCH_LEN, not blocklisted) then reachedfuzzy_match_openrouterand matched the first claude key. The #634 family veto (resolves_unsafe_claude_version) was bypassed becauseclaude-2.1carries no opus/sonnet/haiku/fable token, sorequested_familyisNone.Fixes
strips_claude_numeric_minor: version-segment recognition generalized from all-digits to digits-with-optional-dot (newis_version_segment, plain char loop — file avoids regex), and the guard now also refuses strips that erode a claude-branded id to a bare brand token (candidate left with no digits). Dated forms (claude-3-5-sonnet-20241022) keep stripping — their candidate retains a version.FUZZY_BLOCKLIST(exact-token match inis_fuzzy_eligible) gains"claude"and"anthropic"— a bare brand token can match any model of the brand, so a fuzzy hit from it is never trustworthy.Bug 2 (minor): reseller markup beats canonical pricing for
-fastidsBefore this PR:
instead of canonical
anthropic/claude-opus-4.6-fast/4.7-fastat $30/$150 (OpenRouter).Root cause: in
lookup_auto,exact_match_models_dev_with_provider(model_id)ran before thenormalize_version_separatorpass that converts4-6→4.6and hits the canonical OpenRouter key.Fixes
lookup_autoreordered: separator-normalized exact passes (LiteLLM + OpenRouter) now run before the models.dev model-part pass; models.dev remains the long-tail fallback (covered by a regression test).models_dev_model_partindex (newprefers_model_part_key): when multiple providers share a model part, the winner is now deterministic — exactanthropicprovider first, then shorter key (the historical winner of the insertion-order race, keeping existing resolutions stable), then lexicographic tie-break. Previously HashMap iteration order decided (with real data302ai/beatanthropic/forclaude-3-5-haiku-20241022).Reorder safety diff (real cached datasets, 197 probed ids)
A temporary probe dumped
model_id → matched_key,input_pricefor every distinct id in the lookup.rs test corpus + the Anthropic catalog + ~50 non-Anthropic ids exercising the models.dev model-part path (gpt/gemini/llama/qwen/deepseek/grok/kimi/glm/...), before and after. 10 ids changed, 187 unchanged:claude-2.1,claude-2.0,claude,anthropicclaude-opus-4-6-fast,claude-opus-4-7-fastvenice/...$36/$180anthropic/claude-opus-4.x-fast$30/$150claude-3-5-haiku-20241022302ai/...anthropic/...claude-3-5-haikuopencode/claude-3-5-haiku(Models.dev)anthropic/claude-3.5-haiku(OpenRouter)kimi-k2-5venice/kimi-k2-5$0.56/$3.5moonshotai/kimi-k2.5$0.6/$3.0grok-4-0709jiekou/grok-4-0709abacus/grok-4-0709jiekou/andabacus/; now deterministicThe probe was deleted before committing.
Tests
In
lookup.rs mod tests(fixture style matchingclaude_family_fixture, each fixture contains the tempting wrong candidate so the test is red without the fix):claude_2x_never_fuzzy_matches_modern_models— openrouter fixture with onlyanthropic/claude-opus-4.7-fast;claude-2.1/claude-2.0/claudeall None.claude_2x_still_resolves_when_dataset_prices_it— positive control; matched_keyclaude-2.1, input 8e-6.canonical_fast_price_beats_reseller_markup— venice (36e-6) vs canonical (30e-6); resolves toanthropic/claude-opus-4.6-fastat 30e-6.models_dev_still_covers_long_tail_after_reorder— models.dev-onlysomeprovider/exotic-model-9still resolves.models_dev_provider_choice_is_deterministic_and_prefers_anthropic—302ai/+anthropic/same price; anthropic wins.New
crates/tokscale-core/tests/anthropic_catalog.rs(#[ignore]d, CI skips; run with--ignoredwhen datasets are cached): asserts invariants over the real datasets for the official catalog (fable-5, opus-4-8/4-7/4-6/4-5/4-1, sonnet-4-6/4-5, haiku-4-5, dated and provider forms) — resolved key carries the same family + major-minor version token (both4-8/4.8spellings accepted), price within ±25% of official, andclaude-2.x/bareclaudeare None. Passes locally.Verification
cargo fmt --all --checkclean;cargo clippy --all-targets -- -D warningscleancargo test -p tokscale-core: 914 passed (909 pre-existing + 5 new), 0 failed. One pre-existing assertion changed:test_is_fuzzy_eligibleassertedis_fuzzy_eligible("claude")— that line pinned the exact behavior bug 1 removes, so it now asserts the inverse (plusanthropic). No other existing test was touched.cargo test -p tokscale-core --test anthropic_catalog -- --ignored: passes against the local cachecargo test -p tokscale-cli: 778 passed, 0 failedFound by post-#634 catalog audit.
Summary by cubic
Fixes mispricing by blocking fuzzy matches on brand/generic tokens, preferring canonical
OpenRouter/LiteLLMprices, and honoring provider hints. Tests now flag any resolution of acceptable-None ids.Bug Fixes
claude,anthropic, and genericmodel/router; refuse strips that leave a bare brand. Stopsclaude-2.xmapping to modern models.LiteLLM/OpenRouter) beforemodels.dev; with a hint, run provider-scopedmodels.devbefore unscopedOpenRoutermodel-part and normalized fallbacks. SplitOpenRouterexact into full-key (wins even with a hint) vs model-part (hint overrides). No-match hints fall back to canonical.models.dev: deterministic provider choice (anthropic/first, then shorter key, then lexicographic); skip unpriced rows so zero-cost keys don’t shadow priced ones.Tests
claude-2.x/claude/anthropicresolve to None.OpenRoutermodel-part; updatedis_fuzzy_eligibleto assertclaude/anthropicare blocked.Written for commit 3a408b8. Summary will update on new commits.