fix(pricing): parse modern Claude version from id instead of hardcoding - #634
Merged
Conversation
`normalize_model_name` matched each opus/sonnet/haiku minor version with a hardcoded branch and a catch-all `claude-opus-4`. Every new release (4.5 → 4.6 → 4.7 → 4.8) needed a new branch; a missing one let ids like `aws.claude-opus-4-8` fall through to the catch-all, which OpenRouter then resolved to legacy `anthropic/claude-opus-4` ($15/$75 per M instead of $5/$25) — a ~3x overcharge. This repeated for 4.7 (junhoyeo#580) and again for 4.8. The modern Claude line (major >= 4) follows the regular `claude-{family}-{major}[-{minor}]` scheme, so parse the version straight from the id and build the canonical key dynamically. New minor releases (4.9, 5.0, …) now resolve to their own pricing key with no code change — pricing values still come entirely from the upstream datasets. Boundary contract from the old matcher is preserved: `opus-4-60` → `claude-opus-4` (two-digit minor degrades to major), `opus-14-6` → None (two-digit major is not the modern line), undelimited `opus4`/`opus-4x` → None. The irregular legacy 3.x naming keeps its explicit branches. Tests: - test_normalize_future_minor_versions_resolve_without_hardcoding — 4.9/5.0 and cross-family (sonnet/haiku) parse without a hardcoded entry - test_normalize_modern_claude_boundaries — locks the boundary contract - 4.8 regression tests mirroring the 4.7 set (short/dot/aws forms + cost band) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Resolves the lookup.rs conflict by taking main's normalization region (opus-minor parsing, contains_delimited_major_minor guard) wholesale. The PR's generic parser is reworked on top of main's mechanisms in a follow-up commit instead of being merged as-is, because it degraded two-digit minors to the bare major and dropped reversed-order coverage.
…jors Generalize the opus-only version handling in PricingLookup so every Claude family (opus, sonnet, haiku, fable) gets the same guarantees: - normalize_claude_opus_4_minor -> normalize_claude_family_minor: family/major/minor parsed from the id in both orders (sonnet-4.7, claude-4-6-sonnet), so reversed-order sonnet/haiku ids resolve to their canonical key instead of cross-family fuzzy fallbacks. - normalize_claude_family_bare_major: bare modern majors (claude-opus-5, claude-sonnet-5, fable-5) normalize to a canonical key that resolves only via an exact dataset hit. - contains_delimited_modern_major_minor: the 4.x major-minor None-guard in normalize_model_name now covers majors 4-9, so 4-60 / 5-0 / dated forms stay unresolved instead of degrading. - resolves_unsafe_claude_version (was resolves_different_claude_opus_4_ minor): vetoes cross-family resolutions (bedrock sonnet ids billed as opus), cross-version resolutions for any family (sonnet-4-7 -> sonnet-4.6, haiku-4-6 -> haiku-4.5 / 3.5-haiku), and any modern-Claude resolution for ids whose version could not be parsed. Bare major 4 stays unpinned to preserve existing claude-opus-4 fuzzy behavior. Whether a version is "known" remains dataset-driven, exactly as it was for opus: the canonical key either exact-matches the pricing dataset or the id resolves unpriced. No hardcoded minor lists were added. Reworks PR junhoyeo#634 on top of main's mechanisms instead of merging its parallel parser. Constraint: unknown minors/majors must resolve unpriced, never to a cheaper or different price Constraint: all of main's existing pricing tests must pass unchanged Rejected: PR junhoyeo#634's original generic parser | degrades two-digit minors (opus-4-60 -> claude-opus-4) and drops reversed-order coverage Rejected: extending the hardcoded 4.5/4.6/4.7 allowlist in has_unrecognized_claude_four_minor | recreates the new-minor-release maintenance trap PR junhoyeo#634 set out to remove Rejected: pinning bare major 4 ids (claude-opus-4) to exact-only | changes long-standing dated/regional 4.x resolution covered by existing tests Confidence: high Scope-risk: moderate Directive: requested_claude_version deliberately unpins bare major 4; do not "complete" it to all majors without auditing dated-id (claude-opus-4-20250514) and regional-key resolution Not-tested: provider-hinted lookups (lookup_with_provider) against cross-family adversarial datasets; models.dev reseller keys that legitimately embed reversed-order ids (cortecs/claude-4-6-sonnet) are covered only via real-data probing
junhoyeo
approved these changes
Jun 10, 2026
junhoyeo
left a comment
junhoyeo
left a comment
Owner
There was a problem hiding this comment.
Reviewed: reworked on top of main's #614/#658 design — never-degrade contract now enforced across all Claude families and majors (unknown sonnet/haiku minors and major-5 ids resolve unpriced instead of degrading; cross-family fuzzy matches blocked by a family-token guard). Two of the four fixed defects reproduced against the real shipped dataset (haiku-4-6 → claude-3.5-haiku wrong-generation pricing; claude-opus-5 → opus-4.7-fast $30/M). 13 new tests incl. the PR's boundary suite adapted to the contract; all 143 pre-existing lookup tests pass unchanged; 909+17 crate tests green. Thanks @xczllgit for driving the generic-parsing direction. Merging.
junhoyeo
added a commit
that referenced
this pull request
Jun 14, 2026
…icing sources (#707) * fix(pricing): block brand-token fuzzy matches and prefer canonical sources A post-#634 audit of lookup.rs against the real Anthropic catalog found two mispricing bugs: 1. Retired `claude-2.1`/`claude-2.0` (in historical logs, absent from every dataset) eroded to bare `claude` in try_strip_unknown_suffix (the "2.1" segment failed the all-digits version check) and then fuzzy-matched `anthropic/claude-opus-4.7-fast` at $30/$150. Fixed by (a) generalizing the stripper guard's version-segment recognition to digits-with-optional-dot and refusing strips that erode a claude id to a bare brand token, and (b) blocklisting bare `claude` and `anthropic` from fuzzy matching outright. 2. `claude-opus-4-6-fast` resolved to Models.dev reseller `venice/claude-opus-4-6-fast` ($36/$180) instead of canonical `anthropic/claude-opus-4.6-fast` ($30/$150) because the models.dev model-part pass ran before the separator-normalized OpenRouter exact pass in lookup_auto. Fixed by reordering so canonical separator-normalized passes run first, with models.dev kept as the long-tail fallback; the models.dev model-part index now also picks a deterministic provider (anthropic/ first, then shorter key, then lexicographic) instead of HashMap-iteration-order roulette. Reorder safety diff over the real cached datasets (197 probed ids): only the intended resolutions changed (claude-2.x/claude/anthropic -> None; opus-4-6/4-7-fast -> canonical; identical-price namespace moves; one previously nondeterministic equal-length tie made deterministic). Constraint: ids absent from all datasets must resolve unpriced, never to another model's price Constraint: existing #634 lookup contract must hold (only the assertion pinning the bug itself changed: is_fuzzy_eligible("claude") is now false by design) Rejected: hardcoding claude-2.x ids | the guard must cover future family-less shapes Rejected: pure lexicographic provider choice for shared model parts | would silently move 161 model parts to different-priced providers; anthropic-then-shortest-then-lex preserves historical winners Confidence: high Scope-risk: moderate Directive: lookup_auto pass ordering is load-bearing — canonical (LiteLLM/OpenRouter) separator-normalized exact passes must stay ahead of the models.dev model-part pass Not-tested: providers whose models.dev keys differ only by case * fix(pricing): respect provider hints and priced-key preference in lookup fallbacks Addresses three #707 review findings: 1. Provider-hint bypass: the canonical-source reorder let the unscoped separator-normalized OpenRouter pass return anthropic/... before the provider-scoped models.dev pass could honor a hint like `venice`. Provider-scoped models.dev passes (raw and version-normalized) now run before the unscoped normalized fallback whenever a hint is present, so the reorder only preempts models.dev for unhinted lookups. 2. Bare-brand-token coverage: `anthropic` joins `claude`/`claude-2.x` in the must-resolve-to-None assertions (catalog invariant test and the in-module fuzzy-blocklist regression test). 3. Unpriced-key shadowing: the models.dev model-part index now skips entries without any usable pricing, so the anthropic-first preference cannot pick an unpriced anthropic/<model> row over a priced reseller row and bill usage at zero. The loader already guarantees priced entries (models_dev::cost_to_pricing), but new_with_models_dev is public, so the index guards itself. Constraint: unhinted resolutions must keep the canonical-first behavior the PR introduced (bug 2) Rejected: gating the unscoped raw exact litellm/openrouter passes on the hint too | pre-existing behavior outside this PR's regression, broader blast radius Rejected: filtering zero-cost (0.0) entries from the model-part index | 0.0 is deliberately valid pricing for free models (is_valid_price_value) Confidence: high Scope-risk: narrow Directive: provider-scoped passes must stay ahead of unscoped normalized fallbacks in lookup_auto; both new regression tests fail if either guard is removed Not-tested: hinted lookups where the hint matches multiple models.dev providers sharing a model part * fix(pricing): pin provider hints ahead of unscoped OpenRouter model-part dbec203 pinned provider-hinted lookups ahead of the separator-normalized fallbacks, but the unscoped OpenRouter exact pass still ran first and its model-part fallback could leak: for a dotted id like claude-opus-4.6-fast, exact_match_openrouter matches anthropic/claude-opus-4.6-fast by model-part and returns before the provider-scoped models.dev pass — so a venice-hinted lookup got the canonical price instead of venice's own key. The hyphenated form fell through correctly (its model-part is dotted), which is why dbec203's test missed this. Split exact_match_openrouter into _full_key (the id's own canonical key, which still wins under a hint) and _model_part (matches any provider sharing the model-part, which a hint must override). In lookup_auto the provider- scoped models.dev pass now runs between them. Unhinted lookups are unchanged (full_key.or(model_part) == the old combined match). Constraint: an exact full-key match is the id's own key and stays first even under a hint Constraint: a hint for a provider with no matching key must fall through to the canonical resolution, not None Rejected: gating the unscoped litellm/openrouter full-key passes on the hint too | full-key is the id's own key; broader blast radius for no correctness gain Confidence: high Scope-risk: narrow Directive: provider-scoped models.dev must stay between OpenRouter full-key and model-part passes in lookup_auto; the dotted-id regression test fails if it moves * fix(pricing): block generic "model"/"router" tokens from fuzzy match Harvesting the distinct model ids in real local session data surfaced three ids (`model-zero-usage-v1`, `model-nonzero-usage-v1`, `test-model`) that mispriced to `azure_ai/model_router` ($0.14/MTok). Root cause: after suffix/prefix stripping their only fuzzy-eligible remnant is the bare word `model`, which substring-matches the priced key `azure_ai/model_router` (and `router` matches `kilo/switchpoint/router`). These are generic English words carrying no model identity — the same defect class as the already- blocked bare brand tokens (`claude`, `anthropic`). Add `model` and `router` to FUZZY_BLOCKLIST so such ids stay unpriced (never-degrade) instead of billing at an unrelated key's rate. Exact-key matches (e.g. the real `azure/model-router` id) are unaffected. Constraint: must not change resolution of any real model id — before/after resolution over the full harvested local id set differs only on the three noise ids (now None). Rejected: raise MIN_FUZZY_MATCH_LEN | `model`/`router` are >=5 chars, would not be caught and longer generic words would still slip through. Confidence: high Scope-risk: narrow Not-tested: ids that strip to a generic word longer than `router` * test(pricing): drive catalog regression from real local model ids The catalog invariant test used a hardcoded ~16-id list. Replace its core with the DISTINCT model ids actually harvested from local session data (committed as tests/fixtures/local_model_ids.txt — model-id strings plus family/version/price-band expectations only, NO usage counts or user data). - anthropic_catalog_resolves_to_correct_family_version_and_price now drives every harvested claude-* id (regional/suffix/provider forms incl. vertex_ai/llmgateway/github-copilot variants) through the real cached datasets, asserting family token, major-minor version token, and price within +/-25%. A few hardcoded provider/regional forms remain as guards. - Add real_local_models_all_resolve_sanely: iterates the FULL harvested set (gpt/gemini/grok/glm/kimi/claude) and asserts each id resolves with a matching vendor token or is a documented acceptable-None, and that NO id resolves to a cross-family key. Both stay #[ignore]d (skip-graceful when cache absent). The generic-token misprice fix is covered by a non-ignored unit test in lookup.rs. Constraint: fixture must contain only model-id strings + pricing metadata, no usage counts / session ids / paths / user-identifying content. Confidence: high Scope-risk: narrow Directive: when adding a new model id to the fixture, audit its real resolution first — the version column uses the dashed spelling the matched KEY carries (e.g. claude-3-5 keys need "3-5", not "3.5"). * test(pricing): flag any acceptable-None resolution, not just input-priced The real-data catalog test's acceptable-None gate keyed off input_cost_per_token.is_some(), so a documented-None id resolving to a key with input: None but output: Some(..) would have passed silently. Mirror the sibling catalog test: flag ANY resolution for an acceptable-None id, and surface both input and output prices in the failure message. Issue identified by cubic. Confidence: high Scope-risk: narrow
pinion05
added a commit
to pinion05/tokscale
that referenced
this pull request
Jun 23, 2026
…ng (junhoyeo#634) * fix(pricing): parse modern Claude version from id instead of hardcoding `normalize_model_name` matched each opus/sonnet/haiku minor version with a hardcoded branch and a catch-all `claude-opus-4`. Every new release (4.5 → 4.6 → 4.7 → 4.8) needed a new branch; a missing one let ids like `aws.claude-opus-4-8` fall through to the catch-all, which OpenRouter then resolved to legacy `anthropic/claude-opus-4` ($15/$75 per M instead of $5/$25) — a ~3x overcharge. This repeated for 4.7 (junhoyeo#580) and again for 4.8. The modern Claude line (major >= 4) follows the regular `claude-{family}-{major}[-{minor}]` scheme, so parse the version straight from the id and build the canonical key dynamically. New minor releases (4.9, 5.0, …) now resolve to their own pricing key with no code change — pricing values still come entirely from the upstream datasets. Boundary contract from the old matcher is preserved: `opus-4-60` → `claude-opus-4` (two-digit minor degrades to major), `opus-14-6` → None (two-digit major is not the modern line), undelimited `opus4`/`opus-4x` → None. The irregular legacy 3.x naming keeps its explicit branches. Tests: - test_normalize_future_minor_versions_resolve_without_hardcoding — 4.9/5.0 and cross-family (sonnet/haiku) parse without a hardcoded entry - test_normalize_modern_claude_boundaries — locks the boundary contract - 4.8 regression tests mirroring the 4.7 set (short/dot/aws forms + cost band) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * style: wrap haiku condition to satisfy cargo fmt Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(pricing): enforce never-degrade across all Claude families and majors Generalize the opus-only version handling in PricingLookup so every Claude family (opus, sonnet, haiku, fable) gets the same guarantees: - normalize_claude_opus_4_minor -> normalize_claude_family_minor: family/major/minor parsed from the id in both orders (sonnet-4.7, claude-4-6-sonnet), so reversed-order sonnet/haiku ids resolve to their canonical key instead of cross-family fuzzy fallbacks. - normalize_claude_family_bare_major: bare modern majors (claude-opus-5, claude-sonnet-5, fable-5) normalize to a canonical key that resolves only via an exact dataset hit. - contains_delimited_modern_major_minor: the 4.x major-minor None-guard in normalize_model_name now covers majors 4-9, so 4-60 / 5-0 / dated forms stay unresolved instead of degrading. - resolves_unsafe_claude_version (was resolves_different_claude_opus_4_ minor): vetoes cross-family resolutions (bedrock sonnet ids billed as opus), cross-version resolutions for any family (sonnet-4-7 -> sonnet-4.6, haiku-4-6 -> haiku-4.5 / 3.5-haiku), and any modern-Claude resolution for ids whose version could not be parsed. Bare major 4 stays unpinned to preserve existing claude-opus-4 fuzzy behavior. Whether a version is "known" remains dataset-driven, exactly as it was for opus: the canonical key either exact-matches the pricing dataset or the id resolves unpriced. No hardcoded minor lists were added. Reworks PR junhoyeo#634 on top of main's mechanisms instead of merging its parallel parser. Constraint: unknown minors/majors must resolve unpriced, never to a cheaper or different price Constraint: all of main's existing pricing tests must pass unchanged Rejected: PR junhoyeo#634's original generic parser | degrades two-digit minors (opus-4-60 -> claude-opus-4) and drops reversed-order coverage Rejected: extending the hardcoded 4.5/4.6/4.7 allowlist in has_unrecognized_claude_four_minor | recreates the new-minor-release maintenance trap PR junhoyeo#634 set out to remove Rejected: pinning bare major 4 ids (claude-opus-4) to exact-only | changes long-standing dated/regional 4.x resolution covered by existing tests Confidence: high Scope-risk: moderate Directive: requested_claude_version deliberately unpins bare major 4; do not "complete" it to all majors without auditing dated-id (claude-opus-4-20250514) and regional-key resolution Not-tested: provider-hinted lookups (lookup_with_provider) against cross-family adversarial datasets; models.dev reseller keys that legitimately embed reversed-order ids (cortecs/claude-4-6-sonnet) are covered only via real-data probing --------- Co-authored-by: xiaochaozheng <xiaochaozheng@meituan.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Junho Yeo <i@junho.io>
pinion05
added a commit
to pinion05/tokscale
that referenced
this pull request
Jun 23, 2026
…icing sources (junhoyeo#707) * fix(pricing): block brand-token fuzzy matches and prefer canonical sources A post-junhoyeo#634 audit of lookup.rs against the real Anthropic catalog found two mispricing bugs: 1. Retired `claude-2.1`/`claude-2.0` (in historical logs, absent from every dataset) eroded to bare `claude` in try_strip_unknown_suffix (the "2.1" segment failed the all-digits version check) and then fuzzy-matched `anthropic/claude-opus-4.7-fast` at $30/$150. Fixed by (a) generalizing the stripper guard's version-segment recognition to digits-with-optional-dot and refusing strips that erode a claude id to a bare brand token, and (b) blocklisting bare `claude` and `anthropic` from fuzzy matching outright. 2. `claude-opus-4-6-fast` resolved to Models.dev reseller `venice/claude-opus-4-6-fast` ($36/$180) instead of canonical `anthropic/claude-opus-4.6-fast` ($30/$150) because the models.dev model-part pass ran before the separator-normalized OpenRouter exact pass in lookup_auto. Fixed by reordering so canonical separator-normalized passes run first, with models.dev kept as the long-tail fallback; the models.dev model-part index now also picks a deterministic provider (anthropic/ first, then shorter key, then lexicographic) instead of HashMap-iteration-order roulette. Reorder safety diff over the real cached datasets (197 probed ids): only the intended resolutions changed (claude-2.x/claude/anthropic -> None; opus-4-6/4-7-fast -> canonical; identical-price namespace moves; one previously nondeterministic equal-length tie made deterministic). Constraint: ids absent from all datasets must resolve unpriced, never to another model's price Constraint: existing junhoyeo#634 lookup contract must hold (only the assertion pinning the bug itself changed: is_fuzzy_eligible("claude") is now false by design) Rejected: hardcoding claude-2.x ids | the guard must cover future family-less shapes Rejected: pure lexicographic provider choice for shared model parts | would silently move 161 model parts to different-priced providers; anthropic-then-shortest-then-lex preserves historical winners Confidence: high Scope-risk: moderate Directive: lookup_auto pass ordering is load-bearing — canonical (LiteLLM/OpenRouter) separator-normalized exact passes must stay ahead of the models.dev model-part pass Not-tested: providers whose models.dev keys differ only by case * fix(pricing): respect provider hints and priced-key preference in lookup fallbacks Addresses three junhoyeo#707 review findings: 1. Provider-hint bypass: the canonical-source reorder let the unscoped separator-normalized OpenRouter pass return anthropic/... before the provider-scoped models.dev pass could honor a hint like `venice`. Provider-scoped models.dev passes (raw and version-normalized) now run before the unscoped normalized fallback whenever a hint is present, so the reorder only preempts models.dev for unhinted lookups. 2. Bare-brand-token coverage: `anthropic` joins `claude`/`claude-2.x` in the must-resolve-to-None assertions (catalog invariant test and the in-module fuzzy-blocklist regression test). 3. Unpriced-key shadowing: the models.dev model-part index now skips entries without any usable pricing, so the anthropic-first preference cannot pick an unpriced anthropic/<model> row over a priced reseller row and bill usage at zero. The loader already guarantees priced entries (models_dev::cost_to_pricing), but new_with_models_dev is public, so the index guards itself. Constraint: unhinted resolutions must keep the canonical-first behavior the PR introduced (bug 2) Rejected: gating the unscoped raw exact litellm/openrouter passes on the hint too | pre-existing behavior outside this PR's regression, broader blast radius Rejected: filtering zero-cost (0.0) entries from the model-part index | 0.0 is deliberately valid pricing for free models (is_valid_price_value) Confidence: high Scope-risk: narrow Directive: provider-scoped passes must stay ahead of unscoped normalized fallbacks in lookup_auto; both new regression tests fail if either guard is removed Not-tested: hinted lookups where the hint matches multiple models.dev providers sharing a model part * fix(pricing): pin provider hints ahead of unscoped OpenRouter model-part dbec203 pinned provider-hinted lookups ahead of the separator-normalized fallbacks, but the unscoped OpenRouter exact pass still ran first and its model-part fallback could leak: for a dotted id like claude-opus-4.6-fast, exact_match_openrouter matches anthropic/claude-opus-4.6-fast by model-part and returns before the provider-scoped models.dev pass — so a venice-hinted lookup got the canonical price instead of venice's own key. The hyphenated form fell through correctly (its model-part is dotted), which is why dbec203's test missed this. Split exact_match_openrouter into _full_key (the id's own canonical key, which still wins under a hint) and _model_part (matches any provider sharing the model-part, which a hint must override). In lookup_auto the provider- scoped models.dev pass now runs between them. Unhinted lookups are unchanged (full_key.or(model_part) == the old combined match). Constraint: an exact full-key match is the id's own key and stays first even under a hint Constraint: a hint for a provider with no matching key must fall through to the canonical resolution, not None Rejected: gating the unscoped litellm/openrouter full-key passes on the hint too | full-key is the id's own key; broader blast radius for no correctness gain Confidence: high Scope-risk: narrow Directive: provider-scoped models.dev must stay between OpenRouter full-key and model-part passes in lookup_auto; the dotted-id regression test fails if it moves * fix(pricing): block generic "model"/"router" tokens from fuzzy match Harvesting the distinct model ids in real local session data surfaced three ids (`model-zero-usage-v1`, `model-nonzero-usage-v1`, `test-model`) that mispriced to `azure_ai/model_router` ($0.14/MTok). Root cause: after suffix/prefix stripping their only fuzzy-eligible remnant is the bare word `model`, which substring-matches the priced key `azure_ai/model_router` (and `router` matches `kilo/switchpoint/router`). These are generic English words carrying no model identity — the same defect class as the already- blocked bare brand tokens (`claude`, `anthropic`). Add `model` and `router` to FUZZY_BLOCKLIST so such ids stay unpriced (never-degrade) instead of billing at an unrelated key's rate. Exact-key matches (e.g. the real `azure/model-router` id) are unaffected. Constraint: must not change resolution of any real model id — before/after resolution over the full harvested local id set differs only on the three noise ids (now None). Rejected: raise MIN_FUZZY_MATCH_LEN | `model`/`router` are >=5 chars, would not be caught and longer generic words would still slip through. Confidence: high Scope-risk: narrow Not-tested: ids that strip to a generic word longer than `router` * test(pricing): drive catalog regression from real local model ids The catalog invariant test used a hardcoded ~16-id list. Replace its core with the DISTINCT model ids actually harvested from local session data (committed as tests/fixtures/local_model_ids.txt — model-id strings plus family/version/price-band expectations only, NO usage counts or user data). - anthropic_catalog_resolves_to_correct_family_version_and_price now drives every harvested claude-* id (regional/suffix/provider forms incl. vertex_ai/llmgateway/github-copilot variants) through the real cached datasets, asserting family token, major-minor version token, and price within +/-25%. A few hardcoded provider/regional forms remain as guards. - Add real_local_models_all_resolve_sanely: iterates the FULL harvested set (gpt/gemini/grok/glm/kimi/claude) and asserts each id resolves with a matching vendor token or is a documented acceptable-None, and that NO id resolves to a cross-family key. Both stay #[ignore]d (skip-graceful when cache absent). The generic-token misprice fix is covered by a non-ignored unit test in lookup.rs. Constraint: fixture must contain only model-id strings + pricing metadata, no usage counts / session ids / paths / user-identifying content. Confidence: high Scope-risk: narrow Directive: when adding a new model id to the fixture, audit its real resolution first — the version column uses the dashed spelling the matched KEY carries (e.g. claude-3-5 keys need "3-5", not "3.5"). * test(pricing): flag any acceptable-None resolution, not just input-priced The real-data catalog test's acceptable-None gate keyed off input_cost_per_token.is_some(), so a documented-None id resolving to a key with input: None but output: Some(..) would have passed silently. Mirror the sibling catalog test: flag ANY resolution for an acceptable-None id, and surface both input and output prices in the failure message. Issue identified by cubic. Confidence: high Scope-risk: narrow
t1000040
pushed a commit
to tmobi-internal/tokscale
that referenced
this pull request
Jun 30, 2026
…ng (junhoyeo#634) * fix(pricing): parse modern Claude version from id instead of hardcoding `normalize_model_name` matched each opus/sonnet/haiku minor version with a hardcoded branch and a catch-all `claude-opus-4`. Every new release (4.5 → 4.6 → 4.7 → 4.8) needed a new branch; a missing one let ids like `aws.claude-opus-4-8` fall through to the catch-all, which OpenRouter then resolved to legacy `anthropic/claude-opus-4` ($15/$75 per M instead of $5/$25) — a ~3x overcharge. This repeated for 4.7 (junhoyeo#580) and again for 4.8. The modern Claude line (major >= 4) follows the regular `claude-{family}-{major}[-{minor}]` scheme, so parse the version straight from the id and build the canonical key dynamically. New minor releases (4.9, 5.0, …) now resolve to their own pricing key with no code change — pricing values still come entirely from the upstream datasets. Boundary contract from the old matcher is preserved: `opus-4-60` → `claude-opus-4` (two-digit minor degrades to major), `opus-14-6` → None (two-digit major is not the modern line), undelimited `opus4`/`opus-4x` → None. The irregular legacy 3.x naming keeps its explicit branches. Tests: - test_normalize_future_minor_versions_resolve_without_hardcoding — 4.9/5.0 and cross-family (sonnet/haiku) parse without a hardcoded entry - test_normalize_modern_claude_boundaries — locks the boundary contract - 4.8 regression tests mirroring the 4.7 set (short/dot/aws forms + cost band) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * style: wrap haiku condition to satisfy cargo fmt Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(pricing): enforce never-degrade across all Claude families and majors Generalize the opus-only version handling in PricingLookup so every Claude family (opus, sonnet, haiku, fable) gets the same guarantees: - normalize_claude_opus_4_minor -> normalize_claude_family_minor: family/major/minor parsed from the id in both orders (sonnet-4.7, claude-4-6-sonnet), so reversed-order sonnet/haiku ids resolve to their canonical key instead of cross-family fuzzy fallbacks. - normalize_claude_family_bare_major: bare modern majors (claude-opus-5, claude-sonnet-5, fable-5) normalize to a canonical key that resolves only via an exact dataset hit. - contains_delimited_modern_major_minor: the 4.x major-minor None-guard in normalize_model_name now covers majors 4-9, so 4-60 / 5-0 / dated forms stay unresolved instead of degrading. - resolves_unsafe_claude_version (was resolves_different_claude_opus_4_ minor): vetoes cross-family resolutions (bedrock sonnet ids billed as opus), cross-version resolutions for any family (sonnet-4-7 -> sonnet-4.6, haiku-4-6 -> haiku-4.5 / 3.5-haiku), and any modern-Claude resolution for ids whose version could not be parsed. Bare major 4 stays unpinned to preserve existing claude-opus-4 fuzzy behavior. Whether a version is "known" remains dataset-driven, exactly as it was for opus: the canonical key either exact-matches the pricing dataset or the id resolves unpriced. No hardcoded minor lists were added. Reworks PR junhoyeo#634 on top of main's mechanisms instead of merging its parallel parser. Constraint: unknown minors/majors must resolve unpriced, never to a cheaper or different price Constraint: all of main's existing pricing tests must pass unchanged Rejected: PR junhoyeo#634's original generic parser | degrades two-digit minors (opus-4-60 -> claude-opus-4) and drops reversed-order coverage Rejected: extending the hardcoded 4.5/4.6/4.7 allowlist in has_unrecognized_claude_four_minor | recreates the new-minor-release maintenance trap PR junhoyeo#634 set out to remove Rejected: pinning bare major 4 ids (claude-opus-4) to exact-only | changes long-standing dated/regional 4.x resolution covered by existing tests Confidence: high Scope-risk: moderate Directive: requested_claude_version deliberately unpins bare major 4; do not "complete" it to all majors without auditing dated-id (claude-opus-4-20250514) and regional-key resolution Not-tested: provider-hinted lookups (lookup_with_provider) against cross-family adversarial datasets; models.dev reseller keys that legitimately embed reversed-order ids (cortecs/claude-4-6-sonnet) are covered only via real-data probing --------- Co-authored-by: xiaochaozheng <xiaochaozheng@meituan.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Junho Yeo <i@junho.io>
t1000040
pushed a commit
to tmobi-internal/tokscale
that referenced
this pull request
Jun 30, 2026
…icing sources (junhoyeo#707) * fix(pricing): block brand-token fuzzy matches and prefer canonical sources A post-junhoyeo#634 audit of lookup.rs against the real Anthropic catalog found two mispricing bugs: 1. Retired `claude-2.1`/`claude-2.0` (in historical logs, absent from every dataset) eroded to bare `claude` in try_strip_unknown_suffix (the "2.1" segment failed the all-digits version check) and then fuzzy-matched `anthropic/claude-opus-4.7-fast` at $30/$150. Fixed by (a) generalizing the stripper guard's version-segment recognition to digits-with-optional-dot and refusing strips that erode a claude id to a bare brand token, and (b) blocklisting bare `claude` and `anthropic` from fuzzy matching outright. 2. `claude-opus-4-6-fast` resolved to Models.dev reseller `venice/claude-opus-4-6-fast` ($36/$180) instead of canonical `anthropic/claude-opus-4.6-fast` ($30/$150) because the models.dev model-part pass ran before the separator-normalized OpenRouter exact pass in lookup_auto. Fixed by reordering so canonical separator-normalized passes run first, with models.dev kept as the long-tail fallback; the models.dev model-part index now also picks a deterministic provider (anthropic/ first, then shorter key, then lexicographic) instead of HashMap-iteration-order roulette. Reorder safety diff over the real cached datasets (197 probed ids): only the intended resolutions changed (claude-2.x/claude/anthropic -> None; opus-4-6/4-7-fast -> canonical; identical-price namespace moves; one previously nondeterministic equal-length tie made deterministic). Constraint: ids absent from all datasets must resolve unpriced, never to another model's price Constraint: existing junhoyeo#634 lookup contract must hold (only the assertion pinning the bug itself changed: is_fuzzy_eligible("claude") is now false by design) Rejected: hardcoding claude-2.x ids | the guard must cover future family-less shapes Rejected: pure lexicographic provider choice for shared model parts | would silently move 161 model parts to different-priced providers; anthropic-then-shortest-then-lex preserves historical winners Confidence: high Scope-risk: moderate Directive: lookup_auto pass ordering is load-bearing — canonical (LiteLLM/OpenRouter) separator-normalized exact passes must stay ahead of the models.dev model-part pass Not-tested: providers whose models.dev keys differ only by case * fix(pricing): respect provider hints and priced-key preference in lookup fallbacks Addresses three junhoyeo#707 review findings: 1. Provider-hint bypass: the canonical-source reorder let the unscoped separator-normalized OpenRouter pass return anthropic/... before the provider-scoped models.dev pass could honor a hint like `venice`. Provider-scoped models.dev passes (raw and version-normalized) now run before the unscoped normalized fallback whenever a hint is present, so the reorder only preempts models.dev for unhinted lookups. 2. Bare-brand-token coverage: `anthropic` joins `claude`/`claude-2.x` in the must-resolve-to-None assertions (catalog invariant test and the in-module fuzzy-blocklist regression test). 3. Unpriced-key shadowing: the models.dev model-part index now skips entries without any usable pricing, so the anthropic-first preference cannot pick an unpriced anthropic/<model> row over a priced reseller row and bill usage at zero. The loader already guarantees priced entries (models_dev::cost_to_pricing), but new_with_models_dev is public, so the index guards itself. Constraint: unhinted resolutions must keep the canonical-first behavior the PR introduced (bug 2) Rejected: gating the unscoped raw exact litellm/openrouter passes on the hint too | pre-existing behavior outside this PR's regression, broader blast radius Rejected: filtering zero-cost (0.0) entries from the model-part index | 0.0 is deliberately valid pricing for free models (is_valid_price_value) Confidence: high Scope-risk: narrow Directive: provider-scoped passes must stay ahead of unscoped normalized fallbacks in lookup_auto; both new regression tests fail if either guard is removed Not-tested: hinted lookups where the hint matches multiple models.dev providers sharing a model part * fix(pricing): pin provider hints ahead of unscoped OpenRouter model-part dbec203 pinned provider-hinted lookups ahead of the separator-normalized fallbacks, but the unscoped OpenRouter exact pass still ran first and its model-part fallback could leak: for a dotted id like claude-opus-4.6-fast, exact_match_openrouter matches anthropic/claude-opus-4.6-fast by model-part and returns before the provider-scoped models.dev pass — so a venice-hinted lookup got the canonical price instead of venice's own key. The hyphenated form fell through correctly (its model-part is dotted), which is why dbec203's test missed this. Split exact_match_openrouter into _full_key (the id's own canonical key, which still wins under a hint) and _model_part (matches any provider sharing the model-part, which a hint must override). In lookup_auto the provider- scoped models.dev pass now runs between them. Unhinted lookups are unchanged (full_key.or(model_part) == the old combined match). Constraint: an exact full-key match is the id's own key and stays first even under a hint Constraint: a hint for a provider with no matching key must fall through to the canonical resolution, not None Rejected: gating the unscoped litellm/openrouter full-key passes on the hint too | full-key is the id's own key; broader blast radius for no correctness gain Confidence: high Scope-risk: narrow Directive: provider-scoped models.dev must stay between OpenRouter full-key and model-part passes in lookup_auto; the dotted-id regression test fails if it moves * fix(pricing): block generic "model"/"router" tokens from fuzzy match Harvesting the distinct model ids in real local session data surfaced three ids (`model-zero-usage-v1`, `model-nonzero-usage-v1`, `test-model`) that mispriced to `azure_ai/model_router` ($0.14/MTok). Root cause: after suffix/prefix stripping their only fuzzy-eligible remnant is the bare word `model`, which substring-matches the priced key `azure_ai/model_router` (and `router` matches `kilo/switchpoint/router`). These are generic English words carrying no model identity — the same defect class as the already- blocked bare brand tokens (`claude`, `anthropic`). Add `model` and `router` to FUZZY_BLOCKLIST so such ids stay unpriced (never-degrade) instead of billing at an unrelated key's rate. Exact-key matches (e.g. the real `azure/model-router` id) are unaffected. Constraint: must not change resolution of any real model id — before/after resolution over the full harvested local id set differs only on the three noise ids (now None). Rejected: raise MIN_FUZZY_MATCH_LEN | `model`/`router` are >=5 chars, would not be caught and longer generic words would still slip through. Confidence: high Scope-risk: narrow Not-tested: ids that strip to a generic word longer than `router` * test(pricing): drive catalog regression from real local model ids The catalog invariant test used a hardcoded ~16-id list. Replace its core with the DISTINCT model ids actually harvested from local session data (committed as tests/fixtures/local_model_ids.txt — model-id strings plus family/version/price-band expectations only, NO usage counts or user data). - anthropic_catalog_resolves_to_correct_family_version_and_price now drives every harvested claude-* id (regional/suffix/provider forms incl. vertex_ai/llmgateway/github-copilot variants) through the real cached datasets, asserting family token, major-minor version token, and price within +/-25%. A few hardcoded provider/regional forms remain as guards. - Add real_local_models_all_resolve_sanely: iterates the FULL harvested set (gpt/gemini/grok/glm/kimi/claude) and asserts each id resolves with a matching vendor token or is a documented acceptable-None, and that NO id resolves to a cross-family key. Both stay #[ignore]d (skip-graceful when cache absent). The generic-token misprice fix is covered by a non-ignored unit test in lookup.rs. Constraint: fixture must contain only model-id strings + pricing metadata, no usage counts / session ids / paths / user-identifying content. Confidence: high Scope-risk: narrow Directive: when adding a new model id to the fixture, audit its real resolution first — the version column uses the dashed spelling the matched KEY carries (e.g. claude-3-5 keys need "3-5", not "3.5"). * test(pricing): flag any acceptable-None resolution, not just input-priced The real-data catalog test's acceptable-None gate keyed off input_cost_per_token.is_some(), so a documented-None id resolving to a key with input: None but output: Some(..) would have passed silently. Mirror the sibling catalog test: flag ANY resolution for an acceptable-None id, and surface both input and output prices in the failure message. Issue identified by cubic. Confidence: high Scope-risk: narrow
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
normalize_model_namematched each opus/sonnet/haiku minor version with a hardcoded branch plus a catch-allclaude-opus-4. Every new release (4.5 → 4.6 → 4.7 → 4.8) required a new branch, and a missing one let provider-prefixed ids likeaws.claude-opus-4-8fall through to the catch-all — which OpenRouter then resolved to legacyanthropic/claude-opus-4($15/$75 per M instead of $5/$25), a ~3x overcharge. This already happened for 4.7 (#580) and recurred for 4.8.This PR parses the version straight from the id for the modern Claude line (major >= 4), which follows the regular
claude-{family}-{major}[-{minor}]scheme, and builds the canonical key dynamically. Future minor releases (4.9, 5.0, …) resolve to their own pricing key with no code change — pricing values still come entirely from the upstream LiteLLM/OpenRouter datasets.opus-4-60→claude-opus-4(two-digit minor degrades to major)opus-14-6→None(two-digit major is not the modern line)opus4/opus-4x→Noneclaude-3-7-sonnet,claude-3.5-haiku) keeps its explicit branches.Before / after (real dataset, which already contains
claude-opus-4-8)aws.claude-opus-4-8anthropic/claude-opus-4— $15/$75claude-opus-4-8— $5/$25opus-4.8claude-opus-4-8— $5/$25claude-opus-4-8Verified end-to-end via
tokscale pricing <id>with the fix stashed/unstashed to confirm the regression.Tests
test_normalize_future_minor_versions_resolve_without_hardcoding— 4.9/5.0 and cross-family (sonnet/haiku) parse without a hardcoded entrytest_normalize_modern_claude_boundaries— locks the boundary contract abovecargo test -p tokscale-core --lib pricing::→ 206 passed; clippy cleanTest plan
cargo test -p tokscale-core --lib pricing::cargo clippy -p tokscale-core --libcargo run -p tokscale-cli -- pricing aws.claude-opus-4-8showsclaude-opus-4-8at $5/$25🤖 Generated with Claude Code
Summary by cubic
Parse modern Claude model versions from ids across families and majors, and block cross-family/cross-version fallbacks. Fixes
aws.claude-opus-4-8routing and makes future 4.x/5.x resolve to the correct pricing key only when priced.opus/sonnet/haiku/fable(major ≥ 4), supporting dashed/dotted and reversed forms (e.g.,claude-4-6-sonnet).4-8/4.8); never degrade unparseable or out-of-shape versions (4-60,5-0, dated) → None.claude-opus-5,claude-sonnet-5) but resolve only via exact dataset hits; keep bare major 4’s existing fuzzy behavior.opus4,opus-4x,opus-14-6).Written for commit 0603dc6. Summary will update on new commits.