fix(pricing): resolve custom:<name> providers against models.dev - #60023
Open
robust-WJL wants to merge 3 commits into
Open
fix(pricing): resolve custom:<name> providers against models.dev#60023robust-WJL wants to merge 3 commits into
robust-WJL wants to merge 3 commits into
Conversation
A custom_providers entry is slugged "custom:<name>" (e.g. "custom:friendli"), which never matches a models.dev provider id. get_model_info() therefore missed the catalog and the expensive-model guard fell through to the per-endpoint pricing path. Friendli's /models reports pricing per *million* tokens while that path assumes per-token, so 1.4 -> $1,400,000/M tripped the guard on every Friendli model. Retry the lookup with the bare name when a "custom:" slug misses, so custom endpoints that mirror a known models.dev provider resolve to real pricing and short-circuit before the unreliable endpoint path.
alt-glitch
added
type/bug
Collaborator
Related to the custom-provider cost-guard cluster — but a different mechanism. #54536 / #54422 skip the expensive-model guard for |
Author
|
Hi Hermes team. Any progress on this PR? Let me know if you need any support from my side. |
teknium1
reviewed
Jul 15, 2026
teknium1
left a comment
teknium1
left a comment
Contributor
There was a problem hiding this comment.
Thanks for isolating the catalog-miss path; current main does take it: agent/models_dev.py:703-708 returns no metadata for custom:<name>, and hermes_cli/model_cost_guard.py:75-99 then falls back to endpoint pricing.
Problems
- The retry infers provider identity from a custom endpoint's display name alone.
custom_provider_slug()is derived only from that name (hermes_cli/providers.py:635-642), whileresolve_custom_provider()accepts an independent user-configured URL (hermes_cli/providers.py:667-695). An unrelated endpoint namedfriendliwould receive Friendli pricing, limits, and capabilities. - Catalog misses still reach
agent/usage_pricing.py:813-821, where_pricing_entry_from_metadata()unconditionally multiplies pricing by one million (agent/usage_pricing.py:776-785). The added tests cover metadata lookup but not the guard/fallback path.
Suggested changes
- Use an explicit, validated upstream-provider mapping (or endpoint validation) rather than name-only matching.
- Add a guard-level regression test covering both the mapped custom route and a catalog miss with endpoint pricing metadata.
Automated hermes-sweeper review.
|
|
||
| data = fetch_models_dev() | ||
| pdata = data.get(mdev_id) | ||
| if not isinstance(pdata, dict) and provider_id.startswith("custom:"): |
Contributor
There was a problem hiding this comment.
custom:<name> is derived from an arbitrary user-configured display name, not verified vendor identity (hermes_cli/providers.py:635-695). Retrying the bare name can assign catalog pricing and capabilities to an unrelated endpoint named friendli; please use an explicit/validated upstream-provider mapping instead.
teknium1
added
sweeper:risk-compatibility
…lay name The previous commit (7eb5703) resolved ``custom:<name>`` providers by retrying the catalog lookup with the bare name. That is name-only matching: ``custom_provider_slug`` is derived from the user-chosen display name while ``resolve_custom_provider`` accepts an independent base_url, so an unrelated endpoint named "friendli" pointed elsewhere would inherit Friendli pricing, limits and capabilities (NousResearch#60023 review). Replace it with host-validated resolution. A custom endpoint is mapped to a known models.dev provider only when its base_url host matches that vendor's API host (api.friendli.ai -> friendli), via upstream_provider_id_for_base_url; the cost guard consults that instead of the slug. A genuine Friendli endpoint thus resolves to the per-million models.dev catalog and short-circuits the unit-blind per-endpoint pricing path, so $1.4/M no longer becomes $1,400,000/M and trips the guard - the original integration error. An unrelated same-named endpoint does not validate and falls through to its own endpoint pricing as before. Tests cover both routes the review asked for: the mapped custom route (catalog per-million pricing, no spurious trip) and a catalog miss with endpoint pricing metadata (per-token -> per-million scaling preserved). Co-Authored-By: Claude <noreply@anthropic.com>
Author
|
Hi @teknium1 , thanks for the review. Just pushed a commit to address both of problems.
|
…a hardcoded host map
upstream_provider_id_for_base_url previously matched against a hand-maintained
{_UPSTREAM_PROVIDER_HOSTS} dict (api.friendli.ai -> friendli). That special-cased
Friendli and required a code change + release for every other per-million vendor.
Replace it with a host -> provider id index built lazily from
fetch_models_dev(): each provider's registered ``api`` base URL is canonicalized
to a hostname and, when that host uniquely identifies one provider, mapped to it.
Hosts shared by multiple providers (aggregators exposing many vendors) are left
unindexed so no single upstream identity is mis-assigned. The index tracks the
catalog automatically; the cost guard consults it unchanged.
A genuine Friendli endpoint (https://api.friendli.ai/serverless/v1) still
resolves to the per-million ``friendli`` catalog short-circuiting the unit-blind
endpoint path; an unrelated host and a shared aggregator host return None and
fall through as before. Verified against the live 167-provider registry.
Tests add the shared-host (aggregator) exclusion case and reset the lazy cache
between tests via _reset_upstream_provider_host_index_cache.
Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Selecting a
custom_providersmodel backed by Friendli (e.g.zai-org/GLM-5.2) trips the expensive-model guard with absurd pricing:Root cause
A
custom_providersentry is sluggedcustom:<name>(viacustom_provider_slug), sotarget_providerbecomes e.g.custom:friendli.get_model_info("custom:friendli", ...)never matches the models.dev catalog keyfriendli, so the guard's preferred (correct) models.dev lookup misses and falls through to the per-endpoint pricing path.That path (
_pricing_entry_from_metadata) assumes OpenRouter's per-token convention and multiplies by 1e6. Friendli's/v1/modelsreports pricing per million tokens (prompt: 1.4), so1.4 x 1_000_000 = $1,400,000/M— affecting every Friendli model.Fix
In
get_model_info, when acustom:<name>slug misses, retry the lookup with the bare<name>. Custom endpoints that mirror a known models.dev provider (Friendli, etc.) now resolve to real pricing and short-circuit before the unreliable endpoint path is ever reached. Minimal, additive, no behavior change for non-custom:providers.Verification
Reproduced end-to-end against a real config + live Friendli endpoint:
zai-org/GLM-5.2viacustom:friendli$1400000.00/Min,$4400000.00/Mout1.4 / 4.4from models.devTests:
scripts/run_tests.sh tests/agent/test_models_dev.py tests/hermes_cli/test_model_cost_guard.py tests/agent/test_usage_pricing.py→ 52 passed, 0 failed. Adds 2 invariant tests assertingcustom:friendliyields identical pricing tofriendli, and that an unknowncustom:slug still returns None.Note / follow-up
The underlying units bug in
_pricing_entry_from_metadata(per-million treated as per-token) is still latent for custom endpoints models.dev does not know about. This PR fixes the reported symptom by the smallest change; a magnitude-guard on that function would be a sensible defense-in-depth follow-up.