fix(cli): skip expensive model warning for custom providers (Fixes #54348) - #54536
fix(cli): skip expensive model warning for custom providers (Fixes #54348)#54536AlexFucuson9 wants to merge 1 commit into
Conversation
…usResearch#54348) Custom providers (custom:xxx) use their own pricing that differs from the models.dev catalog (which stores OpenRouter prices). Triggering the expensive-model warning on inaccurate catalog data silently blocks model switching. Skip the cost guard entirely for custom providers.
Related: #54422 (earlier open fix for #54348) already covers this backend |
teknium1
left a comment
There was a problem hiding this comment.
Thanks for isolating the custom-provider cost-guard path.
Problems
hermes_cli/model_cost_guard.py:79only skipscustom:<name>. Current switching also supports a barecustomendpoint (hermes_cli/model_switch.py:1155,:1208), andhermes_cli/runtime_provider.py:623-632supports a configured provider literally namedcustom; that route can still enter the cost guard.- The PR changes only
hermes_cli/model_cost_guard.py;tests/hermes_cli/test_model_cost_guard.pyhas no custom-provider regression case.
Suggested changes
- Cover both
provider == "custom"andprovider.startswith("custom:"). - Add focused guard tests for both forms with expensive metadata, while preserving a warning assertion for a known provider.
Automated hermes-sweeper review.
| # the expensive-model warning on inaccurate catalog data blocks model | ||
| # switching silently — skip the check entirely for custom providers. | ||
| # See #54348. | ||
| if provider and provider.startswith("custom:"): |
There was a problem hiding this comment.
This excludes only named custom:<name> providers. The current switch path also deliberately supports the bare custom endpoint (hermes_cli/model_switch.py:1155,1208), so it can still reach this guard; include provider == "custom" as well.
|
Closing in favor of PR #85951 (#85951), which fixed the same bug (#54348) via #54422 by @dpersek — submitted a day earlier and covering the same skip plus billing-route trust-gating for unknown providers generally, not only the Your diagnosis of the root cause (foreign models.dev pricing judged against custom-provider models) was exactly right — thanks for the report-quality writeup, @AlexFucuson9, and sorry this one was a duplicate. |
Summary
Fixes #54348
Custom providers (e.g.
custom:routerai) cannot switch models because theexpensive_model_warningguard triggers on inaccurate pricing data from the models.dev catalog (which stores OpenRouter prices, not the custom provider's prices).Root Cause
expensive_model_warning()inhermes_cli/model_cost_guard.pylooks up pricing from models.dev or usage_pricing. For custom providers, the models.dev prices are for OpenRouter — wildly different from the custom provider's actual pricing. For example,deepseek-v4-procosts $0.66/M oncustom:routeraibut models.dev shows OpenRouter's $61/M, triggering the warning.When the warning fires,
_apply_model_switchreturnsconfirm_required: true, causing the desktop frontend to silently roll back the model change.Fix
Skip the expensive model check entirely when
providerstarts withcustom:. Custom providers have their own pricing that Hermes cannot know from external catalogs. Users who configure custom providers are already aware of their pricing.Changed files
hermes_cli/model_cost_guard.py: Early return for custom providers (8 lines)