Skip to content

fix(cli): skip expensive model warning for custom providers (Fixes #54348) - #54536

Closed
AlexFucuson9 wants to merge 1 commit into
NousResearch:mainfrom
AlexFucuson9:fix/custom-provider-skip-cost-guard
Closed

fix(cli): skip expensive model warning for custom providers (Fixes #54348)#54536
AlexFucuson9 wants to merge 1 commit into
NousResearch:mainfrom
AlexFucuson9:fix/custom-provider-skip-cost-guard

Conversation

@AlexFucuson9

Copy link
Copy Markdown
Contributor

Summary

Fixes #54348

Custom providers (e.g. custom:routerai) cannot switch models because the expensive_model_warning guard triggers on inaccurate pricing data from the models.dev catalog (which stores OpenRouter prices, not the custom provider's prices).

Root Cause

expensive_model_warning() in hermes_cli/model_cost_guard.py looks up pricing from models.dev or usage_pricing. For custom providers, the models.dev prices are for OpenRouter — wildly different from the custom provider's actual pricing. For example, deepseek-v4-pro costs $0.66/M on custom:routerai but models.dev shows OpenRouter's $61/M, triggering the warning.

When the warning fires, _apply_model_switch returns confirm_required: true, causing the desktop frontend to silently roll back the model change.

Fix

Skip the expensive model check entirely when provider starts with custom:. Custom providers have their own pricing that Hermes cannot know from external catalogs. Users who configure custom providers are already aware of their pricing.

Changed files

  • hermes_cli/model_cost_guard.py: Early return for custom providers (8 lines)

…usResearch#54348)

Custom providers (custom:xxx) use their own pricing that differs from
the models.dev catalog (which stores OpenRouter prices). Triggering
the expensive-model warning on inaccurate catalog data silently blocks
model switching. Skip the cost guard entirely for custom providers.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/cli CLI entry point, hermes_cli/, setup wizard area/billing Account usage, credit usage, billing (cross-cutting) labels Jun 29, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related: #54422 (earlier open fix for #54348) already covers this backend model_cost_guard.py change plus the frontend apps/desktop/use-model-controls.ts rollback handler — so it's the more complete fix. This PR is the narrow backend-only subset (skips the check for custom: providers). Maintainer to pick between the focused backend-only fix here and the full two-part fix in #54422.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for isolating the custom-provider cost-guard path.

Problems

  • hermes_cli/model_cost_guard.py:79 only skips custom:<name>. Current switching also supports a bare custom endpoint (hermes_cli/model_switch.py:1155, :1208), and hermes_cli/runtime_provider.py:623-632 supports a configured provider literally named custom; that route can still enter the cost guard.
  • The PR changes only hermes_cli/model_cost_guard.py; tests/hermes_cli/test_model_cost_guard.py has no custom-provider regression case.

Suggested changes

  • Cover both provider == "custom" and provider.startswith("custom:").
  • Add focused guard tests for both forms with expensive metadata, while preserving a warning assertion for a known provider.

Automated hermes-sweeper review.

# the expensive-model warning on inaccurate catalog data blocks model
# switching silently — skip the check entirely for custom providers.
# See #54348.
if provider and provider.startswith("custom:"):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This excludes only named custom:<name> providers. The current switch path also deliberately supports the bare custom endpoint (hermes_cli/model_switch.py:1155,1208), so it can still reach this guard; include provider == "custom" as well.

@teknium1 teknium1 added sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users labels Jul 15, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Closing in favor of PR #85951 (#85951), which fixed the same bug (#54348) via #54422 by @dpersek — submitted a day earlier and covering the same skip plus billing-route trust-gating for unknown providers generally, not only the custom: prefix.

Your diagnosis of the root cause (foreign models.dev pricing judged against custom-provider models) was exactly right — thanks for the report-quality writeup, @AlexFucuson9, and sorry this one was a duplicate.

@teknium1 teknium1 closed this Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/billing Account usage, credit usage, billing (cross-cutting) comp/cli CLI entry point, hermes_cli/, setup wizard P2 Medium — degraded but workaround exists sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cannot switch model via composer — expensive_model_warning blocks custom providers

3 participants