Skip to content

feat: detect and warn when curated OpenRouter models are removed from live API - #57642

Closed
lincoln-mackay wants to merge 2 commits into
NousResearch:mainfrom
lincoln-mackay:fix/openrouter-removed-model-detection
Closed

feat: detect and warn when curated OpenRouter models are removed from live API#57642
lincoln-mackay wants to merge 2 commits into
NousResearch:mainfrom
lincoln-mackay:fix/openrouter-removed-model-detection

Conversation

@lincoln-mackay

Copy link
Copy Markdown

Summary

already cross-references the curated catalog against OpenRouter's live /v1/models response, but silently skipped models that OpenRouter had removed. This PR adds explicit detection and reporting.

Changes

hermes_cli/models.py (+26 lines)

  1. Log warning when a curated model ID is absent from the live API — fires once per model per process lifetime (deduped by _openrouter_removed_models set)
  2. get_removed_openrouter_models() public accessor returns frozenset[str] of detected removed models for programmatic inspection (WebUI, status checks)

Verified live

On a force_refresh=True call against the current OpenRouter API, detected:

Model Status
openrouter/owl-alpha Removed ✅
openrouter/elephant-alpha Removed ✅
google/gemini-3-pro-preview Removed ✅
tencent/hy3-preview:free Removed ✅
inclusionai/ring-2.6-1t:free Removed ✅

These models were in the curated OPENROUTER_MODELS / remote catalog manifest but absent from OpenRouter's /v1/models. They are automatically excluded from the picker (existing behaviour — no code change needed for that), and now a logger.warning() explains why.

Testing

All 84 existing tests/hermes_cli/test_models.py tests pass.
Syntax-validated via ast.parse.
Import-verified: from hermes_cli.models import fetch_openrouter_models, get_removed_openrouter_models works cleanly.

Adds out-of-band detection for OpenRouter models that are present in the
curated catalog but no longer served by OpenRouter's /v1/models endpoint.

Changes:
- logger.warning() on first detection of each removed model, deduped
  across rebuilds via _openrouter_removed_models set
- get_removed_openrouter_models() public accessor for programmatic
  inspection (WebUI, status checks)
- At cold rebuild time, if the curated model IDs union includes models
  OpenRouter has dropped, they are silently removed from the picker
  and a clear warning is emitted

Verified live against OpenRouter's current API — caught 5 delisted
models: openrouter/owl-alpha, openrouter/elephant-alpha,
google/gemini-3-pro-preview, tencent/hy3-preview:free,
inclusionai/ring-2.6-1t:free
@alt-glitch alt-glitch added type/feature New feature or request comp/cli CLI entry point, hermes_cli/, setup wizard provider/openrouter OpenRouter aggregator P3 Low — cosmetic, nice to have labels Jul 3, 2026
When OpenRouter returns a 404 billing error ('This model is unavailable for
free. The paid version is available now - use this slug instead: <model>'),
the agent now:

1. Correctly classifies it as billing (error_classifier.py _BILLING_PATTERNS)
   - Adds 'unavailable for free', 'paid version', 'use this slug instead'
   - Also updates auxiliary_client.py _is_payment_error for parity

2. Tries the OpenRouter-suggested slug first by parsing 'use this slug
   instead: <slug>' from the error body

3. Falls back to auto-discovering the best free model from the live
   OpenRouter curated catalog (hermes_cli/models.py):
   - find_free_openrouter_model() — fetches live list, prefers :free models
   - parse_openrouter_slug_suggestion() — extracts slug from error text

4. Swaps the agent's model + client in-place and retries (conversation_loop.py)
   - Gated by TurnRetryState.openrouter_free_tier_selfheal_attempted
   - Fires at most once per turn, on billing 404s for OpenRouter/Nous

Integrates with the removed-model detection (PR NousResearch#57642): the auto-discovery
calls fetch_openrouter_models(force_refresh=True), which now logs removed
models as they're detected.
@lincoln-mackay

Copy link
Copy Markdown
Author

Expanded scope: added auto-resolution for free-tier model errors

This PR now also includes the free-tier self-heal feature — when OpenRouter returns a 404 with This model is unavailable for free. The paid version is available now - use this slug instead: <slug>, the agent:

  1. Correctly classifies it as billing (error_classifier.py + auxiliary_client.py parity)
  2. Parses the suggested slug from the error body (parse_openrouter_slug_suggestion)
  3. Auto-discovers the best free model (find_free_openrouter_model — queries live curated catalog, prefers :free suffixed models)
  4. Swaps model + client in-place and retries, gated by TurnRetryState.openrouter_free_tier_selfheal_attempted

Files changed (138 net lines)

File Change
hermes_cli/models.py +52: removed-model detection (prev commit), find_free_openrouter_model, parse_openrouter_slug_suggestion
agent/error_classifier.py +7-2: added "unavailable for free", "paid version", "use this slug instead" to _BILLING_PATTERNS
agent/auxiliary_client.py +3: same patterns in _is_payment_error for parity
agent/turn_retry_state.py +1: openrouter_free_tier_selfheal_attempted guard
agent/conversation_loop.py +76: self-heal block in the error-handling loop
tests/agent/test_turn_retry_state.py +1: contract test update

Verified

  • All 84 tests/hermes_cli/test_models.py pass
  • All 4 tests/agent/test_turn_retry_state.py pass (contract + guards)
  • find_free_openrouter_model(force_refresh=True) returns poolside/laguna-m.1:free (live result)
  • parse_openrouter_slug_suggestion() correctly extracts deepseek/deepseek-v4-flash from the real OpenRouter error format

@teknium1

Copy link
Copy Markdown
Contributor

Thanks for identifying the silent curated-catalog omission: current main does silently skip absent live IDs in hermes_cli/models.py:1481-1484, so the warning portion has a valid premise.

Problems

  • find_free_openrouter_model() falls back to the first catalog entry when no free entry is found. Existing coverage shows that a first/recommended entry can have nonzero pricing (tests/hermes_cli/test_models.py:69-80), so this can switch a free-tier failure to a paid model.
  • The new recovery assigns agent.model before resolve_provider_client() succeeds. If resolution fails, the model changes while the old client remains installed; retry kwargs read agent.model (agent/chat_completion_helpers.py:1041-1045).
  • The PR only updates the TurnRetryState field contract (tests/agent/test_turn_retry_state.py:44-48); it has no behavioral coverage for warning detection, free-model selection, or replacement-client failure.

Suggested changes

  • Split or retain the removed-model warning with deterministic catalog-response tests.
  • Select only a live, tool-capable model explicitly verified free; do not fall back to a recommended/paid entry.
  • Build the replacement client first, then atomically update runtime state only on success, with regression tests for both success and failure.

This is an automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-caching Sweeper risk: may break/degrade prompt caching or cache-key stability (invariant) sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 15, 2026
@lincoln-mackay

Copy link
Copy Markdown
Author

THIS FIX HAS BEEN MERGED INTO PR #6239

Thanks all for your contributions to this fix. To streamline and remove duplication, I consolidated the fixes from PRs:

#5205 (gateway slash command validation) - Now part of #6239
#54895 (fallback indicator UI) - Now part of #6239
#57642 (OpenRouter model detection) - Now part of #6239

All functionality is now available in single, unified PR #6239 which provides:
✅ Silent provider failure retry with exponential backoff
✅ Profile-aware model validation
✅ Gateway infrastructure integration
✅ Enhanced error recovery and user feedback

Hopefully the consolidated fix eliminates overhead while preserving all of the fixes & updates

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard P3 Low — cosmetic, nice to have provider/openrouter OpenRouter aggregator sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-caching Sweeper risk: may break/degrade prompt caching or cache-key stability (invariant) sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants