feat: detect and warn when curated OpenRouter models are removed from live API - #57642
feat: detect and warn when curated OpenRouter models are removed from live API#57642lincoln-mackay wants to merge 2 commits into
Conversation
Adds out-of-band detection for OpenRouter models that are present in the curated catalog but no longer served by OpenRouter's /v1/models endpoint. Changes: - logger.warning() on first detection of each removed model, deduped across rebuilds via _openrouter_removed_models set - get_removed_openrouter_models() public accessor for programmatic inspection (WebUI, status checks) - At cold rebuild time, if the curated model IDs union includes models OpenRouter has dropped, they are silently removed from the picker and a clear warning is emitted Verified live against OpenRouter's current API — caught 5 delisted models: openrouter/owl-alpha, openrouter/elephant-alpha, google/gemini-3-pro-preview, tencent/hy3-preview:free, inclusionai/ring-2.6-1t:free
When OpenRouter returns a 404 billing error ('This model is unavailable for
free. The paid version is available now - use this slug instead: <model>'),
the agent now:
1. Correctly classifies it as billing (error_classifier.py _BILLING_PATTERNS)
- Adds 'unavailable for free', 'paid version', 'use this slug instead'
- Also updates auxiliary_client.py _is_payment_error for parity
2. Tries the OpenRouter-suggested slug first by parsing 'use this slug
instead: <slug>' from the error body
3. Falls back to auto-discovering the best free model from the live
OpenRouter curated catalog (hermes_cli/models.py):
- find_free_openrouter_model() — fetches live list, prefers :free models
- parse_openrouter_slug_suggestion() — extracts slug from error text
4. Swaps the agent's model + client in-place and retries (conversation_loop.py)
- Gated by TurnRetryState.openrouter_free_tier_selfheal_attempted
- Fires at most once per turn, on billing 404s for OpenRouter/Nous
Integrates with the removed-model detection (PR NousResearch#57642): the auto-discovery
calls fetch_openrouter_models(force_refresh=True), which now logs removed
models as they're detected.
Expanded scope: added auto-resolution for free-tier model errorsThis PR now also includes the free-tier self-heal feature — when OpenRouter returns a 404 with
Files changed (138 net lines)
Verified
|
|
Thanks for identifying the silent curated-catalog omission: current main does silently skip absent live IDs in Problems
Suggested changes
This is an automated hermes-sweeper review. |
|
THIS FIX HAS BEEN MERGED INTO PR #6239 Thanks all for your contributions to this fix. To streamline and remove duplication, I consolidated the fixes from PRs: All functionality is now available in single, unified PR #6239 which provides: Hopefully the consolidated fix eliminates overhead while preserving all of the fixes & updates |
Summary
already cross-references the curated catalog against OpenRouter's live
/v1/modelsresponse, but silently skipped models that OpenRouter had removed. This PR adds explicit detection and reporting.Changes
hermes_cli/models.py(+26 lines)_openrouter_removed_modelsset)get_removed_openrouter_models()public accessor returnsfrozenset[str]of detected removed models for programmatic inspection (WebUI, status checks)Verified live
On a
force_refresh=Truecall against the current OpenRouter API, detected:openrouter/owl-alphaopenrouter/elephant-alphagoogle/gemini-3-pro-previewtencent/hy3-preview:freeinclusionai/ring-2.6-1t:freeThese models were in the curated
OPENROUTER_MODELS/ remote catalog manifest but absent from OpenRouter's/v1/models. They are automatically excluded from the picker (existing behaviour — no code change needed for that), and now alogger.warning()explains why.Testing
All 84 existing
tests/hermes_cli/test_models.pytests pass.Syntax-validated via
ast.parse.Import-verified:
from hermes_cli.models import fetch_openrouter_models, get_removed_openrouter_modelsworks cleanly.