Skip to content

fix(models): drop curated OpenRouter ids that are retired or no longer tool-capable - #77203

Closed
jbbottoms wants to merge 2 commits into
NousResearch:mainfrom
jbbottoms:fix/openrouter-catalog-drift
Closed

fix(models): drop curated OpenRouter ids that are retired or no longer tool-capable#77203
jbbottoms wants to merge 2 commits into
NousResearch:mainfrom
jbbottoms:fix/openrouter-catalog-drift

Conversation

@jbbottoms

Copy link
Copy Markdown
Contributor

Summary

Five of the 39 entries in OPENROUTER_MODELS no longer work. Checked against OpenRouter's live /api/v1/models (337 models):

Curated id State
openrouter/elephant-alpha retired, no successor
poolside/laguna-m.1:free retired; family moved to laguna-s-2.1
tencent/hy3:free free tier withdrawn
inclusionai/ring-2.6-1t:free free tier withdrawn
openrouter/pareto-code served, but supported_parameters is []

The first four fail the moment a user picks them.

The fifth is the worse one. openrouter/pareto-code is still served, so it selects and runs fine, and only fails when the agent tries to call a tool. In an agent framework that is a silent trap rather than an error.

Nothing caught this because every existing model test asserts against the curated list itself. That makes the list self-consistent, and says nothing about whether OpenRouter still serves what is in it.

The two :free removals

tencent/hy3 and inclusionai/ring-2.6-1t still exist as paid ids. I removed the :free entries rather than re-pointing them, because moving a user's free selection onto a paid id is a curation decision for you, not something a drift fix should do quietly. tencent/hy3 is already curated separately and is untouched.

Same reasoning for poolside/laguna-m.1:free: laguna-s-2.1 and laguna-xs-2.1 are live, but adding them is a new curation call rather than a repair.

Preventing recurrence

Adds tests/hermes_cli/test_openrouter_catalog_drift_live.py, opt-in and live in the same shape as the other *_live.py tests here:

HERMES_LIVE_TESTS=1 OPENROUTER_API_KEY=sk-or-... \
    pytest tests/hermes_cli/test_openrouter_catalog_drift_live.py -q

Live and opt-in on purpose: it asserts against a third-party catalogue that changes without us, so it must never be able to redden CI on someone else's deploy. It is a maintenance check run deliberately.

It checks the two failure modes separately, because they are not the same problem — one fails on selection, the other on first tool call.

Verification

  • Skips clean with no opt-in — 2 skipped
  • Passes live after this change — 2 passed
  • Non-vacuous: run against the unfixed list it fails and names all five:
    AssertionError: OPENROUTER_MODELS lists ids OpenRouter no longer serves; a user picking one
    gets a failure: ['openrouter/elephant-alpha', 'poolside/laguna-m.1:free', 'tencent/hy3:free',
    'inclusionai/ring-2.6-1t:free']
    assert not ['openrouter/pareto-code']
    
  • scripts/build_model_catalog.py regenerated — openrouter 39 → 34 models
  • pytest tests/hermes_cli/test_model_catalog.py test_model_search.py test_ai_gateway_models.py30 passed
  • ruff check → passed

…r tool-capable

Five of the 39 entries in OPENROUTER_MODELS no longer work, checked against
OpenRouter's live /api/v1/models:

  openrouter/elephant-alpha       retired, no successor
  poolside/laguna-m.1:free        retired; family moved to laguna-s-2.1
  tencent/hy3:free                free tier withdrawn
  inclusionai/ring-2.6-1t:free    free tier withdrawn
  openrouter/pareto-code          served, but supported_parameters is []

The first four fail the moment a user picks them. The fifth is the worse one:
it is still served, so it selects and runs fine, and only fails when the agent
tries to call a tool. In an agent framework that is a silent trap rather than
an error.

Nothing caught this because every existing model test asserts against the
curated list itself, which makes the list self-consistent and says nothing
about whether OpenRouter still serves what is in it.

The two ":free" ids have paid equivalents still live (tencent/hy3,
inclusionai/ring-2.6-1t). They are removed rather than re-pointed: moving a
user's free selection onto a paid id is a curation decision for maintainers,
not something a drift fix should do quietly. tencent/hy3 is already curated
separately and is unaffected.

Adds tests/hermes_cli/test_openrouter_catalog_drift_live.py so this cannot
recur silently. Opt-in and live, in the same shape as the other *_live.py
tests here (HERMES_LIVE_TESTS=1 plus a provider key), because it asserts
against a third-party catalogue that changes without us and must never redden
CI on someone else's deploy.

Verified:
- skips clean with no opt-in: 2 skipped
- passes live after this change: 2 passed
- non-vacuous: against the unfixed list it fails and names all five
- scripts/build_model_catalog.py regenerated: openrouter 39 -> 34 models
- pytest tests/hermes_cli/test_model_catalog.py test_model_search.py
  test_ai_gateway_models.py -> 30 passed
- ruff check -> passed
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/cli CLI entry point, hermes_cli/, setup wizard comp/dashboard Web dashboard / control panel UI (dashboard/, landing) provider/openrouter OpenRouter aggregator labels Aug 3, 2026
CI slice failed because the TokenHub catalog pin still asserted the free tier id after OPENROUTER_MODELS dropped it. Keep paid tencent/hy3; assert :free is absent.
@GottZ

GottZ commented Aug 3, 2026

Copy link
Copy Markdown

This was generated by AI during triage.

Summary

Seven PRs address or materially reference this catalog complex: #3756/#3803 add the original Gemini 3.1 entries, #34581/#73479 update later curated generations, #77159 expands the Gemini subset, #77203 removes retired or non-tool-capable entries and adds a live drift check, and #77254 provides an opt-in full tool-capable catalog for BYOK users.

Related pull requests

Duplicates

#3756 and #3803 contain the same OpenRouter Gemini 3.1 additions; #3756 was closed after being incorporated into the more complete, merged #3803. The remaining PRs represent distinct catalog revisions, drift repair, or opt-in expansion rather than equivalent diffs.

Suggested consolidation

Keep #77203 open with a salvage path focused on the five removals and the opt-in live drift test, and keep #77254 open with its distinct salvage path as the recorded best fix for #76732's BYOK discovery problem. Leave #77159 closed rather than reopening it over the contributor's curation objection and the author's acceptance of that objection; #3756 is already superseded by merged #3803, while #34581 and #73479 remain merged historical baselines rather than closure candidates.

Cross-PR triage: Reviewed 7 pull requests and 2 issues in this complex. Each diff was read against this issue; Assessment working set: 28 kB of PR diffs, 14 kB of issue/PR text, 5 kB of discussion (8 comments), 12 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

@jbbottoms jbbottoms closed this Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard comp/dashboard Web dashboard / control panel UI (dashboard/, landing) P2 Medium — degraded but workaround exists provider/openrouter OpenRouter aggregator type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants