Skip to content

fix(model_switch): filter /model picker for providers with no PROVIDER_REGISTRY entry - #57568

Closed
Ahmett101 wants to merge 1 commit into
NousResearch:mainfrom
Ahmett101:fix/57503-picker-filter-unresolved-providers
Closed

fix(model_switch): filter /model picker for providers with no PROVIDER_REGISTRY entry#57568
Ahmett101 wants to merge 1 commit into
NousResearch:mainfrom
Ahmett101:fix/57503-picker-filter-unresolved-providers

Conversation

@Ahmett101

Copy link
Copy Markdown
Contributor

Summary
list_authenticated_providers() (powers /model in the CLI, gateway, and Telegram picker) emits a row for every slug in PROVIDER_TO_MODELS_DEV whose credential env-var is set. Several of those slugs (notably mistral) have no PROVIDER_REGISTRY entry, so resolve_provider() rejects them with Unknown provider 'mistral' once the user selects a model. The picker shows rows that cannot actually be selected. This is #57503.

Fix: a single resolve-gate in list_authenticated_providers() Section 1 — if PROVIDER_REGISTRY.get(hermes_id) is None, skip the slug. The picker now only lists providers that can actually be switched to. This also resolves the duplicate-Mistral dedup symptom: once the broken-from-models.dev row is filtered, the conflict between PROVIDER_TO_MODELS_DEV['mistral'] and a custom_providers Mistral row becomes moot.

Composes with #50289 (the in-flight first-class-providers plugin merge for mistral): when/if that lands, PROVIDER_REGISTRY gains a mistral entry and the new gate becomes a no-op for it. The two PRs are independent — neither blocks nor breaks the other.

Changes

  • hermes_cli/model_switch.py: 9-line addition in list_authenticated_providers() Section 1: a if not pconfig: continue gate after the existing pconfig.auth_type != "api_key" skip. Documented inline, references the issue.
  • tests/hermes_cli/test_model_switch_filter_unresolved.py (new): 4 regression tests.
    • test_mistral_filtered_when_unregistered_but_api_key_set — pin: MISTRAL_API_KEY set ⇒ mistral MUST NOT appear in picker.
    • test_resolveable_provider_still_appears — regression guard: deepseek (PROVIDER_REGISTRY-backed) with key set MUST still appear.
    • test_resolve_gate_skips_models_dev_only_provider_without_creds — no-key case stays filtered.
    • test_picker_skips_pconfig_none_does_not_break_other_section1_providersxai (REGISTRY-backed) with key set still appears.

How to Test

# new regression suite
pytest tests/hermes_cli/test_model_switch_filter_unresolved.py -xvs
# 4/4 passed

# Verify the test catches the bug: revert the fix and re-run
git stash push -m test hermes_cli/model_switch.py
pytest tests/hermes_cli/test_model_switch_filter_unresolved.py::test_mistral_filtered_when_unregistered_but_api_key_set -xvs
# FAILS with "mistral leaked into /model picker: ['mistral']" -- this is what the
# reporter in #57503 saw on the live picker
git stash pop

# Re-run broader suite to confirm no regressions
pytest tests/hermes_cli/test_model_switch_filter_unresolved.py \
       tests/hermes_cli/test_custom_provider_model_switch.py \
       tests/hermes_cli/test_model_switch_custom_providers.py \
       tests/hermes_cli/test_anthropic_picker_curated.py -q
# 55/55 passed

Checklist

  • Tests pass -- 4 new + 51 existing model-switch picker tests, 55/55 green
  • Follows Conventional Commits (fix(scope):)
  • Changes scoped to a single fix, single-file prod change (+1 test file)
  • Cross-platform impact: None (os.environ read with monkeypatch, no platform-specific code)
  • profile-safe paths: N/A, no paths touched
  • .env not used for non-credential settings: N/A, no settings added
  • No new dependencies
  • Composes cleanly with in-flight feat: add Mistral AI as first-class LLM provider #50289 (different approach, no overlap)

Risk & Impact
Low. Single-file 9-line addition in the picker code path; behaviour change is strictly "drop a row from a list when that row would 100% fail on selection". Users who currently hit the bug get a fixed picker immediately; users with working environments see the picker unchanged (every other provider in PROVIDER_TO_MODELS_DEV is also in PROVIDER_REGISTRY, so they keep appearing).

Type: Bug fix
Closes #57503

Related findings (out of scope here, flagged for maintainers):

  • The deeper fix is the provider-plugin promotion of mistral that feat: add Mistral AI as first-class LLM provider #50289 is already driving -- when that lands, the resolve-gate becomes a no-op for mistral and just keeps defending against the next unregistered models.dev provider that anyone adds to PROVIDER_TO_MODELS_DEV. The gate is intentionally defensive: same code path would have caught a follow-up like "provider X has Y key set but isn't in registry" without a second PR.
  • A future audit pass over PROVIDER_TO_MODELS_DEV against PROVIDER_REGISTRY (one-shot script, no runtime cost) would let us surface any other slugs that historically slipped through, instead of relying on users to file bugs one-by-one. Out of scope for this PR.

…ousResearch#57503)

list_authenticated_providers() emits picker rows for every slug in
PROVIDER_TO_MODELS_DEV that has any credential env-var set. Several of
those slugs (notably 'mistral') have no PROVIDER_REGISTRY entry, so
resolve_provider() rejects them as 'Unknown provider' once the user
selects a model — leaving the picker showing rows that cannot actually
be selected.

Add a resolve-gate in section 1: if PROVIDER_REGISTRY.get(hermes_id)
is None, skip the slug. The picker now only lists providers that can
actually be switched to at runtime.

This automatically resolves the duplicate-Mistral dedup symptom too:
once the broken-from-models.dev row is filtered, the conflict between
PROVIDER_TO_MODELS_DEV['mistral'] and a custom_providers 'Mistral' row
is moot.

Composes with NousResearch#50289 (which promotes mistral to first-class via the
provider-plugin path): when that lands, PROVIDER_REGISTRY gains a
'mistral' entry and the gate becomes a no-op for it. No conflict.

Tests (regression suite):
- tests/hermes_cli/test_model_switch_filter_unresolved.py (new, 4 tests):
  Picker excludes 'mistral' when MISTRAL_API_KEY is set; 'deepseek' and
  'xai' (PROVIDER_REGISTRY-backed) still appear; 'mistral' stays
  excluded when no key is set. Confirmed by reverting the fix and
  seeing the test fail with 'mistral leaked into /model picker'.

Cross-checked against the existing 51 test_model_switch_* and
test_custom_provider_* cases — 55/55 PASS, no regressions.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/cli CLI entry point, hermes_cli/, setup wizard labels Jul 3, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Closing in favor of the earlier, narrower duplicate #57711 for issue #57503. Both add the same PROVIDER_REGISTRY gate in list_authenticated_providers; #57711 is the minimal production change. Thanks for the reproduction and regression coverage.

@teknium1 teknium1 closed this Jul 11, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Reopening as the retained implementation for #57503. #57711 is the later duplicate and is now closed; the unselectable models.dev-only provider bug remains reproducible on main, so one focused PR should stay open for salvage.

@teknium1

Copy link
Copy Markdown
Contributor

Superseded by the improved consolidated salvage PRs prepared in this review batch; contributor credit is preserved in the replacement PR body/commits.

@teknium1 teknium1 closed this Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: mistral listed in /model picker but fails with "Unknown provider 'mistral'"

3 participants