Conversation
… path
resolve_provider_full() (the /model switch + --provider resolver) consulted
only the models.dev catalog, Hermes overlays, and config.yaml providers/
custom_providers. It never consulted the provider-module plugin registry
(plugins/model-providers/<name>/), even though hermes_cli.auth.PROVIDER_REGISTRY
auto-extends from that same source at import time.
Result: a provider declared only as a plugin profile (e.g. a local Anthropic
proxy registered via providers/) resolves fine during runtime startup — so it
works as the configured default model — but switching INTO it via /model fails
with "Unknown provider '<name>'. Check 'hermes model' ...". Two code paths,
two different provider registries.
Reproduction (pre-fix), with a plugin provider declaring api_mode=anthropic_messages:
from hermes_cli.model_switch import switch_model
r = switch_model(raw_input="my-proxy/some-model",
current_provider="openrouter", current_model="x",
current_base_url="", current_api_key="",
is_global=False, explicit_provider="my-proxy")
assert r.success # -> False, "Unknown provider 'my-proxy'"
Fix: add a resolution step (1b) in resolve_provider_full that consults the
providers/ plugin registry via get_provider_profile(), translating the
ProviderProfile into a ProviderDef. api_mode maps back to transport via the
inverse of TRANSPORT_TO_API_MODE (default openai_chat). This reunifies the
switch path with the startup path, so a provider that works as a default also
works as a /model target.
Scope: IN — make the switch resolver see plugin providers. OUT — refactoring
the two registries into one (larger change, own tradeoffs); this PR keeps both
but makes the switch path layer the plugin registry the same way auth.py does.
Verified: resolve_provider_full + switch_model now succeed for a plugin
provider with correct transport/base_url/api_mode; unknown providers still
return None. Pre-existing 3 failures in test_model_switch_custom_providers.py
(model-catalog 403 in sandbox) are unrelated and fail identically on origin/main.
teknium1
left a comment
There was a problem hiding this comment.
Thanks for tracing the split between the profile registry and the switch resolver. The missing lookup is present on current main: resolve_provider_full() falls through at hermes_cli/providers.py:742-779, and /model emits the unknown-provider error at hermes_cli/model_switch.py:862-889.
Problems
hermes_cli/providers.py:701resolves a plugin beforeproviders:/custom_providers:. That can shadow a user-configured endpoint with the same name, despite the file contract that config overrides are merged on top (hermes_cli/providers.py:14-15).- The
api_mode→transportconversion athermes_cli/providers.py:770-771does not reach the runtime switch result.switch_model()later callsresolve_runtime_provider()(hermes_cli/model_switch.py:1213-1220), whose generic registry path defaults tochat_completionsunless config or URL heuristics override it (hermes_cli/runtime_provider.py:1466-1501). - The PR changes only
hermes_cli/providers.py; please add a plugin-profile/modelregression test, including ananthropic_messagesprofile.
Suggested changes
- Preserve user-config precedence, then use profile lookup as fallback.
- Resolve
ProviderProfile.api_modein the runtime path or explicitly carry the resolved transport through switching.
Automated hermes-sweeper review.
| # rejected with "Unknown provider" when the user tries to /model into | ||
| # it. resolve_provider_full and runtime startup resolution were | ||
| # consulting two different registries; this reunifies them. | ||
| plugin_pdef = _resolve_plugin_provider(canonical, name) |
There was a problem hiding this comment.
This fallback is before the existing providers: and custom_providers: paths. A plugin named the same as a configured endpoint will now shadow that user override; keep config resolution ahead of the plugin fallback.
| return None | ||
|
|
||
| api_mode = getattr(profile, "api_mode", "") or "chat_completions" | ||
| transport = _API_MODE_TO_TRANSPORT.get(api_mode, "openai_chat") |
There was a problem hiding this comment.
This mapped transport is not consumed by switch_model after it sets target_provider: runtime resolution independently defaults API-key registry providers to chat_completions. Please propagate ProviderProfile.api_mode into the runtime switch result and cover an anthropic_messages plugin.
|
Thank you for identifying the missing ProviderProfile lookup in the model-switch path. I consolidated this overlap into #52549, kept plugin discovery out of the built-in |
|
@WolframRavenwolf — thank you, and confirming your consolidation from our side after actually checking it rather than taking it on trust.
Verified: More importantly, I checked whether #52549 actually resolves the two defects @teknium1's sweeper raised against this PR, since consolidation only helps if the consolidated version is the better one. It does, on both counts:
So #52549 is strictly the better implementation and this PR should not land in preference to it. One correction to the record, in your favour: an earlier internal triage pass of ours logged this as "already consolidated, nothing to do but close." That was wrong — #52549 is still OPEN, not merged (checked just now). Closing this PR on that basis would have retired the only other open implementation of the fix while the survivor was still unlanded. I'm leaving this open until #52549 merges; happy for a maintainer to close it the moment it does, and I'd rather it close as superseded-by-#52549 than get merged. Nothing needed from you — flagging the open-vs-merged distinction so nobody else on our side acts on the same stale assumption. |
Teach the full CLI provider resolution path to recognize ProviderProfile plugins and aliases without making the built-in get_provider() hot path perform plugin discovery. This consolidates the overlap with NousResearch#34368 while preserving explicit user-config precedence, bare custom endpoint semantics, and provider metadata in /model results. Co-authored-by: Kyzcreig <kyzcreig@users.noreply.github.com>
|
Closing in favor of #52549 per the consolidation above — verified it carries the Co-authored-by trailer ( |
Summary
resolve_provider_full()— the resolver behind/model <name>,--provider, and the interactive model picker — consults the models.dev catalog, Hermes overlays, and config.yamlproviders:/custom_providers:. It never consults the provider-module plugin registry (plugins/model-providers/<name>/), even thoughhermes_cli.auth.PROVIDER_REGISTRYauto-extends from that same source at import time.The two paths therefore use two different provider registries:
runtime_provider→auth.PROVIDER_REGISTRY→ knows plugin providers./modelswitch resolution →resolve_provider_full→get_provider()+ config only → does not know plugin providers.Symptom
A provider declared only as a plugin profile (e.g. a local Anthropic-compatible proxy registered via
providers/) works fine as the configured default model — startup resolves it correctly — but switching into it via/model(or selecting it in the Discord/Telegram picker) fails with:Reproduction (pre-fix)
With a plugin provider under
plugins/model-providers/<name>/declaringapi_mode="anthropic_messages":Fix
Add resolution step 1b in
resolve_provider_fullthat consults theproviders/plugin registry viaget_provider_profile(), translating theProviderProfileinto aProviderDef.api_modemaps back totransportvia the inverse ofTRANSPORT_TO_API_MODE(defaultopenai_chat). Env-var splitting mirrors the same_BASE_URL/_URLheuristicauth.pyalready uses when it auto-extends the registry.This reunifies the switch path with the startup path: a provider that works as a default now also works as a
/modeltarget. Built-in/models.dev/config resolution order is unchanged — the plugin layer slots in after built-ins and before config providers, matching the precedenceauth.pyuses.Scope
auth.pydoes.Verification
A standalone assertion script confirms:
resolve_provider_full("<plugin-provider>")returns aProviderDefwith correcttransport(mapped fromapi_mode),base_url, and key env vars.switch_model(... explicit_provider="<plugin-provider>")succeeds end-to-end with the rightapi_mode/base_url.None/ fails (no over-broad matching).Targeted suites pass (
test_user_providers_model_switch,test_model_switch_opencode_anthropic,test_api_key_providers,test_runtime_provider_resolution, etc.). Three pre-existing failures intest_model_switch_custom_providers.pyare caused by a model-catalog403in the sandbox and fail identically onorigin/mainwith this change reverted — unrelated to this PR.