Skip to content

fix(providers): wire API-key model-provider plugins into runtime and /model - #21695

Open
qWaitCrypto wants to merge 3 commits into
NousResearch:mainfrom
qWaitCrypto:fix/provider-plugin-runtime-picker-21685
Open

fix(providers): wire API-key model-provider plugins into runtime and /model#21695
qWaitCrypto wants to merge 3 commits into
NousResearch:mainfrom
qWaitCrypto:fix/provider-plugin-runtime-picker-21685

Conversation

@qWaitCrypto

Copy link
Copy Markdown
Contributor

What does this PR do?

This fixes a gap in the v0.13.0 model-provider plugin rollout for API-key providers.

Before this change, plugin-backed ProviderProfile entries could be discovered through providers/, but they did not fully integrate with the runtime provider resolver or the shared /model picker:

  • resolve_provider() could still reject a plugin provider slug as unknown unless it also existed in PROVIDER_REGISTRY
  • resolve_api_key_provider_credentials() and get_api_key_provider_status() only handled built-in ProviderConfig entries
  • resolve_runtime_provider() could not route plugin-only API-key providers through the normal generic API-key runtime path
  • /model built its provider list from models.dev mappings, overlays, canonical built-ins, and user config, but not plugin-only provider profiles

This PR wires API-key model-provider plugins into those paths while preserving the existing special-case runtime logic for providers like openrouter and custom.

Related Issue

Fixes #21685

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • Added shared env/.env + credential-pool secret resolution for API-key providers in hermes_cli/auth.py
  • Allowed resolve_provider() to accept plugin-backed API-key provider slugs and aliases
  • Extended resolve_api_key_provider_credentials() and get_api_key_provider_status() to support plugin-backed API-key providers
  • Added guarded runtime handling for plugin-only API-key providers in hermes_cli/runtime_provider.py
  • Preserved the dedicated runtime paths for special providers like openrouter and custom
  • Added a plugin-provider section to list_authenticated_providers() so /model includes plugin-only API-key providers
  • Added regression tests covering plugin credential resolution, alias resolution, runtime resolution, and /model visibility

How to Test

  1. Create an API-key model-provider plugin under $HERMES_HOME/plugins/model-providers/<name>/ with env_vars, base_url, and fallback_models
  2. Set the provider API key env var and run:
    python -m pytest tests/hermes_cli/test_provider_plugin_runtime_and_picker.py -q
  3. Run the related regression suites:
    python -m pytest tests/hermes_cli/test_runtime_provider_resolution.py tests/hermes_cli/test_model_switch_custom_providers.py tests/hermes_cli/test_provider_plugin_runtime_and_picker.py -q
    python -m pytest tests/providers/test_plugin_discovery.py tests/hermes_cli/test_api_key_providers.py -q

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: WSL Ubuntu / Python 3.13

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Screenshots / Logs

  • python -m pytest tests/hermes_cli/test_runtime_provider_resolution.py tests/hermes_cli/test_model_switch_custom_providers.py tests/hermes_cli/test_provider_plugin_runtime_and_picker.py -q132 passed
  • python -m pytest tests/providers/test_plugin_discovery.py tests/hermes_cli/test_api_key_providers.py -q161 passed

@qWaitCrypto
qWaitCrypto force-pushed the fix/provider-plugin-runtime-picker-21685 branch from 82f630e to 250f90c Compare May 15, 2026 02:57
@qWaitCrypto

Copy link
Copy Markdown
Contributor Author

Updated #21695 to cover this path too.
The first version wired plugin-backed API-key providers into runtime resolution and the picker list, but you were right that explicit provider selection still went through switch_model() -> resolve_provider_full(). That meant a plugin provider could appear in /model, then fail when selected with Unknown provider '<name>'.
The PR now adds plugin ProviderProfile support to resolve_provider_full() and a regression test for /model <model> --provider <plugin>, so the picker/listing path and the actual switch path are covered end-to-end.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracing the picker-to-switch failure; that explicit-provider defect still exists on current main (hermes_cli/model_switch.py:862-889hermes_cli/providers.py:700-770).

Problems

  • hermes_cli/runtime_provider.py:1407 only consults a plugin profile when no registry entry exists. Current main auto-registers API-key ProviderProfiles into PROVIDER_REGISTRY (hermes_cli/auth.py:447-477), so this path drops a plugin's declared api_mode after salvage.
  • hermes_cli/auth.py:543-546 accepts every profile env var as a secret. Main explicitly separates *_BASE_URL/*_URL from key vars (hermes_cli/auth.py:464-472); the draft must preserve that distinction. The picker has the same issue at hermes_cli/model_switch.py:1479-1486.
  • tests/providers/test_plugin_discovery.py:49-63 uses a fixed profile count. Current main intentionally uses a directory/registry invariant instead (tests/providers/test_plugin_discovery.py:47-73).

Suggested changes

  • Rework the patch around main's existing registration and canonical-picker paths, retaining the missing resolve_provider_full() profile resolution and profile API-mode propagation.
  • Add a profile with both a key var and base-URL var, plus a non-default API mode, to test the integrated current-main path.

Automated hermes-sweeper review.

Comment thread hermes_cli/runtime_provider.py Outdated
Comment thread hermes_cli/auth.py
Comment thread tests/providers/test_plugin_discovery.py Outdated
@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users labels Jul 13, 2026
Resolve provider plugin runtime and picker review feedback.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cli CLI entry point, hermes_cli/, setup wizard comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: v0.13.0 provider plugin systems do not fully integrate with runtime resolution and /model picker

3 participants