fix(providers): gate plugin api_mode by endpoint (#53054)

[david-bowiegxw]

What does this PR do?

Completes the ProviderProfile.api_mode fix on current main without forcing a
plugin's declared transport onto a user-supplied endpoint override.

main now bridges profile-only providers through get_provider() as a
ProviderDef(source="plugin-profile"), so the old head of this PR is obsolete:
the declared transport already reaches determine_api_mode() and all three
runtime fallback paths. This head is re-derived from current main and retains
only the remaining correctness gap.

Remaining bug on current main

determine_api_mode() currently returns a profile-only provider's transport for
every base_url, including a different user override. For example, a profile
whose default endpoint is native Anthropic Messages can be redirected to an
OpenAI-compatible /v1 endpoint and still be forced onto
anthropic_messages.

That violates the established precedence rule: a different explicit endpoint is
the stronger signal. Recognized hosts remain handled by the host-mandated tier;
an unknown override should retain the conservative chat_completions default.

Fix

• In determine_api_mode(), apply a source="plugin-profile" transport only
  when base_url is absent or matches the profile's declared endpoint.
• Preserve trailing-slash equivalence.
• Leave host-mandated modes, Hermes overlays, models.dev providers, persisted
  config, Nous dual-wire routing, and unknown-provider defaults unchanged.
• Correct providers/README.md to describe the current profile-to-runtime path.

The endpoint-gating design is adopted from @samfoy's #65956.

Regression coverage

The test installs a real provider plugin beneath the isolated test
HERMES_HOME before the first auth import, then exercises the production
chain:

plugin on disk → discovery → ProviderProfile registration → auth registry
extension → get_provider() → determine_api_mode() → runtime resolution.

Coverage includes:

• canonical name and alias resolution;
• pooled, explicit-key, and ordinary no-pool API-key routes;
• persisted-config precedence;
• host-mandated precedence;
• different-base_url override gating and trailing-slash equivalence;
• unchanged overlay and unknown-provider behavior.

Mutation check: removing the new endpoint gate makes the override regression
test fail (anthropic_messages instead of chat_completions).

Verification

tests/hermes_cli/test_plugin_provider_api_mode.py                  9 passed

runtime/provider/model-switch focused selection                168 passed

provider + runtime surrounding suites                          250 passed

ruff (changed Python files)                                      passed
git diff --check                                                 passed

Relationship to the overlapping PRs

• fix(providers): honor ProviderProfile.api_mode in api_mode resolution #65956 still contains distinct unknown-provider/OpenRouter fallback behavior;
  that scope should be reviewed separately from this endpoint-gating fix.
• fix(providers): unify ProviderProfile identity and api mode #52549 combines provider identity and runtime work. Current main already
  contains the profile-only get_provider() bridge this PR now builds on, so
  this focused head does not reintroduce that broader identity change.

Fixes #53054.

[teknium1]

Thanks for tracing the profile-to-runtime gap; the current-head premise is valid: ProviderProfile.api_mode exists at providers/base.py:44, while the auto-registration bridge at hermes_cli/auth.py:466-473 does not retain it.

Problems

• The new fallback is only added to the pooled resolver. The explicit API-key route still resolves from config/URL only at hermes_cli/runtime_provider.py:1493-1501, and the ordinary no-pool API-key route does the same at hermes_cli/runtime_provider.py:2039-2062. Those paths still misroute a non-self-describing plugin endpoint.
• tests/hermes_cli/test_plugin_provider_api_mode.py:152-160 mirrors the production bridge while constructing ProviderConfig; it does not execute auth.py's registration loop, so it cannot catch a bridge regression.

Suggested changes

• Apply the declared-profile fallback consistently to all three API-key resolver paths, preferably through one helper preserving existing config and URL precedence.
• Add a real discovery → auth registry → resolve_runtime_provider regression test for the non-self-describing endpoint, including the no-pool path.

Automated hermes-sweeper review.

[teknium1]

This covers only pooled credentials. _resolve_explicit_runtime and the ordinary no-pool API-key resolver independently apply URL detection and still default to chat_completions; please route all three paths through the same declared-profile fallback.

[teknium1]

This duplicates the production bridge rather than invoking it, so the test passes even if auth.py no longer propagates api_mode. Exercise plugin discovery plus the actual auto-registration loop instead.

[david-bowiegxw]

Both points addressed. The branch is rebuilt on current main (the previous head predated the 0.18.0 resolver refactor), force-pushed as 71f8393ca.

1. All three API-key resolver paths now share one fallback helper. New _default_api_mode_for_provider() in hermes_cli/runtime_provider.py resolves: URL auto-detection first (unchanged), then the profile-declared api_mode bridged into ProviderConfig.api_mode, then chat_completions. It replaces the detection tail in the pooled resolver, the explicit API-key route, and the ordinary no-pool route — persisted-config precedence (including the _provider_supports_explicit_api_mode gate where it existed) is untouched at all three sites.

2. The regression test now executes the real chain, not a mirror. tests/hermes_cli/test_plugin_provider_api_mode.py drops a real plugin directory under $HERMES_HOME/plugins/model-providers/, lets provider discovery import it, then runs the actual registration loop in auth.py — extracted as the re-runnable _extend_registry_from_provider_plugins() (same code, now callable) — and asserts through resolve_runtime_provider(). Coverage: the no-pool route, the explicit API-key route, the pooled route, plus two precedence guards (persisted config still wins over the declared profile; a self-describing URL still wins over the declared profile).

Verification: with the fix reverted, the suite fails (1 failed + 6 errors — the bridge drops the declaration and the registration loop isn't callable); with it applied, all 7 pass, and the surrounding suites stay green (test_runtime_provider*, tests/providers/, test_api_key_providers: 461 passed; test_auth*: 327 passed).

[WolframRavenwolf]

I incorporated your ProviderProfile.api_mode commit into the coordinated #52549 branch with your original authorship and commit message preserved. That branch combines this all-route runtime fix with the overlapping CLI identity work from #34368 and adds an integration regression for the real /model plus auth-registration path. Maintainers can use #52549 as the consolidated landing surface if they prefer one integration PR.

[david-bowiegxw]

Rebuilt on current main (e98bd2aadd). The previous head conflicted after 33a2f29a63, and that commit changes what this PR should be — so this is a re-derivation against the new resolver, not a rebase.

What 33a2f29a63 already fixed

_fallback_api_mode(provider, base_url, model) now routes all three API-key resolver paths — pooled, explicit-key, no-pool — through providers.determine_api_mode. That is the first point of the review above, landed independently for the in-tree overlay case (openai-api on us.api.openai.com). The previous head of this PR reached those same three sites with its own helper, so it is superseded; I've dropped it and kept nothing in runtime_provider.py.

What it did not fix

determine_api_mode never consulted the plugin registry, and get_provider() resolves against models.dev + HERMES_OVERLAYS only — plugin profiles are invisible to both. Probed against 1c39f1c9f9, with a plugin declaring api_mode="anthropic_messages" on a non-self-describing endpoint:

ProviderConfig fields   -> [id, name, auth_type, … base_url_env_var]    # no api_mode
_detect_api_mode_for_url(url)     -> None
determine_api_mode(plugin, url)   -> chat_completions
_fallback_api_mode(plugin, url)   -> chat_completions

So the declaration is still dropped and the request still goes out as {base_url}/chat/completions.

Worth noting providers/README.md already documents the contract this PR implements:

hermes_cli/runtime_provider.py reads profile.api_mode as a fallback when URL detection finds nothing.

Nothing in the tree reads it. This head makes that line true and repoints it at the function that actually does the reading.

This head

One tier inside determine_api_mode, so all three resolver paths inherit it with no change to runtime_provider.py. Overlays keep precedence over a plugin of the same name; the declaration is consulted before the models.dev lookup, whose transport for a catalog-only provider is the generic openai_chat default and would otherwise mask it; a declared chat_completions is treated as no declaration, since returning it early would only skip later tiers without changing the answer.

Second review point — the mirrored test — is addressed the same way as before: auth.py's registration loop is now the named, re-runnable _extend_registry_from_provider_plugins() (same body, same single call at import), so the tests drop a real plugin under $HERMES_HOME/plugins/model-providers/, let discovery import it, run the actual registration loop, and assert through resolve_runtime_provider().

tests/hermes_cli/test_plugin_provider_api_mode.py                 9 passed
  └─ same tests with the providers.py hunk reverted:              5 failed, 4 passed
     (the four precedence guards correctly still pass)

test_runtime_provider*, test_runtime_transport_precedence,
tests/providers/, test_api_key_providers*                       208 passed

tests/hermes_cli/ -k "provider or auth or api_mode
                      or model_switch or transport"            1121 passed, 1 skipped

On #65956

After 33a2f29a63, @samfoy's #65956 and this PR land at the same layer — determine_api_mode — and are close enough that merging both would double-apply. Two things I'd rather say than leave for someone else to find:

• The base_url gating in this head is @samfoy's design, adopted from fix(providers): honor ProviderProfile.api_mode in api_mode resolution #65956 and credited in the code, the commit, and the PR body. It's the right call: a base_url override to a different endpoint is the stronger signal, and forcing the declared mode onto it would break e.g. MiniMax's OpenAI-compatible /v1 route.
• I kept this one moving because it carries the discovery → auth-registration → resolver regression coverage asked for above, and because the original report is [Bug]: provider plugins' ProviderProfile.api_mode dropped at runtime — anthropic_messages plugins with non-/anthropic base_url 404 #53054. That's not an argument that it's the better PR. Either is a fine landing surface; fix(providers): unify ProviderProfile identity and api mode #52549 carries this change with authorship preserved if one consolidated PR is preferred. Maintainers should pick one — fix(providers): honor ProviderProfile.api_mode in api_mode resolution #65956 currently conflicts with main too, so whichever is chosen needs a pass over the new resolver either way.

[david-bowiegxw]

@teknium1 — re-review requested on the new head e2aeb50, based directly on
current main (13ce0c5).

The pull-request workflows are currently marked action_required, so please
also approve the external-contributor Actions runs when reviewing.

This is a clean re-derivation from current main, not a rebase of the stale
implementation. Since main now bridges profile-only providers through
get_provider() as source="plugin-profile", I removed the superseded
registry/lookup work and kept only the remaining correctness gap: a profile's
transport must not be forced onto a different user-supplied base_url.

The production change is now 13 lines in determine_api_mode(). The regression
file still runs the real plugin-on-disk → discovery → auth registration → all
three runtime paths chain, without adding a test-only production hook.

Verification on current main:

• focused regression file: 9 passed;
• selected runtime/provider/model-switch suite: 168 passed;
• surrounding provider/runtime suites: 250 passed;
• Ruff and git diff --check: passed;
• mutation check: removing the endpoint gate fails the override regression.

The PR body now documents how this head relates to #65956 and #52549. In
particular, #65956's distinct unknown-provider/OpenRouter fallback remains out
of scope here.