Skip to content

fix(fallback): honor explicit api_mode on fallback_providers entries - #70141

Open
seagaruda wants to merge 1 commit into
NousResearch:mainfrom
seagaruda:fix/fallback-respects-explicit-api-mode
Open

fix(fallback): honor explicit api_mode on fallback_providers entries#70141
seagaruda wants to merge 1 commit into
NousResearch:mainfrom
seagaruda:fix/fallback-respects-explicit-api-mode

Conversation

@seagaruda

Copy link
Copy Markdown

Problem

try_activate_fallback() in agent/chat_completion_helpers.py re-derives api_mode for a fallback entry purely from provider name / base_url heuristics — it never reads the api_mode field the user explicitly set on that fallback_providers config entry.

For a custom proxy/endpoint whose host doesn't match any of the heuristics (not api.anthropic.com, not api.openai.com, not Azure, not Bedrock), the code silently discards an explicitly configured api_mode: anthropic_messages and falls through to the chat_completions default.

Symptom

Observed in production with a custom Anthropic-compatible proxy configured as a fallback:

fallback_providers:
- provider: custom:MyProxy
  model: claude-sonnet-4-6
  api_key: ${MY_API_KEY}
  base_url: https://my-proxy.example.com
  api_mode: anthropic_messages

When the primary provider hit a 429 and the agent switched to this fallback, the request went to POST /chat/completions with a Bearer auth header (chat_completions wire format) instead of POST /v1/messages with an x-api-key header (the configured anthropic_messages format). The proxy doesn't serve /chat/completions at all, so it returned its frontend's HTML 404 page instead of a JSON API error, and the whole fallback chain failed after 3 retries — even though the endpoint and credentials were both valid.

Fix

Read fb.get("api_mode") first. If the config entry specifies one, use it as-is (it's authoritative — the user is telling us the wire protocol for an endpoint the heuristics can't classify from the URL alone). Only fall back to the existing provider/base_url heuristics when the entry didn't specify api_mode.

Testing

  • python3 -m py_compile agent/chat_completion_helpers.py — syntax OK
  • Verified against my own multi-profile Hermes setup: reproduced the 404-on-fallback failure with an api_mode: anthropic_messages custom fallback provider, applied the fix, restarted the gateway, and confirmed the fallback path now correctly resolves to anthropic_messages instead of defaulting to chat_completions.
  • No test suite changes — did not have time to write a unit test covering try_activate_fallback's api_mode resolution path; happy to add one if requested.

try_activate_fallback() re-derives api_mode for a fallback entry purely
from provider name / base_url heuristics, ignoring any api_mode the
user explicitly set on that fallback_providers config entry. For a
custom proxy/endpoint that the heuristics can't classify (host doesn't
look like api.anthropic.com, api.openai.com, etc.) this silently
discards an explicit api_mode: anthropic_messages and falls through to
chat_completions.

Symptom: fallback requests hit the wrong path (/chat/completions
instead of /v1/messages) with the wrong auth header (Bearer instead of
x-api-key), and the endpoint returns an HTML 404 page instead of a JSON
API error — so the fallback chain fails even though the endpoint and
credentials are both valid.

Fix: read fb.get("api_mode") first and use it as-is when present;
fall back to the existing heuristics only when the config entry didn't
specify one.
@seagaruda
seagaruda force-pushed the fix/fallback-respects-explicit-api-mode branch from e2fae10 to 96f05cb Compare July 23, 2026 15:18
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint area/config Config system, migrations, profiles sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades duplicate This issue or pull request already exists labels Jul 23, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Duplicate of #16346: both repair fallback activation dropping an explicitly configured api_mode in favor of URL/provider heuristics.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for isolating the live fallback path. Current main still has the reported direct-entry bug: agent/chat_completion_helpers.py:1824-1868 never reads fb["api_mode"], so the proposed ordering is directionally correct.

Problems

  • agent/chat_completion_helpers.py PR line 1679 trusts a raw .strip() value. Reuse the canonical validator at hermes_cli/runtime_provider.py:363-369; it rejects non-strings and unsupported modes rather than installing an invalid runtime protocol.
  • This handles only live activation. Gateway auth fallback calls resolve_runtime_provider() without the entry's api_mode or model at gateway/run.py:2463-2467, so the same explicit override remains lost when fallback is selected before normal agent construction.
  • Please add the focused live-fallback regression test in tests/run_agent/test_provider_fallback.py, alongside the existing protocol-selection cases at lines 149-232.

Suggested changes

  • Parse the override canonically and run heuristics only when parsing yields no supported mode.
  • Propagate the validated override through gateway auth fallback, then cover both protocol/client selection and invalid input.

Automated hermes-sweeper review.

# such fallback defaulted to chat_completions, hitting the wrong
# path (/chat/completions instead of /v1/messages) with the wrong
# auth header (Bearer instead of x-api-key) and failing with a 404.
fb_explicit_api_mode = (fb.get("api_mode") or "").strip()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use hermes_cli.runtime_provider._parse_api_mode() here rather than calling .strip() directly. The canonical parser rejects non-string and unsupported YAML values; this expression can raise on a non-string value or set agent.api_mode to a mode with no supported transport.

@teknium1 teknium1 added sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 30, 2026
@seagaruda
seagaruda requested a review from teknium1 August 17, 2026 01:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint duplicate This issue or pull request already exists P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants