fix(fallback): honor explicit api_mode on fallback_providers entries - #70141
fix(fallback): honor explicit api_mode on fallback_providers entries#70141seagaruda wants to merge 1 commit into
Conversation
try_activate_fallback() re-derives api_mode for a fallback entry purely
from provider name / base_url heuristics, ignoring any api_mode the
user explicitly set on that fallback_providers config entry. For a
custom proxy/endpoint that the heuristics can't classify (host doesn't
look like api.anthropic.com, api.openai.com, etc.) this silently
discards an explicit api_mode: anthropic_messages and falls through to
chat_completions.
Symptom: fallback requests hit the wrong path (/chat/completions
instead of /v1/messages) with the wrong auth header (Bearer instead of
x-api-key), and the endpoint returns an HTML 404 page instead of a JSON
API error — so the fallback chain fails even though the endpoint and
credentials are both valid.
Fix: read fb.get("api_mode") first and use it as-is when present;
fall back to the existing heuristics only when the config entry didn't
specify one.
e2fae10 to
96f05cb
Compare
Duplicate of #16346: both repair fallback activation dropping an explicitly configured api_mode in favor of URL/provider heuristics. |
teknium1
left a comment
There was a problem hiding this comment.
Thanks for isolating the live fallback path. Current main still has the reported direct-entry bug: agent/chat_completion_helpers.py:1824-1868 never reads fb["api_mode"], so the proposed ordering is directionally correct.
Problems
agent/chat_completion_helpers.pyPR line 1679 trusts a raw.strip()value. Reuse the canonical validator athermes_cli/runtime_provider.py:363-369; it rejects non-strings and unsupported modes rather than installing an invalid runtime protocol.- This handles only live activation. Gateway auth fallback calls
resolve_runtime_provider()without the entry'sapi_modeor model atgateway/run.py:2463-2467, so the same explicit override remains lost when fallback is selected before normal agent construction. - Please add the focused live-fallback regression test in
tests/run_agent/test_provider_fallback.py, alongside the existing protocol-selection cases at lines 149-232.
Suggested changes
- Parse the override canonically and run heuristics only when parsing yields no supported mode.
- Propagate the validated override through gateway auth fallback, then cover both protocol/client selection and invalid input.
Automated hermes-sweeper review.
| # such fallback defaulted to chat_completions, hitting the wrong | ||
| # path (/chat/completions instead of /v1/messages) with the wrong | ||
| # auth header (Bearer instead of x-api-key) and failing with a 404. | ||
| fb_explicit_api_mode = (fb.get("api_mode") or "").strip() |
There was a problem hiding this comment.
Please use hermes_cli.runtime_provider._parse_api_mode() here rather than calling .strip() directly. The canonical parser rejects non-string and unsupported YAML values; this expression can raise on a non-string value or set agent.api_mode to a mode with no supported transport.
Problem
try_activate_fallback()inagent/chat_completion_helpers.pyre-derivesapi_modefor a fallback entry purely from provider name / base_url heuristics — it never reads theapi_modefield the user explicitly set on thatfallback_providersconfig entry.For a custom proxy/endpoint whose host doesn't match any of the heuristics (not
api.anthropic.com, notapi.openai.com, not Azure, not Bedrock), the code silently discards an explicitly configuredapi_mode: anthropic_messagesand falls through to thechat_completionsdefault.Symptom
Observed in production with a custom Anthropic-compatible proxy configured as a fallback:
When the primary provider hit a 429 and the agent switched to this fallback, the request went to
POST /chat/completionswith aBearerauth header (chat_completions wire format) instead ofPOST /v1/messageswith anx-api-keyheader (the configured anthropic_messages format). The proxy doesn't serve/chat/completionsat all, so it returned its frontend's HTML 404 page instead of a JSON API error, and the whole fallback chain failed after 3 retries — even though the endpoint and credentials were both valid.Fix
Read
fb.get("api_mode")first. If the config entry specifies one, use it as-is (it's authoritative — the user is telling us the wire protocol for an endpoint the heuristics can't classify from the URL alone). Only fall back to the existing provider/base_url heuristics when the entry didn't specifyapi_mode.Testing
python3 -m py_compile agent/chat_completion_helpers.py— syntax OKapi_mode: anthropic_messagescustom fallback provider, applied the fix, restarted the gateway, and confirmed the fallback path now correctly resolves toanthropic_messagesinstead of defaulting tochat_completions.try_activate_fallback's api_mode resolution path; happy to add one if requested.