Skip to content

fix(agent): honor explicit api_mode on fallback chain entries - #80209

Open
kuziTony wants to merge 1 commit into
NousResearch:mainfrom
kuziTony:fix/fallback-explicit-api-mode
Open

fix(agent): honor explicit api_mode on fallback chain entries#80209
kuziTony wants to merge 1 commit into
NousResearch:mainfrom
kuziTony:fix/fallback-explicit-api-mode

Conversation

@kuziTony

@kuziTony kuziTony commented Aug 6, 2026

Copy link
Copy Markdown

Fallback activation re-derived the wire protocol from provider name / base URL / model and ignored an api_mode the user had declared on the chain entry, so a fallback could silently come up on a different protocol than the one configured.

The primary path already honors an explicit value via runtime_provider._parse_api_mode; the fallback path had no equivalent. _iter_fallback_entries already passes the whole entry through, so the field was present in fb all along -- just never read.

Concretely, a self-hosted OpenAI-compatible gateway fronting Claude:

fallback_providers:
  - provider: custom model: claude-opus-4 base_url: http://gateway.internal:3000/v1 api_mode: anthropic_messages

matches none of the anthropic detection branches (provider is not anthropic, the URL neither ends in /anthropic nor resolves to api.anthropic.com), so inference fell through to chat_completions. Requests then went out over /chat/completions, which drops the cache_control blocks the Anthropic path attaches -- zeroing prompt caching for the rest of the conversation with nothing logged.

Observed on a self-hosted gateway: once traffic shifted to the fallback, session_model_usage recorded 77 calls with cache_read_tokens = 0, while the primary model on the same endpoint logged 274 calls / 18.4M cache_read_tokens.

Reuses _parse_api_mode for validation, so invalid or absent values fall through to the existing inference chain unchanged. Hostname detection added in #32243 / #49247 keeps working.

What does this PR do?

Related Issue

Fixes #

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

How to Test

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform:

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

For New Skills

  • This skill is broadly useful to most users (if bundled) — see Contributing Guide
  • SKILL.md follows the standard format (frontmatter, trigger conditions, steps, pitfalls)
  • No external dependencies that aren't already available (prefer stdlib, curl, existing Hermes tools)
  • I've tested the skill end-to-end: hermes --toolsets skills -q "Use the X skill to do Y"

Screenshots / Logs

Fallback activation re-derived the wire protocol from provider name /
base URL / model and ignored an `api_mode` the user had declared on the
chain entry, so a fallback could silently come up on a different protocol
than the one configured.

The primary path already honors an explicit value via
`runtime_provider._parse_api_mode`; the fallback path had no equivalent.
`_iter_fallback_entries` already passes the whole entry through, so the
field was present in `fb` all along -- just never read.

Concretely, a self-hosted OpenAI-compatible gateway fronting Claude:

    fallback_providers:
      - provider: custom
        model: claude-opus-4
        base_url: http://gateway.internal:3000/v1
        api_mode: anthropic_messages

matches none of the anthropic detection branches (provider is not
`anthropic`, the URL neither ends in `/anthropic` nor resolves to
`api.anthropic.com`), so inference fell through to `chat_completions`.
Requests then went out over `/chat/completions`, which drops the
`cache_control` blocks the Anthropic path attaches -- zeroing prompt
caching for the rest of the conversation with nothing logged.

Observed on a self-hosted gateway: once traffic shifted to the fallback,
`session_model_usage` recorded 77 calls with cache_read_tokens = 0, while
the primary model on the same endpoint logged 274 calls / 18.4M
cache_read_tokens.

Reuses `_parse_api_mode` for validation, so invalid or absent values fall
through to the existing inference chain unchanged. Hostname detection
added in NousResearch#32243 / NousResearch#49247 keeps working.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint area/config Config system, migrations, profiles sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants