fix(auxiliary): use active runtime in final main-agent fallback

[Willhong]

What does this PR do?

The final auxiliary main-agent safety net currently re-reads the statically configured main provider/model after an auxiliary route fails. If the conversation has switched or failed over to a different runtime, the safety net can retry the stale provider instead of the provider already serving the session.

This PR threads the normalized session-local main_runtime snapshot through the final synchronous and asynchronous safety-net calls. A complete active runtime takes priority; static main configuration remains the fallback when no runtime snapshot is available.

For openai-codex, the resolver now preserves the token and base URL selected by the active runtime instead of selecting a new route from the credential pool or auth store.

This closes a narrow propagation gap between the layered safety net introduced in #27625 and the session-scoped runtime work merged in #63521.

Related Issue

Fixes #73283

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• Pass the normalized active runtime into the final main-agent fallback in both sync and async auxiliary paths.
• Prefer the active provider/model/authentication context/endpoint, while preserving the existing static-config fallback when runtime state is absent.
• Make the Codex auxiliary client builder honor an explicit runtime token and base URL before credential-pool/auth-store resolution.
• Preserve scoped runtime isolation and handle config-less named custom runtimes through their resolved live endpoint.
• Add regressions for sync/async routing, Codex credential/endpoint reuse, scoped runtime behavior, custom runtime resolution, and legacy config fallback.

How to Test

1. Configure main provider A, then switch or fail over the active conversation to provider B.
2. Make an explicit auxiliary compression route fail with an error handled by the fallback chain.
3. Verify the final main-agent fallback resolves provider B and reuses its active authentication context and endpoint, rather than re-reading provider A from static config.

Validation run on current main:

Focused runtime/safety-net tests: 12 passed
Auxiliary client suite: 373 passed, 1 deselected
Local dual-endpoint probe: configured auxiliary endpoint received one 429 request; active runtime endpoint received one successful request
Ruff, py_compile, and git diff checks: passed

Baseline note: a broader adjacent bundle reports four order-dependent provider-health failures. The same four failures reproduce on a clean current-main worktree, and all four pass in isolation, so they are not introduced by this PR.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: macOS 26.5.1

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — or N/A
• I've updated cli-config.yaml.example if I added/changed config keys — or N/A
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
• I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Screenshots / Logs

Configured auxiliary route -> HTTP 429
Active main runtime route -> valid summary response
Requests observed: auxiliary=1, active-runtime=1

[teknium1]

Thanks for tracing the sync and async safety-net gap. The core premise is confirmed on current main: agent/auxiliary_client.py:4459-4460 still re-reads static main configuration, while the callers at agent/auxiliary_client.py:8563-8565 and 9159-9161 already hold a normalized runtime snapshot.

Problems

• The proposed configured named-custom path does not actually retain its active endpoint or credential. The added branch keeps custom:<name> when it finds a configured entry, but current resolution reads custom_base and custom_key from that entry at agent/auxiliary_client.py:5525-5529; the explicit runtime values passed by the PR are not consumed there. The config-less test does not cover this configured case.

Suggested changes

• Add a configured named-custom regression where the entry differs from main_runtime, and make named-custom resolution honor the active runtime endpoint/key while preserving its API-mode handling.

Automated hermes-sweeper review.

[teknium1]

When a named custom entry exists, this leaves main_provider as custom:<name>, but resolve_provider_client() takes that branch's endpoint and key from custom_entry (agent/auxiliary_client.py:5525-5529) and ignores these runtime explicit values. Please cover a configured named-provider runtime whose endpoint/key differ from config, and preserve the active values without losing the named entry's API-mode behavior.

[Willhong]

Thanks — the named-custom gap was real. Pushed e50d36649.

What was wrong: when a custom:<name> entry exists in config, the safety net kept the full provider string so the named-custom arm would run (preserving its api_mode handling), but that arm reads base_url/api_key from the config entry at agent/auxiliary_client.py:5384-5389 and never looked at the explicit_base_url/explicit_api_key the fallback passed. A conversation that had already failed over to a different host or a rotated credential got retried against the stale configured endpoint.

Fix: the named-custom arm now lets an explicit runtime endpoint/credential override the entry's persisted values, while the entry keeps owning api_mode, model default, and header handling — so api_mode: anthropic_messages still builds an AnthropicAuxiliaryClient against the un-rewritten URL instead of falling into the OpenAI-wire /anthropic→/v1 rewrite. A callable api_key (Azure Entra bearer provider) is passed through unchanged rather than stringified.

Regressions added — tests/agent/test_set_runtime_main_custom_provider.py::TestConfiguredNamedCustomHonorsActiveRuntime, all three against a real config.yaml under a temp HERMES_HOME with a configured custom:rotator entry whose endpoint/key differ from the runtime (no mock on resolve_provider_client; the real client is built and its base_url/api_key asserted):

1. test_safety_net_uses_runtime_endpoint_over_config_entry — entry says https://stale-config.example/v1 + stale-config-key, runtime says https://live-failover.example/v1 + a rotated key; the safety-net client must be built at the live pair.
2. test_runtime_override_preserves_entry_api_mode — same setup with api_mode: anthropic_messages; asserts AnthropicAuxiliaryClient at the live /anthropic URL, no /v1 rewrite.
3. test_config_entry_still_used_when_runtime_carries_no_endpoint — the other direction: a runtime snapshot carrying only provider+model must leave the entry's own endpoint and credential intact.

Verified 1 and 2 fail on the previous commit (both resolve to https://stale-config.example/v1) and pass with the fix.

Test run: tests/agent/test_auxiliary_client.py, test_auxiliary_named_custom_providers.py, test_set_runtime_main_custom_provider.py, test_custom_providers_vision.py, test_image_routing.py, tests/hermes_cli/test_runtime_provider_resolution.py, test_custom_provider_identity.py → 696 passed, 1 failed. The single failure is TestCodexAuxiliaryAdapterTimeout::test_enforces_total_timeout_while_stream_keeps_emitting_events, a wall-clock assertion (< 0.14s, measured 0.178s) that fails identically on the commit before this change — unrelated and pre-existing on this machine.

[dvbaecker]

Independent verification: e50d36649 resolves the configured named-custom case

Reproduced the original bug on released v0.20.0, then re-ran the identical end-to-end reproduction against this branch.

Setup: auxiliary.vision on a configured named custom provider (custom:<name>, OpenAI-compatible vision endpoint), gateway runtime. When the aux call times out, the final safety net fires.

On v0.20.0: _try_main_agent_model_fallback() rebuilds the client from the bare "custom" label without forwarding the session-local main_runtime snapshot. Resolution has no endpoint/key, falls through _try_custom_endpoint / _resolve_api_key_provider, and picks up an unrelated API-key provider from the loaded env (in our case a text-only endpoint). The vision payload then fails with:

400 - {'error': {'message': 'Failed to deserialize the JSON body into the target type:
messages[0]: unknown variant `image_url`, expected `text` ...'}}

On this branch (e50d36649): same reproduction, same runtime snapshot (set_runtime_main("custom", ..., requested_provider="custom:<name>", base_url=..., api_key=..., api_mode="chat_completions")):

1. Fallback client resolves to the active runtime's base_url (the configured named endpoint) instead of env key scanning
2. The vision payload through the fallback client succeeds (real API response)
3. tests/agent/test_set_runtime_main_custom_provider.py: 11 passed

This is the configured named-custom regression requested in the review above (runtime endpoint/key differ from the static config entry). Verified working here against a live gateway-shaped reproduction.

[Willhong]

Rebased onto current main — conflict resolved, branch is MERGEABLE again at 24e1028b5.

The branch had gone CONFLICTING against main (2277 commits ahead of the old merge base). One real conflict, in agent/auxiliary_client.py, in the named-custom arm of resolve_provider_client():

• main (ca5ce1110, route auxiliary-client provider key reads through the profile secret scope) changed the key_env read from os.getenv(custom_key_env, "").strip() to _scoped_key_env(custom_key_env).
• This branch added the active-runtime endpoint/credential override on the lines immediately below it.

Orthogonal changes, so both are kept: the scoped-secret read resolves the entry's own key_env first, and the runtime override then supersedes custom_base / custom_key only when the live snapshot actually carries them. The entry still owns api_mode, model default, and header handling. Nothing else in the diff changed — git diff origin/main --stat is still exactly the three files (agent/auxiliary_client.py, tests/agent/test_auxiliary_client.py, tests/agent/test_set_runtime_main_custom_provider.py, +504/-19).

Rebase rather than a merge commit, so both original commits stay intact on top of main:

24e1028b5 fix(auxiliary): honor active runtime endpoint for configured named-custom fallback
2fac7f322 fix(auxiliary): follow active runtime for main fallback
55505be15 (origin/main) Merge pull request #80770 ...

Test run on the rebased head — tests/agent/test_set_runtime_main_custom_provider.py, test_auxiliary_client.py, test_auxiliary_named_custom_providers.py, test_custom_providers_vision.py, test_image_routing.py, tests/hermes_cli/test_runtime_provider_resolution.py, test_custom_provider_identity.py → 307 passed, 1 failed.

The single failure is TestCodexAuxiliaryAdapterTimeout::test_enforces_total_timeout_while_stream_keeps_emitting_events, the same pre-existing wall-clock flake reported earlier: it asserts < 0.14s and measured 0.178s. It fails identically on a clean origin/main worktree with this branch's changes nowhere in the tree, and it passes when run in isolation on both — machine load sensitivity, unrelated to this PR.

The review feedback from @teknium1 was addressed in e50d36649 (now 24e1028b5 post-rebase) and independently reproduced by @dvbaecker above; this push only re-bases that work onto current main.