Skip to content

feat(agent): expose fallback model awareness to the agent - #26569

Closed
zccyman wants to merge 1 commit into
NousResearch:mainfrom
atyou2happy:feat/fallback-model-awareness-25852
Closed

feat(agent): expose fallback model awareness to the agent#26569
zccyman wants to merge 1 commit into
NousResearch:mainfrom
atyou2happy:feat/fallback-model-awareness-25852

Conversation

@zccyman

@zccyman zccyman commented May 15, 2026

Copy link
Copy Markdown
Contributor

Summary

When Hermes Agent's fallback mechanism activates (primary model fails, backup kicks in), the agent (LLM) now receives a notification in its system prompt indicating the active runtime model. This enables model-aware response behaviors like correct model code prefixes.

Closes #25852

Problem

The fallback mechanism was completely transparent to the agent. When a fallback kicked in, the system prompt still showed the configured primary model. Users who prefix responses with model codes (e.g., G31P-T for gemini-3.1-pro) would get wrong prefixes after fallback.

Solution

Three targeted changes:

  1. run_agent.py - ephemeral_system_prompt injection: When _try_activate_fallback() succeeds, a note is appended to ephemeral_system_prompt (per-call, not cached) telling the agent which fallback model/provider is active. Using ephemeral_system_prompt avoids breaking prefix cache stability.

  2. run_agent.py - cleanup on primary restoration: When _restore_primary_runtime() runs at turn start, the fallback note is stripped from ephemeral_system_prompt so subsequent turns don't carry stale messages.

  3. cli.py - /model status display: When /model is invoked with no args during an active fallback, a status line shows the primary to fallback transition.

Files Changed

File Change
run_agent.py +26 lines: fallback note injection + cleanup on restore
cli.py +9 lines: fallback status in model picker

Test Results

tests/run_agent/test_fallback_model.py - 31 passed (162.92s)

All existing fallback tests pass without modification.

Design Decisions

  • ephemeral_system_prompt over _cached_system_prompt: The cached system prompt is never rebuilt mid-session (prefix cache stability). ephemeral_system_prompt is injected per-API-call, making it the right place for transient runtime state.
  • Append-only note: The fallback note is appended to any existing ephemeral_system_prompt content, preserving user/config-set ephemeral prompts.
  • String marker cleanup: On primary restoration, the note is removed by finding the unique marker string, avoiding fragile regex or line-count assumptions.

Inject fallback notification into ephemeral_system_prompt when the primary
model fails and a fallback activates. Also show fallback status in the
/model picker when a fallback is active.

Closes NousResearch#25852
@alt-glitch alt-glitch added type/feature New feature or request P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard labels May 15, 2026
@zccyman
zccyman force-pushed the feat/fallback-model-awareness-25852 branch from 585a8d7 to e70032a Compare May 18, 2026 07:48
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the clear fallback-awareness proposal. This is already implemented on current main via a cache-safe, broader fix. Automated hermes-sweeper review.

  • c884ff64eaab0b5002e9bb703a9d3075b8dd8387 updates the cached prompt's Model: and Provider: identity on fallback activation at agent/chat_completion_helpers.py:1638-1640.
  • agent/conversation_loop.py:497-520 synchronizes that rewritten identity into the in-flight system message, including ephemeral prompt content, so the fallback retry in the current turn receives it.
  • agent/agent_runtime_helpers.py:1318-1321 restores the primary identity byte-identically for prefix-cache reuse.
  • tests/agent/test_failover_identity.py:36-89 covers fallback identity replacement, restoration, and in-flight message synchronization.
  • The implementing commit is contained in release tag v2026.7.1.

The linked request #25852 was also closed as implemented with this same commit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard P2 Medium — degraded but workaround exists sweeper:implemented-on-main Sweeper: behavior already present on current main type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature Request: Runtime fallback model awareness — expose active model to the agent

3 participants