Skip to content

fix(vision): describe images via aux vision on text-only model failover - #83150

Open
viz-A-viz wants to merge 1 commit into
NousResearch:mainfrom
viz-A-viz:fix/failover-vision-description
Open

viz-A-viz wants to merge 1 commit into
NousResearch:mainfrom
viz-A-viz:fix/failover-vision-description

Conversation

@viz-A-viz

@viz-A-viz viz-A-viz commented Aug 10, 2026

Copy link
Copy Markdown

Problem

When the primary model (vision-capable) dies mid-turn and the fallback chain lands on a text-only model (e.g. DeepSeek), two bugs occur:

  1. Inline image parts in api_messages cause 400 errors — the text-only model rejects image_url parts it cannot process.

  2. vision_analyze tool still uses the native fast path_should_use_native_vision_fast_path() reads stale runtime-main globals (still pointing at the vision-capable primary), so every vision_analyze call returns image bytes that the text-only fallback model cannot process.

Fix

_sync_failover_vision_support (called after every provider failover):

  • Looks up the failover model's vision capability via _lookup_supports_vision
  • If text-only and image parts present — describes each image via the auxiliary vision LLM and replaces the part in place with a text description
  • Falls back to a plaintext placeholder if aux vision fails
  • Sets agent._vision_supported = False
  • Sets module-level _failover_vision_disabled flag in vision_tools

_should_use_native_vision_fast_path (vision_tools.py):

  • Checks _failover_vision_disabled at the top — returns False when set, forcing vision_analyze through the auxiliary vision text path

turn_context.build_turn_context:

  • Clears _failover_vision_disabled on primary restore, re-enabling the native fast path when the vision-capable primary is back

Tests

TestFailoverVisionSupport (5 cases): text-only replacement, aux-vision failure placeholder, vision-capable pass-through, already-unsupported skip, tool-message linkage.

TestFailoverVisionDisablesFastPath (2 cases): text-only failover sets the vision_tools flag, fast path returns False when flag is set.

18 passed in 0.64s

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint tool/vision Vision analysis and image generation labels Aug 10, 2026
When the primary model (vision-capable) dies mid-turn and the fallback
chain lands on a text-only model, two bugs occur:

1. Inline image_url parts in api_messages cause HTTP 400 — the text-only
   model rejects image bytes it cannot process.

2. vision_analyze tool calls still use the native fast path —
   _should_use_native_vision_fast_path reads stale runtime-main globals
   still pointing at the vision-capable primary, so every vision_analyze
   call returns image bytes that the text-only fallback cannot process.

Fix:

_sync_failover_vision_support (called after every provider failover):
- Looks up the failover model's vision capability via
  _lookup_supports_vision
- If text-only and image parts present — describes each image via the
  auxiliary vision LLM and replaces the part in place with text
- Falls back to a plaintext placeholder if aux vision fails
- Sets agent._vision_supported = False
- Sets module-level _failover_vision_disabled flag in vision_tools

_should_use_native_vision_fast_path (vision_tools.py):
- Checks _failover_vision_disabled at the top — returns False when set,
  forcing vision_analyze through the auxiliary vision text path

turn_context.build_turn_context:
- Clears _failover_vision_disabled on primary restore, re-enabling the
  native fast path when the vision-capable primary is back

Co-authored-by: Hermes Agent <hermes-agent[bot]@users.noreply.github.com>
@viz-A-viz
viz-A-viz force-pushed the fix/failover-vision-description branch from 7d4c3e6 to a6e742d Compare August 11, 2026 01:47
@viz-A-viz viz-A-viz changed the title fix(conversation_loop): describe images via aux vision on text-only failover fix(vision): describe images via aux vision on text-only model failover Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists tool/vision Vision analysis and image generation type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants