Skip to content

fix(image-routing): check native vision support before aux override - #44320

Closed
liuhao1024 wants to merge 1 commit into
NousResearch:mainfrom
liuhao1024:fix/aux-vision-blocks-native
Closed

fix(image-routing): check native vision support before aux override#44320
liuhao1024 wants to merge 1 commit into
NousResearch:mainfrom
liuhao1024:fix/aux-vision-blocks-native

Conversation

@liuhao1024

Copy link
Copy Markdown
Contributor

Problem

When auxiliary.vision is explicitly configured in config.yaml, _explicit_aux_vision_override() forces all images through the auxiliary vision LLM text pipeline — even when the main model itself supports vision natively (e.g. mimo-v2.5 with supports_vision=True).

This causes two issues:

  1. Unnecessary latency — images are sent to an auxiliary vision model instead of being handled directly by the main model
  2. Provider failures — providers like Xiaomi MiMo that accept images in user messages but reject them in tool_result messages get HTTP 400 errors when images are routed through the auxiliary pipeline

Root Cause

In agent/image_routing.py, decide_image_input_mode() checks _explicit_aux_vision_override() before checking native vision support:

# Before (buggy)
if _explicit_aux_vision_override(cfg):
    return "text"  # Always, regardless of main model capabilities

supports = _lookup_supports_vision(provider, model, cfg)
if supports is True:
    return "native"  # Never reached when aux is configured

Fix

Swap the check order: verify native vision support first, then fall through to aux override only when the main model does NOT support vision:

# After (fixed)
supports = _lookup_supports_vision(provider, model, cfg)
if supports is True:
    return "native"  # Main model can see images → use it

if _explicit_aux_vision_override(cfg):
    return "text"  # Main model can't see → use aux if configured

Changes

  • agent/image_routing.py — Swap check order in decide_image_input_mode()
  • tests/agent/test_image_routing.py — Update 2 tests to reflect new priority (native wins over aux), add 1 test for aux fallback when main model lacks vision

Testing

77 passed in 0.35s (test_image_routing.py)
30 passed in 0.23s (test_computer_use_vision_routing.py)

Closes #44299

When the main model supports vision natively (supports_vision=True),
use it directly instead of routing through the auxiliary vision LLM.
This fixes a bug where explicit auxiliary.vision config unconditionally
blocked native vision, adding unnecessary latency and causing failures
for providers that reject multimodal tool_result messages (e.g. Xiaomi
MiMo).

The fix swaps the check order in decide_image_input_mode(): native
vision support is checked first, and the aux override only applies
when the main model does NOT support vision.

Closes NousResearch#44299
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint tool/vision Vision analysis and image generation duplicate This issue or pull request already exists labels Jun 11, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Duplicate of #44305 — same reorder of decide_image_input_mode() (check native vision support before _explicit_aux_vision_override), both implementing the bug half of #44299 and descending from #32519/#29135. #44305 was opened earlier; consolidating there.

@liuhao1024

Copy link
Copy Markdown
Contributor Author

Thanks for the flag @alt-glitch. Comparing the two PRs:

Both implement the same fix for #44299. The code changes are equivalent; this PR adds slightly more test coverage. Happy to close if #44305 is preferred.

@liuhao1024

Copy link
Copy Markdown
Contributor Author

Thanks for the flag @alt-glitch. Confirmed — #44305 has the same reorder (check native vision before aux override) and was opened earlier the same day. Closing in favor of #44305.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint duplicate This issue or pull request already exists P2 Medium — degraded but workaround exists tool/vision Vision analysis and image generation type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

auxiliary.vision explicit config blocks native vision + need image-in-user-message injection for providers like Xiaomi MiMo

2 participants