fix(agent): prefer native vision over auxiliary fallback in auto mode (#29135)

[Jiahui-Gu]

Summary

Fixes #29135. In agent.image_input_mode: auto, the routing decision now checks the active main model's vision capability before the auxiliary.vision explicit-override check. This lets a single profile work for both text-only and vision-capable main models:

• Vision-capable main model (e.g. GPT-5.5, Claude, custom with supports_vision: true) -> native multimodal input, full pixels.
• Text-only main model (e.g. DeepSeek) -> falls back to the configured auxiliary.vision backend for text description.

Previously, any explicit auxiliary.vision.provider forced the text pipeline regardless of main-model capability, making the auxiliary block unusable as a true fallback.

Changes

• agent/image_routing.py: in decide_image_input_mode, run _lookup_supports_vision first; only consult _explicit_aux_vision_override when the main model does not support vision. Updated module docstring and _explicit_aux_vision_override docstring to reflect the fallback semantics.
• tests/agent/test_image_routing.py: updated the two tests that encoded the old "aux override wins" precedence, and added regression coverage for Image auto-routing cannot use native vision when auxiliary.vision is configured as fallback #29135 (native preferred when supports_vision=True, aux fallback used when False).

Test plan

• pytest tests/agent/test_image_routing.py -> 59 passed
• Vision-capable main + aux.vision configured -> native
• Text-only main + aux.vision configured -> text
• Explicit agent.image_input_mode: native / text still override auto

[alt-glitch]

Duplicate of #29138 which already fixes the same image_routing.py auto-mode precedence issue (#29135) — checking main model vision capability before auxiliary override.

[ariesmartin]

Thanks for fixing the main routing precedence here. I tested the #29135 scenario locally and agree that auto should prefer native main-model vision when the active model reports supports_vision=True, using auxiliary.vision only as the fallback path for text-only main models.

One extra edge case showed up during local validation that this PR may not cover yet:

tools.vision_tools.check_vision_requirements() currently appears to gate vision_analyze visibility on whether the auxiliary vision client can resolve. That means a stale / unresolvable auxiliary.vision config can still hide the vision_analyze tool even when the active main model has a valid native-vision fast path.

Concrete failing shape:

model:
  provider: openai-codex
  default: gpt-5.5

agent:
  image_input_mode: auto

auxiliary:
  vision:
    provider: dashscope   # or any stale/unresolvable explicit provider
    model: qwen-vl-max-latest

With the routing fix, decide_image_input_mode(...) should now return native for the main model. But if check_vision_requirements() still only checks the auxiliary client chain, it can return False, so the registered vision_analyze tool disappears even though the native fast path does not need an auxiliary client.

I tested a supplemental local patch that makes the requirements check treat native main-model vision as a valid runtime vision path before falling back to auxiliary client resolution:

def check_vision_requirements() -> bool:
    try:
        from agent.auxiliary_client import _read_main_provider, _read_main_model
        from agent.image_routing import decide_image_input_mode
        from hermes_cli.config import load_config

        cfg = load_config()
        provider = _read_main_provider()
        model = _read_main_model()
        if (
            decide_image_input_mode(provider, model, cfg) == "native"
            and _supports_media_in_tool_results(provider, model)
        ):
            return True
    except Exception:
        pass

    # existing auxiliary resolution fallback follows...

I also added tests for:

• native main-model vision surfaces vision_analyze even with no auxiliary client;
• native + auxiliary both unavailable still returns False;
• current Image auto-routing cannot use native vision when auxiliary.vision is configured as fallback #29135 routing behavior remains: vision-capable main model -> native, text-only main model -> text/aux fallback.

Local targeted test result:

venv/bin/python -m pytest tests/agent/test_image_routing.py tests/tools/test_vision_tools.py -q -o 'addopts='
149 passed, 1 warning in 4.84s

Runtime diagnostic after the supplemental patch, with the stale dashscope auxiliary config still present:

main provider/model: openai-codex gpt-5.5
supports vision: True
image mode: native
vision tool ready: True

Without this additional readiness change, #32519 fixes the routing decision but a broken auxiliary vision config can still suppress the tool that loads images into the native path. I think this PR should include that small check_vision_requirements() adjustment, or a follow-up PR should handle it so #29135 is fully closed for stale auxiliary configs.

[ariesmartin]

Thanks for fixing the core routing precedence here. I hit the same issue locally and this PR's agent/image_routing.py change matches the desired behavior: native main-model vision should win in auto mode, and auxiliary.vision should act as fallback.

One related gap I found while testing with a stale/broken auxiliary.vision config: even after routing prefers the native main model, vision_analyze can still be hidden by the tool readiness check.

The second layer is tools/vision_tools.py::check_vision_requirements(): if the main configured model supports native vision, the tool should remain available even when the auxiliary vision client is not resolvable. Otherwise a user can have:

• main provider/model supports native vision
• image routing would choose native
• but vision_analyze is unavailable because the readiness check only validates auxiliary vision setup

Local diagnostic expected in that case:

supports vision: True
image mode: native
vision tool ready: True

So I think the complete fix has two parts:

1. this PR: route auto to native vision before auxiliary fallback
2. supplemental: expose vision_analyze when native main-model vision is available, without requiring a healthy auxiliary vision client

Happy to open a small follow-up PR limited to tools/vision_tools.py + tests if you'd rather keep this PR focused.

[teknium1]

Merged via PR #57650. Your commit was cherry-picked onto current main with your authorship preserved in git log (rebase-merge). Auto mode now prefers native vision on vision-capable main models, with an explicit auxiliary.vision block acting as the fallback for text-only models. Thanks!