Skip to content

fix(agent): prevent text-only main model from leaking into vision provider client (#57948) - #60991

Closed
webtecnica wants to merge 1 commit into
NousResearch:mainfrom
webtecnica:fix/57948-vision-model-fallback-text-only
Closed

webtecnica wants to merge 1 commit into
NousResearch:mainfrom
webtecnica:fix/57948-vision-model-fallback-text-only

Conversation

@webtecnica

Copy link
Copy Markdown
Contributor

Summary

Fixes a bug where vision_analyze returns a 400 "Unexpected item type in content" on its first call when the main model is text-only (e.g. deepseek-chat) and no vision auxiliary model is explicitly configured. The error occurs because the text-only main model leaks into the vision provider's client initialization.

Root Cause

In resolve_provider_client(), when model is None and provider is not "auto", the fallback logic calls:

model = _get_aux_model_for_provider(provider) or _read_main_model() or model

This unconditionally falls back to _read_main_model() — the main runtime model — even when the client is being created for a vision task. If the main model is text-only, it gets injected as the model name for the vision provider client, causing the upstream API to reject the request with a 400 error on the very first message (since a text-only model receives image content it cannot process).

Change

In agent/auxiliary_client.py, the vision-aware fallback now:

  1. Uses _get_aux_model_for_provider(provider) if available (same as before).
  2. Falls back to _read_main_model() only when is_vision is False — vision tasks skip this fallback entirely.
# Before:
model = _get_aux_model_for_provider(provider) or _read_main_model() or model

# After:
aux_model = _get_aux_model_for_provider(provider)
if aux_model:
    model = aux_model
elif not is_vision:
    # Vision tasks must NOT inherit the text-only main model
    model = _read_main_model() or model

Verification

  • 326 tests pass, 9 pre-existing failures (unrelated to this change)
  • The fix targets the exact call path: resolve_provider_client → fallback model assignment for vision tasks
  • No other call sites are affected — the is_vision guard is specific to this single entry point

@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint tool/vision Vision analysis and image generation provider/deepseek DeepSeek API P2 Medium — degraded but workaround exists labels Jul 8, 2026
@teknium1

Copy link
Copy Markdown
Collaborator

Thanks for the focused investigation. Current main already provides the requested protection, so this PR is redundant.

  • agent/auxiliary_client.py:5316-5406 checks whether the active main provider/model supports vision and skips known text-only models before building a client; the resolver then uses the OpenRouter/Nous vision fallback chain.
  • tests/agent/test_vision_routing_31179.py:143-166 covers a DeepSeek text-only main model and asserts that auto vision resolution does not return a client that would receive image content.
  • This behavior landed in 3d66787a04d29beaec16a723e4c2c2bb4dda2cee (fix(vision): ... skip text-only main, fix(vision): route auxiliary.vision.provider=openai to api.openai.com, skip text-only main #31452), included in v2026.5.28.

Automated hermes-sweeper review.

@teknium1 teknium1 closed this Jul 10, 2026
@teknium1 teknium1 added the sweeper:implemented-on-main Sweeper: behavior already present on current main label Jul 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists provider/deepseek DeepSeek API sweeper:implemented-on-main Sweeper: behavior already present on current main tool/vision Vision analysis and image generation type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants