Skip to content

fix(vision): resolve Nous vision model correctly in auto-detect path - #12687

Closed
Ifkellx wants to merge 1 commit into
NousResearch:mainfrom
Ifkellx:fix/nous-vision-model
Closed

Ifkellx wants to merge 1 commit into
NousResearch:mainfrom
Ifkellx:fix/nous-vision-model

Conversation

@Ifkellx

@Ifkellx Ifkellx commented Apr 19, 2026

Copy link
Copy Markdown
Contributor

Problem

The vision auto-detect chain calls resolve_provider_client() with the vision model from _PROVIDER_VISION_MODELS, but resolve_provider_client() always called _try_nous() without vision=True. This caused it to return the default text model instead of the vision-capable xiaomi/mimo-v2-omni, resulting in 404 errors from the Nous inference API when sending images.

Additionally, _PROVIDER_VISION_MODELS was missing an entry for the nous provider.

Root Cause

The auto-detect path in resolve_vision_provider_client():

  1. Looks up _PROVIDER_VISION_MODELS.get("nous") → returns xiaomi/mimo-v2-omni
  2. Calls resolve_provider_client("nous", model="xiaomi/mimo-v2-omni")
  3. resolve_provider_client calls _try_nous() without vision=True
  4. _try_nous() ignores the passed model, returns the default text model

The fallback path (_resolve_strict_vision_backend) worked correctly because it called _try_nous(vision=True) directly.

Fix

  1. _PROVIDER_VISION_MODELS: Added "nous": "xiaomi/mimo-v2-omni" entry so the vision auto-detect chain picks the correct multimodal model.

  2. resolve_provider_client: Auto-detects vision tasks by checking if the requested model matches a value in _PROVIDER_VISION_MODELS or is a known vision model name, then passes vision=True to _try_nous().

Verification

  • xiaomi/mimo-v2-omni returns HTTP 200 with image inputs on Nous inference API
  • google/gemini-3-flash-preview returns 404 with image inputs on Nous inference API
  • Free tier Nous accounts: only Xiaomi models are available, making this fix essential

Impact

Fixes browser_vision and vision_analyze tools for all Hermes users on Nous (both free and paid tiers).

Two changes:
1. _PROVIDER_VISION_MODELS: add 'nous' -> 'xiaomi/mimo-v2-omni' entry
   so the vision auto-detect chain picks the correct multimodal model.

2. resolve_provider_client: detect when the requested model is a vision
   model (from _PROVIDER_VISION_MODELS or known vision model names) and
   pass vision=True to _try_nous().  Previously, _try_nous() was always
   called without vision=True in resolve_provider_client(), causing it to
   return the default text model (gemini-3-flash-preview or mimo-v2-pro)
   instead of the vision-capable mimo-v2-omni.

The _try_nous() function already handled free-tier vision correctly, but
the resolve_provider_client() path (used by the auto-detect vision chain)
never signaled that a vision task was in progress.

Verified: xiaomi/mimo-v2-omni returns HTTP 200 with image inputs on Nous
inference API. google/gemini-3-flash-preview returns 404 with images.
@teknium1

Copy link
Copy Markdown
Collaborator

Merged via PR #13699 (#13699) — your commit was cherry-picked onto current main with your authorship preserved in git log (commit 0301787). Thanks for the fix! Pairs with #13682 (@helix4u) which landed minutes earlier and fixes the stale-credential side of the same vision report Tom-in-Tampa filed.

@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint tool/vision Vision analysis and image generation provider/nous Nous Research API (OAuth) provider/xiaomi Xiaomi MiLM labels Apr 21, 2026
@alt-glitch

Copy link
Copy Markdown
Contributor

Duplicate of #13699 — salvaged and merged as #13699 by teknium1.

aj-nt pushed a commit to aj-nt/hermes-agent that referenced this pull request May 1, 2026
Luminet2023 pushed a commit to Luminet2023/hermes-agent that referenced this pull request May 1, 2026
Luminet2023 added a commit to Luminet2023/hermes-agent that referenced this pull request May 1, 2026
* upstream/main: (160 commits)
  feat(llm-wiki): port provenance markers, source hashing, and quality signals from llm-wiki-compiler (NousResearch#13700)
  fix(vision): restore tier-aware Nous vision model selection (NousResearch#13703)
  fix(vision): route Nous main-provider vision through tier-aware backend
  test(approval): regression guards for thread-local callback contract
  fix(security): TUI approval overlay accepts blind keystrokes, CLI thread-local callback invisible to agent
  chore(release): add Ifkellx to AUTHOR_MAP for PR NousResearch#12687
  fix(vision): resolve Nous vision model correctly in auto-detect path
  docs(delegate): clarify that the parent agent, not the user, populates goal/context (NousResearch#13698)
  fix(auxiliary): refresh Nous runtime credentials after aux 401s
  feat(delegate): orchestrator role and configurable spawn depth (default flat)
  fix(tui): inject VS16 so text-default emoji render as color glyphs
  feat(gateway): recognize .pdf in MEDIA: tag extraction (NousResearch#13683)
  test(delegate): make default_toolsets regression test robust to user config
  docs(delegate): remove default_toolsets from example config and docs
  refactor(delegate): drop dead default_toolsets from CLI default config
  feat(image-gen): add GPT Image 2 to FAL catalog (NousResearch#13677)
  fix(gateway): always inject reply-to pointer, not just when quoted text is absent (NousResearch#13676)
  fix(skills/llama-cpp): concise description, restore python bindings, fix curl
  improve llama.cpp skill
  fix(tui): route skills.manage through the long-handler thread pool
  ...

# Conflicts:
#	tools/file_tools.py
#	uv.lock
02356abc pushed a commit to 02356abc/hermes-agent that referenced this pull request May 14, 2026
gweeteve pushed a commit to gweeteve/hermes-agent that referenced this pull request Jun 2, 2026
waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request Jul 2, 2026
prmartinow pushed a commit to prmartinow/hermes-agent that referenced this pull request Aug 26, 2026
melon-xf added a commit to melon-xf/hermes-agent that referenced this pull request Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint provider/nous Nous Research API (OAuth) provider/xiaomi Xiaomi MiLM tool/vision Vision analysis and image generation type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants