Skip to content

fix: read auxiliary.vision.model from config.yaml in vision/browser tools - #24872

Closed
vanthinh6886 wants to merge 1 commit into
NousResearch:mainfrom
vanthinh6886:fix/vision-config-hardcoded
Closed

fix: read auxiliary.vision.model from config.yaml in vision/browser tools#24872
vanthinh6886 wants to merge 1 commit into
NousResearch:mainfrom
vanthinh6886:fix/vision-config-hardcoded

Conversation

@vanthinh6886

Copy link
Copy Markdown
Contributor

Problem

vision_analyze, browser_vision, and video_analyze tools only read the vision model from the AUXILIARY_VISION_MODEL environment variable, ignoring the auxiliary.vision.model setting in config.yaml.

When users configure a custom vision model via config.yaml:

auxiliary:
  vision:
    model: nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free
    api_key: sk-...

The tools still use the auto-detected default (often google/gemma-4-31b-it:free from Nous Portal free tier), making it impossible to work around rate limits by switching to a different vision model.

Fix

Each tool now reads from config.yaml as fallback when the env var is not set.

Resolution order: env var (AUXILIARY_VISION_MODEL) > auxiliary.vision.model in config.yaml > None (lets auxiliary router auto-detect)

Files changed:

  • tools/browser_tool.py: _get_vision_model() now reads auxiliary.vision.model from config
  • tools/vision_tools.py: _handle_vision_analyze and _handle_video_analyze now read auxiliary.vision.model / auxiliary.video.model from config

Before vs After

Scenario Before After
auxiliary.vision.model set in config, no env var Ignored (auto-detect picks Gemma) Respected ✓
AUXILIARY_VISION_MODEL env var set Used ✓ Used ✓ (still takes precedence)
Neither set Auto-detect Auto-detect (unchanged)

Tests

  • py_compile passes on both modified files
  • 189 vision-related tests pass (0 regressions)
  • Pre-existing test failures (missing optional deps) unchanged

Fixes #24842

…ools

vision_analyze, browser_vision, and video_analyze tools only read the
vision model from the AUXILIARY_VISION_MODEL environment variable,
ignoring the auxiliary.vision.model setting in config.yaml.

When users configure a custom vision model via config.yaml (e.g.
auxiliary.vision.model: nvidia/nemotron-3-nano-omni-30b-a3b-reasoning:free),
the tools would still use the auto-detected default (often Gemma from
Nous Portal free tier), making it impossible to work around rate limits.

Fix: each tool now reads from config.yaml as fallback when the env var
is not set. Resolution order: env var > config.yaml > None (auto-detect).

Files changed:
- tools/browser_tool.py: _get_vision_model() now reads auxiliary.vision.model
- tools/vision_tools.py: _handle_vision_analyze and _handle_video_analyze
  now read auxiliary.vision.model / auxiliary.video.model from config

Fixes NousResearch#24842
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists tool/vision Vision analysis and image generation tool/browser Browser automation (CDP, Playwright) area/config Config system, migrations, profiles labels May 13, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Competing fix for #24842 — also see #24875 which addresses the same vision config routing issue by removing stale env-var bridges entirely in favor of call_llm(task="vision").

@DavidMetcalfe

Copy link
Copy Markdown
Contributor

Superseded by merged #57651 (commit 0e9136c), which incorporates the same config.yaml fallback mechanism. Closing to avoid confusion.

@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the focused configuration-routing fix. This is an automated hermes-sweeper review; current main already provides the requested behavior, so this PR is superseded.

  • tools/browser_tool.py:4125 calls the centralized call_llm(task="vision") path; it only supplies an explicit model when AUXILIARY_VISION_MODEL is set.
  • agent/auxiliary_client.py:5857-5902 resolves an explicit model first, then auxiliary.<task>.model from config.yaml, so browser_vision receives auxiliary.vision.model when no env override is present.
  • tools/vision_tools.py:1498-1510 and tools/vision_tools.py:1873-1886 implement config-first routing for vision_analyze and video_analyze.
  • Commit 149641485c7f5bcb33f6acf11544f0a816d8b054 is on main and is contained in v2026.7.7.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles P2 Medium — degraded but workaround exists sweeper:implemented-on-main Sweeper: behavior already present on current main tool/browser Browser automation (CDP, Playwright) tool/vision Vision analysis and image generation type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: vision_analyze and browser_vision hardcoded to Gemma, ignore auxiliary.vision config

4 participants