Skip to content

fix(vision): respect auxiliary vision routing - #24875

Open
felix-windsor wants to merge 1 commit into
NousResearch:mainfrom
felix-windsor:codex/fix-vision-aux-config
Open

fix(vision): respect auxiliary vision routing#24875
felix-windsor wants to merge 1 commit into
NousResearch:mainfrom
felix-windsor:codex/fix-vision-aux-config

Conversation

@felix-windsor

Copy link
Copy Markdown
Contributor

Summary

  • stop vision_analyze from passing stale AUXILIARY_VISION_MODEL values as explicit model overrides
  • let browser_vision rely on call_llm(task="vision") so auxiliary.vision provider/model/base_url/api_key resolve together
  • update regression tests around stale env bridges and browser screenshot vision routing

Fixes #24842

Tests

  • scripts/run_tests.sh tests/tools/test_vision_tools.py tests/tools/test_browser_console.py tests/tools/test_vision_native_fast_path.py tests/agent/test_vision_resolved_args.py tests/agent/test_auxiliary_config_bridge.py
  • .venv/bin/ruff check tools/vision_tools.py tools/browser_tool.py tests/tools/test_vision_tools.py tests/tools/test_browser_console.py tests/agent/test_auxiliary_config_bridge.py

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint tool/vision Vision analysis and image generation tool/browser Browser automation (CDP, Playwright) labels May 13, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Competing fix for #24842 — also see #24872 which addresses the same vision config routing issue with a simpler env-var-then-config fallback approach.

@felix-windsor

Copy link
Copy Markdown
Contributor Author

Rebased on latest main (as of 2026-05-18) and force-pushed.

Local verification:

  • ./scripts/run_tests.sh tests/tools/test_vision_tools.py

(Still seeing no CI statuses reported on the PR head; may need maintainer Actions approval for fork PRs.)

@felix-windsor
felix-windsor force-pushed the codex/fix-vision-aux-config branch from 00904ef to 51d7342 Compare May 18, 2026 12:36

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for isolating the stale browser model override; that path remains real on current main: tools/browser_tool.py:4103 reads AUXILIARY_VISION_MODEL and :4140-4144 forwards it explicitly, while agent/auxiliary_client.py:5866-5902 gives explicit models precedence over auxiliary.vision.

Problems

  • The vision_analyze change removes the environment-only compatibility override. AUXILIARY_VISION_MODEL remains documented at website/docs/reference/environment-variables.md:801, and current coverage at tests/tools/test_vision_tools.py:220-238 asserts that it is forwarded. The task-config helper reads config only (agent/auxiliary_client.py:5996-6037), so passing None does not preserve this fallback.

Suggested changes

  • Keep config authoritative while retaining environment-only fallback, either in a shared resolver or with config-first resolution at the tool boundary.
  • Cover both config-vs-stale-env precedence and env-only compatibility through the real auxiliary resolution path.

Automated hermes-sweeper review.

Comment thread tools/vision_tools.py
@@ -1043,8 +1044,7 @@ def _handle_vision_analyze(args: Dict[str, Any], **kw: Any) -> Awaitable[str]:
"Fully describe and explain everything about this image, then answer the "
f"following question:\n\n{question}"
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Passing None unconditionally removes the documented environment-only AUXILIARY_VISION_MODEL override. Please retain it as a fallback when auxiliary.vision.model is unset, while making config win when both are present.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades tool/browser Browser automation (CDP, Playwright) tool/vision Vision analysis and image generation type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: vision_analyze and browser_vision hardcoded to Gemma, ignore auxiliary.vision config

3 participants