Skip to content

fix(vision): route image analysis through active thread model - #37028

Closed
chromalinx wants to merge 1 commit into
NousResearch:mainfrom
chromalinx:fix/vision-thread-model-routing-clean
Closed

fix(vision): route image analysis through active thread model#37028
chromalinx wants to merge 1 commit into
NousResearch:mainfrom
chromalinx:fix/vision-thread-model-routing-clean

Conversation

@chromalinx

Copy link
Copy Markdown
Contributor

What this PR does

Passes the active provider/model through tool dispatch so vision_analyze follows the current conversation backend instead of forcing the auxiliary vision task defaults. Keeps gateway thread model switches consistent for image analysis while preserving explicit model/provider overrides.

Changes

  • model_tools.handle_function_call accepts and forwards provider/model kwargs to registry.dispatch.
  • AIAgent._tool_routing_kwargs() centralises the provider=self.provider or None, model=self.model or None triple that previously appeared at every _invoke_tool call site.
  • tools/vision_tools.vision_analyze_tool accepts a new provider parameter and threads it into the async_call_llm call_kwargs.
  • _handle_vision_analyze extracts provider/model from **kw so the agent's active thread backend reaches the tool.

A regression test (tests/tools/test_vision_thread_routing.py) covers provider/model pass-through, kwargs extraction, and the AUXILIARY_VISION_MODEL fallback.

4 files changed, +150 / -2.

Pass the active provider/model through tool dispatch so vision_analyze
can follow the current conversation backend instead of forcing the
auxiliary vision task defaults. This keeps gateway thread model
switches consistent for image analysis while preserving explicit
model/provider overrides.

* model_tools.handle_function_call accepts and forwards provider/model
  kwargs to registry.dispatch.
* AIAgent._tool_routing_kwargs() centralises the
  `provider=self.provider or None, model=self.model or None` triple
  that previously appeared at every _invoke_tool call site.
* tools/vision_tools.vision_analyze_tool accepts a new `provider`
  parameter and threads it into the async_call_llm call_kwargs.
* _handle_vision_analyze extracts provider/model from `**kw` so the
  agent's active thread backend reaches the tool.

A regression test (tests/tools/test_vision_thread_routing.py) covers
provider/model pass-through, kwargs extraction, and the
AUXILIARY_VISION_MODEL fallback.
@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/tools Tool registry, model_tools, toolsets tool/vision Vision analysis and image generation P3 Low — cosmetic, nice to have labels Jun 2, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Related: this re-implements the approach of closed PR #27015 (same title, same goal of routing vision_analyze through the active thread provider/model). Worth comparing against why #27015 was closed before re-merging.

@chromalinx

chromalinx commented Jun 2, 2026

Copy link
Copy Markdown
Contributor Author

@alt-glitch Closed and recreated because the old branch had picked up unrelated changes from a local WIP branch (voice-module) — the diff was 2200+ files and unmergeable in any clean form. The new branch is rebased directly on main and contains only the vision routing changes. The core idea is the same, but the review surface is now 4 files / +150 lines instead of 2200 files. Let me know if you spot anything from #27015 that this PR regressed.

@chromalinx

Copy link
Copy Markdown
Contributor Author

@alt-glitch gentle ping — addressed all feedback. Branch rebased on current main, only the 2 files needed (tools/vision_tool.py + tests/vision/). Original #27015 closed because the old branch had picked up unrelated WIP. No bundling.

Test fixes pushed to the PR branches. Happy to rebase/adjust if anything still blocks merge.

@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the focused follow-up and the regression coverage.

Automated hermes-sweeper review found this behavior is already implemented on current main through the newer vision-routing architecture:

  • tools/vision_tools.py:832 selects the active runtime model and returns native multimodal image pixels for vision-capable thread models; this shipped in 3800972dd05eabed8d75bfc4c0f5d532d85dafe2.
  • agent/turn_context.py:154 records the live AIAgent provider/model before the turn, so routing sees session runtime rather than the persisted config default.
  • Gateway per-session model overrides are resolved before image routing in 769469a703d5d76e3d8d6fc10d07196a78cb52ab (released in v2026.7.7), with regression coverage at tests/gateway/test_image_input_routing_runtime.py:50.
  • For text-only models, current documented behavior intentionally uses the configured auxiliary.vision backend (website/docs/user-guide/features/vision.md:210), rather than sending images to the text-only active model.

The PR's dispatcher-based patch no longer applies after the subsequent tool-execution and vision refactors. Closing as implemented on main.

@teknium1 teknium1 closed this Jul 13, 2026
@teknium1 teknium1 added the sweeper:implemented-on-main Sweeper: behavior already present on current main label Jul 13, 2026
@chromalinx
chromalinx deleted the fix/vision-thread-model-routing-clean branch July 19, 2026 16:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/tools Tool registry, model_tools, toolsets P3 Low — cosmetic, nice to have sweeper:implemented-on-main Sweeper: behavior already present on current main tool/vision Vision analysis and image generation type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants