fix(vision): treat a model without image support as an answer, not a crash

[jackulau]

What does this PR do?

A provider rejecting a request because the model has no vision is a well-formed request the model cannot answer. Three call sites reach the same auxiliary vision model by the same route, and each mishandled that rejection differently. This makes all three treat it as an answer.

The reported traceback is not an unhandled exception. vision_analyze_tool already caught, classified and answered this; it just logged the caught exception at ERROR with exc_info=True first. The stack quoted in #89114 as an "unhandled openai.BadRequestError" is that log line. A handled, classified, user-actionable outcome printed exactly like a crash cost the reporter a triage cycle, and would cost the next person one.

The defect they actually felt is in computer_use. _route_capture_through_aux_vision reads analysis out of vision_analyze_tool's envelope without checking success. That field is populated on the failure path too — with an explanation of why it could not look, not a description of what it saw — so the rejection was merged into the response as vision_analysis. The main model was told the screenshot shows Error code: 400 - {...}.

The fix is small because the graceful degradation the issue asks for already exists in the caller and simply never fired. _capture_response omits the screenshot, flags vision_unavailable: true, and tells the model "element-index actions still work — drive via the element list above." It is guarded on _route_capture_through_aux_vision returning None, which its own docstring says it does "on failure". It never reported this failure. Now it does — items 2 and 3 of the issue's Expected Behavior, using machinery already in the tree.

Why not item 1 (pre-flight capability detection). Deliberately not implemented. There is no reliable capability signal for an arbitrary OpenAI-compatible endpoint — /models does not carry modality for most providers, and the reporter's is a Go proxy in front of another vendor, so a local capability table would be wrong for exactly the setups that hit this. A static allowlist would also fail closed on new multimodal models. The provider's 400 is the capability check; it only has to be read as an answer. That costs one wasted call the first time, which is the better trade than silently refusing a model that would have worked. A per-model+provider negative cache is a reasonable separable follow-up and shouldn't gate this.

Related Issue

Fixes #89114

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

tools/vision_tools.py

• New public is_vision_capability_error(error) — the shared predicate. Public (no leading underscore) because tools/browser_tool.py imports it; the two tools reach the same aux model by the same route and must recognise the same rejection.
• vision_analyze_tool's handler sets classified across the existing if/elif chain and logs after it: WARNING without a stack for a classified failure, ERROR + exc_info=True kept for the unclassified branch, where a traceback earns its place.
• The inline capability-hint tuple is replaced by is_vision_capability_error(e) or "content_policy" in err_str. content_policy is deliberately excluded from the shared predicate — it is not a capability problem, and browser_vision should not tell a user to change auxiliary.vision because of it — but it still routes to the same branch here, so the truth table is unchanged. There is a test pinning exactly that.

tools/browser_tool.py

• browser_vision's handler classifies before responding. Capability rejections become a sentence that names the config key that fixes it (auxiliary.vision); billing rejections get their own remedy; anything unrecognised keeps the raw error text and the traceback. Previously it classified nothing, so the same rejection that produced a sentence through vision_analyze handed the agent a raw Error code: 400 - {...} blob.

tools/computer_use/tool.py

• _route_capture_through_aux_vision returns None when the envelope reports success: false, logging what the aux model said. This routes the failure into the caller's existing vision_unavailable degradation instead of presenting a failed analysis as vision_analysis.
• Docstring corrected: the caller degrades to its AX/SOM payload, not to the multimodal envelope (that changed in the vision_unavailable work and the docstring did not follow).

tests/tools/test_vision_capability_error_surface.py — new, 20 tests.

How to Test

Reproduce the reported setup — point auxiliary.vision at any text-only model and analyse a PNG:

hermes --toolsets vision -q "Use vision_analyze on ./shot.png and describe it"

Before: an ERROR line with a full openai.BadRequestError stack (the traceback in the issue). After: a WARNING with the same message, no stack, and the same success: false JSON the tool has always returned.

The computer_use half — the one that changes behaviour rather than logging:

hermes --toolsets computer_use -q "Capture the screen and tell me what app is in front"

Before: the model receives "vision_analysis": "<model> does not support vision ... Error code: 400 - {...}" under a key promising a description of the screen. After: "vision_unavailable": true, the screenshot omitted, and the AX/SOM element index intact so index-driven actions still work.

Automated:

pytest tests/tools/test_vision_capability_error_surface.py -q      # 20 passed

Mutation proof — each production change is independently load-bearing. Reverting one fails only its own test(s):

Reverted	Tests that fail
is_vision_capability_error → return False	8 (6 predicate cases + vision_analyze message + browser_vision message)
browser_vision's classification block	1 — test_capability_rejection_becomes_a_sentence_not_a_400_blob
the classified log split → unconditional logger.error(..., exc_info=True)	1 — test_classified_failure_logs_warning_without_a_traceback
computer_use's success is False guard	1 — test_capability_failure_degrades_to_the_ax_payload

Baseline over the affected slice (tests/tools/test_vision_*.py, test_browser_content_none_guard.py, test_computer_use*.py, tests/agent/test_vision_routing_31179.py, tests/gateway/test_vision_preprocess.py), run serially with and without the change: 6 failures both ways, the same 6 — all pre-existing and unrelated to this diff. 200 → 220 passed, the delta being exactly the new file.

For the record, since I had to characterise them: two of the six (TestBrowserSourceLinesAreGuarded in test_browser_content_none_guard.py) are a genuine Windows-only test bug — open(path) with no encoding= decodes tools/browser_tool.py as cp1252 and dies on the emoji in the tool registration. Three are Linux/env-specific test_computer_use.py cases, and one is test_vision_routing_31179.py::test_vision_capable_main_used failing to resolve a client in this environment. I left all six alone rather than widen this PR; happy to send the one-line encoding="utf-8" fix separately if that is wanted.

Overlap with open PRs

Searched before building. Five open PRs touch tools/vision_tools.py + tools/browser_tool.py together, and none of them touches the except handlers this PR changes:

PR	What it is	Overlap
#63850	cap browser_vision native embeds	embed sizing on the success path
#62820	pre-flight shrink oversized images	resize before the first call
#65390	split per-tool vision/video gates	which tools advertise which modality
#24875	respect auxiliary vision routing	routing decision
#83865	session-scoped media provenance	provenance, different files
#83150	describe images via aux vision on text-only failover	conversation-loop failover; touches vision_tools.py but not its handler
#72516	fall back to image_url when a provider rejects video_url	a retry on rejection, in the video path

None classifies a provider rejection in a handler, and none touches tools/computer_use/tool.py. #72516 is the closest in spirit — it also reacts to a provider rejection — but it retries a different request shape rather than deciding how to report an unrecoverable one, and it is video-only.

Worth a maintainer's attention: #51551 (route opencode-go / opencode-zen captures to native fast path) is adjacent and this PR makes it safer rather than conflicting with it. #51551 marks the reporter's transport (opencode-go) supports_vision=True so captures take the native path. That is right for the vision-capable models behind that proxy and wrong for mimo-v2.5, which is the one in this issue — per-model overrides still apply, but a user who has not set one would move the same 400 from the aux call onto the main-model turn. With this PR, whichever path is taken, a capability rejection degrades instead of narrating itself. No file conflict: #51551 edits tests/tools/test_computer_use_capture_routing.py, which this PR does not touch.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate — see the Overlap table above
• My PR contains only changes related to this fix/feature (no unrelated commits) — one commit
• I've run the affected slice serially with and without the change (see How to Test). I did not run pytest tests/ -q wholesale: on Windows tests/hermes_cli/ cannot be collected (test_doctor_journal_modes.py calls os.geteuid), so the full-suite result would be meaningless from here. CI runs it.
• I've added tests for my changes — 20, with the mutation proof above
• I've tested on my platform: Windows 11

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — docstrings only; _route_capture_through_aux_vision's return contract was stale and is corrected
• N/A — no config keys added or changed
• N/A — no architecture or workflow change
• I've considered cross-platform impact — no platform-specific code; the change is pure control flow and string classification, and the tests use tmp_path with no shell-outs or path assumptions
• N/A — no tool description or schema change. computer_use's response shape on this path was already documented by the existing vision_unavailable branch; this PR only makes that branch reachable.

Screenshots / Logs

Before (computer_use on a text-only model — what the main model was handed):

{
  "mode": "som",
  "vision_analysis": "mimo-v2.5 does not support vision or our request was not accepted by the server. Error: Error code: 400 - {'error': {'message': 'This model does not support image inputs', ...}}",
  "vision_analysis_routed_via": "auxiliary.vision"
}

After:

{
  "mode": "som",
  "elements": [{"index": 0, "role": "AXButton", "label": "Sign in", ...}],
  "summary": "... (vision unavailable: the auxiliary vision model could not be reached; screenshot omitted. Element-index actions still work — drive via the element list above.)",
  "vision_unavailable": true
}

And the log line the issue was filed on, vision_analyze with the reporter's provider string:

before:  ERROR   tools.vision_tools: Error analyzing image: Error code: 400 - {...}
         Traceback (most recent call last): ...   <- read as a crash
after:   WARNING tools.vision_tools: Error analyzing image: Error code: 400 - {...}

[jackulau]

Disclosure on a self-overlap, since it is the kind of thing that is annoying to discover at merge time: my own open #87867 (feat(computer_use): let capture ask the aux vision model a question) also edits _route_capture_through_aux_vision. It changes the prompt construction and the signature; this PR changes the envelope-parsing block and the Returns: docstring. Different hunks — I test-merged the two branches locally and they combine cleanly (git merge-tree --write-tree returns a single tree, no conflict), in either merge order.

They are also independent on the merits: #87867 is a feature on the success path, this is a correctness fix on the failure path. Neither needs the other, and landing this one first makes #87867 strictly better — a directed question that the aux model cannot answer because it has no vision would otherwise be returned to the main model as if it had been answered.