fix(vision): pre-flight shrink oversized images before first API call

[calamarc]

Problem

Anthropic's Messages API rejects images >5,242,880 bytes (5 MB base64) with a hard HTTP 400:

HTTP 400: messages.0.content.1.image.source.base64: image exceeds 5 MB maximum: 6023876 bytes > 5242880 bytes

Observed in production 3x in one session — real 6.0 MB, 6.3 MB, and 9.9 MB images all burned all 3 retries and never recovered.

Root Cause

The existing shrink recovery in tools/vision_tools.py was reactive-only (fires after a 400) AND used a target of 5 * 1024 * 1024 — exactly Anthropic's ceiling with zero headroom for base64 padding + provider-side rounding. So even when reactive shrink triggered, the resized payload could still trip the wire.

Fix

Two changes, both in tools/vision_tools.py:

1. Tighten _RESIZE_TARGET_BYTES from 5 * 1024 * 1024 to 4_500_000 — real headroom under Anthropic's per-image ceiling.

2. Add proactive pre-flight in vision_analyze_tool: if base64 payload exceeds target, resize BEFORE the first async_call_llm call. Prevents the 3-retry HTTP churn on unrecoverable size errors.

The reactive path (_try_shrink_image_parts_in_messages in run_agent.py) is unchanged — it already uses a 4 MB target and benefits transitively.

Tests

• Updated test_constants_sane to lock in the 4.5 MB target and assert it stays strictly under Anthropic's 5 MB cap.
• Added TestPreFlightShrink — generates a real 6+ MB JPEG and asserts the shrink loop converges under target.

$ pytest tests/tools/test_vision_tools.py tests/run_agent/test_image_shrink_recovery.py tests/run_agent/test_image_rejection_fallback.py
92 passed, 7 skipped in 2.67s

[teknium1]

Thanks for targeting a real failure mode. Current main still sends a 5–20 MB auxiliary-vision image before recovery (tools/vision_tools.py:1173-1189, 1251-1267), so the underlying issue remains.

Problems

• The new regression test only calls _resize_image_for_vision (tests/tools/test_vision_tools.py:938-972). It never calls vision_analyze_tool or inspects its first async_call_llm payload, so it does not prove pre-flight behavior.
• Current main routes encode/resize work through _run_encode_on_cpu_executor (tools/vision_tools.py:176-189, 1178-1188); the proposed direct synchronous resize (tools/vision_tools.py:756) needs to use that path when salvaged.
• browser_vision retains the same full-size-first, exception-only recovery pattern (tools/browser_tool.py:4060-4160).

Suggested changes

• Add an async end-to-end unit test that verifies the first auxiliary request carries a resized image.
• Preserve the bounded executor and apply the policy consistently to the browser auxiliary path.
• Make the lower cap provider-aware so the existing full-resolution behavior remains available for providers that accept larger images.

Automated hermes-sweeper review.

[teknium1]

Current main now routes image encodes and resizes through _run_encode_on_cpu_executor (tools/vision_tools.py:176-189) so CPU-heavy image work cannot block the async event loop. Please preserve that bounded-executor path when salvaging this pre-flight resize.

[teknium1]

This only verifies _resize_image_for_vision; it never invokes vision_analyze_tool or asserts the first async_call_llm payload. Add an async regression test that proves the initial API call receives the already-shrunk data URL.

[calamarc]

Thanks for the review — you were right that main had already moved past what I was patching. Rebased onto latest main and reworked the fix to align with the existing architecture.

What changed

Rebase + rework. The initial 5→4.5 MB constant tighten is gone; main already has _EMBED_TARGET_BYTES = 4 MB and _EMBED_MAX_DIMENSION = 7900px for the native fast-path proactive resize. The real gap was in the auxiliary-LLM paths (vision_analyze_tool and browser_vision), which still sent full-resolution images to the first async_call_llm / call_llm and only reacted after HTTP 400.

Addressing each point:

1. Executor bypass ✓ — Pre-flight in vision_analyze_tool now routes through _run_encode_on_cpu_executor for both the over-bytes check (_image_exceeds_dimension) and the resize itself. Mirrors the native fast-path pattern at vision_tools.py:1018-1037.

2. Test doesn't prove pre-flight behavior ✓ — Replaced the isolated _resize_image_for_vision test with TestAuxVisionPreFlightShrink, two async e2e cases:

   • test_first_api_call_payload_is_pre_shrunk — generates a real 6+ MB JPEG, mocks async_call_llm, asserts the first await_args.kwargs['messages'][0]['content'][1]['image_url']['url'] is ≤ 4 MB. Load-bearing: proves the pre-flight fires end-to-end.
   • test_small_image_untouched_by_preflight — small image is passed through unchanged so aux-vision quality doesn't degrade unnecessarily.

3. browser_vision same bug ✓ — Same pre-flight added to tools/browser_tool.py just before the first call_llm on the non-native path (native fast-path is untouched — it already goes through the _should_use_native_vision_fast_path embed check).

4. Provider-aware cap — Not implemented. My reasoning: the native fast-path already uses the same 4 MB / 7900 px thresholds uniformly across providers, so adding a per-provider knob in the aux paths would create inconsistency between the two paths without concrete evidence any current aux-vision provider suffers from the 4 MB cap. Aux vision routes predominantly to Anthropic Claude Haiku today. Happy to add per-provider logic if there's evidence a specific provider needs more headroom — figured it's cleaner to keep the two paths symmetric until then.

Test results

$ pytest tests/tools/test_vision_tools.py tests/run_agent/test_image_shrink_recovery.py tests/run_agent/test_image_rejection_fallback.py tests/tools/test_browser_lightpanda.py tests/tools/test_browser_hybrid_routing.py -x -q
194 passed in 4.18s

Force-pushed to the same branch.