Skip to content

fix(sse): feed compression pipeline the authoritative vision capability (#7237) - #7560

Merged
diegosouzapw merged 1 commit into
release/v3.8.49from
fix/7237-vision-compression
Jul 17, 2026
Merged

diegosouzapw merged 1 commit into
release/v3.8.49from
fix/7237-vision-compression

Conversation

@diegosouzapw

Copy link
Copy Markdown
Collaborator

Closes #7237

Root cause

open-sse/handlers/chatCore.ts:1330 fed the supportsVision flag into
applyCompressionAsync from isVisionModelId(effectiveModel) — the
deliberately-conservative model-id fragment heuristic in
src/shared/constants/visionModels.ts (its own docstring says to keep it
conservative: newly-released vision models get zero-touch support via the
models.dev sync, not this fallback) — instead of the authoritative
getResolvedModelCapabilities().supportsVision that every other
vision-aware path (notably the vision-bridge guardrail) uses.

gpt-5.5 is registered with supportsVision: true in
src/shared/constants/modelSpecs.ts, but the fragment list has no
gpt-5.x entry, so the heuristic wrongly returned false. That spurious
false reached open-sse/services/compression/lite.ts::replaceImageUrls(),
whose gate is supportsVision !== false — it only strips images when told
explicitly false — so every image_url block was silently replaced with
a [image: png] text placeholder before the request ever reached
AgentRouter/Freemodel.dev (or any other openai-compat passthrough
provider).

This also explains the three symptoms in the report: it reproduces even
though the model is marked vision: true; disabling Vision Bridge changes
nothing (separate code path — lite compression runs earlier); and
"updating model capabilities" in the UI never helped because
modelCapabilityOverrides.ts only supports a max_token override, there
is no vision override to set.

Fix

Swap the heuristic for the authoritative capability at the single call
site:

supportsVision: getResolvedModelCapabilities({ provider, model: effectiveModel })
  .supportsVision,

getResolvedModelCapabilities().supportsVision is boolean | null and
falls back to the same conservative heuristic internally only when no
spec/registry/sync data exists — so genuinely unknown models resolve to
null, which lite.ts's !== false gate already treats as
preserve-by-default. This avoids re-introducing the #4071/#4012 class of
bug (blinding a model that can actually see).

Regression test

tests/unit/vision-compression-authoritative-capability-7237.test.ts —
4 assertions:

  1. documents the drift: isVisionModelId('gpt-5.5') === false while the
    authoritative spec says supportsVision: true
  2. replaceImageUrls preserves the image when fed the authoritative
    capability (the fixed behavior)
  3. sanity-checks the pre-fix bug shape (feeding the raw heuristic value
    strips the image)
  4. end-to-end through applyCompressionAsync(..., 'lite', ...) with
    model: 'gpt-5.5' — the image_url block and its original data URL
    survive intact

RED→GREEN: reproduced the pre-fix behavior standalone (feeding
isVisionModelId('gpt-5.5') — false — into applyCompressionAsync)
and confirmed the placeholder strip:

[{ "type": "text", "text": "[image: png]" }]

Post-fix, the same flow (feeding getResolvedModelCapabilities(...).supportsVision)
preserves the image_url block unchanged — verified both via the
standalone repro and the 4/4 passing regression test.

Gates run (all green)

  • node scripts/check/check-file-size.mjs — OK (no frozen file grew)
  • node scripts/check/check-complexity.mjs — OK (2054 ≤ baseline 2056)
  • node scripts/check/check-cognitive-complexity.mjs — OK (889 ≤ baseline 890)
  • npm run typecheck:core — clean (chatCore.ts is outside the typecheck:core
    allowlist, so also sanity-compiled it against the full tsconfig.json;
    the pre-existing strict-mode errors it reports are all in unrelated
    regions of the file, none touching the changed import or the compression
    call site)
  • npx eslint --suppressions-location config/quality/eslint-suppressions.json open-sse/handlers/chatCore.ts tests/unit/vision-compression-authoritative-capability-7237.test.ts — clean
  • Full related test sweep, 106/106 pass: tests/unit/compression/lite.test.ts,
    tests/unit/vision-detection-consistency.test.ts,
    tests/unit/guardrails/visionBridge.test.ts,
    tests/unit/combo-vision-aware-routing.test.ts,
    tests/unit/model-capabilities-mimo-vision-override.test.ts,
    tests/unit/model-capabilities-mistral-vision-sync-4073.test.ts,
    tests/unit/llm-selector-custom-vision-models.test.ts,
    tests/unit/command-code-vision.test.ts,
    tests/unit/vision-compression-authoritative-capability-7237.test.ts
  • chatCore compression-integration sweep, 85/85 pass:
    tests/unit/chatcore-compression-usage-receipt.test.ts,
    tests/unit/chatcore-translation-paths.test.ts,
    tests/unit/chatcore-compression-settings.test.ts,
    tests/unit/chatcore-compression-combo-predicates.test.ts,
    tests/unit/chatcore-compression-integration.test.ts,
    tests/unit/chatcore-compression-analytics-write.test.ts,
    tests/unit/chatcore-caveman-output-analytics.test.ts,
    tests/unit/chatcore-compression-cache-stats.test.ts

Not confirmed by this mechanism

Per the analysis plan-file, claude-opus-4-6/4-7/4-8 ids match the
existing claude-opus-4 fragment, so this specific mechanism should not
affect them. If the reporter's Claude cases are also broken, that would
be a second, separate cause worth tracking in a follow-up.

…ty (#7237)

chatCore.ts fed applyCompressionAsync's supportsVision option from
isVisionModelId() — the deliberately-conservative model-id fragment
heuristic in src/shared/constants/visionModels.ts — instead of the
authoritative getResolvedModelCapabilities().supportsVision used by every
other vision-aware path (e.g. the vision-bridge guardrail).

gpt-5.5 is registered with supportsVision:true in modelSpecs.ts, but the
fragment list has no gpt-5.x entry, so the heuristic wrongly returned
false. That false reached lite.ts's replaceImageUrls(), whose gate is
`supportsVision !== false`, silently stripping every image_url block
before the request ever reached the executor.

getResolvedModelCapabilities().supportsVision resolves to null (not
false) for genuinely unknown models, which the same !== false gate
already treats as preserve-by-default — matching the conservative
semantics used elsewhere and avoiding the #4071/#4012 class of bug
(blinding a model that can actually see).
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@diegosouzapw
diegosouzapw merged commit f3d92ae into release/v3.8.49 Jul 17, 2026
20 checks passed
@diegosouzapw
diegosouzapw deleted the fix/7237-vision-compression branch July 17, 2026 08:32
HouMinXi pushed a commit to HouMinXi/OmniRoute that referenced this pull request Aug 2, 2026
…ty (cheaper-inference#7237) (cheaper-inference#7560)

chatCore.ts fed applyCompressionAsync's supportsVision option from
isVisionModelId() — the deliberately-conservative model-id fragment
heuristic in src/shared/constants/visionModels.ts — instead of the
authoritative getResolvedModelCapabilities().supportsVision used by every
other vision-aware path (e.g. the vision-bridge guardrail).

gpt-5.5 is registered with supportsVision:true in modelSpecs.ts, but the
fragment list has no gpt-5.x entry, so the heuristic wrongly returned
false. That false reached lite.ts's replaceImageUrls(), whose gate is
`supportsVision !== false`, silently stripping every image_url block
before the request ever reached the executor.

getResolvedModelCapabilities().supportsVision resolves to null (not
false) for genuinely unknown models, which the same !== false gate
already treats as preserve-by-default — matching the conservative
semantics used elsewhere and avoiding the cheaper-inference#4071/cheaper-inference#4012 class of bug
(blinding a model that can actually see).
muhamadgalihsaputra pushed a commit to niyatna/NiyatnaRoute that referenced this pull request Sep 27, 2026
…ty (cheaper-inference#7237) (cheaper-inference#7560)

chatCore.ts fed applyCompressionAsync's supportsVision option from
isVisionModelId() — the deliberately-conservative model-id fragment
heuristic in src/shared/constants/visionModels.ts — instead of the
authoritative getResolvedModelCapabilities().supportsVision used by every
other vision-aware path (e.g. the vision-bridge guardrail).

gpt-5.5 is registered with supportsVision:true in modelSpecs.ts, but the
fragment list has no gpt-5.x entry, so the heuristic wrongly returned
false. That false reached lite.ts's replaceImageUrls(), whose gate is
`supportsVision !== false`, silently stripping every image_url block
before the request ever reached the executor.

getResolvedModelCapabilities().supportsVision resolves to null (not
false) for genuinely unknown models, which the same !== false gate
already treats as preserve-by-default — matching the conservative
semantics used elsewhere and avoiding the cheaper-inference#4071/cheaper-inference#4012 class of bug
(blinding a model that can actually see).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(providers): vision-capable images stop working when routed through OmniRoute (agentrouter/freemodel.dev)

1 participant