feat(vision-bridge): add automatic image description fallback for non-vision models by herjarsa · Pull Request #1476 · cheaper-inference/OmniRoute

herjarsa · 2026-04-21T12:18:36Z

Summary

Implements VisionBridgeGuardrail (priority 5) that intercepts image-bearing requests to non-vision models
Extracts descriptions via a configurable vision model (default: gpt-4o-mini), replaces images with text, and forwards the modified request
Fails open on any error — never blocks valid requests

Details

When a user sends an image to a text-only model (e.g., MiniMax, GLM, DeepSeek), OmniRoute currently either returns an error or silently drops the image. Vision Bridge solves this by:

Detecting image content parts (image_url and base64 image formats) in requests to non-vision models
Extracting descriptions by calling a configurable vision model via OpenAI-compatible API
Replacing image blocks with labeled text descriptions (e.g., [Image 1]: A sunset over the ocean)
Forwarding the modified text-only request to the target model

Files Added (9 files, 1655 lines)

File	Purpose
`src/lib/guardrails/visionBridge.ts`	VisionBridgeGuardrail class (priority 5, DI-based)
`src/lib/guardrails/visionBridgeHelpers.ts`	extractImageParts, callVisionModel, replaceImageParts, resolveImageAsDataUri
`src/shared/constants/visionBridgeDefaults.ts`	Default config + types + merge function
`tests/unit/guardrails/visionBridge.test.ts`	16 guardrail scenario tests
`tests/unit/guardrails/visionBridgeHelpers.*.test.ts`	35 helper tests (4 files)

Files Modified

File	Change
`src/lib/guardrails/registry.ts`	Registered VisionBridgeGuardrail at priority 5

Architecture Decisions

AD1: Priority 5 (runs BEFORE PIIMasker) — needs original image data before PII stripping
AD2: Direct fetch() call — simpler than full executor pipeline for a single vision call
AD3: Fail-open — Vision Bridge is a convenience feature, never blocks requests
AD4: OpenAI-compatible format — portable across vision providers
AD5: Settings-based config — consistent with OmniRoute's existing settings pattern

Configuration

Setting	Default	Description
`visionBridgeEnabled`	`true`	Master toggle
`visionBridgeModel`	`openai/gpt-4o-mini`	Vision model for description extraction
`visionBridgePrompt`	`"Describe this image concisely..."`	Prompt for vision model
`visionBridgeTimeout`	`30000`	Timeout in ms

Disable via header: X-OmniRoute-Disabled-Guardrails: vision-bridge

Test Results

51/51 tests passing (16 guardrail + 35 helpers), 0 TypeScript errors in source files

Closes #1424

…-vision models Implements VisionBridgeGuardrail (priority 5) that intercepts image-bearing requests to non-vision models, extracts descriptions via a configurable vision model (default: gpt-4o-mini), and replaces images with text before forwarding. Fails open on any error. - Add VisionBridgeGuardrail class extending BaseGuardrail - Add visionBridgeHelpers: extractImageParts, callVisionModel, replaceImageParts, resolveImageAsDataUri - Add visionBridgeDefaults with configurable settings - Register VisionBridgeGuardrail in GuardrailRegistry at priority 5 - Add 51 unit tests covering all spec scenarios (VB-S01 through VB-S10) - Dependency injection for getSettings and callVisionModel (testable without SQLite) Closes cheaper-inference#1424

gemini-code-assist

Code Review

This pull request introduces the VisionBridgeGuardrail, which enables text-only models to process image inputs by extracting descriptions via a secondary vision model. The implementation includes comprehensive helper functions for image extraction and replacement, configuration defaults, and a robust suite of unit tests. Additionally, the PR updates error handling for token limits and refines combo target resolution. Feedback highlights critical performance concerns regarding sequential API calls and inefficient database lookups in the chat handler. Technical issues were also identified in the Anthropic API implementation, alongside a project structure violation regarding the placement of the PR documentation file.

gemini-code-assist · 2026-04-21T12:20:10Z

+    const descriptions: string[] = [];
+    const startTime = Date.now();
+
+    for (let i = 0; i < limitedParts.length; i++) {
+      const imagePart = limitedParts[i];
+      try {
+        const description = await callVision(imagePart.imageUrl, config);
+        descriptions.push(`[Image ${i + 1}]: ${description}`);
+      } catch (error) {
+        const message = error instanceof Error ? error.message : String(error);
+        logger?.warn?.(
+          "VISION-BRIDGE",
+          `Failed to get description for image ${i + 1}: ${message}`
+        );
+        descriptions.push(`[Image ${i + 1}]: (unavailable)`);
+      }
+    }


The image descriptions are currently extracted sequentially in a loop. If a request contains multiple images (up to the default limit of 10), this will significantly increase latency as each vision model call must wait for the previous one to complete. Processing these calls in parallel using Promise.all would greatly improve performance.

gemini-code-assist · 2026-04-21T12:20:11Z

+    const baseUrl = config.model.startsWith("anthropic/")
+      ? process.env.ANTHROPIC_API_URL || "https://api.anthropic.com"
+      : process.env.OPENAI_API_URL || "https://api.openai.com/v1";
+
+    const response = await fetch(`${baseUrl}/chat/completions`, {


The callVisionModel function attempts to support Anthropic models by changing the base URL, but it still appends /chat/completions to the endpoint and uses the OpenAI request body format. Anthropic's native API uses the /v1/messages endpoint and a different payload structure. Additionally, Anthropic requires the x-api-key header instead of Authorization: Bearer. This implementation will fail when calling Anthropic models directly. If the intention is to support only OpenAI-compatible providers, the Anthropic-specific logic should be removed or clarified.

gemini-code-assist · 2026-04-21T12:20:11Z

+function deepClone<T>(obj: T): T {
+  if (obj === null || typeof obj !== "object") {
+    return obj;
+  }
+
+  if (Array.isArray(obj)) {
+    return obj.map((item) => deepClone(item)) as T;
+  }
+
+  const cloned = {} as Record<string, unknown>;
+  for (const [key, value] of Object.entries(obj)) {
+    cloned[key] = deepClone(value);
+  }
+  return cloned as T;
+}


The custom deepClone implementation can be replaced with the native structuredClone function, which is more robust and efficient for cloning plain objects and arrays in modern Node.js environments.

gemini-code-assist · 2026-04-21T12:20:11Z

+          const { getCombos } = await import("../../src/lib/localDb");
+          const allCombosData = await getCombos();


Performing a dynamic import and fetching all combos from the database (getCombos()) inside the handleChatCore function on every request is inefficient. This function is in the hot path for chat requests. Consider using a cached version of the combos data or optimizing the lookup to fetch only the specific combo needed.

gemini-code-assist · 2026-04-21T12:20:11Z

+## Summary
+
+- Implements `VisionBridgeGuardrail` (priority 5) that intercepts image-bearing requests to non-vision models
+- Extracts descriptions via a configurable vision model (default: `gpt-4o-mini`), replaces images with text, and forwards the modified request
+- Fails open on any error — never blocks valid requests
+
+## Details
+
+When a user sends an image to a text-only model (e.g., MiniMax, GLM, DeepSeek), OmniRoute currently either returns an error or silently drops the image. Vision Bridge solves this by:
+
+1. **Detecting** image content parts (`image_url` and base64 `image` formats) in requests to non-vision models
+2. **Extracting** descriptions by calling a configurable vision model via OpenAI-compatible API
+3. **Replacing** image blocks with labeled text descriptions (e.g., `[Image 1]: A sunset over the ocean`)
+4. **Forwarding** the modified text-only request to the target model
+
+### Files Added (9 files, 1655 lines)
+
+| File | Purpose |
+|------|---------|
+| `src/lib/guardrails/visionBridge.ts` | VisionBridgeGuardrail class (priority 5, DI-based) |
+| `src/lib/guardrails/visionBridgeHelpers.ts` | extractImageParts, callVisionModel, replaceImageParts, resolveImageAsDataUri |
+| `src/shared/constants/visionBridgeDefaults.ts` | Default config + types + merge function |
+| `tests/unit/guardrails/visionBridge.test.ts` | 16 guardrail scenario tests |
+| `tests/unit/guardrails/visionBridgeHelpers.*.test.ts` | 35 helper tests (4 files) |
+
+### Files Modified
+
+| File | Change |
+|------|--------|
+| `src/lib/guardrails/registry.ts` | Registered VisionBridgeGuardrail at priority 5 |
+
+### Architecture Decisions
+
+- **AD1**: Priority 5 (runs BEFORE PIIMasker) — needs original image data before PII stripping
+- **AD2**: Direct `fetch()` call — simpler than full executor pipeline for a single vision call
+- **AD3**: Fail-open — Vision Bridge is a convenience feature, never blocks requests
+- **AD4**: OpenAI-compatible format — portable across vision providers
+- **AD5**: Settings-based config — consistent with OmniRoute's existing settings pattern
+
+### Configuration
+
+| Setting | Default | Description |
+|---------|---------|-------------|
+| `visionBridgeEnabled` | `true` | Master toggle |
+| `visionBridgeModel` | `openai/gpt-4o-mini` | Vision model for description extraction |
+| `visionBridgePrompt` | `"Describe this image concisely..."` | Prompt for vision model |
+| `visionBridgeTimeout` | `30000` | Timeout in ms |
+
+Disable via header: `X-OmniRoute-Disabled-Guardrails: vision-bridge`
+
+### Test Results
+
+**51/51 tests passing** (16 guardrail + 35 helpers), 0 TypeScript errors in source files
+
+Closes #1424


The file .pr-body.md is located in the project root, which violates the repository's cleanliness rules. According to the style guide, the project root must only contain specific configuration and documentation files (README.md, CHANGELOG.md, AGENTS.md).

References

The Project Root MUST ONLY CONTAIN: Configuration files, Dependency files, Documentation files (README.md, CHANGELOG.md, AGENTS.md), CI/CD files and ignore definitions. ^(link)

…der keys, structuredClone

herjarsa · 2026-04-21T13:15:21Z

Review Response — All Issues Fixed ✅

Commit \de582529\ addresses all review feedback. Here's the resolution for each comment:

1. Sequential image processing → Parallel ✅ Fixed

Replaced the \or\ loop with \Promise.allSettled()\ in \�isionBridge.ts. Each image is processed independently — a failing image produces [Image N]: (unavailable)\ without blocking the others.

2. Anthropic API broken → Provider branching ✅ Fixed

Added provider-aware branching in \callVisionModel():

Anthropic: /v1/messages\ endpoint, \x-api-key\ + \�nthropic-version: 2023-06-01\ headers, different body format { model, messages, max_tokens }, response parsed from \content[0].text\
OpenAI-compatible (default): existing /chat/completions\ format unchanged

3. API key hardcoded to OpenAI → Provider-specific resolution ✅ Fixed

Added
esolveProviderApiKey()\ helper with a provider→env var map:

\�nthropic/\ → \ANTHROPIC_API_KEY\
\google/\ → \GOOGLE_API_KEY\
default → \OPENAI_API_KEY
Explicit \�piKey\ parameter takes precedence over env var lookup.

4. Custom deepClone → structuredClone ✅ Fixed

Replaced custom \deepClone()\ with native \structuredClone(). Project requires Node ≥20.20.2 where it\s available. Removed the \deepClone\ function entirely.

Unrelated comments:

\chatCore.ts\ combo change: Not part of this PR — force-pushed to remove it from the branch.
.pr-body.md\ in root: Removed from the branch.

…-vision models (cheaper-inference#1476) * feat(vision-bridge): add automatic image description fallback for non-vision models Implements VisionBridgeGuardrail (priority 5) that intercepts image-bearing requests to non-vision models, extracts descriptions via a configurable vision model (default: gpt-4o-mini), and replaces images with text before forwarding. Fails open on any error. - Add VisionBridgeGuardrail class extending BaseGuardrail - Add visionBridgeHelpers: extractImageParts, callVisionModel, replaceImageParts, resolveImageAsDataUri - Add visionBridgeDefaults with configurable settings - Register VisionBridgeGuardrail in GuardrailRegistry at priority 5 - Add 51 unit tests covering all spec scenarios (VB-S01 through VB-S10) - Dependency injection for getSettings and callVisionModel (testable without SQLite) Closes cheaper-inference#1424 * fix(vision-bridge): resolve Anthropic API, parallel processing, provider keys, structuredClone

herjarsa requested a review from diegosouzapw as a code owner April 21, 2026 12:18

gemini-code-assist Bot reviewed Apr 21, 2026

View reviewed changes

diegosouzapw changed the base branch from main to release/v3.7.0 April 21, 2026 12:53

fix(vision-bridge): resolve Anthropic API, parallel processing, provi…

de58252

…der keys, structuredClone

herjarsa force-pushed the feat/vision-bridge-guardrail branch from c515e34 to de58252 Compare April 21, 2026 13:13

herjarsa mentioned this pull request Apr 21, 2026

[Feature] Vision Bridge: Automatic Vision Fallback for Non-Vision Models #1424

Closed

diegosouzapw merged commit b709dca into cheaper-inference:release/v3.7.0 Apr 21, 2026
2 checks passed

diegosouzapw mentioned this pull request Apr 22, 2026

Release v3.7.0 #1439

Merged

Tushar49 mentioned this pull request May 27, 2026

Gemini/Antigravity vision: images silently dropped in geminiHelper.convertOpenAIContentToParts (missing rec.image key + no remote URL fetch) #2807

Closed

herjarsa deleted the feat/vision-bridge-guardrail branch June 4, 2026 13:04

Navigation Menu

Repository navigation

feat(vision-bridge): add automatic image description fallback for non-vision models - #1476

feat(vision-bridge): add automatic image description fallback for non-vision models#1476
diegosouzapw merged 2 commits into
cheaper-inference:release/v3.7.0from
herjarsa:feat/vision-bridge-guardrail

herjarsa commented Apr 21, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Apr 21, 2026

Uh oh!

gemini-code-assist Bot Apr 21, 2026

Uh oh!

gemini-code-assist Bot Apr 21, 2026

Uh oh!

gemini-code-assist Bot Apr 21, 2026

Uh oh!

gemini-code-assist Bot Apr 21, 2026

Uh oh!

herjarsa commented Apr 21, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		const { getCombos } = await import("../../src/lib/localDb");
		const allCombosData = await getCombos();

Conversation

herjarsa commented Apr 21, 2026

Summary

Details

Files Added (9 files, 1655 lines)

Files Modified

Architecture Decisions

Configuration

Test Results

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Apr 21, 2026

Choose a reason for hiding this comment

Uh oh!

herjarsa commented Apr 21, 2026

Review Response — All Issues Fixed ✅

1. Sequential image processing → Parallel ✅ Fixed

2. Anthropic API broken → Provider branching ✅ Fixed

3. API key hardcoded to OpenAI → Provider-specific resolution ✅ Fixed

4. Custom deepClone → structuredClone ✅ Fixed

Unrelated comments:

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants