[payne0420]

Summary

Adds image / vision input to OmniRoute's Cursor provider (the agent.v1.AgentService/Run endpoint), plus the supporting prompt-engineering and resilience work developed alongside it. Multimodal OpenAI requests with image_url parts (base64 or remote URL) now reach a vision-capable Cursor model instead of being silently dropped.

Vision input

• Encoder — image_url parts are decoded and inlined as SelectedContext.selected_images[] on the user turn. Field numbers were pinned from the cursor-agent agent.v1 protobuf descriptor (SelectedImage.data oneof = inline bytes, plus uuid, optional Dimension, mime_type), cross-checked against the composer-api encoder shape. The no-image path is byte-identical to before (test-asserted).
• Security (resolveCursorImages) — remote fetches go through the repo's canonical parseAndValidatePublicUrl SSRF guard (always public-only for client-supplied URLs); each image is capped at ≤1 MiB (pre-decode + streaming); image/* content-type enforced; max 12 images per request; all failures return sanitized CursorImageErrors (no stack/path leakage).
• Translator/executor — openai-to-cursor now preserves image_url parts (was dropping them); the executor's buildRequest resolves images and attaches them to the user turn.

Supporting Cursor enhancements (included)

• Tool-commit directive (raises composer-2.5 tool-call rate ~53% → ~88%), tool_choice none/required/specific handling, and output constraints (response_format / max_tokens / stop surfaced as prompt instructions).
• cursorSessionManager: clear pending tool-call mappings on session close.
• cursorVersionDetector: export FALLBACK_VERSION as a single source of truth.

Verification

Validated live against the real endpoint: solid red/green/blue PNGs are correctly identified by gpt-5.2, claude-opus-4-8-low, and composer-2.5 (composer is vision-capable); remote-URL fetch and multi-image messages confirmed.

• Unit — image encoder field layout, byte-identical no-image path, resolver rejections (SSRF / oversize / bad-base64 / too-many), sanitized error body.
• Translator — image_url preservation (object + shorthand forms).
• E2E (gated on CURSOR_E2E_TOKEN) — base64-image + remote-URL image tests.

Notes

• New env vars documented in .env.example + docs/reference/ENVIRONMENT.md: CURSOR_IMAGE_FETCH_TIMEOUT_MS, CURSOR_TOOL_DIRECTIVE.
• Targets release/v3.8.9.

[gemini-code-assist]

Code Review

This pull request introduces vision support for Cursor executors by implementing image extraction, resolution, and security checks, alongside integrating tool-commit directives and output constraints. The code reviewer identified several critical security, performance, and robustness issues, including an SSRF vulnerability due to default redirect-following in fetch, a potential DoS/high CPU usage risk from regex operations on large data URLs, an OOM risk when handling Node.js streams, case-sensitivity bugs with the data: scheme check, and potential NaN timeout values if environment variables are malformed.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

The fetch call follows HTTP redirects by default. If the initial URL is validated by parseAndValidatePublicUrl, a malicious server can redirect the request to an internal/private IP address (e.g., http://127.0.0.1/admin), bypassing the SSRF guard. To prevent this, redirects should be handled manually with redirect: 'manual' and each redirect URL must be validated before fetching.

async function fetchImageBytes(url: string): Promise<{ data: Buffer; mimeType: string }> {
  let currentUrl = url;
  let redirects = 0;
  const maxRedirects = 3;

  while (redirects <= maxRedirects) {
    let parsed: URL;
    try {
      parsed = parseAndValidatePublicUrl(currentUrl);
    } catch (err) {
      if (err instanceof OutboundUrlGuardError) {
        throw new CursorImageError(
          err.code === "OUTBOUND_URL_INVALID"
            ? "Image URL is invalid or uses an unsupported scheme."
            : "Image URL points to a blocked address."
        );
      }
      throw new CursorImageError("Image URL is invalid.");
    }

    const controller = new AbortController();
    const timer = setTimeout(() => controller.abort(), IMAGE_FETCH_TIMEOUT_MS);
    let response: Response;
    try {
      response = await fetch(parsed.toString(), {
        method: "GET",
        signal: controller.signal,
        redirect: "manual",
      });
    } catch {
      clearTimeout(timer);
      throw new CursorImageError("Could not fetch the image URL.");
    }

    try {
      if (response.status >= 300 && response.status < 400) {
        const location = response.headers.get("location");
        if (!location) {
          throw new CursorImageError("Redirect location header is missing.");
        }
        currentUrl = new URL(location, parsed.toString()).toString();
        redirects++;
        continue;
      }

      if (!response.ok) {
        throw new CursorImageError(`Could not fetch the image URL (status ${response.status}).`);
      }

      const contentType = (response.headers.get("content-type") || "").toLowerCase();
      const mimeType = contentType.split(";")[0].trim();
      if (!mimeType.startsWith("image/")) {
        throw new CursorImageError("Image URL did not return an image content type.");
      }

      const declaredLen = Number(response.headers.get("content-length") || "0");
      if (Number.isFinite(declaredLen) && declaredLen > MAX_CURSOR_IMAGE_BYTES) {
        throw new CursorImageError("Image input is too large (max 1 MiB). Resize and retry.");
      }

      const data = await readCapped(response, MAX_CURSOR_IMAGE_BYTES);
      return { data, mimeType };
    } finally {
      clearTimeout(timer);
    }
  }

  throw new CursorImageError("Too many redirects.");
}

[payne0420]

Agreed — real SSRF bypass (the guard only validated the initial URL). Fixed in 5699daa: fetchImageBytes now uses redirect: "manual" and re-validates every hop through parseAndValidatePublicUrl (bounded to 3 redirects). Added tests for redirect→private (blocked), redirect→public (followed), and too-many-redirects. Thanks for catching this.

[gemini-code-assist]

Running payload.replace(/\s/g, '') on an arbitrarily large user-supplied data URL can cause high CPU usage and block the single-threaded Node.js event loop. Since the maximum allowed image size is 1 MiB, we can perform a cheap pre-check on the raw payload.length before executing any regex replacement or decoding.

  if (payload.length > MAX_CURSOR_IMAGE_BYTES * 2) {
    throw new CursorImageError("Image input is too large (max 1 MiB). Resize and retry.");
  }

  const normalized = payload.replace(/\s/g, "");
  // Cheap pre-check: 4 base64 chars -> 3 bytes. Reject obviously oversized
  // payloads before allocating the decode buffer.
  if (Math.floor((normalized.length * 3) / 4) > MAX_CURSOR_IMAGE_BYTES) {
    throw new CursorImageError("Image input is too large (max 1 MiB). Resize and retry.");
  }

[payne0420]

Agreed. Added a raw payload.length > MAX_CURSOR_IMAGE_BYTES * 2 pre-check before the whitespace-strip regex so an oversized data URL can't burn CPU. Fixed in 5699daa.

[gemini-code-assist]

If body is a Node.js stream (e.g., in polyfilled or custom fetch environments), body.getReader will be undefined, causing the code to fall back to response.arrayBuffer(). This will buffer the entire response in memory before checking the size cap, posing an OOM risk for extremely large bodies. We can check if body is an async iterable (which both Node.js Readable streams and Web Streams support in modern Node.js) to consume it safely in chunks.

  const body = response.body;
  if (!body) {
    return Buffer.alloc(0);
  }

  const chunks: Buffer[] = [];
  let total = 0;

  // Handle async iterable (Node.js Readable streams and Web Streams in modern Node)
  if (typeof (body as any)[Symbol.asyncIterator] === "function") {
    for await (const chunk of body as any) {
      const buf = Buffer.from(chunk);
      total += buf.length;
      if (total > cap) {
        throw new CursorImageError("Image input is too large (max 1 MiB). Resize and retry.");
      }
      chunks.push(buf);
    }
    return Buffer.concat(chunks, total);
  }

  if (typeof (body as any).getReader === "function") {
    const reader = (body as any).getReader();
    try {
      for (;;) {
        const { done, value } = await reader.read();
        if (done) break;
        if (value) {
          total += value.byteLength;
          if (total > cap) {
            throw new CursorImageError("Image input is too large (max 1 MiB). Resize and retry.");
          }
          chunks.push(Buffer.from(value));
        }
      }
    } finally {
      try {
        await reader.cancel();
      } catch {}
    }
    return Buffer.concat(chunks, total);
  }

  const buf = Buffer.from(await response.arrayBuffer());
  if (buf.length > cap) {
    throw new CursorImageError("Image input is too large (max 1 MiB). Resize and retry.");
  }
  return buf;

[payne0420]

Agreed. readCapped now consumes the body as an async iterable (Node Readable + Web Streams) or via getReader, capping mid-read; the uncapped arrayBuffer() path is only a last resort for exotic body shapes. Fixed in 5699daa.

[gemini-code-assist]

The scheme check url.startsWith('data:') is case-sensitive. According to RFC 2397, the scheme is case-insensitive, so a data URL starting with DATA: is valid but would fail here. We should perform a case-insensitive check.

Suggested change

	const { data, mimeType } = url.startsWith("data:")
	? decodeDataUrl(url)
	: await fetchImageBytes(url);
	const { data, mimeType } = url.toLowerCase().startsWith("data:")
	? decodeDataUrl(url)
	: await fetchImageBytes(url);

[payne0420]

Agreed — DATA: is valid per RFC 2397. Now matched case-insensitively while passing the original (un-lowercased) URL to the decoder so the base64 payload is preserved. Added an uppercase-DATA: test. Fixed in 5699daa.

[gemini-code-assist]

If process.env.CURSOR_IMAGE_FETCH_TIMEOUT_MS is set to an invalid string, parseInt will return NaN, which can cause unexpected behavior in setTimeout. We should add a defensive fallback check.

Suggested change

	const IMAGE_FETCH_TIMEOUT_MS = parseInt(process.env.CURSOR_IMAGE_FETCH_TIMEOUT_MS || "15000", 10);
	const parsedTimeout = parseInt(process.env.CURSOR_IMAGE_FETCH_TIMEOUT_MS || "15000", 10);
	const IMAGE_FETCH_TIMEOUT_MS = Number.isInteger(parsedTimeout) && parsedTimeout > 0 ? parsedTimeout : 15000;

[payne0420]

Agreed. IMAGE_FETCH_TIMEOUT_MS now falls back to the 15000ms default unless the env value parses to a positive integer. Fixed in 5699daa.

[gemini-code-assist]

If process.env.CURSOR_STREAM_TIMEOUT_MS is set to an invalid string, parseInt will return NaN, which can cause unexpected behavior in setTimeout. We should add a defensive fallback check.

const parsedStreamTimeout = parseInt(process.env.CURSOR_STREAM_TIMEOUT_MS || "300000", 10);
const CURSOR_STREAM_TIMEOUT_MS = Number.isInteger(parsedStreamTimeout) && parsedStreamTimeout > 0 ? parsedStreamTimeout : 300000;

[payne0420]

Agreed. Applied the same positive-integer guard to CURSOR_STREAM_TIMEOUT_MS (falls back to 300000ms). Fixed in 5699daa.

[payne0420]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5699daa0ca

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Resolve image hosts before fetching them

For remote image_url inputs such as https://attacker.example/a.png where DNS resolves or rebinds to 127.0.0.1/169.254.169.254, this path validates only the literal URL hostname via parseAndValidatePublicUrl before handing the hostname to fetch. The guard in src/shared/network/outboundUrlGuard.ts checks url.hostname but does not resolve or pin the destination IP, so attacker-controlled DNS can still make the new client-supplied image fetch hit private or metadata services. Resolve and validate the address for each hop, or use a fetch helper that enforces that before opening the connection.

Useful? React with 👍 / 👎.

[payne0420]

Good catch — agreed. parseAndValidatePublicUrl only checks the hostname string, so a public host that resolves/rebinds to a private or metadata IP would still be fetched. Fixed in f719fd7: each hop now resolves the host via dns.lookup({all:true}) and rejects if any answer is private (isPrivateHost) before connecting; IP literals are skipped (already guarded). Since image_url is fully client-controlled, this is the highest-risk SSRF surface, so the extra hop is worth it. It narrows but doesn't fully close the TOCTOU window vs fetch's own resolution — a connection-time IP filter (e.g. ssrf-req-filter) on the shared outbound guard would close it for every caller, which I've noted as the complete follow-up. Added unit tests for the IP gate + a mocked rebinding case.

[diegosouzapw]

Outstanding work, @payne0420! 🎉 This is a meticulous PR — encoding image_url parts as SelectedContext.selected_images[] with the field numbers pinned from the agent.v1 descriptor, the byte-identical no-image path (asserted in tests), the tool-commit directive bumping composer-2.5's tool-call rate, and the tool_choice/output-constraint handling. The security hardening (SSRF guard via parseAndValidatePublicUrl + DNS-rebinding check, 1 MiB pre-decode cap, image-count cap, protobuf length-overrun guard) and the comprehensive cursor-image-input.test.ts coverage made this easy to trust. Merged into release/v3.8.9 for the next release. Thank you for the depth here!