Skip to content

feat(image_gen): enable image-to-image on the Codex backend - #51838

Closed
iainlane wants to merge 1 commit into
NousResearch:mainfrom
iainlane:feat/codex-image-to-image
Closed

feat(image_gen): enable image-to-image on the Codex backend#51838
iainlane wants to merge 1 commit into
NousResearch:mainfrom
iainlane:feat/codex-image-to-image

Conversation

@iainlane

Copy link
Copy Markdown

What

Enable image-to-image / editing on the ChatGPT/Codex OAuth image-gen backend (openai-codex). Until now it advertised text-to-image only and hard-rejected any source image with a "not capable of image-to-image" error — even though the ChatGPT app itself edits uploaded images.

Why it was rejected, and why that was wrong

The limitation was an implementation gap, not an API one. The backend drives gpt-image-2 through the Responses image_generation tool, and that tool conditions on input_image content parts. The codebase already sends images to /backend-api/codex/responses this way for the main agent's vision (agent/codex_responses_adapter.py). The plugin simply never attached the source image.

What changed

  • Attach each source image as an input_image content part alongside the prompt text in the Responses payload.
  • A source may be a public URL (passed through), a data: URI, or a local file path — what a user-attached image looks like once the gateway has saved it to disk. Local files and data: URIs are validated as real images by magic-byte sniffing (not the extension or data: label), and local files additionally pass the read denylist so a credential file can't be shipped to the image API just because its bytes look like an image.
  • capabilities() now reports ["text", "image"], so the dynamic tool schema offers image_url when this backend is active.

This commit deliberately uses a small in-plugin resolver built on the get_read_block_error + _sniff_mime_from_bytes primitives already on main, so it has no new dependencies and merges independently.

Verification

  • Unit tests cover payload construction (local inlined as input_image, remote passthrough, text-only has no input_image), the denylist, and non-image rejection.
  • Smoke-tested live against a real ChatGPT/Codex subscription: text-to-image and image-to-image both produce images, and the edit clearly conditions on the source (same subject, restyled per the prompt).

@alt-glitch alt-glitch added type/feature New feature or request comp/plugins Plugin system and bundled plugins tool/vision Vision analysis and image generation provider/openai OpenAI / Codex Responses API P3 Low — cosmetic, nice to have labels Jun 24, 2026
The ChatGPT/Codex OAuth image-gen backend advertised text-to-image only
and rejected any source image with a "not capable of image-to-image"
error, even though the ChatGPT app itself edits uploaded images happily.

The limitation was an implementation gap, not an API one. The backend
drives gpt-image-2 through the Responses image_generation tool, and that
tool conditions on input_image content parts — the same mechanism the
rest of the codebase already uses to send images to
/backend-api/codex/responses (see agent/codex_responses_adapter.py). The
plugin simply never attached the source image to the request.

Attach each source as an input_image part alongside the prompt text. A
source may be a public URL (passed through), a data: URI, or a local
file path — what a user-attached image looks like once the gateway has
saved it to disk and surfaced it to the model. Local files and data:
URIs are validated as real images by magic-byte sniffing rather than
trusting the extension or the data: label, and local files additionally
pass the read denylist so a credential store can't be shipped to the
image API just because its bytes happen to look like an image.

capabilities() now reports image support, so the dynamic tool schema
offers image_url when this backend is active.
@iainlane
iainlane force-pushed the feat/codex-image-to-image branch from baa83e5 to 095aff2 Compare June 24, 2026 11:18
@iainlane
iainlane marked this pull request as ready for review June 24, 2026 11:49
@kshitijk4poor

Copy link
Copy Markdown
Collaborator

Closing as a duplicate — Codex OAuth image-to-image / reference-image input is now on main via #57017 (merge commit 019950560).

That PR wires source/reference images through the Codex Responses input_image path on the unified image_generate surface, with magic-byte validation, a 25MB cap, the agent.file_safety read guard, and a 16-image clamp. Your PR tackled the same feature — thank you for the contribution and for helping surface how much demand there was for this. Credit to everyone in the cluster who pushed on it.

If you spot a gap in the merged implementation, please open a follow-up. 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have provider/openai OpenAI / Codex Responses API tool/vision Vision analysis and image generation type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants