feat(image-gen): support reference images in openai-codex backend - #41159
Closed
airmonitor wants to merge 2 commits into
Closed
feat(image-gen): support reference images in openai-codex backend#41159airmonitor wants to merge 2 commits into
airmonitor wants to merge 2 commits into
Conversation
Add an optional `reference_images` parameter to the `image_generate` tool and honor it in the openai-codex backend (gpt-image-2 via the Codex Responses `image_generation` tool). Each reference is an http(s) URL, a data:image/... URL, or an absolute file path; URLs/data-URLs pass through and absolute paths are read and inlined as base64 data URLs. References are forwarded to the Responses request as `input_image` parts, letting a caller preserve the identity of, e.g., a brand mascot in the generated image. - tools/image_generation_tool.py: add optional `reference_images` to the agent schema; plumb it through `_handle_image_generate` and `_dispatch_to_plugin_provider`. Base `ImageGenProvider.generate(**kwargs)` already absorbs it, so backends without image-input support ignore it. - plugins/image_gen/openai-codex: add `_to_input_image_part` normalization (realpath + size cap + mime), thread `image_parts` through `_build_responses_payload`/`_collect_image_b64`, and validate in `generate` (<=4 images, <=10 MiB each) before any network call. No behavior change when `reference_images` is absent (payload is byte-identical to today). - tests: normalization cases, payload threading, dispatch forwarding, schema exposure, and limit/validation errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
alt-glitch
added
type/feature
Refactor `_extract_image_b64` to return both final and partial image b64 values separately, allowing the caller to always prefer the finished image and fall back to a partial only when no final result arrives. This prevents a half-rendered streaming preview from being saved as the final PNG. - plugins/image_gen/openai-codex: update `_extract_image_b64` to return `(final_b64, partial_b64)` tuple and update callers accordingly - plugins/image_gen/openai-codex: change `partial_images` default from 1 to 3 for better streaming preview coverage
Collaborator
|
Closing as a duplicate — Codex OAuth image-to-image / reference-image input is now on That PR wires source/reference images through the Codex Responses If you spot a gap in the merged implementation, please open a follow-up. 🙏 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
image_generateonly exposedprompt+aspect_ratio, and theopenai-codexbackend (gpt-image-2 via the Codex Responses
image_generationtool) sent only atext part — so callers could not pass a reference image to guide/edit generation
(e.g. preserve a brand mascot's identity).
This adds an optional
reference_imagesparameter and honors it in theopenai-codex backend by forwarding each reference as an
input_imagecontent partin the Responses request. gpt-image-2 processes image inputs at high fidelity, so
a single reference is enough to lock identity.
Changes
tools/image_generation_tool.py: optionalreference_images(array of strings)in the agent schema; plumbed through
_handle_image_generateand_dispatch_to_plugin_provider. BaseImageGenProvider.generate(**kwargs)already absorbs it, so backends without image input ignore it.
plugins/image_gen/openai-codex:_to_input_image_partnormalizes eachreference (http(s)/
data:image/...URLs pass through; absolute file paths arerealpath-resolved, size-capped, inlined as base64 data URLs), threaded through
_build_responses_payload/_collect_image_b64, and validated ingenerate(<=4 images, <=10 MiB each) before any network call. No behavior change when
reference_imagesis absent (payload byte-identical).Scope
openai-codex only. Other backends (openai/fal/xai/krea) ignore the kwarg for now.
How to test
Unit:
Manual (Codex OAuth signed in,
image_gen.provider: openai-codex): ask the agentto
image_generatewithreference_images=["https://.../mascot.png"]. Verifiedagainst the live Codex backend — the reference image overrode the text prompt and
drove the output character identity.
Platforms
Tested on macOS (Python 3.11). File-path inputs use
realpath+ binary reads.Related issues
None.
🤖 Generated with Claude Code