feat(image-gen): support reference images in image_generate - #21854
feat(image-gen): support reference images in image_generate#21854NekoPii wants to merge 1 commit into
Conversation
Expose optional input_images/background parameters on the image_generate tool and pass them through to image generation providers. Teach the openai-codex image provider to attach local paths, HTTP(S) URLs, and data URLs as input_image items for Codex image generation. Add focused tests and update provider docs.
There was a problem hiding this comment.
Pull request overview
This PR extends the image_generate tool and image-gen provider interface to support optional reference images (input_images) and an output background preference (background), and wires these through to plugin-based backends (notably openai-codex) with accompanying documentation and tests.
Changes:
- Exposed
input_imagesandbackgroundin theimage_generatetool schema and dispatch path, with explicit errors for the legacy in-tree FAL backend. - Added
openai-codexprovider support for local paths, HTTP(S) URLs, anddata:image/...URLs as referenceinput_imageitems. - Updated provider documentation/ABC guidance and added focused tests for schema + forwarding + Codex request construction.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| website/docs/developer-guide/image-gen-provider-plugin.md | Documents optional provider kwargs (input_images, background) and reference-image behavior expectations. |
| tools/image_generation_tool.py | Adds new tool parameters + schema fields and forwards them to plugin providers; legacy FAL path now errors on unsupported params. |
| tests/tools/test_image_generation.py | Updates schema expectations to include new agent-facing parameters. |
| tests/tools/test_image_generation_input_images.py | Adds tests for schema exposure, plugin forwarding, and Codex input-image/background wiring. |
| plugins/image_gen/openai-codex/init.py | Implements reference-image normalization + local/data/remote handling and sends background preference to Codex Responses API. |
| agent/image_gen_provider.py | Updates provider ABC docstring to describe current optional kwargs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| def image_generate_tool( | ||
| prompt: str, | ||
| aspect_ratio: str = DEFAULT_ASPECT_RATIO, | ||
| input_images: Optional[list[str]] = None, | ||
| background: Optional[str] = None, | ||
| num_inference_steps: Optional[int] = None, | ||
| guidance_scale: Optional[float] = None, | ||
| num_images: Optional[int] = None, |
| @@ -324,6 +408,8 @@ def generate( | |||
| prompt=prompt, | |||
| size=size, | |||
| quality=meta["quality"], | |||
| input_images=input_images, | |||
| background=background, | |||
| ) | |||
| except Exception as exc: | |||
| logger.debug("Codex image generation failed", exc_info=True) | |||
alt-glitch
added
type/feature
|
Closing as a duplicate — Codex OAuth image-to-image / reference-image input is now on That PR wires source/reference images through the Codex Responses If you spot a gap in the merged implementation, please open a follow-up. 🙏 |
Summary
input_imagesandbackgroundparameters on theimage_generatetool.openai-codexprovider support for local paths, HTTP(S) URLs, anddata:image/...URLs asinput_imageitems for Codex image generation.Test Plan
uv run pytest tests/tools/test_image_generation.py tests/tools/test_image_generation_plugin_dispatch.py tests/tools/test_image_generation_input_images.py -q -o addopts=''