Skip to content

feat: image_edit tool — prompt-guided img2img editing with OpenAI Codex - #32248

Closed
CipherFrame wants to merge 1 commit into
NousResearch:mainfrom
CipherFrame:feat/image-edit-codex
Closed

feat: image_edit tool — prompt-guided img2img editing with OpenAI Codex#32248
CipherFrame wants to merge 1 commit into
NousResearch:mainfrom
CipherFrame:feat/image-edit-codex

Conversation

@CipherFrame

Copy link
Copy Markdown
Contributor

Summary

Adds an image_edit tool that enables prompt-guided, image-to-image editing via the OpenAI Codex provider (GPT Image 2). Agents can now intelligently route to image_edit when the user provides a reference image, instead of falling through to text-to-image generation.

What it does

  • New image_edit tool (tools/image_edit_tool.py): dispatches to any provider that supports edit()
  • Codex provider edit() method: passes a reference image through the Codex Responses API (input_image + text prompt)
  • Provider interface extension: supports_edit() and edit() on ImageGenProvider base, defaulting to unsupported for existing providers
  • Full wiring: model_tools, toolsets, prompt_builder, CLI config, ACP adapter, MCP server transport

Files changed

tools/image_edit_tool.py                              | 224 ++++++ (new)
tests/tools/test_image_edit_tool.py                   | 195 ++++++ (new)
plugins/image_gen/openai-codex/__init__.py            | 273 +++++++-
tests/plugins/image_gen/test_openai_codex_provider.py | 100 ++++
agent/image_gen_provider.py                           |  32 +
agent/display.py                                      |   4 +-
agent/prompt_builder.py                               |   1 +
agent/transports/hermes_tools_mcp_server.py           |   1 +
hermes_cli/tools_config.py                            |   2 +-
model_tools.py                                        |   2 +-
toolsets.py                                           |   2 +-
acp_adapter/tools.py                                  |   7 +-

12 files, 825 insertions, 18 deletions. 40 new tests, 141 existing pass — zero regressions.

Test plan

  • Schema validation (name, required fields, parameter types)
  • Provider dispatch — routes to correct backend
  • Codex edit(): auth errors, missing/invalid image, nonexistent paths
  • Codex edit(): streams input_image through Responses API, validates response
  • Unsupported provider returns graceful error
  • Real-world smoke test: historical B&W photo → restored + colorized (verified in testing environment)

Demo

Before (B&W damaged historical photo) → After (restored, colorized):

Smoke-tested in Hermes Telegram gateway — gpt-image-2-high successfully processed a 1890s football team photograph with scratch removal, stain cleanup, and natural colorization.

Why

Closes the gap where image_generate (text-to-image) was the only option. Agents now see image_edit as a mandatory tool when a reference image is present ("Mandatory: when the user provides, uploads, links, or names any reference/source/product/person image, use this image-to-image tool rather than image_generate/text-to-image"). This enables historical photo restoration, style transfer, product variant editing, and other reference-guided workflows.


Submitted from CipherFrame fork — happy to iterate based on review feedback.

- New image_edit tool: prompt-guided image-to-image editing
- Codex provider: edit() method via Responses API with input_image
- Provider interface: supports_edit() + edit() on ImageGenProvider base
- 40 new tests: schema, handler, provider validation, error paths
- Tool wired into model_tools, toolsets, prompt_builder, CLI config
- ACP adapter + MCP server transport updated for tool schema

Motivation: enables historical photo restoration, style transfer,
and reference-guided image editing without leaving Hermes.

Closes the gap where image_generate (text-to-image) was the only
option — now agents can intelligently choose image_edit when a
reference image is provided.
@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/tools Tool registry, model_tools, toolsets comp/plugins Plugin system and bundled plugins tool/vision Vision analysis and image generation provider/openai OpenAI / Codex Responses API labels May 25, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Saturated image_edit feature cluster — competing with multiple open PRs:

Also see closed #19261 (first-class image_edit tool, same architecture with supports_edit()/edit() on ImageGenProvider ABC).

@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the Codex image-editing implementation. The member's cluster note correctly identified overlapping work; automated hermes-sweeper review found that this capability is now implemented on main through the unified image-generation surface.

  • tools/image_generation_tool.py:1175 exposes image_url and reference_image_urls on image_generate; :1525 forwards them to the active provider.
  • plugins/image_gen/openai-codex/__init__.py:460 advertises Codex image-input support, and :525 turns source/reference inputs into Responses input_image parts before submission.
  • tests/plugins/image_gen/test_openai_codex_provider.py:169 covers source/reference payload construction and :207 covers the 16-image cap.
  • ecffd290a and 460235d58 shipped in v2026.7.7.

Current main deliberately uses one image_generate model-facing tool for both text-to-image and image-to-image editing, so the separate image_edit surface in this PR is redundant.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/plugins Plugin system and bundled plugins comp/tools Tool registry, model_tools, toolsets P3 Low — cosmetic, nice to have provider/openai OpenAI / Codex Responses API sweeper:implemented-on-main Sweeper: behavior already present on current main tool/vision Vision analysis and image generation type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants