Skip to content

feat(image-gen): add image edit tool - #19261

Closed
BeliefanX wants to merge 3 commits into
NousResearch:mainfrom
BeliefanX:feat/image-edit-tool
Closed

feat(image-gen): add image edit tool#19261
BeliefanX wants to merge 3 commits into
NousResearch:mainfrom
BeliefanX:feat/image-edit-tool

Conversation

@BeliefanX

@BeliefanX BeliefanX commented May 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a first-class image_edit tool for prompt-guided image-to-image editing through configured image-generation backends, with OpenAI Codex / GPT Image 2 as the initial backend implementation.

What changed

  • Add ImageGenProvider.supports_edit() and default edit() fallback for backwards-compatible provider capability detection.
  • Add tools/image_edit_tool.py and expose it through the image_gen, CLI, messaging, and API-server toolsets.
  • Implement OpenAI Codex / GPT Image 2 edit calls using the Responses image_generation tool with action: edit and an input_image content part.
  • Accept local image paths, HTTP(S) URLs, and image data URLs as reference images.
  • Guard local/data image inputs:
    • local files must be PNG/JPEG/WEBP/GIF by magic bytes / MIME;
    • local files and data URLs have a size cap;
    • safe toolset intentionally excludes image_edit because it can read/upload local reference images.
  • Harden image size / aspect-ratio / model-quality override dispatch for both generation and edit paths.
  • Add tests for tool dispatch, Codex edit payloads, validation failures, toolset exposure, and docs references.

Testing

  • python -m py_compile tools/image_edit_tool.py agent/image_gen_provider.py plugins/image_gen/openai-codex/__init__.py acp_adapter/tools.py toolsets.py tests/plugins/image_gen/test_openai_codex_provider.py tests/gateway/test_api_server_toolset.py
  • python -m pytest tests/plugins/image_gen tests/tools/test_image_generation.py tests/tools/test_image_generation_plugin_dispatch.py tests/tools/test_image_edit_tool.py tests/tools/test_registry.py tests/gateway/test_api_server_toolset.py -q -o 'addopts='
    • 196 passed
  • cd website && npm run lint:diagrams
    • 290 files checked, 22 boxes found, 0 errors
  • ruff check agent/image_gen_provider.py tools/image_edit_tool.py plugins/image_gen/openai/__init__.py plugins/image_gen/openai-codex/__init__.py tests/plugins/image_gen/test_openai_provider.py

CI compatibility notes

  • Kept the branch-only docs diagram cleanup so docs-site-checks passes on current main.
  • This docs change only removes/rewrites ASCII box diagrams that the repository's ascii-guard job rejects; it is not part of the image-edit feature and can be split or dropped if maintainers prefer a separate docs PR.

Notes

This PR keeps scope narrow: it adds the edit capability abstraction, one tool, and one backend implementation. Other image providers keep working through the default unsupported-edit fallback until they opt in.

@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/tools Tool registry, model_tools, toolsets comp/plugins Plugin system and bundled plugins tool/vision Vision analysis and image generation labels May 3, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Related to #17034 — this PR adds and properly exposes the image_edit tool that was previously registered but not wired into toolsets.

1 similar comment
@alt-glitch

Copy link
Copy Markdown
Collaborator

Related to #17034 — this PR adds and properly exposes the image_edit tool that was previously registered but not wired into toolsets.

@BeliefanX

Copy link
Copy Markdown
Contributor Author

Thanks for linking #17034.

I think this PR covers the main gap described there: adding a first-class image_edit tool, wiring it through the image_gen/toolset/API surfaces, and including the initial OpenAI Codex / GPT Image 2 backend implementation with regression coverage.

One intentional boundary: safe excludes image_edit, since edit can read/upload local reference images.

@BeliefanX
BeliefanX force-pushed the feat/image-edit-tool branch 2 times, most recently from 5aadfdf to d2e6030 Compare May 4, 2026 13:37
@BeliefanX
BeliefanX force-pushed the feat/image-edit-tool branch from d2e6030 to 147cb3b Compare May 5, 2026 10:32
@BeliefanX

Copy link
Copy Markdown
Contributor Author

CI note:

The remaining failing test check appears to match the current upstream main full-suite baseline rather than this PR's image-edit changes.

For this PR head (523d4ba21c95364e32d40086a7cdd38cf717ebf5):

  • docs-site-checks: passing
  • nix (ubuntu-latest) / nix (macos-latest): passing
  • e2e: passing
  • supply-chain audit: passing
  • focused image-gen/edit regression suite passes locally: 196 passed

The full test job failure is broad and mostly overlaps with the latest failing main Tests run I checked:

  • PR test job: 30 failed, 19716 passed, 49 skipped
  • latest main Tests run checked: 33 failed, 19679 passed, 49 skipped

The failures are concentrated in unrelated areas such as Bedrock headers, cron script prompts, gateway/update restart tests, browser Chromium checks, delegate provider resolution / heartbeat, Dockerfile TUI deps, credential env fallback, and sandbox cwd wrappers.

I kept this PR focused on image-edit support and only included the minimal docs diagram cleanup needed for the current docs-site-checks job. Happy to rebase after the main test baseline is restored, or split/drop the docs compatibility cleanup if maintainers prefer.

@BeliefanX

Copy link
Copy Markdown
Contributor Author

I opened #21765 as a smaller replacement for the core image-edit capability from this PR.

This older branch had drifted behind current main, had conflicts/red checks, and also mixed the core image_edit work with some adjacent cleanup. Rather than ask reviewers to reason through the stale branch, #21765 keeps the focused pieces only:

  • image_edit tool plumbing
  • provider edit capability and OpenAI/OpenAI-Codex gpt-image-2 edit support
  • reference-image validation for local/data inputs
  • ACP/toolset/display integration
  • focused regression tests

Closing this PR in favor of the clean replacement. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/plugins Plugin system and bundled plugins comp/tools Tool registry, model_tools, toolsets P3 Low — cosmetic, nice to have tool/vision Vision analysis and image generation type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants