Support follow-up edits for generated images#5712
Conversation
pre-commit-ci
Bot
requested review from
danielhanchen and
rolandtannous
as code owners
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request implements support for OpenAI's image_generation_call references, allowing follow-up prompts to refine generated images without resending the full payload. Key changes include updating the backend to handle these call references in streaming responses and tool events, adding new Pydantic models for content parts, and modifying the frontend chat adapter to collect and forward these references in outbound messages. A potential issue was identified in the frontend where empty message content might bypass validation checks and lead to backend errors.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6045257e35
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
💡 Codex Reviewunsloth/studio/frontend/src/features/chat/api/chat-adapter.ts Lines 955 to 956 in 271c20e  Preserve pending image edit reference until request can run
ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 328816ccea
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9ee5c9be8d
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8db2f5eb07
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
for more information, see https://pre-commit.ci
…260526-104415 # Conflicts: # studio/backend/core/inference/external_provider.py # studio/frontend/src/features/chat/api/chat-adapter.ts
for more information, see https://pre-commit.ci
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
1 similar comment
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
Main moved forward 17 commits during PR review (latest: 953c8bf). Real conflicts in five files; resolved by combining both branches' changes. studio/backend/core/inference/external_provider.py - Add fast_mode (Anthropic Opus 4.6/4.7 speed flag, unslothai#5715) to stream_chat_completion and Anthropic-branch call site, alongside existing Gemini tools/tool_choice forwarding. - Add _openai_image_generation_tool() helper (action:"edit" for follow- up image edits, unslothai#5712) and use it inside the existing _responses_hosted_builtins_allowed gate so the forced-function / tool_choice="none" suppression added in rounds 21+ still applies. - Keep Anthropic web_fetch gated on _anthropic_hosted_builtins_allowed (round 19+ hosted-builtin gate) while taking main's per-model version selector (web_fetch_20260209 vs _20250910). studio/backend/routes/inference.py - Add `openai = provider_type == "openai"` (used by main's reasoning content forwarding for follow-up image edits). - Keep the round 25/26 Gemini filter chain (_filter_tool_calls drops synthetic server-builtin cards, marks tc_id so the matching role="tool" follow-up gets skipped, extra_content gated to native Gemini host). - Forward fast_mode alongside tools/tool_choice. studio/backend/tests/test_openai_image_generation.py - Combine assertions: both _server_tool: True (PR) and openai_image_generation_call_id (main) are present on the tool_start arguments. studio/frontend/src/features/chat/shared-composer.tsx - Add supportsBuiltinWebFetch declaration (separate Fetch pill from unslothai#5742) before the PR's isExternalGemini constant so both the Gemini image-tier gating and the standalone Anthropic Fetch pill compile. studio/frontend/src/features/chat/api/chat-adapter.ts - Add main's normalizeOpenAIReasoningItem, toOpenAIImageEditReferenceMessage, isAnthropicRefusalMessage helpers alongside PR's collectAssistantToolCalls, collectToolResultMessages, SerializedMessage, collectAssistantTextThoughtSignature. - toOpenAIMessages (PR) now also early-returns on isAnthropicRefusalMessage so refused turns get pruned from outbound history. - Add a thin toOpenAIMessage (singular) wrapper for the OpenAI image- edit replay path's flat .map() usage. - Merge per-turn enable flags: keep PR's imageGenerationEnabledForThisTurn, geminiImageModeForThisTurn, codeExecEnabledForThisTurn !geminiImageMode gate; take main's webFetchEnabledForThisTurn (sourced from independent webFetchToolsEnabled pill state). - Outbound build chains main's anthropic_refusal survivingMessages prune, then flatMap(toOpenAIMessages) (PR), then PR's selectedImageEditReference reference message prepend; image-edit unavailable toast from main fires before any of that when the pill is off. - tool_end merge: do main's nextArgs spread first, then PR's Gemini native_part parts concat so both OpenAI image-call ids and Gemini executableCode/codeExecutionResult/inlineData round-trip. - Cumulative + final yields: orderAssistantContent(pinTextThoughtSignature(...)) composes main's tool-vs-text ordering with PR's per-text thoughtSignature pin. Tests: gemini provider 148/148; openai_responses_translation + openai_code_execution + openai_image_generation + anthropic_code_execution + anthropic_web_fetch + external_provider_usage_chunk + providers_api: 50 passed, 42 skipped; main's new anthropic_fast_mode + citations + openai_citation_markers + openai_tool_result_fallbacks suites all 43/43.
2 participants
Summary
Testing