Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .agentsroom/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# AgentsRoom: personal files (not committed to git)
*-personal.json
agents-local.json
sessions/
10 changes: 10 additions & 0 deletions .agentsroom/agents.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
[
{
"role": "fullstack",
"model": "opus",
"customName": "Full-Stack Developer",
"isPersonal": false,
"id": "agent-1776361243376-3sekdc",
"claudeSessionId": "96773a93-be2a-45a9-a732-ceb224d3d0e5"
}
]
4 changes: 4 additions & 0 deletions .agentsroom/prompts.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"folders": [],
"prompts": []
}
21 changes: 21 additions & 0 deletions .changeset/media-edit-from.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
---
'@tanstack/ai': minor
'@tanstack/ai-openai': minor
'@tanstack/ai-gemini': minor
'@tanstack/ai-grok': minor
'@tanstack/ai-fal': minor
'@tanstack/ai-client': minor
---

feat: first-class follow-up edits for generated media.

`generateVideo({ ..., previousJobId })` edits a previously generated video instead of generating from scratch. Callers always pass the prior generation's job id; adapters decide how to consume it via a `VideoAdapter` edit-kind map:

- `'job'` — reference the id server-side (OpenAI Sora 2 / Sora 2 Pro remix; Gemini Omni Flash, which maps `previousJobId` onto the Interactions API's `previous_interaction_id` wire field — that field is omitted from Omni `modelOptions`)
- `'media'` — resolve the finished clip via `getVideoUrl(previousJobId)` (xAI `grok-imagine-video` → `/videos/edits`; fal video-to-video endpoints such as `xai/grok-imagine-video/edit-video` and Seedance 2.0 reference-to-video). Fal generate endpoints with a known edit sibling (e.g. Grok text/image-to-video) resolve on the generate model, then submit to the edit endpoint.

Non-editing models (Veo, `grok-imagine-video-1.5`) reject `previousJobId` at compile time. Sora remix and Grok edits accept only a prompt — `size` / `duration` / media inputs are rejected because the output inherits them from the source video.

`generateImage({ ..., previousImage })` is the image-side counterpart: pass a prior result's `GeneratedImage` (or an array, or the whole result) and it is prepended to the prompt as an image part, flowing through each adapter's existing edit path; type-gated to models that accept image inputs.

Breaking for hand-rolled (non-`BaseVideoAdapter`) `VideoAdapter` implementations: the interface gains `supportedEditKind(): 'job' | 'media' | undefined` (and a 7th, defaulted `TModelEditByName` generic — existing 6-argument instantiations keep compiling). `BaseVideoAdapter` supplies a default returning `undefined`, plus `resolvePreviousJobUrl(previousJobId)`. New exports include `VideoEditKind`, `ModelEditKindByName`, `VideoPreviousJobIdForAdapter`, `ImagePreviousSource`, `ImagePreviousImageForModel`, `generatedImageToImagePart`, `generatedVideoUrlToVideoPart`. Client wire types: `VideoGenerateInput.previousJobId`, `ImageGenerateInput.previousImage`.
4 changes: 2 additions & 2 deletions docs/config.json
Original file line number Diff line number Diff line change
Expand Up @@ -480,13 +480,13 @@
"label": "Image Generation",
"to": "media/image-generation",
"addedAt": "2026-04-15",
"updatedAt": "2026-08-19"
"updatedAt": "2026-08-21"
},
{
"label": "Video Generation",
"to": "media/video-generation",
"addedAt": "2026-04-15",
"updatedAt": "2026-08-20"
"updatedAt": "2026-08-21"
},
{
"label": "Generation Hooks",
Expand Down
78 changes: 71 additions & 7 deletions docs/media/image-generation.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,13 +99,14 @@ Image URLs expire after 24 hours; pass `response_format: 'b64_json'` in `modelOp

All image adapters support these common options:

| Option | Type | Description |
| ---------------- | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `adapter` | `ImageAdapter` | Image adapter instance with model (required) |
| `prompt` | `string \| MediaPromptPart[]` | Description of the image to generate (required). A plain string, or — on models that support image-conditioned generation — an ordered array of content parts interleaving text with image inputs. See [Image-Conditioned Generation](#image-conditioned-generation) below. |
| `numberOfImages` | `number` | Number of images to generate |
| `size` | `string` | Size of the generated image in WIDTHxHEIGHT format |
| `modelOptions?` | `object` | Model-specific options (renamed from `providerOptions`) |
| Option | Type | Description |
| ----------------- | ------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `adapter` | `ImageAdapter` | Image adapter instance with model (required) |
| `prompt` | `string \| MediaPromptPart[]` | Description of the image to generate (required). A plain string, or — on models that support image-conditioned generation — an ordered array of content parts interleaving text with image inputs. See [Image-Conditioned Generation](#image-conditioned-generation) below. |
| `numberOfImages` | `number` | Number of images to generate |
| `size` | `string` | Size of the generated image in WIDTHxHEIGHT format |
| `previousImage?` | `GeneratedImage \| GeneratedImage[] \| { images }` | A previously generated image (or images) to edit — prepended to the prompt as image input(s). Only offered (at compile time) for models that accept image inputs — see [Editing generated images](#editing-generated-images-previousimage). |
| `modelOptions?` | `object` | Model-specific options (renamed from `providerOptions`) |

### Size Options

Expand Down Expand Up @@ -217,6 +218,69 @@ The accepted part types are narrowed **per model at compile time**: passing
an image part to a text-only model (e.g. `dall-e-3`, Imagen) is a type
error, not just a runtime throw.

### Editing generated images (previousImage)

To run a **follow-up edit** on something you just generated, pass the prior
result's image as `previousImage` — sugar that prepends it to the prompt as an
image part, so it flows through the model's regular edit path (OpenAI
`/images/edits`, Gemini `generateContent`, xAI `/images/edits`, fal).

**Server:**

```typescript
import { generateImage } from '@tanstack/ai'
import { openaiImage } from '@tanstack/ai-openai'

const adapter = openaiImage('gpt-image-2')

const first = await generateImage({ adapter, prompt: 'A city street at dusk' })

const edited = await generateImage({
adapter,
prompt: 'Same scene, but make it rain',
previousImage: first.images[0],
})
```

`previousImage` accepts a single `GeneratedImage`, an array of them, or the
whole prior result (`{ images }`). URL results pass through as `url`
sources (`data:` URLs are decomposed into raw bytes for adapters that
upload files); `b64Json` results become `data` sources with the mime type
sniffed from the payload (defaulting to `image/png`). Like image parts, the
option is offered **per model at compile time** — text-only models
(`dall-e-3`, Imagen) reject it as a type error.

**Client** — the hook's `ImageGenerateInput.previousImage` is a wire-friendly
`{ url? }` / `{ b64Json? }` shape; your server route should narrow it back
to a `GeneratedImage` before calling `generateImage`:

```tsx
import { useGenerateImage, fetchServerSentEvents } from '@tanstack/ai-react'

function ImageEditor() {
const { generate, result, isLoading } = useGenerateImage({
connection: fetchServerSentEvents('/api/generate/image'),
})

const handleEdit = () => {
const image = result?.images[0]
if (!image) return
void generate({
prompt: 'Same scene, but make it rain',
previousImage: image.url
? { url: image.url }
: { b64Json: image.b64Json },
})
Comment on lines +253 to +273

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Include the server endpoint half of this example.

The client snippet sends a wire-friendly { url? } / { b64Json? } value and explicitly says the server must narrow it, but no route showing that conversion is provided. Add a minimal server endpoint that validates the payload and passes a GeneratedImage to generateImage.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/media/image-generation.md` around lines 219 - 239, Extend the
image-generation example with a minimal server endpoint matching the client
request, validating the wire-friendly previousImage shape, narrowing it to a
GeneratedImage, and passing it to generateImage. Reference the route handler and
generateImage usage, and show handling for invalid payloads and both URL and
base64 image variants.

Source: Coding guidelines

}

return (
<button onClick={handleEdit} disabled={isLoading || !result}>
Edit
</button>
)
}
```

### Referencing images from your prompt

**Your prompt text is always sent verbatim — the SDK never injects or
Expand Down
Loading
Loading