Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .changeset/generate-world.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
'@tanstack/ai': minor
'@tanstack/ai-event-client': patch
'@tanstack/ai-reactor': minor
'@tanstack/ai-fal': minor
---

Add `generateWorld()` and `generateLiveVideo()` for prompt-steerable sessions, plus a first-party Reactor adapter (`reactorWorld`, `reactorVideo`) and fal `falLiveVideo()` for H3 Max Director. Reactor returns a session JWT. falLiveVideo returns the WMA app id on `result.model` so the browser can call `wma(live.model)`. `generateVideo()` stays the job path that polls for a file URL.
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,7 @@ testing/

examples/ # Example applications
├── ts-react-chat/ # React chat example
├── ts-react-media/ # Image, video, live, and world generation
├── ts-solid-chat/ # Solid chat example
├── ts-vue-chat/ # Vue chat example
├── ts-svelte-chat/ # Svelte chat example
Expand Down
31 changes: 16 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,7 +190,7 @@ Learn more in the

- [Generations](https://tanstack.com/ai/latest/docs/media/generations) - one
pattern for image generation, text-to-speech, transcription, summarization,
audio generation, and video generation.
audio generation, video generation, and world generation.
- [Realtime Voice Chat](https://tanstack.com/ai/latest/docs/media/realtime-chat) -
build low-latency realtime voice experiences.
- [Code Mode](https://tanstack.com/ai/latest/docs/code-mode/code-mode) - let
Expand All @@ -202,20 +202,21 @@ Learn more in the

Official adapters include:

| Package | Use it for |
| ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------ |
| [`@tanstack/ai-openrouter`](https://tanstack.com/ai/latest/docs/adapters/openrouter) | 300+ models through one OpenRouter API, with per-request cost tracking |
| [`@tanstack/ai-openai`](https://tanstack.com/ai/latest/docs/adapters/openai) | OpenAI chat, image, video, speech, transcription, realtime, and provider tools |
| [`@tanstack/ai-anthropic`](https://tanstack.com/ai/latest/docs/adapters/anthropic) | Anthropic Claude chat, thinking, tools, structured outputs, and Vertex Claude |
| [`@tanstack/ai-gemini`](https://tanstack.com/ai/latest/docs/adapters/gemini) | Google Gemini chat, image, speech, and audio generation |
| [`@tanstack/ai-vertex`](https://tanstack.com/ai/latest/docs/adapters/vertex) | Gemini on Vertex AI with regional endpoints and Google Cloud credentials |
| [`@tanstack/ai-ollama`](https://tanstack.com/ai/latest/docs/adapters/ollama) | Local Ollama models |
| [`@tanstack/ai-grok`](https://tanstack.com/ai/latest/docs/adapters/grok) | xAI Grok chat, images, and realtime |
| [`@tanstack/ai-groq`](https://tanstack.com/ai/latest/docs/adapters/groq) | Groq low-latency inference |
| [`@tanstack/ai-elevenlabs`](https://tanstack.com/ai/latest/docs/adapters/elevenlabs) | ElevenLabs realtime voice, speech, transcription, music, and sound effects |
| [`@tanstack/ai-byteplus`](https://tanstack.com/ai/latest/docs/adapters/byteplus) | BytePlus Seed chat, Seedance video, Seedream image, and Seed Speech TTS/ASR |
| [`@tanstack/ai-fal`](https://tanstack.com/ai/latest/docs/adapters/fal) | fal.ai image, video, audio, speech, and transcription models |
| [`@tanstack/ai-cloudflare`](https://tanstack.com/ai/latest/docs/adapters/cloudflare) | Cloudflare Workers AI chat, embeddings, image, speech, transcription, and AI Gateway |
| Package | Use it for |
| ------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------- |
| [`@tanstack/ai-openrouter`](https://tanstack.com/ai/latest/docs/adapters/openrouter) | 300+ models through one OpenRouter API, with per-request cost tracking |
| [`@tanstack/ai-openai`](https://tanstack.com/ai/latest/docs/adapters/openai) | OpenAI chat, image, video, speech, transcription, realtime, and provider tools |
| [`@tanstack/ai-anthropic`](https://tanstack.com/ai/latest/docs/adapters/anthropic) | Anthropic Claude chat, thinking, tools, structured outputs, and Vertex Claude |
| [`@tanstack/ai-gemini`](https://tanstack.com/ai/latest/docs/adapters/gemini) | Google Gemini chat, image, speech, and audio generation |
| [`@tanstack/ai-vertex`](https://tanstack.com/ai/latest/docs/adapters/vertex) | Gemini on Vertex AI with regional endpoints and Google Cloud credentials |
| [`@tanstack/ai-ollama`](https://tanstack.com/ai/latest/docs/adapters/ollama) | Local Ollama models |
| [`@tanstack/ai-grok`](https://tanstack.com/ai/latest/docs/adapters/grok) | xAI Grok chat, images, and realtime |
| [`@tanstack/ai-groq`](https://tanstack.com/ai/latest/docs/adapters/groq) | Groq low-latency inference |
| [`@tanstack/ai-elevenlabs`](https://tanstack.com/ai/latest/docs/adapters/elevenlabs) | ElevenLabs realtime voice, speech, transcription, music, and sound effects |
| [`@tanstack/ai-byteplus`](https://tanstack.com/ai/latest/docs/adapters/byteplus) | BytePlus Seed chat, Seedance video, Seedream image, and Seed Speech TTS/ASR |
| [`@tanstack/ai-fal`](https://tanstack.com/ai/latest/docs/adapters/fal) | fal.ai image, video, audio, speech, and transcription models |
| [`@tanstack/ai-reactor`](https://tanstack.com/ai/latest/docs/adapters/reactor) | Reactor live world and video generation (Orbis, Happy Oyster, LingBot, Helios, FastH3) |
| [`@tanstack/ai-cloudflare`](https://tanstack.com/ai/latest/docs/adapters/cloudflare) | Cloudflare Workers AI chat, embeddings, image, speech, transcription, and AI Gateway |

The adapter system is tree-shakeable by activity. Import `openaiText` for chat,
`openaiImage` for images, `falVideo` for video, `geminiSpeech` for TTS, and so
Expand Down
21 changes: 20 additions & 1 deletion docs/adapters/fal.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ keywords:
- adapter
---

The fal.ai adapter provides access to 600+ models on the fal.ai platform for image, video, audio, speech, and transcription. Unlike text-focused adapters, the fal adapter is **media-focused** — it supports `generateImage()`, `generateVideo()`, `generateAudio()`, `generateSpeech()`, and `generateTranscription()` but does not support `chat()` or tools.
The fal.ai adapter provides access to 600+ models on the fal.ai platform for image, video, live, audio, speech, and transcription. Unlike text-focused adapters, the fal adapter is **media-focused**. It supports `generateImage()`, `generateVideo()`, `generateLiveVideo()`, `generateAudio()`, `generateSpeech()`, and `generateTranscription()`. It does not support `chat()` or tools.

For a full working example, see the [fal.ai example app](https://github.com/TanStack/ai/tree/main/examples/ts-react-media).

Expand Down Expand Up @@ -396,6 +396,12 @@ const sfx = await generateAudio({
| `fal-ai/ltx-2/text-to-video/fast` | Text-to-Video | Fast text-to-video |
| `fal-ai/ltx-2/image-to-video/fast` | Image-to-Video | Fast image-to-video animation |

### Live Models

| Model | Mode | Description |
|-------|------|-------------|
| `minimax/h3-max/director` | Live | Steerable live stream. Use `falLiveVideo()` with `generateLiveVideo()`. |

### Text-to-Speech Models

| Model | Description |
Expand Down Expand Up @@ -473,6 +479,19 @@ Creates a fal.ai video adapter using the `FAL_KEY` environment variable or an ex

**Returns:** A `FalVideoAdapter` instance for use with `generateVideo()` and `getVideoJobStatus()`.

### `falLiveVideo(model, config?)`

Creates a fal.ai live-video adapter for H3 Max Director. `generateLiveVideo()` returns the WMA app id on `result.model` (`fal-ai/minimax-h3-max-director`). Open that id with `wma(live.model)` through a server proxy that attaches `FAL_KEY`. Do not send `live.token` as `Key` credentials. Call `allowedFalLiveVideoProxyTarget()` in the proxy so it forwards only WMA `/ice`, `/session`, `/session/heartbeat`, and Director `/ice`.

**Parameters:**

- `model` - `"minimax/h3-max/director"`
- `config.apiKey?` - Your fal.ai API key (falls back to `FAL_KEY` env var)

**Returns:** A `FalLiveVideoAdapter` instance for use with `generateLiveVideo()`.

See [Live Generation](../media/live-generation) for the browser connect step.

### `falSpeech(model, config?)`

Creates a fal.ai text-to-speech adapter.
Expand Down
178 changes: 178 additions & 0 deletions docs/adapters/reactor.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,178 @@
---
title: Reactor
id: reactor-adapter
description: "Generate live worlds and video with Reactor models in TanStack AI via the @tanstack/ai-reactor adapter."
keywords:
- tanstack ai
- reactor
- world generation
- live generation
- orbis
- helios
- generateWorld
- generateLiveVideo
- adapter
---

Reactor hosts live world and video models. You describe a scene. Then you open a session and stream video. You can steer the stream with a new prompt.

Use `reactorWorld()` with `generateWorld()` for navigable worlds. Use `reactorVideo()` with `generateLiveVideo()` for live video models. Both mint a session token. Neither supports `chat()`.

## Installation

<!-- ::start:tabs variant="package-manager" mode="install" -->

react: @tanstack/ai-reactor
vue: @tanstack/ai-reactor
solid: @tanstack/ai-reactor
svelte: @tanstack/ai-reactor
preact: @tanstack/ai-reactor
angular: @tanstack/ai-reactor
vanilla: @tanstack/ai-reactor
octane: @tanstack/ai-reactor

<!-- ::end:tabs -->

Peer dependency: `@tanstack/ai`. The browser also needs `@reactor-team/js-sdk` to connect and play the stream. See [World Generation](../media/world-generation) and [Live Generation](../media/live-generation).

A full working app is in [`examples/ts-react-media`](https://github.com/TanStack/ai/tree/main/examples/ts-react-media). Open the World or Live tab.

## API key

Create a key in the [Reactor dashboard](https://www.reactor.inc/dashboard). Keys start with `rk_`.

The example app uses [Bring Your Own Key](../advanced/byok). The browser pastes the key. The relay reads `x-byok-reactor`, then `REACTOR_API_KEY`.

```ts
import { generateWorld } from '@tanstack/ai'
import { byokMissing, getByokKey } from '@tanstack/ai/byok/server'
import { reactorWorld } from '@tanstack/ai-reactor'
import { reactorByok } from '@tanstack/ai-reactor/byok'

export async function POST(request: Request) {
const apiKey = getByokKey(request, reactorByok)
if (!apiKey) return byokMissing(reactorByok)

const world = await generateWorld({
adapter: reactorWorld('visko-orbis-stable', { apiKey }),
prompt: 'A neon cyberpunk city at night, slow aerial drift',
})

return Response.json({
token: world.token,
model: world.model,
prompt: world.prompt,
expiresAt: world.expiresAt,
})
}
```

The adapter mints a **session-scoped** token for that model only. Hand `world.token`, `world.model`, and `world.prompt` to the browser. Do not put the API key in the JSON body.

## Models

Pass a string literal so TypeScript can narrow options.

```ts
import { reactorWorld } from '@tanstack/ai-reactor'

const adapter = reactorWorld('visko-orbis-stable')
```

| Id | Connect slug |
| --- | --- |
| `visko-orbis-stable` | `reactor/visko-orbis-stable` |
| `visko-orbis-dynamic` | `reactor/visko-orbis-dynamic` |
| `happy-oyster-adventure` | `reactor/happy-oyster-adventure` |
| `happy-oyster-director` | `reactor/happy-oyster-director` |
| `lingbot-world-2` | `reactor/lingbot-world-2` |
| `lingbot` | `reactor/lingbot` |
| `helios` | `reactor/helios` |

`world.model` is the connect slug. Pass it to `new Reactor({ modelName })`.

## Live video

Reactor video is a live stream, not a finished file. `generateLiveVideo()` returns a token. The browser connects, sets the prompt, and plays the track.

```ts
import { generateLiveVideo } from '@tanstack/ai'
import { reactorVideo } from '@tanstack/ai-reactor'

const apiKey = process.env.REACTOR_API_KEY ?? ''
const live = await generateLiveVideo({
adapter: reactorVideo('helios', { apiKey }),
prompt: 'A neon cyberpunk city at night, slow aerial drift',
})
```

Hand `live.token`, `live.model`, and `live.prompt` to the browser. Connect as shown in [Live Generation](../media/live-generation).

| Id | Connect slug |
| --- | --- |
| `helios` | `reactor/helios` |
| `fast-h3` | `reactor/fast-h3` |
| `longlive-v2` | `reactor/longlive-v2` |
| `ltx2` | `reactor/ltx2` |

`helios` also works with `reactorWorld()`. Pick `generateLiveVideo()` when you want a video session. Pick `generateWorld()` when you want a navigable world.

Pass a text prompt to `generateWorld()` / `generateLiveVideo()`. Those calls mint a token. They do not send resolution or a seed image.

After `connect`, send browser commands. LingBot starts from a seed image. Pass a `File` from `<input type="file">`. The SDK uploads it and returns a `FileRef`. Do not send base64. `start` still needs `set_prompt`. Send a short default, then steer after the first frame.

```ts
import { Reactor } from '@reactor-team/js-sdk'

const reactor = new Reactor({ modelName: 'reactor/lingbot' })
const picker = document.querySelector('input[type="file"]')
if (!(picker instanceof HTMLInputElement)) {
throw new Error('Pick a seed image')
}
const file = picker.files?.[0]
if (file === undefined) {
throw new Error('Pick a seed image')
}
const image = await reactor.uploadFile(file)
await reactor.sendCommand('set_image', { image })
await reactor.sendCommand('set_prompt', { prompt: 'Follow the seed image.' })
await reactor.sendCommand('start', {})
```

Helios can take the same `File` with `set_conditioning` so prompt and image land together.

## Browser session options

Orbis reads these on the next `start`. Keep them in client state. Send them with `sendCommand` after `connect`. They are not token-mint fields.

```ts
import { Reactor } from '@reactor-team/js-sdk'

const reactor = new Reactor({ modelName: 'reactor/visko-orbis-stable' })
await reactor.sendCommand('set_resolution', { resolution: '2k' })
await reactor.sendCommand('set_seed', { seed: 42 })
await reactor.sendCommand('set_audio_enabled', { enabled: true })
```

| Option | Command | Meaning |
| --- | --- | --- |
| `resolution` | `set_resolution` | `1080p`, `2k`, or `4k` delivery tier |
| `seed` | `set_seed` | RNG seed for the next run |
| `audioEnabled` | `set_audio_enabled` | When `false`, skip audio compute |
| `audioPrompt` | `set_audio_prompt` | Sound description, or `""` for picture-driven audio |

## Custom endpoint

```ts
import { reactorWorld } from '@tanstack/ai-reactor'

const apiKey = process.env.REACTOR_API_KEY ?? ''
const adapter = reactorWorld('visko-orbis-stable', {
apiKey,
baseUrl: 'https://api.reactor.inc',
})
```

## What you have now

A server call that mints a scoped Reactor token for one world or video model. Next: connect in the browser as shown in [World Generation](../media/world-generation) or [Live Generation](../media/live-generation).
4 changes: 3 additions & 1 deletion docs/advanced/otel.md
Original file line number Diff line number Diff line change
Expand Up @@ -215,7 +215,7 @@ otelMiddleware({

## Beyond chat: media activities

`otelMiddleware` is not chat-only. The media activities `generateImage`, `generateVideo`, `generateAudio`, `generateSpeech`, and `generateTranscription`accept the **same** `otelMiddleware` value on their `middleware` option. Each is a single request response (or submit poll for video), so the middleware emits one span per call instead of the chat span tree:
`otelMiddleware` is not chat-only. The media activities (`generateImage`, `generateVideo`, `generateLiveVideo`, `generateWorld`, `generateAudio`, `generateSpeech`, and `generateTranscription`) accept the **same** `otelMiddleware` value on their `middleware` option. Each is a single request to response (or submit then poll for video), so the middleware emits one span per call instead of the chat span tree:

```ts
import { generateImage } from '@tanstack/ai'
Expand Down Expand Up @@ -246,6 +246,8 @@ Each media call produces one `CLIENT` span tagged with the activity's `gen_ai.op
| `generateAudio` | `audio_generation` |
| `generateSpeech` | `text_to_speech` |
| `generateTranscription` | `transcription` |
| `generateWorld` | `world_generation` |
| `generateLiveVideo` | `live_video_generation` |
| `summarize` | `summarize` |

The span carries `gen_ai.system` and `gen_ai.request.model` at start and, on finish, the same `gen_ai.usage.*` / `tanstack.ai.usage.*` attributes documented above — including the `tanstack.ai.usage.billed_quantity` / `tanstack.ai.usage.billed_unit` pair for unit-billed media. When a `Meter` is supplied it records the `gen_ai.client.operation.duration` histogram, tagged per activity. For streaming video the span covers the full create → poll → complete lifecycle. Non-streaming video is two calls, so the submit itself emits no span — the run opens once the provider accepts the job, and the `getVideoJobStatus()` poll that observes a terminal state ends it. If a streaming video consumer abandons the stream before completion, the span is ended via `onAbort` (status `ERROR`, `tanstack.ai.completion.reason = cancelled`) rather than leaked.
Expand Down
4 changes: 4 additions & 0 deletions docs/advanced/tree-shaking.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@ import { generateImage } from '@tanstack/ai' // Image generation
import { generateSpeech } from '@tanstack/ai' // Text-to-speech
import { generateTranscription } from '@tanstack/ai' // Audio transcription
import { generateVideo } from '@tanstack/ai' // Video generation
import { generateLiveVideo } from '@tanstack/ai' // Live video sessions
import { generateWorld } from '@tanstack/ai' // Live world sessions
```

### Example: Chat Only
Expand Down Expand Up @@ -274,6 +276,8 @@ Each adapter type implements a specific interface:
- `TTSAdapter` - Provides `generateSpeech()` method for text-to-speech
- `TranscriptionAdapter` - Provides `generateTranscription()` method for audio transcription
- `VideoAdapter` - Provides `generateVideo()` method for video generation
- `LiveVideoAdapter` - Provides `createLiveVideo()` method for live video sessions
- `WorldAdapter` - Provides `createWorld()` method for live world sessions

All adapters have a `kind` property that indicates their type:

Expand Down
2 changes: 2 additions & 0 deletions docs/advanced/typed-options.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,8 @@ Each helper mirrors the activity it pairs with. Same options, same return type.
| `createImageOptions` | `generateImage()` | image adapter (e.g. `openaiImage`, `falImage`) |
| `createAudioOptions` | `generateAudio()` | audio adapter (e.g. `falAudio`, `geminiAudio`) |
| `createVideoOptions` | `generateVideo()` / `getVideoJobStatus()` | video adapter (e.g. `falVideo`, `openaiVideo`) |
| `createLiveVideoOptions` | `generateLiveVideo()` | live adapter (e.g. `reactorVideo`, `falLiveVideo`) |
| `createWorldOptions` | `generateWorld()` | world adapter (e.g. `reactorWorld`) |
| `createSpeechOptions` | `generateSpeech()` | speech adapter (e.g. `openaiSpeech`, `elevenlabsSpeech`) |
| `createTranscriptionOptions` | `generateTranscription()` | transcription adapter (e.g. `openaiTranscription`, `falTranscription`) |

Expand Down
Loading
Loading