feat: add generateWorld() and generateLiveVideo() for prompt-steerable sessions #1321
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| --- | ||
| '@tanstack/ai': minor | ||
| '@tanstack/ai-event-client': patch | ||
| '@tanstack/ai-reactor': minor | ||
| --- | ||
|
|
||
| Add `generateWorld()` for live, prompt-steerable world sessions, plus a first-party Reactor adapter (`reactorWorld`) that mints a session-scoped token for Orbis and other Reactor world models. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,112 @@ | ||
| --- | ||
| title: Reactor | ||
| id: reactor-adapter | ||
| description: "Generate live worlds with Reactor models in TanStack AI via the @tanstack/ai-reactor adapter." | ||
| keywords: | ||
| - tanstack ai | ||
| - reactor | ||
| - world generation | ||
| - orbis | ||
| - generateWorld | ||
| - adapter | ||
| --- | ||
|
|
||
| Reactor hosts live world models. You describe a scene. Then you open a session and stream video. You can steer the stream with a new prompt. This adapter is for `generateWorld()`. It does not support `chat()`. | ||
|
|
||
| ## Installation | ||
|
|
||
| <!-- ::start:tabs variant="package-manager" mode="install" --> | ||
|
|
||
| react: @tanstack/ai-reactor | ||
| vue: @tanstack/ai-reactor | ||
| solid: @tanstack/ai-reactor | ||
| svelte: @tanstack/ai-reactor | ||
| preact: @tanstack/ai-reactor | ||
| angular: @tanstack/ai-reactor | ||
| vanilla: @tanstack/ai-reactor | ||
| octane: @tanstack/ai-reactor | ||
|
|
||
| <!-- ::end:tabs --> | ||
|
|
||
| Peer dependency: `@tanstack/ai`. The browser also needs `@reactor-team/js-sdk` to connect and play the stream. See [World Generation](../media/world-generation). | ||
|
|
||
| A full working app is in [`examples/ts-react-world`](https://github.com/TanStack/ai/tree/main/examples/ts-react-world). | ||
|
|
||
| ## API key | ||
|
|
||
| Create a key in the [Reactor dashboard](https://www.reactor.inc/dashboard). Keys start with `rk_`. Set `REACTOR_API_KEY` on the server, or pass `apiKey` to the adapter. | ||
|
|
||
| ```ts | ||
| import { generateWorld } from '@tanstack/ai' | ||
| import { reactorWorld } from '@tanstack/ai-reactor' | ||
|
|
||
| const apiKey = process.env.REACTOR_API_KEY | ||
| if (!apiKey) { | ||
| throw new Error('REACTOR_API_KEY is not set') | ||
| } | ||
|
|
||
| const world = await generateWorld({ | ||
| adapter: reactorWorld('visko-orbis-stable', { apiKey }), | ||
| prompt: 'A neon cyberpunk city at night, slow aerial drift', | ||
| }) | ||
| ``` | ||
|
|
||
| The adapter mints a **session-scoped** token for that model only. Hand `world.token`, `world.model`, and `world.prompt` to the browser. Do not send the API key. | ||
|
|
||
| ## Models | ||
|
|
||
| Pass a string literal so TypeScript can narrow options. | ||
|
|
||
| ```ts | ||
| import { reactorWorld } from '@tanstack/ai-reactor' | ||
|
|
||
| const adapter = reactorWorld('visko-orbis-stable') | ||
| ``` | ||
|
|
||
| | Id | Connect slug | | ||
| | --- | --- | | ||
| | `visko-orbis-stable` | `reactor/visko-orbis-stable` | | ||
| | `visko-orbis-dynamic` | `reactor/visko-orbis-dynamic` | | ||
| | `happy-oyster-adventure` | `reactor/happy-oyster-adventure` | | ||
| | `happy-oyster-director` | `reactor/happy-oyster-director` | | ||
| | `lingbot-world-2` | `reactor/lingbot-world-2` | | ||
| | `lingbot` | `reactor/lingbot` | | ||
| | `helios` | `reactor/helios` | | ||
|
|
||
| `world.model` is the connect slug. Pass it to `new Reactor({ modelName })`. | ||
|
|
||
| ## Provider options | ||
|
|
||
| Orbis reads these on the next `start`. Put them in `modelOptions`. The browser applies them with `sendCommand` before `start`. | ||
|
|
||
| ```ts | ||
| const world = await generateWorld({ | ||
| adapter: reactorWorld('visko-orbis-stable'), | ||
| prompt: 'Black volcanic cliffs, slow aerial camera', | ||
| modelOptions: { | ||
| resolution: '2k', | ||
| seed: 42, | ||
| audioEnabled: true, | ||
| }, | ||
| }) | ||
| ``` | ||
|
|
||
| | Option | Meaning | | ||
| | --- | --- | | ||
| | `resolution` | `1080p`, `2k`, or `4k` delivery tier | | ||
| | `seed` | RNG seed for the next run | | ||
| | `audioEnabled` | When `false`, skip audio compute | | ||
| | `audioPrompt` | Sound description, or `""` for picture-driven audio | | ||
|
|
||
| ## Custom endpoint | ||
|
|
||
| ```ts | ||
| const adapter = reactorWorld('visko-orbis-stable', { | ||
| apiKey, | ||
| baseUrl: 'https://api.reactor.inc', | ||
| }) | ||
| ``` | ||
|
|
||
| ## What you have now | ||
|
|
||
| A server call that mints a scoped Reactor token for one world model. Next: connect in the browser as shown in [World Generation](../media/world-generation). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,122 @@ | ||
| --- | ||
| title: World Generation | ||
| id: world-generation | ||
| order: 7 | ||
| description: "Open a live, prompt-steerable world session with generateWorld(). Mint a token on the server, connect in the browser, and steer the stream with new prompts." | ||
| keywords: | ||
| - tanstack ai | ||
| - world generation | ||
| - generateWorld | ||
| - reactor | ||
| - orbis | ||
| - infinite world | ||
| - live video | ||
| - experimental | ||
| --- | ||
|
|
||
| # World Generation (Experimental) | ||
|
|
||
| You want a world that generates while the viewer watches or changes the prompt. A finite video job stops. `generateWorld()` opens a live session instead. | ||
|
|
||
| Call `generateWorld()` on the server. It returns a short-lived token, a model slug, and the prompt. The browser connects, sets the prompt, and starts the stream. | ||
|
|
||
| > **Experimental.** The API can change. World models bill per session-second while a GPU is held. | ||
|
|
||
| ## 1. Mint a session on the server | ||
|
|
||
| Keep the API key on the server. Never send it to the browser. | ||
|
|
||
| ```ts | ||
| import { generateWorld } from '@tanstack/ai' | ||
| import { reactorWorld } from '@tanstack/ai-reactor' | ||
|
|
||
| export async function POST(request: Request) { | ||
| const body = await request.json() | ||
| const prompt = typeof body.prompt === 'string' ? body.prompt : '' | ||
| if (prompt.length === 0) { | ||
| return Response.json({ error: 'prompt is required' }, { status: 400 }) | ||
| } | ||
|
|
||
| const world = await generateWorld({ | ||
| adapter: reactorWorld('visko-orbis-stable'), | ||
| prompt, | ||
| }) | ||
|
|
||
| return Response.json({ | ||
| token: world.token, | ||
| model: world.model, | ||
| prompt: world.prompt, | ||
| expiresAt: world.expiresAt, | ||
| }) | ||
| } | ||
| ``` | ||
|
|
||
| `REACTOR_API_KEY` must be set, or pass `apiKey` in the adapter config. | ||
|
|
||
| ## 2. Connect in the browser | ||
|
|
||
| Install `@reactor-team/js-sdk`. Connect with the token. Then set the prompt and start. | ||
|
|
||
| ```ts | ||
| import { Reactor } from '@reactor-team/js-sdk' | ||
|
|
||
| const video = document.querySelector('video') | ||
| if (!video) { | ||
| throw new Error('Missing video element') | ||
| } | ||
|
|
||
| const world = await fetch('/api/world', { | ||
| method: 'POST', | ||
| headers: { 'Content-Type': 'application/json' }, | ||
| body: JSON.stringify({ | ||
| prompt: | ||
| 'A dramatic coastline of black volcanic cliffs at golden hour, a single unbroken take.', | ||
| }), | ||
| }).then(async (response) => { | ||
| if (!response.ok) { | ||
| throw new Error('World session failed') | ||
| } | ||
| return response.json() | ||
| }) | ||
|
|
||
| const token = typeof world.token === 'string' ? world.token : '' | ||
| const model = typeof world.model === 'string' ? world.model : '' | ||
| const prompt = typeof world.prompt === 'string' ? world.prompt : '' | ||
| if (token.length === 0 || model.length === 0 || prompt.length === 0) { | ||
| throw new Error('World payload is incomplete') | ||
| } | ||
|
|
||
| const reactor = new Reactor({ modelName: model }) | ||
|
|
||
| reactor.on('trackReceived', (name, _track, stream) => { | ||
| if (name !== 'main_video') return | ||
| video.srcObject = stream | ||
| void video.play() | ||
|
coderabbitai[bot] marked this conversation as resolved.
Outdated
|
||
| }) | ||
|
|
||
| await reactor.connect(token) | ||
| await reactor.sendCommand('set_prompt', { prompt }) | ||
| await reactor.sendCommand('start', {}) | ||
| ``` | ||
|
|
||
| The video element now plays a live world. A new `set_prompt` during the run morphs the scene at the next chunk. | ||
|
|
||
| ## Models | ||
|
|
||
| `reactorWorld()` accepts these ids. The result `model` field is the Reactor connect slug. | ||
|
|
||
| | Id | Connect slug | What it does | | ||
| | --- | --- | --- | | ||
| | `visko-orbis-stable` | `reactor/visko-orbis-stable` | Steerable video with realtime audio | | ||
| | `visko-orbis-dynamic` | `reactor/visko-orbis-dynamic` | Same family, live resolution switch | | ||
| | `happy-oyster-adventure` | `reactor/happy-oyster-adventure` | Explorable world, held controls | | ||
| | `lingbot-world-2` | `reactor/lingbot-world-2` | Image-anchored navigable world | | ||
| | `helios` | `reactor/helios` | Interactive realtime video | | ||
|
Contributor
There was a problem hiding this comment. 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win Remove The table states that these IDs are accepted by 🤖 Prompt for AI Agents |
||
|
|
||
| See the [Reactor adapter](../adapters/reactor) for API keys, token scope, and provider options. | ||
|
|
||
| A full app lives in [`examples/ts-react-world`](https://github.com/TanStack/ai/tree/main/examples/ts-react-world). | ||
|
|
||
| ## What you have now | ||
|
|
||
| A server route that mints a world session, and a browser that streams it. Change the prompt while the video plays to steer the world. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift
Denial of Service (CWE-400): Uncontrolled Resource Consumption
Reachability: External
Protect the token-minting route from resource abuse.
If this handler is externally reachable, require authentication, prompt-size limits, rate limits, and concurrent-session limits before calling
generateWorld(). The current validation only checks thatpromptis non-empty.🤖 Prompt for AI Agents