feat(gateway): seedance 2.0 reference video input

apps/docs/content/features/video-generation.mdx

@@ -41,6 +41,8 @@ LLMGateway currently supports a focused subset of the OpenAI video API.
 | `last_frame`       | object  | no       | Optional ending frame when `image` is provided                                                                             |
 | `reference_images` | array   | no       | One to three provider-specific image inputs                                                                                |
 | `input_reference`  | object  | no       | Alias for one or more `reference_images`                                                                                   |
+| `reference_videos` | array   | no       | One to three reference video HTTPS URLs (Seedance 2.0 only, see below)                                                     |
+| `reference_audios` | array   | no       | One to three reference audio HTTPS URLs (Seedance 2.0 only, see below)                                                     |
 | `callback_url`     | string  | no       | LLMGateway extension for completion webhooks                                                                               |
 | `callback_secret`  | string  | no       | LLMGateway extension used to sign webhook deliveries                                                                       |
 
@@ -54,6 +56,90 @@ LLMGateway currently supports a focused subset of the OpenAI video API.
 
 Requests return `400` when the selected provider cannot serve the requested `size` or `seconds`. Seedance derives `aspect_ratio` from the requested `size` (16:9 for landscape, 9:16 for portrait).
 
+### Reference-guided generation (Seedance 2.0)
+
+Seedance 2.0 (`seedance-2-0`, `seedance-2-0-fast`) can generate a video that is guided by reference **images**, **videos**, and **audio** — sometimes called omni-reference. You attach references as top-level fields in the same `POST /v1/videos` payload; the gateway forwards each one to the provider tagged with the correct role, so you don't set roles yourself.
+
+| Reference type | Payload field                                | Count | Accepted input                   | Available on                                         |
+| -------------- | -------------------------------------------- | ----- | -------------------------------- | ---------------------------------------------------- |
+| Image          | `reference_images` (`input_reference` alias) | 1–3   | HTTPS URL **or** base64 data URL | Seedance 2.0, Veo 3.1 (`google-vertex`, `avalanche`) |
+| Video          | `reference_videos`                           | 1–3   | HTTPS URL only                   | Seedance 2.0                                         |
+| Audio          | `reference_audios`                           | 1–3   | HTTPS URL only                   | Seedance 2.0                                         |
+
+Each list item accepts either a bare URL string or an object form:
+
+- `reference_images`: `"https://…/subject.png"` or `{ "image_url": "https://…/subject.png" }`
+- `reference_videos`: `"https://…/motion.mp4"` or `{ "video_url": "https://…/motion.mp4" }`
+- `reference_audios`: `"https://…/track.mp3"` or `{ "audio_url": "https://…/track.mp3" }`
+
+You can mix all three reference types in one request. The `prompt` can be a light instruction (for example `"adapt this to show more detail"`) — the references drive the result.
+
+#### Rules and limits
+
+- **HTTPS only for video and audio.** `reference_videos` and `reference_audios` must be publicly reachable HTTPS URLs (the provider fetches them). base64 data URLs are rejected for video/audio; images may be HTTPS URLs or base64 data URLs.
+- **Reference video resolution.** Seedance requires reference video frames to be at least ~409,600 pixels (roughly 480p or larger). Low-resolution clips such as 360p are rejected with a `400`.
+- **Not combinable with frames.** Reference inputs (`reference_images`, `reference_videos`, `reference_audios`) cannot be combined with the first/last frame inputs (`image`, `last_frame`).
+- **Provider scope.** Reference videos and audio are only supported on Seedance 2.0 models; sending them to other models returns a `400`.
+- **Moderation still applies.** The output is subject to the provider's content moderation. Blocked generations finish as `failed` and are logged with a `content_filter` finish reason.
+
+#### Examples
+
+Reference images only (subjects / style):
+
+```bash
+curl -X POST "https://api.llmgateway.io/v1/videos" \
+  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "seedance-2-0",
+    "prompt": "The subject walks through a neon-lit market at night",
+    "seconds": 5,
+    "size": "1280x720",
+    "reference_images": [
+      { "image_url": "https://example.com/subject.png" },
+      { "image_url": "https://example.com/style.png" }
+    ]
+  }'
+```
+
+Reference video only (motion / scene — let the clip drive the output):
+
+```bash
+curl -X POST "https://api.llmgateway.io/v1/videos" \
+  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "seedance-2-0",
+    "prompt": "adapt this to show more detail",
+    "seconds": 5,
+    "size": "1280x720",
+    "reference_videos": ["https://example.com/reference-motion.mp4"]
+  }'
+```
+
+All three reference types combined:
+
+```bash
+curl -X POST "https://api.llmgateway.io/v1/videos" \
+  -H "Authorization: Bearer $LLM_GATEWAY_API_KEY" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "seedance-2-0",
+    "prompt": "The subject performs the choreography from the reference video",
+    "seconds": 5,
+    "size": "1280x720",
+    "reference_images": [
+      { "image_url": "https://example.com/subject.png" }
+    ],
+    "reference_videos": [
+      "https://example.com/reference-motion.mp4"
+    ],
+    "reference_audios": [
+      "https://example.com/reference-track.mp3"
+    ]
+  }'
+```
+
 ### Not supported yet
 
 - multipart uploads

apps/gateway/src/chat/tools/get-finish-reason-from-error.ts

@@ -1,5 +1,7 @@
 import { hasInvalidProviderCredentialError } from "@/lib/provider-auth-errors.js";
 
+import { isContentFilterErrorText } from "@llmgateway/shared";
+
 /**
  * Determines the appropriate finish reason based on HTTP status code and error message
  * 5xx status codes indicate upstream provider errors
@@ -31,21 +33,10 @@ export function getFinishReasonFromError(
 		return "upstream_error";
 	}
 
-	// Azure OpenAI content filter (ResponsibleAIPolicyViolation)
-	if (errorText?.includes("ResponsibleAIPolicyViolation")) {
-		return "content_filter";
-	}
-
-	// ByteDance / DeepSeek provider moderation block
-	if (errorText?.includes("SensitiveContentDetected")) {
-		return "content_filter";
-	}
-
-	// Alibaba / DashScope moderation block
-	if (
-		errorText?.includes("data_inspection_failed") ||
-		errorText?.includes("Input data may contain inappropriate content")
-	) {
+	// Provider content-moderation / safety blocks (Azure ResponsibleAIPolicyViolation,
+	// ByteDance/DeepSeek SensitiveContentDetected, Alibaba data_inspection_failed,
+	// Azure content management policy, OpenAI safety system rejection, etc.)
+	if (isContentFilterErrorText(errorText)) {
 		return "content_filter";
 	}
 
@@ -57,17 +48,6 @@ export function getFinishReasonFromError(
 		return "content_filter";
 	}
 
-	// Azure OpenAI prompt-side content filter (distinct from ResponsibleAIPolicyViolation,
-	// which fires on the response side and includes inner_error details)
-	if (errorText?.includes("Microsoft's content management policy")) {
-		return "content_filter";
-	}
-
-	// OpenAI safety system rejection (e.g. gpt-image-2 image generation)
-	if (errorText?.includes("Your request was rejected by the safety system")) {
-		return "content_filter";
-	}
-
 	// 401/403 and known provider credential payloads indicate bad provider keys.
 	if (
 		statusCode === 401 ||

apps/gateway/src/videos/videos.spec.ts

@@ -171,6 +171,201 @@ describe("videos", () => {
 		);
 	});
 
+	test("/v1/videos rejects non-https reference videos", async () => {
+		await db.insert(tables.apiKey).values({
+			id: "token-id",
+			token: "real-token",
+			projectId: "project-id",
+			description: "Test API Key",
+			createdBy: "user-id",
+		});
+
+		const res = await app.request("/v1/videos", {
+			method: "POST",
+			headers: {
+				"Content-Type": "application/json",
+				Authorization: "Bearer real-token",
+			},
+			body: JSON.stringify({
+				model: "seedance-2-0",
+				prompt: "Reproduce the camera move from the reference clip",
+				size: "1280x720",
+				seconds: 5,
+				reference_videos: ["http://example.com/reference-motion.mp4"],
+			}),
+		});
+
+		expect(res.status).toBe(400);
+		const json = await res.json();
+		expect(JSON.stringify(json)).toContain("reference_videos");
+	});
+
+	test("/v1/videos rejects combining frames with reference videos", async () => {
+		await db.insert(tables.apiKey).values({
+			id: "token-id",
+			token: "real-token",
+			projectId: "project-id",
+			description: "Test API Key",
+			createdBy: "user-id",
+		});
+
+		const res = await app.request("/v1/videos", {
+			method: "POST",
+			headers: {
+				"Content-Type": "application/json",
+				Authorization: "Bearer real-token",
+			},
+			body: JSON.stringify({
+				model: "seedance-2-0",
+				prompt: "Blend these inputs",
+				size: "1280x720",
+				seconds: 5,
+				image: { image_url: "data:image/png;base64,aGVsbG8=" },
+				reference_videos: ["https://example.com/reference-motion.mp4"],
+			}),
+		});
+
+		expect(res.status).toBe(400);
+		const json = await res.json();
+		expect(JSON.stringify(json)).toContain("Frame inputs");
+	});
+
+	test("/v1/videos rejects reference videos on non-bytedance models", async () => {
+		await db.insert(tables.apiKey).values({
+			id: "token-id",
+			token: "real-token",
+			projectId: "project-id",
+			description: "Test API Key",
+			createdBy: "user-id",
+		});
+
+		await db.insert(tables.providerKey).values({
+			id: "provider-key-id",
+			token: "sk-test-key",
+			provider: "avalanche",
+			organizationId: "org-id",
+			baseUrl: mockServerUrl,
+		});
+
+		const res = await app.request("/v1/videos", {
+			method: "POST",
+			headers: {
+				"Content-Type": "application/json",
+				Authorization: "Bearer real-token",
+			},
+			body: JSON.stringify({
+				model: "avalanche/veo-3.1-fast-generate-preview",
+				prompt: "Reproduce this motion",
+				size: "1920x1080",
+				seconds: 8,
+				reference_videos: ["https://example.com/reference-motion.mp4"],
+			}),
+		});
+
+		expect(res.status).toBe(400);
+		const json = await res.json();
+		expect(JSON.stringify(json)).toContain(
+			"reference videos are currently only supported on bytedance",
+		);
+	});
+
+	test("/v1/videos rejects non-https reference audios", async () => {
+		await db.insert(tables.apiKey).values({
+			id: "token-id",
+			token: "real-token",
+			projectId: "project-id",
+			description: "Test API Key",
+			createdBy: "user-id",
+		});
+
+		const res = await app.request("/v1/videos", {
+			method: "POST",
+			headers: {
+				"Content-Type": "application/json",
+				Authorization: "Bearer real-token",
+			},
+			body: JSON.stringify({
+				model: "seedance-2-0",
+				prompt: "Align the motion to the reference track",
+				size: "1280x720",
+				seconds: 5,
+				reference_audios: ["http://example.com/reference-track.mp3"],
+			}),
+		});
+
+		expect(res.status).toBe(400);
+		const json = await res.json();
+		expect(JSON.stringify(json)).toContain("reference_audios");
+	});
+
+	test("/v1/videos rejects reference audio on non-bytedance models", async () => {
+		await db.insert(tables.apiKey).values({
+			id: "token-id",
+			token: "real-token",
+			projectId: "project-id",
+			description: "Test API Key",
+			createdBy: "user-id",
+		});
+
+		await db.insert(tables.providerKey).values({
+			id: "provider-key-id",
+			token: "sk-test-key",
+			provider: "avalanche",
+			organizationId: "org-id",
+			baseUrl: mockServerUrl,
+		});
+
+		const res = await app.request("/v1/videos", {
+			method: "POST",
+			headers: {
+				"Content-Type": "application/json",
+				Authorization: "Bearer real-token",
+			},
+			body: JSON.stringify({
+				model: "avalanche/veo-3.1-fast-generate-preview",
+				prompt: "Align this motion",
+				size: "1920x1080",
+				seconds: 8,
+				reference_audios: ["https://example.com/reference-track.mp3"],
+			}),
+		});
+
+		expect(res.status).toBe(400);
+		const json = await res.json();
+		expect(JSON.stringify(json)).toContain(
+			"reference audio is currently only supported on bytedance",
+		);
+	});
+
+	test("/v1/videos restricts reference inputs to Seedance 2.0 models", async () => {
+		await db.insert(tables.apiKey).values({
+			id: "token-id",
+			token: "real-token",
+			projectId: "project-id",
+			description: "Test API Key",
+			createdBy: "user-id",
+		});
+
+		const res = await app.request("/v1/videos", {
+			method: "POST",
+			headers: {
+				"Content-Type": "application/json",
+				Authorization: "Bearer real-token",
+			},
+			body: JSON.stringify({
+				model: "bytedance/seedance-1-5-pro",
+				prompt: "Reproduce this motion",
+				size: "1280x720",
+				seconds: 5,
+				reference_videos: ["https://example.com/reference-motion.mp4"],
+			}),
+		});
+
+		expect(res.status).toBe(400);
+		const json = await res.json();
+		expect(JSON.stringify(json)).toContain("Seedance 2.0");
+	});
+
 	test("/v1/videos uses routing metrics to pick the best eligible provider", async () => {
 		const originalGoogleCloudProject = process.env.LLM_GOOGLE_CLOUD_PROJECT;
 		const originalRuntimeGoogleCloudProject = process.env.GOOGLE_CLOUD_PROJECT;