Repository navigation
feat(gateway): seedance 2.0 reference video input - #2538
Merged
Merged
Conversation
Add reference_videos (HTTPS URL) input and enable reference image + video generation for ByteDance Seedance 2.0 (seedance-2-0, seedance-2-0-fast) via the omni-reference content roles (first_frame, reference_image, reference_video). Reference videos are restricted to Seedance 2.0 and cannot be combined with frame inputs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
Warning Review limit reached
More reviews will be available in 3 minutes and 40 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (6)
WalkthroughAdds HTTPS-only ChangesSeedance 2.0 omni-reference (video & audio)
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Adds first-class support in the gateway for ByteDance Seedance 2.0 “omni-reference” video generation by introducing reference_videos (HTTPS URLs) and wiring those reference inputs through request validation, provider eligibility checks, and the ByteDance upstream request payload shape.
Changes:
- Added
reference_videosrequest field (1–3 HTTPS URLs) and integrated it into request-mode detection/validation. - Updated provider-constraint evaluation to (a) allow reference inputs for Seedance 2.0 while (b) rejecting reference videos on non-Seedance 2.0 providers/models with 400s.
- Extended the ByteDance upstream request builder to emit ModelArk
contententries with the expectedrolevalues; added unit tests and docs.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| apps/gateway/src/videos/videos.ts | Adds zod schema + routing/constraint plumbing for reference_videos, and sends reference videos/images to ByteDance with correct content roles. |
| apps/gateway/src/videos/videos.spec.ts | Adds unit tests covering reference_videos validation and model/provider constraints. |
| apps/docs/content/features/video-generation.mdx | Documents reference_videos and Seedance 2.0 omni-reference usage. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
294
to
299
| value.image !== undefined || value.last_frame !== undefined; | ||
| const hasReferenceInput = | ||
| value.reference_images !== undefined || | ||
| value.input_reference !== undefined; | ||
| value.input_reference !== undefined || | ||
| value.reference_videos !== undefined; | ||
|
|
Comment on lines
+855
to
863
| if (referenceVideoCount > 0) { | ||
| reasons.push( | ||
| "reference videos are currently only supported on bytedance Seedance 2.0 models", | ||
| ); | ||
| } | ||
|
|
||
| if (isGoogleVertexVideoProvider(provider.providerId)) { | ||
| if (provider.externalId !== "veo-3.1-generate-001") { | ||
| reasons.push( |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
apps/gateway/src/videos/videos.spec.ts (1)
272-299: ⚡ Quick winAdd ByteDance provider key setup for consistency.
This test validates a ByteDance provider-specific constraint (Seedance 2.0 model restriction). For consistency with test 3 (lines 233–270), which sets up an Avalanche provider key to test provider-specific constraints, this test should also set up a ByteDance provider key to ensure the constraint validation logic is properly reached.
🔧 Suggested provider key setup
Add after line 279:
}); + await db.insert(tables.providerKey).values({ + id: "provider-key-id", + token: "sk-test-key", + provider: "bytedance", + organizationId: "org-id", + baseUrl: mockServerUrl, + }); + const res = await app.request("/v1/videos", {🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/gateway/src/videos/videos.spec.ts` around lines 272 - 299, The test "test(\"/v1/videos restricts reference inputs to Seedance 2.0 models\")" is missing a ByteDance provider key setup; add a db.insert into tables.providerKey (or the same provider-key table used in the Avalanche test) immediately after inserting the API key so the provider-specific validation runs — create a provider key record referencing the API key (e.g., provider: "bytedance", apiKeyId: "token-id", and include the model/provider details used by the validation such as "bytedance/seedance-2-0" or the equivalent fields your schema expects).
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@apps/gateway/src/videos/videos.spec.ts`:
- Around line 272-299: The test "test(\"/v1/videos restricts reference inputs to
Seedance 2.0 models\")" is missing a ByteDance provider key setup; add a
db.insert into tables.providerKey (or the same provider-key table used in the
Avalanche test) immediately after inserting the API key so the provider-specific
validation runs — create a provider key record referencing the API key (e.g.,
provider: "bytedance", apiKeyId: "token-id", and include the model/provider
details used by the validation such as "bytedance/seedance-2-0" or the
equivalent fields your schema expects).
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 9b2a51b3-90d4-4fb5-aee2-2d613d00cb28
📒 Files selected for processing (3)
apps/docs/content/features/video-generation.mdxapps/gateway/src/videos/videos.spec.tsapps/gateway/src/videos/videos.ts
Enable reference image + video inputs for Seedance 2.0 in the video playground and add an HTTPS reference-video URL control wired to the reference_videos request field. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Add reference_audios (HTTPS URL) input for ByteDance Seedance 2.0 omni-reference, sending audio_url content items with role reference_audio. Mirrors reference_videos across the gateway, docs, and playground (with a reference-audio URL control). Reference audio is restricted to Seedance 2.0 and cannot be combined with frame inputs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
apps/gateway/src/videos/videos.ts (1)
357-362:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winUpdate the frame/reference conflict error message to cover all reference inputs.
The rule now blocks
reference_videosandreference_audiostoo, but the message still says “reference image inputs,” which is misleading for clients.💡 Suggested patch
- "Frame inputs (image/last_frame) cannot be combined with reference image inputs", + "Frame inputs (image/last_frame) cannot be combined with reference inputs (reference_images/input_reference/reference_videos/reference_audios)",🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/gateway/src/videos/videos.ts` around lines 357 - 362, The error message in the validation branch that triggers when hasFrameInput && hasReferenceInput is misleading because it only mentions "reference image inputs"; update the ctx.addIssue message (in the block using hasFrameInput, hasReferenceInput and ctx.addIssue) to refer to all reference inputs—e.g., "Frame inputs (image/last_frame) cannot be combined with reference inputs (reference_images, reference_videos, reference_audios)"—so clients understand the rule applies to videos and audios as well.
🧹 Nitpick comments (1)
apps/gateway/src/videos/videos.spec.ts (1)
301-338: ⚡ Quick winAdd a Seedance 2.0 gating test for
reference_audios(ByteDance non-2.0).You cover HTTPS and non-ByteDance rejection, but there’s no explicit regression test that rejects
reference_audiosonbytedance/seedance-1-5-pro.✅ Suggested test addition
+ test("/v1/videos restricts reference audio inputs to Seedance 2.0 models", async () => { + await db.insert(tables.apiKey).values({ + id: "token-id", + token: "real-token", + projectId: "project-id", + description: "Test API Key", + createdBy: "user-id", + }); + + const res = await app.request("/v1/videos", { + method: "POST", + headers: { + "Content-Type": "application/json", + Authorization: "Bearer real-token", + }, + body: JSON.stringify({ + model: "bytedance/seedance-1-5-pro", + prompt: "Align this motion to the reference track", + size: "1280x720", + seconds: 5, + reference_audios: ["https://example.com/reference-track.mp3"], + }), + }); + + expect(res.status).toBe(400); + const json = await res.json(); + expect(JSON.stringify(json)).toContain("Seedance 2.0"); + });🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/gateway/src/videos/videos.spec.ts` around lines 301 - 338, Add a gating test in videos.spec.ts that posts to "/v1/videos" with Authorization "Bearer real-token" and a body including model "bytedance/seedance-1-5-pro" (or the exact non-2.0 ByteDance model string used elsewhere), seconds/size/prompt and reference_audios set to an HTTPS URL, then assert the response status is 400 and the JSON contains the same rejection message ("reference audio is currently only supported on bytedance" or the exact error string used by the handler). Reuse the existing setup pattern (inserting apiKey and providerKey) and the same request flow used in the test "/v1/videos rejects reference audio on non-bytedance models" to ensure consistent fixture usage.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/playground/src/components/playground/video-controls.tsx`:
- Around line 108-139: The addReferenceVideo and addReferenceAudio handlers
currently validate only an https:// prefix; update both to reliably validate a
real HTTPS URL by attempting to construct a URL object from referenceVideoDraft
/ referenceAudioDraft (in a try/catch) and ensuring url.protocol === 'https:'
before proceeding; keep existing behavior of trimming, resetting frame inputs
via setFrameInputs, and updating reference lists via setReferenceVideos /
setReferenceAudios, and clear the drafts only when the URL is valid.
- Around line 108-139: When adding a reference URL in addReferenceVideo and
addReferenceAudio, also flip the upload mode to reference so subsequent
pasted/dropped images are treated as references: inside addReferenceVideo and
addReferenceAudio (after validating url and before clearing
referenceVideoDraft/referenceAudioDraft), call setUploadTarget("reference") in
addition to setFrameInputs({ start: null, end: null }) and updating
setReferenceVideos/setReferenceAudios so the UI stays in reference mode; ensure
you only invoke this when a URL is accepted (i.e., the same place you push the
URL into setReferenceVideos/setReferenceAudios).
---
Outside diff comments:
In `@apps/gateway/src/videos/videos.ts`:
- Around line 357-362: The error message in the validation branch that triggers
when hasFrameInput && hasReferenceInput is misleading because it only mentions
"reference image inputs"; update the ctx.addIssue message (in the block using
hasFrameInput, hasReferenceInput and ctx.addIssue) to refer to all reference
inputs—e.g., "Frame inputs (image/last_frame) cannot be combined with reference
inputs (reference_images, reference_videos, reference_audios)"—so clients
understand the rule applies to videos and audios as well.
---
Nitpick comments:
In `@apps/gateway/src/videos/videos.spec.ts`:
- Around line 301-338: Add a gating test in videos.spec.ts that posts to
"/v1/videos" with Authorization "Bearer real-token" and a body including model
"bytedance/seedance-1-5-pro" (or the exact non-2.0 ByteDance model string used
elsewhere), seconds/size/prompt and reference_audios set to an HTTPS URL, then
assert the response status is 400 and the JSON contains the same rejection
message ("reference audio is currently only supported on bytedance" or the exact
error string used by the handler). Reuse the existing setup pattern (inserting
apiKey and providerKey) and the same request flow used in the test "/v1/videos
rejects reference audio on non-bytedance models" to ensure consistent fixture
usage.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 618b2919-6ee7-4c63-bae5-2a1b21eaa757
📒 Files selected for processing (7)
apps/docs/content/features/video-generation.mdxapps/gateway/src/videos/videos.spec.tsapps/gateway/src/videos/videos.tsapps/playground/src/components/playground/video-controls.tsxapps/playground/src/components/playground/video-page-client.tsxapps/playground/src/lib/video-gen.spec.tsapps/playground/src/lib/video-gen.ts
Comment on lines
+108
to
+139
| const addReferenceVideo = useCallback(() => { | ||
| const url = referenceVideoDraft.trim(); | ||
| if (!url || !/^https:\/\//i.test(url)) { | ||
| return; | ||
| } | ||
| setFrameInputs({ start: null, end: null }); | ||
| setReferenceVideos((prev) => | ||
| prev.length >= 3 || prev.includes(url) ? prev : [...prev, url], | ||
| ); | ||
| setReferenceVideoDraft(""); | ||
| }, [referenceVideoDraft, setFrameInputs, setReferenceVideos]); | ||
|
|
||
| const removeReferenceVideo = useCallback( | ||
| (index: number) => { | ||
| setReferenceVideos((prev) => prev.filter((_, i) => i !== index)); | ||
| }, | ||
| [setReferenceVideos], | ||
| ); | ||
|
|
||
| const [referenceAudioDraft, setReferenceAudioDraft] = useState(""); | ||
|
|
||
| const addReferenceAudio = useCallback(() => { | ||
| const url = referenceAudioDraft.trim(); | ||
| if (!url || !/^https:\/\//i.test(url)) { | ||
| return; | ||
| } | ||
| setFrameInputs({ start: null, end: null }); | ||
| setReferenceAudios((prev) => | ||
| prev.length >= 3 || prev.includes(url) ? prev : [...prev, url], | ||
| ); | ||
| setReferenceAudioDraft(""); | ||
| }, [referenceAudioDraft, setFrameInputs, setReferenceAudios]); |
Contributor
There was a problem hiding this comment.
Validate an actual HTTPS URL, not just an https:// prefix.
The current regex accepts strings that are not valid URLs, so the UI can store values the gateway will always reject. The server contract is “valid URL” and HTTPS, not just a matching prefix.
Suggested fix
+function isValidHttpsUrl(value: string): boolean {
+ try {
+ const parsed = new URL(value);
+ return parsed.protocol === "https:";
+ } catch {
+ return false;
+ }
+}
+
const addReferenceVideo = useCallback(() => {
const url = referenceVideoDraft.trim();
- if (!url || !/^https:\/\//i.test(url)) {
+ if (!url || !isValidHttpsUrl(url)) {
return;
}
setFrameInputs({ start: null, end: null });
setReferenceVideos((prev) =>
prev.length >= 3 || prev.includes(url) ? prev : [...prev, url],
@@
const addReferenceAudio = useCallback(() => {
const url = referenceAudioDraft.trim();
- if (!url || !/^https:\/\//i.test(url)) {
+ if (!url || !isValidHttpsUrl(url)) {
return;
}
setFrameInputs({ start: null, end: null });
setReferenceAudios((prev) =>
prev.length >= 3 || prev.includes(url) ? prev : [...prev, url],
@@
<Button
variant="outline"
size="sm"
onClick={addReferenceVideo}
disabled={
isGenerating ||
referenceVideos.length >= 3 ||
- !/^https:\/\//i.test(referenceVideoDraft.trim())
+ !isValidHttpsUrl(referenceVideoDraft.trim())
}
@@
<Button
variant="outline"
size="sm"
onClick={addReferenceAudio}
disabled={
isGenerating ||
referenceAudios.length >= 3 ||
- !/^https:\/\//i.test(referenceAudioDraft.trim())
+ !isValidHttpsUrl(referenceAudioDraft.trim())
}Also applies to: 549-603
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/playground/src/components/playground/video-controls.tsx` around lines
108 - 139, The addReferenceVideo and addReferenceAudio handlers currently
validate only an https:// prefix; update both to reliably validate a real HTTPS
URL by attempting to construct a URL object from referenceVideoDraft /
referenceAudioDraft (in a try/catch) and ensuring url.protocol === 'https:'
before proceeding; keep existing behavior of trimming, resetting frame inputs
via setFrameInputs, and updating reference lists via setReferenceVideos /
setReferenceAudios, and clear the drafts only when the URL is valid.
Keep the control in reference mode after a URL ref is added.
Adding a reference video/audio clears frames, but it never flips uploadTarget to "reference". The next pasted or dropped image still follows the old frame target, so it gets treated as a frame input and wipes the just-added URL refs instead of combining with them.
Suggested fix
const addReferenceVideo = useCallback(() => {
const url = referenceVideoDraft.trim();
if (!url || !/^https:\/\//i.test(url)) {
return;
}
+ setUploadTarget("reference");
setFrameInputs({ start: null, end: null });
setReferenceVideos((prev) =>
prev.length >= 3 || prev.includes(url) ? prev : [...prev, url],
);
setReferenceVideoDraft("");
}, [referenceVideoDraft, setFrameInputs, setReferenceVideos]);
const addReferenceAudio = useCallback(() => {
const url = referenceAudioDraft.trim();
if (!url || !/^https:\/\//i.test(url)) {
return;
}
+ setUploadTarget("reference");
setFrameInputs({ start: null, end: null });
setReferenceAudios((prev) =>
prev.length >= 3 || prev.includes(url) ? prev : [...prev, url],
);
setReferenceAudioDraft("");
}, [referenceAudioDraft, setFrameInputs, setReferenceAudios]);🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/playground/src/components/playground/video-controls.tsx` around lines
108 - 139, When adding a reference URL in addReferenceVideo and
addReferenceAudio, also flip the upload mode to reference so subsequent
pasted/dropped images are treated as references: inside addReferenceVideo and
addReferenceAudio (after validating url and before clearing
referenceVideoDraft/referenceAudioDraft), call setUploadTarget("reference") in
addition to setFrameInputs({ start: null, end: null }) and updating
setReferenceVideos/setReferenceAudios so the UI stays in reference mode; ensure
you only invoke this when a URL is accepted (i.e., the same place you push the
URL into setReferenceVideos/setReferenceAudios).
Failed video jobs were always logged as upstream_error. ByteDance content-moderation blocks (e.g. OutputVideoSensitiveContentDetected on Seedance 2.0) are now classified as content_filter. Extracts the content-filter signal detection into @llmgateway/shared so the gateway's getFinishReasonFromError and the worker video-job logger share one source. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Expand the video docs with a reference-guided generation guide: a reference-type table (field, count, accepted input, provider scope), string vs object item forms, rules/limits (HTTPS-only video/audio, min reference-video resolution, frame exclusivity, moderation), and per-type curl examples. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Nice to see. |
This was referenced Jun 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds support for reference-guided video generation on ByteDance Seedance 2.0 (
seedance-2-0,seedance-2-0-fast) — the model's "omni / all-round reference" feature.reference_videos: 1–3 reference video HTTPS URLs (base64 data URLs are rejected for videos; URLs are passed straight through to ByteDance per the chosen design).reference_images/input_reference) for Seedance 2.0, which was previously blocked for thebytedanceprovider./contents/generations/taskscontentarray with the correct roles:first_frame,reference_image, andreference_video(type: "video_url").image/last_frameframe inputs.400.Validation / constraints
reference_videosmust be HTTPS (zod refine).400(existing rule extended toreference_videos).400("reference videos are currently only supported on bytedance Seedance 2.0 models").400.Tests
Added 4 unit tests in
videos.spec.tscovering the validation/constraint paths above (no upstream mock required). Fullvideos.spec.tssuite passes (26 tests).Docs
Updated
video-generation.mdxwith thereference_videosfield and a Seedance 2.0 omni-reference section with an example request.Notes
The official BytePlus ModelArk docs are a JS-rendered SPA that couldn't be scraped directly; model IDs (
dreamina-seedance-2-0-260128,dreamina-seedance-2-0-fast-260128), the endpoint, and thecontent-arrayroleshape were cross-checked against the IDs already present inpackages/modelsand ByteDance developer references.🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation
UI
Tests