feat: video operations in runware

[TejasGhatte]

Summary

Implements video generation, retrieval, and download for the Runware provider, replacing the previous stub implementations that returned "unsupported operation" errors. Also consolidates the separate RunwareImageInferenceRequest and RunwareImageResult types into unified RunwareInferenceRequest and RunwareResult types that cover both image and video modalities.

Changes

• Replaced RunwareImageInferenceRequest with RunwareInferenceRequest, a single struct covering image and video task fields. PositivePrompt, Width, and Height are now pointer fields to support optional omission (required for video tasks where some fields may not apply).
• Replaced RunwareImageResult with RunwareResult, adding video-specific fields (VideoUUID, VideoURL, Status).
• Added taskTypeVideoInference and taskTypeGetResponse task type constants, and a deliveryMethodAsync constant used to queue video tasks asynchronously.
• Extracted a sendTaskArray helper that wraps a single task in the Runware array envelope, posts it, and returns the decoded response — eliminating duplicated HTTP logic across image and video operations.
• Added videos.go with ToRunwareVideoGenerationRequest (text-to-video and image-to-video via frameImages) and ToBifrostVideoGenerationResponse (maps Runware status strings to Bifrost VideoStatus values).
• VideoGeneration submits a videoInference task asynchronously and returns the queued job ID.
• VideoRetrieve polls a previously submitted task using a getResponse task type.
• VideoDownload calls VideoRetrieve to confirm completion, then fetches the video bytes from the returned URL.
• Added default video dimensions (1920x1080) separate from the image defaults (1024x1024), with a comment explaining why square dimensions are rejected by video models.
• Updated the test config to use runware:102@1 (FLUX Fill) for image editing and added a klingai:kling-video@3-pro video model entry with video scenarios disabled by default.

Type of change

• Bug fix
• Feature
• Refactor
• Documentation
• Chore/CI

Affected areas

• Core (Go)
• Transports (HTTP)
• Providers/Integrations
• Plugins
• UI (React)
• Docs

How to test

Enable the video scenarios in runware_test.go by setting VideoGeneration, VideoRetrieve, and VideoDownload to true, then run with a valid Runware API key:

go test ./core/providers/runware/... -v -run TestRunware

Expected outcomes:

• VideoGeneration returns a response with a non-empty ID and a queued/in-progress status.
• VideoRetrieve using the returned ID returns the task status (poll until completed).
• VideoDownload returns raw video bytes with a video/mp4 content type once the task is complete.

Image generation and edit tests remain enabled and should continue to pass unchanged.

Breaking changes

• Yes
• No

RunwareImageInferenceRequest and RunwareImageResult have been removed and replaced by RunwareInferenceRequest and RunwareResult. Any code outside this package referencing those types directly will need to be updated to use the new names.

Related issues

Security considerations

No new auth mechanisms are introduced. The existing Bearer token flow is reused for all video endpoints. Video bytes are fetched via a direct GET to the Runware-provided URL without forwarding user credentials.

Checklist

• I read docs/contributing/README.md and followed the guidelines
• I added/updated tests where appropriate
• I updated documentation where needed
• I verified builds succeed (Go and UI)
• I verified the CI pipeline passes locally if applicable

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 3ae28c0f-517b-4c28-b178-d7fceeed6164

📥 Commits

Reviewing files that changed from the base of the PR and between 2697006 and a6be983.

📒 Files selected for processing (6)

• core/providers/runware/images.go
• core/providers/runware/runware.go
• core/providers/runware/runware_test.go
• core/providers/runware/types.go
• core/providers/runware/utils.go
• core/providers/runware/videos.go

---

📝 Walkthrough

Summary by CodeRabbit

• New Features
  • Enabled full end-to-end Runware video support: video generation, async status polling, and MP4 download (instead of previously unsupported operations).
  • Added support for video reference images (image-to-video) and improved request handling for prompts, negative prompts, seeds, and optional duration.
  • Video generation now uses default 16:9 (1080p) dimensions when no size is specified.
• Bug Fixes
  • Improved conversion of provider task results into consistent Bifrost video statuses and error reporting during retrieval and download.

Walkthrough

The Runware provider's image-specific types (RunwareImageInferenceRequest, RunwareImageResult) are replaced with a unified RunwareInferenceRequest and RunwareResult supporting both image and video modalities. A new videos.go adds request/response conversion helpers, and runware.go implements VideoGeneration, VideoRetrieve, and VideoDownload using a new sendTaskArray HTTP helper.

Changes

Runware Video Support

Layer / File(s)	Summary
Unified inference types and constants core/providers/runware/types.go	Replaces RunwareImageInferenceRequest/RunwareImageResult with RunwareInferenceRequest (pointer fields, video-only fields), RunwareGetResponseRequest for async polling, unified RunwareResult covering image and video outputs, new task-type and async delivery constants, and RunwareFrameImage for frame anchoring.
Image requests adapted to unified types core/providers/runware/images.go, core/providers/runware/utils.go	ToRunwareImageGenerationRequest and ToRunwareImageEditRequest updated to return *RunwareInferenceRequest with pointer-based PositivePrompt/Width/Height. Default video dimension constants (defaultRunwareVideoWidth 1920, defaultRunwareVideoHeight 1080) added to utils.go.
Video request/response conversion helpers core/providers/runware/videos.go	New file adds ToRunwareVideoGenerationRequest (validates input, sets async delivery, maps prompt/reference image/size/duration/extra params to RunwareInferenceRequest) and ToBifrostVideoGenerationResponse (maps Runware statuses to Bifrost video states, emits MP4 URL output).
Provider VideoGeneration, VideoRetrieve, VideoDownload core/providers/runware/runware.go	Implements all three video provider methods: sendTaskArray helper wraps a task in the Runware array envelope and POSTs; VideoGeneration submits a videoInference task and converts the result; VideoRetrieve polls via getResponse; VideoDownload calls VideoRetrieve, validates completion, and downloads bytes via HTTP GET. firstVideoResult surfaces task-level errors.
Test config for video scenarios core/providers/runware/runware_test.go	ComprehensiveTestConfig updated: ImageEditModel changed to runware:102@1, VideoGenerationModel field added, and video scenarios (VideoGeneration, VideoRetrieve, VideoDownload) explicitly set to false.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant BifrostProvider as Runware Provider
  participant sendTaskArray
  participant RunwareAPI as Runware API

  rect rgba(70, 130, 180, 0.5)
    Note over Client,RunwareAPI: VideoGeneration
    Client->>BifrostProvider: VideoGeneration(BifrostVideoGenerationRequest)
    BifrostProvider->>sendTaskArray: videoInference task array
    sendTaskArray->>RunwareAPI: POST /inference [taskType=videoInference, async]
    RunwareAPI-->>sendTaskArray: RunwareResponse{Data[0]: taskUUID, status}
    sendTaskArray-->>BifrostProvider: decoded response + latency
    BifrostProvider->>BifrostProvider: firstVideoResult → ToBifrostVideoGenerationResponse
    BifrostProvider-->>Client: BifrostVideoGenerationResponse (taskUUID+suffix, status=queued)
  end

  rect rgba(34, 139, 34, 0.5)
    Note over Client,RunwareAPI: VideoRetrieve (polling)
    Client->>BifrostProvider: VideoRetrieve(taskUUID+suffix)
    BifrostProvider->>sendTaskArray: getResponse task array (stripped UUID)
    sendTaskArray->>RunwareAPI: POST /inference [taskType=getResponse]
    RunwareAPI-->>sendTaskArray: RunwareResponse{Data[0]: videoURL, status}
    sendTaskArray-->>BifrostProvider: decoded response
    BifrostProvider-->>Client: BifrostVideoGenerationResponse (status=completed, videoURL)
  end

  rect rgba(178, 34, 34, 0.5)
    Note over Client,RunwareAPI: VideoDownload
    Client->>BifrostProvider: VideoDownload(taskUUID)
    BifrostProvider->>BifrostProvider: VideoRetrieve → validate completed + videoURL
    BifrostProvider->>RunwareAPI: HTTP GET videoURL
    RunwareAPI-->>BifrostProvider: video bytes + Content-Type
    BifrostProvider-->>Client: BifrostVideoDownloadResponse (bytes, content type)
  end

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Possibly related PRs

• maximhq/bifrost#4601: Both PRs modify the Runware provider's request/response modeling, with earlier PR establishing image-specific infrastructure that this PR extends to unified inference types for both image and video modalities.

Suggested reviewers

• akshaydeo
• danpiths

Poem

🐇 Hop hop, the pixels now move and flow,
From still images to videos aglow!
A unified type, both image and reel,
videoInference tasks — what a deal!
The rabbit queues tasks, then polls with glee,
Downloads the bytes — watch the magic, see! 🎬

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'feat: video operations in runware' clearly and concisely describes the main feature addition—video functionality for the Runware provider.
Description check	✅ Passed	The pull request description is comprehensive and well-structured. It includes a clear summary, detailed changes, type of change selection, affected areas, testing instructions, breaking changes acknowledgment, security considerations, and a completed checklist.
Docstring Coverage	✅ Passed	Docstring coverage is 83.33% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch 06-22-feat_video_operations_in_runware

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.12.2)

level=error msg="[linters_context] typechecking error: pattern ./...: directory prefix . does not contain main module or its selected dependencies"

---

_{Comment @coderabbitai help to get the list of available commands.}

[TejasGhatte]

• feat: video operations in runware #4615 👈 (View in Graphite)
• feat: runware provider support #4601
• feat: runway image operations #4503
• dev

This stack of pull requests is managed by Graphite. Learn more about stacking.

[greptile-apps]

Confidence Score: 2/5

Not safe to merge — the runware package does not compile due to invalid new(value) call sites in videos.go that were flagged in a prior review and remain unfixed.

The entire runware package fails to build because videos.go calls new() with constants and runtime string values instead of types (e.g., new(deliveryMethodAsync), new(defaultRunwareVideoWidth), new("first"), new(result.VideoURL)). These were called out in a previous review round and are still present in the head commit. Until these are corrected, neither the new video paths nor the existing image-generation paths can be exercised.

core/providers/runware/videos.go — all five new(value) call sites must be replaced with local-variable + address-of patterns before the package can compile.

Important Files Changed

Filename	Overview
core/providers/runware/videos.go	New video converter functions — contains several new(value) call sites that use constants/values instead of types, causing compile errors in the entire package (flagged in a previous review); also hardcodes ContentType: "video/mp4" regardless of the requested OutputFormat
core/providers/runware/runware.go	Adds VideoGeneration, VideoRetrieve, VideoDownload and the sendTaskArray helper; HTTP acquire/release and wait() defer ordering are correct, but handleImageInference is not yet refactored to use sendTaskArray (duplicate logic)
core/providers/runware/types.go	Replaces the split image/result types with unified RunwareInferenceRequest and RunwareResult; new constants and RunwareFrameImage struct look correct
core/providers/runware/images.go	Migrated to pointer-based Width/Height on the unified RunwareInferenceRequest; local variable intermediary pattern for pointer addresses is correct
core/providers/runware/utils.go	Adds defaultRunwareVideoWidth/Height constants; comment correctly describes 1080p 16:9 defaults
core/providers/runware/runware_test.go	Updates image edit model to FLUX Fill and adds video model entry; video scenarios are disabled by default and require a valid API key to run

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Caller
    participant Bifrost
    participant sendTaskArray
    participant RunwareAPI
    participant RunwareCDN

    Caller->>Bifrost: VideoGeneration(req)
    Bifrost->>Bifrost: ToRunwareVideoGenerationRequest()
    Bifrost->>sendTaskArray: [videoInference task]
    sendTaskArray->>RunwareAPI: "POST /v1 [{taskType:videoInference, deliveryMethod:async}]"
    RunwareAPI-->>sendTaskArray: "{data:[{taskUUID, status:queued}]}"
    sendTaskArray-->>Bifrost: taskUUID, status
    Bifrost-->>Caller: "BifrostVideoGenerationResponse{ID, status=queued}"

    Caller->>Bifrost: VideoRetrieve(ID)
    Bifrost->>sendTaskArray: [getResponse task]
    sendTaskArray->>RunwareAPI: "POST /v1 [{taskType:getResponse, taskUUID}]"
    RunwareAPI-->>sendTaskArray: "{data:[{taskUUID, videoURL, status:success}]}"
    sendTaskArray-->>Bifrost: result with videoURL
    Bifrost-->>Caller: "BifrostVideoGenerationResponse{ID, status=completed, Videos[URL]}"

    Caller->>Bifrost: VideoDownload(ID)
    Bifrost->>Bifrost: VideoRetrieve(ID) [internal]
    Bifrost->>RunwareCDN: GET videoURL (no auth header)
    RunwareCDN-->>Bifrost: video bytes + Content-Type
    Bifrost-->>Caller: "BifrostVideoDownloadResponse{Content, ContentType}"

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Caller
    participant Bifrost
    participant sendTaskArray
    participant RunwareAPI
    participant RunwareCDN

    Caller->>Bifrost: VideoGeneration(req)
    Bifrost->>Bifrost: ToRunwareVideoGenerationRequest()
    Bifrost->>sendTaskArray: [videoInference task]
    sendTaskArray->>RunwareAPI: "POST /v1 [{taskType:videoInference, deliveryMethod:async}]"
    RunwareAPI-->>sendTaskArray: "{data:[{taskUUID, status:queued}]}"
    sendTaskArray-->>Bifrost: taskUUID, status
    Bifrost-->>Caller: "BifrostVideoGenerationResponse{ID, status=queued}"

    Caller->>Bifrost: VideoRetrieve(ID)
    Bifrost->>sendTaskArray: [getResponse task]
    sendTaskArray->>RunwareAPI: "POST /v1 [{taskType:getResponse, taskUUID}]"
    RunwareAPI-->>sendTaskArray: "{data:[{taskUUID, videoURL, status:success}]}"
    sendTaskArray-->>Bifrost: result with videoURL
    Bifrost-->>Caller: "BifrostVideoGenerationResponse{ID, status=completed, Videos[URL]}"

    Caller->>Bifrost: VideoDownload(ID)
    Bifrost->>Bifrost: VideoRetrieve(ID) [internal]
    Bifrost->>RunwareCDN: GET videoURL (no auth header)
    RunwareCDN-->>Bifrost: video bytes + Content-Type
    Bifrost-->>Caller: "BifrostVideoDownloadResponse{Content, ContentType}"

Loading

_{Reviews (3): Last reviewed commit: "feat: video operations in runware" | Re-trigger Greptile}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@core/providers/runware/runware_test.go`:
- Around line 29-35: The Scenarios struct in the test configuration has
VideoGenerationModel set but all video scenario flags (VideoGeneration,
VideoRetrieve, VideoDownload) are disabled. To exercise the new Runware video
provider methods in tests, change at least the VideoGeneration flag (and ideally
VideoRetrieve as well) from false to true in the Scenarios configuration.
Alternatively, if video test coverage is intentionally deferred, add an explicit
SkipReason field explaining why these scenarios are disabled.

In `@core/providers/runware/runware.go`:
- Around line 382-427: The VideoDownload method accepts a request parameter with
a Variant field, but never uses it when selecting which video asset to download.
Update the method to use the request.Variant to select the appropriate video
from taskDetails.Videos instead of always using the first element at index 0.
Add validation to ensure the requested variant exists in the response, and
return an appropriate error if the variant is not found or if variants are
expected but none are available in the taskDetails.

In `@core/providers/runware/videos.go`:
- Around line 69-73: The `ToBifrostVideoGenerationResponse` function is
incorrectly setting the `CreatedAt` field to `time.Now().Unix()`, which causes
incorrect timestamps when the function is used for retrieve/polling operations.
Instead of synthesizing the current time, use the actual creation timestamp from
the result object (which should contain the original creation time). Replace the
`CreatedAt: time.Now().Unix()` assignment with the corresponding creation
timestamp field from the result parameter.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 85007a50-d5b7-440a-856d-3dbe0b759648

📥 Commits

Reviewing files that changed from the base of the PR and between 55dbf0a and 2c9c1c9.

📒 Files selected for processing (6)

• core/providers/runware/images.go
• core/providers/runware/runware.go
• core/providers/runware/runware_test.go
• core/providers/runware/types.go
• core/providers/runware/utils.go
• core/providers/runware/videos.go

[akshaydeo]

Merge activity

• Jun 23, 1:28 PM UTC: A user started a stack merge that includes this pull request via Graphite.
• Jun 23, 1:31 PM UTC: Graphite rebased this pull request as part of a merge.
• Jun 23, 1:32 PM UTC: @akshaydeo merged this pull request with Graphite.

The base branch was changed.