Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 53 additions & 22 deletions .github/workflows/omnidreams-demo-runtime.yml
Original file line number Diff line number Diff line change
Expand Up @@ -103,15 +103,62 @@ jobs:

log_dir="${ARTIFACT_DIR}/logs"
output_dir="${ARTIFACT_DIR}/outputs"
launch_dir="${ARTIFACT_DIR}/launch_manifests"
summary="${ARTIFACT_DIR}/summary.md"
status_file="${ARTIFACT_DIR}/command-status.env"
mkdir -p "${log_dir}" "${output_dir}"
mkdir -p "${log_dir}" "${output_dir}" "${launch_dir}"
: > "${status_file}"

odemo() {
uv run --no-sync --package flashdreams-omnidreams omnidreams-demo "$@"
frun() {
uv run --no-sync --package flashdreams-omnidreams flashdreams-run "$@"
}

runner="omnidreams-sv-2steps-chunk2-loc6-lightvae-lighttae"
null_manifest="${launch_dir}/null.yaml"
precomputed_manifest="${launch_dir}/precomputed-mp4.yaml"
ludus_manifest="${launch_dir}/ludus-mp4.yaml"

cat > "${null_manifest}" <<EOF
schema_version: 1
runner: ${runner}
mode: "null"
runner_overrides:
device: cuda:0
scenario:
total_blocks: ${NULL_BLOCKS}
EOF

cat > "${precomputed_manifest}" <<EOF
schema_version: 1
runner: ${runner}
mode: mp4
runner_overrides:
device: cuda:0
scenario:
example_data: true
example_data_uuid: ${EXAMPLE_DATA_UUID}
total_blocks: ${PRECOMPUTED_BLOCKS}
output:
path: ${GITHUB_WORKSPACE}/${output_dir}/omnidreams-demo-precomputed-20s.mp4
fps: ${FPS}
EOF

cat > "${ludus_manifest}" <<EOF
schema_version: 1
runner: ${runner}
mode: mp4
runner_overrides:
device: cuda:0
scenario:
conditioning_mode: ludus-scene-driving
keyboard_trace: ${GITHUB_WORKSPACE}/${LUDUS_TRACE}
scene_uuid: ${LUDUS_SCENE_UUID}
total_blocks: ${LUDUS_BLOCKS}
output:
path: ${GITHUB_WORKSPACE}/${output_dir}/omnidreams-demo-ludus-20s.mp4
fps: ${FPS}
EOF

run_demo() {
local name="$1"
shift
Expand Down Expand Up @@ -143,29 +190,13 @@ jobs:
} > "${summary}"

run_demo null \
odemo replay \
--output-mode null \
--device cuda:0 \
--total-blocks "${NULL_BLOCKS}"
frun "${runner}" null --manifest "${null_manifest}"

run_demo precomputed-mp4 \
odemo replay \
--device cuda:0 \
--example-data \
--example-data-uuid "${EXAMPLE_DATA_UUID}" \
--total-blocks "${PRECOMPUTED_BLOCKS}" \
--fps "${FPS}" \
--output "${output_dir}/omnidreams-demo-precomputed-20s.mp4"
frun "${runner}" mp4 --manifest "${precomputed_manifest}"

run_demo ludus-mp4 \
odemo replay \
--conditioning-mode ludus-scene-driving \
--keyboard-trace "${LUDUS_TRACE}" \
--device cuda:0 \
--scene-uuid "${LUDUS_SCENE_UUID}" \
--seed 42 \
--total-blocks "${LUDUS_BLOCKS}" \
--output "${output_dir}/omnidreams-demo-ludus-20s.mp4"
frun "${runner}" mp4 --manifest "${ludus_manifest}"

- name: Validate OmniDreams demo artifacts
run: |
Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,8 @@ pip install flashdreams

### Try the interactive driving demo

Drive a world model in real time with the OmniDreams `interactive-drive` demo. See the
Drive a world model in real time with the unified OmniDreams `local-window` or
`webrtc` launch mode. See the
**[interactive demo guide](https://nvidia.github.io/flashdreams/main/models/omnidreams.html#launch-the-interactive-demo)**.
Comment on lines +106 to 107

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: we will need to update the website accordingly.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will do another round of updating


## Supported models
Expand Down
14 changes: 14 additions & 0 deletions docs/launch_manifests/lingbot_mp4.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
schema_version: 1
runner: lingbot-world-fast
mode: mp4

scenario:
example_data: true
example_idx: 0
total_blocks: 20
pixel_width: 832
pixel_height: 464

output:
path: ../../outputs/lingbot-replay.mp4
fps: 16
16 changes: 16 additions & 0 deletions docs/launch_manifests/lingbot_webrtc.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
schema_version: 1
runner: lingbot-world-fast
mode: webrtc

scenario:
example_idx: 0

output:
host: 0.0.0.0
port: 8080
fps: 16
video_width: 832
video_height: 464
warmup_chunks: 10
warmup_timeout_s: 600
client_liveness_timeout_s: 30
10 changes: 10 additions & 0 deletions docs/launch_manifests/omnidreams_local_window.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
schema_version: 1
runner: omnidreams-sv-2steps-chunk2-loc6-lightvae-lighttae-perf
mode: local-window

scenario:
auto_start: false
preload_scenes: false

output:
world_model_manifest_path: ../../integrations/omnidreams/omnidreams/interactive_drive/configs/example_world_model_perf.yaml
13 changes: 13 additions & 0 deletions docs/launch_manifests/omnidreams_mp4.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
schema_version: 1
runner: omnidreams-sv-2steps-chunk2-loc6-lightvae-lighttae
mode: mp4

scenario:
example_data: true
total_blocks: 60
pixel_width: 1280
pixel_height: 704

output:
path: ../../outputs/omnidreams-replay.mp4
fps: 30
10 changes: 10 additions & 0 deletions docs/launch_manifests/omnidreams_null.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
schema_version: 1
runner: omnidreams-sv-2steps-chunk2-loc6-lightvae-lighttae
mode: "null"

scenario:
example_data: true
total_blocks: 10
pixel_width: 1280
pixel_height: 704
fps: 30
21 changes: 21 additions & 0 deletions docs/launch_manifests/omnidreams_webrtc.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
schema_version: 1
runner: omnidreams-sv-2steps-chunk2-loc6-lightvae-lighttae
mode: webrtc

runner_overrides:
device: cuda:0

scenario:
scene_uuid: 0d404ff7-2b66-498c-b047-1ed8cded60d4
scene_variant: default
camera_name: camera_front_wide_120fov

output:
host: 0.0.0.0
port: 8089
fps: 30
video_width: 1280
video_height: 704
warmup_chunks: 10
warmup_timeout_s: 600
client_liveness_timeout_s: 10
17 changes: 15 additions & 2 deletions docs/source/api/cli.rst
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,24 @@ Inspect one runner's full options:

uv run flashdreams-run self-forcing-wan2.1-t2v-1.3b-taehv --help

Run a single-GPU inference:
Run a single-GPU inference (``run`` is the default mode):

.. code-block:: bash

uv run flashdreams-run self-forcing-wan2.1-t2v-1.3b-taehv --total-blocks 7

Launch a WebRTC demo from a versioned manifest:

.. code-block:: bash

uv run flashdreams-run lingbot-world-fast webrtc \
--manifest docs/launch_manifests/lingbot_webrtc.yaml

The common command shape is ``flashdreams-run <runner> [mode]``. A runner only
advertises modes it implements; unsupported pairs fail before CUDA
initialization. Shared modes are ``run``, ``mp4``, ``null``, ``webrtc``, and
``local-window``.

Run a multi-GPU inference:

.. code-block:: bash
Expand All @@ -51,7 +63,7 @@ Resolve config only (no model instantiation):

.. code-block:: bash

uv run flashdreams-run --no-instantiate self-forcing-wan2.1-t2v-1.3b-taehv
uv run flashdreams-run self-forcing-wan2.1-t2v-1.3b-taehv --no-instantiate

Post-processing presets
-----------------------
Expand All @@ -73,5 +85,6 @@ See also
--------

- :doc:`/quickstart/index`
- :doc:`/api/launch_manifests`
- :doc:`/developer_guides/config_system`
- :doc:`/api/infra`
1 change: 1 addition & 0 deletions docs/source/api/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ Reference pages for the FlashDreams command-line interface and Python APIs.
:maxdepth: 1

cli
launch_manifests
core
infra
integrations
Expand Down
106 changes: 106 additions & 0 deletions docs/source/api/launch_manifests.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
.. SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
.. SPDX-License-Identifier: Apache-2.0

Launch manifests
================

Every FlashDreams model launch starts with the same command shape:

.. code-block:: bash

uv run flashdreams-run <runner-slug> [mode] [--manifest PATH]

``run`` is the default mode. Integrations may additionally expose ``mp4``,
``null``, ``webrtc``, or ``local-window``. Inspect a resolved launch without
loading checkpoints or initializing CUDA with ``--no-instantiate``.

Schema
------

Launch manifests are strict, versioned YAML documents:

.. code-block:: yaml

schema_version: 1
runner: omnidreams-sv-2steps-chunk2-loc6-lightvae-lighttae
mode: webrtc

runner_overrides:
device: cuda:0

scenario:
scene_uuid: 0d404ff7-2b66-498c-b047-1ed8cded60d4
scene_variant: default

output:
host: 0.0.0.0
port: 8089

``schema_version``, ``runner``, and ``mode`` are required. The remaining
sections are optional mappings:

.. note::

Quote the null-output mode as ``mode: "null"`` in YAML; an unquoted
``null`` is YAML's null scalar rather than the FlashDreams mode name.

``runner_overrides``
Recursive overrides for the registered runner configuration. The same
runner fields remain available as explicit CLI flags.

``scenario``
Inputs and controls such as prompts, example data, scenes, traces, and
rollout length. The selected integration validates the accepted fields.

``output``
Transport or artifact settings such as output path, frame rate, WebRTC
bind address, warmup, and local-window presentation settings.

Relative paths in keys named ``path`` or ending in ``_path``, ``_paths``, or
``_dir`` resolve relative to the manifest file, which makes checked-in launch
manifests reproducible from any working directory. Unknown top-level or
integration-specific fields fail before CUDA initialization.

Precedence
----------

Settings resolve in this order, from lowest to highest precedence:

.. code-block:: text

registered runner preset
< manifest runner_overrides
< manifest scenario/output
< explicit CLI runner flags and --host/--port

The runner and positional mode must agree with the manifest. For example, this
fails instead of silently launching a different preset:

.. code-block:: bash

uv run flashdreams-run lingbot-world-fast mp4 \
--manifest docs/launch_manifests/lingbot_webrtc.yaml

Examples
--------

.. code-block:: bash

# WebRTC
uv run flashdreams-run lingbot-world-fast webrtc \
--manifest docs/launch_manifests/lingbot_webrtc.yaml

# MP4 replay
uv run flashdreams-run lingbot-world-fast mp4 \
--manifest docs/launch_manifests/lingbot_mp4.yaml

# Resolve an OmniDreams launch without loading the model
uv run flashdreams-run \
omnidreams-sv-2steps-chunk2-loc6-lightvae-lighttae webrtc \
--manifest docs/launch_manifests/omnidreams_webrtc.yaml \
--no-instantiate

OmniDreams local-window also accepts the existing
``example_world_model*.yaml`` format directly as a compatibility input. New
automation should use a versioned launch manifest whose
``output.world_model_manifest_path`` references that model-specific file.
Loading