Add Cosmos3 model support

docs/.nav.yml

@@ -17,6 +17,7 @@ nav:
     - Offline Inference:
       - BAGEL-7B-MoT: user_guide/examples/offline_inference/bagel.md
       - GLM-Image Multistage End-to-End Inference: user_guide/examples/offline_inference/glm_image.md
+      - Cosmos3: user_guide/examples/offline_inference/cosmos3.md
       - Helios Video Generation: user_guide/examples/offline_inference/helios.md
       - HunyuanImage-3.0 Image-to-Text Inference: user_guide/examples/offline_inference/hunyuan_image3.md
       - Image-To-Image: user_guide/examples/offline_inference/image_to_image.md
@@ -34,6 +35,7 @@ nav:
     - Online Serving:
       - BAGEL-7B-MoT: user_guide/examples/online_serving/bagel.md
       - vLLM-Omni Helm Chart: user_guide/examples/online_serving/chart-helm.md
+      - Cosmos3: user_guide/examples/online_serving/cosmos3.md
       - Diffusers Backend Adapter: user_guide/examples/online_serving/diffusers_pipeline_adapter.md
       - GLM-Image Online Serving: user_guide/examples/online_serving/glm_image.md
       - Image-To-Image: user_guide/examples/online_serving/image_to_image.md

docs/models/supported_models.md

@@ -32,6 +32,7 @@ th {
 | `ZImagePipeline` | Z-Image | `Tongyi-MAI/Z-Image-Turbo` | ✅︎ | ✅︎ | ✅︎ | ✅︎ |
 | `WanPipeline` | Wan2.1-T2V, Wan2.2-T2V, Wan2.2-TI2V | `Wan-AI/Wan2.1-T2V-1.3B-Diffusers`, `Wan-AI/Wan2.1-T2V-14B-Diffusers`, `Wan-AI/Wan2.2-T2V-A14B-Diffusers`, `Wan-AI/Wan2.2-TI2V-5B-Diffusers` | ✅︎ | ✅︎ | ✅︎ | ✅︎ |
 | `WanImageToVideoPipeline` | Wan2.2-I2V | `Wan-AI/Wan2.2-I2V-A14B-Diffusers` | ✅︎ | ✅︎ | ✅︎ | ✅︎ |
+| `Cosmos3OmniDiffusersPipeline` | Cosmos3 T2I, T2V, I2V, T2V with sound, action policy | `nvidia/Cosmos3-Nano` | ✅︎ | | | |
 | `WanSpeechToVideoPipeline` | Wan2.2-S2V | `Wan-AI/Wan2.2-S2V-14B` | ✅︎ | ✅︎ | ✅︎ | ✅︎ |
 | `Wan22VACEPipeline` | Wan2.1-VACE | `Wan-AI/Wan2.1-VACE-1.3B-diffusers`, `Wan-AI/Wan2.1-VACE-14B-diffusers` | ✅︎ | ✅︎ | ✅︎ | ✅︎ |
 | `LTX2Pipeline` | LTX-2-T2V | `Lightricks/LTX-2` | ✅︎ | ✅︎ | | |

docs/user_guide/diffusion/cache_acceleration/cache_dit.md

@@ -128,6 +128,21 @@ python image_edit.py \
 
 See the [image_edit.py](https://github.com/vllm-project/vllm-omni/blob/main/examples/offline_inference/image_to_image/image_edit.py) for detailed configuration options.
 
+For Cosmos3 text-to-video or image-to-video, use the video examples with the Cosmos3 pipeline class:
+
+```bash
+cd examples/offline_inference/text_to_video
+
+python text_to_video.py \
+    --model nvidia/Cosmos3-Nano \
+    --model-class-name Cosmos3OmniDiffusersPipeline \
+    --prompt "A small warehouse robot moves a blue box across a clean floor." \
+    --cache-backend cache_dit \
+    --num-inference-steps 35
+```
+
+Cosmos3 Cache-DiT wraps the GEN denoising path. TeaCache is not implemented for Cosmos3.
+
 ### Online Serving
 
 ```bash
@@ -138,6 +153,11 @@ vllm serve Qwen/Qwen-Image --omni --port 8091 --cache-backend cache_dit
 vllm serve Qwen/Qwen-Image --omni --port 8091 \
   --cache-backend cache_dit \
   --cache-config '{"Fn_compute_blocks": 1, "residual_diff_threshold": 0.12}'
+
+# Cosmos3
+vllm serve nvidia/Cosmos3-Nano --omni --port 8091 \
+  --model-class-name Cosmos3OmniDiffusersPipeline \
+  --cache-backend cache_dit
 ```
 
 ---

docs/user_guide/diffusion/cpu_offload_diffusion.md

@@ -194,10 +194,12 @@ Factory function `get_offload_backend()` selects the appropriate backend based o
 | OvisImagePipeline | `AIDC-AI/Ovis-Image-7B` | `OvisImageTransformer2DModel` | - | ✓ | `"transformer"` |
 | QwenImagePipeline | `Qwen/Qwen-Image` | `QwenImageTransformer2DModel` | ✓ | ✓ | `"transformer_blocks"` |
 | StableDiffusion3Pipeline | `stabilityai/stable-diffusion-3.5-medium` | `SD3Transformer2DModel` | - | ✓ | `"transformer_blocks"` |
+| Cosmos3OmniDiffusersPipeline | `nvidia/Cosmos3-Nano` | `Cosmos3VFMTransformer` | - | ✓ | `"gen_layers"` |
 | Wan22I2VPipeline | `Wan-AI/Wan2.2-I2V-A14B-Diffusers` | `WanTransformer3DModel` | ✓ | ✓ | `"blocks"` |
 | Wan22Pipeline | `Wan-AI/Wan2.2-T2V-A14B-Diffusers` | `WanTransformer3DModel` | ✓ | ✓ | `"blocks"` |
 | BagelPipeline | `ByteDance-Seed/BAGEL-7B-MoT` | `Qwen2MoTModel` | - | ✓ | `"layers"`, `"customized modules"` |
 
 **Notes:**
 - Model-Level Offloading is expected to be supported by all common diffusion models (DiT and encoders) naturally
 - Layerwise Offloading requires DiT class to define `_layerwise_offload_blocks_attrs` pointing to transformer blocks
+- Cosmos3 uses the singular `_layerwise_offload_blocks_attr` compatibility path and offloads GEN decoder layers.

docs/user_guide/diffusion/parallelism/cfg_parallel.md

@@ -144,6 +144,15 @@ sampling_params = OmniDiffusionSamplingParams(
 )
 ```
 
+For Cosmos3, use `guidance_scale` rather than `true_cfg_scale`:
+
+```python
+sampling_params = OmniDiffusionSamplingParams(
+    num_inference_steps=35,
+    guidance_scale=4.0,
+)
+```
+
 2. **Add negative prompt:**
 ```python
 outputs = omni.generate(

docs/user_guide/diffusion_features.md

@@ -108,6 +108,7 @@ The following tables show which models support each feature:
 | Model | ⚡TeaCache | ⚡Cache-DiT | 🔀SP (Ulysses & Ring) | 🔀CFG-Parallel | 🔀Tensor-Parallel | 🔀HSDP | 💾CPU Offload (Layerwise) | 💾VAE-Patch-Parallel | 💾Quantization | 🔄Step Execution |
 |-------|:----------:|:-----------:|:---------------------:|:--------------:|:-----------------:|:------:|:------------------------:|:--------------------:|:--------------:|:----------------:|
 | **Bagel** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅  | ❌ | ❌ | ❌ |
+| **Cosmos3 (T2I)** | ❌ | ✅ | ✅ (Ulysses) | ✅ | ✅ | ✅ | ✅ | ✅ (decode) | ✅ | ❌ |
 | **FLUX.1-dev** | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ |
 | **FLUX.1-schnell** | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ |
 | **FLUX.2-klein** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ❌ |
@@ -135,19 +136,24 @@ The following tables show which models support each feature:
 > Notes:
 > 1. Nextstep_1(T2I) does not support cache acceleration methods such as TeaCache or Cache-DiT.
 > 2. `Tongyi-MAI/Z-Image-Turbo` and `SII-GAIR/daVinci-MagiHuman-Base-1080p` are distilled models with minimal NFEs; CFG-Parallel is not necessary.
+> 3. Cosmos3 T2I uses `Cosmos3OmniDiffusersPipeline` with `modalities=["image"]`. Model-level CPU offload is not supported; use layerwise offload.
 
 ### VideoGen
 
 | Model | ⚡TeaCache | ⚡Cache-DiT | 🔀SP (Ulysses & Ring) | 🔀CFG-Parallel | 🔀Tensor-Parallel | 🔀HSDP | 💾CPU Offload (Layerwise) | 💾VAE-Patch-Parallel | 💾Quantization | 🔄Step Execution |
 |-------|:----------:|:-----------:|:---------------------:|:--------------:|:-----------------:|:------:|:------------------------:|:--------------------:|:--------------:|:----------------:|
 | **Wan2.2** | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ (encode/decode) | ❌ | ❌ |
+| **Cosmos3 (T2V/I2V)** | ❌ | ✅ | ✅ (Ulysses) | ✅ | ✅ | ✅ | ✅ | ✅ (encode/decode) | ✅ | ❌ |
 | **Wan2.1-VACE** | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ (decode) | ❌ | ❌ |
 | **LTX-2** | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
 | **LTX-2.3** | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
 | **Helios** | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
 | **HunyuanVideo-1.5 T2V I2V** | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ (decode) | ✅ | ❌ |
 | **DreamID-Omni** | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ |
 
+> Notes:
+> 1. Cosmos3 T2V and I2V use `Cosmos3OmniDiffusersPipeline` with video output. I2V is selected when the request includes an input image. Model-level CPU offload is not supported; use layerwise offload.
+
 **Frame Interpolation Support**
 
 - **Supported**: Wan2.2 text-to-video, image-to-video, and TI2V pipelines

docs/user_guide/examples/offline_inference/cosmos3.md

@@ -0,0 +1,90 @@
+# Cosmos3
+
+Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/offline_inference/cosmos3>.
+
+
+Cosmos3 uses `Cosmos3OmniDiffusersPipeline` for text-to-image, text-to-video, image-to-video, video-with-sound, and action generation. Examples default to the `nvidia/Cosmos3-Nano` Hugging Face repo, but you can override the checkpoint with `--model` or by exporting `COSMOS3_MODEL` to a local Diffusers-format checkpoint.
+
+```bash
+cd examples/offline_inference/cosmos3
+```
+
+## Text-to-Image
+
+```bash
+python end2end.py \
+  --task t2i \
+  --prompt "A small warehouse robot carrying a blue box, clean product photography" \
+  --negative-prompt "blurry, distorted, low quality" \
+  --output cosmos3_t2i.png
+```
+
+## Text-to-Video
+
+```bash
+python end2end.py \
+  --task t2v \
+  --prompt "A small warehouse robot moves a blue box across a clean floor." \
+  --negative-prompt "blurry, distorted, low quality" \
+  --output cosmos3_t2v.mp4
+```
+
+## Image-to-Video
+
+Download an example image or provide your own image path.
+
+```bash
+wget https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/cherry_blossom.jpg
+
+python end2end.py \
+  --task i2v \
+  --image cherry_blossom.jpg \
+  --prompt "Cherry blossoms swaying gently in the breeze, petals falling, smooth motion" \
+  --negative-prompt "blurry, distorted, low quality" \
+  --output cosmos3_i2v.mp4
+```
+
+## Video With Sound
+
+This path requires a sound-capable Cosmos3 checkpoint with `sound_gen` weights.
+
+```bash
+python end2end.py \
+  --task t2v_sound \
+  --prompt "A small warehouse robot rolls across the floor with soft motor sounds." \
+  --negative-prompt "blurry, distorted, low quality" \
+  --sound-duration 3.4 \
+  --output cosmos3_t2v_sound.mp4
+```
+
+## Action Policy
+
+This path requires an action-capable Cosmos3 checkpoint with `action_gen` weights. The example returns a video plus an action JSON payload. Pass either `--domain-name` or `--domain-id`.
+
+```bash
+python end2end.py \
+  --task action_policy \
+  --image cherry_blossom.jpg \
+  --prompt "Predict the robot action for moving toward the target." \
+  --domain-name bridge_orig_lerobot \
+  --raw-action-dim 2 \
+  --action-chunk-size 16 \
+  --output cosmos3_action_policy.mp4 \
+  --action-output cosmos3_action_policy_action.json
+```
+
+## Common Options
+
+- `--enable-layerwise-offload`: use layerwise offload for memory-constrained runs.
+- `--cache-backend cache_dit`: enable Cache-DiT where supported.
+- `--cfg-parallel-size 2`, `--ulysses-degree`, `--tensor-parallel-size`, `--use-hsdp`: enable parallel execution options.
+- `--height`, `--width`, `--num-frames`, `--num-inference-steps`, `--guidance-scale`, `--fps`: override task defaults.
+
+Do not use model-level `--enable-cpu-offload` for Cosmos3. Use `--enable-layerwise-offload` instead.
+
+## Example materials
+
+??? abstract "end2end.py"
+    ``````py
+    --8<-- "examples/offline_inference/cosmos3/end2end.py"
+    ``````

docs/user_guide/examples/online_serving/cosmos3.md

@@ -0,0 +1,92 @@
+# Cosmos3
+
+Source <https://github.com/vllm-project/vllm-omni/tree/main/examples/online_serving/cosmos3>.
+
+
+This example shows Cosmos3 online serving with `Cosmos3OmniDiffusersPipeline`.
+
+The server defaults to the `nvidia/Cosmos3-Nano` Hugging Face repo. Override the checkpoint by exporting `MODEL` or `COSMOS3_MODEL` to a local Diffusers-format checkpoint.
+
+```bash
+cd examples/online_serving/cosmos3
+bash run_server.sh
+```
+
+`run_server.sh` accepts these environment overrides:
+
+- `MODEL`: checkpoint path or Hugging Face repo, defaults to `nvidia/Cosmos3-Nano` (or `COSMOS3_MODEL` if set)
+- `PORT`: server port, defaults to `8091`
+- `CACHE_BACKEND`: set to `cache_dit` to enable Cache-DiT
+- `ENABLE_LAYERWISE_OFFLOAD`: set to `1` to enable layerwise offload
+- `CFG_PARALLEL_SIZE`, `TENSOR_PARALLEL_SIZE`, `ULYSSES_DEGREE`, `USE_HSDP`: parallel execution controls
+- `ALLOWED_LOCAL_MEDIA_PATH`: local media access path, defaults to `/`
+
+## Text-to-Image
+
+```bash
+bash run_curl_t2i.sh
+```
+
+The script calls `POST /v1/images/generations`, which selects Cosmos3 text-to-image through `modalities=["image"]` internally.
+
+## Text-to-Video
+
+```bash
+bash run_curl_t2v.sh
+```
+
+## Image-to-Video
+
+Download an example image or set `IMAGE_PATH` to your own image:
+
+```bash
+wget https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/cherry_blossom.jpg
+IMAGE_PATH=cherry_blossom.jpg bash run_curl_i2v.sh
+```
+
+## Video With Sound
+
+This path requires a sound-capable Cosmos3 checkpoint with `sound_gen` weights.
+
+```bash
+bash run_curl_t2v_sound.sh
+```
+
+The script passes `generate_sound=true` and `sound_duration` to the video endpoint.
+
+## Action Policy
+
+This path requires an action-capable Cosmos3 checkpoint with `action_gen` weights. Pass either `domain_name` or `domain_id` through `extra_params`.
+
+```bash
+IMAGE_PATH=cherry_blossom.jpg bash run_curl_action_policy.sh
+```
+
+The script uses the asynchronous `POST /v1/videos` job endpoint so it can download the MP4 and save the returned action metadata JSON.
+
+## Example materials
+
+??? abstract "run_curl_action_policy.sh"
+    ``````sh
+    --8<-- "examples/online_serving/cosmos3/run_curl_action_policy.sh"
+    ``````
+??? abstract "run_curl_i2v.sh"
+    ``````sh
+    --8<-- "examples/online_serving/cosmos3/run_curl_i2v.sh"
+    ``````
+??? abstract "run_curl_t2i.sh"
+    ``````sh
+    --8<-- "examples/online_serving/cosmos3/run_curl_t2i.sh"
+    ``````
+??? abstract "run_curl_t2v.sh"
+    ``````sh
+    --8<-- "examples/online_serving/cosmos3/run_curl_t2v.sh"
+    ``````
+??? abstract "run_curl_t2v_sound.sh"
+    ``````sh
+    --8<-- "examples/online_serving/cosmos3/run_curl_t2v_sound.sh"
+    ``````
+??? abstract "run_server.sh"
+    ``````sh
+    --8<-- "examples/online_serving/cosmos3/run_server.sh"
+    ``````

examples/offline_inference/cosmos3/README.md

@@ -0,0 +1,80 @@
+# Cosmos3
+
+Cosmos3 uses `Cosmos3OmniDiffusersPipeline` for text-to-image, text-to-video, image-to-video, video-with-sound, and action generation. Examples default to the `nvidia/Cosmos3-Nano` Hugging Face repo, but you can override the checkpoint with `--model` or by exporting `COSMOS3_MODEL` to a local Diffusers-format checkpoint.
+
+```bash
+cd examples/offline_inference/cosmos3
+```
+
+## Text-to-Image
+
+```bash
+python end2end.py \
+  --task t2i \
+  --prompt "A small warehouse robot carrying a blue box, clean product photography" \
+  --negative-prompt "blurry, distorted, low quality" \
+  --output cosmos3_t2i.png
+```
+
+## Text-to-Video
+
+```bash
+python end2end.py \
+  --task t2v \
+  --prompt "A small warehouse robot moves a blue box across a clean floor." \
+  --negative-prompt "blurry, distorted, low quality" \
+  --output cosmos3_t2v.mp4
+```
+
+## Image-to-Video
+
+Download an example image or provide your own image path.
+
+```bash
+wget https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/cherry_blossom.jpg
+
+python end2end.py \
+  --task i2v \
+  --image cherry_blossom.jpg \
+  --prompt "Cherry blossoms swaying gently in the breeze, petals falling, smooth motion" \
+  --negative-prompt "blurry, distorted, low quality" \
+  --output cosmos3_i2v.mp4
+```
+
+## Video With Sound
+
+This path requires a sound-capable Cosmos3 checkpoint with `sound_gen` weights.
+
+```bash
+python end2end.py \
+  --task t2v_sound \
+  --prompt "A small warehouse robot rolls across the floor with soft motor sounds." \
+  --negative-prompt "blurry, distorted, low quality" \
+  --sound-duration 3.4 \
+  --output cosmos3_t2v_sound.mp4
+```
+
+## Action Policy
+
+This path requires an action-capable Cosmos3 checkpoint with `action_gen` weights. The example returns a video plus an action JSON payload. Pass either `--domain-name` or `--domain-id`.
+
+```bash
+python end2end.py \
+  --task action_policy \
+  --image cherry_blossom.jpg \
+  --prompt "Predict the robot action for moving toward the target." \
+  --domain-name bridge_orig_lerobot \
+  --raw-action-dim 2 \
+  --action-chunk-size 16 \
+  --output cosmos3_action_policy.mp4 \
+  --action-output cosmos3_action_policy_action.json
+```
+
+## Common Options
+
+- `--enable-layerwise-offload`: use layerwise offload for memory-constrained runs.
+- `--cache-backend cache_dit`: enable Cache-DiT where supported.
+- `--cfg-parallel-size 2`, `--ulysses-degree`, `--tensor-parallel-size`, `--use-hsdp`: enable parallel execution options.
+- `--height`, `--width`, `--num-frames`, `--num-inference-steps`, `--guidance-scale`, `--fps`: override task defaults.
+
+Do not use model-level `--enable-cpu-offload` for Cosmos3. Use `--enable-layerwise-offload` instead.