Skip to content

[Recipe] Add Cosmos3-Edge - #5313

Merged
lishunyang12 merged 1 commit into
vllm-project:mainfrom
lishunyang12:cosmos3-edge-recipe
Jul 26, 2026
Merged

lishunyang12 merged 1 commit into
vllm-project:mainfrom
lishunyang12:cosmos3-edge-recipe

Conversation

@lishunyang12

@lishunyang12 lishunyang12 commented Jul 22, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

#5001 added support for the Cosmos3 Edge checkpoint; this adds the recipe for it. recipes/cosmos3/ covered only Nano and Super.

Edge is worth its own page rather than a note on an existing one, because its defaults differ enough that copying Nano's or Super's parameters lands on the wrong operating point:

Edge Nano / Super
T2I resolution 640×640 1024²
T2V / I2V resolution 480×832 1280×720
Video guidance_scale 5.0 6.0
Video flow_shift 3.0 10.0

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

@lishunyang12

lishunyang12 commented Jul 22, 2026 •

Copy link
Copy Markdown
Collaborator Author

@bastefaniak PTAL cc @MaciejBalaNV

Comment thread recipes/cosmos3/Cosmos3-Edge.md Outdated

- Vendor: NVIDIA
- Model: `nvidia/Cosmos3-Edge`
- Task: Text-to-image (T2I), text-to-video (T2V) and image-to-video (I2V) generation

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action is supported as well

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noted.

Comment thread recipes/cosmos3/Cosmos3-Edge.md Outdated

## GPU

### 1x B300 (Offline generation)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why only B300?

@lishunyang12 lishunyang12 Jul 22, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for reminder, i don't think B300 recipe examplifies downstream usage. I will remove it, and leave follow-up prs to provide benchmarking results.

Comment thread recipes/cosmos3/Cosmos3-Edge.md Outdated
Comment on lines +135 to +136
- T2I 640² @ 50 steps → **~38 s**
- T2V 480×832, 49 frames @ 35 steps → **~41 s**

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These generation number look way too high

@hsliuustc0106 hsliuustc0106 added the documentation Improvements or additions to documentation label Jul 23, 2026
@MaciejBalaNV

Copy link
Copy Markdown
Contributor

Can we add the mention about action generation? Other than that, LGTM

@david6666666
david6666666 force-pushed the cosmos3-edge-recipe branch from 986cadf to f54c987 Compare July 25, 2026 16:10
@lishunyang12

Copy link
Copy Markdown
Collaborator Author

Can we add the mention about action generation? Other than that, LGTM

Done

@lishunyang12 lishunyang12 added the ready label to trigger buildkite CI label Jul 25, 2026
@lishunyang12

Copy link
Copy Markdown
Collaborator Author

Self-review notes after checking this page's claims against main (post-#5001). Flagging what I found so it's visible to reviewers; I'll push fixes.

Verified accurate: defaults table matches the constants exactly (pipeline_cosmos3.py:173-178 — 640×640 T2I, 480×832 T2V, guidance 5.0, flow_shift 3.0), including the I2V path (:491-495); backbone auto-selection works as described (:232-237, nothing to pass); the three rejections are real (_validate_edge_generation_mode, :1751-1765); both guardrail mechanisms are correct (serve.py:741 for --no-guardrails, text_to_image.py:447 for the "guardrails" extra-body key).

1. The acceleration note is wrong (Cosmos3-Edge.md, Support notes)

Cache-DiT is a silent no-op (no _cache_dit_adapter_config); Sequence Parallel is unvalidated (no _sp_plan)

Both parentheticals are false. Cosmos3EdgeVFMTransformer subclasses Cosmos3VFMTransformer (transformer_cosmos3_edge.py:303), which defines _cache_dit_adapter_config at transformer_cosmos3.py:989 and _sp_plan at :1020. Edge overrides neither. The adapter config targets gen_layers, which Edge has (validate_loaded_weights requires gen_layers.{i}.mlp.*), and the SP plan targets the GEN pathway Edge shares.

Cache-DiT is therefore wired on Edge, not a no-op — as written the page tells users to skip a feature that works. Will re-test and rewrite this section based on measured behavior rather than attribute presence.

2. The ffprobe expected output can't occur (Verification)

The default writer emits [STREAM] / key=value lines, so video,832,480,49 never prints. The field order is also flipped — the command requests codec_type,nb_frames,width,height, the expected line reads codec_type, width, height, nb_frames. With -of csv=p=0 it would be video,49,832,480.

3. Every tuned parameter in the examples restates a default

T2I passes --num-inference-steps 50 --guidance-scale 7.0 + flow_shift: 3.0; constants are 50 / 7.0 / 3.0 (:168-170). T2V passes 35 / 5.0 / 3.0; constants are 35 / 5.0 / 3.0 (:161,175,176). Only --num-frames 49 and max_sequence_length are non-default. The page's own advice is to omit these to get the Edge defaults, so the examples work against it — and the values will drift silently when the constants change.

4. 1x GPU is not a useful hardware cell (recipes/README.md)

Every other row names a part and VRAM. For the checkpoint whose pitch is constrained deployment, that column is the number readers come for. Will fill in the GPU actually tested.

Minor

  • nvidia/Cosmos3-Edge-Policy-DROID — needs confirming, I haven't verified this ID resolves.
  • Ordering: Edge sits between Nano and Super in the table, but the page describes it as the smallest checkpoint.

Add a concise user-guide recipe for nvidia/Cosmos3-Edge, the small
Nemotron-based Cosmos3 checkpoint: T2I / T2V / I2V and action-conditioned
(Physical-AI) generation, Edge-specific defaults, rejected modes,
acceleration caveats, and offline/online usage.

Signed-off-by: lishunyang12 <lishunyang03@gmail.com>
@david6666666
david6666666 force-pushed the cosmos3-edge-recipe branch from f54c987 to 5f093e5 Compare July 25, 2026 16:41
@david6666666

david6666666 commented Jul 25, 2026 •

Copy link
Copy Markdown
Collaborator

LGTM

@princepride princepride left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@lishunyang12
lishunyang12 enabled auto-merge (squash) July 26, 2026 09:22
@lishunyang12 lishunyang12 added ready label to trigger buildkite CI and removed ready label to trigger buildkite CI labels Jul 26, 2026
@lishunyang12
lishunyang12 merged commit 2bb6189 into vllm-project:main Jul 26, 2026
6 of 7 checks passed
nagisa-kunhah pushed a commit to nagisa-kunhah/vllm-omni that referenced this pull request Jul 30, 2026
Signed-off-by: lishunyang12 <lishunyang03@gmail.com>
Co-authored-by: lishunyang12 <lishunyang03@gmail.com>
khairulkabir1661 pushed a commit to khairulkabir1661/vllm-omni that referenced this pull request Sep 25, 2026
Signed-off-by: lishunyang12 <lishunyang03@gmail.com>
Co-authored-by: lishunyang12 <lishunyang03@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation ready label to trigger buildkite CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants