Add composable world-model pipelines and Cosmos3 export - #466
Draft
xiaoyu-work wants to merge 7 commits into
Draft
Add composable world-model pipelines and Cosmos3 export#466xiaoyu-work wants to merge 7 commits into
xiaoyu-work wants to merge 7 commits into
Conversation
Introduce executable pipeline schema 1.1 with typed generated-input programs, recurrent state lifecycle, scheduler and autoregressive controls, parameterized transforms, runtime assets, and dtype/EP hints. Implement complete Cosmos3 Omni and Edge exports with the shared MoT generator, Wan video VAE, optional AVAE audio tokenizer, domain-aware action heads, real checkpoint weight routing, CLI support, documentation, and numerical tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 997ab62c-69cb-4802-a5c6-fddd51bfd583 Signed-off-by: Xiaoyu Zhang <xiaoyuzhang@microsoft.com>
|
|
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
|
|
||
| import onnx_ir as ir | ||
|
|
||
| from mobius._configs.per_model import _cosmos3_edge_vision # noqa: F401 |
| from mobius.tasks._cosmos3_audio import select_cosmos3_audio_task | ||
| from mobius.tasks._wan_vae import WanVAETask | ||
|
|
||
| logger = logging.getLogger(__name__) |
Focus the world-model guides on export commands, package contents, supported checkpoints, runtime usage, and practical options. Remove internal architecture and manifest implementation details from the user guide. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 997ab62c-69cb-4802-a5c6-fddd51bfd583 Signed-off-by: Xiaoyu Zhang <xiaoyuzhang@microsoft.com>
Performance Comparison
|
🏗️ Architecture Diff
No architecture changes detected. ✅ Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
Add model-agnostic build, generation, and pipeline configuration for world-model exporters, plus safe shared helpers for diffusers-style checkpoint inspection and asset resolution. Keep LLM ArchitectureConfig and every model/component implementation unchanged; Cosmos3 Omni and Edge only consume the new outer pipeline configuration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 997ab62c-69cb-4802-a5c6-fddd51bfd583 Signed-off-by: Xiaoyu Zhang <xiaoyuzhang@microsoft.com>
Match upstream chunk-zero temporal resampling so Cosmos3 can decode a single latent frame without running Conv on an empty temporal extent. Preserve bit-identical multi-frame behavior and add ORT/PyTorch parity coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 997ab62c-69cb-4802-a5c6-fddd51bfd583 Signed-off-by: Xiaoyu Zhang <xiaoyuzhang@microsoft.com>
Replace the fixed SigLIP1-style vision graph with the published packed variable-resolution SigLIP2 contract, correct block-major projection ordering, and use interleaved multimodal RoPE. Add image/video feature routing, processor metadata, real-checkpoint numerical parity tests, and image/video understanding coverage while preserving text-only behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 997ab62c-69cb-4802-a5c6-fddd51bfd583 Signed-off-by: Xiaoyu Zhang <xiaoyuzhang@microsoft.com>
Describe conditioned video encoding, classifier-free guidance, prompt packing, and official Cosmos3 Edge generation recipes in pipeline.json. Ship the recommended negative prompt when available and preserve checkpoint-specific action behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 997ab62c-69cb-4802-a5c6-fddd51bfd583 Signed-off-by: Xiaoyu Zhang <xiaoyuzhang@microsoft.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Apply the repository Ruff formatter to resolve CI formatting findings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 997ab62c-69cb-4802-a5c6-fddd51bfd583 Signed-off-by: Xiaoyu Zhang <xiaoyuzhang@microsoft.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduce executable pipeline schema 1.1 with typed generated-input programs, recurrent state lifecycle, scheduler and autoregressive controls, parameterized transforms, runtime assets, and dtype/EP hints.
Implement complete Cosmos3 Omni and Edge exports with the shared MoT generator, Wan video VAE, optional AVAE audio tokenizer, domain-aware action heads, real checkpoint weight routing, CLI support, documentation, and numerical tests.