[Model] Add circlestone-labs/Anima by akshatvishu · Pull Request #4083 · vllm-project/vllm-omni

akshatvishu · 2026-06-02T22:25:25Z

Resolves #3658

Adds native diffusion support for circlestone-labs/Anima, a Cosmos-style text-to-image model as a local single-file safetensors checkpoint.

The new AnimaPipeline loads the transformer and text-conditioner weights from the checkpoint, maps the original Cosmos-style keys when needed and reuses the existing Diffusers-style component directory for the text encoder, tokenizers, VAE and scheduler.

TP, SP, CFG parallel, HSDP, Cache-DiT/TeaCache, quantization, CPU/layerwise offload and step execution are left for follow-up work.

Key Changes

Native Anima Pipeline

Adds vllm_omni/diffusion/models/anima/ with AnimaPipeline, AnimaTransformer3DModel and AnimaTextConditioner.
Loads Anima directly from a local single-file safetensors checkpoint.
Converts the original Cosmos-style transformer keys before strict native loading.
Adds prompt encoding, true CFG, denoising, VAE decode, and Anima post-processing.
Uses custom_pipeline_args for Anima-specific paths, for example components_path.

Native Single-File Resolution

Registers AnimaPipeline in the diffusion registry.
Maps Anima single-file aliases, including AnimaModularPipeline, to AnimaPipeline.
Lets local Anima single-file checkpoints use the default diffusion load format and native pipeline path.
Lets local Anima single-file serving use the default single-stage config without a deploy YAML.

Worker / Custom Pipeline Args

Updates the custom_pipeline_args worker path so skip_load_model and re_init_pipeline() only run when pipeline_class is set.
This lets native pipelines pass model-specific paths through custom_pipeline_args without being treated as dynamic custom pipelines.

Tests:

Adds unit tests for native Anima loading, alias resolution, component path handling, CFG behavior and worker wrapper behavior.
Adds offline and online Anima serving tests.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1e64bd1902

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-06-02T22:30:32Z

+            pipeline.enable_vae_tiling()
+
+        self._pipeline = pipeline
+        self._accept_call_kwargs = set(inspect.signature(pipeline.__call__).parameters.keys())


Preserve ModularPipeline runtime kwargs

When the native Anima path is used, pipeline is a Diffusers ModularPipeline, whose __call__ signature is generic (state, output, **kwargs) rather than listing model inputs like prompt, height, or num_inference_steps. Caching that signature here makes _build_call_kwargs() later reject and drop the actual request fields, so a normal text-to-image request reaches the modular blocks without the required prompt and fails before generation. For modular pipelines this needs to allow block input names (or accept all kwargs) instead of using inspect.signature(pipeline.__call__) directly.

Useful? React with 👍 / 👎.

akshatvishu · 2026-06-04T21:41:07Z

Doing a major refactoring , not ready for review yet!

timzsu · 2026-06-13T08:23:00Z

Hi @akshatvishu, may I ask when this PR will be ready?

akshatvishu · 2026-06-13T11:57:54Z

@timzsu It's ready review!

The benchmarking code is included temporarily for validating this port. Once we're happy with the implementation, I'll run the benchmarks against the native diffuser implementation and remove the benchmarking code afterward before merging!

timzsu · 2026-06-13T12:14:08Z

Hi @akshatvishu, is it possible to split the performance optimizations from the model support? The current PR is too big (>3k lines) and hard to review. I suggest keeping the first PR as an integration with no extra optimizations. Then you can create separate PRs for offloading, quantization, and cache based on it.

Signed-off-by: akshatvishu <akshatnayak197@gmail.com>

vllm-omni-review-bot · 2026-08-31T11:29:35Z

Omni ReviewBot triage note

Automated triage of commit ff9057781caf produced:

Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

vllm-omni-review-bot · 2026-09-16T10:16:17Z

Omni ReviewBot: no human activity for 15 days

@akshatvishu this pull request has had no human commit, comment or review since 2026-08-31. Per repository policy it may be closed if it stays inactive.

To keep it moving, any one of these is enough: push an update, reply to the open blocker, or post the current plan and timeline.

Signed-off-by: akshatvishu <akshatnayak197@gmail.com> # Conflicts: # docs/models/supported_models.md # examples/offline_inference/text_to_image/text_to_image.py # tests/entrypoints/test_utils.py # vllm_omni/diffusion/data.py # vllm_omni/diffusion/registry.py # vllm_omni/entrypoints/utils.py

akshatvishu · 2026-09-16T14:04:30Z

Hey @alex-jw-brooks @tzhouam @yuanheng-zhao , following up on the suggestion to replace the Anima allowlist with a local-file plus explicit-class check.

My concern is that an explicit class name does not guarantee its loader supports checkpoint files. We could declare that support in the existing DiffusionModelMetadata -> avoiding the pipeline import required by my earlier proposal. However, that still adds a shared field and requires each supported model to opt in.

Would you prefer the simpler file-plus-class rule or the explicit support check? The native loading path and previously suggested worker guard would stay as they are.

tzhouam · 2026-09-16T15:39:14Z

fix the CI error please, thanks

Signed-off-by: akshatvishu <akshatnayak197@gmail.com>

akshatvishu · 2026-09-16T16:44:47Z

@tzhouam CI was failing because Anima was missing shared tiny-model test settings; I added them and also fixed a CLI bug that treated local checkpoint files as HF model IDs. Can you please re-run the CI?

tzhouam · 2026-09-17T08:43:02Z

still failed, please pass the tests locally
also, please deal with the conflict

Signed-off-by: akshatvishu <akshatnayak197@gmail.com> # Conflicts: # vllm_omni/config/resolver.py # vllm_omni/entrypoints/cli/serve.py

akshatvishu · 2026-09-17T10:58:38Z

Fixed the conflicts! The CI show 3 failure , none of which is related to changes made in this PR @tzhouam ! Short summary of CI failures:

Failure	Existing report / fix	Status
NVIDIA NIXL	Issue #7699 includes this exact failure. PR #7088 contains the test fix.	Open
AMD MAGI2	PR #7606 adds the correct CUDA platform guard.	Open
Intel disk full	Issue #7587 reports the same runner failure.	Open

Checked against main at f5e4f5fa4 .

Signed-off-by: akshatvishu <akshatnayak197@gmail.com>

Signed-off-by: akshatvishu <akshatnayak197@gmail.com> Signed-off-by: Matthieu Laneuville <matthieu.laneuville@surf.nl>

Signed-off-by: akshatvishu <akshatnayak197@gmail.com>

akshatvishu requested review from Gaohan123, Isotr0py, RuixiangMa, SamitHuang, ZJY0516, david6666666, hsliuustc0106, princepride, tzhouam, wtomin and yenuo26 as code owners June 2, 2026 22:25

chatgpt-codex-connector Bot reviewed Jun 2, 2026

View reviewed changes

akshatvishu force-pushed the anima branch from 1e64bd1 to 77d77f4 Compare June 2, 2026 22:33

akshatvishu requested review from yuanheng-zhao and ywang96 as code owners June 4, 2026 22:02

akshatvishu requested review from ZeldaHuang, linyueqian and lishunyang12 as code owners June 5, 2026 12:33

akshatvishu force-pushed the anima branch from 1fda9a7 to e6cde7b Compare June 5, 2026 19:34

akshatvishu force-pushed the anima branch from 403b9fa to 17d6815 Compare June 13, 2026 09:54

akshatvishu requested review from congw729 and gcanlin as code owners June 13, 2026 09:54

akshatvishu force-pushed the anima branch from 17d6815 to f6cfe9d Compare June 13, 2026 09:59

akshatvishu changed the title ~~[WIP] Add circlestone-labs/Anima~~ [Model] Add circlestone-labs/Anima Jun 13, 2026

Add Anima Pipeline

223e441

Signed-off-by: akshatvishu <akshatnayak197@gmail.com>

akshatvishu force-pushed the anima branch from 164f3b5 to 223e441 Compare June 13, 2026 12:58

akshatvishu requested review from Sy0307, alex-jw-brooks and xuechendi as code owners July 3, 2026 13:51

hsliuustc0106 added enhancement New feature or request new model add new model labels Jul 17, 2026 — with ChatGPT Codex Connector

akshatvishu requested a review from NickCao as a code owner September 16, 2026 13:59

tzhouam added the ready label to trigger buildkite CI label Sep 16, 2026

[BugFix] Add Anima tiny-model coverage and fix native CLI validation

a56c95f

Signed-off-by: akshatvishu <akshatnayak197@gmail.com>

hsliuustc0106 added the high priority high priority issue, needs to be done asap label Sep 17, 2026

tzhouam added ready label to trigger buildkite CI and removed ready label to trigger buildkite CI labels Sep 17, 2026

Merge remote-tracking branch 'upstream/main' into anima

8dcd74e

Signed-off-by: akshatvishu <akshatnayak197@gmail.com> # Conflicts: # vllm_omni/config/resolver.py # vllm_omni/entrypoints/cli/serve.py

yuanheng-zhao added ready label to trigger buildkite CI and removed ready label to trigger buildkite CI labels Sep 17, 2026

linyueqian mentioned this pull request Sep 17, 2026

[CI][Flaky] test_nixl_connector.py::test_unknown_key_is_queried_once fails with ZMQError "Socket operation on non-socket" in Simple · Other Test #7615

Open

tzhouam enabled auto-merge (squash) September 18, 2026 01:49

[CI] Retrigger CI

ff90577

Signed-off-by: akshatvishu <akshatnayak197@gmail.com>

tzhouam added ready label to trigger buildkite CI and removed ready label to trigger buildkite CI labels Sep 19, 2026

tzhouam merged commit 415005b into vllm-project:main Sep 19, 2026
6 of 9 checks passed

mlaneuville pushed a commit to mlaneuville/vllm-omni that referenced this pull request Sep 22, 2026

[Model] Add circlestone-labs/Anima (vllm-project#4083)

d61efc6

Signed-off-by: akshatvishu <akshatnayak197@gmail.com> Signed-off-by: Matthieu Laneuville <matthieu.laneuville@surf.nl>

khairulkabir1661 pushed a commit to khairulkabir1661/vllm-omni that referenced this pull request Sep 25, 2026

[Model] Add circlestone-labs/Anima (vllm-project#4083)

cf1d304

Signed-off-by: akshatvishu <akshatnayak197@gmail.com>

Conversation

akshatvishu commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Key Changes

Native Anima Pipeline

Native Single-File Resolution

Worker / Custom Pipeline Args

Tests:

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

akshatvishu commented Jun 4, 2026

Uh oh!

timzsu commented Jun 13, 2026

Uh oh!

akshatvishu commented Jun 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

timzsu commented Jun 13, 2026

Uh oh!

vllm-omni-review-bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Omni ReviewBot triage note

Uh oh!

vllm-omni-review-bot commented Sep 16, 2026

Omni ReviewBot: no human activity for 15 days

Uh oh!

akshatvishu commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tzhouam commented Sep 16, 2026

Uh oh!

akshatvishu commented Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tzhouam commented Sep 17, 2026

Uh oh!

akshatvishu commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

akshatvishu commented Jun 2, 2026 •

edited

Loading

akshatvishu commented Jun 13, 2026 •

edited

Loading

vllm-omni-review-bot commented Aug 31, 2026 •

edited

Loading

akshatvishu commented Sep 16, 2026 •

edited

Loading

akshatvishu commented Sep 16, 2026 •

edited

Loading

akshatvishu commented Sep 17, 2026 •

edited

Loading