[Model] Add two stages inference for model LTX-2 distilled.

[Songrui625]

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

This PR add two stages inference for model LTX-2 refer to huggingface/diffusers#12934

Two stages inference is recommended approach to achieve production quality generation, we provide LTX2TwoStagesPipeline (for T2V) and LTX2ImageToVideoTwoStagesPipeline (for I2V) to do this. The pipeline is composed of two stages:

Stage 1: Generate a video at the target resolution using diffusion sampling with classifier-free guidance (CFG). This stage produces a coherent low-noise video sequence that respects the text/image conditioning.
Stage 2: Upsample the Stage 1 output by 2 and refine details using a distilled LoRA model to improve fidelity and visual quality. Stage 2 may apply lighter CFG to preserve the structure from Stage 1 while enhancing texture and sharpness.

For now it only supported distilled model rootonchair/LTX-2-19b-distilled. The distilled LoRA model ltx-2-19b-distilled-lora-384.safetensors is in the main repo but doesn't have the related adpater config, which cause we couldn't load it directly.

How to use

• model: rootonchair/LTX-2-19b-distilled
• Set option --model-class-name to value LTX2TwoStagesPipeline (for T2V) or LTX2ImageToVideoTwoStagesPipeline (for I2V)
• Set option --guidance-scale to value 1.0

Test Plan

Regression tests are all passed to make sure no breaking change. And we also need to bench the acceleration methods for two stages pipeline.

Text-To-Video

LTX-2 distilled T2V single stage sample

python3 text_to_video.py --model /data00/models/LTX-2-19b-distilled --prompt 'A close-up shot of a young waitress in a retro 1950s diner, her warm brown eyes meeting the camera with a gentle smile. She wears a black polka-dot dress with an elegant cream lace collar, her reddish-brown hair styled in an elaborate updo with delicate curls framing her freckled face. Soft, warm light from overhead fixtures illuminates her features as she stands behind a yellow counter. The camera begins slightly to her side, then slowly pushes in toward her face, revealing the subtle rosy blush on her cheeks. In the blurred background, the soft teal walls and a glowing red "Diner" sign create a nostalgic atmosphere. The ambient sounds of clinking dishes, distant conversations, and the gentle hum of a jukebox fill the air. She tilts her head slightly and says in a friendly, warm voice: "Welcome to Rosie'\''s. What can I get for you today?" The mood is inviting, timeless, and full of classic American diner charm.' --negative-prompt "shaky, glitchy, low quality, worst quality, deformed, distorted, disfigured, motion smear, motion artifacts, fused fingers, bad anatomy, weird hand, ugly, transition, static." --guidance-scale 1.0 --width 768 --height 512 --num-inference-steps 8 --num-frames 121 --frame-rate 24 --seed 42 --enforce-eager --output ltx2_t2v_sample.mp4

LTX-2 distilled T2V two stages sample

python3 text_to_video.py --model /data00/models/LTX-2-19b-distilled --model-class-name LTX2TwoStagesPipeline --prompt 'A close-up shot of a young waitress in a retro 1950s diner, her warm brown eyes meeting the camera with a gentle smile. She wears a black polka-dot dress with an elegant cream lace collar, her reddish-brown hair styled in an elaborate updo with delicate curls framing her freckled face. Soft, warm light from overhead fixtures illuminates her features as she stands behind a yellow counter. The camera begins slightly to her side, then slowly pushes in toward her face, revealing the subtle rosy blush on her cheeks. In the blurred background, the soft teal walls and a glowing red "Diner" sign create a nostalgic atmosphere. The ambient sounds of clinking dishes, distant conversations, and the gentle hum of a jukebox fill the air. She tilts her head slightly and says in a friendly, warm voice: "Welcome to Rosie'\''s. What can I get for you today?" The mood is inviting, timeless, and full of classic American diner charm.' --negative-prompt "shaky, glitchy, low quality, worst quality, deformed, distorted, disfigured, motion smear, motion artifacts, fused fingers, bad anatomy, weird hand, ugly, transition, static." --guidance-scale 1.0 --width 768 --height 512 --num-inference-steps 8 --num-frames 121 --frame-rate 24 --seed 42 --enforce-eager --output ltx2_t2v_2st_sample.mp4

Image-To-Video

LTX-2 distilled I2V single stage sample

python3 image_to_video.py --model /data00/models/LTX-2-19b-distilled --image /data00/ltx2_i2v_input.png --model-class-name LTX2ImageToVideoTwoStagesPipeline --prompt "A close-up shot of a young waitress in a retro 1950s diner, her warm brown eyes meeting the camera with a gentle smile. She wears a black polka-dot dress with an elegant cream lace collar, her reddish-brown hair styled in an elaborate updo with delicate curls framing her freckled face. Soft, warm light from overhead fixtures illuminates her features as she stands behind a yellow counter. The camera begins slightly to her side, then slowly pushes in toward her face, revealing the subtle rosy blush on her cheeks. In the blurred background, the soft teal walls and a glowing red \"Diner\" sign create a nostalgic atmosphere. The ambient sounds of clinking dishes, distant conversations, and the gentle hum of a jukebox fill the air. She tilts her head slightly and says in a friendly, warm voice: \"Welcome to Rosie's. What can I get for you today?\" The mood is inviting, timeless, and full of classic American diner charm." --negative-prompt "shaky, glitchy, low quality, worst quality, deformed, distorted, disfigured, motion smear, motion artifacts, fused fingers, bad anatomy, weird hand, ugly, transition, static." --guidance-scale 1.0 --width 768 --height 512 --num-inference-steps 8 --num-frames 121 --fps 24 --seed 45 --enforce-eager --output ltx2_i2v_2st_sample.mp4

LTX-2 distilled I2V two stages sample

python3 image_to_video.py --model /data00/models/LTX-2-19b-distilled --image /data00/ltx2_i2v_input.png --model-class-name LTX2ImageToVideoPipeline --prompt "A close-up shot of a young waitress in a retro 1950s diner, her warm brown eyes meeting the camera with a gentle smile. She wears a black polka-dot dress with an elegant cream lace collar, her reddish-brown hair styled in an elaborate updo with delicate curls framing her freckled face. Soft, warm light from overhead fixtures illuminates her features as she stands behind a yellow counter. The camera begins slightly to her side, then slowly pushes in toward her face, revealing the subtle rosy blush on her cheeks. In the blurred background, the soft teal walls and a glowing red \"Diner\" sign create a nostalgic atmosphere. The ambient sounds of clinking dishes, distant conversations, and the gentle hum of a jukebox fill the air. She tilts her head slightly and says in a friendly, warm voice: \"Welcome to Rosie's. What can I get for you today?\" The mood is inviting, timeless, and full of classic American diner charm." --negative-prompt "shaky, glitchy, low quality, worst quality, deformed, distorted, disfigured, motion smear, motion artifacts, fused fingers, bad anatomy, weird hand, ugly, transition, static." --guidance-scale 1.0 --width 768 --height 512 --num-inference-steps 8 --num-frames 121 --fps 24 --seed 45 --enforce-eager --output ltx2_i2v_sample.mp4

Test Result

Text-To-Video

single stage inference:

ltx2_t2v_sample.mp4

two stages inference

ltx2_t2v_2st_sample.mp4

Image-To-Video

single stage inference:

ltx2_i2v_sample.mp4

two stages inference:

ltx2_i2v_2st_sample.mp4

Benchmark

We bench it by text_to_video.py and image_to_video.py both with option '--enforce-eager' on NVIDIA H20.

Sampling Parameters:

• model: rootonchair/LTX-2-19b-distilled
• prompt: A close-up shot of a young waitress in a retro 1950s diner, her warm brown eyes meeting the camera with a gentle smile. She wears a black polka-dot dress with an elegant cream lace collar, her reddish-brown hair styled in an elaborate updo with delicate curls framing her freckled face. Soft, warm light from overhead fixtures illuminates her features as she stands behind a yellow counter. The camera begins slightly to her side, then slowly pushes in toward her face, revealing the subtle rosy blush on her cheeks. In the blurred background, the soft teal walls and a glowing red "Diner" sign create a nostalgic atmosphere. The ambient sounds of clinking dishes, distant conversations, and the gentle hum of a jukebox fill the air. She tilts her head slightly and says in a friendly, warm voice: "Welcome to Rosie's. What can I get for you today?" The mood is inviting, timeless, and full of classic American diner charm.
• width: 768
• height: 512
• guidance_scale: 1.0
• num_frames: 121
• frame_rate: 24
• seed: 45
• image(for I2V):
  

	Cache-Dit	Ulysses-SP 2	Ring Attention 2	CFG Parallel 2
Text-To-Video(LTX2TwoStatgesPipeline)	49.5590 s	31.7546s	31.4847s	51.3056 s
Image-To-Video(LTX2ImageToVideoTwoStatgesPipeline)	52.7701 s	32.0508 s	34.0860 s	52.6714 s

Future Work

• L4 E2E test for LTX-2
• Distilled LoRA support that Lightricks/LTX-2 could apply two stages inference

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
• The test results. Please paste the results comparison before and after, or the e2e results.
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
• (Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9551a1b24d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[Songrui625]

@david6666666 PTAL! Thanks!

[lishunyang12]

please open a new RFC and attach your design doc using this template in your RFC

[Songrui625]

please open a new RFC and attach your design doc using this template in your RFC

Sorry to bother you. This PR is more of a model-related implementation than a new feature. I have already changed both the PR title and commit title.

[david6666666]

I believe we need to test all acceleration methods of LTX-2 T2V and I2V to ensure functionality is unaffected. Alternatively, we could add an E2E L4 level test for monitoring, similar to #2087.

[Songrui625]

I believe we need to test all acceleration methods of LTX-2 T2V and I2V to ensure functionality is unaffected. Alternatively, we could add an E2E L4 level test for monitoring, similar to #2087.

Thanks, David, all acceleration methods regression tests (Ulysses-SP, Ring Attention, Cache-DiT and CFG-Parallel) are passed. And I have attached the benchmark results of two stages pipeline in the PR description. PTAL again.

[david6666666]

@SamitHuang help take a look, thx

[Songrui625]

@SamitHuang Hi, this PR is ready to go on. Please take a look, thanks!

[Songrui625]

@lishunyang12 @SamitHuang @wtomin This PR is ready. PTAL, thanks! 😊

[lishunyang12]

LGTM

[Songrui625]

LGTM. Can you help to create a L4 e2e test on follow pr, covering the existing diffusion features supported (See #1217). As for how to create a L4 e2e test, please refer to #1832 .

The PR about L4 e2e tests has committed. PTAL, thanks. #2815