[Model] Add HSDP support for LTX-2

[fywc]

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

Add LTX-2 HSDP support.

This PR resolves the LTX-2 gap in HSDP support from RFC #1217 by:

• defining shard conditions for LTX2VideoTransformer3DModel so HSDP can shard repeated transformer blocks
• integrating LTX-2 into the existing FSDP2-based HSDP flow without changing HSDP core logic
• updating tests and documentation to reflect LTX-2 HSDP support

Test Plan

Test Result

INFO 04-18 07:58:31 [diffusers_loader.py:324] Loading weights took 2.77 seconds
INFO 04-18 07:58:31 [hsdp.py:128] HSDP Inference: replicate_size=1, shard_size=2, world_size=2, rank=0, fs_world_size=2, fs_rank=0
INFO 04-18 07:58:34 [platform.py:77] Defaulting to diffusion attention backend FLASH_ATTN
INFO 04-18 07:58:37 [hsdp.py:128] HSDP Inference: replicate_size=1, shard_size=2, world_size=2, rank=1, fs_world_size=2, fs_rank=1
INFO 04-18 07:58:39 [hsdp.py:202] Sharded 912 modules + root
INFO 04-18 07:58:39 [hsdp.py:173] HSDP applied to model: FSDPLTX2VideoTransformer3DModel
INFO 04-18 07:58:40 [diffusion_model_runner.py:142] Model loading took 45.5504 GiB and 23.878847 seconds
INFO 04-18 07:58:40 [diffusion_model_runner.py:147] Model runner: Model loaded successfully.
INFO 04-18 07:58:40 [diffusion_model_runner.py:188] Model runner: Initialization complete.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
• The test results. Please paste the results comparison before and after, or the e2e results.
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
• (Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[hsliuustc0106]

can you profile with HSDP and check whether there are some free bubble in the first a few blocks with FSDP on? We notice some similar issues with WAN 2.2

[gcanlin]

Only one issue needs to be fixed. Otherwise LGTM.

[hsliuustc0106]

Well-structured PR adding HSDP support to LTX-2. Good test coverage and documentation.

[fywc]

can you profile with HSDP and check whether there are some free bubble in the first a few blocks with FSDP on? We notice some similar issues with WAN 2.2

I’ll profile LTX-2 with HSDP enabled for this problem as soon as possible, is there any related issues already?

[fywc]

can you profile with HSDP and check whether there are some free bubble in the first a few blocks with FSDP on? We notice some similar issues with WAN 2.2

I’ll profile LTX-2 with HSDP enabled for this problem as soon as possible, is there any related issues already?

Hi，I did a profiling analysis on LTX-2 to investigate this problem, not sure if this is helpful or matches what you have observed with WAN2.2 ?

Test environment:

• 2 × NVIDIA H200 GPU
• HSDP shard size = 2

Test Flow：

• Initialize the LTX-2 model with & without HSDP.
• Send one request to warmup the model.
• Profile the next one request and analysis data.

Key findings:

Enabling HSDP introduces a noticeable latency increase at the block level:

• Per-block latency increases from ~10 ms → ~27 ms (~2.5–3x).
• The overhead is consistent across all transformer blocks, not a one-off effect.

Breaking down the block:

• Both Attention and FFN stages are slower (~2–3x).
• The main overhead comes from RowParallelLinear / ColumnParallel ops, each increasing from ~100 µs → ~1.2–1.4 ms.

According to the stream timeline :

• HSDP introduces additional NCCL all_gather streams.
• However, these comm streams are not effectively overlapped with compute, leading to visible pipeline bubbles.

As a result, Maybe the free bubbles are mainly due to lack of comm/compute overlap, rather than compute inefficiency itself.

PS:
2 snapshots showing timeline with one block:

• without HSDP

• with HSDP

[hsliuustc0106]

can you profile with HSDP and check whether there are some free bubble in the first a few blocks with FSDP on? We notice some similar issues with WAN 2.2

I’ll profile LTX-2 with HSDP enabled for this problem as soon as possible, is there any related issues already?

Hi，I did a profiling analysis on LTX-2 to investigate this problem, not sure if this is helpful or matches what you have observed with WAN2.2 ?

Test environment:

• 2 × NVIDIA H200 GPU
• HSDP shard size = 2

Test Flow：

• Initialize the LTX-2 model with & without HSDP.
• Send one request to warmup the model.
• Profile the next one request and analysis data.

Key findings:

Enabling HSDP introduces a noticeable latency increase at the block level:

• Per-block latency increases from ~10 ms → ~27 ms (~2.5–3x).
• The overhead is consistent across all transformer blocks, not a one-off effect.

Breaking down the block:

• Both Attention and FFN stages are slower (~2–3x).
• The main overhead comes from RowParallelLinear / ColumnParallel ops, each increasing from ~100 µs → ~1.2–1.4 ms.

According to the stream timeline :

• HSDP introduces additional NCCL all_gather streams.
• However, these comm streams are not effectively overlapped with compute, leading to visible pipeline bubbles.

As a result, Maybe the free bubbles are mainly due to lack of comm/compute overlap, rather than compute inefficiency itself.

PS: 2 snapshots showing timeline with one block:

• without HSDP

* with HSDP

do you mean HSDP introduces negative latency impact for LTX-2?

[hsliuustc0106]

how about the peak vram consumption?

[fywc]

how about the peak vram consumption?

Peak vram remains the same with HSDP enabled, no noticeable difference observed.

[fywc]

can you profile with HSDP and check whether there are some free bubble in the first a few blocks with FSDP on? We notice some similar issues with WAN 2.2

I’ll profile LTX-2 with HSDP enabled for this problem as soon as possible, is there any related issues already?

Hi，I did a profiling analysis on LTX-2 to investigate this problem, not sure if this is helpful or matches what you have observed with WAN2.2 ?
Test environment:

• 2 × NVIDIA H200 GPU
• HSDP shard size = 2

Test Flow：

• Initialize the LTX-2 model with & without HSDP.
• Send one request to warmup the model.
• Profile the next one request and analysis data.

Key findings:
Enabling HSDP introduces a noticeable latency increase at the block level:

• Per-block latency increases from ~10 ms → ~27 ms (~2.5–3x).
• The overhead is consistent across all transformer blocks, not a one-off effect.

Breaking down the block:

• Both Attention and FFN stages are slower (~2–3x).
• The main overhead comes from RowParallelLinear / ColumnParallel ops, each increasing from ~100 µs → ~1.2–1.4 ms.

According to the stream timeline :

• HSDP introduces additional NCCL all_gather streams.
• However, these comm streams are not effectively overlapped with compute, leading to visible pipeline bubbles.

As a result, Maybe the free bubbles are mainly due to lack of comm/compute overlap, rather than compute inefficiency itself.
PS: 2 snapshots showing timeline with one block:

• without HSDP

* with HSDP

do you mean HSDP introduces negative latency impact for LTX-2?

I think so. Possibly similar behavior may appear on other models as well, I can follow up with additional tests and analysis.

[fywc]

cc @hsliuustc0106 @gcanlin

[hsliuustc0106]

thanks, I hope you can profile higher resolution inputs which HSDP may benefit more