[WIP][Feature] Pipeline Parallelism & Stream Batch for Real-Time Video (#2280) by mnasser02 · Pull Request #2 · zzhang-fr/vllm-omni

mnasser02 · 2026-04-16T05:58:30Z

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

Implements vllm-project#2280: depth-partitioned pipeline parallelism + Stream Batch temporal scheduling for streaming video diffusion. Targets 40+ FPS on 4×H100, sub-500 ms TTFF.

Stacks on vllm-project#2322.

Phase	Deliverable	Status
1	`SupportsStepExecution` on `Wan22Pipeline`	✅
2	`StreamBatchScheduler` (warmup/steady/cooldown) driving micro-step execution pipeline	🔲
3	SLO-adaptive step count + config	🔲
4	`StreamVAE` with chunk-wise decode + 3D conv caching	🔲
5	CFG parallel, `DistributedKVCache` stub (vllm-project#1987), VAE-as-stage (vllm-project#2089)	🔲
6	Benchmarks vs SP baseline, PP×SP hybrid, docs	🔲

Test Plan

Test Result

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan. Please provide the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
The test results. Please paste the results comparison before and after, or the e2e results.
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
(Optional) Release notes update. If your change is user-facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

Signed-off-by: Rustam Khadipash <16683750+hadipash@users.noreply.github.com>

Signed-off-by: Rustam Khadipash <16683750+hadipash@users.noreply.github.com> # Conflicts: # docs/user_guide/diffusion_features.md

Signed-off-by: Rustam Khadipash <16683750+hadipash@users.noreply.github.com>

Signed-off-by: Rustam Khadipash <16683750+hadipash@users.noreply.github.com> # Conflicts: # vllm_omni/diffusion/models/wan2_2/pipeline_wan2_2_ti2v.py # vllm_omni/diffusion/models/wan2_2/wan2_2_transformer.py

…#2524) Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>

…llm-project#2690) Signed-off-by: Sy03 <1370724210@qq.com> Signed-off-by: Yueqian Lin <linyueqian@outlook.com> Co-authored-by: Yueqian Lin <linyueqian@outlook.com> Co-authored-by: Yueqian Lin <70319226+linyueqian@users.noreply.github.com>

…ject#2750) Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>

Signed-off-by: Rustam Khadipash <16683750+hadipash@users.noreply.github.com> # Conflicts: # docs/user_guide/diffusion_features.md

Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>

Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>

…t#2735) Signed-off-by: samithuang <285365963@qq.com>

…e single-GPU performance (vllm-project#2604) Signed-off-by: Ian Carrasco <ian.carrasco@baseten.co>

Signed-off-by: Zhang <jianmusings@gmail.com> Signed-off-by: Zhang Jian <jianmusings@gmail.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…-N-senders to support Bagel TP/CFG parallel (vllm-project#2731) Signed-off-by: natureofnature <wzliu@connect.hku.hk>

Signed-off-by: rongfu.leng <lenronfu@gmail.com>

…llm-project#2598) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>

Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>

Signed-off-by: neptune <neptune@hust.edu.cn> Co-authored-by: neptune <neptune@hust.edu.cn>

Signed-off-by: wangyu <410167048@qq.com>

Signed-off-by: Alex Brooks <albrooks@redhat.com>

…ed task Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

… pipeline (B and T hardcoded for now) Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

…instead of sync send/recv Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Plain P2P on size-2 PG triggers lazy sub-comm creation that requires the peer present. Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

- Separate comms stream - Double buffering - Set rcv buffers for a new req - Revert changes regarding Async structs Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

hadipash and others added 30 commits April 10, 2026 15:40

add pipeline parallel to Wan 2.2

a25e9a1

Signed-off-by: Rustam Khadipash <16683750+hadipash@users.noreply.github.com>

add async communication for last -> first rank

70fb785

Signed-off-by: Rustam Khadipash <16683750+hadipash@users.noreply.github.com>

bug fix

ea5aa03

Signed-off-by: Rustam Khadipash <16683750+hadipash@users.noreply.github.com>

bug fix

95c9db8

Signed-off-by: Rustam Khadipash <16683750+hadipash@users.noreply.github.com>

align isend with vLLM's implementation

8a62204

Signed-off-by: Rustam Khadipash <16683750+hadipash@users.noreply.github.com>

add CFG parallel support

79a83db

Signed-off-by: Rustam Khadipash <16683750+hadipash@users.noreply.github.com>

add unit tests and docs

0db0f8f

Signed-off-by: Rustam Khadipash <16683750+hadipash@users.noreply.github.com>

add online serving

7244874

Signed-off-by: Rustam Khadipash <16683750+hadipash@users.noreply.github.com>

remove unused pipeline

eb9d9ac

Signed-off-by: Rustam Khadipash <16683750+hadipash@users.noreply.github.com>

Merge branch 'main' into wan_pipe_parallel

eb1190b

Signed-off-by: Rustam Khadipash <16683750+hadipash@users.noreply.github.com> # Conflicts: # docs/user_guide/diffusion_features.md

add safeguard

154a4db

Signed-off-by: Rustam Khadipash <16683750+hadipash@users.noreply.github.com>

align UT with other tests

8c394da

Signed-off-by: Rustam Khadipash <16683750+hadipash@users.noreply.github.com>

bug fixes and align UT with other tests

2be4cc9

Signed-off-by: Rustam Khadipash <16683750+hadipash@users.noreply.github.com>

Merge branch 'main' into wan_pipe_parallel

169ad84

Signed-off-by: Rustam Khadipash <16683750+hadipash@users.noreply.github.com> # Conflicts: # vllm_omni/diffusion/models/wan2_2/pipeline_wan2_2_ti2v.py # vllm_omni/diffusion/models/wan2_2/wan2_2_transformer.py

[Voxtral TTS] Correct decode steps param in Voxtral TTS (vllm-project…

c9e2e3e

…#2524) Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>

[Voxtral TTS] Fix Voxtral TTS input with text and ref_audio (vllm-pro…

dd13891

…ject#2750) Signed-off-by: Chen-Yo Sun <chenyo.sun@mistral.ai>

Merge branch 'main' into wan_pipe_parallel

46ac762

Signed-off-by: Rustam Khadipash <16683750+hadipash@users.noreply.github.com> # Conflicts: # docs/user_guide/diffusion_features.md

[CI] Qwen image edit performance benckmark (vllm-project#2216)

8d23549

Signed-off-by: Huang, Zeyu <11222265+fhfuih@users.noreply.github.com>

[BugFix] Remove stage_configs_path validation (vllm-project#2741)

a5b38b5

Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>

[Perf] Optimize MP4 encoding latency in video generation (vllm-projec…

644edac

…t#2735) Signed-off-by: samithuang <285365963@qq.com>

[Qwen3-TTS] Remove hardcoded distributed_executor_backend to improv…

48c30bc

…e single-GPU performance (vllm-project#2604) Signed-off-by: Ian Carrasco <ian.carrasco@baseten.co>

[Test] Add Stable Audio offline e2e TeaCache Test (vllm-project#2377)

17acd05

Signed-off-by: Zhang <jianmusings@gmail.com> Signed-off-by: Zhang Jian <jianmusings@gmail.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

[Omni Connector] Omni Transfer Engine Connector: Enable 1-receiver-to…

6d01a8b

…-N-senders to support Bagel TP/CFG parallel (vllm-project#2731) Signed-off-by: natureofnature <wzliu@connect.hku.hk>

[skip ci] fix docs, gdown remove --id param (vllm-project#2787)

3229bae

Signed-off-by: rongfu.leng <lenronfu@gmail.com>

[Tests][Qwen3-Omni]Add test cases for long videos and long audios. (v…

159d655

…llm-project#2598) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>

[skip ci]add skills (vllm-project#2710)

f87674a

Signed-off-by: hsliuustc0106 <liuhongsheng4@huawei.com>

[Misc] clean Temporary CI Configs (vllm-project#2784)

bcd5f16

Signed-off-by: neptune <neptune@hust.edu.cn> Co-authored-by: neptune <neptune@hust.edu.cn>

[CI][Bugfix] Update thresholds for accuracy tests (vllm-project#2725)

5ce0a43

Signed-off-by: wangyu <410167048@qq.com>

[CI/BugFix] Fix Flaky Test for Qwen Omni Perf (vllm-project#2754)

cf1fcd5

Signed-off-by: Alex Brooks <albrooks@redhat.com>

mnasser02 added 4 commits April 24, 2026 09:04

Add execute_micro_step execution path that works on a per-rank assign…

5716eb6

…ed task Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Add StreamBatchScheduler that controls the flow of chunks through the…

9c08c56

… pipeline (B and T hardcoded for now) Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Add StreamBatchScheduler tests

283ca17

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Add micro-step execution pipeline tests

0084f89

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

mnasser02 force-pushed the stream-diffusion branch from 9967162 to 1d341e8 Compare April 24, 2026 07:07

mnasser02 added 18 commits April 24, 2026 10:40

Add stream_batch arg

def46ff

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Set reply rank to 0

255263b

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Fix blocking send/recv resulting in deadlock by registering metadata …

528f3be

…instead of sync send/recv Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

bugfix

5bc3c78

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

bugfix

f0af7a9

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

bugfix

017abee

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Implement fully async isend/recv_dicts and migrate PPMixin to use them

1a9b42c

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Rmv stale code

a9745ed

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

bugfix

a81b233

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Make sending dict schema unblocking

cdac8ec

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Add warmup for nccl comms on init

958d020

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Use batch_isend_irecv instead of plain isend/recv

aaa0609

Plain P2P on size-2 PG triggers lazy sub-comm creation that requires the peer present. Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Handle tensor receive as a task

462ffe9

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Chunk-aware buffers

cfd1cd3

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

fix

87059ee

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Improve and fix PP communication.

05f5020

- Separate comms stream - Double buffering - Set rcv buffers for a new req - Revert changes regarding Async structs Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

bugfixes

07842fd

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

Remove stale tests

4c8e2b7

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

mnasser02 force-pushed the stream-diffusion branch from 1d341e8 to 4c8e2b7 Compare April 24, 2026 08:40

mnasser02 added 4 commits April 27, 2026 13:52

Add unit tests for StreamBatchScheduler

98145a1

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

add SupportsMicroStepExecution

dac11a9

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

add unit tests for micro-step execution pipeline

7e4be1d

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

edit

b34cf19

Signed-off-by: Mahdi Nasser <94046147+mnasser02@users.noreply.github.com>

mnasser02 force-pushed the stream-diffusion branch from 8a0f16b to b34cf19 Compare April 27, 2026 11:52

mnasser02 closed this Apr 27, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP][Feature] Pipeline Parallelism & Stream Batch for Real-Time Video (#2280)#2

[WIP][Feature] Pipeline Parallelism & Stream Batch for Real-Time Video (#2280)#2
mnasser02 wants to merge 155 commits into
mainfrom
stream-diffusion

mnasser02 commented Apr 16, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

mnasser02 commented Apr 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

mnasser02 commented Apr 16, 2026 •

edited

Loading