[tzhouam]

Adds TaoMate-H3 realtime video generation with live prompt updates through /v1/realtime/video, including FP8 deployment with CUDA graphs on USP2 and BF16 deployment on USP4.

Implementation

• AR-Diffusion owns clean KV and request scheduling. Immutable segments use the shared allocator; TaoMate specifies its sink/recent-history and audio-reset policy. The step scheduler waits for interactions or host deadlines without dummy denoise steps, retaining admission and session state. Generation tracking handles early updates and timeout races; cancellation releases held sessions immediately.
• The worker owns one DiffusionLoRAManager. TaoMate's checkpoint contains ordinary low-rank A/B pairs in a native serialization format. The loader validates all 208 targets and normalizes them to LoRAModel/PEFTHelper. Fused QKV/FC1 reuse the existing single-slice wrappers, preserving one shrink and one expand per projection, BF16 factors, and epilogue alpha/r scaling. TaoMate QKV B is already in loaded projection order; FastH3's reordering and permanent fusion are not used.
• Teacher/student switching uses the shared manager. Nested exception-safe suspension keeps adapter buffers resident. Optional merged student weights remain in a separate bank, finalized after quantization and switched by reference; captured teacher weights are never modified in place. Early manager initialization occurs after final weight processing and before offload/compile, KV allocation, and warmup. Closing invalidates graphs before releasing banks and wrappers.
• Model-specific work stays in TaoMate. Teacher/student algorithms, native-format validation, AV/RoPE layout, incremental VAE state, graph management, and shadow-weight preparation remain local. Text/embedding computation reuses MiniMax. Existing CLI/YAML names, USP2/USP4, prompt updates, finite/infinite hold, fallback, and cancellation remain supported. hold_poll_seconds remains a compatibility no-op; max_num_seqs remains 1.

Review units and size

Four appended commits cover shared LoRA support, model/worker integration, redundant-state cleanup and tests, and documentation. There is no rebase or history rewrite.

The second refactor delta from babd5517c is below; this is not the full PR diff against main:

Area	Files	Added/deleted	Net
Production	15	+436/-482	-46
Tests	12	+697/-117	+580
Documentation	3	+7/-9	-2

Production also decreases by 4 non-comment/nonblank/non-docstring lines. Tests increase because native disk-to-manager integration, lifecycle failure paths, non-unit alpha, actual CUDA graph switching, and production eviction/alias paths now have coverage. The independent TaoMate LoRA hook/controller and redundant format probe are removed; lora.py decreases from 432 to 289 lines. Four scheduler state maps are consolidated into _StepProgress, and staging allocation/session cleanup are shared. Both deployment YAML files are byte-identical to the baseline.

Validation

Validated frozen candidate 5dfe4ee2e against babd5517c on the same four reserved GPUs, pinned environment, checkpoint, configurations, and seeds.

• Six baseline and six candidate real-checkpoint runs: USP2/USP4 each have three paired thirteen-window runs, including prompt changes, long prompts, cancellation and restart. All 12 paired warmup/long-stream video files are byte-identical to baseline. Every timing sample is retained.
• All four finite/indefinite hold cases pass. Existing production counters prove cached graph replay on both USP2 ranks after server readiness. In the finite case, each rank has 84 teacher replays versus two new captures, and six text replays versus two new captures; the indefinite case also proves reuse.
• 10 actual CUDA tests pass: segmented storage plus merged/unmerged teacher graph switching at alpha/r = 0.5, 1 and 1.5, stable addresses and cleanup. 53 additional GPU regressions pass, including FastH3, worker wake-up and compiled paged writes.
• Real checkpoint merged mode passes on baseline and candidate with USP4 BF16, a prompt update and a following request; both paired videos are byte-identical. This is a functional smoke, separate from the latency gate.
• CPU regression groups: 608 passed / 33 skips in the CPU-only run, 155 worker/runner/step-pipeline cases passed, and 152 final KV/runner cases passed. These groups overlap and are not summed. The CPU-only selection excludes full-model LoRA serving tests; CUDA-built vLLM uses a test-runner-only PIN_MEMORY=False setting.
• Applicable non-mypy pre-commit hooks pass. The repository-pinned changed-file mypy comparison has no new diagnostics: baseline 50, candidate 44. Both deployment YAML files remain byte-identical.

Topology	Window median change	Window P95 change	Median startup change
USP2	-0.162%	+5.142%	+4.545%
USP4	+1.374%	+1.490%	-0.002%

All gates pass: median ≤5%, pooled P95 ≤10%, startup median ≤10%, with no new OOM. The fixed summary pools 36 warm completion intervals per revision/topology (12 intervals from each thirteen-window run); startup uses the median of three runs. The largest individual-round P95 increase is 8.073% in USP2 round 2, retained in the pooled result. Memory polling started during baseline and cannot establish exact allocator peaks.

The prior NVIDIA/AMD grouped-step test failure was traced to the new AR cleanup override touching AR-only state on the generic fallback path; the fallback now delegates base cleanup and returns before accessing AR state. The prior Intel failure was a shared uv cache-lock timeout before tests.

MP4 transport in this PR is video-only. Synchronized audio/video transport is separate work in #8314, and full /v1/videos requests are in #8676; neither is included here.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/diffusion/index.md, docs/design/module/quantization.md.

Module owners: @david6666666 @Isotr0py @princepride

Routing: @david6666666 via module of the changed files, module named in the PR description, semantic router, model owner, CODEOWNERS; @Isotr0py via module of the changed files, module named in the PR description; @princepride via module of the changed files, module named in the PR description

@tzhouam, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot: superseded

The CI failure noted on 19762dbf0373 refers to an earlier head; the pull request now points at 23fe6b44d8e6.

[vllm-omni-review-bot]

Omni ReviewBot routing record

Assigned Strict under experiment vllm-omni-strict-5050-20260829.

[vllm-omni-review-bot]

Omni ReviewBot review

PR description

Adds TaoMate-H3 as a new AR-Diffusion streaming pipeline (rank-128 LoRA over MiniMax-H3) with causal four-phase generation, persistent clean AV KV, switchable LoRA student/teacher, and USP2/USP4 deploy profiles. Shared runtime changes enable step-mode async media packing and fragmented MP4 with AAC so each phase can stream video+audio; serving and quantization factory hooks support the realtime configs.

Change flow

flowchart TD
  A["[EXISTING] WS /v1/realtime/video"]:::existing
  B["[NEW] TaoMateH3Pipeline<br/>streaming phases + LoRA/KV"]:::new
  C["[CHANGED] MiniMaxH3 DiT<br/>attention_cls hook"]:::changed
  D["[CHANGED] Worker/executor<br/>step_async_output"]:::changed
  E["[CHANGED] FragmentedMP4<br/>video+AAC mux"]:::changed
  F["[NEW] USP2/USP4 deploy<br/>+ recipe knobs"]:::new
  A --> B
  C --> B
  F --> B
  B --> D
  D --> E
  classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
  classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
  classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
  classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px

Loading

CI at 372f7d2d27ac (2026-09-29T20:34:45.920441+00:00): required check(s) blocking: buildkite/vllm-omni (missing).

See inline comments below.

---

_{🤖 This review was generated by InferMatrix Copilot, an open-source repo-maintenance agent for PR review, CI debugging and issue triage. Try it on your own repo, and ⭐ star it if it helped!}

[vllm-omni-review-bot]

[P1] New CPU tests lack core_model/cpu markers, so CI will not select them

Guard step Simple · Diffusion Test in .buildkite/cuda/test-merge.yml and test-ready.yml runs pytest -sv tests/diffusion -m 'core_model and cpu'. Every new file under tests/diffusion/models/taomate_h3/ (and tests/quantization/test_online_factory.py for the Online factory path via Simple · Other Test) has no pytest.mark.core_model / pytest.mark.cpu (or equivalent). With --strict-markers, unmarked tests are deselected, so the PR's claimed 46+ CPU contract tests and online-factory tests will not run in the covering L1/L2 Simple Test steps even when Buildkite is present. Add the same marker pair used by other diffusion CPU contract tests (and keep selectors aligned with those steps).

[vllm-omni-review-bot]

[P2] Non-test diff far over review budget; needs a split plan or exemption

Supplied diff_size: non_test_added_lines=5518 vs budget 1000 (over_budget=true, 33 counted files). Largest contributors: vllm_omni/diffusion/models/taomate_h3/pipeline.py (1684), lora.py (432), audio_teacher.py (397), kv_cache.py (371), cuda_graph.py (346). The PR body describes the full model+opt stack but does not give a split plan or a concrete size exemption (e.g. upstream sync/generated). Please either split (e.g. core pipeline/KV/LoRA vs shared worker/mux/quant factory vs USP2 opt knobs) or state an explicit exemption for this landing.

[vllm-omni-review-bot]

[P2] USP2 realtime profile leaves almost no H200 memory headroom

PR/recipe state USP2 steady memory is 139–143 GiB on 144 GiB cards, and note that text_encoder: {method: fp8_per_channel} frees ~4 GiB/rank. The shipped taomate_h3_usp2_realtime.yaml keeps text_encoder: null (BF16) while enabling FP8 DiT, CUDA graphs, stack tiling, and step_async_output. That leaves roughly 1–5 GiB for transient peaks (graph capture growth, VAE decode, KV buffer growth, fragmentation). For a published realtime profile this is an OOM cliff under small workload drift. Either enable the documented text-encoder FP8 (or other reserved headroom) in the USP2 profile, or document a hard fail-closed memory gate / lower-res default before calling the config realtime-ready.

[vllm-omni-review-bot]

Omni ReviewBot: finding feedback

[p1] New CPU tests lack core_model/cpu markers, so CI will not select them — tests/diffusion/models/taomate_h3/test_taomate_h3_cpu.py:41

Guard step Simple · Diffusion Test in .buildkite/cuda/test-merge.yml and test-ready.yml runs pytest -sv tests/diffusion -m 'core_model and cpu'. Every new file under tests/diffusion/models/taomate_h3/ (and tests/quantization/test_online_factory.py for the Online factory path via Simple · Other Test) has no pytest.mark.core_model / pytest.mark.cpu (or equivalent). With --strict-markers, unmarked tests are deselected, so the PR's claimed 46+ CPU contract tests and online-factory tests will not run in the covering L1/L2 Simple Test steps even when Buildkite is present. Add the same marker pair used by other diffusion CPU contract tests (and keep selectors aligned with those steps).

This finding appears in the bot's COMMENT review, but GitHub could not place it as an inline diff comment. If you are the PR author and disagree, react 👎 to this comment. The disagreement will be shown to the maintainer; it does not approve or merge the PR.

[vllm-omni-review-bot]

Omni ReviewBot: superseded

The CI failure noted on 5dfe4ee2e671 refers to an earlier head; the pull request now points at c1c7631944f8.

[tzhouam]

Validation follow-up at c431a88ba.

@Dong1017, following up on your prompt-marker question: started_event_ids, active_event_ids, and completed_event_ids acknowledge installation of the prompt embeddings into session state. TaoMate uses an immediate embedding transition, so all three can acknowledge the update at the same boundary. The generation window snapshots its conditioning at its first phase; the remaining phases of an existing window continue using that window's prior prompt.

I documented that distinction and extended the existing shorter/longer-prompt regression: the event is acknowledged, phase index 3 still reads the old window conditioning, and phase index 4 reads the new prompt. Both parameterized cases pass. This checks the state/conditioning boundary; the markers do not specify the first changed pixel or an exact media timestamp because the VAE retains temporal context and blends overlap.

Other review follow-ups:

• CPU CI selection: 104 TaoMate CPU cases selected; combined validation with three KV admission cases and one online-quantization case passed 108 tests.
• USP2 configuration constructor: actual YAML constructs successfully with the current factory and pinned vLLM 0.31.0. This is not a fresh full-model startup claim.
• Unsupported references: 10 rejection cases pass before encoding or session replacement.
• Size: the complete PR remains large; no size exemption is claimed.
• Memory: later historical sampled measurements are scoped to 5dfe4ee2e; current-head startup, graph/long-stream memory and USP2/USP4 runs are queued. This concern remains open.

Applicable local pre-commit checks, including documentation lint and the repository's changed-file mypy hook, pass. The added commit changes only documentation and regression coverage; production remains at 6479200be. GPU results from earlier revisions are not being presented as current-head verification.

[vllm-omni-review-bot]

Omni ReviewBot: superseded

The CI failure noted on c431a88ba31c refers to an earlier head; the pull request now points at 010e89781212.

[princepride]

LGTM