[Feature] Add official LTX multimodal guidance and precision guard

[mglyn]

Purpose

This PR builds on the shared LTX-2/LTX-2.3 runtime at #5147. It adds official one-stage multimodal guidance, aligns inference with the Lightricks reference, and replaces the Diffusers-based LTX-2.3 accuracy check with a pinned official reference guard.

Follow-up two-stage/HQ support, and subsequent LTX runtime work are tracked continuously in #4985.

LTX2Pipeline is now the unified one-stage entry for LTX-2/LTX-2.3 T2V and I2V: checkpoint metadata selects the model version, while image= selects I2V. The former one-stage registry names LTX2ImageToVideoPipeline, LTX23Pipeline, and LTX23ImageToVideoPipeline are intentionally removed. The distilled two-stage path is now exposed as LTX2DistilledPipeline, replacing the legacy two-stage names; the regular LTX2TwoStagePipeline is reserved for a follow-up PR. The LTX-2 recipe documents the keyword-only API and registry-name migrations.

The official recipe uses up to four Transformer passes per denoise step: positive (cond), negative CFG (uncond), STG perturbation (ptb), and cross-modality perturbation (mod). This intentional compute increase replaces the previous partial guidance behavior.

Model	Steps	Video CFG	Audio CFG	STG	Modality	Rescale	STG block
LTX-2	40	3.0	7.0	1.0	3.0	0.7	29
LTX-2.3	30	3.0	7.0	1.0	3.0	0.7	28

It also provides independent video/audio guidance controls, request-level custom sigma schedules, shared connector/denoiser attention selection, validation for incompatible request batches and CFG parallelism, consolidated LTX documentation, and a raw video/audio accuracy guard against pinned source and model revisions.

The distilled-only two-stage path remains positive-only. Full two-stage/HQ guidance is a separate follow-up.

Test Plan

• Run the consolidated LTX suite and adjacent DMD2, model-extra, and attention regressions.
• Run changed-file pre-commit hooks and git diff --check.
• Compare raw Omni video/audio outputs with the pinned official pipeline using the same checkpoint, inputs, guidance batch, and SDPA FlashAttention backend.
• Run official-resolution LTX-2/LTX-2.3 T2V/I2V E2E generation with full guidance and CFG-only guidance on NVIDIA A100 80GB.

vLLM Version: 0.25.0

vLLM-Omni Commit: ebe35d5d86093d5ea52ae2d6e1f2bd9ca42e226a

Test Result

LTX + DMD2 + model extras + attention config: 285 passed

Numerical behavior changed to match the official pipeline

Area	Alignment
Guidance	Combine CFG, STG, modality guidance, and rescale in official x0 space
Latents	Sample directly in packed token space
Conditioning	Batch positive/negative Gemma encoding and preserve official connector ordering
Denoising	Use official sigma schedule, per-token timesteps, and independent video/audio sigmas
Scheduler	Preserve official fp32 conversion and Euler arithmetic order
Attention	Match connector processing, RoPE dtype order, audio Q/K/V, RMSNorm, and cross-modal gates
Padding	Drop the connector mask only after learned registers replace padding tokens

These are pipeline behavior fixes, not tolerance adjustments. Packed-token sampling changes the random-number layout, while separate positive/negative Gemma forwards do not reproduce the official batched conditioning output.

Official precision guard

The old guard compared against Diffusers. The new guard runs a pinned Lightricks checkout and compares raw video/audio tensors, avoiding MP4 writer and codec differences.

• Official source: 9377758131b1ffde4b7f766804590a6617bf2ab9.
• LTX-2 and LTX-2.3 model/checkpoint revisions are pinned.
• Prompts, seed, shape, steps, and one-stage guidance are identical.
• Both runtimes use PyTorch SDPA FlashAttention.
• Official max_batch_size=4 matches Omni's fused four-pass batch.

Output	Metric	Threshold
Video	Mean/minimum SSIM	>= 0.95 / 0.90
Video	Mean PSNR	>= 30.0 dB
Audio	Relative L2	<= 0.20
Audio	Cosine similarity	>= 0.95

Local single-L20 probe (512x384, 25 frames, 20 steps, full guidance):

Metric	Result
Video mean/minimum SSIM	0.999543 / 0.999390
Video mean PSNR	58.32 dB
Audio relative L2	0.02758
Audio cosine similarity	0.999622

Official-resolution E2E result videos

Configuration: single NVIDIA L20, layerwise-offload, eager execution, 768x512, 121 frames, 24 fps, seed 42, LTX-2 40 steps, and LTX-2.3 30 steps. Full guidance uses the official one-stage recipe.

Prompt:

A large orbital rocket ignites and lifts off vertically from the launch pad, powerful bright exhaust flames and dense billowing smoke expanding below, the rocket rises steadily into the dark blue sky, fixed camera, realistic cinematic footage, continuous natural motion, highly detailed

LTX-2 T2V

req_0000.mp4

LTX-2 I2V

req_0000.mp4

LTX-2.3 T2V

req_0000.mp4

LTX-2.3 I2V

req_0000.mp4

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[david6666666]

LGTM now