[mglyn]

Summary

Repeated LTX vocoder calls with identical mel inputs can produce different audio when cuDNN selects nondeterministic convolution algorithms. Temporarily enable cudnn.deterministic during CUDA vocoder execution and restore its previous value on both success and failure. Other cuDNN settings and the existing BWE FP32 handling are preserved.

Apply the same setting to the pinned official BWE vocoder used by the accuracy reference runner. This change is independent of DiffVAE tiling and operator optimizations.

Validation

Similarity failure motivating this fix

The eager NVIDIA H200 run of test_ltx25_distilled_two_stage_matches_official (task=t2v) failed only the audio relative-L2 threshold.

Metric	Failing run	Unfixed rerun	Deterministic vocoders	Acceptance
Audio relative-L2	0.10105968 (FAIL)	0.01237781	0.00607631	<= 0.10
Audio cosine	0.99490147	0.99992417	0.99998216	>= 0.99
Audio max absolute error	0.00053406	0.00004160	0.00002670	Diagnostic only

The deterministic distilled-T2V case was run independently twice: the initial targeted validation and the subsequent full six-case rerun. Both produced exactly audio relative-L2 = 0.006076306110274279 and audio cosine = 0.9999821597405034. SHA-256 checks also match across runs for the official audio artifacts and, separately, for the Omni audio artifacts. Thus the repeated runs reproduce identical audio outputs within each implementation, not merely the same rounded metric; the remaining official-versus-Omni difference is stable in these runs.

Passing an unfixed rerun did not resolve the instability. Subsequent fixed-mel probes isolated non-bitwise output in both the Diffusers and pinned-official vocoders without CUDA RNG consumption. Enabling cuDNN determinism yielded bitwise-equal repeated outputs within and across two fresh processes for each implementation.

These are historical investigation results, not a fresh similarity run of this PR head.

Current branch checks

• 25 existing CPU VAE tests and 4 direct deterministic-context tests passed.
• The new tests check that the flag is enabled inside the context, restored to either initial value after normal or exceptional exit, and that other cuDNN settings are preserved. They do not simulate a vocoder or claim to reproduce numerical nondeterminism.
• The pre-existing CUDA vocoder precision test is unchanged.
• Ruff, formatting and diff checks passed. The new CPU cases are collected by the existing Simple · Diffusion Test CI job.
• Actual audio determinism is supported by the similarity and fixed-mel experiments above.

cuDNN policy remains process-wide; the temporary setting assumes serialized model execution within a worker.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/diffusion/diffusion_model_integration.md.

Module owners: @fhfuih @Bounty-hunter @wtomin @RuixiangMa

@mglyn, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.