test: define Wan fused BF16 parity budget

[andyluo7]

Summary

• keep exact parity for the non-fused path and FP32 fused path
• validate the BF16 fused path with explicit maximum-error and RMS-error budgets
• retain the same checks for both local and two-rank sharded decoding

Root cause

FusedWanRMSNormSiLU returns BF16 activations immediately, while the eager Wan norm keeps its affine and SiLU result in FP32 until the following convolution rounds it. Requiring bit identity, or a generic elementwise relative tolerance near zero, treats that intentional precision boundary as a regression.

Across repeated AMD failures the observed maximum absolute errors were 0.0195 and 0.0273. The new contract caps the worst output error at four BF16 epsilons (0.03125) and the whole-output RMS error at one BF16 epsilon (0.0078125). Decoder outputs are tanh-bounded to [-1, 1], so these absolute limits are directly interpretable.

Examples:

• https://buildkite.com/vllm/vllm-omni-amd-ci/builds/12825#01a0dc12-13d6-470a-a366-fbfa0d24382c
• https://buildkite.com/vllm/vllm-omni-amd-ci/builds/12825#01a0dc12-13d7-4a60-8e63-7a2a90952f5a
• https://buildkite.com/vllm/vllm-omni-amd-ci/builds/12825#01a0dc12-13d7-4d87-bdd7-36aaa19282a0

Validation

• python3 -m py_compile tests/diffusion/distributed/test_wan_decoder_fast_path.py
• git diff --check origin/main...HEAD
• exact head b34e9bfb5e45648b7f92d1031ab66372f2035db0, synchronized with current main after test: keep Hunyuan LoRA CPU path on native linear #8166:
  • pre-commit/Ruff, Python 3.11 and 3.12 wheel builds, DCO, and docs passed
  • generic Buildkite #16263 passed
  • Intel Buildkite #9220 passed
  • NPU Buildkite #7984 passed
  • AMD Buildkite #12898: all three BF16-sensitive Wan cases and the dedicated Wan22 job passed; the previous Hunyuan LoRA failures are gone
  • the AMD aggregate is red only for an unrelated hard Ulysses one-GPU routing failure fixed by ci: route Ulysses UAA test to multi-GPU lanes #8165; five unrelated Breeze failures in R2-01 are quarantined/soft-failed and addressed by test: honor Breeze eager RNG fallback on ROCm #8190

Signed-off-by: andyluo7 andy.luo@amd.com

[andyluo7]

Self-review: I checked the current 6f48d7d7 patch against current main. Exact parity is still required for the exact path, and FP32 fused parity remains tight. Only the BF16 fused path uses the explicit error budget, matching its intentional extra cast boundary; both max-error and RMS-error assertions prevent a broad tolerance from hiding widespread drift. py_compile and git diff --check pass. Exact-head AMD runtime validation is requested.

[vllm-omni-review-bot]

This PR appears to be related to model: Wan.

Model owners: @bjf-frz @NickCao @yenuo26

Routing: @bjf-frz via semantic router, model owner; @NickCao via CODEOWNERS; @yenuo26 via CODEOWNERS

@andyluo7, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit b34e9bfb5e45 produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[vllm-omni-review-bot]

Omni ReviewBot: superseded

The CI failure noted on d37f77f62950 refers to an earlier head; the pull request now points at b34e9bfb5e45.

[vllm-omni-review-bot]

Omni ReviewBot routing record

Assigned Strict under experiment vllm-omni-strict-5050-20260829.

[vllm-omni-review-bot]

Omni ReviewBot review

PR description

This pull request changes only the Wan decoder CPU fast-path test. The exact streaming path still has to match the untouched decoder bit-for-bit, and the fused path in FP32 still uses a tight assert_close. Under BF16 autocast, fused outputs are now judged by a pair of absolute budgets: worst-element error at four BF16 epsilons and whole-tensor RMS error at one BF16 epsilon, instead of a generic allclose. The same fused check is used for the local decoder and the two-rank gloo shard.

Change flow

flowchart TD
  A["[EXISTING] WanStreamingDecoder CPU chunks"]:::existing
  B["[EXISTING] install_wan_decoder_fast_path exact/fused"]:::existing
  C["[CHANGED] _check_pair local and 2-rank compare"]:::changed
  D["[NEW] _assert_fused_output_close BF16 max+RMS budget"]:::new
  E["[EXISTING] exact torch.equal; FP32 fused assert_close"]:::existing
  A --> B --> C
  C -->|"fused bf16"| D
  C -->|"exact / fused fp32"| E
  classDef existing fill:#e5e7eb,stroke:#6b7280,color:#111827
  classDef changed fill:#fef3c7,stroke:#d97706,color:#451a03,stroke-width:2px
  classDef new fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:2px
  classDef removed fill:#fee2e2,stroke:#dc2626,color:#450a0a,stroke-width:2px

Loading

CI at d37f77f62950 (2026-09-27T20:39:55.130997+00:00): required check(s) blocking: buildkite/vllm-omni (failed), and -5 more. Observed Buildkite: buildkite/vllm-omni-amd-ci (failed), buildkite/vllm-omni-npu-ci (passed), buildkite/vllm-omni (failed), and 1 more.

No actionable findings.

[hsliuustc0106]

CI failed

[andyluo7]

Fresh exact-head CI after syncing with current main (including merged #8166): b34e9bfb5e45648b7f92d1031ab66372f2035db0.

Wan BF16 validation passed on AMD build #12898:

• test_fused_level_plumbing_on_cpu[True-RMSNormVAE] passed in Simple Diffusion shard 1.
• test_fused_level_plumbing_on_cpu[True-default] passed in shard 2.
• test_sharded_fast_path_on_every_rank[True-fused] passed in shard 3.
• The dedicated Diffusion · Wan22 Test passed.
• The previous Hunyuan LoRA TP=1/2/4 failures are gone after test: keep Hunyuan LoRA CPU path on native linear #8166.

The AMD aggregate is red only for failures outside this one-file Wan test diff:

• Hard failure: test_ulysses_uaa_2d_mask_layout_matches_baseline[global] ran on an mi300_1 worker and rank 1 failed with invalid device ordinal; this is the routing issue fixed by ci: route Ulysses UAA test to multi-GPU lanes #8165.
• Quarantined/soft failure (will_fail=false): five Breeze graph-expectation tests in R2-01; these are addressed by test: honor Breeze eager RNG fallback on ROCm #8190.

All other required checks passed: generic Buildkite #16263, Intel #9220, NPU #7984, pre-commit/Ruff, both wheel builds, DCO, and docs.

There are no failures attributable to #8188. The remaining gates are the known #8165 baseline/routing dependency and maintainer approval.

[yenuo26]

LGTM