[fusheng-ji]

Purpose and status

Add experimental stage-local TeaCache integration for real pretrained Wan2.1-T2V-1.3B, including PP=2 and CFG=2. Production acceptance has failed so far; keep this PR in draft. No qualified default cache profile is enabled. This branch includes dependencies #8475 and #8483.

Native forward and the TeaCache extractor share input preparation, local block execution, and output projection. Each PP stage caches only its own block residual; cache hits retain the required PP input/output handling and final projection. PP prediction uses #8483's explicit CFG branch helper. Branch state resets between requests, and no cache-decision collective is added between sequential PP stages.

The supported experimental scope is BF16, single-expert T2V 1.3B, PP/CFG sizes 1 or 2, TP=SP=1, without quantization. Unsupported variants are rejected; explicit coefficients are required through the existing cache configuration. The branch also fixes a pre-existing PP VAE non-owner (None,) output crash, reproduced on the #8475 baseline with CPU and pretrained GPU runs.

Current-head update (b47ded2b9)

Stage-specific coefficients, negative-prediction clamping, calibrated-input-range guards and warmup policy now run in the production TeaCache hook through optional cache_config.wan_profile. Profiles validate the pinned Wan revision, BF16, PP/CFG topology and stage. The validation harness no longer replaces cache decisions; forced full compute remains test-only. Formal paired timing uses a separate minimal-observation run of the same profile, and independent acceptance is blocked unless calibration passes every unchanged gate. Calibration includes the historical 30-step warmup strategy, with a newly frozen independent acceptance prompt set.

• Exact current SHA b47ded2b94f7a4f7e0b80216ccf12844c0ffd2ff: focused profile/config/guard/acceptance-gate CPU suite 25 passed, 0 failed, 0 errors, 0 skipped (4 GPU cases deselected).
• Local pre-commit checks pass with mypy-3.10 skipped as in CI. Two validation JSON EOF violations were fixed. All commits on the branch now carry a Signed-off-by trailer (history rewritten with identical tree), and the DCO check passes.
• Broader CPU cache diagnostic comparison: candidate 222 passed / 8 failed / 17 errors, baseline f4eda6347 208 passed / 8 failed / 17 errors. Failure identities and error messages match (CPU/CUDA dependencies and missing cpu_linear). The candidate worktree was edited during that run, so it is not exact-current-SHA acceptance evidence; the 25-case suite above was rerun on an immutable current-SHA checkout.
• No GPU suite has run on this head yet. Overflow jobs 1105534–1105542 are pending: hook/full-model equivalence, four topology calibration campaigns, then four conditional acceptance campaigns. All four calibration jobs must succeed before independent acceptance starts. Queueing, dependency waits and timeouts do not count as passes.
• Detailed GPU cost profiling, optimization, visual inspection and all four production quality/performance gates remain outstanding. This PR remains draft/experimental, with no production-qualified default profile.

Historical evidence (not current-head GPU validation)

Protocol and reproduction · Results, profiles, JUnit and compressed logs/rank traces

• Original [Bugfix][Diffusion] Keep TeaCache CFG branches separate in the shared CFG helper #8475 SHA 7e88fc3e: four B200, PP2/CFG2, 4 passed / 0 failed / 0 skipped, including numerical prediction/scheduler alignment and CFG on → off → on flags on all four ranks. Results on #8475.
• Feature regression suites: CFG/TeaCache CPU 34 passed; PP CPU 29 passed; Wan pipeline CPU 22 passed; latest variant guards 4 passed; actual small Wan hook FP32/BF16 full/reuse 4 passed; latest PP explicit-branch regression 1 passed. These overlap and were run at different implementation stages; they are not a count of unique tests on the final reporting commit.
• Real pretrained, mounted-hook/full-compute smoke equivalence passed in all four PP/CFG topologies: maximum decoded pixel error 0. These runs use 256×256, 5 frames, 6 steps and are not quality acceptance.
• Full calibration uses 24 prompts × 2 seeds, 512×512, 17 frames, 50 steps, guidance 5.0. Both PP topologies collected 4,704 transition pairs per stage. All 18 completed threshold/warmup candidates failed at least one gate. Frozen diagnostic profiles were selected using calibration data only.
• Independent evaluation runtime is frozen at ef6379755f1f12a64a41336efa70b62e43b933f6. Subsequent coefficient-collection changes and the main-branch merge have not been validated by these GPU runs; results must not be attributed to the current PR head.

Independent evaluation	Paired videos	Mean / worst SSIM	Temporal error	Cached/native latency	Result
PP1/CFG1, job 1096032	36	0.988863 / 0.971094	0.163825	1.027440	2 gates pass, 2 fail
PP1/CFG2, job 1096033	36	0.988863 / 0.971094	0.163825	1.044654	2 gates pass, 2 fail
PP2/CFG1, retry job 1096862	36	0.974147 / 0.938729	0.250010	1.066345	2 gates pass, 2 fail
PP2/CFG2, job 1096101	36	0.974147 / 0.938729	0.250010	1.101729	2 gates pass, 2 fail

Required: mean SSIM ≥0.95, worst video SSIM ≥0.90, temporal error ≤0.10, latency ratio ≤0.90. Each completed PP1 run has 110 real local-block-stack cache hits per CFG branch, finite paired latents, distinct branch states and request resets. Scheduling and failed startup attempts are not counted as passing tests.

For the historical runs below the earlier implementation boundary, the evaluation harness added stage-specific polynomial selection, negative-prediction clamping and an observed-input-range guard. These are not production defaults, so its results alone cannot qualify the production backend. Performance includes rank-trace overhead, excludes loading/warmup, and alternates native/cached requests in one loaded engine. Exact temporal-error definition, per-video latent errors and all failed gates are retained in the report.

Ruff passes. Targeted mypy retains 12 pre-existing PP mixin errors (14 on the merged dependency baseline); it is not reported as a clean type-check run. Intermittent multiprocessing SemLock startup failures occurred on both baseline and feature attempts and are retained in the logs.

AI assistance: Codex implemented this draft and executed the validation. This remains an experimental draft, not a production-ready claim.

Historical evaluation update (2026-10-12; frozen source stated above): all four topologies completed, 144 paired requests / 288 videos. All four pass both SSIM gates and fail temporal-error and latency gates (8 measured gate passes, 8 failures, 0 skips). Each PP2 run has stage-0 positive/negative hits of 168/168 and stage-1 hits of 281/279, finite paired latents, distinct branch states and a passing request-reset audit. These are real local block-stack cache hits.

PP2/CFG1 retry 1096862 completed in 8m41s with the same frozen source, model, profile and dependency environment. Original job 1096100 timed out without complete video pairs and reported a ShmRingBuffer shared_memory AttributeError; root cause remains unresolved. Shutdown resource-tracker warnings remain in successful logs. Final raw PP2 reports, JUnit, videos and traces are retained in the shared wan-teacache-validation/validation-1096101 and validation-1096862 directories. The source-tree evidence summary is updated; generated artifacts remain outside the current source tree, following the existing archive convention. This PR stays a draft with production defaults disabled.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-10-18T16:38:22.200137Z	3341880	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[vllm-omni-review-bot]

This PR appears to belong to: docs/design/module/diffusion/index.md, docs/design/module/diffusion/parallelism.md.

Module owners: @princepride @wtomin @david6666666

Routing: @princepride via module of the changed files, module named in the PR description, CODEOWNERS; @wtomin via module of the changed files, module named in the PR description, CODEOWNERS; @david6666666 via module of the changed files, module named in the PR description

@fusheng-ji, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

[vllm-omni-review-bot]

Omni ReviewBot triage note

Automated triage of commit 2405abe52a48 produced:

• Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

[fusheng-ji]

Hi guys. Could you help review the experimental Wan TeaCache implementation at 7b1bbc4?

• @0z5a: could you recheck the branch-local coefficient collection fix and its regression coverage?
• @princepride: please check stage/CFG cache isolation, request resets, and PP communication on cache hits.
• @Bounty-hunter: please check native/cache forward equivalence and the four PP/CFG combinations.
  Feedback on the profile and acceptance protocol would also help. The linked GPU results belong to older commits; current-head qualification evidence remains outstanding.

[princepride]

@fusheng-ji You forgot sign-off😂

[fusheng-ji]

@fusheng-ji You forgot sign-off😂

Hi, thanks for reminding me. I have fixed the sign-off issue; now it should be ready for review.

[vllm-omni-review-bot]

Omni ReviewBot: no human activity for 7 days

@fusheng-ji this pull request has had no human commit, comment or review since 2026-10-29. Please confirm the current plan and next step. The author or a maintainer decides whether to change the PR state.

To keep it moving, any one of these is enough: push an update, reply to the open blocker, or post the current plan and timeline.