Repository navigation
[Bugfix][Diffusion] Fix Cosmos3 transfer caching artifacts with control-CFG consensus - #7738
yzhautouskay wants to merge 2 commits into
Conversation
Signed-off-by: Yuliya Zhautouskaya <yzhautouskay@nvidia.com>
|
This PR appears to belong to: docs/design/module/diffusion/index.md. Module owners: @Isotr0py @david6666666 @princepride Routing: @Isotr0py via module of the changed files, module named in the PR description, CODEOWNERS; @david6666666 via module of the changed files, module named in the PR description; @princepride via module of the changed files, module named in the PR description @yzhautouskay, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer. Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment. |
Signed-off-by: Yuliya Zhautouskaya <yzhautouskay@nvidia.com>
|
Self-review note: FP8 runs with Cosmos3 Transfer + SeaCache implementation from this PR exposed a transfer quality issue when This PR addresses and ultimately solves this issue by synchronizing cache decisions across control-CFG branches, aligning residual refresh and extrapolation steps while retaining separate branch residuals. |
|
Closing this PR in favor of #7939 |
Purpose
Fix Cosmos3 Transfer caching artifacts by synchronizing SeaCache decisions across control-CFG branches. Original implementation in the PR: #6922
Control CFG strengthens adherence to a control hint (such as edge, depth, or segmentation) by contrasting predictions with and without that hint, while keeping the text prompt unchanged. Unlike text-CFG branches, these branches receive different vision inputs, so their SeaCache indicators can produce different skip decisions
Independent decisions can create mismatched residual refresh steps and extrapolation histories, distorting the branch differences amplified by guidance.
This change:
Test Plan
Test Result
Current cache quality issue is input-dependent because control-CFG branch schedules can diverge very differently across examples, independent of the control type alone. Some show little degradation, while others develop severe temporal/block artifacts. Effect is amplified with FP8-quantized checkpoints. This PR resolves the observed caching artifacts, from mild flickering to severe degradation, in both BF16 and FP8 runs
Rows: BF16 (top), FP8 (bottom).
Columns, left to right: uncached / current cache / this PR fix (consensus):
seg_six_arms_fix.mp4
edge_six_arms_fix.mp4
vLLM Version:
0.29.0vLLM-Omni Commit:
896f4466BEFORE SUBMITTING: read CONTRIBUTING.md and run the precheck-pr skill with the code agent for a self-check against project conventions.
(anything written below this line will be removed by GitHub Actions)