Repository navigation
[CI] [AMD] diffusion/flux3: Make exact-quality VAE convolutions deterministic on ROCm - #46947
Merged
michaelzhang-ai merged 2 commits intoNov 3, 2026
Conversation
…ministic on ROCm MIOpen's default fp32 convolution kernels are not run-to-run deterministic for some FLUX.3 VAE shapes, so an image edit with quality="exact" encodes the same reference to slightly different latents on each request and the denoiser amplifies that into a visibly different image. This breaks TestDisaggFlux3Image.test_t2i_and_edit_match_monolithic on ROCm. Set torch.backends.cudnn.deterministic around the FLUX.3 VAE encode and decode for exact requests on ROCm only. Other platforms are unchanged.
michaelzhang-ai
marked this pull request as ready for review
October 28, 2026 17:24
michaelzhang-ai
requested review from
AgainstEntropy,
BBuf,
HaiShaw,
kevin-mii,
mickqian,
niehen6174,
ping1jing2 and
yichiche
as code owners
October 28, 2026 17:24
HaiShaw
approved these changes
Oct 29, 2026
HaiShaw
reviewed
Oct 29, 2026
HaiShaw
previously requested changes
Oct 29, 2026
…act VAE calls With SGLANG_USE_ROCM_CUDNN_BENCHMARK set, cudnn.benchmark picks the convolution kernel by timing, which can undo the deterministic choice. Turn it off for the exact-quality VAE call and restore both flags after, matching the existing Sana and MiniMax-H3 deterministic helpers. Add unit tests for the flag handling on ROCm and on other platforms.
Collaborator
Author
|
@mickqian could you take a look? |
This was referenced Oct 30, 2026
3 tasks done
2 of 5 tasks
QwenApp
pushed a commit
to QwenApp/sglang
that referenced
this pull request
Nov 2, 2026
…atibility Install ml-dtypes explicitly for the legacy MUSA vendor bundle, whose tilelang-musa wheel is installed without dependency resolution. Backport the exact ROCm VAE convolution context and regression tests from sgl-project#46947. Preserve the exact image assertions and restore backend flags after normal completion or exceptions. Backport the paired fork-worker preload API from merged sgl-project#47592 so the current CI caller is compatible with the package installed from this branch. Validated the production VAE context and all three call sites on CPU, the real runner subprocess/fork paths, nine MUSA helper tests, and the actual installer command with isolated offline wheel metadata fixtures. Normal pre-commit checks pass. These local checks do not establish GPU/NPU CI acceptance or resolve the outstanding PP transport and NPU host allocator failures.
4 of 5 tasks
4 of 5 tasks
michaelzhang-ai
enabled auto-merge (squash)
November 3, 2026 05:34
yichiche
approved these changes
Nov 3, 2026
yichiche
left a comment
yichiche
left a comment
Collaborator
There was a problem hiding this comment.
Guard with is_hip, LGTM.
This was referenced Nov 4, 2026
Yuxingwang-intel
pushed a commit
to Yuxingwang-intel/sglang
that referenced
this pull request
Nov 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
On ROCm, run the FLUX.3 VAE encode and decode with
torch.backends.cudnn.deterministic = Trueforquality="exact"requests. The flag is set only for the duration of the VAE call and restored afterwards. Other platforms and non-exact tiers are unchanged.Why
TestDisaggFlux3Image::test_t2i_and_edit_match_monolithichas failed on every scheduled PR Test (AMD) rocm10 run since #45309 landed, e.g. this job: text-to-image matches exactly, but the edit differs in ~57% of pixels (max 22).The disaggregated path isn't the problem. A monolithic server gives a different edit each time the same request is sent (diagnostic run). Text embeddings, noise and the prepared reference are bit-identical between runs. The reference's VAE latents are not: 5-9 of 30720 values differ by one bf16 ulp, and the denoiser amplifies that across the image.
Per-module hooks on the VAE alone (run) show the first divergence at fp32
Conv2dlayers whose inputs are identical between runs (encoder.down.3.block.0.conv1,decoder.mid.block_1.conv1). MIOpen picks a non-deterministic kernel for those shapes. GroupNorm and the attention block are deterministic. Eight repeated calls on MI300X:cudnn.deterministicTest plan
multimodal-gen-test-2-gpu-amd, rocm10, dispatched on 177e8f7: both shards pass, includingTestDisaggFlux3Image(run, shard 0 job).multimodal-gen-test-2-gpu-amdshards pass.TestDisaggFlux3Image::test_t2i_and_edit_match_monolithicis PASSED in shard 0, along with the other 3 tests intest_disagg_server.py(job). The run is red only because oftest_ming_image.py::test_ming_norm_fusion_replay_and_native_fallbackin 1-GPU shard 3, which this change doesn't touch and which passes on main now.test_flux3_image_stages.pypass in the rocm10 image (10 passed).CI States
Latest PR Test (Base): ✅ Run #40814696873
Latest PR Test (Extra): ❌ Run #40814696951
Latest PR Test (AMD ROCm 10): ❌ Run #40814696838