Repository navigation
[diffusion] Warm up CI cases at the request's quality level - #42958
Merged
Merged
Conversation
Motivation: since sgl-project#42370 consistency-checked CI requests pin quality=exact while warmup runs the default lossless level, which mounts lossless-tier fusions. The timed request unmounts them and runs a path warmup never touched; the LTX refinement stages, already near their limits, now fail on every retry. Modifications: when a case's request names a quality level, merge the same level into --warmup-sampling-params, keeping existing warmup params and any level the case set itself. Validation: H200 with H100 free memory emulated, eager, round 1: warming up at exact brings LTX2RefinementStage back from 836 to 749 ms (ltx_2_two_stage_t2v, baseline commit 800 ms) and from 964 to 860 ms (ltx_2.3_two_stage_t2v_2gpus, 869 ms). New unit tests; the CI-case one fails without this change.
niehen6174
marked this pull request as ready for review
October 7, 2026 11:18
niehen6174
requested review from
AgainstEntropy,
BBuf,
HaiShaw,
kevin-mii,
mickqian,
ping1jing2 and
yichiche
as code owners
October 7, 2026 11:18
niehen6174
added
run-ci
Collaborator
Author
|
/tag-and-rerun-ci |
mickqian
approved these changes
Oct 7, 2026
Collaborator
Author
|
/rerun-failed-ci |
1 similar comment
Collaborator
Author
|
/rerun-failed-ci |
mickqian
pushed a commit
that referenced
this pull request
Oct 7, 2026
Picks up #42958, which warms CI cases up at the request's quality level. Without it the LTX-2 two-stage cases on the 2-GPU partition run their timed request on a path warmup never touched and exceed their refinement limits; main itself fails the same way. The two streamed-encode tests move to python/sglang/multimodal_gen/test/unit/ with the rest of the diffusion unit tests (#42706), dropping their register_cuda_ci call as the moved files did: that directory is collected whole, so new files need no registration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mickqian
pushed a commit
that referenced
this pull request
Oct 7, 2026
… the streamed encode Encoding the MP4 while the VAE decodes moves the encode inside DecodingStage, and with it inside e2e (which is the sum of the stages). On main the encode ran after the last stage, so neither number covered it: the 1.07-1.10 s DecodingStage baselines and the e2e baselines measured less work than the request does. The stage timers also do not synchronize, so the old DecodingStage figure was mostly the CPU-side launch (0.30 s against 1.48 s of GPU decode on a synchronized H200 run); the encoder's close() now waits for the decode inside the stage, which is the first time this stage reports what it costs. Whole requests are not slower: same-process A/B with the streamed encode off and on gives 1.817 s against 1.729 s for decode plus encode, and the fastwan request total (MP4 written) is 3.29 s against 3.55 s on main. New values are the medians of the 7 retries of each case on H100, in the first run with #42958 merged (37645482900): fastwan2_2_ti2v_5b DecodingStage 1073.10 -> 2173.40, e2e 2444.17 -> 3073.51 wan2_2_ti2v_5b DecodingStage 1098.67 -> 2172.17, e2e 14853.56 -> 15599.56 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mickqian
pushed a commit
to adenzhou1350/sglang
that referenced
this pull request
Oct 7, 2026
mickqian
pushed a commit
that referenced
this pull request
Oct 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Since #42370, consistency-checked CI requests pin
quality=exact, but server warmup still runs the defaultlosslesslevel. Warmup therefore mounts thelossless-tier fusions, and the timedexactrequest unmounts them and runs a path warmup never touched.The LTX refinement stages, already close to their limits, now exceed them on every retry: 2-GPU partition 2 passed 1 of 17 diffusion PR runs after 10-07 03:00 UTC (
ltx_2_two_stage_t2v,ltx_2.3_two_stage_t2v_2gpus,ltx_2_3_two_stage_ti2v_2gpus). The server log shows the mismatch directly: warmup logsMounted LTX-2 fused RMSNorm+modulate for quality=lossless(and two more fusions), then the request runs atexact.The same mismatch shows up for other models. The LTX cases fail first because they have the least headroom:
losslessltx_2_two_stage_t2v,ltx_2.3_two_stage_t2v_2gpus,ltx_2_3_two_stage_ti2v_2gpusexactwan2_1_i2v_14b_480P_2gpu,wan2_1_t2v_1.3b_teacache_enabled,wan2_1_t2v_1.3b_upscaling_4x,wan2_1_t2v_1_3b_lora_1gpuexactflux_image_t2i_2_gpus,flux1_modelopt_fp8_t2iexactqwen_image_edit_2511_ti2i,qwen_image_layered_i2iexactlossless-tier fusion sites in the model codeModifications
--warmup-sampling-params. Existing warmup params are kept, a level the case sets itself is kept, and aliases are normalized (extra-high->lossless).Validation
Same H200 with H100 free memory emulated (80,900 MiB per GPU,
originaltwo-stage mode), eager as in CI, 3 interleaved rounds.LTX2RefinementStage, per-run and median:49bafb58cd)c52dcc5a01)ltx_2_two_stage_t2vltx_2.3_two_stage_t2v_2gpuslosslessinstead would be faster still, but fails consistency in all 3 rounds, so the request pin from [diffusion] Name the quality tiers after what they guarantee #42370 stays.CI States
Latest PR Test (Base): ❌ Run #37613001428
Latest PR Test (Extra): 🚫 Run #37613296122
Latest PR Test (AMD ROCm 10): ❌ Run #37613001149