Repository navigation
Conversation
Contributor
Preview approval requiredThis pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit |
BBuf
marked this pull request as ready for review
October 29, 2026 03:23
BBuf
requested review from
AgainstEntropy,
HaiShaw,
JustinTong0323,
kevin-mii,
mickqian,
niehen6174,
ping1jing2,
sogalin,
wisclmy0611,
yichiche and
zijiexia
as code owners
October 29, 2026 03:23
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
4 of 5 tasks
Collaborator
Author
|
/rerun-test test/registered/kernels/ops/diffusion/test_model_fast_paths.py::test_qwen21_bf16_mlp_packing_restore_and_lora |
Contributor
|
🚀 |
Collaborator
Author
|
/rerun-failed-ci |
1 similar comment
Collaborator
Author
|
/rerun-failed-ci |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Current-head update (2026-10-30):
188169fa0db0bb2d524d56a97184590f948cca07passed the frozen 512 gates but failed the 1024 cleanup gate for T2I and editing. See the full revalidation and actual images below. The original report measuresefcb88a2; it does not establish that the current head passed cleanup.Qwen-Image 2.1 runs separate BF16 gate and up projections in its image MLP. Packing them into one projection reduces saved-request E2E latency by 3.16% for 512² T2I and 2.63% for 512² editing on RTX PRO 6000 Blackwell. The original 1024² cases improve by 1.08% and 0.94%; those smaller gains are retained below.
The change shares gate/up storage while preserving checkpoint parameter names and shapes, then reuses the existing packed SiLU-multiply operator. It adds no kernel or public flag. Eligibility is limited to SM120, resident unquantized TP1 projections, BF16 batch-one targets of 1024 or 4096 tokens, and
quality="high". Prefixes, exact/lossless requests, LoRA wrappers, incompatible storage/layouts andtorch.compileretain separate projections. Views are derived from parameters, with the existing_applylifecycle repacking after CPU/GPU transfers.Sources and comparison setup
af21e2b989e482cfdce39689126562f2209d53ac.efcb88a201a41b1fecc516a60df0ca0a34fb6dd5.817d4c17feb702289a2d2b2cbacfd594c262b7a7.Qwen/Qwen-Image-2.1@d26bb61231c349cf6b7896fa83353113880e1ba3; all 24 runtime files hash-verified.The same physical GPU0 was used serially: RTX PRO 6000 Blackwell Server Edition, SM120, 97887 MiB, driver 595.84. GPU1 remained idle; the shared host has no exclusive lease, so ownership was checked at boundaries and during requests. Batch/concurrency 1, 50 steps, CFG 1, CPU seed 1101, identical prompts/reference image, empty negative prompt, BF16 native components with mixed-precision internals, no quantization.
SGLang uses native resident eager with cuDNN transformer attention, compile/BCG off. Omni retains its native regional compilation and decode graphs. Its default passes both 512 tasks and 1024 editing. Default 1024 T2I fails the Chinese-image gate (RGB PSNR 24.18 dB against SGLang cuDNN); the native compiled
TORCHINDUCTOR_EMULATE_PRECISION_CASTS=1variant passes and is used there. A separate valid-variant screen selected default for the other three cases. Failed variants remain archived. Equivalent native PNG compression is level 0: SGLangoutput_compression=0, Omni100.SGLang uses Torch 2.14.1+cu130 / Triton 3.8 / cuDNN 9.24; Omni uses Torch 2.13+cu130 / Triton 3.7.1 / cuDNN 9.20. Dependency versions are pinned in the evidence. These are measured native stacks, not identical compiler environments.
Final-head results
Primary E2E measures HTTP submission through receipt of complete response bytes; client decoding and artifact writes are outside the timer. Each task follows main/PR/Omni/Omni/PR/main, two fresh processes per arm. Each process first qualifies short, long, Chinese and repeated-short prompts. Then 512 uses 6 warmups + 8 measured requests; 1024 uses 3 + 6. The performance request is the frozen short prompt. Existing disk/compiler caches remain. Nothing was rerolled or filtered.
¹ Separate diagnostic runs, 4 measured requests per case/arm: SGLang synchronized native stages; Omni native pipeline timer plus a metadata-only observer. No Torch profiler during stage measurements. These scopes have instrumentation overhead and are excluded from primary ratios; do not subtract them from primary HTTP time. ² Reciprocal mean latency at concurrency 1, not saturation throughput. ³ NVML sampled every 250 ms; transient peaks can be missed. p95 is descriptive over 16 requests per arm at 512 and 12 at 1024.
Ratios below are geometric means of the two temporal pairs' per-process median ratios, rather than ratios of pooled p50s.
Main A/A endpoint drift was at most 0.089% at 512 and 0.020% at 1024. All final main/PR process CVs and main endpoint drift meet the original 0.5% limits. Omni fails that CV screen in one 512 edit process (0.621%) and both 1024 T2I processes (0.553% / 1.044%). Those failures and every sample remain recorded. Both pairs favor PR; even the fastest observed Omni request is slower than the slowest PR request in each case. The separate admission records explicitly use this post-screen interpretation, not a silent threshold change. Finite-sample separation is not a population guarantee.
Cached launch-to-ready ranges were 44.03–44.04 s for SGLang at 512, 52.04–54.04 s at 1024, and 36.02–40.02 s for Omni. This excludes the four qualification requests and explicit warmups, whose individual costs are recorded in the summary. Eager has no BCG capture cost. At 1024 each SGLang process warms only 1024: a preceding 512 warmup disables the pre-existing native packed-QKV path through its bit-exact gate in both main and PR. No unrelated QKV change is included.
Profile, cleanup and compatibility
Final representative traces show 64 separate gate/up GEMMs → 32 packed GEMMs, plus 32 existing packed SiLU calls per denoise step. At 512 total GPU kernels fall 480 → 448; fresh 1024-only traces retain native packed QKV and fall 416 → 384. Matched complete GPU step durations, diagnostic only:
Cleanup narrows the original broader length predicate to the two model-qualified targets and simplifies comments/docs. Two fresh-process ABBA screens retained T2I pairs at +0.1338% and +0.1688%, exceeding the fixed 0.1% cleanup screen. These are failed screens, not discarded warmups. Complete production AST comparison finds only the length predicate changed; it is equivalent at the tested shapes. A separately preregistered within-process diagnostic alternated the actual archived precleanup/final functions at request boundaries: 8 ABBA quartets per 512 task and 6 per 1024 task, after 8 warmups. All 144 outputs match main. Final/pre ratios are 0.999757 / 1.000049 at 512 and 1.000071 / 1.000069 at 1024; each descriptive per-case t95 upper bound is below 1.001. This supports no measurable direct cleanup penalty within that bound, while retaining the inconclusive fresh-process evidence. It does not prove zero effect or provide a familywise guarantee. These instrumented timings never enter the performance table.
Final compatibility validation covers 68 saved requests, including 12 separate profiles: high/lossless/exact/high-return, both shapes, short/long/Chinese/repeated prompts. High T2I executes one prefill plus 49 BCG replays; each profiled replay contains 32 packed SiLU calls and 33 graph launches. No serving capture occurs. Editing exceeds the native conditioning bucket and remains eager; non-high requests also remain eager and restore main pixels with no packed-MLP calls. The two-shape warmup creates 10 actual graph entries. A single cached diagnostic observed 47480 vs 39782 MiB before requests and 58.04 vs 56.04 s ready time for BCG/eager. This is compatibility and cost evidence, not a BCG performance win.
Quality and real outputs
Every final saved SGLang qualification/measurement output has RGBA pixels byte-identical to its corresponding main reference. High PR/main uses fixed SSIM 0.99 / PSNR 40 dB gates; cross-framework RGB and alpha use 0.95 / 28 dB separately. Minimum final selected-Omni metrics:
CPU processor inputs, token IDs, reference pixels and initial noise were checked. Effective BF16 model timesteps match at 32 recorded boundaries; the native sigma schedule differs by at most two FP32 ULP from
numpy.expvsmath.exp. This is not a bound on all solver arithmetic. T2I encoder hidden states match; edit hidden states differ before collapse. The runtime precision audit retains those differences. Final Omni native-stage observations verify BF16 scheduler input/output boundaries with autocast off. Cross-framework intermediate identity is not claimed.Each panel below contains three actual output pairs (short, long, Chinese), main left / final PR right. Original RGBA pixels are kept at 1:1 with labels outside them; GitHub may scale the display. The source hashes, inputs and visual review are archived. Editing changes the reference red teapot to blue. These are generated results, not illustrations.
Validation, reproduction and limits
The real SM120 regression passed both 1024/4096-token cases: numeric/dispatch checks, original-path restoration, prefix fallback, independent storage, CPU/GPU round trip, actual dynamic LoRA weights and
torch.compile(backend="eager", fullgraph=True)fallback. The latter checks frontend fallback, not full-model Inductor performance. Ruff 0.15.1/isort, Python/diff checks and Mint validation/broken-links passed. Final diff: 3 files, 114 additions / 7 deletions, including the regression and scoped docs.Existing INT8 packing #44196 remains unchanged. Closed, unmerged #40918 is BF16/QKV/VAE prior art; this PR supplies a scoped BF16 path with parameter/lifecycle preservation and fresh SM120 model evidence. Latest main checked was
5cc9c933fc8acdffb5fc847d3b4824f082cd3297: its Qwen encoder is unchanged; the DiT differs from frozen main only in the reviewed BCG attention-output allocation. Three-way textual merge checks pass for the three changed files; this is not a new benchmark or runtime test of merged main.Use separate pinned environments with
/taskcontaining the archived inputs/scripts, verified checkpoint at/task/cache/common/Qwen-Image-2.1, main at/task/repos/sglang, final at/task/candidates/qwen21-bf16-gate-up-v4, and Omni at/task/repos/vllm-omni. Start one server at a time and stop only its recorded owned process before the next arm. Example 512 launches:Against the running PR server, qualify first with a fresh output label:
Then use
bench_qwen21_serving.pywith the same arm/contract/repo/SHA,--quality high --warmups 6 --repeats 8, a fresh--run-id, and--allowed-gpu-pidscontaining only the current owned server's NVML-visible PID(s). Recreate the A/A and paired orders. For 1024 use the original non-512 contracts,launch_sgl_qwen21_candidate.shfor main,launch_sgl_qwen21_v4_1024.shfor PR,--warmups 3 --repeats 6, and the native casts launcher for Omni T2I only. Every exact executed command and request is archived.Controllers contain original host/container paths and reject overwritten output IDs; adapt them to a fresh namespace, recreate quality references, and never reuse recorded PIDs. Primary runs use
SYNC_STAGE_PROFILING=0; separate denoise launchers use synchronized native stages. Profile requests are diagnostic only.Performance claims cover these single-GPU, single-reference, batch/concurrency-one shapes. Other lengths, GPUs, quantized checkpoints, full-model offload/LoRA/compile combinations, broader caches and multi-GPU cases are not validated performance claims. The broader frozen model matrix remains incomplete.
Evidence: final numbers and warmup costs, cleanup admission and failed screens, runtime precision, paired plans, raw results and exact commands, actual traces, reproduction scripts.
Initial CI (before the ready-for-review update): runtime workflows stopped at the explicit Draft gate. The initial all-file lint job fails Ruff F821 (
MLATokenToKVPool) inpython/sglang/srt/managers/cache_controller.py:885. That file has the same blobeb71dc2709c1ede1c0060bb082731833c405dc9din base286e0185and the tested merge8471aa33; it is outside this three-file PR. Other pre-commit hooks pass, later docs checks are skipped. Local scoped validation is reported above; that initial CI run was not green.Cleanup validation (2026-10-29)
Cleanup at
188169fa0: renamed the existing packed-BF16 dispatch flags and made their call sites explicit; packing eligibility, arithmetic, and kernels are unchanged. On RTX PRO 6000 Blackwell (SM120), PyTorch 2.14.1+cu130, the Qwen-Image 2.1 non-VAE fast-path selection passed 8 tests, including both MLP token sizes, restoration, offload, compile fallback, and dynamic LoRA. Changed-file pre-commit checks pass. The model timings and images above remain measurements of the previously documented head; this cleanup did not repeat full-model benchmarks.Current-head revalidation: 1024 cleanup gate failed
Current head
188169fa0db0bb2d524d56a97184590f948cca07passes the frozen 512 gates but fails the frozen 1024 cleanup gate in both T2I and editing. All samples and original thresholds are retained. This report does not qualify the current head. The original PR report remains historical evidence forefcb88a201a41b1fecc516a60df0ca0a34fb6dd5.Frozen SGLang main
af21e2b989e482cfdce39689126562f2209d53ac; native vLLM-Omni817d4c17feb702289a2d2b2cbacfd594c262b7a7; checkpointQwen/Qwen-Image-2.1@d26bb61231c349cf6b7896fa83353113880e1ba3, all 24 runtime files hash verified. Same physical RTX PRO 6000 Blackwell Server GPU0 (GPU-1565b45e-a702-2015-9776-61f68c36fd59), GPU1 idle, driver 595.84. SGLang Torch2.14.1+cu130, Omni Torch2.13.0+cu130. BF16 native components with recorded internal mixed precision, CPU seed1101, 50steps, CFG1, batch/concurrency1, fixed inputs and RGBA PNG semantics.New primary runs use native resident SGLang eager, SYNC0, without a profiler or observer. Omni retains its native regional compilation and decode graphs. 512 uses native default; 1024 T2I uses the previously quality-qualified native
emulate_precision_casts, while 1024 editing uses native default. The earlier default1024 T2I quality failure remains in the original PR evidence; it is not removed or counted as a win.Each resolution has a fresh two-process main A/A followed by the frozen paired primary schedule. Per task: two main and two native Omni processes (16 measured requests each), four original and four current processes (32 each); each process/task also retains four qualification requests, six excluded warmups and eight measured short requests. The second half reverses task order. Total primary: 12 processes/432 PNGs at512 and14 processes/432 PNGs at1024. A/A adds72 PNGs per resolution. Process pairs, not individual requests, are the independent units in paired confidence intervals.
Primary E2E and memory
Serial response rate is reciprocal mean serial latency, not a saturation-throughput result. Peak NVML includes process/device residency; it is not tensor allocator size.
Fixed gates and uncertainty
Cleanup requires mean regression no greater than fresh AA noise AND a four-pair log-t95 regression upper bound below0.1%. Both1024 means exceed AA noise; T2I also fails the upper-bound test. The two1024 cleanup intervals include1: this does not prove a nonzero population regression or a source-code cause, and does not satisfy the frozen no-regression assurance. No unchanged primary rerun or relaxed gate was used.
Both512 main/current intervals include1 despite their frozen gates passing; only two process pairs support these intervals. Do not claim all main/current intervals exclude1. Both1024 main/current and all four Omni/current intervals exceed1. Independent NumPy/SciPy reviews recomputed all statistics from actual raw results without the admission calculator.
Startup and excluded warmups
Readiness includes native startup/warmup and, for Omni, native compilation/graph setup; these costs are not included in primary saved-request E2E. The same server may handle both tasks; per-task startup entries are not additive.
Quality
The twelve displayed fresh main/current primary image pairs have identical PNG bytes and RGBA pixels (RGB/alpha SSIM1, PSNR∞). Three distinct prompts per task/shape include short, long and Chinese text; repeated short requests are retained. Pixel equality does not establish semantic perfection. Cross-framework RGB and alpha are assessed separately against0.95 SSIM/28dB thresholds. Below are componentwise minima over the three distinct prompts in the first fresh native Omni process; the full per-prompt records identify potentially different worst samples.
Separate failure investigation
The complete original→current source diff is two files,16 insertions/12 deletions. Production changes rename the enable flags and pass one outer block flag by keyword instead of position. Arithmetic, kernels and weight storage are unchanged. The actual SM120 packing/restoration/LoRA regression passed at1024 and4096 tokens,2 tests/0 skips. This does not establish full-model LoRA or GPU compile performance.
A separately frozen original/current1024 native high run completed two fresh processes,44 PNGs and four Torch traces, all hash verified. Each task/source retains four qualification requests, two excluded warmups, four SYNC1 diagnostic requests and one separate profiled request. All44 images equal corresponding primary main RGBA outputs. Three complete interior profile steps (#2,#3,#4) per trace have identical GPU kernel sequence/launch geometry, CPU operator sequence/shapes and CUDA runtime counts:384 kernels,32 packed gate/up GEMMs and32 packed SiLU kernels per step,zero graph launches. Original/current sampled peaks are identical within this diagnostic (39339884544B T2I;45669089280B edit). This finds no operator/storage regression mechanism; Python argument-binding overhead is not isolated by these traces.
These timings are diagnostic, one process per source, and cannot replace primary E2E or prove a performance repair. New resolution-specific main/current stage diagnostics and high→lossless→exact→high/BCG compatibility plans were not run after primary failure. No current-head BCG net benefit, new scenario or complete compatibility validation is claimed. No source patch is justified solely by matching traces.
Reproduction and archive
Use the pinned commits/checkpoint and an isolated owned GPU0, with the exact native launchers, contracts, warmups, process ordering and pair indices in
evidence/plans/.run_qwen21_current_head_blocks1.pyruns each AA or primary plan once; archive/verify/audit AA before primary. Preserve SYNC0 for primary. The separately namedqwen21-cleanup-1024-forensics1plan uses the unmodified generic diagnostic runner with SYNC1 and explicitly pins the failed primary receipt. Do not treat it as an admission retry. Local root-level remote scripts map toscripts/and archivedevidence/reproduction/paths; remote root is/data/xyzhang/diffusion-rtx6000-20261026, mounted/task.Complete archives remain local and are identified by public per-member hashes; public PNGs are deduplicated by content hash, complete raw numerical results/requests/telemetry are retained, response copies omit repeated base64 data with original hashes recorded, and traces are published separately from production code. All historical failures remain in the original PR evidence. The full frozen81-cell matrix is not claimed complete.
Complete evidence and numerical report · Archive hashes · Profile investigation
Actual fresh main/current outputs
These twelve panels are actual primary qualification outputs; full source pixels and alpha are preserved at 1:1. Labels state the failed cleanup status. The original historical media above remain unchanged.
Media provenance · Local visual inspection
CI States
Latest PR Test (Base): ❌ Run #42123750135
Latest PR Test (Extra): ✅ Run #42123749864
Latest PR Test (AMD ROCm 10): ⏳ Run #42123749967
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: