Repository navigation
[Model] Support SenseNova-U1.5-8B-MoT and its distilled 8-step LoRA - #6516
Conversation
The distill LoRA backend accepted --lora-path and then warned that the pipeline could not load it, so generation ran without the adapter. Add load_lora_weights to SenseNovaU1Pipeline, and raise when nothing matches. The shared stacked-delta helper picked fused mappings by substring and did not stop at the first hit, so .qkv_proj and .qkv_proj_mot_gen both fired and produced a delta twice the height of the parameter. Match on the tail and stop at the first hit. The fused names shipped by QwenImage and Wan2.2 do not overlap, so this selects the same shards for them. Replace the eager RMSNorm cast chain with F.rms_norm: -7.47% at 1024x1024 and -7.56% at 1536x1536 on one A800, and closer to a float64 reference in every shape tested. Signed-off-by: MrlixiangWE <mrdanaer@gmail.com>
|
Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits. |
|
This PR appears to belong to: docs/design/module/diffusion/diffusion_model_integration.md. Module owners: @fhfuih @Bounty-hunter @wtomin @RuixiangMa @MrlixiangWE, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer. Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment. |
|
Same prompt and seed, 1024x1024, on this branch.
10973.6 ms / 1026.6 ms / 1024.2 ms. |
recipes/ sits outside the mkdocs docs tree, so the relative link warned and fail_on_warning failed the Read the Docs build. Every other recipe in this table is linked by absolute GitHub URL. Signed-off-by: MrlixiangWE <mrdanaer@gmail.com>
|
@hsliuustc0106 this PR is ready for review. |
hsliuustc0106
left a comment
hsliuustc0106
left a comment
There was a problem hiding this comment.
Single-GPU validation passed (36 tests; full-model latency -11.7%), but TP=2 distilled LoRA startup crashes on both ranks.
| state_dict = {k: v.to(torch.float32) for k, v in state_dict.items()} | ||
| state_dict = _apply_diffusers_lora_alpha_scaling(state_dict) | ||
|
|
||
| loaded_keys = self.load_lora_into_module(state_dict, self, prefix="") |
There was a problem hiding this comment.
This breaks distilled LoRA with --tensor-parallel-size 2: both ranks load a 3,072-row local QKV weight, but _prepare_lora_delta builds a 6,144-row global delta and params.add_ raises. Please slice the delta through vLLM's parameter weight-loader/TP metadata and add a TP=2 regression test.
There was a problem hiding this comment.
Fixed in a312a94 — the delta is written through the parameter's own weight loader, so each rank takes only its shard. test_fused_delta_is_sharded_through_the_weight_loader pins it on CPU, and TP=2 with the distilled LoRA now runs on two A800s.
There was a problem hiding this comment.
Verified all five at be42af5 — the delta now routes through _prepare_lora_delta per stacked shard (with the loader test), the RMSNorm test asserts the fused path is closer to float64, only the adapter-name sentinel survives fusion, the cache-cleared cold-start table plus the 85acb30 warmup covers the readiness question, and 9e0561c documents the paged-cache scope and deletion condition. Thanks. Two things remain: my 08-27 A/B-parity comment is still open, and 0d68f32 adds ~460 lines of unreviewed 8B-MoT support to this PR — intentional, or should that land as a follow-up PR?
There was a problem hiding this comment.
Intentional. U1.5 shares SenseNovaU1Pipeline with U1 -- same
model_type: neo_chat, and SenseNovaU1Config already reads use_pixel_head
and noise_scale_max_value. Supporting it is making the distilled LoRA path
correct. The 460 lines:
| File | + | What |
|---|---|---|
| tests (3 files) | 258 | tests |
recipes/SenseNova/SenseNova-U1.5.md |
114 | recipe |
pipeline_sensenova_u1.py |
67 | load_lora_weights, stacked_params_mapping |
vllm_omni/diffusion/lora/loader.py |
14 | _prepare_lora_delta fix |
sensenova_u1_transformer.py |
5 | F.rms_norm |
| two doc table rows | 2 | supported-models, diffusion-features |
Tests on the latest commit are running here; I will reply on each thread when
they finish, about six hours if nothing breaks.
There was a problem hiding this comment.
Delta round at e32770a — reviewed the four commits since my last pass (3149b71 one-step resize publish, d7aae00 LoRA-wrapper capture gating, dcd1109 keyword-only flash-attn args, e32770a fusion-failure doc); all clean, and the fault-injected test_a_failed_grow_leaves_the_cache_on_its_old_buffers is a nice touch. The A/B-parity ask is resolved for me: the same-seed table above plus 8fb8683's whole-loop decode comparison is exactly the durable coverage I wanted — please push that commit into this PR so the test and the recipe index are actually in-tree. Still outstanding from my 08-28 review: is 0d68f32's ~460-line 8B-MoT support intentional for this PR, or should it land as a follow-up?
There was a problem hiding this comment.
Intentional. This PR closes #6471, which asks for SenseNova-U1.5-8B-MoT support; the model already runs on the existing pipeline, and 0d68f32, the PR's first commit, is what makes its distilled LoRA correct, so it belongs here rather than in a follow-up. Per-file breakdown in #6516 (comment).
8fb8683 is in the PR already, commit 17 of 25: recipes/README.md +1/-0 and the whole-loop decode test +53/-0.
There was a problem hiding this comment.
Conceded — my framing was wrong on both counts: 0d68f32 was this PR's first commit (the #6471 support itself, predating my review), and the growth I attributed to it was actually the nine 6194f41→be42af5 commits answering review threads. With the scope question resolved and the A/B coverage question reduced to pushing 8fb8683, the only remaining item from my side is that push. Thanks for the detailed accounting.
There was a problem hiding this comment.
One more ask before approval: an e2e stage-attribution table in the same format #6714 and #6607 already use, so the paged-decode and LoRA claims are isolated rather than blended. The VLLM_OMNI_SENSENOVA_PAGED_DECODE switch makes this a clean A/B on the same head:
Conditions: 1×A800 80GB, same prompt + seed + resolution across columns, 2 warmups then N≥10 measured, P50 (P100). Baseline = main <sha>, candidates = head <sha> with the switch as shown.
| Metric | main eager (50-step) | PR paged=0 (50-step) | PR paged=1 (50-step) | PR paged=1 + LoRA (8-step) |
|---|---|---|---|---|
| Requested inference steps | 50 | 50 | 50 | 8 |
e2e_total_ms |
||||
| Stage 0 — AR prefill + decode (incl. think decode, tokens) | ||||
| Stage handoff (e2e − stages) | ||||
| Stage 1 — diffusion execution | ||||
| Postprocess | ||||
| MP4/PNG response encoding | ||||
| Approx request → output | ||||
| Sanity: output pixels / generated tokens | identical | identical | identical | identical |
main vs paged=0 proves the loading/fusion changes cost nothing at equal steps; paged=0 vs paged=1 isolates the paged-KV + CUDA-graph decode gain (and doubles as the scheduler-overhead measurement for the model-local-cache question); the LoRA column separates the step-reduction claim. A small companion table — requests that hit a new decode bucket and the max capture stall — would keep the mid-serving re-capture tail honest, since P50s hide it.
There was a problem hiding this comment.
Stage attribution. One note on the shape first: stage_metrics reports a single stage named diffusion for this model, because SenseNovaU1Pipeline, which U1.5 shares with U1, runs its whole autoregressive loop inside one pipeline forward and the decode steps never reach the scheduler — the same fact behind the model-local cache. The AR/diffusion split below is therefore measured inside that stage, not across two of them.
Conditions: sensenova/SenseNova-U1.5-8B-MoT, 1×A800 80GB, TP=1, same prompt + seed 42 + 1024×1024 across columns, think on in every column, 2 warmups then N=10 measured, P50 (P100). Baseline = main 262ff14, candidates = head e32770a with the switch as shown.
| Metric | main eager (50-step) | PR paged=0 (50-step) | PR paged=1 (50-step) | PR paged=1 + LoRA (8-step) |
|---|---|---|---|---|
| Requested inference steps | 50 | 50 | 50 | 8 |
e2e_total_ms |
27.747 s (28.159) | 16.500 s (16.702) | 13.930 s (13.944) | 4.455 s (4.485) |
| Stage 0 — AR prefill + think decode | 16.281 s (16.681) | 6.283 s (6.461) | 3.604 s (3.612) | 3.593 s (3.603) |
| · AR prefill | 69.0 ms (78.0) | 38.9 ms (39.1) | 39.3 ms (39.5) | 39.0 ms (41.9) |
| · think decode | 16.146 s (16.545) | 6.213 s (6.391) | 3.534 s (3.539) | 3.522 s (3.533) |
| Stage 0 — diffusion execution | 11.460 s (11.492) | 10.251 s (10.271) | 10.317 s (10.335) | 0.854 s (0.877) |
| Stage handoff (e2e − stage_gen) | 1.5 ms (2.4) | 1.5 ms (2.1) | 1.3 ms (1.5) | 1.0 ms (1.3) |
| Postprocess | 0.0 ms | 0.0 ms | 0.0 ms | 0.0 ms |
| Sanity: output pixels | 1048576 | 1048576 | 1048576 | 1048576 |
| Sanity: think chars | 1233 | 1233 | 1233 | 1233 |
Isolated that way:
| Comparison | e2e | think decode | diffusion |
|---|---|---|---|
| main → paged=0 (loading/fusion, RMSNorm, RoPE) | -40.53% | -61.52% | -10.56% |
| paged=0 → paged=1 (paged KV + CUDA graph) | -15.57% | -43.11% | +0.65% |
The earlier same-seed A/B ran with think off, where the decode loop this switch serves barely executes — hence +0.02% there against -15.57% here.
Stage handoff stays at 1.0–1.5 ms across all four, which is also the scheduler-overhead number for the model-local-cache question.
There is no PNG encoding row and no separate request → output row: the offline path returns the image in-process, so nothing encodes a response and e2e_total_ms is that number.
Capture tail
Every capture timed individually, since P50s hide it. paged=1, 1 warmup + 5 measured per row.
| workload | requests | captures | bucket growths | max capture stall |
|---|---|---|---|---|
| T2I + think | 5 | 2 | 1 | 29.3 ms |
| edit, 2 images | 5 | 1 | 0 | 28.9 ms |
| image-to-text | 5 | 1 | 0 | 30.1 ms |
Three prefill lengths against the same buckets (512, 1024, 2048, 4096, 8192), to reach the growth path rather than report a stall we never triggered:
| prefill | bucket growths | captures | max stall | trail |
|---|---|---|---|---|
| 237 chars | 1 | 2 | 30.1 ms | 512 → 1024 at length 513 |
| 2007 chars | 1 | 1 | 34.0 ms | 1024 → 2048 at length 1025 |
| 4367 chars | 0 | 0 | — | starts in a bucket wide enough |
One re-capture costs 30–34 ms against a 14 s request. Five edit requests captured once between them, which is the cross-request reuse.
The other dispatch entry points
forward dispatches three ways, and the table above covers _forward_t2i only. The other two, same head, switch as the only difference, 1 warmup + 5 measured:
| entry point | workload | paged=1 | paged=0 | delta | output identical |
|---|---|---|---|---|---|
_forward_it2i |
edit, 2 images | 24.050 s | 30.158 s | -20.25% | image, not compared here |
_forward_text |
image-to-text | 7.263 s | 12.532 s | -42.04% | no — 2031 vs 2190 chars |
_forward_text |
text-to-text | 7.014 s | 12.640 s | -44.51% | no — 2750 vs 2789 chars |
The two text divergences are the greedy-argmax ties already measured on the same-seed A/B at #6516 (comment): the first differing step has a top1-top2 margin of 0.000 and 0.125 against a |Δlogit| of 0.188 and 0.164.
| want = _reference(x, norm.weight, norm.variance_epsilon) | ||
| assert got.dtype == dtype | ||
| assert got.shape == x.shape | ||
| torch.testing.assert_close(got.to(torch.float32), want.to(torch.float32), atol=atol, rtol=atol) |
There was a problem hiding this comment.
Restoring the pre-PR eager RMSNorm implementation still leaves all 14 tests in this file green, so the claimed low-precision rounding improvement is not regression-protected. Please add a deterministic bf16/fp16 case that distinguishes F.rms_norm from the legacy path, or compare both errors against the float64 reference.
There was a problem hiding this comment.
You were right that the suite stayed green with the old implementation restored. test_rmsnorm_is_closer_to_float64_than_the_cast_chain now measures both implementations against the float64 reference and asserts the fused one is closer, so restoring the cast chain fails it.
| loaded_keys = self.load_lora_into_module(state_dict, self, prefix="") | ||
| if not loaded_keys: | ||
| raise ValueError(f"Distilled LoRA {paths[0]} matched no parameter of this pipeline.") | ||
| self.lora_loaded[adapter_name] = state_dict |
There was a problem hiding this comment.
This keeps the full fp32 adapter state alive even though SenseNova has no unload path. The 814.9 MB BF16 file expands to 1.52 GiB of tensors and raised steady process-tree RSS by 1.95 GiB locally; keep only an adapter-name sentinel after one-way fusion unless unload is added.
There was a problem hiding this comment.
Fixed in a312a94 — only an adapter-name sentinel survives the one-way fusion. test_fusion_does_not_retain_the_adapter_tensors asserts what is kept is neither a dict nor a tensor, and that a second load is still a no-op.
Under tensor parallelism the delta read from the checkpoint spans the whole layer while the parameter holds only this rank's shard, so adding it directly raised a shape error and startup failed on every rank. Route each delta through the parameter's own weight loader, which is the same code that placed the base weights, into a zeroed copy of the parameter, and add that. Fusion is one-way and this pipeline has no unload path, so keep only a sentinel instead of the fp32 state dict, which measured 1.52 GiB resident. The RMSNorm tests passed with the previous implementation restored, so they did not protect the accuracy claim. Compare both implementations against the float64 reference and assert the fused one is closer. Signed-off-by: MrlixiangWE <mrdanaer@gmail.com>
|
I profiled one 1024×1024 step with Nsight Systems 2026.1.3 on an L20X, capturing only the third worker-side forward after the engine warmup and one user warmup.
Inter-launch gaps overlap GPU execution and are not additive to kernel time; they measure host intervals between consecutive launch APIs. This is host-dispatch/kernel-launch limited, not device-compute limited:
Can you optimize the launch path before merge? The runtime currently reports that regional compilation is skipped because _repeated_blocks = ["SenseNovaU1DecoderLayer"]Please validate both UND/GEN specializations, resolutions, TP, and distilled LoRA, then provide the same-input A/B with:
A successful change should materially reduce the 6,372 launches, especially copy/cast and elementwise kernels. If regional compile still graph-breaks or leaves the count unchanged, the next target should be CUDA Graph replay for stable resolution/sequence/CFG buckets. GEMM tuning alone has a low ceiling here because GEMMs are only about 11-15% of end-to-end wall time. Nsight collection overhead was ~6 ms. The 9-11 second application timing printed during capture is invalid because |
…che and a CUDA graph Decode is about 80% of a think request, and almost all of the GPU idle in that window sits inside no CUDA call at all -- it is Python/ATen dispatch. A CUDA graph retires those dispatches, but capture needs static shapes and DynamicCache grows K/V with torch.cat on every step. Padding to a bucket and masking the tail does give static shapes, but the mask costs more than the graph saves: 7.96-11.55 ms per step against 1.03 ms unmasked, measured on an A800 at real decode shapes. A paged cache gives both. flash_attn_varlen_func takes the used length as a tensor (seqused_k) alongside a block_table, so the buffers stay bucket-sized and capturable while the kernel reads only the valid prefix, and one capture serves every length in the bucket. Three per-layer costs go with it. The 3D RoPE tables were rebuilt inside all 84 modules -- 1,008 calls producing 17 distinct tables -- and are now built once per forward. K/V are no longer expanded before the attention backend, which was hiding the GQA shape from the backend's own fused-kernel check. Single-token decode no longer builds an all-zeros mask whose only effect was to keep SDPA off those kernels. The path is gated by VLLM_OMNI_SENSENOVA_PAGED_DECODE and by a signature probe on flash_attn_varlen_func, so a wheel without seqused_k/block_table falls back to SDPA instead of raising at the first decode step. Signed-off-by: MrlixiangWE <mrdanaer@gmail.com>
VLLM_OMNI_SENSENOVA_PAGED_DECODE=0 forces the ordinary cache; the automatic fallback covers devices and wheels that cannot support the paged path. Signed-off-by: MrlixiangWE <mrdanaer@gmail.com>
|
Both suggestions are in. Taking the buckets first: they are not where the idle time is. Corrected GPU busy — kernel time from the profiled run, wall clock from a clean one — is 92.0% for text-to-image and 68.5% for understanding/think. Replay over stable resolution/sequence/CFG buckets is worth at most ~7.5% there; the idle sits in AR decode, at 31.5%. Its attribution is 8.7% The blocker was the cache. Whole PR, main -> this branch (1x A800-80G, seed 42, n=5, one session)
On main the distilled adapter is silently dropped, so that row is the user-facing before/after rather than a like-for-like speed comparison. Launch count, the criterion you setSame input, one profiled generation of T2I 1024x1024, 8 steps, cfg 1.0, distilled LoRA.
Tensor parallelism (a312a94 -> this branch, 2x A800, SYS/PCIe, no NVLink)
The baseline is a312a94 rather than main because this configuration is the one that crashed at startup before that commit. The T2I leg is bit-identical; think with the adapter loaded also captures and replays cleanly. Memory, same bytes with only
|
| Measure | Off | On | Delta |
|---|---|---|---|
| Peak host RSS | 7345 MiB | 8146 MiB | +801 MiB, during capture |
| Resident after the run | 3589 MiB | 3612 MiB | +23 MiB |
| Peak VRAM, schedule pinned to the 8192 bucket | 35107 MiB | 35971 MiB | +864 MiB (+2.46%) |
Everything I tried
| Change | Verdict | Evidence |
|---|---|---|
| Build the 3D RoPE tables once per forward | kept | 1,008 calls producing 17 distinct tables |
| Stop expanding K/V before the backend | kept | a +2.55% regression on its own; only pays once the mask is gone |
| Drop the all-zeros decode mask | kept | -5.66%, -8.19% together with the row above |
| Paged decode under a CUDA graph | kept | -29.93% / -31.27% on two prompts, same bytes, switch only, n=5 |
_repeated_blocks regional compile (your first suggestion) |
dropped | launches 19,974 -> 19,974, zero triton kernels, hits recompile_limit (8) and falls back to eager permanently, +2.9% |
Removing the forced sdpa_fallback |
dropped | init dies: the other backends invert a boolean mask and SenseNova passes an additive float one, so the line is load-bearing |
Removing the .item() sync |
dropped | 0.2% of the idle |
| Bucketing plus a mask | dropped | 7.96-11.55 ms per step against 1.03 ms unmasked |
think, main vs this branch (same prompt, same seed)
think output text does change: paged FA2 and SDPA are equally accurate against a float64 reference (mean error identical to four significant figures at n = 271/512/900/1500/3000), but they agree with each other only to 7.63e-06, and in a 382-step argmax loop that eventually flips a token. All four T2I cases are bit-identical between a312a94 and this branch.
Tests
tests/diffusion/models/sensenova_u1/ — 84 passed, 16 new. Verified by reverting: dropping the forward_und branch turns exactly one test red, weakening the flash_attn_varlen_func capability probe turns one red, and restoring the Python index in attend() turns one red. Bucket size does not affect results — pinning the schedule to 512, 2048 or 8192 produces the same png — and growth plus re-capture is covered end to end with a subdivided schedule that triggers two consecutive growths.
|
The TP LoRA sharding, retained adapter state, and RMSNorm regression test are addressed at
I stopped the full-model A/B at that failed feasibility gate, so the reported switch-only paged-decode speedup remains unconfirmed locally. |
_generate_text drives text-to-text and image-to-text, and it called _ar_step without the decode context, so neither path ever reached the paged cache the recipe documents for them. Build the context there as well. Unlike the think loop there is no write-back, because the caller returns the decoded string and never reads the cache again. The maskless-GQA accuracy gate demanded that every one of eight fixed-seed trials come out closer to the float64 reference, which a single tie breaks: on an L20X with CUDA 12.9 it reported 7/8 at kv_len=512. Compare the aggregate instead, which is the property the change actually claims, and print both means and the per-trial count when it fails. Measured margins on an A800: 2.18% at kv_len 271, 2.54% at 512, 5.72% at 2048. Signed-off-by: MrlixiangWE <mrdanaer@gmail.com>
|
Rechecked Environment: NVIDIA L20X, PyTorch 2.13.0+cu129, vLLM 0.27.0. All runs used one excluded warmup followed by three measured requests.
Timed NVML peak increased by 366 MiB (+1.01%) for TP=1 and 402 MiB/GPU (+2.02%) for TP=2. The real TP=2 distilled-LoRA path initialized and generated successfully on both ranks. The updated suites also pass locally: 16 CPU tests and 5 CUDA tests. AR output diverges across implementations, so image hashes/SSIM are not parity metrics, but all compared images were visually sharp and prompt-faithful. This confirms the performance claims and keeps memory below the 5% regression gate. |
| paged_cache=self.cache, | ||
| ).logits | ||
|
|
||
| def _capture(self, key): |
There was a problem hiding this comment.
In a fresh-cache feasibility run, engine readiness completed before this path was exercised. The first graph-enabled think request took 76.9 s, with the first capture occurring about 69 s after request start; subsequent post-warmup requests were 7.566 ± 0.131 s.
Can we warm a representative AR decode/capture before reporting readiness, or document and measure this cold first-request penalty separately? The current dummy diffusion warmup does not appear to cover paged decode.
There was a problem hiding this comment.
The tens of seconds is the language model's torch.compile, and where it lands depends on whether the startup warmup reaches that path. One session on an A800, inductor / triton / vLLM / CUDA JIT caches cleared for every arm, three requests each. "full" is diffusion_compile_granularity=full and "fixed" adds the warmup from 85acb30.
| Warmup | Compile | Readiness | First request | Steady | Above steady |
|---|---|---|---|---|---|
| active | default | 38.7 s | 6.12 s | 5.41 s | 0.71 s |
| active | full | 68.8 s | 13.13 s | 5.35 s | 7.77 s |
| active | full, fixed | 75.2 s | 10.44 s | 5.41 s | 5.03 s |
| skipped | full | 35.2 s | 45.67 s | 5.48 s | 40.19 s |
| skipped | default | 35.6 s | 8.78 s | 5.43 s | 3.35 s |
The fourth row is the shape you saw: readiness completes, then the first request pays tens of seconds. Your 76.9 s against a 7.566 s steady state is 69.3 s above steady, which matches the 69 s you measured to the first capture, and is 9.2x steady; that row is 40.19 s above steady, 7.3x. With the warmup active the same work is absorbed into readiness instead, so a run showing tens of seconds inside the request is one the warmup reached none of.
The default configuration compiles nothing at all: granularity is regional and regionally_compile returns early without _repeated_blocks, so the runner's "configured for lazy torch.compile" line is misleading.
85acb30 makes the dummy run warm the decode shape, which the text-to-image dummy request never touches: rows two and three, 7.77 s above steady before and 5.03 s after. It runs on both ranks under tensor parallelism.
A warmup cannot cover the capture half of your question: _decode_context builds the cache and the runner per request, so a startup capture is bound to buffers the first real request replaces — about 188 ms of every request rather than a one-off.
| return ((length + BLOCK_SIZE - 1) // BLOCK_SIZE) * BLOCK_SIZE | ||
|
|
||
|
|
||
| class PagedDecodeCache: |
There was a problem hiding this comment.
Can you document why this allocation remains model-local instead of using DiffusionKVCacheManager or ARDiffusionModelRunner?
PagedDecodeCache owns per-request tensors and an identity block table, so it bypasses scheduler admission, pooling, eviction, prefix reuse, and continuous-batching lifecycle. That is reasonable for the current max_num_seqs=1 path, but please state the limitation and link a follow-up/deletion condition before this is treated as general vLLM paged-KV support.
There was a problem hiding this comment.
Documented in the module docstring at 9e0561c, with the deletion condition: once the loop is driven by a scheduler-visible runner, or the pipeline serves more than one sequence per forward, this file goes and the manager takes over.
|
…rst request PagedDecodeCache is model-local because DiffusionKVCacheManager reserves once per scheduler request and ARDiffusionModelRunner lives under experimental, while SenseNova runs its whole autoregressive loop inside one pipeline forward, where the decode steps never surface to the scheduler at all. Say so in the module docstring, together with the condition under which this file should be deleted in favour of the manager, so it is not mistaken for general paged-KV support. Also record what a cold first request costs. Measured on one A800 with cold inductor and triton caches, the first request sits 1.43 s above steady state with the paged path on and 1.67 s above it with the path off, so the graph capture is not what dominates a cold start. Signed-off-by: MrlixiangWE <mrdanaer@gmail.com>
…cache cleared The earlier note cleared only the inductor and triton caches, which left vLLM's own compile cache warm and made the number unrepresentative of a cold start. With VLLM_CACHE_ROOT cleared as well, one A800 gives 1.43 s above steady state with the paged path on and 1.28 s with it off, so the capture accounts for about 150 ms and the rest is what a first request costs anyway. Signed-off-by: MrlixiangWE <mrdanaer@gmail.com>
…st capture The previous note put the first-request delta down to the graph capture. It is not: the capture is per request, twice for a think request that grows past the 512 bucket, at 150 ms for the first one in a process and about 94 ms after that, so about 188 ms of every request including steady state. The 1.3 s a first request costs above steady state appears with the paged path off as well. Signed-off-by: MrlixiangWE <mrdanaer@gmail.com>
The engine warms the model with a dummy text-to-image request, and think is off for it, so that request exercises the prefill shape and never the single-token decode one. Whatever a deploy config compiles for decode therefore lands on the first real request rather than on readiness. Detect the dummy request the way the other pipelines do and drive a two-token prefill followed by one decode step through the language model. Measured on an A800 with the inductor, triton, vLLM and CUDA JIT caches cleared and diffusion_compile_granularity=full, the first request drops from 7992.4 ms above steady state to 4846.0 ms while readiness absorbs 5.7 s of it. With the default granularity nothing compiles and the warmup costs a two-token forward. The warmup is best effort: a failure logs and leaves startup alone. Signed-off-by: MrlixiangWE <mrdanaer@gmail.com>
|
Ci failed |
|
is there any CPU overhead for client e2e time? for the kernels, what's the optimization opportunity? |
I checked the buildkite log. The error is |
All numbers below are re-measured, on 1x A800-80G with an Intel Xeon Gold 6336Y, TP=1, 1024x1024, seed 42, head CPU overhead2.31% of the AR decode window and 3.97% of the diffusion window at this head. Before the graph it was 37.29%. Three arms, one session, each differing from the previous by one thing. B is not a switch, only the arm that separates the cache from the graph.
Profiled: think on, 8 steps. Kernel time from the profiled run, wall clock from a clean one; idle is the difference.
Kernel time is flat across the three arms, so the idle that disappears is host time and not device work. Timing, think on, 50 steps, 2 warmups then n=10, P50:
On the decode loop A -> B is -1.10% and B -> C is -45.79%: the paged cache buys the static shape, the graph buys the time. Outside the model, wall - stage is 2.8-3.3 ms, 0.02% of e2e. KernelsGEMM. Both windows are above 96% busy, so there is little left to take on the host side.
|
Thanks for the review on this PR. Signed-off-by: MrlixiangWE <mrdanaer@gmail.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
| # A pipeline may hold captures of its own across requests. Level 2 | ||
| # discards the memory those were recorded against, so they have to | ||
| # go with `graph_runners`. | ||
| release = getattr(model, "release_captured_graphs", None) |
There was a problem hiding this comment.
The cleanup is necessary, but this establishes a second graph-lifecycle protocol in the infrastructure worker: it clears model_runner.graph_runners, then duck-types an individual pipeline for a hidden method. DiffusionModelRunner owns model compilation, caches, and execution resources, so please expose one model_runner.release_captured_graphs() method that clears its runners and delegates internally, or register the paged runner with its graph registry. Otherwise every future pipeline that retains captures must discover this unwritten hook, and an omission will silently survive level-2 sleep.
There was a problem hiding this comment.
Fixed in 519c9b2, with the single runner method.
|
Tested PR head Environment
Focused tests
Real-model smoke
The same-seed 8-step LoRA output differs from the no-LoRA control in 1,048,574/1,048,576 pixels (MAE 58.62), so the adapter is not a no-op. All generated PNGs were nonblank and had shape Online smoke The PR-source server reached The expected startup messages about the checkpoint lacking @MrlixiangWE Could you add the H200 validation to |
The worker cleared `model_runner.graph_runners` and then looked for a `release_captured_graphs` method on the pipeline, so a pipeline that keeps captures across requests had to be found through an unwritten hook, and an omission would have survived level-2 sleep silently. `DiffusionModelRunner` owns compilation and execution resources, so it owns the release: one `release_captured_graphs()` clears its own runners and delegates to the pipeline, and the worker asks it once. Signed-off-by: MrlixiangWE <mrdanaer@gmail.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Environment, commands, peak memory and the 50-step and 8-step timings measured on 1x NVIDIA H200. The recipe documented only the A800 numbers before. Signed-off-by: MrlixiangWE <mrdanaer@gmail.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
The tolerance case only called the method, so it would have passed against an implementation that created `graph_runners` on the way past. Signed-off-by: MrlixiangWE <mrdanaer@gmail.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
The A800 block listed only the OS and the vLLM version, so its numbers could not be placed against the H200 ones. Both blocks now carry the same fields, and the software stack turns out to be identical. Signed-off-by: MrlixiangWE <mrdanaer@gmail.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
The table listed the cases but not how to reproduce them. Signed-off-by: MrlixiangWE <mrdanaer@gmail.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
hsliuustc0106
added
ready
…ntory
The VSA backend's native Blackwell opt-in reads FASTVIDEO_VSA_SM100A, but
the name was never classified, so the inventory drift check failed:
FAILED tests/config/test_environment_variables.py::
test_statically_resolvable_environment_accesses_are_classified
AssertionError: assert {'FASTVIDEO_VSA_SM100A'} == set()
Classify it as model-specific with a "promote" disposition, matching the
other per-model kernel toggles (MING_CFM_CUDAGRAPH, OMNIVOICE_CUDA_GRAPH,
VLLM_OMNI_SENSENOVA_PAGED_DECODE): it selects a kernel route for the
MiniMax-H3 FastH3 path and belongs in typed diffusion attention config
rather than an environment switch.
Bump the reviewed snapshot counts accordingly. The reference page's
model-specific counts were already one behind main after vllm-project#6516, so refresh
them to the real values (58 model-specific, 35 promote) instead of leaving
a second inconsistency.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: princepride <wangzhipeng628@gmail.com>
…llm-project#6516) Signed-off-by: MrlixiangWE <mrdanaer@gmail.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
…llm-project#6516) Signed-off-by: MrlixiangWE <mrdanaer@gmail.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com> Signed-off-by: ZhengWG <zwg0606@gmail.com>
Resolve conflict in pipeline_sensenova_u1.py by combining SupportsStepExecution with new LoraLoaderMixin from vllm-project#6516. Signed-off-by: MaxCao <86maxcao@gmail.com>
Re-apply only the optimizations that measured a real win on the v0.24.0 work; experiments that showed no gain are not carried over (dual-stream CFG Plan A, row-chunk pipeline, batched CFG), and the MC2 MatmulAllReduce platform registration is dropped because the vllm-ascend snapshot in the v0.28.0 container removed the whole feature. Kept: - Phase 1: gate the CUDA-only Triton qk-norm+rope kernel to non-NPU platforms (pristine v0.28.0 verified to segfault on NPU model load) - Phase 2: npu_fused_rmsnorm_rope.py - npu_rms_norm x4 (fp32) + npu_rotary_mul x3 replacing ~34 unfused kernels - Phase 3: forward_fa_npu bool-mask conversion (True=attend) + explicit softmax scale for the mindiesd fused attention call - SDPA fallback restricted to non-NPU platforms - Fix A/B/C host-sync removal: exist_und/exist_gen kwargs, timesteps_py python mirror, grid_pairs python ints - FIA attention via npu_fused_infer_attention_score (opt-out with VLLM_OMNI_SENSENOVA_FIA=0), kept on top of the build-once position_embeddings threading from vllm-project#6516 (which supersedes the earlier id()-keyed RoPE cache approach) Verified E2E in quay.io/ascend/vllm-omni:v0.28.0 (CANN 9.1.0), cards 0/1 only, 2048x2048 / 50 steps / cfg 7.0 / seed 42, clean-protocol run (proc+port verified before each round, APIServer pid re-checked after): TP2 FIA=0 50.33s md5 72e1638c (= v0.24 baseline, bit-exact) TP2 FIA=on 49.00s md5 9beb7c94 (= v0.24 FIA, bit-exact, -2.7%) TP1 54.00s md5 03d629b8 TP1+cfg-par2 30.13s md5 03d629b8 (cfg-parallel, -40% vs TP2) Per-config md5s match the v0.24.0 stack exactly, i.e. the port is output-identical across versions. NPU sampling during the FIA=on run: cards 0/1 average ~69% AICore / ~225W across the whole 49s window. No FIA fallback / rope-fallback / segfault lines in any serve log. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: ljy19911228 <ljy19911228@163.com>
…llm-project#6516) Signed-off-by: MrlixiangWE <mrdanaer@gmail.com> Co-authored-by: Hongsheng Liu <liuhongsheng4@huawei.com>
Purpose
FIX #6471
SenseNova-U1.5-8B-MoT runs on the existing pipeline, but its distilled 8-step LoRA loaded
without error and changed no weight. This fixes that, and cuts denoise latency.
Propose
--lora-backend distillwas accepted and then warnedPipeline does not support loading distilled LoRA weights for now, generating without the adapter. Implementload_lora_weightson
SenseNovaU1Pipeline, and raise when nothing matches instead of loading a no-op._prepare_lora_deltapicked stacked mappings withif param_name not in base_keyand did notstop at the first hit.
.qkv_projis a substring of.qkv_proj_mot_gen, so both rules firedand the delta came out twice the height of the parameter. Match on the tail and stop at the
first hit. The fused names shipped by QwenImage and Wan2.2 do not overlap, so this selects the
same shards for them; a test pins that.
Qwen3RMSNorm.forwardused an eager cast chain. UseF.rms_norm, which keeps the accumulationin fp32 through the weight multiply instead of rounding to bf16 first.
mypyflags inloader.py; the file is now clean.recipes/SenseNova/SenseNova-U1.5.md, and list U1.5 in supported-models anddiffusion-features.
Test Plan
Test Result
Latency, 1x A800-80G, seed 42, n=5 medians
Peak VRAM unchanged, 34364 -> 34340 MiB. Five-run spread 0.3-3.7%.
Distilled LoRA, 1024x1024, 8 steps, cfg 1.0
--lora-pathgiven--lora-path--lora-pathgiven--lora-pathOn main the first two rows differ by +0.48%: the adapter is dropped, so both runs use the base
weights. With this PR it is applied, 588 LoRA keys into 168 parameters, and still costs nothing
against the same-step control, -0.23%, inside the 1.16% and 3.70% run spreads.
Accuracy of the RMSNorm change
Mean relative error against a float64 reference, same bf16/fp16 inputs:
F.rms_normLower in all 12 shape/dtype combinations tested: the eager path rounds to bf16 before the weight
multiply,
F.rms_normrounds once. CFG-parallel output is byte-identical to single GPU.Tests
tests/diffusion/models/tests/diffusion/lora/core_model and cpuCompared as failing-test-ID sets rather than counts: the two sides are identical in all three, at 7,
2 and 139 ids. The pass-count differences are the tests this PR adds.
sensenova_u1/alone is 86passed, and restoring the pre-PR implementation under these tests turns 20 of them red.
Serving
vLLM Version: 0.27.0
vLLM-Omni Commit: 3781214