Repository navigation
Conversation
|
This PR appears to belong to: docs/design/module/diffusion/diffusion_model_integration.md, docs/design/module/diffusion/index.md. Module owners: @david6666666 @wtomin @Isotr0py Routing: @david6666666 via module of the changed files, module named in the PR description, CODEOWNERS; @wtomin via module of the changed files, module named in the PR description, CODEOWNERS; @Isotr0py via module of the changed files, module named in the PR description @yuweih205, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer. Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment. |
64ba7eb to
4461158
Compare
|
Self-review: the double-stream attention takes an explicit |
Dong1017
left a comment
Dong1017
left a comment
There was a problem hiding this comment.
Review (static + H200 unit run at 4461158f)
What this solves: the per-block eager chain (video RMSNorm → RoPE, text RMSNorm, three cats) across HunyuanVideo 1.5's 54 dual-stream blocks becomes one Triton launch producing the joint [B, S_v+S_t, H, D] QKV attention consumes. Consumer-side increment over #7560: per-forward packed rope table with identity rows for the un-rotated text tokens, a no-allocation gate (SP / non-bf16 / CPU / token floor), and per-call dispatch with the eager chain preserved line-for-line as fallback.
Verification: I ran the 4 tests on H200 (vllm 0.29.0, torch 2.13.0+cu130, triton 3.7.1, clean head): 4 passed. Note the head tree requires vLLM 0.29 (compute_layout_strides, diffusion_kv/layout.py:23) — the hybrid pre-0.29 tree in the test plan is not what the ready pipeline will execute. Quality/perf A/B is same-tree, same-workload, A B B A with a rounding-order control arm — baseline consistent; e2e numbers remain author-reported.
Findings: no blockers. Two inline items (test wording vs "bitwise" claim; second private copy of the table helper). Verified correct: [video|text] order end-to-end, identity-row exactness (fp32 x·1 − pair·0), weight slots/eps/coeff dtype, interleaved convention (is_neox_style=False ⇔ kernel interleaved=True, rotary 128 = head_dim), SP and fallback paths.
RFC #7382: fits the §4.2 QK/RoPE consumer-adoption track (H3 → Boogu → Flux.2 → this) with per-consumer validation and retained references as required; this is not the §4.3 gated-residual delivery. Two alignment items: (1) the RFC's public entry is diffusion/layers/ops/ plus a public eligibility query — this PR adds the second consumer of private _fused_cuda_supported, so I'd land that migration before a third consumer appears; (2) HunyuanVideo 1.5's gated-residual adoption is named in §4.3 but stays out of this PR — worth an explicit pointer in the RFC's PR mapping.
Coordination: merge order behind #7560; ops-package/eligibility-query ownership; CUDA CI wiring for the tests/diffusion/layers/ op suite (only the NPU variant runs today); remaining dual-stream candidates (flux1/sd3/qwen_image/cosmos3/longcat/ovis) each need text-rows (identity vs real angles), rope style, head-count/eps contract, and SP routing checked before adoption.
4461158 to
9833942
Compare
|
@Dong1017 Thanks for the H200 run. Done: private packer removed ( |
9833942 to
eaa7d58
Compare
Omni ReviewBot triage noteResolved as of |
hsliuustc0106
added
the
Kernel optimization
Omni ReviewBot: no human activity for 9 days@yuweih205 this pull request has had no human commit, comment or review since 2026-09-19. Please confirm the current plan and next step. The author or a maintainer decides whether to change the PR state. To keep it moving, any one of these is enough: push an update, reply to the open blocker, or post the current plan and timeline. |
Omni ReviewBot routing recordAssigned Strict on cursor (cursor-grok-4.6-high) under experiment |
Signed-off-by: HuangYuwei <yuweih205@gmail.com>
f84ebbd to
01752b9
Compare
Signed-off-by: HuangYuwei <yuweih205@gmail.com>
Signed-off-by: HuangYuwei <yuweih205@gmail.com>
01752b9 to
f7e730a
Compare
Signed-off-by: HuangYuwei <yuweih205@gmail.com>
4b25e59 to
3020de9
Compare
Supplemental eager bitwise measurements and queued production checkExisting PR performance results remain as recorded. The retained operator scripts and raw results stay pinned to their original artifact revision. Measured operator case: H200; B1, 2048 image/video + 256 text tokens, 16 heads, head dimension 128; BF16 activations/weights/RoPE table, full interleaved RoPE on image/video rows and identity text rows, epsilon
Eager CUDA-event medians: six balanced-order samples, 20 warmup calls, 100 calls per sample. These durations cover Q/K/V preparation with the packed RoPE table reused; table packing is excluded. They establish operator parity and latency for this case. Prepared production setting: change Actual r3 target: prepared source The three eager arms use the same initialized model/state/inputs:
H200, vLLM Source manifests and reproduction instructions and the eight-worker runner pin the settings. The eight-GPU r3 job is queued in MOVA2.0纯交付分区; full-model parity/performance is not yet established. |
Fresh eager measurements: bitwise reference parity with retained speedupThe H200 performance follow-up is now complete. The measured CUDA-reference numerical preset has zero output storage-byte mismatches versus the original eager chain, with a measurable transformer-forward saving. Tested implementation: prepared source Settings: Scope: eager, random-weight, full-width shallow HunyuanVideo1.5 transformer, 16 heads × 128 dimensions, BF16, 4096 image/video tokens and 512 configured text tokens, B1/B2, seed
The original RMSNorm provider is pinned and checked as Timing: H200 141 GB, vLLM Raw records: B1, B2. See the reproduction procedure and all raw results. The fast arm and its separate timing/parity records are retained in those JSON files. AI assistance: Codex ran and checked the GPU measurements and prepared this additive evidence note. |
Omni ReviewBot attempt recordReview attempt ended as failed (failed; retrying strict/cursor/cursor-grok-4.6-high in 120s (try 2 of 3)). |
Omni ReviewBot attempt recordReview attempt ended as failed (failed; retrying strict/cursor/cursor-grok-4.6-high in 600s (try 3 of 3)). |
Omni ReviewBot attempt recordReview attempt ended as failed (failed; falling back to direct/cursor/auto). |
Signed-off-by: HuangYuwei <yuweih205@gmail.com>
…e fused Q/K path HunyuanVideo-1.5 passes `cuda_rms_norm=True` to the joint op: the fused path is now bit-identical to vLLM RMSNorm + RoPE (identity rows for text) in eager mode instead of differing by the Triton reduction order. - `fused_joint_qkv_norm_rope` takes `cuda_rms_norm` like the single-stream op (BF16, head_dim = rotary_dim = 128, FMA contraction off) and drops `interleaved`: its half-split mode was never reachable in production. - `pack_qk_norm_rope_table(cuda_rms_norm=True)` returns None outside that geometry, so consumers never fuse outside the bitwise contract. - Shared tests: joint op bitwise against vLLM's CUDA RMSNorm, the Flux.2 eager-chain test exact under `cuda_rms_norm`, compile and CUDA-graph coverage for both modes. Signed-off-by: HuangYuwei <yuweih205@gmail.com>
…nels One-warp programs of eight heads instead of four are 8-15% faster under CUDA-graph replay on H200 across 26 joint and single-stream geometries (320 to 16K tokens, 8 to 64 heads, fast and CUDA-matching modes) with bitwise-identical output. The only slower case is 20 heads in the fast mode (+2.6%), where eight heads pad the head count to 24 slots.
4 participants
Purpose
Fuse video/text Q/K preparation and concatenation. Video tokens rotate; trailing text tokens use identity rotations. Keep video-first ordering and the existing sequence-parallel path.
The fusion paths are selected automatically; there is no fusion environment-variable override. Unsupported devices, dtypes and geometry retain their existing paths. All six PRs contain identical shared operator and test files and only their own model integration; any PR can merge first.
In eager mode the fused path reproduces the vLLM CUDA RMSNorm reduction order and keeps the rotation uncontracted (
cuda_rms_norm=True), so video/text Q/K/V (identity rows for text) are bit-identical to the unfused chain. Compiled graphs (regionaltorch.compile,dynamic=True) keep using the fused custom op. There the unfused chain still runs vLLM's CUDA RMSNorm kernel as an opaque op while Inductor generates the RoPE/concat arithmetic; on the measured shallow shapes the fused output was byte-identical to that compiled chain as well (see Test Result).cuda_rms_norm=Falseat the fused calls restores the previous revision's Triton-order numerics (kernel-level cost of the CUDA-order reduction measured below).Test Plan
9d96e8566.Test Result
On 2026-10-17 (infra-debug NVIDIA H200; Python 3.12, vLLM 0.31.0, Torch 2.13.0+cu130, CUDA 13.0, Triton 3.7.1; eager, world/TP/SP size 1.)
tests/diffusion/models/hunyuan_video/test_hunyuan_video_15_fused_qk_norm_rope.py: 4 passed, 0 failed/skipped.Eager parity and timing (original arm = table packing disabled, i.e. the unfused chain; bitwise arm = this head's default,
cuda_rms_norm=True; fast arm = the same launch withcuda_rms_norm=False, i.e. the previous revision's Triton-order numerics; CUDA-event medians over three ABBA rounds, each fused arm interleaved with the original arm; exact byte comparison of the full output before and after timing):cuda_rms_norm=False)Kernel alone (joint op, 16 heads, 512 text + 4096 image tokens per sample; CUDA-graph replay medians from the heads-per-program campaign on this operator (26 geometries × fast/bitwise × 4/8/16 heads per program, infra-debug H200, job
hyw-vomni-hp-validation-1017-r1; eight heads were bitwise-identical to four in all 52 path/mode/geometry cells and faster in 51)): four → eight heads per program moves B=1: bitwise 42.6 → 37.6 µs, fast 38.3 → 34.8 µs; the bitwise default is now +8.0% of the fast kernel and −1.8% of the previous revision's fast kernel.Model level, the bitwise arm's latency is +0.21% (B=1) / +0.22% (B=2) relative to the fast arm — inside the ±0.5% run-to-run spread of this harness. To take the previous numerics instead, pass
cuda_rms_norm=Falseat the fused calls invllm_omni/diffusion/models/hunyuan_video/hunyuan_video_15_transformer.py(joint launch, table packer); this is a code-level switch, no environment variable.Production compile (same harness,
regionally_compile(transformer, dynamic=True)as the diffusion model runner applies it, production IR op priority['vllm_c', 'native']; the previous public head of this PR was run the same way in the same job):Eager in the same run: −4.68% / −4.59%. Both heads compile with the same 11 graph breaks per forward (all from
_vllm_fa2_C.varlen_fwdhaving no fake implementation, which the non-fullgraph production path tolerates), no recompilation across repeated calls, 2 fused launches per forward, and vLLM'srms_norm_kernel(8 launches per forward) present in the unfused chain in both eager and compiled runs.The model-level and request-level rows above were measured at the previous head (
a3cd4526c, four heads per program); the eight-heads commit changes only the kernel launch shape with bitwise-identical output, so the parity results carry over and the fused arms gain the kernel delta on top of the figures shown.These are operator, component and forward-level regressions on the stated shapes; pretrained end-to-end accuracy/performance on other models was not rerun for this update.
AI assistance: Claude Code contributed the initial implementation and this revision; Codex simplified automatic selection, implemented and checked CUDA-reference numerics, and ran earlier validation.