[Quantization] FP8 quantization framework for diffusion attention#1413

Draft

lishunyang12 wants to merge 48 commits intovllm-project:mainfrom

lishunyang12:fp8-kv-quantization

Commits on Mar 31, 2026

[Quantization] Add FP8 KV quantization for diffusion attention layers
lishunyang12
committed

Commits on Apr 2, 2026

Merge branch 'main' into fp8-kv-quantization
lishunyang12
authored
[Quantization] Align FP8 attention with design doc: Q quantization, CLI rename, joint scales
lishunyang12
committed
Add --kv-cache-dtype flag to text_to_video.py
lishunyang12
committed
Fix FP8 attention not activating: lazy-resolve config from forward context
lishunyang12
committed
Add debug logging for FP8 attention resolution
lishunyang12
committed
Wire kv_cache_dtype through default stage config
lishunyang12
committed
Use vLLM fused CUDA kernel for FP8 quantization
lishunyang12
committed
Fix dequant dtype: cast result to output_dtype after f32 scale multiply
lishunyang12
committed
Skip FP8 quantization when padding mask is present
lishunyang12
committed
Add debug scripts for FA3 FP8 capability check and mask stats
lishunyang12
committed
Fix FA3 FP8 descale param names and enable FP8 varlen path
lishunyang12
committed
Expand debug script: micro-benchmarks, varlen FP8, layer breakdown
lishunyang12
committed
Fix total_mem -> total_memory for PyTorch 2.10+
lishunyang12
committed
Fix descale shape: FA3 requires (batch, num_kv_heads) not scalar
lishunyang12
committed
Add debug_shapes.py: log actual Q/K/V shapes during inference
lishunyang12
committed
Fix debug_shapes.py: add __main__ guard for multiprocessing spawn
lishunyang12
committed
Update benchmarks with actual HunyuanVideo shapes: 50345 tokens, 16 heads
lishunyang12
committed
Implement delayed scaling: cache scales to skip amax on subsequent calls
lishunyang12
committed
Disable delayed scaling (green output bug), use dynamic only
lishunyang12
committed
Test KV-only FP8: keep Q in BF16, only quantize K/V
lishunyang12
committed
Skip varlen for FP8: use regular FA3 path to avoid varlen descale bug
lishunyang12
committed
Revert to QKV FP8: FA3 requires same dtype for Q/K/V
lishunyang12
committed
Zero out padding K/V before FP8 quantization
lishunyang12
committed
Auto-fallback to BF16 for sequences >16K tokens
lishunyang12
committed
Use num_splits for FP8 accuracy at long sequences instead of seqlen cutoff
lishunyang12
committed
Add debug_fa3_version.py: check FA3 builds and two-level accumulation
lishunyang12
committed
Add debug_vllm_fa.py: check vLLM's bundled flash-attention for FP8 fix
lishunyang12
committed
Switch FP8 path to vLLM's bundled FA3 (has two-level accumulation fix)
lishunyang12
committed

Commits on Apr 5, 2026

Merge branch 'main' into fp8-kv-quantization
lishunyang12
authored

Commits on Apr 6, 2026

Optimize FP8 attention: zero-overhead quantization, direct FA3 path
lishunyang12
committed

Commits on Apr 7, 2026

Commits on Apr 8, 2026

Update vllm_omni/diffusion/attention/backends/abstract.py

lishunyang12
and
gcanlin
authored

Commits on Apr 9, 2026