Add comprehensive NVFP4 KVCache technical documentation and upstream PR/issue analysis

[Copilot]

Motivation

NVFP4 KVCache implementation details were scattered across source code, upstream PRs, issues, and documentation. No single reference existed covering design rationale, implementation architecture, and development history.

Modifications

Documentation Created

NVFP4_KVCACHE_DESIGN_SUMMARY.md (20KB) - Technical reference:

• E2M1 quantization format: 4-bit FP with block-based microscaling (16 elements/block, 8-bit scale factors)
• KVFP4QuantizeUtil implementation: batched_quantize/batched_dequantize with @torch.compile
• Memory pool integration for MLA and MHA token pools
• Backend compatibility matrix: MLA (FlashInfer, FlashMLA, Cutlass, TRTLLM, FA4) and MHA (FA4, Triton, Torch Native, TRTLLM)
• Memory savings: ~3.56× vs BF16 (4.5 effective bits/value including scale overhead)
• Accuracy benchmarks: Large models (200B+) show minimal degradation on simple tasks, more pronounced on complex reasoning

NVFP4_KVCACHE_PRS_AND_ISSUES.md (18KB) - Upstream analysis:

• 15+ PRs from sgl-project/sglang:
  • PR feat: Add FP4 (E2M1) KV Cache Support with Quantization Utilities for MLA sgl-project/sglang#10078 (Sep 2025): Foundational FP4 MLA support by @JackChuang
  • PR feat: Add FP4 (E2M1) KV Cache Support for MHA sgl-project/sglang#12612 (Nov 2025): Extended to MHA by @JackChuang
  • PR Enable native ModelOpt quantization support (3/3) sgl-project/sglang#10154: ModelOpt integration by @Edwardf0t1
  • PR NVFP4 KV Cache for SM120 sgl-project/sglang#18314 (Jan 2026): Latest TRTLLM optimizations (550µs→70µs) by @samuellees
• 8+ issues tracking features, bugs, and strategic roadmap
• Four-phase timeline: Foundation (Jul-Sep 2025) → Expansion (Oct-Nov 2025) → Refinement (Dec 2025) → Recent work (Jan 2026)

NVFP4_SUMMARY_README.md (4.4KB) - Navigation guide with quick links to implementation files and key takeaways.

Technical Details Documented

# Quantization: [B, M, N] → [B, M, N/2] packed + [B, M*N/16] scale factors
# E2M1 values: [0, 0.5, 1, 1.5, 2, 3, 4, 6]
cache_k_fp4, scale_factors = KVFP4QuantizeUtil.batched_quantize(cache_k)
cache_k_dequant = KVFP4QuantizeUtil.batched_dequantize(cache_k_fp4, scale_factors)

Sources: kvfp4_tensor.py, memory_pool.py, attention backends, upstream PRs sgl-project#10078/sgl-project#12612, existing docs.

Accuracy Tests

N/A - Documentation only.

Benchmarking and Profiling

N/A - Documentation only.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review Process

1. Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
4. After green CI and required approvals, ask Merge Oncalls to merge.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.