Enable SM86 MXFP4 sparse-indexer cache

benchmarks/kernels/deepseek_v4/fp4_indexer_sm86/CAPACITY-BASELINE.md

@@ -0,0 +1,45 @@
+# FP4 DS-MLA capacity baseline
+
+This report records the allocation baseline before enabling an MXFP4 sparse-indexer cache on RTX 3090. The runtime uses `fp4_ds_mla` for the main MLA cache and FP8 for the sparse-indexer cache.
+
+## Pinned inputs
+
+- Runtime source base: `Whamp/vllm@81593507f`
+- FP4 integration source: `Whamp/vllm@633815f6889d9d033aefa04bf40cb270d5b6a3f1`
+- Accounting image: `sha256:4f11d99672280c34ad32c271c20195bb76aaad35c3cdc780177f946dd5cfacd6`
+- Main FP4 image: `sha256:eb94d5049bf4d8d55c335ac1d2445382a811b7312d28e3e73088011a8103e181`
+- Hardware: four RTX 3090 GPUs, TP=4
+- Safety policy: 230 W power limit and 210-1650 MHz graphics-clock range
+- Cache profile: `fp4_ds_mla`, `max_num_seqs=2`, `max_num_batched_tokens=256`
+
+## Measured capacity
+
+| Configured context | KV tokens | KV allocation | Packed blocks | Near-ceiling NIAH | Serving swap | Stress headroom |
+| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| 160,000 | 179,743 | 841,727,040 B | 965 | 156,649 tokens | 0 KiB | 27 MiB |
+| 170,000 | 178,887 | 825,154,176 B | 946 | 166,470 tokens | 0 KiB | 27-28 MiB |
+| 175,000 | 178,050 | 814,687,104 B | 934 | 173,058 tokens | 0 KiB | 27-28 MiB |
+
+All three profiles reached API readiness and passed the fast stress probes, tool and reasoning checks, and exact needle retrieval near their configured ceilings. The planner's storage size matched storage-deduplicated runtime allocation on every rank with a zero-byte reconciliation delta.
+
+None is release-safe under the project's 1 GiB free-VRAM gate. The 175K profile is the highest tested functional ceiling, not a promotion candidate.
+
+## Packed layout at 175K
+
+The planner packs 167 cache specs into five groups. Every allocated block uses the largest group stride, 872,256 bytes.
+
+Group 0 contains:
+
+- 21 ratio-4 FP8 indexer rows: 132 physical bytes per token, 8,704-byte pages
+- 21 ratio-4 FP4 MLA rows: 368 physical bytes per token, 23,584-byte pages
+- 20 ratio-128 FP4 MLA rows: 368 physical bytes per token, 1,056-byte pages
+
+Those specs use 699,168 bytes of each group-0 block. The 872,256-byte global stride leaves 173,088 bytes at the end of group 0.
+
+Changing the indexer row from 132-byte FP8 to 68-byte MXFP4 is predicted to reduce its aligned page from 8,704 to 4,608 bytes. Group 0 would shrink to 613,152 bytes, but its tail would grow to 259,104 bytes. The global stride would remain 872,256 bytes because group 3 is still the largest group.
+
+At 175K, the indexer change should reduce logical model-length bytes by 58,834,944 bytes per rank. It should not increase the physical KV pool or token capacity unless packing or group composition also changes. The post-port runtime report must confirm this prediction.
+
+## Evidence
+
+`evidence/capacity_before_indexer/` contains the resolved Compose files, image inspection, startup and allocation logs, planner JSON, GPU and swap snapshots, verification output, final container state, and release snapshots for all three profiles. `SHA256SUMS` binds every file.

benchmarks/kernels/deepseek_v4/fp4_indexer_sm86/RESULTS.md

@@ -0,0 +1,122 @@
+# DeepSeek V4 SM86 MXFP4 indexer results
+
+## Decision
+
+The SM86 MXFP4 sparse-indexer cache is a validated opt-in capacity mode. It must not replace the existing FP8 indexer default.
+
+It increases tested long-context capacity materially and preserves the focused behavioral-quality result. It also reduces prefill throughput, especially at deep context, and every tested capacity profile remains below the project's 1 GiB free-VRAM release gate.
+
+## Compared configurations
+
+Both arms use the same DeepSeek V4 GGUF-TP model, four RTX 3090 GPUs, TP=4, `fp4_ds_mla` for the main MLA cache with its retained BF16 RoPE section, `max_num_seqs=2`, `max_num_batched_tokens=256`, CUDA graphs, native Ampere FlashMLA, hierarchical all-reduce, and the 230 W / 210-1650 MHz GPU safety policy.
+
+The only cache-format difference is the 21 compression-ratio-4 sparse-indexer caches:
+
+- Baseline: 128 FP8 values plus one FP32 scale, 132 bytes per row.
+- Candidate: 64 packed E2M1 bytes plus four UE8M0 scale bytes, 68 bytes per row.
+
+The candidate uses dedicated SM86 Triton prefill and paged-decode logits kernels. Existing FP8 and DeepGEMM branches remain unchanged.
+
+## Allocation accounting
+
+### Indexer row and page
+
+| Quantity | FP8 indexer | MXFP4 indexer | Change |
+| --- | ---: | ---: | ---: |
+| Semantic key width | 128 values | 128 values | Equal |
+| Physical row | 132 bytes | 68 bytes | -48.5% |
+| Real 64-token page | 8,448 bytes | 4,352 bytes | -48.5% |
+| Allocated page | 8,704 bytes | 4,608 bytes | -47.1% |
+| Per-layer logical bytes at 175K | 5,953,536 | 3,151,872 | -47.1% |
+| All 21 indexer layers at 175K | 125,024,256 | 66,189,312 | -58,834,944 bytes |
+
+The 256-byte page-padding quantum remains unchanged.
+
+### Packed groups
+
+The packed layout still has five groups and an 872,256-byte global block stride. Group 0 shrinks from 699,168 to 613,152 bytes per block, but another group still determines the global stride. The indexer row change therefore does not reduce bytes per allocated packed block.
+
+The 175K request's logical model-length requirement falls from 478,230,912 to 419,395,968 bytes per rank, an exact 58,834,944-byte saving. Available KV-pool memory rises by 111,214,592 bytes, from 815,005,369 to 926,219,961 bytes per rank.
+
+The accounting proves both changes but does not separately attribute the remaining 52,379,648-byte increase in available pool memory. Smaller FP4 indexer profiling and gather workspaces are a source-level candidate, not a measured attribution. Because available pool memory increases while the block stride stays fixed, vLLM allocates more 872,256-byte blocks.
+
+### Runtime reconciliation
+
+| Profile | Indexer | Available pool | Blocks | Allocated pool | KV tokens | Planned versus observed |
+| --- | --- | ---: | ---: | ---: | ---: | --- |
+| 175K baseline | FP8 | 815,005,369 B | 934 | 814,687,104 B | 178,050 | exact |
+| 175K candidate | MXFP4 | 926,219,961 B | 1,061 | 925,463,616 B | 202,260 | exact |
+| 200K candidate | MXFP4 | 884,276,921 B | 1,013 | 883,595,328 B | 199,409 | exact |
+
+At matched 175K, reported KV-token capacity rises **13.60%**. The larger configured context consumes more non-KV runtime memory, so the 200K profile has fewer available blocks than the 175K profile.
+
+## Long-context correctness
+
+Before the indexer port, the 175K baseline recalled the exact needle at 173,058 prompt tokens. The 200K MXFP4-indexer candidate recalled exact needles at 194,812 and 195,812 prompt tokens. This is a **13.15%** increase in the highest directly validated prompt length and a **14.29%** increase in configured context.
+
+The 200K profile is a functional capacity ceiling, not a release-safe operating point:
+
+- 25-26 MiB free VRAM per RTX 3090 after near-ceiling work
+- zero serving-process swap after normalization
+- zero VRAM growth across the ceiling ladder
+- exact needle recall through 195,812 tokens
+
+The normal release gate requires 1 GiB free VRAM per card. None of the pre-indexer 160K, 170K, or 175K profiles, nor the post-indexer 175K or 200K profiles, meets that margin.
+
+## Performance
+
+Canonical protocol: three warmups and five measured narrative/code runs, followed by three cache-busted runs at each prefill depth.
+
+| Metric | FP8-indexer baseline | MXFP4-indexer candidate | Change |
+| --- | ---: | ---: | ---: |
+| Narrative decode | 80.36 tok/s | 77.32 tok/s | -3.78% |
+| Code decode | 80.37 tok/s | 77.32 tok/s | -3.80% |
+| 10K prefill | 524.87 tok/s | 499.84 tok/s | -4.77% |
+| 90K prefill | 495.79 tok/s | 348.35 tok/s | -29.74% |
+| Concurrency-2 aggregate | 127.27 tok/s | 122.30 tok/s | -3.91% |
+| Concurrency-2 per stream | 63.74 tok/s | 61.24 tok/s | -3.92% |
+| Post-run VRAM growth | 0 MiB | 0 MiB | equal |
+
+The deep-prefill regression is the main reason not to promote MXFP4 indexer caching as the default. The kernel is correctness-oriented SM86 enablement rather than a tuned prefill implementation. The concurrency-2 comparison uses the identical three-warmup/five-measured 512-token pair harness in both arms; every measured completion ended by the length cap.
+
+## Behavioral quality
+
+The matched BenchLocal quick gate produced:
+
+| Indexer | Pass@1 | Pass@3 |
+| --- | ---: | ---: |
+| FP8 baseline | 27/30 | 27/30 |
+| MXFP4 candidate | 26/30 | 27/30 |
+
+The candidate had one additional pass@1 miss, while pass@3 remained equal. A single 30-case sampled run does not establish a quality difference. This result supports opt-in testing but not a stronger quality-equivalence claim.
+
+## Kernel and runtime gates
+
+Passed on RTX 3090 SM86:
+
+- E2M1/UE8M0 numerical comparison using a table-distance, ties-to-even reference independent of the production threshold cascade
+- Partial sequence, non-block-aligned context, `next_n=4`, and paged block-table cases
+- Top-k set and pairwise-order comparison, downstream gathered-output comparison, and a tied-boundary case
+- Software E2M1 query and cache writers
+- Deterministic CUDA-Graph replay
+- Targeted paged-decode and fused-query-writer Compute Sanitizer memcheck: zero errors
+- Targeted paged-decode and fused-query-writer Compute Sanitizer racecheck: zero hazards
+- Runtime-generated `_mxfp4_mqa_logits_kernel.sm_86.cubin` and `_mxfp4_paged_mqa_logits_kernel.sm_86.cubin`
+- Full TP=4 model load and runtime dispatch
+- `verify-full.sh`
+- `verify-stress.sh`, including tool, reasoning, coding-agent, and exact long-context integration probes; NIAH is integration evidence, not a numerical-format oracle
+
+The MXFP4 logits kernels reuse the inherited FP8 autotune configuration and are not described as tuned SM86 kernels. The measured deep-prefill regression remains the performance evidence for that limitation.
+
+## Release recommendation
+
+Keep the FP8 indexer as the production default. Ship the MXFP4 indexer as an explicit, documented capacity experiment for users who value approximately 13% more validated context enough to accept about 4% decode/shallow-prefill loss and about 30% deep-prefill loss.
+
+Do not describe the 200K profile as release-safe. Its 25-26 MiB physical VRAM margin is a measured ceiling. A future default requires either a tuned MXFP4 prefill kernel or another VRAM reclaim that restores the 1 GiB margin without sacrificing the demonstrated context gain.
+
+## Evidence
+
+- `evidence/capacity_before_indexer/`: 160K, 170K, and 175K startup, allocation, and stress evidence.
+- `evidence/baseline_comparison/`: exact pre-indexer benchmark, BenchLocal result, image identity, and launch provenance.
+- `evidence/after_indexer/`: kernel, sanitizer, cubin, TP=4 startup, allocation, quality, benchmark, 175K stress, 200K stress, swap, safety, and release evidence.
+- Each evidence directory contains a `SHA256SUMS` manifest.

benchmarks/kernels/deepseek_v4/fp4_indexer_sm86/REVIEW.md

@@ -0,0 +1,81 @@
+# Independent review record
+
+## Scope
+
+- Fixed point: `Whamp/vllm@81593507f`
+- Accounting commit: `3f54512de`
+- Initial indexer commit: `187116cef`
+- Review axes: repository/Python standards, implementation spec, and adversarial GPU-kernel safety
+
+## Reviewers
+
+- OpenAI Codex GPT-5.6 Sol High: standards and spec review
+- Cursor Grok 4.6 Extra High: adversarial kernel review
+- Z.ai GLM 5.3 Max: unavailable; the provider returned repeated HTTP 429 quota-exhaustion responses and produced no review
+- OpenAI Codex GPT-5.6 Sol Medium: fix-verification synthesis
+
+## Findings and resolution
+
+### Fail-closed platform handling
+
+**Finding:** `assert` guarded the general architecture boundary, so optimized Python could remove it outside the earlier SM8x model-selection check.
+
+**Resolution:** `supports_mxfp4_indexer_cache` now also requires an NVIDIA CUDA platform and accepts only SM86 or SM100-family devices. Metadata construction raises `ValueError` unconditionally on unsupported platforms. The unfused MXFP4 insertion branch raises `NotImplementedError` rather than relying on `assert`. Seven architecture cases cover SM86, SM100, SM80, SM89, SM90, SM120, and non-CUDA platforms.
+
+### Top-k and downstream-output evidence
+
+**Finding:** The initial test permitted one wrong top-k ID, tolerated broad order disagreement, and used a weak four-dimensional periodic output proxy.
+
+**Resolution:** The test now requires exact top-k set equality. It requires pairwise order agreement whenever the reference margin exceeds twice the observed maximum logit error, leaving only numerically ambiguous near ties unordered. It gathers deterministic random 64-dimensional value rows through the selected set and requires bit-exact set-reduction equality. A separate tied-boundary case remains.
+
+### Deployed `clean_logits=False` path
+
+**Finding:** The paged test only exercised `clean_logits=True`, while production leaves out-of-range tail storage uninitialized and relies on sequence-length-bounded top-k.
+
+**Resolution:** Tests now cover both modes for `next_n=1` and `next_n=4`. The false mode poisons unwritten logits with maximum FP32 values, checks valid logits, then passes the unsliced poisoned tensor through the real `_C.top_k_per_row_decode` operation with production sequence lengths and requires exact selected sets. CUDA-Graph replay remains deterministic.
+
+### Oracle independence
+
+**Finding:** The first test reference copied the production E2M1 threshold cascade.
+
+**Resolution:** The reference now selects from a hard-coded E2M1 value table by minimum distance and resolves exact ties by even code parity. This is algorithmically independent of the production threshold cascade. UE8M0 decode and downstream FP32 accumulation remain explicit in the reference.
+
+### Fused insertion-to-gather integration
+
+**Finding:** Direct logits and writer tests did not prove that the existing cache gather operation interpreted the new segregated page correctly.
+
+**Resolution:** Every FP4 fused indexer-writer case now gathers through `cp_gather_indexer_k_quant_cache` and requires byte-exact packed-value and UE8M0-scale equality with the independent writer reference.
+
+### Baseline provenance
+
+**Finding:** Initial durable evidence omitted the exact FP8-indexer benchmark and BenchLocal inputs used for reported deltas.
+
+**Resolution:** `evidence/baseline_comparison/` now contains the exact canonical benchmark log, raw BenchLocal JSON, benchmarked image inspection, final-equivalent resolved profile, provenance note, and SHA-256 manifest. The note distinguishes the benchmarked dev6 image from the later reproducibly rebuilt equivalent image.
+
+### Report claims
+
+**Finding:** The first report over-attributed the increase in available pool memory to smaller profiling/gather workspaces and did not narrowly scope sanitizer evidence.
+
+**Resolution:** The report now separates the exact 58,834,944-byte logical indexer saving from the 111,214,592-byte observed available-pool increase and labels the residual attribution unresolved. Sanitizer claims are limited to the paged-decode and fused-query-writer tests actually run. NIAH is labeled integration evidence, not a numerical oracle. The kernels are explicitly described as reusing FP8 autotune configurations and not as tuned SM86 kernels.
+
+### Public documentation and logging
+
+**Finding:** The flag docstring still said unsupported, and the full allocation JSON logged at INFO.
+
+**Resolution:** The flag now documents a backend-gated MXFP4 indexer cache. Full allocation JSON logs at DEBUG; normal capacity summaries remain at INFO.
+
+### Commit trailers
+
+**Finding:** The pushed accounting commit lacked the repository's requested attribution trailers.
+
+**Resolution:** Published history was not rewritten. Subsequent local commits use `Assisted-by` and `Signed-off-by` trailers. The original process violation remains recorded here rather than hidden by force-pushing a published commit.
+
+## Findings not changed
+
+- The backend's supported head sizes describe semantic attention widths, not physical quantized row bytes; adding physical width 68 would mix two interfaces.
+- `next_n > 2` flattening on SM86 and inherited FP8 autotune choices are performance opportunities, not correctness defects. Measured deep-prefill loss keeps the feature opt-in.
+- Full allocation-report construction remains available for deterministic accounting, while its large serialized form is DEBUG-only.
+
+## Verdict
+
+Proceed as an explicit capacity experiment after the strengthened GPU tests, repository gates, packaging checks, and healthy final-state audit pass. Do not promote it as the default sparse-indexer format.