[Perf] Audit and recover DFlash2 native prefill routes

[yangzhuxinyzx]

Purpose

Restore and document the missing native prefill routes in the frozen QUASAR + DFlash2 performance launch. The service lacked the FA2 sidecar and forced serial GDN prefill. Loading the matching FA2 library and selecting the existing original FlashQLA GDN route recovers 64K prefill to 3651 tokens/s and 128K to 3015 tokens/s. This PR adds four-rank native-library/configuration checks, verified cold-prefill counters and separate q8/late-prefill tracing. It does not change production kernels or serving defaults.

This scope does not duplicate #548 (native prefill implementation), #556 (the accepted verification optimizations), #566 (TP2), or #561 (shared weights).

Test Plan

Serving source stays frozen at a7cc5ae. Preserve physical GPUs 4–7, TP4/B1/q8, 4096 prefill budget, 262144 total capacity, E4M3 target KV, FP32 logits/state, weights, sampling and CUDA graphs. Add the explicitly selected FA2 sidecar from #548 with matching source hashes, then select original GDN prefill. Measurements stop at 128K at the user's request.

Reset prefix caches per length and require computed-prefill tokens to equal full input length. Record actual FA2 bridge hits on four ranks and the original-GDN runtime log hit. Keep unprofiled timings separate from graph-node traces. Use one cold request and three repeated 256-token latency windows with EOS honored. These bounded probes do not establish dataset quality or three-startup performance acceptance.

Test Result

Input	Cold prefill before/after GDN restoration, tokens/s	Restored cold prefill, s	Complete round, ms	Pure decode, tokens/s	Accepted drafts/round	Emitted tokens/round
1K	3222 / 3595	0.285	16.214	291.24	3.7778	4.7407
32K	3388 / 4070	8.051	26.068	148.21	2.8939	3.8788
64K	3096 / 3651	17.949	34.633	139.84	3.8627	4.8627
128K	2630 / 3015	43.476	53.133	81.34	3.3390	4.3390

• Existing v37/E4M3 GPU tests: 23 passed, 2 256K cases deselected. Initial harness import failure is retained; the corrected launcher used frozen serving imports.
• Scoped pre-commit passes. CLI dependency and mixed-rank forced-serial GDN rejection checks: 2 passed. The GDN configuration guard was added after the performance run; its runtime hit is evidenced separately by the worker log.
• All four requests within each length repeat with identical token IDs. Versus serial GDN, 1K is unchanged and 32K/64K/128K first differ at indices 199/126/21. The 64K candidate ends naturally after 248 tokens with a complete function passing the three supplied examples; the control reaches its 256-token cap mid-expression. Other restored lengths reach the cap.
• Accepted drafts/round change from 3.3729/4.0784/3.5536 to 2.8939/3.8627/3.3390 at 32K/64K/128K. Sampled continuations differ, and acceptance-length non-inferiority is not established. Long-context decode is not claimed as restored by the prefill switch.
• Independent FP64 GDN screen: original TileLang output relative L2 is about 2x native; state error has a larger ratio against the tiny FP32 baseline. The user accepted sub-order-of-magnitude output-error growth and requested speed restoration first. This is not a model-quality promotion.
• Separate late-128K trace: attention 665.067 ms, native GDN 210.862 ms GPU service per 3296-token chunk. The 64K q8 trace instead identifies compensated grouped attention as 18.798 ms per rank/round. These trace sums are not endpoint timings.
• Precision-preserving private candidates retain 48/48 bitwise output/state matches and improve the GDN operator from 4.842 to 2.696 ms. Prefetch8 is rejected at 5.698 ms despite 12/12 exact cases. None is enabled as a production kernel.

Harness base: 80545c0. Native hashes, report names, failed paths and remaining gates are recorded in docs/design/sm70_dflash2_context_cost_20260909.md. The owned service is stopped; 256K requests remain held.

AI assistance was used for diagnostics and reporting.

Assisted-by: Codex

Main integration

The user explicitly requested merging the current DFlash2 optimization work to main on 2026-09-10. Merge order: #586 → #587 → #589. The combined tree against main 0a0d4d6754f553ab00307eac807e9b5963b58b37 merges cleanly and passes CUDA_VISIBLE_DEVICES='' .venv/bin/python -m pytest -q tests/v1/worker/test_sm70_long_attention_graphs.py tests/kernels/core/test_sm70_dflash2_state_audit.py (26 passed, 1 skipped) and scoped pre-commit. This merge preserves explicit opt-in switches and does not assert the unfinished 7-ms attention or 22-ms round targets. It adds no default activation or new service admission.