[Kernel][SM70] Integrate FP32 v37 prefill and exact E4M3 bridge

[yangzhuxinyzx]

Current status: merged at explicit user direction

Merged into main on 2026-09-07 at merge commit 8d9c3518992059105d89939e8a46d75184505d8e, after the requester was informed of the model-parity failure and explicitly directed the merge. This is a known-issue integration, not a passed strict numerical/model-quality gate. The 256K natural-generation token-109 divergence remains unresolved.

Both pre-merge GitHub checks passed. The fetched merge tree is identical to the tested PR head 05f28efcf364e441146e58c6618a1cf294de1ed4; no additional source changes were introduced by the merge. No MTP, GPU jobs, runtime deployment, or service restart was performed for this merge.

The following pre-merge hold and evidence are retained as a historical record; their not-merged/hold statements describe the previous decision, not the current PR state.

Historical pre-merge model-parity hold

Not merged. All planned runtime tests have completed. Speed, operator
precision, build and memory-safety checks pass, but a matched-native256K
natural summary first differs at token109 (157 versus158 tokens). Both
answers are coherent and retrieval-correct; the strict identity gate fails.
See the committed integration audit and qualification comments below.

CPU141 passed/1 GPU skip; GPU33 passed;17 additional dynamic v37 endpoints
are bitwise equal. No source-level cause of model variation is established.
Do not replace this gate with a forced post-EOS or kernel-only check.

Purpose

Integrate the precision-qualified v37 SM70 long-prefill implementation and
an exact E4M3 paged-to-FP16 bridge. Preserve FP32 QK/PV accumulation and
prefix/tail state, support the scheduler's tile-aligned query family, and
retain bounded workspace, explicit rollback and exact fallbacks.

This is prefill-only, not the grouped/MTP work in Draft #524. No MTP was
enabled for this scope. The global fp8 encoding and weight quantization
defaults are not changed. Existing E4M3 decode wave kernels are unchanged;
the acceptance launch explicitly enables their existing flags so a slow
control and a slow candidate cannot falsely establish speed parity.

Original integration base: 2946e0ed664e0dca452b38745df601c1d9f4404d.
Rebased integration base: 6ccf581e127af61dd1ae5cd5b7ef708a5d0ccd70.
Owned implementation head: 70c12826460af3dd3fa5560922c3357eea68e7ad.

Implementation

• Parent-owned FA2 translation units and separately named v37 operator;
  a stale legacy extension cannot masquerade as v37.
• SM70 FP16/B1/Hq6/Hkv1/D256 causal, Q64–8192 in multiples of64,
  Q<KV<=262144, KV multiple32; preserve real-query causal offsets when padded.
• FP32 prefix/tail state and final merge; probabilities and final output
  retain FP16 storage. No claim of real-arithmetic exactness or no KV loss.
• Exact E4M3 byte conversion with unit/non-unit scales and live-length graph tests.
• Per-device workspace serialization and exception-safe tail lifetime.

Test plan and results

Completed before Draft publication:

• Full changed-file pre-commit, including Ruff, Clang format, mypy and SPDX.
• CPU attention-policy tests: 138 passed, 1 GPU-only skip.
• CMake-built FA2 target: 26 GPU kernel/bridge/OOM tests passed.
• Compute Sanitizer: zero errors for supported raw endpoints and E4M3 bridge.
• Private-v37 vs port: bitwise real-QKV equality; 100-ABBA paired speed ratios
  0.998963/0.998585 at Q8000/KV128000/256000. This is a port-regression test,
  not a speedup attributed to new arithmetic.
• Clean native Flash rebuild vs retained native dependency: four matched
  prefill/decode endpoints bitwise equal, finite, max-abs0.
• E4M3 long-wave diagnostic: six real no-MTP decode replays, FP64 on identical
  KV, graph=eager, L2 about0.023–0.027%. In FP8 TP4 model diagnostics,
  restoring existing wave flags improves 256000-input decode16.76→43.06tok/s;
  historical50.38tok/s is a different NVFP4-weight contract, not this baseline.

Focused public commands in an SM70-built environment:

.venv/bin/python -m pytest tests/v1/attention/test_sm70_v37_prefill.py tests/v1/attention/test_sm70_flash_v100_policy.py -q
.venv/bin/python -m pytest tests/kernels/attention/test_sm70_v37_prefill.py -q
.venv/bin/python -m pytest tests/v1/cudagraph/test_cudagraph_dispatch.py -k fp8_e4m3_b1_wave_graph_routing_on_sm70 -q

Completed model gate, held: matched retained/release TP4 model runs on
physical GPUs0–3 only, MTP off, FP8weights/E4M3KV, graph on, no prefix reuse.
Includes four context lengths, natural-EOS short/long output, recorded
logprobs, and the exact262144 final-context boundary. The controlled result
still has the token109 summary divergence described above. Do not promote
or merge based on forced post-EOS continuation or operator tests alone.

Final attention libraries are rebuilt; shared _C/stable dependencies remain
pinned. This is an explicitly identified source-overlay test, not a claim
that a full newly built wheel has been qualified.

Duplicate check and AI assistance

The open-PR area scan found no other v37 prefill integration. Draft #524
addresses grouped/speculative precision and is deliberately not included.
Codex provided AI-assisted implementation, review and testing at the fork
owner's request. No independent human line-by-line review is claimed.

Detailed numerical scope, failed variants and initial endpoint caveats are
retained in docs/design/sm70_flash_v37_prefill.md. Final model results are
recorded there; this PR is not ready for promotion.

[yangzhuxinyzx]

Final qualification update: keeping this PR in Draft.

• Rebased CPU suite:141 passed/1 GPU skip; GPU suite:33 passed.
• Both full TP4/no-MTP model runs completed, including natural-EOS128K/256K
  summaries and the262144 final-context boundary. Prefill/decode latency is
  retained; release256000-input decode is about43.1tok/s.
• Five of six natural output streams match exactly, including both long
  summaries. The72-input-token reasoning case first differs at token279
  (388 vs394 output tokens). The top two words are a near-tie; both answers
  remain mathematically correct. This still fails the strict parity gate.
• That comparison also refreshed native Flash/paged-helper dependencies,
  which are not modified by this PR, so it does not isolate the FA2/v37
  change. A third controlled run now holds those dependencies identical and
  changes only retained FA2/JITv37 versus the parent-owned FA2/v37 port.
  The original failed comparison is retained, not relabelled as passing.

No merge until this discrepancy is localized and the relevant gate passes.
All runs keep MTP disabled and use at most four physical GPUs.

[yangzhuxinyzx]

Controlled qualification completed; the strict model token-parity gate still fails, so this PR remains Draft and is not merged.

With native Flash and paged-helper binaries held identical, retained/release
256000-input prefill is106.985/107.000s and decode43.136/43.109tok/s. The
262144 final-context boundary completes. Five of six natural-EOS outputs
match exactly. The256K summary first differs at token109 and finishes with
157/158tokens; both texts are coherent and retrieval-correct. This is not
evidence of a semantic failure, but does not satisfy the current identity gate.

Additional isolation:17 real-derived dynamic v37 endpoints are bitwise equal
(includingQ8192/KV256K); eight dense D256 native-prefill kernel instruction
dumps match; six real native decode replays have identical error metrics.
These results do not yet establish why the complete model differs. All raw
failing and passing evidence is retained. MTP remained off, at most four
physical GPUs were used, and all task GPU processes have now exited.

The integration document and migration-control entry record the unresolved
gate. The next investigation must localize complete-model variation rather
than repeating already-passed isolated operator tests or lowering the gate.

[yangzhuxinyzx]

The requester explicitly directed merging PR #548 into main after being informed that the controlled 256K natural-generation comparison first differs at token 109. Proceeding as a requester-directed integration with this known issue, not as a passed numerical/model-quality gate. The strict token-identity result remains FAILED and its cause is unresolved; original evidence and failed results are preserved. Current head is 05f28ef; both GitHub checks pass and the branch is cleanly mergeable onto main. This merge adds no further source/default changes, no MTP, and no service restart. Follow-up model-level localization remains outstanding.