[Kernel][WIP] Integrate native FlashInfer SM70 batch GDN and QSA probes

[yangzhuxinyzx]

Purpose

Current status (2026-09-06 follow-up): still Draft / default off. The
20--30% end-to-end target and expanded quality gate are not met. Historical
results below are retained; the latest ablations and component evidence are
summarized at the end. Nothing here is a production-performance admission.

Integrate the operator-screen winners from #513 and #515 with the pinned #504 batch HC/TP baseline, behind a default-off VLLM_SM70_FLASHINFER_BATCH probe. This branch combines those component branches; the new work is the model-execution bridge, not a release default change.

• Fuse BA projection, convolution and GDN at the existing opaque input/core boundary, preserving FP16 materialization and FP32 recurrent state.
• Dispatch supported small-batch QSA to native FlashInfer SM70 decode/merge, preserving sparse index order/duplicates and the existing output gate.
• Preserve M1, prefill/mixed, speculative and unsupported-layout fallbacks. No global TP, KV dtype, max-seqs or chunk-size binding.
• Retain existing HC kernels: both new HC variants lost their microbenchmark despite passing numerical checks.
• Preserve CUDA Graph derived-weight pointers across reloads; keep scratch call-local.

Test Plan

• Targeted CPU dispatch and GPU bridge tests, graph replay, PAD/slot recycling and independent recurrent-state evolution.
• Reuse component sanitizer checks and fixed native-library provenance.
• One paired control/candidate E2E campaign on GPU 4–7 using the unchanged 8192-input/256-output C1/4/8/16 no-MTP workload, TP4, 256K capacity, prefix cache + Mamba align, FP16 KV and FP32 GDN state.
• Reuse the fixed GSM8K first-16 health screen and 80-case BFCL/schema manifest. Offline model/parser testing is not HTTP/SSE validation or an official leaderboard score.
• Release all owned GPU workers after testing. Do not modify remote services.

Test Result

• Initial GPU bridge suite: 21 passed (11.67 seconds), including numerical/graph/state checks.
• CPU dispatch/reload suite after adding pointer-preservation coverage: 19 passed.
• Component GDN/HC suite: 20 passed. Targeted GDN memcheck: 0 errors; racecheck: 0 errors / 0 warnings.
• Pre-commit checks passed for integration commit 190e5f02250f98080905250337cd3fce93901b8e.
• Native GDN and QSA libraries built in isolated task caches. Uses pinned existing vLLM/Flash-V100/HC native references, not a newly built release wheel.
• E2E control reproduced the prior baseline: C1/4/8/16 = 87.750 / 217.026 / 364.521 / 587.789 tok/s.
• First candidate only selected QSA: C16 612.823 tok/s (+4.3%), not a GDN gain. GDN was prepared but bypassed by a prefill-shape specialization outside the opaque boundary.
• Commit 45711b5e21 moves shape/metadata dispatch inside the compiled boundary and preserves the existing FP16 input projection on fallback. Reintroducing the legacy guard demonstrably fails dynamic-range export. 2273c24d7d additionally resolves empty AOT state placeholders using the existing recurrent-core contract; explicit state buffers are not replaced. Final expanded preflight: 29 passed, including four GPU cases.
• Quality not admitted: control / QSA-only GSM8K 15/16 / 15/16, no truncations; offline BFCL 52/64 / 50/64, JSON Schema 16/16 / 16/16. Preserve the negative signal; do not enable by default or infer noninferiority from this small screen.
• All first-comparison model workers exited. Respect unrelated GPU leases. Keep Draft; no default or deployed API change.

Final combined model result (source f45c673898)

GDN and QSA both hit their native runtime routes. Fixed 8K/256 no-MTP TP4 workload, prefix cache + Mamba align, FP16 KV, FP32 recurrent state, CUDA Graph, GPU 4–7:

Concurrency	Control tok/s	Combined tok/s	Gain	Fixed-70 efficiency / target
1	87.750	88.556	+0.92%	n/a
4	217.026	232.169	+6.98%	82.92% / 85%
8	364.521	390.046	+7.00%	69.65% / 75%
16	587.789	630.996	+7.35%	56.34% / 65%

C16 complete engine-step mean: 27.221 -> 25.357 ms. The three throughput targets remain unmet. This is one campaign, not a cross-run stability study.

Not admitted for quality: combined GSM8K 15/16 and JSON Schema 16/16, but offline BFCL 49/64 vs control 52/64. No truncations. Five control successes become failures and two failures become successes. Both candidate variants regress on the same two irrelevance cases. This is a localization lead, not proof of causality or broad quality degradation. No HTTP/SSE, coding, PPL or long-context quality admission is claimed.

Keep this PR Draft and the integration default off. Do not count the throughput gain as accepted production performance. Next use retained real trajectories to isolate QSA arithmetic/output-gate drift and then GDN; do not seek favorable seeds to clear the gate.

All task workers exited; cleanup confirmed GPU 4–7 at 7 MiB each with no compute processes. The user-authorized old local API remains inactive/disabled. Remote services and unrelated GPU 0–3 work were untouched. Raw artifacts and every rejected/failed attempt are recorded in the worklog.

See docs/design/sm70_flashinfer_batch_integration.md for contracts, negative results and provenance. AI-assisted with Codex; DCO sign-offs included. Human review required before merge.

Follow-up: device-planned MQA and QSA quality attribution

This remains WIP, not completion of the full hot-chain plan. New source:
64dd679aa1; reviewed #494 dependency cherry-picked without conflict as
3e0f7a40c1 (original author and DCO retained).

• Native device-planned MQA scoring now has a formal separate CMake/wheel
  fragment, caller-owned graph schedule, int64 position support and runtime
  capability dispatch behind the existing default-off parent switch.
• Actual MQA + production exact top-k + index expansion, fixed 256K capacity:
  8K C4/C8/C16 = 49.17/61.15/92.04 -> 40.56/42.27/54.00 us.
  This EXCLUDES sparse attention and the model; it is not an E2E speed claim.
  Six 8K/64K cases pass changing-replay index equality and have positive
  five-block paired within-run timing intervals.
• MQA: 31 GPU tests, including tile-visit ownership, FP64 oracle, graph length
  changes and independent streams. CPU routing/statistics/compile-boundary:
  36 tests. Final source commit's applicable pre-commit hooks pass.
• Corrected full-width GDN split-phase screen preserves 256-step state/output
  history exactly, but loses at C4/C8/C16. Do not select that variant there.
  HC full-chain B4/B8/B16 benchmark extension is prepared; GPU test pending.
• The frozen performance control itself has an allocator-order QSA prefill
  defect. Four-rank captures first diverge at layer 3 QSA despite identical
  Q/K/V, gate, selected logical tokens and effective K/V. Small local error
  grows to final hidden relative L2 0.2236.
• A diagnostic-only canonical planner makes 12 physical relocations identical,
  and restores all four ranks' final hidden tensors across four cold-prefill
  step pairs plus all 16 first-token logprobs. This is causal evidence, NOT
  broad quality admission.
• Overlap review found existing [Bugfix][SM70] Stabilize QSA page4 order under KV relocation #494 already fixes that defect, including
  shared-page union and XQA tails. The competing diagnostic implementation was
  retired to ignored artifacts and is NOT in the delivered source/build.
  Reuse [Bugfix][SM70] Stabilize QSA page4 order under KV relocation #494 instead of a duplicate PR. Its Flash-V100 source builds and imports
  locally; its GPU regressions in this integration are still pending.
• The new NVFP4 diagnostic supports the defect attribution independently of
  [Bugfix][SM70] Stabilize QSA page4 order under KV relocation #494's prior AWQ evidence. Do NOT report the retired alternative's tests as
  validation of [Bugfix][SM70] Stabilize QSA page4 order under KV relocation #494's exact implementation.

The original adverse BFCL results are retained. No new E2E result, expanded
dataset-quality admission, full-wheel acceptance, HC/TP overlap win or MoE
compaction win is claimed. Targets remain C4/C8/C16 = 238/420/728 tok/s,
C16 <=21.978 ms; they are NOT yet reached. Parent switch stays default off,
remote services untouched, no direct main push. All model diagnosis workers
were cleaned up; subsequent GPU work respects other tasks' leases.

Purpose/Test Plan/Test Result details and raw artifact provenance are retained
in docs/design/sm70_flashinfer_batch_integration.md. AI-assisted with Codex;
human review required before any promotion or merge.

Follow-up: stable planning, quality ablations, native GDN component

Source head: d1c7e6489a81b0b2f520358b066aafd54892cf72 (DCO signed;
AI attribution included). All applicable staged and commit-time hooks pass.
Publishing preflight observed main 4366d9d5fe80eeaf79575b51ec36a6a032673df0;
these measurements remain on the frozen integration base, not an untested
silent rebase. Final GDN component rebuild imports without CUDA initialization
and has identical disassembled device instructions to the tested component.

Purpose: continue the fixed FlashInfer full-hot-chain plan without promoting
an unadmitted model route. Integration base remains
755baae1d075ee04fa9096b23fc0225b23589a86; #494 is the reviewed dependency,
not a competing implementation. No direct main push or remote deployment.

Implementation and focused Test Plan

• Reuse [Bugfix][SM70] Stabilize QSA page4 order under KV relocation #494's physical-page deduplication/logical-owner order; compact the
  live device union and select 512/1024/2048/4096/8192-entry sorts, with the
  same graph addresses and attention arithmetic. Test changing live counts,
  relocation, graph replay, actual captured NVFP4 inputs and memcheck.
• Package GDN as its own CMake/setup component with isolated Q4/V12,
  Q8/V24 and Q16/V48 namespaces. Test head-partition isolation and state
  ownership at B1/2/4/8/16/32/64. This is not a distributed TP test.
• Prevent duplicate native registration, preserve derived-weight addresses,
  and reject unsupported reload contracts rather than retaining stale
  prepared weights. Initial unsupported shapes still fall back locally.
• Keep HC/MoE experiments out of the quality truth; repeat the original
  fixed 80 cases only for causal component ablations. The preregistered
  expanded three-seed BFCL/schema/GSM8K/HumanEval and HTTP/SSE gates remain
  required before admission.

Test Result

• [Bugfix][SM70] Stabilize QSA page4 order under KV relocation #494's exact rebuilt library: 78 QSA/MQA tests pass. Adaptive follow-up:
  80 pass, including live-sort-boundary changes in both directions under
  one graph. Two targeted graph cases pass memcheck with zero errors.
• Twelve real-input physical relocations are bitwise identical to [Bugfix][SM70] Stabilize QSA page4 order under KV relocation #494.
  Captured 2048-row planner+attention+gate: 653.21 -> 458.50 us (-29.81%),
  five-block within-run paired 95% reduction interval [29.76%,29.85%].
  This recovers correctness-fix overhead, remains slower than the unsafe
  433-us allocator-dependent reference, and is NOT a model TPS result.
• Native GDN CMake component builds/imports all three geometries; seven
  changing-state graph tests pass. A model ablation resolves the staged
  component through package discovery, without external GDN-library paths.
  Full release-wheel/clean-install validation remains pending.
• CPU routing/statistics/compiled-boundary suite: 42 passed, including
  missing/preloaded native components and unsupported prepared reloads.

Repaired original-80 arm	BFCL /64	Schema /16	New failures vs control
Production control (HC/MoE experiments off)	51	16	n/a
MQA only	51	16	none; all 80 complete token lists identical
GDN only	52	16	none
Native-component GDN + MQA	52	16	none
GDN + MQA + native sparse QSA	50	16	parallel_1: missing second call

All arms use the same repaired #494 binary, weights, prompts, sampling,
natural EOS and 16K limit, with zero truncations. GDN-only and native GDN+MQA
produce identical complete token lists. The sparse-QSA combined result is
still a failure: paired BFCL delta -1.5625 pp, conservative 95% interval
[-9.5612,+6.5981] pp. The GDN-only / GDN+MQA delta is +1.5625 pp with interval
[-6.5981,+9.5612] pp; do not interpret this small screen as noninferiority.
Scoring is offline model/parser diagnostic, not official BFCL or HTTP/SSE.

Rejected paths / remaining work

• Shared-parameter GDN preserves 256-step state/output exactly but loses at
  target C8/C16; the separate split-phase variant also remains rejected.
• HC B4 fails its unchanged 96-module chain numerical gate. A single-GPU,
  communication-free replay exactly reproduces the four-rank failure and
  isolates projection arithmetic association. No new HC/TP speed admitted.
• Two device-compact MoE loop designs lose on actual weights/captured
  routes. Corrected W2-only C16 full chain: 131.411 -> 141.158 us. Production
  MoE source is restored unchanged; rejected code is benchmark-only.
• Sparse QSA has different FP16 probability materialization and reduction
  boundaries from production. That is a source lead, not proof of which
  arithmetic change caused the missing tool call. Keep the negative result.
• Native sparse-QSA packaging, broader state/IPC/long-context gates, expanded
  quality, full wheel, prefill and new fixed-contract E2E remain unfinished.
  Do not add isolated service-time savings into a claimed model gain.

The worklog records source/library provenance, confidence-interval scope,
all negative attempts and local artifacts. Keep
VLLM_SM70_FLASHINFER_BATCH=0 by default and require human review.

Numerical isolation follow-up (1a5a7f7e7b)

No new serving kernel, runtime default or E2E throughput result in this
follow-up. Last measured C4/C8/C16 remains 232.169/390.046/630.996 tok/s,
not quality-admitted.

• The repaired parallel_1 failure shares the first 51 output tokens with
  MQA+GDN, including one complete valid tool call. At offset 51 the reference
  continues with newline 198 and a second call; the sparse-QSA arm emits
  EOS 248046. Lengths are 105 versus 52. This is generation termination, not
  a parser discarding a generated second call. The numerical cause of the
  different EOS draw remains unproved.
• Added a CPU FP64 counterfactual that changes only probability rounding
  before PV, retaining reference tile/split and output/gate boundaries.
  All 36 retained real QSA inputs get closer to the recorded reference with
  its FP16 probability materialization: median relative L2
  1.27622e-4 -> 3.82219e-5, median per-case reduction 70.34%.
  These are older shadow inputs, NOT the repaired EOS step. This is neither
  CUDA emulation nor model-quality evidence, and does not justify blindly
  reducing native precision.
• Added a single-GPU 96-module HC arithmetic harness separating down-only,
  up-only and full sharding. It requires reproducing the retained four-rank
  negative input first and does not relax the 3e-3 envelope. No distributed
  timing is inferred from the single-GPU simulation.
• CPU routing/compiled-boundary/statistics/counterfactual suite: 45 pass,
  including three new invalid/NaN/duplicate/denominator tests. Applicable
  source/commit hooks pass; DCO and AI attribution included.
• The original 16-case parallel cohort is registered for a short teacher
  conditional-probability capture on all four ranks with A/A repeats. This
  diagnostic is artifact-only, explicit opt-in and NOT a quality score.
  CPU spawn-import checks pass without CUDA initialization.

The HC GPU launcher exhausted its ten-minute availability window with exit
75; a subsequent process check still found foreign GPU work. The new HC
GPU and EOS model diagnostics have NOT run. The waiter exited, no GPU
process/service was left resident, and foreign work was not terminated.
Resume those focused checks after actual availability, then the fixed
quality suites, then the concentrated frozen-contract E2E campaign.

Follow-up: implemented numerical repairs (6dbb62937e)

The preceding lease-only status is superseded: resources became available,
the registered HC and four-rank EOS diagnostics completed, and the combined
natural quality campaign is now running. Keep this PR Draft/default-off.

• HC: the saved failure is reproduced by down-only and combined projection
  sharding, but up-only is bitwise identical at all arithmetic intermediates
  for 16 inputs across 96 real weights. The runtime now preserves full down
  GEMM and only shards up. TP4 full HC Graph chains at B4/8/16 pass the
  existing every-intermediate 3e-3 gate. Five paired blocks: 0.422%/0.213%/
  -0.092% chain-time reduction. B16 has no speed admission; correctness did
  not preserve the previous sharded-down speedup.
• Sparse QSA: new native Volta WMMA experiment restores production FP16 P
  materialization, 16-token partition and FP32 normalization, retaining
  pinned FlashInfer virtual-page preparation/cascade and the output gate.
  36 real-input median relative L2 improves 1.27593e-4 -> 5.34201e-5.
  Full-QSA micro reductions over production: B4 6.62%, B8 38.27%, B16 49.57%;
  positive five-block CIs. It is slower than old, numerically different SIMT.
• Reuse the production profile and maximum-useful splits for intermediate
  live widths, multiple KV heads and short selections. The incomplete first
  wrapper model attempt was explicitly stopped before quality/speed output.
  Expanded routing/HC CPU tests: 118 passed. GPU output-gate/Graph bridge
  tests at B4/8/9/15/16, one/two KV heads, widths15/65/2051: 20 passed.
• EOS attribution: four ranks and both A/A repeats agree bitwise. First-token
  raw logits are identical across arms. At offset51, old SIMT QSA shifts EOS
  probability 0.760651 -> 0.839734 and newline 0.239349 -> 0.160266. The same
  stateless draw flips from newline to EOS. This is decode probability drift,
  not a parser, seed, or first-prefill issue. No EOS masking was added.
• Combined natural original80 run uses unchanged inputs/seeds/max16K/EOS,
  grouped-MoE experiments off. Its frozen 8K/256 C1/4/8/16 speed probe is
  gated on each suite meeting the repaired production control and the EOS
  case passing. This is not yet expanded multi-seed quality admission.

Fresh model quality and E2E results are pending. New QSA is still an explicit
prebuilt experimental component, not a released wheel/default. Full raw
provenance and all negative results remain in the worklog.

[yangzhuxinyzx]

Quality attribution follow-up recorded in a25c23f. This is evidence for the existing #523 integration scope, not a duplicate component PR or a request to merge.

• Same-input eager shadows: 144 GDN + 48 QSA comparisons. GDN Z/conv state exact; maximum recurrent-state relative L2 1.777e-5. QSA candidate is closer to the FP64 reference in all 48 captured comparisons. This alone is not model-quality admission.
• Retained 80 prompts/seeds/sampling, 16K natural-EOS limit, fixed-cohort diagnostic: control BFCL 51/64 and 52/64; candidate 50/64 and 51/64; JSON Schema 16/16 throughout; no truncation. The original adverse 52 -> 49 continuous-admission screen remains recorded.
• Control itself changes 22/80 complete outputs and 8/80 first tokens across repeats. Candidate changes 23/80 and 8/80 respectively.
• Fixed-trajectory comparison over 2,225 tokens: first-token mean absolute NLL delta is 0.542 between candidate/control, but 0.801 in an unchanged-control A/A; full-token mean NLL deltas +0.003106 vs +0.004358. Request/position traces and CPU/GPU seeds agree. These diagnostic continuations are not a PPL benchmark or task score.
• The remaining priority is shared prefill/first-token computation and state initialization/reuse, not a demonstrated catastrophic error in the new GDN/QSA arithmetic. The exact causal operator is not yet isolated, and noninferiority is NOT established.
• 25 routing/compiled-boundary CPU tests pass; commit hooks pass. No production math/defaults changed. Keep Draft/default-off; prior ~7% speed gain remains experimental. Local diagnostic workers exited and GPUs 4-7 returned to 7 MiB each; no remote service changed.

AI-assisted investigation (Codex); human review is still required before merge. Full provenance, aborted diagnostic attempts, raw result locations and next action are in docs/design/sm70_flashinfer_batch_integration.md.

[yangzhuxinyzx]

集成审计更新：#494 已合入 main（642f95d15f）。请继续复用该修复，不另造 planner。已测版本约 7% 的收益有价值，但旧控制自身的分配顺序问题以及 52/64→49/64 的负信号都应保留。当前 3e0f7a4 又加入新 MQA 原生实现和打包变更，不能直接继承旧版本整机/质量结论。下一步优先在修复后的同条件基线做最小质量归因与集成验证；不要求 greedy/hash 一致，也不重跑无决策价值的整套 E2E。本轮保持 open，不关闭。