chore(squad): consolidate GEMV 6th-win + honest-ceiling campaign state

.squad/agents/deckard/history.md

@@ -44,3 +44,24 @@ Luv's current-main (`923dc592`) re-probe found captured verify B* still far abov
 
 Scribe recorded the family-trio closure: Deckard's qwen3.5-0.8b text-only export via graph surgery merged in PR #1456 (`169febb1f`), establishing the hybrid graph-block moat at the small end. Sebastian's `CudaDropIdentityCast` PR #1459 (`792958ecf`) is a general byte-identical cleanup win (+3.0% short ctx) for hybrid graphs. Wallace completed the fair numeric 0.8B row, closing the {0.8B, 2B, 9B} qwen3.5 evidence set.
 
+
+## 2026-08-19T16:40Z — fp16 fair race and per-op profiling (deckard-8)
+
+- **fp16 export root-cause:** "Native fp16 slower than fp32" was entirely a `keep_io_types=True` export artifact. fp16io (keep_io_types=False) captures cleanly at 236 tok/s (+10% over fp32). fp16act (keep_io_types=True) declines CUDA-graph capture → eager penalty ~3× (72 tok/s). No native kernel regression.
+- **Fair fp16io per-op profile (nsys captured-graph):** Native 4.769 ms/tok, 209.7 tok/s. #1 inefficiency: SSM-reduce fragmentation 20.7% (cuDNN fp32 ReduceSum + casts + LinearAttention). GQA 2.0%, argmax 0.17% — cheap and NOT the problem. Fix hypothesis (fuse f16 ReduceSum) confirmed by Gaff (+7.4%). ORT captured baseline unverifiable on-box.
+- **gpt-oss-20b moat retracted:** fp32 export artifact; honest native lead ~1.5× GPU-vs-GPU. File upstream exporter change (bf16 activations), not an ORT kernel fix.
+- Read-only turn; no source code changes, no PRs.
+
+## 2026-08-19T18:20Z — Fair ORT fp16io baseline + kernel attribution complete (deckard-8)
+
+- **Fair baseline:** ORT eager ~284–296 tok/s on qwen3.5-0.8b-hybrid fp16io (1019 CUDA / 50 CPU nodes; no structural kernel gap). Native post-#1486 = 233.6 tok/s → **~1.24× ORT-ahead**. 2.1× / 496 tok/s retired; now.md corrected in PR #1493.
+- **Attribution:** Gap is entirely GPU-busy kernel time (+1269 µs/step). #1 lever: native's decomposed LinearAttention subsystem (~700–900 µs/step; fp32 cuDNN reduce + data-shuffle + gating chain). #4 lever: int4 GEMV latency-bound (+193 µs; ORT kernel 33%/call faster on M=1). Gap is fusion-closable; more cuda-graph coverage will not help. Banked in decisions.md.
+
+## 2026-08-19T21:15Z — Post-#1503 re-attribution delivered; data-shuffle lever corrected
+
+Delivered fresh post-#1503 gap attribution: native captured 253.9 tok/s vs ORT eager 279.1 tok/s (1.099x ORT-ahead, ~356 us/step), initially ranking data-shuffle +289 us/step as #1. Gaff's follow-up refuted that lever for captured decode: the count was an eager standalone-kernel artifact; captured replay amortizes movement kernels and future attribution must use graph-node profiling.
+
+
+## 2026-08-20T00:15Z — Captured-replay attribution corrected; honest ceiling set
+
+Corrected the post-#1503 native-vs-ORT attribution using captured-replay-aware profiling only. Native vs ORT had near-identical kernel counts (**795 vs 793**) but native carried **+617 µs/step GPU-busy**; every hot kernel was occupancy/latency-bound, not roofline-bound. Data-shuffle is retired as a dead lever because elision fragments capture. The one remaining buildable lever was int4/int8 GEMV occupancy tuning; Gaff landed it in #1516. Honest practical ceiling on this arch is **~260–270 tok/s**, with the remaining qwen3.5-0.8b batch=1 gap being launch/latency floor.

.squad/agents/gaff/history.md

@@ -51,3 +51,12 @@ Deckard's kernel attribution (deckard-8, 2026-08-19) identified the **gated-delt
 - **End-to-end:** +7.4–10.6 tok/s (~240 → ~249 tok/s, +3.2%). Native ~249 vs ORT ~290 (~1.16× ORT-ahead). Coordinator re-validated golden lock GPU0 (35.01 s). Admin-merged (squash, 2026-08-19). Opt-out: `ONNX_GENAI_CUDA_DISABLE_LINATTN_WARP_COOP=1`. Worktrees swept.
 - **CRITICAL FINDING — lock tests assert token-IDs, NOT bit-exact bytes:** `qwen35_*_text_decode_lock` assert `Vec<u32>` argmax. Warp-shuffle reduction reorder is ULP-divergent but argmax-stable. qwen3.5-0.8b + qwen3.5-2b PASS, oracle-parity suite PASS (all update rules, d_k=128/96/130). **De-risk: future ULP-divergent kernel rewrites (warp reductions, reduction-order changes) are not barred by the lock gate — validate ≥2 models for argmax stability, not bit-identity.**
 - **5-win session arc complete (#1486/#1495/#1496/#1501/#1503):** ~214 → ~249 tok/s (+16.4%). Remaining gap to ORT is per-layer launch/fusion structure — diminishing-returns territory.
+
+## 2026-08-19T21:15Z — Shuffle-fusion honest floor (gaff-16, PR #1511 closed)
+
+Attempted CUDA no-op Transpose zero-copy views for the data-shuffle lever; no code shipped. The view path fired but mutates buffer lifetimes mid-capture, aborting graph recording and fragmenting decode capture 16→70 segments. `movement.rs` was reverted and docs-only PR #1511 was closed. Reusable finding: captured decode already amortizes tiny movement kernels; attribute with captured-replay profiling, not eager kernel counts.
+
+
+## 2026-08-20T00:15Z — Symmetric int8 GEMV split-K grid-fill (PR #1516)
+
+Landed the final GEMV lever: symmetric int8 `MatMulNBits` now uses split-K for grid-starved decode projections, general and capture-safe, opt-out `ONNX_GENAI_CUDA_DISABLE_INT8_SYMMETRIC_SPLITK=1`. Bias-fusion was honestly a no-op (already on main; Qwen3.5 exports bias-free). Result: qwen3.5-0.8b-hybrid fp16io **254.0 → 263.8 tok/s** (+3.9%, +9.8 tok/s); ncu N=1024 **11424→8049 ns** and N=2048 **6065→4877 ns** with occupancy roughly doubled. Golden token-ID locks passed on 0.8b and 2b; PR #1516 merged at `ec695d897`.

.squad/decisions.md

@@ -196,3 +196,40 @@ Rewrote the gated-delta `LinearAttention` decode kernel (`linear_attention.rs`)
 Occupancy gate for int4 MatMulNBits GEMV entry selection. Routing rule: if `ceil(N / cols_per_cta) >= mp_count * 32` (well-occupied, e.g. LM head N=248320) → plain/low-register entry (−14% latency: 98.8→85.0 µs). Otherwise → prefetch-pipelined entry (grid-starved projections already at byte-identical floor ~4.3–4.7 µs/call, under ORT's 4733 ns). Keys on N, launch-width, and SM count only — capture-safe. Byte-identical golden lock PASS. Coordinator independently re-validated on GPU0 (33.06 s). Admin-merged (squash, 2026-08-19). Opt-out: `ONNX_GENAI_GEMV_PIPE_WELLOCC=0`.
 
 **Result (H200, qwen3.5-0.8b fp16io, end-to-end medians):** +1.3–1.6 tok/s (~242→243.5 tok/s). Grid-starved projections at byte-identical floor; split-K would raise occupancy but reorders fp32 reduction (not byte-identical — barred under the golden-lock rule).
+
+---
+
+### 2026-08-19: data-shuffle lever investigation — attribution refuted by captured-graph honest floor
+
+**By:** Deckard (deckard-8) + Gaff (gaff-16)
+
+**Attribution:** After PR #1503 on current `origin/main` (`4bbd9152c`), Deckard re-measured qwen3.5-0.8b-hybrid fp16io on H200: native captured decode **253.9 tok/s** (3.939 ms/tok, 8 captures, 1008 replays, fallbacks=0) vs ORT eager **279.1 tok/s** (3.583 ms/tok), leaving **1.099x ORT-ahead / ~356 us per step**. The eager kernel-count attribution ranked `data_shuffle` as apparent #1: native 341.2 us/step vs ORT 52.3 us/step (**+288.9 us/step**) across transpose/split/gather/concat/where copy-like kernels; int4 GEMV was #2 (+176.1 us/step), and linear attention residual #3 (+58.7 us/step).
+
+**Attempt:** Gaff tested the data-shuffle fusion lever on branch `squad/shuffle-fusion` by using CUDA `TransposeKernel::view_outputs()` for seq_len=1 no-op transposes, gated during the experiment by `ONNX_GENAI_CUDA_DISABLE_SHUFFLE_FUSION=1`. The elision path fired correctly, but zero-copy view installation mutates device-buffer lifetimes and is illegal during CUDA graph capture.
+
+**Correction / decision:** Deckard's **+289 us/step data-shuffle lever is refuted for the captured decode path**. The standalone eager kernel count is real, but its cost attribution is an eager-mode/nsys artifact: inside captured replay, the 60 no-op transposes are ~2 KB copies and cost roughly **~40 ns/step** (~0.001% of a ~4 ms step). Installing views mid-capture aborts graph recording, quarantines Transpose, and fragments the decode graph from **16 captured segments / 15 eager seams** to **70 captured segments / 69 eager seams** — a regression, not a buildable win.
+
+**Outcome:** No code shipped. `movement.rs` was reverted, origin/main behavior is unchanged, and docs-only PR **#1511** was closed by the coordinator without merge. Do **not** treat data-shuffle fusion as the top buildable lever. Future attribution for captured decode must use captured-replay profiling (`ncu --graph-profiling node` or `nsys --cuda-graph-trace=node`) rather than eager standalone kernel counts.
+
+
+---
+
+### 2026-08-20: Corrected captured-replay attribution — honest native ceiling is the batch=1 launch/latency floor
+
+**By:** Deckard
+
+**What:** Re-ran qwen3.5-0.8b-hybrid fp16io native-vs-ORT decode attribution using captured-replay-aware tooling only (`nsys --cuda-graph-trace=node`, `cuda_gpu_kern_sum`, and `ncu --graph-profiling node`) after Gaff refuted the data-shuffle hypothesis. Native captured replay measured ~249.6 tok/s (run-to-run 249–254; later campaign baseline ~254) vs ORT eager 279.1 tok/s, with GPU-busy time **native 2516 µs/step vs ORT 1899 µs/step (+617 µs)** and near-identical kernel counts (**795 vs 793**). Every hot native kernel is occupancy/latency-bound rather than bandwidth/compute-roofline-bound: main int8/int4 GEMV ~22% occupancy, `transpose_bytes` ~12% occupancy and near-zero bytes moved, and post-#1503 `linear_attention_f16_coop` ~21.5% occupancy.
+
+**Why:** The previous data-shuffle attribution was real GPU timeline cost but not a buildable lever. The transposes are tiny seq_len=1 floor kernels; eliding them mid-capture fragments CUDA graph replay and regresses. Deckard retired data-shuffle and int64-index chasing, identified **int4/int8 GEMV occupancy tuning** as the single remaining credible lever, and set the honest practical ceiling for this arch at roughly **260–270 tok/s**: ORT parity minus the small-batch launch/latency floor tax.
+
+---
+
+### 2026-08-20: Symmetric int8 MatMulNBits split-K grid-fill landed — sixth clean token-stable win (PR #1516, merged)
+
+**By:** Gaff
+
+**What:** PR **#1516** landed the final buildable GEMV lever on `origin/main` (`ec695d897`): symmetric int8 `MatMulNBits` GEMV now uses the existing split-K/grid-fill path for grid-starved decode projections, keyed only on `K`, `N`, bits, zero-point presence, and live SM count. The bias-fusion sub-lever was a no-op: native already has `CudaMatMulNBitsBiasFusion`, and all Qwen3.5 exports checked here are bias-free. The shipped change is general, capture-safe, and opt-out guarded by `ONNX_GENAI_CUDA_DISABLE_INT8_SYMMETRIC_SPLITK=1`.
+
+**Result:** H200 qwen3.5-0.8b-hybrid fp16io paired A/B improved **254.0 → 263.8 tok/s (+3.9%, +9.8 tok/s)**. ncu confirmed the targeted grid-starved int8 projections improved: N=1024 **11424 → 8049 ns** with occupancy **11.6% → 23.4%**; N=2048 **6065 → 4877 ns** with occupancy **22.2% → 45.4%**. Golden token-ID locks passed on qwen3.5-0.8b and qwen3.5-2b, and the coordinator independently re-validated both as `ok`.
+
+**Campaign state:** The decode campaign now has **six clean token-identical/token-stable wins**: #1486, #1495, #1496, #1501, #1503, and #1516. Native is now ~**263.8 tok/s** vs ORT **279.1 tok/s** (**~1.057× ORT-ahead**), narrowed from ~1.24× at session start. Per Deckard's corrected attribution, the int4/int8 GEMV lever was the **last genuine buildable lever**; the remaining gap is the batch=1 0.8b launch/latency floor, not an unmined roofline or data-shuffle opportunity.