now.md: fair baseline resolved — qwen3.5 gap ~1.2-1.4x ORT-ahead (496 retired)

.squad/identity/now.md

@@ -4,9 +4,9 @@
 
 ## Latest verdicts — 2026-08-19 (fp16 fairness + native decode fusions)
 
-- **HONEST FAIR-TEST REVERSAL (qwen3.5 hybrid): the fp32 "moats" were EXPORT ARTIFACTS.** The context-scaling moat numbers (3–7×) came from fp32-activation exports that force ORT to drop GQA/recurrent ops to CPU (no fp32 CUDA kernel), inserting Memcpy nodes and blocking CUDA-graph capture. Re-export fp16 (`keep_io_types=False` = **fp16io**, NOT fp16act) → ORT keeps ops on GPU, captures, and the gap collapses. On qwen3.5-0.8b-hybrid same-graph fp16io: native ~236 vs ORT ~496 tok/s = **~2.1× ORT-AHEAD** (honest). No fabricated ORT PR filed. **This is the fair-test pattern seen 3× (DeepSeek, gpt-oss, qwen3.5) — fp32-activation "moats" must be re-checked on fp16io before banking.**
+- **HONEST FAIR-TEST REVERSAL (qwen3.5 hybrid): the fp32 "moats" were EXPORT ARTIFACTS.** The context-scaling moat numbers (3–7×) came from fp32-activation exports that force ORT to drop GQA/recurrent ops to CPU (no fp32 CUDA kernel), inserting Memcpy nodes and blocking CUDA-graph capture. Re-export fp16 (`keep_io_types=False` = **fp16io**, NOT fp16act) → ORT keeps ops on GPU and the gap collapses. **This is the fair-test pattern seen 3× (DeepSeek, gpt-oss, qwen3.5) — fp32-activation "moats" must be re-checked on fp16io before banking.**
 - **fp16 export subtlety (critical):** `keep_io_types=True` (fp16act) leaves KV/recurrent-state fp32 at the graph boundary → NATIVE declines CUDA-graph capture → eager (3× slower). Use fp16io so KV/state are fp16 → native captures → native fp16 is HEALTHY and faster than native fp32 (236 vs 214). Earlier "native fp16 slow" was this artifact.
-- **⚠️ 2.1× magnitude is UNVERIFIED on-box:** the 496 tok/s ORT fp16io captured baseline could NOT be reproduced (genai can't load the custom hybrid arch; crate ORT is eager-only; raw python ORT hits a pathological growing-KV GQA path ~20 tok/s). Needs a real ORT captured harness (buffer-shared KV) before citing 2.1×. Todo `verify-ort-fp16io-baseline`.
+- **✅ FAIR BASELINE RESOLVED (Deckard, verified): gap is ~1.2–1.4× ORT-ahead, NOT 2.1×.** The 496 tok/s ORT figure is **NOT reproducible** on the fp16io export and is RETIRED. Correct device-resident harness (H200, ORT 1.28 CUDA EP, greedy tokens byte-match): **ORT eager = ~284–296 tok/s, FLAT across depth 16→1040** (the old "~20 tok/s" was just the COLD first run / cuDNN warmup). ORT node placement = **1019 CUDA / 50 CPU** — the 50 CPU nodes are trivial mrope/mask index ops placed by ORT's *cost heuristic* (NOT missing kernels); all 18 LinearAttention + 18 CausalConvWithState + 6 GQA + 187 MatMulNBits run on CUDA. **→ "ORT can't run the hybrid recurrent ops on GPU" is DISPROVEN; there is NO structural whole-graph-capture moat for this arch.** ORT cuda-graph is not correctly achievable here (no GQA share-buffer wiring), so best correct ORT = eager ~290. **Honest ratio: native captured 233.6 (post-#1486) ÷ ORT ~290 ≈ 0.80× → ORT ~1.24× ahead; native does not win at any depth.** THE REAL "beat ORT" LEVER is native's own decode efficiency (int4 MatMulNBits GEMV 35.8% + glue tail), NOT any ORT deficiency. Gap to close ≈ 24%; PR #1486 already banked +8.9% of it. Next todo `native-vs-ort-290-attrib` (kernel-level attribution of the remaining gap).
 - **PR #1486 (native decode fusions, +8.9% on qwen3.5-0.8b-hybrid, byte-identical, capture preserved):** Gaff landed two general fusions on the honest fair fp16io path. (1) **SSM/linear-attn fp16 reduction** — root cause: cuDNN rejects fp16 comp-type → forced fp32 3-kernel round-trip (cast/ReduceSum/cast) through a full-size fp32 temp (~20.7% of decode GPU time, 18×/step); fix routes fp16 sum/mean to the existing NVRTC single-kernel block reduction (f32 register accum, exact ONNX semantics, shape/dtype-general). (2) **`Sqrt`+`Reciprocal`→fused `rsqrt`** in the normalize-scale glue. Combined **214.4→233.6 tok/s (+8.9%)**, `generated_token_ids` byte-identical, captures=4/fallbacks=0. Honest ceiling: mul 9% / transpose+split ~8% glue-tail not cleanly/generally fusable in surgical scope — left alone. **Lesson banked (repo memory): keep fp16 reductions OFF cuDNN → NVRTC path.**
 - **#1474 memory-stack verification (Bryant, comment posted):** stack NOT fully green on H200 — **6 stack-owned GPU tests fail deterministically** (reproduced in isolation), ALL rooted in the Phase 4/7 **async deferred-release / stream-ordered release** mechanism that was `#[ignore]`'d on the GPU-less dev box and thus never executed before. Looks like test-harness/timing gaps (sync asserts / no release-queue installed), NOT observed memory corruption. Sites: device_allocator_gpu.rs:304 (injected.frees()==0), :433 (zero-byte leak + worker panic), vmm_release_quarantine_gpu.rs:310, weight_paging vmm_retained/admission (no deferred-release queue), provider.rs:3681 (count race). Criterion-10 decode: Phase 6→7 **+2.1%, no regression**. CPU side + authority/teardown logic hold. Follow-up todo `fix-1474-async-release-tests` (belongs to #1186 owner — surfaced, not auto-fixed).