diff --git a/.squad/agents/wallace/history.md b/.squad/agents/wallace/history.md index 5b60b4c5c5..26826a04d1 100644 --- a/.squad/agents/wallace/history.md +++ b/.squad/agents/wallace/history.md @@ -72,4 +72,8 @@ Full pre-compaction history in `history-archive.md`. - Reconfirmed the 2026-08-17 dense int4 fairness decomposition by attempting ORT CUDA graph mode on Phi-4-mini, qwen2.5-7b, and qwen2.5-14b-zp. - Graph-vs-graph remains unattainable: Phi hard-rejects control flow, qwen7b errors on an unconstructed `ort_value`, and qwen14b-zp accepts graph mode but no-ops due to CPU shape nodes. - Eager-vs-eager native/ORT is 0.85× / 0.77× / 1.19×; keep the 1.33× / 1.14× / 1.83× deployment headline only with the architectural capture + on-GPU argmax asterisk. +## 2026-08-18T15:58:57Z — Different-architecture GPU probe +- Probed qwen3.5-2b hybrid linear-attn, Phi-3.5-mini, and qwen3.5-9b multimodal paths in background, measure-only mode with no commits. +- Verdict: qwen3.5-2b flips small-qwen2.5 eager framing because native owns GPU kernels/capture for linear-attn ops while ORT falls back; native eager/capture are 1.10×/1.65× vs ORT eager. +- Follow-up lever: fp16 activations plus fused head_size=256 GQA decode to remove the 31% reference-attention hotspot; Phi needs native `If`/subgraph support and qwen3.5-9b needs a multimodal harness path. diff --git a/.squad/decisions.md b/.squad/decisions.md index 7db252e28f..18c7ce3dd0 100644 --- a/.squad/decisions.md +++ b/.squad/decisions.md @@ -156,3 +156,144 @@ The decision does **not** change: speculative decode remains shelved. B\* improv Luv completed the real prefill/TTFT A/B for `ONNX_GENAI_MARLIN_M_GT_1=1` versus the portable tiled GEMM path, closing the thread opened by the two prior Gate-3 Marlin entries: “Gate-3 speculative verify remains shelved after Marlin” and “Gate-3 Marlin M>1 opt-in follow-up still NO-GO.” The verdict is **NO-GO to flip the default**: Marlin M>1 stays opt-in. E2E `profile_native` TTFT showed only marginal-to-neutral qwen2.5-14b-zp movement (**0.976× / 0.988× / 0.999×** Marlin/portable at M=128/512/2048) and neutral-to-worse qwen2.5-7b movement (**1.005× / 1.013× / 1.001×**). Argmax matched every arm, but full-vocab token-0 logprob dumps were not byte-identical (max Δ **0.017** qwen14, **0.168** qwen7), so the silent-default byte-identity bar fails. Treat the Marlin-M>1 vein as mined out: not a spec-decode win, not a prefill/TTFT win, and not eligible for a silent default. + +## 2026-08-18 — Wallace small-model and different-architecture GPU probe + +### 2026-08-18: Different-architecture GPU probe — qwen3.5 (hybrid linear-attn) + Phi-3.5 + small qwen2.5 regression sanity + +**By:** Wallace (inference-engine specialist) + +**What (this addendum — the user steer: pivot off qwen2.5 to genuinely DIFFERENT architectures):** +Regression-sanity re-confirmed the two small qwen2.5 rows below are unchanged (native capture wins: +0.5b 1053.5 ≥ ORT 572; 1.5b 692.6 ≥ ORT 436 — no regression). Then pivoted to NEW architectures on the +same H200/GPU7, same fairness pinning (native `ONNX_GENAI_ONGPU_ARGMAX=1`; ORT `ONNX_GENAI_ORT_LIB` → +`.ort-cuda-1.27` CUDA build + `ONNX_GENAI_EP_FALLBACK=1`; 128 tok, warmups 2, `--steady --decode-skip 1`, +medians of 5). Worktree `.worktrees/wallace-small-probe` @ `origin/main 774b256c`, measure-only. + +**Models probed:** +| Model | Arch | Export | Native CUDA EP? | +|---|---|---|---| +| qwen3.5-2b-text (`.../qwen3.5-2b-text-generic-cpu-1/v1`) | **hybrid linear attn** (18 `linear_attention` + 6 `full_attention`, DeltaNet/mamba-style: `CausalConvWithState`+`LinearAttention` ops, `attn_output_gate`, GQA 8/2, head_dim 256) | int4 weights, **fp32 activations** (generic-cpu) | ✅ loads, runs fully on GPU, **captures** | +| Phi-3.5-mini-instruct (`.../Phi-3.5-mini-instruct-generic-cpu-2/v2`) | dense GQA, hidden 3072, head 96 | int4-rtn-blk32, fp32 acts | ❌ **native load FAILS — `If` control-flow op unsupported** | +| qwen3.5-9b (`.../qwen3.5-9b-generic-cpu-2/v2`) | multimodal (embedding+text+vision, `inputs_embeds` path) | int4, fp32 acts | ⚠️ not measured — profile_native harness expects a single decoder .onnx; multimodal split needs the embeds pipeline (harness limitation, not an EP gap) | + +**qwen3.5-2b standing row (tok/s, H200, int4-wt/fp32-act, 128 tok, greedy; medians of 5):** +| Model | native_cap | native_eager (+argmax) | ORT eager | eager-vs-eager | capture-vs-eager | capture uplift | +|---|--:|--:|--:|:--:|:--:|:--:| +| qwen3.5-2b hybrid | **100.9** (sd 0.46) | 67.5 (sd 0.96) | 61.2 (sd 4.24, noisy) | 1.10× (overlaps) | **1.65×** (clean) | 1.49× | + +**Verdict — qwen3.5 hybrid FLIPS the small-qwen2.5 story, and surfaces genuinely NEW levers:** +- **Native runs the entire hybrid graph (linear-attn conv + recurrent-state + full-attn) on GPU and + captures it cleanly** (`captures replays fallbacks=0`), eager 67.5 → capture 100.9. **ORT CUDA EP + CANNOT keep the linear-attention ops on GPU** — it inserts **25 Memcpy nodes** and falls nodes to CPU + (log: "25 Memcpy nodes added… might be unable to run CUDA graph"; strict-no-fallback init *errors*, + only runs with EP fallback). So ORT is structurally handicapped on qwen3.5 (like the DeepSeek-QMoE + finding, milder): ~61 tok/s, high variance. Native beats ORT in BOTH framings here (eager 1.10× + within noise; capture 1.65× clean) — UNLIKE qwen2.5 where native eager LOST. Root cause of the flip: + ORT lacks GPU kernels for the `LinearAttention`/`CausalConvWithState` contrib ops; native has them. +- **NEW LEVER #1 — the linear-attention path is NOT the bottleneck (it's efficient).** nsys on native + eager: `linear_attention_f32` = 5.8% + `causal_conv_with_state_f32` = 0.8% (≈6.6% combined). The repo's + linear-attn kernels are cheap. So the "linear-attention decode inefficiency" the mandate worried about + does NOT exist here — that path is well-optimized. +- **NEW LEVER #2 (the real headroom) — this export runs entirely in FP32.** Every kernel carries the + `_f32` suffix (vs qwen2.5's `_f16`). The ONNX export is int4-weight / **fp32-activation** (generic-cpu + profile: all KV, conv_state, recurrent_state, logits are `FLOAT`). fp32 activations ≈ 2× the + bandwidth/FLOPs of fp16 on the compute-bound parts. A proper fp16/CUDA-profile export (or on-the-fly + fp16 activation casting) is a large, unclaimed lever the qwen2.5 (fp16 cuda-gpu-4) campaign never saw. +- **NEW LEVER #3 — the full-attention layers use an unoptimized REFERENCE kernel.** `gqa_attention_reference_f32` + is the **#1 hotspot at 31.2%** (avg 328 µs/call). head_dim=256 falls outside the fused + `gqa_decode_attention_f16` fast path qwen2.5 (head 128/64) hits → reference fallback. Extending the + fused GQA decode kernel to head_size=256 (+ fp16) would kill ~⅓ of decode time. +- **NEW LEVER #4 — giant vocab lm_head.** vocab **248320** (1.6× qwen2.5's 152k); the lm_head GEMV + `matmul_nbits_gemv_int8_f32` = 730 µs/step, 11.6%, in fp32. On-GPU argmax already applied; fp16 + a + wider-K GEMV would help. +- **Support-gap finding (Phi-3.5-mini generic-cpu):** native decode backend does not support the **`If`** + control-flow op present in this export (all its other ops — MatMulNBits, GroupQueryAttention, + SkipSimplifiedLayerNorm, Sigmoid — are supported). Same control-flow class that blocks ORT CUDA-graph + on Phi. Needs `ONNX_GENAI_BACKEND=ort` to run, or native `If`/subgraph support to be added. + +**Bottom line for the campaign:** qwen3.5 hybrid is a NEW native win vs ORT (ORT can't place linear-attn +on GPU; native runs + captures it, 1.65×). But it also exposes that our qwen3.5 support currently runs +the *generic-cpu fp32 export* — the biggest untapped lever is an fp16 path + a fused head_size=256 GQA +decode kernel (kills the 31% reference-attention hotspot). Linear-attention itself is already efficient. +Owners: fp16-activation path + head256 fused GQA decode → CUDA-kernel owner (Leon/Deckard), reviewer-gated. + +--- + +### 2026-08-18: Small-model (qwen2.5-0.5b / 1.5b) native-vs-ORT GPU decode probe — the launch-bound regime + +**By:** Wallace (inference-engine specialist) + +**What:** +Benchmarked two never-measured small Foundry Local CUDA-GPU models — qwen2.5-0.5b-instruct +and qwen2.5-1.5b-instruct int4 (`~/.foundry/cache/models/Microsoft/qwen2.5-{0.5b,1.5b}-instruct-cuda-gpu-4/v4`) +— base decode, single-stream greedy, on an idle H200. Same fairness-correct methodology as the +now.md standing table: native `ONNX_GENAI_ONGPU_ARGMAX=1`; ORT pinned to the genuine CUDA build +(`ONNX_GENAI_ORT_LIB=$ORT_ROOT/lib/libonnxruntime.so` = `.ort-cuda-1.27`, `ONNX_GENAI_EP_FALLBACK=1` +— no conda CPU false baseline; CUDAExecutionProvider confirmed). 128 tok, warmups 2, `--steady +--decode-skip 1`, medians of 5 rounds (`--runs 3` each). Fresh worktree off `origin/main 774b256c`. +GPU pinned `CUDA_VISIBLE_DEVICES=7` (nvidia-smi: all 8 idle first). + +**Standing-table rows (tok/s, H200, int4, 128 tok, greedy; medians of 5, sd in-line):** + +| Model | native_cap (capture) | native_eager (+argmax) | ORT eager | eager-vs-eager (ne/oe) | capture-vs-eager (nc/oe) | capture uplift (nc/ne) | +|---|---:|---:|---:|:--:|:--:|:--:| +| qwen2.5-0.5b | **1053.5** (sd 0.97) | 321.5 (sd 1.22) | 572.0 (sd 6.88) | **0.56×** (ORT faster) | **1.84×** | **3.28×** | +| qwen2.5-1.5b | **692.6** (sd 1.02) | 324.8 (sd 0.89) | 435.7 (sd 1.45) | **0.75×** (ORT faster) | **1.59×** | **2.13×** | + +Overlap checks clean: capture-vs-eager min_native_cap > max_ort_eager on both (1053>585, 692>436). +eager-vs-eager decisively ORT: min_ort_eager > max_native_eager on both (568>323, 432>326). + +**Verdict:** +- **Deployment default (graph capture ON, our shipping config): native WINS big — 1.84× (0.5b) and + 1.59× (1.5b) over ORT's best (eager).** The capture moat is *largest in the small regime*: uplift + 3.28× on 0.5b and 2.13× on 1.5b, vs only ~1.5× on the 7b/14b. So native has HEADROOM here and still + beats ORT under the deployment default. This is the strongest capture win in the whole campaign. +- **BUT eager-vs-eager (pure per-kernel), native LOSES to ORT on both small models — 0.56× (0.5b) and + 0.75× (1.5b).** This is a real, honest regime where our per-kernel/per-step path is behind ORT. It + extends the pattern from the dense trio (native eager also lost on Phi 0.85× and qwen7b 0.77×; only + qwen14b-zp won eager 1.19×) — and the smaller the model, the WORSE native eager looks relative to ORT. + +**Why (root cause — a NEW headroom lever, quantified):** +Native eager decode is **fixed-per-step-overhead (launch/host-sync) bound** in the small regime: +- native_eager throughput is **model-size-INVARIANT**: 321.5 tok/s (0.5b) vs 324.8 tok/s (1.5b) — a + flat **~3.1 ms/step floor** across a 3× model-size change. A compute-bound path would scale with size. + ORT eager DOES scale (1.75 → 2.30 ms/step) and native_cap DOES scale (0.95 → 1.44 ms/step) — only + native *eager* is pinned. That flat floor is the signature of fixed per-step overhead dominating. +- **nsys (`--cuda-graph-trace=node`) on 0.5b native eager confirms it:** total GPU kernel busy ≈ 101 ms + over 128 decode steps = **~0.79 ms/step GPU-active vs 3.11 ms/step wall → only ~25% GPU-active**, + ~75% is host-side gaps between launches. No single kernel dominates (largest is the lm_head + `matmul_nbits_gemv_f16_scales_f16_pipe` at 79.7 µs/step = ~10% of GPU time); it's **death-by-many- + small-launches** — ~48 `matmul_nbits_gemv` + 48 `skip_rmsnorm` + 24 attention kernels per step, each + 2–5 µs, so per-launch overhead (~2 µs) rivals the kernel itself. This is precisely the regime CUDA + graph capture was built for: capture collapses the ~2.16 ms/step of host launch/sync overhead + (3.11 → 0.95 ms/step on 0.5b), landing native_cap near the GPU-active floor. + +**The lever:** native's eager per-step overhead (~2 ms fixed, ~75% of a small-model step) is the gap. +Options if we ever want eager competitive on small models (NOT needed for the deployment claim, which +is capture-ON): reduce host-side launch overhead — kernel fusion (fuse the per-layer gemv/rmsnorm +chains further), fewer host round-trips / stream syncs per step, or a lighter dispatch path. But the +honest framing: **for shipping (capture ON) native already wins 1.6–1.8× and this is the biggest +capture headroom we've measured — the small regime is a native STRENGTH via capture, and an eager +WEAKNESS via per-step overhead.** Same architectural story as the dense/ORT-fairness decomposition: +the moat is graph-capture + on-GPU argmax, not per-kernel eager speed. + +**Notes:** +- Did not pursue the `v4-bs128` 0.5b variant: it's a batch-128 export, irrelevant to this single- + stream per-step-overhead question (v4 already gave a decisive result). +- ORT could not be graph-captured here either (consistent with the 2026-08-18 graph-vs-graph finding: + ORT CUDA-graph structurally blocked on these dynamic-KV int4 decode exports) — so ORT's best is eager, + which is the column compared. Base decode greedy only; not a spec-decode basis. + +**Reproduce:** +```bash +source /home/justinchu/onnx-genai/.cudaenv.sh +export CUDA_VISIBLE_DEVICES=7 ONNX_GENAI_ONGPU_ARGMAX=1 \ + ONNX_GENAI_ORT_LIB=$ORT_ROOT/lib/libonnxruntime.so ONNX_GENAI_EP_FALLBACK=1 +M=~/.foundry/cache/models/Microsoft/qwen2.5-0.5b-instruct-cuda-gpu-4/v4 +ONNX_GENAI_CUDA_GRAPH=1 target/release/profile_native --model $M --ep cuda --backend native --tokens 128 --warmups 2 --runs 5 --steady --decode-skip 1 # native capture +ONNX_GENAI_CUDA_GRAPH=0 target/release/profile_native --model $M --ep cuda --backend native --tokens 128 --warmups 2 --runs 5 --steady --decode-skip 1 # native eager +ONNX_GENAI_CUDA_GRAPH=0 target/release/profile_native --model $M --ep cuda --backend ort --tokens 128 --warmups 2 --runs 5 --steady --decode-skip 1 # ORT eager +``` +Worktree `.worktrees/wallace-small-probe` @ `origin/main 774b256c`; profile_native built +`--features bench-native,bench-ort,cuda`. No source edits, no commits (measure-only). GPU7 returned idle. diff --git a/.squad/identity/now.md b/.squad/identity/now.md index 839fb44cc4..fd192ae067 100644 --- a/.squad/identity/now.md +++ b/.squad/identity/now.md @@ -22,8 +22,10 @@ Fresh steady-state decode (medians of 5, 128 tok, on-GPU argmax ON), native CUDA | qwen2.5-14b-zp | 153.3 | 85.2 | **1.80×** | | DeepSeek-V2-Lite int4 (MoE) | 101.68 (capture) | 0.17† | **capability gap** | | GLM-4-9B int4 (dense GQA) | 211.74 (capture) / 128.82 (eager) | can't load‡ | **capability gap** | +| qwen3.5-2b (hybrid linear-attn, head_dim=256) | 100.9 (capture) / 67.5 (eager) | 61.2 (eager, noisy) | **1.65× capture / 1.10× eager**§ | **Native is ahead of ORT CUDA on all three dense int4 models under real deployment conditions.** +§ qwen3.5-2b (Wallace 2026-08-18, H200, `qwen3.5-2b-text-generic-cpu-1`, int4-wt/**fp32-act**, medians of 5): FLIPS the qwen2.5 story — native beats ORT in BOTH framings (eager 1.10× within noise; capture 1.65× clean, `captures replays fallbacks=0`). Root cause: ORT CUDA EP **cannot place the hybrid `LinearAttention`/`CausalConvWithState` ops on GPU** (25 Memcpy nodes, CUDA-graph blocked, strict-no-fallback errors) — native has GPU kernels + captures the whole hybrid graph. Linear-attn path is already efficient (6.6% of decode). New levers surfaced: (#1, in flight) fused **general head-size** GQA decode kernel — head_dim=256 falls out of the head≤128 fused fast path → slow `gqa_attention_reference_f32` = **31% of decode**; (#2) generic-cpu export is **fp32-activation** (~2× fp16 cost); (#3) giant vocab 248320 lm_head fp32 GEMV 11.6%. † DeepSeek-V2-Lite: ORT CUDA EP **structurally cannot run it on GPU** — the export has 26× `com.microsoft::QMoE` and ORT 1.27 has no CUDA kernel for it → every MoE layer forced to CPU EP (104 Memcpy bridge nodes, ORT itself warns "unable to run CUDA graph") → 0.17 tok/s CPU-fallback artifact. The 598× (capture) / 336× (eager) ratios are **GPU-vs-CPU-fallback, NOT per-kernel speed** — reported honestly as a hard capability gap (Wallace 2026-08-18, ORT pinned to CUDA 1.27 build, H200). Native is the ONLY engine serving int4 QMoE on GPU (same class as GLM-4-9B). Base decode, greedy, opt-in capture flag. ‡ GLM-4-9B: ORT cannot load the dense int4 export at all (native-only). Native graph-capture engages cleanly with NO code change — the V2-Lite capture stack (#1171 classifier + #1181 `_d1` + #1189 long-context) already covers GLM's dense GQA topology (its attention-mask cone terminates at a genuine capacity-form `Attention[3]` with present-KV, so the #1171 classifier passes). Measured `captures=3 replays=185 fallbacks=0`, byte-identical 0/256 tokens (independent lock `glm4_9b_decode_lock.rs`), capture **211.74** vs eager **128.82** = **1.64×**, zero-overlap (min-ON 211.57 > max-OFF 129.05), pstdev ≤0.33 (Wallace 2026-08-18, H200 GPU6, `cohaagen-glm-4-9b-int4-cuda-post434`). Base decode, greedy, opt-in capture flag. The older 98.2 eager figure predates #1189+argmax on this box. @@ -84,6 +86,8 @@ Whole family is **Long-Scoreboard (global int4 weight-load latency) bound**, DRA Batch-1 kernel vein mined out (5 NO-GOs). Three parallel fronts, all higher-value than another batch-1 micro-opt: +- **🎯 Deckard — general head-size fused GQA decode kernel (IN FLIGHT, qwen3.5-2b lever):** the fused f32 decode fast path is capped at `MAX_HEAD_DIM=128` (32 lanes × `GQA_MAX_DPL=4`) → qwen3.5-2b's head_dim=256 falls to `gqa_attention_reference_f32` = **31% of decode**. Scoped (user directive: support VARIOUS head sizes, not one special-case) to generalize the fast path over head_dim {64,80,96,112,128,192,256} with per-dim byte-identical validation vs the CPU f64 reference + a head-64/128 regression guard, headline perf A/B on qwen3.5-2b. Also auditing **asymmetric "dual head size"** (v_head_dim ≠ qk_head_dim — Gemma family, DeepSeek MLA): add if cheap, else document exact follow-up. Byte-identical-eligible; GO gated on beating the reference kernel without register spill. +- **Support gaps surfaced by the qwen3.5 probe (Wallace 2026-08-18):** (a) **Phi-3.5-mini-instruct generic-cpu** native load FAILS on the `If` control-flow op (all other ops supported) — same control-flow class that blocks ORT CUDA-graph; needs native `If`/subgraph support or `ONNX_GENAI_BACKEND=ort`. (b) **qwen3.5-9b multimodal** (embedding+text+vision, `inputs_embeds` path) unmeasured — `profile_native` expects a single decoder .onnx; needs the embeds pipeline (harness limitation, not an EP gap). - **Wallace — ORT fairness follow-up (in flight):** THE open question on the headline claim. Native leads ORT (1.33/1.13/1.80×) but native runs a captured CUDA graph while ORT runs eager. Get ORT into CUDA-graph mode (static-shape IO binding) for graph-vs-graph, or isolate the kernel-only delta via eager-vs-eager. Either outcome is honest. - **🎯 V2-Lite CUDA-graph-capture UNLOCK (top lever, 2-STAGE — stage 1 done, stage 2 IN FLIGHT):** Wallace scoped GO → Deckard implemented the mask classifier → but a SECOND pre-existing blocker gates the real win. - **Stage 1 ✅ MERGED — Wallace's corrected classifier (PR #1171, main @ `bc1e97ff`):** topology-gated capacity-classifier in `executor/{geometry,build}.rs`. Deckard's original was 🔴 REJECTED (2 safety blockers); Deckard LOCKED OUT, **Wallace revised** (same capacity-policy surface he scoped), Rachael 🟢 re-approved, admin-merged. Both blockers fixed, strictly TIGHTENING: (1) `is_capacity_form_attention_mask_input` requires present-KV `inputs[4]`+`inputs[5]` (grounded `standard_attention.rs:822-840`), test `mask_builder_to_attention_without_past_kv_is_rejected`; (2) dropped `value != mask` exception so a root mask that escapes as graph output is rejected, test `mask_binding_that_is_graph_output_is_rejected`. GLM-5.2 `Add` exclusion preserved.