research(decode): Lever B Phase-0 capture-stability probe — gated-GO on Increment-0 by justinchuby · Pull Request #948 · justinchuby/onnx-genai

justinchuby · 2026-08-14T14:44:23Z

Phase-0 de-risking probe for Lever B (capture-stable padded M=K verify), on glm-4-9b-int4 / H200. Throwaway #[ignore]d un-wired microbench + findings doc + decision drop. No pipeline wiring.

Three pass-criteria measured:

(b) capture stability — PASS: 1000 M=1 steps → captures=3/replays=994/invalidations=3, 90.3 tok/s, no thrash (vs eager verify 6→280).
(a) M=K instantiates capture-safe — FAIL (build gap, not kernel veto): real M=K try_capture = NotCapturable because [1,K,vocab] logits materialized (binding is [1,1,vocab]) + 724 alloc/722 free transient workspaces at M=8. Both are the missing Increment-0 (persistent [1,K_max,vocab] logits binding + alloc-free M=K workspace + KV-symbol pin).
(c) captured M=K wall ≈ M=1 wall — UNMEASURED (blocked by a): eager proxy M=1 13.5ms → M=8 91.5ms (6.77×), a step function: M=1→M=2 cliff = 66.9ms, near-flat M=2→M=8 tail = 1.85ms/row. Cliff composition (dispatch vs generic-GEMM+alloc) is the pivotal unknown.

Verdict: NO-GO to an unconditional multi-week Lever B commit; gated-GO on a cheap Increment-0 (days) — build the persistent logits binding + alloc-free workspace + KV pin, re-run this probe for the decisive captured-M=8-vs-M=1 replay number. If captured M=K ≈ M=1 → GO full build; if the ~80ms floor persists under capture → promote Lever A (Marlin, unconditional 1.3–1.6×) to primary.

Docs + probe + decision drop only.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

Cheap, throwaway, #[ignore]d, UN-WIRED measurement probe answering the one load-bearing Lever B question on glm-4-9b-int4 (H200): can a fixed-shape padded M=K forward be CUDA-graph captured, replay ~1 dispatch/verify across KV bucket growth, and cost ~= one M=1 replay? Result by strict "PASS = all three": (b) PASS, (a) FAIL, (c) UNMEASURED -> NO-GO to an unconditional multi-week commit; GATED-GO on a cheap Increment-0. - (b) 1000 M=1 steps: captures=3 replays=994 invalidations=3 growths=2, 90.3 tok/s captured, no thrash. Capture machine stable across bucket growth. - (a) real M=K try_capture NotCapturable: [1,K,vocab] logits materialized + ~722 transient workspace allocs. Both are the missing Increment-0 build (persistent [1,K_max,vocab] logits binding + alloc-free M=K workspace + KV-symbol pin), not a kernel veto. - (c) blocked by (a). Eager proxy: M=1 13.5ms, M=8 91.5ms = 6.77x, a step function: M=1->M=2 cliff 66.9ms, near-flat M=2->M=8 tail 1.85ms/row. The cliff's composition (dispatch vs generic-GEMM+alloc) is the pivotal unknown Phase-0 could not resolve. M=8 logit readback 4.62MB (small). Recommend: fund Increment-0, re-run this probe for the decisive captured M=8-vs-M=1 replay number, then GO/NO-GO. If the ~80ms floor persists under capture, promote Lever A (Marlin, unconditional 1.3-1.6x) to primary. Adds: - native_decode/leverb_phase0_probe.rs (#[ignore]d probe, cfg-gated) - #[cfg(test)] leverb_phase0_capture_attempt + LeverBPhase0CaptureAttempt (cuda.rs) - #[cfg(test)] load_with_cuda_options_and_io_spec (load.rs) - docs/research/leverb-phase0-capture-probe.md (findings) - .squad/decisions/inbox/deckard-leverb-phase0.md (decision drop) None of the above is wired into any decode path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

codecov · 2026-08-14T14:52:42Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.81%. Comparing base (2689e8e) to head (9aeab26).
⚠️ Report is 6 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #948      +/-   ##
==========================================
- Coverage   78.46%   77.81%   -0.65%     
==========================================
  Files         354      354              
  Lines      148153   148153              
  Branches   148153   148153              
==========================================
- Hits       116246   115287     -959     
- Misses      27410    28372     +962     
+ Partials     4497     4494       -3

Flag	Coverage Δ
cli-ort-linux	`83.42% <ø> (ø)`
cli-ort-windows	`82.92% <ø> (ø)`
mlas	`80.96% <ø> (ø)`
offline	`77.55% <ø> (-0.69%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 6 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…uda feature The leverb_phase0_probe module is #[cfg(all(test, feature = "cuda"))], but the LeverBPhase0CaptureAttempt struct and leverb_phase0_capture_attempt method it consumes were only #[cfg(test)]. Under the native-backend (non-CUDA) test build, they compiled with no consumer, tripping dead_code as clippy -D warnings in the 'Check the native backend compiles' CI step. Align their gate to the module's. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-08-14T15:30:06Z

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	`matmul/small_generic_f32_threads=8/1x256x256`	31.76 µs	53.19 µs	+67.5%
🔴	`block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024`	71.58 µs	94.76 µs	+32.4%
🔴	`block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024`	767.04 µs	1.00 ms	+30.4%
⚠️	`matmul/medium_generic_f16_threads=8/32x512x512`	28.13 µs	36.13 µs	+28.4%
⚠️	`matmul/small_generic_bf16_threads=8/1x256x256`	29.74 µs	37.38 µs	+25.7%
⚠️	`matmul/medium_generic_f32_threads=8/32x512x512`	893.99 µs	1.12 ms	+24.8%
⚠️	`matmul/small_generic_bf16_threads=1/1x256x256`	30.36 µs	37.73 µs	+24.3%
⚠️	`matmul/small_generic_f16_threads=1/1x256x256`	27.70 µs	34.30 µs	+23.8%
⚠️	`matmul/medium_generic_f16_threads=1/32x512x512`	27.97 µs	34.03 µs	+21.7%
⚠️	`tokenization/decode_tokens_per_second`	6.42 ms	7.80 ms	+21.5%
⚠️	`add/large_f32_threads=1-internal/4194304`	45.49 ms	54.57 ms	+20.0%
⚠️	`gather/large_f32_threads=1-internal/131072`	24.92 µs	29.41 µs	+18.0%
⚠️	`add/medium_bf16_threads=1-internal/262144`	2.80 ms	3.27 ms	+16.7%
⚠️	`kv_cache/alloc_dealloc_pages`	37.09 µs	43.22 µs	+16.5%
⚠️	`matmul/medium_generic_f32_threads=1/32x512x512`	2.16 ms	2.51 ms	+16.1%
⚠️	`sampling_latency/greedy_per_token`	3.03 µs	3.50 µs	+15.4%
⚠️	`matmul/small_generic_f16_threads=8/1x256x256`	29.06 µs	33.51 µs	+15.3%
✅	`gather/small_f32_threads=1-internal/4096`	622.5 ns	704.5 ns	+13.2%
✅	`gather/medium_f32_threads=1-internal/32768`	3.60 µs	4.07 µs	+12.9%
✅	`block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024`	54.72 µs	61.70 µs	+12.8%
✅	`matmul/small_generic_f32_threads=1/1x256x256`	34.02 µs	38.25 µs	+12.4%
✅	`gather/medium_f16_threads=1-internal/32768`	2.26 µs	2.49 µs	+10.2%
✅	`reduce_mean/medium_f32_threads=1-internal/65536`	228.36 µs	250.85 µs	+9.8%
✅	`add/small_f16_threads=1-internal/1024`	13.30 µs	14.54 µs	+9.3%
✅	`qwen3_sampling_processors/top_k_top_p_fast`	646.70 µs	700.77 µs	+8.4%
✅	`matmul/medium_generic_bf16_threads=1/32x512x512`	533.49 µs	568.43 µs	+6.5%
✅	`add/large_bf16_threads=1-internal/4194304`	41.31 ms	43.89 ms	+6.2%
✅	`qwen3_sampling_processors/top_k_top_p_full_sort_baseline`	5.54 ms	5.85 ms	+5.7%
✅	`gather/large_bf16_threads=1-internal/131072`	11.24 µs	11.85 µs	+5.4%
✅	`grammar_masking/llguidance_compute_mask/32`	89.45 µs	94.31 µs	+5.4%
✅	`sampling_latency/top_p_per_token`	387.63 µs	405.64 µs	+4.6%
✅	`qwen3_sampling_processors/top_k_partial_selection`	145.34 µs	152.04 µs	+4.6%
✅	`gather/small_f16_threads=1-internal/4096`	489.8 ns	511.9 ns	+4.5%
✅	`qwen3_sampling_processors/top_p_fast_after_top_k`	531.20 µs	554.66 µs	+4.4%
✅	`sampling_latency/top_k_per_token`	55.83 µs	58.23 µs	+4.3%
✅	`block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1`	82.63 µs	85.76 µs	+3.8%
✅	`reduce_mean/large_f32_threads=1-internal/262144`	954.03 µs	985.86 µs	+3.3%
✅	`logit_processing/seven_processor_chain_per_step`	347.36 µs	356.28 µs	+2.6%
✅	`tokenization/encode_tokens_per_second`	407.71 µs	417.26 µs	+2.3%
✅	`matmul/medium_generic_bf16_threads=8/32x512x512`	565.22 µs	573.93 µs	+1.5%
✅	`gather/medium_bf16_threads=1-internal/32768`	2.25 µs	2.28 µs	+1.4%
✅	`qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline`	3.56 ms	3.60 ms	+1.3%
✅	`sampling_latency/min_p_per_token`	216.20 µs	217.19 µs	+0.5%
✅	`add/large_f16_threads=1-internal/4194304`	44.09 ms	44.19 ms	+0.2%
✅	`qwen3_sampling_processors/top_k_full_sort_baseline`	2.25 ms	2.24 ms	-0.4%
✅	`add/small_bf16_threads=1-internal/1024`	13.83 µs	13.70 µs	-0.9%
✅	`reduce_mean/small_f32_threads=1-internal/4096`	15.45 µs	15.26 µs	-1.3%
✅	`matmul/large_generic_f32_threads=8/32x1024x1024`	4.33 ms	4.23 ms	-2.4%
✅	`matmul/large_generic_bf16_threads=1/32x1024x1024`	2.23 ms	2.15 ms	-3.8%
✅	`matmul/large_generic_f16_threads=8/32x1024x1024`	94.96 µs	91.08 µs	-4.1%
✅	`matmul/large_generic_f16_threads=1/32x1024x1024`	93.78 µs	89.93 µs	-4.1%
✅	`add/medium_f16_threads=1-internal/262144`	3.00 ms	2.83 ms	-5.9%
✅	`gather/large_f16_threads=1-internal/131072`	12.16 µs	11.38 µs	-6.4%
✅	`matmul/large_generic_bf16_threads=8/32x1024x1024`	1.90 ms	1.77 ms	-6.7%
✅	`matmul/large_generic_f32_threads=1/32x1024x1024`	10.64 ms	9.87 ms	-7.2%
✅	`add/medium_f32_threads=1-internal/262144`	2.65 ms	2.46 ms	-7.3%
✅	`block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1`	555.18 µs	513.22 µs	-7.6%
✅	`add/small_f32_threads=1-internal/1024`	234.6 ns	214.5 ns	-8.6%
🟢	`gather/small_bf16_threads=1-internal/4096`	558.0 ns	459.0 ns	-17.7%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 5.33 4.26 6.55 }

What this cannot catch

Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
Sub-threshold regressions that compound over multiple PRs
Performance changes that only manifest under GPU execution
Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

…→ NO-GO, promote Marlin (#949) Follow-up to #948. Built the 3 capture-enablement fixes Phase-0 identified (persistent padded `[1,K_max,vocab]` logits binding + alloc-free pre-capture warm forward + inherited KV-symbol pin) as a throwaway `#[ignore]`d, un-wired test overlay, then took the decisive measurement on glm-4-9b-int4 / H200 (reproduced 3×). **Criteria now:** - **(a) instantiates capture-safe → PASS.** M=8: `captured=true`, `capture_alloc=(0,0)`, `decline=None` (Phase-0's raw M=8 was NotCapturable). The recipe works. - **(b) stable replay across bucket growth → PASS** (unchanged). - **(c) captured M=K wall ≈ captured M=1 wall → FAIL.** Captured M=8 = **87.2 ms** vs captured M=1 = **10.2 ms** = **8.58×**. The ~80 ms floor persists *under capture*. **Smoking gun:** `segments=41` at M=8 vs `1` at M=1 — the M=K forward does NOT whole-graph capture. Every hot kernel opts out at query-width >1: `GroupQueryAttention[KernelCaptureUnsupported]×40`, `MatMulNBits×240`, `SkipSimplifiedLayerNormalization×80`. "Captured" M=K degrades to ~361 eager per-op relaunches — the dispatch-bound regime the lever meant to escape. **Verdict: NO-GO for Lever B. Promote Lever A (Marlin int4 relayout, unconditional 1.3–1.6×) to primary.** Lever B is gated behind a multi-family kernel-capture-support program (make GQA / MatMulNBits / SkipSimplifiedLayerNorm capture-safe at M>1); this exact probe can be re-run after that lands. Throwaway probe + docs + decision drop only; no pipeline wiring. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Scribe state-only batch: merged 2 decision drops (#948 leverb-phase0, #949 leverb-increment0) into decisions.md as the resolved 'Decode perf REOPENED — RESOLVED' section (Lever B tested and NO-GO, Marlin/Lever A promoted to primary — supersedes the #938 'build Lever B first' recommendation), archived older bodies to keep decisions.md under 50 KB (48,169 → 40,842 B), appended deckard history + logs. State-only; no code. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…d behind Marlin (#957) ## Speculative decoding under CUDA-graph capture — feasibility (DESIGN-ONLY) Answers Justin's question — *"想办法把 speculative decoding 的 capture 做好？有办法吗？"* — with a concrete, honest staged path. **No kernels touched; adds one research doc + one decision drop.** ### Verdict: CONDITIONAL-GO, and the single condition is Marlin (Lever A) — which we are already building as the primary decode lever. - **Today (measured, #948/#949):** captured M=8 verify = **87.2 ms = 8.58× captured M=1 (10.2 ms)** ⇒ acceptance break-even **B\* = 8.5** ⇒ physically impossible ⇒ guaranteed loss (matches the settled speculative KILL). - **Root cause (code-grounded, no new GPU run):** the ~80 ms floor is **dominated by the 240 `MatMulNBits` generic 16×16 CUDA-core tiled GEMMs** — M>1 leaves the tuned fused int4 GEMV (`matmul_nbits.rs:4803-4869`). GQA (efficient flash) and norm (same kernel) live in the cheap 1.85 ms/row tail. - **Marlin is the enabler, not just a speedup:** static launch grid ⇒ capture-safe by construction (kills the 240 seams) + weights reused across the M rows ⇒ collapses the 67 ms cliff. **Estimated** post-Marlin verify ~12–20 ms ⇒ break-even **B\* ≈ 1.2–2.0**, cleared by prompt-lookup (B≈2–4) and EAGLE-3/MTP. - **Residual blockers after Marlin are cheap:** SkipSimplifiedLayerNorm = a `num_groups==1` **signature relaxation** (`normalization.rs:2306`); GQA = **pre-warmed fixed-M flash workspace** (`group_query_attention.rs:3099`). Not kernel rewrites. ### Staged plan with hard gates 1. **Marlin** (already primary; ref `docs/research/decode-remaining-levers-feasibility.md`) + a new "capture-safe at fixed M=K" contract item. 2. **GQA + norm M>1 capture-support** (the two cheap fixes) — gate: seam census collapses to ~1 segment. 3. **Re-run THIS Increment-0 probe** for the true post-Marlin captured M=K wall — **fund/kill gate: measured B\* ≤ ~2 ⇒ GO, ≳4 ⇒ NO-GO.** 4. **Only then** build the verify-graph + capture-stable KV-commit + near-tie guard + draft sources. ### Context - Refines (does not overturn) the settled **Lever B NO-GO** — #948 (Phase-0) / #949 (Increment-0). - Marlin scoping: `docs/research/decode-remaining-levers-feasibility.md`. - Portability (Rule 11): Marlin is SM-version-specific ⇒ arch guard + split-K fallback + degrade-to-disabled on non-Marlin tiers. H200/glm-4-9b-int4-scoped. ⚠️ Design/research doc — no production code changed. A squad member should sanity-check the post-Marlin verify-wall estimate (clearly labeled as an estimate; Stage-3 measures it). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Scribe consolidation for the Marlin int4 GEMM program landing (#960/#961/#962). - Merge 5 decision inbox drops (deckard/pris/sebastian kernel+bench+numerics + chew/gaff reviews) into one consolidated Marlin program entry; corrected per-model attribution (glm block-128 → gate_up fusion no-op; 2.16× was the general split-K retune). - Archive resolved #957/#948/#949 spec-capture + Lever B entries to decisions-archive/2026-08.md (decisions.md 45,991 → 42,104 B, under the 50 KB gate). - Orchestration logs (deckard/sebastian/pris/chew/gaff), session log, cross-agent history updates; summarized sebastian history (>8 chronicle threshold). State-only change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

justinchuby merged commit c5f4fa6 into main Aug 14, 2026
15 checks passed

justinchuby deleted the squad/leverb-phase0-capture-probe branch August 14, 2026 15:20

justinchuby mentioned this pull request Aug 14, 2026

research(decode): Lever B Increment-0 — decisive captured M=K number → NO-GO, promote Marlin #949

Merged

justinchuby mentioned this pull request Aug 14, 2026

chore(scribe): consolidate Lever B NO-GO decisions, archive, logs #953

Merged

justinchuby mentioned this pull request Aug 14, 2026

docs(research): speculative-capture feasibility — CONDITIONAL-GO gated behind Marlin #957

Merged

justinchuby mentioned this pull request Aug 14, 2026

chore(scribe): consolidate Marlin int4 GEMM decisions + logs #972

Merged

Navigation Menu

Repository navigation

research(decode): Lever B Phase-0 capture-stability probe — gated-GO on Increment-0 - #948

research(decode): Lever B Phase-0 capture-stability probe — gated-GO on Increment-0#948
justinchuby merged 2 commits into
mainfrom
squad/leverb-phase0-capture-probe

justinchuby commented Aug 14, 2026

Uh oh!

codecov Bot commented Aug 14, 2026 •

edited

Loading

Uh oh!

Uh oh!

github-actions Bot commented Aug 14, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

justinchuby commented Aug 14, 2026

Uh oh!

codecov Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

github-actions Bot commented Aug 14, 2026

🔴 Benchmark Regression Detected

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

codecov Bot commented Aug 14, 2026 •

edited

Loading