diff --git a/.squad/agents/cohaagen/history-archive.md b/.squad/agents/cohaagen/history-archive.md new file mode 100644 index 0000000000..ee43c8dcc1 --- /dev/null +++ b/.squad/agents/cohaagen/history-archive.md @@ -0,0 +1,36 @@ +# cohaagen — History Archive + +_Archived by Scribe round 9, 2026-07-31T10:24:07Z._ + +## 2026-07-29T00:45:00+0000 — PR #380 re-review +- Re-reviewed Melina's encoder-decoder fixture correction for issue #377 and approved PR #380, merged as `47c3331d`. +- Ran the CLI ORT E2E gate (23/23); metadata/I/O-detection reviews require that gate alongside engine/native unit tests. + +## 2026-07-30T09:16:00Z — 7B native CUDA perf findings + +- The Foundry baseline reports native CUDA ahead of ORT; 7B tracing localized o_proj at 19.5% of kernel time. +- Reverted the two-way o_proj split-K gate after a repeatable 0.59% 7B regression; do not retry that lever without a new higher-split kernel experiment. + +## 2026-07-30T13:36:00Z — Issue #63 increment 2 delivered; #87 assessment complete + +- PR #444 merged: wired GPU weight pager into live decode hot-path with `ONNX_GENAI_WEIGHT_OFFLOAD=1`. Added `CudaWeightResidency` LRU device-page cache, env-gated device policy, extended lazy boundary to `com.microsoft::QMoE` + `MatMulNBits`. +- Test: 0.6B native CUDA on 2MiB budget: 32-token decode 2.79 → 2.30 tok/s (1.21× slowdown); tokens identical; paging counters confirm page-ins/evictions. +- Issue #87: assessed async H2D prefetch infrastructure — all async/copy-stream machinery exists and is GPU-tested. Missing only live wiring (residency currently synchronous). Inc1 = switch to async, Inc2 = double-buffered look-ahead. Unblocked, awaiting signal. + +## 2026-07-30T15:20:00Z — PR #480 merged (CUDA CausalConvWithState + GBQ coverage) + +- PR #480 merged (Melina APPROVED): NVRTC `CausalConvWithState` kernel (fp32/fp16/bf16, f32-accum to match ORT/CPU) + `GatherBlockQuantized` coverage declaration (#67). Oracle chain CUDA->CPU EP->ORT; advertised ops 161 -> 163. Closed the registered-but-undeclared GBQ coverage-of-coverage hole. +- Empirical #67 audit: classic transformer decode is 100% covered on CUDA; control-flow (If/Loop/Scan) is executor-handled and must NOT be added to the EP; remaining real gaps are the Qwen3.5 hybrid family (CausalConvWithState landed, LinearAttention next). +- Follow-up (safe-to-defer, fail-closed): GBQ bits=4 odd-blocks-per-row. In flight: LinearAttention (rank-2 gap), PR pending. + +## 2026-07-30T21:15:00Z — PR #484 + #525 merged (CUDA LinearAttention; #67 coverage-polish) + +- PR #484 merged: CUDA LinearAttention (Gated DeltaNet) kernel — per-thread f32-register-column state; 4/4 parity; qwen3.5 hybrid node placement 0→18/18/24. Coordinator resolved a merge conflict with #480 in kernels/mod.rs/provider.rs/docs (union of ops). +- PR #525 merged (Melina APPROVED): #67 coverage-polish — RotaryEmbedding com.microsoft + fixed dtype-check bug (Int64 position_ids compared vs float); Bool NonZero on both EPs via `to_dense_bool`; GatherBlockQuantized odd-blocks-per-row LOUD fail-closed gate + honest doc softening. Op counts unchanged (no new op names). +- Qwen3.5 hybrid recurrent op set now fully CUDA-covered (CausalConvWithState #480 + LinearAttention #484 + RoPE-contrib/NonZero #525). + +## 2026-07-31T00:25:00Z — PR #529 merged (qwen3.5-0.8b hybrid 100% CUDA placement) + +- PR #529 merged: qwen3.5-0.8b hybrid places 100% on CUDA — split package embedding.onnx 24 nodes + text.onnx 1265 = 1289 nodes, 0 declines (after #480/#484/#525). Regression lock `qwen35_0_8b_placement_lock`. +- E2e decode still BLOCKED on the loader: `Engine::from_dir` rejects the 3-onnx split; `from_pipeline_dir` refuses during vision `smart_resize` admission. Parity harness `qwen35_0_8b_hybrid_native_cuda_e2e` graceful-skips until fixed. +- In flight (cohaagen-4): loader-unblock — admit the text-only split hybrid for decode, flip the e2e parity harness active. diff --git a/.squad/agents/cohaagen/history.md b/.squad/agents/cohaagen/history.md index eeb24dfae2..7a22e95351 100644 --- a/.squad/agents/cohaagen/history.md +++ b/.squad/agents/cohaagen/history.md @@ -1,47 +1,25 @@ # cohaagen — History -## 2026-07-29T00:45:00+0000 — PR #380 re-review -- Re-reviewed Melina's encoder-decoder fixture correction for issue #377 and approved PR #380, merged as `47c3331d`. -- Ran the CLI ORT E2E gate (23/23); metadata/I/O-detection reviews require that gate alongside engine/native unit tests. +## Role +CUDA performance and kernel specialist. Owns CUDA EP op-coverage (#67), weight-offload (#63/#87), kernel tuning (o_proj split-K, CausalConvWithState, LinearAttention), and decode profiling. -## 2026-07-30T09:16:00Z — 7B native CUDA perf findings - -- The Foundry baseline reports native CUDA ahead of ORT; 7B tracing localized o_proj at 19.5% of kernel time. -- Reverted the two-way o_proj split-K gate after a repeatable 0.59% 7B regression; do not retry that lever without a new higher-split kernel experiment. - -## 2026-07-30T13:36:00Z — Issue #63 increment 2 delivered; #87 assessment complete - -- PR #444 merged: wired GPU weight pager into live decode hot-path with `ONNX_GENAI_WEIGHT_OFFLOAD=1`. Added `CudaWeightResidency` LRU device-page cache, env-gated device policy, extended lazy boundary to `com.microsoft::QMoE` + `MatMulNBits`. -- Test: 0.6B native CUDA on 2MiB budget: 32-token decode 2.79 → 2.30 tok/s (1.21× slowdown); tokens identical; paging counters confirm page-ins/evictions. -- Issue #87: assessed async H2D prefetch infrastructure — all async/copy-stream machinery exists and is GPU-tested. Missing only live wiring (residency currently synchronous). Inc1 = switch to async, Inc2 = double-buffered look-ahead. Unblocked, awaiting signal. - -## 2026-07-30T15:20:00Z — PR #480 merged (CUDA CausalConvWithState + GBQ coverage) - -- PR #480 merged (Melina APPROVED): NVRTC `CausalConvWithState` kernel (fp32/fp16/bf16, f32-accum to match ORT/CPU) + `GatherBlockQuantized` coverage declaration (#67). Oracle chain CUDA->CPU EP->ORT; advertised ops 161 -> 163. Closed the registered-but-undeclared GBQ coverage-of-coverage hole. -- Empirical #67 audit: classic transformer decode is 100% covered on CUDA; control-flow (If/Loop/Scan) is executor-handled and must NOT be added to the EP; remaining real gaps are the Qwen3.5 hybrid family (CausalConvWithState landed, LinearAttention next). -- Follow-up (safe-to-defer, fail-closed): GBQ bits=4 odd-blocks-per-row. In flight: LinearAttention (rank-2 gap), PR pending. - -## 2026-07-30T21:15:00Z — PR #484 + #525 merged (CUDA LinearAttention; #67 coverage-polish) - -- PR #484 merged: CUDA LinearAttention (Gated DeltaNet) kernel — per-thread f32-register-column state; 4/4 parity; qwen3.5 hybrid node placement 0→18/18/24. Coordinator resolved a merge conflict with #480 in kernels/mod.rs/provider.rs/docs (union of ops). -- PR #525 merged (Melina APPROVED): #67 coverage-polish — RotaryEmbedding com.microsoft + fixed dtype-check bug (Int64 position_ids compared vs float); Bool NonZero on both EPs via `to_dense_bool`; GatherBlockQuantized odd-blocks-per-row LOUD fail-closed gate + honest doc softening. Op counts unchanged (no new op names). -- Qwen3.5 hybrid recurrent op set now fully CUDA-covered (CausalConvWithState #480 + LinearAttention #484 + RoPE-contrib/NonZero #525). -- In flight: hybrid-e2e. - -## 2026-07-31T00:25:00Z — PR #529 merged (qwen3.5-0.8b hybrid 100% CUDA placement) - -- PR #529 merged: qwen3.5-0.8b hybrid places 100% on CUDA — split package embedding.onnx 24 nodes + text.onnx 1265 = 1289 nodes, 0 declines (after #480/#484/#525). Regression lock `qwen35_0_8b_placement_lock`. -- E2e decode still BLOCKED on the loader: `Engine::from_dir` rejects the 3-onnx split; `from_pipeline_dir` refuses during vision `smart_resize` admission. Parity harness `qwen35_0_8b_hybrid_native_cuda_e2e` graceful-skips until fixed. -- In flight (cohaagen-4): loader-unblock — admit the text-only split hybrid for decode, flip the e2e parity harness active. +_Entries before 2026-07-31T03:03:15Z archived to `history-archive.md` (Scribe round 9). Archived: PR #380 re-review; 7B perf (o_proj split-K revert); #63 inc2 (#444 weight pager); #480 (CausalConvWithState+GBQ); #484+#525 (LinearAttention+RoPE-contrib/NonZero); #529 (qwen3.5 100% CUDA placement)._ ## 2026-07-31T03:03:15Z — PR #535 merged: text-only decode pipeline synthesis (hybrid loader unblock) -- PR #535 merged: unblocks the split VLM package whose image preprocessing is unrepresentable (`smart_resize`). New `GenAiConfigError::UnrepresentablePreprocessing` (distinct from `IncompletePipeline`) → `to_strict_text_only_pipeline_metadata` synthesizes an embedding→decoder AR pipeline with NO vision component (positions rank-3 `linear_increment`, decoder `inputs_embeds`). Modality-driven, NOT a model-name case. Resolves the symbolic leading (batch) axis in `decode/values.rs`. Flips the `qwen35_0_8b_hybrid_native_cuda_e2e` parity harness active (paved the way for Mary's #543 landmark). -- Weight-offload chain (#63): #444 first increment merged — `page_lazy_weight` dispatch-layer seam + `CudaWeightResidency` LRU, gated behind `ONNX_GENAI_WEIGHT_OFFLOAD=1`, token-identical, ~1.21× slowdown at 2 MiB. NEXT LEVER: #87 async page-in (PLAN-ONLY, awaiting Justin green-light). -- Guardrail proven: o_proj 2-way split-K (K_SPLIT=2) REGRESSES the 7B o_proj GEMV (−0.59%, repeatable) — do NOT re-try; reduction tax > sub-wave grid-fill. A K_SPLIT>2 new kernel with its own A/B is the future candidate. +- PR #535 merged: unblocks the split VLM package whose image preprocessing is unrepresentable (`smart_resize`). New `GenAiConfigError::UnrepresentablePreprocessing` → `to_strict_text_only_pipeline_metadata` synthesizes an embedding→decoder AR pipeline with NO vision component. Modality-driven, NOT a model-name case. Flips the `qwen35_0_8b_hybrid_native_cuda_e2e` parity harness active. +- Weight-offload chain (#63): #444 first increment merged — `page_lazy_weight` dispatch-layer seam + `CudaWeightResidency` LRU, gated behind `ONNX_GENAI_WEIGHT_OFFLOAD=1`. NEXT: #87 async page-in (PLAN-ONLY, awaiting Justin green-light). +- Guardrail proven: o_proj 2-way split-K (K_SPLIT=2) REGRESSES the 7B o_proj GEMV (−0.59%) — **do NOT re-try**. A K_SPLIT>2 new kernel is the future candidate. ## 2026-07-31T08:48:28Z — PR #544 async page-in MERGED; #552 observability MERGED; GQA capture validated - PR #544 merged (Harry test fix; Melina RC→APPROVE): async `cuMemcpyHtoDAsync`+`compute_wait_fence` page-in; eviction re-serializes (WAR); ~1% win at 2 MiB only; byte-identical all budgets. `ONNX_GENAI_WEIGHT_OFFLOAD_ASYNC_PAGEIN` default ON. - PR #552 merged (Lori APPROVE): `NativeDecodeSession::load_with_resolved_io` — genai_config decoders now emit `cuda_graph captures/fallbacks/fallback_report` + `--trace` capture-reject reasons. `profile_native` simple path now fully observable. - GQA capture default-on validated (verification-only, no code change): Qwen2.5-0.5B int4 GQA — ON 2.14× eager, byte-identical, zero declines. Standing question answered. Gemma-4-E2B unmeasurable: stale HF export + GAP-3. Capture default-on for multi-component deferred until mary's native pipeline + re-export. + +## 2026-07-31T10:24:07Z — 27B decode roofline profile (profile-only, no code changed) + +- Profiled Qwen3.6-27B int4 native-CUDA resident decode on H200. Steady-state: 168 ms/tok (~35× off roofline). +- Root cause: `Scan` (48 LinearAttention blocks/step) = **56.5%** of decode step — structurally un-capturable control-flow. MatMulNBits already at roofline (4.4 ms, 2%) — **do NOT touch for this**. +- Capture engages (fallbacks=0); Transpose per-call sync 9.2% is next lever in Cohaagen's lane (separate from Scan fix). +- Fix is Justin's / pipeline-control-flow area. This was a profile+identify task only; no worktree remains. diff --git a/.squad/agents/mary/history.md b/.squad/agents/mary/history.md index 99ad8beed8..e4438a1943 100644 --- a/.squad/agents/mary/history.md +++ b/.squad/agents/mary/history.md @@ -29,3 +29,10 @@ _Entries before 2026-07-30T21:15 archived to `history-archive.md` (Scribe round - **27B offload A/B (H200):** Qwen3.6-27B int4 (497 MatMulNBits). 2 GiB budget → 6.2 GiB peak VRAM (2.9× vs 17.7 GiB resident), byte-exact at all budgets. Cliff: ≤12 GiB → 0.11 tok/s (bandwidth-bound, whole working set evicted/token). cuda_graph auto-off under offload. Action item: add `model.io` block to canonical int4-cuda package. - **#553 bug found (#554 MERGED, Harry APPROVED):** `NativeDecodeSession` 2nd+ generation returns garbage — conv_state/recurrent_state not zeroed between generates. Not graph-capture related. Fixed. - **Native pipeline keystone test:** `native_full_pipeline_parity` drives `tiny-gemma4-vlm` with native embedding + native decoder simultaneously → `[0,5,6,7] == ORT`. Closes composite-native safety gap ahead of 35B-A3B wiring. + +## 2026-07-31T10:24:07Z — Scan-capture increment assessed: STOP+report; right-sized workstream proposed + +- Reproduced 176.7 ms/tok baseline (matches Cohaagen ±5%). Confirmed `--trace` Scan seam. +- Single-trip Scan inline is NOT an increment: (1) shared prefill+decode plan — static inline corrupts prefill; (2) control-flow structurally declined at `provider.rs:458`; (3) no child-body-fold machinery. +- **Right-sized: Approach 1 (runtime dual-path)** — 1a correctness behind flag, 1b body enters capture. Blast radius: #443/#543 core. Reference decode tokens locked. **Awaiting Justin go-ahead.** +- No code changed; worktree `wt-mary-scan` removed. diff --git a/.squad/decisions-archive/2026-07.md b/.squad/decisions-archive/2026-07.md index 7867a368a7..f0e1a3f6f9 100644 --- a/.squad/decisions-archive/2026-07.md +++ b/.squad/decisions-archive/2026-07.md @@ -23293,4 +23293,17 @@ narrative below). Wave: #535, #540, #541, #543 all MERGED to origin/main; contex for #377; native side complete via #382/#412). Held: **#534** (Harry, server contracts #481/#482, Melina APPROVED) targets Justin's branch `feat/genai-demo-dashboard` (PR #476), not on main. In flight: gemma-3n vision-optional text-only decode; #87 async weight prefetch (awaiting -green-light); 35B-A3B routed-expert MoE representation. \ No newline at end of file +green-light); 35B-A3B routed-expert MoE representation. +--- + +## 2026-07-31 — Scribe consolidation (round 8) + +**By:** Scribe. Merged: #544 #552 #554. Wave narrative in archive. + +- **#544 async page-in inc1** (Cohaagen; Harry test fix; Melina RC→APPROVE): `cuMemcpyHtoDAsync`+`compute_wait_fence`; eviction re-serializes (WAR). ~1% at 2 MiB; byte-identical all budgets. `ONNX_GENAI_WEIGHT_OFFLOAD_ASYNC_PAGEIN` default ON. +- **#552 profile_native observability** (Cohaagen; Lori APPROVE): `load_with_resolved_io` — genai_config decoders print `cuda_graph captures/fallbacks/fallback_report` + `--trace` capture-reject reasons. +- **GQA capture validated** (Cohaagen): Qwen2.5-0.5B int4 GQA ON 2.14× eager, byte-identical, zero declines. Gemma-4-E2B: stale export (vision fn 2 vs 4 outputs) + GAP-3 (native pipeline unimplemented). +- **27B offload A/B** (Mary, H200): Qwen3.6-27B int4, 2 GiB → 6.2 GiB peak VRAM (2.9× vs 17.7 GiB), byte-exact. Cliff: ≤12 GiB → 0.11 tok/s. cuda_graph off under offload. +- **#553/#554 session-reuse fix** (Mary; Harry APPROVED): conv_state/recurrent_state not zeroed between generates in `NativeDecodeSession`. Not graph-capture related. +- **Native pipeline keystone** (Mary): `native_full_pipeline_parity` — native embedding + native decoder → `[0,5,6,7] == ORT` (`tiny-gemma4-vlm`). +In flight: #87 inc2 double-buffer; native paged-KV; 35B-A3B MoE; gemma-3n text-only. \ No newline at end of file diff --git a/.squad/decisions.md b/.squad/decisions.md index 88efa73bfc..c71c0f2497 100644 --- a/.squad/decisions.md +++ b/.squad/decisions.md @@ -1,6 +1,6 @@ # Decisions — live standing directives -Last consolidated: 2026-07-31T08:48:28Z (Scribe round 8 — #544 async page-in inc1, #552 observability, #554 session-reuse fix, 27B offload A/B, GQA capture; round-7+MoE in archive) +Last consolidated: 2026-07-31T10:24:07Z (Scribe round 9 — 27B decode profile Scan=56.5%, Scan-capture scoping PENDING JUSTIN, #554 session-reuse merged; round-8 in archive) Standing governance rules and constraints. Dated wave records and historical ledger updates are archived to `.squad/decisions-archive/2026-07.md`. @@ -260,14 +260,21 @@ NEVER `to_raw_bytes()`. Unblocked the gemma-3n Bool audio mask. repeatable). Do NOT re-try that lever** (reduction tax > sub-wave grid-fill); a K_SPLIT>2 new kernel with its own A/B is the future candidate. -## 2026-07-31 — Scribe consolidation (round 8) +## 2026-07-31 — 27B decode profile + Scan-capture scoping (round 9) -**By:** Scribe. Merged: #544 #552 #554. Wave narrative in archive. +**By:** Scribe. Round-8 (#544/#552/#554/27B-A/B/GQA) → archive. -- **#544 async page-in inc1** (Cohaagen; Harry test fix; Melina RC→APPROVE): `cuMemcpyHtoDAsync`+`compute_wait_fence`; eviction re-serializes (WAR). ~1% at 2 MiB; byte-identical all budgets. `ONNX_GENAI_WEIGHT_OFFLOAD_ASYNC_PAGEIN` default ON. -- **#552 profile_native observability** (Cohaagen; Lori APPROVE): `load_with_resolved_io` — genai_config decoders print `cuda_graph captures/fallbacks/fallback_report` + `--trace` capture-reject reasons. -- **GQA capture validated** (Cohaagen): Qwen2.5-0.5B int4 GQA ON 2.14× eager, byte-identical, zero declines. Gemma-4-E2B: stale export (vision fn 2 vs 4 outputs) + GAP-3 (native pipeline unimplemented). -- **27B offload A/B** (Mary, H200): Qwen3.6-27B int4, 2 GiB → 6.2 GiB peak VRAM (2.9× vs 17.7 GiB), byte-exact. Cliff: ≤12 GiB → 0.11 tok/s. cuda_graph off under offload. -- **#553/#554 session-reuse fix** (Mary; Harry APPROVED): conv_state/recurrent_state not zeroed between generates in `NativeDecodeSession`. Not graph-capture related. -- **Native pipeline keystone** (Mary): `native_full_pipeline_parity` — native embedding + native decoder → `[0,5,6,7] == ORT` (`tiny-gemma4-vlm`). +- **27B native-CUDA decode: Scan is bottleneck** (Cohaagen; profile-only): 168 ms/tok (~35× off roofline). `Scan` (48 LinearAttention blocks/step) = **56.5%**, structurally un-capturable. MatMulNBits at roofline (4.4 ms, 2%). Ceiling: **~15–30× speedup** if Scan enters capture/fuse. NOT a kernel fix. +- **#554 MERGED** (Mary; Harry APPROVED): `DecodeCudaState.rewind(0)` re-zeroes `fixed_state_binding_range`; pure-KV models unaffected (empty range). + +## ⚑ PENDING JUSTIN: 27B Scan→CUDA-capture (Mary; no code changed, awaiting go-ahead) + +Structurally larger than an increment — blockers: (1) shared prefill+decode plan; seq=1 inline corrupts prefill. (2) Control-flow declined at `provider.rs:458`, no trip_count exemption. (3) Child bodies never fold into parent plan. + +**Approach 1 — runtime dual-path** (only correct+feasible path): +- **1a** (flag-gated): inline body into parent plan alongside Scan; runtime trip_count==1 picks body. Correctness-only, no capture. +- **1b**: body enters capture; validate captures/replays rise; assert 27B tokens byte-identical to locked reference. +- Blast radius: #443/#543 core. Prefill MUST be validated (shared-plan is the correctness tripwire). + +**Baseline + locked reference tokens ready.** Awaiting Justin go-ahead. In flight: #87 inc2 double-buffer; native paged-KV; 35B-A3B MoE; gemma-3n text-only. \ No newline at end of file diff --git a/.squad/decisions/inbox/mary-session-reuse-reset.md b/.squad/decisions/inbox/mary-session-reuse-reset.md deleted file mode 100644 index 90f60cf552..0000000000 --- a/.squad/decisions/inbox/mary-session-reuse-reset.md +++ /dev/null @@ -1,119 +0,0 @@ -# Decision: native-CUDA session-reuse recurrent-state reset (LinearAttention corruption) - -- **Author:** Mary (Lead, large-model memory-offload workstream) -- **Date:** 2026-07-31T08:11:36+0000 -- **Requested by:** Justin (@justinchuby) -- **Branch:** `squad/native-session-reuse-reset` -- **References:** #384 (native-CUDA large-model decode); discovered during the - 27B weight-offload A/B (see `mary-27b-native-offload-ab.md`). Possible link to - Justin's "repeated / degenerate sentence" report on multi-turn native decode. - -## Scope (decided EARLY, per Step 0) - -**LinearAttention-only — NOT a general session-reuse bug.** - -Step-0 blast-radius test on a plain GQA transformer (KV-only, no conv/recurrent -state), `qwen2.5-0.5b-instruct-cuda`, `profile_native --runs 2`: - -- GQA: **clean / deterministic** — gen#2 byte-identical to gen#1. -- 27B hybrid LinearAttention (`conv_state`/`recurrent_state`): **corrupts** gen#2+. - -So the general KV/position/decode-step reset path is fine for everyone. The bug -is confined to models that carry fixed-size recurrent/conv state. - -## Root cause - -`DecodeCudaState` (`crates/onnx-genai-engine/src/native_decode/cuda.rs`) appends -the fixed recurrent/conv-state device bindings after the growable KV bindings. -Those fixed-state bindings are `memset_zero`'d **once, in `new()`**, and never -again. `reset()` → `rewind(0)` (called at the start of every `generate()`) only -re-zeroed `bindings[0]` (the attention mask) and reset `logical_len`. - -Growable KV is safe on reuse because it is masked and length-tracked (stale -slots are inert). Fixed recurrent state is a **wholesale, unmasked rolling -cache**: on the 2nd generation it inherited generation #1's *terminal* recurrent -state, so decode #2 started from garbage → non-deterministic degenerate output. - -This reproduces both with CUDA-graph capture ON and with `ONNX_GENAI_CUDA_GRAPH=0` -(eager), confirming it is a state-reset bug, not a capture bug. - -## Fix (general, no model-specific special-casing) - -In `cuda.rs`: -- Added `fixed_state_binding_range: Range` to `DecodeCudaState`, - populated in `new()` as `kv_end..fixed_state_end` (empty for pure-KV models). -- In `rewind()`, when `target_len == 0`, re-zero every binding in - `fixed_state_binding_range` via `native_cuda_memset_zero` (same zero-init the - constructor applies; `state_pairs` declare `init: zeros`). Only at the reset - boundary — speculative recurrent rewind to a non-zero length is intentionally - unsupported, mirroring the CPU path. - -No model names, no architecture switches: any decoder that declares fixed -recurrent/conv state pairs is reset correctly. Pure-KV decoders have an empty -range and are entirely unaffected (verified: GQA path unchanged). - -CPU native was checked and is clean here — its full reset clears the recurrent -state via the `past` map on `rewind(0)` — so the regression test is CUDA-gated. - -## Regression test (non-vacuous) - -`native_cuda_reused_session_rezeros_recurrent_state` -(`crates/onnx-genai-engine/src/native_decode/tests.rs`, `#[cfg(feature="cuda")]`, -gated by `ONNX_GENAI_RUN_CUDA_SMOKE=1`). - -Builds a synthetic recurrent decoder whose **`logits` are a direct function of -the incoming `conv_state`** (`logits = ReduceSum(Cast(conv_state))`), and whose -next state accumulates the decoded token id. Decodes a fixed 3-token sequence, -`reset()`s, decodes it again, and asserts the two logits sequences are identical. -It also asserts the per-step logits strictly grow, proving the state actually -feeds the logits (guards against a vacuous pass). - -**Non-vacuity proof** — with the re-zero disabled: - -``` -gen#1 [0.0, 60.0, 144.0] != gen#2 [252.0, 312.0, 396.0] → FAIL -``` - -With the fix, gen#2 == gen#1 → PASS. (The existing `profile_native --runs 2` -determinism check is the natural end-to-end CI guard on real models.) - -## End-to-end verification (real models, device 4/6, greedy) - -Prompt `"The capital of France is"`, `--tokens 16 --warmups 0 --runs 2`: - -| model | capture | gen#1==gen#2 | output | -|---|---|---|---| -| 27B int4 LinearAttention | ON (captures=2) | ✅ deterministic | `" Paris.\n\n\n\n\n\nThat is correct. Paris is the capital and"` | -| 27B int4 LinearAttention | OFF (`ONNX_GENAI_CUDA_GRAPH=0`) | ✅ deterministic | identical to above | -| qwen2.5-0.5b GQA (regression) | ON | ✅ deterministic | `" Paris. It is the largest city in France and the second largest in the European"` | - -Before the fix the 27B failed: `native greedy decode was not deterministic: -first=[11751,...] rerun=[279,6511,...]`. - -## Exact repro command - -```bash -source .cudaenv.sh -export ONNX_GENAI_ORT_LIB=$ORT_ROOT/lib/libonnxruntime.so.1.27.0 -export ONNX_GENAI_REQUIRE_CUDA=1 CUDA_VISIBLE_DEVICES=4 -./target/release/profile_native \ - --model <27b-int4-native-pkg> --ep cuda --backend native \ - --tokens 16 --warmups 0 --runs 2 --prompt "The capital of France is" -``` - -(27B package: `model.onnx`/`model.onnx.data`/`tokenizer.json` from -`~/mary-models/qwen3.6-27b-int4-cuda/` + `inference_metadata.yaml` carrying the -`io:` block copied from the `qwen36-ortref` sibling, as in the offload A/B.) - -Unit regression: -```bash -ONNX_GENAI_RUN_CUDA_SMOKE=1 CUDA_VISIBLE_DEVICES=6 \ - cargo test --release -p onnx-genai-engine --lib --features "cuda native-backend" \ - native_cuda_reused_session_rezeros_recurrent_state -``` - -## Guardrails honored - -- Did not touch `weight_paging.rs` / `provider.rs` (harry-6, #544 / squad/87-async-pagein). -- Did not switch the main checkout branch; all work in worktree `wt-mary-session-reset` off `origin/main` (fa1afed3). -- This is the sync page-in / native-decode session-state domain only.