[justinchuby]

The decode loop was already shared. Prefix reuse was not.

What was wrong

One run_decode_loop, one PipelineDecodeLoopBackend, both backends behind &mut dyn PipelineDecoderComponent — the loop is properly unified. But #1152 added a second prefix-reuse implementation inline on the native path, alongside reusable_prefix_len used by ORT.

Two copies of one policy, already drifted in opposite directions:

guard	native copy	ORT copy
use_kv gate	missing	present
clamp shared against the KV's real length	present	missing

Each was missing a guard the other had. Neither omission can surface as a crash: a prefix-reuse bug produces fluent text conditioned on the wrong context, so no test fails and no benchmark moves.

What this does

• pipeline::prefix_reuse::plan_prefix_reuse — the decision, as a pure function.
• apply_prefix_reuse over a KvPrefixStore seam — the application.
• Only the mechanism differs by backend now: a native decoder rewinds its session-resident cache (rewind_kv), ORT slices pipeline-held past tensors (truncate_past). Both adapters live in flat_autoregressive.rs and are ~20 lines each.

Both backends now get both guards.

A latent hazard this closes

Sharing the application means a declined rewind now always resets. Reporting "reuse nothing" while leaving the previous turn's tokens in the cache would prefill the new prompt on top of them. The native path had exactly that shape — unreachable today only because NativePipelineDecoder::rewind_kv never returns false, which is a property of the current sole implementor, not of the contract.

Validation

• cargo test -p onnx-genai-engine --features native-backend --lib — 507 passed, 0 failed, 1 ignored. The pre-existing ORT rewind-semantics tests (failed_rewind_of_ort_owned_kv_leaves_session_unchanged, failed_rewind_of_runner_backed_state_leaves_session_unchanged, failed_rewind_of_windowed_evicted_position_leaves_session_unchanged) still pass — that is the no-regression evidence for the ORT path.
• cargo clippy -p onnx-genai-engine --features native-backend --all-targets -- -D warnings — clean.
• The policy is now testable without a model or a device: 11 unit tests where there were none.

Falsified, not just written. Both new guards were deliberately broken and confirmed to redden:

• removing shared.min(current) → the_cache_length_bounds_the_reuse_when_the_context_claims_more and an_empty_cache_reuses_nothing_however_much_the_prompt_shares FAIL
• removing the reset-on-decline → a_declined_rewind_empties_the_cache_rather_than_leaving_its_tail FAILS

Performance

No behaviour change on either backend for any input the two copies already agreed on. Same work per turn; one added trait dispatch per request (not per token), which is unmeasurable against a prefill.

[justinchuby]

Self-review

Reviewed my own diff against the one failure this change could plausibly introduce, rather than re-describing what it does.

The risk: an ordering dependency. The ORT adapter's reset() sets decoder_state = None, and this now happens in strictly more cases than before (previously only the non-linear-positions and truncate-decline branches dropped the state; now every Reset verdict does). Line 314 does expect("autoregressive pipeline has decode state"). If any path could reach 314 with None, this refactor turns a working request into a panic.

Traced it:

• The ORT adapter runs at line 212, and only in the else arm — i.e. only when !paged_enabled.
• The rebuild is at line 303, guarded by reused.is_empty() && paged_session.is_none().
• paged_session is assigned only inside if paged_enabled && !positions_routed (line 265). When the ORT adapter ran, paged_enabled is false, so paged_session is necessarily None.
• Therefore every path that can set None satisfies the rebuild guard and hits 303 before 314.
• Conversely, when reuse succeeds (Keep/Rewind), reset() never ran, so the state is still Some.

No path reaches the expect with None.

Behaviour deltas I accepted, deliberately:

1. The native path gains a use_kv gate. No-op today — NativePipelineDecoder::use_kv() returns true unconditionally — but correct for any future decoder that carries no KV.
2. The ORT path gains shared.min(current). No-op for a correctly-maintained retained context, which is the point: it only bites when the context and the cache have already disagreed, and it is that disagreement, not the clamp, that would be the bug.
3. NativeKvStore::reset() errors if the decoder declines to rewind to empty, where the old code silently reported zero reuse. Unreachable today (rewind_kv always returns Ok(true)). I chose loud over silent: the alternative is prefilling a new prompt on top of a previous turn's KV, which produces fluent, wrong text and no signal.

Verified: cargo test -p onnx-genai-engine --features native-backend --lib — 508 passed / 0 failed / 1 ignored. cargo clippy --all-targets -- -D warnings clean. The three pre-existing ORT rewind-semantics tests still pass, which is the no-regression evidence for that path.

Not verified, stated plainly: this box has no CUDA runtime, so the native adapter was never exercised against a real device-resident KV — only against the trait. And the performance claim is an argument, not a measurement: the work per turn is unchanged and the added cost is one trait dispatch per request, which is not resolvable against a prefill. I did not benchmark it and am not claiming I did.

The mechanisms (second commit)

Extracting the policy stops the two copies that exist; it does not stop the third. Two mechanisms now do, and both were falsified rather than merely written:

• Compile time — ReusedPrefix has a private field and its only constructors live in prefix_reuse. Probe: constructing one in paged_decode.rs gives error[E0603]: tuple struct constructor `ReusedPrefix` is private. The inline computation added in feat(cuda): retain native KV across pipeline turns #1152 would not compile today.
• Test time — the_kv_rewind_primitives_are_only_driven_by_the_shared_policy scans the crate source and fails if .rewind_kv( / .truncate_past( / .current_kv_len( is called outside the KvPrefixStore adapters. Probe: a call in paged_decode.rs reddens it with pipeline\paged_decode.rs:311. It runs under cargo test, so it cannot be missed by not looking at CI.

The allowlist is three entries and the failure message tells you to route the new backend through apply_prefix_reuse rather than widen it. Widening it is exactly how the duplication this module exists to prevent gets back in.

Merging.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.54%. Comparing base (73deffa) to head (f317ffe).
⚠️ Report is 17 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1170   +/-   ##
=======================================
  Coverage   80.53%   80.54%           
=======================================
  Files         357      357           
  Lines      154203   154282   +79     
  Branches   154203   154282   +79     
=======================================
+ Hits       124183   124262   +79     
+ Misses      25476    25474    -2     
- Partials     4544     4546    +2     

Flag	Coverage Δ
offline	80.54% <ø> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 8 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

✅ Benchmarks — No Regression

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
✅	kv_cache/alloc_dealloc_pages	39.66 µs	43.36 µs	+9.3%
✅	qwen3_sampling_processors/top_k_partial_selection	141.48 µs	151.26 µs	+6.9%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	5.88 ms	6.28 ms	+6.7%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	525.32 µs	558.16 µs	+6.3%
✅	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.63 ms	3.79 ms	+4.4%
✅	qwen3_sampling_processors/top_k_top_p_fast	701.18 µs	729.62 µs	+4.1%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	481.79 µs	499.08 µs	+3.6%
✅	matmul/medium_generic_bf16_threads=8/32x512x512	366.45 µs	371.53 µs	+1.4%
✅	matmul/small_generic_f16_threads=8/1x256x256	28.56 µs	28.94 µs	+1.3%
✅	matmul/medium_generic_f32_threads=8/32x512x512	891.80 µs	901.82 µs	+1.1%
✅	grammar_masking/llguidance_compute_mask/32	82.26 µs	82.96 µs	+0.9%
✅	add/medium_f32_threads=1-internal/262144	23.54 µs	23.61 µs	+0.3%
✅	matmul/small_generic_f16_threads=1/1x256x256	27.88 µs	27.95 µs	+0.2%
✅	matmul/small_generic_bf16_threads=8/1x256x256	30.44 µs	30.45 µs	+0.0%
✅	matmul/small_generic_f32_threads=1/1x256x256	34.03 µs	34.02 µs	-0.0%
✅	matmul/medium_generic_f16_threads=1/32x512x512	28.66 µs	28.64 µs	-0.1%
✅	matmul/medium_generic_f16_threads=8/32x512x512	27.97 µs	27.92 µs	-0.2%
✅	matmul/small_generic_bf16_threads=1/1x256x256	28.76 µs	28.69 µs	-0.2%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	8.81 ms	8.76 ms	-0.6%
✅	matmul/large_generic_f32_threads=8/32x1024x1024	3.72 ms	3.69 ms	-0.8%
✅	reduce_mean/medium_f32_threads=1-internal/65536	242.95 µs	238.73 µs	-1.7%
✅	gather/small_f32_threads=1-internal/4096	643.9 ns	620.6 ns	-3.6%
✅	add/small_bf16_threads=1-internal/1024	432.1 ns	412.0 ns	-4.6%
✅	matmul/small_generic_f32_threads=8/1x256x256	36.07 µs	34.02 µs	-5.7%
✅	logit_processing/seven_processor_chain_per_step	351.12 µs	329.62 µs	-6.1%
✅	add/small_f32_threads=1-internal/1024	207.5 ns	194.2 ns	-6.4%
✅	reduce_mean/large_f32_threads=1-internal/262144	969.05 µs	907.01 µs	-6.4%
✅	sampling_latency/min_p_per_token	217.03 µs	201.52 µs	-7.1%
✅	gather/small_bf16_threads=1-internal/4096	473.0 ns	437.9 ns	-7.4%
✅	gather/medium_bf16_threads=1-internal/32768	2.58 µs	2.37 µs	-8.0%
✅	gather/small_f16_threads=1-internal/4096	475.0 ns	435.8 ns	-8.2%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	2.16 ms	1.98 ms	-8.6%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.40 ms	2.18 ms	-9.0%
✅	add/medium_f16_threads=1-internal/262144	107.78 µs	98.01 µs	-9.1%
✅	tokenization/encode_tokens_per_second	397.03 µs	357.54 µs	-9.9%
✅	add/large_bf16_threads=1-internal/4194304	1.72 ms	1.54 ms	-10.5%
✅	sampling_latency/top_k_per_token	56.17 µs	50.18 µs	-10.7%
✅	matmul/large_generic_bf16_threads=8/32x1024x1024	1.54 ms	1.37 ms	-10.9%
✅	sampling_latency/greedy_per_token	3.37 µs	2.99 µs	-11.3%
✅	matmul/large_generic_f16_threads=1/32x1024x1024	83.83 µs	73.58 µs	-12.2%
✅	matmul/large_generic_f16_threads=8/32x1024x1024	90.37 µs	78.96 µs	-12.6%
✅	add/large_f16_threads=1-internal/4194304	1.75 ms	1.52 ms	-13.3%
✅	reduce_mean/small_f32_threads=1-internal/4096	16.08 µs	13.85 µs	-13.9%
✅	add/medium_bf16_threads=1-internal/262144	109.05 µs	93.75 µs	-14.0%
✅	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	183.03 µs	156.46 µs	-14.5%
✅	tokenization/decode_tokens_per_second	6.63 ms	5.64 ms	-14.9%
🟢	sampling_latency/top_p_per_token	425.95 µs	359.12 µs	-15.7%
🟢	add/small_f16_threads=1-internal/1024	491.4 ns	414.1 ns	-15.7%
🟢	gather/medium_f16_threads=1-internal/32768	2.72 µs	2.29 µs	-15.8%
🟢	gather/large_bf16_threads=1-internal/131072	13.16 µs	10.76 µs	-18.2%
🟢	matmul/large_generic_bf16_threads=1/32x1024x1024	2.32 ms	1.81 ms	-22.0%
🟢	gather/large_f32_threads=1-internal/131072	34.72 µs	24.69 µs	-28.9%
🟢	add/large_f32_threads=1-internal/4194304	772.27 µs	539.97 µs	-30.1%
🟢	gather/medium_f32_threads=1-internal/32768	5.09 µs	3.44 µs	-32.3%
🟢	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	676.46 µs	434.29 µs	-35.8%
🟢	gather/large_f16_threads=1-internal/131072	18.40 µs	9.87 µs	-46.3%
🟢	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	1.15 ms	530.33 µs	-53.8%
🟢	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	100.26 µs	42.18 µs	-57.9%
🟢	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	135.10 µs	40.89 µs	-69.7%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 3.22 3.47 5.88 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)