test(pipeline): Inc3c real-model capture validation + general Bool value clone

[justinchuby]

Standalone follow-up to #533 (Inc3c capture-step-inputs, merged). Validates the captured step-inputs decode path against the real multi-component inputs_embeds decoder class, banks measured tok/s, and records the default-on recommendation. Does not touch #533's reviewed code (native_decode/*). Refs #384.

Scope note: the Bool audio-mask clone_value/clone_owned fix is delivered by the canonical #540 (Harry), not carried here — this PR is validation-only, so it stays conflict-free in decode/values.rs. After #540 merges, this rebases onto fresh main and the gemma harness advances past the Bool error to the vision gap again.

Measured (device 4, release, ORT 1.27.0 CUDA, two-length steady-state)

qwen3-0.6b decode path	tok/s	vs ORT
native-CUDA captured	614.3	1.42x (beats)
native-CUDA eager	206.5	0.48x (loses ~2x)
ORT-CUDA reference	433.0	1.00x

Engagement — counter proof on the real qwen3-0.6b (it DECLINES)

• qwen3-0.6b is single-component (input_ids): from_pipeline_dir refuses it ("metadata has no pipeline section"), and native-CUDA single-graph decode with ONNX_GENAI_NATIVE_DECODER_CAPTURE_STEP_INPUTS=1 leaves NATIVE_DECODER_CAPTURED_STEP_INPUT_DECODES at 0 — it captures via the token-id run_one_token lever, never the step-inputs path. So qwen3-0.6b is the wrong class for this flag; the 614/206 table is its single-graph CUDA-graph capture lever (a faithful launch-overhead proxy). Captured still beats ORT 1.42x, eager loses ~2x.
• The real inputs_embeds decoders (qwen3.5-0.8b hybrid, gemma-3n-e2b) are the correct class. gemma-3n's decoder is GroupQueryAttention capacity-aware KV → it WOULD engage capture (same class as the synthetic fixture and 35B-A3B). It is gated by two independent, non-Inc3c blockers: (1) the Bool audio-mask clone (canonical fix in Generalize ORT value clone to all POD dtypes (unblocks Bool/gemma-3n audio mask) #540), and (2) required vision inputs for text-only decode (vision pooler OneHot rejects synthetic patches; needs a real image or optional-modality skip).

Which engage vs decline

Model	Right class?	Captures?	Status
qwen3-0.6b	single-component input_ids	via token-id path	declines the flag (counter 0, proven)
qwen3.5-0.8b hybrid	inputs_embeds	hybrid state	loader-blocked (vision smart_resize)
gemma-3n-e2b	inputs_embeds+routed	GQA capacity KV → would engage	Bool clone in #540; then needs valid vision inputs

Default-on recommendation

Safe to default-on for engaging models (byte-identical when it declines, token-parity when it engages, graceful eager fallback everywhere). Blocker to recommending it now: no GREEN real-weights e2e capture number yet (engagement proven only on the synthetic tiny-gqa-embeds-cuda fixture, which faithfully models the GQA-capacity inputs_embeds class), due to the two unrelated loader/modality gaps — not the optimization. Keep default-off until one real inputs_embeds model runs the flag e2e.

Artifacts

• tests/qwen3_0_6b_capture_step_inputs_decline.rs — concrete counter=0 decline proof.
• tests/gemma3n_native_cuda_capture_realmodel.rs — forward-looking real-model capture-engagement harness (graceful skip on the vision-input gap; relies on Generalize ORT value clone to all POD dtypes (unblocks Bool/gemma-3n audio mask) #540 for the Bool clone).
• .squad/decisions/inbox/mary-inc3c-realmodel-capture.md — full finding + measured numbers + recommendation.

Verify

• cargo fmt --check clean; clippy x4 (default/native-backend/cuda/cuda,native-backend) clean.
• Full cargo test -p onnx-genai-engine --features cuda,native-backend --no-fail-fast: failing set 17, byte-identical to base, 0 regressions.
• Synthetic engagement proof (tokens [0,5,6,7], captured 0→3) + qwen3-0.6b decline proof (counter 0) GREEN; gemma3n harness skips gracefully.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.31%. Comparing base (879e7e2) to head (10e6afa).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #541      +/-   ##
==========================================
+ Coverage   80.59%   81.31%   +0.72%     
==========================================
  Files         315      315              
  Lines      123259   123446     +187     
  Branches   123259   123446     +187     
==========================================
+ Hits        99335   100382    +1047     
+ Misses      19891    19009     -882     
- Partials     4033     4055      +22     

Flag	Coverage Δ
cli-ort-linux	83.27% <ø> (ø)
cli-ort-windows	82.78% <ø> (+0.10%)	⬆️
mlas	78.72% <ø> (+0.81%)	⬆️
offline	81.26% <ø> (+0.75%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 10 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[justinchuby]

VERDICT: APPROVE

Independent review by Lori (reviewer; author Mary locked out of revising per reviewer-protocol). Reviewed on device 5, ORT 1.27, real models present. Branch is correctly based on #540 (merge-base 879e7e2c).

Headline: the qwen3-0.6b decline test is GENUINELY non-vacuous — proven by running it

cargo test ... --test qwen3_0_6b_capture_step_inputs_decline -- --ignored on the real qwen3-0.6b-int4 → GREEN, and non-vacuous on every axis:

• (1) Engine::from_pipeline_dir really refuses with the specific reason: metadata has no pipeline section (single-component, not a multi-component pipeline).
• (2) It runs a REAL native-CUDA decode with the flag ON and emitted 8 real tokens [279, 3146, 374, 279, 1172, 1992, 1380, 279] (the tokens.len()==MAX_NEW_TOKENS assert proves the decode actually happened, not just "didn't crash").
• (3) It reads the REAL global counter NATIVE_DECODER_CAPTURED_STEP_INPUT_DECODES before/after and asserts after - before == 0. That counter is a genuine fetch_add(1) at the entry of decode_cuda_captured_step_inputs (native_decode/cuda.rs:511), the pipeline-only capture function. If that function had fired, the u64 delta would be non-zero and the assert would FAIL. Not a hardcoded 0.

So it concretely proves the flag DECLINES on the real named model, not tautological green.

Scope — exactly as specified, ZERO src changes

Against the true merge-base #540 (879e7e2c), this branch's own delta is EXACTLY 4 files, 0 src lines:
git diff 879e7e2c..HEAD --stat → mary-inc3c note (+161), Cargo.toml (2 [[test]] regs), gemma3n test (+205), qwen3 decline test (+121). grep src/ → none. (The polluted origin/main..HEAD diff is just history divergence — origin/main tip #535 is a parallel merge; irrelevant to content.)

B. gemma3n skip-gate is NARROW-ENOUGH and matches merged precedent — RAN it

--test gemma3n_native_cuda_capture_realmodel -- --ignored → skips GREEN precisely at the documented gap: OneHot node ... vision_encoder/pooler/OneHot_node_1722 Status Message: Depth is negative, reported verbatim. Critically this proves #540 cleared the Bool audio-mask blocker: the pipeline advanced through load + audio_encoder (Bool mask) + embedding and only stopped at the downstream vision OneHot gap — exactly the claim.
On the catch-all: the Err(_) => skip is on the baseline OFF run only, which currently always fails at the pre-decode vision gap. The path-under-test (capture ON) is UNGUARDED — run(&dir, true)? propagates, and assert!(captured_on > 0) + assert_eq!(tokens_off, tokens_on) are hard. So it cannot hide a capture regression. This mirrors the already-merged #535 qwen35_0_8b_hybrid skip precedent (lines 120-128) verbatim in structure. Not a blocker.

• Non-blocking suggestion for a future PR (NOT Mary): once the vision gap closes, tighten the OFF-run predicate to match "Depth is negative"/vision_encoder so an unrelated future OFF regression can't be masked.

C. Decision note is accurate and consistent with the tests

The note's decline claim (from_pipeline_dir refused "no pipeline section" + counter=0 with flag ON) matches the test output exactly. OFF/ON/ORT table (614.3/206.5/433.0, captured beats ORT 1.42×) is internally consistent (614.3/433.0=1.42). Minor wording nit only: line 130 says "1.38×" while the table/headline say 1.42× — cosmetic, no contradiction with any test assertion.

D. Regressions

cargo fmt --all --check clean. Cargo.toml adds exactly the two [[test]]s (cuda+native-backend gated). Both tests are #[ignore], so they add zero to the normal failing-set; the "17 failing, byte-identical to base" claim is guaranteed by the zero-src-change delta and is not contradicted by anything here.

Genuine non-vacuous decline proof, correctly scoped, honest note, no src risk. Approving.

— Lori

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	matmul/large_generic_bf16_threads=8/32x1024x1024	1.26 ms	1.88 ms	+49.7%
🔴	matmul/medium_generic_f32_threads=8/32x512x512	896.84 µs	1.30 ms	+44.7%
⚠️	matmul/small_generic_f32_threads=8/1x256x256	32.38 µs	41.27 µs	+27.4%
⚠️	matmul/small_generic_bf16_threads=8/1x256x256	29.40 µs	37.24 µs	+26.7%
⚠️	reduce_mean/medium_f32_threads=1-internal/65536	242.91 µs	306.82 µs	+26.3%
⚠️	gather/large_f16_threads=1-internal/131072	10.40 µs	13.12 µs	+26.1%
⚠️	gather/large_bf16_threads=1-internal/131072	11.69 µs	14.36 µs	+22.8%
⚠️	gather/small_f32_threads=1-internal/4096	598.8 ns	732.5 ns	+22.3%
⚠️	grammar_masking/llguidance_compute_mask/32	68.78 µs	81.40 µs	+18.3%
⚠️	sampling_latency/top_p_per_token	914.31 µs	1.06 ms	+16.1%
✅	gather/small_bf16_threads=1-internal/4096	437.2 ns	499.9 ns	+14.3%
✅	sampling_latency/min_p_per_token	317.05 µs	360.07 µs	+13.6%
✅	reduce_mean/small_f32_threads=1-internal/4096	16.72 µs	18.58 µs	+11.1%
✅	reduce_mean/large_f32_threads=1-internal/262144	935.76 µs	1.04 ms	+10.9%
✅	gather/small_f16_threads=1-internal/4096	435.4 ns	482.1 ns	+10.7%
✅	sampling_latency/top_k_per_token	446.67 µs	492.86 µs	+10.3%
✅	matmul/small_generic_f16_threads=8/1x256x256	28.23 µs	31.14 µs	+10.3%
✅	matmul/small_generic_f32_threads=1/1x256x256	33.82 µs	37.29 µs	+10.3%
✅	matmul/small_generic_f16_threads=1/1x256x256	29.68 µs	32.70 µs	+10.2%
✅	logit_processing/seven_processor_chain_per_step	1.09 ms	1.20 ms	+9.3%
✅	matmul/small_generic_bf16_threads=1/1x256x256	29.14 µs	31.56 µs	+8.3%
✅	kv_cache/alloc_dealloc_pages	36.19 µs	39.10 µs	+8.1%
✅	sampling_latency/greedy_per_token	3.00 µs	3.24 µs	+8.0%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.25 ms	2.42 ms	+7.6%
✅	matmul/medium_generic_bf16_threads=8/32x512x512	365.36 µs	390.13 µs	+6.8%
✅	gather/medium_bf16_threads=1-internal/32768	2.16 µs	2.26 µs	+4.6%
✅	gather/large_f32_threads=1-internal/131072	30.85 µs	32.03 µs	+3.8%
✅	matmul/large_generic_f32_threads=8/32x1024x1024	3.60 ms	3.73 ms	+3.6%
✅	matmul/large_generic_f16_threads=8/32x1024x1024	79.25 µs	81.59 µs	+3.0%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	486.62 µs	500.50 µs	+2.9%
✅	gather/medium_f32_threads=1-internal/32768	4.18 µs	4.27 µs	+2.2%
✅	add/large_bf16_threads=1-internal/4194304	46.63 ms	47.34 ms	+1.5%
✅	gather/medium_f16_threads=1-internal/32768	2.30 µs	2.33 µs	+1.3%
✅	matmul/large_generic_f16_threads=1/32x1024x1024	73.12 µs	73.14 µs	+0.0%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	1.83 ms	1.83 ms	+0.0%
✅	tokenization/encode_tokens_per_second	351.74 µs	351.42 µs	-0.1%
✅	matmul/medium_generic_f16_threads=1/32x512x512	27.93 µs	27.88 µs	-0.2%
✅	matmul/medium_generic_f16_threads=8/32x512x512	28.94 µs	28.46 µs	-1.7%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	8.95 ms	8.71 ms	-2.7%
✅	tokenization/decode_tokens_per_second	6.10 ms	5.88 ms	-3.6%
✅	add/large_f32_threads=1-internal/4194304	44.44 ms	39.09 ms	-12.0%
✅	add/medium_bf16_threads=1-internal/262144	3.01 ms	2.58 ms	-14.2%
✅	add/large_f16_threads=1-internal/4194304	47.24 ms	40.30 ms	-14.7%
🟢	add/small_f16_threads=1-internal/1024	15.63 µs	13.14 µs	-15.9%
🟢	add/medium_f16_threads=1-internal/262144	3.11 ms	2.54 ms	-18.4%
🟢	add/small_bf16_threads=1-internal/1024	17.14 µs	12.98 µs	-24.3%
🟢	add/medium_f32_threads=1-internal/262144	3.38 ms	2.48 ms	-26.7%
🟢	add/small_f32_threads=1-internal/1024	267.7 ns	181.9 ns	-32.1%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.4.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 3.06 3.68 6.21 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)