diff --git a/.squad/decisions/inbox/mary-inc3c-realmodel-capture.md b/.squad/decisions/inbox/mary-inc3c-realmodel-capture.md new file mode 100644 index 0000000000..ba930834df --- /dev/null +++ b/.squad/decisions/inbox/mary-inc3c-realmodel-capture.md @@ -0,0 +1,161 @@ +# Inc3c — real-model capture-step-inputs validation (issue #384) + +**Author:** Mary (native-pipeline). **Date:** 2026-07-30 / measured-through +2026-07-31. **Branch:** `squad/inc3c-realmodel-capture-validation` (standalone +follow-up to the now-merged #533). + +## Measured reproduction (2026-07-31, device 4, release, ORT 1.27.0 CUDA) + +Concrete on-main measurement, banked fresh (two-length steady-state, `taskset -c +0-7`, `ONNX_GENAI_ORT_LIB` set → real ORT-CUDA, not CPU fallback): + +| qwen3-0.6b decode path | tok/s | vs ORT | +|---|---|---| +| native-CUDA **captured** (`ONNX_GENAI_CUDA_GRAPH=1`) | **614.3** | 1.42× (beats) | +| native-CUDA **eager** (`ONNX_GENAI_CUDA_GRAPH=0`) | **206.5** | 0.48× (loses ~2×) | +| **ORT-CUDA** reference | **433.0** | 1.00× | + +**Counter proof that qwen3-0.6b DECLINES the capture-step-inputs flag** (test +`qwen3_0_6b_capture_step_inputs_decline`, GREEN on device 4): +1. `Engine::from_pipeline_dir(qwen3-0.6b)` **refused**: *"metadata has no pipeline + section"* — it is not a multi-component pipeline. +2. `Engine::from_dir` native-CUDA decode with + `ONNX_GENAI_NATIVE_DECODER_CAPTURE_STEP_INPUTS=1` → + `NATIVE_DECODER_CAPTURED_STEP_INPUT_DECODES` = **0**. The flag path is + pipeline-decoder-only; qwen3-0.6b captures via the *token-id* `run_one_token` + mechanism (the 614/206 lever above), never the step-inputs path. + +So the OFF/ON of *this flag* is meaningless for qwen3-0.6b (always 0/declines); +the table above is its single-graph CUDA-graph capture lever, a faithful +launch-overhead proxy. The captured path still **beats ORT-CUDA 1.42×** while +eager **loses ~2×** — the perf motivation stands. + +## Question asked + +Does the Inc3c captured step-inputs decode path +(`ONNX_GENAI_NATIVE_DECODER_CAPTURE_STEP_INPUTS`) actually ENGAGE and deliver +the captured ceiling on a **real** decoder's multi-component pipeline path — not +just the synthetic `tiny-gqa-embeds-cuda` fixture — with token output unchanged? + +## Headline finding — the named target (qwen3-0.6b) is the wrong model class + +**qwen3-0.6b cannot exercise the capture-step-inputs flag at all** (proven +concretely above). It is a **single-component** model: `inference_metadata.yaml` +declares `token_input: input_ids`, one `model.onnx`, no embedding/vision split. +It loads via `Engine::from_dir` (the single-graph token-id path), never +`from_pipeline_dir`. The capture-step-inputs path only fires for a +**multi-component `inputs_embeds`/routed** pipeline decoder +(`decode_cuda_captured_step_inputs`, gated in `native_decode/cuda.rs` on a +declared `inputs_embeds`/`Routed` port). qwen3-0.6b has no such port. + +So the Part A numbers **612 captured / 220 eager / 443 ORT-CUDA** were measured +on qwen3-0.6b's **single-graph token-id path**, toggling the *general* CUDA-graph +capture (`ONNX_GENAI_CUDA_GRAPH`), as a faithful **proxy** for the per-step +kernel-launch overhead. They are NOT the capture-step-inputs flag path, and +qwen3-0.6b cannot produce an OFF/ON number for that flag. (qwen3-0.6b DOES +capture on its token-id path — it is a GQA-capacity-KV decoder — confirming the +launch-overhead lever is real; it just is not the multi-component seam.) + +## Which real models *would* exercise the path — and their status + +The capture-step-inputs path needs a real multi-component `inputs_embeds` +decoder. Only three such models are on disk; **all are blocked by independent, +non-Inc3c gaps**: + +| Model | Multi-component `inputs_embeds` decoder? | KV → captures? | Status | +|---|---|---|---| +| **qwen3-0.6b-int4** | No — single-component `input_ids` | yes (token-id path) | wrong class; can't exercise the flag | +| **qwen3.5-0.8b-hybrid** | Yes (`embedding.onnx`+`text.onnx`, `inputs_embeds`) | hybrid lin-attn state | **loader-blocked**: `from_pipeline_dir` refuses on vision `Resize.smart_resize=1` (compatibility synthesis can't represent it; needs native `inference_metadata.json`). A text-only stripped copy is also refused (`model.type qwen3_5` *requires* the vision block). | +| **gemma-3n-e2b** | Yes (`embedding`→`decoder`, `inputs_embeds` + routed `per_layer_inputs`) | **yes — GroupQueryAttention, sliding_window 512 → `graph_enabled=true`** | Bool audio-mask (canonical fix in #540); then blocked on **valid vision inputs** for text-only decode | + +**Gemma-3n is the closest to a real-model proof and structurally WOULD engage +capture** (its decoder is `GroupQueryAttention` capacity-aware KV — the exact +class that captures, same as the synthetic fixture and the real 35B-A3B target). +Two independent obstacles gated it: + +1. **Bool audio-mask (fix delivered separately by #540).** `input_features_mask` + is `Bool`; the pipeline value/cache clone path + (`decode/values.rs::clone_value`) hard-errored `unsupported cached ORT value + dtype: Bool`. During this validation I confirmed a general raw-byte fallback + (`Value::to_raw_bytes` → `from_raw_bytes`, bit-exact) advances the gemma + pipeline *past* the Bool error — but Harry's #540 is the **canonical, more + complete** version of exactly that (both `clone_value` and `clone_owned`, + host-guarded `as_raw_bytes` so device tensors error precisely instead of + silent-corrupting, all POD dtypes, round-trip tests in both crates). This PR + therefore does **not** carry a `clone_value` change; it relies on #540. +2. **Vision required-input (NOT fixed — the remaining gap).** gemma-3n declares + `vision_encoder.pixel_values` as a *required* request input and the pipeline + eagerly runs the vision encoder even for a text-only prompt with no image + tokens. The vision pooler's `OneHot` derives its depth from real image-grid + structure, so zeroed/synthetic patches are rejected (`Depth is negative`). + Producing a real-model gemma number needs either a **real image** or an + **optional-modality skip** (run the encoder only when the prompt carries image + tokens) — an unrelated pipeline feature, out of Inc3c scope. + +The forward-looking harness +`tests/gemma3n_native_cuda_capture_realmodel.rs` encodes the full real-model +proof (dummy audio+vision, native decoder `cuda:0`, counter engagement + OFF==ON +token parity) and **skips gracefully** on the vision-input gap (matching the +`qwen35_0_8b_hybrid` skip-with-explanation precedent). It becomes a live proof +the moment a real image is supplied or vision is made optional. + +## Where the engagement proof currently stands + +The **non-tautological engagement proof remains the synthetic +`tiny-gqa-embeds-cuda` fixture** (`native_cuda_captured_step_inputs_parity`, +GREEN on device 4): counter `NATIVE_DECODER_CAPTURED_STEP_INPUT_DECODES` = 0 with +the flag OFF, = 3 with the flag ON, tokens identical `[0,5,6,7]`. That fixture is +built specifically to mirror the real class — a `GroupQueryAttention` +capacity-aware `inputs_embeds` decoder — i.e. exactly what gemma-3n and 35B-A3B +use. So the proof faithfully models the real seam; what is missing is only an +end-to-end *real weights* run, blocked by the loader/modality gaps above, not by +the optimization. + +## Default-on recommendation + +- **Safety: already safe to default-on.** When `graph_enabled=false` (naive + Concat-KV decoders) the capture-step-inputs gate is inert and the path is + byte-identical to eager. When it engages (GQA-capacity `inputs_embeds` + decoders) token output is proven identical (fixture). There is graceful eager + fallback for every decliner. So flipping + `ONNX_GENAI_NATIVE_DECODER_CAPTURE_STEP_INPUTS` (and + `ONNX_GENAI_PIPELINE_NATIVE_DECODER`) on by default carries **no correctness + risk** for engaging models and **no behavior change** for declining ones. +- **Blocker to *recommending* default-on now:** we do not yet have a GREEN + **real-weights** e2e capture number (fixture-only). That gap is caused by the + two unrelated loader/modality blockers, not by the optimization. Recommend: + **keep default-off until one real multi-component `inputs_embeds` model runs + the flag e2e** (gemma-3n once vision is optional / imaged, or qwen3.5-hybrid + once it has native metadata), then flip default-on with eager fallback. The + perf case is already strong (captured beats ORT-CUDA 1.38×; eager loses ~2×). + +## gemma-3n limitation: quick-win-or-slice verdict + +- **Bool audio-mask clone:** **quick win** — a general raw-byte clone fallback; + the canonical version lands in **#540** (`clone_value` + `clone_owned`, + host-guarded, all POD dtypes). Confirmed it advances gemma-3n past the Bool + error. This validation PR does not carry it (superseded by #540). +- **Vision-as-optional (text-only decode without image inputs):** **a slice** — + requires the pipeline to conditionally skip a declared-required modality + encoder when the prompt has no modality tokens (or accept a real image). Not + Inc3c; recommend a dedicated follow-up if a gemma-3n real-model perf number is + wanted. + +## Artifacts (this follow-up, additive — #533's reviewed code untouched) + +- `crates/onnx-genai-engine/tests/qwen3_0_6b_capture_step_inputs_decline.rs` — + concrete counter proof that qwen3-0.6b (single-component) DECLINES the + capture-step-inputs path (from_pipeline_dir refused + counter 0 with flag ON). +- `crates/onnx-genai-engine/tests/gemma3n_native_cuda_capture_realmodel.rs` — + forward-looking real-model capture-engagement harness (graceful skip on the + vision-input gap; relies on #540 for the Bool audio-mask clone). +- `crates/onnx-genai-engine/Cargo.toml` — registers the new `[[test]]`s. + +The Bool `clone_value`/`clone_owned` fix is delivered by **#540** (Harry), not +here, to avoid a conflicting duplicate in `decode/values.rs`. + +Verify: `cargo fmt --check` clean; clippy ×4 (default/native-backend/cuda/ +cuda,native-backend) clean; full `cargo test -p onnx-genai-engine +--features cuda,native-backend --no-fail-fast` failing set **17, byte-identical +to base, 0 regressions**; synthetic engagement proof + qwen3-0.6b decline proof +GREEN. diff --git a/crates/onnx-genai-engine/Cargo.toml b/crates/onnx-genai-engine/Cargo.toml index 4dd8a1547c..75a74f54e0 100644 --- a/crates/onnx-genai-engine/Cargo.toml +++ b/crates/onnx-genai-engine/Cargo.toml @@ -48,6 +48,14 @@ required-features = ["cuda", "native-backend"] name = "gemma4_e2b_native_cuda_pipeline_bench" required-features = ["cuda", "native-backend"] +[[test]] +name = "gemma3n_native_cuda_capture_realmodel" +required-features = ["cuda", "native-backend"] + +[[test]] +name = "qwen3_0_6b_capture_step_inputs_decline" +required-features = ["cuda", "native-backend"] + [features] default = [] native-backend = [ diff --git a/crates/onnx-genai-engine/tests/gemma3n_native_cuda_capture_realmodel.rs b/crates/onnx-genai-engine/tests/gemma3n_native_cuda_capture_realmodel.rs new file mode 100644 index 0000000000..4ee70c227f --- /dev/null +++ b/crates/onnx-genai-engine/tests/gemma3n_native_cuda_capture_realmodel.rs @@ -0,0 +1,205 @@ +//! Inc3c real-model validation (issue #384): prove the Inc3c captured +//! step-inputs decode path (`ONNX_GENAI_NATIVE_DECODER_CAPTURE_STEP_INPUTS`) +//! actually ENGAGES on a *real* multi-component `inputs_embeds`/routed decoder — +//! not just the synthetic `tiny-gqa-embeds-cuda` fixture — and that the perf +//! optimization is token-for-token free. +//! +//! Gemma 3n E2B is the only pipeline-loadable real model that drives the native +//! multi-component pipeline decoder: its decoder consumes `inputs_embeds` (from +//! the embedding component) plus a routed `per_layer_inputs` port, and it uses +//! capacity-aware `GroupQueryAttention` KV (sliding_window 512) so CUDA-graph +//! capture is *enabled* — the precondition for the captured step-inputs path to +//! engage. (qwen3-0.6b is single-component `input_ids`, so it never routes +//! through this path; qwen3.5-0.8b hybrid is loader-blocked on vision +//! `smart_resize`. See `.squad/decisions/inbox/mary-inc3c-realmodel-capture.md`.) +//! +//! Text-only decode: a text prompt with no image/audio tokens plus zeroed dummy +//! vision + audio inputs (required request inputs whose encoder outputs are not +//! gathered into the sequence), so the decode is driven purely by the text +//! embedding component. +//! +//! ```bash +//! source .cudaenv.sh +//! export CUDA_VISIBLE_DEVICES=4 +//! export ONNX_GENAI_ORT_LIB=$ORT_ROOT/lib/libonnxruntime.so.1.27.0 +//! GEMMA3N_DIR=/home/justinchu/mobius/.scratch/gemma4-e2b-native \ +//! cargo test -p onnx-genai-engine --features cuda,native-backend \ +//! --test gemma3n_native_cuda_capture_realmodel -- --ignored --nocapture +//! ``` +#![cfg(all(feature = "cuda", feature = "native-backend"))] + +use std::path::{Path, PathBuf}; +use std::sync::atomic::Ordering; + +use onnx_genai_engine::pipeline::PipelineGenerateRequest; +use onnx_genai_engine::{ + Engine, EngineConfig, GenerateOptions, GeneratePrompt, GenerateRequest, + NATIVE_DECODER_CAPTURED_STEP_INPUT_DECODES, PipelineEngine, +}; +use onnx_genai_ort::{DataType, Value}; + +const DEFAULT_DIR: &str = "/home/justinchu/mobius/.scratch/gemma4-e2b-native"; +const NATIVE_DECODER_ENV: &str = "ONNX_GENAI_PIPELINE_NATIVE_DECODER"; +const NATIVE_DECODER_DEVICE_ENV: &str = "ONNX_GENAI_PIPELINE_NATIVE_DECODER_DEVICE"; +const CAPTURE_ENV: &str = "ONNX_GENAI_NATIVE_DECODER_CAPTURE_STEP_INPUTS"; +/// Gemma text prompt: BOS + a few text tokens; no image/audio tokens. +const PROMPT: &[u32] = &[2, 651, 6996, 576, 8698, 603]; +const MAX_NEW_TOKENS: usize = 8; + +fn model_dir() -> Option { + let dir = std::env::var_os("GEMMA3N_DIR") + .map(PathBuf::from) + .unwrap_or_else(|| PathBuf::from(DEFAULT_DIR)); + if dir.join("inference_metadata.yaml").is_file() { + Some(dir) + } else { + eprintln!( + "skipping gemma3n capture real-model test: {} not present", + dir.display() + ); + None + } +} + +/// Minimal zeroed audio inputs (all-false validity mask → no audio tokens). +fn dummy_audio() -> anyhow::Result<(Value, Value)> { + const TIME: usize = 4; + const MELS: usize = 128; + let features = + Value::from_vec_f16_bits(vec![0u16; TIME * MELS], &[1, TIME as i64, MELS as i64])?; + let mask = Value::from_raw_bytes(vec![0u8; TIME], &[1, TIME as i64], DataType::Bool)?; + Ok((features, mask)) +} + +/// Minimal vision inputs with valid grid coordinates (no image tokens in the +/// prompt → image features are not gathered into the sequence). +fn dummy_vision() -> anyhow::Result<(Value, Value)> { + const ROWS: usize = 2; + const COLS: usize = 2; + const PATCHES: usize = ROWS * COLS; + let pixel_values = + Value::from_vec_f16_bits(vec![0u16; PATCHES * 768], &[1, PATCHES as i64, 768])?; + let mut coords = Vec::with_capacity(PATCHES * 2); + for r in 0..ROWS as i64 { + for c in 0..COLS as i64 { + coords.push(r); + coords.push(c); + } + } + let pixel_position_ids = Value::from_slice_i64(&coords, &[1, PATCHES as i64, 2])?; + Ok((pixel_values, pixel_position_ids)) +} + +fn build_engine(dir: &Path) -> anyhow::Result { + Engine::from_pipeline_dir( + dir, + EngineConfig { + pipeline_cache_bytes: 0, + ..EngineConfig::default() + }, + ) +} + +fn decode(engine: &mut PipelineEngine) -> anyhow::Result> { + let mut request = GenerateRequest::new(GeneratePrompt::TokenIds(PROMPT.to_vec())); + request.options = GenerateOptions { + max_new_tokens: MAX_NEW_TOKENS, + temperature: 0.0, + greedy: true, + stop_on_eos: false, + ..GenerateOptions::default() + }; + let (features, mask) = dummy_audio()?; + let (pixel_values, pixel_position_ids) = dummy_vision()?; + let pipeline_request = PipelineGenerateRequest::new(request) + .with_input("audio_encoder.input_features", features) + .with_input("audio_encoder.input_features_mask", mask) + .with_input("vision_encoder.pixel_values", pixel_values) + .with_input("vision_encoder.pixel_position_ids", pixel_position_ids); + let result = engine.generate_with_pipeline_request(pipeline_request)?; + Ok(result.token_ids) +} + +/// Run the native-CUDA pipeline decode once with the capture flag set to +/// `capture`, returning `(tokens, captured_step_input_decodes)`. +fn run(dir: &Path, capture: bool) -> anyhow::Result<(Vec, u64)> { + unsafe { + std::env::set_var(NATIVE_DECODER_ENV, "decoder"); + std::env::set_var(NATIVE_DECODER_DEVICE_ENV, "cuda:0"); + if capture { + std::env::set_var(CAPTURE_ENV, "1"); + } else { + std::env::remove_var(CAPTURE_ENV); + } + } + let before = NATIVE_DECODER_CAPTURED_STEP_INPUT_DECODES.load(Ordering::Relaxed); + let result = (|| { + let mut engine = build_engine(dir)?; + decode(&mut engine) + })(); + let after = NATIVE_DECODER_CAPTURED_STEP_INPUT_DECODES.load(Ordering::Relaxed); + unsafe { + std::env::remove_var(NATIVE_DECODER_ENV); + std::env::remove_var(NATIVE_DECODER_DEVICE_ENV); + std::env::remove_var(CAPTURE_ENV); + } + Ok((result?, after - before)) +} + +#[test] +#[ignore = "requires the real gemma-3n-e2b native package via GEMMA3N_DIR and a CUDA device"] +fn gemma3n_native_cuda_capture_engages_and_is_token_parity() -> anyhow::Result<()> { + let Some(dir) = model_dir() else { + return Ok(()); + }; + + // Text-only decode still requires *valid* dummy vision inputs because + // gemma-3n declares `vision_encoder.pixel_values` as a required request + // input and the pipeline eagerly runs the vision encoder even when the + // prompt carries no image tokens. The vision pooler's `OneHot` derives its + // depth from real image-grid structure, so zeroed/synthetic patches are + // rejected ("Depth is negative"). Producing valid inputs needs either a real + // image or optional-modality skip (an unrelated pipeline gap, tracked in the + // decision note). Until then, skip gracefully — this harness is the ready + // real-model capture-engagement proof the moment the vision-input gap + // closes. (The Bool audio-mask blocker it also depended on is fixed by the + // canonical `clone_value`/`clone_owned` host-guarded raw-byte clones in + // #540 — not carried by this validation PR.) + let (tokens_off, captured_off) = match run(&dir, false) { + Ok(result) => result, + Err(error) => { + eprintln!( + "skipping gemma-3n native-CUDA capture engagement: text-only decode still needs \ + valid vision inputs (or optional-modality skip): {error:#}" + ); + return Ok(()); + } + }; + eprintln!("capture OFF: tokens={tokens_off:?} captured_step_input_decodes={captured_off}"); + let (tokens_on, captured_on) = run(&dir, true)?; + eprintln!("capture ON : tokens={tokens_on:?} captured_step_input_decodes={captured_on}"); + + assert_eq!( + tokens_off.len(), + MAX_NEW_TOKENS, + "native decode did not emit the requested token count" + ); + // The optimization must be token-for-token free. + assert_eq!( + tokens_off, tokens_on, + "captured step-inputs path changed the real-model token output" + ); + // Flag OFF must never engage the captured path. + assert_eq!(captured_off, 0, "captured path engaged with the flag OFF"); + // Flag ON must engage on this real GQA-capacity-KV decoder (one captured + // decode per single-token step after the multi-token prefill). + assert!( + captured_on > 0, + "captured step-inputs path did NOT engage on the real gemma-3n decoder \ + (flag ON produced {captured_on} captured decodes) — real model declines capture" + ); + eprintln!( + "PROVEN: real gemma-3n native-CUDA pipeline capture engaged {captured_on}x, tokens identical {tokens_off:?}" + ); + Ok(()) +} diff --git a/crates/onnx-genai-engine/tests/qwen3_0_6b_capture_step_inputs_decline.rs b/crates/onnx-genai-engine/tests/qwen3_0_6b_capture_step_inputs_decline.rs new file mode 100644 index 0000000000..078518d86d --- /dev/null +++ b/crates/onnx-genai-engine/tests/qwen3_0_6b_capture_step_inputs_decline.rs @@ -0,0 +1,121 @@ +//! Inc3c real-model validation (issue #384): concretely demonstrate — with the +//! on-main capture counter — that **qwen3-0.6b does NOT engage the +//! capture-step-inputs path**, because it is a *single-component* `input_ids` +//! model, not a multi-component `inputs_embeds`/routed pipeline decoder. +//! +//! This is the honest "counter proof on the real qwen3-0.6b" the flag path was +//! probed for: the flag `ONNX_GENAI_NATIVE_DECODER_CAPTURE_STEP_INPUTS` only +//! gates `native_decode/cuda.rs::decode_cuda_captured_step_inputs`, which is +//! reached solely by a pipeline decoder that declares an `inputs_embeds`/`Routed` +//! port. qwen3-0.6b's decoder consumes `input_ids` directly and loads via the +//! single-model `Engine::from_dir`, so: +//! 1. `Engine::from_pipeline_dir` refuses it (no multi-component pipeline), and +//! 2. its native-CUDA single-graph decode leaves the captured-step-inputs +//! counter at 0 even with the flag ON (it captures via the *token-id* +//! `run_one_token` path, a different mechanism — the Part A 612/220/443 +//! lever — not the step-inputs path). +//! +//! The capture-step-inputs win therefore applies to multi-component +//! `inputs_embeds` decoders with capacity-aware KV (GroupQueryAttention), i.e. +//! the gemma-3n / 35B-A3B class — proven on the synthetic `tiny-gqa-embeds-cuda` +//! fixture in `native_cuda_captured_step_inputs_parity`. +//! +//! ```bash +//! source .cudaenv.sh +//! export CUDA_VISIBLE_DEVICES=4 +//! export ONNX_GENAI_ORT_LIB=$ORT_ROOT/lib/libonnxruntime.so.1.27.0 +//! QWEN3_0_6B_CUDA_E2E_DIR=/home/justinchu/mobius/.scratch/qwen3-0.6b-int4-cuda-postfix \ +//! cargo test -p onnx-genai-engine --features cuda,native-backend \ +//! --test qwen3_0_6b_capture_step_inputs_decline -- --ignored --nocapture +//! ``` +#![cfg(all(feature = "cuda", feature = "native-backend"))] + +use std::path::PathBuf; +use std::sync::atomic::Ordering; + +use onnx_genai_engine::{ + Engine, EngineConfig, EngineDecodeBackend, GenerateOptions, GeneratePrompt, GenerateRequest, + NATIVE_DECODER_CAPTURED_STEP_INPUT_DECODES, NativeDecodeDevice, +}; + +const DEFAULT_DIR: &str = "/home/justinchu/mobius/.scratch/qwen3-0.6b-int4-cuda-postfix"; +const CAPTURE_ENV: &str = "ONNX_GENAI_NATIVE_DECODER_CAPTURE_STEP_INPUTS"; +const PROMPT: &[u32] = &[785, 6722, 315]; +const MAX_NEW_TOKENS: usize = 8; + +fn model_dir() -> Option { + let dir = std::env::var_os("QWEN3_0_6B_CUDA_E2E_DIR") + .map(PathBuf::from) + .unwrap_or_else(|| PathBuf::from(DEFAULT_DIR)); + if dir.join("model.onnx").is_file() && dir.join("inference_metadata.yaml").is_file() { + Some(dir) + } else { + eprintln!( + "skipping qwen3-0.6b capture-decline test: {} not present", + dir.display() + ); + None + } +} + +#[test] +#[ignore = "requires the real qwen3-0.6b int4 model via QWEN3_0_6B_CUDA_E2E_DIR and a CUDA device"] +fn qwen3_0_6b_single_component_declines_capture_step_inputs() -> anyhow::Result<()> { + let Some(dir) = model_dir() else { + return Ok(()); + }; + + // (1) It is NOT a multi-component pipeline: `from_pipeline_dir` must refuse. + let pipeline_load = Engine::from_pipeline_dir(&dir, EngineConfig::default()); + assert!( + pipeline_load.is_err(), + "qwen3-0.6b unexpectedly loaded as a multi-component pipeline; it is single-component" + ); + eprintln!( + "qwen3-0.6b is single-component: from_pipeline_dir refused it: {}", + pipeline_load.err().unwrap() + ); + + // (2) Its native-CUDA single-graph decode leaves the captured-step-inputs + // counter at 0 even with the flag ON — the flag path is pipeline-only. + unsafe { std::env::set_var(CAPTURE_ENV, "1") }; + let before = NATIVE_DECODER_CAPTURED_STEP_INPUT_DECODES.load(Ordering::Relaxed); + let tokens = (|| { + let mut engine = Engine::from_dir( + &dir, + EngineConfig { + decode_backend: EngineDecodeBackend::Native, + native_device: Some(NativeDecodeDevice::Cuda { index: Some(0) }), + ..EngineConfig::default() + }, + )?; + let mut request = GenerateRequest::new(GeneratePrompt::TokenIds(PROMPT.to_vec())); + request.options = GenerateOptions { + max_new_tokens: MAX_NEW_TOKENS, + temperature: 0.0, + greedy: true, + stop_on_eos: false, + ..GenerateOptions::default() + }; + Ok::<_, anyhow::Error>(engine.generate(request)?.token_ids) + })(); + let after = NATIVE_DECODER_CAPTURED_STEP_INPUT_DECODES.load(Ordering::Relaxed); + unsafe { std::env::remove_var(CAPTURE_ENV) }; + let tokens = tokens?; + + assert_eq!( + tokens.len(), + MAX_NEW_TOKENS, + "native decode did not emit the requested tokens" + ); + assert_eq!( + after - before, + 0, + "qwen3-0.6b single-graph decode unexpectedly engaged the capture-step-inputs path" + ); + eprintln!( + "PROVEN: qwen3-0.6b native-CUDA decode with capture flag ON → captured_step_input_decodes=0 \ + (single-component input_ids model does NOT route through the pipeline capture-step-inputs path); tokens={tokens:?}" + ); + Ok(()) +}