[justinchuby]

Problem

A dynamic Compress output shape is the count of selected entries, so computing it requires reading the condition tensor. The dynamic shape reader only admitted the small tensors that normally carry shape data, so a vision graph whose condition spans a full image failed shape inference:

no inferred shape ... produced by op Compress

Serving a 448x448 image through the Muse vision encoder returned HTTP 500 as a result.

Fix

Read a Compress condition through its own bounded path, so the one operand the shape genuinely depends on is materialized while an unrelated large tensor still is not. The bound is an explicit element cap rather than the general shape-data limit.

Testing

• New regression test covering a condition larger than the general shape-data limit.
• cargo test -p onnx-runtime-session --lib compress passes.
• Verified end to end: a 448x448 image request that previously 500'd now succeeds against the native CUDA server.

Note

Stacked on #1159, which introduces the Compress shape-inference path this extends.

[codecov]

Codecov Report

❌ Patch coverage is 35.29412% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.56%. Comparing base (e8b2986) to head (7d03319).

Files with missing lines	Patch %	Lines
...ates/onnx-runtime-session/src/executor/dispatch.rs	29.03%	21 Missing and 1 partial ⚠️

Additional details and impacted files

@@                     Coverage Diff                      @@
##           split/native-cuda-vision    #1195      +/-   ##
============================================================
+ Coverage                     79.84%   80.56%   +0.71%     
============================================================
  Files                           359      359              
  Lines                        157265   157958     +693     
  Branches                     157265   157958     +693     
============================================================
+ Hits                         125563   127252    +1689     
+ Misses                        27140    26085    -1055     
- Partials                       4562     4621      +59     

Flag	Coverage Δ
mlas	85.49% <ø> (+0.16%)	⬆️
offline	80.46% <35.29%> (+0.73%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...ates/onnx-runtime-session/src/executor/geometry.rs	92.65% <100.00%> (+1.70%)	⬆️
...ates/onnx-runtime-session/src/executor/dispatch.rs	67.68% <29.03%> (-1.45%)	⬇️

... and 28 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	560.90 µs	1.09 ms	+94.1%
🔴	matmul/large_generic_f32_threads=8/32x1024x1024	4.00 ms	6.94 ms	+73.6%
🔴	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	52.89 µs	87.45 µs	+65.4%
🔴	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	57.51 µs	93.65 µs	+62.8%
🔴	matmul/large_generic_f16_threads=1/32x1024x1024	81.76 µs	120.99 µs	+48.0%
🔴	matmul/large_generic_bf16_threads=8/32x1024x1024	1.34 ms	1.95 ms	+45.2%
⚠️	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	484.34 µs	618.72 µs	+27.7%
⚠️	matmul/large_generic_f16_threads=8/32x1024x1024	94.27 µs	119.50 µs	+26.8%
⚠️	matmul/large_generic_f32_threads=1/32x1024x1024	9.38 ms	11.79 ms	+25.6%
⚠️	sampling_latency/min_p_per_token	206.38 µs	241.26 µs	+16.9%
⚠️	matmul/medium_generic_f16_threads=1/32x512x512	30.46 µs	35.58 µs	+16.8%
✅	sampling_latency/top_k_per_token	51.70 µs	58.52 µs	+13.2%
✅	qwen3_sampling_processors/top_k_partial_selection	142.49 µs	161.15 µs	+13.1%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	2.10 ms	2.36 ms	+12.5%
✅	gather/small_bf16_threads=1-internal/4096	514.3 ns	575.0 ns	+11.8%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	2.19 ms	2.43 ms	+11.2%
✅	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	168.10 µs	185.48 µs	+10.3%
✅	tokenization/decode_tokens_per_second	6.30 ms	6.94 ms	+10.1%
✅	matmul/medium_generic_f16_threads=8/32x512x512	30.64 µs	33.54 µs	+9.5%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	5.70 ms	6.19 ms	+8.6%
✅	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.55 ms	3.82 ms	+7.8%
✅	sampling_latency/top_p_per_token	378.16 µs	406.52 µs	+7.5%
✅	gather/small_f16_threads=1-internal/4096	542.9 ns	576.5 ns	+6.2%
✅	qwen3_sampling_processors/top_k_top_p_fast	661.97 µs	702.37 µs	+6.1%
✅	sampling_latency/greedy_per_token	3.27 µs	3.42 µs	+4.3%
✅	matmul/small_generic_bf16_threads=8/1x256x256	30.86 µs	31.84 µs	+3.2%
✅	grammar_masking/llguidance_compute_mask/32	76.59 µs	78.80 µs	+2.9%
✅	matmul/small_generic_f32_threads=8/1x256x256	35.25 µs	35.83 µs	+1.6%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	531.80 µs	538.28 µs	+1.2%
✅	matmul/medium_generic_bf16_threads=8/32x512x512	402.42 µs	401.70 µs	-0.2%
✅	gather/medium_f16_threads=1-internal/32768	2.42 µs	2.42 µs	-0.2%
✅	matmul/small_generic_f16_threads=1/1x256x256	31.69 µs	31.52 µs	-0.5%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	534.58 µs	526.45 µs	-1.5%
✅	gather/medium_f32_threads=1-internal/32768	3.87 µs	3.78 µs	-2.3%
✅	matmul/medium_generic_f32_threads=8/32x512x512	981.48 µs	949.97 µs	-3.2%
✅	gather/medium_bf16_threads=1-internal/32768	2.51 µs	2.42 µs	-3.8%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.40 ms	2.29 ms	-4.6%
✅	kv_cache/alloc_dealloc_pages	44.32 µs	42.02 µs	-5.2%
✅	tokenization/encode_tokens_per_second	425.92 µs	400.97 µs	-5.9%
✅	reduce_mean/small_f32_threads=1-internal/4096	15.98 µs	14.86 µs	-7.0%
✅	gather/large_f32_threads=1-internal/131072	33.08 µs	30.44 µs	-8.0%
✅	matmul/small_generic_bf16_threads=1/1x256x256	35.34 µs	31.80 µs	-10.0%
✅	matmul/small_generic_f16_threads=8/1x256x256	33.71 µs	30.30 µs	-10.1%
✅	logit_processing/seven_processor_chain_per_step	357.76 µs	321.10 µs	-10.2%
✅	gather/small_f32_threads=1-internal/4096	766.4 ns	680.3 ns	-11.2%
🟢	gather/large_bf16_threads=1-internal/131072	14.92 µs	12.59 µs	-15.6%
🟢	matmul/small_generic_f32_threads=1/1x256x256	44.40 µs	36.91 µs	-16.9%
🟢	reduce_mean/medium_f32_threads=1-internal/65536	301.22 µs	248.16 µs	-17.6%
🟢	reduce_mean/large_f32_threads=1-internal/262144	1.30 ms	979.09 µs	-24.4%
🟢	add/large_bf16_threads=1-internal/4194304	2.34 ms	1.69 ms	-28.0%
🟢	add/medium_bf16_threads=1-internal/262144	145.83 µs	104.24 µs	-28.5%
🟢	gather/large_f16_threads=1-internal/131072	16.14 µs	11.52 µs	-28.6%
🟢	add/large_f16_threads=1-internal/4194304	2.43 ms	1.72 ms	-29.1%
🟢	add/medium_f32_threads=1-internal/262144	37.60 µs	24.85 µs	-33.9%
🟢	add/medium_f16_threads=1-internal/262144	163.36 µs	106.81 µs	-34.6%
🟢	add/large_f32_threads=1-internal/4194304	1.05 ms	671.06 µs	-35.9%
🟢	add/small_bf16_threads=1-internal/1024	724.5 ns	444.8 ns	-38.6%
🟢	add/small_f16_threads=1-internal/1024	786.1 ns	445.0 ns	-43.4%
🟢	add/small_f32_threads=1-internal/1024	356.8 ns	192.4 ns	-46.1%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 4.62 4.10 6.55 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

[justinchuby]

Verified and landed as #1253 (merged cdf5ff3), which is this fix rebased onto current main (now including #1251, the replacement for the stacked base #1159) plus a follow-up commit adding the falsifying test the original was missing.

The shipped test only exercised the bounded_compress_condition predicate, so reverting the dispatch routing (the behavioural half of the fix) left it green. I added a behavioural end-to-end test (image-sized 1524-element Bool condition, run on the CPU EP) that goes RED with exactly UnresolvedShape { value: "y", op: "Compress" } when the routing is reverted, plus boundary coverage (condition longer/shorter than axis, all-false empty, all-true full). Full runtime-session suite: 185 passed, 0 failed. Closing in favour of #1253.