perf(cpu): spread the decode CPU budget one thread per physical core by justinchuby · Pull Request #1232 · justinchuby/onnx-genai

justinchuby · 2026-08-18T07:36:00Z

The CPU budget was buying hyperthreads, not cores

ONNX_GENAI_CPU_DECODE_THREADS=N confines the process to N logical CPUs.
choose_budget_cpus picked them as the N lowest indices of the chosen NUMA
node. Every host we run on numbers SMT siblings adjacently — 0-1, 2-3, … on
AMD EPYC and on Intel since Skylake-SP — so a budget of N landed on N/2
physical cores.

The symptom was unmistakable once measured. Int4 MatMulNBits 4096×6144, 128
tokens, native-only, 20 runs, on the 16-core/32-thread EPYC 9V74:

budget	wall per run	user CPU
1	79.4 ms	2.04 s
2	81.2 ms	4.10 s
3	40.4 ms	2.05 s
4	53.9 ms	3.35 s

A budget of 2 was exactly as slow as a budget of 1 while burning two cores. A
budget of 3 beat a budget of 4. taskset confirms the cause directly — 0,2
(two cores) ran in 50.1 ms against 0,1 (one core, two threads) at 81.8 ms,
using 36% less CPU.

The fix

A ranking, not a widening:

scatter_across_cores orders the candidate pool by
CoreTopology::leaders_within — one CPU per physical core first, siblings
after — and truncates after ranking. The result stays a subset of the same
pool, so the cpuset/cgroup guarantee is unchanged and a full-width budget
returns exactly what it did before.
smt_scaled_request sizes the NUMA node search in cores rather than logical
CPUs, falling back to the old logical sizing when no node is that large, so a
budget that fits on one core-rich node still stays on one node.
the cross-node top-up now happens before ranking, so a second node's fresh
cores can displace the first node's SMT siblings instead of being appended to
an already-full mask.
order_pin_targets applies the same order to the flat decode Rayon pool's pin
list (the builder pins worker i to cpus[i % len], so the order decides
whether an 8-worker pool occupies 8 cores or 4). The persistent SPMD pool and
the numa-split sub-pools are deliberately left compact — their workers spin,
and spreading spinning workers one per core is the experiment already recorded
in core_topology's module docs, where it measured worse.

choose_budget_cpus gained a cores: Option<&CoreTopology> parameter so the
policy stays pure and unit-testable; with None (no discoverable SMT map) every
function here is the identity and the old behaviour stands exactly.

Result

42 cells (20 GEMM, 22 transform) × 6 widths × 5 trials, paired arms in one
driver invocation. Native-only view — the mask moves ORT's threads too, so only
native-vs-native isolates the change. Geomean of before/after, >1 is faster:

width	GEMM	>1.05×	<0.95×	Transforms	>1.05×	<0.95×
1	0.996	0	1	0.997	1	3
2	1.769	19	0	1.191	12	0
4	1.642	20	0	1.095	11	2
8	1.441	16	1	0.913	4	9
16	1.245	15	1	0.961	6	9
32	0.976	8	6	1.062	9	4

The two ends are the control: a budget of 1 cannot be re-ranked and a budget of
32 is the whole machine, so both must be flat — and both are. At 2 and 4 threads
not one cell out of 42 regressed.

The transform column at t=8/16 reads as a loss and is not one: those cells are
20–500 µs at five trials on a shared box. Re-measured at 40 runs × 5 warmups
with the arms strictly interleaved, they are wins — sm_bert_b8_s128 by 1.9× at
both widths, rope_gptj_il_s512 by 1.35× at t=8. The five-trial grid is left in
the docs unedited so the distinction stays visible; §34.5 has the raw pairs.

Validation

cargo test -p onnx-runtime-ep-cpu — 1391 passed, 0 failed
cargo clippy -p onnx-runtime-ep-cpu --all-targets — clean
cargo fmt --all --check — clean
8 new unit tests, all of which fail if the ranking is reverted, including the
cpuset-containment case (a leader that exists in the topology but not in the
process's allowed set must not be invented), the asymmetric-node case that
exercises smt_scaled_request through choose_budget_cpus, the
top-up-before-ranking case, and the "a budget that fits one node does not
spill across nodes for cores" case.

Documented as §34 Phase 14 in
docs/benchmarks/2026-08-15-cpu-ep-vs-ort-attention-moe.md.

Rebase and convergence (2026-08-18)

#1201/#1202/#1207 were squash-merged, so this branch's ancestry no longer
reached main. Its three commits were cherry-picked onto main at c55a3fab3;
all applied cleanly, and the diff is now self-contained: 3 files, +465/−29.
Review the whole PR, not "the last two commits".

Defect found and fixed during the rebase: the benchmark section was numbered
## 32. Phase 12, which collided with the existing §32 (Phase 12, erf
Estrin) on main. Renumbered to ## 34. Phase 14 (subsections 34.1–34.6),
and the internal back-reference "Phase 11 built a task runtime" corrected to
"Phase 13 (§33) built a task runtime" — §33 is where the task runtime actually
landed. The squad decision note's §32 cross-reference was updated to match.

Re-validated on the rebased head (rustc 1.97.1, the same toolchain CI resolves):

cargo test -p onnx-runtime-ep-cpu --lib — 1426 passed, 0 failed, 17 ignored
cargo clippy -p onnx-runtime-ep-cpu --all-targets — clean
cargo fmt --all -- --check — clean

CI note: the repository's Actions queue is saturated (every recent run is
queued, nothing in_progress), so the two required checks — Fast (Linux x86_64) and Rust quality — cannot report. Both lanes were reproduced locally,
step for step, from .github/workflows/ci.yml. main itself fails both of
them today; #1346 is the fix for that and lands first.

## What `main` at 4d231ea fails two blocking CI steps on current stable (rustc/rustfmt 1.9.0, 1.97.1): ``` $ cargo fmt --all -- --check Diff in crates/onnx-runtime-ep-cuda/src/provider.rs Diff in crates/onnx-runtime-memory-api/src/allocator.rs Diff in crates/onnx-runtime-memory-api/src/capability.rs $ cargo clippy --locked --all-targets -p onnx-runtime-session -- -D warnings error: field `0` is never read --> crates/onnx-runtime-session/src/executor/mod.rs:175:45 ``` Both landed while the runner queue was saturated (71 queued / 1 in progress at the time of writing, nothing completed on `main` since 08:18Z), so no PR has seen a red check yet. Every open PR in the repo currently inherits both failures. ## Why these fixes **rustfmt** — mechanical normalisation, no semantic change. **`ActivationPlanForTest`** (from #1226) is a tuple struct whose single field is the `globals_lock()` `MutexGuard`. `dead_code` does not model "this field's value is its `Drop`", so it fires. The guard must stay: releasing it early is exactly the leaked-planner-gate race the struct was added to prevent. So the lint is silenced with a comment explaining the RAII intent, rather than the field removed. ## Verification | check | result | | --- | --- | | `cargo fmt --all -- --check` | clean | | `cargo clippy --locked --all-targets -p onnx-runtime-session -- -D warnings` | clean | | `cargo test --locked -p onnx-runtime-session --lib` | 181 passed, 0 failed | Found while trying to land the CPU task-runtime stack (#1201 → #1202 → #1207 → #1232 → #1238); this is unrelated to that work and is deliberately kept out of it so it can go in on its own. Working as sebastian (Performance Engineer) --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-08-18T14:01:19Z

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	`block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024`	46.68 µs	120.05 µs	+157.2%
🔴	`block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1`	60.94 µs	124.17 µs	+103.8%
🔴	`block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024`	608.39 µs	1.20 ms	+97.1%
🔴	`gather/large_bf16_threads=1-internal/131072`	16.12 µs	28.52 µs	+76.9%
🔴	`matmul/large_generic_f32_threads=8/32x1024x1024`	4.12 ms	7.22 ms	+75.0%
🔴	`block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024`	60.67 µs	102.44 µs	+68.9%
🔴	`matmul/large_generic_bf16_threads=8/32x1024x1024`	1.43 ms	2.28 ms	+59.2%
🔴	`matmul/medium_generic_bf16_threads=8/32x512x512`	392.06 µs	604.10 µs	+54.1%
🔴	`block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1`	399.84 µs	608.77 µs	+52.3%
🔴	`gather/large_f16_threads=1-internal/131072`	16.42 µs	22.04 µs	+34.3%
🔴	`matmul/medium_generic_f32_threads=8/32x512x512`	1.32 ms	1.73 ms	+31.0%
⚠️	`gather/medium_f32_threads=1-internal/32768`	4.75 µs	6.15 µs	+29.6%
⚠️	`matmul/medium_generic_bf16_threads=1/32x512x512`	531.42 µs	676.30 µs	+27.3%
⚠️	`kv_cache/alloc_dealloc_pages`	35.88 µs	45.33 µs	+26.3%
⚠️	`matmul/small_generic_f16_threads=8/1x256x256`	32.95 µs	40.34 µs	+22.4%
⚠️	`sampling_latency/top_k_per_token`	51.56 µs	63.10 µs	+22.4%
⚠️	`sampling_latency/min_p_per_token`	215.09 µs	259.76 µs	+20.8%
⚠️	`matmul/medium_generic_f16_threads=1/32x512x512`	33.15 µs	39.63 µs	+19.5%
⚠️	`grammar_masking/llguidance_compute_mask/32`	69.75 µs	83.25 µs	+19.4%
⚠️	`qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline`	3.55 ms	4.18 ms	+17.7%
⚠️	`qwen3_sampling_processors/top_k_top_p_fast`	633.76 µs	744.07 µs	+17.4%
⚠️	`gather/small_bf16_threads=1-internal/4096`	530.2 ns	611.9 ns	+15.4%
✅	`sampling_latency/top_p_per_token`	368.61 µs	417.30 µs	+13.2%
✅	`gather/small_f16_threads=1-internal/4096`	502.5 ns	567.2 ns	+12.9%
✅	`logit_processing/seven_processor_chain_per_step`	302.79 µs	341.61 µs	+12.8%
✅	`matmul/small_generic_bf16_threads=8/1x256x256`	40.84 µs	45.80 µs	+12.1%
✅	`matmul/large_generic_f32_threads=1/32x1024x1024`	9.90 ms	11.03 ms	+11.4%
✅	`qwen3_sampling_processors/top_p_fast_after_top_k`	502.21 µs	557.73 µs	+11.1%
✅	`matmul/medium_generic_f32_threads=1/32x512x512`	2.40 ms	2.67 ms	+10.9%
✅	`qwen3_sampling_processors/top_k_top_p_full_sort_baseline`	5.87 ms	6.35 ms	+8.3%
✅	`matmul/large_generic_f16_threads=8/32x1024x1024`	88.03 µs	95.20 µs	+8.2%
✅	`gather/large_f32_threads=1-internal/131072`	46.57 µs	49.91 µs	+7.2%
✅	`qwen3_sampling_processors/top_k_partial_selection`	147.29 µs	154.54 µs	+4.9%
✅	`tokenization/encode_tokens_per_second`	419.61 µs	439.92 µs	+4.8%
✅	`tokenization/decode_tokens_per_second`	6.68 ms	6.99 ms	+4.7%
✅	`add/small_bf16_threads=1-internal/1024`	619.0 ns	646.9 ns	+4.5%
✅	`matmul/large_generic_bf16_threads=1/32x1024x1024`	2.18 ms	2.24 ms	+2.4%
✅	`gather/small_f32_threads=1-internal/4096`	737.4 ns	753.2 ns	+2.1%
✅	`sampling_latency/greedy_per_token`	3.32 µs	3.39 µs	+2.0%
✅	`matmul/large_generic_f16_threads=1/32x1024x1024`	85.16 µs	86.39 µs	+1.5%
✅	`qwen3_sampling_processors/top_k_full_sort_baseline`	2.28 ms	2.31 ms	+1.3%
✅	`matmul/small_generic_f32_threads=1/1x256x256`	41.28 µs	41.64 µs	+0.9%
✅	`matmul/small_generic_bf16_threads=1/1x256x256`	36.20 µs	36.38 µs	+0.5%
✅	`reduce_mean/large_f32_threads=1-internal/262144`	1.07 ms	1.07 ms	+0.2%
✅	`add/large_bf16_threads=1-internal/4194304`	1.81 ms	1.80 ms	-0.3%
✅	`matmul/medium_generic_f16_threads=8/32x512x512`	38.30 µs	36.69 µs	-4.2%
✅	`add/small_f16_threads=1-internal/1024`	553.3 ns	529.7 ns	-4.3%
✅	`matmul/small_generic_f32_threads=8/1x256x256`	55.27 µs	52.85 µs	-4.4%
✅	`gather/medium_f16_threads=1-internal/32768`	2.95 µs	2.78 µs	-5.9%
✅	`gather/medium_bf16_threads=1-internal/32768`	3.05 µs	2.80 µs	-8.1%
✅	`add/medium_f32_threads=1-internal/262144`	35.61 µs	31.33 µs	-12.0%
✅	`matmul/small_generic_f16_threads=1/1x256x256`	39.24 µs	34.43 µs	-12.3%
🟢	`add/small_f32_threads=1-internal/1024`	256.1 ns	212.0 ns	-17.2%
🟢	`reduce_mean/small_f32_threads=1-internal/4096`	20.34 µs	16.06 µs	-21.0%
🟢	`add/medium_f16_threads=1-internal/262144`	165.46 µs	130.31 µs	-21.2%
🟢	`add/large_f16_threads=1-internal/4194304`	2.47 ms	1.93 ms	-21.9%
🟢	`reduce_mean/medium_f32_threads=1-internal/65536`	329.58 µs	251.05 µs	-23.8%
🟢	`add/large_f32_threads=1-internal/4194304`	1.05 ms	795.59 µs	-24.0%
🟢	`add/medium_bf16_threads=1-internal/262144`	172.57 µs	116.78 µs	-32.3%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 5.12 3.60 5.74 }

What this cannot catch

Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
Sub-threshold regressions that compound over multiple PRs
Performance changes that only manifest under GPU execution
Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

#1232 lands its own phase section and takes §34/Phase 14, so this PR's section is renumbered to §35/Phase 15 to keep the ledger's numbering unique and in merge order. Heading text only; the `34.1×` figure in the matrix is a speed ratio, not a section reference, and is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

`ONNX_GENAI_CPU_DECODE_THREADS=N` confines the process to N CPUs, and `choose_budget_cpus` picked them as the N lowest indices of the chosen NUMA node. Every host we run on numbers SMT siblings adjacently -- `0-1`, `2-3`, ... on AMD EPYC and on Intel since Skylake-SP -- so a budget of N landed on N/2 physical cores. Two threads sharing one core's front end are not two cores' throughput, so the mask silently halved the machine and then burned twice the CPU proving it. Measured on the 16-core/32-thread EPYC 9V74, `MatMulNBits` 4096x6144 int4 with 128 tokens, native only, 20 runs (before -> after): | budget | native ms | user CPU | | ------ | --------- | -------- | | 2 | 81.4 -> 39.6 (2.05x) | 4.11 s -> 2.09 s | | 4 | 54.9 -> 26.7 (2.05x) | 3.39 s -> 2.06 s | | 8 | 24.3 -> 11.4 (2.14x) | 3.91 s -> 2.02 s | | 16 | 11.9 -> 6.8 (1.75x) | 3.90 s -> 2.09 s | Twice the throughput at half the CPU time, at every budget below the whole machine. The prior symptom was stark: at a budget of 2 the run took *exactly* as long as at a budget of 1 while burning two cores, because the second thread was the first thread's hyperthread. The fix is a ranking, not a widening. `scatter_across_cores` orders the candidate pool by `CoreTopology::leaders_within` -- one CPU per physical core first, siblings afterwards -- and only then truncates to the budget, so the result is still a subset of the process cpuset and the full-width budget is unchanged. `smt_scaled_request` sizes the NUMA node search in cores rather than logical CPUs so a budget that fits on one core-rich node still stays on one node, and falls back to the old logical sizing when no node is that large. The cross-node top-up now happens before the ranking, so a second node's CPUs compete for core slots instead of being appended to an already-full mask. `order_pin_targets` applies the same rule to the *flat* decode Rayon pool's pin list, whose builder pins worker `i` to `cpus[i % len]` -- the order of that list, not just its contents, decides whether an 8-worker pool occupies 8 cores or 4. The persistent SPMD pool and the `numa-split` sub-pools are deliberately left compact: their workers spin, and spreading spinning workers one per core is the experiment already recorded in `core_topology`'s module docs, where it measured worse. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Numbered §34 / Phase 14: the document already has a §32 (Phase 12, erf's Estrin reassociation) and a §33 (Phase 13, the CPU task runtime), both of which landed while this branch was open. Filing this as §32 would have produced two sections with the same number and two different Phase 12s. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

justinchuby · 2026-08-19T00:24:08Z

Convergence report — rebased, defect fixed, validated, merging

Rebase

#1201/#1202/#1207 were squash-merged, so this branch's ancestry no longer
reached main. Its three commits were cherry-picked onto main at
c5d10941b (clean). The diff is self-contained: 3 files, +465/−29. Review the
whole PR — the "review the last two commits only" note in the original
description no longer applies and has been removed.

Defect found and fixed

The benchmark section was numbered ## 32. Phase 12, which collides with the
existing §32 (Phase 12, erf Estrin) already on main. Renumbered to
## 34. Phase 14 with subsections 34.1–34.6, and the internal back-reference
"Phase 11 built a task runtime" corrected to "Phase 13 (§33) built a task
runtime" — §33 is where the task runtime actually landed. The squad decision
note's §32 cross-reference was updated to match.

Implementation review

scatter_across_cores ranks the candidate pool by CoreTopology::leaders_within
and truncates after ranking, so the result is always a subset of the pool the
cpuset/cgroup already allows — it reorders, it never widens. With
cores = None every helper is the identity, so a host with no discoverable SMT
map keeps the old behaviour exactly. The cross-node top-up moving before
ranking is the part that actually matters: appending a second node's cores to
an already-full mask could never displace the first node's SMT siblings.
order_pin_targets is applied only to the flat decode Rayon pool (the builder
pins worker i → cpus[i % len]), and deliberately not to the spinning
SPMD/numa-split pools — spreading spinning workers one per core is the
experiment already recorded in core_topology's module docs, where it measured
worse.

Validation on the rebased head (rustc 1.97.1, the toolchain CI resolves)

check	result
`cargo test -p onnx-runtime-ep-cpu --lib`	1427 passed, 0 failed, 17 ignored
`cargo clippy -p onnx-runtime-ep-cpu --all-targets -- -D warnings`	clean
`cargo fmt --all -- --check`	clean
`check_publish_order` / `check_profile_table` / `check_platform_naming`	OK
`check_dispatch_reachability` / `check_dispatch_manifest`	OK
`check_feature_gate_coverage` / `verify_documented_env_vars`	OK
`workspace_test_packages.py verify`	OK

That is every step of the Rust quality lane, run step for step out of
.github/workflows/ci.yml.

CI

The Actions queue is saturated — every recent run is queued, nothing is
in_progress repo-wide — so Fast (Linux x86_64) and Rust quality cannot
report on this PR. main was red on Rust quality twice today for unrelated
reasons (#1346, #1352, both merged); it is green now, and this branch is green
on the same reproduction. Merging on that evidence.

Working as sebastian (CPU perf).

perf(cpu): run the MLAS prefill tiling on the CPU task runtime The `m > 1` MLAS SQNBit prefill tiling (`run_mlas_shards`) issued a single `tiles.par_iter()` fan-out on global Rayon, sized by `rayon::current_num_threads()`. With a co-resident ORT intra-op pool spinning on the same cores, every parked-Rayon wake-up lands behind a spinning thread, which is what made `gemm_nbits_*_t8` at t=32 the worst cell in the benchmark ledger. This routes that fan-out through the CPU task runtime from #1201 instead, with a work-size policy for the one case where the SMT-capped pool leaves hardware threads idle. ## MLAS remains opt-in and non-load-bearing This does **not** enable MLAS anywhere. The changed code lives entirely inside the pre-existing `m > 1 && active > 1 && !mlas_prefill_serial()` path that already called `mlas_sys::sqnbit_gemm_into`. No Cargo feature, `#[cfg(feature = "mlas")]` gate, or default-feature set is touched: `mlas` is still opt-in (`default = ["full"]`, and `full` does not include `mlas`). The new policy helpers (`prefill_fan_out`, `prefill_tile_grain`, `PrefillFanOut`) are pure integer arithmetic marked `#[cfg_attr(not(feature = "mlas"), allow(dead_code))]` so they compile and their unit tests run on the default (mlas-off) CI lane even though only the MLAS path consults them. MLAS stays a labelled reference arm. ## What changed - `prefill_fan_out(macs, lanes, wide)`: below `WIDE_PREFILL_MACS` (512 Mi MACs), or whenever global Rayon is not actually wider than the pool, fan out on the task runtime (cheap ~5 us dispatch, topology-aware, no fight with a co-resident ORT pool). Above it, use the wider global Rayon path -- a prefill tile is a multi-ms MLAS call whose dequantise step has enough load latency that SMT siblings pay off, so the SMT cap costs more than a 226 us park wake-up (0.25% of a 90 ms fan-out). - `prefill_tile_grain`: a per-task tile floor so no task gets less than `MIN_PREFILL_TASK_MACS` (512 Ki) of arithmetic. ## Verification Hardware: Intel Core i7-13800H, 14 physical / 20 logical (6 P + 8 E). Baseline: `main` at the rebase point. Toolchain: cargo 1.97.1. - **Bit-identity (the important one).** This is pure scheduling: the `run_tile` closure and the `sqnbit_gemm_into` call are byte-for-byte the same, only the executor and grain differ, and every tile writes a disjoint `[row, row+rows) x [shard.start, shard.start+len)` window so order cannot matter. Confirmed empirically under `--features mlas`: `mlas_prefill_parallel_dispatch_matches_serial` and `mlas_prefill_dispatch_parity_subprocess` pass -- the routed parallel tiling matches the serial reference. - **Policy tests (default features).** All six `prefill_*` unit tests pass. - **Falsified.** Flipping the threshold comparison in `prefill_fan_out` from `<` to `<=` turns `large_prefill_work_takes_the_wide_fan_out` RED (`left: TaskRuntime, right: Wide` at exactly `WIDE_PREFILL_MACS`) -- the test is non-vacuous and guards the boundary. Restored to green. - **`--features mlas` compiles clean;** clippy `--all-targets -D warnings` clean on default features. ## Perf The §35 (Phase 15) tables in the benchmark doc record up to 13.5x at t=32 on the small int4 cells, dropping `gemm_nbits_*_t8` from 22-34x ORT to 2.6-2.8x. Those tables mix the author's EPYC 9V74 (16c/32t) and the laptop measurements; per the repo's measurement rule they are peers, named by hardware. The mechanism (the work-size policy, the grain floor, the disjoint-window safety) and correctness are verified, and the policy decisions are reproduced in unit tests; no full ORT A/B sweep was re-run on the laptop, so the headline speedup magnitudes are the author's, not independently re-measured there. ## Rebase and convergence (2026-08-18) #1201/#1202/#1207 and #1143 were **squash-merged**, so this branch's ancestry no longer reached `main`. Its three commits were cherry-picked onto `main` at `c55a3fab3` and applied cleanly. The diff is now self-contained: 2 files, +355/-25 (`matmul_nbits.rs` and the benchmark doc). It is **not** stacked on #1232 any more. Section numbering: #1232 lands first and takes §34/Phase 14, so this PR's section was renumbered to **§35/Phase 15**. (The `34.1×` figure in the §35.4 matrix is a speed ratio, not a section reference, and is unchanged.) Re-validated on the rebased head (rustc 1.97.1, the toolchain CI resolves): * `cargo test -p onnx-runtime-ep-cpu --lib` — **1433 passed, 0 failed, 17 ignored** (on top of #1232) * `cargo clippy -p onnx-runtime-ep-cpu --all-targets` — clean * `cargo fmt --all -- --check` — clean * under `--features mlas`, the parity test that actually guards this change, `mlas_prefill_parallel_dispatch_matches_serial`, **passes** ### The two `--features mlas` failures are pre-existing on `main` Running with `--features mlas` fails two tests: `feature_default_guard::mlas_is_not_a_default_feature` (it panics *because* `--features mlas` was passed explicitly) and `kernels::simd_activations::mlas_ab::mlas_matches_rust_simd_on_special_values` (a 1-ULP Erf disagreement between the MLAS and pure-Rust SIMD routes). **Both reproduce identically on `main` at `c55a3fab3` with this PR's changes absent**, so they are baseline-equivalent, not regressions, and they are out of scope here. Neither runs on any required lane: `mlas` is not a default feature. ### The red-criterion comment is runner noise The criterion report flags regressions including `tokenization/decode_tokens_per_second` -47.6%. This diff cannot reach the tokenizer, and every changed line of `matmul_nbits.rs` is inside the `#[cfg(feature = "mlas")]` `run_mlas_shards` path, which the benchmark build (default features) does not compile in. The default-feature build is behaviourally identical to `main`; the deltas are shared-runner variance. ### CI The repository's Actions queue is saturated (every recent run is `queued`, nothing `in_progress`), so the two required checks — `Fast (Linux x86_64)` and `Rust quality` — cannot report. Both lanes were reproduced locally, step for step, from `.github/workflows/ci.yml`. `main` itself fails **both** of them today; #1346 is the fix for that and lands first. Closes #1238. Working as sebastian (CPU perf). --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

codecov · 2026-08-19T12:42:43Z

Codecov Report

❌ Patch coverage is 96.10390% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.60%. Comparing base (06c62e0) to head (f375f25).
⚠️ Report is 77 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/onnx-runtime-ep-cpu/src/decode_affinity.rs	96.10%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1232      +/-   ##
==========================================
- Coverage   80.88%   80.60%   -0.28%     
==========================================
  Files         364      364              
  Lines      160729   160888     +159     
  Branches   160729   160888     +159     
==========================================
- Hits       130005   129691     -314     
- Misses      26069    26540     +471     
- Partials     4655     4657       +2

Flag	Coverage Δ
mlas	`85.22% <ø> (ø)`
offline	`80.52% <96.10%> (-0.29%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
crates/onnx-runtime-ep-cpu/src/decode_affinity.rs	`89.08% <96.10%> (+1.28%)`	⬆️

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@sebastian

#1244) ## What this is The per-`Run` cost of dispatching a **single-node fused subgraph** through the plugin path, with the kernel removed from the picture. On a 4 Ki f32 elementwise node our arm spends ~3.5 µs per `Run` where plain ORT spends ~2.6 µs, and the kernels themselves account for well under a microsecond of that — the same kernel on a 1 Mi tensor runs at 0.51x–1.03x of ORT. The difference is fixed dispatch overhead, and it is why every cheap elementwise op sits at 1.1x–1.5x of ORT at 4 Ki while the identical kernel wins at 1 Mi. This PR removes four pieces of that overhead. No kernel is touched, no numerics change, and nothing about assignment or execution ownership changes: the CPU EP still executes every node it claims, locally, with no ORT CPU fallback anywhere. ## The four cuts **1. `device_mem_info` is no longer resolved eagerly (6 ORT FFI calls per `Run`).** `compute_execute` resolved `scratch_mem_info` at the top of *every* call: ```rust let scratch_mem_info = unsafe { device_mem_info(api_ref, kernel_context, exported.device_staging.as_ref()) }; ``` `device_mem_info` calls `KernelContext_GetInputCount`, then for each input `KernelContext_GetInput` + `GetTensorMemoryInfo` + `GetMemoryInfoDeviceType`, then falls through to two more calls for input 0. For a one-input node that is **six FFI calls**. Who reads it? Two consumers. `intermediate_scratch` — routed multi-node path only. And `PlacementSources::subgraph_fallback`, which `operand_mem_info` reads **only when the node binds no ORT operands at all**, and which `prepare_workspace` only reaches past its zero-byte and lifetime gates. An elementwise node has operands and needs no workspace, so it reached neither. Now the routed path resolves it once per `Run` (unchanged) and passes `SubgraphFallback::Resolved`; the single-node path passes `SubgraphFallback::Deferred(staging)` and `operand_mem_info` calls `device_mem_info` itself, with the same arguments, if it ever actually needs the answer. Deferring is safe: the function only *reads* input memory info, so running it after outputs are allocated cannot see a different answer. **2. `allocate_output` takes `want_mem_info` (1 ORT FFI call per output per `Run`).** `OwnedOutput::mem_info` has exactly one consumer in the tree — `stage_host_boundary_inputs`, called under `if let Some(staging) = exported.device_staging.as_ref()`. A host EP has no staging context, so every `Run` made a `GetTensorMemoryInfo` call per output and dropped the result. Both call sites now pass `exported.device_staging.is_some()`, so a device EP is bit-for-bit unchanged and a host EP stops making the call. **3. `staging_log(&format!(..))` → `staging_log!(..)` (one `String` per `Run`).** `staging_log` checks `ONNX_GENAI_PLUGIN_TRANSFER_TRACE` *inside* the function, so the `format!` was evaluated whether or not the trace was on — and one of these sites is at the top of `compute_execute`, formatting three fields into a heap `String` on every dispatch. The macro checks the gate first. All 12 sites converted, so the footgun is gone rather than papered over at one site. **4. Absent-output bookkeeping is built only when there are absent outputs (5 allocations per `Run`).** ```rust let absent_shapes: Vec<Vec<usize>> = output_shapes.clone(); let absent_strides_storage: Vec<Vec<i64>> = absent_shapes.iter().map(|s| contiguous_strides(s)).collect(); let mut ort_views: Vec<TensorMut<'_>> = owned_outputs.iter_mut().map(|o| o.view_mut()).collect(); let mut ort_view_iter = ort_views.drain(..); ``` That storage exists solely to back the `TensorMut`s of *absent* output slots. A node with no absent outputs — every elementwise op — cloned every output shape and built a stride vector per output to lend them to nobody. It is now conditional on `!absent_bufs.is_empty()`. The `ort_views` `Vec` was collected and immediately drained; the iterator is taken directly from `owned_outputs.iter_mut()`. Related: placement operands are now described by an `OrtOperands` enum, so the single-node path lends `entry.input_slots` (`&[Option<usize>]`, flattened lazily) instead of collecting a fresh `Vec<usize>` per call for a consumer that almost never runs. The routed path passes its already-resolved slice. Per `Run` on a 1-in/1-out elementwise node this is **7 fewer ORT FFI calls** (16 → 9) and **7 fewer heap allocations**. ## A/B, one thread `taskset -c 8-15`, this commit vs its parent, **interleaved** (A,B,A,B,A,B with a rebuild before each arm, so drift hits both arms equally), 400 iterations after 50 warmup, three rounds. Ratio is **ours/ORT, lower is better**; the p50 column is the median of the three rounds' p50 ratios, p90 likewise. | case | before p50 | after p50 | before p90 | after p90 | |---|---|---|---|---| | `sqrt_f32_4k` | 1.132 | **1.026** | 1.153 | **1.033** | | `sigmoid_f32_4k` | 1.484 | **1.346** | 1.517 | **1.362** | | `tanh_f32_4k` | 1.534 | **1.406** | 1.595 | **1.434** | | `erf_f32_4k` | 1.760 | **1.510** | 1.779 | **1.512** | Best-of-three (the contention-robust statistic on this shared box) agrees: sqrt 1.130 → 1.023, sigmoid 1.481 → 1.344, tanh 1.503 → 1.387, erf 1.570 → 1.480. 12 of 12 arm-pairs favour the change; there is no round in which any case regressed. In absolute terms `tanh_f32_4k` goes 0.0046 ms → 0.0042 ms, i.e. **~0.4 µs off a ~0.9 µs gap**. ## A/B, threaded Same protocol, `taskset -c 0-15`, two rounds, `NXRT_MM_BENCH_THREADS` = `ONNX_GENAI_MLAS_THREADPOOL_THREADS` = `RAYON_NUM_THREADS`. | case | 4t before | 4t after | 16t before | 16t after | |---|---|---|---|---| | `sqrt_f32_4k` | 1.257 | **1.129** | 1.262 | **1.156** | | `sigmoid_f32_4k` | 1.520 | **1.355** | 1.499 | **1.340** | | `tanh_f32_4k` | 1.551 | **1.418** | 1.548 | **1.455** | | `erf_f32_4k` | 1.776 | **1.671** | 1.777 | **1.624** | The overhead is per call, not per element or per worker, so the gain is the same absolute number of microseconds at every thread count. ## Drift control: the 1 Mi grid is unchanged Same protocol, `_f32_1m`, three interleaved rounds, one thread. A large tensor amortises the per-call cost away, so these must *not* move — and they don't: | case | before | after | |---|---|---| | `relu_f32_1m` | 1.032 | 1.034 | | `exp_f32_1m` | 1.021 | 1.022 | | `sigmoid_f32_1m` | 1.066 | 1.063 | | `tanh_f32_1m` | 1.117 | 1.115 | | `gelu_tanh_f32_1m` | 1.243 | 1.241 | | `gelu_exact_f32_1m` | 1.424 | 1.420 | | `fastgelu_f32_1m` | 1.239 | 1.224 | | `erf_f32_1m` | 1.461 | 1.439 | | `quickgelu_f32_1m` | 0.809 | 0.811 | | `sqrt_f32_1m` | 0.514 | 0.511 | Ten of ten within ±0.5 %, which is this box's noise floor. That is the shape of a per-call fix. ## Correctness **New test, with a verified falsifier.** `output_memory_info_is_queried_only_when_the_caller_asked_for_it` drives `allocate_output` against a hand-built `OrtApi` whose `GetTensorMemoryInfo` counts its calls, and asserts 0 calls for `want_mem_info: false` and 1 for `true`. Falsifier: making `allocate_output` ignore the flag fails it with `left: 1, right: 0` — checked by breaking the code, not by inspection. **Behaviour preserved, argued per cut.** (1) `device_mem_info` is called with identical arguments, only later and only when read; it reads inputs, which output allocation cannot change. (2) The gate on the memory-info query is *the same condition* as the gate on its only consumer. (3) The macro's only difference is when the `format!` runs. (4) The absent storage is only ever indexed for absent slots. **Suites.** `-p onnx-runtime-ep-plugin`: 247 unit tests pass (246 before, +1 new). `-p onnx-runtime-ep-cpu-plugin` with `NXRT_REQUIRE_ORT_TESTS=1`: the full e2e suite passes, including all 54 `plugin_ort_e2e` cases — the routed multi-node fixtures (`conformance_chain_add_mul`, `..._repeated_runs_do_not_leak_stale_intermediates`, `conformance_mixed_partition`) exercise the `SubgraphFallback::Resolved` arm, and `every_assigned_node_is_also_executed_by_this_ep` passes, so every node this EP claims is still executed here with ORT CPU fallback disabled. `cargo fmt --all` and `cargo clippy --release --all-targets -p onnx-runtime-ep-cpu -p onnx-runtime-ep-cpu-plugin -p onnx-runtime-ep-plugin` are clean. **Build identity.** Pure native CPU EP: no MLAS at runtime, no ORT CPU EP fallback, no new dependency. AVX2/FMA host (`avx2 fma f16c`, no AVX-512), so ORT/MLAS and we are on the same instruction footing. ## What is left The remaining ~0.5 µs is, as far as I can attribute it without CPU counters (`perf_event_paranoid` is 4 on this box, so `perf record` is not available and everything here is A/B attribution): * **~9 ORT FFI calls that are genuinely needed.** `read_inputs` costs 7 per input — `KernelContext_GetInput`, `GetTensorTypeAndShape` (which allocates an ORT object we then release), `GetTensorElementType`, `GetDimensionsCount`, `GetDimensions`, `ReleaseTensorTypeAndShapeInfo`, `GetTensorData` — and there is no cheaper spelling in the stable C API. ORT's own CPU kernels reach the same data through `OpKernelContext` with no FFI at all, which is a structural part of what a plugin EP pays. * **~7 remaining allocations**: `OwnedInput`'s shape and strides per input, `kernel_inputs`, `infer_shapes`'s `Vec<Vec<usize>>`, `slot_map`, `output_views`, `prepare_workspace`'s metadata, `allocate_output`'s dims, and the `Box<HostPool>` in `host_pool::install`. Each is worth ~25–35 ns. Removing them needs either an inline-capacity vector type or per-session caching of the parts that cannot change between `Run`s; both are worth doing and neither belongs in this PR. I deliberately did **not** touch `host_pool::install` — the per-call `Box` is one allocation, and that file is @sebastian's 16-thread scheduling work; a change there should come from him or after his PRs land. --- ## Refreshed against `main` (2026-08-18) The branch was behind `main` and its whole red CI wall came from that, not from this change: `crates/onnx-runtime-session/src/executor/mod.rs:175` failed `-D dead-code` on current stable, which `ca32b3adf` ("fix(ci): unbreak the Rust quality lane on current stable", #1239) fixed on `main` after this branch forked. `origin/main` (`c55a3fab3`) is merged in — no rebase, no force-push — and the diff this PR owns is unchanged at 2 files, +285/-79. Revalidated on the merge commit, AVX2/FMA host, no AVX-512: * `cargo test --release -p onnx-runtime-ep-plugin` — **247 passed, 0 failed**. * `NXRT_REQUIRE_ORT_TESTS=1 cargo test --release -p onnx-runtime-ep-cpu-plugin` — every suite green, including all **55** `plugin_ort_e2e` cases. The ones that matter to this change all pass on the merge: `every_assigned_node_is_also_executed_by_this_ep`, `no_supported_node_is_ever_left_to_the_ort_cpu_ep`, `no_matmul_family_node_escapes_to_the_ort_cpu_ep`, and `every_fixture_loads_with_cpu_fallback_disabled` — so assigned still equals executed with ORT CPU fallback off. * `cargo fmt` clean for the crates this PR touches. The one `cargo fmt --all` hunk on this tree is in `onnx-runtime-ep-cuda/src/kernels/standard_attention.rs`, which arrived from `main` untouched by this PR and is a local rustfmt-version difference, not a branch defect. ## Independent review Reviewed by **Claude Opus 4.8**, read-only, with the four cuts and the absent-slot history stated as the priority list. Verdict **APPROVE**, no blockers. It independently confirmed: * `SlotKind::Absent(idx)` is pushed into `slot_map` only in the same branch that pushes into `absent_bufs`, so `has_absent == false` implies `slot_map` holds no `Absent` and the skipped storage is never indexed; and the surviving indices are the same full-slot indices as before. * `ort_view_iter` from `owned_outputs.iter_mut()` has the identical borrow structure as the old `collect()` + `drain(..)`, so nothing borrows a temporary. * `OrtOperands::Slots(..).indices()` yields the same elements in the same order as the removed `.iter().flatten().copied().collect()`, `None` skipped. * `operand_mem_info` is the only reader of the deferred value, only reachable when the node binds **zero** ORT inputs, which makes the timing of the deferred `device_mem_info` moot rather than merely argued. * Both `allocate_output` call sites gate on the same predicate as the only reader of `OwnedOutput::mem_info`, so a device EP is unchanged. Its one correction is applied above: the prose said 14 `staging_log` sites; the real count in `origin/main` is **12**, and all 12 are converted. --- ## Refreshed against `main` @ `6a855d5e0`, and measured as a stack `origin/main` moved a long way while this sat in the CI queue (#1346, #1352 and #1361 on the quality lane; #1154, #1232, #1238 on the CPU side). Merged in normally — no rebase — and re-measured from scratch against the new baseline. Production pure-native A/B, plain ORT as the control arm. No MLAS, no ORT CPU fallback, no deferral. `taskset -c 8-15`, one thread, 400 iterations, five interleaved rounds out of two worktrees, started only once cores 8-15 were >=93% idle. **Ratio is ours/ORT, lower is better.** `before` is `main` at `6a855d5e0`; `after` is #1244 + #1246 together, since #1246 is stacked on #1244 and the pair is what a user gets. | case | ratio p50 main | ratio p50 stack | Δ | ratio p90 main | ratio p90 stack | ours us | ORT drift | rounds won | |---|---|---|---|---|---|---|---|---| | `thresholdedrelu_f32_4k` | 1.512 | **1.216** | -19.6% | 1.519 | **1.216** | 3.6 → **2.8** | -4.2% | 5/5 | | `tanh_f32_4k` | 1.511 | **1.279** | -15.4% | 1.513 | **1.284** | 4.6 → **3.9** | +0.0% | 5/5 | | `sigmoid_f32_4k` | 1.475 | **1.248** | -15.4% | 1.484 | **1.256** | 4.7 → **4.0** | +0.0% | 5/5 | | `erf_f32_4k` | 1.470 | **1.363** | -7.3% | 1.489 | **1.364** | 7.5 → **7.0** | +0.0% | 5/5 | | `hardsigmoid_f32_4k` | 1.416 | **1.127** | -20.4% | 1.426 | **1.133** | 3.5 → **2.8** | +0.0% | 5/5 | | `leakyrelu_f32_4k` | 1.361 | **1.094** | -19.6% | 1.371 | **1.104** | 3.5 → **2.8** | +0.0% | 5/5 | | `sqrt_f32_4k` | 1.141 | **0.946** | -17.1% | 1.150 | **0.955** | 4.0 → **3.4** | -2.8% | 5/5 | | `log_f32_4k` | 0.767 | **0.698** | -9.0% | 0.776 | **0.703** | 7.9 → **7.2** | +0.0% | 5/5 | | `selu_f32_4k` | 0.505 | **0.440** | -12.9% | 0.512 | **0.444** | 5.6 → **4.9** | +0.0% | 5/5 | | `elu_f32_4k` | 0.480 | **0.416** | -13.3% | 0.488 | **0.421** | 5.3 → **4.6** | +0.0% | 5/5 | | `celu_f32_4k` | 0.470 | **0.411** | -12.6% | 0.478 | **0.418** | 5.7 → **5.0** | +0.0% | 5/5 | | `mish_f32_4k` | 0.276 | **0.264** | -4.3% | 0.278 | **0.268** | 17.3 → **16.6** | +0.0% | 5/5 | **Every case, every round.** The two rows with a moving control (`sqrt` -2.8%, `thresholdedrelu` -4.2%) are reported rather than dropped; both won 5/5 anyway and their absolute time fell by the same ~0.7 us as everything else. That constant ~0.7 us is the point. It is not proportional to tensor size — the same absolute amount comes off `hardsigmoid` (3.5 -> 2.8 us) as off `mish` (17.3 -> 16.6 us) — which is what a fixed per-`Run` cost looks like when you remove some of it. It moves the cheap ops the most because they had the least to hide it behind, and `sqrt` crosses from 1.141 to **0.946**, from a loss to a win. ### Where the remaining time goes Measured directly, by instrumenting `compute_execute` segment by segment on top of this stack (temporary probe, not committed; `perf` is unavailable on this host — `perf_event_paranoid=4`). Per `Run`, one-in/one-out elementwise node, 4096 `f32`, microseconds: | segment | us | note | |---|---|---| | `KernelContext_GetOutput` | 0.35 | ORT's own API — ours to call, not to optimise | | `read_inputs` | 0.15 | 4 ORT FFI calls, already one shape call after #1246 | | rest of `allocate_output` | 0.13 | `GetTensorMutableData` + strides | | `prepare_workspace` | 0.09 | metadata vector + plan-cache lookup, for a kernel needing 0 bytes | | `host_pool::install` | 0.05 | @sebastian's, not touched | | `infer_shapes` | 0.05 | | | `kernel_inputs` | 0.04 | | | `output_views` | 0.04 | | Non-kernel node cost is **~1.25 us and near-constant across all twelve operators** (0.28 to 15.2 us of kernel time), which is the direct confirmation that small-node ratios on this EP are dispatch-bound rather than kernel-bound. There is no single large item left — the biggest, `KernelContext_GetOutput`, is ORT's. The rest is a long tail of 0.04-0.15 us items, which is what #1358 (`InlineVec`) starts on. ### And nothing breaks at 1 Mi Same harness, 1048576 elements, 120 iterations, 3 rounds. A fixed per-`Run` cost should be invisible here, and it is: | case | ratio p50 main | ratio p50 stack | ours us | ORT drift | |---|---|---|---|---| | `celu_f32_1m` | 0.141 | 0.140 | 382.1 → 377.9 | -0.0% | | `elu_f32_1m` | 0.138 | 0.136 | 347.7 → 343.6 | +0.1% | | `erf_f32_1m` | 0.671 | 0.670 | 595.2 → 594.6 | +0.1% | | `exp_f32_1m` | 0.606 | 0.590 | 247.0 → 240.5 | +0.1% | | `fastgelu_f32_1m` | 0.643 | 0.648 | 415.3 → 413.2 | -1.7% | | `gelu_exact_f32_1m` | 0.572 | 0.592 | 706.1 → 710.6 | -2.8% ⚠ | | `gelu_tanh_f32_1m` | 0.657 | 0.651 | 414.8 → 410.5 | +0.3% | | `hardsigmoid_f32_1m` | 0.376 | 0.372 | 88.0 → 69.6 | -0.9% | | `leakyrelu_f32_1m` | 0.421 | 0.430 | 90.1 → 87.3 | -2.2% ⚠ | | `log_f32_1m` | 0.270 | 0.274 | 616.4 → 612.4 | +1.7% | | `mish_f32_1m` | 0.105 | 0.105 | 1626.0 → 1626.0 | -0.2% | | `quickgelu_f32_1m` | 0.455 | 0.453 | 321.5 → 320.0 | -0.5% | | `relu_f32_1m` | 1.034 | 1.022 | 131.7 → 130.2 | +0.0% | | `selu_f32_1m` | 0.147 | 0.148 | 374.1 → 371.8 | -1.5% | | `sigmoid_f32_1m` | 0.471 | 0.606 | 231.6 → 229.0 | -23.4% ⚠ | | `sqrt_f32_1m` | 0.314 | 0.302 | 148.1 → 143.9 | +1.1% | | `tanh_f32_1m` | 0.644 | 0.631 | 226.2 → 221.9 | -0.2% | | `thresholdedrelu_f32_1m` | 0.500 | 0.486 | 70.9 → 68.6 | -0.3% | Flat, as predicted — 0.7 us against 70-1626 us of work. Absolute time is equal or better in 16 of 18 cases. The two ⚠ rows had the control move more than the effect: `sigmoid` is unusable (ORT itself moved -23.4%; our own absolute went 231.6 -> 229.0 us), and `gelu_exact`'s +0.6% absolute sits inside its -2.8% control. Reported rather than dropped. This is the coverage claim for the change: it buys ~0.7 us at every size, which is 20% of a small node and nothing at all of a large one, and it costs nothing anywhere. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@sebastian

) ## What this is `read_inputs` runs **once per input per `Run`**, and it asked ORT for the element type and the dimensions through the classic five-call sequence: ``` GetTensorTypeAndShape → GetTensorElementType → GetDimensionsCount → GetDimensions → ReleaseTensorTypeAndShapeInfo ``` The first of those makes ORT **heap-allocate** an `OrtTensorTypeAndShapeInfo` and copy the shape into it; the last frees it. Five FFI crossings and one allocation on ORT's side of the boundary, per input, per `Run`, to read data the `OrtValue` already owns. `GetTensorElementTypeAndShapeDataReference` (ORT C API, since 1.24) returns the element type **and a reference to the value's own shape array** in one call, allocating nothing. This PR routes `read_inputs` through it. Stacked on #1244 (since merged; this branch now targets `main` directly). No kernel is touched, no numerics change, and node assignment and execution ownership are unchanged: the CPU EP still executes every node it claims, locally, with ORT CPU fallback off. ## The three cuts **1. Five ORT calls and one ORT-side allocation per input → one call, no allocation.** The plugin already fails closed below API 27, so the hook is always present in practice. The five-call sequence is kept as a fallback for a host that leaves it null, and a host offering **neither** now fails closed with a message naming both — it does not silently read garbage shapes. The borrowed pointer is used only to build the owned `shape`/`strides` of the `OwnedInput`; nothing retains it. ORT spells a scalar as a **null** pointer with count 0, and `slice::from_raw_parts` is UB on null, so that case is guarded explicitly — see *Correctness* below for how that guard is actually enforced. **2. `validate_dims` takes a lazily-formatted label, not `&str`.** Every call site passed `&format!("input {i}")`, i.e. a heap `String` built on the success path purely to label an error that almost never happens. Callers now pass `format_args!(..)` and the string is only materialised inside the error branches. One `String` per input per `Run` gone. All call sites converted, including `transfer.rs` and the unit tests; no message text changed. (The merge with `main` took main's `impl std::fmt::Display` signature here, which is a superset of this branch's original `Arguments<'_>` and keeps both callers working unchanged.) **3. `allocate_output` converts the output shape to `i64` on the stack.** `let dims: Vec<i64> = shape.iter().map(|&d| d as i64).collect()` was a heap allocation per output per `Run` whose entire lifetime was the `KernelContext_GetOutput` call below it. Rank ≤ 8 now goes to a stack array; a taller tensor still works, through the `Vec`. For a 1-in/1-out elementwise node that is **5 fewer ORT FFI calls**, one fewer ORT-side allocation, and **2 fewer heap allocations** per `Run`, on top of #1244. ## A/B, one thread `taskset -c 8-15`, this branch vs its merge parent (`origin/main` + #1244), **interleaved** (A,B,A,B,… so drift hits both arms equally), 400 iterations after 50 warmup, **five rounds**. Ratio is **ours/ORT, lower is better**; each column is the median of the five rounds. "ORT drift" is the control: ORT's own p50 measured in the two arms, which must not move for the comparison to mean anything. | case | before p50 | after p50 | before p90 | after p90 | ORT drift | rounds won | |---|---|---|---|---|---|---| | `sqrt_f32_4k` | 1.023 | **0.952** | 1.031 | **0.964** | +2.9 % | 4/5 | | `leakyrelu_f32_4k` | 1.202 | **1.097** | 1.210 | **1.104** | 0.0 % | 5/5 | | `hardsigmoid_f32_4k` | 1.213 | **1.144** | 1.226 | **1.149** | 0.0 % | 5/5 | | `thresholdedrelu_f32_4k` | 1.316 | **1.209** | 1.314 | **1.220** | 0.0 % | 5/5 | | `sigmoid_f32_4k` | 1.328 | **1.252** | 1.333 | **1.258** | 0.0 % | 5/5 | | `tanh_f32_4k` | 1.368 | **1.287** | 1.369 | **1.291** | 0.0 % | 5/5 | | `erf_f32_4k` | 1.383 | **1.327** | 1.387 | **1.329** | 0.0 % | 5/5 | | `celu_f32_4k` | 0.430 | **0.411** | 0.435 | **0.418** | 0.0 % | 4/5 | | `elu_f32_4k` | 0.437 | **0.416** | 0.443 | **0.422** | −1.8 % | 4/5 | | `selu_f32_4k` | 0.467 | **0.441** | 0.471 | **0.446** | 0.0 % | 5/5 | | `log_f32_4k` | 0.727 | **0.700** | 0.728 | **0.706** | 0.0 % | 5/5 | | `mish_f32_4k` | 0.268 | **0.264** | 0.271 | **0.266** | 0.0 % | 4/5 | 12 of 12 cases improve at p50 and p90; 56 of 60 individual arm-pairs favour the change, and no case regressed in its median. In absolute terms it is **0.2–0.3 µs off every `Run`**: `tanh_f32_4k` 4.20 µs → 3.90 µs against ORT's 3.10 µs, `thresholdedrelu_f32_4k` 3.10 µs → 2.80 µs against ORT's 2.30 µs. That is the shape of a fixed per-call cost being removed, which is what it is. `sqrt_f32_4k` crosses below 1.00 — the plugin path now dispatches that op faster than plain ORT does. ## A/B, four threads Same protocol, `NXRT_MM_BENCH_THREADS` = `ONNX_GENAI_MLAS_THREADPOOL_THREADS` = `RAYON_NUM_THREADS` = 4, three rounds. | case | before p50 | after p50 | before p90 | after p90 | |---|---|---|---|---| | `leakyrelu_f32_4k` | 1.069 | **0.966** | 1.075 | **0.976** | | `hardsigmoid_f32_4k` | 1.152 | **1.050** | 1.164 | **1.060** | | `thresholdedrelu_f32_4k` | 1.212 | **1.091** | 1.228 | **1.106** | | `sqrt_f32_4k` | 1.121 | **1.036** | 1.130 | **1.045** | | `sigmoid_f32_4k` | 1.346 | **1.264** | 1.359 | **1.263** | | `tanh_f32_4k` | 1.434 | **1.319** | 1.453 | **1.325** | | `erf_f32_4k` | 1.581 | **1.502** | 1.588 | **1.510** | | `celu_f32_4k` | 0.460 | **0.437** | 0.476 | **0.445** | | `elu_f32_4k` | 0.481 | **0.448** | 0.486 | **0.468** | | `selu_f32_4k` | 0.500 | **0.458** | 0.482 | **0.464** | | `log_f32_4k` | 0.766 | **0.731** | 0.781 | **0.734** | | `mish_f32_4k` | 0.332 | **0.322** | 0.345 | **0.339** | 36 of 36 arm-pairs favour the change. The cost is per call, not per element or per worker, so the gain is the same absolute microseconds at every thread count. ## Drift control: the 1 Mi grid does not move Same protocol, `_f32_1m`, 200 iterations after 30 warmup, five interleaved rounds, one thread. A 1 Mi tensor amortises a fixed per-call cost away, so **these must not move** — and they don't: | case | before | after | | case | before | after | |---|---|---|---|---|---|---| | `relu_f32_1m` | 1.024 | 1.028 | | `sqrt_f32_1m` | 0.313 | 0.309 | | `exp_f32_1m` | 0.580 | 0.576 | | `log_f32_1m` | 0.264 | 0.258 | | `sigmoid_f32_1m` | 0.588 | 0.584 | | `mish_f32_1m` | 0.100 | 0.099 | | `tanh_f32_1m` | 0.618 | 0.620 | | `celu_f32_1m` | 0.127 | 0.129 | | `erf_f32_1m` | 0.651 | 0.668 | | `elu_f32_1m` | 0.129 | 0.129 | | `gelu_exact_f32_1m` | 0.585 | 0.584 | | `selu_f32_1m` | 0.136 | 0.138 | | `gelu_tanh_f32_1m` | 0.602 | 0.646 | | `hardsigmoid_f32_1m` | 0.455 | 0.453 | | `fastgelu_f32_1m` | 0.608 | 0.639 | | `leakyrelu_f32_1m` | 0.409 | 0.417 | | `quickgelu_f32_1m` | 0.429 | 0.438 | | `thresholdedrelu_f32_1m` | 0.567 | 0.573 | The per-case round-win counts here are 0/5–4/5 with a median of 2/5, i.e. a coin flip — the signature of noise, not of an effect. The two largest movers, `gelu_tanh` (+7 %) and `fastgelu` (+5 %), are the two cases whose **ORT** side also drifted most in the same rounds (−4.4 % and −2.8 %), so the ratio moved because the denominator did. This host's floor is roughly ±5 % on the 1 Mi grid and I am not claiming anything below it. ## Correctness **Five new tests, three with a verified falsifier**, driving `read_inputs` and `allocate_output` against a hand-built `OrtApi` whose hooks count their calls. | test | what it pins | falsifier | |---|---|---| | `input_shapes_come_from_one_call_when_ort_offers_the_reference_hook` | 1 reference call, **0** legacy calls, and the shape/strides/dtype that come out | forcing the legacy route fails it with `left: 0, right: 1` (run) | | `the_five_call_fallback_produces_the_same_input_as_the_reference_hook` | the fallback is exercised and its `OwnedInput` matches the reference path field for field | — (it *is* the parity check) | | `a_borrowed_scalar_shape_never_dereferences_null` | ORT's documented scalar spelling — null pointer, count 0 — takes the borrowed route and yields rank 0 | deleting the null guard makes **Miri** report `out-of-bounds pointer use: null pointer is a dangling pointer` at `kernel_ctx.rs:280` (run) | | `read_inputs_fails_closed_when_no_shape_route_exists` | the error names **both** routes; no silent garbage shapes | — | | `output_dims_are_identical_on_the_inline_and_heap_ranks` | ranks 0/1/8/9/12 all reach ORT with every dimension intact | dropping the heap arm delivers rank 9 truncated to 8 dims (run) | **The null guard is now actually enforced, not just asserted.** Natively, `from_raw_parts(null, 0)` returns an empty slice, so a test asserting the resulting shape passes with or without the guard — the guarantee only exists if Miri sees it. `onnx-runtime-ep-plugin` was not in the Miri matrix, so this PR adds `kernel_ctx::` as a lane in `.github/workflows/miri.yml` (the crate's other modules dlopen ORT and are not Miri-tractable, which is why the lane is scoped to the module rather than the crate). 25 tests pass under Miri in 4.15 s; with the guard deleted the lane fails. That pairing is what makes the test non-vacuous. **Suites.** `-p onnx-runtime-ep-plugin`: **252** unit tests pass (247 before, +5 new). `-p onnx-runtime-ep-cpu-plugin` with `NXRT_REQUIRE_ORT_TESTS=1`: every suite green, including all **55** `plugin_ort_e2e` cases — `every_assigned_node_is_also_executed_by_this_ep`, `no_supported_node_is_ever_left_to_the_ort_cpu_ep`, `no_matmul_family_node_escapes_to_the_ort_cpu_ep` and `every_fixture_loads_with_cpu_fallback_disabled` all pass, so **assigned still equals executed** with ORT CPU fallback disabled. `cargo clippy --release --all-targets -p onnx-runtime-ep-plugin` and `cargo fmt` are clean. **Build identity.** Pure native CPU EP: no MLAS at runtime, no ORT CPU EP fallback, no new dependency. AVX2/FMA host (`avx2 fma f16c`, no AVX-512), so ORT and we are on the same instruction footing. ## Independent review Reviewed by **Claude Opus 4.8**, read-only, briefed with the exact ORT contract for the borrowed pointer and asked specifically to hunt UB and vacuous tests. Verdict **APPROVE**, no blockers. It confirmed the null/scalar guard is unreachable-by-construction for `from_raw_parts`, that the borrow cannot outlive the `OrtValue` (its only consumer copies), that `ReleaseTensorTypeAndShapeInfo` still runs on every legacy error path — and noted that moving `DataType::from_onnx` after the match incidentally closes a **pre-existing** leak of `type_shape` on the unsupported-dtype path. It raised two test-quality defects, both real and both **fixed** in `6027f6e16`: 1. The three counting tests shared two process-wide `AtomicUsize`es while asserting exact counts, and libtest runs them in parallel — one test's reset could land inside another's assertion window. They now take a shared `SHAPE_COUNTER_LOCK` that resets both counters under the guard. 2. `a_borrowed_scalar_shape_never_dereferences_null` was **vacuous** with respect to its name, for exactly the reason above, and the crate was not under Miri. Hence the Miri lane, and the test now also asserts the scalar went through the borrowed route so it cannot pass on the fallback. It also flagged that `OrtStatus` is not released on error paths in this file. That is pre-existing and repo-wide in `kernel_ctx.rs` — the old five-call code leaked identically — so it is not touched here; it belongs in its own change. ## What is left After #1244 and this PR, a 1-in/1-out elementwise `Run` is down to roughly `KernelContext_GetInputCount` + `GetInput` + the one shape reference + `GetTensorData` + `GetOutput` + `GetTensorMutableData`. What remains: * **`OwnedInput`'s `shape` and `strides` `Vec`s**, `kernel_inputs`, `infer_shapes`'s `Vec<Vec<usize>>`, `slot_map`, `output_views`, and the `Box<HostPool>` in `host_pool::install` — each worth ~25–35 ns. Removing them needs an inline-capacity vector type or per-session caching of the parts that cannot change between `Run`s. Both are worth doing; neither belongs here. * **The `HostPool` box** specifically is @sebastian's 16-thread scheduling file and should come from him or after his PRs land. --- ## Refreshed against `main` @ `6a855d5e0`, and measured as a stack `origin/main` moved a long way while this sat in the CI queue (#1346, #1352 and #1361 on the quality lane; #1154, #1232, #1238 on the CPU side). Merged in normally — no rebase — and re-measured from scratch against the new baseline. Production pure-native A/B, plain ORT as the control arm. No MLAS, no ORT CPU fallback, no deferral. `taskset -c 8-15`, one thread, 400 iterations, five interleaved rounds out of two worktrees, started only once cores 8-15 were >=93% idle. **Ratio is ours/ORT, lower is better.** `before` is `main` at `6a855d5e0`; `after` is #1244 + #1246 together, since #1246 is stacked on #1244 and the pair is what a user gets. | case | ratio p50 main | ratio p50 stack | Δ | ratio p90 main | ratio p90 stack | ours us | ORT drift | rounds won | |---|---|---|---|---|---|---|---|---| | `thresholdedrelu_f32_4k` | 1.512 | **1.216** | -19.6% | 1.519 | **1.216** | 3.6 → **2.8** | -4.2% | 5/5 | | `tanh_f32_4k` | 1.511 | **1.279** | -15.4% | 1.513 | **1.284** | 4.6 → **3.9** | +0.0% | 5/5 | | `sigmoid_f32_4k` | 1.475 | **1.248** | -15.4% | 1.484 | **1.256** | 4.7 → **4.0** | +0.0% | 5/5 | | `erf_f32_4k` | 1.470 | **1.363** | -7.3% | 1.489 | **1.364** | 7.5 → **7.0** | +0.0% | 5/5 | | `hardsigmoid_f32_4k` | 1.416 | **1.127** | -20.4% | 1.426 | **1.133** | 3.5 → **2.8** | +0.0% | 5/5 | | `leakyrelu_f32_4k` | 1.361 | **1.094** | -19.6% | 1.371 | **1.104** | 3.5 → **2.8** | +0.0% | 5/5 | | `sqrt_f32_4k` | 1.141 | **0.946** | -17.1% | 1.150 | **0.955** | 4.0 → **3.4** | -2.8% | 5/5 | | `log_f32_4k` | 0.767 | **0.698** | -9.0% | 0.776 | **0.703** | 7.9 → **7.2** | +0.0% | 5/5 | | `selu_f32_4k` | 0.505 | **0.440** | -12.9% | 0.512 | **0.444** | 5.6 → **4.9** | +0.0% | 5/5 | | `elu_f32_4k` | 0.480 | **0.416** | -13.3% | 0.488 | **0.421** | 5.3 → **4.6** | +0.0% | 5/5 | | `celu_f32_4k` | 0.470 | **0.411** | -12.6% | 0.478 | **0.418** | 5.7 → **5.0** | +0.0% | 5/5 | | `mish_f32_4k` | 0.276 | **0.264** | -4.3% | 0.278 | **0.268** | 17.3 → **16.6** | +0.0% | 5/5 | **Every case, every round.** The two rows with a moving control (`sqrt` -2.8%, `thresholdedrelu` -4.2%) are reported rather than dropped; both won 5/5 anyway and their absolute time fell by the same ~0.7 us as everything else. That constant ~0.7 us is the point. It is not proportional to tensor size — the same absolute amount comes off `hardsigmoid` (3.5 -> 2.8 us) as off `mish` (17.3 -> 16.6 us) — which is what a fixed per-`Run` cost looks like when you remove some of it. It moves the cheap ops the most because they had the least to hide it behind, and `sqrt` crosses from 1.141 to **0.946**, from a loss to a win. ### Where the remaining time goes Measured directly, by instrumenting `compute_execute` segment by segment on top of this stack (temporary probe, not committed; `perf` is unavailable on this host — `perf_event_paranoid=4`). Per `Run`, one-in/one-out elementwise node, 4096 `f32`, microseconds: | segment | us | note | |---|---|---| | `KernelContext_GetOutput` | 0.35 | ORT's own API — ours to call, not to optimise | | `read_inputs` | 0.15 | 4 ORT FFI calls, already one shape call after #1246 | | rest of `allocate_output` | 0.13 | `GetTensorMutableData` + strides | | `prepare_workspace` | 0.09 | metadata vector + plan-cache lookup, for a kernel needing 0 bytes | | `host_pool::install` | 0.05 | @sebastian's, not touched | | `infer_shapes` | 0.05 | | | `kernel_inputs` | 0.04 | | | `output_views` | 0.04 | | Non-kernel node cost is **~1.25 us and near-constant across all twelve operators** (0.28 to 15.2 us of kernel time), which is the direct confirmation that small-node ratios on this EP are dispatch-bound rather than kernel-bound. There is no single large item left — the biggest, `KernelContext_GetOutput`, is ORT's. The rest is a long tail of 0.04-0.15 us items, which is what #1358 (`InlineVec`) starts on. ### And nothing breaks at 1 Mi Same harness, 1048576 elements, 120 iterations, 3 rounds. A fixed per-`Run` cost should be invisible here, and it is: | case | ratio p50 main | ratio p50 stack | ours us | ORT drift | |---|---|---|---|---| | `celu_f32_1m` | 0.141 | 0.140 | 382.1 → 377.9 | -0.0% | | `elu_f32_1m` | 0.138 | 0.136 | 347.7 → 343.6 | +0.1% | | `erf_f32_1m` | 0.671 | 0.670 | 595.2 → 594.6 | +0.1% | | `exp_f32_1m` | 0.606 | 0.590 | 247.0 → 240.5 | +0.1% | | `fastgelu_f32_1m` | 0.643 | 0.648 | 415.3 → 413.2 | -1.7% | | `gelu_exact_f32_1m` | 0.572 | 0.592 | 706.1 → 710.6 | -2.8% ⚠ | | `gelu_tanh_f32_1m` | 0.657 | 0.651 | 414.8 → 410.5 | +0.3% | | `hardsigmoid_f32_1m` | 0.376 | 0.372 | 88.0 → 69.6 | -0.9% | | `leakyrelu_f32_1m` | 0.421 | 0.430 | 90.1 → 87.3 | -2.2% ⚠ | | `log_f32_1m` | 0.270 | 0.274 | 616.4 → 612.4 | +1.7% | | `mish_f32_1m` | 0.105 | 0.105 | 1626.0 → 1626.0 | -0.2% | | `quickgelu_f32_1m` | 0.455 | 0.453 | 321.5 → 320.0 | -0.5% | | `relu_f32_1m` | 1.034 | 1.022 | 131.7 → 130.2 | +0.0% | | `selu_f32_1m` | 0.147 | 0.148 | 374.1 → 371.8 | -1.5% | | `sigmoid_f32_1m` | 0.471 | 0.606 | 231.6 → 229.0 | -23.4% ⚠ | | `sqrt_f32_1m` | 0.314 | 0.302 | 148.1 → 143.9 | +1.1% | | `tanh_f32_1m` | 0.644 | 0.631 | 226.2 → 221.9 | -0.2% | | `thresholdedrelu_f32_1m` | 0.500 | 0.486 | 70.9 → 68.6 | -0.3% | Flat, as predicted — 0.7 us against 70-1626 us of work. Absolute time is equal or better in 16 of 18 cases. The two ⚠ rows had the control move more than the effect: `sigmoid` is unusable (ORT itself moved -23.4%; our own absolute went 231.6 -> 229.0 us), and `gelu_exact`'s +0.6% absolute sits inside its -2.8% control. Reported rather than dropped. This is the coverage claim for the change: it buys ~0.7 us at every size, which is 20% of a small node and nothing at all of a large one, and it costs nothing anywhere. --- ## Status after merging latest `main` (2026-08-19) Validated on latest `main` after the six-PR #1077 stack landed (#1387, #1409, #1412, #1430, #1433, #1472). Full evidence in the PR comment below; the measured summary: **Deterministic counters, `relu_1_tiny`** — `OrtFfiCall` **10 → 6 per `Run`**, `DispatchAlloc` **24 → 20**. Four fewer round trips for one input, i.e. the predicted 7 → 3 per input. **Timing A/B**, production build, 3 clean reps (a 4th discarded for contention — it read 0.973, which would have flattered us): | case | main | this PR | |---|---|---| | `relu_1_tiny` | 1.397 | **1.271** | | `relu_10_tiny` | 1.250 | **1.173** | | `relu_100_tiny` | 1.163 | **1.110** | The gain shrinks as depth grows — the signature of a fixed-cost fix, since this removes FFI calls per `Run`, not per node. Fixed per-`Run` overhead **1.38 → 1.23** against ORT; per-node slope unchanged. **Test coverage gap this merge exposed and closed:** the pinned costs (`1 + 7` calls, `1 + 3` allocations) run against a zeroed `fake_api()`, where the reference hook is null — so they describe the *legacy fallback*, not the fast path this PR exists to add. Added `the_reference_hook_path_costs_exactly_three_ort_calls_per_input` (pins `1 + 3`) and `the_reference_hook_path_allocates_less_than_the_legacy_path`. `dispatch_probe`'s FFI-coverage table updated to 12 members / 12 `ort_call()` sites, which is the guard that flagged the new API member in the first place. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Resch <resch@squad.local>

## The build we ship had no integer GEMM at all `QLinearMatMul` in the default build did this per call: 1. `read_quantized` widened operand `A` to a `Vec<i32>`, then did it again for operand `B`. For a 2048x2048 `B` that is a 16 MiB allocation and fill on every single call, thrown away at the end of it. 2. A scalar rank-1 update walked `A` row by row, and each row re-streamed the whole of `B`. At `m = 128` that is 512 MiB of traffic for 1 GFLOP of work. The result was 11.8x ORT at `m = 1` and 12.1x at `m = 128` — the largest single loss on the x86-64 CPU EP. The performance doc's `QLinearMatMul` rows never described this build: they were taken with `--features mlas`, which is a research build we do not ship. That is now called out in the doc. This adds `kernels/qgemm_native.rs`, a native byte-operand integer GEMM, and points `qlinear_matmul.rs` at it. Nothing defers, and nothing falls back. ## Two kernels, chosen by `m` | shape | kernel | why | | --- | --- | --- | | `m <= 4` (decode) | pack-free fused | one pass over `A` means a packed panel of `B` is never reused, so packing is pure cost. Accumulators stay in registers across a 256-row `k` block. | | `m > 4` (prefill) | packed | `KC/2` pairs of `NC` columns (`KC = 512`, `NC = 256`) is 256 KiB of `B`, which stays in L2 while every row of `A` sweeps it. | The inner tile is `vpmaddwd` over `NR = 16` columns and `MR = 4` rows, `k` consumed two rows at a time. ## Why `vpmaddwd` and not `vpmaddubsw` MLAS gets 32 MACs from two instructions using `vpmaddubsw`, which **saturates**: it needs a sign-domain translation of `B` and its intermediate is only nominally exact. `vpmaddwd` needs four instructions for the same 32 MACs, but with centred `a` in `[-255, 255]` and raw `b` in `[-128, 255]` a product is at most 65025 and a pair sum at most 130050, so it cannot saturate and cannot overflow. No sign-domain flip, no reasoning about clamped intermediates. That instruction-count difference is the whole of the residual gap at `m = 1`. Closing it means giving up exact integer arithmetic, which is not a trade I am willing to make for a quantized kernel whose entire value is that it is exact. ## Determinism is structural, not tested-in The kernel computes `sum_k (a - za)(b - zb)` as `sum_k (a - za) * b - zb * sum_k (a - za)`, with every accumulation a **wrapping** `i32` add. Wrapping addition is arithmetic mod 2^32, which is associative and commutative, so *any* blocking, tiling, column split, row split or thread count gives bit-identical output — including on overflow, where the wrap itself is reproducible. `wrapping_overflow_is_reordering_invariant` and `the_thread_count_cannot_change_the_result` assert exactly that, and the SIMD path is checked bit-for-bit against a portable scalar oracle (`the_simd_kernel_is_bit_identical_to_the_portable_loop`, and separately for the fused path). ## Numbers Session A/B against plain ORT, `K = N = 2048`, u8 x u8, ratio is `ours / ORT`, **lower is better**, p50 of 61 iterations. ORT's own timings moved under 1.5% between the two arms at 1 and 4 threads, which is the control that makes the comparison mean anything. | M | threads | before | after | ours before | ours after | | ---: | ---: | ---: | ---: | ---: | ---: | | 1 | 1 | 12.20x | **2.17x** | 1.402 ms | 0.226 ms | | 128 | 1 | 11.90x | **1.20x** | 99.84 ms | 9.99 ms | | 1 | 4 | 37.11x | **4.03x** | 1.379 ms | 0.170 ms | | 128 | 4 | 14.44x | **1.47x** | 31.03 ms | 3.14 ms | | 1 | 16 | 83.12x | 35.63x | 2.366 ms | 1.336 ms | | 128 | 16 | 42.28x | 15.05x | 51.05 ms | 16.55 ms | `i8_m1` goes 0.206 ms to **0.049 ms** at one thread. Kernel-level scaling (`bench_qgemm_ab`, `taskset -c 0-15`), with the portable scalar arm as the control: | shape | 1t | 2t | 4t | 8t | 16t | portable 1t | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | 1x2048x2048 | 0.229 ms | 0.136 | 0.090 | 0.098 | 0.166 | 4.92 ms (21x) | | 4x2048x2048 | 0.565 ms | 0.311 | 0.199 | 0.237 | 0.345 | 4.58 ms (8.1x) | | 128x2048x2048 | 8.911 ms | 4.773 | 2.755 | 2.780 | 1.991 | — | | 128x5120x5120 | 53.56 ms | 27.06 | 14.35 | 8.98 | 11.51 | — | The task grid splits rows as well as columns. Columns alone gave only `n / NC` tasks — eight for `n = 2048` — so a sixteen-worker pool left half of itself spinning; `128x2048x2048` was 2.69 ms at sixteen threads against 1.62 ms at eight. Splitting columns further would shrink the panel and re-walk `B`; splitting rows duplicates only the pack, about a percent of the GEMM it feeds. ## Things I measured and rejected - **Software prefetch** of the next `B` rows (`PREFETCH_ROWS = 8`): a consistent **8% regression** with a stable `m = 128` control. The hardware prefetcher already has the sequential stream. - **Permuting inside the fused inner loop**: replaced by accumulators held in the permuted order with a single `vperm2i128` fixup per `k`-block flush. Saves eight instructions per 32 MACs. ## Left open, deliberately - **Constant-`B` packed cache.** The pack is repeated per call. Caching it would remove it from prefill entirely, but any new weight-derived cache has to go through `kernels/governed_weight_cache.rs` to satisfy the "New weight-derived caches must be governed" gate. That is a separate PR with its own eviction story, not a rider on this one. - **The session-level threading gap.** At four threads the session takes 0.170 ms while the kernel alone does 0.090 ms, and past eight threads both arms get worse. That is the pre-existing oversubscription item — it is present before and after this change, so it is not a regression here, and it is the next thing I am working on. ## Validation - `cargo test --release -p onnx-runtime-ep-cpu --lib` — 1340 passed, 0 failed. - Every `onnx-runtime-ep-cpu-plugin` suite with `NXRT_REQUIRE_ORT_TESTS=1`, including the 53-test `plugin_ort_e2e` ORT conformance suite with CPU fallback disabled. - `cargo clippy -p onnx-runtime-ep-cpu --all-targets` clean, `cargo fmt --all --check` clean. - `cargo check -p onnx-runtime-ep-cpu --lib --features mlas` — the research build still compiles. - Reviewed by Claude Opus 4.8 against the memory-safety, lane-semantics, determinism and edge-extent claims above; no blockers, two documentation fixes applied. --- ## Refreshed against `main` (2026-08-18) The branch was behind `main` and its red CI wall came from that, not from this change: `crates/onnx-runtime-session/src/executor/mod.rs:175` failed `-D dead-code` on current stable, fixed on `main` by `ca32b3adf` (#1239) after this branch forked. `origin/main` (`c55a3fab3`) is merged in — no rebase, no force-push. One conflict, in `docs/performance/CPU_MATMUL_ASSIGNMENT.md`, resolved as a **union**: this branch's `#### 3b` (the native integer GEMM) and `main`'s `### 4` (the f32 `M = 1` GEMV becoming the default, #1091) were both new sections appended after 3a. Both are kept, in that order. Taking either side would have silently deleted the other's record. Revalidated on the merge commit, AVX2/FMA host, no AVX-512: * `cargo test --release -p onnx-runtime-ep-cpu --lib` — **1424 passed, 0 failed**, 18 ignored, including `qgemm_i32_matches_the_integer_oracle_for_every_signedness`, `the_simd_kernel_is_bit_identical_to_the_portable_loop`, `wrapping_overflow_is_reordering_invariant` and `the_thread_count_cannot_change_the_result`. The measurements in this PR were taken before the merge; nothing in the merged range touches `qgemm_native.rs`, `qlinear_matmul.rs`, or the CPU threadpool, so they stand as recorded. The `main` change that did land in this range (#1091's f32 `M = 1` GEMV default) is on a different kernel family and is documented in the section-4 text kept above. --- ## Refreshed again against `main` @ `6a855d5e0`, and a real branch bug found `main` moved again while this was queued (#1346/#1352/#1361 on the quality lane, #1154/#1232/#1238 on the CPU side). Merged in normally — no rebase — and revalidated. The revalidation caught something the earlier ones had not. Running `-p onnx-runtime-ep-cpu --lib` in a **debug** profile rather than `--release` fails: ``` kernels::qgemm_native::tests::degenerate_extents_do_nothing assertion `left == right` failed left: 0 right: 4 ``` `degenerate_extents_do_nothing` called `qgemm` with an empty `b_zero_points` and `n == 4`. `qgemm` opens with `debug_assert_eq!(b_zero_points.len(), n)`, so that call is not one the function accepts — the test was exercising the `m == 0` early return through an argument list the contract forbids. It passed every previous run here only because `debug_assert` compiles out under `--release`, which is how I had been validating this branch locally. A debug test profile fails it, and this is branch-caused: `qgemm_native.rs` is new in this PR. Fixed in `9ca99e538` by sizing the test's zero points to `m` and `n`, not by weakening the assertion — the assertion states the contract the kernel's indexing depends on, and a caller whose `m` is zero still has `n` columns and still knows their zero points. **`-p onnx-runtime-ep-cpu --lib`, debug profile: 1440 passed, 0 failed** (was 1439 passed, 1 failed). This is the second time on this stack that the profile a test runs under decided whether it caught anything. Worth remembering: `--release` silently disables every `debug_assert` in the crate under test, so a local `cargo test --release` is not a substitute for what CI runs. --- ## Re-validated on latest `main` (`e0aedd0fa`), 2026-08-19 Latest `main` merged in normally (no rebase). Full re-measurement, 1 thread pinned, `K = N = 2048`, 61 iters / 10 warmup, 2 reps, `ours_p50 / ort_p50`: | case | `main` ours | `main` ratio | this PR ours | this PR ratio | speedup | |---|---|---|---|---|---| | `bench_qlinear_u8_m1` | 1.418 / 1.435 ms | 11.83x / 12.51x | **0.121 / 0.123 ms** | **1.16x / 1.18x** | **11.7x** | | `bench_qlinear_u8_m128` | 29.55 / 29.56 ms | 3.57x / 3.57x | **3.055 / 3.065 ms** | **0.372x / 0.373x** | **9.6x** | | `bench_qlinear_i8_m1` | 1.516 / 1.497 ms | 0.215x / 0.212x | **0.209 / 0.212 ms** | **0.030x / 0.030x** | **7.2x** | ORT-side drift between the two arms was 0.7% at `m = 128` and 0.0% on `i8`, which is the control that makes the comparison mean anything. **At `m = 128` we are now 2.7x faster than ORT outright**, and `m = 1` closes from 11.8x to 1.16x. These are better than the numbers originally posted above because the dispatch work in #1077 landed in between. ## Review fixes (`987aa0c5c`) An independent review found no blockers but two things worth fixing: 1. **aarch64 built with 5 warnings** — `NR`/`MR`/`NC`/`KC`/`FUSED_KC` are read only by the x86 kernels, so every non-x86 target warned on all five. CI builds with `-D warnings`, so this was a branch-caused CI failure waiting to happen; the local x86 clippy run could never have caught it. Now `#[cfg]`-gated alongside the code that uses them: **0 warnings on both x86-64 and aarch64**. 2. **The fused-parallel path had no end-to-end coverage.** Every `m <= 4` shape in `qlinear_matmul_reordered_accumulation_is_bit_identical` sat below `PARALLEL_MIN_WORK`, so the pack-free kernel's column split was only ever checked at the kernel level, never through `requantize_rows`. Added `(4, 1029, 1100)`, which forks both. The review independently re-derived the register-shuffle math in numpy (`cvtep*_epi16`, `permute4x64_epi64(0xD8)`, `unpacklo/hi_epi16`, `madd_epi16`, `permute2x128`) against a plain per-column dot product over 2000 tiles with extreme values — 0 mismatches — and confirmed the `vpmaddwd` non-saturation bound for all four operand combos, the wrapping-add determinism claim, and the absence of out-of-bounds access in every tail path. ## Validation on the merged base - `cargo fmt` clean; `cargo clippy --all-targets -D warnings` clean - **1553 `onnx-runtime-ep-cpu` tests**, debug profile (so `debug_assert`s are live) - **55 plugin conformance tests** (`NXRT_REQUIRE_ORT_TESTS=1`, release) - `every_assigned_node_is_also_executed_by_this_ep` and `every_fixture_loads_with_cpu_fallback_disabled` green — nothing defers, nothing falls back to the ORT CPU EP - **aarch64-unknown-linux-gnu** cross-check clean, 0 warnings --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Resch <resch@squad.local>

justinchuby force-pushed the squad/sebastian-cpu-affinity-smt branch from f818b10 to 30c46b4 Compare August 18, 2026 07:48

justinchuby force-pushed the squad/sebastian-cpu-task-runtime-transforms branch from ebcc6cd to b4caf90 Compare August 18, 2026 08:05

justinchuby force-pushed the squad/sebastian-cpu-affinity-smt branch from ad4707b to f33b17e Compare August 18, 2026 08:05

justinchuby marked this pull request as ready for review August 18, 2026 08:16

justinchuby mentioned this pull request Aug 18, 2026

perf(cpu): run the MLAS prefill tiling on the CPU task runtime #1238

Merged

justinchuby force-pushed the squad/sebastian-cpu-task-runtime-transforms branch from b4caf90 to 8fbb204 Compare August 18, 2026 09:12

justinchuby force-pushed the squad/sebastian-cpu-affinity-smt branch from 3d619d0 to 52f0b2a Compare August 18, 2026 09:12

justinchuby mentioned this pull request Aug 18, 2026

fix(ci): unbreak the Rust quality lane on current stable #1239

Merged

justinchuby mentioned this pull request Aug 18, 2026

perf(cpu): add a CPU task runtime with an adaptive-spin native pool #1201

Merged

justinchuby force-pushed the squad/sebastian-cpu-task-runtime-transforms branch from 8fbb204 to 064afde Compare August 18, 2026 16:29

Base automatically changed from squad/sebastian-cpu-task-runtime-transforms to main August 18, 2026 16:31

justinchuby force-pushed the squad/sebastian-cpu-affinity-smt branch 2 times, most recently from aadb0d8 to cb357f6 Compare August 18, 2026 23:57

justinchuby and others added 3 commits August 19, 2026 00:22

docs(squad): record the CPU-budget-is-cores decision for the Scribe

f375f25

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

justinchuby force-pushed the squad/sebastian-cpu-affinity-smt branch from cb357f6 to f375f25 Compare August 19, 2026 00:23

justinchuby merged commit acb8566 into main Aug 19, 2026
9 checks passed

justinchuby deleted the squad/sebastian-cpu-affinity-smt branch August 19, 2026 00:24

This was referenced Aug 19, 2026

style(cuda-ep): restore rustfmt compliance in CudaRuntime::new #1396

Closed

bench: add a decode-shaped CPU benchmark harness (gap, warm-up, park/wake) #1395

Merged

Navigation Menu

Repository navigation

perf(cpu): spread the decode CPU budget one thread per physical core - #1232

perf(cpu): spread the decode CPU budget one thread per physical core#1232
justinchuby merged 3 commits into
mainfrom
squad/sebastian-cpu-affinity-smt

justinchuby commented Aug 18, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 18, 2026 •

edited

Loading

Uh oh!

justinchuby commented Aug 19, 2026

Uh oh!

Uh oh!

codecov Bot commented Aug 19, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

justinchuby commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The CPU budget was buying hyperthreads, not cores

The fix

Result

Validation

Rebase and convergence (2026-08-18)

Uh oh!

github-actions Bot commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔴 Benchmark Regression Detected

Uh oh!

justinchuby commented Aug 19, 2026

Convergence report — rebased, defect fixed, validated, merging

Rebase

Defect found and fixed

Implementation review

Validation on the rebased head (rustc 1.97.1, the toolchain CI resolves)

CI

Uh oh!

Uh oh!

codecov Bot commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

justinchuby commented Aug 18, 2026 •

edited

Loading

github-actions Bot commented Aug 18, 2026 •

edited

Loading

codecov Bot commented Aug 19, 2026 •

edited

Loading