Repository navigation
perf(cpu): push native CPU EP toward ORT/foundry parity on int4 decode - #105
Merged
Merged
Conversation
Expose existing per-stage profiling in profile_native after warmup reset and record the 7B CPU kernel/thread breakdown. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Record 24-token end-to-end and 128-token steady native, ORT, and OGA measurements, and separate native M=8 compute from reset overhead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Profile-first (RULES.md rule 4) re-evaluation of int4 MatMulNBits M=1 decode at real Qwen2.5-Coder-7B shapes on Sapphire Rapids (Xeon 8480C). The task hypothesis -- that MLAS SQNBit CompInt8 wins M=1 and closes the ~2.3x gap to ORT -- is refuted: with cold, DRAM-streamed distinct weights the hand int4 GEMV and MLAS tie at ~90 GB/s (decode is memory-bandwidth bound). The prior "MLAS wins 1.7-1.9x" microbench was an L3-cache-resident artifact. perf shows the gap is ~27% per-op Rayon fork-join plus NUMA locality, not the kernel. Changes (routing stays shape/dtype-driven, never model identity -- rule 2): - Rename NXRT_SQNBIT_PREFILL_MIN -> NXRT_SQNBIT_DECODE_MIN (default 16) with measured rationale in the docstrings. - M=1 gate is now dtype/accuracy-driven: keep the fast hand path only for bits==4 && accuracy_level==4; route the slow dequant-to-f32 M=1 case (accuracy_level!=4) to MLAS SQNBit CompFp32 -- a genuine generic win. g_idx and 2-bit still fall back to the hand path (MLAS can't serve them). - build_mlas_packed remains cached once per constant weight (OnceLock); decode steps stay pack-free. Tests (rule 8, same commit): - Add matmulnbits_try_mlas_serves_slow_dequant_decode (m=1 bits=4 acc=0 routes to MLAS, matches f32 reference within 2e-3). - Add ignored matmulnbits_mlas_decode_step cold distinct-buffer 7B decode-step probe (hand vs MLAS). - Rename/update matmulnbits_resolve_decode_min_parses_or_defaults and matmulnbits_try_mlas_gates_decode_by_m_threshold. The bits4/acc4 M=1 path is unchanged, so 7B decode output is bit-identical. Docs: update BENCH_MLAS_INT4_E2E.md (env rename + 7B addendum with cold-tie table, perf split, NUMA +25%) and PROGRESS.md env rename. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Optimizes the M=1 decode hot path in GroupQueryAttentionKernel::execute with three targeted changes: 1. Attended-window scoring: allocate scores[attended] instead of scores[total_sequence_length], skipping unattended masked positions entirely. 2. SIMD dot-product (dot_f32 / dot_avx2_fma): AVX2+FMA with two parallel 8-wide accumulators for the QK score computation, runtime-gated by has_simd_x86(), scalar fallback for non-x86 targets. Bounded FP reordering (n × f32::EPSILON × max_val); greedy token ids unchanged. 3. Cache-friendly P·V AXPY (axpy_f32 / axpy_avx2_fma): reorder P·V from d-outer/ks-inner (stride-128 V access → cache miss per ks) to ks-outer/ d-inner (contiguous V row access + AVX2 FMADD). Normalize softmax scores once before the P·V loop to eliminate per-element division. Precision contract (RULES.md §4) preserved: softmax uses the exact f64 exp + f32 cast path unchanged. Benchmarks (dev workstation, Qwen2.5-Coder-7B int4, 32 threads): - Short context (~32 tokens): GQA 1.77-2.37 ms vs 3.3-7.6 ms baseline (1.9x-3.2x); e2e 9.23 vs 8.73 tok/s (+5.7%) - Long context (~1000-token prompt): GQA 67 ms vs 86 ms per step (1.28x); e2e 0.41 vs 0.36 tok/s (+14%) Greedy token ids verified identical (16 tokens, long-context prompt). Tests (RULES.md §8): - gqa_decode_long_context_matches_reference - dot_f32_matches_scalar_reference_for_various_lengths - axpy_f32_matches_scalar_reference_for_various_lengths Numerics review (Rule 9): Chew to verify AVX2 dot-product accumulator reordering is within tolerance for production use. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sebastian's profile: MatMulNBits is 77% of the M=1 decode step. The isolated 7B MatMulNBits op mix runs in ~33ms, but the real decode bucket is 64ms -- ~30ms of per-op glue. Part of it was a serial, non-vectorized per-element strided walk: every op ran to_dense_f32 on its activation and write_dense_f32 on its output, incrementing multi-dim indices one element at a time (~2.5M serial iters/token, off the parallel path). Add a contiguous row-major bulk-copy fast path to to_dense_f32 and write_dense_f32: contiguous tensors (the common decode/prefill case) use copy_from_slice / extend_from_slice; non-contiguous views keep the exact strided walk. Benefits every f32 kernel. Bit-identical output. End-to-end Qwen2.5-Coder-7B decode (32 threads, 6 runs, noisy shared host): best step 104.0 -> 87.8 ms (best 9.61 -> 11.39 tok/s, ~+11% median). Text unchanged/coherent. Also settle the MLAS-vs-hand M=1 question definitively: extend the matmulnbits_mlas_decode_step probe to a 3-way hand / CompInt8 / CompFp32 comparison at cold distinct 7B shapes. hand (29.5 tok/s) and CompInt8 (30.6 tok/s) tie within host noise; CompFp32 (23.8 tok/s) is clearly worst -- so the hand path is kept for M=1 accuracy_level=4 (rule 2, shape/dtype-driven). Tests (rule 8, same commit): - write_dense_contiguous_bulk_copies, write_dense_strided_matches_logical_order cover the new fast path and the retained strided path. - existing dense_roundtrip_contiguous / dense_reads_transposed_view already prove fast/strided read agreement. Remaining gap to ORT (20.12 tok/s) is per-op Rayon fork-join, executor dispatch, and NUMA locality -- cross-crate; see decision note for the ranked follow-up (gate+up projection grouping, fewer fork-join barriers, NUMA-aware placement). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Decode streams the full int4 weight set every token, so throughput is dominated by memory latency and by the fork-join barrier that closes each of the ~141 per-token MatMulNBits projections. When the bounded decode workers float across both sockets, every barrier pays a cross-socket cache-coherency round trip and the weights land on whatever node first touched them. Add ONNX_GENAI_CPU_DECODE_AFFINITY to pin the decode pool workers to the CPUs of a single NUMA node. Topology is queried at runtime from /sys/devices/system/node (no hardcoded socket/core counts, Rule 2); the switch is explicit and inspectable (Rule 5); a single-node/non-Linux host or a restricted cgroup transparently falls back to the unpinned global behavior, logged once (Rule 1). Because the packed int4 weights are lazily first-touched inside with_decode_pool_scope on a pinned worker, both the barrier traffic and the weight stream become node-local. Modes: off (default), compact (smallest node covering the thread count), node:<index>. Greedy token ids are bit-identical; this only changes placement, not math. Measured (Xeon 8480C 2x48c, Qwen2.5-Coder-7B int4, 32 threads, steady M=1, 5 runs x 3 rounds): off median 13.1 tok/s (12.6-14.4, jittery) vs compact median 16.3 tok/s (16.3-16.4, stable), ~+25% median and far lower run-to-run variance. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ion (review fixes for 32a122e) Address Gaff's rejection of 32a122e on the M=1 decode-pool NUMA affinity: - Size the sched_setaffinity mask from the runtime CPU index via build_cpu_mask() so a CPU id >= CPU_SETSIZE grows the buffer instead of writing out of bounds; overflow falls back to unpinned, and unsafe is reduced to the single syscall. - Make every invalid ONNX_GENAI_CPU_DECODE_AFFINITY value report the rejected value, all accepted modes, and the available-node list (or a topology- unavailable statement) via DecodeAffinity::resolve + shared helpers; unknown nodes are now reported even on single-node / /sys-unavailable hosts. - Select the smallest-index fitting node for compact (find over the ordered BTreeMap) to match the documented policy, replacing min_by_key(len). - Add unit tests for diagnostic content, high CPU-index mask safety, and the compact smallest-index policy; keep the existing four tests green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add ONNX_GENAI_CPU_DECODE_AFFINITY=numa-split: build one node-pinned Rayon sub-pool per NUMA node and row-shard each M=1 MatMulNBits projection across nodes, joined by a two-level barrier (node-local par_chunks_mut, then a single cross-node combine) instead of a flat cross-socket per-op fork-join. Weight row-shards are first-touched on their owning node so each node streams a node-local slice. Reuses the existing MLAS SQNBit / hand VNNI kernels on each row-slice (no new matmul). Row-sharding a GEMV is exactly associative, so greedy token output is bit-identical to the single-node path (verified). Opt-in only; default path unchanged. Topology queried from /sys at runtime (no hardcoded node/core counts). Falls back once-logged to flat single-node decode on single-node/non-Linux hosts or THREADS=0. Additive over Deckard's affinity fix (046414b): numa-split parse arm flows through resolve()'s pass-through; reuses the unchanged pin_current_thread_to_cpu; extends the centralized ACCEPTED_MODES menu to list numa-split. Steady M=1 int4 (Qwen2.5-Coder-7B, SPR 8480C, 5 interleaved rounds, T=32): compact 16.87 tok/s median -> numa-split 18.42 tok/s median (best 18.51, +9%), bit-parity exact and lower run-to-run variance. Peak at 16 workers/node; T=64 regresses (cross-node barrier cost). Still ~1.6 tok/s short of the 20 target. Tests: cargo test -p onnx-runtime-ep-cpu --features mlas = 675 passed. Clippy clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…Bits Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Windows: NUMA topology via GetLogicalProcessorInformationEx(RelationNumaNode) and per-thread pinning via SetThreadGroupAffinity, group-aware so >64-CPU hosts (multiple processor groups) pin correctly (cpu = group*64 + bit). macOS/other: documented graceful (logged) no-op; crate compiles+lints clean on linux, x86_64-pc-windows-gnu, and aarch64-apple-darwin. Auto-enable (gap #1): when ONNX_GENAI_CPU_DECODE_AFFINITY is unset, a multi-node host on a pinning-capable OS defaults to compact (the safe ~+25% win, not numa-split), logged once at info; single-node stays unpinned; users can still force off or any explicit mode. Bit-parity holds (identical greedy token ids, auto vs off). cpuset safety (gap #7): intersect discovered topology with the process's allowed CPU set (Linux sched_getaffinity, Windows GetThreadGroupAffinity) before pinning; if <2 nodes remain, auto-enable declines and logs once, so containers/taskset never pin outside their cpuset. Applies to auto, explicit flat, and numa-split paths. Adds unit tests for intersection, auto-enable decisions, and explicit-mode validation; all pure logic runs on every host, OS-specific FFI is cfg-gated. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ECODE_PERSISTENT_POOL Native CPU int4 M=1 decode is per-op fork-join-barrier + memory-latency bound, not kernel-compute bound: ~141 MatMulNBits ops/token, each a separate rayon parallel region whose join barrier dominates once the int4 kernel is L3-resident (profile in .squad/decisions/inbox/pris-decode-profile.md). Add a persistent SPMD decode pool (decode_spmd.rs) that keeps one worker set hot for the whole decode loop and drives each op with a lightweight reusable barrier whose completion counters are per-node, so barrier traffic stays node-local (the cross-socket barrier is what sank the naive dual-node pool). The actual GEMV still runs the existing packed int4 / MLAS SQNBit kernels (rule 4) - only the orchestration/barrier structure changes. Mirrors decode_numa's two-level layout for node-local weight placement and exact reduction order. Opt-in, default OFF (rule 5): ONNX_GENAI_CPU_DECODE_PERSISTENT_POOL=1. Falls back to the existing bounded pool on single-node / non-Linux / cgroup-refused pinning (rule 2). Portable std atomics + thread::park; only optional CPU pinning is Linux-specific. Measured (Xeon 8480C, 2x48c/2 nodes, Qwen2.5-Coder-7B int4, 32 threads, steady 96-tok, interleaved A/B, noisy shared host): 17.71 tok/s median (best 18.37) vs 16.42 numa-split prior best -- +7.9% median / +8.3% best, wins 4/4 rounds. Greedy token ids byte-identical flag ON vs OFF (rule 8). Full ep-cpu suite (692) + clippy -D warnings clean. Fixes a startup-race deadlock (worker starting after first dispatch missed the op) with a readiness barrier + regression tests. Details/parity proof: .squad/decisions/inbox/pris-decode-barrier.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…panic-safe barrier (Chew/Gaff review) Keep numa-split precedence explicit and logged when both strategies are requested. On single-node or unpinnable hosts, persistent mode uses one unpinned SPMD group rather than the bounded Rayon pool. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Scribe housekeeping: merge Pris/Sebastian/Gaff/Chew SPMD review notes into decisions.md, untrack the gitignored .squad/decisions/inbox/ files that were accidentally committed, refresh agent histories + orchestration/session logs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Widen Float16 activations to f32 for normalization and narrow all outputs back to their declared dtype. Add f16 coverage for normalized and residual outputs; native 0.5B Foundry CUDA-export decode matches ORT CPU greedy tokens. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Root cause: try_mlas_sqnbit gated ALL MLAS SQNBit use on CpuBackend::auto_detect() == CpuBackend::Mlas. On an AVX2 host auto_detect() returns SimdX86 (never Mlas unless NXRT_CPU_GEMM_BACKEND=mlas is forced), so MatMulNBits never reached MLAS by default. For accuracy_level != 4 int4 weights (e.g. Foundry cuda-gpu exports that set accuracy_level=0, "implementation's choice") the kernel then fell to the slow m==1 full-f32-dequant GEMV (gemv_nk), reading the whole weight dequantized to f32 -- ~4x the bandwidth of int4 -- which is why the 0.5B decoded at ~3.7 tok/s despite having 14x fewer weight bytes than the 7B. Fix: MLAS SQNBit is a specialized blockwise-quantized kernel, orthogonal to the dense-f32 GEMM microkernel that CpuBackend selects. Require the backend==Mlas gate only for accuracy_level==4 (its fast hand int8/int4 path owns MatMulNBits and matches ORT CompInt8). For every other accuracy level, route to MLAS SQNBit CompFp32 whenever MLAS has a kernel -- matching ORT/onnxruntime-genai, which treat accuracy_level 0/1 as CompFp32. General (no model/shape hardcoding); also fixes acc levels 1/2/3. accuracy_level==4 is a strict no-op: use_mlas = backend_is_mlas || acc != 4, and on the default SimdX86 backend acc==4 still returns None exactly as before. Evidence (qwen2.5-0.5b-instruct-cuda-gpu-4/v4, acc=0, 8 threads, steady decode): - MatMulNBits op time per token ~48ms -> ~14.5ms (~3.3x). - End-to-end is GroupQueryAttention-bound (~88% / ~158ms/token), so overall throughput moves modestly and is host-noise dominated (baseline 3.8-4.4 -> 4.0-5.4 tok/s @8t). Residual bottleneck = GQA (separate issue). - 0.5B greedy opener matches ORT CPU reference [271,40,1079,264,48948,304,13027,323,358,1079,4460,311,...]. No regression, 7B qwen2.5-coder-7b-instruct-generic-cpu-4 (acc=4): opener unchanged [48298,271,9707,0,2585,646,358,7789,498,3351,30,151645,...], 11.5 tok/s @8t / 14.75 @32t. Tests: cargo test -p onnx-runtime-ep-cpu --features mlas -> 697 pass (+1 new, matmulnbits_try_mlas_routes_acclevel0_without_mlas_backend, asserting acc=0 routes to MLAS with a non-MLAS backend and matches the f32 reference, while acc=4 stays on the hand path). Two prepack-cache tests updated to check whichever prepack cache the routed path populates. Clippy clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
GroupQueryAttention, not MatMulNBits, dominated 7B int4 CPU decode (~47% / ~32 ms/token per ONNX_GENAI_PROFILE_OPS, which perturbs wall by only ~6%). Root cause: the persistent SPMD decode scope's dispatcher is the engine thread (not a Rayon worker), so GQA's `par_chunks_mut` fell to the *global* 96-thread Rayon pool and contended with the SPMD pool's 32 pinned, spinning workers. The numa-split and flat decode scopes install the forward onto a bounded Rayon pool, so they never hit this contention. Route GQA's decode-time row parallelism through the *active* decode pool instead of the global pool: - add `SpmdDecodePools::dispatch_output_row_blocks`, which shards fixed-width (per-head) output rows across the resident SPMD workers under the existing lightweight barrier; - add generic `decode_parallel_output_row_blocks`, which selects the pool the same way `parallel_output_rows` does (SPMD when its scope is active, otherwise `par_chunks_mut` on the already-bounded numa/flat pool); - GQA's parallel branch calls it. Routing keys off which decode scope is active, never off op or model identity (RULES.md §2). Each attention row is independent, so row-sharding reproduces the single-threaded result bit-for-bit. 7B qwen2.5-coder int4 (accuracy_level=4, f32 activations), 2-socket Xeon 8480C, 32 threads, greedy, persistent SPMD pool, steady 120 tok, tokens=128 runs=5, interleaved A/B median: 16.4 -> 26.2 tok/s (+60%), beating onnxruntime-genai 0.14.1 (21.30 tok/s) by ~23%. Greedy opener unchanged: [48298,271,9707,0,2585,646,358,7789,498,3351,30,151645,...]. Tests: `cargo test -p onnx-runtime-ep-cpu --features mlas` 698+10 green (new `dispatch_output_row_blocks_matches_flat_computation` bit-parity test; all GQA f16 / long-context / prefill parity tests pass). `cargo clippy -p onnx-runtime-ep-cpu --features mlas --all-targets -- -D warnings` clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Flip ONNX_GENAI_CPU_DECODE_PERSISTENT_POOL from default-OFF (only `=1` on) to default-ON with opt-out. Out-of-box 7B int4 decode now beats ORT. Default-on policy (rule 5): single source of truth `persistence_mode()` (Off/Auto/Forced). Unset -> Auto (on), `=0` -> Off (flat + auto-compact legacy path), `=1` -> Forced (bypasses the safe-host gate). One-time activation log announces the default and the opt-out. Safe auto-enable gate: auto only fires on hosts with >= 4 logical CPUs; tiny/single-core hosts and THREADS=0 fall back to the flat path (build_from_env still degrades to one unpinned group on single-node/non-NUMA/no-pinning hosts). Thread-count decision (rule 2, topology-derived): the persistent pool's unset default is now half the logical CPUs (configured_persistent_decode_threads), not the flat pool's 8-worker ceiling. The flat cap exists because per-op fork/join regresses past 8; the persistent pool replaces that with one hot barrier and keeps scaling. 7B sweep (128 tok, 5-run median, 2-socket Xeon 8480C): 8->12.95, 16->19.78, 24->23.86, 32->26.18, 40->27.90, 48->28.71, 64->28.67, 96(all)->1.36 tok/s (a fully-subscribed spinning pool starves the dispatcher). Half sits at the plateau and avoids the cliff; explicit ONNX_GENAI_CPU_DECODE_THREADS is honored and clamped to the host. Out-of-box A/B (nothing set): before (flat, 8 workers) ~11.1 tok/s -> after (persistent, 48 workers) 28.8 tok/s, beating onnxruntime-genai 0.14.1 (21.30) and raw ORT (~26.9). PERSISTENT_POOL=0 restores ~11.1 (legacy), proving opt-out. Auto-affinity reconciliation (rule 5): when persistent SPMD is active the flat pool is never built, so decode_affinity's auto-`compact` never fires (verified: the auto-compact log is absent in the default path); the SPMD pool does its own per-node pinning. Precedence: explicit numa-split env > persistent SPMD default > flat + auto-compact. Precedence log messages reworded for default-on. Parity: greedy 7B openers byte-identical across default, THREADS=1/8/48, PERSISTENT_POOL=0/1, and numa-split; the kernel-level subprocess parity test now compares `=0` (legacy) vs `=1` and is byte-identical. Tests: enabled()/mode default-on + only-`0`-opts-out, auto-enable gate, and the half-cores thread-count policy. 702+10 green; clippy -D warnings clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Address Chew's cross-platform review (2 blocking) and Gaff's concurrency cleanups (2 non-blocking) on top of the persistent-default flip (770d0c5). Chew #1 -- explicit affinity was silently ignored under the default. Since Auto routes to the persistent SPMD pool before the flat pool is built, an explicitly-set ONNX_GENAI_CPU_DECODE_AFFINITY (off/compact/node:<n>/malformed) no longer reached plan_decode_affinity, regressing an explicit user control. New single-source-of-truth decode_spmd::auto_defers_to_flat (consumed by build_from_env/pools): in Auto mode an explicit non-numa-split affinity now DEFERS to the flat path so off=unpinned, compact, node:<n> are honored and malformed values still error, exactly as before. Precedence: numa-split > (Auto default, unless explicit non-numa-split affinity defers) persistent SPMD > flat + auto-compact. numa-split keeps top precedence; =1 (Forced) overrides the defer and keeps the persistent pool (its own per-node pinning); =0 (Off) and the true out-of-box case (no affinity env) are unchanged. A one-time log announces the defer. Chew #2 -- docs falsely claimed the single-node/non-Linux fallback uses an UNPINNED SPMD group. Reality: node_shards passes allowed_cpus() into the fallback shard and SpmdDecodePools::build best-effort pins every worker to those CPUs on Linux/Windows; only an unknown allowed-set or a no-pinning platform (macOS/unsupported) is truly unpinned. Docs corrected; code unchanged. Gaff #1 -- removed dead configured_decode_threads_public (decode_spmd uses configured_persistent_decode_threads now). Gaff #2 -- the numa-split "mutually exclusive" diagnostic now logs only when the user EXPLICITLY forced the pool (decode_spmd::is_forced), not under the Auto default; removed the now-dead decode_spmd::enabled. Tests (707+10 green, was 702+10; clippy -D warnings clean): unit coverage for auto_defers_to_flat, and subprocess routing tests -- Auto+off/node:0/compact build no SPMD pool (flat), Forced+off still builds SPMD, Auto+malformed defers and still errors. Proof (7B qwen2.5-coder int4, shared host): out-of-box (nothing set) 27.25 tok/s median (128 tok, 3 runs), beats ORT (~26.9); DECODE_AFFINITY=off default now emits the defer log and NOT the SPMD default-on log (flat path honors off); PERSISTENT_POOL=0 ~11.5 tok/s (legacy), =1 30.45 tok/s. Greedy opener [48298,271,9707,0,2585,646,358,7789,498,3351,30,151645,...] byte-identical for default, DECODE_AFFINITY=off, =0, =1, THREADS=1. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Scribe consolidated the default-flip decision (Leon impl + Deckard affinity-defer revision, reviewed by Gaff + Chew) and merged the gaff-gqa-pool + roy-f16-layernorm review notes into decisions.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Native decode of f16-activation int4 foundry cuda-gpu exports was GroupQueryAttention-bound. Profiling (per-op + within-GQA phase timers) showed ~92% of GQA wall time was scalar f16<->f32 conversion of the WHOLE growing KV cache every token (widen ~47%, narrow/out ~45%), not attention math (~6%). GQA was ~54% of the 0.5B decode. Fix (targeted, general, EP-agnostic; f32 generic-cpu path untouched): 1. F16C-vectorize bulk f16<->f32 in dtype.rs (_mm256_cvtph_ps / _mm256_cvtps_ph, round-to-nearest-even). Bit-identical to scalar half (exact widen; RNE narrow) - locked by test over all 65536 f16 patterns. Runtime-detected; scalar fallback off-x86/non-contiguous. 2. group_query_attention.rs: replace eager full-cache widen (Cow<[f32]>) with a lazy PastSrc enum (F32 borrow / F16 raw u16 / Dense fallback) and widen_run() that widens each per-head run DIRECTLY into `present`, removing the intermediate materialize and the second f32 copy paid every token. Skip the present zero-fill when there is no tail (steady decode). Results (tokens 128, runs 3, median): - 0.5b cuda-gpu: 2.55 -> 6.56 tok/s (2.57x); GQA share 54% -> ~14% (MatMulNBits now dominant ~82%). - 1.5b cuda-gpu: 1.15 -> 3.58 tok/s (3.11x). - Opener byte-identical for full 128 tokens; f32 accumulation throughout. - generic-cpu 7B unchanged (~29 tok/s) - f32 caches take PastSrc::F32 borrow+copy, same behavior. Tests: 709 + 10 golden green; added multi-step incremental-vs-full-widen lock test + F16C bit-identity test. Clippy -D warnings clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Scribe merged roy-f16-gqa-decode (2.5-3.1x f16 GQA fix, eedbf93, Gaff+Chew APPROVE) and sebastian-foundry-cpu-comparison (7B native 28.6 vs ort 21.0 = 1.36x; qwen3.5-9b is a VLM package) into decisions.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Close Chew's nonblocking eedbf93 review nit with a multi-step old-vs-new cache and attention parity test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds no_tail_with_past_present_independently_bit_exact, closing Chew's reject on 8638ec6. Pris's full-widen parity test only exercises the !has_tail (uninitialized set_len) fast path at step zero, where the past is empty and both the production call and the old-style reference share the same present-construction; a shared offset/read-before-write bug would corrupt both sides identically and still pass. The new test forces has_tail == false AND past_len > 0 for every batch (batch=2, kv_heads=2, q_heads=4, head_dim=7, past_len=3, present_len=4) and asserts the f16 present_k/present_v bit patterns equal a present cache assembled BY HAND from the known past f16 bits + the current-step K/V in BNSH order -- never routing the expected side through the production fast path. Attention output is checked bit-exact against the old full-widen reference. head_dim=7 exercises the F16C widen tail; q_heads>kv_heads exercises GQA group broadcast. TEST-ONLY: production kernel untouched (diff inside mod tests). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Scribe merged pris-f16-widen-parity-test + gaff-notail-widen-test: bit-exact old-vs-new parity tests for the f16 GQA lazy-widen (eedbf93), incl. independent no-tail-with-past assertion guarding the uninitialized set_len fast path (Chew REJECT->APPROVE). Merged 80b09c3. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Speeds up decode of f16-activation (accuracy_level=0) int4 models on CPU
by fixing thread oversubscription in the MLAS SQNBit path. Does not touch
the shipped generic-cpu (f32-activation, acc=4) path.
Profile (0.5B cuda-gpu, steady, decode-skip 8, 128 tok) BEFORE:
MatMulNBits ~79% of the token (~115-135 ms, 121 calls/token), GQA ~16%.
Within-MatMulNBits phase timers (ONNX_GENAI_PROFILE_MM=1):
gemv 99.6%, widen(f16->f32) 0.1%, narrow(f32->f16) 0.3%.
=> Activation f16<->f32 conversion is NOT the bottleneck (the F16C bulk
widen/narrow already make it negligible); the whole cost is the MLAS
SQNBit GEMV call itself.
Root cause: for f16/acc=0 every MatMulNBits routes through MLAS SQNBit
CompFp32. sqnbit_gemm(multithread=true) forks MLAS's rayon_parallel_for
on the CURRENT pool. Under the persistent SPMD decode scope the forward
runs inline on the (non-worker) dispatcher thread, so current_num_threads
returns the GLOBAL pool (96) and MLAS forks 96 tiles for a tiny M=1 GEMV
while the ~48 pinned SPMD workers spin-wait -> catastrophic
oversubscription, x121/token. (RAYON_NUM_THREADS=8 -> 23.8 tok/s and
DECODE_PERSISTENT_POOL=0 -> 28.6 tok/s both confirm contention.)
Fix: run the M=1 decode GEMV on the SPMD workers themselves.
- mlas-sys: add sqnbit_gemm_into(.., c: *mut f32, ldc, ..), a strided
wrapper so one N-shard writes its columns into a shared [m, ldc]
output; sqnbit_gemm delegates with ldc == n (matmul_nbits.rs).
- decode_spmd.rs: add output_column_segments(n) and
dispatch_output_rows_indexed(result, compute) passing each worker its
global index so the caller picks the matching pre-packed shard.
- matmul_nbits.rs: pre-pack the constant weight once per N-segment into
mlas_shards (MlasShard{start,len,packed}); decode dispatches one shard
per resident, pinned, node-local worker running sqnbit_gemm(mt=false)
under one barrier; prefill/no-pool loops shards with
sqnbit_gemm_into(mt=true) at stride n. No persistent pool => single
[(0,n)] shard == old single multithread=true call (no off-path change).
ONNX_GENAI_CPU_MM_MLAS_NO_SHARD=1 forces the full-width path for A/B.
Results (median, low load; shared Xeon 8480C, 96 logical / 2 NUMA):
0.5B cuda-gpu: 6.2-6.6 -> 33.1 tok/s; MatMulNBits 79% -> 10.4%
1.5B cuda-gpu: 3.58 -> 15.08 tok/s; MatMulNBits ~79% -> 10.6%
GroupQueryAttention is now the dominant op (~71-73%).
Parity: 0.5B and 1.5B full 128 greedy tokens byte-identical between the
sharded path and ONNX_GENAI_CPU_MM_MLAS_NO_SHARD=1 (full-width MLAS ref)
on the same binary. f32 accumulation preserved. MLAS CompFp32 SIMD
column-tiling is not bit-stable across an N-partition boundary (~1 ULP,
below argmax resolution); mlas-sys test sqnbit_int4_n_shards_match_full
asserts near-equality (1e-3) and documents this.
No regression: generic-cpu 7B ~27.8 tok/s (baseline ~28.6, within shared-
host noise), opener [48298,271,9707,0,2585,646,358,7789,...] unchanged.
Tests: onnx-runtime-ep-cpu 710 passed +3 ignored (+10 integration);
mlas-sys green. clippy -D warnings clean on both crates.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Close Chew finding 1 with MLAS block-size shard parity and finding 2 with cached SPMD route parity. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two regressions on the CPU-EP perf branch broke the executor's reject-before-materialize contract on PR #105: 1. Dynamic shape propagation downloaded every bounded *float* input to host (added for Resize scales), materializing an unrelated tensor before an invalid integer shape input could fail resolution. Gate the float-value read to inputs an op actually consumes (Resize scales) via `reads_float_shape_input`, so no host copy precedes the rejection. 2. Always-on CPU-EP optimizer passes now run their whole-graph postcondition + infer_graph, pre-empting load-time validation: a cycle surfaced as a wrapped PostconditionFailed, an unsupported op as an opset-import invariant, and a Slice step-0 aborted the build instead of run. Run the DAG (topological_order) and operator-support checks before EP passes (mirroring the control-flow signature check), and make the post-pass infer_graph best-effort so data-dependent invalidities are rejected at run time as before. Collapse the `if !nested { if let ... }` per clippy collapsible_if. The zero-copy top-level output move (decode fast path) is unchanged. Fixes 5 executor tests + slice_zero_step; session crate green, session clippy -D warnings clean, ep-cpu (881+10) unaffected, fmt clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… 🟢) — greens Rust test + quality jobs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… + session logs) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…=1 asym int8 CI runners are AVX2-class (not AVX-512); two crates/mlas-sys tests hard-coded AVX-512-only expectations and were RED on CI while passing by luck on AVX-512 dev hosts. Root cause (reproduced under Intel SDE: -hsw fails, -skx passes): MLAS's AVX2 M=1 CompInt8 SQNBit kernel with a zero point (SQ4BitGemmM1Kernel_CompInt8_avx2) is numerically broken for asymmetric int4 -- ~46% error (mlas=6.09 vs ref=11.29) for every block length. The AVX-512 M=1 kernel and every AVX2 M>1 kernel (zero point applied via the QuantBBlkSum block-sum correction) are correct, so packing and the test are fine; only the AVX2 M=1 subtract-zp compute path diverges. Changes: - mlas-sys: rename avx512_kernel_is_selected -> best_available_float_kernel_is_selected and assert the per-ISA kernel id the host actually supports (512/3/1/-1/0), verified across SDE -skx/-hsw/-snb/-mrm. - mlas-sys: gate the M=1 asymmetric CompInt8 case of sqnbit_int4_compint8_matches_reference to AVX-512 hosts (documented); all symmetric and all M>1 asymmetric cases still run on every CPU. No tolerance loosened. - ep-cpu: try_mlas_sqnbit refuses M=1 asymmetric CompInt8 on non-AVX-512 hosts and falls back to the correct hand int8 decode kernel (host_supports_avx512f helper). Default routing already kept M=1 on the hand path via the sqnbit_decode_min()>=2 crossover; this closes the latent hole where NXRT_SQNBIT_DECODE_MIN<=1 could reach the broken kernel. - ep-cpu: add matmulnbits_accuracy4_m1_asymmetric_matches_fp32_reference regression test (fails on AVX2 without the guard, proven under SDE). Production is correct on all CPUs (default and under the decode-min override); prefill M>1 asymmetric int4 continues to use MLAS CompInt8 (correct on AVX2). Upstream MLAS bug draft recorded for filing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…pley, Lambert review 🟢) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…o perf/cpu-ep-mlas Resolves conflict in onnx-runtime-ep-cuda normalization.rs (kept origin/main's new bf16_bytes + run_bf16_norm_gpu test helpers; our CPU-EP branch had none). Verified post-merge on dev host: session 69+32/0, ep-cpu 883/0 +10 golden, CUDA compile + clippy -D warnings exit 0, session/shape-inference clippy clean, fmt clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…AS sgemm(trans_b) Native CPU prefill (time-to-first-token) on quantized MatMulNBits models was ~9-22x slower than ONNX Runtime, dominating whole-model latency even though decode was already competitive. Op profiling attributed ~95% of prefill to MatMulNBits; scoped timers showed the cost was the dense-dequant fallback for batched (m>1) calls dequantizing the weight into the transposed Kn layout -- a strided-scatter (stride N per K step) that thrashes cache (~4773ms) while the actual GEMM was only ~204ms. This path is hit by 8-bit weights (the fast SQNBit MLAS route only handles 4-bit) and other generic/grouped cases. Fix (MLAS backend): dequantize once into the natural contiguous Nk layout, cache it in the existing weight_nk OnceLock (shared with the m==1 GEMV path), and run MLAS's cache-tiled sgemm with trans_b=true so it consumes Nk as B^T directly -- no materialized transpose, and each weight row is streamed once and reused across all m activation rows. Non-MLAS hosts keep the prior direct-Kn path. Also parallelize dequantize_weight's Nk/no-group-indices path across output rows (was single-threaded) to speed the one-time cold dequant. Bit-identical to the old dense GEMM (A*B_nk^T == A*B_kn); native greedy token IDs are byte-identical before/after. Prefill ~5335ms -> ~545ms (~9.8x); MatMulNBits op ~6475ms -> ~204ms; native-vs-ORT prefill gap ~25x -> ~2.6x. Decode unchanged (~25ms/token; only the m>1 branch is touched). Adds regression test matmulnbits_8bit_prefill_batched_matches_dequant_f32_oracle (m=16/32/100, symmetric+asymmetric) asserting RMSE<=1e-5 and per-row argmax parity vs an independent dequant-to-f32 GEMM oracle. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rd, Kano review 🟢) native_available non-x86 stub + dot_u8_i16_scalar were dead code on aarch64 under -D warnings. Proper cfg gating; portable test coverage retained. Verified: aarch64 lib+tests clean, x86 884/0, clippy(mlas+default)+fmt clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… (Voight, Chew review 🟢) The persistent SPMD decode pool was a structural 2.5-3x decode regression: its per-op broadcast/counting barrier hard-spins over ~197 tiny M=1 ops/token. Make it opt-in via ONNX_GENAI_CPU_DECODE_PERSISTENT_POOL=1; unset (Auto) and =0 (Off) both take the flat Rayon path by construction, so the default can never be slower than pool-OFF. Removed 4 tests targeting deleted auto-enable heuristics; added only_forced_mode_enables_the_persistent_pool. Explicit-affinity behavior stays covered by surviving integration tests. Chew (non-author, opus) reproduced ~2.8-3x regression interleaved (flat 33-38 vs pool 12-13 tok/s), byte-identical tokens across default/=0/=1. Gates: ep-cpu 881+10 pass, clippy (mlas + no-mlas) clean, fmt clean, aarch64 -D warnings clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Cherry-pick ea31e26+7e70fcf (Holden/Pris, reviewed 🟢 by Dietrich). qwen3-0.6b + phi3.5-mini int4 divergence regression tests using an fp32-activation oracle (accuracy_level 4->1) proving native argmax is the higher-precision choice. phi adds a real m=1 decode-loop lock. Tests are #[ignore]+env-gated; compile under --features mlas. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Cherry-pick 59ea1ab (Ripley, reviewed 🟢 by Gorman). int4_dot_row VNNI paths use independent f32 accumulators (4 avx512 / 2 avxvnni) + weight prefetch, breaking the loop-carried add_ps latency chain. Per-block integer dots are bit-identical; only the final f32 scale-accumulate order reassociates (within 1e-4 tol). Token-exact on qwen3-0.6b + Phi-3.5-mini, zero-regression, positive on compute-bound cases. New non-vacuous argmax guard. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Cherry-pick cdf1091 (Vasquez, bench_generic separable-part 🟢 by Drake). Generic single-inference native-vs-ORT bench (interleaved timing + output parity) for traditional-ML ONNX models (resnet/yolo). Reusable generality harness. required-features=[mlas] gates it off non-x86 builds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…(opt-in) Cherry-pick 69a4463 (parker sizing) + ab3fd65 (hicks non-vacuous test). numa_pools() sized from configured_persistent_decode_threads() (was capped at 8); default flat decode path unchanged (opt-in via numa-split env). Token-exact (row-shard concat, no cross-node reduction). Test proven non-vacuous. Reviewer: Gorman (opus) approved (non-author). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Cherry-pick 710e18f (apone). compute_activations_cow borrows contiguous host f32 activations in place instead of alloc+memcpy every execute(); owns/widens for f16/bf16/strided/device. Eliminates ~12.9% warm-prefill copy on qwen3-0.6b/phi3.5-mini. Bit-identical (same from_raw_parts the old contiguous fast path used); token-exact; regression test proven non-vacuous. Reviewer: Drake (opus) approved (non-author). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Jul 24, 2026
…ky deleter test) (#108) ## Summary - replace shared static DLPack import-deleter counters and their serialization lock with an `Arc<AtomicUsize>` owned by each fake producer/test - apply the same hermetic counter pattern to legacy and versioned managed-tensor headers - leave production deleter/idempotency logic unchanged ## Root cause The Windows ARM64 Rust job intermittently observed `IMPORT_DROPS == 2` after one explicit deleter call. `ManagedTensorOwner::call_deleter` already sets its `deleted` guard before invoking the foreign deleter, so this was not a production double-run. The shared static test counter could instead be contaminated by a deferred deleter from another test despite lock-based reset/observe serialization. Main has been red on this pre-existing flaky failure across recent pushes. Each fake producer now carries its own counter through `manager_ctx`, so its `extern "C"` deleter can only increment the counter belonging to that test. This removes cross-test contamination by construction on every platform. Found while integrating `origin/main` into the CPU-EP performance branch (PR #105). ## Validation - `cargo test -p onnx-runtime-dlpack` (3 default-parallel runs) - `cargo test -p onnx-runtime-dlpack -- --test-threads=1` - `cargo clippy -p onnx-runtime-dlpack --all-targets -- -D warnings` - `cargo fmt --check` - `cargo check -p onnx-runtime-dlpack --target aarch64-unknown-linux-gnu` Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
# Conflicts: # crates/mlas-sys/src/lib.rs
justinchuby
added a commit
that referenced
this pull request
Jul 24, 2026
## Summary Fixes the **flaky Windows ARM64 CI crash** in `spmd_real_multi_op_int4_is_bit_identical_at_odd_worker_count` (introduced by #105). **Test-only change** — no production/kernel code touched. ### Root cause The parity test hard-coded `DECODE_THREADS=31` + `RAYON_NUM_THREADS=31` for both child subprocesses. On the constrained native **Windows ARM64** runner this oversubscribed the host and intermittently hard-crashed the child with `0xC0000005 STATUS_ACCESS_VIOLATION` (empty stderr, truncated stdout — a hard abort, not a Rust panic or a byte mismatch). Same main SHA both passed and failed → flaky. macOS arm64 and Windows x64 always passed. Ruled out (with evidence): not an aarch64 kernel bug (`selected_dot_kernel()` returns the safe-Rust `Scalar` path on aarch64 — no `unsafe`, no NEON/SVE), not a data race (full suite TSan-clean at 31 workers), not a numeric/token divergence. Also fixed a **determinism defect**: effective worker count was `min(31, cores)`, so "odd worker count" actually depended on the runner's core-count parity. ### Fix New `parity_worker_count()` = largest **odd** integer ≤ host CPUs, capped at 15. Never 0, never exceeds host CPUs (no oversubscription), deterministic (depends only on `available_parallelism()`). The ON child self-verifies the odd count; the byte-identical SPMD-vs-flat parity assertion still runs on **every platform** across all 6 sequential int4 ops. Coverage preserved. ### Review 🟢 **APPROVED** by Drake (non-author). Independently confirmed: test-only, always-odd/≥1/≤CPUs, deterministic, parity assertion runs on all platforms and is non-vacuous (byte-XOR perturbation → FAILED → revert → passed), SPMD routing asserted (≥6 dispatches). Non-blocking nit: on 1–2 CPU hosts the uneven-sharding sub-invariant degrades silently (byte-parity + routing asserts still run); optional follow-up to emit a skip-notice. ### Gates (x86_64) - `cargo test -p onnx-runtime-ep-cpu` (default, NO mlas — the failing CI config): 859 passed; parity test 5×: 5/5 no crash - `cargo test -p onnx-runtime-ep-cpu --features mlas`: 888 passed - clippy with & without `--features mlas` (`-D warnings`): clean - `cargo fmt --check`: clean (the sole mlas-sys:1211 diff is pre-existing/unrelated, a local rustfmt-version artifact) - `RUSTFLAGS="-D warnings" cargo check --target aarch64-unknown-linux-gnu`: clean Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Jul 26, 2026
justinchuby
pushed a commit
that referenced
this pull request
Aug 17, 2026
Seven session-lifetime, weight-scaled caches exist in the CPU EP and only three were ever declared to the memory-strategy plan (#1056). Each ungoverned one was written by someone who did not know the rule existed -- a bare `OnceLock<Vec<T>>` field is easy to add and invisible to the plan. Two of the three that are governed were governed only after doing damage: one took a 14B int4 model to ~66 GB, another took its peak working set from 8.17 GB to 25.5 GB while the ledger read zero. A seventh landed on main today (#1086) after I asked for accounting. `GovernedWeightCache<T>` (21cd05b) makes the accounting structural, but nothing stopped a new bare `OnceLock` field. This is the crude defence: the pattern was found by grep in twenty minutes, so a grep can guard it. It asks the author one question rather than blocking them -- if the buffer does not scale with weight size, say so and add the `weight-cache-reviewed` label to record the judgement. Validated against real history, four commits: | commit | contains | verdict | | --- | --- | --- | | `d0fdfa47` (#105) | introduces `packed_filter`/`packed_bias` | flagged | | `ac394fd6` (#1086) | introduces the QLinearMatMul pre-pack | flagged | | `21cd05b3` | adds `GovernedWeightCache` itself | passes | | `af04a613` | unrelated Exp vectorisation | passes | My first pattern missed both real cases -- it matched `OnceLock<Vec<..>>` but not `OnceLock<Option<Vec<..>>>` or `OnceLock<Option<(Key, PackedB)>>`. Worth recording because it is the same failure mode as a vacuous test: a guard that cannot fire on the cases that motivated it looks like protection and is not. The falsification against known-bad commits is what caught it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d60eb808-7cc6-4abc-b48d-2a6dd3841624
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Native onnx-genai CPU EP performance + generality work to match/beat ONNX Runtime (ORT / onnxruntime-genai / foundry-local) across per-op, whole-model decode, and generation loop — while staying general (cross-model / OS / processor) and token-exact.
All changes land here (base
main) via reviewed cherry-picks; a second squad works onorigin/main(untouched). Each change is non-author reviewed with parity + regression tests as a hard gate.Landed in this PR
Decode / LLM performance (int4/int8 MatMulNBits, GQA, RoPE, SPMD)
QuantizeARow_CompInt8) — resolves slow-drift argmax flips.Generality (enable models to run natively)
CNN / traditional-ML performance
2.5×, MobileNetV2 4.6×→1.3× (best ~1.05×), top-1 parity holds.Correctness / tests
copy_nonoverlappingUB in merged LinearAttention. (pending final review)Test state
cargo test -p onnx-runtime-ep-cpu --features mlas→ 812 lib tests green;mlas-sys20 green; clippy clean on changed code. Parity gates deterministic/load-independent.Notes
main.origin/main.Session update 2026-07-23 (cont.)
Clean load-gated scoreboard: native BEATS ORT on Qwen2.5-0.5B (2.5x) and coder-7B (2.2x) int4 decode; qwen3-0.6b now correct+2.6x-faster-8bit; Phi-3.5 large-KV gap being closed via in-place KV (follow-up).