fix(engine): accept Cast-backed QMoE scales by justinchuby · Pull Request #771 · justinchuby/onnx-genai

justinchuby · 2026-08-11T04:22:54Z

Summary

accepts one-hop Cast(initializer) scale tensors during static QMoE placement
includes native CUDA KV capacity prerequisite needed by the real DeepSeek-V2-Lite decode lock
records GLM/DeepSeek native load-gap tractability memo

Validation

CUDA_VISIBLE_DEVICES=2 cargo test -p onnx-genai-engine --features native-backend,cuda native_kv_reservation -- --nocapture
CUDA_VISIBLE_DEVICES=2 cargo test -p onnx-genai-engine --features native-backend,cuda qmoe_static_placement -- --nocapture
ONNX_GENAI_CUDA_KV_MAX_LEN=4096 DEEPSEEK_V2_LITE_CUDA_DIR=/home/justinchu/ds-e2e-artifacts/deepseek-v2-lite-real-int4 CUDA_VISIBLE_DEVICES=2 cargo test -q -p onnx-genai-engine --features native-backend,cuda --test deepseek_v2_lite_decode_lock -- --ignored --nocapture

DeepSeek native CUDA token stream: [11, 304, 608, 245].

Trace one-hop Cast(initializer) inputs when planning static QMoE weight placement so DeepSeek-V2-Lite's fp16 scale initializers cast to fp32 remain placement-classifiable. Includes the native KV reservation cap prerequisite needed for the real DeepSeek decode lock. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

codecov · 2026-08-11T04:29:32Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.36%. Comparing base (1179f7b) to head (2ba9b2d).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #771      +/-   ##
==========================================
- Coverage   79.88%   79.36%   -0.53%     
==========================================
  Files         331      331              
  Lines      137462   137462              
  Branches   137462   137462              
==========================================
- Hits       109812   109096     -716     
- Misses      23306    24026     +720     
+ Partials     4344     4340       -4

Flag	Coverage Δ
cli-ort-linux	`83.41% <ø> (ø)`
cli-ort-windows	`82.90% <ø> (ø)`
mlas	`80.96% <ø> (ø)`
offline	`79.17% <ø> (-0.56%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 4 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Treat usize::MAX CUDA KV capacity as unbounded rather than reservable so models without a declared max context retain the warning-and-load behavior. Keep finite runtime caps preferred over metadata. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-08-11T05:03:00Z

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	`block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024`	42.65 µs	132.75 µs	+211.3%
🔴	`block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024`	531.34 µs	1.36 ms	+155.3%
🔴	`matmul/medium_generic_f32_threads=8/32x512x512`	889.22 µs	2.22 ms	+149.7%
🔴	`block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024`	47.98 µs	95.32 µs	+98.7%
🔴	`matmul/large_generic_bf16_threads=8/32x1024x1024`	1.33 ms	2.61 ms	+95.6%
🔴	`matmul/medium_generic_bf16_threads=8/32x512x512`	363.37 µs	680.94 µs	+87.4%
🔴	`matmul/large_generic_f32_threads=8/32x1024x1024`	3.54 ms	6.63 ms	+86.9%
🔴	`matmul/small_generic_f32_threads=8/1x256x256`	31.57 µs	51.02 µs	+61.6%
🔴	`gather/large_bf16_threads=1-internal/131072`	9.06 µs	14.58 µs	+61.0%
🔴	`matmul/medium_generic_f16_threads=8/32x512x512`	28.43 µs	44.61 µs	+56.9%
🔴	`gather/small_f16_threads=1-internal/4096`	518.9 ns	808.9 ns	+55.9%
🔴	`block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1`	349.04 µs	542.41 µs	+55.4%
🔴	`matmul/large_generic_f16_threads=8/32x1024x1024`	78.59 µs	120.41 µs	+53.2%
🔴	`matmul/medium_generic_f16_threads=1/32x512x512`	27.67 µs	39.56 µs	+43.0%
🔴	`matmul/small_generic_bf16_threads=8/1x256x256`	29.34 µs	39.49 µs	+34.6%
🔴	`matmul/small_generic_bf16_threads=1/1x256x256`	32.15 µs	43.24 µs	+34.5%
🔴	`matmul/large_generic_f16_threads=1/32x1024x1024`	72.83 µs	96.65 µs	+32.7%
🔴	`matmul/small_generic_f16_threads=8/1x256x256`	28.60 µs	37.26 µs	+30.3%
⚠️	`gather/large_f16_threads=1-internal/131072`	10.75 µs	13.58 µs	+26.4%
⚠️	`matmul/small_generic_f16_threads=1/1x256x256`	28.74 µs	35.80 µs	+24.6%
⚠️	`tokenization/decode_tokens_per_second`	6.25 ms	7.44 ms	+19.0%
⚠️	`matmul/medium_generic_f32_threads=1/32x512x512`	2.21 ms	2.60 ms	+17.5%
⚠️	`matmul/large_generic_bf16_threads=1/32x1024x1024`	1.90 ms	2.20 ms	+15.9%
✅	`grammar_masking/llguidance_compute_mask/32`	74.93 µs	85.56 µs	+14.2%
✅	`gather/medium_f32_threads=1-internal/32768`	3.69 µs	4.21 µs	+14.2%
✅	`block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1`	71.95 µs	82.11 µs	+14.1%
✅	`matmul/medium_generic_bf16_threads=1/32x512x512`	568.08 µs	644.81 µs	+13.5%
✅	`add/small_bf16_threads=1-internal/1024`	14.20 µs	16.08 µs	+13.2%
✅	`matmul/large_generic_f32_threads=1/32x1024x1024`	8.82 ms	9.93 ms	+12.6%
✅	`sampling_latency/top_p_per_token`	383.92 µs	430.87 µs	+12.2%
✅	`gather/large_f32_threads=1-internal/131072`	27.91 µs	31.28 µs	+12.1%
✅	`qwen3_sampling_processors/top_k_full_sort_baseline`	2.19 ms	2.42 ms	+10.8%
✅	`sampling_latency/min_p_per_token`	216.75 µs	236.68 µs	+9.2%
✅	`qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline`	3.62 ms	3.91 ms	+8.1%
✅	`gather/medium_bf16_threads=1-internal/32768`	2.29 µs	2.45 µs	+7.1%
✅	`gather/small_f32_threads=1-internal/4096`	793.1 ns	846.8 ns	+6.8%
✅	`matmul/small_generic_f32_threads=1/1x256x256`	36.05 µs	38.14 µs	+5.8%
✅	`sampling_latency/top_k_per_token`	51.59 µs	54.22 µs	+5.1%
✅	`add/medium_f32_threads=1-internal/262144`	2.87 ms	3.01 ms	+4.8%
✅	`kv_cache/alloc_dealloc_pages`	39.43 µs	40.45 µs	+2.6%
✅	`tokenization/encode_tokens_per_second`	487.86 µs	495.58 µs	+1.6%
✅	`qwen3_sampling_processors/top_p_fast_after_top_k`	524.59 µs	532.58 µs	+1.5%
✅	`gather/small_bf16_threads=1-internal/4096`	529.3 ns	535.3 ns	+1.1%
✅	`add/large_f32_threads=1-internal/4194304`	43.14 ms	43.12 ms	-0.0%
✅	`qwen3_sampling_processors/top_k_partial_selection`	158.16 µs	158.08 µs	-0.1%
✅	`qwen3_sampling_processors/top_k_top_p_fast`	668.66 µs	668.05 µs	-0.1%
✅	`sampling_latency/greedy_per_token`	3.38 µs	3.29 µs	-2.6%
✅	`gather/medium_f16_threads=1-internal/32768`	2.47 µs	2.40 µs	-2.7%
✅	`add/large_bf16_threads=1-internal/4194304`	48.63 ms	45.85 ms	-5.7%
✅	`qwen3_sampling_processors/top_k_top_p_full_sort_baseline`	6.46 ms	5.89 ms	-8.9%
✅	`reduce_mean/large_f32_threads=1-internal/262144`	1.23 ms	1.10 ms	-10.7%
✅	`add/small_f16_threads=1-internal/1024`	17.20 µs	15.34 µs	-10.8%
✅	`logit_processing/seven_processor_chain_per_step`	357.89 µs	316.62 µs	-11.5%
✅	`add/large_f16_threads=1-internal/4194304`	50.98 ms	44.66 ms	-12.4%
🟢	`add/medium_bf16_threads=1-internal/262144`	3.44 ms	2.76 ms	-19.6%
🟢	`reduce_mean/medium_f32_threads=1-internal/65536`	325.42 µs	261.65 µs	-19.6%
🟢	`reduce_mean/small_f32_threads=1-internal/4096`	20.11 µs	15.70 µs	-21.9%
🟢	`add/medium_f16_threads=1-internal/262144`	3.55 ms	2.75 ms	-22.5%
🟢	`add/small_f32_threads=1-internal/1024`	426.2 ns	226.9 ns	-46.8%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 4.14 3.99 6.21 }

What this cannot catch

Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
Sub-threshold regressions that compound over multiple PRs
Performance changes that only manifest under GPU execution
Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

## Summary - gate the native KV reservation helper behind `native-backend` - gate the four helper-specific unit tests the same way - fixes ORT-only CLI clippy dead_code from #770/#771 ## Validation - Reproduced before fix: `cargo clippy --locked -p onnx-genai-cli --all-targets -- -D warnings` failed with dead_code on `native_kv_reservation_max_context` - After fix: `cargo clippy --locked -p onnx-genai-cli --all-targets -- -D warnings` - After fix: `cargo clippy --locked -p onnx-genai-engine --no-default-features --features cuda-13000 -- -D warnings` - After fix: `CUDA_VISIBLE_DEVICES=2 LD_LIBRARY_PATH=/home/tlwu/cudnn9.19_cuda13/lib:$LD_LIBRARY_PATH cargo clippy -p onnx-genai-engine --features "cuda native-backend" -- -D warnings` - `cargo fmt --all` Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

justinchuby · 2026-08-11T06:02:46Z

Why not constant cast first?

…-27B native CUDA (#779) ## Summary Enables **Qwen/Qwen3.6-27B** (`model_type: qwen3_5`, hybrid GDN `LinearAttention`/`CausalConvWithState` + periodic `GroupQueryAttention` + int4 `MatMulNBits`) to load and decode on the **native CUDA EP**, byte-exact vs an fp32 oracle. This was the last model the ORT-vs-native benchmark flagged as _"Unsupported native load: model needs unsupported native operators"_. ## Root cause (not a kernel gap, not a bad graph) The 27B `model.onnx` is a **correct** hybrid graph — 64 layers = 48 GDN linear-attention layers (`conv_state`/`recurrent_state`) + 16 periodic full-attention layers (dense `key`/`value`) — and the native kernels already exist. The blocker is the artifact's **thin `inference_metadata.yaml`**: it declares only `grouped_query_attention` and **no `io` port contract**. On the native load path the Resource Governor derives per-layer KV page byte geometry from `model.io.kv_inputs`/`kv_outputs`; with no `io` block `resolve_kv_layers` returns `None` (it deliberately never guesses KV pairing from tensor names), so load failed with `cannot derive the KV memory budget because per-layer KV page geometry is unknown`. ## Fix (DRY, attribute/shape-driven — no model-name gate) The native loader now auto-derives the decoder `io` port contract from the ONNX graph's own port inventory when the sidecar declares none, via `GenAiConfig::derive_decoder_io_from_graph` — the exact proven derivation the native decode step driver already uses in `derive_fallback_io`. New helper `maybe_fill_hybrid_io_from_graph` (`engine/load.rs`) fills the sparse dense `kv_inputs`/`kv_outputs` + fixed recurrent `state_pairs` (and token/mask/ position/logits port names by presence). It is gated on a non-empty derived `state_pairs` (the recurrent-hybrid case shape inference cannot classify); a declared `io` block always wins and pure-dense decoders are untouched. Added `Default` to `ModelCapabilities` so the derived spec can attach when `model` is absent. This makes **all** future hybrid GDN models with a correct graph but thin metadata auto-run the optimal native path. ## Validation (H200) - Native-CUDA fp16 greedy of `"The capital of France is"` → `[11751, 13, 271, 248068, 271, 248069, 271, 4639]` = `" Paris.\n\n<think>\n\n</think>\n\nThat"` (coherent). - Teacher-forced next token (fresh engine): native-CUDA fp16 = **11751** " Paris" (logprob −0.6080). - **fp32 oracle** (fp16→fp32 up-conversion, int4 preserved, native CPU) teacher-forced argmax = **11751** " Paris" (logprob −0.6100, top-1 margin **2.549** nats). Byte-exact parity ✅ — fp16/fp32 logits near-identical. Locked by `crates/onnx-genai-engine/tests/qwen35_27b_hybrid_native_cuda_e2e.rs` (ignored; env `QWEN35_27B_DIR`, `QWEN35_27B_FP32_ORACLE_DIR`), mirroring the 35B QMoE oracle lock. 🤖 Working as Cohaagen (native-model-enablement). Follows the GLM (#770) / DeepSeek-V2-Lite (#771) playbook. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…detail (#825) Justin asked to update `docs/PROGRESS.md` with the latest progress and drop stale info. It had grown to **1166 lines** of day-by-day changelog spanning 2026-07-15 → 2026-07-27, with the entire recent arc missing. ## Changes (1166 → 303 lines) - **Added recent milestones (2026-07-28 → 2026-08-12), newest-first:** - EP plugins run inside ORT via the plugin-EP C ABI (#762) and ship on PyPI — **`nxrt-ep-cpu` + `nxrt-ep-cuda` 0.1.0.dev5 both LIVE** (#819/#824/#820). - Absent-optional-output memory-safety wave (#762 review; Miri-clean). - VMM / KV-layout / offload / batching residency wave — managed no-spill VMM default, layout-governed KV floor (768× reduction), offload+capture coexist, #736 over-reservation audit (#755–#814). - Issue triage + autonomous fixes (#784/#785/#786/#788/#789); CPU-EP TopK correctness+perf (#774/#775). - Qwen3.5/3.6-27B hybrid Gated-DeltaNet native CUDA (#779); GLM-4-9B + DeepSeek-V2-Lite native + ORT 1.28 3-config fairness (#766/#770/#771). - Fused QMoE decode kernel + QMoE arc concluded; CUDA-graph capture trilogy, version-selectable CUDA (#708/#728/#757/#760), megakernel study (#769). - **Compressed stale detail:** the ~40 dated July day-by-day sections + verbose diffusion/LoRA/ORT2 narrative collapsed into a ~35-line "Foundational milestones" spine; superseded perf-wave numbers dropped; §1–§40 table condensed to a compact grid. - **Header refreshed:** last-updated → 2026-08-12; Published line now lists both EP plugins on PyPI + Miri CI. - **Preserved & updated** the DeepSeek/GLM known-gaps checklists and added new open items. Docs-only; full narrative remains in `.squad/decisions-archive/` and `docs/benchmarks/`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

justinchuby enabled auto-merge (squash) August 11, 2026 04:22

justinchuby merged commit 6b1e02e into main Aug 11, 2026
13 of 15 checks passed

justinchuby deleted the squad/deepseek-qmoe-cast-placement branch August 11, 2026 04:46

justinchuby mentioned this pull request Aug 11, 2026

fix(engine): gate native-only KV helper behind native-backend #773

Merged

justinchuby mentioned this pull request Aug 11, 2026

feat(native): derive hybrid decoder io from graph to load Qwen3.5/3.6-27B native CUDA #779

Merged

justinchuby mentioned this pull request Aug 12, 2026

docs: refresh PROGRESS.md — Aug 2026 milestones, compress stale July detail #825

Merged

Navigation Menu

Repository navigation

fix(engine): accept Cast-backed QMoE scales - #771

fix(engine): accept Cast-backed QMoE scales#771
justinchuby merged 2 commits into
mainfrom
squad/deepseek-qmoe-cast-placement

justinchuby commented Aug 11, 2026

Uh oh!

codecov Bot commented Aug 11, 2026 •

edited

Loading

Uh oh!

Uh oh!

github-actions Bot commented Aug 11, 2026

Uh oh!

justinchuby commented Aug 11, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

justinchuby commented Aug 11, 2026

Summary

Validation

Uh oh!

codecov Bot commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

github-actions Bot commented Aug 11, 2026

🔴 Benchmark Regression Detected

Uh oh!

justinchuby commented Aug 11, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

codecov Bot commented Aug 11, 2026 •

edited

Loading