docs(#1295): MoE VMM granularity trade curve — one granule, two experts (not commit-and-subdivide)

[justinchuby]

#1295 follow-up — MoE VMM sub-granule packing: the trade curve (analysis, no impl)

Reporting before implementing, as the owner asked. This answers "can VMM commit more at once and subdivide ourselves?" from the already-measured granite router-skew distribution — no packing code was written.

The question and the trap

The 2 MiB CUDA granule wastes 2.7× on granite int4 experts (~0.75 MiB). The proposed escape — commit a large region (tens of MiB), pack ~85 experts, subdivide ourselves — is favourable on granule waste alone. But committing is also the unit of residency, so the real question is whether sub-division keeps selectivity.

Finding (measured, granite-3.0-1b-a400m, 32e/top-8/24L, real IBM router, CPU EP, 192 decode steps; expert 0.75 MiB, granule 2 MiB)

The granularity that minimises resident mapped physical — and the owner's own waste × transferred product — is ONE 2 MiB granule holding TWO experts (E=2), not a large region.

E	waste	resident MiB/layer/step	transfer MiB	sel%
1 (per-expert)	2.67×	16.00	6.00	100%
2 ← optimum	1.33×	13.34	10.01	167%
8	1.00×	19.07	19.07	318%
32 (whole bank)	1.00×	24.00	24.00	400%

Resident physical is U-shaped with its minimum at E=2 and rises back to whole-bank levels for every larger chunk — the "tens of MiB" proposal is at the worst end. E=2 beats per-expert (16.00→13.34, −17%) because the second expert in a granule you already pay for is free physical.

Why large regions are fatal (inferred)

Every decode step activates all 24 layers, so any chunk spanning layers is hot in 100% of steps. A 64 MiB region (~85 experts) can't fit one layer's 24 MiB bank, so it must span layers → always resident → whole-bank pinning. The "near-zero waste" arithmetic is real but irrelevant; the binding objective is resident physical.

The other answers

• Hotness-sorted packing rescues selectivity, modestly (~5–14% vs arbitrary at every E; does not rescue large chunks).
• Static packing is stable: decays only 3.4–7.1% across prose/code/math vs a per-prompt oracle — so packing once is justified. Layer-1/2 always-on experts stay free pins.
• Interaction with Streaming-regime batch-N ceiling is VMM map/unmap churn, not the batching path (N_max~4-5 vs projected ~19 on 8GB) #1295/vmm(#1295): usable-VRAM admission cap (cliff->plateau) + vram_free_ms attribution fix #1325: span count is not the cliff driver, so E=2 does not help the cliff — but lower peak mapped physical raises the admission-cap plateau height (not N_max). Single shared authority preserved. This is not conflated in the doc.

Decisive framing (which architecture this assumes)

• If we keep our own VMM paging: the minimal correct change is E=2 (two experts/granule, hotness-packed, within-layer) — provably the trade-curve optimum, and small (still commits exactly one granule). Do not build large-region commit-and-subdivide — dominated on every metric.
• If OS hinting works, sub-division is unnecessary: 4 KiB OS paging dissolves the granule problem (zero waste + full per-expert selectivity, strictly better than any VMM packing).
• My Streaming-regime batch-N ceiling is VMM map/unmap churn, not the batching path (N_max~4-5 vs projected ~19 on 8GB) #1295 finding leans toward OS paging: the binding cost is WDDM fault-in beyond usable VRAM, not our bookkeeping — so we may be fighting for control we don't benefit from. The condition to not build even E=2 is the OS-hint agent confirming WDDM honours residency hints with an adequate prediction window. I did not measure hinted-vs-unhinted fault-in (their job) but recorded that per-page fault-in is exactly what a prefetch hint would target.

Reproduction

scripts/moe_granularity_analysis.py reuses the exact moe_router_skew.py probe, captures per-step selection sets (the committed JSON has only aggregates), and prints the curve for both packings. Deterministic, bit-for-bit.

Box: i7-13800H, RTX 4060 Laptop 8GB, driver 591.55, CUDA 13.1, WDDM. Analysis is CPU-EP (the which experts question is dtype-/EP-independent); granule is the CUDA device granule. No GPU decode claimed. Doc-only + reproducible script; no source change, nothing to test beyond the script's deterministic output.

Follow-up to #1295 (closed by #1325).

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	48.97 µs	139.64 µs	+185.2%
🔴	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	45.73 µs	128.64 µs	+181.3%
🔴	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	45.98 µs	88.72 µs	+92.9%
🔴	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	563.51 µs	950.89 µs	+68.7%
🔴	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	364.25 µs	610.25 µs	+67.5%
🔴	gather/large_f16_threads=1-internal/131072	9.49 µs	15.84 µs	+67.0%
🔴	gather/medium_f32_threads=1-internal/32768	3.53 µs	5.63 µs	+59.6%
🔴	matmul/small_generic_f32_threads=1/1x256x256	33.91 µs	51.22 µs	+51.0%
🔴	gather/large_bf16_threads=1-internal/131072	9.57 µs	14.05 µs	+46.8%
🔴	matmul/medium_generic_f32_threads=8/32x512x512	886.45 µs	1.26 ms	+41.8%
🔴	matmul/small_generic_bf16_threads=8/1x256x256	29.09 µs	41.01 µs	+41.0%
🔴	gather/large_f32_threads=1-internal/131072	23.24 µs	31.66 µs	+36.2%
🔴	matmul/medium_generic_bf16_threads=8/32x512x512	365.96 µs	497.69 µs	+36.0%
⚠️	gather/medium_f16_threads=1-internal/32768	2.25 µs	2.93 µs	+30.0%
⚠️	matmul/medium_generic_f16_threads=1/32x512x512	27.74 µs	35.75 µs	+28.9%
⚠️	gather/small_f16_threads=1-internal/4096	438.2 ns	549.9 ns	+25.5%
⚠️	matmul/small_generic_f16_threads=1/1x256x256	27.87 µs	34.93 µs	+25.3%
⚠️	matmul/small_generic_bf16_threads=1/1x256x256	29.28 µs	36.45 µs	+24.5%
⚠️	gather/small_bf16_threads=1-internal/4096	440.4 ns	542.5 ns	+23.2%
⚠️	matmul/large_generic_f32_threads=1/32x1024x1024	8.60 ms	10.59 ms	+23.1%
⚠️	reduce_mean/large_f32_threads=1-internal/262144	921.04 µs	1.12 ms	+22.0%
⚠️	matmul/small_generic_f32_threads=8/1x256x256	31.57 µs	38.09 µs	+20.6%
⚠️	gather/small_f32_threads=1-internal/4096	619.5 ns	746.8 ns	+20.5%
⚠️	matmul/medium_generic_f16_threads=8/32x512x512	27.19 µs	32.71 µs	+20.3%
⚠️	grammar_masking/llguidance_compute_mask/32	70.52 µs	84.08 µs	+19.2%
⚠️	logit_processing/seven_processor_chain_per_step	319.71 µs	378.77 µs	+18.5%
⚠️	gather/medium_bf16_threads=1-internal/32768	2.31 µs	2.73 µs	+18.1%
✅	matmul/large_generic_f16_threads=8/32x1024x1024	78.52 µs	89.14 µs	+13.5%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	490.01 µs	556.03 µs	+13.5%
✅	matmul/small_generic_f16_threads=8/1x256x256	28.55 µs	32.22 µs	+12.8%
✅	kv_cache/alloc_dealloc_pages	39.52 µs	44.13 µs	+11.7%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.15 ms	2.39 ms	+11.5%
✅	qwen3_sampling_processors/top_k_top_p_fast	675.27 µs	748.66 µs	+10.9%
✅	matmul/large_generic_f16_threads=1/32x1024x1024	73.78 µs	81.39 µs	+10.3%
✅	matmul/large_generic_f32_threads=8/32x1024x1024	3.67 ms	4.01 ms	+9.1%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	1.85 ms	2.02 ms	+9.1%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	5.98 ms	6.50 ms	+8.6%
✅	matmul/large_generic_bf16_threads=8/32x1024x1024	1.32 ms	1.40 ms	+6.1%
✅	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.36 ms	3.55 ms	+5.5%
✅	add/medium_f16_threads=1-internal/262144	95.61 µs	99.62 µs	+4.2%
✅	reduce_mean/medium_f32_threads=1-internal/65536	239.12 µs	246.37 µs	+3.0%
✅	reduce_mean/small_f32_threads=1-internal/4096	13.80 µs	14.14 µs	+2.4%
✅	sampling_latency/min_p_per_token	210.50 µs	214.40 µs	+1.9%
✅	add/large_bf16_threads=1-internal/4194304	1.51 ms	1.54 ms	+1.6%
✅	add/small_f16_threads=1-internal/1024	416.9 ns	423.5 ns	+1.6%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	2.15 ms	2.15 ms	-0.0%
✅	add/medium_bf16_threads=1-internal/262144	95.43 µs	94.45 µs	-1.0%
✅	sampling_latency/top_p_per_token	401.26 µs	395.70 µs	-1.4%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	541.32 µs	531.64 µs	-1.8%
✅	add/small_bf16_threads=1-internal/1024	408.2 ns	400.7 ns	-1.8%
✅	add/large_f16_threads=1-internal/4194304	1.56 ms	1.52 ms	-2.2%
✅	add/medium_f32_threads=1-internal/262144	22.73 µs	22.20 µs	-2.3%
✅	add/small_f32_threads=1-internal/1024	191.3 ns	179.0 ns	-6.5%
✅	tokenization/encode_tokens_per_second	426.11 µs	390.29 µs	-8.4%
✅	sampling_latency/top_k_per_token	61.06 µs	54.49 µs	-10.8%
✅	add/large_f32_threads=1-internal/4194304	654.46 µs	576.92 µs	-11.8%
✅	qwen3_sampling_processors/top_k_partial_selection	171.57 µs	149.87 µs	-12.6%
✅	sampling_latency/greedy_per_token	4.01 µs	3.43 µs	-14.4%
🟢	tokenization/decode_tokens_per_second	9.20 ms	6.66 ms	-27.6%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 4.66 3.50 4.06 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.00%. Comparing base (1bd7e9e) to head (fce05a6).
⚠️ Report is 56 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##           main    #1332       +/-   ##
=========================================
+ Coverage      0   80.00%   +80.00%     
=========================================
  Files         0      362      +362     
  Lines         0   157408   +157408     
  Branches      0   157408   +157408     
=========================================
+ Hits          0   125935   +125935     
- Misses        0    26839    +26839     
- Partials      0     4634     +4634     

Flag	Coverage Δ
offline	80.00% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.
see 362 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.