Repository navigation
docs(bench): MoE expert-cache policy headroom — oracle vs static hot-pin (trace-driven) - #1321
Merged
Merged
Conversation
… static hot-pin)
Offline decision aid for the over-subscribed MoE regime (expert bank > VRAM):
replays a real trained-router expert-selection trace (granite-3.0-1b-a400m, 32
experts, top-8, 24 layers, 192 decode steps across prose/code/math) against a
bounded, GLOBALLY-shared expert cache and compares eviction policies.
Adds:
- scripts/dump_moe_expert_trace.py per-step per-layer selection trace capture
- scripts/moe_expert_trace.json the captured trace (reproduced 2026-08-18)
- scripts/moe_expert_cache_sim.py the simulator (global budget; oracle/MIN,
random, FIFO, LRU, LFU, static hot-pin,
hybrid; per-layer equal-split baseline)
- scripts/moe_cache_sim_results.json results
- docs/benchmarks/2026-08-18-moe-expert-cache-policy-headroom.md writeup
Key measured findings (bytes/token @16 MiB, 25-50% budget regime):
- Headroom is real: oracle transfers 46-67% fewer bytes than random -> a policy
roughly halves transfer. The residency-policy line is NOT dead (unlike dense).
- Static global frequency-pin closes ~50-84% of the random->oracle gap with zero
runtime prediction, beats LRU/FIFO/LFU and the hybrid, and is durable across
prompt type (cross-prompt miss decay only +0.07..+0.13). Ship it first.
- Knee at ~50% budget (oracle miss 0.68->0.37->0.16 at 10/25/50%); below 25% you
page most accesses regardless of policy.
- Global budget beats per-layer equal split by ~31% (25%) / ~17% (50%) for the
oracle -> hot layers (1-2 always-on) deserve more slots.
Caveats recorded: 75-90% rows are cold-start-dominated over a 64-token trace;
generalisation to DeepSeek-class 8-of-256 routers is INFERRED (likely more
headroom, not measured); says nothing about wall-clock or paging-mechanism cost.
No runtime policy built; reported to owner first per directive.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 18, 2026
…ling (trace-driven) (#1322) ## What & why Extends the single-tier policy-headroom study (#1321) to test **three claims from an external MoE-offload analysis** against the same real trained-router trace, so the owner can decide tiering/scheduling direction on **measurement, not assumption**. Trace behind every number: `granite-3.0-1b-a400m-instruct` (real IBM trained router, 32 experts, top-8, 24 layers), 3 prompts × 64 greedy decode steps, **CPU-only replay — no GPU**. Box (provenance, house rule §32.2): **i7-13800H (14C/20T), RTX 4060 8 GB, WDDM**. Measured vs inferred kept strictly separate. ## Arm 1 — Concentration curve (Zipf premise): **MILDER than assumed** The analysis assumes "top 12.5% of experts → ~80% of traffic". Measured on granite: **top 12.5% of keys carry 27.1%**, top 25% → 45.6%. Per-layer top-8/32 = **45.4%** (reproduces prior), Gini **0.343** (cross-checks the independent 0.334 → capture validated). Skew is real and exploitable but **tiering wins are smaller than 80/20 implies** on a 32-expert router. *(Inferred: DeepSeek-class 8-of-256 likely more concentrated — not measurable here.)* ## Arm 2 — Three-tier VRAM/DRAM/SSD: **DRAM removes the SSD from the hot path** Exclusive two-level LRU, separating compulsory (first-touch) from capacity SSD traffic. At **DRAM ≥ 100% of bank, avoidable SSD traffic → 0** (only compulsory remains). Confirms the analysis's central claim: **the SSD leaves the critical path; DirectStorage/GDS is a cold-miss backstop, not the main event.** But **PCIe DRAM→VRAM then dominates** (~1.8–2.8 GB/token @16 MiB), tying back to VRAM residency/pinning: *tier to delete the SSD, then pin to shrink the PCIe bill*. *(Bandwidth constants PCIe/NVMe/DDR5 are **assumed/inferred**, not measured on this box; ms/token is relative only.)* ## Arm 3 — Route-aware scheduling: **real, but batch size is the bigger lever and reordering has a fairness price** Each decode step = a request needing its 192 keys; batch loads the union once. Two schedulers over a lookahead window Q: FIFO vs overlap-greedy. - **Batching alone (FIFO) is the biggest, cost-free win:** W=2→8 cuts bytes/token 3072→1208 (**−61%**) at zero reorder cost. - **Route-aware adds 17–39% over FIFO** and grows with Q — *as the analysis claims* — **but reordering buys bandwidth with latency**: full-lookahead W=8 saves 39% yet pushes p99 delay to **102 positions** (a request waits behind ~half the queue). **Bounded lookahead Q≈2–4×W** captures ~half the reorder win for p99 delay 7–25 — the defensible operating point. Unbounded overlap maximisation is not a serving win. ## Files - `scripts/moe_concentration_curve.py`, `scripts/moe_tier_sim.py`, `scripts/moe_route_aware_sched.py` + `*_results.json` - `docs/benchmarks/2026-08-18-moe-tiering-and-route-aware-scheduling.md` (writeup; links the merged router-skew record & single-tier study) Reproduce: `python scripts/dump_moe_expert_trace.py && python scripts/moe_concentration_curve.py && python scripts/moe_tier_sim.py && python scripts/moe_route_aware_sched.py`. No runtime policy built; measurement reported to owner first per directive. Analysis/measurement only — ours to merge. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 18, 2026
… tok/s (#1326) ## What & why Composes the two prior trace-driven studies (#1321 single-tier residency, #1322 tiering) into **the number a deployer needs**: how much VRAM a large MoE needs to hit a target tok/s. Assumes a **DRAM tier holds the whole bank** (so SSD is out of the critical path — #1322 result), leaving **PCIe DRAM→VRAM as the only per-step expert-transfer cost**, and applies the best practical policy (**static hot-pin**) from #1321. Routing is **measured** on the granite trace (32 experts, top-8, 24 layers, 192 decode steps, real trained router; **CPU-only, no GPU**). Every **bandwidth constant is inferred/assumed**, not measured on this box. Box: **i7-13800H (14C/20T), RTX 4060 8 GB, WDDM**. ## Headline (measured routing, inferred bandwidth; 16 MiB experts) **VRAM 25%, batch 1, PCIe 4.0 x8 (~13 GB/s): 1.63 GB/token → ≈ 8 tok/s from expert transfer ALONE, before any compute.** A bank that overflows VRAM is PCIe-bound in the single-digit-to-low-double-digit tok/s range. **DirectStorage/faster-SSD cannot help — the SSD is already out of the path; the wall is PCIe.** ## The curve & the two levers - Static hot-pin sits **~17–27% above the Belady oracle** across the range (consistent with #1321). - **No sharp knee.** Concentration is milder than the assumed 80/20 Zipf, so VRAM must cover a **large** share of the bank (~50–65%) to make transfer cheap — *later* than a Zipf assumption predicts. Even at 50% VRAM you still stream ~49 experts/token. - **Batching** W=8 roughly halves GB/token (25%: 1.63 → 0.81). ## Rule of thumb — min VRAM % for target tok/s (transfer floor only, inferred BW) | batch | link | 5 | 10 | 20 | 40 tok/s | |---|---|---|---|---|---| | W=1 | x8 (13) | 5% | 35% | 65% | 75% | | W=1 | x16 (26) | 5% | 5% | 35% | 65% | | W=8 | x16 (26) | 5% | 5% | 5% | 50% | | W=8 | PCIe5 (55) | 5% | 5% | 5% | 5% | Batch size and PCIe generation matter as much as VRAM. ## Caveats (recorded in the doc) - **The W=8 byte savings are NOT free in wall-clock.** A sibling measurement finds **M=1→2 costs ~5.4×/step for 2× work** (batch-decode penalty, GEMV excluded) — cross-referenced. The batching win holds in **bytes**, not yet in tokens/second. - Trace-driven sim gives no wall-clock, no paging-mechanism cost, and no proof skew generalises to a 256-expert router (expected *more* concentrated → earlier knee; granite is the conservative flat end). - Mechanism choice (VMM vs OS page cache vs DirectStorage) is a separate slice; this strengthens the cheapest-first (OS page cache) instinct there. ## Files - `scripts/moe_pcie_floor.py` + `scripts/moe_pcie_floor_results.json` - `docs/benchmarks/2026-08-18-moe-offload-pcie-floor.md` (links #1321/#1322 docs) Reproduce: `python scripts/moe_pcie_floor.py`. Analysis/measurement only — ours to merge. No runtime policy, no #82/#87 wiring. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Aug 18, 2026
…e page cache, PCIe wall, what to build (#1333) ## MoE expert-offload architecture — synthesis of three converged workstreams Composes the mechanism, policy, and batching findings into one authoritative document, `docs/memory/MOE_OFFLOAD_ARCHITECTURE.md`, in the honesty register of `MEMORY_MANAGEMENT_MODEL_DESIGN.md` (measured vs inferred separated; hardware/model/regime on every figure; per-agent attribution so any claim traces to its evidence). ### The architectural conclusion of the week **Three regimes** decide whether a routing-aware cache is worth building (measured on this box — RTX 4060/WDDM, 63.8 GB DDR5-6400, gemma-3-27b, routing-skewed pattern under swept memory pressure): | regime | condition | OS captures | headroom | |---|---|---|---| | 1 | cache ≥ full bank | ~100% | ≈0 | | 2 | hot-set < cache < bank | ~88% (LRU exploits skew for free) | small | | 3 | cache ≲ hot-set | **33% hot-warm** | **~3× pin win** | - **Host/PCIe crossover:** warm host reads (~10.5 GB/s) exceed every host→device path (zero-copy 5.9, staged 10.9 GB/s), so in regimes 1–2 the host tier is not the bottleneck — **PCIe is**. Only regime-3 eviction-miss traffic drags host throughput below PCIe. - **CUDA UM is empirically closed on WDDM** (~0.6 GB/s, `concurrent=0`, `cuMemAdvise_v2`/`cuMemPrefetchAsync_v2` return `INVALID_DEVICE`) — recorded as a hard negative so nobody re-proposes it. - **Layout constraint gates the whole storage family:** only canonical MatMulNBits int4 / `BlockQuantizedMoE` are file-backed-usable; Marlin and MLAS-prepacked are excluded. - **DirectStorage/GDS** are PCIe-bound Regime-B backstops (bank > host RAM), not breakthroughs; the D3D12→CUDA interop link is flagged inferred-not-measured. ### Composed with the other workstreams (cited, not re-derived) - **Policy** (MoE agent, #1321/#1322/#1326/#1331): concentration milder than 80/20 (25% → 45.6%); static hot-pin within 17–27% of Belady; PCIe floor **1.63 GB/token → ~8 tok/s** (25% VRAM, batch 1, x8); no knee; **dynamic prefetch foreclosed** by a ≈0 intra-layer window. - **Batching**: **−61% bytes/token** (W=2→8) is the largest lever, but **blocked** behind a ~5.4× M=1→2 step overhead. ### What to build / not build 1. Routing-aware **static** hot-pin — **regime 3 only** (near-worthless in 1–2; dynamic prefetch foreclosed). 2. Target (hundreds-of-GB MoE) sits in regime 3 → worth building, but **cannot be validated on this 8 GB laptop**; the doc states what would (Linux datacentre GPU + real MoE). 3. **Quantisation** = best return per complexity (linear PCIe relief, no policy), at a stated accuracy cost. 4. DirectStorage/GDS = Regime-B backstop only. 5. Never re-propose CUDA UM on WDDM. Cross-links the result back into `MEMORY_MANAGEMENT_MODEL_DESIGN.md` (which said MoE "should be measured before any residency policy is designed"); its dense conclusions (`reads_per_step=1.000` #944, ~30× OS win #864/#874) kept intact and reframed as regime 1 with no reuse. Docs-only; nothing under `docs/` is consumed by a build script (verified). Measured on RTX 4060 / driver 591.55 / CUDA 13.1 / WDDM. Closes #1266. Co-authored-by: justinchuby <223556219+Copilot@users.noreply.github.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1321 +/- ##
=========================================
+ Coverage 0 80.11% +80.11%
=========================================
Files 0 364 +364
Lines 0 160729 +160729
Branches 0 160729 +160729
=========================================
+ Hits 0 128775 +128775
- Misses 0 27301 +27301
- Partials 0 4653 +4653
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
|
| Status | Scenario | Base | PR | Change |
|---|---|---|---|---|
matmul/large_generic_bf16_threads=8/32x1024x1024 |
2.29 ms | 2.91 ms | +27.0% | |
qwen3_sampling_processors/top_p_fast_after_top_k |
521.42 µs | 659.68 µs | +26.5% | |
qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline |
3.49 ms | 4.32 ms | +23.7% | |
block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024 |
1.37 ms | 1.67 ms | +22.0% | |
sampling_latency/greedy_per_token |
3.20 µs | 3.90 µs | +21.9% | |
| ✅ | matmul/large_generic_f16_threads=1/32x1024x1024 |
95.63 µs | 104.41 µs | +9.2% |
| ✅ | tokenization/decode_tokens_per_second |
5.96 ms | 6.37 ms | +6.9% |
| ✅ | qwen3_sampling_processors/top_k_partial_selection |
144.87 µs | 154.86 µs | +6.9% |
| ✅ | logit_processing/seven_processor_chain_per_step |
312.46 µs | 329.98 µs | +5.6% |
| ✅ | sampling_latency/top_k_per_token |
53.17 µs | 55.99 µs | +5.3% |
| ✅ | qwen3_sampling_processors/top_k_top_p_full_sort_baseline |
5.61 ms | 5.88 ms | +4.9% |
| ✅ | block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024 |
96.19 µs | 100.55 µs | +4.5% |
| ✅ | kv_cache/alloc_dealloc_pages |
38.93 µs | 40.69 µs | +4.5% |
| ✅ | matmul/large_generic_f32_threads=1/32x1024x1024 |
9.89 ms | 10.27 ms | +3.8% |
| ✅ | grammar_masking/llguidance_compute_mask/32 |
73.13 µs | 75.81 µs | +3.7% |
| ✅ | qwen3_sampling_processors/top_k_full_sort_baseline |
2.10 ms | 2.13 ms | +1.3% |
| ✅ | tokenization/encode_tokens_per_second |
428.29 µs | 425.73 µs | -0.6% |
| ✅ | sampling_latency/min_p_per_token |
218.25 µs | 212.56 µs | -2.6% |
| ✅ | qwen3_sampling_processors/top_k_top_p_fast |
685.39 µs | 664.07 µs | -3.1% |
| ✅ | gather/small_f32_threads=1-internal/4096 |
705.4 ns | 676.5 ns | -4.1% |
| ✅ | reduce_mean/small_f32_threads=1-internal/4096 |
18.01 µs | 16.59 µs | -7.9% |
| ✅ | sampling_latency/top_p_per_token |
429.02 µs | 394.17 µs | -8.1% |
| ✅ | reduce_mean/large_f32_threads=1-internal/262144 |
1.04 ms | 948.01 µs | -8.9% |
| ✅ | matmul/medium_generic_f32_threads=1/32x512x512 |
2.43 ms | 2.17 ms | -10.4% |
| ✅ | reduce_mean/medium_f32_threads=1-internal/65536 |
299.69 µs | 266.67 µs | -11.0% |
| 🟢 | add/small_f16_threads=1-internal/1024 |
502.5 ns | 417.7 ns | -16.9% |
| 🟢 | add/small_bf16_threads=1-internal/1024 |
505.5 ns | 418.8 ns | -17.2% |
| 🟢 | add/small_f32_threads=1-internal/1024 |
232.1 ns | 191.5 ns | -17.5% |
| 🟢 | add/large_bf16_threads=1-internal/4194304 |
1.92 ms | 1.57 ms | -18.6% |
| 🟢 | matmul/medium_generic_f16_threads=1/32x512x512 |
36.98 µs | 29.94 µs | -19.0% |
| 🟢 | matmul/small_generic_f32_threads=1/1x256x256 |
44.21 µs | 35.57 µs | -19.5% |
| 🟢 | matmul/small_generic_bf16_threads=1/1x256x256 |
36.20 µs | 28.21 µs | -22.1% |
| 🟢 | matmul/small_generic_f16_threads=1/1x256x256 |
36.26 µs | 28.23 µs | -22.1% |
| 🟢 | matmul/medium_generic_bf16_threads=1/32x512x512 |
658.17 µs | 507.59 µs | -22.9% |
| 🟢 | matmul/large_generic_bf16_threads=1/32x1024x1024 |
2.81 ms | 2.14 ms | -23.7% |
| 🟢 | matmul/small_generic_bf16_threads=8/1x256x256 |
40.24 µs | 29.66 µs | -26.3% |
| 🟢 | add/medium_f32_threads=1-internal/262144 |
35.60 µs | 25.19 µs | -29.2% |
| 🟢 | matmul/large_generic_f16_threads=8/32x1024x1024 |
139.82 µs | 98.02 µs | -29.9% |
| 🟢 | add/large_f16_threads=1-internal/4194304 |
2.26 ms | 1.59 ms | -29.9% |
| 🟢 | block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1 |
590.19 µs | 407.79 µs | -30.9% |
| 🟢 | matmul/medium_generic_f32_threads=8/32x512x512 |
1.67 ms | 1.15 ms | -31.2% |
| 🟢 | matmul/small_generic_f16_threads=8/1x256x256 |
41.45 µs | 28.38 µs | -31.5% |
| 🟢 | gather/medium_bf16_threads=1-internal/32768 |
3.71 µs | 2.44 µs | -34.1% |
| 🟢 | block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1 |
99.17 µs | 65.34 µs | -34.1% |
| 🟢 | gather/small_f16_threads=1-internal/4096 |
689.1 ns | 451.0 ns | -34.5% |
| 🟢 | gather/medium_f16_threads=1-internal/32768 |
3.50 µs | 2.25 µs | -35.6% |
| 🟢 | matmul/medium_generic_f16_threads=8/32x512x512 |
43.31 µs | 27.79 µs | -35.8% |
| 🟢 | matmul/large_generic_f32_threads=8/32x1024x1024 |
6.31 ms | 3.93 ms | -37.7% |
| 🟢 | add/medium_f16_threads=1-internal/262144 |
156.78 µs | 97.51 µs | -37.8% |
| 🟢 | gather/large_f32_threads=1-internal/131072 |
42.70 µs | 26.15 µs | -38.8% |
| 🟢 | block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024 |
126.54 µs | 75.26 µs | -40.5% |
| 🟢 | matmul/small_generic_f32_threads=8/1x256x256 |
54.06 µs | 31.64 µs | -41.5% |
| 🟢 | gather/small_bf16_threads=1-internal/4096 |
756.7 ns | 442.8 ns | -41.5% |
| 🟢 | gather/medium_f32_threads=1-internal/32768 |
6.61 µs | 3.80 µs | -42.5% |
| 🟢 | add/medium_bf16_threads=1-internal/262144 |
174.05 µs | 96.22 µs | -44.7% |
| 🟢 | add/large_f32_threads=1-internal/4194304 |
1.03 ms | 565.87 µs | -45.1% |
| 🟢 | matmul/medium_generic_bf16_threads=8/32x512x512 |
667.00 µs | 352.96 µs | -47.1% |
| 🟢 | gather/large_bf16_threads=1-internal/131072 |
19.48 µs | 9.49 µs | -51.3% |
| 🟢 | gather/large_f16_threads=1-internal/131072 |
22.96 µs | 10.09 µs | -56.1% |
Visual flags:
Host info
CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 3.79 3.78 6.84 }
What this cannot catch
- Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
- Sub-threshold regressions that compound over multiple PRs
- Performance changes that only manifest under GPU execution
- Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Offline decision aid for the over-subscribed MoE regime (expert bank > VRAM), the problem the owner actually cares about. Replays a real trained-router expert-selection trace against a bounded, globally-shared expert cache and measures how much any residency policy could win and how close the cheapest policy gets to the oracle — before committing to hardware we do not have. No runtime policy is built; this is the measurement the owner asked to see first.
Trace / method (house rule §32.2)
granite-3.0-1b-a400m-instruct— 32 experts, top-8, 24 layers, real IBM trained router, f16 Mobius. 3 prompts (prose/code/math) × 64 greedy tokens = 192 decode steps.(layer, expert)(768 objects); budget is GLOBAL across all 24 layers (VRAM is one shared pool). Per-layer equal-split kept only as a baseline-to-beat. Deterministic; reproduced 2026-08-18.Key measured findings (25–50% budget regime)
Honesty / limits
Files
scripts/dump_moe_expert_trace.py,scripts/moe_expert_trace.json— trace capture + datascripts/moe_expert_cache_sim.py,scripts/moe_cache_sim_results.json— simulator + resultsdocs/benchmarks/2026-08-18-moe-expert-cache-policy-headroom.md— writeup, cites the merged router-skew wiki page and churn benchmarkMeasured vs inferred separated throughout. Reproduction:
python scripts/dump_moe_expert_trace.py && python scripts/moe_expert_cache_sim.py.