[justinchuby]

Two problems with taking #907 to the ORT team as written.

The motivation was written for us, not for them

It named no ORT mechanism, so an ORT reader could reasonably conclude this is a GenAI-side concern that does not touch their code. And it never answered the first objection anyone who knows ORT's memory model will raise: these are configuration knobs — set gpu_mem_limit, choose an arena extend strategy, size OrtArenaCfg, share allocators through OrtEnv.

Rewritten around that objection, because it is a fair one and the answer is the whole point of the design: ORT gives each pool a bound; it gives no holder a way to lend to another, and no way to reclaim from a holder that is over-provisioned right now. It now names the concrete mechanisms — BFC high-water behaviour and precisely where end-of-Run shrinkage does and does not help, custom device allocators not being BFC-wrapped (so the governed path and the arena path are two implementations today, not one policy), and RegisteredAllocator leaking because ORT does not expose the last session user.

Also puts three measurements up front. An architecture proposal with no evidence of harm reads as speculation, and we now have the harm:

• a load-time device_budget − kv_bytes_per_token × max_context split that stranded 1.611 GB from the first token; making it elastic cut streamed bytes 1.68× (Weight budget is reduced by the full-context KV reservation and never reclaims it: 1.611 GB idle while weights stream 3.94 GB/step (2.75x floor reduction available) #857/Make CUDA weight-residency budget elastic against KV occupancy (#857) #866)
• weight bytes measured from the external-data file rather than referenced extents — wrong by exactly 2.00×, so every derived budget was wrong (total_weight_bytes uses file size, not referenced bytes: qwen14b-zp's external data is 50% dead space, inflating W by 2.00x #853/fix(loader): report weights the graph references, not external-data file size (#853) #856)
• managing single-touch weights measured 30× slower than letting the OS demand-page them; deferring recovered ~100× (Managed offload is 30x SLOWER than WDDM demand paging on Windows — and it is the default: the copy-to-VRAM buys nothing for a single-touch access pattern #864/feat(memory): prefer WDDM shared-memory over managed weight streaming by default (#864) #874)

The first two are accounting failures that a single authority prevents by construction, which is the argument for the design. The third is why it also has to say when not to manage.

The diagram had a real omission

The appendix assigns transient-peak reporting to InferenceSession/EP, and I8 requires admission to derive exact bytes including transient peaks — but there was no edge from Session to ProcessMemoryManager at all. That is the principal ORT-side integration point, so leaving it out made the ORT-side work look smaller than it is and hid the one interface ORT actually has to add.

Added the edge, added the matching entry to Required architecture changes, and stated explicitly that the session is a reporter, not a holder, so the new edge is not misread as an ownership change. Also split control-plane (solid) from data-plane (dotted) edges, since only the control plane is new work and a reviewer should be able to see that without reading the prose.

One overclaim of my own, corrected

I first wrote that the BFC arena "does not return memory". That is wrong — ORT supports arena shrinkage at a Run boundary, and an ORT reviewer would have caught it immediately and discounted everything around it. The accurate criticism is narrower and still holds: shrinkage is caller-scheduled at a Run boundary, not demand-driven, so it cannot answer pressure that arrives mid-generation.

Docs only.

[justinchuby]

Rendered the diagram with @mermaid-js/mermaid-cli\ before merging rather than assuming the syntax was fine — this file is going to another team, and a broken diagram would discredit the rest of it. Clean 43 KB SVG, the new \ ransient peak + granularity report\ edge present, no syntax error (the only \�rror\ strings in the output are mermaid's own \�rror-icon/\�rror-text\ CSS classes, which every mermaid SVG carries).

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.74%. Comparing base (a5493fa) to head (c5b54ff).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #917      +/-   ##
==========================================
- Coverage   77.78%   77.74%   -0.05%     
==========================================
  Files         351      352       +1     
  Lines      140874   145872    +4998     
  Branches   140874   145872    +4998     
==========================================
+ Hits       109582   113403    +3821     
- Misses      26885    27988    +1103     
- Partials     4407     4481      +74     

Flag	Coverage Δ
cli-ort-linux	83.42% <ø> (ø)
cli-ort-windows	82.92% <ø> (?)
offline	77.52% <ø> (-0.05%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 33 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	47.69 µs	124.49 µs	+161.0%
🔴	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	39.11 µs	93.34 µs	+138.7%
🔴	matmul/medium_generic_f32_threads=8/32x512x512	897.48 µs	1.74 ms	+94.3%
🔴	matmul/large_generic_f32_threads=8/32x1024x1024	3.72 ms	6.65 ms	+78.8%
🔴	matmul/medium_generic_bf16_threads=8/32x512x512	376.73 µs	636.58 µs	+69.0%
🔴	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	705.72 µs	1.16 ms	+64.1%
🔴	matmul/small_generic_f32_threads=8/1x256x256	31.89 µs	50.17 µs	+57.3%
🔴	matmul/medium_generic_f16_threads=8/32x512x512	28.38 µs	44.14 µs	+55.5%
🔴	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	358.90 µs	529.42 µs	+47.5%
🔴	matmul/small_generic_bf16_threads=8/1x256x256	29.55 µs	38.89 µs	+31.6%
🔴	matmul/large_generic_f16_threads=8/32x1024x1024	77.57 µs	101.58 µs	+31.0%
⚠️	matmul/medium_generic_f16_threads=1/32x512x512	27.48 µs	35.42 µs	+28.9%
⚠️	add/large_f32_threads=1-internal/4194304	41.41 ms	52.40 ms	+26.6%
⚠️	matmul/large_generic_f16_threads=1/32x1024x1024	72.82 µs	91.10 µs	+25.1%
⚠️	matmul/small_generic_bf16_threads=1/1x256x256	28.36 µs	35.38 µs	+24.7%
⚠️	matmul/small_generic_f16_threads=8/1x256x256	27.86 µs	34.60 µs	+24.2%
⚠️	matmul/medium_generic_f32_threads=1/32x512x512	2.16 ms	2.60 ms	+20.5%
⚠️	matmul/large_generic_f32_threads=1/32x1024x1024	8.68 ms	10.13 ms	+16.6%
⚠️	matmul/medium_generic_bf16_threads=1/32x512x512	491.72 µs	568.42 µs	+15.6%
⚠️	matmul/large_generic_bf16_threads=1/32x1024x1024	1.85 ms	2.13 ms	+15.2%
✅	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	72.14 µs	79.86 µs	+10.7%
✅	matmul/small_generic_f16_threads=1/1x256x256	31.78 µs	34.39 µs	+8.2%
✅	matmul/small_generic_f32_threads=1/1x256x256	42.28 µs	45.15 µs	+6.8%
✅	tokenization/decode_tokens_per_second	6.19 ms	6.58 ms	+6.4%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	5.82 ms	6.06 ms	+4.1%
✅	matmul/large_generic_bf16_threads=8/32x1024x1024	2.14 ms	2.22 ms	+3.7%
✅	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.58 ms	3.71 ms	+3.5%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	534.54 µs	539.88 µs	+1.0%
✅	gather/large_f32_threads=1-internal/131072	34.04 µs	34.36 µs	+0.9%
✅	tokenization/encode_tokens_per_second	387.35 µs	388.92 µs	+0.4%
✅	qwen3_sampling_processors/top_k_top_p_fast	698.31 µs	693.44 µs	-0.7%
✅	logit_processing/seven_processor_chain_per_step	340.48 µs	335.65 µs	-1.4%
✅	gather/large_f16_threads=1-internal/131072	14.22 µs	13.92 µs	-2.1%
✅	gather/medium_f16_threads=1-internal/32768	2.56 µs	2.50 µs	-2.6%
✅	gather/small_bf16_threads=1-internal/4096	504.6 ns	490.4 ns	-2.8%
✅	gather/medium_f32_threads=1-internal/32768	4.39 µs	4.24 µs	-3.3%
✅	add/medium_bf16_threads=1-internal/262144	3.17 ms	3.06 ms	-3.5%
✅	add/large_f16_threads=1-internal/4194304	45.05 ms	43.26 ms	-4.0%
✅	reduce_mean/large_f32_threads=1-internal/262144	1.04 ms	993.19 µs	-4.3%
✅	sampling_latency/greedy_per_token	3.53 µs	3.36 µs	-4.8%
✅	grammar_masking/llguidance_compute_mask/32	84.64 µs	80.45 µs	-5.0%
✅	gather/large_bf16_threads=1-internal/131072	14.08 µs	13.23 µs	-6.1%
✅	kv_cache/alloc_dealloc_pages	43.20 µs	40.35 µs	-6.6%
✅	sampling_latency/top_k_per_token	60.63 µs	56.58 µs	-6.7%
✅	sampling_latency/min_p_per_token	228.11 µs	210.91 µs	-7.5%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	2.36 ms	2.16 ms	-8.2%
✅	add/small_f16_threads=1-internal/1024	14.03 µs	12.79 µs	-8.8%
✅	reduce_mean/medium_f32_threads=1-internal/65536	285.52 µs	248.37 µs	-13.0%
✅	add/small_bf16_threads=1-internal/1024	15.51 µs	13.32 µs	-14.1%
✅	gather/small_f32_threads=1-internal/4096	825.7 ns	707.4 ns	-14.3%
🟢	gather/medium_bf16_threads=1-internal/32768	3.03 µs	2.57 µs	-15.1%
🟢	add/medium_f16_threads=1-internal/262144	3.33 ms	2.79 ms	-16.1%
🟢	sampling_latency/top_p_per_token	473.54 µs	389.97 µs	-17.6%
🟢	add/medium_f32_threads=1-internal/262144	3.02 ms	2.48 ms	-17.8%
🟢	add/small_f32_threads=1-internal/1024	245.1 ns	199.4 ns	-18.7%
🟢	add/large_bf16_threads=1-internal/4194304	61.40 ms	46.14 ms	-24.9%
🟢	reduce_mean/small_f32_threads=1-internal/4096	21.36 µs	16.04 µs	-24.9%
🟢	gather/small_f16_threads=1-internal/4096	657.6 ns	488.2 ns	-25.8%
🟢	qwen3_sampling_processors/top_k_partial_selection	197.16 µs	145.90 µs	-26.0%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 4.45 3.55 6.27 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)