llama: improve TENSOR_READ_LAZY handling by ngxson · Pull Request #27837 · ggml-org/llama.cpp

ngxson · 2026-08-27T23:35:14Z

Overview

Make sure --tensor-read-lazy take full precedence over --load-mode or -ot: If tensor is decided to be "lazy-read", it will be mmap'ed no matter what. This is because some use cases use --load-mode none for faster weight offloading to GPU, but that doesn't mean user want to offload the PLE tensor
Context fit pass now correctly exclude the PLE tensor
If system doesn't support mmap, print a WARN while loading the whole tensor onto RAM --> not sure if it worth properly fixing this case, I feel like will be too much work to do
Logic is now contained inside substruct llama_model_loader::lazy_read

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: I own the idea, code is AI written

tarruda · 2026-08-28T12:26:50Z

@ngxson can you also implement the flag on llama-bench? Would be useful to compare how it affects tg/pp (see also: #27864)

ngxson · 2026-08-28T13:32:11Z

hmm ok I thought llama-bench uses the same common/arg, turns out we need to add it manually here

…y_improve

Kononnable · 2026-08-29T08:30:59Z

As --tensor-read-lazy is enabled by default(auto) it might be a good idea to add a warning that -ot on those tensors is ignored (explicitly set -ot will be overridden by implicit --tensor-read-lazy).

Nekotekina · 2026-08-29T13:36:09Z

Maybe add LLAMA_LAZY_MODE_ALL or something to disable prefetching completely.
#27928
I'm at bad timings again...
It can be useful to load any model like this.

jintakhan · 2026-08-30T14:33:11Z

Previewing this PR, definitely seeing a --load-mode none prefill speed boost of 15% over --load-mode mmap. Not quite the 2x speed of loading the whole table to RAM, but a boost nonetheless, and the PLE tables stay on disk as intended, so that's a win in my book. Disabling --tensor-read-lazy (which will be renamed --lazy-mode -lzm per #27969) along with --load-mode none shows the expected behavior of loading PLE table to RAM, the old no-mmap behavior incl. the full 2x speed gain and RAM use.

ServeurpersoCom

LGTM. The precedence rule is the right call. Tested on Qwen3.8-Flash-Next: with and without -ot per_layer_token_embd=CPU gives the same numbers, so it lands in auto and the -ot is correctly ignored.

ngxson · 2026-08-30T14:59:40Z

@Kononnable there should already been a LLAMA_LOG_INFO if the tensor is determined to have lazy read = on (get logged no matter auto or on), that should be enough

jintakhan · 2026-08-30T18:11:35Z

One more update on performance gains. The mmap bottleneck isn't architectural overhead from memory mapping, it's probably just latency sensitive. I run llama.cpp in a Docker VM and moving the model files from the vdisk to direct storage on PCIe passthrough restores the no-mmap PLE lazy load prefill performance to speeds with the full model+PLE in resident memory.
This patch will let me use these big models while freeing up enough memory to run a second faster MoE model on another GPU with expert offload — and still have enough RAM left over to keep the workstation usable as a general purpose computer. Awesome stuff.

This reverts upstream 2578138. It costs 73% of prefill on this box. The commit routes every lazily-read tensor to the generic CPU buffer type (lazy_read::buft() returns ggml_backend_dev_buffer_type of the CPU device). Before it, our 28.8 GB per_layer_token_embd lived in CPU_Mapped -- mmap'd host memory that GB10's GPU reads directly, since the part reports pageableMemoryAccessUsesHostPageTables=1. Forced into a plain CPU buffer, the per-layer PLE gathers get scheduled on the CPU backend instead, and the GPU sits idle waiting on them: an nsys capture of the regressed build shows 31.5% GPU-busy and 22-38 W board power during prefill, with the SM clock pinned at 2190 MHz and no throttle reason active. The work is not slower, it is on the wrong device. Found by bisecting the 21 commits between c589f0e and a7cc83b on pure upstream, with none of our code in the tree: 0b5be7e hip: tune rdna 3 mmq config 741.98 t/s f1793c1 CUDA: fast mm_ids_helper for any n_eu 812.70 t/s +9.5% 2578138 llama: improve TENSOR_READ_LAZY 222.70 t/s -72.6% a7cc83b (tip) 211.67 t/s a7cc83b + this revert 813.86 t/s llama-bench pp4096, ub4096, UD-IQ4_XS. The revert restores the number exactly, which is the evidence that this commit and nothing else in that range is responsible. Worth being precise about what this is not: the mm_ids_helper commit immediately before it is a real +9.5% win here (n_expert_used = 10 was excluded from the fast path by the old warp_size % n_expert_used == 0 gate), and it survives the revert. The two are independent. This is a local deviation, not a claim the upstream change is wrong in general -- it plausibly fixes lazy reads on discrete-GPU systems where a device-buffer lazy tensor cannot work. It is wrong specifically for a UMA part that can read mmap'd host pages from the GPU, which is the only configuration we serve.

… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().

This reverts upstream 2578138. It costs 73% of prefill on this box. The commit routes every lazily-read tensor to the generic CPU buffer type (lazy_read::buft() returns ggml_backend_dev_buffer_type of the CPU device). Before it, our 28.8 GB per_layer_token_embd lived in CPU_Mapped -- mmap'd host memory that GB10's GPU reads directly, since the part reports pageableMemoryAccessUsesHostPageTables=1. Forced into a plain CPU buffer, the per-layer PLE gathers get scheduled on the CPU backend instead, and the GPU sits idle waiting on them: an nsys capture of the regressed build shows 31.5% GPU-busy and 22-38 W board power during prefill, with the SM clock pinned at 2190 MHz and no throttle reason active. The work is not slower, it is on the wrong device. Found by bisecting the 21 commits between c589f0e and a7cc83b on pure upstream, with none of our code in the tree: 0b5be7e hip: tune rdna 3 mmq config 741.98 t/s f1793c1 CUDA: fast mm_ids_helper for any n_eu 812.70 t/s +9.5% 2578138 llama: improve TENSOR_READ_LAZY 222.70 t/s -72.6% a7cc83b (tip) 211.67 t/s a7cc83b + this revert 813.86 t/s llama-bench pp4096, ub4096, UD-IQ4_XS. The revert restores the number exactly, which is the evidence that this commit and nothing else in that range is responsible. Worth being precise about what this is not: the mm_ids_helper commit immediately before it is a real +9.5% win here (n_expert_used = 10 was excluded from the fast path by the old warp_size % n_expert_used == 0 gate), and it survives the revert. The two are independent. This is a local deviation, not a claim the upstream change is wrong in general -- it plausibly fixes lazy reads on discrete-GPU systems where a device-buffer lazy tensor cannot work. It is wrong specifically for a UMA part that can read mmap'd host pages from the GPU, which is the only configuration we serve.

… mmap When a tensor is read lazily (per_layer_token_embd on the CPU) but sits between tensors that go to the device, the device buffer created from the host pointer spanned it. On Apple silicon Metal pins every page of a mapped buffer, so the whole 27 GiB table stayed wired even though nothing on the device reads it. Split the range around such tensors instead; a file may now map to more than one buffer, and load_all_data picks the one that holds each tensor. Rebased onto b10712: lazy tensors now live in their own context (ggml-org#27837), so the split is skipped for lazy contexts and the ranges come from ml.lazy.for_file().

Since upstream ggml-org#27837 a TENSOR_READ_LAZY tensor is forced onto a CPU mmap buffer and read on demand. The qwen4exp PLE table (27 GiB, MADV_RANDOM) is gathered by a single-threaded CPU get_rows, so every needed row was a major fault served one at a time from disk (~160 us each): 8192 rows per pp512 ubatch cost ~1.3 s and starved the GPU. That is the "bimodal" 410 vs 200 t/s prefill: fast before ggml-org#27837 (table resident), slow after. set_input() already computes the row indices on the host, so advise the kernel (MADV_WILLNEED) on the needed rows right there, merged into page spans. The reads are then issued at high queue depth and overlap the graph before the gather runs. A GPU-resident table (lazy off) skips the hint. Measured on Strix Halo (gfx1151, Vulkan, nl-oproj8, pp512/tg64, r=3): lazy auto before: 205 t/s / 28.6 t/s, ~36000 major faults per run lazy auto after: 410-416 t/s / 31.1-31.3 t/s, 0-14 major faults lazy off: 445-451 t/s / 31.4 t/s (unchanged, hint skipped) The table stays on disk: the load still reads 66 GiB, not 94 GiB. Assisted-by: Claude Claude-Session: https://claude.ai/code/session_01KXWaojVXitKAUbG1LUNGr2

* force lazy tensor on cpu if lazy is on * llama: improve TENSOR_READ_LAZY handling (cherry picked from commit 2578138)

KV restore batching (ggml-org#27991), kv-cells seq-scan early stop (ggml-org#28011), MOE fusion to specdec + multi-token (ggml-org#27621), mm_ids_helper templated fast path (ggml-org#27978), qwen4exp recurrent state rollback (ggml-org#28123), n_layer_nextn load order (ggml-org#28159), FA K/V XOR-swizzle smem tiles (ggml-org#25635), --lazy-mode -lzm (ggml-org#27837/ggml-org#27969). TQ3/TurboQuant stack and vitriol-* integration auto-merged clean; no conflicts. Experiment E1 of mining-experiment-master-plan-2026-09-01.

This reverts upstream 2578138. It costs 73% of prefill on this box. The commit routes every lazily-read tensor to the generic CPU buffer type (lazy_read::buft() returns ggml_backend_dev_buffer_type of the CPU device). Before it, our 28.8 GB per_layer_token_embd lived in CPU_Mapped -- mmap'd host memory that GB10's GPU reads directly, since the part reports pageableMemoryAccessUsesHostPageTables=1. Forced into a plain CPU buffer, the per-layer PLE gathers get scheduled on the CPU backend instead, and the GPU sits idle waiting on them: an nsys capture of the regressed build shows 31.5% GPU-busy and 22-38 W board power during prefill, with the SM clock pinned at 2190 MHz and no throttle reason active. The work is not slower, it is on the wrong device. Found by bisecting the 21 commits between c589f0e and a7cc83b on pure upstream, with none of our code in the tree: 0b5be7e hip: tune rdna 3 mmq config 741.98 t/s f1793c1 CUDA: fast mm_ids_helper for any n_eu 812.70 t/s +9.5% 2578138 llama: improve TENSOR_READ_LAZY 222.70 t/s -72.6% a7cc83b (tip) 211.67 t/s a7cc83b + this revert 813.86 t/s llama-bench pp4096, ub4096, UD-IQ4_XS. The revert restores the number exactly, which is the evidence that this commit and nothing else in that range is responsible. Worth being precise about what this is not: the mm_ids_helper commit immediately before it is a real +9.5% win here (n_expert_used = 10 was excluded from the fast path by the old warp_size % n_expert_used == 0 gate), and it survives the revert. The two are independent. This is a local deviation, not a claim the upstream change is wrong in general -- it plausibly fixes lazy reads on discrete-GPU systems where a device-buffer lazy tensor cannot work. It is wrong specifically for a UMA part that can read mmap'd host pages from the GPU, which is the only configuration we serve.

ngxson added 2 commits August 28, 2026 01:06

force lazy tensor on cpu if lazy is on

a7cfd88

llama: improve TENSOR_READ_LAZY handling

280e764

ngxson requested review from CISC and ggerganov as code owners August 27, 2026 23:35

ngxson requested a review from ServeurpersoCom August 27, 2026 23:35

ngxson changed the title ~~Xsn/tensor read lazy improve~~ llama: improve TENSOR_READ_LAZY handling Aug 27, 2026

ngxson mentioned this pull request Aug 28, 2026

bench: add --tensor-read-lazy #27881

Merged

danielhanchen mentioned this pull request Aug 28, 2026

llama: batched readahead for lazily read gather tables unslothai/llama.cpp#137

Open

Merge remote-tracking branch 'origin/master' into xsn/tensor_read_laz…

cda6f9e

…y_improve

ggerganov approved these changes Aug 29, 2026

View reviewed changes

CISC approved these changes Aug 29, 2026

View reviewed changes

sdougbrown added a commit to sdougbrown/r9700-llama-cpp that referenced this pull request Aug 30, 2026

llama: improve TENSOR_READ_LAZY handling (PR ggml-org#27837)

ce8a88c

antoinezambelli mentioned this pull request Aug 30, 2026

vulkan: large top-k support via argsort #28005

Closed

ServeurpersoCom approved these changes Aug 30, 2026

View reviewed changes

ngxson merged commit 2578138 into master Aug 30, 2026
23 of 27 checks passed

jintakhan mentioned this pull request Aug 30, 2026

Feature Request: Make mmap vs read decision per tensor instead of per file. #27766

Closed

4 tasks

ilariofebi mentioned this pull request Sep 1, 2026

Regression: --lazy-mode auto halves pp512 for qwen4exp on Vulkan (AMD iGPU) #28160

Open

Navigation Menu

Uh oh!

llama: improve TENSOR_READ_LAZY handling - #27837

llama: improve TENSOR_READ_LAZY handling#27837
ngxson merged 3 commits into
masterfrom
xsn/tensor_read_lazy_improve

ngxson commented Aug 27, 2026

Uh oh!

tarruda commented Aug 28, 2026

Uh oh!

ngxson commented Aug 28, 2026

Uh oh!

Kononnable commented Aug 29, 2026

Uh oh!

Nekotekina commented Aug 29, 2026

Uh oh!

jintakhan commented Aug 30, 2026

Uh oh!

ServeurpersoCom left a comment

Uh oh!

ngxson commented Aug 30, 2026

Uh oh!

Uh oh!

jintakhan commented Aug 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Uh oh!

Conversation

ngxson commented Aug 27, 2026

Overview

Requirements

Uh oh!

tarruda commented Aug 28, 2026

Uh oh!

ngxson commented Aug 28, 2026

Uh oh!

Kononnable commented Aug 29, 2026

Uh oh!

Nekotekina commented Aug 29, 2026

Uh oh!

jintakhan commented Aug 30, 2026

Uh oh!

ServeurpersoCom left a comment

Choose a reason for hiding this comment

Uh oh!

ngxson commented Aug 30, 2026

Uh oh!

Uh oh!

jintakhan commented Aug 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants