llama: model_loader: add TENSOR_READ_LAZY by ngxson · Pull Request #27794 · ggml-org/llama.cpp

ngxson · 2026-08-27T11:08:41Z

Overview

Models having PLE and engrams embeddings don't actually need to load the whole embedding table onto RAM. It can be lazily read via mmap

The behavior can be controlled via --tensor-read-lazy on|off|auto, with auto means lazy if tensor size if > 4 GiB. This is to make sure we don't degrade performance of small models, see below

For small models like gemma 4, doing this will have a significant impact on performance as the read delay is significant compared to token generation. However, bigger models like qwen4, the effect will be minor

Tested using unsloth/gemma-4-E4B-it-GGUF:Q4_K_M

Lazy tensor: per_layer_token_embd.weight, Q5_K, [10752, 262144], 1.94 GB = 39% of the 4.96 GB file

config	gguf region resident	peak RSS	pp (t/s)	tg (t/s)
baseline (eager prefetch)	4.6 G / 4.6 G	7.37 GB	571-583	105.8
lazy: skip prefetch + MADV_RANDOM	2.8 G / 4.6 G	6.16 GB	514-547	94.2-94.7 (-10.7%)
lazy: skip prefetch only	2.8 G / 4.6 G	-	561-563	97.0-97.3 (-8%)

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: yes

Green-Sky · 2026-08-27T11:12:30Z

And the fact that ngrams are given qwen4 on the second layer, not the first, can be used to hide the lazy nature.

ggerganov

I guess you've tested on a Mac? I can test on DGX Spark. What commands do you use and what numbers do you monitor?

ggerganov · 2026-08-27T11:34:58Z

+    llama_mmap(struct llama_file * file, size_t prefetch = (size_t) -1, bool numa = false,
+               const std::vector<std::pair<size_t, size_t>> & lazy_ranges = {});


Typedef this std::vector<std::pair<>> into something like llama_mmap::ranges

ggerganov · 2026-08-27T11:37:51Z

+    if ((flags & TENSOR_READ_LAZY) && use_mmap && tensor_read_lazy != LLAMA_TENSOR_READ_LAZY_OFF) {
+        // in auto mode, small tensors are cheap enough to keep resident
+        constexpr size_t auto_lazy_min_size = 4ull * 1024 * 1024 * 1024;
+        if (tensor_read_lazy == LLAMA_TENSOR_READ_LAZY_ON || ggml_nbytes(cur) > auto_lazy_min_size) {
+            const auto & w = require_weight(tn.str().c_str());
+            lazy_tensor_ranges[w.idx].emplace_back(w.offs, w.offs + ggml_nbytes(cur));
+        }
+    }


Should add a log here to help debugging easier.

ngxson · 2026-08-27T11:59:24Z

I guess you've tested on a Mac? I can test on DGX Spark. What commands do you use and what numbers do you monitor?

yes I'm testing on my max M5

for DGX (linux), you can try using the model as usual, e.g. llama-cli -hf unsloth/gemma-4-E4B-it-GGUF:Q4_K_M --tensor-read-lazy (on|off) then grab the PID of the process

pmap -x $PID | grep gguf will give the RSS reading. for lazy-read off, expect >4GB, the whole model is mmap'ed; and with on it should fall to 2.8GB as the PLE tensor is excluded
While doing prefill, this number should grow, but should never be 4GB

ngxson · 2026-08-27T12:01:08Z

one thing not 100% obvious to me though, the GPU offloading logic is a bit above my head. this PR assumes that the PLE tensor is not offloaded to GPU by default (as part of LLM_TENSOR_LAYER_INPUT), which seems to be the case, but I'm not sure if there are any edge cases

pwilkin · 2026-08-27T12:15:33Z

@ngxson not sure if there shouldn't be a limit based on relative tensor size as well. Some tensors for the really big models might also hit 4GB.

ngxson · 2026-08-27T12:38:01Z

@pwilkin only tensors explicitly marked as TENSOR_READ_LAZY will go through this path

obviously expert tensors can be as big, but they should not be marked as TENSOR_READ_LAZY

ggerganov · 2026-08-27T13:03:49Z

Tested on the DGX Spark and RTX 5090 and confirm the RSS readings.

one thing not 100% obvious to me though, the GPU offloading logic is a bit above my head. this PR assumes that the PLE tensor is not offloaded to GPU by default (as part of LLM_TENSOR_LAYER_INPUT), which seems to be the case, but I'm not sure if there are any edge cases

Yes, I can't say it's completely clear to me either. Let's see how it goes.

* llama: model_loader: add TENSOR_GET_ROW_LAZY * add --tensor-read-lazy * rename to TENSOR_READ_LAZY * gen docs * address comments

Gemma-3N, Gemma-4 E-series and Qwen4-Exp carry a per-layer (n-gram) embedding table alongside their weights. It is a large fraction of the file — 40% of gemma-4-E4B-it-IQ4_NL, and 97.7 GiB unquantized on Qwen3.8-Flash-Next — and llama.cpp can read its rows from the file on demand rather than holding it resident (--tensor-read-lazy, added upstream in ggml-org/llama.cpp#27794). Reaching that through the extra-flags box meant knowing the flag exists and spelling the tensor right, and the documentation in circulation names a tensor (ple_ngram_embd) that does not match the implementation (per_layer_token_embd). Give it a selector instead: Auto, Stream from model file, Keep resident. The selector only appears for models that have such a table, detected by reading the GGUF tensor-info block rather than by matching architecture names — llama.cpp assigns lazy-read eligibility per tensor and the set of architectures using it grows, so a name match picks the next one up for free. Tensor sizes come from the gaps between data offsets rather than from dimensions and quantization type, which avoids carrying a table of ggml block sizes that would need editing for every new quant. The VRAM estimate excludes a table that is streamed. It used whole file size as the weight proxy, so a model with a 28 GB table read the same as one that needed 28 GB more VRAM than it does. The conditions mirror llama.cpp's own: auto streams only above 4 GiB, and streaming needs mmap so direct I/O counts the table again. Verified against a real gemma-4-E4B-it-IQ4_NL: the scanner returns 1937768448 bytes for its table, byte-identical to computing it independently, and 40% of the file matches the ~39% ngxson measured upstream. That table is under the 4 GiB threshold, so auto leaves it resident and no existing estimate moves. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Go9An1JPRDAuj8nVYwpXLZ

Gemma-3N, Gemma-4 E-series and Qwen4-Exp carry a per-layer (n-gram) embedding table alongside their weights. It is a large fraction of the file — 40% of gemma-4-E4B-it-IQ4_NL, and 97.7 GiB unquantized on Qwen3.8-Flash-Next — and llama.cpp can read its rows from the file on demand rather than holding it resident (--tensor-read-lazy, added upstream in ggml-org/llama.cpp#27794). Reaching that through the extra-flags box meant knowing the flag exists and spelling the tensor right, and the documentation in circulation names a tensor (ple_ngram_embd) that does not match the implementation (per_layer_token_embd). Give it a selector instead: Auto, Stream from model file, Keep resident. The selector only appears for models that have such a table, detected by reading the GGUF tensor-info block rather than by matching architecture names — llama.cpp assigns lazy-read eligibility per tensor and the set of architectures using it grows, so a name match picks the next one up for free. Tensor sizes come from the gaps between data offsets rather than from dimensions and quantization type, which avoids carrying a table of ggml block sizes that would need editing for every new quant. The VRAM estimate excludes a table that is streamed. It used whole file size as the weight proxy, so a model with a 28 GB table read the same as one that needed 28 GB more VRAM than it does. The conditions mirror llama.cpp's own: auto streams only above 4 GiB, and streaming needs mmap so direct I/O counts the table again. Verified against a real gemma-4-E4B-it-IQ4_NL: the scanner returns 1937768448 bytes for its table, byte-identical to computing it independently, and 40% of the file matches the ~39% ngxson measured upstream. That table is under the 4 GiB threshold, so auto leaves it resident and no existing estimate moves. Claude-Session: https://claude.ai/code/session_01Go9An1JPRDAuj8nVYwpXLZ Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

…742) We merged PR ggml-org#27742 at its llama_memory_hybrid_idx commit. Upstream merged it as 6c84c7d after ~30 more review commits. This brings those in, plus the two upstream-master PRs they depend on (ggml-org#27762 KV token tracking, ggml-org#27794 TENSOR_READ_LAZY) and ggml-org#27795, which replaced our own quantizer buffer fix. Cost and correctness, in rough order of what it is worth here: - QSA bias is now per block, not per cell. It was f32[n_kv, n_tokens], which is about 1 MB per token at -c 262144. The per-cell half is the attention mask, which the graph already has, so only the per-block half is uploaded. - Layers sharing a compress ratio now share one QSA input set. 48 layers each allocated their own cell_blk / blk_cells / blk_pos / bias. - can_reuse() on the QSA and PLE graph inputs, so decode reuses the graph. - The full memory context now carries its indexer cache. graph_reserve() walks a full context and qwen4exp builds sparse attention only when one is present, so the reserved worst case was the smaller dense graph and ggml-alloc grew the compute buffer past the size reported at load. - Output rows are trimmed at the last layer instead of after the final mixer. Our MTP sidecar reads the hyper-connection streams by raw token position, so that trim is skipped when unmasked nextn embeddings are requested and applied after the mixer instead, the same way deepseek4.cpp does it. - QSA supports a rotated/quantized KV cache: q/k/v are hadamard-rotated in build_attn_qsa rather than asserting self_k_rot is null. Inert while LLAMA_ATTN_ROT_DISABLE defaults on for TurboQuant, but the assert is gone. - The PLE n-gram history moves out of a map on llama_model and is recovered from the KV cells (llama_kv_cell_ext::tok, get_prev_tokens). A llama_model is shared by every context and the map was keyed only by seq_id, so two contexts on one model overwrote each other's window; it was also in no state blob, so a restored context hashed against EOS padding. - The PLE conv history gets its own mirrored recurrent row (cache_ple_r_l), and n_embd_r() returns to n_conv. The row exists only on PLE layers, so the recurrent footprint drops rather than grows. - The indexer cache gets its own tensor names (cache_idx_k_l%d); it was matching the attention split pattern. - Indexer state save/restore, whole-context restore clears once, mirrored slot layout is checked rather than assumed. - PLE hparams shrink: is_ple_impl becomes a bitset (2048 bytes to 64), head offsets and vocab sizes narrow to uint32. llama_hparams is held by value in llm_graph_params, which is a stack local on every worker thread. - Converter streams the 128-shard PLE table through LazyChunkedTensor; the quantizer dequantizes in row bands. Drops our -sm tensor deny for qwen4exp (1cecef9). That existed because the PR shipped no split rules for the hyper-connection, PLE and indexer tensors. Upstream wrote them, so the deny is obsolete and was blocking a capability we can use. test-llama-archs -a qwen4exp: R9700 OK 9.58e-14 roundtrip OK RX 6900 XT OK 9.95e-14 roundtrip OK CPU OK 0.00e+00 roundtrip OK Meta OK 9.81e-14 (was SKIP under the deny) No regressions across the families the kv-cache and recurrent changes touch: qwen3next qwen35 qwen35moe minimax-01 falcon-h1 mamba mamba2 nemotron_h granitehybrid lfm2 plamo2 deepseek32 glm-dsa dots3note all pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NjfM4FxGPEhoApv6yvWAzD

* llama: model_loader: add TENSOR_GET_ROW_LAZY * add --tensor-read-lazy * rename to TENSOR_READ_LAZY * gen docs * address comments

El tensor per_layer_token_embd (26.8 GiB en UD-IQ3_XXS) quedaba residente en RAM aunque el loader soportara lectura diferida: el PR ggml-org#27794 solo marcaba gemma4. Con el marcador, -lm mmap --tensor-read-lazy auto mantiene la tabla en SSD (~1 GiB residente) y libera ~31 GiB en Strix Halo (128 GB), habilitando el IQ4_XS con contexto completo. Validado: contenido temp-0 idéntico, tg 27-30 t/s (-5-10% vs modo RAM), aceptación MTP sin cambio.

* llama: model_loader: add TENSOR_GET_ROW_LAZY * add --tensor-read-lazy * rename to TENSOR_READ_LAZY * gen docs * address comments (cherry picked from commit e56b889)

* llama: model_loader: add TENSOR_GET_ROW_LAZY * add --tensor-read-lazy * rename to TENSOR_READ_LAZY * gen docs * address comments

… gather prefetch Ports upstream ggml-org#27794 (fac889f) and reconciles it with the env-gated implementation this branch carried. Upstream's structure is taken wholesale: the loader collects the byte ranges of tensors an arch marks TENSOR_READ_LAZY, hands them to llama_mmap, and the constructor skips MAP_POPULATE, issues WILLNEED over the complement and MADV_RANDOM over the ranges. That replaces advise_random_range()/prefetch_except() and the post-load advise pass, all removed here. A per-tensor arch flag with a CLI beats an environment variable, and matching upstream's names means anything a user reads elsewhere transfers. What upstream dropped and this keeps is prefetch_rows(), the batched readahead issued ahead of a sparse gather. Their own commit measured MADV_RANDOM WITHOUT it at 94.4 s against 36.7 s for an untouched mapping, and the pair together at 34.1 s: suppressing the kernel's readahead only pays if you replace it. Because of that, an arch that marks a tensor without also wiring the prefetch would get precisely the losing half. gemma4 is marked upstream and has no prefetch path, so a >4 GiB Gemma table would have regressed under the new default. Ranges whose tensor is not returned by gather_tables() are therefore dropped before init_mappings() acts on them -- the advice is never applied without the prefetch that pays for it. Also marks qwen4exp's PLE/engram table. Upstream marked only gemma4, so on the model this was written for the flag did nothing. The LLAMA_MMAP_RANDOM env var and its `drop` mode are gone. `drop` measured inside the spread of plain `on` (353.7 vs 352.5 pp512); memory pressure evicts those pages anyway. Measured on gfx1151, from a clean build. Qwen3.8-Flash-Next-Q3KEXP-PLEf16 (~95 GiB f16 PLE table against 62 GB RAM), -ngl 99 -ncmoe 0 -ub 256 -p 512 -n 128 -r 3, cold page cache per cell, counterbalanced: --tensor-read-lazy off default (auto) pp512 99.01 / 95.79 352.38 / 352.50 3.62x tg128 26.09 / 26.04 33.41 / 33.41 1.28x nvme read 217.6 GiB 113.1 GiB 1.92x less gemma-4-E2B-it-Q8_0, whose 2.32 GiB table the guard declines, is unchanged either way: pp512 3901.77 off vs 3928.75 on, tg128 75.33 vs 75.24. The win decays with depth (2.53x at d32768 on the pre-port build) and depends on the table not fitting in page cache; where it fits, expect far less. Output is unaffected: gated at temperature 0 over 5 prompts x 192 tokens, byte-identical, with a passing determinism control. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

ggml-org#27794 wires --tensor-read-lazy through common/arg.cpp, which serves llama-cli and llama-server. llama-bench has its own parser and never learned it: error: invalid parameter for argument: --tensor-read-lazy That matters more than usual for this flag, because its effect depends on whether the marked tensor fits in page cache, so anyone reproducing or refuting the result has to A/B it, and llama-bench is where that is done. Mirrors the existing -lm/--load-mode wiring, as a scalar rather than a swept axis: a behaviour switch belongs in separate counterbalanced launches, not inside one process where -r would understate the error. The scalar needs a default member initialiser, NOT just an entry in cmd_params_defaults. parse_cmd_params() starts from a fresh cmd_params and restores fields from the defaults via .empty() checks, which only reach the vector members -- every other param here is a vector, so the pattern is invisible until a scalar is added. Without the initialiser the field zero-initialises to LLAMA_TENSOR_READ_LAZY_OFF and llama-bench silently overrides the library's AUTO default, disabling the feature for every run that does not pass the flag explicitly. Found by measurement, not by review: parsing worked, the explicit values worked, and only the untyped default was wrong. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* llama: model_loader: add TENSOR_GET_ROW_LAZY * add --tensor-read-lazy * rename to TENSOR_READ_LAZY * gen docs * address comments (cherry picked from commit fac889f)

ngxson added 2 commits August 27, 2026 12:54

llama: model_loader: add TENSOR_GET_ROW_LAZY

e072597

add --tensor-read-lazy

bb07e28

github-actions Bot added the model Model specific label Aug 27, 2026

rename to TENSOR_READ_LAZY

f3bb761

ngxson changed the title ~~llama: model_loader: add TENSOR_GET_ROW_LAZY~~ llama: model_loader: add TENSOR_READ_LAZY Aug 27, 2026

ngxson marked this pull request as ready for review August 27, 2026 11:24

ngxson requested review from a team, CISC and ggerganov as code owners August 27, 2026 11:24

gen docs

420c032

ngxson requested a review from a team as a code owner August 27, 2026 11:29

github-actions Bot added documentation Improvements or additions to documentation server labels Aug 27, 2026

ngxson mentioned this pull request Aug 27, 2026

model: add Qwen3.8-Flash-Next (qwen4exp) #27742

Merged

ggerganov approved these changes Aug 27, 2026

View reviewed changes

address comments

937fe54

voidpush mentioned this pull request Aug 27, 2026

llama : stream MoE routed experts from disk #25294

Open

ngxson merged commit fac889f into master Aug 27, 2026
23 of 26 checks passed

giveen mentioned this pull request Aug 27, 2026

feat: import qwen4exp (Qwen3.8-Flash-Next) support from upstream PR #27742 TheTom/llama-cpp-turboquant#324

Merged

ppenatra pushed a commit to ppenatra/llama.cpp that referenced this pull request Aug 27, 2026

llama: model_loader: add TENSOR_READ_LAZY (ggml-org#27794)

674cf4f

* llama: model_loader: add TENSOR_GET_ROW_LAZY * add --tensor-read-lazy * rename to TENSOR_READ_LAZY * gen docs * address comments

tmac1973 mentioned this pull request Aug 27, 2026

feat(models): first-class control for per-layer embedding tables tmac1973/llama-toolchest#149

Merged

KAFKA2306 mentioned this pull request Aug 27, 2026

ローカルLLM用PCを実測し、最小費用の構成を決める KAFKA2306/agent-resources#237

Open

16 tasks

ngxson mentioned this pull request Aug 27, 2026

llama: improve TENSOR_READ_LAZY handling #27837

Merged

danielhanchen mentioned this pull request Aug 28, 2026

llama: batched readahead for lazily read gather tables unslothai/llama.cpp#137

Open

x1250 pushed a commit to x1250/llama.cpp that referenced this pull request Aug 28, 2026

llama: model_loader: add TENSOR_READ_LAZY (ggml-org#27794)

52b982b

* llama: model_loader: add TENSOR_GET_ROW_LAZY * add --tensor-read-lazy * rename to TENSOR_READ_LAZY * gen docs * address comments

Green-Sky mentioned this pull request Aug 29, 2026

llama-model: Disable mmap prefetching #27928

Closed

Nekotekina mentioned this pull request Aug 31, 2026

llama-mmap: don't use MAP_POPULATE on Linux #28055

Open

ServeurpersoCom mentioned this pull request Sep 4, 2026

Misc. bug: Qwen3.8-Flash-Next (qwen4_exp): 51B N-gram embedding table not loaded after build 10665, causing extremely slow prefill #28355

Closed

ggerganov mentioned this pull request Sep 4, 2026

llama.cpp : bump version to 0.4.0 #28386

Merged

Navigation Menu

Uh oh!

llama: model_loader: add TENSOR_READ_LAZY - #27794

llama: model_loader: add TENSOR_READ_LAZY#27794
ngxson merged 5 commits into
masterfrom
xsn/tensor_rows_lazy

ngxson commented Aug 27, 2026 •

edited

Loading

Uh oh!

Green-Sky commented Aug 27, 2026

Uh oh!

ggerganov left a comment

Uh oh!

ggerganov Aug 27, 2026

Uh oh!

ggerganov Aug 27, 2026

Uh oh!

ngxson commented Aug 27, 2026 •

edited

Loading

Uh oh!

ngxson commented Aug 27, 2026 •

edited

Loading

Uh oh!

pwilkin commented Aug 27, 2026

Uh oh!

ngxson commented Aug 27, 2026

Uh oh!

ggerganov commented Aug 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		llama_mmap(struct llama_file * file, size_t prefetch = (size_t) -1, bool numa = false,
		const std::vector<std::pair<size_t, size_t>> & lazy_ranges = {});

Uh oh!

Conversation

ngxson commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Requirements

Uh oh!

Green-Sky commented Aug 27, 2026

Uh oh!

ggerganov left a comment

Choose a reason for hiding this comment

Uh oh!

ggerganov Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

ggerganov Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

ngxson commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ngxson commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pwilkin commented Aug 27, 2026

Uh oh!

ngxson commented Aug 27, 2026

Uh oh!

ggerganov commented Aug 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ngxson commented Aug 27, 2026 •

edited

Loading

ngxson commented Aug 27, 2026 •

edited

Loading

ngxson commented Aug 27, 2026 •

edited

Loading