llama: add default load-mode auto, which avoids mmap on iGPUs by 0cc4m · Pull Request #26081 · ggml-org/llama.cpp

0cc4m · 2026-07-24T13:49:55Z

Overview

Follow-up to #20834 and #26023

mmap is detrimental on iGPUs that have to copy the model to device-visible, but still shared memory, as that means during loading it will have the model in RAM twice, doubling the memory requirements temporarily and slowing down the process if it overflows. This is currently the default behaviour, leading to issues for users that don't know about this.

Add load-mode auto and default to it. It keeps the current behaviour for most cases, except when a device reports that it doesn't support mmap. This is up to the backends to flag, through a new capability boolean (apologies for the mass-pinging this causes..). I only disabled it for CUDA/ROCm and Vulkan iGPUs for now, because I'm not familiar enough with the other backends.

Please let me know if this implementation is feasible or if there's a better way to handle it, and also if there are more backends I should include.

In the future further features could be added, e.g. automatic use of direct-io, if we know when it can definitely be enabled.

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES, Claude prototyped the code, I manually reviewed and tested.

pwilkin

This is sorely needed for ROCm, where the --mmap mode is additionally bugged beyond 64GB.

0cc4m · 2026-07-28T10:14:09Z

I addressed the feedback and added llama-bench support.

0cc4m · 2026-07-29T07:57:47Z

Sign in to view

+    if (ml.use_mmap && params.load_mode == LLAMA_LOAD_MODE_AUTO) {
+        for (const auto & dev : devices) {
+            ggml_backend_dev_props props;
+            ggml_backend_dev_get_props(dev.dev, &props);


Pretty sure the compiler figures this out no problem.

ORippler

AI tells me we may wish to bump APIR_PROTOCOL_MINOR as we change ggml_backend_dev_props to encode one more bool. Pinging @kpouget as I have no clue about virtgpu backend.
Please run llama-gen-docs to regenerate/update docs to reflect this change

Validated this change on DGX Spark. Would love to change default to -dio on iGPU systems in future to achieve faster load times

kpouget · 2026-07-29T12:26:27Z

AI tells me we may wish to bump APIR_PROTOCOL_MINOR as we change ggml_backend_dev_props to encode one more bool. Pinging @kpouget as I have no clue about virtgpu backend.

thanks for the ping,
yes, the bump makes sense
I'll start testing the PR with the virtgpu backend 👍🏻

…etal iGPU is used

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>

ORippler

Thanks, LGTM!

kpouget · 2026-08-05T17:46:00Z

LGTM for the virt-gpu part (@taronaeo)

elchic00 · 2026-08-08T15:03:18Z

Confirming this is worth automating on gfx1151 (Strix Halo APU, Framework Desktop Ryzen AI Max+ 395, ROCm/HIP). Without --direct-io, GPU-layer offload for a 35B-A3B MoE model at Q8 was capped at 45/99 layers — pushing further didn't degrade gracefully, it fell off a cliff to ~0.01 t/s (effectively hung). Looked like GTT page-fault thrashing from mmap's default region-pinning once the resident working set crossed a threshold, not an actual VRAM limit (it's an APU, so the GPU reaches system RAM through GTT). --direct-io (skips mmap entirely) gave a stable full 99-layer offload, not just a faster one. Happy to share exact config/benchmarks if useful for wiring this into auto-detection.

…, multi-output backend sampling, pocket-tts Merges 45 upstream commits (0865990..ebb546b, b10362-16-gebb546b7e) into synori/llama-update-mtp-fit. Zero conflicts; all vendored patches carried over untouched. Primary motivation — new Meta architecture: * 62bf73d model: Muse Glimmer Support (ggml-org#26841) LLM_ARCH_MUSE_GLIMMER + src/models/muse-glimmer.cpp + the mtmd vision tower in tools/mtmd/models/muse-glimmer.cpp and conversion/muse_glimmer.py. Other notable changes that touch our public API surface: * dd1ea52 llama : support multi-output backend sampling (ggml-org#25532) llama_context_params gains n_outputs_max_per_seq; llama_sampler_i.backend_init takes it as a third argument; new backend_reset / copy_state vtable slots and llama_sampler_copy(). * 153d324 llama : default load-mode auto, avoids mmap on iGPUs (ggml-org#26081) llama_load_mode gains LLAMA_LOAD_MODE_AUTO = -1 (enum is now signed). * 6e62ba5 mtmd: support pocket-tts (ggml-org#26871) mtmd_gen_inp/mtmd_gen_out gain seed/temp/feats/is_eos; new mtmd_gen_inp_default(); mtmd_helper_gen_audio_step_gen() takes out_stop. * 157b81f model : Granite-Switch Architecture (ggml-org#25107) * 7a20b41 model: MTP support for Nemotron (ggml-org#26725) and cc078b4 Dflash support for nemotron-3.5 (ggml-org#26905) * e23e944 vendor : cpp-httplib 0.53.0, 4c6766f vendor : subprocess.h sync Vendored patches preserved: * 919fde3 feat(rpc): thread-local last_error accessor — intact, upstream touched ggml-rpc.cpp by one unrelated line. * 3679b23 Fixes (RPC) — intact. * 2aa76c7 fix(metal): drop stray kernel_pad_f32 — still applies; upstream has since refactored pad into a templated kernel_pad_impl<T>, so the duplicate definition that referenced the nonexistent kargs_pad.s0..s3 is gone on both sides and nothing had to be re-applied.

…rg#26081) * llama: add new default load-mode auto which picks mmap unless a non-Metal iGPU is used * Update ggml/src/ggml-hexagon/ggml-hexagon.cpp Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com> * set mmap_support to false on OpenCL backend * fix order of load modes * use -1 for auto * resolve load mode auto earlier to correctly pick gpu host or cpu memory * add load mode auto to llama-bench * bump virtgpu api version, regenerate docs --------- Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

0cc4m requested review from a team, CISC, JohannesGaessler, ggerganov and marty1885 as code owners July 24, 2026 13:49

pwilkin approved these changes Jul 24, 2026

View reviewed changes

ngxson approved these changes Jul 24, 2026

View reviewed changes

taronaeo approved these changes Jul 24, 2026

View reviewed changes

ORippler self-assigned this Jul 27, 2026

github-actions Bot added the examples label Jul 28, 2026

0cc4m force-pushed the 0cc4m/mmap-auto branch from 6bcf37a to 36b9da0 Compare July 28, 2026 10:10

woodpenker reviewed Jul 29, 2026

View reviewed changes

ORippler reviewed Jul 29, 2026

View reviewed changes

0cc4m and others added 8 commits August 5, 2026 08:03

llama: add new default load-mode auto which picks mmap unless a non-M…

01649e6

…etal iGPU is used

Update ggml/src/ggml-hexagon/ggml-hexagon.cpp

9d8bd14

Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>

set mmap_support to false on OpenCL backend

4b4fada

fix order of load modes

2f2d246

use -1 for auto

9391a09

resolve load mode auto earlier to correctly pick gpu host or cpu memory

18c2399

add load mode auto to llama-bench

f00aaaa

bump virtgpu api version, regenerate docs

125b6c6

0cc4m force-pushed the 0cc4m/mmap-auto branch from 36b9da0 to 125b6c6 Compare August 5, 2026 06:12

0cc4m requested a review from a team as a code owner August 5, 2026 06:12

github-actions Bot added documentation Improvements or additions to documentation server labels Aug 5, 2026

ORippler approved these changes Aug 5, 2026

View reviewed changes

0cc4m added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Aug 10, 2026

Merge branch 'master' into 0cc4m/mmap-auto

233ec61

ggerganov merged commit 153d324 into master Aug 11, 2026
53 of 61 checks passed

ggerganov deleted the 0cc4m/mmap-auto branch August 11, 2026 06:20

claude Bot mentioned this pull request Aug 12, 2026

fix(container): update image ghcr.io/ggml-org/llama.cpp (server-cuda-b10362 ➔ server-cuda-b10380) drag0n141/home-ops#7239

Merged

1 task

fboudra mentioned this pull request Aug 12, 2026

common: migrate the deprecated --mmap/--no-mmap to --load-mode #26934

Merged

1 task

Navigation Menu

Uh oh!

llama: add default load-mode auto, which avoids mmap on iGPUs - #26081

llama: add default load-mode auto, which avoids mmap on iGPUs#26081
ggerganov merged 9 commits into
masterfrom
0cc4m/mmap-auto

0cc4m commented Jul 24, 2026 •

edited

Loading

Uh oh!

pwilkin left a comment

Uh oh!

0cc4m commented Jul 28, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

0cc4m Jul 29, 2026

Uh oh!

ORippler left a comment

Uh oh!

kpouget commented Jul 29, 2026

Uh oh!

ORippler left a comment

Uh oh!

kpouget commented Aug 5, 2026

Uh oh!

elchic00 commented Aug 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

Uh oh!

Conversation

0cc4m commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Requirements

Uh oh!

pwilkin left a comment

Choose a reason for hiding this comment

Uh oh!

0cc4m commented Jul 28, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

0cc4m Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

ORippler left a comment

Choose a reason for hiding this comment

Uh oh!

kpouget commented Jul 29, 2026

Uh oh!

ORippler left a comment

Choose a reason for hiding this comment

Uh oh!

kpouget commented Aug 5, 2026

Uh oh!

elchic00 commented Aug 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

0cc4m commented Jul 24, 2026 •

edited

Loading