metal : add residency sets keep-alive heartbeat by ggerganov · Pull Request #17766 · ggml-org/llama.cpp

ggerganov · 2025-12-04T19:33:24Z

So something changed in MacOS recently because the fix from #11427 no longer works - the memory wiring/unwiring (a.k.a. throttling) after 1 second of being idle is back. Maybe this happened with the update to MacOS Tahoe - not sure.

Here are the results on master:

make -j && ./bin/llama-idle -m ../models/llama-3.1-70b/ggml-model-f16.gguf

Details

0.06.147.450 I ggml_metal_init: allocating
0.06.147.477 I ggml_metal_init: found device: Apple M2 Ultra
0.06.147.483 I ggml_metal_init: picking default device: Apple M2 Ultra
0.06.147.486 I ggml_metal_init: use fusion         = true
0.06.147.486 I ggml_metal_init: use concurrency    = true
0.06.147.487 I ggml_metal_init: use graph optimize = true
0.06.147.505 I llama_context:        CPU  output buffer size =     0.49 MiB
0.06.151.959 I llama_kv_cache:      Metal KV buffer size =   160.00 MiB
0.06.159.411 I llama_kv_cache: size =  160.00 MiB (   512 cells,  80 layers,  1/1 seqs), K (f16):   80.00 MiB, V (f16):   80.00 MiB
0.06.160.621 I llama_context: Flash Attention was auto, set to enabled
0.06.170.950 I llama_context:      Metal compute buffer size =   282.50 MiB
0.06.170.952 I llama_context:        CPU compute buffer size =    17.01 MiB
0.06.170.952 I llama_context: graph nodes  = 2487
0.06.170.952 I llama_context: graph splits = 2
  - decode time:   193.05 ms
  - decode time:   192.43 ms
  - decode time:   192.13 ms
  - decode time:   192.62 ms
  - decode time:   193.71 ms
  - decode time:   192.02 ms
  - decode time:   191.88 ms
  - decode time:   193.06 ms
  - decode time:   192.33 ms
  - decode time:   191.80 ms
iters:   10, pause:   200 ms, avg decode time:   192.50 +/- 0.61 ms
  - decode time:   192.67 ms
  - decode time:   192.49 ms
  - decode time:   191.15 ms
  - decode time:   192.62 ms
  - decode time:   192.98 ms
  - decode time:   191.24 ms
  - decode time:   191.12 ms
  - decode time:   191.50 ms
  - decode time:   191.59 ms
  - decode time:   192.20 ms
iters:   10, pause:   400 ms, avg decode time:   191.95 +/- 0.71 ms
  - decode time:   191.18 ms
  - decode time:   191.54 ms
  - decode time:   190.83 ms
  - decode time:   191.12 ms
  - decode time:   190.08 ms
  - decode time:   190.78 ms
  - decode time:   191.07 ms
  - decode time:   191.04 ms
  - decode time:   196.06 ms
  - decode time:   192.10 ms
iters:   10, pause:   600 ms, avg decode time:   191.58 +/- 1.66 ms
  - decode time:   192.49 ms
  - decode time:   191.13 ms
  - decode time:   191.46 ms
  - decode time:   191.73 ms
  - decode time:   191.92 ms
  - decode time:   193.68 ms
  - decode time:   192.04 ms
  - decode time:   191.72 ms
  - decode time:   192.06 ms
  - decode time:   191.81 ms
iters:   10, pause:   800 ms, avg decode time:   192.01 +/- 0.69 ms
  - decode time:   193.58 ms
  - decode time:   194.51 ms
  - decode time:   192.86 ms
  - decode time:   433.47 ms
  - decode time:   190.99 ms
  - decode time:   193.74 ms
  - decode time:   199.78 ms
  - decode time:   190.71 ms
  - decode time:   191.84 ms
  - decode time:   193.31 ms
iters:   10, pause:  1000 ms, avg decode time:   217.48 +/- 75.93 ms
  - decode time:   412.50 ms
  - decode time:   309.28 ms
  - decode time:   360.40 ms
  - decode time:   387.33 ms
  - decode time:   370.75 ms
  - decode time:   376.98 ms
  - decode time:  1377.47 ms
  - decode time:  1300.97 ms
  - decode time:  1298.22 ms
  - decode time:  1327.39 ms
iters:   10, pause:  1200 ms, avg decode time:   752.13 +/- 495.04 ms
  - decode time:  1067.61 ms
  - decode time:  1125.08 ms
  - decode time:  1119.94 ms
  - decode time:  1315.78 ms
  - decode time:  1087.12 ms
  - decode time:  1091.60 ms
  - decode time:  1087.27 ms
  - decode time:  1084.48 ms
  - decode time:  1087.62 ms
  - decode time:  1087.20 ms
iters:   10, pause:  1400 ms, avg decode time:  1115.37 +/- 72.44 ms
  - decode time:  1598.81 ms
  - decode time:  1659.83 ms
  - decode time:  1704.90 ms
  - decode time:  1576.85 ms
  - decode time:  1682.92 ms
  - decode time:  1660.09 ms
  - decode time:  1685.85 ms
  - decode time:  1616.58 ms
  - decode time:  1660.40 ms
  - decode time:  1683.09 ms
iters:   10, pause:  1600 ms, avg decode time:  1652.93 +/- 41.88 ms
  - decode time:  1451.54 ms
  - decode time:  1429.19 ms
  - decode time:  1448.97 ms
  - decode time:  1472.25 ms
  - decode time:  1437.39 ms
  - decode time:  1455.94 ms
  - decode time:  1419.29 ms
  - decode time:  1471.90 ms
  - decode time:  1430.76 ms
  - decode time:  1461.42 ms
iters:   10, pause:  1800 ms, avg decode time:  1447.86 +/- 18.27 ms
2.36.527.021 I ggml_metal_free: deallocating

And here are the results with this PR:

Details

0.06.159.343 I ggml_metal_init: allocating
0.06.159.377 I ggml_metal_init: found device: Apple M2 Ultra
0.06.159.383 I ggml_metal_init: picking default device: Apple M2 Ultra
0.06.159.386 I ggml_metal_init: use fusion         = true
0.06.159.386 I ggml_metal_init: use concurrency    = true
0.06.159.387 I ggml_metal_init: use graph optimize = true
0.06.159.403 I llama_context:        CPU  output buffer size =     0.49 MiB
0.06.164.393 I llama_kv_cache:      Metal KV buffer size =   160.00 MiB
0.06.171.847 I llama_kv_cache: size =  160.00 MiB (   512 cells,  80 layers,  1/1 seqs), K (f16):   80.00 MiB, V (f16):   80.00 MiB
0.06.173.129 I llama_context: Flash Attention was auto, set to enabled
0.06.183.473 I llama_context:      Metal compute buffer size =   282.50 MiB
0.06.183.482 I llama_context:        CPU compute buffer size =    17.01 MiB
0.06.183.482 I llama_context: graph nodes  = 2487
0.06.183.484 I llama_context: graph splits = 2
  - decode time:   191.36 ms
  - decode time:   191.88 ms
  - decode time:   192.17 ms
  - decode time:   190.04 ms
  - decode time:   191.18 ms
  - decode time:   191.01 ms
  - decode time:   192.39 ms
  - decode time:   192.18 ms
  - decode time:   193.01 ms
  - decode time:   192.78 ms
iters:   10, pause:   200 ms, avg decode time:   191.80 +/- 0.90 ms
  - decode time:   191.12 ms
  - decode time:   191.23 ms
  - decode time:   191.20 ms
  - decode time:   192.33 ms
  - decode time:   190.71 ms
  - decode time:   192.59 ms
  - decode time:   192.66 ms
  - decode time:   192.14 ms
  - decode time:   190.35 ms
  - decode time:   191.60 ms
iters:   10, pause:   400 ms, avg decode time:   191.59 +/- 0.80 ms
  - decode time:   193.46 ms
  - decode time:   194.52 ms
  - decode time:   191.91 ms
  - decode time:   190.64 ms
  - decode time:   194.11 ms
  - decode time:   191.17 ms
  - decode time:   192.81 ms
  - decode time:   192.62 ms
  - decode time:   192.73 ms
  - decode time:   191.18 ms
iters:   10, pause:   600 ms, avg decode time:   192.51 +/- 1.29 ms
  - decode time:   199.77 ms
  - decode time:   195.51 ms
  - decode time:   192.30 ms
  - decode time:   190.35 ms
  - decode time:   193.33 ms
  - decode time:   190.30 ms
  - decode time:   193.18 ms
  - decode time:   192.25 ms
  - decode time:   192.48 ms
  - decode time:   193.67 ms
iters:   10, pause:   800 ms, avg decode time:   193.31 +/- 2.74 ms
  - decode time:   193.74 ms
  - decode time:   194.80 ms
  - decode time:   192.59 ms
  - decode time:   192.08 ms
  - decode time:   194.69 ms
  - decode time:   191.29 ms
  - decode time:   191.55 ms
  - decode time:   196.23 ms
  - decode time:   191.95 ms
  - decode time:   192.46 ms
iters:   10, pause:  1000 ms, avg decode time:   193.14 +/- 1.64 ms
  - decode time:   221.25 ms
  - decode time:   222.57 ms
  - decode time:   219.05 ms
  - decode time:   218.60 ms
  - decode time:   191.75 ms
  - decode time:   223.81 ms
  - decode time:   217.56 ms
  - decode time:   192.10 ms
  - decode time:   219.01 ms
  - decode time:   217.57 ms
iters:   10, pause:  1200 ms, avg decode time:   214.33 +/- 11.99 ms
  - decode time:   227.78 ms
  - decode time:   218.82 ms
  - decode time:   212.44 ms
  - decode time:   222.10 ms
  - decode time:   225.00 ms
  - decode time:   229.71 ms
  - decode time:   226.76 ms
  - decode time:   218.06 ms
  - decode time:   222.88 ms
  - decode time:   220.53 ms
iters:   10, pause:  1400 ms, avg decode time:   222.41 +/- 5.19 ms
  - decode time:   226.02 ms
  - decode time:   227.07 ms
  - decode time:   224.31 ms
  - decode time:   221.87 ms
  - decode time:   224.10 ms
  - decode time:   226.71 ms
  - decode time:   225.18 ms
  - decode time:   228.20 ms
  - decode time:   224.98 ms
  - decode time:   221.39 ms
iters:   10, pause:  1600 ms, avg decode time:   224.98 +/- 2.18 ms
  - decode time:   225.10 ms
  - decode time:   225.45 ms
  - decode time:   229.04 ms
  - decode time:   229.66 ms
  - decode time:   229.84 ms
  - decode time:   221.65 ms
  - decode time:   232.24 ms
  - decode time:   229.59 ms
  - decode time:   227.63 ms
  - decode time:   227.49 ms
iters:   10, pause:  1800 ms, avg decode time:   227.77 +/- 3.03 ms
1.55.608.188 I ggml_metal_free: deallocating

It seems that attaching the residency sets to the Metal queue mostly eliminates the unwiring of the memory. Although, every now and then, it still seems to occur - not sure if this was the case before on MacOS Sequoia.

Edit: Just attaching the residency sets to the Metal queue is not enough. Ended up implementing a background thread that periodically calls MTLResidencySet::requestResidency() in order to keep the memory buffers wired. The thread loops as a heartbeat in the background and requests residency for all sets approximately every 500ms.

By default, this heartbeat stops after 3 minutes of inactivity. It can be controlled with the environment variable:

# keep the memory wired for 30 seconds after last activity (e.g. graph computation)
GGML_METAL_RESIDENCY_KEEP_ALIVE_S=30 ...

ggerganov · 2025-12-04T20:33:24Z

Yes, definitely something happened with the new MacOS Tahoe. It keep unwiring the memory, even with the patch from this PR. The only way to force it to not unwire it is by running:

sudo sysctl iogpu.disable_wired_collector=1

But this is quite inconvenient. Bummer..

MacOS Sequoia 15.5 (no unwiring)
MacOS Tahoe 26.1 (memory gets unwired 1 second after model usage)
MacOS Tahoe 26.1 (with iogpu.disable_wired_collector=1 there is no unwiring)

Edit: fixed by periodically requesting residency in a background thread

CISC · 2025-12-05T19:48:15Z

https://github.com/ggml-org/llama.cpp/actions/runs/19971070125/job/57275662870#step:5:1733

* examples : add idle * metal : attach residency sets to queue * idle : add link * idle : adjust intervals * metal : add residency sets keep-alive heartbeat * cont : adjust default keep-alive time

ggerganov added 3 commits December 4, 2025 21:22

examples : add idle

84c2a6d

metal : attach residency sets to queue

a103ca1

idle : add link

5074a21

loci-dev mentioned this pull request Dec 4, 2025

UPSTREAM PR #17766: metal : attach residency sets to queue auroralabs-loci/llama.cpp#437

Open

github-actions Bot added examples ggml changes relating to the ggml tensor library for machine learning Apple Metal https://en.wikipedia.org/wiki/Metal_(API) labels Dec 4, 2025

ggerganov added 2 commits December 5, 2025 10:29

idle : adjust intervals

5f52ca5

metal : add residency sets keep-alive heartbeat

b2d7fa6

ggerganov changed the title ~~metal : attach residency sets to queue~~ metal : add residency sets keep-alive heartbeat Dec 5, 2025

cont : adjust default keep-alive time

df4f195

ggerganov merged commit c41bde6 into master Dec 5, 2025
76 of 78 checks passed

ggerganov deleted the gg/idle-new branch December 5, 2025 17:38

ggerganov mentioned this pull request Dec 5, 2025

metal : fix build #17799

Merged

gabe-l-hart mentioned this pull request Dec 10, 2025

feat: llama.cpp bump (17f7f4) for SSM performance improvements ollama/ollama#13408

Merged

ggerganov mentioned this pull request Dec 19, 2025

Feature Request: Idle model unload timeout (router mode / config.ini) #18189

Closed

4 tasks

netshade mentioned this pull request Jan 27, 2026

Eval bug: GGML_ASSERT([rsets->data count] == 0) #19137

Closed

This was referenced May 1, 2026

Metal: ggml_metal_rsets_free assertion fires deterministically on device free (missing rsets_remove in buffer rset free path) #22593

Open

metal : pair rsets_add with rsets_rm on buffer free (fix #22593) #22595

Open

FuJacob mentioned this pull request May 28, 2026

Bump llama.cpp to b9310 to fix Metal deadlock on macOS 26 FuJacob/cotabbyinference#2

Merged

ggerganov mentioned this pull request Jun 3, 2026

metal : reduce rset heartbeat from 500ms -> 5ms #24074

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

metal : add residency sets keep-alive heartbeat#17766

metal : add residency sets keep-alive heartbeat#17766
ggerganov merged 6 commits into
masterfrom
gg/idle-new

ggerganov commented Dec 4, 2025 •

edited

Loading

Uh oh!

ggerganov commented Dec 4, 2025 •

edited

Loading

Uh oh!

Uh oh!

CISC commented Dec 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

ggerganov commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ggerganov commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

CISC commented Dec 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ggerganov commented Dec 4, 2025 •

edited

Loading

ggerganov commented Dec 4, 2025 •

edited

Loading