rpc: support apple RDMA as an RPC transport by ryan5rdx · Pull Request #26421 · ggml-org/llama.cpp

ryan5rdx · 2026-08-01T23:50:15Z

Overview

Extending the RPC transport layer to support Apple's RDMA implementation (alongside the existing linux one). A lot of the implementation details here are the result of what's outlined in Apple's tech note TN3205, while trying to stay as close to the existing linux implementation as possible. IIUC, the linux RDMA path should be unchanged, but I don't have linux hardware to test so would appreciate someone verifying that everything (still)works fine there.

Additional information

From the inline comment: the core differences in this implementation vs linux RDMA are:

UC(unreliable connection) on Apple vs RC(reliable connection) QP transport types on Linux (though in practice UC on Apple is still lossless)
fixed 128KiB stride on Apple vs variable chunk size on Linux
relying on Apple's hardware credit based flow control vs RNR NAKs + retries on Linux

on Apple a SEND and its corresponding RECV must cover the same number of 4 KiB Thunderbolt frames, so every SEND posts a whole 128KiB stride over the wire, even when partially filled. (In testing 128KiB was the best performing among 32, 64, 128, 256)

There are some micro optimizations included here per AI suggestion (eg. no flush on RPC_CMD_SET_TENSOR for coalescing, forcing a flush on ggml_backend_rpc_synchronize) In testing they didn't seem to improve much, but leaving them in for now because they seem harmless+correct 🙃

Testing

Testing with Qwen3-0.6B-UD-Q4_K_XL.gguf, M3 Ultra, layer parallel:

TCP baseline

Nodes	Prefill t/s	Decode t/s
1	7620	304.8
2	5296	115.6
3	4498	101.1
4	4007	87.7

Apple RDMA

Nodes	Prefill t/s	Decode t/s
1	7620	304.8
2	7227	261.3
3	7233	164.2
4	6250	133.5

Performance vs TCP

nodes	prefill	decode
2	136.46%	226.04%
3	160.80%	162.41%
4	155.98%	152.22%

These numbers are so high because for this tiny model the RPCs are a big chunk of the per-token-time relative to compute - I'll try to also try to benchmark with qwen3.6 27B shortly

Note that when testing - on workers with multiple active TB interfaces (often the case), pass the one connected to your client via the GGML_RDMA_DEV env var, eg:

GGML_RDMA_DEV=rdma_en7 ./bin/ggml-rpc-server   --host 0.0.0.0 --port 50052 -d MTL0 -c

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: Yes - Used claude to assist here - I've tested + reviewed every line and am prepared to discuss them.

ryan5rdx · 2026-08-02T19:23:39Z

benchmarks for Unsloth's Qwen 3.6 27B Q4_K_XL for something more practical:

cli:

./bin/llama-cli -m ~/Downloads/Qwen3.6-27B-MTP-UD-Q4_K_XL.gguf  -c 100000 --reasoning on -fa on --rpc <ip>:50052,...  -np 1 --jinja --reasoning-preserve -ub 512 -b 4096  --no-mmap  -ngl 999

workers:

./bin/ggml-rpc-server   --host 0.0.0.0 --port 50052 -d MTL0 -c  -t 12

TCP Baseline

Nodes	Prefill t/s	Decode t/s
1	245.2	24.4
2	221.4	18
3	206.7	17.3
4	196	17.7

Apple RDMA

Nodes	Prefill t/s	Decode t/s
1	245.2	24.4
2	237.5	22.5
3	232.2	20.5
4	227.8	19.8

Performance vs TCP

Nodes	Prefill	Decode
2	107.27%	125.00%
3	112.34%	118.50%
4	116.22%	111.86%

ryan5rdx · 2026-08-04T00:33:01Z

Another datapoint, using https://huggingface.co/ggml-org/DeepSeek-V4-Flash-0731-GGUF . I don't have hardware to test a 1 node baseline so pulled that from @ggerganov 's result here

I'm unable to test with >2 nodes because deepseek v4 fails to init with 3, 4 RPC backends currently..

Apple RDMA

Nodes	Prefill t/s	Decode t/s
1	408.93	27.6
2	290	22.37

TCP

Nodes	Prefill t/s	Decode t/s
1	408.93	27.6
2	275.1	14.7

Performance vs TCP

Nodes	Prefill	Decode
2	105.42%	152.18%

cc @rgerganov given this is RPC work, let me know if there's anything I can do to move this forward. Been daily driving this for a few weeks now with other models.

am17an · 2026-08-05T05:33:24Z

@ryan5rdx can you try #26610?

ggerganov · 2026-08-05T06:59:26Z

@ryan5rdx None of the current maintainers has Macs with RDMA connection, so it might be a while before we can review these changes properly. If you are interested in helping with maintaining the RDMA transport implementation, I can invite you as a maintainer of the RPC backend. With the ongoing work in #26610 we can use your help with testing and fixing issues on Apple clusters. LMK

ryan5rdx · 2026-08-05T07:18:32Z

@ryan5rdx None of the current maintainers has Macs with RDMA connection, so it might be a while before we can review these changes properly. If you are interested in helping with maintaining the RDMA transport implementation, I can invite you as a maintainer of the RPC backend. With the ongoing work in #26610 we can use your help with testing and fixing issues on Apple clusters. LMK

sure feel free to add me - happy to help test where I can.

rgerganov

thanks for the patch; I don't have the hardware to test this but I am fine accepting this as long as it doesn't change the transport interface without good reasons and doesn't break the linux implementation

ryan5rdx · 2026-08-05T11:03:36Z

@rgerganov thanks for the review! Reverted the set_tensor optimization + socket pinning, quick smoke test resulted in same performance, so they didn't seem to be doing much anyway.

rgerganov · 2026-08-05T11:23:43Z

@@ -308,7 +311,7 @@ static bool send_rpc_cmd(socket_ptr sock, enum rpc_cmd cmd, const void * input,
    if (!sock->send_data(input, input_size)) {
        return false;
    }
-    return true;
+    return sock->flush();


can you try to measure the performance impact of having an explicit flush()? i.e. flushing on every send vs manual flushing like here

measured - without this manual flush the coalesing breaks somewhat (we end up sending ~two more empty TB frames per RPC command).

flushing on every send: 21.17 t/s tg 2048
manual flush: 22.41 t/s tg 2048

prefill appears to be a little slower, but within ~margin of error 290 vs ~282 t/s pp2048

rgerganov

no CI build with RPC enabled is currently triggered on MacOS; please fix build-apple.yml to trigger CI builds on changes in ggml/src/ggml-rpc/**

ryan5rdx · 2026-08-06T18:04:47Z

arm64 apple build looks good - I see other apple failures (x86, tvos?) but those look like infra issues, and I can't seem to retrigger (though I'm not sure if those should work anyway):


Error: Service Unavailable
Error: Failed to resolve action download info.

ryan5rdx · 2026-08-09T06:16:11Z

@rgerganov let me know if there's anything I can do here!

rgerganov

I am fine to merge this, just need to do some final testing on the spark cluster that I have. I'm travelling this week, will post updates after 15th. Thanks.

ryan5rdx · 2026-08-19T18:32:00Z

@rgerganov thanks! Hoping to merge this soon!

Once this is in I plan to also put a PR up to support similar polled-release fence to what I've put up here: antirez/ds4#743 which should result in some additional big reductions in apple TB RPC latency

ryan5rdx · 2026-08-24T17:17:27Z

@rgerganov sorry to keep prodding here! Maybe @ggerganov could take a look? Looking forward to brining the metal RPC topology up to speed, esp now that #26490 has landed!

ggerganov · 2026-08-25T14:34:59Z

@ryan5rdx Could you rebase the branch on latest master?

srigi · 2026-08-27T12:42:03Z

This change renders llama.cpp not working anymore in MacOS Sequoia:

❯ ./llama-server --list-devices
dyld[52025]: Library not loaded: /usr/lib/librdma.dylib
  Referenced from: <8884FFC7-AE3A-31EE-9433-58C08295B8FA> /Users/srigi/Downloads/llama-b10628/libggml-rpc.0.22.0.dylib
  Reason: tried: '/usr/lib/librdma.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/lib/librdma.dylib' (no such file), '/usr/lib/librdma.dylib' (no such file, not in dyld cache)
[1]    52025 abort      ./llama-server --list-devices

You pretty much locked all Sequoia users to the previous version without any future updates.

ryan5rdx · 2026-08-27T16:24:37Z

This change renders llama.cpp not working anymore in MacOS Sequoia:

❯ ./llama-server --list-devices
dyld[52025]: Library not loaded: /usr/lib/librdma.dylib
  Referenced from: <8884FFC7-AE3A-31EE-9433-58C08295B8FA> /Users/srigi/Downloads/llama-b10628/libggml-rpc.0.22.0.dylib
  Reason: tried: '/usr/lib/librdma.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/lib/librdma.dylib' (no such file), '/usr/lib/librdma.dylib' (no such file, not in dyld cache)
[1]    52025 abort      ./llama-server --list-devices

You pretty much locked all Sequoia users to the previous version without any future updates.

opps sorry - mind testing #27815 ?

srigi · 2026-08-27T16:58:02Z

opps sorry - mind testing #27815 ?

Your branch compiled successfully and is working OK. Thank you.

ggerganov · 2026-08-28T11:56:46Z

@ryan5rdx Have you seen this error when running with RDMA:

ryan5rdx · 2026-08-28T16:03:13Z

@ryan5rdx Have you seen this error when running with RDMA:

could you send your command? and hardware? IIUC this error is cosmetic( RDMA buffer/QP cleanup on destroy), so I suspect your workflow is creating/destroying many RDMA sockets over and over, I'll look into it once I have the repro.

I used to see it on llama-cpp server/cli shutdown sometimes as I was developing

ggerganov · 2026-08-28T16:26:57Z

On MacBook M5 Max 48GB:

./bin/ggml-rpc-server --host 0.0.0.0 --port 50052 -d MTL0 -c

On MacBook M5 Max 128GB:

./bin/llama-server -hf ggml-org/Qwen3.8-27B-GGUF:Q8_0 --rpc 192.168.100.1:50052 --host 0.0.0.0 --port 8044 -lv 4 -c 65536

ryan5rdx · 2026-08-28T20:27:37Z

On MacBook M5 Max 48GB:

./bin/ggml-rpc-server --host 0.0.0.0 --port 50052 -d MTL0 -c

On MacBook M5 Max 128GB:

./bin/llama-server -hf ggml-org/Qwen3.8-27B-GGUF:Q8_0 --rpc 192.168.100.1:50052 --host 0.0.0.0 --port 8044 -lv 4 -c 65536

#27908 should address this - doing some final tests now.

* rpc: support apple RDMA as an RPC transport * remove set_tensor micro optimization, rpc socket pinning per CR * remove transparent reconnect * trigger apple builds on RPC changes --------- Co-authored-by: Ryan Churaman <rschu@meta.com>

radekg · 2026-09-03T20:29:53Z

What would be great is to revisit the decision of having to push all tensors from the server to worker nodes on cold start. It would be much better if one could distribute GGUF files to every node with a worker and only send metadata from the server to workers.

I've been experimenting with custom TCP over JACCL before I learned about this PR and in all my experiments it still takes minutes before a model is distributed. I know about the cache but in certain cases cache might not be preferred.

Instead, workers should be able to hold their copy of the model, the server should be telling the worker "these tensors from this GGUF need to be loaded", and the worker should simply trust the server. The main bottleneck appears to be sequential processing of every byte, and tensor reconstruction on the other side. I'd be happy to help getting that in, I also can help testing on 4x M3 Ultra.

I bet this would also solve this problem #28126.

Great to see this landing here, though! /cc @ggerganov @ryan5rdx

cdvankammen · 2026-09-04T21:22:31Z

What would be great is to revisit the decision of having to push all tensors from the server to worker nodes on cold start. It would be much better if one could distribute GGUF files to every node with a worker and only send metadata from the server to workers.

I've been experimenting with custom TCP over JACCL before I learned about this PR and in all my experiments it still takes minutes before a model is distributed. I know about the cache but in certain cases cache might not be preferred.

Instead, workers should be able to hold their copy of the model, the server should be telling the worker "these tensors from this GGUF need to be loaded", and the worker should simply trust the server. The main bottleneck appears to be sequential processing of every byte, and tensor reconstruction on the other side. I'd be happy to help getting that in, I also can help testing on 4x M3 Ultra.

I bet this would also solve this problem #28126.

Great to see this landing here, though! /cc @ggerganov @ryan5rdx

I really think this would be a great idea i like this do you have a request you have made for this i can add my vote to it

github-actions Bot added documentation Improvements or additions to documentation examples ggml changes relating to the ggml tensor library for machine learning labels Aug 1, 2026

ryan5rdx marked this pull request as ready for review August 1, 2026 23:56

ryan5rdx requested a review from a team as a code owner August 1, 2026 23:56

ryan5rdx mentioned this pull request Aug 5, 2026

RPC: add -sm tensor #26610

Open

rgerganov reviewed Aug 5, 2026

View reviewed changes

Comment thread ggml/src/ggml-rpc/ggml-rpc.cpp Outdated

Comment thread ggml/src/ggml-rpc/ggml-rpc.cpp Outdated

ryan5rdx force-pushed the ryan5rdx/apple-rdma-transport branch from c7b1b67 to 7cdd3fe Compare August 5, 2026 11:01

ryan5rdx requested a review from rgerganov August 5, 2026 11:17

rgerganov reviewed Aug 5, 2026

View reviewed changes

ryan5rdx requested a review from rgerganov August 6, 2026 02:40

rgerganov reviewed Aug 6, 2026

View reviewed changes

ryan5rdx requested a review from a team as a code owner August 6, 2026 15:32

github-actions Bot added the devops improvements to build systems and github actions label Aug 6, 2026

ryan5rdx requested a review from rgerganov August 6, 2026 15:44

rgerganov reviewed Aug 9, 2026

View reviewed changes

Kononnable mentioned this pull request Aug 24, 2026

Feature Request: RDMA support for rpc back ends #9493

Open

4 tasks

rgerganov approved these changes Aug 25, 2026

View reviewed changes

rgerganov added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Aug 25, 2026

Ryan Churaman and others added 4 commits August 25, 2026 09:51

rpc: support apple RDMA as an RPC transport

28bd8e8

remove set_tensor micro optimization, rpc socket pinning per CR

1f625d4

remove transparent reconnect

b3e5281

trigger apple builds on RPC changes

0de310d

ryan5rdx force-pushed the ryan5rdx/apple-rdma-transport branch from a8e7767 to 0de310d Compare August 25, 2026 16:51

ggerganov merged commit b114b47 into ggml-org:master Aug 25, 2026
29 of 31 checks passed

ryan5rdx mentioned this pull request Aug 27, 2026

fix pre-rdma macOS versions #27815

Merged

ryan5rdx mentioned this pull request Aug 28, 2026

rpc: fix apple rdma error spew on teardown #27908

Merged

marksmeltzer mentioned this pull request Sep 3, 2026

Is this repository even actively being watched and maintained in the last 4 months? exo-explore/exo#2258

Open

This was referenced Sep 4, 2026

ggml : bump version to 0.23.0 ggml-org/ggml#1618

Merged

llama.cpp : bump version to 0.4.0 #28386

Merged

BrewTestBot mentioned this pull request Sep 4, 2026

ggml 0.23.0 Homebrew/homebrew-core#302393

Merged

1 task

Uh oh!

Conversation

ryan5rdx commented Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

Testing

TCP baseline

Apple RDMA

Performance vs TCP

Requirements

Uh oh!

ryan5rdx commented Aug 2, 2026

TCP Baseline

Apple RDMA

Performance vs TCP

Uh oh!

ryan5rdx commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Apple RDMA

TCP

Performance vs TCP

Uh oh!

am17an commented Aug 5, 2026

Uh oh!

ggerganov commented Aug 5, 2026

Uh oh!

ryan5rdx commented Aug 5, 2026

Uh oh!

rgerganov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ryan5rdx commented Aug 5, 2026

Uh oh!

rgerganov Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

ryan5rdx Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rgerganov left a comment

Choose a reason for hiding this comment

Uh oh!

ryan5rdx commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ryan5rdx commented Aug 9, 2026

Uh oh!

rgerganov left a comment

Choose a reason for hiding this comment

Uh oh!

ryan5rdx commented Aug 19, 2026

Uh oh!

ryan5rdx commented Aug 24, 2026

Uh oh!

ggerganov commented Aug 25, 2026

Uh oh!

Uh oh!

srigi commented Aug 27, 2026

Uh oh!

ryan5rdx commented Aug 27, 2026

Uh oh!

srigi commented Aug 27, 2026

Uh oh!

ggerganov commented Aug 28, 2026

Uh oh!

ryan5rdx commented Aug 28, 2026

Uh oh!

ggerganov commented Aug 28, 2026

Uh oh!

ryan5rdx commented Aug 28, 2026

Uh oh!

radekg commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cdvankammen commented Sep 4, 2026

ryan5rdx commented Aug 1, 2026 •

edited

Loading

ryan5rdx commented Aug 4, 2026 •

edited

Loading

ryan5rdx Aug 6, 2026 •

edited

Loading

ryan5rdx commented Aug 6, 2026 •

edited

Loading

radekg commented Sep 3, 2026 •

edited

Loading