rpc: parallelize cached tensor hashing during model load

[chuyqa]

Overview

Related to #25890.

Up to 20-60% lower RPC model load time by adding GGML_RPC_LOAD_THREADS.

On RPC-cache loads, the client fully FNV-hashes each large RPC tensor one at a time before dispatching it. For large models, this can pin a single CPU core at 100% while the remaining cores sit mostly idle.

This change:

• Uses an opt-in worker pool (GGML_RPC_LOAD_THREADS) to hash large mmap-backed RPC tensors concurrently.
• Serializes complete set_tensor transactions, because RPC devices for the same endpoint may share a socket (e.g. RPC0 and RPC1 will share a socket).
• Makes no changes to the default loading path, RPC protocol, or cache format

Additional information

Main host: Ryzen 9 3900X, 128 GB RAM, 2× RTX 4060 Ti 16 GB,
RPC host: Ryzen 5 9600X, 192 GB RAM, 1× RTX 4060 Ti 16 GB
Connected via 2.5gbps peer to peer lan.
Model: Kimi-K2.7-Code UD-IQ2_M, 318 GiB, 8 files
RPC CPU allocation: approximately 143 GiB
GGML_RPC_LOAD_THREADS=12

On this very limited hardware, we see a near 3x improvement to the loading phase:

On b10173 - "state":"loading" 4min54sec

load-mode = mmap

[55965] 1.57.697.172 I load_tensors: RPC0[10.44.0.2:50052] model buffer size =   594.13 MiB
[55965] 1.57.697.173 I load_tensors: RPC1[10.44.0.2:50052] model buffer size = 143149.18 MiB
[55965] cmd_child_to_router:state:{"state":"loading","payload":{"stages":["text_model"],"current":"text_model","value":0.0}}
[55965] cmd_child_to_router:state:{"state":"loading","payload":{"stages":["text_model"],"current":"text_model","value":1.0}}
[55965] 6.51.217.230 I cmn  common_init_: added [EOS] logit bias = -inf

With this PR and GGML_RPC_LOAD_THREADS 12 - "state":"loading" 1min38sec

-- env GGML_RPC_LOAD_THREADS 12 
load-mode = mmap

[49635] 1.58.187.278 I load_tensors: RPC0[10.44.0.2:50052] model buffer size =   594.13 MiB
[49635] 1.58.187.278 I load_tensors: RPC1[10.44.0.2:50052] model buffer size = 143149.18 MiB
[49635] cmd_child_to_router:state:{"state":"loading","payload":{"stages":["text_model"],"current":"text_model","value":0.0}}
[49635] cmd_child_to_router:state:{"state":"loading","payload":{"stages":["text_model"],"current":"text_model","value":0.5258322954177856}}
[49635] 2.06.980.323 I load_all_data: preloading 20 RPC tensors (0.39 GiB across 1 GGUF shards) with 12 mmap workers
[49635] 2.07.513.035 I load_all_data: parallel RPC preload complete in 0.53 s (0.73 GiB/s)
[49635] cmd_child_to_router:state:{"state":"loading","payload":{"stages":["text_model"],"current":"text_model","value":0.5276646614074707}}
[49635] 2.08.208.854 I load_all_data: preloading 265 RPC tensors (138.07 GiB across 7 GGUF shards) with 12 mmap workers
[49635] 3.25.386.061 I load_all_data: parallel RPC preload complete in 77.17 s (1.79 GiB/s)
[49635] cmd_child_to_router:state:{"state":"loading","payload":{"stages":["text_model"],"current":"text_model","value":1.0}}
[49635] 3.36.552.404 I cmn  common_init_: added [EOS] logit bias = -inf

With this PR and GGML_RPC_LOAD_THREADS not set - "state":"loading" 4min47sec

Test the existing code path to ensure we dont regress

load-mode = mmap

 [59581] 2.07.956.707 I load_tensors: RPC0[10.44.0.2:50052] model buffer size =   594.13 MiB
[59581] 2.07.956.707 I load_tensors: RPC1[10.44.0.2:50052] model buffer size = 143149.18 MiB
[59581] cmd_child_to_router:state:{"state":"loading","payload":{"stages":["text_model"],"current":"text_model","value":0.0}}
[59581] cmd_child_to_router:state:{"state":"loading","payload":{"stages":["text_model"],"current":"text_model","value":1.0}}
[59581] 6.54.513.444 I cmn  common_init_: added [EOS] logit bias = -inf

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure:
  YES.
  Grok to find the problem / single threaded slowdown.
  GPT to find a solution
  Grok to refine the solution
  GPT to optimize and test.

[ggml-gh-bot]

Hi @chuyqa, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[jaholmesuk]

Tested on the 96-core Arm box from #25890. It works: 1.55x here, and it moves the bottleneck to the server.

Two nodes, 96-core Arm Neoverse-N2, 672 GB each, connected at 100 GbE and comfortably capable of 70+ Gbps sustained node-to-node. GLM-4.5-Air Q4_K_M (68 GB), -ngl 20, so 27 GB sits on the RPC worker. Model and tensor cache both on tmpfs. Base 3018a11e79e4 vs this PR's head. Time from server start to /health 200.

Build	-c	GGML_RPC_LOAD_THREADS	Load
base	on (warm)	unset	74 / 73 / 73 s
this PR	on (warm)	64	47 / 46 / 49 / 48 s
this PR	on (warm)	unset	73 s
base	off	unset	86 s
this PR	off	64	66 s

No regression with the env var unset. It also helps with no cache at all (86 to 66 s), since the client hashes before the server can answer "miss", so that is a broader win than the description suggests.

Sampling /proc/stat every 250 ms: the client goes from 2 cores busy to 52 cores for a 0.75 s burst. Immediately after, the server sits on one core for 17 s walking its cache. 27 GB at ~1.5 GB/s single-threaded is about that, so the remaining time is now server-side, and there looks to be another 17 s in it if that path gets the same treatment. The link ran at about 5.9 Gbps through all of this, well under 10% of what it does, so none of it is transport-bound. Your 2.5 Gbps link would hide that phase, which is probably why it did not show up for you.

Worth noting the loader is single-threaded regardless of RPC: same box, no --rpc, model already in RAM on tmpfs, 68 GB takes 44 s at 2 cores. Detail on #25890.

Happy to share the per-core traces or screenshots if useful, and to re-run on a larger model.

[chuyqa]

After looking more at the server-side bottleneck, I think the client-side load changes in this PR may be better kept separate from a larger RPC/server rework.

The current change is self-contained:

• parallelize the read/hash work while keeping the existing synchronous RPC protocol.

The server side changes are heavier and after a few attempts I see it as requiring:

• Request/response ID mapping.
• Reworking the current shared get_socket() model into reusable per-endpoint connections / multiple sockets.
• Avoiding unnecessary staging/copies on cache hits, e.g. reading a cached tensor directly into a host-backed destination buffer instead of file -> std::vector -> tensor

This feels like a heavier/follow up PR work effort.

One question for this PR @ggerganov : should the load worker count be a new env var like GGML_RPC_LOAD_THREADS, or should it inherit the existing --threads value? The load/hash phase is fairly different from generation, so there may be value in tuning it independently, but using an existing setting would avoid adding another knob and give a better out-of-box load experience.

[jaholmesuk]

Scope split sounds right. The client side win stands on its own and the server phase is a separable problem.

On the env var question: a dedicated variable is better. Load parallelism wants as many threads as I/O allows, while --threads is tuned for inference compute, and on big machines those are very different numbers (we run inference at 0.4-0.6x cores but would want load at full width). Inheriting --threads would couple them silently.

One datapoint for the follow-up's motivation: this week we loaded a 1.34TB model (plain mmap, no RPC) on a 96-core box and the load sat at ~92% of one core for 9+ minutes with the disk mostly idle. At that scale the single threaded phase dominates time to first token regardless of transport. Happy to benchmark the server side follow-up on the same hardware class when it exists.

[ttimbrook2018]

Worked for me. Super helpful in iterating llama build changes. From my agent who did all the lift (gpt 5.6 sol):

Community QA datapoint: I manually ported the functional changes from 3c0073e6 onto our existing llama.cpp fork at 87cec54e (so this was not a clean build of PR head, for transparency).

Setup:

• GLM-5.2 Q4_K_XL, 10 GGUF shards
• 16x AMD MI50: 8 local and 8 on one RPC host
• one RPC endpoint over direct 10 GbE, MTU 9000
• mmap loading
• time measured from launcher start until /health succeeded

I ran a cache-disabled A-B-A-B comparison using the same client/server package and launcher configuration, changing only GGML_RPC_LOAD_THREADS:

Run	Threads	Launch to health
Control 1	unset	754.485 s
Enabled 1	8	537.716 s
Control 2	unset	754.562 s
Enabled 2	8	540.936 s

The serial path averaged 754.524 s and the 8-worker path averaged 539.326 s, saving 215.198 s (28.5%, or about 3m35s per restart). The controls differed by only 0.077 s, and the A-B-A-B ordering makes filesystem warming an unlikely explanation.

The unset/default path reached healthy twice, so I did not observe a startup regression. Both threaded starts also reached healthy without client or RPC errors. After each enabled run, our normal smoke checks passed: vision OCR, exact post-image arithmetic, structured tool calling, and MTP drafting/acceptance. Prompt/decode performance remained in its expected range.

I also tested a populated RPC tensor cache:

Workers	Launch to health
8	332.961 s
12	321.710 s
16	317.827 s
12 repeat	317.499 s

We retained 12 workers in production since 16 was effectively tied. Warm production restarts now average about 5m20s, versus roughly 12m35s for the earlier serial/cache-off setup. That overall reduction includes the cache benefit, while the cache-disabled 28.5% result is the cleaner measurement of this change itself.

This definitely addressed the long-load interruption that motivated us to try it. Happy to provide the raw timing or smoke-test logs if useful.

[chuyqa]

@rgerganov looking for feedback

[gopinath87607]

hi can you also check this issues too #27585

edit
btw someone should also add the support for sm tensor flag . this could have help the local ai who has the multiple system which connected via lan. currently the decode is not a problem since we have multiple speculative decode option but prefill is the only things thats hurt hope we will find easy way deal with it one day

[chuyqa]

Now that PR #18626 has merged, the mutex safe workaround in ggml-rpc.cpp shouldnt be needed anymore.

I'll need to test the existing PR's llama-loader changes, as they should still be applicable: The loader kicking off 12 ggml_backend_tensor_set workers, each running its own request->hash = fnv_hash

[chuyqa]

Retested after #18626 on master @ bf94216, with all ggml-rpc.cpp changes removed from #26291
The speed improvements are still very much applicable.

Model: Kimi-K2.7-Code UD-IQ2_M
Main host: Ryzen 9 3900X, 128 GB RAM, 2× RTX 4060 Ti 16 GB,
RPC host: Ryzen 5 9600X, 192 GB RAM, 1× RTX 4060 Ti 16 GB

Polling /models every 0.25 sec until "loaded":

Build	GGML_RPC_LOAD_THREADS	Load times (s)	Avg (s)	vs master
master @ bf94216	unset	435.743	435.743	baseline
#26291 on top of bf94216	unset	436.544	436.544	+0.18%
#26291 on top of bf94216	12	209.961, 244.274	227.118	−47.88%