Implement events and async processing for RPC backend, enabling pipeline parallel input processing

[sredman]

Overview

Implement events and async processing for RPC backend, enabling pipeline parallel input processing.

Note this does NOT change the single-threaded design of rpc-server, meaning if you have one rpc-server handling multiple GPUs, those GPUs will effectively not be pipelined. Easy workaround: Use multiple rpc-server instances, one per GPU.

I think I've laid out the code such that if the rpc-server were handling multiple devices in parallel, it would "just work", but that is necessarily uncertain since it is not implemented.

Additional information

Results:

Using two 4-thread CPU workers:

~/src/llama.cpp/build-x64-linux-gcc-release/bin/llama-bench -rpc localhost:9090,localhost:9091 -m /opt/localai-models/Phi-4-mini-instruct-Q4_K_L.gguf --split-mode layer --tensor-split 1/1 -ngl 99 --ubatch-size 256 --batch-size 2048 --device RPC0,RPC0/RPC1 -p 4096 -n 0 -r 3

model	size	params	backend	ngl	n_ubatch	dev	ts	test	t/s
phi3 3B Q4_K - Medium	2.45 GiB	3.84 B	RPC	99	256	RPC0	1.00/1.00	pp4096	37.96 ± 0.14
phi3 3B Q4_K - Medium	2.45 GiB	3.84 B	RPC	99	256	RPC0/RPC1	1.00/1.00	pp4096	59.84 ± 0.74

Baseline: same CPU with 8 threads:

~/src/llama.cpp/build-x64-linux-gcc-release/bin/llama-bench -rpc localhost:9090,localhost:9091 -m /opt/localai-models/Phi-4-mini-instruct-Q4_K_L.gguf --split-mode layer -ngl 99 --ubatch-size 256 --batch-size 2048 -p 4096 -n 0 -r 3 --threads 8

model	size	params	backend	threads	n_ubatch	test	t/s
phi3 3B Q4_K - Medium	2.45 GiB	3.84 B	CPU	8	256	pp4096	89.69 ± 1.17

Using 2x AMD v320 (MI25) workers (with Vulkan backend):

~/src/llama.cpp/build-x64-linux-gcc-vulkan-release/bin/llama-bench -rpc localhost:9090,localhost:9091 -m /opt/localai-models/microsoft_Phi-4-mini-instruct-Q4_K_M.gguf --split-mode layer --tensor-split 1/1 -ngl 99 --ubatch-size 256 --batch-size 2048 --device RPC0,RPC0/RPC1 -p 4096 -n 0 -r 3

model	size	params	backend	ngl	n_ubatch	dev	ts	test	t/s
phi3 3B Q4_K - Medium	2.31 GiB	3.84 B	Vulkan,RPC	99	256	RPC0	1.00/1.00	pp4096	527.89 ± 1.51
phi3 3B Q4_K - Medium	2.31 GiB	3.84 B	Vulkan,RPC	99	256	RPC0/RPC1	1.00/1.00	pp4096	904.55 ± 2.96

Baseline: direct Vulkan without RPC:

~/src/llama.cpp/build-x64-linux-gcc-vulkan-release/bin/llama-bench -m /opt/localai-models/microsoft_Phi-4-mini-instruct-Q4_K_M.gguf --split-mode layer --tensor-split 1/1 -ngl 99 --ubatch-size 256 --batch-size 2048 --device Vulkan2,Vulkan2/Vulkan3 -p 4096 -n 0 -r 3

model	size	params	backend	ngl	n_ubatch	dev	ts	test	t/s
phi3 3B Q4_K - Medium	2.31 GiB	3.84 B	Vulkan	99	256	Vulkan2	1.00/1.00	pp4096	533.37 ± 0.55
phi3 3B Q4_K - Medium	2.31 GiB	3.84 B	Vulkan	99	256	Vulkan2/Vulkan3	1.00/1.00	pp4096	860.53 ± 0.10

Not shown: master build comparison, since those very boringly show the same t/s with two devices as with one, since master branch does not support pipeline parallelism in RPC!

This also enables pipeline parallelism in hybrid scenarios with a mix of local devices and RPC devices.

Tested in my lab with a mix of local devices plus RPC CPU and GPU workers. Previously I was getting about 5 t/s with GLM5.1, with pipeline parallelism I get about 20t/s, with four CPU workers holding the bulk of the sparse layers. I need more GPUs!

Requirements

• I have read and agree with the contributing guidelines - Yes
• AI usage disclosure: YES - Used Github Copilot with Claude Opus 4.6 to research appropriate locations for code changes, implement the design, and debug various issues which were preventing pipeline parallel dispatch. The engineering work was human-derived and I am prepared to discuss it.

[sredman]

@ggerganov - Sorry to ping you directly. You seem to span ownership between RPC and pipeline parallelism. Would you be able to review this change?

[sredman]

@ggerganov - Gentle reminder. If you think this is a bad idea overall, I'm happy to hear that too so I can stop thinking about it 🙂

[mndodd]

FWIW, I tested this a while back and it seemed to work, though I don't have any rigorous results.

[am17an]

I'm trying to add TP support via RDMA and this PR would be essential in doing that

[sredman]

@am17an , if you're familiar with this code area, could you give this PR a review? You may not be able to approve, but you could help find any dumb stuff before ggerganov spends time on it 🙂

[sredman]

I'd especially be interested to hear if you think this implementation can stretch to cover RDMA and TP. I am being a bit lazy in this implementation and relying on TCP and the rpc-server for synchronization, but I hope/planned that switching those bits out should be a relatively self-contained change.

[sredman]

This conflicts with #25912 -- The merge is not difficult. We both touch the rpc_cmd enum, and there's a new call to ctx->sock which I need to change to my ctx->cmd_queue->submit_rpc_sync( syntax. That said, it compiles but I haven't tested it. I will at least do a simple test before pushing.

[rgerganov]

@sredman sorry for not being able to look into this for so long; I also have a long-standing PR which implements events and async for RPC: #18626

I will try to find some time to look into your PR soon

[rgerganov]

ok, so both this PR and PR #18626 implement the same design which is one thread per endpoint which dispatches commands from a queue. I am obviously biased but I find my implementation much better because it handles all commands in a generic way, without code duplication.

One critical thing that I was missing was having a cache for the RPC_CMD_GET_ALLOC_SIZE responses and waiting for this was basically killing the pipeline overlap. Once I added this, I started getting the same performance improvements with Phi-4-mini-instruct-Q4_K_M.gguf.

@sredman thanks a lot for your work, it really helped my to understand the performance issue with my PR. I'd appreciate if you can review and test PR #18626 so we can finally implement PP support in the RPC backend.

[sredman]

@rgerganov , I am biased to your implementation too. You are much more familiar with working within llama.cpp than I am, not least you hold the mechanical understanding of how all the ggml* types work together, which I do not ;)

I'm glad I was able to provide the missing puzzle piece for your branch. We can thank Claude too. I provided the instruction, but Claude did the hard work of grinding through the logs to figure out where the blocking was 🙂 .

I'll review your PR this week.