rpc : implement event and async backend APIs - #18626
Conversation
github-actions
Bot
added
the
ggml
I think we have the following analogy between CUDA and RPC:
If this is correct, then I believe the |
According to this comment from @slaren, The server side is a bit tricky as we may create multiple |
ggerganov
left a comment
ggerganov
left a comment
There was a problem hiding this comment.
AFAIU since all communication for a given endpoint is funneled through the same socket, then effectively the RPC backends for a given endpoint on the client are all associated with a single shared "stream" (i.e. the network socket). So if this is correct, then my guess would be that event_wait() would have to be a noop. Doing dispatcher->event_synchronize() would not make it incorrect, though probably it's not needed.
From reading how the CUDA backend and toolkit work, I think that event_wait() only makes sense with parallel streams. It is effectively a mechanism to tell one stream to wait for another stream. So in the context of the RPC backend, it should be a noop.
Apart from that, I think the approach is OK.
|
I think it will be useful to check that extra heap allocation of each request and dispatching logic doesn't hurt TG speed when pipeline parallelism is not needed/disabled |
|
Seems to work fine, together with #16753 I'm getting nice speed improvement. |
|
Apologize. Know everyone is busy but any updates on this implementation? Thank you! |
I'm working on the implementation of |
|
thanks for all the good work! @rgerganov Let me know when testing is required, I have a real franken-setup here with 2xB50 and a 6900XT (via RPC) - can assist in testing once the async work is ready. |
|
maybe a stupid question, but would this be an enabler for tensor parallel via RPC as well? |
|
@rgerganov -- No big comments from me. You and I converged on the same design, so implicit +1 from me there :) I see the same speedup as on my branch with my hardware, so functionally it seems good. |
|
Here are some performance results coming from Nvidia 4xA100 HF space:
Adding a second RPC server running on localhost gives us 30-70% PP improvement, depending on the model. I expect these numbers to be very similar with servers running on different hosts, using RDMA transport. Adding a third RPC server makes things worse, I guess due to the communication overhead between the nodes. Nevertheless, I think these results justify the complexity being added in the RPC backend and there is still room for improvement (e.g. implementing One mistake I made when I've been testing this in the past was the assumption that I can get performance improvements with two RPC servers using the same physical GPU accelerator. This is simply not the case (at least with CUDA). |
|
I recently upgraded to four AMD V620. I also see negative speedup with more than two RPC GPUs. They're all on the same host, so it's kind of a bummer. |
|
@rgerganov Can you do a test with |
These are the results:
|
|
I see - not much difference. Good to merge? |
I am not happy with the |
rgerganov
added
the
merge ready
* rpc : implement event and async backend APIs * cache responses from RPC_CMD_GET_ALLOC_SIZE
34 commits, 9 conflicted files. Resolutions worth knowing: - ggml-rpc: KEPT THE FORK'S RPC LAYER, deferred upstream ggml-org#18626. Upstream replaced the client socket model with an rpc_dispatcher + message queue and bumped the wire to 6.0.0. The fork has 4035 lines layered on the old model (full-duplex lanes, peer push, proto-v5 graph cache, imatrix sqsum, bf16 wire); porting them is a project, not a merge step, so ggml-rpc.{h,cpp} stay on the fork's side at proto 5.1.3. Client and servers deploy together, so the fabric stays self-consistent. - Vulkan lightning_indexer: both sides had a complete implementation and BOTH SURVIVED the automerge (duplicate structs, two dispatch functions, two switch cases, two shader registrations -- the documented "compiles clean with both copies" seam). Kept upstream's shader and its k-type array, which supports 9 K types instead of f32/f16, and layered the fork's measured fast_f16 path back on top: it keeps its own push-constant struct (renamed vk_op_lightning_indexer_fast_push_constants), its own dispatch branch, and the -inf-masked-row skip that fixed the DSV4 cold request (942c45c). - qwen4exp (new upstream arch): read hparams.n_ff_exp as a plain field. It is a per-layer accessor here; raw field is n_ff_exp_impl. The recurring seam. - server-context: n_ctx_slot became a method (--kv-unified-per-slot). - speculative: DFlash2's !is_dflash2 masking kept alongside the fork's Laguna causal-attention probe; DFlash2 is never "laguna" so it keeps non-causal. - llama-model: kept upstream's M-RoPE branch ahead of the fork's DFLASH dsv4-backbone rope selection. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JwEmGb6MgaXhJVy46bKNLk
The planner shipped one number for splitting a model that already fits on one node: 0.92x, a loss, always. That is still correct for a llama.cpp whose RPC backend predates ggml-org/llama.cpp#18626 ("rpc: implement event and async backend APIs", merged 2026-08-26), which is what the fork currently builds. Without that commit the RPC backend advertises neither async nor events, ggml_backend_sched refuses to pipeline across RPC devices, and the two halves run strictly one after the other. With it the answer stops being a constant, because what overlaps is prefill. Measured end to end on two Sparks, Qwen3-27B Q4_K_XL, same binary in both arms so the only variable is whether the model is split: prompt tokens | c=1 c=4 c=8 128 | 0.94x 0.95x 0.95x 256 | 0.98x 1.00x 1.00x <- break-even 512 | 0.96x 1.05x 1.07x 1024 | 1.02x 1.12x 1.17x 2048 | 1.07x 1.23x 1.29x 4096 | 1.11x 1.35x 1.45x Decode is 0.93-0.98x throughout and cannot be otherwise: a layer split moves the same weight bytes per token, so the entire gain is prefill and the entire question is prompt length. Below ~256 tokens splitting costs 2-6%; above ~1024 it wins, growing with both prompt length and concurrency. So it is a loss for chat-shaped traffic and a win for prompt-heavy work, which is the opposite of a single verdict. `layer_split_speedup()` defaults to `async_rpc=False` and returns the conservative 0.92x, so nothing promises users a speedup their build cannot deliver. With `async_rpc=True` and no prompt length it returns None rather than guessing, and it snaps down to the nearest measured row rather than interpolating: these are six measured points, not a fitted curve. The old constant is kept and still used, so no existing caller changes behaviour.
7 participants
This patch implements the event and async APIs from the backend interface. The intent is to enable pipeline parallelism with the RPC backend.
Here is a summary of the changes:
rpc_dispatcherwhich allows sending commands both synchronously and asynchronously to an RPC endpoint. There is one dispatcher per RPC endpoint and every dispatcher is using a background thread for its network communication.rpc_dispatcherwithshared_ptrs to avoid making copies of the input data.RPC_CMD_FREE_BUFFER,RPC_CMD_BUFFER_CLEAR,RPC_CMD_INIT_TENSORto not send any response which simplifies the logic in the dispatcher. Hence the protocol version bump (3.7.0).TODO:
event_synchronize()andevent_wait()equivalent here?cpy_tensor_async(). This is a hard one as RPC servers need to to talk to each other (see PR rpc : copy tensors across servers #8032). Pipeline parallelism relies on bothgraph_compute()andcpy_tensor_async()being async and I am not sure if we are going to see improvements only with the former.@ggerganov I will appreciate some early feedback on this.