rpc: avoid serializing buffers from other servers - #26500
Conversation
github-actions
Bot
added
testing
6e24f65 to
f0c483c
Compare
|
Tested this on 2x DGX Spark (GB10, sm_121, arm64, CUDA 13.0, driver 580.159.03) — it fixes the failure here. Before the patch, both master ( Only the remote worker dies; the loopback worker on the client machine logs nothing unusual, and a single-worker setup never reproduces it — which lines up with the ownership problem described here. With
The 144 GiB quant does not fit in a single node's 121 GiB of unified memory, so more than one RPC worker is the only way to run it at all. That is the case this patch unblocks for me. Beyond Same signature is reported in #26820; I left the reproduction details there. |
|
I had the same issue ( ggml-org/llama.cpp:master with this PR fixes the issue for DeekSeek V4 Flash and the unslothai/llama.cpp:kimi-k3-text-base with manually merging this PR does also fix running Kimi K3 on multiple CPU-only RPC nodes in my tests. |
|
Confirming this also fixes GLM-5.2 multi-node RPC crashes (#26583), deterministic invalid data ptr at graph node 30 before, fully working after, on a mixed SM120 head + SM121 workers cluster with RDMA. MTP-over-RPC works too. Nice fix, thanks |
|
I have the same problem with DS4 REAP with RTX 4060Ti + 2x BC250 over RPC all on Vulkan. @hmirin maybe resolving the branch conflict could help get this PR merged? Thanks for the fix :) |
f0c483c to
e144634
Compare
|
Updated to keep up with the main branch. |
Only include remote buffer pointers when the buffer belongs to the RPC dispatcher receiving the graph. Add a two-server regression test for cross-server tensor serialization. Assisted-by: Codex
e144634 to
41d0d5f
Compare
ggerganov
added
the
merge ready
[no ci]
68 upstream commits, 9 conflicted files. Brings qwen4exp fixes needed for Qwen3.8-Flash-Next: recurrent state rollback (ggml-org#28123), seq_cp/block position keying/mtmd input (ggml-org#27941), indexer head slicing (ggml-org#28023), and the central n_layer_nextn load (ggml-org#28159). Resolutions: - ggml-rpc.cpp: resolved entirely to the fork side again. Upstream's rpc_dispatcher/proto-6.0 port stays deferred; the fork keeps 5.1.3. Note that upstream ggml-org#26500 now implements the same foreign-endpoint skip in serialize_graph that the fork carries, so that one is a candidate to drop when the dispatcher port finally happens. - llama-graph.cpp: adopted upstream's fused ggml_swiglu_clamp for both the dense and MoE clamp paths and kept the fork's LLM_ARCH_GLM5NEXT in the condition. The fused kernel is min(gate,limit) / clamp(up,+-limit) / swiglu_split, i.e. bit-identical to the three-op sequence it replaces, and Vulkan implements it, so the boards get it too. Costs the ffn_*_clamped cb() trace names, same tradeoff upstream took for DSV4. - speculative.cpp: upstream ggml-org#27310 folded the DFlash encoder into the injection decode and deleted features_buf. The fork's non-finite feature scrub moves onto batch_inject.embd in place; the last_tap_nextn tap automerged. - dflash.cpp: kept the fork's Laguna kv_inp norm and deferred gated o_proj, layered on upstream's newly-populated wk_s/wv_s/wo_s scale arguments (previously NULL). The gated path latches wo_s alongside wo. - nemotron-h.cpp: dropped the now-duplicate NEXTN_PREDICT_LAYERS get_key, kept the fork's stricter assert and n_layer_nextn_per_head derivation. - llama-model-loader.cpp: kept the fork's rpc_preloaded skip ahead of upstream's use_mmap || lazy.has(cur) predicate. - tests: kept both sets of flash-attn cases; ported the fork's GLM5NEXT test_dsa_kpool/test_mtp onto upstream's int-verbosity logging convention. Windows CUDA build green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01U3H5motr51eTWujztSXykc
6 participants
Overview
The bug caused the following error:
Additional Information
invalid data ptrRPC error.Requirements