RPC: Pipeline parallelism

[amangupta-tether]

Overview

https://app.asana.com/0/0/1217186633202282/f

Tested on 2x DGX Sparks with -ub 512 -p 2048 on llama-bench. 47% better throughput with PP compared to no layer parallelism

model	size	params	backend	ngl	split	pipeline	fa	dev	ts	dio	test	t/s
deepseek4 MXFP4 MoE	145.63 GiB	284.33 B	CUDA,RPC	999	layer	0	1	RPC0/RPC1	1.00/1.00	1	pp2048	398.47 +/- 13.87
deepseek4 MXFP4 MoE	145.63 GiB	284.33 B	CUDA,RPC	999	layer	1	1	RPC0/RPC1	1.00/1.00	1	pp2048	587.11 +/- 5.76
deepseek4 MXFP4 MoE	145.63 GiB	284.33 B	CUDA,RPC	999	tensor	0	1	RPC0/RPC1	1.00/1.00	1	pp2048	500.00 +/- 12.15
deepseek4 MXFP4 MoE	145.63 GiB	284.33 B	CUDA,RPC	999	layer	0	1	RPC0/RPC1	1.00/1.00	1	tg128	14.98 +/- 0.09
deepseek4 MXFP4 MoE	145.63 GiB	284.33 B	CUDA,RPC	999	layer	1	1	RPC0/RPC1	1.00/1.00	1	tg128	14.48 +/- 0.14
deepseek4 MXFP4 MoE	145.63 GiB	284.33 B	CUDA,RPC	999	tensor	0	1	RPC0/RPC1	1.00/1.00	1	tg128	18.17 +/- 0.60

Additional information

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure:

[gianni-cor]

Please add multi-node support for GGML_RPC_DIRECT_COPY.

Layer pipelines with more than two stages need the middle server to communicate with both neighbours: A↔B and B↔C. The current server state holds only one peer per device, so after A↔B is initialized, B cannot create the B↔C connection. That prevents direct server-to-server activation copies in a multi-stage pipeline.

Please store direct-copy peers per remote endpoint/session, with clear lifetime management, and add a three-endpoint test that verifies A→B and B→C direct copies in the same run.

[amangupta-tether]

I have dropped GGML_RPC_DIRECT_COPY feature entirely as it was providing no measurable benefit, we can revisit at a later stage if required.

[gianni-cor]

@amangupta-tether could you take a look at the Windows CI failure here?

windows (x64-openblas) is failing on ctest #27 test-thread-safety with 0xC0000374 (STATUS_HEAP_CORRUPTION):

27: All threads finished without errors.
26/38 Test #27: test-thread-safety ....Exit code 0xc0000374 ***Exception: 50.12 sec
The following tests FAILED:
	 27 - test-thread-safety (Exit code 0xc0000374)

job log

Note the ordering: the test prints "All threads finished without errors" and only then aborts, so the heap corruption is being detected during teardown rather than in the test body. The two sibling matrix jobs (x64-vulkan, x64-cpu-static) show The operation was canceled — that's just fail-fast collateral from this one, not separate failures.

Where I'd start looking: this destructor joins worker, and the queue instances are kept in a function-local static std::unordered_map<std::string, std::weak_ptr<rpc_command_queue>> in get_command_queue() below. On Windows, a thread joined from static-destruction / DLL-unload order is a very common source of exactly this signature. Worth ruling out first — I haven't confirmed it from a stack trace, so it's a starting point rather than a diagnosis.

Two data points on scope:

• RPC: Tensor Parallel  #201 (the parent branch) passes this job, and unrelated PRs fix: surface a failed K-shift instead of decoding over stale K #213/QVAC-24112 fit: budget against real memory availability #214 pass it too, so it looks introduced here rather than pre-existing or flaky.
• RPC: More TP improvments #210 is stacked on this branch and fails the same job the same way, so fixing it here should clear that one as well.

For what it's worth, the other red checks on this PR are not code: cpu-x64-high-perf ran all suites green (53/53 and 56/56) and then died with exit code 127 / '/opt/actions-runner-3/externals/node24/bin/node' ... No such file or directory, which is self-hosted runner breakage.

[amangupta-tether]

@gianni-cor please see

The failure is real, but the proposed RPC diagnosis is not supported.

• The same queue/destructor code passed x64-openblas twice immediately beforehand.
• Since that passing head, only CUDA kernels and RPC server tensor deserialization changed. Neither runs in test-thread-safety.
• The test never adds an RPC server, so no rpc_command_queue worker is created.
• The static map stores weak_ptrs, so its destruction cannot trigger queue destruction or worker.join().

The post-test crash indicates teardown-time corruption, but not RPC teardown specifically. Treat this as a likely Windows/OpenBLAS flake and rerun the failed job; no PR 202 code change is justified from this evidence
alone.