layer-split pipeline parallel: run a contiguous layer range per node, bit-exact

[danielhanchen]

Splits a model's decoder stack across two machines so a GGUF larger than one node's memory can be served, and runs each contiguous layer range in its own llama_context. Built for two DGX Sparks cabled over ConnectX-7, but nothing here is Spark specific.

This is marked draft because prefill works and batched decode does not. Details and numbers below, including the part that failed.

Approach

Each node loads only its own layer range. llama-model-loader skips out of range weights at load time, so a node never allocates memory for layers it will not run. That skip is generic across architectures rather than per model, with qwen3 and qwen3moe (the 235B architecture) wired through.

Stage A runs layers [0, k) and emits hidden states. Stage B accepts them through batch.embd, runs [k, n_layer) and produces logits. tools/pp-stage serves one stage over a minimal binary TCP protocol.

Also included: the RPC backend gains async submit plus events, so ggml_backend_sched can pipeline across RPC devices instead of blocking on each transfer.

Numerical equivalence

The split is bit exact, not approximately equal. tests/test-layer-split.cpp checks the whole chain against the unsplit model on the same machine:

stage A vs t_layer_inp[32] of the full model: max|dh|=0 rmse=0
stage B run-to-run max|dlogit| = 0  (deterministic)
reference argmax :  11751 " Paris"  logit=17.59603
split     argmax :  11751 " Paris"  logit=17.59603
max |dlogit| = 0.000000   rmse = 0.000000
PASS: layer split is numerically equivalent

Qwen3-27B Q4_K_XL, 64 layers, split at layer 32.

Cache correctness was checked separately on hardware, since a split that quietly breaks the KV or prefix cache would still look right on a single prompt. A batched group of 8 with caching on is token identical to the unsplit model, a prefix cache hit (300 of 332 tokens reused, and reported as reused so the cache is provably still in play) equals cold recompute, and a mixed batch of lengths 128/512/61/300 equals each sequence run alone.

Measured on two nodes, Qwen3-27B

Model buffers land at 7683.83 MiB on stage A against 8716.68 MiB on stage B, so the pair holds one copy of the model, not two.

batch depth	prefill	decode against a properly batched single node
1	0.97x	1.00x
2	1.26x	0.73x
4	1.52x	0.51x
8	1.69x	0.37x

Prefill scales with batch depth. Decode does not, and gets worse. Reporting the second number rather than shipping the first one alone.

Why decode is slow, and what would fix it

Not the weights. An earlier version reloaded weights per sequence; that is fixed, and mean_group is now identical on the split and the control at every depth, so each stage moves its weights once per batch.

The remaining deficit is that the two stages never overlap. At depth 8, TPOT is 362 ms split against 132 ms single, and per stage utilisation sampled inside the split arm only is 72% on A and 44% on B. That sums to 116%, not 200%: the stages are taking turns and the gap is pipeline bubble.

The fix is two or more microbatches in flight (1F1B) in the driver. The stage engine already batches correctly and needs no change for it. I did not get there, which is why this is draft.

One caveat for anyone measuring this: the batching fix moved the control from 40.3 to 60.7 tok/s at depth 8, after which it agrees with llama-batched-bench within 0.5%. Measured against the old denominator the split would have posted a fraudulent 1.5x. If you benchmark this, check your single node control is actually batching first.

Requests within a stage are sorted by seq_id rather than arrival order, because the unsplit single node otherwise produced three different greedy streams across three identical runs. Note that argmax on knife edge logits can legitimately differ between serial and batched execution in llama.cpp CUDA regardless of splitting, so use high margin prompts for strict comparisons.

Also fixed

The split guard rejected vocab only models, which broke llama-tokenize in this build whether or not anything was being split.

[danielhanchen]

Pushed three fixes from a pre-merge regression audit. None are about layer splitting; all three are about not changing behaviour for people who are not using it.

1. RPC pipelining was opt-out, and it should not have been

This was the real problem with the PR. rpc_pp_enabled() defaulted to true (disabled only by GGML_RPC_NO_PIPELINE) and it advertises async and events in props->caps. Pre-PR both were hard false, and llama-context.cpp uses exactly those to veto pipeline parallelism. So the PR silently enabled n_copies = 4 and graph_reuse_disable for every existing multi-RPC-device user, none of whom are splitting layers.

Measured on two loopback rpc-server instances (zero network latency, so a best case for the change), same binary, Qwen3.5-4B, --device RPC0,RPC1 -sm layer -ngl 99:

	compute buffers per device	sched copies	tok/s
PR default	~1100 MiB	4	39.7 / 30.8
pipelining off (= pre-PR)	320 MiB	1	58.5 / 55.4

About 3.5x the compute-buffer memory and 35 to 45% less throughput, by default. Worse on a real LAN, since disabling graph reuse means re-serializing and re-sending the whole graph every token.

The reason it does not pay off is that the events are not really events. ggml_backend_rpc_event_record is a no-op and event_synchronize calls rpc_barrier, which drains the entire endpoint command stream, so the per-split synchronize degenerates into a full device drain. Safe, but it removes the overlap that n_copies = 4 is paying for.

Now opt-in via GGML_RPC_PIPELINE. That also restores ggml_backend_rpc_synchronize to the no-op it used to be, which matters independently: the new one issues a GET_ALIGNMENT round trip under RPC_STATUS_ASSERT, so a transient socket failure during a synchronize would GGML_ABORT the process, where previously it could not fail at all.

2. qwen35.cpp no longer relaxes output_norm

It was passing TENSOR_NOT_REQUIRED for output_norm, which is both redundant and harmful. Redundant because llama_model_loader::create_tensor() already ORs in TENSOR_NOT_REQUIRED | TENSOR_SKIP for a non-final stage, which is exactly why qwen3.cpp and qwen3moe.cpp still pass 0 and work correctly. Harmful because it also let a qwen3.5 GGUF genuinely missing output_norm.weight load with a null tensor, and build_norm() treats a null weight as "no weight" and returns an unweighted RMS norm. That converts a clear load-time error into silently wrong output, for unsplit users.

3. Two asserts restored

llm_graph_input_out_ids::set_input and llm_graph_input_attn_no_cache::set_input had GGML_ASSERT(out_ids) and GGML_ASSERT(self_kq_mask) replaced by the new liveness check. That collapsed two different cases. A null tensor is a graph-construction bug and must still abort loudly; a present-but-unallocated tensor is the new legitimate case, where a non-final stage registers an input it never uses and sched leaves it unbacked. Both asserts are back, with the liveness check after them, so only the second case is skipped.

Confirmed unaffected

• llama.h and all of ggml/include/ are untouched, so there is no C API ABI change. The new il_load_beg / il_load_end and cparams fields are in internal headers under src/.
• The load-time weight skip is a genuine no-op when not splitting: both hparams default to 0, the guard is il_load_beg > 0 || il_load_end > 0, and nothing is touched otherwise. Checked for 1-layer and vocab-only models too.
• qwen3moe.cpp is a pure no-op when unsplit.
• Clean builds of base and head with -Wall -Wextra -Wpedantic: zero new warnings.
• Base vs head, identical args, greedy --temp 0 --seed 1: byte-identical generated text.

After the fixes, test-layer-split still reports max |dlogit| = 0.000000 rmse = 0.000000 on Qwen3-27B split at layer 32, and llama-tokenize still works on the vocab-only path.

Still draft: batched decode remains 0.37x, as described in the PR body.

[danielhanchen]

Closing this. It is superseded by an already-merged upstream commit, and measurement says the upstream path is better on every axis I can measure.

What changed

The premise of this PR was that llama.cpp could not split a model across two machines usefully, so core needed modifying. That premise was wrong, and the reason is #18626, rpc : implement event and async backend APIs, merged 2026-08-26. Pipeline parallelism in ggml requires the backend to advertise async operations and events, which the RPC backend historically did not. ggml-org#18626 implements them properly, with a real per-endpoint dispatcher on a background thread.

This fork does not have it. Verified directly: fork RPC_PROTO 5.1 with zero rpc_dispatcher occurrences and .async/.events hardcoded false, against upstream 6.0 with 25 occurrences and both unconditionally true. So every earlier comparison in this PR was measured against an RPC backend predating upstream's async work, which made a custom path look necessary when it was not.

The measurement

Cherry-picking d0132a68 (ggml-org#18626) onto the same base, so the RPC implementation is the only variable. Same model, same CUDA kernels, control re-run on the same binary. Qwen3-27B Q4_K_XL, two DGX Sparks over ConnectX-7.

End to end, for a real workload rather than a component ratio:

workload	one Spark	RPC + ggml-org#18626	this PR
npp 512, depth 8	21.77 s	20.68 s (1.05x faster)	48.0 s (2.18x slower)
npp 2048, depth 8	38.22 s	29.21 s (1.31x faster)	not measured

Component ratios, npp 2048: prefill 1.54x / 1.67x / 1.80x / 1.89x at depth 1 / 2 / 4 / 8, with decode at 0.94x to 0.98x throughout. This PR reached 1.68x prefill but 0.37x decode, which is why it loses badly end to end: the prefill win saves about 2 seconds and the decode loss gives back 28.

Memory also favours upstream: +10.9 / +11.3 GiB per node against this PR's +12.4 / +13.3, because this PR duplicates tok_embd and a 682 MiB host buffer on both stages.

Correctness, and the part that matters most

Run through llama-server on both configurations and diffed: greedy serial output, a prefix-cache hit against cold recompute, and a four-way mixed-length concurrent batch each compared against the same request run alone. Every generated string is byte-identical between the single node and the RPC split, including a knife-edge case that the unsplit node itself answers differently between serial and batched execution. Prefix reuse is real, not bypassed: 2399 ms cold against 199 ms warm on the split.

The deeper point is structural. Under RPC there is one llama_context, one KV cache and one control plane, and the split sits below all of it at tensor placement. There is no second stage that can disagree about seq_id-to-slot mapping, prefix-reuse counts or eviction. That entire class of bug exists in this PR only because this PR created two independent contexts. Defending a hazard I introduced was never going to beat not introducing it.

What was actually worth finding

• -sm row cannot load in this fork at all, and -sm tensor runs at 0.55x decode and crashes the remote server at depth 8 in rpc_server::deserialize_tensor. Corroborated upstream by open issues Eval bug: DFlash2: --split-mode tensor aborts with SPLIT_AXIS_UNKNOWN on ROCm (Qwen3.8-27B, 2x RX 7900 XTX) ggml-org/llama.cpp#27829 and Eval bug: ggml-backend-meta.cpp:541: GGML_ASSERT(src_ss[0].axis != GGML_BACKEND_SPLIT_AXIS_0) failed ggml-org/llama.cpp#24015. So tensor parallelism, which is the only axis that could raise decode, is not viable in llama.cpp today. That is worth knowing and I will report the crash separately.
• The gap that stops RPC stages from truly overlapping is upstream rpc : copy tensors across servers ggml-org/llama.cpp#8032, rpc : copy tensors across servers, open since 2025. Without server-to-server copies everything routes through the client.
• The RPC async commit in this PR reimplemented rpc : implement event and async backend APIs ggml-org/llama.cpp#18626 worse, using a barrier as a fake event. It should never have been written, and it is deleted along with the rest.

Next

The useful deliverable is not this PR, it is cherry-picking ggml-org#18626 into the fork so fork users get the faster two-node path. That will go up separately.

Nothing here is a criticism of layer splitting as an idea. The split in this PR is numerically exact and always was. It simply is not needed, because upstream already supports the same thing better through a backend that does not require touching core inference at all.