llama : disable graph reuse with pipeline parallelism

[ggerganov]

The following repro demonstrates the issue:

make -j && ./bin/llama-perplexity -hf ggml-org/Qwen3-0.6B-GGUF -f wiki.test.raw --chunks 16 -ngl 99 -ub 512 -b 2048

PPL = 2382.4719 +/- 246.20903

The problem seems to occur when 2 consecutive pp ubatches both output logits, which occurs in perplexity runs with the above parameters: 4 ubatches of size 512, the second 2 ubtaches output logits. I think the graph reuse logic somehow conflicts with the scheduler's logic for tracking the current copy:

llama.cpp/ggml/src/ggml-backend.cpp

Lines 1775 to 1780 in 557fe2d

	GGML_ASSERT(!sched->is_alloc);
	sched->cur_copy = sched->next_copy;
	sched->next_copy = (sched->next_copy + 1) % sched->n_copies;
	ggml_backend_sched_split_graph(sched, graph);

For now disabling graph reuse when pipeline parallelism is active to workaround. Proper investigation is necessary.

---

Additionally, after #17795, only disabling the graph reuse is not enough to fix the issue. So for now also reverting that change.

Note that both commits in this PR are needed. Neither one fixes the issue alone.

cc @aendk @gaugarg-nv

[ORippler]

Let's take the time to understand what's going on here

[Superbobo75]

hi,

Starting from version c8323 of llama.cpp, where the graph parameter was disabled, the inference speed of the model Unsloth/Qwen3.5-35B-A3B-Q5_K_M.gguf dropped significantly. On my specific hardware setup—consisting of 2x RTX 5060 Ti (16 GB VRAM) and 64 GB DDR4 RAM—the performance fell from approximately 85 tokens per second (t/s) to around 50 t/s.

Due to this substantial regression, I am currently sticking with the last known functional version prior to this change. I have not yet tested other models to see if this issue is isolated to this specific architecture or configuration.