ggml: use dynamic allocation for split graph inputs - #22789
Conversation
Replace fixed-size GGML_SCHED_MAX_SPLIT_INPUTS arrays with dynamically allocated buffers in the backend scheduler. This fixes crashes when loading wide MoE models (Gemma 4, Qwen MoE, Mixtral, DeepSeek) on multi-backend setups where graph splits exceed 30 input tensors. - split->inputs: dynamic array with grow-on-demand - sched->graph_inputs: dynamic array with grow-on-demand - graph_size calculation now uses actual input count instead of fixed constant
github-actions
Bot
added
the
ggml
|
Possible HIP/ROCm issue with First, thank you for working on this patch. I’m seeing two related failure modes on ROCm/HIP when changing the memory options (using fix/dynamic-split-inputs), so I wanted to report them Environment:
The default mmap path appears to work for me. When I add Example command: ROCR_VISIBLE_DEVICES=0,1 /opt/src/llama.cpp-dynamic-split-inputs/build-r9700-hip/bin/llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:Q8_K_XL --host 0.0.0.0 --port 8080 --gpu-layers all --jinja --api-key REDACT -fa on -ts 1,1 -kvu -kvo --no-mmapI also found a possibly related failure mode with --no-host. Even without --no-mmap, adding --no-host does not crash immediately, but generation becomes corrupted and the model starts repeating itself badly after some output. Current behavior matrix: default mmap path: appears to work Can anyone else confirm this? Thanks |
|
Confirming both the bug in #24132 and this PR's fix on a low-VRAM CUDA setup — the hardware class mentioned above as having no current options. Setup: Quadro T1000 4GB (Turing), CUDA backend, Linux (NixOS); b10063 + this patch (applies cleanly); Without the patch: With the patch:
Vision (image describe) and audio (8 s speech transcription) outputs are consistent with the CPU-only baseline — no quality change observed. Re the ROCm Testing was done with AI assistance (Claude). |
|
I can reproduce this assertion consistently with the official Windows Vulkan build on two different Intel integrated GPUs. The failure occurs only with partial CPU/GPU offload. CPU-only and full GPU offload both work correctly. Common setup
Command used: .\llama-server.exe `
-m .\gemma-4-E2B-it-qat-UD-Q4_K_XL.gguf `
--host 127.0.0.1 `
--port 8081 `
-c 32768 `
-t <N> `
-tb <N> `
-np 1 `
--no-ui `
--cache-prompt `
--cache-reuse 256 `
--reasoning auto `
--reasoning-budget 3072 `
-ngl <0|18|999> `
-dev Vulkan0The thread count was adjusted to match each machine's logical processor count. Everything else was identical. Result on both machines
The exact assertion is: Machine A
Results:
Crash-adjacent log: This does not present as an out-of-memory failure. The process aborts specifically on Machine B
Two Vulkan devices were detected: All tests were explicitly pinned to Results:
Crash-adjacent log: This also does not present as an out-of-memory failure. The same assertion occurs with 15,413 MiB of Vulkan device memory reported free. ObservationsBoth machines hit the same assertion in the same source file and line, using the same model and the same partial-offload setting. The machines use different Intel GPU generations and materially different driver versions, but the result is identical: This makes a hardware-specific or single-driver-specific cause less likely and points toward the shared graph-splitting path used when the model spans both CPU and GPU backends. The failure pattern appears consistent with PR #22789, which replaces the fixed I have not yet tested a Vulkan build containing PR #22789, so I cannot confirm that the patch resolves this on Vulkan. This report only confirms that the same fixed-size split-input assertion is reproducible on Vulkan across two separate Intel integrated GPUs. The current workaround is to avoid partial offload and use either Thanks, |
|
🫡
…________________________________
From: Georgi Gerganov ***@***.***>
Sent: Monday, August 3, 2026 5:03 PM
To: ggml-org/llama.cpp ***@***.***>
Cc: Peter Haughie ***@***.***>; Author ***@***.***>
Subject: Re: [ggml-org/llama.cpp] ggml: use dynamic allocation for split graph inputs (PR #22789)
Merged #22789<#22789> into master.
—
Reply to this email directly, view it on GitHub<#22789?email_source=notifications&email_token=CDLOWBPZYM6JZJD2ZWKUHJL5ICSUTA5CNFSNUABQM5UWIORPF5TWS5BNNB2WEL2JONZXKZKFOZSW45CON52GSZTJMNQXI2LPNYXTEOBYHE2TONBQGI2TTJTSMVQXG33OUZQXK5DIN5ZKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#event-28895740259>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/CDLOWBPFUOGOYDDRN45MCWL5ICSUTAVCNFSNUABFKJSXA33TNF2G64TZHM3DCMRTGU2DOOBUHNEXG43VMU5TIMZZG42DQMJSGE32C5QC>.
Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS<https://github.com/notifications/mobile/ios/CDLOWBMBGMVB7N7HQXNHHAD5ICSUTA5CNFSNUABQM5UWIORPF5TWS5BNNB2WEL2JONZXKZKFOZSW45CON52GSZTJMNQXI2LPNYXTEOBYHE2TONBQGI2TTJTSMVQXG33OUZQXK5DIN5ZKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android<https://github.com/notifications/mobile/android/CDLOWBOGKW6ZH2SGSNU6WBL5ICSUTA5CNFSNUABQM5UWIORPF5TWS5BNNB2WEL2JONZXKZKFOZSW45CON52GSZTJMNQXI2LPNYXTEOBYHE2TONBQGI2TTJTSMVQXG33OUZQXK5DIN5ZKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today!
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
|
Just wanted to report: The commit fixes loading DSv4 Flash + DSpark draft model on dual RTX 6000 Pro Max-Q. No manual raising of GGML_SCHED_MAX_SPLIT_INPUTS needed anymore. Thanks a lot! |
* ggml: use dynamic allocation for split graph inputs Replace fixed-size GGML_SCHED_MAX_SPLIT_INPUTS arrays with dynamically allocated buffers in the backend scheduler. This fixes crashes when loading wide MoE models (Gemma 4, Qwen MoE, Mixtral, DeepSeek) on multi-backend setups where graph splits exceed 30 input tensors. - split->inputs: dynamic array with grow-on-demand - sched->graph_inputs: dynamic array with grow-on-demand - graph_size calculation now uses actual input count instead of fixed constant * cont : clean-up --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
ggml-org#22789 replaced the fixed 30-entry split input array with a growable one and, in the same edit, changed the split-cutting heuristic from the constant to split->inputs_capacity: - if (split->n_inputs == GGML_SCHED_MAX_SPLIT_INPUTS) { + if (split->n_inputs >= split->inputs_capacity) { inputs_capacity starts at GGML_SCHED_MAX_SPLIT_INPUTS but doubles on demand and is never reset for the life of the sched, so once a split slot grows, the scheduler stops cutting there and the cut point ratchets up for every later graph build. Longer splits mean every cross-backend input copy is materialised at the split's start and stays live to its last use inside it, which raises the peak the compute-buffer allocator has to cover - n_copies times over under pipeline parallelism. Only multi-backend configurations can reach this. Keep the growable array, which is what fixes the original >30-input assert, and cut on the constant again as before ggml-org#22789. >= rather than == so the check keeps firing for splits that did have to grow. DeepSeek-V4-Flash UD-IQ3_XXS, gfx1151, -c 400000 -ub 2048 -fa 1 --fit off: Vulkan0 compute buffer 4714.00 MiB and 9157 graph nodes, byte-identical to the unpatched tree, and neither run grows a split past 30 inputs. Expected - one Vulkan device plus the CPU backend cannot exercise the path on this box. The reported case is 3 devices with pipeline parallelism. test-backend-ops -o FLASH_ATTN_EXT, run alone on gfx1151: 13257/13295 on both this and the unpatched tree, with the same 38 failing cases (identical case list, all type_K=q8_0 prec=def kv_view=1). Pre-existing on the branch, not touched by this change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* ggml: use dynamic allocation for split graph inputs Replace fixed-size GGML_SCHED_MAX_SPLIT_INPUTS arrays with dynamically allocated buffers in the backend scheduler. This fixes crashes when loading wide MoE models (Gemma 4, Qwen MoE, Mixtral, DeepSeek) on multi-backend setups where graph splits exceed 30 input tensors. - split->inputs: dynamic array with grow-on-demand - sched->graph_inputs: dynamic array with grow-on-demand - graph_size calculation now uses actual input count instead of fixed constant * cont : clean-up --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* ggml: use dynamic allocation for split graph inputs Replace fixed-size GGML_SCHED_MAX_SPLIT_INPUTS arrays with dynamically allocated buffers in the backend scheduler. This fixes crashes when loading wide MoE models (Gemma 4, Qwen MoE, Mixtral, DeepSeek) on multi-backend setups where graph splits exceed 30 input tensors. - split->inputs: dynamic array with grow-on-demand - sched->graph_inputs: dynamic array with grow-on-demand - graph_size calculation now uses actual input count instead of fixed constant * cont : clean-up --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
74 upstream commits, 14 in our paths. Two textual conflicts, both expected, plus ONE silent-drift resolution that did NOT conflict -- which is the whole reason upstream-check.sh warns about auto-merges. ☠☠ fattn.cpp -- THE ONE THAT AUTO-MERGED CLEAN (upstream 66fa168 / ggml-org#25874) ggml-org#25874 extends oneDNN SDPA to non-F16 KV (Q4_0..Q8_0) and inserts its gate ABOVE the MKL gate -- which our 0801 sync had already deferred to MMA for exactly this reason. Its envelope (Q->ne[1] >= 32, K->ne[1] >= 1024, quantized KV now accepted) matches our deploy prefill exactly: q8_0 KV, ubatch 2048, any real depth. Verbatim it would have silently swapped the measured MMA kernel (2.87x served prefill at 43k) for an unmeasured one and staged the whole q8_0 KV cache to dense F16 first. ⇒ added the SAME conjunct, same wording, as the MKL gate below it: !ggml_sycl_fattn_mma_supported(dst). oneDNN keeps its full envelope everywhere MMA declines. A/B reachable both ways without a rebuild (GGML_SYCL_FATTN_MMA=0 / GGML_SYCL_FA_ONEDNN=0 -- the latter is checked inside _supported(), so it still governs the new early return). This is a precedence choice, not a revert, and it is UNMEASURED on this box. B59 is the finding that prices it. The three conflicts git DID flag in this file were cosmetic (debug kname lines + upstream re-calling get_best_fattn_kernel where we hoist it). Kept ours, added upstream's ONEDNN kname lines. concat.cpp -- upstream 6c8dcaa (ggml-org#25852) is a duplicate of our own fix Kept OURS: strict superset (same launch geometry + the GGML_SYCL_CONCAT_WG door + the o[dim] hoist + the i64 loop var). Ours deliberately omits their WARP_SIZE floor, which would make WG=1 unreachable and destroy the one-binary positive control for the 8.36x launch fix. Noted in the source so the next sync does not resolve toward upstream. Assessed, no action needed: 272700b (ggml-org#26105) iGPU classification -- NULL here, B70 is discrete and still reports TYPE_GPU. Relevant later for PVC/Max-1100 and B51. dbadb68 (ggml-org#22789) dynamic split-graph inputs -- mechanical, auto-merged beside our GGML_SCHED_HANDOFF_CENSUS. 596a579 (ggml-org#25784) DeepseekV4 MTP + DSpark -- ⚠ changes SHARED MTP plumbing (n_embd_out vs n_embd_inp in llama_context::decode, set_embeddings_layer_inp bounds, embeddings_layer_inp sized n_layer+1). We run MTP every step. NOT yet gated -- see the build/gate that follows. ☠ NOT YET BUILT, NOT YET GATED. Every absolute in f306 predates this merge.
…y (nathanw1014) ggml-org#22789 replaced the fixed 30-entry split input array with a growable one and in the same edit changed the split-cutting heuristic from the constant to split->inputs_capacity. inputs_capacity starts at GGML_SCHED_MAX_SPLIT_INPUTS but doubles on demand and is never reset, so once a split slot grows, the scheduler stops cutting there and the cut point ratchets up for every later graph build. Longer splits mean every cross-backend input copy is materialised at the split's start and stays live to its last use inside it, which raises the peak the compute-buffer allocator has to cover - n_copies times over under pipeline parallelism. Keep the growable array (fixes the original >30-input assert), and cut on the constant again. Cherry-picked from nathanw1014/llama.cpp strix-halo-vulkan (76ad2ba).
* ggml: use dynamic allocation for split graph inputs Replace fixed-size GGML_SCHED_MAX_SPLIT_INPUTS arrays with dynamically allocated buffers in the backend scheduler. This fixes crashes when loading wide MoE models (Gemma 4, Qwen MoE, Mixtral, DeepSeek) on multi-backend setups where graph splits exceed 30 input tensors. - split->inputs: dynamic array with grow-on-demand - sched->graph_inputs: dynamic array with grow-on-demand - graph_size calculation now uses actual input count instead of fixed constant * cont : clean-up --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* ggml: use dynamic allocation for split graph inputs Replace fixed-size GGML_SCHED_MAX_SPLIT_INPUTS arrays with dynamically allocated buffers in the backend scheduler. This fixes crashes when loading wide MoE models (Gemma 4, Qwen MoE, Mixtral, DeepSeek) on multi-backend setups where graph splits exceed 30 input tensors. - split->inputs: dynamic array with grow-on-demand - sched->graph_inputs: dynamic array with grow-on-demand - graph_size calculation now uses actual input count instead of fixed constant * cont : clean-up --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
6 participants
Overview
Replace fixed-size GGML_SCHED_MAX_SPLIT_INPUTS arrays with dynamically allocated buffers in the backend scheduler. This fixes crashes when loading wide MoE models (Gemma 4, Qwen MoE, Mixtral, DeepSeek) on multi-backend setups where graph splits exceed 30 input tensors. Done with the lightest touch possible to the fewest files to fix the issue.
Additional information
Requirements