Conversation
Commit 2446419 added ggml_build_forward_expand immediately after layer normalization to enable multi-GPU graph partitioning. However, for recurrent linear attention models, this prematurely closed the graph branch before the internal state updates (convolution and SSM states) were fully constructed. This caused the graph engine to schedule state cache writes out of order or independently, leading to data races or stale reads in the convolution and SSM states. The bug manifested as intermittent generation corruption, such as alignment errors in code generation. This commit moves the ggml_build_forward_expand call to immediately after the linear attention layer computation for all recurrent branches. This ensures the entire layer computation, including internal state updates, is part of the forward graph before being expanded for partitioning, restoring the correct topological barriers. This was found and verified with a single GPU only using the synthmerge_bench. The multi GPU split is untested. Fixes: 2446419 ("models : fix graph splits (ggml-org#19866)")
|
This change would again break multi-GPU support. I am not sure that the explanation is correct. Looking at the order of the nodes when running with |
|
Hello, should I try to provide a self contained reproducer to facilitate the debugging? I tried also to only add an extra expand after linear attention as a flush synchronization and that didn't fix it. Another data point, I'm running the model on a single GPU and it only happens with vulkan backend, I couldn't reproduce on rocm. I suppose vulkan must be leveraging the expand somehow while rocm doesn't? In fact I didn't initially expect such multi gpu optimization to make any difference to my setup, given I have a single GPU... |
|
cc/ @0cc4m @jeffbolznv |
|
It might affect what we do in graph_optimize or what nodes run in parallel, but it's hard to speculate. |
|
@aagit Yes, a repro would be quite useful. |
|
https://gitlab.com/-/snippets/5989463 Here's a few more details: I can reproduce with vulkan on gfx1201 with Qwen3.6-35B-A3B-UD-Q6_K.gguf (from huggingface unsloth page) -reasoning off (reasoning off is likely required to reproduce). gbnf is in the reproducer, but it's not the problem it happens also without. Most llama.cpp parameters don't affect it. I didn't test if others setups can reproduce it. |
Fixes run-to-run non-determinism observed on Qwen3.6 35B-A3B UD on ROCm (Strix Halo): two consecutive runs of the same prompt produced different outputs at long context, and the stock build itself was not bit-stable (stock vs stock KL divergence 0.00384 without the fix, 0 with it). The layer loop called ggml_build_forward_expand(gf, cur) on the attention-norm output before building the recurrent attention layer. build_layer_attn_linear() and build_recurrent_attn() add the GDN state updates (ggml_cpy into the convolution and SSM state caches) later, and those writes are sink nodes with no consumers in the compute DAG, so their scheduling position is fixed only by the graph expansion order. Closing the graph branch at the pre-layer norm left the state cache writes ordered against the wrong node set, which could schedule them independently of the following layers and race with the state reads of the next token or layer. Move the expansion to after the recurrent layer is fully built. The full-attention branch keeps its original expansion position, so only the recurrent path changes. Matches upstream PR ggml-org#22661 (models: fix linear attention state corruption in recurrent layers, introduced by commit 2446419), which reported intermittent generation corruption on Vulkan. On this ROCm machine the same ordering surfaced as non-determinism; after the fix, repeated runs are bit-identical, and both the isolated LDS decode step and the MMQ-pair prefill show zero KL divergence against the stock reference. No throughput regression (tg128 47.08/45.21/41.71 t/s at 0/10k/30k, prefill unchanged). Assisted-by: OpenCode (cherry picked from commit be236e33d1e50be5b599cdc64c8b00a163963acf)
The layer loop called ggml_build_forward_expand(gf, cur) on the attention-norm output before building the recurrent attention layer. build_layer_attn_linear() adds the GDN state updates - the ggml_cpy into the convolution and SSM state caches, via delta-net-base.cpp - only afterwards, and those writes are sink nodes with no consumers in the compute DAG, so their scheduling position is fixed solely by graph expansion order. Closing the branch at the pre-layer norm left the state-cache writes ordered against the wrong node set, where they can be scheduled independently of the following layers and race with the state reads of the next token or layer. Move the expansion to after the recurrent layer is fully built. The full-attention branch keeps its original expansion position, so only the recurrent path changes. All three delta-net models carry the same shape and share the same delta-net-base.cpp sinks, so all three are fixed. Qwen3.8-27B is qwen35, not qwen35moe. Matches upstream PR ggml-org#22661, which reported intermittent generation corruption on Vulkan. Measured neutral here: tg64 within +0.4% at depths 0, 16384 and 65536 on a Q6_K MoE, and prefill unchanged - which is the right outcome for a fix about ordering rather than speed. Assisted-by: Claude Opus 5
Fixes run-to-run non-determinism observed on Qwen3.6 35B-A3B UD on ROCm (Strix Halo): two consecutive runs of the same prompt produced different outputs at long context, and the stock build itself was not bit-stable (stock vs stock KL divergence 0.00384 without the fix, 0 with it). The layer loop called ggml_build_forward_expand(gf, cur) on the attention-norm output before building the recurrent attention layer. build_layer_attn_linear() and build_recurrent_attn() add the GDN state updates (ggml_cpy into the convolution and SSM state caches) later, and those writes are sink nodes with no consumers in the compute DAG, so their scheduling position is fixed only by the graph expansion order. Closing the graph branch at the pre-layer norm left the state cache writes ordered against the wrong node set, which could schedule them independently of the following layers and race with the state reads of the next token or layer. Move the expansion to after the recurrent layer is fully built. The full-attention branch keeps its original expansion position, so only the recurrent path changes. Matches upstream PR ggml-org#22661 (models: fix linear attention state corruption in recurrent layers, introduced by commit 2446419), which reported intermittent generation corruption on Vulkan. On this ROCm machine the same ordering surfaced as non-determinism; after the fix, repeated runs are bit-identical, and both the isolated LDS decode step and the MMQ-pair prefill show zero KL divergence against the stock reference. No throughput regression (tg128 47.08/45.21/41.71 t/s at 0/10k/30k, prefill unchanged). Assisted-by: OpenCode
Commit 2446419 added ggml_build_forward_expand immediately after layer normalization to enable multi-GPU graph partitioning. However, for recurrent linear attention models, this prematurely closed the graph branch before the internal state updates (convolution and SSM states) were fully constructed.
This caused the graph engine to schedule state cache writes out of order or independently, leading to data races or stale reads in the convolution and SSM states. The bug manifested as intermittent generation corruption, such as alignment errors in code generation.
This commit moves the ggml_build_forward_expand call to immediately after the linear attention layer computation for all recurrent branches. This ensures the entire layer computation, including internal state updates, is part of the forward graph before being expanded for partitioning, restoring the correct topological barriers.
This was found and verified with a single GPU only using the synthmerge_bench. The multi GPU split is untested.
Fixes: 2446419 ("models : fix graph splits (#19866)")
Overview
Fixes computation of linear attention with recurrent layers.
Additional information
While running the synthmerge_bench in parallel with another synthmerge testcase, I was getting failures about failed validation of the head context. When validation fails logging shows a diff against the expected output, which looked particulary odd with just an alignment error off by whitespace for all lines. That was not the normal error an LLM does that I would expect. I guessed it was a bad quantization issue first and changing quantization made the synthmerge testcase work fine, until I noticed it wasn't the change of quantization fixing generation, but the restart of llama.cpp.
Requirements
Coded with rg-edit:
rg-edit regexp: ggml_build_forward_expand.{1,500}is_recurrent.*? }
rg-edit directive: Rewrite: move expand after the linear attention (gated delta net or KDA) layers, otherwise call the expand before the full attention
All files matching regexp added to gptel-context.