Skip to content

models: fix linear attention state corruption in recurrent layers - #22661

Open
aagit wants to merge 1 commit into
ggml-org:masterfrom
aagit:linear-attention-corruption
Open

aagit wants to merge 1 commit into
ggml-org:masterfrom
aagit:linear-attention-corruption

Conversation

@aagit

@aagit aagit commented May 4, 2026

Copy link
Copy Markdown
Contributor

Commit 2446419 added ggml_build_forward_expand immediately after layer normalization to enable multi-GPU graph partitioning. However, for recurrent linear attention models, this prematurely closed the graph branch before the internal state updates (convolution and SSM states) were fully constructed.

This caused the graph engine to schedule state cache writes out of order or independently, leading to data races or stale reads in the convolution and SSM states. The bug manifested as intermittent generation corruption, such as alignment errors in code generation.

This commit moves the ggml_build_forward_expand call to immediately after the linear attention layer computation for all recurrent branches. This ensures the entire layer computation, including internal state updates, is part of the forward graph before being expanded for partitioning, restoring the correct topological barriers.

This was found and verified with a single GPU only using the synthmerge_bench. The multi GPU split is untested.

Fixes: 2446419 ("models : fix graph splits (#19866)")

Overview

Fixes computation of linear attention with recurrent layers.

Additional information

While running the synthmerge_bench in parallel with another synthmerge testcase, I was getting failures about failed validation of the head context. When validation fails logging shows a diff against the expected output, which looked particulary odd with just an alignment error off by whitespace for all lines. That was not the normal error an LLM does that I would expect. I guessed it was a bad quantization issue first and changing quantization made the synthmerge testcase work fine, until I noticed it wasn't the change of quantization fixing generation, but the restart of llama.cpp.

Requirements

Coded with rg-edit:

rg-edit regexp: ggml_build_forward_expand.{1,500}is_recurrent.*? }
rg-edit directive: Rewrite: move expand after the linear attention (gated delta net or KDA) layers, otherwise call the expand before the full attention

All files matching regexp added to gptel-context.

Commit 2446419 added ggml_build_forward_expand immediately after
layer normalization to enable multi-GPU graph partitioning. However,
for recurrent linear attention models, this prematurely closed the
graph branch before the internal state updates (convolution and SSM
states) were fully constructed.

This caused the graph engine to schedule state cache writes out of
order or independently, leading to data races or stale reads in the
convolution and SSM states. The bug manifested as intermittent
generation corruption, such as alignment errors in code generation.

This commit moves the ggml_build_forward_expand call to immediately
after the linear attention layer computation for all recurrent
branches. This ensures the entire layer computation, including
internal state updates, is part of the forward graph before being
expanded for partitioning, restoring the correct topological barriers.

This was found and verified with a single GPU only using the
synthmerge_bench. The multi GPU split is untested.

Fixes: 2446419 ("models : fix graph splits (ggml-org#19866)")
@aagit
aagit requested a review from CISC as a code owner May 4, 2026 01:25
@github-actions github-actions Bot added the model Model specific label May 4, 2026
@CISC
CISC requested a review from ggerganov May 4, 2026 07:11
@ggerganov

Copy link
Copy Markdown
Member

This change would again break multi-GPU support.

I am not sure that the explanation is correct. Looking at the order of the nodes when running with GGML_SCHED_DEBUG=2 and -lv 5, I don't see a problem in the ordering on master.

@aagit

aagit commented May 4, 2026

Copy link
Copy Markdown
Contributor Author

Hello, should I try to provide a self contained reproducer to facilitate the debugging? I tried also to only add an extra expand after linear attention as a flush synchronization and that didn't fix it. Another data point, I'm running the model on a single GPU and it only happens with vulkan backend, I couldn't reproduce on rocm. I suppose vulkan must be leveraging the expand somehow while rocm doesn't? In fact I didn't initially expect such multi gpu optimization to make any difference to my setup, given I have a single GPU...

@CISC

CISC commented May 4, 2026

Copy link
Copy Markdown
Member

cc/ @0cc4m @jeffbolznv

@jeffbolznv

Copy link
Copy Markdown
Contributor

It might affect what we do in graph_optimize or what nodes run in parallel, but it's hard to speculate.

@ggerganov

Copy link
Copy Markdown
Member

@aagit Yes, a repro would be quite useful.

@aagit

aagit commented May 5, 2026

Copy link
Copy Markdown
Contributor Author

https://gitlab.com/-/snippets/5989463

Here's a few more details: I can reproduce with vulkan on gfx1201 with Qwen3.6-35B-A3B-UD-Q6_K.gguf (from huggingface unsloth page) -reasoning off (reasoning off is likely required to reproduce). gbnf is in the reproducer, but it's not the problem it happens also without. Most llama.cpp parameters don't affect it. I didn't test if others setups can reproduce it.

Nathanw1014 pushed a commit to Nathanw1014/llama.cpp that referenced this pull request Aug 21, 2026
Fixes run-to-run non-determinism observed on Qwen3.6 35B-A3B UD on
ROCm (Strix Halo): two consecutive runs of the same prompt produced
different outputs at long context, and the stock build itself was not
bit-stable (stock vs stock KL divergence 0.00384 without the fix, 0
with it).

The layer loop called ggml_build_forward_expand(gf, cur) on the
attention-norm output before building the recurrent attention layer.
build_layer_attn_linear() and build_recurrent_attn() add the GDN state
updates (ggml_cpy into the convolution and SSM state caches) later,
and those writes are sink nodes with no consumers in the compute DAG,
so their scheduling position is fixed only by the graph expansion
order. Closing the graph branch at the pre-layer norm left the state
cache writes ordered against the wrong node set, which could schedule
them independently of the following layers and race with the state
reads of the next token or layer.

Move the expansion to after the recurrent layer is fully built. The
full-attention branch keeps its original expansion position, so only
the recurrent path changes.

Matches upstream PR ggml-org#22661 (models: fix linear attention state
corruption in recurrent layers, introduced by commit 2446419), which
reported intermittent generation corruption on Vulkan. On this ROCm
machine the same ordering surfaced as non-determinism; after the fix,
repeated runs are bit-identical, and both the isolated LDS decode step
and the MMQ-pair prefill show zero KL divergence against the stock
reference. No throughput regression (tg128 47.08/45.21/41.71 t/s at
0/10k/30k, prefill unchanged).

Assisted-by: OpenCode
(cherry picked from commit be236e33d1e50be5b599cdc64c8b00a163963acf)
LaurentZuijdwijk pushed a commit to LaurentZuijdwijk/llama.cpp that referenced this pull request Aug 25, 2026
The layer loop called ggml_build_forward_expand(gf, cur) on the attention-norm
output before building the recurrent attention layer. build_layer_attn_linear()
adds the GDN state updates - the ggml_cpy into the convolution and SSM state
caches, via delta-net-base.cpp - only afterwards, and those writes are sink
nodes with no consumers in the compute DAG, so their scheduling position is
fixed solely by graph expansion order. Closing the branch at the pre-layer norm
left the state-cache writes ordered against the wrong node set, where they can
be scheduled independently of the following layers and race with the state
reads of the next token or layer.

Move the expansion to after the recurrent layer is fully built. The
full-attention branch keeps its original expansion position, so only the
recurrent path changes.

All three delta-net models carry the same shape and share the same
delta-net-base.cpp sinks, so all three are fixed. Qwen3.8-27B is qwen35, not
qwen35moe.

Matches upstream PR ggml-org#22661, which reported intermittent generation corruption
on Vulkan. Measured neutral here: tg64 within +0.4% at depths 0, 16384 and
65536 on a Q6_K MoE, and prefill unchanged - which is the right outcome for a
fix about ordering rather than speed.

Assisted-by: Claude Opus 5
ilmmatias pushed a commit to ilmmatias/llama.cpp that referenced this pull request Sep 2, 2026
Fixes run-to-run non-determinism observed on Qwen3.6 35B-A3B UD on
ROCm (Strix Halo): two consecutive runs of the same prompt produced
different outputs at long context, and the stock build itself was not
bit-stable (stock vs stock KL divergence 0.00384 without the fix, 0
with it).

The layer loop called ggml_build_forward_expand(gf, cur) on the
attention-norm output before building the recurrent attention layer.
build_layer_attn_linear() and build_recurrent_attn() add the GDN state
updates (ggml_cpy into the convolution and SSM state caches) later,
and those writes are sink nodes with no consumers in the compute DAG,
so their scheduling position is fixed only by the graph expansion
order. Closing the graph branch at the pre-layer norm left the state
cache writes ordered against the wrong node set, which could schedule
them independently of the following layers and race with the state
reads of the next token or layer.

Move the expansion to after the recurrent layer is fully built. The
full-attention branch keeps its original expansion position, so only
the recurrent path changes.

Matches upstream PR ggml-org#22661 (models: fix linear attention state
corruption in recurrent layers, introduced by commit 2446419), which
reported intermittent generation corruption on Vulkan. On this ROCm
machine the same ordering surfaced as non-determinism; after the fix,
repeated runs are bit-identical, and both the isolated LDS decode step
and the MMQ-pair prefill show zero KL divergence against the stock
reference. No throughput regression (tg128 47.08/45.21/41.71 t/s at
0/10k/30k, prefill unchanged).

Assisted-by: OpenCode
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

model Model specific

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants