Deepseek 4: `-sm tensor` by am17an · Pull Request #26490 · ggml-org/llama.cpp

am17an · 2026-08-03T04:28:00Z

Overview

Add -sm tensor for DeepSeek4. The FA is mirrored because there is only 1 K head (Q has 64). Apart from that add memset_tensor is added to the meta backend.

Additional information

on 4x 4090s I see a PP increase of 50% but TG does not increase.

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: Yes, paired with Codex

am17an · 2026-08-03T04:29:05Z

@gaugarg-nv since Johannes is busy would you mind reviewing this PR?

gaugarg-nv

Could you test this with other GPU counts like 2 and 3? Also test with uneven tesnor-split values like {0.7,0.3}? This often helps uncover bugs related to tensor splitting.

am17an · 2026-08-03T16:34:19Z

Could you test this with other GPU counts like 2 and 3? Also test with uneven tesnor-split values like {0.7,0.3}? This often helps uncover bugs related to tensor splitting.

Thanks, that indeed turn up a bug which is now fixed with 3bbafa2

iSevenDays · 2026-08-03T16:49:38Z

Is this change compatible with draft model?

E llama_model_load: error loading model hyperparameters: key not found in model: dflash.attention.sliding_window_pattern
E failed to load draft model, 'DeepseekV4-Flash-20260731-DSpark.gguf'

/root/llama.cpp/build/bin/llama-server \
  --model DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
  -ngl 99 -ngld 99 \
  --main-gpu 1 \
  -ot 'blk\.([5-9]|[12][0-9]|3[0-7])\.ffn_(gate|up|down)_exps\.=CPU' \
  --split-mode tensor -ts 43,43 \
  -b 8192 -ub 8192 -fit off -fa on -c 262144 \
  --host 0.0.0.0 --port 8002 --parallel 1 --jinja --reasoning on \
  --no-mmap --log-verbosity 4 \
  --spec-type draft-dspark --spec-draft-n-max 5 \
  --model-draft DeepseekV4-Flash-20260731-DSpark.gguf \
  --spec-draft-p-min 0.5

main model loads fine under --split-mode tensor, but the DSpark draft fails

the same command works on current master (build 10248 / e8e06f7)

am17an · 2026-08-05T09:15:17Z

Would appreciate if other people can also test this and we can merge. This has been tested on 2x, 4x and 5x GPUs

ggerganov · 2026-08-05T09:34:00Z

+                if (tensor->view_src != nullptr) {
+                    // full-tensor view created with ggml_view_tensor, transparent for the split state
+                    split_state = ggml_backend_meta_get_split_state(stc, tensor->view_src, assume_sync);
+                } else {
+                    split_state = {GGML_BACKEND_SPLIT_AXIS_MIRRORED, {0}, {1}, 1};
+                }


Why do we need to handle the views differently here?

S3kundes · 2026-08-05T15:13:48Z

Were the review comments that @JohannesGaessler left on https://github.com/ggml-org/llama.cpp/pull/25860/changes taken into account here?

A lot of the implementation still looks the same to me, while the original PR was clearly presented as an incomplete proof of concept. What changed to address those earlier issues?

Also, since the KV cache and DSV4 state are still mirrored, am I right that context memory isn’t distributed across GPUs and 1M-token contexts still aren’t possible in tensor split mode?

I tested this version and I’m seeing roughly the same PP and TG improvements as with #25860. Could the lack of TG improvement on your setup be related to PCIe bandwidth—x8 vs x16, the available CPU PCIe lanes, or the risers being used?

am17an · 2026-08-05T15:22:57Z

@S3kundes Yup I see roughly the same speeds. Feel free to close the other PR

kiuckhuang · 2026-08-06T02:44:34Z

Would appreciate if other people can also test this and we can merge. This has been tested on 2x, 4x and 5x GPUs

Tested with 5 x GPU and 8 x GPU, old Turing 24GB, both are working.
Also tested with --spec-type draft-dspark,ngram-mod and --spec-type draft-mtp,ngram-mod, all are working, but slower than non speculative decoding, maybe due to non-NVlink and PCIE3-X16 bus bandwidth

Performance-wise:
5x (same PIX) > 8x > 5x+dspark >= 5x+mtp

wOvAN · 2026-08-07T14:58:50Z

3090 (x8) - pci-e 4 (x8)

same query, it became slower, prefill the same, decode fell from 38 tps to 20 with tensor

wOvAN · 2026-08-07T15:04:44Z

for reference layer with draft-dspark

mgabor3141 · 2026-08-07T20:04:51Z

[AGENT] Independent test on four consumer GPUs, since you asked for outside testing before merge. Summary: the prefill gain reproduces strongly, decode regresses slightly, and VRAM distribution becomes even. One numerical question at the end.

Setup

4x RTX 3090 24 GB, sm_86, 220 W each
Threadripper 1950X on ASUS PRIME X399-A, PCIe 3.0, x8/x16/x8/x16 across two NUMA nodes
Peer access is enabled via a patched NVIDIA 610.43.03 open driver with 32 GiB BAR1, so all six GPU pairs report OK. Stock drivers report GNS on this topology, so results without that patch may differ.
Built your branch at b70a47a8a2dab17e018cf37301885729b3e4d246 with .devops/cuda.Dockerfile --target server, CUDA_DOCKER_ARCH=86

Model: unsloth/DeepSeek-V4-Flash-0731-GGUF, UD-IQ2_M, 84.68 GiB, revision fbbb5b93fb787c21338159b0af3318bb3f4d9768.

Server flags, identical apart from --split-mode:

--ctx-size 16384 --parallel 1 --n-gpu-layers 999 \
--tensor-split 1,1,1,1 --flash-attn on --batch-size 512 --ubatch-size 256

Results

Both rows are from the same build, so this isolates layout rather than engine version. Decode is a 256-token deterministic run, prefill is a 15,412-token prompt.

Layout	Decode tok/s	15K prefill tok/s	Loaded VRAM
`--split-mode layer`	40.32	368.88	21.0 to 23.0 GiB, uneven
`--split-mode tensor`	38.53	635.51	23.24 GiB, even

Prefill: +72.3%
Decode: -4.4%

So the prompt-processing win holds up on PCIe 3.0 without NVLink. Decode was slightly worse rather than flat, which is consistent with more inter-GPU traffic per token on a slow interconnect.

The even VRAM distribution is a worthwhile secondary benefit that I did not see mentioned. Layer split left one card about 2 GiB lighter than another, and on a four-card build the fullest card is what caps context.

For reference, mainline b10290 cannot use --split-mode row with this model at all:

device CUDA0 does not support split buffers

Quality check

I ran a 75-scenario deterministic benchmark (tool calling, instruction following, structured output, data extraction, reasoning/math) at pass@1 under both layouts:

layer split: 67/75
tensor split: 66/75

The single differing scenario was a reasoning item that passed 2 of 3 times when rerun in isolation under tensor split, so I read that as sampling variance, not a regression.

One thing I could not resolve

A temperature-zero, fixed-seed, 256-token completion produced a different output hash under tensor split than under layer split, on the same build and same model file. Both outputs were coherent and on-topic, and an exact-instruction canary returned the expected token, so this does not look like corruption.

My assumption is that splitting tensors changes floating-point reduction order, one token diverges, and the rest follow. That would be expected rather than a bug. But I wanted to flag it explicitly in case exact parity with layer split is something you consider a requirement, since it is the one claim I could not verify.

Happy to rerun anything specific on this hardware if useful.

cb88 · 2026-08-07T20:53:41Z

2xMI50 32GB +AMD EPYC 7352 128GB ram (-sm layer works at about 10t/s note only half DDR4 channels populated) -sm tensor with --n-gpu-layers 18 results about 7t/s

0.14.793.130 I load_tensors: Meta() model buffer size = 29769.34 MiB
0.14.793.131 I load_tensors: ROCm_Host model buffer size = 89308.16 MiB

Something seems not right about this though as the Meta() buffer should be the combined size of each GPU right? When I was running the model 29+GB was used on each GPU. Also with SM layer it was offloading about 42layers to the GPUs. Trying to let it auto fit just resulted in it crashing with.

0.00.301.944 I load_tensors: loading model tensors, this can take a while... (load_mode = dio)
/home/cb88/llama.cpp/ggml/src/ggml-backend-meta.cpp:1721: GGML_ASSERT(meta_buf_ctx->bufs[i]) failed
0.00.331.629 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 74451.44 MiB on device 0: cudaMalloc failed: out of memory
0.00.331.635 E alloc_tensor_range: failed to allocate ROCm0 buffer of size 78067998336

my script it quantizing the context or not didn't seem to affect the outcomes:
HIP_VISIBLE_DEVICES=0,1 ./llama.cpp/build/bin/llama-server --host 0.0.0.0
-fa on --load-mode dio --api-key llama
-sm tensor
-c 10000
-lv 4 --tools all
-ctk q8_0 -ctv q8_0
-m $@

Edit: also both GPUs are on PCIE 3.0 @ 16x links

blakemartz · 2026-08-08T00:35:54Z

[AGENT]

Tester datapoint from a CPU-expert-offload rig (Ampere Altra aarch64 + 2× RTX 4090, DeepSeek-V4-Flash-0731 MXFP4, experts on CPU): -sm tensor aborts at model load with expert tensors overridden to CPU, in both override styles:

with a regex -ot '...ffn_(gate|up|down)_exps=CPU': GGML_ASSERT(meta_buf_ctx->bufs[i]) at ggml-backend-meta.cpp:1721
with --n-cpu-moe 43: GGML_ASSERT(tensor->view_offs == 0 && ggml_are_same_shape(tensor, tensor->view_src)) at ggml-backend-meta.cpp:855

Given cb88's MI50 result above loads with --n-gpu-layers-style partial offload, the failure looks specific to tensor-buft-override expert offload rather than partial offload generally. Happy to retest when/if override support lands — on this rig the PP win would matter (long-context ingest is the pain point).

Skinpack · 2026-08-08T07:43:54Z

2x ROCm R9700 with a partial CPU MOE offload hits the same GGML_ASSERT(tensor->view_offs == 0 && ggml_are_same_shape(tensor, tensor->view_src)) assert for me as well.

./llama-server --load-mode mlock --ubatch-size 512 --batch-size 512 --cache-reuse 256 -fa on -np 1 --ctx-size 384000 --model DeepSeek-V4-Flash-0731-UD-IQ3_XXS-00001-of-00004.gguf --reasoning on --fit off -sm tensor -ts 320,320 --n-cpu-moe 21

am17an · 2026-08-17T18:05:07Z

@Skinpack @cb88 the latest commit should solve the crash

thomasjfox · 2026-08-17T18:46:38Z

@am17an: I quickly tested the almost-final PR (f48c77bfc9b4fe96c4edc918880039f25d23ec71) on 2x RTX 6000 Pro and compared to master:

master: Assertion upon start with -sm tensor
This PR: Works fine. Hammered it with two jobs in parallel

PP seems to go down from 1600 toks/s to about 1380 toks/s during the brief test case. Not sure what's going on there. TG is about the same. From a stability viewpoint the branch is stable. ✅

Log from -sm tensor:

8.29.457.768 I slot print_timing: id 1 | task 2285 | prompt processing, n_tokens = 10240, progress = 0.56, t = 7.41 s / 1381.62 tokens per second
8.31.025.900 I slot print_timing: id 1 | task 2285 | prompt processing, n_tokens = 12288, progress = 0.67, t = 8.98 s / 1368.42 tokens per second
8.32.657.857 I slot print_timing: id 1 | task 2285 | prompt processing, n_tokens = 14336, progress = 0.79, t = 10.61 s / 1350.96 tokens per second

Compared to default split mode:

1.35.209.929 I slot print_timing: id 0 | task 79 | prompt processing, n_tokens = 10210, progress = 0.53, t = 6.28 s / 1626.08 tokens per second
1.36.574.051 I slot print_timing: id 0 | task 79 | prompt processing, n_tokens = 12252, progress = 0.63, t = 7.64 s / 1603.03 tokens per second
1.37.988.100 I slot print_timing: id 0 | task 79 | prompt processing, n_tokens = 14294, progress = 0.74, t = 9.06 s / 1578.21 tokens per second

Tested with DSpark enabled and --spec-draft-n-max 5, though that should not matter for PP imho.

am17an · 2026-08-17T19:09:07Z

Can you try without dspark?

wOvAN · 2026-08-19T11:14:50Z

I pr'ed similar #26636 to solve spec-draft-device

am17an · 2026-08-22T16:39:31Z

The exps and the shared exps can have a delayed all-reduce similar to earlier add pattern. Doing this reduces all-reduce per layer from 3 to 2, it seems to help performance

am17an · 2026-08-23T15:35:52Z

Latest results on 4x 4090s, -sm tensor is now better here too for TG

model	size	params	backend	ngl	sm	fa	test	t/s
deepseek4 ?B IQ3_XXS - 3.0625 bpw	82.87 GiB	284.33 B	CUDA	-1	layer	1	pp512	1133.36 ± 213.87
deepseek4 ?B IQ3_XXS - 3.0625 bpw	82.87 GiB	284.33 B	CUDA	-1	layer	1	tg128	72.68 ± 0.63
deepseek4 ?B IQ3_XXS - 3.0625 bpw	82.87 GiB	284.33 B	CUDA	-1	layer	1	pp512 @ d10000	1042.45 ± 37.58
deepseek4 ?B IQ3_XXS - 3.0625 bpw	82.87 GiB	284.33 B	CUDA	-1	layer	1	tg128 @ d10000	69.92 ± 0.58
deepseek4 ?B IQ3_XXS - 3.0625 bpw	82.87 GiB	284.33 B	CUDA	-1	layer	1	pp512 @ d20000	961.87 ± 42.93
deepseek4 ?B IQ3_XXS - 3.0625 bpw	82.87 GiB	284.33 B	CUDA	-1	layer	1	tg128 @ d20000	67.49 ± 0.61
deepseek4 ?B IQ3_XXS - 3.0625 bpw	82.87 GiB	284.33 B	CUDA	-1	tensor	1	pp512	1499.61 ± 685.73
deepseek4 ?B IQ3_XXS - 3.0625 bpw	82.87 GiB	284.33 B	CUDA	-1	tensor	1	tg128	78.25 ± 1.80
deepseek4 ?B IQ3_XXS - 3.0625 bpw	82.87 GiB	284.33 B	CUDA	-1	tensor	1	pp512 @ d10000	1160.70 ± 484.89
deepseek4 ?B IQ3_XXS - 3.0625 bpw	82.87 GiB	284.33 B	CUDA	-1	tensor	1	tg128 @ d10000	74.97 ± 4.17
deepseek4 ?B IQ3_XXS - 3.0625 bpw	82.87 GiB	284.33 B	CUDA	-1	tensor	1	pp512 @ d20000	1147.64 ± 403.52
deepseek4 ?B IQ3_XXS - 3.0625 bpw	82.87 GiB	284.33 B	CUDA	-1	tensor	1	tg128 @ d20000	72.90 ± 4.11

checkpoint51677 · 2026-08-24T13:46:29Z

I test -sm tensor -ts 43,43 on dual rtx pro 5000，but the prefill was worse then layer split, i use b10608 release. So what the problem?

* DSV4: sm tensor * set coarser granularity for head splits * fix dspark * add model saving for dsv4 + allow dflash to return on specific device * add comment about dsv4 seq_rm * simplify * add shared expert delayed allreduce * remove special test for dsv4

am17an · 2026-09-04T09:25:21Z

I found a bug here with dspark which can be fixed by bumping GGML_SCHED_MAX_SPLIT_INPUTS to 64. Proper fix incoming

JohannesGaessler · 2026-09-04T12:37:42Z

As I already commented in #25860 , the attention should not be mirrored. If there only is a single attention head that will at least not gimp the performance but it will still result in overallocation of the KV cache. If this exact architecture only exists with a single K/V head it may make sense to implement support for a split state that is explicitly assigning everything to a single GPU since implementing and maintaining the theoretical logic for >1 attention heads would not be worthwhile.

* DSV4: sm tensor * set coarser granularity for head splits * fix dspark * add model saving for dsv4 + allow dflash to return on specific device * add comment about dsv4 seq_rm * simplify * add shared expert delayed allreduce * remove special test for dsv4

am17an requested review from CISC and JohannesGaessler as code owners August 3, 2026 04:28

github-actions Bot added testing Everything test related ggml changes relating to the ggml tensor library for machine learning labels Aug 3, 2026

am17an changed the title ~~DSV4: sm tensor~~ Deepseek 4: -sm tensor Aug 3, 2026

am17an requested a review from gaugarg-nv August 3, 2026 04:28

gaugarg-nv reviewed Aug 3, 2026

View reviewed changes

Comment thread tests/test-llama-archs.cpp Outdated

ggerganov mentioned this pull request Aug 5, 2026

RPC: add -sm tensor #26610

Open

am17an force-pushed the dsv4-sm-tensor branch from 3bbafa2 to bfc7bf0 Compare August 5, 2026 08:49

ggerganov reviewed Aug 5, 2026

View reviewed changes

github-actions Bot added the model Model specific label Aug 5, 2026

am17an force-pushed the dsv4-sm-tensor branch 2 times, most recently from f055961 to f48c77b Compare August 17, 2026 17:58

am17an force-pushed the dsv4-sm-tensor branch from f48c77b to 6b29fd8 Compare August 17, 2026 18:28

am17an force-pushed the dsv4-sm-tensor branch from 6b29fd8 to a060c66 Compare August 20, 2026 16:09

am17an force-pushed the dsv4-sm-tensor branch 2 times, most recently from a6cd579 to cee363e Compare August 23, 2026 13:31

am17an added 7 commits August 23, 2026 15:35

DSV4: sm tensor

ae34ad1

set coarser granularity for head splits

8f6e63f

fix dspark

832fd0e

add model saving for dsv4 + allow dflash to return on specific device

a67a1ca

add comment about dsv4 seq_rm

d6a392e

simplify

d428657

add shared expert delayed allreduce

46daefb

am17an force-pushed the dsv4-sm-tensor branch from cee363e to 46daefb Compare August 23, 2026 15:31

ggerganov approved these changes Aug 23, 2026

View reviewed changes

am17an added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Aug 23, 2026

ggerganov reviewed Aug 24, 2026

View reviewed changes

Comment thread tests/test-llama-archs.cpp Outdated

remove special test for dsv4

f9e2f8c

ggerganov merged commit bf0a29c into ggml-org:master Aug 24, 2026
23 of 26 checks passed

ryan5rdx mentioned this pull request Aug 24, 2026

rpc: support apple RDMA as an RPC transport #26421

Merged

ggerganov mentioned this pull request Aug 25, 2026

llama.cpp : bump version to 0.3.0 #27696

Merged

BrewTestBot mentioned this pull request Aug 25, 2026

ggml 0.22.0 Homebrew/homebrew-core#300553

Merged

1 task

lapy mentioned this pull request Aug 25, 2026

Eval bug: llama.cpp-b9568/ggml/src/ggml-cuda/fattn.cu:579: fatal error #24324

Open

NeedForSpeed73 mentioned this pull request Sep 3, 2026

Misc. bug: --tensor-split causes intermittent output degeneration on long-context MoE inference, independent of client (reproducible via raw API calls) #28185

Open

Uh oh!

Conversation

am17an commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

Requirements

Uh oh!

am17an commented Aug 3, 2026

Uh oh!

gaugarg-nv left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

am17an commented Aug 3, 2026

Uh oh!

iSevenDays commented Aug 3, 2026

Uh oh!

am17an commented Aug 5, 2026

Uh oh!

Uh oh!

Uh oh!

ggerganov Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

S3kundes commented Aug 5, 2026

Uh oh!

am17an commented Aug 5, 2026

Uh oh!

kiuckhuang commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wOvAN commented Aug 7, 2026

Uh oh!

wOvAN commented Aug 7, 2026

Uh oh!

mgabor3141 commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Setup

Results

Quality check

One thing I could not resolve

Uh oh!

cb88 commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blakemartz commented Aug 8, 2026

Uh oh!

Skinpack commented Aug 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

am17an commented Aug 17, 2026

Uh oh!

thomasjfox commented Aug 17, 2026

Uh oh!

am17an commented Aug 17, 2026

Uh oh!

wOvAN commented Aug 19, 2026

Uh oh!

am17an commented Aug 22, 2026

Uh oh!

am17an commented Aug 23, 2026

Uh oh!

Uh oh!

Uh oh!

checkpoint51677 commented Aug 24, 2026

Uh oh!

am17an commented Sep 4, 2026

Uh oh!

JohannesGaessler commented Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

16 participants

am17an commented Aug 3, 2026 •

edited

Loading

gaugarg-nv left a comment •

edited

Loading

kiuckhuang commented Aug 6, 2026 •

edited

Loading

mgabor3141 commented Aug 7, 2026 •

edited

Loading

cb88 commented Aug 7, 2026 •

edited

Loading

Skinpack commented Aug 8, 2026 •

edited

Loading