DeepseekV4: Add fused hyper-connection ops by am17an · Pull Request #25585 · ggml-org/llama.cpp

am17an · 2026-07-12T11:54:54Z

Overview

Add the sinkhorn ops to the DeepseekV4 graph. Graph nodes go from 29k to 8k with this change. Big increases in the TG + PP

Additional information

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES, CUDA kernels and backend tests are auto-generated, I have reviewed and tested them

jadenmach2 · 2026-07-13T15:44:48Z

@am17an looks like there's an overlap with #25421 . I added the Sinkhorn step as a standalone SINKHORN_NORM op, you've fused the whole HC block, which is the better call.

I came at this from the AMD side. Profiling deepseek-v4-flash on an MI250X, the decomposed Sinkhorn was ~30% of gpu kernel time across ~250k tiny 4×4 dispatches, fusing it gave me +60.9% decode / +7.4% prefill on 4 GCDs.

Since we're solving the same thing, want to join forces? I'm happy to close mine and help on this one, especially validating/tuning it on MI250X if that's not already done/ any other additions you would like to delegate ?

am17an · 2026-07-13T16:10:17Z

@jadenmach2 sure, thanks! I've only tested these on nvidia gpus

ggerganov · 2026-07-17T10:51:15Z

@am17an The new ops are producing nans sometimes:

https://github.com/ggml-org/llama.cpp/actions/runs/29572046650/job/87857884185?pr=25816#step:3:1965

fairydreaming · 2026-07-17T12:23:21Z

@am17an The new ops are producing nans sometimes:

https://github.com/ggml-org/llama.cpp/actions/runs/29572046650/job/87857884185?pr=25816#step:3:1965

This is just my wild guess, but isn't that because of uninitialized sentinel tensors? test_dsv4_hc implements its own initialize_tensors() where it skips initialization if tensor_range() returns false - and it does it based on the tensor name.

am17an · 2026-07-17T12:26:52Z

Yeah I think that's the most likely cause. AFK in case @fairydreaming you want to pick this up, otherwise I will fix in a few hours

fairydreaming · 2026-07-17T12:57:10Z

@ggerganov @am17an This should fix it: #25822

Squash-rebase of the ds4-flash-experiments branch (post-cleanup: 8 dead flags + MOE_TILE/FP4_RT ops removed, LID_CACHE_MXFP4 default-on) onto current upstream, which had independently evolved DeepSeek-V4 (fused HC ops ggml-org#25585, kv_stream cache refactor ggml-org#25702, seq_rm fix ggml-org#25588, lightning-indexer ggml-org#24231). Reconciliation (per Teej's calls — keep our tuned versions, adopt upstream only where cleanly additive): - KV cache: adopted upstream's kv_stream per-stream views; kept our kv_stash MTP frontier-rewind AND our frontier-aware seq_rm (upstream's ggml-org#25588 seq_rm rejects the 1-token-tail eviction our server/MTP post_decode relies on -> aborts; caught in re-validation, reverted to ours). - HC fusion: kept OUR validated bandwidth-minimal HC op; upstream's HC_PRE/COMB/ POST sit dormant. Renamed our colliding ggml_dsv4_hc_post -> _hc_fused_post (+ test struct); fixed models.h class decl to match our deepseek4 impl. - Server: adapted our --cache-disk L2 tier to upstream's split prompt-cache (server_prompt vs server_prompt_cache_state / server_prompt_data). - Kept all unique work: LID indexer CUDA kernels, CSA_TILE, FA_SPLIT/FA_MERGE, fp4-mma + packed MXFP4 container, radix/int8/dec, power governor, MTP, CVEC. - experiments/ untracked (repo-local ignore) — branch delta is code only. Verified on the rebased tree: - build clean (cli + server + tests) on upstream 86d86ed - DSV4 backend-ops 2/2 all kept ops (LID_TOPK/UNION/MEMB, HC_FUSED, QAT_SET_ROWS, FA_MERGE) - llama-cli shallow smoke c8192: coherent, byte-identical greedy to pre-rebase - llama-server c32768 + MTP + --cache-disk (LAN): MTP 21.5 t/s (draft 47/95 accepted), 3 requests no crash, --cache-disk spill + 660-tok disk restore Not yet run: deep-context (>=131k) serving — defer to an attended run (wedge risk).

…, OpenCL Q6_K/Adreno, CORS, checkpoint min-step, prompt cache refactor, MoE expert API stays) Upstream highlights since 6be7459: - model: DFlash speculative with KV rotation (ggml-org#25823) - model: Hy3 (hy_v3) with MTP speculative decoding (ggml-org#25395) - model: DeepseekV4 with fused hyper-connection ops (ggml-org#25585) - ggml: 0.17.0, LIGHTNING_INDEXER, out_prod, f16 set_rows - vulkan: Q2_0 support, native e2m1/e4m3 conversions, transfer-queue race fix - CUDA: MMQ kernel config refactor (ggml-org#24127), tighter MMQ src1 buffer for fp4 (ggml-org#25613), CUDA graphs on Volta/Turing, MoE gate/up dedup, CUDA Virtual Devices - ROCm: hexagon L2 cache rework, native fp4, FP16/INT8 coopmat on AMD - SYCL: Battlemage flash attention via oneDNN XMX, XIELU op, fp16 conv2d_dw - OpenCL: Q6_K GEMM/GEMV fix, ragged-tile MoE prefill FP16, Adreno vectorized LD/ST, A7x optimizations, ABS op - kleidiai: SME2 f32 kernel, SME vs SME2 dispatch - server: refactor prompt cache state ownership (ggml-org#25649) - new server_prompt_cache_state separates prompt metadata from KV data - server: evict checkpoints within min-step (ggml-org#25472) - server: text-only slot save/restore with mtmd (ggml-org#25076) - server: --cors-* options (ggml-org#25655) - server: refactored server_stream (ggml-org#25541) - server: respect min-step when splitting prompt batches (ggml-org#25420) - server: move chat-template thinking probe inside init try/catch (ggml-org#24093) - common: auto-download dflash/eagle3 HF sidecars (ggml-org#25811), drop --stdin mutual-exclusion, align tokenize usage - conversion: BitNetForCausalLM, dflash tokenizer fix, split MTP export for HY V3 - llama-quant: exclude i32 ffn_gate_tid2eid routing table, allow manual tensor types with --pure - llama-batch: fix allowed decreasing pos in a seq (ggml-org#25449), n_keep_tail in split_equal for recurrent - llama: refactor fused ops (ggml-org#24646), TP fix for Phi3/Bert/Plamo2/3/ChatGLM - ui: agentic content UX, reasoning effort on mobile add sheet, MCP panel fixes, thinking menu fix - vendor: BoringSSL 0.20250713.0 - tests: actually exercise test-recurrent-state-rollback, ds_v4_hc sentinel init, export-graph-ops graceful exit CachyLLama preservation work (conflict resolution): 1. tools/server/server-task.h: Accept upstream's server_prompt refactor (no data member, clear() method). Move our t_last_used field from server_prompt to server_prompt_cache_state (where it now lives after the refactor). server_prompt_cache_state already has the size() method, so our old size() on server_prompt is no longer needed. 2. tools/server/server-context.cpp (create_checkpoint): Take upstream's min-step eviction pre-filter as the FIRST pass, then keep our existing highest-pos_min eviction as the capacity overflow fallback. These are complementary: min-step removes redundant checkpoints from the same task; highest-pos_min keeps the rec-window-friendly checkpoints when at cap. 3. tools/server/server-context.cpp (handle_completions_impl): Keep our std::vector<server_task> tasks batching for multi-prompt requests and per-user concurrency check, AND take upstream's res->set_req(&req) for spipe ownership transfer. 4. tools/server/server-task.cpp: Fix references to entry.tokens -> entry.prompt.tokens, entry.checkpoints -> entry.prompt.checkpoints, entry.n_tokens() -> entry.prompt.n_tokens(). Update find_eviction_candidate return type from list<server_prompt>::iterator to list<server_prompt_cache_state>::iterator. 5. ggml/src/ggml-cuda/mmq.cuh + new mmq-config-rdna3_5.cuh: Upstream's massive MMQ refactor moved per-architecture config into separate files but did NOT add RDNA3.5 (gfx1150/1/2/3, Strix Halo). Create mmq-config-rdna3_5.cuh (231 CASE entries) derived from rdna2 with nthreads=128 (4 warps) and I=48 (smaller X tile) matching our original Strix Halo tuning. Wire into both host and device dispatch paths before the RDNA4 / RDNA2 fallback. 6. README.md and AGENTS.md: Keep CachyLLama-specific links and project context where upstream added parallel content. Verified: - cmake --build builds clean (Release, CPU-only) - llama-server starts, --help shows all CachyLLama flags preserved: --cache-ssd-hot-ram, --cache-ssd-warm-ram, --cache-ssd-system-prompts, --cache-ssd-system-max-days, --cache-ssd-no-fsync, --cache-ssd-max-conversations, --max-concurrent-per-user - /expert-stats and /expert-tracking endpoints preserved - 55/58 tests pass; 3 failures unrelated to merge: - test-tokenizers-ggml-vocabs: missing model downloads - test-jinja-py: missing jinja2 Python module - test-quant-type-selection: snapshot mismatch on upstream's new MXFP4_MOE heuristic Custom CachyLLama files untouched (no upstream conflicts): - common/kv-ssd-cache.{cpp,h}, common/kv-ssd-posix.h, common/kv-ssd-system-cache.{cpp,h} - common/kv_page_manager.{cpp,h} - tools/server/server-context-page-manager.{cpp,h} - tools/server/server-context-ssd-cache.{cpp,h} - test_kv_page_manager.cpp, tests/test-ssd-cache-caps.cpp - STRIX_HALO_NOTES.md, docs/development/user-isolation-design.md - .github/workflows/build-cpu.yml, build-cuda-windows.yml, build-vulkan.yml

* dsv4 hc-ops * add missing files; * add cparams * update rpc version * address review comments * address review comments

Merge upstream ggml-org/llama.cpp master (876a432) into fork master. Fork point was 2026-07-06 (20a04b2): 330 commits behind, 965 ahead. 37 files conflicted; all resolved by hand (never --theirs), plus fixes for cleanly-merged files that referenced APIs changed elsewhere. Brings in DSpark speculative decoding (8407527) and the DeepSeek V4 work that landed after our fork point, including the fused hyper-connection ops (0dc74e3, ggml-org#25585), DS4 seq_rm fix, graph-split reduction, MTP tensor loading, dflash K/V rotation and sidecar auto-download. Upstream removals that forced fork-side ports: * -sm row / CUDA split buffers removed (74976e1). Dropped our copy of the split-buffer implementation (no fork code in it) and rewrote ggml_cuda_mul_mat onto upstream's early-return dispatch, re-inserting the ML8_FP8 route, the F8_E4M3 guard and the TQ4_1S/TQ3_1S kernels. ggml_cuda_Memcpy2DPeerAsync is kept -- it carries our no-P2P host staging. * mmq.cuh rewritten upstream (<type,J,fallback>, y_scale, stream-k helper). Ported the MAD-88 routed-expert pointer hook to the new kernel signature, args struct and both launch sites. * WMMA flash-attention kernel deleted upstream; our RDNA4 path now leads. * use_mmap/use_mlock/use_direct_io collapsed into llama_load_mode. Weight paging now strips only the mmap bit instead of forcing no-mmap. * llama_context auto-FA/GDN resolution folded into resolve_fused_ops(); the WP attention-island guard is re-injected there, scoped to flash-attn. * server: draft/MTP context now built by common_speculative_init_from_params, so --spec-draft-n-ctx and the MTP tier-disable move onto params_dft; slot memory ops go through slot.mem; migrated two subprocess sites to common_subproc; folded our byte-backpressure into upstream's server_pipe (upstream's drops the oldest item, which would corrupt a proxied body). Fork-visible behaviour changes: * GGML_TYPE_Q2_0 is 56 here, not upstream's 42 -- 42 is our TURBO3_0 and type ids are on-disk. Our turbo/ml8 GGUFs stay readable; an upstream Q2_0 GGUF needs reconversion. gguf-py kept in sync. * DS4 per-layer output renamed l_out -> l_last upstream; added l_last to the WP FFN-island pin or it silently stops firing on DeepSeek V4. * Dropped our -ffast-math on HIP: upstream sets -funsafe-math-optimizations instead because -ffast-math implies -ffinite-math-only, which breaks ggml's INFINITY masking. Do not reinstate. * --flash-attn off with a turbo/quantized cache is now an error rather than a silent override, matching upstream. * DFlash conversion delegates to the target model's vocab class instead of hardcoding the deepseek-v3 pre-tokenizer. ABI: llama_model_params gained load_mtp. Every binary that consumes it must be rebuilt on BOTH machines (llama, llama-server, llama-wp-expert-worker, test-wp-expert-worker) -- a partial target list is what crash-looped the fleet on 2026-07-30. Verified: no conflict markers; fork-marker counts vs backup flat or up (kv-tier, wp_, weight_pager, mt_pagedattn, mt::, MAD-, routed_expert, ml8, spec-draft-n-ctx); every hip_xdev function survives; all fork flags still register in arg.cpp; CPU build (llama, llama-server, llama-cli) clean. Backup: backup/pre-upstream-sync-2026-07-31 (e6b6856). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018kbRS3KuJquSpjpZXdtSNp

* dsv4 hc-ops * add missing files; * add cparams * update rpc version * address review comments * address review comments

Merges 251 upstream commits on top of the fork's 392. Base was 22b208b (2026-07-15). What this brings in for DeepSeek V4: - CUDA kernels for the hyper-connection ops and the lightning indexer (dsv4-hc.cu, lightning-indexer.cu, upstream ggml-org#25585 and ggml-org#25545). These landed upstream after our base, so the graph no longer needs a CPU fallback for those ops. - MTP and DSpark support (ggml-org#25784), the wo_a reshape fix on load, and the same-K/V-cache-type enforcement (ggml-org#25871). - Exclusion of the i32 ffn_gate_tid2eid routing table from quantization, which the fork did not carry. Conflict resolution kept both architectures everywhere the two sides touched the same code: - llama-kv-cache: kept the fork's default-off attention-rotation policy and its env overrides, took upstream's GLM_DSA addition to the DSA indexer arch list. - llama-context: moved the TurboQuant flash-attention auto-enable above upstream's generic quantized-V check, which would otherwise reject turbo cache types under -fa off, and dropped the fork's older V-cache check in favour of upstream's. - mmq.cuh: kept the fork's int64 offsets in all three of upstream's new NVFP4 branches. - fattn.cu: dropped the WMMA block, since upstream removed that kernel and its helpers entirely; kept the RDNA4 turbo path. - ggml-cuda.cu: kept the host-staged cross-device copy and routed its peer copy through upstream's new virtual-to-physical device mapping. - chat.cpp: rebuilt on upstream's file with the fork's Inkling and Laguna parsers and the leading-whitespace tolerance reapplied; thinking_end_tag became thinking_end_tags upstream. - laguna.cpp/laguna.py and mtmd-image.cpp: took upstream, which already carries the fork's own upstreamed review fixes plus later refinements. - Removed the inherited upstream workflows again, per 0c9a069. GGML_OP_COUNT is 103: upstream's 101 plus the fork's TURBO_WHT and FLASH_ATTN_EXT_BANDED. Also drops a duplicate LLM_ARCH_LAGUNA case in test-llama-archs that the merge would otherwise have left in moe_mandatory.

* dsv4 hc-ops * add missing files; * add cparams * update rpc version * address review comments * address review comments

am17an added 3 commits July 12, 2026 13:50

dsv4 hc-ops

eba2cf2

add missing files;

c9a7c7b

add cparams

7b2b731

am17an requested review from a team, CISC, JohannesGaessler and ggerganov as code owners July 12, 2026 11:54

github-actions Bot added model Model specific testing Everything test related ggml changes relating to the ggml tensor library for machine learning CUDA Related to the CUDA backend labels Jul 12, 2026

update rpc version

7e92401

drrros mentioned this pull request Jul 12, 2026

Eval bug: DSV4-Flash churned-reuse SWA KV-cache exhaustion (crash + stall) #25452

Open

JohannesGaessler reviewed Jul 13, 2026

View reviewed changes

Comment thread ggml/src/ggml-cuda/dsv4-hc.cu Outdated

Comment thread ggml/src/ggml-cuda/dsv4-hc.cu Outdated

Comment thread ggml/src/ggml-cuda/dsv4-hc.cu Outdated

address review comments

4360e1d

ggerganov self-assigned this Jul 16, 2026

ggerganov reviewed Jul 16, 2026

View reviewed changes

Comment thread src/models/deepseek4.cpp Outdated

Comment thread src/models/deepseek4.cpp Outdated

Comment thread tests/test-backend-ops.cpp Outdated

Comment thread ggml/include/ggml.h Outdated

address review comments

b9a885c

ggerganov approved these changes Jul 16, 2026

View reviewed changes

am17an requested a review from fairydreaming July 16, 2026 15:53

fairydreaming approved these changes Jul 16, 2026

View reviewed changes

am17an merged commit 0dc74e3 into ggml-org:master Jul 16, 2026
26 of 31 checks passed

eapache mentioned this pull request Jul 17, 2026

Eval bug: GGML_ASSERT(n_inputs < GGML_SCHED_MAX_SPLIT_INPUTS) failed on DS v4 Flash on single GPU #25800

Closed

fairydreaming mentioned this pull request Jul 17, 2026

tests : initialize all tensors in test_dsv4_hc to avoid NaNs in sentinel tensors #25822

Merged

SamuelOliveirads mentioned this pull request Jul 18, 2026

IK's experiments with DS4 ikawrakow/ik_llama.cpp#2147

Closed

CowboyTim pushed a commit to aardbeiplantje/llama.cpp that referenced this pull request Jul 21, 2026

DeepseekV4: Add fused hyper-connection ops (ggml-org#25585)

f58a354

* dsv4 hc-ops * add missing files; * add cparams * update rpc version * address review comments * address review comments

This was referenced Jul 22, 2026

ggml: add DeepSeek V4 hyperconnection + KV ops (CPU) #23122

Open

ggml : add fused SINKHORN_NORM op (Sinkhorn-Knopp normalization) #25421

Closed

ggerganov pushed a commit to am17an/llama.cpp that referenced this pull request Jul 28, 2026

DeepseekV4: Add fused hyper-connection ops (ggml-org#25585)

59d5f8d

* dsv4 hc-ops * add missing files; * add cparams * update rpc version * address review comments * address review comments

smalinin pushed a commit to smalinin/llama.cpp that referenced this pull request Aug 4, 2026

DeepseekV4: Add fused hyper-connection ops (ggml-org#25585)

5f406f6

* dsv4 hc-ops * add missing files; * add cparams * update rpc version * address review comments * address review comments

Navigation Menu

Uh oh!

DeepseekV4: Add fused hyper-connection ops - #25585

DeepseekV4: Add fused hyper-connection ops#25585
am17an merged 6 commits into
ggml-org:masterfrom
am17an:hc_ops

am17an commented Jul 12, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jadenmach2 commented Jul 13, 2026

Uh oh!

am17an commented Jul 13, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ggerganov commented Jul 17, 2026

Uh oh!

fairydreaming commented Jul 17, 2026

Uh oh!

am17an commented Jul 17, 2026

Uh oh!

fairydreaming commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

am17an commented Jul 12, 2026

Overview

Additional information

Requirements

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jadenmach2 commented Jul 13, 2026

Uh oh!

am17an commented Jul 13, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ggerganov commented Jul 17, 2026

Uh oh!

fairydreaming commented Jul 17, 2026

Uh oh!

am17an commented Jul 17, 2026

Uh oh!

fairydreaming commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants