Wip/deepseek v4 support - #22378
Closed
nisparks wants to merge 81 commits into
Closed
Conversation
github-actions
Bot
added
model
|
Are you aware of https://github.com/antirez/llama.cpp-deepseek-v4-flash ? |
|
The values in |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Improve DeepSeek V4 conversion hot paths and add generalized converter controls for writer buffering, temp-file copying, and PyTorch thread tuning. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add the upstream-based DeepSeek V4 runtime graph, memory path, activation parity ops, and CUDA smoke-performance fixes on top of the existing GGUF/native mixed FP8/MXFP4 conversion support. Load the MoE routing scale metadata required for sane outputs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Serialize DeepSeek4 runtime memory for prompt-cache and checkpoint paths, validate tensor metadata on restore, and make sequence clears reset runtime tensor data instead of leaving stale compressed state behind. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add DeepSeek4-shaped F8 performance cases and use direct CUDA bit construction for E8M0 block-scale decode to avoid the CUDA 12.8 BF16 conversion path on NVIDIA devices. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> (cherry picked from commit feb527a)
Bring the validated WIP-compatible performance pieces from the experimental branch into DeepSeek4 support: restore the fast F8 MMVQ VDR and shared-LUT path, specialize Q8_1 activation quantization for native F8/FP4 matvecs, and reduce DeepSeek4 graph overhead with fast top-k, sum_rows expert reduction, and fewer unnecessary contiguous/repeat nodes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add F8 MUL_MAT_VEC_FUSION coverage and make perf mode repeat whole fusion graphs instead of only the final output node. Reuse the shared F8 decode LUT for non-small-K fused MMVQ paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use a warp-local top-k kernel for common small expert-count shapes so TOP_K avoids the argsort fallback on DeepSeek4 routing shapes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use two output rows per CUDA block for one-token, non-small-K F8 MMVQ. This preserves other quantized paths while reducing full-model F8 kernel time in the DeepSeek4 layer-split profile. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use an explicit inverse scale for Q8_1 activation quantization so quantized matvec setup multiplies by the reciprocal scale instead of dividing by the stored scale in every lane. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use a smaller copy transpose tile and a 512-thread RMSNorm launch for exact 1024-column rows in the CUDA backend. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reshape the hc_post vector directly to the broadcast shape instead of materializing a transposed copy. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds a DeepSeek4-specific hot-expert manager that, when DS4_HOT_PROFILE_JSON is set in the environment, reads a per-layer hot-expert ID profile (produced by ds4-expert-profile + ds4-hot-experts.py) and extracts the K hot rows of each CPU-resident MoE expert tensor into a separate GPU buffer. This is Phase 1 of the topic-aware hot-expert pinning architecture. Phase 2 (dual hot/cold mul_mat_id dispatch in build_expert_mix) is still pending. Behaviour: - Reads DS4_HOT_PROFILE_JSON env var; no-op if unset - Skips already-GPU-resident layers (e.g., layers covered by -ot or default -ncmoe placement) so the saved VRAM goes only toward layers that were CPU-MoE-bound - Spreads hot-tensor allocations across CUDA0/CUDA1/CUDA2 using a per-device free-memory budget tracker, picking the device with the most remaining capacity each tensor; gracefully drops layers that don't fit - Tolerates the params-fit memory probe by skipping layers whose tensors haven't been backed yet - Handles both combined (ffn_gate_up_exps) and separate (ffn_gate_exps + ffn_up_exps) MoE tensor shapes; DS4-Flash uses the separate variant Verified with hot-code-k16.json on the current best config (3-GPU, -ncmoe 29): ds4-hot: pinned hot experts for 19/29 CPU-MoE layers, ~3876 MiB on GPU across 3 buffers (k=16, category=code) Server still produces correct output and the unused hot tensors do not affect throughput (PP 19.20 / TG 18.11), confirming the loader is benign when not yet wired to compute. Phase 2 work (next): in src/models/deepseek4.cpp::build_expert_mix, add a parallel mul_mat_id branch using these hot tensors with masked routing weights so hot-pick activations only flow through the GPU subset. The hot manager exposes per-layer state via ds4_hot::instance().get(il). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Wires the GPU-pinned hot expert tensors from Phase 1 into the model graph so that on each MoE layer the K hot picks are computed on GPU against a small K-expert subset and only the cold picks are computed on CPU against the original 256-expert tensor (with hot picks redirected to a single sentinel cold expert that mul_mat_id dedupes). Implementation outline (src/models/deepseek4.cpp::build_expert_mix): hot_ids = get_rows(hot_remap_table_GPU, sel_flat) -> [0, K) cold_ids = get_rows(cold_remap_table_CPU, sel_flat) -> [0, N) cold is_hot = get_rows(is_hot_mask_GPU, sel_flat) -> 0/1 is_cold = get_rows(is_cold_mask_CPU, sel_flat) -> 0/1 out_h = mul_mat_id(hot_*_exps_GPU, x, hot_ids) ... swiglu, down, etc. out_c = mul_mat_id(layer.ffn_*_exps_CPU, x, cold_ids) ... same out = out_h * weights * is_hot + out_c * weights * is_cold Hot tensors are extracted at load time with a +1 padding slot at the end to give the CUDA mmq kernel safe room to prefetch past the last hot expert (mirrors the LRU MoE cache layout). Without this padding the kernel crashes with an illegal memory access. Hot-side lookup tables live on the same GPU as the hot weights; cold-side tables live on CPU so the cold mul_mat_id stays on CPU. Three env-driven diagnostic toggles let us bisect the dispatch: DS4_HOT_DISPATCH=0 - skip dispatch entirely (Phase 1 only) DS4_HOT_DISPATCH_MODE=cold - cold path only (hot output zeroed) DS4_HOT_DISPATCH_MODE=hot - hot path only (cold output zeroed) DS4_HOT_USE_FULL_WEIGHTS=1 - hot path uses full CPU tensor Status: - Phase 1 alloc + dispatch infrastructure: clean. - Cold-only path with cold_remap: clean, correct output, PP/TG ~ baseline. - Hot-only path with K+1 padding: clean (output is partial, masked). - Dual hot+cold dispatch: works for some prompts (e.g. 'hi', 'What is 17 + 25' without the question mark - returns correct '42' with PP=22.07) but crashes on others (e.g. with the '?' tail token) in launch_mul_mat_q with an illegal memory access. The crash is prompt-content sensitive, suggesting a specific expert-ID routing pattern triggers a residual scheduler / kernel issue. - The default behaviour for users who set DS4_HOT_PROFILE_JSON is the full dual dispatch path, which means Phase 2 is currently NOT production-safe; users should keep using the profile loaded at K=16 with no Phase 2 dispatch (DS4_HOT_DISPATCH=0) until the bug is resolved. Next debugging steps: - Identify the specific token / expert-ID pattern that triggers the crash. The '?' token routes through a particular mix of hot/cold experts that breaks something downstream. - Capture the actual kernel that crashes by running with CUDA_LAUNCH_BLOCKING=1 and a tighter inference loop. - Investigate whether multi-GPU placement (hot tensors spread across CUDA0/CUDA1/CUDA2) interacts badly with the scheduler when both hot path and cold path consume the same activation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two safety/debug improvements following the Phase 2 prompt-content-sensitive
crash investigation:
1. Phase 2 dispatch is now OPT-IN (DS4_HOT_DISPATCH=1 to enable). Previously
the dispatch ran by default whenever DS4_HOT_PROFILE_JSON was set. Since
the crash on certain prompts is unresolved, defaulting to opt-in keeps
Phase 1 (alloc only) safe for users who just want the +55% PP win that
Phase 1 alone delivers.
2. Added DS4_HOT_DEVICE=CUDAN env var to pin all hot tensors onto a single
GPU for debugging. Tested with DS4_HOT_DEVICE=CUDA1 + DS4_HOT_DISPATCH=1
on the failing prompt: still crashes, ruling out multi-device scheduler
interactions as the cause of the residual bug.
Bisection results:
- DS4_HOT_DISPATCH=1 -> crashes on '?' prompt
- DS4_HOT_DISPATCH=1 DS4_HOT_USE_FULL_WEIGHTS=1
-> works, PP 35-37, TG 17-19
(math approximate but model output
is coherent and correct on the
test prompts)
- DS4_HOT_DISPATCH=1 DS4_HOT_DEVICE=CUDA1 -> still crashes
- DS4_HOT_DISPATCH=1 DS4_HOT_DISPATCH_MODE=cold
-> works (cold-only path verified)
- DS4_HOT_DISPATCH=1 DS4_HOT_DISPATCH_MODE=hot
-> works on safe prompts but
produces partial output (cold
contribution masked to 0)
Conclusion: the crash is specifically caused by the GPU-resident K-subset
tensor + the helper kernel's expert-bounds processing for certain expert ID
distributions. The dispatch graph topology and masking math are confirmed
correct. Future work needs to either (a) fix the K-subset tensor's
per-expert padding to fully match the LRU-cache layout, or (b) instrument
ggml_cuda_launch_mm_ids_helper to identify which specific input pattern
trips the illegal memory access.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The previous 256 MiB margin was too small: prompts longer than ~30 tokens need ~1.1 GiB compute buffers per device for the batched-prefill path (LLAMA_DEEPSEEK4_BATCH_PREFILL=1) and would OOM with a Phase 1 profile loaded. Also adds DS4_HOT_MARGIN_MIB env override for tuning per workload (e.g., users with shorter typical prompts can lower it to pin more layers). Trade-off: with the larger margin, fewer layers fit (e.g., 3 instead of 18 at K=16 on the current 3-GPU config). This is necessary for the server to remain stable on real-world prompts. NOTE: empirical re-testing confirmed that Phase 1 alone (alloc without Phase 2 dispatch) provides effectively zero throughput benefit. Both 'profile loaded' and 'no profile' modes converge to the same PP/TG after warmup. The earlier +55% PP claim was measurement variance from comparing a cold first-run to a warm later-run on different prompts. Phase 2 dispatch remains the only path to actual throughput gain, but still has the unresolved crash on certain expert ID patterns. Users who do not also set DS4_HOT_DISPATCH=1 should leave DS4_HOT_PROFILE_JSON unset to avoid pointlessly consuming VRAM. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Root cause (kudos to rubber-duck for the diagnosis): the CUDA
mm_ids_helper kernel (ggml/src/ggml-cuda/mmid.cu lines 43-103) dedups
(token, expert) pairs - it stores at most one iex_used per token-expert.
When my hot remap mapped multiple cold picks within a token to the same
sentinel hot-index 0, the helper compacted those to a single entry. But
the downstream quantize_mmq_mxfp4_cuda was still launched for the full
P*T rows (P=n_expert_used=6, T=n_tokens) and read uninitialized tail
entries of ids_src1, hitting illegal memory access when those garbage
indices pointed past the activation tensor.
This was reproducibly prompt-content sensitive: prompts whose top-K
expert routings happened to put many picks in cold experts (mapping all
to sentinel 0 in the hot path) tripped the bug, while prompts that hit
mostly-hot experts did not.
Fix: per-pick unique dummy expert IDs in the hot path.
- Hot tensor now allocated with K + P + 1 expert slots:
[0, K) - real hot experts (extracted from CPU)
[K, K+P) - per-pick dummy zero-weighted experts
[K+P] - trailing prefetch padding slot
The dummy slots are zero-initialized, so cold picks routed there
contribute zero to the output naturally - no output mask needed on the
hot path.
- Lookup tables changed to f32 so the graph can do per-pick arithmetic:
hot_remap_table[e] = remap_hot[e] (in [0,K)) for hot, K for cold
cold_remap_table[e] = e for cold, 0 for hot
is_hot_mask[e] = 1.0 if hot else 0.0
is_cold_mask[e] = 1.0 if cold else 0.0
hot_pick_arange = [0, 1, ..., P-1] (per-pick offset)
cold_pick_sentinel = [cold_ids[0], ..., cold_ids[P-1]] (per-pick
cold sentinels - defensive uniqueness for the
CPU mul_mat_id too)
- Dispatch graph constructs per-pick unique IDs:
hot_ids[k,t] = hot_remap[sel] + is_cold[k,t] * hot_pick_arange[k]
= remap_hot[sel] (hot) or K + k (cold, unique per pick)
cold_ids[k,t] = cold_remap[sel] + is_hot[k,t] * cold_pick_sentinel[k]
= sel (cold) or cold_ids[k] (hot, unique per pick)
Then ggml_cast(*, GGML_TYPE_I32) to feed mul_mat_id.
- Also handles the warmup/reserve graph case where selected_experts has
ne[0] = n_expert (256) instead of n_picks (6) - falls back to the
single-path code in that case (would assert in ggml_mul otherwise).
- llama-context.cpp passes model.hparams.n_expert_used to the hot
manager so it knows the right P for the dummy expert allocation.
Verified correctness: 'What is 17 + 25?' -> '42', 'What is 18 + 25?'
-> '43', and high-quality Fibonacci-with-type-hints Python code, all
with DS4_HOT_DISPATCH=1 and no CUDA errors.
Performance status: only 3/29 CPU-MoE layers fit in the current 1.5 GiB
margin per GPU at K=16, so the dispatch covers a small fraction of the
expert work and end-to-end throughput is similar to baseline. Smaller K
or tighter VRAM tuning will increase coverage; that is a separate
optimization on top of this correctness fix.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The CPU mul_mat_id correctly handles duplicate expert IDs within a token (its matrix_row_counts dedup), so the cold path can use a single shared cold sentinel for all hot picks within a token. This collapses the bandwidth-saving dedup (multiple hot picks all map to one cold sentinel expert in the cold path). The CUDA mm_ids_helper bug only affects the GPU hot path; per-pick unique dummy experts there are still required. Result on real long-context (~11k token) prefill: - Baseline (no profile, no dispatch): PP=17.50, TG=14.94 - Dispatch with per-pick cold sentinels (10 layers): PP=16.77, TG=14.55 - Dispatch with shared cold sentinel (10 layers): PP=16.74, TG=14.57 So even with the correct CPU bandwidth saving, the dispatch with current 10/29 layer coverage does not improve long-context PP. The upper bound on Phase 2 gain is limited by: - DeepSeek4's expert routing is approximately uniform (top-32 covers only ~66% of activations - per perf_observations note deepseek4-routing-skew-by-prompt-type-20260501) - Cold path with 10 of 29 layers covered still does most of the work - Hot path adds GPU dispatch overhead that's not amortized at current K=16 with small batch counts per chunk - For batched prefill chunks of 2048 tokens, nearly all 256 experts are activated by at least one (token, pick) combination so the dedup trick saves only ~6% bandwidth at most Phase 2 dispatch remains opt-in (DS4_HOT_DISPATCH=1). Default behavior is unchanged for users. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When non-standard attention architectures (e.g. DeepSeek V4 Flash CSA+HCA) are used, llama_memory_seq_pos_min() may return -1 even with n_past > 0. The custom KV cache layout is incompatible with standard prompt cache restoration, causing a hard crash. Replace GGML_ABORT with graceful fallback: set n_past=0 and pos_next=0 to force full prompt re-evaluation, same as the SWA/hybrid memory path. Verified: 60+ concurrent requests on 8xA100, zero crashes. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The AVX2 fast path of ggml_vec_dot_mxfp4_q8_0 explicitly expanded the
mul_add_epi8 + madd_epi16 + cvtepi32_ps chain instead of going through
mul_sum_i8_pairs_float, so AVX-512 VNNI / AVX-VNNI / AVX-VNNI-INT8
hosts (Zen 4, Sapphire Rapids, Granite Rapids, ...) fell back to the
non-VNNI path even when those instructions are available.
Switching to mul_sum_i8_pairs_float lets the existing dpbusd helpers
do the work in one fused instruction and removes the dead 16-bit
intermediate. test-backend-ops MUL_MAT mxfp4 still passes against the
CUDA reference on a non-VNNI host, so fallback semantics are
preserved.
Verified by compiling ggml/src/ggml-cpu/arch/x86/quants.c with
"-march=znver4 -mavx512vnni -mavx512vl" (the toolchain's default with
GGML_NATIVE=ON on a Zen 4 host) and inspecting the inner loop of
ggml_vec_dot_mxfp4_q8_0:
before (wip/deepseek-v4-support head):
vpdpbusd: 0
vpmaddubsw: 2 (legacy AVX2 path emitted)
after (this commit):
vpdpbusd: 2 (VNNI fused mul-add active)
vpmaddubsw: 0
GGML_TYPE_W4A16_AUTOROUND and GGML_FTYPE_MOSTLY_W4A16_AUTOROUND were referenced from the type-traits and ftype-to-type tables but were never declared in ggml.h or backed by quantize/dequantize routines, which broke the C build on this branch. Drop the stub entries so the branch builds. The type can be reintroduced as a complete change if and when the AutoRound W4A16 support is plumbed all the way through.
deepseek4_batch_prefill_enabled() is opt-in via env var, and its split helper (split_seq_deepseek4_prefill) is not present on this branch. Force the loop to use split_seq(1) regardless until that helper is restored, so the build link succeeds and the default code path (single-token ubatches) still works.
Two field-feedback fixes for the DeepSeek V4 conversion path: 1. The conditional registration of torch.float8_e8m0fnu populated _dtype_byteswap_map and _dtype_str_map but missed _dtype_map (the one used by LazyTorchTensor.numpy()). On torch >= 2.7, this surfaced as a KeyError when the converter materialized an E8M0 scale tensor. Add the third entry so e8m0 round-trips work. 2. transformers does not (yet) recognize model_type=deepseek_v4, so AutoTokenizer.from_pretrained() in DeepseekV2Model.set_vocab fails inside AutoConfig before tokenizer files are touched. The V4 tokenizer is a plain PreTrainedTokenizerFast, so override set_vocab to fall back to a direct PreTrainedTokenizerFast load if the parent path fails.
The auto-FA logic in llama_context::sched_reserve runs a worst-case
graph with n_tokens=1 and inspects each FLASH_ATTN_EXT op to decide
whether FA can stay on the layer's KV device. For DeepSeek V4 this
check has been failing for two compounding reasons:
1. build_attn_v4 only created kq_mask when work_tokens > 1, so the
reservation graph (work_tokens=1) called build_attn_mha with
mask=nullptr.
2. The CUDA FA kernel for K[0]=512 (V4's MLA latent KV) only fires
when 'gqa_opt_applies' is true, which requires a non-null mask
plus K->ne[1] % FATTN_KQ_STRIDE (256) == 0.
With both conditions failing on the reservation graph, CUDA reported
the FA op as unsupported, the scheduler placed it on the CPU backend,
and the device-mismatch detector disabled FA globally for the entire
context. Every subsequent attention call then ran through the dense
Q*K^T -> softmax -> *V path, which materializes a [work_tokens,
n_kv_total] matrix per layer per ubatch.
This commit:
- Always allocates kq_mask of shape [n_kv_total_padded, work_tokens],
even for work_tokens=1.
- Pads n_kv_total up to a 256 multiple via ggml_pad on kv_prefix.
The padded K/V slots are masked out with -INFINITY in the new
set_input pass, so they contribute zero to softmax/V.
- Shares one kq_mask per (n_kv_total_padded, work_tokens) shape
across all V4 layers via a small map keyed in the input object.
With 43 layers each creating its own mask we previously hit
GGML_SCHED_MAX_SPLIT_INPUTS (30); after the dedup we usually
end up with one or two unique shapes.
- Casts the mask to F16 when cparams.flash_attn is set, matching
the standard llama-graph kq_mask handling.
Local validation on a 3-GPU host (2x RTX 3090 + 1x 2080 Ti):
- sched_reserve now logs 'Flash Attention was auto, set to enabled'
for V4 (previously: 'set to disabled').
- test-llama-archs -a deepseek4 still passes for all 4 backend
configurations (NMSE 1.63e-12, Roundtrip OK).
- llama-cli on a 4000-word prompt: PP 34.7 / TG 29.6 t/s with
FA on, vs 32.5 / 26.6 t/s with FA off (same build).
ggml_compute_forward_hc_weighted_sum and ggml_compute_forward_sinkhorn_4x4 asserted on src0->ne[2] == 1, which limited V4's per-layer Hadamard weighted-sum and 4x4 routing-combine ops to single-token ubatches. This was harmless under the current single-token prefill but blocks any future batched-prefill work in the V4 graph (HC_WEIGHTED_SUM is called from build_attn_v4's weighted_sum_hc helper for any ubatch with work_tokens > 1; SINKHORN_4X4 is called per layer in build_moe_v4). Both ops are now batched along ne[2]: HC distributes work across (n_embd * n_batch) output elements; SINKHORN distributes the per-batch 4x4 problems across worker threads. Single-token behaviour is preserved bit-for-bit.
Until now V4 prefill processed every prompt token in its own ubatch.
For a 4k token prompt this meant 4000 separate compute graphs, each
incurring kernel-launch and graph-scheduling overhead. Per-token cost
ended up dominated by infrastructure rather than actual compute.
This commit teaches the V4 graph builder, memory module, and indexer
path to handle work_tokens > 1, gated on the existing
LLAMA_DEEPSEEK4_BATCH_PREFILL=1 env var:
- llama_memory_deepseek4::init_batch() now uses
balloc.split_seq(n_ubatch) when batch_prefill is on AND the batch
is a real prefill (n_outputs != n_tokens). Decode and the legacy
path keep the single-token semantics so logit reads stay in
bounds.
- compression_ape_rows handles any (start_pos, work_tokens)
combination by decomposing the slice into a partial start
window + complete windows + partial end window and concatenating
the corresponding ape rows. The previous code only handled the
aligned multi-window case and aborted otherwise.
- The indexer scoring path keeps work_tokens as a separate
dimension through the hadamard mul_mat and FP4 quant, then
aggregates per-query scores into a single ubatch-wide top-k by
summing over (indexer_n_head * work_tokens) at the end. This is
a 'shared top-k' approximation: every query in the ubatch
attends to the same selected compressed prefix. Adjacent prefill
tokens overwhelmingly want the same top-k slots so the loss is
in practice small, and it lets the rest of the attention path
stay batch-friendly.
- The non-indexer single-window path was already mostly batch-safe
via build_attn_mha; we just needed the kq_mask + KV padding from
the FA-unblock fix and the matching CPU op support for HC and
sinkhorn from the previous commit.
Local benchmark on a 3-GPU host (2x RTX 3090 + 1x 2080 Ti, IQ1_S DSv4
fully on GPU, -fa on, -ub 64, single-stream prefill):
pp single-token batched speedup
----- ------------ ------- -------
256 32.24 71.46 2.22x
1024 33.94 109.30 3.22x
4096 30.01 153.26 5.11x
8192 28.78 178.38 6.20x
16384 28.42 191.67 6.74x
test-llama-archs -a deepseek4 with LLAMA_DEEPSEEK4_BATCH_PREFILL=1
still passes for all 4 backend configurations (NMSE 1.70e-12,
roundtrip OK). The synthetic test fixture uses n_outputs=n_tokens
so the new code path is exercised by the runtime smoke test, not
the test-archs fixture; both behave identically there.
Inverts the LLAMA_DEEPSEEK4_BATCH_PREFILL gate so users get the ~7x
prefill speedup without setting an env var. The escape hatch for the
single-token path is now LLAMA_DEEPSEEK4_BATCH_PREFILL=0.
Final benchmark on 2x RTX 3090 + 1x 2080 Ti, IQ1_S DSv4 fully on GPU,
-fa on, -ub 128:
pp single-token batched speedup
----- ------------ ------- -------
256 32.10 81.35 2.53x
1024 33.44 121.37 3.63x
4096 29.90 170.05 5.69x
8192 28.87 197.57 6.84x
16384 28.46 210.42 7.39x
Decode (tg64) is unaffected: 26.76 t/s with batched, 26.56 t/s
without (within noise). The init_batch gate also requires
n_outputs != n_tokens before splitting multi-token ubatches, so
generation correctly stays on the single-token path.
NMSE vs CPU: 1.85e-12 across all four backends (within noise of the
single-token baseline 1.64e-12).
… budget
Two fixes that together raise the safe ubatch ceiling for batched prefill:
1) Indexer score collapse-Q: when work_tokens > 1, sum-reduce indexer_q
along the work_tokens axis BEFORE the score mul_mat instead of after.
The previous per-query path materialized a [n_comp, n_head, work_tokens]
intermediate that grew to ~134 MB per V4 layer at ub=512; with 21
r=4 layers that easily exceeded the GPU budget and OOMed at ub=512+.
Collapsing first turns each indexer score op back into the same
2D shape decode uses ([n_comp, n_head]). The approximation is
small since shared top-k already collapses across queries.
2) graph_max_nodes bump for DSv4: max(n_tokens * 256, 128*n_tensors)
-> max(n_tokens * 512, 256*n_tensors) so the ggml metadata pool
no longer hits GGML_ASSERT(obj_new) on the bigger ubatch graphs.
Test results on EPYC 7C13 + 2x RTX 3090 + RTX 2080 Ti (mixed CPU/GPU,
IQ1_S model, batched prefill enabled):
pp=4096:
ub=128: 245 t/s
ub=384: 328 t/s (was 312)
ub=512: 340 t/s (was OOM)
ub=768: 346 t/s (NEW)
ub=1024: 349 t/s (NEW best)
pp=8192 ub=768: 362 t/s
pp=16384 ub=512: 364 t/s
NMSE versus CPU still passes with the new collapse: 1.27e-12 (better
than the old per-query path's 1.63e-12).
CUDA graph capture/instantiation memory budget is exceeded on the
DeepSeek V4 batched prefill graph at ub>=768 on dual RTX 3090 + 2080 Ti,
producing 'CUDA error: out of memory' before the graph can run, even
though direct execution of the same graph fits in VRAM.
Detect by scanning all non-view ops in the cgraph and looking at any
operand or output dimension >= 384. When a wide prefill ubatch is
present, fall back to direct execution.
This recovers the speedup that was previously only available with a
manual GGML_CUDA_DISABLE_GRAPHS=1 override:
pp=4096 mixed CPU/GPU:
ub=512: 327 -> 340 t/s
ub=768: OOM -> 346 t/s
ub=1024: OOM -> 349 t/s
pp=8192 ub=768: 362 t/s (was OOM)
pp=16384 ub=512: 364 t/s
The threshold leaves single-token decode (max ne[d]=1) and modest
prompt batches (ub<=256) entirely on the existing CUDA graph path.
…_4X4 The DeepSeek V4 batched-prefill graph emits HC_WEIGHTED_SUM and SINKHORN_4X4 with shapes [n_embd, hc_mult, n_batch] and [4, 4, n_batch] respectively (n_batch == work_tokens). The CUDA kernels asserted ne[2] == 1, so when work_tokens > 1 the scheduler had to fall back to CPU for these ops, forcing GPU<->CPU sync per layer per ubatch. Extend both kernels to handle n_batch > 1: HC_WEIGHTED_SUM: launch grid is now (n_embd/block_size, n_batch). Each block handles one batch index using src0->nb[2], src1->nb[1], dst->nb[1] strides. SINKHORN_4X4: launch grid is (ceil(n_batch/64),) with 64 threads per block. Each thread handles one independent 4x4 problem. Update the corresponding supports() entries to advertise the new shapes. Single-batch (decode) shape continues to work unchanged. Measured prefill on EPYC 7C13 + 2x RTX 3090 + 2080 Ti, IQ1_S, mixed CPU/GPU, batched prefill on, before/after this commit: pp=4096 ub=512: 340 -> 411 t/s (+21%) pp=4096 ub=1024: 349 -> 426 t/s (+22%) pp=8192 ub=768: 362 -> 447 t/s (+23%) pp=16384 ub=512: 364 -> 440 t/s (+21%) pp=16384 ub=768: -- -> 460 t/s (NEW) NMSE versus CPU still passes (2.04e-12).
The compression update inside build_attn_v4 used to loop n_comp_windows times (up to work_tokens/comp_ratio = 256 iterations per layer at ub=1024) emitting ~13 graph nodes per iteration per layer per ubatch. With 21 indexer-eligible layers and the same loop firing for both the attention and the indexer compression, this added ~70k graph nodes per ubatch at ub=512 and dominated graph build / dispatch overhead. Replace the loop with a single batched pass that builds the per-window [head_dim, 2*comp_ratio, n_comp_windows] tensor via two strided 3D views (prev half + cur half) of comp_kv/comp_score, concatenates them along dim 1, and runs the soft-attention pool, RMS norm, RoPE, FP8/FP4 quant, and set_rows on all windows at once. The host already provides the position and cache-index vectors; we strided-view them at offset (comp_ratio - 1) with stride comp_ratio to pick out the slot for each window's last token. Removing both loops cuts graph node count for an ub=512 prefill ubatch from ~86k to ~16k. Measured prefill on EPYC 7C13 + 2x RTX 3090 + 2080 Ti, IQ1_S, mixed CPU/GPU, batched prefill on, before/after this commit: pp=4096 ub=512: 411 -> 565 t/s (+38%) pp=4096 ub=1024: 426 -> 629 t/s (+48%) pp=8192 ub=768: 447 -> 622 t/s (+39%) pp=8192 ub=1024: 452 -> 643 t/s (+42%) pp=16384 ub=512: 440 -> 584 t/s (+33%) pp=16384 ub=768: 461 -> 628 t/s (+36%) pp=16384 ub=1024: OOM -> 650 t/s NMSE versus CPU still passes (2.01e-12).
The previous heuristic scanned every dim of every non-view op for values >= 384, which caused decode graphs to be disabled at long context (the FA op's K[1] = n_kv_total can exceed 384 well before prefill ubatch sizes get there) and also tripped on V4's HC_POST batched mixer matmul (output ne[1] = n_embd = 4096 regardless of work_tokens). Replace with a tighter signal: only disable when MUL_MAT_ID's ne[2] >= 384, where ne[2] is exactly work_tokens for MoE expert dispatch. This is the dimension that grows with the prefill ubatch width and is the actual cause of the CUDA-graph capture memory blowup. Decode (work_tokens = 1) and modest prompt batches now keep CUDA graphs enabled regardless of context length. NMSE versus CPU still passes.
The collapse-Q approximation (sum indexer_q across queries before the
score mul_mat) was producing wrong KV slot selections at long context.
Field testing on a 67K-token coding-review prompt showed the model
producing degenerate output ('2.0', 'The code: Yes, this code,...')
where the per-query path produces coherent answers.
Mathematically:
collapse-Q: top_k(relu(kv * sum_q indexer_q) * sum_q index_weights)
per-query: top_k(sum_q (relu(kv * indexer_q) * index_weights))
These are NOT equivalent because of the relu. At long context with
retrieval-style queries, the collapse-Q path picks generic 'average
relevance' KV slots instead of slots specific to the model's queries,
and the model can't recover.
Switch the default back to per-query (which is what was originally
shipped in 94ec5be). Add LLAMA_DEEPSEEK4_INDEXER_COLLAPSE_Q=1 as
an opt-in for tight-VRAM hosts that prefer the speed at the cost of
long-context correctness.
The per-query path uses more VRAM (peak score tensor is
[n_comp, n_head, work_tokens] which scales with the ubatch). On
24GB 3090s long-context per-query needs ub <= 256; on 96GB Blackwell
it should fit at ub=512 (the original field-tester config).
NMSE versus CPU still passes for both modes.
|
Not sure if this branch was only supposed to work with your model on HuggingFace, but I just tried to run this against the official model and ended up getting: Downloading your model now and will try again. |
Models like DeepSeek V4 have a fixed-size sliding-window + indexer KV
state that can't be partially evicted (llama_memory_seq_rm returns
false for partial range removal). When the prompt cache is enabled
and the server tries to do prefix-matched cache reuse on such models,
update_slots() eventually hits
GGML_ABORT('pos_min == -1, but n_past > 0 - should not happen')
because llama_memory_seq_pos_min returns -1 for these models even
when the cache contains data. Symptom is the server happily serving
a few requests then crashing the second time it tries to reuse a
prompt prefix.
common_context_can_seq_rm() already detects the partial-removal
limitation (logging 'the target context does not support partial
sequence removal'). Use that signal at server startup to force
cache_ram_mib = 0, with a clear log line. User can still pass
--cache-ram N to override at their own risk.
Verified on DeepSeek V4: server now stable across many short
multi-prompt sessions where it previously aborted.
fix(server): prevent GGML_ABORT when prompt cache pos_min == -1 for non-standard attention architectures
This was referenced May 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note: this PR is purely for reference. If someone wants to pick up the work to cherry pick individual changes into a proper PR, by all means, go ahead.
Overview
Additional information
Requirements