Truncate prompt if longer than context + n_predict by matthew-mcallister · Pull Request #78 · ggml-org/llama.cpp

matthew-mcallister · 2023-03-13T04:55:27Z

I think this is an improvement over the current behavior of outputting nothing at all when the prompt is too long. It's slightly ugly to see the truncated prompt in the terminal, but it is helpful to see exactly what was included in the prompt so you can tell what was truncated.

Maybe there's a better solution, IDK, just throwing a PR your way.

ggerganov · 2023-03-19T18:18:27Z

I have a better solution in mind. Will close this for now

* Faster q6_0 on AVX2 PP-512 goes up by 3.4%. * q6_0: this is slightly better --------- Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>

@bingh0

…nt (ggml-org#78) Post-attention V-padded reshape in build_attn was using hparams.n_head_kv(il), but cur returned from build_attn_mha has shape (n_embd_head * n_head, n_tokens) — n_head is the Q-head count. On GQA models where n_head != n_head_kv (e.g. Qwen2.5-0.5B with head_dim=64 padded → 128, n_head=14, n_head_kv=2), the reshape element count fails the assertion in ggml_reshape_3d and the process aborts. Symptom: GGML_ASSERT(ggml_nelements(a) == ne0*ne1*ne2) at ggml.c:3656. Reported and diagnosed by @bingh0 in TheTom#78. Verified locally on Qwen2.5-7B (head_dim=128, no padding, regression check passes) and on AMD MI300X with Qwen2.5-0.5B (head_dim=64, was crashing pre-fix). Three sites fixed (lines 2285, 2412, 2532 — same idiom in three build_attn overloads). Closes ggml-org#78. Likely also closes ggml-org#108 (speculative decoding hits the same assertion). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

fix: build_attn V-padded reshape uses Q-head count, not KV-head count (ggml-org#78)

Switched from atomicmilkshake/feature/triattention (KV-cache focus) to TheTom/feature/turboquant-kv-cache (broader scope: KV cache + new weight quants TQ3_1S/TQ4_1S + Vulkan/HIP/Metal coverage + active development). ID renumbering (preserves Q2_0=42 / MOSTLY_Q2_0=41 GGUF ABI): GGML_TYPE_Q1_0_g128 = 41 (= TheTom Q1_0 on disk; semantic match) GGML_TYPE_Q2_0 = 42 (PrismML — DO NOT MOVE, GGUF ABI) GGML_TYPE_Q1_0 = 43 (local 32-block 1-bit) GGML_TYPE_TURBO3_0 = 44 (TheTom KV cache) GGML_TYPE_TURBO4_0 = 45 (TheTom KV cache) GGML_TYPE_TURBO2_0 = 46 (TheTom KV cache) GGML_TYPE_TQ3_1S = 47 (TheTom 3-bit weight, WHT-rotated) GGML_TYPE_TQ4_1S = 48 (TheTom 4-bit weight, WHT-rotated) GGML_TYPE_COUNT = 49 Same shape mirrored in include/llama.h LLAMA_FTYPE_MOSTLY_* (40-44) and gguf-py/gguf/constants.py. Conflict resolution strategy: - TheTom-as-source-of-truth for: turbo-innerq.cu/cuh, turbo-quant.cuh, turbo-wht.cu, ggml-turbo-quant.c, test-turbo-quant.c, ggml-vulkan.cpp (replaces atomicmilkshake versions wholesale) - Union for additive switch/case files (ggml.c, ggml-cpu.c, convert.cu, fattn.cu, fattn-vec.cuh, ggml-cuda.cu, metal*, etc.) - Manual fix for union-induced duplicates: * ggml-cpu.c: dropped duplicate ggml_vec_dot_turbo{2,3,4}_0_f32 bodies (HEAD malloc-based versions superseded by TheTom stack-array versions) * llama-context.cpp: dropped narrower TURBO3/4-only flash-attn check (TheTom's TURBO2/3/4 check is the superset) * llama-graph.cpp: dropped duplicate `n_head_v` decl (kept TheTom's Fix PrismML-Eng#78 version that uses n_head not n_head_kv for GQA models) * llama-kv-cache.cpp: dropped HEAD's old extern "C" InnerQ block (TheTom's #ifdef GGML_USE_CUDA pattern with weak fallback wins) * common/arg.cpp: removed orphan spec-default lambda fragments inside triattention-log lambda body (HEAD's --spec-default add_opt was eaten by union, only fragments remained) - atomicmilkshake's TriAttention args/code preserved; TheTom doesn't add or remove TriAttention so it remains as a separate feature. CPU build verified end-to-end. CUDA build pending. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

@bingh0

…nt (ggml-org#78) Post-attention V-padded reshape in build_attn was using hparams.n_head_kv(il), but cur returned from build_attn_mha has shape (n_embd_head * n_head, n_tokens) — n_head is the Q-head count. On GQA models where n_head != n_head_kv (e.g. Qwen2.5-0.5B with head_dim=64 padded → 128, n_head=14, n_head_kv=2), the reshape element count fails the assertion in ggml_reshape_3d and the process aborts. Symptom: GGML_ASSERT(ggml_nelements(a) == ne0*ne1*ne2) at ggml.c:3656. Reported and diagnosed by @bingh0 in TheTom#78. Verified locally on Qwen2.5-7B (head_dim=128, no padding, regression check passes) and on AMD MI300X with Qwen2.5-0.5B (head_dim=64, was crashing pre-fix). Three sites fixed (lines 2285, 2412, 2532 — same idiom in three build_attn overloads). Closes ggml-org#78. Likely also closes ggml-org#108 (speculative decoding hits the same assertion). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Port the layer-adaptive KV precision block from TheTom's TQ-KV state (5aeb2fd llama-kv-cache.cpp lines 267-326) plus mode 7 from da4a02e. Steps 1-4's cumulative ports landed everything else recon/08 §Step 5 intended; Step 5 is reduced to this single block (gap analysis in body). Modes: - 0 = uniform (default, no behavior change) - 1 = q8_0 K+V for first 4 + last 4 layers (turbo K+V outer-protection) - 2 = q8_0 K+V for last 8 layers - 5 = first2+last2 V=TURBOQ4_0, rest V=TURBOQ3_0 (K unchanged) - 6 = last8 V=TURBOQ4_0, rest V=TURBOQ3_0 (K unchanged) - 7 = Boundary V (recommended): first2+last2 V=q8_0, rest V=TURBOQ3_0 (K unchanged) Yggdrasil adjustments: - TURBO2_0 (TQ-KV slot 42) substituted with TURBOQ3_0 in the "middle" slot of modes 5/6/7. TURBOQ2_0 is deferred per phase-1 scope; when it lands, restore TURBO2_0 -> TURBOQ2_0 in those three branches to recover TQ-KV's original compression ratios. - TQ-KV's auto-enable on V=TURBO2_0 is intentionally dropped per recon/08 §Step 5 validation ("Boundary V default-off — no behavioral change without explicit flag"). Mode 0 stays the default. - Banner LLAMA_LOG_INFO uses a per-constructor-call local flag, not a static, because fit-probe + real construction each invoke the constructor; logger output is suppressed during the probe pass. Gap analysis (recon/08 §Step 5 commits, post-Step-3 cumulative-port state): - f2b3936 Metal asymmetric K/V: SKIP — Metal-only, no Metal backend. - 0fa61d7 PR ggml-org#115 auto-asymmetric merge: already absorbed via Step 3 (5aeb2fd). fattn-vec.cuh has K_is_turbo/V_is_turbo; is_kv_compat permits asymmetric pairs of {TURBOQ3_0, TURBOQ4_0, Q8_0, F16, BF16}. - b6f8e7f ggml-org#78 GQA reshape fix: N/A — targets padded_v_head / orig_v_head block in build_attn that doesn't exist in yggdrasil (mainline post-April refactor never had it). - e9f05a9 64-elem WHT + MLA Q rotation: partially absorbed (CUDA kernel supports group_size {128,64,32}); ggml_turbo_wht C-API stays 3-arg with hardcoded group_size=128 + assert ne[0]%128==0. The K-only build_attn Q rotation + %64 fallback are MLA-only; Phase 1 baseline (Qwen3.5-9B head_dim=128) doesn't exercise them. - 53ca939 MLA inverse WHT group_size from K: MLA-only. Deferred to a future phase that adds an MLA model (DeepSeek-V2 head_dim 192, GLM-4.7 head_dim 576): C-API extension of ggml_turbo_wht to take explicit group_size, Q rotation in inp_attn_k build_attn, %64 fallback, 53ca939's K-derived inverse-WHT guard. PPL parity gate (Qwen3.5-9B-BF16, 32-chunk c=512, eval-shared wikitext): | Backend | mode | PPL | |---------|---------|--------| | ROCm | default | 7.5939 | (= session 11, bit-identical: no regression) | Vulkan | default | 7.6065 | (= session 11, bit-identical: no regression) Cross-backend Δ = +0.17% (well within <0.5% threshold per design decision ggml-org#9). Mode 7 activation spot-check (4-chunk): | Backend | mode | PPL | |---------|------|--------| | ROCm | 0 | 8.2261 | | ROCm | 7 | 8.1180 | (-1.31%; boundary q8_0-V improves quality) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

@bingh0

…nt (ggml-org#78) Post-attention V-padded reshape in build_attn was using hparams.n_head_kv(il), but cur returned from build_attn_mha has shape (n_embd_head * n_head, n_tokens) — n_head is the Q-head count. On GQA models where n_head != n_head_kv (e.g. Qwen2.5-0.5B with head_dim=64 padded → 128, n_head=14, n_head_kv=2), the reshape element count fails the assertion in ggml_reshape_3d and the process aborts. Symptom: GGML_ASSERT(ggml_nelements(a) == ne0*ne1*ne2) at ggml.c:3656. Reported and diagnosed by @bingh0 in TheTom#78. Verified locally on Qwen2.5-7B (head_dim=128, no padding, regression check passes) and on AMD MI300X with Qwen2.5-0.5B (head_dim=64, was crashing pre-fix). Three sites fixed (lines 2285, 2412, 2532 — same idiom in three build_attn overloads). Closes ggml-org#78. Likely also closes ggml-org#108 (speculative decoding hits the same assertion). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Port the layer-adaptive KV precision block from TheTom's TQ-KV state (5aeb2fd llama-kv-cache.cpp lines 267-326) plus mode 7 from da4a02e. Steps 1-4's cumulative ports landed everything else recon/08 §Step 5 intended; Step 5 is reduced to this single block (gap analysis in body). Modes: - 0 = uniform (default, no behavior change) - 1 = q8_0 K+V for first 4 + last 4 layers (turbo K+V outer-protection) - 2 = q8_0 K+V for last 8 layers - 5 = first2+last2 V=TURBOQ4_0, rest V=TURBOQ3_0 (K unchanged) - 6 = last8 V=TURBOQ4_0, rest V=TURBOQ3_0 (K unchanged) - 7 = Boundary V (recommended): first2+last2 V=q8_0, rest V=TURBOQ3_0 (K unchanged) Yggdrasil adjustments: - TURBO2_0 (TQ-KV slot 42) substituted with TURBOQ3_0 in the "middle" slot of modes 5/6/7. TURBOQ2_0 is deferred per phase-1 scope; when it lands, restore TURBO2_0 -> TURBOQ2_0 in those three branches to recover TQ-KV's original compression ratios. - TQ-KV's auto-enable on V=TURBO2_0 is intentionally dropped per recon/08 §Step 5 validation ("Boundary V default-off — no behavioral change without explicit flag"). Mode 0 stays the default. - Banner LLAMA_LOG_INFO uses a per-constructor-call local flag, not a static, because fit-probe + real construction each invoke the constructor; logger output is suppressed during the probe pass. Gap analysis (recon/08 §Step 5 commits, post-Step-3 cumulative-port state): - f2b3936 Metal asymmetric K/V: SKIP — Metal-only, no Metal backend. - 0fa61d7 PR ggml-org#115 auto-asymmetric merge: already absorbed via Step 3 (5aeb2fd). fattn-vec.cuh has K_is_turbo/V_is_turbo; is_kv_compat permits asymmetric pairs of {TURBOQ3_0, TURBOQ4_0, Q8_0, F16, BF16}. - b6f8e7f ggml-org#78 GQA reshape fix: N/A — targets padded_v_head / orig_v_head block in build_attn that doesn't exist in yggdrasil (mainline post-April refactor never had it). - e9f05a9 64-elem WHT + MLA Q rotation: partially absorbed (CUDA kernel supports group_size {128,64,32}); ggml_turbo_wht C-API stays 3-arg with hardcoded group_size=128 + assert ne[0]%128==0. The K-only build_attn Q rotation + %64 fallback are MLA-only; Phase 1 baseline (Qwen3.5-9B head_dim=128) doesn't exercise them. - 53ca939 MLA inverse WHT group_size from K: MLA-only. Deferred to a future phase that adds an MLA model (DeepSeek-V2 head_dim 192, GLM-4.7 head_dim 576): C-API extension of ggml_turbo_wht to take explicit group_size, Q rotation in inp_attn_k build_attn, %64 fallback, 53ca939's K-derived inverse-WHT guard. PPL parity gate (Qwen3.5-9B-BF16, 32-chunk c=512, eval-shared wikitext): | Backend | mode | PPL | |---------|---------|--------| | ROCm | default | 7.5939 | (= session 11, bit-identical: no regression) | Vulkan | default | 7.6065 | (= session 11, bit-identical: no regression) Cross-backend Δ = +0.17% (well within <0.5% threshold per design decision ggml-org#9). Mode 7 activation spot-check (4-chunk): | Backend | mode | PPL | |---------|------|--------| | ROCm | 0 | 8.2261 | | ROCm | 7 | 8.1180 | (-1.31%; boundary q8_0-V improves quality) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…fork note (ggml-org#78) * ci(release): ship full self-contained Windows Vulkan/HIP bundles readme: add Prism fork note (start with Bonsai-demo, main caveats) * readme: fix Q2_0 model-file guidance (fork=Q2_0, mainline=Q2_0_g64, PQ2_0 future) + link demo status; ASCII punctuation

Truncate prompt if longer than context + n_predict

1133eea

matthew-mcallister mentioned this pull request Mar 13, 2023

Longer and infinite output #71

Closed

ggerganov closed this Mar 19, 2023

Bearsaerker mentioned this pull request Mar 12, 2025

Eval bug: Gemma 3 extremly slow prompt processing when using quantized kv cache. #12352

Closed

LifesLight pushed a commit to LifesLight/custom.llama.cpp that referenced this pull request May 2, 2026

Merge pull request ggml-org#116 from TheTom/fix/issue-78-gqa-reshape

cde3e1a

fix: build_attn V-padded reshape uses Q-head count, not KV-head count (ggml-org#78)

Copilot AI mentioned this pull request Aug 26, 2026

Harden workflows and server paths/output to resolve Code Scanning alerts #76–#100 kerrrang9214-tech/llama.cpp#3

Draft

Navigation Menu

Uh oh!

Truncate prompt if longer than context + n_predict - #78

Truncate prompt if longer than context + n_predict#78
matthew-mcallister wants to merge 1 commit into
ggml-org:masterfrom
matthew-mcallister:context-overflow

matthew-mcallister commented Mar 13, 2023

Uh oh!

ggerganov commented Mar 19, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

matthew-mcallister commented Mar 13, 2023

Uh oh!

ggerganov commented Mar 19, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants