Bug: std::out_of_range error for codegeex4-all-9b-GGUF

[99991]

What happened?

llama-cli crashes with an std::out_of_range error for model https://huggingface.co/bartowski/codegeex4-all-9b-GGUF/tree/main
I have tested Q8_0 and Q4_K_M, probably applies to other quantizations as well.
This happens when building from source or when using release b3371, probably also with other releases. Here are steps to reproduce this with Ubuntu 22.04 in Docker:

sudo apt update && sudo apt install wget
wget -nc https://huggingface.co/bartowski/codegeex4-all-9b-GGUF/resolve/main/codegeex4-all-9b-Q4_K_M.gguf?download=true -O codegeex4-all-9b-Q4_K_M.gguf
docker run --rm -ti --volume "$PWD":/data ubuntu:22.04
apt update && apt install wget unzip libgomp1 libcurl4 -yy
wget https://github.com/ggerganov/llama.cpp/releases/download/b3371/llama-b3371-bin-ubuntu-x64.zip
unzip llama-b3371-bin-ubuntu-x64.zip
./build/bin/llama-cli -m /data/codegeex4-all-9b-Q4_K_M.gguf
# crashes here

The faulty index 18446744073709551615 is equal to 0xffffffffffffffff or -1. My guess would be that someone passed -1 to a function expecting a value of type size_t, which usually is an unsigned 64 bit integer.

Name and Version

version: 3371 (9a55ffe)
built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu

What operating system are you seeing the problem on?

Linux

Relevant log output

# ./build/bin/llama-cli -m /data/codegeex4-all-9b-Q4_K_M.gguf
Log start
main: build = 3371 (9a55ffe6)
main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
main: seed  = 1720699310
llama_model_loader: loaded meta data with 27 key-value pairs and 283 tensors from /data/codegeex4-all-9b-Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = chatglm
llama_model_loader: - kv   1:                               general.name str              = codegeex4-all-9b
llama_model_loader: - kv   2:                     chatglm.context_length u32              = 131072
llama_model_loader: - kv   3:                   chatglm.embedding_length u32              = 4096
llama_model_loader: - kv   4:                chatglm.feed_forward_length u32              = 13696
llama_model_loader: - kv   5:                        chatglm.block_count u32              = 40
llama_model_loader: - kv   6:               chatglm.attention.head_count u32              = 32
llama_model_loader: - kv   7:            chatglm.attention.head_count_kv u32              = 2
llama_model_loader: - kv   8:   chatglm.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv   9:                          general.file_type u32              = 15
llama_model_loader: - kv  10:               chatglm.rope.dimension_count u32              = 64
llama_model_loader: - kv  11:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  12:                     chatglm.rope.freq_base f32              = 5000000.000000
llama_model_loader: - kv  13:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  14:                         tokenizer.ggml.pre str              = chatglm-bpe
llama_model_loader: - kv  15:                      tokenizer.ggml.tokens arr[str,151552]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  16:                  tokenizer.ggml.token_type arr[i32,151552]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  17:                      tokenizer.ggml.merges arr[str,151073]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  18:            tokenizer.ggml.padding_token_id u32              = 151329
llama_model_loader: - kv  19:                tokenizer.ggml.eos_token_id u32              = 151329
llama_model_loader: - kv  20:                tokenizer.ggml.eot_token_id u32              = 151336
llama_model_loader: - kv  21:            tokenizer.ggml.unknown_token_id u32              = 151329
llama_model_loader: - kv  22:               general.quantization_version u32              = 2
llama_model_loader: - kv  23:                      quantize.imatrix.file str              = /models/codegeex4-all-9b-GGUF/codegee...
llama_model_loader: - kv  24:                   quantize.imatrix.dataset str              = /training_data/calibration_datav3.txt
llama_model_loader: - kv  25:             quantize.imatrix.entries_count i32              = 160
llama_model_loader: - kv  26:              quantize.imatrix.chunks_count i32              = 125
llama_model_loader: - type  f32:  121 tensors
llama_model_loader: - type q5_0:   20 tensors
llama_model_loader: - type q8_0:   20 tensors
llama_model_loader: - type q4_K:   81 tensors
llama_model_loader: - type q5_K:   40 tensors
llama_model_loader: - type q6_K:    1 tensors
llm_load_vocab: special tokens cache size = 223
llm_load_vocab: token to piece cache size = 0.9732 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = chatglm
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 151552
llm_load_print_meta: n_merges         = 151073
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 131072
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_layer          = 40
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 2
llm_load_print_meta: n_rot            = 64
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 16
llm_load_print_meta: n_embd_k_gqa     = 256
llm_load_print_meta: n_embd_v_gqa     = 256
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 13696
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 5000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 131072
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 9B
llm_load_print_meta: model ftype      = Q4_K - Medium
llm_load_print_meta: model params     = 9.40 B
llm_load_print_meta: model size       = 5.82 GiB (5.31 BPW) 
llm_load_print_meta: general.name     = codegeex4-all-9b
llm_load_print_meta: EOS token        = 151329 '<|endoftext|>'
llm_load_print_meta: UNK token        = 151329 '<|endoftext|>'
llm_load_print_meta: PAD token        = 151329 '<|endoftext|>'
llm_load_print_meta: LF token         = 128 'Ä'
llm_load_print_meta: EOT token        = 151336 '<|user|>'
llm_load_print_meta: max token length = 1024
llm_load_tensors: ggml ctx size =    0.14 MiB
llm_load_tensors:        CPU buffer size =  5955.59 MiB
........................................................................................
llama_new_context_with_model: n_ctx      = 131072
llama_new_context_with_model: n_batch    = 2048
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 5000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:        CPU KV buffer size =  5120.00 MiB
llama_new_context_with_model: KV self size  = 5120.00 MiB, K (f16): 2560.00 MiB, V (f16): 2560.00 MiB
llama_new_context_with_model:        CPU  output buffer size =     0.58 MiB
llama_new_context_with_model:        CPU compute buffer size =  8481.01 MiB
llama_new_context_with_model: graph nodes  = 1606
llama_new_context_with_model: graph splits = 1

system_info: n_threads = 28 / 56 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 0 | 
terminate called after throwing an instance of 'std::out_of_range'
  what():  vector::_M_range_check: __n (which is 18446744073709551615) >= this->size() (which is 151552)
Aborted (core dumped)

[ggerganov]

Could you build in Debug and post a stack trace

[99991]

Could you build in Debug and post a stack trace

I guess I can.

Commands to reproduce stack trace

sudo apt update && sudo apt install wget
wget -nc https://huggingface.co/bartowski/codegeex4-all-9b-GGUF/resolve/main/codegeex4-all-9b-Q4_K_M.gguf?download=true -O codegeex4-all-9b-Q4_K_M.gguf
docker run --rm -ti --volume "$PWD":/data ubuntu:22.04
apt update && apt install wget unzip libgomp1 libcurl4 build-essential gdb -yy
wget https://github.com/ggerganov/llama.cpp/archive/refs/tags/b3378.zip
unzip b3378.zip
cd llama.cpp-b3378/
make -j LLAMA_DEBUG=1
echo -e "run\nbt full\nquit\n" > gdb_commands
gdb -x gdb_commands --args ./llama-cli -m /data/codegeex4-all-9b-Q4_K_M.gguf 2>&1 | tee stacktrace

GDB output

GNU gdb (Ubuntu 12.1-0ubuntu1~22.04.2) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./llama-cli...
warning: Error disabling address space randomization: Operation not permitted
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Log start
main: build = 0 (unknown)
main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
main: seed  = 1720776731
llama_model_loader: loaded meta data with 27 key-value pairs and 283 tensors from /data/codegeex4-all-9b-Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = chatglm
llama_model_loader: - kv   1:                               general.name str              = codegeex4-all-9b
llama_model_loader: - kv   2:                     chatglm.context_length u32              = 131072
llama_model_loader: - kv   3:                   chatglm.embedding_length u32              = 4096
llama_model_loader: - kv   4:                chatglm.feed_forward_length u32              = 13696
llama_model_loader: - kv   5:                        chatglm.block_count u32              = 40
llama_model_loader: - kv   6:               chatglm.attention.head_count u32              = 32
llama_model_loader: - kv   7:            chatglm.attention.head_count_kv u32              = 2
llama_model_loader: - kv   8:   chatglm.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv   9:                          general.file_type u32              = 15
llama_model_loader: - kv  10:               chatglm.rope.dimension_count u32              = 64
llama_model_loader: - kv  11:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  12:                     chatglm.rope.freq_base f32              = 5000000.000000
llama_model_loader: - kv  13:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  14:                         tokenizer.ggml.pre str              = chatglm-bpe
llama_model_loader: - kv  15:                      tokenizer.ggml.tokens arr[str,151552]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  16:                  tokenizer.ggml.token_type arr[i32,151552]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  17:                      tokenizer.ggml.merges arr[str,151073]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  18:            tokenizer.ggml.padding_token_id u32              = 151329
llama_model_loader: - kv  19:                tokenizer.ggml.eos_token_id u32              = 151329
llama_model_loader: - kv  20:                tokenizer.ggml.eot_token_id u32              = 151336
llama_model_loader: - kv  21:            tokenizer.ggml.unknown_token_id u32              = 151329
llama_model_loader: - kv  22:               general.quantization_version u32              = 2
llama_model_loader: - kv  23:                      quantize.imatrix.file str              = /models/codegeex4-all-9b-GGUF/codegee...
llama_model_loader: - kv  24:                   quantize.imatrix.dataset str              = /training_data/calibration_datav3.txt
llama_model_loader: - kv  25:             quantize.imatrix.entries_count i32              = 160
llama_model_loader: - kv  26:              quantize.imatrix.chunks_count i32              = 125
llama_model_loader: - type  f32:  121 tensors
llama_model_loader: - type q5_0:   20 tensors
llama_model_loader: - type q8_0:   20 tensors
llama_model_loader: - type q4_K:   81 tensors
llama_model_loader: - type q5_K:   40 tensors
llama_model_loader: - type q6_K:    1 tensors
llm_load_vocab: special tokens cache size = 223
llm_load_vocab: token to piece cache size = 0.9732 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = chatglm
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 151552
llm_load_print_meta: n_merges         = 151073
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 131072
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_layer          = 40
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 2
llm_load_print_meta: n_rot            = 64
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 16
llm_load_print_meta: n_embd_k_gqa     = 256
llm_load_print_meta: n_embd_v_gqa     = 256
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 13696
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 5000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 131072
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: model type       = 9B
llm_load_print_meta: model ftype      = Q4_K - Medium
llm_load_print_meta: model params     = 9.40 B
llm_load_print_meta: model size       = 5.82 GiB (5.31 BPW) 
llm_load_print_meta: general.name     = codegeex4-all-9b
llm_load_print_meta: EOS token        = 151329 '<|endoftext|>'
llm_load_print_meta: UNK token        = 151329 '<|endoftext|>'
llm_load_print_meta: PAD token        = 151329 '<|endoftext|>'
llm_load_print_meta: LF token         = 128 'Ä'
llm_load_print_meta: EOT token        = 151336 '<|user|>'
llm_load_print_meta: max token length = 1024
llm_load_tensors: ggml ctx size =    0.14 MiB
llm_load_tensors:        CPU buffer size =  5955.59 MiB
........................................................................................
llama_new_context_with_model: n_ctx      = 131072
llama_new_context_with_model: n_batch    = 2048
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 5000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:        CPU KV buffer size =  5120.00 MiB
llama_new_context_with_model: KV self size  = 5120.00 MiB, K (f16): 2560.00 MiB, V (f16): 2560.00 MiB
llama_new_context_with_model:        CPU  output buffer size =     0.58 MiB
ggml_gallocr_reserve_n: reallocating CPU buffer from size 0.00 MiB to 8481.01 MiB
llama_new_context_with_model:        CPU compute buffer size =  8481.01 MiB
llama_new_context_with_model: graph nodes  = 1606
llama_new_context_with_model: graph splits = 1
[New Thread 0x7f7bf1d9a640 (LWP 4972)]
[New Thread 0x7f7bf1599640 (LWP 4973)]
[New Thread 0x7f7bf0d98640 (LWP 4974)]
[New Thread 0x7f7bf0597640 (LWP 4975)]
[New Thread 0x7f7befd96640 (LWP 4976)]
[New Thread 0x7f7bef595640 (LWP 4977)]
[New Thread 0x7f7beed94640 (LWP 4978)]
[New Thread 0x7f7bee593640 (LWP 4979)]
[New Thread 0x7f7bedd92640 (LWP 4980)]
[New Thread 0x7f7bed591640 (LWP 4981)]
[New Thread 0x7f7becd90640 (LWP 4982)]
[New Thread 0x7f7bec58f640 (LWP 4983)]
[New Thread 0x7f7bebd8e640 (LWP 4984)]
[New Thread 0x7f7beb58d640 (LWP 4985)]
[New Thread 0x7f7bead8c640 (LWP 4986)]
[New Thread 0x7f7bea58b640 (LWP 4987)]
[New Thread 0x7f7be9d8a640 (LWP 4988)]
[New Thread 0x7f7be9589640 (LWP 4989)]
[New Thread 0x7f7be8d88640 (LWP 4990)]
[New Thread 0x7f7be8587640 (LWP 4991)]
[New Thread 0x7f7be7d86640 (LWP 4992)]
[New Thread 0x7f7be7585640 (LWP 4993)]
[New Thread 0x7f7be6d84640 (LWP 4994)]
[New Thread 0x7f7be6583640 (LWP 4995)]
[New Thread 0x7f7be5d82640 (LWP 4996)]
[New Thread 0x7f7be5581640 (LWP 4997)]
[New Thread 0x7f7be4d80640 (LWP 4998)]

system_info: n_threads = 28 / 56 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | 

Thread 1 "llama-cli" received signal SIGABRT, Aborted.
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140190861064128) at ./nptl/pthread_kill.c:44
44	./nptl/pthread_kill.c: No such file or directory.
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140190861064128) at ./nptl/pthread_kill.c:44
        tid = <optimized out>
        ret = 0
        pd = 0x7f80ba7977c0
        old_mask = {__val = {1, 4, 18446744073709551615, 1720776738, 34, 0, 322315820728320, 0, 2676586395008836901, 18446744073709551312, 34, 94326094595648, 1, 0, 2, 140190861759571}}
        ret = <optimized out>
        pd = <optimized out>
        old_mask = <optimized out>
        ret = <optimized out>
        tid = <optimized out>
        ret = <optimized out>
        resultvar = <optimized out>
        resultvar = <optimized out>
        __arg3 = <optimized out>
        __arg2 = <optimized out>
        __arg1 = <optimized out>
        _a3 = <optimized out>
        _a2 = <optimized out>
        _a1 = <optimized out>
        __futex = <optimized out>
        resultvar = <optimized out>
        __arg3 = <optimized out>
        __arg2 = <optimized out>
        __arg1 = <optimized out>
        _a3 = <optimized out>
        _a2 = <optimized out>
        _a1 = <optimized out>
        __futex = <optimized out>
        __private = <optimized out>
        __oldval = <optimized out>
        result = <optimized out>
#1  __pthread_kill_internal (signo=6, threadid=140190861064128) at ./nptl/pthread_kill.c:78
No locals.
#2  __GI___pthread_kill (threadid=140190861064128, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
No locals.
#3  0x00007f80ba7de476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
        ret = <optimized out>
#4  0x00007f80ba7c47f3 in __GI_abort () at ./stdlib/abort.c:79
        save_stage = 1
        act = {__sigaction_handler = {sa_handler = 0x7ffc793b0400, sa_sigaction = 0x7ffc793b0400}, sa_mask = {__val = {94326058037312, 1063, 94326058041016, 94326058036979, 1, 140722342396976, 
              140722342397584, 140722342396992, 94326056269328, 140722342397008, 94326055844747, 140722342397040, 93462767727634, 140722342397063, 140722342397599, 140722342397072}}, 
          sa_flags = -15600424, sa_restorer = 0xf}
        sigs = {__val = {32, 94326056271244, 3690189933722141584, 6883906998192698624, 94326094595648, 140722342397584, 140722342397120, 1, 15, 94326059197304, 140190867263552, 140190861477791, 
            206158430248, 140722342397088, 140722342396896, 6883906998192698624}}
#5  0x000055c9ff0d3903 in std::__replacement_assert (__file=0x55c9ff2eb040 "/usr/include/c++/11/bits/stl_vector.h", __line=1063, 
    __function=0x55c9ff2ebeb8 "std::vector<_Tp, _Alloc>::const_reference std::vector<_Tp, _Alloc>::operator[](std::vector<_Tp, _Alloc>::size_type) const [with _Tp = llama_vocab::token_data; _Alloc = std::allocator<llama_vocab::toke"..., __condition=0x55c9ff2eaef3 "__n < this->size()") at /usr/include/x86_64-linux-gnu/c++/11/bits/c++config.h:514
No locals.
#6  0x000055c9ff113f0f in std::vector<llama_vocab::token_data, std::allocator<llama_vocab::token_data> >::operator[] (this=0x55ca014857a0, __n=18446744073709551615)
    at /usr/include/c++/11/bits/stl_vector.h:1063
        __PRETTY_FUNCTION__ = "std::vector<_Tp, _Alloc>::const_reference std::vector<_Tp, _Alloc>::operator[](std::vector<_Tp, _Alloc>::size_type) const [with _Tp = llama_vocab::token_data; _Alloc = std::allocator<llama_vocab::toke"...
#7  0x000055c9ff0b7fe7 in llama_token_get_attr (model=0x55ca01484a80, token=-1) at src/llama.cpp:21027
No locals.
#8  0x000055c9ff0b8718 in llama_token_to_piece (model=0x55ca01484a80, token=-1, buf=0x7ffc793b0690 "", length=15, lstrip=0, special=true) at src/llama.cpp:21134
        attr_special = 9
        attr = (LLAMA_TOKEN_ATTR_NORMAL | LLAMA_TOKEN_ATTR_CONTROL | LLAMA_TOKEN_ATTR_USER_DEFINED | LLAMA_TOKEN_ATTR_BYTE | LLAMA_TOKEN_ATTR_NORMALIZED | LLAMA_TOKEN_ATTR_LSTRIP | LLAMA_TOKEN_ATTR_RSTRIP | LLAMA_TOKEN_ATTR_SINGLE_WORD | unknown: 0x7c00)
        _try_copy = {__lstrip = 2033911168, __length = 32764, __buf = 0x5500ff11f412 <error: Cannot access memory at address 0x5500ff11f412>}
#9  0x000055c9ff20b7dc in llama_token_to_piece[abi:cxx11](llama_context const*, int, bool) (ctx=0x55ca014c79b0, token=-1, special=true) at common/common.cpp:2631
        piece = '\000' <repeats 14 times>
        n_chars = -15628542
#10 0x000055c9ff2c5020 in LOG_TOKENS_TOSTR_PRETTY<llama_context*, std::vector<int, std::allocator<int> > > (ctx=@0x7ffc793b0928: 0x55ca014c79b0, tokens=std::vector of length 1, capacity 2 = {...})
    at common/log.h:641
        detokenized = '\000' <repeats 14 times>
        token = @0x55ca03a4c0f0: -1
        __for_range = std::vector of length 1, capacity 2 = {-1}
        __for_begin = -1
        __for_end = 0
        buf = <incomplete type>
        first = false
#11 0x000055c9ff2baa03 in main (argc=3, argv=0x7ffc793b2f08) at examples/main/main.cpp:293
        params = {seed = 1720776731, n_threads = 28, n_threads_draft = -1, n_threads_batch = -1, n_threads_batch_draft = -1, n_predict = -1, n_ctx = 0, n_batch = 2048, n_ubatch = 512, n_keep = 0, 
          n_draft = 5, n_chunks = -1, n_parallel = 1, n_sequences = 1, p_split = 0.100000001, n_gpu_layers = -1, n_gpu_layers_draft = -1, main_gpu = 0, tensor_split = {0 <repeats 128 times>}, 
          grp_attn_n = 1, grp_attn_w = 512, n_print = -1, rope_freq_base = 0, rope_freq_scale = 0, yarn_ext_factor = -1, yarn_attn_factor = 1, yarn_beta_fast = 32, yarn_beta_slow = 1, 
          yarn_orig_ctx = 0, defrag_thold = -1, cb_eval = 0x0, cb_eval_user_data = 0x0, numa = GGML_NUMA_STRATEGY_DISABLED, split_mode = LLAMA_SPLIT_MODE_LAYER, 
          rope_scaling_type = LLAMA_ROPE_SCALING_TYPE_UNSPECIFIED, pooling_type = LLAMA_POOLING_TYPE_UNSPECIFIED, attention_type = LLAMA_ATTENTION_TYPE_UNSPECIFIED, sparams = {n_prev = 64, 
            n_probs = 0, min_keep = 0, top_k = 40, top_p = 0.949999988, min_p = 0.0500000007, tfs_z = 1, typical_p = 1, temp = 0.800000012, dynatemp_range = 0, dynatemp_exponent = 1, 
            penalty_last_n = 64, penalty_repeat = 1, penalty_freq = 0, penalty_present = 0, mirostat = 0, mirostat_tau = 5, mirostat_eta = 0.100000001, penalize_nl = false, seed = 4294967295, 
            samplers_sequence = std::vector of length 6, capacity 6 = {llama_sampler_type::TOP_K, llama_sampler_type::TFS_Z, llama_sampler_type::TYPICAL_P, llama_sampler_type::TOP_P, 
              llama_sampler_type::MIN_P, llama_sampler_type::TEMPERATURE}, grammar = "", cfg_negative_prompt = "", cfg_scale = 1, logit_bias = std::unordered_map with 0 elements, 
            penalty_prompt_tokens = std::vector of length 0, capacity 0, use_penalty_prompt_tokens = false}, model = "/data/codegeex4-all-9b-Q4_K_M.gguf", model_draft = "", model_alias = "unknown", 
          model_url = "", hf_token = "", hf_repo = "", hf_file = "", prompt = "", prompt_file = "", path_prompt_cache = "", input_prefix = "", input_suffix = "", logdir = "", 
          lookup_cache_static = "", lookup_cache_dynamic = "", logits_file = "", rpc_servers = "", in_files = std::vector of length 0, capacity 0, antiprompt = std::vector of length 0, capacity 0, 
          kv_overrides = std::vector of length 0, capacity 0, lora_adapter = std::vector of length 0, capacity 0, lora_base = "", control_vectors = std::vector of length 0, capacity 0, 
          verbosity = 0, control_vector_layer_start = -1, control_vector_layer_end = -1, ppl_stride = 0, ppl_output_type = 0, hellaswag = false, hellaswag_tasks = 400, winogrande = false, 
          winogrande_tasks = 0, multiple_choice = false, multiple_choice_tasks = 0, kl_divergence = false, usage = false, use_color = false, special = false, interactive = false, 
          interactive_first = false, conversation = false, prompt_cache_all = false, prompt_cache_ro = false, escape = true, multiline_input = false, simple_io = false, cont_batching = true, 
          flash_attn = false, input_prefix_bos = false, ignore_eos = false, logits_all = false, use_mmap = true, use_mlock = false, verbose_prompt = false, display_prompt = true, infill = false, 
          dump_kv_cache = false, no_kv_offload = false, warmup = true, check_tensors = false, cache_type_k = "f16", cache_type_v = "f16", mmproj = "", image = std::vector of length 0, capacity 0, 
          embedding = false, embd_normalize = 2, embd_out = "", embd_sep = "\n", port = 8080, timeout_read = 600, timeout_write = 600, n_threads_http = -1, hostname = "127.0.0.1", public_path = "", 
          chat_template = "", system_prompt = "", enable_chat_template = true, api_keys = std::vector of length 0, capacity 0, ssl_file_key = "", ssl_file_cert = "", endpoint_slots = true, 
          endpoint_metrics = false, log_json = false, slot_save_path = "", slot_prompt_similarity = 0.5, is_pp_shared = false, n_pp = std::vector of length 0, capacity 0, 
          n_tg = std::vector of length 0, capacity 0, n_pl = std::vector of length 0, capacity 0, context_files = std::vector of length 0, capacity 0, chunk_size = 64, chunk_separator = "\n", 
          n_junk = 250, i_pos = -1, out_file = "imatrix.dat", n_out_freq = 10, n_save_freq = 0, i_chunk = 0, process_output = false, compute_ppl = true, n_pca_batch = 100, n_pca_iterations = 1000, 
          cvector_dimre_method = DIMRE_METHOD_PCA, cvector_outfile = "control_vector.gguf", cvector_positive_file = "examples/cvector-generator/positive.txt", 
          cvector_negative_file = "examples/cvector-generator/negative.txt", spm_infill = false}
        sparams = @0x7ffc793b12f0: {n_prev = 64, n_probs = 0, min_keep = 0, top_k = 40, top_p = 0.949999988, min_p = 0.0500000007, tfs_z = 1, typical_p = 1, temp = 0.800000012, dynatemp_range = 0, 
          dynatemp_exponent = 1, penalty_last_n = 64, penalty_repeat = 1, penalty_freq = 0, penalty_present = 0, mirostat = 0, mirostat_tau = 5, mirostat_eta = 0.100000001, penalize_nl = false, 
          seed = 4294967295, samplers_sequence = std::vector of length 6, capacity 6 = {llama_sampler_type::TOP_K, llama_sampler_type::TFS_Z, llama_sampler_type::TYPICAL_P, 
            llama_sampler_type::TOP_P, llama_sampler_type::MIN_P, llama_sampler_type::TEMPERATURE}, grammar = "", cfg_negative_prompt = "", cfg_scale = 1, 
          logit_bias = std::unordered_map with 0 elements, penalty_prompt_tokens = std::vector of length 0, capacity 0, use_penalty_prompt_tokens = false}
        __func__ = "main"
        rng = {static state_size = 624, _M_x = {1720776731, 3433942083, 1486869826, 2567241330, 1077039668, 3712732142, 1854163335, 4029921253, 2166072262, 3840376925, 1570370336, 3804271888, 
            2638674827, 3059519002, 2100750214, 289511250, 2753140074, 1692703257, 2068816778, 1639835882, 3019876299, 925287266, 814254016, 3972222679, 2248773052, 114399759, 1720504069, 
            695390383, 1422064679, 3068645659, 2081087483, 1509763009, 291925728, 920624769, 328438279, 2760858598, 1475888408, 27042818, 3399468784, 1730578694, 79285995, 427255264, 985665930, 
            2851063197, 3618901735, 238937121, 3865553459, 1212798239, 2214601478, 3420910789, 1149667408, 3997049128, 1248860459, 3285673927, 283104394, 2027074729, 1046949504, 471324857, 
            3513942583, 1766273727, 779137842, 2696574199, 3235529191, 3768255283, 3895259696, 3401439584, 1274134353, 1358056147, 1029695262, 2203532571, 3100408867, 2818679372, 2158151438, 
            1578234885, 348613086, 2003210977, 2217857708, 4230561011, 2020702974, 814808554, 1786537890, 805314208, 1456452978, 3794936242, 2846229545, 132622412, 3247051346, 3950875468, 
            2005493123, 1326816931, 1943685188, 2611900564, 573354378, 2037048783, 1288300708, 1370232440, 4198145821, 437900599, 667717141, 3931130796, 1341621103, 33470667, 3421539197, 687830813, 
            1632863193, 3530136737, 686521940, 265276303, 1439946455, 1650870491, 2236442736, 4153227113, 2034538306, 4090988768, 243236097, 3375946456, 814386139, 2189320412, 1808074252, 
            2605930648, 1182949178, 352527808, 28419386, 3105372765, 2858737399, 2848986150, 383465138, 3953120953, 3608101346, 3161286470, 1716068950, 580656086, 1029079538, 608814847, 3516070433, 
            2244403697, 3892785255, 3866598397, 1537046976, 730878640, 4009807868, 4271540008, 4219099013, 2630172269, 4047767899, 4198484041, 2405408708, 2745499569, 1590996275, 251008847, 
            2428074433, 4088096134, 3197516561, 946305560, 399639570, 2122792373, 3947670688, 1511544812, 4129523231, 2576147883, 2401045837, 3015608268, 2138755816, 380823440, 3346099316, 
            2383182488, 180275560, 3102088879, 3958353897, 3793396219, 2197266306, 2358079019, 1927243737, 3617493221, 3949216108, 597375098, 550277842, 117854091, 4141598089, 2687617573, 
            3554976023, 3404939673, 55200888, 3330565135, 4162446708, 944060332, 1406010262, 492059214, 630243202, 1562779911, 2531343900, 1810731157, 455425828, 2390240757, 4223947317, 1810716177, 
            2988541204, 588007539, 3854770469, 46361285...}, _M_p = 624}
        model = 0x55ca01484a80
        ctx = 0x55ca014c79b0
        ctx_guidance = 0x0
        chat_msgs = std::vector of length 0, capacity 0
        n_ctx_train = 131072
        n_ctx = 131072
        path_session = ""
        session_tokens = std::vector of length 0, capacity 0
        add_bos = false
        embd_inp = std::vector of length 1, capacity 2 = {-1}
        guidance_inp = std::vector of length 46239617908736, capacity 46240691641925 = {<error reading variable guidance_inp (Cannot access memory at address 0x51de0000fa15)>
        guidance_offset = 63979
        original_prompt_len = 63979
        n_matching_session_tokens = 274856432171515
        ga_i = 21533
        ga_n = 63986
        ga_w = 63986
        is_antiprompt = 249
        input_echo = false
        display = false
        need_to_save_session = 226
        n_past = 63982
        n_remain = 63982
        n_consumed = 29136
        n_session_consumed = 63983
        n_past_guidance = 63983
        input_tokens = std::vector of length 1073733577, capacity -35384088068095 = {<error reading variable input_tokens (Cannot access memory at address 0xfa180000fa18)>
        output_tokens = std::vector of length -35334695935954, capacity 1073750061 = {<error reading variable output_tokens (Cannot access memory at address 0xfa1b00007965)>
        output_ss = <incomplete type>
        assistant_ss = <incomplete type>
        embd = std::vector of length 34462817583104, capacity 34464965058920 = {<error reading variable embd (Cannot access memory at address 0x7cbe0000fa1d)>
        embd_guidance = std::vector of length 3221218357, capacity -29113435815935 = {<error reading variable embd_guidance (Cannot access memory at address 0xfa220000fa22)>
        antiprompt_ids = std::vector of length -4454417955742, capacity 178958092 = {<error reading variable antiprompt_ids (Cannot access memory at address 0xfa2a000090fd)>
        ctx_sampling = 0xfa0200005ea6
A debugging session is active.

	Inferior 1 [process 4969] will be killed.

Quit anyway? (y or n) [answered Y; input not from terminal]

Looks like the variable token is -1 here:

https://github.com/ggerganov/llama.cpp/blob/f53226245f421bd01b47cce43a47e791de82c636/src/llama.cpp#L21027

[ggerganov]

Should be fixed with #8456 - give it a try

[99991]

With this fix, I get error: input is empty for all models I tried (for example https://huggingface.co/failspy/Meta-Llama-3-8B-Instruct-abliterated-v3-GGUF/tree/main), even those which worked previously.

[ggerganov]

You have to add some prompt with -p "some prompt". The models that you use have add_bos == false in their GGUF header which means that if you provide no input prompt then there is nothing to process. Before #8456, the BOS was added unconditionally upon empty input even if the token didn't exist or was not supposed to be added

[99991]

Ah, I see. Thank you. I am using llama-cli for interactive chatting and it is hard to keep up with the frequent changes. I guess it works now if you can figure out the magic words of the day.