Eval bug: Crash with Mistral Small 2506

[allo-]

Name and Version

version: 4731 (0f2bbe6)
built with cc (Debian 14.2.0-16) 14.2.0 for x86_64-linux-gnu

Operating systems

Linux

GGML backends

CUDA

Hardware

Device 0: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes

Models

Mistral Small 2506 Q4_K_M

Problem description & steps to reproduce

Crash when running llama-server. See log.

First Bad Commit

No response

Relevant log output

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 5060 Ti, compute capability 12.0, VMM: yes
build: 6445 (00681dfc1) with cc (Debian 14.2.0-19) 14.2.0 for x86_64-linux-gnu
system info: n_threads = 8, n_threads_batch = 8, total_threads = 16

system_info: n_threads = 8 (n_threads_batch = 8) / 16 | CUDA : ARCHS = 1200 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 

main: binding port with default address family
main: HTTP server is listening, hostname: 127.0.0.1, port: 5000, http threads: 15
main: loading model
srv    load_model: loading model 'models/mistralai_Mistral-Small-3.2-24B-Instruct-2506-Q4_K_M.gguf'
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 5060 Ti) - 14691 MiB free
llama_model_loader: loaded meta data with 46 key-value pairs and 363 tensors from models/mistralai_Mistral-Small-3.2-24B-Instruct-2506-Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Mistral Small 3.2 24B Instruct 2506
llama_model_loader: - kv   3:                            general.version str              = 2506
llama_model_loader: - kv   4:                           general.finetune str              = Instruct
llama_model_loader: - kv   5:                           general.basename str              = Mistral-Small-3.2
llama_model_loader: - kv   6:                         general.size_label str              = 24B
llama_model_loader: - kv   7:                            general.license str              = apache-2.0
llama_model_loader: - kv   8:                   general.base_model.count u32              = 1
llama_model_loader: - kv   9:                  general.base_model.0.name str              = Mistral Small 3.1 24B Base 2503
llama_model_loader: - kv  10:               general.base_model.0.version str              = 2503
llama_model_loader: - kv  11:          general.base_model.0.organization str              = Mistralai
llama_model_loader: - kv  12:              general.base_model.0.repo_url str              = https://huggingface.co/mistralai/Mist...
llama_model_loader: - kv  13:                               general.tags arr[str,1]       = ["image-text-to-text"]
llama_model_loader: - kv  14:                          general.languages arr[str,24]      = ["en", "fr", "de", "es", "pt", "it", ...
llama_model_loader: - kv  15:                          llama.block_count u32              = 40
llama_model_loader: - kv  16:                       llama.context_length u32              = 131072
llama_model_loader: - kv  17:                     llama.embedding_length u32              = 5120
llama_model_loader: - kv  18:                  llama.feed_forward_length u32              = 32768
llama_model_loader: - kv  19:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv  20:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  21:                       llama.rope.freq_base f32              = 1000000000.000000
llama_model_loader: - kv  22:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  23:                 llama.attention.key_length u32              = 128
llama_model_loader: - kv  24:               llama.attention.value_length u32              = 128
llama_model_loader: - kv  25:                           llama.vocab_size u32              = 131072
llama_model_loader: - kv  26:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  27:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  28:                         tokenizer.ggml.pre str              = tekken
llama_model_loader: - kv  29:                      tokenizer.ggml.tokens arr[str,131072]  = ["<unk>", "<s>", "</s>", "[INST]", "[...
llama_model_loader: - kv  30:                  tokenizer.ggml.token_type arr[i32,131072]  = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv  31:                      tokenizer.ggml.merges arr[str,269443]  = ["Ġ Ġ", "Ġ t", "e r", "i n", "Ġ Ä...
llama_model_loader: - kv  32:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  33:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  34:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  35:            tokenizer.ggml.padding_token_id u32              = 11
llama_model_loader: - kv  36:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  37:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  38:                    tokenizer.chat_template str              = {%- set today = strftime_now("%Y-%m-%...
llama_model_loader: - kv  39:            tokenizer.ggml.add_space_prefix bool             = false
llama_model_loader: - kv  40:               general.quantization_version u32              = 2
llama_model_loader: - kv  41:                          general.file_type u32              = 15
llama_model_loader: - kv  42:                      quantize.imatrix.file str              = /models_out/Mistral-Small-3.2-24B-Ins...
llama_model_loader: - kv  43:                   quantize.imatrix.dataset str              = /training_dir/calibration_datav3.txt
llama_model_loader: - kv  44:             quantize.imatrix.entries_count u32              = 280
llama_model_loader: - kv  45:              quantize.imatrix.chunks_count u32              = 499
llama_model_loader: - type  f32:   81 tensors
llama_model_loader: - type q4_K:  241 tensors
llama_model_loader: - type q6_K:   41 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 13.34 GiB (4.86 BPW) 
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load:   - 2 ('</s>')
load: special tokens cache size = 1000
load: token to piece cache size = 0.8498 MB
print_info: arch             = llama
print_info: vocab_only       = 0
print_info: n_ctx_train      = 131072
print_info: n_embd           = 5120
print_info: n_layer          = 40
print_info: n_head           = 32
print_info: n_head_kv        = 8
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 4
print_info: n_embd_k_gqa     = 1024
print_info: n_embd_v_gqa     = 1024
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 32768
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 0
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 131072
print_info: rope_finetuned   = unknown
print_info: model type       = 13B
print_info: model params     = 23.57 B
print_info: general.name     = Mistral Small 3.2 24B Instruct 2506
print_info: vocab type       = BPE
print_info: n_vocab          = 131072
print_info: n_merges         = 269443
print_info: BOS token        = 1 '<s>'
print_info: EOS token        = 2 '</s>'
print_info: UNK token        = 0 '<unk>'
print_info: PAD token        = 11 '<pad>'
print_info: LF token         = 1010 'Ċ'
print_info: EOG token        = 2 '</s>'
print_info: max token length = 150
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 34 repeating layers to GPU
load_tensors: offloaded 34/41 layers to GPU
load_tensors:        CUDA0 model buffer size = 10775.66 MiB
load_tensors:   CPU_Mapped model buffer size =  2886.70 MiB
...............................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 32768
llama_context: n_ctx_per_seq = 32768
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = enabled
llama_context: kv_unified    = false
llama_context: freq_base     = 1000000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (32768) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
llama_context:        CPU  output buffer size =     0.50 MiB
llama_kv_cache:      CUDA0 KV buffer size =  2312.00 MiB
llama_kv_cache:        CPU KV buffer size =   408.00 MiB
llama_kv_cache: size = 2720.00 MiB ( 32768 cells,  40 layers,  1/1 seqs), K (q8_0): 1360.00 MiB, V (q8_0): 1360.00 MiB
llama_context:      CUDA0 compute buffer size =   791.00 MiB
llama_context:  CUDA_Host compute buffer size =    74.01 MiB
llama_context: graph nodes  = 1247
llama_context: graph splits = 70 (with bs=512), 3 (with bs=1)
common_init_from_params: added </s> logit bias = -inf
common_init_from_params: setting dry_penalty_last_n to ctx_size = 32768
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
Failed to generate tool call example: Value is not callable: null at row 2, column 42:
{%- set today = strftime_now("%Y-%m-%d") %}
{%- set yesterday = (today | as_datetime - timedelta(days=1)).strftime("%Y-%m-%d") %}
                                         ^
{%- set default_system_message = "You are Mistral Small 3, a Large Language Model (LLM) created by Mistral AI, a French startup headquartered in Paris.\nYour knowledge base was last updated on 2023-10-01. The current date is " + today + ".\n\nWhen you're not sure about some information or when the user's request requires up-to-date or specific data, you must use the available tools to fetch the information. Do not hesitate to use tools whenever they can provide a more accurate or complete response. If no relevant tools are available, then clearly state that you don't have the information and avoid making up anything.
 at row 2, column 61:
{%- set today = strftime_now("%Y-%m-%d") %}
{%- set yesterday = (today | as_datetime - timedelta(days=1)).strftime("%Y-%m-%d") %}
                                                            ^
{%- set default_system_message = "You are Mistral Small 3, a Large Language Model (LLM) created by Mistral AI, a French startup headquartered in Paris.\nYour knowledge base was last updated on 2023-10-01. The current date is " + today + ".\n\nWhen you're not sure about some information or when the user's request requires up-to-date or specific data, you must use the available tools to fetch the information. Do not hesitate to use tools whenever they can provide a more accurate or complete response. If no relevant tools are available, then clearly state that you don't have the information and avoid making up anything.
 at row 2, column 63:
{%- set today = strftime_now("%Y-%m-%d") %}
{%- set yesterday = (today | as_datetime - timedelta(days=1)).strftime("%Y-%m-%d") %}
                                                              ^
{%- set default_system_message = "You are Mistral Small 3, a Large Language Model (LLM) created by Mistral AI, a French startup headquartered in Paris.\nYour knowledge base was last updated on 2023-10-01. The current date is " + today + ".\n\nWhen you're not sure about some information or when the user's request requires up-to-date or specific data, you must use the available tools to fetch the information. Do not hesitate to use tools whenever they can provide a more accurate or complete response. If no relevant tools are available, then clearly state that you don't have the information and avoid making up anything.
 at row 2, column 1:
{%- set today = strftime_now("%Y-%m-%d") %}
{%- set yesterday = (today | as_datetime - timedelta(days=1)).strftime("%Y-%m-%d") %}
^
{%- set default_system_message = "You are Mistral Small 3, a Large Language Model (LLM) created by Mistral AI, a French startup headquartered in Paris.\nYour knowledge base was last updated on 2023-10-01. The current date is " + today + ".\n\nWhen you're not sure about some information or when the user's request requires up-to-date or specific data, you must use the available tools to fetch the information. Do not hesitate to use tools whenever they can provide a more accurate or complete response. If no relevant tools are available, then clearly state that you don't have the information and avoid making up anything.
 at row 1, column 1:
{%- set today = strftime_now("%Y-%m-%d") %}
^
{%- set yesterday = (today | as_datetime - timedelta(days=1)).strftime("%Y-%m-%d") %}

srv          init: initializing slots, n_slots = 1
slot         init: id  0 | task -1 | new slot n_ctx_slot = 32768
[New LWP 225306]
[New LWP 225305]
[New LWP 225304]
[New LWP 225303]
[New LWP 225302]
[New LWP 225301]
[New LWP 225300]
[New LWP 225299]
[New LWP 225298]
[New LWP 225297]
[New LWP 225296]
[New LWP 225295]
[New LWP 225294]
[New LWP 225293]
[New LWP 225292]
[New LWP 225291]
[New LWP 225290]
[New LWP 225289]
[New LWP 225288]
[New LWP 225287]
[New LWP 225286]
[New LWP 225285]
[New LWP 225284]
[New LWP 225283]
[New LWP 225282]
[New LWP 225281]
[New LWP 225276]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
__syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
warning: 56	../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S: No such file or directory
#0  __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
56	in ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S
#1  0x00007f2289299668 in __internal_syscall_cancel (a1=<optimized out>, a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=a5@entry=0, a6=a6@entry=0, nr=61) at ./nptl/cancellation.c:49
warning: 49	./nptl/cancellation.c: No such file or directory
#2  0x00007f22892996ad in __syscall_cancel (a1=<optimized out>, a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=a5@entry=0, a6=a6@entry=0, nr=61) at ./nptl/cancellation.c:75
75	in ./nptl/cancellation.c
#3  0x00007f2289304787 in __GI___wait4 (pid=<optimized out>, stat_loc=<optimized out>, options=<optimized out>, usage=<optimized out>) at ../sysdeps/unix/sysv/linux/wait4.c:30
warning: 30	../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#4  0x00007f2289b37d6b in ggml_print_backtrace () from llama.cpp/build/bin/Release/libggml-base.so
#5  0x00007f2289b48ab9 in ggml_uncaught_exception() () from llama.cpp/build/bin/Release/libggml-base.so
#6  0x00007f22894b344a in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#7  0x00007f22894a15e9 in std::terminate() () from /lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007f22894b36c8 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6
#9  0x0000556a33e03a51 in minja::TemplateNode::render(std::__cxx11::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >&, std::shared_ptr<minja::Context> const&) const ()
#10 0x0000556a33e303a0 in minja::chat_template::apply[abi:cxx11](minja::chat_template_inputs const&, minja::chat_template_options const&) const ()
#11 0x0000556a33df0c96 in apply(minja::chat_template const&, templates_params const&, std::optional<nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> >, void> > const&, std::optional<nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> >, void> > const&, std::optional<nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::vector<unsigned char, std::allocator<unsigned char> >, void> > const&) ()
#12 0x0000556a33df12a2 in common_chat_params_init_without_tools(minja::chat_template const&, templates_params const&) ()
#13 0x0000556a33df8fdd in common_chat_templates_apply_jinja(common_chat_templates const*, common_chat_templates_inputs const&) ()
#14 0x0000556a33df94d5 in common_chat_templates_apply(common_chat_templates const*, common_chat_templates_inputs const&) ()
#15 0x0000556a33dfa014 in common_chat_templates_support_enable_thinking(common_chat_templates const*) ()
#16 0x0000556a33d7cf33 in server_context::init() ()
#17 0x0000556a33ce43e1 in main ()
[Inferior 1 (process 225275) detached]
terminate called after throwing an instance of 'std::runtime_error'
  what():  Value is not callable: null at row 2, column 42:
{%- set today = strftime_now("%Y-%m-%d") %}
{%- set yesterday = (today | as_datetime - timedelta(days=1)).strftime("%Y-%m-%d") %}
                                         ^
{%- set default_system_message = "You are Mistral Small 3, a Large Language Model (LLM) created by Mistral AI, a French startup headquartered in Paris.\nYour knowledge base was last updated on 2023-10-01. The current date is " + today + ".\n\nWhen you're not sure about some information or when the user's request requires up-to-date or specific data, you must use the available tools to fetch the information. Do not hesitate to use tools whenever they can provide a more accurate or complete response. If no relevant tools are available, then clearly state that you don't have the information and avoid making up anything.
 at row 2, column 61:
{%- set today = strftime_now("%Y-%m-%d") %}
{%- set yesterday = (today | as_datetime - timedelta(days=1)).strftime("%Y-%m-%d") %}
                                                            ^
{%- set default_system_message = "You are Mistral Small 3, a Large Language Model (LLM) created by Mistral AI, a French startup headquartered in Paris.\nYour knowledge base was last updated on 2023-10-01. The current date is " + today + ".\n\nWhen you're not sure about some information or when the user's request requires up-to-date or specific data, you must use the available tools to fetch the information. Do not hesitate to use tools whenever they can provide a more accurate or complete response. If no relevant tools are available, then clearly state that you don't have the information and avoid making up anything.
 at row 2, column 63:
{%- set today = strftime_now("%Y-%m-%d") %}
{%- set yesterday = (today | as_datetime - timedelta(days=1)).strftime("%Y-%m-%d") %}
                                                              ^
{%- set default_system_message = "You are Mistral Small 3, a Large Language Model (LLM) created by Mistral AI, a French startup headquartered in Paris.\nYour knowledge base was last updated on 2023-10-01. The current date is " + today + ".\n\nWhen you're not sure about some information or when the user's request requires up-to-date or specific data, you must use the available tools to fetch the information. Do not hesitate to use tools whenever they can provide a more accurate or complete response. If no relevant tools are available, then clearly state that you don't have the information and avoid making up anything.
 at row 2, column 1:
{%- set today = strftime_now("%Y-%m-%d") %}
{%- set yesterday = (today | as_datetime - timedelta(days=1)).strftime("%Y-%m-%d") %}
^
{%- set default_system_message = "You are Mistral Small 3, a Large Language Model (LLM) created by Mistral AI, a French startup headquartered in Paris.\nYour knowledge base was last updated on 2023-10-01. The current date is " + today + ".\n\nWhen you're not sure about some information or when the user's request requires up-to-date or specific data, you must use the available tools to fetch the information. Do not hesitate to use tools whenever they can provide a more accurate or complete response. If no relevant tools are available, then clearly state that you don't have the information and avoid making up anything.
 at row 1, column 1:
{%- set today = strftime_now("%Y-%m-%d") %}
^
{%- set yesterday = (today | as_datetime - timedelta(days=1)).strftime("%Y-%m-%d") %}

Aborted (core dumped)

[CISC]

The template (specifically the yesterday parsing) is unsupported, you have to use the template found in models/templates instead, using the --chat-template-file option.

[allo-]

I'll try. But I think it still shouldn't crash. And an helpful error message or fallback would be nice!

[allo-]

I also think it is a regression. I just looked and I am using the same file since Jun 21, which worked with older llama.cpp. Maybe the older version didn't use the template in the file at all?

[CISC]

Not really a regression, just that a working template is required for various features to work, it does give an error message about what the problem is: Failed to generate tool call example: Value is not callable: null at row 2, column 42, this is currently a limitation in minja, which is used to parse the template.

Just use --chat-template-file with the working template in models/templates.

[allo-]

The same model with the same start script worked for me some revisions ago. Now llama.cpp crashes with a stacktrace.

I do not care (in this bug) about not working tool calls. I care about the llama-server program crashing. If llama.cpp would quit with an error message, it would probably okay, even though it worked (probably without tool calls) before. But I don't think it should crash on unsupported templates.

[CISC]

Ok, I see what the issue is now, I'll make a PR.

[allo-]

Thanks! You can ask me to test when it's merged. I thought about trying to bisect the commits anyway but hadn't the time yet, so if you think you found it I'd happy to test.

[CISC]

Thanks! You can ask me to test when it's merged. I thought about trying to bisect the commits anyway but hadn't the time yet, so if you think you found it I'd happy to test.

See linked PR, you'll have to build it yourself until Release CI is working again.

[allo-]

The fix works for me.

[allo-]

I guess now the question is, what needs to be done to deal with --jinja and invalid/unsupported templates, such that they result in an error message and not in a crash.

[CISC]

I guess now the question is, what needs to be done to deal with --jinja and invalid/unsupported templates, such that they result in an error message and not in a crash.

Why would you want that? If the template doesn't work you cannot use --jinja.

[allo-]

A crash is a crash and should be fixed? If something is unsupported a program should output an error message and either continue without the feature or quit with an error status.