Imatrix quantization bug: OLMo-2-0325-32B-Instruct found nan value

[bartowski1182]

Name and Version

b4896

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-quantize

Command line

./llama-quantize --imatrix /models/OLMo-2-0325-32B-Instruct-GGUF/OLMo-2-0325-32B-Instruct.imatrix /models/OLMo-2-0325-32B-Instruct-GGUF/OLMo-2-0325-32B-Instruct-bf16.gguf /models/OLMo-2-0325-32B-Instruct-GGUF/OLMo-2-0325-32B-Instruct-Q4_K_M.gguf Q4_K_M

Problem description & steps to reproduce

Affects Q2_K, Q4_K, and Q5_K on the same blocks

Does not seem to affect Q6_K, IQ4_NL, IQ4_XS, Q4_0, Q3_K, or any IQ3/2_N quant levels

Imatrix file uploaded here for testing:

https://huggingface.co/bartowski/PleaseIgnore_uploaded_for_testing/blob/main/allenai_OLMo-2-0325-32B-Instruct.imatrix

BF16 conversion also uploaded for testing convenience:

https://huggingface.co/bartowski/PleaseIgnore_uploaded_for_testing/tree/main/OLMo-2-0325-32B-Instruct-bf16

First Bad Commit

No response

Relevant log output

[ 466/ 707]                 blk.42.attn_k.weight - [ 5120,  1024,     1,     1], type =   bf16, converting to q4_K .. ggml_validate_row_data: found nan value at block 40
ggml_validate_row_data: found nan value at block 20
ggml_validate_row_data: found nan value at block 40
ggml_validate_row_data: found nan value at block 20
ggml_validate_row_data: found nan value at block 20
ggml_validate_row_data: found nan value at block 40

[bartowski1182]

Likely related to #11764

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.