llama-quantize: --dry-run option

[ikawrakow]

Prints the tensor types and resulting tensor sizes, but does not run the quantization, so it is very fast.

Useful for experimenting with --custom-q before running the actual quantization.

Enable with --dry-run, which needs to appear before the model name as all other optional llama-quantize arguments.

[Nexesenex]

I was about to (try to) copyport it myself.
Thanks!

[ubergarm]

Tested and confirmed this branch is working and does not create/overwrite any specified output file when using --dry-run.

Here is an example quantize command like I generally use, and now with --dry-run I can confirm final model size and tweak as desired to hit a target breakpoint with more confidence.

👈 Details

#!/usr/bin/env bash

custom="
# 60 Repeating Layers [0-59]

## Gated Attention/Delta Net [Blended 0-59]
blk\..*\.attn_gate\.weight=q8_0
blk\..*\.attn_qkv\.weight=q8_0
blk\..*\.attn_output\.weight=q8_0
blk\..*\.attn_q\.weight=q8_0
blk\..*\.attn_k\.weight=q8_0
blk\..*\.attn_v\.weight=q8_0
blk\..*\.ssm_ba\.weight=q8_0
blk\..*\.ssm_out\.weight=q8_0

# Shared Expert Layers [0-59]
blk\..*\.ffn_down_shexp\.weight=q8_0
blk\..*\.ffn_(gate|up)_shexp\.weight=q8_0

# Routed Experts Layers [0-59]
blk\..*\.ffn_down_exps\.weight=iq2_kt
blk\..*\.ffn_(gate|up)_exps\.weight=iq1_kt

# Non-Repeating Layers
token_embd\.weight=iq4_k
output\.weight=iq6_k
"

custom=$(
  echo "$custom" | grep -v '^#' | \
  sed -Ez 's:\n+:,:g;s:,$::;s:^,::'
)

    #--exclude-weights ffn_gate_exps \
    #--exclude-weights ffn_up_exps \
numactl -N ${SOCKET} -m ${SOCKET} \
./build/bin/llama-quantize \
    --dry-run \
    --custom-q "$custom" \
    --imatrix /mnt/data/models/ubergarm/Qwen3-Coder-Next-GGUF/imatrix-Qwen3-Coder-Next-BF16.dat \
    /mnt/data/models/ubergarm/Qwen3-Coder-Next-GGUF/Qwen3-Coder-Next-512x2.5B-BF16-00001-of-00004.gguf \
    /mnt/data/models/ubergarm/Qwen3-Coder-Next-GGUF/Qwen3-Coder-Next-IQ1_KT.gguf \
    IQ1_KT \
    128

It does not let you know if there will be errors due to imatrix related stuff, but that is fine with me. For example. it seems like the qwen3next models don't like very low quantization types e.g. iq1_kt, iq2_kt, iq2_kl, iq1_s at least for ffn_(gate|up)_exps (but ik2_ks is fine). But this is not specific to this PR. fwiw seems to error out even when skipping imatrix data for those, just noting the observation:

converting to q8_0 .. size =    16.00 MiB ->     8.50 MiB
[  11/ 843]           blk.0.ffn_down_exps.weight - [  512,  2048,   512,     1], type =   bf16, Using custom type iq2_kt for tensor blk.0.ffn_down_exp
s.weight
converting to iq2_kt .. size =  1024.00 MiB ->   140.00 MiB
[  12/ 843]           blk.0.ffn_gate_exps.weight - [ 2048,   512,   512,     1], type =   bf16, Using custom type iq1_kt for tensor blk.0.ffn_gate_exp
s.weight
converting to iq1_kt .. Oops: jbest = -1 for cluster 98 with 1160 points
/home/w/projects/ik_llama.cpp/ggml/src/iqk/iqk_quantize.cpp:8334: GGML_ASSERT(false) failed
Oops: jbest = -1 for cluster 98 with 1160 points
/home/w/projects/ik_llama.cpp/ggml/src/iqk/iqk_quantize.cpp:8334: GGML_ASSERT(false) failed
Oops: jbest = -1 for cluster 240 with 1024 points
.
.
.

Yes thanks for porting this feature and thanks to @ddh0 on ggml-org/llama.cpp#19526 which I was using a rough "vibe port" of myself already xD

[ikawrakow]

It does not let you know if there will be errors due to imatrix related stuf

It doesn't? It is supposed to. Commenting out the check in examples/quantize/quantize.cpp (so we can get past the imatrix police) and running

./bin/llama-quantize --dry-run $some_model junk iq2_xxs

I get

main: quantizing '../models/il32_3B/Llama-3.2-3B-Instruct-BF16.gguf' to 'mist.bin' as IQ2_XXS
llama_model_loader: loaded meta data with 31 key-value pairs and 255 tensors from ../models/il32_3B/Llama-3.2-3B-Instruct-BF16.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Llama 3.2 3B Instruct
llama_model_loader: - kv   3:                           general.finetune str              = Instruct
llama_model_loader: - kv   4:                           general.basename str              = Llama-3.2
llama_model_loader: - kv   5:                         general.size_label str              = 3B
llama_model_loader: - kv   6:                            general.license str              = llama3.2
llama_model_loader: - kv   7:                               general.tags arr[str,6]       = ["facebook", "meta", "pytorch", "llam...
llama_model_loader: - kv   8:                          general.languages arr[str,8]       = ["en", "de", "fr", "it", "pt", "hi", ...
llama_model_loader: - kv   9:                          llama.block_count u32              = 28
llama_model_loader: - kv  10:                       llama.context_length u32              = 131072
llama_model_loader: - kv  11:                     llama.embedding_length u32              = 3072
llama_model_loader: - kv  12:                  llama.feed_forward_length u32              = 8192
llama_model_loader: - kv  13:                 llama.attention.head_count u32              = 24
llama_model_loader: - kv  14:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv  15:                       llama.rope.freq_base f32              = 500000.000000
llama_model_loader: - kv  16:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  17:                 llama.attention.key_length u32              = 128
llama_model_loader: - kv  18:               llama.attention.value_length u32              = 128
llama_model_loader: - kv  19:                          general.file_type u32              = 32
llama_model_loader: - kv  20:                           llama.vocab_size u32              = 128256
llama_model_loader: - kv  21:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv  22:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  23:                         tokenizer.ggml.pre str              = llama-bpe
llama_model_loader: - kv  24:                      tokenizer.ggml.tokens arr[str,128256]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  25:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  26:                      tokenizer.ggml.merges arr[str,280147]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv  27:                tokenizer.ggml.bos_token_id u32              = 128000
llama_model_loader: - kv  28:                tokenizer.ggml.eos_token_id u32              = 128009
llama_model_loader: - kv  29:                    tokenizer.chat_template str              = {{- bos_token }}\n{%- if custom_tools ...
llama_model_loader: - kv  30:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   58 tensors
llama_model_loader: - type bf16:  197 tensors
[   1/ 255]                    rope_freqs.weight - [   64,     1,     1,     1], type =    f32, size =    0.000 MB
[   2/ 255]                    token_embd.weight - [ 3072, 128256,     1,     1], type =   bf16, converting to iq4_k .. size =   751.50 MiB ->   211.36 MiB
[   3/ 255]               blk.0.attn_norm.weight - [ 3072,     1,     1,     1], type =    f32, size =    0.012 MB
[   4/ 255]                blk.0.ffn_down.weight - [ 8192,  3072,     1,     1], type =   bf16, converting to q2_K .. size =    48.00 MiB ->     7.88 MiB
[   5/ 255]                blk.0.ffn_gate.weight - [ 3072,  8192,     1,     1], type =   bf16, 

============================================================
Missing importance matrix for tensor blk.0.ffn_gate.weight in a very low-bit quantization
The result will be garbage, so bailing out
============================================================

llama_model_quantize: failed to quantize: Missing importance matrix for tensor blk.0.ffn_gate.weight in a very low-bit quantization

so yes, it will abort if there are missing entries in the imatrix data.

[ubergarm]

so yes, it will abort if there are missing entries in the imatrix data.

Ah yes, so it does. The error I'm getting is unrelated to imatrix then, I can open an issue specific for that, thanks for confirming!