change quant conf example to use act_maxabs_pow2_weights_pcs_opt_pow2

[HolyFalafel]

change quant conf example to use act_maxabs_pow2_weights_pcs_opt_pow2

[regisss]

@HolyFalafel What's the difference between both methods?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[HolyFalafel]

@regisss

• maxabs_hw - Scale is calculated from the maxabs measurement and then aligned to the corresponding HW accelerated scale.
• maxabs_hw_opt_weight - Scale of model params (weights) is chosen as the scale that provides minimal mean-square-error between quantized and unquantized weights, from all possible HW accelerated scales. Scale of activations is calculated in same manner as maxabs_hw.
• maxabs_pow2 - Scale is calculated from the maxabs measurement and then rounded to the power of 2.
• act_maxabs_pow2_weights_pcs_opt_pow2 - Scale of model params (weights) is calculated per-channel of the params tesnor. The scale per-channel is calculated in same manner as maxabs_hw_opt_weight. Scale of activations is calculated in same manner as maxabs_pow2.
• act_maxabs_hw_weights_pcs_maxabs_pow2 - Scale of model params (weights) is calculated per-channel of the params tesnor. The scale per-channel is calculated in same manner as maxabs_pow2. Scale of activations is calculated in same manner as maxabs_hw.

The method we changed into gave us better accuracy

[regisss]

Thanks for the explanation!

Your branch is quite far from the head of the main branch (it still relies on Transformers v4.34), could you rebase it so that I can test it please?

[HolyFalafel]

I wanted to minimize the number of commits. I'll rebase

[regisss]

Just to make sure I test it correctly, I first need to measure with

QUANT_CONFIG=./quantization_config/maxabs_measure.json python ../gaudi_spawn.py --use_deepspeed --world_size 8 run_generation.py \
  --model_name_or_path meta-llama/Llama-2-70b-hf \
  --attn_softmax_bf16 \
  --use_hpu_graphs \
  --trim_logits \
  --use_kv_cache \
  --reuse_cache \
  --bf16 \
  --batch_size 1

and then run the quantized model with

QUANT_CONFIG=./quantization_config/act_maxabs_hw_weights_pcs_maxabs_pow2_quant.json python ../gaudi_spawn.py --use_deepspeed --world_size 8 run_generation.py \
  --model_name_or_path meta-llama/Llama-2-70b-hf \
  --attn_softmax_bf16 \
  --use_hpu_graphs \
  --trim_logits \
  --use_kv_cache \
  --reuse_cache \
  --bf16 \
  --batch_size 1 \
  --fp8

right?

[regisss]

The second command returns this error:

File "/root/workspace/fork/examples/text-generation/utils.py", line 246, in setup_distributed_model
    habana_quantization_toolkit.prep_model(model)
  File "/usr/local/lib/python3.10/dist-packages/habana_quantization_toolkit/prepare_quant/prepare_model.py", line 12, in prep_model
    prepare_model(model)  # registers hooks
  File "/usr/local/lib/python3.10/dist-packages/habana_quantization_toolkit/_hook_method/__init__.py", line 45, in prepare_model
    return quantize_hooks(model, mod_list)
  File "/usr/local/lib/python3.10/dist-packages/habana_quantization_toolkit/_hook_method/quantize.py", line 67, in quantize_hooks
    scale_convert_method = scale_convert_methods[scaling_method_name]
KeyError: 'act_maxabs_pts_pow2_weights_opt_pcs_pow2'

Is it supposed to work with v1.14 or should I wait for v1.15?

[HolyFalafel]

The second command returns this error:

File "/root/workspace/fork/examples/text-generation/utils.py", line 246, in setup_distributed_model
    habana_quantization_toolkit.prep_model(model)
  File "/usr/local/lib/python3.10/dist-packages/habana_quantization_toolkit/prepare_quant/prepare_model.py", line 12, in prep_model
    prepare_model(model)  # registers hooks
  File "/usr/local/lib/python3.10/dist-packages/habana_quantization_toolkit/_hook_method/__init__.py", line 45, in prepare_model
    return quantize_hooks(model, mod_list)
  File "/usr/local/lib/python3.10/dist-packages/habana_quantization_toolkit/_hook_method/quantize.py", line 67, in quantize_hooks
    scale_convert_method = scale_convert_methods[scaling_method_name]
KeyError: 'act_maxabs_pts_pow2_weights_opt_pcs_pow2'

Is it supposed to work with v1.14 or should I wait for v1.15?

Yes, it's a change done in v1.15, so let's wait for it

Just to make sure I test it correctly, I first need to measure with

QUANT_CONFIG=./quantization_config/maxabs_measure.json python ../gaudi_spawn.py --use_deepspeed --world_size 8 run_generation.py \
  --model_name_or_path meta-llama/Llama-2-70b-hf \
  --attn_softmax_bf16 \
  --use_hpu_graphs \
  --trim_logits \
  --use_kv_cache \
  --reuse_cache \
  --bf16 \
  --batch_size 1

and then run the quantized model with

QUANT_CONFIG=./quantization_config/act_maxabs_hw_weights_pcs_maxabs_pow2_quant.json python ../gaudi_spawn.py --use_deepspeed --world_size 8 run_generation.py \
  --model_name_or_path meta-llama/Llama-2-70b-hf \
  --attn_softmax_bf16 \
  --use_hpu_graphs \
  --trim_logits \
  --use_kv_cache \
  --reuse_cache \
  --bf16 \
  --batch_size 1 \
  --fp8

right?

This looks correct. @bgoldberg-habana any remarks?

[HolyFalafel]

@regisss
These are the commands for the whole process:

Please notice that we measure with run_lm_eval.py to catch more data.

Measurement:

QUANT_CONFIG=./quantization_config/maxabs_measure.json python ../gaudi_spawn.py --use_deepspeed --world_size 8 run_lm_eval.py \
  --model_name_or_path meta-llama/Llama-2-70b-hf \
  --attn_softmax_bf16 \
  --use_hpu_graphs \
  --trim_logits \
  --use_kv_cache \
  --reuse_cache \
  --bf16 \
  --batch_size 1 \
  -o 70b_bs1_measure.txt

Per Tensor Quantization (PTQ):

QUANT_CONFIG=./quantization_config/maxabs_quant.json python ../gaudi_spawn.py --use_deepspeed --world_size 8 run_lm_eval.py \
  --model_name_or_path meta-llama/Llama-2-70b-hf \
  --attn_softmax_bf16 \
  --use_hpu_graphs \
  --trim_logits \
  --use_kv_cache \
  --reuse_cache \
  --bf16 \
  --batch_size 1 \
  --fp8 \
  -o 70b_bs1_ptq_quant.txt

Per Channel Quantization (PCQ):

QUANT_CONFIG=./quantization_config/act_maxabs_hw_weights_pcs_maxabs_pow2_quant.json  python ../gaudi_spawn.py --use_deepspeed --world_size 8 run_lm_eval.py \
  --model_name_or_path meta-llama/Llama-2-70b-hf \
  --attn_softmax_bf16 \
  --use_hpu_graphs \
  --trim_logits \
  --use_kv_cache \
  --reuse_cache \
  --bf16 \
  --batch_size 1 \
  --fp8 \
  -o 70b_bs1_pcq_quant.out

You can also run on 7B

[regisss]

LGTM!

Waiting for the release of Synapse v1.15 to merge.

[regisss]

@HolyFalafel I guess we can close this PR as this change was present in #765 too?

[HolyFalafel]

@HolyFalafel I guess we can close this PR as this change was present in #765 too?

Right