Lm eval accuracy regression fix

[12010486]

What does this PR do?

Fix a small regression, ~-0.39% in some of the fp8 eval harness.

For example, if you execute:

PT_HPU_LAZY_MODE=1 HF_DATASETS_TRUST_REMOTE_CODE=true QUANT_CONFIG=/root/optimum-habana/examples/text-generation//quantization_config//maxabs_measure.json  TQDM_DISABLE=1 python3  run_lm_eval.py --model_name_or_path meta-llama/Llama-3.1-8B-Instruct --warmup 0 --use_hpu_graphs -o test_results_measure.json --bf16 --batch_size 1 --use_kv_cache --trim_logits --attn_softmax_bf16 --bucket_size=128 --bucket_internal --trust_remote_code --tasks hellaswag lambada_openai piqa winogrande mathqa pubmedqa arc_easy arc_challenge

and

PT_HPU_LAZY_MODE=1 HF_DATASETS_TRUST_REMOTE_CODE=true QUANT_CONFIG=/root/optimum-habana/examples/text-generation//quantization_config//act_maxabs_pow2_weights_pcs_opt_pow2_quant.json  TQDM_DISABLE=1  python3  run_lm_eval.py --model_name_or_path meta-llama/Llama-3.1-8B-Instruct --warmup 0 --use_hpu_graphs -o test_results_quant.json --bf16 --batch_size 1 --use_kv_cache --trim_logits --attn_softmax_bf16 --bucket_size=128 --bucket_internal --trust_remote_code --tasks hellaswag lambada_openai piqa winogrande mathqa pubmedqa arc_easy arc_challenge     --show_config

with this fix we have an average accuracy of 66.683, while before we dropped to 66.422