Sarkar/mistral optimizations

[ssarkar2]

What does this PR do?

Pull in optimizations from llama for mistral

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[regisss]

I left several comments. Can you also provide a command that I can use for testing please?

[ssarkar2]

I left several comments. Can you also provide a command that I can use for testing please?

command line to test (as tested on gaudi 2):
For gaudi1 the batch sizes will be smaller
#cfg 1
python run_generation.py --model_name_or_path mistralai/Mistral-7B-v0.1 --use_hpu_graphs --use_kv_cache --bf16 --batch_size 500 --max_input_tokens 128 --max_new_tokens 128 --reuse_cache --trim_logits --attn_softmax_bf16

#cfg 2:
python run_generation.py --model_name_or_path mistralai/Mistral-7B-v0.1 --use_hpu_graphs --use_kv_cache --bf16 --batch_size 116 --max_input_tokens 128 --max_new_tokens 2048 --reuse_cache --trim_logits --bucket_internal --bucket_size 256 --attn_softmax_bf16

#cfg 3:
python run_generation.py --model_name_or_path mistralai/Mistral-7B-v0.1 --use_hpu_graphs --use_kv_cache --bf16 --batch_size 116 --max_input_tokens 2048 --max_new_tokens 128 --reuse_cache --trim_logits --attn_softmax_bf16 --limit_hpu_graphs

#cfg 4
python run_generation.py --model_name_or_path mistralai/Mistral-7B-v0.1 --use_hpu_graphs --use_kv_cache --bf16 --batch_size 46 --max_input_tokens 2048 --max_new_tokens 2048 --reuse_cache --trim_logits --attn_softmax_bf16 --limit_hpu_graphs --bucket_internal --bucket_size 256

[regisss]

The commented line is still there