Support mixtral kvcache reuse and remove kv_cache_fp8

[jychen21]

What does this PR do?

1. Support reuse_cache and bucket_internal;
2. Remove kv_cache_fp8;
3. Modify 'whitelist' and 'blacklist' to 'allowlist' and 'blocklist' in quantization_config/maxabs_quant_mixtral.json

Before

QUANT_CONFIG=./quantization_config/maxabs_quant_mixtral.json python run_generation.py --model_name_or_path mistralai/Mixtral-8x7B-v0.1 --use_hpu_graphs --use_kv_cache --bucket_size 128 --max_new_tokens 256 --bf16 --fp8 --batch_size 32 --max_input_tokens 128

input 32: ('Peace is the only way',)
output 1: ('Peace is the only way to end the war in Syria.\n\nThe Syrian people have suffered enough.\n\nThe war has killed hundreds of thousands of people, displaced millions, and destroyed the country’s infrastructure.\n\nThe international community must do everything in its power to end the conflict and bring peace to Syria.\n\nThe United Nations Security Council should pass a resolution calling for an immediate ceasefire and the start of negotiations between the Syrian government and the opposition.\n\nThe international community should also provide humanitarian assistance to the Syrian people and help rebuild the country.\n\nOnly by working together can we hope to achieve peace in Syria.\n\n## What is the best way to end the war in Syria?\n\nThe best way to end the war in Syria is to find a political solution that includes all parties involved. This would require a ceasefire and negotiations between the Syrian government and the opposition. The international community should also provide humanitarian assistance to the Syrian people and help rebuild the country.\n\n## How can the international community help to end the war in Syria?\n\nThe international community can help to end the war in Syria by providing humanitarian assistance to the Syrian people and helping to rebuild the country.',)

Stats:
Throughput (including tokenization) = 1136.5631648005235 tokens/second
Number of HPU graphs = 183
Memory allocated = 58.67 GB
Max memory allocated = 60.26 GB
Total memory available = 94.62 GB
Graph compilation duration = 227.78881454397924 seconds

After

QUANT_CONFIG=./quantization_config/maxabs_quant_mixtral.json python run_generation.py --model_name_or_path mistralai/Mixtral-8x7B-v0.1 --use_hpu_graphs --use_kv_cache --reuse_cache --bucket_internal --bucket_size 128 --max_new_tokens 256 --bf16 --fp8 --batch_size 32 --max_input_tokens 128

input 32: ('Peace is the only way',)
output 1: ('Peace is the only way to end the war in Syria.\n\nThe Syrian people have suffered enough.\n\nThe war has killed hundreds of thousands of people, displaced millions, and destroyed the country’s infrastructure.\n\nThe international community must do everything in its power to end the conflict and bring peace to Syria.\n\nThe United Nations Security Council should pass a resolution calling for an immediate ceasefire and the start of negotiations between the Syrian government and the opposition.\n\nThe international community should also provide humanitarian assistance to the Syrian people and help rebuild the country.\n\nOnly by working together can we hope to achieve peace in Syria.\n\n## What is the best way to end the war in Syria?\n\nThe best way to end the war in Syria is to find a political solution that includes all parties involved. This would require a ceasefire and negotiations between the Syrian government and the opposition. The international community should also provide humanitarian assistance to the Syrian people and help rebuild the country.\n\n## How can the international community help to end the war in Syria?\n\nThe international community can help to end the war in Syria by providing humanitarian assistance to the Syrian people and helping to rebuild the country.',)

Stats:
Throughput (including tokenization) = 1221.6872883013737 tokens/second
Number of HPU graphs = 180
Memory allocated = 52.36 GB
Max memory allocated = 52.45 GB
Total memory available = 94.62 GB
Graph compilation duration = 216.56420183897717 seconds

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

[jychen21]

Break PR #836 into small pieces, based on PR #892

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[jychen21]

Hi @regisss, could you help review this PR, it's based on your PR 'linear and dynamic rope', thanks!

[ZhaiFeiyue]

LGTM!!!

[ZhaiFeiyue]

@jychen-habana nice PR and we could enable reuse cache for Mixtral!
@regisss How do you think if we draft a guide about how to tuning accuracy with HQT for LLM that others can refer, since we have changed the allowlist and blocklist?

[jychen21]

Hi @regisss could you please review and merge this PR?