Enable fused rmsnorm in bf16 for llama

[puneeshkhanna]

What does this PR do?

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

[puneeshkhanna]

@regisss - please review. We can enable fused rmsnorm in lower precision too and this gives a boost in performance too.

Command -> python ../gaudi_spawn.py --use_deepspeed --world_size 8 run_generation.py --model_name_or_path /software/data/llama_inference/Llama-2-70b-hf/ --max_new_tokens ?? --bf16 --n_iterations 3 --use_hpu_graphs --use_kv_cache --batch_size ?? --reuse_cache --limit_hpu_graphs --trim_logits --warmup 2 --attn_softmax_bf16

See below table for improved perf results:

<style> </style>

70B 8x	BS	Max new tokens	Default Perf	Perf with rmsnorm fix	% Improvement over default perf
	1	100	55.7	57.995	3.68
	40	100	1893.22	1969.95	3.5
	1	2048	60.53	62.19	2.58
	40	2048	1686.35	1726.47	2.5
	60	2048	2207.2	2256.79	2.2
	1	4096	59.825	61.4	2.48
	40	4096	1366.72	1393.36	1.95
	60	4096	1688.21	1716.64	1.74
7B 1x	1	4096	124.147	126.8	2.15
	4	4096	354.48	357.28	0.8
13B 1x	1	4096	68.86	69.79	1.37
	4	4096	203.77	204.89	0.56

[bgoldberg-habana]

LGTM, was also verified in FP8 runs.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[regisss]

Nice! Does it generate the same outputs as before?

[mandy-li]

@regisss , let me check finetuning as well for perf and accuracy.

[puneeshkhanna]

@mandy-li - thanks for checking finetuning. I guess there were no issues there too. Did perf also improve in finetuning too ?
@regisss - thanks for merging.