Enable fast softmax mode in FusedSDPA

[wszczurekhabana]

Support for setting fast softmax mode in FusedSDPA operator. This is a tradeoff: performance vs accuracy.

Data on performance:

Ratio	Max input tokens	Max new tokens	Batch size	Throughput without fast softmax [tokens/s]	Throughput with fast softmax [tokens/s]	Improvement %
97%	31744	1042	12	139.08	147.97	6.4%
75%	24576	8192	16	431.09	437.95	1.6%
50%	16384	16384	24	653.39	656.38	0.5%

Data on accuracy (using mlperf test from: https://gerrit.habana-labs.com/plugins/gitiles/mlperf_inference/+/refs/heads/master_next/code/llama/llama_greedy.py
and https://gerrit.habana-labs.com/plugins/gitiles/mlperf_inference/+/refs/heads/master_next/code/llama/evaluation.py):

	rouge1	rouge2	rougeL	rougeLsum	accuracy
without fast softmax	44.4279	22.0536	28.6362	42.0044	99.99
with fast softmax	44.4065	22.0229	28.6156	41.9858	99.94

[dudilester]

This ModuleFusedSDPA forward API change will require changes in the HQT patched module for quantization. which means it will break the nightly testing once merged. Im not sure we support regular-softmax for 8bit, we need to consider the appropriate behavior when user requests both quantization and regular-softmax, should we ignore the quantization or the softmax? or assert on that configuration.

[wszczurekhabana]

Discussed offline. Relevant change for quantization toolkit: https://gerrit.habana-labs.com/#/c/411008/ pushed by @dudilester is in review.

[wszczurekhabana]

Change in https://gerrit.habana-labs.com/#/c/411008/ is merged. @dvarshney-habana @puneeshkhanna @dudilester I think we can merge this PR now.

[dudilester]

Change in https://gerrit.habana-labs.com/#/c/411008/ did not pass promotion yet, we need to wait till it will pass before we merge this PR.

[dudilester]

FYI, commit https://gerrit.habana-labs.com/#/c/411008/ was promoted since my previous comment, and is included in builds since CD 1.16.0-328 release build

[wszczurekhabana]

Thanks, I was not tracking it closely. @dvarshney-habana can we merge it?

[wszczurekhabana]

upstreamed in: huggingface#972