lm_eval to 0.4.9.1 and support for new args - rebased

[12010486]

Cherry-picked from #2193 to have it on main.

Changes are as follow:
Argument parsing and evaluation configuration:

• Added new command-line arguments in run_lm_eval.py for controlling evaluation, including support for generation kwargs, few-shot and multi-turn settings, metadata, system instructions, chat template application, and sample selection. This allows for more granular and customizable evaluation runs.
• Add try_parse_json to robustly handle JSON or string input for generation arguments.
• Updated main evaluation logic to pass new arguments through to the evaluator, and added validation for combinations of options (e.g., requiring chat template when using multi-turn few-shot).

Model adapter enhancements:

• Added support for softmax_dtype, think_end_token, enable_thinking, and chat_template_args in HabanaModelAdapter initialization, enabling more advanced generation and prompt formatting features.
• Improved bucket selection logic for static shape generation, and switched to using max_new_tokens for generation instead of max_length, aligning with HuggingFace's recommended API usage.

Dependency update:

• Upgraded the lm-eval package to version 0.4.9.1 to support new features and bug fixes.

General improvements:

• Minor refactoring for imports and typing to support new features.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[AKloniecki]

We're currently working on running these changes in our CI, to check results after the upgrade. Will report back as soon as we get the results.

[AKloniecki]

Please update get_model_info() function to match current version at https://github.com/EleutherAI/lm-evaluation-harness/blob/v0.4.9.1/lm_eval/models/huggingface.py#L1522
It seems to have changed since our last update.

[12010486]

@AKloniecki, thank you very much for the review. On this specific point, it might be that we can actually remove the whole get_model_info(), as that add was due to a glitch in neural_compressor_pt, I'm checking in the background if it was solved with v 3.3.1, the latest on SynapseAI 1.21

[12010486]

I can confirm the error that required the get_model_info() patch is gone. Hence, my last commit is removing the function

[regisss]

LGTM!