Adding TinyBench

[clefourrier]

Should be good to review.

Test with:

python run_evals_accelerate.py --model_args "pretrained=EleutherAI/pythia-70m" --tasks "extended|tiny:winogrande|0|0,extended|tiny:gsm8k|0|0,extended|tiny:hellaswag|0|0,extended|tiny:arc|0|0,extended|tiny:truthfulqa|0|0" --extended_tasks extended_tasks --output_dir "./evals"

[clefourrier]

Before this fix, we were overwritting the aggregations! (which was never a problem since we tend to use the same aggregation for similar evals, like np.mean for acc)

[lewtun]

Maybe good to add your sample usage command here? Otherwise it's hard for users to know how to run it

[clefourrier]

Will do, thanks for the edit!

[lewtun]

I'm getting this error when running the command in the description:

Traceback (most recent call last):
  File "/fsx/lewis/git/hf/lighteval/run_evals_accelerate.py", line 125, in <module>
    main(args)
  File "/fsx/lewis/git/hf/lighteval/src/lighteval/logging/hierarchical_logger.py", line 166, in wrapper
    return fn(*args, **kwargs)
  File "/fsx/lewis/git/hf/lighteval/src/lighteval/main_accelerate.py", line 83, in main
    task_dict = Registry(cache_dir=env_config.cache_dir).get_task_dict(
  File "/fsx/lewis/git/hf/lighteval/src/lighteval/tasks/registry.py", line 141, in get_task_dict
    custom_tasks_module.extend(load_extended_tasks_modules(extended_tasks_path=extended_tasks))
  File "/fsx/lewis/git/hf/lighteval/src/lighteval/tasks/registry.py", line 161, in load_extended_tasks_modules
    cur_module = create_custom_tasks_module(os.path.join(extended_tasks_path, folder, "main.py"))
  File "/fsx/lewis/git/hf/lighteval/src/lighteval/tasks/registry.py", line 181, in create_custom_tasks_module
    return importlib.import_module(dataset_module.module_path)
  File "/fsx/lewis/miniconda3/envs/lighteval/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/fsx/h4/.cache/datasets_modules/datasets/main/e1b8c49ef9a80f593418f38b008371bd75e4d5f4455a007fed4f3723c6810948/main.py", line 214, in <module>
    task_params["prompt"],
TypeError: list indices must be integers or slices, not str

[clefourrier]

I know what the problem is, thanks a lot for your report, fixing. None of my modifs were pushed on the main file, should be good now @lewtun
(& thanks a lot for taking a first look, sorry about that)