Skip to content

Add CLI to evaluate models, list supported tasks etc #53

@lewtun

Description

@lewtun

If would be nice if one could launch evaluation jobs directly from the command line with something like:

lighteval --tasks="lighteval|hellaswag|5|1" --output_dir "/scratch/evals" --model_args "pretrained=gpt2

By default we could look for the default accelerate config, but allow users to override this if needed with

lighteval --accelerate_config=path/to/accelerate/config --tasks="lighteval|hellaswag|5|1" --output_dir "/scratch/evals" --model_args "pretrained=gpt2

It would also be nice if the CLI would produce a list of supported tasks with something like

lighteval --list-tasks

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions