Multilingual Reading Comprehension tasks (#333)

* add multilignaul dynamic generative metrics

* draft

* finish multichoice config

* update tokenizers + install nltk reqs

* use punkt tab

* Update src/lighteval/utils/imports.py

Co-authored-by: Nathan Habib <[email protected]>

* Update src/lighteval/metrics/normalizations.py

Co-authored-by: Nathan Habib <[email protected]>

* fix imports

* remove unused import

* finish implementation of templates + move stuff around

* resolve nits

* when in rome do as romans do (handle error messages the same way)

* fix utils

* nicers tests + fix them

* nicer todo

* add nice doscrings 📃

* add even more docstring

* nit

* fix test

* add multilingual to dev group

* merge nli, add languagees to literals

* translation literals

* add nli

* add copa tasks + fix tranlation literals

* add hellaswag tasks

* remove custom telgu hellaswag

* remove hindi hellaswag

* add rc tasks + small nits

* add rcb + chinese nli

* Update src/lighteval/tasks/multilingual/tasks.py

Co-authored-by: Clémentine Fourrier <[email protected]>

* Update src/lighteval/tasks/multilingual/tasks.py

Co-authored-by: Clémentine Fourrier <[email protected]>

* Update src/lighteval/tasks/multilingual/tasks.py

Co-authored-by: Clémentine Fourrier <[email protected]>

* Update src/lighteval/tasks/multilingual/tasks.py

Co-authored-by: Clémentine Fourrier <[email protected]>

* Update src/lighteval/tasks/multilingual/tasks.py

Co-authored-by: Clémentine Fourrier <[email protected]>

* Update src/lighteval/tasks/multilingual/tasks.py

Co-authored-by: Clémentine Fourrier <[email protected]>

* Update src/lighteval/tasks/multilingual/tasks.py

Co-authored-by: Clémentine Fourrier <[email protected]>

* add two new tasks + docs

* add nice docs

* update hellaswag with docs

* move hellaswag to lighteval suite

* add desc to tasks

* Update src/lighteval/tasks/multilingual/tasks.py

Co-authored-by: Clémentine Fourrier <[email protected]>

* enable returning none from templates + better typing

* change unoficial hellaswag names to have community_prefix + unify hellaswag preprocesisng

* let strip be optional in hellaswag

* nits

* add comment

* update the datasets after changing ownership

---------

Co-authored-by: Nathan Habib <[email protected]>
Co-authored-by: Hynek Kydlicek <[email protected]>
Co-authored-by: Clémentine Fourrier <[email protected]>

Author: 4 people

src/lighteval/metrics/dynamic_metrics.py

@@ -119,7 +119,7 @@ def multilingual_quasi_f1_score_metric(
     Returns:
         F1 score metric.
     """
-    metric_name = f"f1_{language}"
+    metric_name = f"f1_{language.value}"
 
     multilang_normalizer = get_multilingual_normalizer(language)
     return SampleLevelMetric(
@@ -153,7 +153,7 @@ def multilingual_quasi_exact_match_metric(
     Returns:
         Exact match metric.
     """
-    metric_name = f"exact_match_{language}_{match_type}"
+    metric_name = f"exact_match_{language.value}_{match_type}"
     multilang_normalizer = get_multilingual_normalizer(language)
     return SampleLevelMetric(
         metric_name=metric_name,