Multilingual Hellaswag tasks #332

hynky1999 · 2024-09-25T18:06:20Z

Goal

Add Hellaswag tasks supporting about 36 unique languages.

mlmm_hellaswag https://github.com/nlp-uoregon/mlmm-evaluation
hellaswag_{tha/tur} No paper for these, but we found them to work good enough

Since it uses template it support all 3 types of formulation out of the box :)

How to test:

lighteval accelerate --output_dir=./tmp --custom_tasks="lighteval.tasks.multilingual.tasks" --tasks="custom|{task}|0|0" --model_args=pretrained=gpt2 --override_batch_size=1 --max_samples=100 --save_details

where task in hellaswag_tel_cf|0|0 hellaswag_tur_cf|0|0 hellaswag_tha_cf|0|0

Comments

We talked about removing the suite all together. Since right now we use suite I decided to use custom suite. I can switch to multilingual or whatever

Co-authored-by: Nathan Habib <[email protected]>

src/lighteval/tasks/templates/hellaswag.py

src/lighteval/tasks/multilingual/tasks.py

Co-authored-by: Clémentine Fourrier <[email protected]>

…l into multilnag_nli_tasks

Co-authored-by: Clémentine Fourrier <[email protected]>

…laswag preprocesisng

clefourrier · 2024-10-01T12:36:10Z

src/lighteval/tasks/templates/utils/translation_literals.py

    Language.SINDHI: TranslationLiterals(language=Language.SINDHI),
    Language.TURKMEN: TranslationLiterals(language=Language.TURKMEN),
    Language.SOUTH_AZERBAIJANI: TranslationLiterals(language=Language.SOUTH_AZERBAIJANI),
    Language.SORANI: TranslationLiterals(language=Language.SORANI),


In the future, would be nice to sort this one alphabetically to avoid duplicates

* add multilignaul dynamic generative metrics * draft * finish multichoice config * update tokenizers + install nltk reqs * use punkt tab * Update src/lighteval/utils/imports.py Co-authored-by: Nathan Habib <[email protected]> * Update src/lighteval/metrics/normalizations.py Co-authored-by: Nathan Habib <[email protected]> * fix imports * remove unused import * finish implementation of templates + move stuff around * resolve nits * when in rome do as romans do (handle error messages the same way) * fix utils * nicers tests + fix them * nicer todo * add nice doscrings 📃 * add even more docstring * nit * fix test * add multilingual to dev group * merge nli, add languagees to literals * translation literals * add nli * add copa tasks + fix tranlation literals * add hellaswag tasks * remove custom telgu hellaswag * remove hindi hellaswag * add rcb + chinese nli * Update src/lighteval/tasks/multilingual/tasks.py Co-authored-by: Clémentine Fourrier <[email protected]> * Update src/lighteval/tasks/multilingual/tasks.py Co-authored-by: Clémentine Fourrier <[email protected]> * Update src/lighteval/tasks/multilingual/tasks.py Co-authored-by: Clémentine Fourrier <[email protected]> * Update src/lighteval/tasks/multilingual/tasks.py Co-authored-by: Clémentine Fourrier <[email protected]> * Update src/lighteval/tasks/multilingual/tasks.py Co-authored-by: Clémentine Fourrier <[email protected]> * Update src/lighteval/tasks/multilingual/tasks.py Co-authored-by: Clémentine Fourrier <[email protected]> * Update src/lighteval/tasks/multilingual/tasks.py Co-authored-by: Clémentine Fourrier <[email protected]> * add two new tasks + docs * add nice docs * update hellaswag with docs * move hellaswag to lighteval suite * Update src/lighteval/tasks/multilingual/tasks.py Co-authored-by: Clémentine Fourrier <[email protected]> * enable returning none from templates + better typing * change unoficial hellaswag names to have community_prefix + unify hellaswag preprocesisng * let strip be optional in hellaswag --------- Co-authored-by: Nathan Habib <[email protected]> Co-authored-by: Hynek Kydlicek <[email protected]> Co-authored-by: Clémentine Fourrier <[email protected]>

hynky1999 and others added 30 commits September 5, 2024 17:40

add multilignaul dynamic generative metrics

5c69eb0

Merge branch 'main' into geneartive_dynamic_metrics

39c4220

Merge branch 'geneartive_dynamic_metrics' into config_templates

2a5cdca

draft

2df9a08

finish multichoice config

95729ee

Merge branch 'main' into geneartive_dynamic_metrics

3aa0579

update tokenizers + install nltk reqs

b8f90a9

use punkt tab

f5a8717

Update src/lighteval/utils/imports.py

227f572

Co-authored-by: Nathan Habib <[email protected]>

Update src/lighteval/metrics/normalizations.py

d80b3ba

Co-authored-by: Nathan Habib <[email protected]>

fix imports

532bdad

remove unused import

75f7ac5

Merge branch 'main' into geneartive_dynamic_metrics

f99e330

Merge branch 'main' into geneartive_dynamic_metrics

92daf90

Merge branch 'main' into geneartive_dynamic_metrics

f2a801d

finish implementation of templates + move stuff around

91d9d4f

resolve nits

9356cc6

when in rome do as romans do (handle error messages the same way)

0fbc731

fix utils

fa1fa83

Merge branch 'geneartive_dynamic_metrics' into config_templates

db36e16

nicers tests + fix them

44aeecf

nicer todo

2bff963

add nice doscrings 📃

3c9eb21

add even more docstring

4216ae2

nit

d8f56b8

fix test

f26e88c

add multilingual to dev group

111d615

merge nli, add languagees to literals

7ca4239

translation literals

22eeddb

add nli

7faaa8a

clefourrier reviewed Sep 27, 2024

View reviewed changes

src/lighteval/tasks/templates/hellaswag.py Outdated Show resolved Hide resolved

clefourrier reviewed Sep 27, 2024

View reviewed changes

src/lighteval/tasks/multilingual/tasks.py Show resolved Hide resolved

hynky1999 and others added 15 commits September 30, 2024 18:21

Update src/lighteval/tasks/multilingual/tasks.py

ca865bd

Co-authored-by: Clémentine Fourrier <[email protected]>

Update src/lighteval/tasks/multilingual/tasks.py

1cc1187

Co-authored-by: Clémentine Fourrier <[email protected]>

Update src/lighteval/tasks/multilingual/tasks.py

d64251f

Co-authored-by: Clémentine Fourrier <[email protected]>

Update src/lighteval/tasks/multilingual/tasks.py

9806fab

Co-authored-by: Clémentine Fourrier <[email protected]>

Update src/lighteval/tasks/multilingual/tasks.py

35d7e6d

Co-authored-by: Clémentine Fourrier <[email protected]>

Update src/lighteval/tasks/multilingual/tasks.py

e560738

Co-authored-by: Clémentine Fourrier <[email protected]>

Update src/lighteval/tasks/multilingual/tasks.py

99524c5

Co-authored-by: Clémentine Fourrier <[email protected]>

add two new tasks + docs

150c76f

Merge branch 'multilnag_nli_tasks' of github.com:huggingface/lighteva…

4e6100d

…l into multilnag_nli_tasks

Merge remote-tracking branch 'origin/main' into multilnag_nli_tasks

7b561fe

Merge branch 'multilnag_nli_tasks' into multilang_copa_task

f9c7134

add nice docs

b219e07

Merge branch 'multilang_copa_task' into hellaswag_tasks

8899a1c

update hellaswag with docs

fcad0e8

move hellaswag to lighteval suite

43809ab

hynky1999 changed the base branch from multilang_copa_task to main September 30, 2024 18:16

hynky1999 and others added 3 commits September 30, 2024 20:19

Update src/lighteval/tasks/multilingual/tasks.py

a673d65

Co-authored-by: Clémentine Fourrier <[email protected]>

Merge remote-tracking branch 'origin/main' into hellaswag_tasks

3c453f7

enable returning none from templates + better typing

815e897

hynky1999 requested a review from clefourrier September 30, 2024 22:11

hynky1999 changed the title ~~Multilingual Hellaswg tasks~~ Multilingual Hellaswag tasks Oct 1, 2024

hynky1999 added 2 commits October 1, 2024 11:56

change unoficial hellaswag names to have community_prefix + unify hel…

eec2444

…laswag preprocesisng

let strip be optional in hellaswag

6284fe5

clefourrier approved these changes Oct 1, 2024

View reviewed changes

Merge branch 'main' into hellaswag_tasks

9bf53e9

hynky1999 merged commit 994fe73 into main Oct 1, 2024
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Multilingual Hellaswag tasks #332

Multilingual Hellaswag tasks #332

Uh oh!

hynky1999 commented Sep 25, 2024

Uh oh!

Uh oh!

Uh oh!

clefourrier Oct 1, 2024

Uh oh!

Uh oh!

Uh oh!

Multilingual Hellaswag tasks #332

Multilingual Hellaswag tasks #332

Uh oh!

Conversation

hynky1999 commented Sep 25, 2024

Goal

Comments

Uh oh!

Uh oh!

Uh oh!

clefourrier Oct 1, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!