Adding support for Arabic benchmarks : AlGhafa benchmarking suite #95
Conversation
|
Do you want us to wait for Alghafa 2 to merge this? |
Yes please @clefourrier , i will take some time before Saturday to add the new version of the benchmark |
|
No hurries, take your time! |
|
Hello @clefourrier , I believe this PR is ready to be merged |
Add Support for the AlGhafa benchmarking suite
Adding support to the AlGhafa benchmarking suite
remove translated from AlGhafa
This file now contains all the arabic tasks including tasks not present in OALL_tasks.txt
Add support for ALGHAFA TRANSLATED tasks
Add support to AlGhafa Translated benchmark suite (11 subsets)
minor fixes flagged by the pre-commit hook
forgot to remove `community|Alghafa:multiple_choice_copa_translated_task|5|1` & `community|Alghafa:multiple_choice_openbookqa_translated_task|5|1` from ALGHAFA NATIVE
forgot to remove `community|Alghafa:multiple_choice_copa_translated_task|5|1` & `community|Alghafa:multiple_choice_openbookqa_translated_task|5|1` from ALGHAFA NATIVE
no need
There was a problem hiding this comment.
LGTM but you need to homogeneize your naming:
- Prompt names such as
boolq_functionwill be unclear long term. For such functions, you could either useboolq_prompt_arabicor justboolq_arabic. (You need to specify the language since there is already aboolqprompt function by default.) - You also need to homogeneize
Alghafa, which exists with several different casings, and fit it to Python style casing. For the prompt fonction, I'd keep it asalghafa_promptoralghafa, for the class,CustomAlGhafaTask, and here for the name I'd keep it lower case
[CustomAlGhafaTask(name=f"alghafa:{subset}", hf_subset=subset) for subset in ALGHAFA_SUBSETS]
homogeneize naming according to the following comments :
####
Prompt names such as boolq_function will be unclear long term. For such functions, you could either use boolq_prompt_arabic or just boolq_arabic. (You need to specify the language since there is already a boolq prompt function by default.)
You also need to homogeneize Alghafa, which exists with several different casings, and fit it to Python style casing. For the prompt fonction, I'd keep it as alghafa_prompt or alghafa, for the class, CustomAlGhafaTask, and here for the name I'd keep it lower case
[CustomAlGhafaTask(name=f"alghafa:{subset}", hf_subset=subset) for subset in ALGHAFA_SUBSETS]
####
homogeneize AlGhafa naming : `Alghafa` to `alghafa`
homogeneize AlGhafa naming : `Alghafa` to `alghafa`
Co-authored-by: Clémentine Fourrier <[email protected]>
Co-authored-by: Clémentine Fourrier <[email protected]>
Co-authored-by: Clémentine Fourrier <[email protected]>
Co-authored-by: Clémentine Fourrier <[email protected]>
use the standard camel casing for classes: (remove) class CustomALGHAFATask(LightevalTaskConfig): (add) class CustomAlGhafaTask(LightevalTaskConfig): Co-authored-by: Clémentine Fourrier <[email protected]>
Fixes based on Clementine's comments
|
@clefourrier I hope this answers to your comments, plz feel free to ping me if i missed anything (i have a tendency to forget 😅) |
|
Looks better thank you! |
Yes @clefourrier , I tested gpt2 using |
|
Sounds good, feel free to ping me whenever :) |
Fix ValueError: Prompt query
|
AlGhafa eval dataset is no longer available on Huggingface, any alternatives ? |
Hi there, Can you plz provide more context ? I have checked the eval code and it seems it works fine |
Hi, yesterday the datasets disappeared from the OALL Huggingface account, now i can see them, thanks |
OOH I see, i had to make the datasets private for about 20 min yesterday cuz i was testing something, what a coincidence you checked it at the same time 😅 |
- Add Support for the AlGhafa benchmarking suite --------- Co-authored-by: Clémentine Fourrier <[email protected]>
- Add Support for the AlGhafa benchmarking suite --------- Co-authored-by: Clémentine Fourrier <[email protected]>
3 participants
AlGhafa benchmarking suite, consist of 11 dataset presented in this paper and hosted in this repo in the Hub