Multilingual Reading Comprehension tasks

[hynky1999]

Goal

Add RC tasks supporting about 130 unique languages/scripts.

• xquad https://arxiv.org/abs/1910.11856
• thaiqa
• sber_squad https://arxiv.org/abs/1912.09723
• arcd https://arxiv.org/pdf/1906.05394
• kenswquad https://arxiv.org/abs/2205.02364
• chinese_squad https://github.com/pluto-junzeng/ChineseSquad
• cmrc2018 https://arxiv.org/abs/1810.07366
• indicqa https://arxiv.org/abs/2407.13522
• fquad_v2 https://arxiv.org/abs/2002.06071
• tquad_v2
• tydiqa https://arxiv.org/abs/2003.05002
• beleble https://arxiv.org/abs/2308.16884

How to test:

lighteval accelerate --output_dir=./tmp --custom_tasks="lighteval.tasks.multilingual.tasks" --tasks="custom|{task}|0|0" --model_args=pretrained=gpt2 --override_batch_size=1 --max_samples=100 --save_details

where task in xquad_eng|0|0 thaiqa_tha|0|0 sber_squad_rus|0|0 arcd_ara|0|0 kenswquad_swa|0|0 chinese_squad_zho|0|0 cmrc2018_zho|0|0 indicqa_hin|0|0 fquadv2_fra|0|0 tquadv2_tur|0|0 tydiqa_eng|0|0 belebele_acm_Arab|0|0

Comments

• We talked about removing the suite all together. Since right now we use suite I decided to use custom suite. I can switch to multilingual or whatever