v0.13.0

What's Changed

Breaking !!

• One file one task definition by @NathanHB in #1059

New Features 🎉

• add a task dump in registry for better documentation of tasks by @NathanHB in #1052
• Add style bot and other quality of life by @NathanHB in #1076

Enhancement ⚙️

• run all hf providers with :all by @NathanHB in #1039
• remove suites and make fewshot optional by @NathanHB in #1038
• neater bundle and logdir for inspect-ai by @NathanHB in #1043
• Fix set using wrong syntax by @f14-bertolotti in #1057
• [FT] Add py.typed to lighteval by @akshathmangudi in #1071

New Tasks

• Adding AA Omniscience task by @NathanHB in #1066

Task and Metrics changes 🛠️

• add to inspect by @NathanHB in #1065

Bug Fixes 🐛

• remove suites from serbian_eval.py by @S-Y-A-N in #1044
• not forcing use_logits at True by @f14-bertolotti in #1050
• wrong attribute self.k -> self.n by @f14-bertolotti in #1049
• Remove suites in task configs example and fix task with hf_filters by @NathanHB in #1051
• Fix: correct argument order in MajAtN.compute by @WeiKangda in #1058
• Update LiteLLM configuration for hosted_vllm provider by @abhiram1809 in #1060
• Use correct hf subset for ifbench multiturn by @sam-paech-liquid in #1061
• Fix autobatch size issue by @clefourrier in #1073
• batched metric was not aggregated properly by @f14-bertolotti in #1067

Other Changes

• Fix task config metric typing to accept Metric enums by @emmanuel-ferdman in #1018

New Contributors

• @S-Y-A-N made their first contribution in #1044
• @f14-bertolotti made their first contribution in #1050
• @WeiKangda made their first contribution in #1058
• @abhiram1809 made their first contribution in #1060
• @sam-paech-liquid made their first contribution in #1061
• @emmanuel-ferdman made their first contribution in #1018
• @akshathmangudi made their first contribution in #1071

Full Changelog: v0.12.0...v0.13.0

What's Changed

New Features 🎉

• add a task dump in registry for better documentation of tasks by @NathanHB in #1052
• Add style bot and other quality of life by @NathanHB in #1076

Enhancement ⚙️

• Fix set using wrong syntax by @f14-bertolotti in #1057
• [FT] Add py.typed to lighteval by @akshathmangudi in #1071

New Tasks

• Adding AA Omniscience task by @NathanHB in #1066

Task and Metrics changes 🛠️

• add to inspect by @NathanHB in #1065

Bug Fixes 🐛

• Fix: correct argument order in MajAtN.compute by @WeiKangda in #1058
• Update LiteLLM configuration for hosted_vllm provider by @abhiram1809 in #1060
• Use correct hf subset for ifbench multiturn by @sam-paech-liquid in #1061
• Fix autobatch size issue by @clefourrier in #1073
• batched metric was not aggregated properly by @f14-bertolotti in #1067
• Fix task config metric typing to accept Metric enums by @emmanuel-ferdman in #1018

New Contributors

• @WeiKangda made their first contribution in #1058
• @abhiram1809 made their first contribution in #1060
• @sam-paech-liquid made their first contribution in #1061
• @emmanuel-ferdman made their first contribution in #1018
• @akshathmangudi made their first contribution in #1071

Full Changelog: v0.12.2...v0.13.0