feat: supports evaluation of multiple-choice benchmarks#559

Merged

parthchadha merged 46 commits intoNVIDIA-NeMo:mainfrom

xxman-google:xx/new_eval

Jul 3, 2025

Commits on Jun 30, 2025

Adds multiple choice eval datasets.
xxman-google
committed
Add a verify worker for multiple-choice problems.
xxman-google
committed
add prompts for MMLU and GPQA.
xxman-google
committed
modifies eval script to support multiple-choice questions.
xxman-google
committed
add eval config files.
xxman-google
committed
add unit tests.
xxman-google
committed
add AIME 2024 dataset.
xxman-google
committed
add GPQA main version.
xxman-google
committed
fix: remove reference_model_buffers in fsdp2 (#558)

yuki-97
authored and
xxman-google
committed
fix: Add assertion if async is disabled when using pp with vllm (#565)

parthchadha
authored and
xxman-google
committed
fix: remove visualization code (#566)

authored and
xxman-google
committed
Allow uneven shards for multi-GPU inference in vllm backend (#494)

authored and
xxman-google
committed
add GPQA main version.
xxman-google
committed
updates doc.
xxman-google
committed
feat: vllm Model diagnostic test checking long generation quality (#516)

authored and
xxman-google
committed
feat: Log code in wandb (#175)

authored and
xxman-google
committed
fix: add dynamic_batching key to SFT OpenMathInstruct config (#570)

ashors1
authored and
xxman-google
committed
feat: support async in non-colocated (#523)

yuki-97
authored and
xxman-google
committed
fix: correct mcore dtype + assertion on activation_func (#572)

terrykong
authored and
xxman-google
committed
fix: move core ray port from 6379 -> 54258 to reduce port collision freq (#574)

terrykong
authored and
xxman-google
committed
fix: fix overlap param gather (#561)

ashors1
authored and
xxman-google
committed
docs: fix some typos on nsys/model-quirk pages (#560)

terrykong
authored and
xxman-google
committed
feat: Add megatron to hf converter (#555)

ashors1
authored and
xxman-google
committed
docs: Add a note on supported backends (#553)

ashors1
authored and
xxman-google
committed
feat: Support pass@k (#536)

peri044
authored and
xxman-google
committed
fix: Megatron config fixes (#576)

SahilJain314
authored and
xxman-google
committed
update docs for the new eval.
xxman-google
committed
docs: move training backends section (#580)

ashors1
authored and
xxman-google
committed
docs: Add a note on supported backends (#553)

ashors1
authored and
xxman-google
committed
docs: move training backends section (#580)

ashors1
authored and
xxman-google
committed
Update more docs for the new eval.
xxman-google
committed
Merge branch 'main' into xx/new_eval
yuki-97
committed

Commits on Jul 2, 2025

fix lint errors.
xxman-google
committed
add missing copyright statements.
xxman-google
committed
add missing copyright statements.
xxman-google
committed
docs: Add missing arguments to DeepScaler evaluation (#502)

butsugiri
authored and
xxman-google
committed
fix: prevent divisible error by dropping last batch in loader (#583)

wedu-nvidia
authored and
xxman-google
committed
feat: improve worker group args/kwargs (#539)

yuki-97
authored and
xxman-google
committed
fix: update gemma3 prefix (#585)

ashors1
authored and
xxman-google
committed
fix: Added copyright to functest (#584)

SahilJain314
authored and
xxman-google
committed
chore: Update github url after org transfer (#512)

authored and
xxman-google
committed
feat: add OpenAI format dataset for SFT (#485)

AtsunoriFujita
authored and
xxman-google
committed
fix: load HF model only on rank 0 (#544)

parthchadha
authored and
xxman-google
committed
feat: support async in non-colocated (#523)

yuki-97
authored and
xxman-google
committed
feat: Add megatron to hf converter (#555)

ashors1
authored and
xxman-google
committed
Merge branch 'main' into xx/new_eval
xxman-google
committed