Add ComputeEval Dataset Support#1124

Closed

blahblahasdf wants to merge 34 commits intoNVIDIA-NeMo:mainfrom

blahblahasdf:add-compute-eval

Commits on Jan 8, 2026

Add ComputeEval skill.
blahblahasdf
committed
Split out evaluator from generation task.
blahblahasdf
committed
Adding clan PR with AudioBench and Librispeech PC. (#1103)

Jorjeous
authored and
blahblahasdf
committed
Schema overrides for tool-calling (#1118)

gwarmstrong
authored and
blahblahasdf
committed
FIX tool call error handling and search tool errors (#1120)

gwarmstrong
authored and
blahblahasdf
committed
Use run.Script for generate pipeline (#1052)

gwarmstrong
authored and
blahblahasdf
committed
Port ICPC changes to IOI (#1046)

authored and
blahblahasdf
committed
replace raise error with LOG.warning in AA LCR dataset prepare (#1119)

authored and
blahblahasdf
committed
FIX tavily search results return type (#1123)

gwarmstrong
authored and
blahblahasdf
committed
Address CodeRabbit review comments.
blahblahasdf
committed
Add exception handling during eval.
blahblahasdf
committed
Remove unncessary HuggingFace Hub call.
blahblahasdf
committed
Push `Problem` data transformation logic in to the prepare script and move `ComputeEvalGenerationTask` to be more on the rails of the `process_single_datapoint` approach.
blahblahasdf
committed
Revert "Use run.Script for generate pipeline (#1052)" (#1125)

gwarmstrong
authored and
blahblahasdf
committed
Fix: add serialized_output on bad request (#1127)

gwarmstrong
authored and
blahblahasdf
committed
update paper link (#1128)

wedu-nvidia
authored and
blahblahasdf
committed
update paper link, references to dataset, self-correction differences (#1129)

stephencge
authored and
blahblahasdf
committed
FIX ioi ignore (#1131)

gwarmstrong
authored and
blahblahasdf
committed
download AA-LCR_extracted-text.zip via hf_hub_download (#1126)

authored and
blahblahasdf
committed
Evaluation on Livecodebench-pro (#1115)

wasiahmad
authored and
blahblahasdf
committed
Evaluation support for SWE-rebench (#1102)

authored and
blahblahasdf
committed
Trust remote code in tokenizer (#1146)

authored and
blahblahasdf
committed
Resolve broken links in docs (#1150)

activatedgeek
authored and
blahblahasdf
committed
Introduced vLLM_multimodal model to save multimodal outputs (#1136)

authored and
blahblahasdf
committed
add swe-rebench to excluded datasets (#1154)

gwarmstrong
authored and
blahblahasdf
committed
Fix run.Script refactor (#1133)

gwarmstrong
authored and
blahblahasdf
committed
BIRD Benchmark (Text-to-SQL) (#1132)

redoctopus
authored and
blahblahasdf
committed
o3-mini-20250131 -> o3-mini-2025-01-31 (#1149)

authored and
blahblahasdf
committed
Update compute-eval version.
blahblahasdf
committed
Another update to compute eval version that removes the pinned urllib3.
blahblahasdf
committed
Set compute eval dep to explicit hash.
blahblahasdf
committed
Merge branch 'main' into add-compute-eval
blahblahasdf
authored
Address a bunch of the automated PR comments.
blahblahasdf
committed
Add compute eval documentation.
blahblahasdf
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ComputeEval Dataset Support#1124

Add ComputeEval Dataset Support#1124
blahblahasdf wants to merge 34 commits intoNVIDIA-NeMo:mainfrom
blahblahasdf:add-compute-eval

Commits on Jan 8, 2026

Add ComputeEval skill.

Split out evaluator from generation task.

Adding clan PR with AudioBench and Librispeech PC. (#1103)

Schema overrides for tool-calling (#1118)

FIX tool call error handling and search tool errors (#1120)

Use run.Script for generate pipeline (#1052)

Port ICPC changes to IOI (#1046)

replace raise error with LOG.warning in AA LCR dataset prepare (#1119)

FIX tavily search results return type (#1123)

Address CodeRabbit review comments.

Add exception handling during eval.

Remove unncessary HuggingFace Hub call.

Push `Problem` data transformation logic in to the prepare script and move `ComputeEvalGenerationTask` to be more on the rails of the `process_single_datapoint` approach.

Revert "Use run.Script for generate pipeline (#1052)" (#1125)

Fix: add serialized_output on bad request (#1127)

update paper link (#1128)

update paper link, references to dataset, self-correction differences (#1129)

FIX ioi ignore (#1131)

download AA-LCR_extracted-text.zip via hf_hub_download (#1126)

Evaluation on Livecodebench-pro (#1115)

Evaluation support for SWE-rebench (#1102)

Trust remote code in tokenizer (#1146)

Resolve broken links in docs (#1150)

Introduced vLLM_multimodal model to save multimodal outputs (#1136)

add swe-rebench to excluded datasets (#1154)

Fix run.Script refactor (#1133)

BIRD Benchmark (Text-to-SQL) (#1132)

o3-mini-20250131 -> o3-mini-2025-01-31 (#1149)

Update compute-eval version.

Another update to compute eval version that removes the pinned urllib3.

Set compute eval dep to explicit hash.

Merge branch 'main' into add-compute-eval

Address a bunch of the automated PR comments.

Add compute eval documentation.