feat: add custom judge type support for external repo integration#1274

Merged

gwarmstrong merged 24 commits intomainfrom

peri044/external_judge

Feb 25, 2026

Commits on Feb 24, 2026

Remove incorrect presence-penalty setting (#1259)
Kipok
committed
Add reference to internal benchmarks repo (#1261)
Kipok
committed
Rename custom -> external benchmarks (#1262)
Kipok
committed
Add aime26 (#1256)

bzantium
authored and
Kipok
committed
fix deps (#1258)

yqwangustc
authored and
Kipok
committed
feat: add custom judge type support for external repo integration

peri044
authored and
Kipok
committed
chore: Address review comments

peri044
authored and
Kipok
committed
Remove deprecated dataset group (#1263)
Kipok
committed
chore: Fix argument

peri044
authored and
Kipok
committed
chore: address review comments

peri044
authored and
Kipok
committed
chore: address review comments

peri044
authored and
Kipok
committed
chore: rename judge_step to judge_path

peri044
authored and
Kipok
committed
Fix incorrect prompt tokens count due to HF api update (#1264)
Kipok
committed
Fix no_answer metric overcounting in _compute_pass_at_k (#1245)

committed
Add CritPt benchmark (#1200)

jiacheng-xu
and
Kipok
committed
Add DSBench-DA evaluation (#1254)

sgunasekar
authored and
Kipok
committed
Numb3rs ds addition (#1174)

Jorjeous
authored and
Kipok
committed
style: fix ruff format for judge_path line

peri044
authored and
Kipok
committed
Small renaming
Kipok
committed
Update tests with new api
Kipok
committed
Fix judge assignment
Kipok
committed
Merge branch 'main' into peri044/external_judge
Kipok
committed
Year change
Kipok
committed
Merge branch 'main' into peri044/external_judge
Kipok
authored