feat: add additional goosebench evals - #1571

Merged

ahau-square merged 35 commits into

mainfrom

Mar 10, 2025

Commits on Mar 10, 2025

ollama model + structured output shim

alicehau
authored and
Alice Hau
committed
41a9daa
tool shim generic

alicehau
authored and
Alice Hau
committed
92d5443
refactoring toolshim

alicehau
authored and
Alice Hau
committed
863a133
refactor

alicehau
authored and
Alice Hau
committed
20e3810
ollama toolshim ready

alicehau
authored and
Alice Hau
committed
f93d5c5
clean up

alicehau
authored and
Alice Hau
committed
78a3fa8
lint

alicehau
authored and
Alice Hau
committed
da64947
format

alicehau
authored and
Alice Hau
committed
24d8f69
add toolshim to openai provider

alicehau
authored and
Alice Hau
committed
0387207
update toolshim prompt:
Alice Hau
committed
10966a1
prompt update

alicehau
authored and
Alice Hau
committed
b0984c9
refactor
Alice Hau
committed
e81af6d
undo change to reference agent
Alice Hau
committed
fced390
undo ref agent changes
Alice Hau
committed
80ebac7
refactor config
Alice Hau
committed
7025113
refactor
Alice Hau
committed
a17a4da
check env vars
Alice Hau
committed
152bbe2
fix tests
Alice Hau
committed
b01091b
clean up
Alice Hau
committed
aaae84b
add small model evals
Alice Hau
committed
8b21c75
add initial evals
Alice Hau
committed
d4743f8
updated suite assignments
Alice Hau
committed
6899fe1
rebase
Alice Hau
committed
2ca7030
updatae external extensions str
Alice Hau
committed
ed701ae
add time and tools metrics and write to file utils

alicehau
authored and
Alice Hau
committed
1cbb48f
add copying of session file into workdir

alicehau
authored and
Alice Hau
committed
cff52a8
add metric to check tool use

alicehau
authored and
Alice Hau
committed
1fee63a
rename suite

alicehau
authored and
Alice Hau
committed
b4c236b
update name, account for non bool metrics

alicehau
authored and
Alice Hau
committed
595beca
format

alicehau
authored and
Alice Hau
committed
85d9bc3
update benchmark script to allow toolshim

alicehau
authored and
Alice Hau
committed
90849cd
remove changes from toolshim branch
Alice Hau
committed
da3b4a4
add back token usage mods
Alice Hau
committed
53de867
instruct flappy bird to use pyenv
Alice Hau
committed
a5b42b9
remove toolshiim from this PR
Alice Hau
committed
59666fc