adds aime24, 25 and math500 #586
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| stop_tokens = dataset[0].stop_sequence | ||
|
|
||
| max_new_tokens = dataset[0].generation_size # could be none | ||
| max_new_tokens = ( |
There was a problem hiding this comment.
By default, we would want the dataset to set the max length (or to cap it at max_tokens if asked by the user)
There was a problem hiding this comment.
It is set by the task by default is is overiden if the user sets it
There was a problem hiding this comment.
We don't want to possibly override to above the generation size imo
There was a problem hiding this comment.
I think it's good to have sensible defaults in the task definition (e.g. 4096 tokens is good for most models), but give users the ability to override at runtime, e.g. if they want to evaluate a long CoT model that requires 32k tokens
There was a problem hiding this comment.
Issue is going to be task repro
There was a problem hiding this comment.
I'm OK with leaving it if a warning message is displayed
There was a problem hiding this comment.
I'm not sure why task repro would be an issue here. all is clearly defined wither in task config or model config, some model (thinking) require something like 8k context length to generate what shoul be a much smaller response
There was a problem hiding this comment.
Well, you know how much people share their configs in eval ^^
If people assume they can reproduce the results of a model on a task, but actually the model config used overwrote the generation length of the task to allow the model to generate more, it will lead to a mismatch.
There was a problem hiding this comment.
I would personally not expect my model config to overwrite the maximum allowed length for a given task
* commit * Apply suggestions from code review * commit * add prompt to math 500 * add prompt to math 500
* commit * Apply suggestions from code review * commit * add prompt to math 500 * add prompt to math 500
No description provided.