[WIP] Bump lighteval with proper pass@1 #584
Merged
+35
−239
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lewtun
commented
Apr 8, 2025
| ```shell | ||
| MODEL=deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B | ||
| MODEL_ARGS="pretrained=$MODEL,dtype=bfloat16,max_model_length=32768,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}" | ||
| MODEL_ARGS="pretrained=$MODEL,dtype=bfloat16,max_model_length=32768,max_num_batched_tokens=32768,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}" |
There was a problem hiding this comment.
This is now needed to avoid lighteval errors
lewtun
commented
Apr 8, 2025
| echo "Running lighteval script ..." | ||
| echo "Eval results will be saved to $OUTPUT_DIR" | ||
| # Check if "custom" is a substring of TASKS | ||
| if [[ $TASKS == *"custom"* ]]; then |
There was a problem hiding this comment.
We no longer have custom evals, so this can be removed
edbeeching
approved these changes
Apr 8, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR bumps
lightevalto give us access to a newpass@1metric that estimates the accuracy fromNsamples per prompt.From 10 different seeds, I found that
N=16gave the best tradeoff in terms of variance / compute cost: huggingface/lighteval#661I am running the evals to sanity check nothing is changed dramatically, but the code should be good for review.
TODO