Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compute the standard deviation of COMET scores for training student models #885

Closed
gregtatum opened this issue Oct 18, 2024 · 1 comment
Closed
Labels
experiment A training experiment with hypothesis and results

Comments

@gregtatum
Copy link
Member

gregtatum commented Oct 18, 2024

In #771 I had a bug in my code, where the training data wasn't actually truncated as expected, and so the models were trained with identical data. The lucky benefit of this was that we can compute a sample standard deviation of scores from that.

The result was: +/- 00.12 difference.

Type Value
COMET 89.48
COMET 89.24
COMET 89.41
COMET 89.50
STDDEV 00.12
@gregtatum gregtatum added the experiment A training experiment with hypothesis and results label Oct 18, 2024
@gregtatum
Copy link
Member Author

TODO - Re-run the COMET evaluations.

These should be the same values given the same random seeds.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
experiment A training experiment with hypothesis and results
Projects
None yet
Development

No branches or pull requests

1 participant