Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

higher_is_better for InfoLM should change depending on information_measure #2659

Closed
grahamannett opened this issue Jul 29, 2024 · 2 comments · Fixed by #2674
Closed

higher_is_better for InfoLM should change depending on information_measure #2659

grahamannett opened this issue Jul 29, 2024 · 2 comments · Fixed by #2674
Labels
bug / fix Something isn't working good first issue Good for newcomers help wanted Extra attention is needed v1.4.x

Comments

@grahamannett
Copy link
Contributor

Not sure if there are other metrics this will apply to but for InfoLM, if the information_measure is l2, then the distance will always positive, but infolm.higher_is_better shows True. I am pretty sure that is no longer the case compared to the base measure which is KL in which case trending towards 0 from vals < 0 is improvement

@grahamannett grahamannett added bug / fix Something isn't working help wanted Extra attention is needed labels Jul 29, 2024
@Borda Borda changed the title higher_is_better for InfoLM should change depending on information_measure higher_is_better for InfoLM should change depending on information_measure Aug 2, 2024
@Lightning-AI Lightning-AI deleted a comment from github-actions bot Aug 2, 2024
@Borda Borda added the v1.4.x label Aug 2, 2024
@Borda
Copy link
Member

Borda commented Aug 2, 2024

That is a good point and should be fixed. Are you interested in sending a PR?
cc: @lucadiliello

@Borda Borda added the good first issue Good for newcomers label Aug 2, 2024
@grahamannett
Copy link
Contributor Author

Sure, will try and send a PR in the next few days.

Was thinking/wondering if its an issue for other metrics but not sure if I will have time to go through and look into others/understand if it applies in other cases. This one specifically made some metrics for runs very confusing!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug / fix Something isn't working good first issue Good for newcomers help wanted Extra attention is needed v1.4.x
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants