Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Result summarization wrong? #57

Open
tsly123 opened this issue Dec 23, 2023 · 0 comments
Open

Result summarization wrong? #57

tsly123 opened this issue Dec 23, 2023 · 0 comments

Comments

@tsly123
Copy link

tsly123 commented Dec 23, 2023

Hi @Tsingularity @KMnP ,

Thank you for the great work.
In the get_df function which summarizes the results. Line 179

        metric_b = "val_top1"
        best_epoch = np.argmax(eval_results[metric_b])

Here, everything is correct. However, when there are many "val_top1" that have the same value, the np.argmax will return the first index for that value. A concrete example is with Caltech101, I got many eval_results["val_top1"] = 100, and the np.argmax(eval_results["val_top1"]) returns the first index where eval_results["val_top1"] = 100. This would make the eval_results["b-test-top1"] much lower, leading to incorrect summarization. Consider the minimal example below:

>>> import numpy as np
>>> val_arr = np.array([5, 10, 15, 20, 100, 100, 100, 100, 100])
>>> test_arr = np.arange(len(val_arr))
>>> best_epoch = np.argmax(val_arr)
>>> test_arr[best_epoch]
4

Am I correct? Or Do I misunderstand something here?

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant