-
Notifications
You must be signed in to change notification settings - Fork 41
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improved handling of source weights in NN ensemble #458
Conversation
…malized weights and compensate for the smaller weights within NN ensemble
65e68a8
to
90f3f76
Compare
Kudos, SonarCloud Quality Gate passed! 0 Bugs No Coverage information |
Codecov Report
@@ Coverage Diff @@
## master #458 +/- ##
=======================================
Coverage 99.41% 99.41%
=======================================
Files 65 65
Lines 4627 4630 +3
=======================================
+ Hits 4600 4603 +3
Misses 27 27
Continue to review full report at Codecov.
|
There's also the question what happens if you use an old model, trained with the current/old NN ensemble code, while running the new code. I tested that with the example model from #457 that originally achieved an average F1@5 score of 0.4418; when evaluated with the new code, the score was 0.4330, which is a bit worse, but not catastrophically so - and in fact for some of the individual data sets the scores had improved. I think it's enough to state in the release notes that retraining NN ensemble models is recommended, but not absolutely necessary. |
I verified that this PR doesn't change the results of a simple ensemble. (The Still need to double-check that PAV isn't affected |
I verified this once more with the Annif tutorial data sets. The results improved significantly for the STW data set, but declined slightly for the YSO one (though depends on metric). Double-checked that PAV is unaffected. Merging this now. |
This PR contains fixes aiming at improving the way the NN ensemble handles sources that have been assigned non-default weights.
Fixes #457