-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implementation of ATKD #9
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Everything looks good to me.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not completely aware of this method but approving
Implement #1 as of now and let's see how the experiments go. In the meanwhile, you can look into the decaying. Are they using any schedulers? |
@navidpanchi @SharathRaparthy
|
Adding implementation of Attention Transfer KD (ATKD) which is an ICLR '17 paper. Some inspiration for the source code is from their official implementation.
Please go through the code (in case there are any obvious mistakes).
I ran one experiment with ResNet10 full data and the result is 92% validation accuracy (comparable to 92.2% of simultaneous KD for same settings).