Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

EMA update for codebook #11

Open
j-min opened this issue Mar 3, 2020 · 2 comments
Open

EMA update for codebook #11

j-min opened this issue Mar 3, 2020 · 2 comments

Comments

@j-min
Copy link

j-min commented Mar 3, 2020

Is NearestEmbedEMA for EMA codebook update working?
I tried training by replacing NearestEmbed with NearestEmbed, but the loss did not converge. NearestEmbed worked for my case.
@sylqiu
Copy link

sylqiu commented Jun 10, 2020

I am also curious about this. Can you try initializing cluster_size to be ones instead of zeros? Although the latter is done in the Sonnet's official implementation, I actually don't understand why it should be initialized to zeros, since in case no feature is an NN of a code, that code will seem to be multiplied by a big constant.

@nadavbh12
Copy link
Owner

Haven't tried running it with ones.
If it works for you, feel free to send a pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@nadavbh12 @j-min @sylqiu