-
Notifications
You must be signed in to change notification settings - Fork 124
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Freeze embeddings #136
base: master
Are you sure you want to change the base?
Freeze embeddings #136
Conversation
@adrian, do you want to do the initial review? |
As for loading, I think the model should be stored as is (including the frozen parameters). Not sure what kind of "loading functionality" is required for this, though. Only the embedder is affected, right? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good. Just some small points.
We should check if resuming still works correctly with frozen embeddings. Also we need to extend the package functionality (maybe not in this PR) to handle models with frozen embeddings.
48975fc
to
57a651e
Compare
57a651e
to
2317dbd
Compare
As it is now, models are saved with the frozen parameters and resuming the training works. |
Functionality to freeze embeddings during training as discussed with Adrian. A file containing ids for the entities or relations can be loaded, these embeddings are held constant during training. What is an open question is how the save/resume functionality for frozen models should be. Right now, the frozen and not-frozen parameters are saved. This requires loading functionality for a frozen model. An alternative would be to save the model as a standard model e.g. by unfreezing it first. This would, however, loose the optimizer state and a training process where parameters are held constant could not be continued.