Fix #905

[leezu]

Description

enforce_max_size function incorrectly overwrote an internal TokenEmbedding datastructure, breaking the use of unknown_token's embedding for '<pad>'. When this script was written, modifying TokenEmbedding's internals was required. Based on #750 we can now use proper API to make the required changes.

In general however, vocab = nlp.Vocab(nlp.data.count_tokens(tokens)) can be replaced with vocab = nlp.Vocab(nlp.data.count_tokens(tokens), unknown_token=token_embedding_.unknown_token, padding_token=None, bos_token=None, eos_token=None). I'm opening a PR for these two changes.

Unfortunately this error slipped through the tests. I'm also extending the testcase.

Fixes #905

Checklist

Essentials

• PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage
• Code is well-documented

Changes

• Fix word_embeddings/evaluate_pretrained.py script when --analogy-max-vocab-size is used

[codecov]

Codecov Report

❗ No coverage uploaded for pull request head (fixevaluatepretrained@6224bcd). Click here to learn what that means.
The diff coverage is n/a.

[codecov]

Codecov Report

Merging #906 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #906      +/-   ##
==========================================
+ Coverage   90.48%   90.48%   +<.01%     
==========================================
  Files          66       66              
  Lines        6400     6401       +1     
==========================================
+ Hits         5791     5792       +1     
  Misses        609      609

Impacted Files	Coverage Δ
src/gluonnlp/vocab/vocab.py	97.32% <100%> (+0.01%)	⬆️

[mli]

Job PR-906/1 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-906/1/index.html

[mli]

Job PR-906/2 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-906/2/index.html

[mli]

Job PR-906/13 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-906/13/index.html