[Numpy Refactor] BART #1282

zheyuye · 2020-07-29T14:19:44Z

Description

As #1241 requeted, BART model from fairseq is now available as well as its conversion script. Most of the test cases for Bart are covered by test_models_transformer.py

TODO

BART Inference model for Machine Translation
Denosing training scripts of BART
Fine-tuning result of BART on Machine Translation, Querstion Answering and Text Classification

cc @dmlc/gluon-nlp-team

src/gluonnlp/models/bart.py

src/gluonnlp/models/transformer.py

sxjscience · 2020-07-29T17:18:26Z

src/gluonnlp/models/bart.py

+    @classmethod
+    def from_cfg(cls, cfg,
+                 use_pooler=False,
+                 classifier_activation=False):


Is it possible to remove the classifier_activation flag?

We could remove this and consider applying an activation layer as default setting for the pooler.

src/gluonnlp/models/transformer.py

commit 510d991 Author: ZheyuYe <[email protected]> Date: Thu Jul 30 02:33:22 2020 +0800 test commit 1b5fa7b Author: ZheyuYe <[email protected]> Date: Thu Jul 30 01:48:01 2020 +0800 fix comment1 commit 6533601 Author: ZheyuYe <[email protected]> Date: Thu Jul 30 01:27:44 2020 +0800 fix comment commit a8853f9 Author: ZheyuYe <[email protected]> Date: Thu Jul 30 01:10:06 2020 +0800 Squashed commit of the following: commit 232e0b6 Author: ZheyuYe <[email protected]> Date: Thu Jul 30 01:05:17 2020 +0800 update commit 995e5d7 Author: ZheyuYe <[email protected]> Date: Thu Jul 30 01:01:56 2020 +0800 fix commit 9623240 Author: ZheyuYe <[email protected]> Date: Thu Jul 30 00:52:17 2020 +0800 fix commit d9c4140 Author: ZheyuYe <[email protected]> Date: Wed Jul 29 23:07:10 2020 +0800 fix transformer commit e49fbe1 Author: ZheyuYe <[email protected]> Date: Wed Jul 29 22:18:12 2020 +0800 update commit 1f75b26 Author: ZheyuYe <[email protected]> Date: Wed Jul 29 22:04:08 2020 +0800 test bart commit 5bab516 Author: ZheyuYe <[email protected]> Date: Wed Jul 29 21:34:47 2020 +0800 fix cfg commit 6c62a29 Merge: 3366cf3 033214e Author: ZheyuYe <[email protected]> Date: Wed Jul 29 21:33:10 2020 +0800 Merge remote-tracking branch 'upstream/numpy' into bart commit 033214e Author: Xingjian Shi <[email protected]> Date: Wed Jul 29 00:36:57 2020 -0700 [Numpy] Fix SQuAD + Fix GLUE downloading (dmlc#1280) * Update run_squad.py * Update run_squad.py * Update prepare_glue.py commit 3c87457 Author: Xingjian Shi <[email protected]> Date: Tue Jul 28 18:03:21 2020 -0700 Add layout + compute_layout support: TransformerNMT, BERT, ALBERT, ELECTRA, MobileBERT, RoBERTA, XLMR (dmlc#1258) * Add layout support * fix test * Update transformer.py * Update transformer.py * Update README.md * try to add set_layout * update test case * fix * update * update * update * Update bert.py * fix bug * update * Update test_models_bert.py * Update tokenizers.py * add compute layout * Update xlmr.py * Update test_models_bert.py * revise test cases * Update layers.py * move jieba to try import * fix * Update transformer.py * fix * Update bert.py * Update setup.py * Update test_models_bert.py * Update test_models_bert.py * fix * update * Revise * Update electra.py * Update electra.py * Update test_models_electra.py * fix * fix bug * Update test_models_albert.py * add more testcases * fix * Update albert.py * Update albert.py * fix bug * fix testcase * Update test_models_electra.py * Update bert.py * update * Update test_models_electra.py * Update mobilebert.py * Update mobilebert.py * update mobilebert * Update test_models_mobilebert.py * Update mobilebert.py * fix bug * Update roberta.py * fix roberta * update * update * fix import * fix bug * update * reduce test workloads * address comment * address comment commit 4d43f82 Author: Sheng Zha <[email protected]> Date: Mon Jul 27 20:21:00 2020 -0700 add subversion/wget to docker, add readme (dmlc#1279) commit d76897b Author: phile <[email protected]> Date: Tue Jul 28 10:10:13 2020 +0800 Add embedding related methods in numpy version (dmlc#1263) * A draft for embedding * fix embed_loader * add hyperbolic space and some updates * revise evaluation * fix * simple fixes * move l2norm to op.py * new features * fix * update * add tests, update * newline

scripts/conversion_toolkits/convert_fairseq_bart.py

commit 9e1ffde Author: ZheyuYe <[email protected]> Date: Thu Jul 30 11:42:01 2020 +0800 todo commit 9a7c343 Author: ZheyuYe <[email protected]> Date: Thu Jul 30 10:53:15 2020 +0800 revert gelu commit 0425346 Author: ZheyuYe <[email protected]> Date: Thu Jul 30 10:49:52 2020 +0800 re-upload bart commit 516ae84 Author: ZheyuYe <[email protected]> Date: Thu Jul 30 03:32:35 2020 +0800 use_qkv_bias for transformer commit 9d60cda Author: ZheyuYe <[email protected]> Date: Thu Jul 30 03:17:28 2020 +0800 classifier_activation commit 510d991 Author: ZheyuYe <[email protected]> Date: Thu Jul 30 02:33:22 2020 +0800 test commit 1b5fa7b Author: ZheyuYe <[email protected]> Date: Thu Jul 30 01:48:01 2020 +0800 fix comment1 commit 6533601 Author: ZheyuYe <[email protected]> Date: Thu Jul 30 01:27:44 2020 +0800 fix comment commit a8853f9 Author: ZheyuYe <[email protected]> Date: Thu Jul 30 01:10:06 2020 +0800 Squashed commit of the following: commit 232e0b6 Author: ZheyuYe <[email protected]> Date: Thu Jul 30 01:05:17 2020 +0800 update commit 995e5d7 Author: ZheyuYe <[email protected]> Date: Thu Jul 30 01:01:56 2020 +0800 fix commit 9623240 Author: ZheyuYe <[email protected]> Date: Thu Jul 30 00:52:17 2020 +0800 fix commit d9c4140 Author: ZheyuYe <[email protected]> Date: Wed Jul 29 23:07:10 2020 +0800 fix transformer commit e49fbe1 Author: ZheyuYe <[email protected]> Date: Wed Jul 29 22:18:12 2020 +0800 update commit 1f75b26 Author: ZheyuYe <[email protected]> Date: Wed Jul 29 22:04:08 2020 +0800 test bart commit 5bab516 Author: ZheyuYe <[email protected]> Date: Wed Jul 29 21:34:47 2020 +0800 fix cfg commit 6c62a29 Merge: 3366cf3 033214e Author: ZheyuYe <[email protected]> Date: Wed Jul 29 21:33:10 2020 +0800 Merge remote-tracking branch 'upstream/numpy' into bart commit 033214e Author: Xingjian Shi <[email protected]> Date: Wed Jul 29 00:36:57 2020 -0700 [Numpy] Fix SQuAD + Fix GLUE downloading (dmlc#1280) * Update run_squad.py * Update run_squad.py * Update prepare_glue.py commit 3c87457 Author: Xingjian Shi <[email protected]> Date: Tue Jul 28 18:03:21 2020 -0700 Add layout + compute_layout support: TransformerNMT, BERT, ALBERT, ELECTRA, MobileBERT, RoBERTA, XLMR (dmlc#1258) * Add layout support * fix test * Update transformer.py * Update transformer.py * Update README.md * try to add set_layout * update test case * fix * update * update * update * Update bert.py * fix bug * update * Update test_models_bert.py * Update tokenizers.py * add compute layout * Update xlmr.py * Update test_models_bert.py * revise test cases * Update layers.py * move jieba to try import * fix * Update transformer.py * fix * Update bert.py * Update setup.py * Update test_models_bert.py * Update test_models_bert.py * fix * update * Revise * Update electra.py * Update electra.py * Update test_models_electra.py * fix * fix bug * Update test_models_albert.py * add more testcases * fix * Update albert.py * Update albert.py * fix bug * fix testcase * Update test_models_electra.py * Update bert.py * update * Update test_models_electra.py * Update mobilebert.py * Update mobilebert.py * update mobilebert * Update test_models_mobilebert.py * Update mobilebert.py * fix bug * Update roberta.py * fix roberta * update * update * fix import * fix bug * update * reduce test workloads * address comment * address comment commit 4d43f82 Author: Sheng Zha <[email protected]> Date: Mon Jul 27 20:21:00 2020 -0700 add subversion/wget to docker, add readme (dmlc#1279) commit d76897b Author: phile <[email protected]> Date: Tue Jul 28 10:10:13 2020 +0800 Add embedding related methods in numpy version (dmlc#1263) * A draft for embedding * fix embed_loader * add hyperbolic space and some updates * revise evaluation * fix * simple fixes * move l2norm to op.py * new features * fix * update * add tests, update * newline

codecov · 2020-07-30T04:42:46Z

Codecov Report

Merging #1282 into numpy will increase coverage by 0.06%.
The diff coverage is 87.36%.

@@            Coverage Diff             @@
##            numpy    #1282      +/-   ##
==========================================
+ Coverage   84.14%   84.21%   +0.06%     
==========================================
  Files          41       42       +1     
  Lines        6157     6316     +159     
==========================================
+ Hits         5181     5319     +138     
- Misses        976      997      +21

Impacted Files	Coverage Δ
src/gluonnlp/models/bart.py	`85.16% <85.16%> (ø)`
src/gluonnlp/layers.py	`86.78% <100.00%> (ø)`
src/gluonnlp/models/transformer.py	`98.52% <100.00%> (+0.43%)`	⬆️

src/gluonnlp/layers.py

src/gluonnlp/models/bart.py

sxjscience

Minor comments about the embedding initializer. LGTM overall.

zheyuye added 30 commits July 27, 2020 15:28

init

5274e63

fix convert roberta

886b1ac

rename TransformerNMTModel as TransformerModel

f6d0d35

update bart

db7d8db

fix

01d9945

fix

dc31a44

update init

5807642

add layernorm_embedding for transformer

6bb255e

convert script

41be41e

encoder

5fa5108

fix

a06293d

fix vocab

8a112d5

fix roberta

6f8335a

fix

dc1e79b

fix electra

52121c3

add conversion bash for roberta and xlmr

be3b2d9

ELECTRA SETUP

4afc84f

convert bart decoder

179ee73

fix

4c54128

update

cf1e20d

testing output

a44b03a

remove arange_like for embeddings

041bf8d

fix

b47ca86

update

917dcce

use_pooler for bart

0bda11d

fix

e3c61f3

upload params for bart

a5a91e0

add test_models_bart

3366cf3

Merge remote-tracking branch 'upstream/numpy' into bart

6c62a29

fix cfg

5bab516

zheyuye added 2 commits July 29, 2020 22:05

test bart

1f75b26

update

e49fbe1

zheyuye added enhancement New feature or request numpyrefactor labels Jul 29, 2020

fix transformer

d9c4140