GluonNLP 0.8 BERT output is different from GluonNLP 0.9

[davisliang]

Description

Using the same BERT model parameters, GluonNLP0.8 and GluonNLP0.9 output drastically different encoder representations. Tested on both MxNet1.5, MxNet1.6.

Error Message

GluonNLP 0.8 output:
[[[-0.14241228 0.13353696 -0.12907042 ... -0.3596797 -0.05622234
0.36050126]
[-0.3506479 0.10419717 0.62444484 ... -0.17610289 0.48340234
0.06443496]
[-0.24513118 -0.15731761 0.69451797 ... -0.5654461 -0.08939961
-0.18564416]
[-0.824786 -0.9119225 -0.65607095 ... 0.50742507 -0.19388743
-0.1658766 ]
[ 0.87665254 0.03524816 -0.12331399 ... 0.2720159 -0.63690007
-0.15850069]]]
<NDArray 1x5x768 @cpu(0)>

GluonNLP 0.9 output:
[[[-0.8164635 -0.18977094 -0.44616854 ... -0.9124998 0.02381709
0.5544555 ]
[-1.0949776 -0.41612968 0.5249134 ... -0.706112 0.14636786
-0.38614586]
[-0.8410385 -0.45054507 0.3945069 ... -0.9239115 -0.3351414
0.05241004]
[-0.8435936 -1.2706859 -0.5667961 ... -0.40249282 -0.1447221
-0.12529008]
[ 0.75693136 -1.1236286 -0.2741627 ... 0.08363507 -0.71608377
0.29923674]]]
<NDArray 1x5x768 @cpu(0)>

To Reproduce

First, install the packages:
!pip install mxnet-cu102
!pip install gluonnlp==0.9.1
!pip install gluonnlp==0.8

import gluonnlp as nlp; import mxnet as mx;
model, vocab = nlp.model.get_model('bert_12_768_12', dataset_name='book_corpus_wiki_en_uncased', use_classifier=False, use_decoder=False);
tokenizer = nlp.data.BERTTokenizer(vocab, lower=True);
transform = nlp.data.BERTSentenceTransform(tokenizer, max_seq_length=512, pair=False, pad=False);
sample = transform(['Hello world!']);
words, valid_len, segments = mx.nd.array([sample[0]]), mx.nd.array([sample[1]]), mx.nd.array([sample[2]]);
seq_encoding, cls_encoding = model(words, segments, valid_len);

print(seq_encoding)

Steps to reproduce

(Paste the commands you ran that produced the error.)

1. Install the package (either GluonNLP 0.8 or 0.9, as detailed in To Reproduce)
2. Run the example code snippet, taken from the GluonNLP BERT tutorial

What have you tried to solve it?

1. I hypothesized there might be something different in the handing of parameters but it doesn't seem like it.
2. I hypothesized there might be something different with tokenization and transformations but it doesn't seem like that's the issue either.

Environment

Any EC2

We recommend using our script for collecting the diagnositc information. Run the following command and paste the outputs below:

curl --retry 10 -s https://raw.githubusercontent.com/dmlc/gluon-nlp/master/tools/diagnose.py | python

# paste outputs here

[leezu]

I did a git bisect on the 0.9.x branch with start commit 76fdafb and got the following result: 75c29a3

75c29a3518ee42b98cc651b6922cbae85d2e961e is the first bad commit
commit 75c29a3518ee42b98cc651b6922cbae85d2e961e
Author: Haibin Lin <[email protected]>
Date:   Fri Feb 7 13:51:21 2020 -0800

    [API] use softmax with length, and interleaved matmul for BERT (#1136)

cc @eric-haibin-lin

[sxjscience]

@davisliang Is it solved by #1296?

[davisliang]

@sxjscience yes, looks like it solves the issue. Thanks for the quick turnaround!