Skip to content

Latest commit

 

History

History
67 lines (44 loc) · 3.12 KB

README.md

File metadata and controls

67 lines (44 loc) · 3.12 KB
Raw

An unofficial implementation of R-net in PyTorch and AllenNLP.

Natural Language Computing Group, MSRA: R-NET: Machine Reading Comprehension with Self-matching Networks

Actually, I didn't reproduce the model of this paper exactly because some details are not very clear to me and the dynamic attention in self-matching requires too much memory. Instead, I implemented the variant of R-Net according to HKUST-KnowComp/R-Net (in Tensorflow).

The biggest difference between the original R-net and HKUST R-net is that:

  • The original R-net performs attention at each RNN step, which means that the hidden states are involved in the attention calculation. I call it dynamic attention.
  • In HKUST R-Net, attentions (in pair encoder and self-matching encoder) are calculated before performing RNN. I call it static attention.

Some details in HKUST-KnowComp/R-Net that improves performance:

  • Question and Passage share the same GRU sentence encoder instead of using two GRU encoders respectively.
  • The sentence encoder has three layers, but its output is the concat of the three layers instead of the output of the top layer.
  • The GRUs in the pair encoder and the self-matching encoder have only one layer instead of three layers.
  • Variational dropouts are applied to (1) the inputs of RNNs (2) inputs of attentions

Furthermore, this repo added ELMo word embeddings, which further improved the model's performance.

Dependency

  • Python == 3.6
  • AllenNLP == 0.7.2
  • PyTorch == 1.0

Usage

git clone https://github.com/matthew-z/R-net.git
cd R-net
python main.py train configs/squad/r-net/hkust.jsonnet  // HKUST R-Net

Note that the batch size may be a bit too large for 11GB GPUs. Please try 64 in case of OOM Error by adding the following arg: -o '{"iterator.batch_size": 64}'

Configuration

The models and hyperparameters are declared in configs/

  • the HKUST R-Net: configs/r-net/hkust.jsonnet (79.4 F1)
    • +ELMo: configs/r-net/hkust+elmo.jsonnet (82.2 F1)
  • the original R-Net: configs/r-net/original.jsonnet (currently not workable)

Performance

This implementation of HKUST R-Net can obtain 79.4 F1 and 70.5 EM on the validation set.

  • ELMo: 82.2 F1 and 74.4 EM.

The visualization of R-Net + Elmo Training: Red: training score, Green: validation score

Note that validation score is higher than training because each validation has three acceptable answers, which makes validation easier than training.

Future Work

  • Add BERT: A preliminary implementation is in configs/r-net/hkust+bert.jsonnet
  • Add ensemble training
  • Add FP16 training

Acknowledgement

Thank HKUST-KnowComp/R-Net for sharing their Tensorflow implementation of R-net. This repo is based on their work.