Evaluating the R-NET Model
- This is the implementation of paper R-NET: MACHINE READING COMPREHENSION WITH SELF-MATCHING NETWORKS.
- Code is adapted from https://github.com/minsangkim142/R-net
- Please contact the original authors for questions and suggestions.
R-NET is an end-to-end neural networks model for reading comprehension style question answering, which aims to answer questions from a given passage. R-NET first matches the question and passage with gated attention-based recurrent networks to obtain the question-aware passage representation. Then R-NET applies a self-matching attention mechanism to refine the representation by matching the passage against itself, which effectively encodes information from the whole passage. Finally, the R-NET employ the pointer networks to locate the positions of answers from the passages.
- The R-NET Model
The model consists of five different layers as Shown in Figure 1.
- The R-NET Model Layers
Figure 1 gives an overview of R-NET. First, the question and passage are processed by a bidirectional recurrent network (Mikolov et al., 2010) separately. We then match the question and passage with gated attention-based recurrent networks, obtaining question-aware representation for the passage. On top of that, we apply self-matching attention to aggregate evidence from the whole passage and refine the passage representation, which is then fed into the output layer to predict the boundary of the answer span.
- R-NET Training Dataset
There are two training datasets for the R-NET model. We can use any of them to train the model. We mainly focus on the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016) and Microsoft Machine Reading Comprehension (MS-MARCO) dataset, two large-scale datasets for reading comprehension and question answering which both are manually created through crowdsourcing.
a. R-NET is trained on Standford Question Answering Dataset (SQUAD) as just as the BiDAF model described earlier. SQUAD requires answering questions given a passage. It constrains answers to the space of all possible spans within the reference passage, which is different from cloze-style reading comprehension datasets (Hermann et al., 2015; Hill et al., 2016) in which answers are single words or entities. Moreover, SQuAD requires different forms of logical reasoning to infer the answer (Rajpurkar et al., 2016).
b. Microsoft Machine Reading Comprehension (MS MARCO) is a new large-scale dataset for reading comprehension and question answering. In MS MARCO, all questions are sampled from real anonymized user queries. The context passages, from which answers in the dataset are derived, are extracted from real web documents using the most advanced version of the Bing search engine. The answers to the queries are human generated if they could summarize the answer. MS-MARCO provides several related documents collected from Bing Index for a question. The answer to the question in MS-MARCO is generated by human and the answer words cannot only come from the given text.
To train the model, please follow instrucion in the link here
- R-NET Test
The R-NET paper authors provided us with a SQUAD-trained R-NET model and some sample demo code ( Figure 2 ).
- Creating a QA-Bot with R-NET Model for our comparison study
Instead of trying QA on multiple disjoint documents, we wanted to create a QA-Bot for a big corpus using the trained R-NET model. For creating our test corpus, we choose the book Future Computed by Harry Shum and Brad Smith. We converted the online book PDF to a word format and removed all images and diagrams from the book. Our test corpus now consists of text only. We wrote a bot script where we use this corpus only for testing any question coming from the bot UI. We operationalized the bot and tested it with several questions on the topic of Artificial Intelligence (AI).
-
Existing Resources
Paper: https://www.microsoft.com/en-us/research/wp-content/uploads/2017/05/r-net.pdf
GitHub: https://github.com/minsangkim142/R-net