skip_thoughts

Skip-Thought Vectors

This is a TensorFlow implementation of the model described in:

Jamie Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, Sanja Fidler. Skip-Thought Vectors. In NIPS, 2015.

Contact

Code author: Chris Shallue

Pull requests and issues: @cshallue

Contents

• Model Overview
• Getting Started
  • Install Required Packages
  • Download Pretrained Models (Optional)
• Training a Model
  • Prepare the Training Data
  • Run the Training Script
  • Track Training Progress
• Expanding the Vocabulary
  • Overview
  • Preparation
  • Run the Vocabulary Expansion Script
• Evaluating a Model
  • Overview
  • Preparation
  • Run the Evaluation Tasks
• Encoding Sentences

Model overview

The Skip-Thoughts model is a sentence encoder. It learns to encode input sentences into a fixed-dimensional vector representation that is useful for many tasks, for example to detect paraphrases or to classify whether a product review is positive or negative. See the Skip-Thought Vectors paper for details of the model architecture and more example applications.

A trained Skip-Thoughts model will encode similar sentences nearby each other in the embedding vector space. The following examples show the nearest neighbor by cosine similarity of some sentences from the movie review dataset.

Input sentence	Nearest Neighbor
Simplistic, silly and tedious.	Trite, banal, cliched, mostly inoffensive.
Not so much farcical as sour.	Not only unfunny, but downright repellent.
A sensitive and astute first feature by Anne-Sophie Birot.	Absorbing character study by André Turpin .
An enthralling, entertaining feature.	A slick, engrossing melodrama.

Getting Started

Install Required Packages

First ensure that you have installed the following required packages:

• Bazel (instructions)
• TensorFlow (instructions)
• NumPy (instructions)
• scikit-learn (instructions)
• Natural Language Toolkit (NLTK)
  • First install NLTK (instructions)
  • Then install the NLTK data (instructions)
• gensim (instructions)
  • Only required if you will be expanding your vocabulary with the word2vec model.

Download Pretrained Models (Optional)

You can download model checkpoints pretrained on the BookCorpus dataset in the following configurations:

• Unidirectional RNN encoder ("uni-skip" in the paper)
• Bidirectional RNN encoder ("bi-skip" in the paper)

# Directory to download the pretrained models to.
PRETRAINED_MODELS_DIR="${HOME}/skip_thoughts/pretrained/"

mkdir -p ${PRETRAINED_MODELS_DIR}
cd ${PRETRAINED_MODELS_DIR}

# Download and extract the unidirectional model.
wget "http://download.tensorflow.org/models/skip_thoughts_uni_2017_02_02.tar.gz"
tar -xvf skip_thoughts_uni_2017_02_02.tar.gz
rm skip_thoughts_uni_2017_02_02.tar.gz

# Download and extract the bidirectional model.
wget "http://download.tensorflow.org/models/skip_thoughts_bi_2017_02_16.tar.gz"
tar -xvf skip_thoughts_bi_2017_02_16.tar.gz
rm skip_thoughts_bi_2017_02_16.tar.gz

You can now skip to the sections Evaluating a Model and Encoding Sentences.

Training a Model

Prepare the Training Data

To train a model you will need to provide training data in TFRecord format. The TFRecord format consists of a set of sharded files containing serialized tf.Example protocol buffers. Each tf.Example proto contains three sentences:

• encode: The sentence to encode.
• decode_pre: The sentence preceding encode in the original text.
• decode_post: The sentence following encode in the original text.

Each sentence is a list of words. During preprocessing, a dictionary is created that assigns each word in the vocabulary to an integer-valued id. Each sentence is encoded as a list of integer word ids in the tf.Example protos.

We have provided a script to preprocess any set of text-files into this format. You may wish to use the BookCorpus dataset. Note that the preprocessing script may take 12 hours or more to complete on this large dataset.

# Comma-separated list of globs matching the input input files. The format of
# the input files is assumed to be a list of newline-separated sentences, where
# each sentence is already tokenized.
INPUT_FILES="${HOME}/skip_thoughts/bookcorpus/*.txt"

# Location to save the preprocessed training and validation data.
DATA_DIR="${HOME}/skip_thoughts/data"

# Build the preprocessing script.
bazel build -c opt skip_thoughts/data/preprocess_dataset

# Run the preprocessing script.
bazel-bin/skip_thoughts/data/preprocess_dataset \
  --input_files=${INPUT_FILES} \
  --output_dir=${DATA_DIR}

When the script finishes you will find 100 training files and 1 validation file in DATA_DIR. The files will match the patterns train-?????-of-00100 and validation-00000-of-00001 respectively.

The script will also produce a file named vocab.txt. The format of this file is a list of newline-separated words where the word id is the corresponding 0- based line index. Words are sorted by descending order of frequency in the input data. Only the top 20,000 words are assigned unique ids; all other words are assigned the "unknown id" of 1 in the processed data.

Run the Training Script

Execute the following commands to start the training script. By default it will run for 500k steps (around 9 days on a GeForce GTX 1080 GPU).

# Directory containing the preprocessed data.
DATA_DIR="${HOME}/skip_thoughts/data"

# Directory to save the model.
MODEL_DIR="${HOME}/skip_thoughts/model"

# Build the model.
bazel build -c opt skip_thoughts/...

# Run the training script.
bazel-bin/skip_thoughts/train \
  --input_file_pattern="${DATA_DIR}/train-?????-of-00100" \
  --train_dir="${MODEL_DIR}/train"

Track Training Progress

Optionally, you can run the track_perplexity script in a separate process. This will log per-word perplexity on the validation set which allows training progress to be monitored on TensorBoard.

Note that you may run out of memory if you run the this script on the same GPU as the training script. You can set the environment variable CUDA_VISIBLE_DEVICES="" to force the script to run on CPU. If it runs too slowly on CPU, you can decrease the value of --num_eval_examples.

DATA_DIR="${HOME}/skip_thoughts/data"
MODEL_DIR="${HOME}/skip_thoughts/model"

# Ignore GPU devices (only necessary if your GPU is currently memory
# constrained, for example, by running the training script).
export CUDA_VISIBLE_DEVICES=""

# Run the evaluation script. This will run in a loop, periodically loading the
# latest model checkpoint file and computing evaluation metrics.
bazel-bin/skip_thoughts/track_perplexity \
  --input_file_pattern="${DATA_DIR}/validation-?????-of-00001" \
  --checkpoint_dir="${MODEL_DIR}/train" \
  --eval_dir="${MODEL_DIR}/val" \
  --num_eval_examples=50000

If you started the track_perplexity script, run a TensorBoard server in a separate process for real-time monitoring of training summaries and validation perplexity.

MODEL_DIR="${HOME}/skip_thoughts/model"

# Run a TensorBoard server.
tensorboard --logdir="${MODEL_DIR}"

Expanding the Vocabulary

Overview

The vocabulary generated by the preprocessing script contains only 20,000 words which is insufficient for many tasks. For example, a sentence from Wikipedia might contain nouns that do not appear in this vocabulary.

A solution to this problem described in the Skip-Thought Vectors paper is to learn a mapping that transfers word representations from one model to another. This idea is based on the "Translation Matrix" method from the paper Exploiting Similarities Among Languages for Machine Translation.

Specifically, we will load the word embeddings from a trained Skip-Thoughts model and from a trained word2vec model (which has a much larger vocabulary). We will train a linear regression model without regularization to learn a linear mapping from the word2vec embedding space to the Skip-Thoughts embedding space. We will then apply the linear model to all words in the word2vec vocabulary, yielding vectors in the Skip- Thoughts word embedding space for the union of the two vocabularies.

The linear regression task is to learn a parameter matrix W to minimize || X - Y * W ||², where X is a matrix of Skip-Thoughts embeddings of shape [num_words, dim1], Y is a matrix of word2vec embeddings of shape [num_words, dim2], and W is a matrix of shape [dim2, dim1].

Preparation

First you will need to download and unpack a pretrained word2vec model from this website (direct download link). This model was trained on the Google News dataset (about 100 billion words).

Also ensure that you have already installed gensim.

Run the Vocabulary Expansion Script

# Path to checkpoint file or a directory containing checkpoint files (the script
# will select the most recent).
CHECKPOINT_PATH="${HOME}/skip_thoughts/model/train"

# Vocabulary file generated by the preprocessing script.
SKIP_THOUGHTS_VOCAB="${HOME}/skip_thoughts/data/vocab.txt"

# Path to downloaded word2vec model.
WORD2VEC_MODEL="${HOME}/skip_thoughts/googlenews/GoogleNews-vectors-negative300.bin"

# Output directory.
EXP_VOCAB_DIR="${HOME}/skip_thoughts/exp_vocab"

# Build the vocabulary expansion script.
bazel build -c opt skip_thoughts/vocabulary_expansion

# Run the vocabulary expansion script.
bazel-bin/skip_thoughts/vocabulary_expansion \
  --skip_thoughts_model=${CHECKPOINT_PATH} \
  --skip_thoughts_vocab=${SKIP_THOUGHTS_VOCAB} \
  --word2vec_model=${WORD2VEC_MODEL} \
  --output_dir=${EXP_VOCAB_DIR}

Evaluating a Model

Overview

The model can be evaluated using the benchmark tasks described in the Skip-Thought Vectors paper. The following tasks are supported (refer to the paper for full details):

• SICK semantic relatedness task.
• MSRP (Microsoft Research Paraphrase Corpus) paraphrase detection task.
• Binary classification tasks:
  • MR movie review sentiment task.
  • CR customer product review task.
  • SUBJ subjectivity/objectivity task.
  • MPQA opinion polarity task.
  • TREC question-type classification task.

Preparation

You will need to clone or download the skip-thoughts GitHub repository by ryankiros (the first author of the Skip-Thoughts paper):

# Folder to clone the repository to.
ST_KIROS_DIR="${HOME}/skip_thoughts/skipthoughts_kiros"

# Clone the repository.
git clone [email protected]:ryankiros/skip-thoughts.git "${ST_KIROS_DIR}/skipthoughts"

# Make the package importable.
export PYTHONPATH="${ST_KIROS_DIR}/:${PYTHONPATH}"

You will also need to download the data needed for each evaluation task. See the instructions here.

For example, the CR (customer review) dataset is found here. For this task we want the files custrev.pos and custrev.neg.

Run the Evaluation Tasks

In the following example we will evaluate a unidirectional model ("uni-skip" in the paper) on the CR task. To use a bidirectional model ("bi-skip" in the paper), simply pass the flags --bi_vocab_file, --bi_embeddings_file and --bi_checkpoint_path instead. To use the "combine-skip" model described in the paper you will need to pass both the unidirectional and bidirectional flags.

# Path to checkpoint file or a directory containing checkpoint files (the script
# will select the most recent).
CHECKPOINT_PATH="${HOME}/skip_thoughts/model/train"

# Vocabulary file generated by the vocabulary expansion script.
VOCAB_FILE="${HOME}/skip_thoughts/exp_vocab/vocab.txt"

# Embeddings file generated by the vocabulary expansion script.
EMBEDDINGS_FILE="${HOME}/skip_thoughts/exp_vocab/embeddings.npy"

# Directory containing files custrev.pos and custrev.neg.
EVAL_DATA_DIR="${HOME}/skip_thoughts/eval_data"

# Build the evaluation script.
bazel build -c opt skip_thoughts/evaluate

# Run the evaluation script.
bazel-bin/skip_thoughts/evaluate \
  --eval_task=CR \
  --data_dir=${EVAL_DATA_DIR} \
  --uni_vocab_file=${VOCAB_FILE} \
  --uni_embeddings_file=${EMBEDDINGS_FILE} \
  --uni_checkpoint_path=${CHECKPOINT_PATH}

Output:

[0.82539682539682535, 0.84084880636604775, 0.83023872679045096,
 0.86206896551724133, 0.83554376657824936, 0.85676392572944293,
 0.84084880636604775, 0.83023872679045096, 0.85145888594164454,
 0.82758620689655171]

The output is a list of accuracies of 10 cross-validation classification models. To get a single number, simply take the average:

ipython  # Launch iPython.

In [0]:
import numpy as np
np.mean([0.82539682539682535, 0.84084880636604775, 0.83023872679045096,
         0.86206896551724133, 0.83554376657824936, 0.85676392572944293,
         0.84084880636604775, 0.83023872679045096, 0.85145888594164454,
         0.82758620689655171])

Out [0]: 0.84009936423729525

Encoding Sentences

In this example we will encode data from the movie review dataset (specifically the sentence polarity dataset v1.0).

ipython  # Launch iPython.

In [0]:

# Imports.
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import numpy as np
import os.path
import scipy.spatial.distance as sd
from skip_thoughts import configuration
from skip_thoughts import encoder_manager

In [1]:
# Set paths to the model.
VOCAB_FILE = "/path/to/vocab.txt"
EMBEDDING_MATRIX_FILE = "/path/to/embeddings.npy"
CHECKPOINT_PATH = "/path/to/model.ckpt-9999"
# The following directory should contain files rt-polarity.neg and
# rt-polarity.pos.
MR_DATA_DIR = "/dir/containing/mr/data"

In [2]:
# Set up the encoder. Here we are using a single unidirectional model.
# To use a bidirectional model as well, call load_model() again with
# configuration.model_config(bidirectional_encoder=True) and paths to the
# bidirectional model's files. The encoder will use the concatenation of
# all loaded models.
encoder = encoder_manager.EncoderManager()
encoder.load_model(configuration.model_config(),
                   vocabulary_file=VOCAB_FILE,
                   embedding_matrix_file=EMBEDDING_MATRIX_FILE,
                   checkpoint_path=CHECKPOINT_PATH)

In [3]:
# Load the movie review dataset.
data = []
with open(os.path.join(MR_DATA_DIR, 'rt-polarity.neg'), 'rb') as f:
  data.extend([line.decode('latin-1').strip() for line in f])
with open(os.path.join(MR_DATA_DIR, 'rt-polarity.pos'), 'rb') as f:
  data.extend([line.decode('latin-1').strip() for line in f])

In [4]:
# Generate Skip-Thought Vectors for each sentence in the dataset.
encodings = encoder.encode(data)

In [5]:
# Define a helper function to generate nearest neighbors.
def get_nn(ind, num=10):
  encoding = encodings[ind]
  scores = sd.cdist([encoding], encodings, "cosine")[0]
  sorted_ids = np.argsort(scores)
  print("Sentence:")
  print("", data[ind])
  print("\nNearest neighbors:")
  for i in range(1, num + 1):
    print(" %d. %s (%.3f)" %
          (i, data[sorted_ids[i]], scores[sorted_ids[i]]))

In [6]:
# Compute nearest neighbors of the first sentence in the dataset.
get_nn(0)

Output:

Sentence:
 simplistic , silly and tedious .

Nearest neighbors:
 1. trite , banal , cliched , mostly inoffensive . (0.247)
 2. banal and predictable . (0.253)
 3. witless , pointless , tasteless and idiotic . (0.272)
 4. loud , silly , stupid and pointless . (0.295)
 5. grating and tedious . (0.299)
 6. idiotic and ugly . (0.330)
 7. black-and-white and unrealistic . (0.335)
 8. hopelessly inane , humorless and under-inspired . (0.335)
 9. shallow , noisy and pretentious . (0.340)
 10. . . . unlikable , uninteresting , unfunny , and completely , utterly inept . (0.346)