LAB3_RNN/LAB3_RNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 style=\"text-align:center\">Deep Learning   </h1>\n",
    "\n",
    "<h1 style=\"text-align:center\"> Sentiment Analysis with Recurrent Neural Networks</h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The aim of this session is to practice with VanillaRNN and Gated Recurrent Units (GRU). Each group should fill and run appropriate notebook cells. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section 1: Sentiment Analysis with a Vanilla RNN\n",
    "\n",
    "You will work on a corpus of 3,000 user comments taken from IMDb (1,000), Amazon (1,000) and Yelp (1,000). These comments are split into two categories: positive comments (denoted by \"1\") and negative comments (denoted by \"0\"). For each website, 500 comments are positive and 500 comments are negative. This corpus has been created for the paper <i>From Group to Individual Labels using Deep Features</i> by Kotzias <i>et al</i> (KDD '15 Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Pages 597-606, Sydney, NSW, Australia — August 10 - 13, 2015, ACM New York, NY, USA ©2015  ISBN: 978-1-4503-3664-2 doi>10.1145/2783258.2783380).\n",
    "\n",
    "In this lab, we split this dataset into a training set of 2,520 comments (420 positive comments and 420 negative comments from each website), a validation set of 240 comments (40 positive comments and 40 negative comments from each website) and a test set of 240 comments (40 positive comments and 40 negative comments from each website).\n",
    "\n",
    "Your goal will be to classify automatically these sentences by training a Vanilla RNN and then a GRU. Please note that we use the word2vec method to convert words into vectors (Embedding of 300 dimensions in this lab): these vectors are designed so that they reflect the semantic and the syntactic functions of words. You can read more about word2vec in the paper <i>Distributed representations of words and phrases and their compositionality</i> by Mikolov <i>et al.</i> (NIPS'13 Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, Pages 3111-3119, Lake Tahoe, Nevada — December 05 - 10, 2013).\n",
    "\n",
    "First of all, please run the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/liangjianzhong/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    }
   ],
   "source": [
    "# Imports\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import utils\n",
    "\n",
    "# Parameters\n",
    "epsilon = 1e-10\n",
    "max_l = 32 # Max length of sentences\n",
    "\n",
    "train, val, test, word2vec = utils.load_data()\n",
    "data = utils.Dataset(train, val, test, word2vec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### look into data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(['great', 'for', 'the', 'jawbone'], 1.0)\n",
      "the dimension of word2vec: (300,)\n"
     ]
    }
   ],
   "source": [
    "print(train[1])\n",
    "print(\"the dimension of word2vec:\",word2vec['is'].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following cell, we define a VanillaRNN class. Please read its code carefully before running the cell because you will need to implement a similar class for the GRU.\n",
    "\n",
    "If our sentence is represented by the sequence $(x_1, ..., x_L)$, the hidden states $h_t$ of the Vanilla RNN are defined as\n",
    "\n",
    "<div align=\"center\">$h_0 = 0$</div>\n",
    "<div align=\"center\">$h_{t+1} = f(W_h h_t + W_x x_{t+1} + b)$</div>\n",
    "\n",
    "where $W_h$, $W_x$ and $b$ are trainable parameters and $f$ is an activation function.\n",
    "<img src=\"images/RNN.png\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class VanillaRNN:\n",
    "\n",
    "    def __init__(self, input_size, hidden_states, activation=None, name=None):\n",
    "        self._hidden_states = hidden_states\n",
    "        self._input_size = input_size\n",
    "        self._activation = activation or tf.tanh\n",
    "        self._name = (name or \"vanilla_rnn\") + \"/\"\n",
    "        self._candidate_kernel = tf.get_variable(self._name + \"candidate/weights\",\n",
    "                                                 shape=[input_size + self._hidden_states, self._hidden_states])\n",
    "        self._candidate_bias = tf.get_variable(self._name + \"candidate/bias\", shape=[self._hidden_states])\n",
    "        tf.summary.histogram(\"weights\", self._candidate_kernel)\n",
    "        tf.summary.histogram(\"biases\", self._candidate_bias)\n",
    "\n",
    "    def state_size(self):\n",
    "        return self._hidden_states\n",
    "    \n",
    "    # generate zero initial hidden state\n",
    "    def zero_state(self, inputs):\n",
    "        batch_size = tf.shape(inputs)[0]\n",
    "        return tf.zeros([batch_size, self.state_size()], dtype=tf.float32)\n",
    "\n",
    "    # run over one time step\n",
    "    def __call__(self, inputs, state):\n",
    "        candidate = tf.matmul(tf.concat([inputs, state], 1), self._candidate_kernel)\n",
    "        candidate = tf.nn.bias_add(candidate, self._candidate_bias)\n",
    "        new_h = self._activation(candidate)\n",
    "        tf.summary.histogram(\"activations\", candidate)\n",
    "        tf.summary.histogram(\"hiddenstate\", new_h)\n",
    "        return new_h"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we define our model. Please read the code of the process_sequence() function to understand the utility of the MaskData placeholder. If $h_L$ is the last hidden state of the Vanilla RNN, then we define our final prediction $p$ as\n",
    "\n",
    "<div align=\"center\">$p = \\sigma (W_{pred} h_L + b_{pred})$</div>\n",
    "\n",
    "where $W_{pred}$ and $b_{pred}$ are trainable parameters and $\\sigma$ denotes the sigmoid function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "# Parameters\n",
    "learning_rate = 0.001\n",
    "training_epochs = 30\n",
    "batch_size = 128\n",
    "hidden_states = 50\n",
    "\n",
    "# graph\n",
    "tf.reset_default_graph()\n",
    "tf.set_random_seed(123)\n",
    "model_path = \"models/vanilla.ckpt\"\n",
    "# tf Graph Input:  sentiment analysis data\n",
    "\n",
    "# Sentences are padded with zero vectors\n",
    "x = tf.placeholder(tf.float32, [None, max_l, 300], name='InputData')\n",
    "# masks: necessary as we have different sentence lengths\n",
    "m = tf.placeholder(tf.float32, [None, max_l, 1], name='MaskData')\n",
    "# positive (1) or negative (0) labels\n",
    "y = tf.placeholder(tf.float32, [None, 1], name='LabelData')\n",
    "\n",
    "with tf.name_scope('RNN'):\n",
    "    # we define our VanillaRNN cell\n",
    "    vanilla = VanillaRNN(300, hidden_states)\n",
    "    # we retrieve its last output\n",
    "    vanilla_output = utils.process_sequence(vanilla, x, m)\n",
    "\n",
    "with tf.name_scope('Predict'):\n",
    "    W = tf.Variable(tf.zeros([hidden_states, 1]), name='Weights')\n",
    "    b = tf.Variable(tf.zeros([1]), name='Bias')\n",
    "    # we make the final prediction\n",
    "    pred = tf.nn.sigmoid(tf.matmul(vanilla_output, W) + b)\n",
    "    tf.summary.histogram(\"weights\", W)\n",
    "    tf.summary.histogram(\"biases\", b)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Eventually, we train our model using a cross-entropy loss and the Adam optimizer. At each epoch we check the validation accuracy, and save the model if that accuracy increased. At the end, we load the best model on validation, and print its accuracy on the test set.\n",
    "\n",
    "We test our model using a $\\tanh$ activation function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training started\n",
      "Accuracy on validation: 0.5\n",
      "        Model saved in file: models/vanilla.ckpt\n",
      "Epoch:  01   =====> Loss= 0.694689268\n",
      "Accuracy on validation: 0.6666667\n",
      "        Model saved in file: models/vanilla.ckpt\n",
      "Epoch:  02   =====> Loss= 0.690510916\n",
      "Accuracy on validation: 0.7416667\n",
      "        Model saved in file: models/vanilla.ckpt\n",
      "Epoch:  03   =====> Loss= 0.662441825\n",
      "Accuracy on validation: 0.69166666\n",
      "Epoch:  04   =====> Loss= 0.582807086\n",
      "Accuracy on validation: 0.77916664\n",
      "        Model saved in file: models/vanilla.ckpt\n",
      "Epoch:  05   =====> Loss= 0.528961472\n",
      "Accuracy on validation: 0.79583335\n",
      "        Model saved in file: models/vanilla.ckpt\n",
      "Epoch:  06   =====> Loss= 0.471243090\n",
      "Accuracy on validation: 0.8\n",
      "        Model saved in file: models/vanilla.ckpt\n",
      "Epoch:  07   =====> Loss= 0.430939939\n",
      "Accuracy on validation: 0.8208333\n",
      "        Model saved in file: models/vanilla.ckpt\n",
      "Epoch:  08   =====> Loss= 0.417647840\n",
      "Accuracy on validation: 0.8208333\n",
      "Epoch:  09   =====> Loss= 0.395544462\n",
      "Accuracy on validation: 0.8208333\n",
      "Epoch:  10   =====> Loss= 0.373513074\n",
      "Accuracy on validation: 0.825\n",
      "        Model saved in file: models/vanilla.ckpt\n",
      "Epoch:  11   =====> Loss= 0.357039759\n",
      "Accuracy on validation: 0.84166664\n",
      "        Model saved in file: models/vanilla.ckpt\n",
      "Epoch:  12   =====> Loss= 0.356905489\n",
      "Accuracy on validation: 0.8333333\n",
      "Epoch:  13   =====> Loss= 0.343876859\n",
      "Accuracy on validation: 0.8375\n",
      "Epoch:  14   =====> Loss= 0.338211777\n",
      "Accuracy on validation: 0.8208333\n",
      "Epoch:  15   =====> Loss= 0.336021914\n",
      "Accuracy on validation: 0.85833335\n",
      "        Model saved in file: models/vanilla.ckpt\n",
      "Epoch:  16   =====> Loss= 0.334060202\n",
      "Accuracy on validation: 0.82916665\n",
      "Epoch:  17   =====> Loss= 0.312384519\n",
      "Accuracy on validation: 0.82916665\n",
      "Epoch:  18   =====> Loss= 0.300098574\n",
      "Accuracy on validation: 0.81666666\n",
      "Epoch:  19   =====> Loss= 0.320890601\n",
      "Accuracy on validation: 0.8375\n",
      "Epoch:  20   =====> Loss= 0.298526885\n",
      "Accuracy on validation: 0.82916665\n",
      "Epoch:  21   =====> Loss= 0.287694277\n",
      "Accuracy on validation: 0.8375\n",
      "Epoch:  22   =====> Loss= 0.282658980\n",
      "Accuracy on validation: 0.8375\n",
      "Epoch:  23   =====> Loss= 0.271541340\n",
      "Accuracy on validation: 0.84166664\n",
      "Epoch:  24   =====> Loss= 0.270977124\n",
      "Accuracy on validation: 0.8541667\n",
      "Epoch:  25   =====> Loss= 0.265912869\n",
      "Accuracy on validation: 0.82916665\n",
      "Epoch:  26   =====> Loss= 0.271027016\n",
      "Accuracy on validation: 0.84583336\n",
      "Epoch:  27   =====> Loss= 0.248016080\n",
      "Accuracy on validation: 0.8541667\n",
      "Epoch:  28   =====> Loss= 0.235057430\n",
      "Accuracy on validation: 0.8541667\n",
      "Epoch:  29   =====> Loss= 0.234967814\n",
      "Accuracy on validation: 0.82916665\n",
      "Epoch:  30   =====> Loss= 0.224477890\n",
      "INFO:tensorflow:Restoring parameters from models/vanilla.ckpt\n",
      "Accuracy: 0.84583336\n"
     ]
    }
   ],
   "source": [
    "with tf.name_scope('Loss'):\n",
    "    # Minimize error using cross entropy\n",
    "    # We use tf.clip_by_value to avoid having too low numbers in the log function\n",
    "    cost = tf.reduce_mean(-y*tf.log(tf.clip_by_value(pred, epsilon, 1.0)) - (1.-y)*tf.log(tf.clip_by_value((1.-pred), epsilon, 1.0)))\n",
    "\n",
    "with tf.name_scope('Adam'):\n",
    "    # Gradient Descent\n",
    "    optimizer = tf.train.AdamOptimizer(learning_rate).minimize(cost)\n",
    "with tf.name_scope('Accuracy'):\n",
    "    # Accuracy\n",
    "    pred_tmp = tf.stack([pred, 1.-pred])\n",
    "    y_tmp = tf.stack([y, 1.-y])\n",
    "    acc = tf.equal(tf.argmax(pred_tmp, 0), tf.argmax(y_tmp, 0))\n",
    "    acc = tf.reduce_mean(tf.cast(acc, tf.float32))\n",
    "\n",
    "# Initializing the variables\n",
    "init = tf.global_variables_initializer()\n",
    "saver = tf.train.Saver()\n",
    "tf.summary.scalar(\"loss\", cost)\n",
    "tf.summary.scalar(\"accuracy\", acc)\n",
    "merged_summary_op = tf.summary.merge_all()\n",
    "\n",
    "\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "    summary_writer = tf.summary.FileWriter('logfiles/RNN', graph=tf.get_default_graph())\n",
    "    # Training cycle\n",
    "    print(\"Training started\")\n",
    "    best_val_acc = 0.\n",
    "    for epoch in range(training_epochs):\n",
    "        avg_cost = 0.\n",
    "        total_batch = int(len(train)/batch_size)\n",
    "        # Loop over all batches\n",
    "        for i in range(total_batch):\n",
    "            batch_xs, batch_ms, batch_ys = data.next_batch(batch_size)\n",
    "            # Run optimization op (backprop), cost op (to get loss value)\n",
    "            # and summary nodes\n",
    "            _, c, summary = sess.run([optimizer, cost, merged_summary_op],\n",
    "                                     feed_dict={x: batch_xs, y: batch_ys, m: batch_ms})\n",
    "            summary_writer.add_summary(summary, epoch * total_batch + i)\n",
    "            \n",
    "            # Compute average loss\n",
    "            avg_cost += c / total_batch\n",
    "        # Display logs per epoch step\n",
    "        val_xs, val_ms, val_ys = data.val_batch()\n",
    "        val_acc = acc.eval({x: val_xs, m: val_ms, y: val_ys})\n",
    "        print(\"Accuracy on validation:\", val_acc)\n",
    "        if val_acc > best_val_acc:\n",
    "            best_val_acc = val_acc\n",
    "            save_path = saver.save(sess, model_path)\n",
    "            print(\"        Model saved in file: %s\" % save_path)\n",
    "        print(\"Epoch: \", '%02d' % (epoch+1), \"  =====> Loss=\", \"{:.9f}\".format(avg_cost))\n",
    "\n",
    "    # Test model\n",
    "    # Calculate accuracy\n",
    "    saver.restore(sess, model_path)\n",
    "    test_xs, test_ms, test_ys = data.test_batch()\n",
    "    print(\"Accuracy:\", acc.eval({x: test_xs, m: test_ms, y: test_ys}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Did you understand everything? If so, you can move towards Section 2.\n",
    "\n",
    "# Section 2: GRU\n",
    "\n",
    "<img src=\"images/GRU.png\">\n",
    "\n",
    "<b>Question 1</b> - Recall the formulas defining the hidden states of a GRU.<br>\n",
    "<div class='alert alert-success'>\n",
    "<ul>\n",
    "<li><b>Remember gate:</b><br>\n",
    "$ r^t = sigmoid(W_r \\cdot [h^{t-1}, x^t] + b_r) $</li><br>\n",
    "<li><b>Input:</b><br>\n",
    "$ h^{'t} = tanh(W_i \\cdot [r^t \\otimes h^{t-1}, x^t] + b_i) $</li><br>\n",
    "<li><strong>Update gate:</strong><br>\n",
    "$ z^t = sigmoid(W_z \\cdot [h^{t-1}, x^t] + b_z) $</li>\n",
    "<br>\n",
    "$ h^{t} = z^t \\otimes h^{'t} + (1-z^t) \\otimes h^{t-1} $\n",
    "</ul>\n",
    "<img src=\"https://cdnpythonmachinelearning.azureedge.net/wp-content/uploads/2017/11/GRU.png?x31195\" width=\"400\">\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Question 2</b> - Define a GRU similar to the Vanilla RNN that we defined in Section 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class GRU:\n",
    "\n",
    "    def __init__(self, input_size, hidden_states, activation=None, name=None):\n",
    "        self._hidden_states = hidden_states\n",
    "        self._input_size = input_size\n",
    "        self._activation = activation or tf.tanh\n",
    "        self._name = (name or \"gru\") + \"/\"\n",
    "        \n",
    "        ############ CODE NEEDED ############\n",
    "        # Define trainable parameters here  #\n",
    "        #####################################\n",
    "        # R\n",
    "        self._candidate_kernel_R = tf.get_variable(self._name + \"candidate/weightsR\",\n",
    "                                                 shape=[input_size + self._hidden_states, self._hidden_states])\n",
    "        self._candidate_bias_R = tf.get_variable(self._name + \"candidate/biasR\", shape=[self._hidden_states])\n",
    "        \n",
    "        # I\n",
    "        self._candidate_kernel_I = tf.get_variable(self._name + \"candidate/weightsI\",\n",
    "                                                 shape=[input_size + self._hidden_states, self._hidden_states])\n",
    "        self._candidate_bias_I = tf.get_variable(self._name + \"candidate/biasI\", shape=[self._hidden_states])\n",
    "        \n",
    "        # Z\n",
    "        self._candidate_kernel_Z = tf.get_variable(self._name + \"candidate/weightsZ\",\n",
    "                                                 shape=[input_size + self._hidden_states, self._hidden_states])\n",
    "        self._candidate_bias_Z = tf.get_variable(self._name + \"candidate/biasZ\", shape=[self._hidden_states])\n",
    "        \n",
    "        tf.summary.histogram(\"R_W\", self._candidate_kernel_R)\n",
    "        tf.summary.histogram(\"R_b\", self._candidate_bias_R)\n",
    "        tf.summary.histogram(\"I_W\",self._candidate_kernel_I)\n",
    "        tf.summary.histogram(\"I_b\", self._candidate_bias_I)\n",
    "        tf.summary.histogram(\"Z_W\",self._candidate_kernel_Z)\n",
    "        tf.summary.histogram(\"Z_b\", self._candidate_bias_Z)        \n",
    "\n",
    "\n",
    "\n",
    "    def state_size(self):\n",
    "        return self._hidden_states\n",
    "\n",
    "    def output_size(self):\n",
    "        return self._hidden_states\n",
    "\n",
    "    def zero_state(self, inputs):\n",
    "        batch_size = tf.shape(inputs)[0]\n",
    "        return tf.zeros([batch_size, self.state_size()], dtype=tf.float32)\n",
    "\n",
    "    \n",
    "    def __call__(self, inputs, state):\n",
    "        \n",
    "        # Remember\n",
    "        candidate_R = tf.matmul(tf.concat([inputs, state], 1), self._candidate_kernel_R)\n",
    "        candidate_R = tf.nn.bias_add(candidate_R, self._candidate_bias_R)\n",
    "        r = tf.sigmoid(candidate_R)\n",
    "        \n",
    "        # Input\n",
    "        candidate_I = tf.matmul(tf.concat([inputs, tf.multiply(r, state)], 1), self._candidate_kernel_I)\n",
    "        candidate_I = tf.nn.bias_add(candidate_I, self._candidate_bias_I)\n",
    "        i = tf.tanh(candidate_I)\n",
    "        \n",
    "        # Update\n",
    "        candidate_Z = tf.matmul(tf.concat([inputs, tf.multiply(r, state)], 1), self._candidate_kernel_Z)\n",
    "        candidate_Z = tf.nn.bias_add(candidate_Z, self._candidate_bias_Z)\n",
    "        z = tf.sigmoid(candidate_Z)\n",
    "        \n",
    "        new_h = tf.multiply(z, i) + tf.multiply(1-z, state)\n",
    "        return new_h"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Question 3</b> - Train that GRU with a $tanh$ activation function and print its accuracy on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training started\n",
      "Accuracy on validation: 0.5833333\n",
      "        Model saved in file: models/gru.ckpt\n",
      "Epoch:  01   =====> Loss= 0.691213316\n",
      "Accuracy on validation: 0.75\n",
      "        Model saved in file: models/gru.ckpt\n",
      "Epoch:  02   =====> Loss= 0.663495892\n",
      "Accuracy on validation: 0.78333336\n",
      "        Model saved in file: models/gru.ckpt\n",
      "Epoch:  03   =====> Loss= 0.545318318\n",
      "Accuracy on validation: 0.81666666\n",
      "        Model saved in file: models/gru.ckpt\n",
      "Epoch:  04   =====> Loss= 0.446474080\n",
      "Accuracy on validation: 0.825\n",
      "        Model saved in file: models/gru.ckpt\n",
      "Epoch:  05   =====> Loss= 0.399288366\n",
      "Accuracy on validation: 0.825\n",
      "Epoch:  06   =====> Loss= 0.378474634\n",
      "Accuracy on validation: 0.825\n",
      "Epoch:  07   =====> Loss= 0.356144936\n",
      "Accuracy on validation: 0.84166664\n",
      "        Model saved in file: models/gru.ckpt\n",
      "Epoch:  08   =====> Loss= 0.339461015\n",
      "Accuracy on validation: 0.84583336\n",
      "        Model saved in file: models/gru.ckpt\n",
      "Epoch:  09   =====> Loss= 0.323130208\n",
      "Accuracy on validation: 0.8541667\n",
      "        Model saved in file: models/gru.ckpt\n",
      "Epoch:  10   =====> Loss= 0.313221033\n",
      "Accuracy on validation: 0.85833335\n",
      "        Model saved in file: models/gru.ckpt\n",
      "Epoch:  11   =====> Loss= 0.302376412\n",
      "Accuracy on validation: 0.82916665\n",
      "Epoch:  12   =====> Loss= 0.295511073\n",
      "Accuracy on validation: 0.84166664\n",
      "Epoch:  13   =====> Loss= 0.286810462\n",
      "Accuracy on validation: 0.85833335\n",
      "Epoch:  14   =====> Loss= 0.278383929\n",
      "Accuracy on validation: 0.84583336\n",
      "Epoch:  15   =====> Loss= 0.267085533\n",
      "Accuracy on validation: 0.8541667\n",
      "Epoch:  16   =====> Loss= 0.256256917\n",
      "Accuracy on validation: 0.8625\n",
      "        Model saved in file: models/gru.ckpt\n",
      "Epoch:  17   =====> Loss= 0.250856972\n",
      "Accuracy on validation: 0.8375\n",
      "Epoch:  18   =====> Loss= 0.247814431\n",
      "Accuracy on validation: 0.85\n",
      "Epoch:  19   =====> Loss= 0.238662619\n",
      "Accuracy on validation: 0.85833335\n",
      "Epoch:  20   =====> Loss= 0.225614341\n",
      "Accuracy on validation: 0.8541667\n",
      "Epoch:  21   =====> Loss= 0.220624724\n",
      "Accuracy on validation: 0.85833335\n",
      "Epoch:  22   =====> Loss= 0.203089611\n",
      "Accuracy on validation: 0.8375\n",
      "Epoch:  23   =====> Loss= 0.198559469\n",
      "Accuracy on validation: 0.85\n",
      "Epoch:  24   =====> Loss= 0.188314638\n",
      "Accuracy on validation: 0.8375\n",
      "Epoch:  25   =====> Loss= 0.177515809\n",
      "Accuracy on validation: 0.85833335\n",
      "Epoch:  26   =====> Loss= 0.188248508\n",
      "Accuracy on validation: 0.84583336\n",
      "Epoch:  27   =====> Loss= 0.181234477\n",
      "Accuracy on validation: 0.85833335\n",
      "Epoch:  28   =====> Loss= 0.166815496\n",
      "Accuracy on validation: 0.8375\n",
      "Epoch:  29   =====> Loss= 0.162653621\n",
      "Accuracy on validation: 0.85\n",
      "Epoch:  30   =====> Loss= 0.147546547\n",
      "INFO:tensorflow:Restoring parameters from models/gru.ckpt\n",
      "Accuracy: 0.8666667\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "tf.set_random_seed(123)\n",
    "model_path = \"models/gru.ckpt\"\n",
    "board_path = \"logfiles/GRU\"\n",
    "\n",
    "# tf Graph Input:  sentiment analysis data\n",
    "x = tf.placeholder(tf.float32, [None, max_l, 300], name='InputData')\n",
    "# masks\n",
    "m = tf.placeholder(tf.float32, [None, max_l, 1], name='MaskData')\n",
    "# Positive (1) or Negative (0) labels\n",
    "y = tf.placeholder(tf.float32, [None, 1], name='LabelData')\n",
    "\n",
    "with tf.name_scope('RUN_GRU'):\n",
    "    gru = GRU(300, hidden_states)\n",
    "    gru_output = utils.process_sequence(gru, x, m)\n",
    "with tf.name_scope('Projection'):\n",
    "    W = tf.Variable(tf.zeros([hidden_states, 1]), name='Weights')\n",
    "    b = tf.Variable(tf.zeros([1]), name='Bias')\n",
    "    pred = tf.nn.sigmoid(tf.matmul(gru_output, W) + b)\n",
    "    tf.summary.histogram(\"P_W\",W)\n",
    "    tf.summary.histogram(\"P_b\", b)\n",
    "\n",
    "with tf.name_scope('Loss'):\n",
    "    # Minimize error using cross entropy\n",
    "    # We use tf.clip_by_value to avoid having too low numbers in the log function\n",
    "    cost = tf.reduce_mean(-y*tf.log(tf.clip_by_value(pred, epsilon, 1.0)) - (1.-y)*tf.log(tf.clip_by_value((1.-pred), epsilon, 1.0)))\n",
    "\n",
    "with tf.name_scope('Adam'):\n",
    "    # Gradient Descent\n",
    "    optimizer = tf.train.AdamOptimizer(learning_rate).minimize(cost)\n",
    "with tf.name_scope('Accuracy'):\n",
    "    # Accuracy\n",
    "    pred_tmp = tf.stack([pred, 1.-pred])\n",
    "    y_tmp = tf.stack([y, 1.-y])\n",
    "    acc = tf.equal(tf.argmax(pred_tmp, 0), tf.argmax(y_tmp, 0))\n",
    "    acc = tf.reduce_mean(tf.cast(acc, tf.float32))\n",
    "    \n",
    "tf.summary.scalar(\"loss\", cost)\n",
    "tf.summary.scalar(\"accuracy\", acc)\n",
    "merged_summary_op = tf.summary.merge_all()\n",
    "    \n",
    "    \n",
    "# Initializing the variables\n",
    "init = tf.global_variables_initializer()\n",
    "saver = tf.train.Saver()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "    summary_writer = tf.summary.FileWriter(board_path, graph=tf.get_default_graph())\n",
    "    # Training cycle\n",
    "    print(\"Training started\")\n",
    "    best_val_acc = 0.\n",
    "    for epoch in range(training_epochs):\n",
    "        avg_cost = 0.\n",
    "        total_batch = int(len(train)/batch_size)\n",
    "        # Loop over all batches\n",
    "        for i in range(total_batch):\n",
    "            batch_xs, batch_ms, batch_ys = data.next_batch(batch_size)\n",
    "            # Run optimization op (backprop), cost op (to get loss value)\n",
    "            # and summary nodes\n",
    "            _, c, summary = sess.run([optimizer, cost, merged_summary_op],\n",
    "                                     feed_dict={x: batch_xs, y: batch_ys, m: batch_ms})\n",
    "            summary_writer.add_summary(summary, epoch * total_batch + i)\n",
    "            \n",
    "            # Compute average loss\n",
    "            avg_cost += c / total_batch\n",
    "        # Display logs per epoch step\n",
    "        val_xs, val_ms, val_ys = data.val_batch()\n",
    "        val_acc = acc.eval({x: val_xs, m: val_ms, y: val_ys})\n",
    "        print(\"Accuracy on validation:\", val_acc)\n",
    "        if val_acc > best_val_acc:\n",
    "            best_val_acc = val_acc\n",
    "            save_path = saver.save(sess, model_path)\n",
    "            print(\"        Model saved in file: %s\" % save_path)\n",
    "        print(\"Epoch: \", '%02d' % (epoch+1), \"  =====> Loss=\", \"{:.9f}\".format(avg_cost))\n",
    "\n",
    "    # Test model\n",
    "    # Calculate accuracy\n",
    "    saver.restore(sess, model_path)\n",
    "    test_xs, test_ms, test_ys = data.test_batch()\n",
    "    print(\"Accuracy:\", acc.eval({x: test_xs, m: test_ms, y: test_ys}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Question 4</b> - Comment on your findings:\n",
    "<div class='alert alert-success'>\n",
    "<b>GRU</b> (together with LSTM) faces the problem of the <b>vanishing</b> (or exploding) gradient. Vanilla RNN indeed, especially for the long chain of layers which is composed of (thinking of it in a \"unrolled\" fashion), has the big issue of having a very small gradient during the backpropagation phase: this issue prevents the weights of the network to change their value.<br>\n",
    "The idea of LSTM is that the network can keep <b>memory</b> (or in the case of the GRU, the <b>hidden state</b>) for a long term in order to avoid the gradient to vanish or explode.<br>\n",
    "<img src=\"https://qph.fs.quoracdn.net/main-qimg-d4725497028c5a60241e1524c32f60de\">\n",
    "<br>\n",
    "While the LSTM uses memory units, the GRU exposes always the entire hidden state (<a href=\"https://arxiv.org/pdf/1502.02367.pdf\">source</a>). Of course, even if accuracy is generally in line with the LSTM, the GRU is more efficient from the computational point of view, because of the lack of complex memory structures.<br>\n",
    "<br>\n",
    "In our case the performances are better!<br>\n",
    "<ul>\n",
    "    <li>Vanilla RNN: <b>84.6%</b></li>\n",
    "    <li>GRU RNN: <b>86.7%</b></li>\n",
    "</ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}