worksheet06_solutions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# COMP90051 Workshop 6\n",
    "## Convolutional neural net (CNN) in TensorFlow\n",
    "***\n",
    "In the previous worksheet, we implemented a logistic regression classifier for the MNIST data set in TensorFlow, which achieved a test accuracy of 92%. \n",
    "In this worksheet, we hope to improve upon this accuracy by implementing a convolutional neural network (CNN)—a model that is more naturally suited to image data.\n",
    "We'll assume familiarity with the TensorFlow fundamentals covered previously.\n",
    "By the end of this worksheet you should be able to:\n",
    "* build more complex computation graphs\n",
    "* apply composite operators (e.g. those available under [`tf.layers`](https://www.tensorflow.org/api_docs/python/tf/layers))\n",
    "* monitor computations in [TensorBoard](https://www.tensorflow.org/guide/summaries_and_tensorboard) (a web app that's included with TensorFlow)\n",
    "\n",
    "*Note: this worksheet is draws on material from the following tutorials: [link 1](https://www.tensorflow.org/tutorials/estimators/cnn) and [link 2](https://codelabs.developers.google.com/codelabs/cloud-tensorflow-mnist/).*\n",
    "\n",
    "Let's begin by importing the required packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Resuming from Worksheet 5\n",
    "We're going to use the same MNIST data set as in Worksheet 5, so that we can compare the accuracy of the CNN with logistic regression.\n",
    "\n",
    "In Worksheet 5, we unrolled the 2D image arrays into feature vectors, as was required for logistic regression. However, here we leave the image arrays intact, as the CNN assumes images as input (it exploits spatial locality between the pixels). \n",
    "We again apply a rescaling transformation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import mnist\n",
    "(images_train, labels_train), (images_test, labels_test) = mnist.load_data()\n",
    "\n",
    "# Rescale\n",
    "images_train = images_train.astype('float32')/255\n",
    "images_test = images_test.astype('float32')/255"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we define some constants related to the data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "IM_WIDTH = images_train.shape[1]      # width of an image in pixels\n",
    "IM_HEIGHT = images_train.shape[2]     # height of an image in pixels\n",
    "NUM_CLASSES = 10                      # number of classes (0-9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We again make use of the `DatasetIterator` defined in Worksheet 5, which provides an interface for drawing randomised mini-batches from the training set.\n",
    "Note that we continue to use a batch size of 100 (you may consider changing this)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DatasetIterator:\n",
    "    \"\"\"\n",
    "    An iterator that returns randomized batches from a data set (with features and labels)\n",
    "    \"\"\"\n",
    "    def __init__(self, features, labels, batch_size):\n",
    "        assert(features.shape[0]==labels.shape[0])\n",
    "        assert(batch_size > 0 and batch_size <= features.shape[0])\n",
    "        self.features = features\n",
    "        self.labels = labels\n",
    "        self.num_instances = features.shape[0]\n",
    "        self.batch_size = batch_size\n",
    "        self.num_batches = self.num_instances//self.batch_size\n",
    "        if (self.num_instances%self.batch_size!=0):\n",
    "            self.num_batches += 1\n",
    "        self._i = 0\n",
    "        self._rand_ids = None\n",
    "\n",
    "    def __iter__(self):\n",
    "        self._i = 0\n",
    "        self._rand_ids = np.random.permutation(self.num_instances)\n",
    "        return self\n",
    "    \n",
    "    def next(self):\n",
    "        self.__next__(self)\n",
    "    \n",
    "    def __next__(self):\n",
    "        if self.num_instances - self._i >= self.batch_size:\n",
    "            this_rand_ids = self._rand_ids[self._i:self._i + self.batch_size]\n",
    "            self._i += self.batch_size\n",
    "            return self.features[this_rand_ids], self.labels[this_rand_ids]\n",
    "        elif self.num_instances - self._i > 0:\n",
    "            this_rand_ids = self._rand_ids[self._i::]\n",
    "            self._i = self.num_instances\n",
    "            return self.features[this_rand_ids], self.labels[this_rand_ids]\n",
    "        else:\n",
    "            raise StopIteration()\n",
    "            \n",
    "batch_size = 100\n",
    "train_iterator = DatasetIterator(images_train, labels_train, batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Placeholders for data input\n",
    "Following Worksheet 5, we again define placeholders for inputting data (images + labels) into the graph.\n",
    "This time we group the placeholders for the images and labels under a [variable scope](https://www.tensorflow.org/api_docs/python/tf/variable_scope) called `'input'`.\n",
    "By using variable scopes, we can simplify the graph visualisation in TensorBoard."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.variable_scope('input'):\n",
    "    X = tf.placeholder(dtype=tf.float32, shape=[None, IM_WIDTH, IM_HEIGHT], name='images')\n",
    "    Y = tf.placeholder(dtype=tf.int32, shape=[None,], name='labels')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. CNN architecture\n",
    "Due to hardware and time constraints, we must limit the size of our CNN, otherwise it will take too long to train.\n",
    "\n",
    "For the convolutional layers, we follow the \"convolutional pyramid\" design principle—i.e. successive layers have decreasing spatial dimensions, but increasing depth. (This architecture is biologically motivated.)\n",
    "The reduction in the spatial dimensions is achieved through max pooling.\n",
    "\n",
    "After the convolutional layers, we add two densely-connected layers which combine the higher-level features to make a classification.\n",
    "We also make use of dropout (a regularization method whereby random units are removed from the network) to prevent overfitting.\n",
    "Note that the final layer is similar to the logistic regression model (although the input differs).\n",
    "\n",
    "**Exercise (Advanced/Optional):** If you're interested in learning more about dropout (not examinable), you may like to read the following paper:\n",
    "> Srivastava et al. \"Dropout: a simple way to prevent neural networks from overfitting.\" JMLR 15.1 (2014): 1929-1958. [link](http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf)\n",
    "\n",
    "**Architecture overview**\n",
    "1. *Convolutional Layer #1* | 8 5×5 filters with a stride of 1 and a ReLU activation function.\n",
    "2. *Pooling Layer #1* | Max pooling with a 2×2 filter and stride of 2 (implies pooled regions do not overlap).\n",
    "3. *Convolutional Layer #2* | 16 5×5 filters with a stride of 1 and a ReLU activation function.\n",
    "4. *Pooling Layer #2* | Same specs as pooling layer #1.\n",
    "5. *Dense Layer #1* | 256 neurons, with dropout regularization rate of 0.4 (probability of 0.4 that any given element will be dropped during training)\n",
    "6. *Dense Layer #2* | Logits Layer. 10 neurons, one for each digit target class (0–9)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "DEPTH_C1 = 8       # depth of convolutional layer #1\n",
    "DEPTH_C2 = 16      # depth of convolutional layer #2\n",
    "UNITS_D1 = 256     # number of neurons in dense layer #1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fill in the missing parts of the model (Convolutional Layer #2 and Pooling Layer #2) below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.variable_scope('cnn_model'):\n",
    "    # Boolean placeholder which is set to True for training, and False for inference.\n",
    "    # This is required to implement dropout. \n",
    "    training_mode = tf.placeholder(dtype=tf.bool, name='training_mode')\n",
    "    \n",
    "    # Input Layer\n",
    "    input_layer = tf.reshape(X, [-1, IM_WIDTH, IM_HEIGHT, 1])\n",
    "\n",
    "    # Convolutional Layer #1\n",
    "    conv1 = tf.layers.conv2d(inputs=input_layer, filters=DEPTH_C1, kernel_size=[5, 5], \n",
    "                             padding='same', activation=tf.nn.relu, use_bias=True, \n",
    "                             name='conv_layer_1')\n",
    "\n",
    "    # Pooling Layer #1\n",
    "    pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2, \n",
    "                                    name='pool_layer_1')\n",
    "\n",
    "    # Convolutional Layer #2 \n",
    "    conv2 = tf.layers.conv2d(inputs=pool1, filters=DEPTH_C2, kernel_size=[5, 5], \n",
    "                             padding='same', activation=tf.nn.relu, use_bias=True,\n",
    "                             name='conv_layer_2') # fill in\n",
    "\n",
    "    # Pooling Layer #2\n",
    "    pool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2, \n",
    "                                    name='pool_layer_2') # fill in\n",
    "\n",
    "    # Dense Layer #1\n",
    "    pool2_flat = tf.reshape(pool2, shape=[-1, 7*7*DEPTH_C2], name='pool_layer_2_flat')\n",
    "    dense = tf.layers.dense(inputs=pool2_flat, units=UNITS_D1, activation=tf.nn.relu, \n",
    "                            name='dense_layer_1')\n",
    "    dropout = tf.layers.dropout(inputs=dense, rate=0.4, training=training_mode, name='dropout')\n",
    "\n",
    "    # Dense Layer #2 (Logits Layer)\n",
    "    logits = tf.layers.dense(inputs=dropout, units=NUM_CLASSES, use_bias=True,\n",
    "                             name='dense_layer_2')\n",
    "    \n",
    "    # Predicted labels\n",
    "    predictions = tf.argmax(logits, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question:** What is the shape of the tensor output at each layer? Assume a single training instance is passed through the network. It may be helpful to review the lecture slides describing convolutional and max pooling layers.\n",
    "*(Hint: the `padding='same'` option for `tf.layers.conv2d` adds a border of zeros around the input so that the width/height of the output = width/height of the input.)*\n",
    "\n",
    "**Answer:** \n",
    "* *Output of Convolutional Layer #1: (1, 28, 28, 8).*\n",
    "* *Output of Pooling Layer #1: (1, 14, 14, 8).*\n",
    "* *Output of Convolutional Layer #2: (1, 14, 14, 16).*\n",
    "* *Output of Pooling Layer #2: (1, 7, 7, 16). This is then flattened to (1, 784).*\n",
    "* *Output of Dense Layer #1: (1, 256)*\n",
    "* *Output of Dense Layer #2: (1, 10)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Minimizing the empirical loss\n",
    "To measure the discrepancy between the predicted class distribution and the true labels, we use the softmax cross entropy—the same loss we used for logistic regression in Worksheet 5.\n",
    "\n",
    "Fill in the blank below to calculate the loss from the true labels `Y` and the output of the network `logits`.\n",
    "(Hint: The built-in losses can be found under the [`tf.losses`](https://www.tensorflow.org/api_docs/python/tf/losses) namespace.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.variable_scope('loss'):\n",
    "    loss = tf.losses.sparse_softmax_cross_entropy(labels=Y, logits=logits) # fill in"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To minimize the loss, we'll use the built-in Adam optimizer (it tends to converge more rapidly than gradient descent).\n",
    "Note: we've defined the `global_step` variable to keep track of how many parameter updates have been performed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.variable_scope('train'):\n",
    "    opt = tf.train.AdamOptimizer(learning_rate=0.001)\n",
    "    global_step = tf.Variable(0, name='global_step', trainable=False)\n",
    "    train_op = opt.minimize(loss=loss, global_step=global_step)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Evaluation and TensorBoard summaries\n",
    "As in Worksheet 5, we'll use accuracy to evaluate the CNN.\n",
    "When using the built-in [`tf.metrics.accuracy`](https://www.tensorflow.org/api_docs/python/tf/metrics/accuracy) implementation, `acc_op` must be called to update the accuracy—if `acc` is called then an out-of-date value (computed from internal local variables) may be returned.\n",
    "\n",
    "Since we're going to use TensorBoard to monitor training progress, we need to define some Summary operations (available under the [`tf.summary`](https://www.tensorflow.org/api_docs/python/tf/summary) namespace).\n",
    "Below we define `loss_summary` and `acc_summary` to monitor the loss and accuracy.\n",
    "Then we merge the summaries into a single Summary operation `eval_summaries` (for simplicity)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.variable_scope('evaluation'):\n",
    "    acc, acc_op = tf.metrics.accuracy(labels=Y, predictions=predictions, name='accuracy')\n",
    "    loss_summary = tf.summary.scalar('loss', loss)\n",
    "    acc_summary = tf.summary.scalar('accuracy', acc)\n",
    "    eval_summaries = tf.summary.merge([loss_summary, acc_summary])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'd also like to monitor some of the filters (a.k.a. kernels) in the first convolutional layer of the network. These will show up in the 'Images' tab in TensorBoard.\n",
    "To do this we:\n",
    "* extract the kernel tensor from `conv_layer_1`\n",
    "* rescale the kernel tensor so that all values are on the unit interval\n",
    "* transpose the kernel tensor so that the depth dimension is first\n",
    "* define an image Summary operator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.variable_scope('cnn_model/conv_layer_1', reuse=True):\n",
    "    kernel = tf.get_variable('kernel')\n",
    "    with tf.variable_scope('visualization'):\n",
    "        # scale weights to [0 1]\n",
    "        x_min = tf.reduce_min(kernel)\n",
    "        x_max = tf.reduce_max(kernel)\n",
    "        kernel_0_to_1 = (kernel - x_min) / (x_max - x_min)\n",
    "\n",
    "        # to tf.summary.image format\n",
    "        kernel_transposed = tf.transpose(kernel_0_to_1, [3, 0, 1, 2])\n",
    "\n",
    "        # this will display 5 filters from the 8 in conv_layer_1\n",
    "        filter_summary = tf.summary.image('filters', kernel_transposed, max_outputs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Running TensorBoard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before opening a session to train the CNN, you should start TensorBoard so that you can monitor progress.\n",
    "\n",
    "To do this on the lab machine:\n",
    "\n",
    "1. Start an Anaconda Prompt in the `workshop06` directory and run the command: `python -m tensorboard.main --logdir=mnist_log --host=localhost`\n",
    "2. Navigate to http://localhost:6006/ in your web browser.\n",
    "3. If successful, you should see the following web page. Later on, this will be populated with useful info.\n",
    "\n",
    "![TensorBoard](https://screenshotscdn.firefoxusercontent.com/images/8941dbec-7dfb-4e5a-b015-225345f7615f.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can finally start training the CNN. \n",
    "Below we specify the log directory for TensorBoard and the number of epochs (full sweeps through the training data).\n",
    "You'll soon see that training is slow on the CPU, so we're limited to a small number of epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "LOG_DIR = os.path.join(os.curdir, 'mnist_log')\n",
    "NUM_EPOCHS = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to create an initializer for the global and local variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "init = tf.group(tf.global_variables_initializer(), tf.local_variables_initializer())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And open a session to run operations on the graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting epoch 0.\n",
      "\tTraining accuracy at step 100: 0.8704533576965332.\n",
      "\tTraining accuracy at step 200: 0.9113644957542419.\n",
      "\tTraining accuracy at step 300: 0.929695725440979.\n",
      "\tTraining accuracy at step 400: 0.9406428337097168.\n",
      "\tTraining accuracy at step 500: 0.9478151798248291.\n",
      "\tTraining accuracy at step 600: 0.9531726241111755.\n",
      "Starting epoch 1.\n",
      "\tTraining accuracy at step 700: 0.9572432637214661.\n",
      "\tTraining accuracy at step 800: 0.9604693055152893.\n",
      "\tTraining accuracy at step 900: 0.9631503224372864.\n",
      "\tTraining accuracy at step 1000: 0.9652717113494873.\n",
      "\tTraining accuracy at step 1100: 0.967065691947937.\n",
      "\tTraining accuracy at step 1200: 0.9685727953910828.\n",
      "Starting epoch 2.\n",
      "\tTraining accuracy at step 1300: 0.9699966311454773.\n",
      "\tTraining accuracy at step 1400: 0.9712287187576294.\n",
      "\tTraining accuracy at step 1500: 0.9722719788551331.\n",
      "\tTraining accuracy at step 1600: 0.9732027649879456.\n",
      "\tTraining accuracy at step 1700: 0.97404944896698.\n",
      "\tTraining accuracy at step 1800: 0.9748313426971436.\n",
      "Starting epoch 3.\n",
      "\tTraining accuracy at step 1900: 0.9755547642707825.\n",
      "\tTraining accuracy at step 2000: 0.9762388467788696.\n",
      "\tTraining accuracy at step 2100: 0.9768573045730591.\n",
      "\tTraining accuracy at step 2200: 0.9774211049079895.\n",
      "\tTraining accuracy at step 2300: 0.9779422283172607.\n",
      "\tTraining accuracy at step 2400: 0.9784550666809082.\n",
      "Starting epoch 4.\n",
      "\tTraining accuracy at step 2500: 0.9789355993270874.\n",
      "\tTraining accuracy at step 2600: 0.9793869853019714.\n",
      "\tTraining accuracy at step 2700: 0.9798157811164856.\n",
      "\tTraining accuracy at step 2800: 0.9801973104476929.\n",
      "\tTraining accuracy at step 2900: 0.9805859327316284.\n",
      "\tTraining accuracy at step 3000: 0.9809461236000061.\n",
      "Optimization complete.\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "    # Instantiate writers for TensorBoard (for saving serialized summaries to disk)\n",
    "    train_summary_writer = tf.summary.FileWriter(os.path.join(LOG_DIR, 'train'), sess.graph)\n",
    "    test_summary_writer = tf.summary.FileWriter(os.path.join(LOG_DIR, 'test'), sess.graph)\n",
    "    \n",
    "    # Run optimizer for multiple epochs\n",
    "    for epoch in range(NUM_EPOCHS):\n",
    "        print(\"Starting epoch {}.\".format(epoch))\n",
    "        for X_batch, Y_batch in train_iterator:\n",
    "            # Run a training step\n",
    "            _, step = sess.run([train_op, global_step],\n",
    "                               feed_dict={X: X_batch, Y: Y_batch, training_mode: True})\n",
    "            # Every 100 batches compute the accuracy on the training set and save the filters in the first convolutional layer\n",
    "            if (step % 100 == 0 and step > 0):\n",
    "                train_accuracy, eval_s, filter_s = sess.run([acc_op, eval_summaries, filter_summary], \n",
    "                                  feed_dict={X: images_train, Y: labels_train, training_mode: False})\n",
    "                train_summary_writer.add_summary(eval_s, global_step=step)\n",
    "                train_summary_writer.add_summary(filter_s, global_step=step)\n",
    "                print(\"\\tTraining accuracy at step {}: {}.\".format(step, train_accuracy))\n",
    "            # Every 10 batches compute the accuracy on the test set.\n",
    "            if (step % 10 == 0):\n",
    "                test_accuracy, eval_s = sess.run([acc_op, eval_summaries], \n",
    "                                 feed_dict={X: images_test, Y: labels_test, training_mode: False})\n",
    "                test_summary_writer.add_summary(eval_s, global_step=step)\n",
    "    print(\"Optimization complete.\")\n",
    "    \n",
    "    train_summary_writer.close()\n",
    "    test_summary_writer.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question:** Are 5 training epochs sufficient for this problem?\n",
    "\n",
    "**Answer:** *For this particular run (note that there is variability due to randomisation), the optimiser does not appear to have converged after 5 epochs. \n",
    "Both the loss and accuracy curves in TensorBoard were continuing to improve (albeit at a slow rate) when the optimisation was terminated.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Extension activities\n",
    "* Count the number of scalar parameters in the CNN model. How does this compare to logistic regression (from Worksheet 5)? *\\[Answer: This CNN has 206,954 parameters, whereas logistic regression had only 7,850 parameters.\\]*\n",
    "* Remove dropout from the architecture. What happens to the train/test curves? Does the model now overfit?\n",
    "* Vary `DEPTH_C1`, `DEPTH_C2` and/or `UNITS_D1`. How do these parameters affect the goodness of fit?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}