data-augmentation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Augmentation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Imports as usual\n",
    "library(keras)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deep neural networks, or even shallow neural networks require a lot (thousands, hundred of thousands of exampels) of training data to achieve good results. What must one do if there is not enough data? Think of a data set of digits whereby the task is to develop a model that can recognise digits. Now think of a hand written number on paper, say number 3. Now image that same 3 rotated by a tiny bit, it still looks like a 3. In this tutorial we will cover simple data augmentation techniques that will result in a lot more data for training. Here are some examples of images that were generated automatically, what do you think of them? Do you think that having more images like this in a small dataset could add value?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Here is a Crab](img/aug_1.jpeg)\n",
    "![Here is a Crab](img/aug_2.jpeg)\n",
    "![Here is a Crab](img/aug_3.jpeg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Load Keras' MNIST data\n",
    "mnist<- dataset_mnist()\n",
    "\n",
    "# Read the training data\n",
    "x_train <- mnist$train$x\n",
    "y_train <- mnist$train$y\n",
    "\n",
    "# Read the test data\n",
    "x_test <- mnist$test$x\n",
    "y_test <- mnist$test$y\n",
    "\n",
    "# Convert the labels into their one-hot encoded equivalents\n",
    "# MNIST has 10 classes\n",
    "y_train_hot<-to_categorical(y_train,num_classes = 10)\n",
    "y_test_hot<-to_categorical(y_test,num_classes=10) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Just for now, let's only use a small subset of the data by selecting 10 examples from the data.Since MNIST is greyscale, we need to add a \"1\" to the number of channels. If it were a colour dataset then the colour channel would be a \"3\" instead. In order to use the built in data augmentation function, our data has to be of rank 4 (i.e. 4 dimensions). (number of examples, width, height, channels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x_train_small <- x_train[1:10,,]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a look at the dimensions of our sampled dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dim(x_train_small)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MNIST is already by default in (60000, 28,28) but you can see that it is missing one more dimension\n",
    "and we fix this below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dim(x_train_small) <- c(nrow(x_train_small), 28, 28, 1) \n",
    "dim(x_train_small)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can you notice the difference between the dimension before and after?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Select a subset of training labels\n",
    "y_train_small <- y_train_hot[1:10,]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Select a subset of test examples\n",
    "x_test_small <- x_test[1:10,,]\n",
    "\n",
    "# Add greyscale channel value of 1\n",
    "dim(x_test_small) <- c(nrow(x_test_small), 28, 28, 1) \n",
    "\n",
    "# Check dimensions\n",
    "dim(x_test_small)\n",
    "\n",
    "# Select a subset of test labels\n",
    "y_test_small <- y_test_hot[1:10,]\n",
    "\n",
    "# Check dimensions\n",
    "dim(y_test_small)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build a neural network model. In this case a CNN for MNIST so the input shape must be 28,28,1 (1 for greyscale)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Define a sequential model\n",
    "model<-keras_model_sequential()\n",
    "\n",
    "# Create the network architecture\n",
    "model %>%\n",
    "  layer_conv_2d(filters = 32,                      # number of convolution filters in conv layer 1\n",
    "                kernel_size = c(3,3),              # use 3 x 3 convolution filter in conv layer 1\n",
    "                input_shape = c(28, 28, 1)) %>%    # shape of input data\n",
    "  layer_activation('relu') %>%                     # activation function in conv layer 1\n",
    "  layer_dropout(rate = 0.20) %>%                   # apply 20% dropout after conv layer 1\n",
    "  layer_conv_2d(filters = 64,                      # number of convolution filters in conv layer 2\n",
    "                kernel_size = c(3,3)) %>%          # also use 3 x 3 filter in conv layer 2\n",
    "  layer_activation('relu') %>%                     # activation function in conv layer 2\n",
    "  layer_max_pooling_2d(pool_size = c(2, 2)) %>%    # apply max pooling after conv layer 2\n",
    "  layer_flatten() %>%                              # flatten output into a vector\n",
    "  layer_dense(units = 10, activation = 'softmax')  # fully connected to output layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compile the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model %>% compile(\n",
    "  loss = 'categorical_crossentropy',\n",
    "  optimizer = 'rmsprop',\n",
    "  metrics = c('accuracy')\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create data generator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we use the image_data_generator() function to create additional images. This function can flip, shift and rotate images. Read more here https://tensorflow.rstudio.com/keras/reference/image_data_generator.html In this dataset, should horizontal_flip be set to TRUE or FALSE?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "gen_images <- image_data_generator(featurewise_center = TRUE,\n",
    "                                   featurewise_std_normalization = TRUE,\n",
    "                                   rotation_range = 10,\n",
    "                                   width_shift_range = 0.30,\n",
    "                                   height_shift_range = 0.30,\n",
    "                                   horizontal_flip = FALSE  )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we take some images from the dataset and use this to fit some parameters in the image generator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "gen_images %>% fit_image_data_generator(x_train_small)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the following snippet of code, we can generate some images and then save those images to the hard drive. Note that the code uses gen_images which we defined just above. The batch size represents the number of images which will be generated, in this case, 9."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "images_iter <- flow_images_from_data(\n",
    "  x=x_train_small, y=y_train_small,\n",
    "  generator=gen_images,\n",
    "  batch_size=9,\n",
    "  save_to_dir='data/Images/', # create this folder if it doesn't exist\n",
    "  save_prefix=\"aug\",\n",
    "  save_format=\"jpeg\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're now ready to generate images and the model together. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model %>% fit_generator(\n",
    "  images_iter,\n",
    "  steps_per_epoch = 1, epochs = 1,\n",
    "  validation_data = list(x_test_small, y_test_small) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we evaluate on the test set. To be correct, we should evaluate on the validation set and then once happy with the results apply the best model to the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model %>% evaluate(x_test_small, y_test_small, batch_size=32, verbose = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do these results mean? Nice explanation online: https://stackoverflow.com/questions/34518656/how-to-interpret-loss-and-accuracy-for-a-machine-learning-model\n",
    "\n",
    "Of course the results are bad because we used a tiny subset of the data. Try use the whole MNIST training data with a large batch size for the image generator and see how the results change. Is there an improvement?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## --------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## This is the start of the second part.\n",
    "## Data augmentation on structured folder data\n",
    "It's possible to start directly from this point and skip everything above. <b> In which case, remember to import keras! </b>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial assumes that you can load your data directly and easily into x_train, y_train, x_test and y_test. Sometimes this isn't the case if preprocessing steps are hard. One way to overcome this is to organise your images into folders in your hard drive and to use a slight variation in the code.\n",
    "\n",
    "\n",
    "Now let's apply data augmentation to an invasive species image dataset. Data source: https://www.kaggle.com/c/invasive-species-monitoring/data and also see some of the code [here](https://www.kaggle.com/ogurtsov/0-99-with-r-and-keras-inception-v3-fine-tune)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Specify the folder locations for the training, validation and test data\n",
    "train_directory <- \"data/invasives/sample/train/\"\n",
    "validation_directory <- \"data/invasives/sample/validation/\"\n",
    "test_directory <- \"data/invasives/sample/test/\"\n",
    "\n",
    "# once you are satisfied the code is working, run full dataset. Remove the # symbols to uncomment the code\n",
    "# train_directory <- \"data/invasives/train/\"\n",
    "# validation_directory <- \"data/invasives/validation/\"\n",
    "# test_directory <- \"data/invasives/test/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And work out how many images we have."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Count the training examples\n",
    "train_samples <- length(list.files(paste(train_directory,\"invasive\",sep=\"\"))) +\n",
    "    length(list.files(paste(train_directory,\"non_invasive\",sep=\"\")))\n",
    "\n",
    "# Count the validation examples\n",
    "validation_samples <- length(list.files(paste(validation_directory,\"invasive\",sep=\"\"))) +\n",
    "    length(list.files(paste(validation_directory,\"non_invasive\",sep=\"\")))\n",
    "\n",
    "# Count the test examples\n",
    "test_samples <- length(list.files(paste(test_directory,\"invasive\",sep=\"\"))) +\n",
    "    length(list.files(paste(test_directory,\"non_invasive\",sep=\"\")))\n",
    "\n",
    "train_samples\n",
    "validation_samples\n",
    "test_samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Specify the image dimension and batch size. We need to specify this so that we can generate new images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "img_height <- 224\n",
    "img_width <- 224\n",
    "batch_size <- 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data generator\n",
    "Here we define the data generator. We choose a small rotation and no horizontal flipping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "datagen_invasive <- image_data_generator(featurewise_center = TRUE,\n",
    "    rotation_range = 1,\n",
    "    width_shift_range = 0.05,\n",
    "    height_shift_range = 0.05,\n",
    "    horizontal_flip = FALSE\n",
    "  )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train/validation/test generators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is the code on how to train the model on the invasive species data. We will create another generator so as not to keep saving images to the disk. If we use the one above, \"train_generator_invasive\" we will end up saving a lot of images to disk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_generator_invasive <- flow_images_from_directory(\n",
    "    train_directory, \n",
    "    generator = datagen_invasive,\n",
    "    target_size = c(img_height, img_width),\n",
    "    color_mode = \"rgb\",\n",
    "    class_mode = \"binary\", \n",
    "    classes = c('non_invasive', 'invasive'),\n",
    "    batch_size = batch_size,\n",
    "    shuffle = TRUE,\n",
    "    seed = 123)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need a validation generator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "validation_generator <- flow_images_from_directory(\n",
    "    validation_directory, \n",
    "    generator = datagen_invasive,\n",
    "    target_size = c(img_height, img_width),\n",
    "    color_mode = \"rgb\",\n",
    "    class_mode = \"binary\", \n",
    "    batch_size = batch_size, \n",
    "    shuffle = TRUE,\n",
    "    seed = 123)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And also a test generator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_generator <- flow_images_from_directory(\n",
    "  test_directory, \n",
    "  generator = image_data_generator(),\n",
    "  target_size = c(img_height, img_width), \n",
    "  color_mode = \"rgb\", \n",
    "  class_mode = \"binary\", \n",
    "  batch_size = 1,\n",
    "  shuffle = FALSE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model<-keras_model_sequential()\n",
    "\n",
    "model %>%\n",
    "  layer_conv_2d(filters = 32,                      # number of convolution filters in conv layer 1\n",
    "                kernel_size = c(3,3),              # use 3 x 3 convolution filter in conv layer 1\n",
    "                input_shape = c(img_height, img_width, 3)) %>%    # shape of input data\n",
    "  layer_activation('relu') %>%                     # activation function in conv layer 1\n",
    "  layer_dropout(rate = 0.20) %>%                   # apply 20% dropout after conv layer 1\n",
    "  layer_max_pooling_2d(pool_size = c(2, 2)) %>%    # apply max pooling after conv layer 2\n",
    "  layer_flatten() %>%                              # flatten output into a vector\n",
    "  layer_dense(units = 64, activation = \"relu\") %>% \n",
    "  layer_dense(units = 1, activation = \"sigmoid\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print out a summary of the network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "summary(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compile the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model %>% compile(\n",
    "  loss = \"binary_crossentropy\",\n",
    "  optimizer = optimizer_sgd(lr = 0.0001, \n",
    "                            momentum = 0.9, \n",
    "                            decay = 1e-5),\n",
    "  metrics = \"accuracy\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model %>% fit_generator(\n",
    "  train_generator_invasive,\n",
    "  steps_per_epoch = 1, # can have this as 1 due to small dataset, else see previous notebooks \n",
    "  epochs = 1, \n",
    "  validation_data = validation_generator,\n",
    "  validation_steps = 1,\n",
    "  verbose = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model %>% evaluate_generator(\n",
    "    test_generator,\n",
    "    steps = test_samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "In summary:\n",
    "\n",
    "Read in the data.\n",
    "\n",
    "Create a data generator and specify the rotation, flipping parameters.\n",
    "\n",
    "If you are using data which are already in the x and y variables then use the flow_images_from_data() like in the first example on MNIST.\n",
    "\n",
    "Otherwise, if you're using a structured folder, use flow_images_from_directory() like in the invasive species dataset. In this case also remember to create a generator for the validation and test data."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.4.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}