3_write_convnet_in_chainer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to write ConvNet models in Chainer\n",
    "\n",
    "In this notebook, you will learn how to write\n",
    "\n",
    "- A small convolutional network with a model class that is inherited from Chain,\n",
    "- A large convolutional network that has several building block networks with ChainList.\n",
    "\n",
    "A convolutional network (ConvNet) is mainly comprised of convolutional layers. This type of network is commonly used for various visual recognition tasks, e.g., classifying hand-written digits or natural images into given object classes, detecting objects from an image, and labeling all pixels of an image with the object classes (semantic segmentation), and so on.\n",
    "\n",
    "In such tasks, a typical ConvNet takes a set of images whose shape is $(N,C,H,W)$, where\n",
    "\n",
    "- $N$ denotes the number of images in a mini-batch,\n",
    "- $C$ denotes the number of channels of those images,\n",
    "- $H$ and $W$ denote the height and width of those images,\n",
    "\n",
    "respectively. Then, it typically outputs a fixed-sized vector as membership probabilities over the target object classes. It also can output a set of feature maps that have the corresponding size to the input image for a pixel labeling task, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's import the necessary packages for using Chainer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import chainer\n",
    "from chainer import cuda, Function, gradient_check, report, training, utils, Variable\n",
    "from chainer import datasets, iterators, optimizers, serializers\n",
    "from chainer import Link, Chain, ChainList\n",
    "import chainer.functions as F\n",
    "import chainer.links as L\n",
    "from chainer.training import extensions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LeNet5\n",
    "\n",
    "Here, let’s start by defining LeNet5 [[LeCun98]](#LeCun98) in Chainer. This is a ConvNet model that has 5 layers comprised of 3 convolutional layers and 2 fully-connected layers. This was proposed to classify hand-written digit images in 1998. In Chainer, the model can be written as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class LeNet5(Chain):\n",
    "    def __init__(self):\n",
    "        super(LeNet5, self).__init__()\n",
    "        with self.init_scope():\n",
    "            self.conv1 = L.Convolution2D(\n",
    "                in_channels=1, out_channels=6, ksize=5, stride=1)\n",
    "            self.conv2 = L.Convolution2D(\n",
    "                in_channels=6, out_channels=16, ksize=5, stride=1)\n",
    "            self.conv3 = L.Convolution2D(\n",
    "                in_channels=16, out_channels=120, ksize=4, stride=1)\n",
    "            self.fc4 = L.Linear(None, 84)\n",
    "            self.fc5 = L.Linear(84, 10)\n",
    "\n",
    "    def __call__(self, x):\n",
    "        h = F.sigmoid(self.conv1(x))\n",
    "        h = F.max_pooling_2d(h, 2, 2)\n",
    "        h = F.sigmoid(self.conv2(h))\n",
    "        h = F.max_pooling_2d(h, 2, 2)\n",
    "        h = F.sigmoid(self.conv3(h))\n",
    "        h = F.sigmoid(self.fc4(h))\n",
    "        if chainer.config.train:\n",
    "            return self.fc5(h)\n",
    "        return F.softmax(self.fc5(h))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A typical way to write your network is creating a new class inherited from [Chain](https://docs.chainer.org/en/latest/reference/core/generated/chainer.Chain.html#chainer.Chain) class. When defining your model in this way, typically, all the layers which have trainable parameters are registered to the model by assigning the objects of [Link](https://docs.chainer.org/en/latest/reference/core/generated/chainer.Link.html#chainer.Link) as an attribute.\n",
    "\n",
    "The model class is instantiated before the forward and backward computations. To give input images and label vectors simply by calling the model object like a function, `__call__()` is usually defined in the model class. This method performs the forward computation of the model. Chainer uses the powerful autograd system for any computational graphs written with [Function](https://docs.chainer.org/en/latest/reference/core/generated/chainer.Function.html#chainer.Function)s and [Link](https://docs.chainer.org/en/latest/reference/core/generated/chainer.Link.html#chainer.Link)s (actually a [Link](https://docs.chainer.org/en/latest/reference/core/generated/chainer.Link.html#chainer.Link) calls a corresponding [Function](https://docs.chainer.org/en/latest/reference/core/generated/chainer.Function.html#chainer.Function) inside of it), so that you don’t need to explicitly write the code for backward computations in the model. Just prepare the data, then give it to the model. The way this works is the resulting output [Variable](https://docs.chainer.org/en/latest/reference/core/generated/chainer.Variable.html#chainer.Variable) from the forward computation has a [backward()](https://docs.chainer.org/en/latest/reference/core/generated/chainer.Variable.html#chainer.Variable.backward) method to perform autograd. In the above model, `__call__()` has a `if` statement at the end to switch its behavior by the Chainer’s running mode, i.e., training mode or not. Chainer presents the running mode as a global variable `chainer.config.train`. When it’s in training mode, `__call__()` returns the output value of the last layer as is to compute the loss later on, otherwise it returns a prediction result by calculating [softmax()](https://docs.chainer.org/en/latest/reference/generated/chainer.functions.softmax.html#chainer.functions.softmax)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ways to calculate loss\n",
    "\n",
    "When you train the model with label vector `t`, the loss should be calculated using the output from the model. There also are several ways to calculate the loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = LeNet5()\n",
    "\n",
    "# Input data and label\n",
    "x = np.random.rand(32, 1, 28, 28).astype(np.float32)\n",
    "t = np.random.randint(0, 10, size=(32,)).astype(np.int32)\n",
    "\n",
    "# Forward computation\n",
    "y = model(x)\n",
    "\n",
    "# Loss calculation\n",
    "loss = F.softmax_cross_entropy(y, t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a primitive way to calculate a loss value from the output of the model. On the other hand, the loss computation can be included in the model itself by wrapping the model object ([Chain](https://docs.chainer.org/en/latest/reference/core/generated/chainer.Chain.html#chainer.Chain) or [ChainList](https://docs.chainer.org/en/latest/reference/core/generated/chainer.ChainList.html#chainer.ChainList) object) with a class inherited from Chain. The outer [Chain](https://docs.chainer.org/en/latest/reference/core/generated/chainer.Chain.html#chainer.Chain) should take the model defined above and register it with [init_scope()](https://docs.chainer.org/en/latest/reference/core/generated/chainer.Chain.html#chainer.Chain.init_scope). [Chain](https://docs.chainer.org/en/latest/reference/core/generated/chainer.Chain.html#chainer.Chain) is actually inherited from Link, so that [Chain](https://docs.chainer.org/en/latest/reference/core/generated/chainer.Chain.html#chainer.Chain) itself can also be registered as a trainable [Link](https://docs.chainer.org/en/latest/reference/core/generated/chainer.Link.html#chainer.Link) to another Chain. Actually, [Classifier](https://docs.chainer.org/en/latest/reference/generated/chainer.links.Classifier.html#chainer.links.Classifier) class to wrap the model and add the loss computation to the model already exists. Actually, there is already a [Classifier](https://docs.chainer.org/en/latest/reference/generated/chainer.links.Classifier.html#chainer.links.Classifier) class that can be used to wrap the model and include the loss computation as well. It can be used like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = L.Classifier(LeNet5())\n",
    "\n",
    "# Foward & Loss calculation\n",
    "loss = model(x, t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This class takes a model object as an input argument and registers it to a `predictor` property as a trained parameter. As shown above, the returned object can then be called like a function in which we pass `x` and `t` as the input arguments and the resulting loss value (which we recall is a [Variable](https://docs.chainer.org/en/latest/reference/core/generated/chainer.Variable.html#chainer.Variable)) is returned.\n",
    "\n",
    "See the detailed implementation of [Classifier](https://docs.chainer.org/en/latest/reference/generated/chainer.links.Classifier.html#chainer.links.Classifier) from here: [chainer.links.Classifier](https://docs.chainer.org/en/latest/reference/generated/chainer.links.Classifier.html#chainer.links.Classifier) and check the implementation by looking at the source.\n",
    "\n",
    "From the above examples, we can see that Chainer provides the flexibility to write our original network in many different ways. Such flexibility intends to make it intuitive for users to design new and complex models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# VGG16\n",
    "\n",
    "Next, let’s write some larger models in Chainer. When you write a large network consisting of several building block networks, [ChainList](https://docs.chainer.org/en/latest/reference/core/generated/chainer.ChainList.html#chainer.ChainList) is useful. First, let’s see how to write a VGG16 [[Simonyan14]](#Simonyan14) model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class VGG16(chainer.ChainList):\n",
    "\n",
    "    def __init__(self):\n",
    "        w = chainer.initializers.HeNormal()\n",
    "        super(VGG16, self).__init__(\n",
    "            VGGBlock(64),\n",
    "            VGGBlock(128),\n",
    "            VGGBlock(256, 3),\n",
    "            VGGBlock(512, 3),\n",
    "            VGGBlock(512, 3, True))\n",
    "\n",
    "    def __call__(self, x):\n",
    "        for f in self.children():\n",
    "            x = f(x)\n",
    "        if chainer.config.train:\n",
    "            return x\n",
    "        return F.softmax(x)\n",
    "\n",
    "\n",
    "class VGGBlock(chainer.Chain):\n",
    "    \n",
    "    def __init__(self, n_channels, n_convs=2, fc=False):\n",
    "        w = chainer.initializers.HeNormal()\n",
    "        super(VGGBlock, self).__init__()\n",
    "        with self.init_scope():\n",
    "            self.conv1 = L.Convolution2D(None, n_channels, 3, 1, 1, initialW=w)\n",
    "            self.conv2 = L.Convolution2D(\n",
    "                n_channels, n_channels, 3, 1, 1, initialW=w)\n",
    "            if n_convs == 3:\n",
    "                self.conv3 = L.Convolution2D(\n",
    "                    n_channels, n_channels, 3, 1, 1, initialW=w)\n",
    "            if fc:\n",
    "                self.fc4 = L.Linear(None, 4096, initialW=w)\n",
    "                self.fc5 = L.Linear(4096, 4096, initialW=w)\n",
    "                self.fc6 = L.Linear(4096, 1000, initialW=w)\n",
    "\n",
    "        self.n_convs = n_convs\n",
    "        self.fc = fc\n",
    "\n",
    "    def __call__(self, x):\n",
    "        h = F.relu(self.conv1(x))\n",
    "        h = F.relu(self.conv2(h))\n",
    "        if self.n_convs == 3:\n",
    "            h = F.relu(self.conv3(h))\n",
    "        h = F.max_pooling_2d(h, 2, 2)\n",
    "        if self.fc:\n",
    "            h = F.dropout(F.relu(self.fc4(h)))\n",
    "            h = F.dropout(F.relu(self.fc5(h)))\n",
    "            h = self.fc6(h)\n",
    "        return h"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That’s it. VGG16 is a model which won the 1st place in [classification + localization task at ILSVRC 2014](http://www.image-net.org/challenges/LSVRC/2014/results#clsloc), and since then, has become one of the standard models for many different tasks as a pre-trained model. This has 16-layers, so it’s called “VGG-16”, but we can write this model without writing all layers independently. Since this model consists of several building blocks that have the same architecture, we can build the whole network by re-using the building block definition. Each part of the network is consisted of 2 or 3 convolutional layers and activation function ([relu()](https://docs.chainer.org/en/latest/reference/generated/chainer.functions.relu.html#chainer.functions.relu)) following them, and [max_pooling_2d()](https://docs.chainer.org/en/latest/reference/generated/chainer.functions.max_pooling_2d.html#chainer.functions.max_pooling_2d) operations. This block is written as VGGBlock in the above example code. And the whole network just calls this block one by one in sequential manner."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ResNet152\n",
    "\n",
    "How about ResNet? ResNet [[He16]](#He16) came in the following year’s ILSVRC. It is a much deeper model than VGG16, having up to 152 layers. This sounds super laborious to build, but it can be implemented in almost same manner as VGG16. In the other words, it’s easy. One possible way to write ResNet-152 is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class ResNet152(chainer.Chain):\n",
    "    \n",
    "    def __init__(self, n_blocks=[3, 8, 36, 3]):\n",
    "        w = chainer.initializers.HeNormal()\n",
    "        super(ResNet152, self).__init__(\n",
    "            conv1=L.Convolution2D(\n",
    "                None, 64, 7, 2, 3, initialW=w, nobias=True),\n",
    "            bn1=L.BatchNormalization(64),\n",
    "            res2=ResBlock(n_blocks[0], 64, 64, 256, 1),\n",
    "            res3=ResBlock(n_blocks[1], 256, 128, 512),\n",
    "            res4=ResBlock(n_blocks[2], 512, 256, 1024),\n",
    "            res5=ResBlock(n_blocks[3], 1024, 512, 2048),\n",
    "            fc6=L.Linear(2048, 1000))\n",
    "\n",
    "    def __call__(self, x):\n",
    "        h = self.bn1(self.conv1(x))\n",
    "        h = F.max_pooling_2d(F.relu(h), 2, 2)\n",
    "        h = self.res2(h)\n",
    "        h = self.res3(h)\n",
    "        h = self.res4(h)\n",
    "        h = self.res5(h)\n",
    "        h = F.average_pooling_2d(h, h.shape[2:], stride=1)\n",
    "        h = self.fc6(h)\n",
    "        if chainer.config.train:\n",
    "            return h\n",
    "        return F.softmax(h)\n",
    "\n",
    "\n",
    "class ResBlock(chainer.ChainList):\n",
    "    \n",
    "    def __init__(self, n_layers, n_in, n_mid, n_out, stride=2):\n",
    "        w = chainer.initializers.HeNormal()\n",
    "        super(ResBlock, self).__init__()\n",
    "        self.add_link(BottleNeck(n_in, n_mid, n_out, stride, True))\n",
    "        for _ in range(n_layers - 1):\n",
    "            self.add_link(BottleNeck(n_out, n_mid, n_out))\n",
    "\n",
    "    def __call__(self, x):\n",
    "        for f in self.children():\n",
    "            x = f(x)\n",
    "        return x\n",
    "\n",
    "\n",
    "class BottleNeck(chainer.Chain):\n",
    "    \n",
    "    def __init__(self, n_in, n_mid, n_out, stride=1, proj=False):\n",
    "        w = chainer.initializers.HeNormal()\n",
    "        super(BottleNeck, self).__init__()\n",
    "        with self.init_scope():\n",
    "            self.conv1x1a = L.Convolution2D(\n",
    "                n_in, n_mid, 1, stride, 0, initialW=w, nobias=True)\n",
    "            self.conv3x3b = L.Convolution2D(\n",
    "                n_mid, n_mid, 3, 1, 1, initialW=w, nobias=True)\n",
    "            self.conv1x1c = L.Convolution2D(\n",
    "                n_mid, n_out, 1, 1, 0, initialW=w, nobias=True)\n",
    "            self.bn_a = L.BatchNormalization(n_mid)\n",
    "            self.bn_b = L.BatchNormalization(n_mid)\n",
    "            self.bn_c = L.BatchNormalization(n_out)\n",
    "            if proj:\n",
    "                self.conv1x1r = L.Convolution2D(\n",
    "                    n_in, n_out, 1, stride, 0, initialW=w, nobias=True)\n",
    "                self.bn_r = L.BatchNormalization(n_out)\n",
    "        self.proj = proj\n",
    "\n",
    "    def __call__(self, x):\n",
    "        h = F.relu(self.bn_a(self.conv1x1a(x)))\n",
    "        h = F.relu(self.bn_b(self.conv3x3b(h)))\n",
    "        h = self.bn_c(self.conv1x1c(h))\n",
    "        if self.proj:\n",
    "            x = self.bn_r(self.conv1x1r(x))\n",
    "        return F.relu(h + x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the BottleNeck class, depending on the value of the proj argument supplied to the initializer, it will conditionally compute a convolutional layer `conv1x1r` which will extend the number of channels of the input `x` to be equal to the number of channels of the output of `conv1x1c`, and followed by a batch normalization layer before the final ReLU layer. Writing the building block in this way improves the re-usability of a class. It switches not only the behavior in `__class__()` by flags but also the parameter registration. In this case, when `proj` is `False`, the `BottleNeck` doesn’t have `conv1x1r` and `bn_r` layers, so the memory usage would be efficient compared to the case when it registers both anyway and just ignore them if `proj` is `False`.\n",
    "\n",
    "Using nested `Chain`s and `ChainList` for sequential part enables us to write complex and very deep models easily."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Use Pre-trained Models\n",
    "\n",
    "Various ways to write your models were described above. It turns out that VGG16 and ResNet are very useful as general feature extractors for many kinds of tasks, including but not limited to image classification. So, Chainer provides you with the pre-trained VGG16 and ResNet-50/101/152 models with a simple API. You can use these models as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading from http://www.robots.ox.ac.uk/%7Evgg/software/very_deep/caffe/VGG_ILSVRC_16_layers.caffemodel...\n",
      "Now loading caffemodel (usually it may take few minutes)\n"
     ]
    }
   ],
   "source": [
    "from chainer.links import VGG16Layers\n",
    "\n",
    "model = VGG16Layers()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When [VGG16Layers](https://docs.chainer.org/en/latest/reference/generated/chainer.links.VGG16Layers.html#chainer.links.VGG16Layers) is instantiated, the pre-trained parameters are automatically downloaded from the author’s server. So you can immediately start to use VGG16 with pre-trained weight as a good image feature extractor. See the details of this model here: [chainer.links.VGG16Layers](https://docs.chainer.org/en/latest/reference/generated/chainer.links.VGG16Layers.html#chainer.links.VGG16Layers)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the case of ResNet models, there are three variations differing in the number of layers. We have `chainer.links.ResNet50`, `chainer.links.ResNet101`, and `chainer.links.ResNet152` models with easy parameter loading feature. ResNet’s pre-trained parameters are not available for direct downloading, so you need to download the weight from the author’s web page first, and then place it into the dir `$CHAINER_DATSET_ROOT/pfnet/chainer/models` or your favorite place. Once the preparation is finished, the usage is the same as VGG16:\n",
    "\n",
    "```\n",
    "from chainer.links import ResNet152Layers\n",
    "\n",
    "model = ResNet152layers()\n",
    "```\n",
    "\n",
    "Please see the details of usage and how to prepare the pre-trained weights for ResNet here: [chainer.links.ResNet50](https://docs.chainer.org/en/latest/reference/generated/chainer.links.ResNet50Layers.html#chainer.links.ResNet50Layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References\n",
    "\n",
    "<a id=\"LeCun98\"></a>\n",
    "[LeCun98]\tYann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278–2324, 1998.\n",
    "\n",
    "<a id=\"Simonyan14\"></a>\n",
    "[Simonyan14]\tSimonyan, K. and Zisserman, A., Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv preprint arXiv:1409.1556, 2014.\n",
    "\n",
    "<a id=\"He16\"></a>\n",
    "[He16]\tKaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun. Deep Residual Learning for Image Recognition. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.\n",
    "Next  Previous\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}