Text_Classification_CNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Classification Using CNN - Extension\n",
    "\n",
    "## NOTE:\n",
    "- Since we are using NLTK here, it would be advisable to download ALL NLTK corpora/samples\n",
    "- This code uses the **Low-Level Symbol API** - the code therefore tends to be messy\n",
    "- Default models (in part 1):\n",
    "    - Do not include dropout layers\n",
    "    - Do not do learning rate decay\n",
    "- This code is adapted and extended from [MXNet's Tutorial](https://mxnet.incubator.apache.org/versions/master/tutorials/nlp/cnn.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# these are the modules you are allowed to work with. \n",
    "\n",
    "import nltk\n",
    "from nltk import word_tokenize\n",
    "import re\n",
    "import numpy as np\n",
    "import mxnet as mx\n",
    "import sys, os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-processing functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "208 1298\n"
     ]
    }
   ],
   "source": [
    "'''\n",
    "First job is to clean and preprocess the social media text. (5)\n",
    "\n",
    "1) Replacing URLs by  \"_URL_\" and mentions (i.e strings which are preceeded with @) by \"_MENT_\"\n",
    "2) Segment #hastags \n",
    "3) Remove emoticons and other unicode characters\n",
    "'''\n",
    "\n",
    "def preprocess_tweet(string):\n",
    "    '''\n",
    "    Input: The input string read directly from the file\n",
    "    \n",
    "    Output: Pre-processed tweet text\n",
    "    '''\n",
    "    string = re.sub(r\"((http:\\/\\/)?www\\.|(https:\\/\\/)?www\\.|http:\\/\\/|https:\\/\\/)+[^\\s]+\", \"_URL_\", string) #replacing URLs\n",
    "    string = re.sub(r\"(@)+[^\\s]+\", \"_MENT_\", string) #replacing Mentions\n",
    "    #search with hash-tags\n",
    "    hash_tag_p = re.compile(r\"(#)+[^\\s]+\")\n",
    "    matches={}\n",
    "    for x in re.finditer(hash_tag_p,string):\n",
    "        tag=x.group()\n",
    "        new_tag=tag.strip('#')\n",
    "        new_tag=\" \".join([match.group() for match in re.finditer(\"[A-Z][a-z]*[^(A-Z)]\",new_tag)])\n",
    "        string=string.replace(tag,new_tag)\n",
    "    #removing emoticons\n",
    "    emoji_pattern = re.compile(\"[\"\n",
    "        u\"\\U0001F600-\\U0001F64F\"  # emoticons\n",
    "                           \"]+\", flags=re.UNICODE)\n",
    "    emoji_pattern.sub(\"\",string)\n",
    "    #removing unicode characters\n",
    "    string = string.encode('ascii', 'ignore').decode(\"utf-8\")    \n",
    "    \n",
    "    string = re.sub(r\"[^A-Za-z]\", \" \", string) #removing punctuations & numerical values\n",
    "    string = string.strip().lower()\n",
    "    return string\n",
    "\n",
    "\n",
    "# reading the input file and separating the set of positive examples and negative examples. \n",
    "\n",
    "file=open('cancer_data.tsv','r')\n",
    "pos_data=[]\n",
    "neg_data=[]\n",
    "\n",
    "for line in file:\n",
    "    line = line.strip().split('\\t')\n",
    "    text2 = preprocess_tweet(line[0]).strip().split()\n",
    "    if line[1] == 'yes':\n",
    "        pos_data.append(text2)\n",
    "    if line[1] == 'no':\n",
    "        neg_data.append(text2)\n",
    "\n",
    "print(len(pos_data), len(neg_data))     \n",
    "\n",
    "sentences= list(pos_data)\n",
    "sentences.extend(neg_data)\n",
    "pos_labels= [1 for _ in pos_data]\n",
    "neg_labels= [0 for _ in neg_data]\n",
    "y=list(pos_labels)\n",
    "y.extend(neg_labels)\n",
    "y=np.array(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "**After this we obtain the following :**\n",
    "\n",
    "1. sentences =  List of sentences having the positive and negative examples with all the positive examples first\n",
    "2. y = List of labels with the positive labels first.\n",
    "\n",
    "Before running the CNN there are a few things one needs to take care of: (5)\n",
    "\n",
    "1. Pad the sentences so that all of them are of the same length\n",
    "2. Build a vocabulary comprising all unique words that occur in the corpus\n",
    "3. Convert each sentence into a corresponding vector by replacing each word in the sentence with the index in the vocabulary. \n",
    "\n",
    "Example :\n",
    "S1 = a b a c\n",
    "S2 = d c a \n",
    "\n",
    "Step 1:  S1= a b a c, \n",
    "         S2 =d c a </s> \n",
    "         (Both sentences are of equal length). \n",
    "\n",
    "Step 2:  voc={a:1, b:2, c:3, d:4, </s>: 5}\n",
    "\n",
    "Step 3:  S1= [1,2,1,3]\n",
    "         S2= [4,3,1,5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pad_sentences(sentences, padding_word=\"</s>\"):\n",
    "    \"\"\"\n",
    "    Pads all sentences to be the length of the longest sentence.\n",
    "    Returns padded sentences.\n",
    "    \"\"\"\n",
    "    sequence_length = max(len(x) for x in sentences)\n",
    "    padded_sentences = []\n",
    "    for i in range(len(sentences)):\n",
    "        sentence = sentences[i]\n",
    "        num_padding = sequence_length - len(sentence)\n",
    "        new_sentence = sentence + [padding_word] * num_padding\n",
    "        padded_sentences.append(new_sentence)\n",
    "\n",
    "    #still, a list of strings(each string is a \"sentence\")\n",
    "    return padded_sentences\n",
    "\n",
    "\n",
    "\n",
    "def build_vocab(sentences):\n",
    "    \"\"\"\n",
    "    Builds a vocabulary mapping from token to index based on the sentences.\n",
    "    Returns vocabulary mapping and inverse vocabulary mapping.\n",
    "    \"\"\"\n",
    "    tokens=[]\n",
    "    # Mapping from index to word\n",
    "    for sent in sentences:\n",
    "        tokens.extend(sent)\n",
    "    token_counts_dict={}\n",
    "    for word in tokens:\n",
    "        if word not in token_counts_dict:\n",
    "            token_counts_dict[word]=tokens.count(word)\n",
    "        else:\n",
    "            continue\n",
    "    \n",
    "    vocabulary_inv=[]\n",
    "    \n",
    "    #iterates in decreasing order by keys\n",
    "    for k,v in token_counts_dict.items():\n",
    "        vocabulary_inv.append(k)\n",
    "        \n",
    "    # Mapping from word to index\n",
    "    vocabulary = {vocabulary_inv[i]: i for i in range(len(vocabulary_inv))}\n",
    "    \n",
    "    return vocabulary, vocabulary_inv\n",
    "\n",
    "\n",
    "def create_word_vectors(sentences):\n",
    "    '''\n",
    "    Input: List of sentences\n",
    "    Output: List of word vectors corresponding to each sentence, vocabulary\n",
    "    '''\n",
    "    sentences=pad_sentences(sentences)\n",
    "    vocab,inv_vocab=build_vocab(sentences)\n",
    "    sent_vectors = np.array([\n",
    "            [vocab[word] for word in sentence]\n",
    "            for sentence in sentences])\n",
    "    return sent_vectors, vocab\n",
    "\n",
    "\n",
    "def create_shuffle(x,y):\n",
    "    '''\n",
    "    Create an equal distribution of the positive and negative examples. \n",
    "    Please do not change this particular shuffling method.\n",
    "    '''\n",
    "    pos_len= len(pos_data)\n",
    "    neg_len= len(neg_data)\n",
    "    # ensuring proper mix of +ve & -ve samples\n",
    "    pos_len_train= int(0.8*pos_len)\n",
    "    neg_len_train= int(0.8*neg_len)\n",
    "    train_data= [(x[i],y[i]) for i in range(0, pos_len_train)]\n",
    "    train_data.extend([(x[i],y[i]) for i in range(pos_len, pos_len+ neg_len_train )])\n",
    "    test_data=[(x[i],y[i]) for i in range(pos_len_train, pos_len)]\n",
    "    test_data.extend([(x[i],y[i]) for i in range(pos_len+ neg_len_train, len(x) )])\n",
    "\n",
    "    np.random.shuffle(train_data)\n",
    "    x_train=[i[0] for i in train_data]\n",
    "    y_train=[i[1] for i in train_data]\n",
    "    np.random.shuffle(test_data)\n",
    "    x_test=[i[0] for i in test_data]\n",
    "    y_test=[i[1] for i in test_data]\n",
    "    \n",
    "    x_train=np.array(x_train)\n",
    "    y_train=np.array(y_train)\n",
    "    x_test= np.array(x_test)\n",
    "    y_test= np.array(y_test)\n",
    "    return x_train, y_train, x_test, y_test\n",
    "# obtaining the data\n",
    "x,vocab=create_word_vectors(sentences)\n",
    "x_train, y_train, x_test, y_test= create_shuffle(x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'divide': 'warn', 'over': 'warn', 'under': 'ignore', 'invalid': 'warn'}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# default parameters\n",
    "n_epochs=10\n",
    "batch_size=20\n",
    "size_embed=200\n",
    "num_filter=100\n",
    "vocab_size=len(vocab)\n",
    "sentence_len=x_train.shape[1]\n",
    "filters=[2,3,4,5]\n",
    "learn_rate=0.005\n",
    "np.seterr(all='ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "***\n",
    "\n",
    "### Declaring and visualizing the graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input shape:  (1204, 112)\n",
      "Shape of embed layer o/p:  [(1204, 112, 200)]\n"
     ]
    }
   ],
   "source": [
    "#placeholders\n",
    "input_x = mx.sym.Variable('data')\n",
    "input_y = mx.sym.Variable('softmax_label')\n",
    "print (\"input shape: \",x_train.shape)\n",
    "\n",
    "#embeddings layer - for converting to n-dimension vector representations while training\n",
    "embed = mx.sym.Embedding(data=input_x, input_dim=vocab_size, output_dim=size_embed, name='embed_layer')\n",
    "#inferring shape of the output:\n",
    "embed_op_shape=embed.infer_shape(data=x_train.shape)[1]\n",
    "print(\"Shape of embed layer o/p: \",embed_op_shape)\n",
    "\n",
    "#reshaping for further layers:\n",
    "conv_input = mx.sym.Reshape(data=embed, shape=(batch_size, 1, sentence_len, size_embed))\n",
    "\n",
    "#conv layers: sizes:2,3,4,5 - bi-,tri-, 4-, 5- grams considered - creating multiple PARALLEL convolutional layers... \n",
    "# ...  - to be later combined\n",
    "pooled_outputs = []\n",
    "for filter_size in filters:\n",
    "    convi = mx.sym.Convolution(data=conv_input, kernel=(filter_size, size_embed), num_filter=num_filter)\n",
    "    relui = mx.sym.Activation(data=convi, act_type='relu')\n",
    "    pooli = mx.sym.Pooling(data=relui, pool_type='max', kernel=(sentence_len - filter_size + 1, 1), stride=(1, 1))\n",
    "    pooled_outputs.append(pooli)\n",
    "\n",
    "# combining pooled outputs\n",
    "total_filters = num_filter * len(filters)\n",
    "concat = mx.sym.Concat(*pooled_outputs, dim=1)  # ('*' unpacks the list)\n",
    "\n",
    "# reshaping for next layer\n",
    "h_pool = mx.sym.Reshape(data=concat, shape=(batch_size, total_filters))\n",
    "\n",
    "# fully connected layer\n",
    "cls_weight = mx.sym.Variable('cls_weight')\n",
    "cls_bias = mx.sym.Variable('cls_bias')\n",
    "\n",
    "fc = mx.sym.FullyConnected(data=h_pool, weight=cls_weight, bias=cls_bias, num_hidden=2)\n",
    "\n",
    "# softmax output\n",
    "sm = mx.sym.SoftmaxOutput(data=fc, label=input_y, name='softmax')\n",
    "cnn = sm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.30.1 (20180420.1509)\n",
       " -->\n",
       "<!-- Title: plot Pages: 1 -->\n",
       "<svg width=\"623pt\" height=\"912pt\"\n",
       " viewBox=\"0.00 0.00 623.00 912.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 908)\">\n",
       "<title>plot</title>\n",
       "<polygon fill=\"white\" stroke=\"white\" points=\"-4,5 -4,-908 620,-908 620,5 -4,5\"/>\n",
       "<!-- data -->\n",
       "<g id=\"node1\" class=\"node\"><title>data</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"309\" cy=\"-29\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"309\" y=\"-25.3\" font-family=\"Times,serif\" font-size=\"14.00\">data</text>\n",
       "</g>\n",
       "<!-- embed_layer -->\n",
       "<g id=\"node2\" class=\"node\"><title>embed_layer</title>\n",
       "<ellipse fill=\"#fccde5\" stroke=\"black\" cx=\"309\" cy=\"-123\" rx=\"47.0782\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"309\" y=\"-119.3\" font-family=\"Times,serif\" font-size=\"14.00\">embed_layer</text>\n",
       "</g>\n",
       "<!-- embed_layer&#45;&gt;data -->\n",
       "<g id=\"edge1\" class=\"edge\"><title>embed_layer&#45;&gt;data</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M309,-83.7443C309,-75.2043 309,-66.2977 309,-58.2479\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"309,-93.8971 304.5,-83.897 309,-88.8971 309,-83.8971 309,-83.8971 309,-83.8971 309,-88.8971 313.5,-83.8971 309,-93.8971 309,-93.8971\"/>\n",
       "</g>\n",
       "<!-- reshape0 -->\n",
       "<g id=\"node3\" class=\"node\"><title>reshape0</title>\n",
       "<ellipse fill=\"#fdb462\" stroke=\"black\" cx=\"309\" cy=\"-217\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"309\" y=\"-213.3\" font-family=\"Times,serif\" font-size=\"14.00\">reshape0</text>\n",
       "</g>\n",
       "<!-- reshape0&#45;&gt;embed_layer -->\n",
       "<g id=\"edge2\" class=\"edge\"><title>reshape0&#45;&gt;embed_layer</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M309,-177.744C309,-169.204 309,-160.298 309,-152.248\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"309,-187.897 304.5,-177.897 309,-182.897 309,-177.897 309,-177.897 309,-177.897 309,-182.897 313.5,-177.897 309,-187.897 309,-187.897\"/>\n",
       "</g>\n",
       "<!-- convolution0 -->\n",
       "<g id=\"node4\" class=\"node\"><title>convolution0</title>\n",
       "<ellipse fill=\"#fb8072\" stroke=\"black\" cx=\"96\" cy=\"-311\" rx=\"58.3554\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"96\" y=\"-314.8\" font-family=\"Times,serif\" font-size=\"14.00\">Convolution</text>\n",
       "<text text-anchor=\"middle\" x=\"96\" y=\"-299.8\" font-family=\"Times,serif\" font-size=\"14.00\">2x200/1, 100</text>\n",
       "</g>\n",
       "<!-- convolution0&#45;&gt;reshape0 -->\n",
       "<g id=\"edge3\" class=\"edge\"><title>convolution0&#45;&gt;reshape0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M148.467,-287.338C186.421,-270.945 236.829,-249.172 270.958,-234.431\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"139.104,-291.382 146.5,-283.286 143.694,-289.4 148.284,-287.417 148.284,-287.417 148.284,-287.417 143.694,-289.4 150.068,-291.548 139.104,-291.382 139.104,-291.382\"/>\n",
       "</g>\n",
       "<!-- activation0 -->\n",
       "<g id=\"node5\" class=\"node\"><title>activation0</title>\n",
       "<ellipse fill=\"#ffffb3\" stroke=\"black\" cx=\"79\" cy=\"-405\" rx=\"48.4635\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"79\" y=\"-408.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"79\" y=\"-393.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- activation0&#45;&gt;convolution0 -->\n",
       "<g id=\"edge4\" class=\"edge\"><title>activation0&#45;&gt;convolution0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M86.0345,-365.931C87.6437,-357.222 89.3253,-348.122 90.8389,-339.931\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"84.1929,-375.897 81.585,-365.246 85.1015,-370.98 86.0101,-366.064 86.0101,-366.064 86.0101,-366.064 85.1015,-370.98 90.4352,-366.881 84.1929,-375.897 84.1929,-375.897\"/>\n",
       "</g>\n",
       "<!-- pooling0 -->\n",
       "<g id=\"node6\" class=\"node\"><title>pooling0</title>\n",
       "<ellipse fill=\"#80b1d3\" stroke=\"black\" cx=\"70\" cy=\"-499\" rx=\"70.2314\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"70\" y=\"-502.8\" font-family=\"Times,serif\" font-size=\"14.00\">Pooling</text>\n",
       "<text text-anchor=\"middle\" x=\"70\" y=\"-487.8\" font-family=\"Times,serif\" font-size=\"14.00\">max, 111x1/1x1</text>\n",
       "</g>\n",
       "<!-- pooling0&#45;&gt;activation0 -->\n",
       "<g id=\"edge5\" class=\"edge\"><title>pooling0&#45;&gt;activation0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M73.7424,-459.744C74.5778,-451.204 75.4491,-442.298 76.2366,-434.248\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"72.7492,-469.897 69.2443,-459.506 73.2361,-464.921 73.7229,-459.945 73.7229,-459.945 73.7229,-459.945 73.2361,-464.921 78.2015,-460.383 72.7492,-469.897 72.7492,-469.897\"/>\n",
       "</g>\n",
       "<!-- convolution1 -->\n",
       "<g id=\"node7\" class=\"node\"><title>convolution1</title>\n",
       "<ellipse fill=\"#fb8072\" stroke=\"black\" cx=\"241\" cy=\"-311\" rx=\"58.3554\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"241\" y=\"-314.8\" font-family=\"Times,serif\" font-size=\"14.00\">Convolution</text>\n",
       "<text text-anchor=\"middle\" x=\"241\" y=\"-299.8\" font-family=\"Times,serif\" font-size=\"14.00\">3x200/1, 100</text>\n",
       "</g>\n",
       "<!-- convolution1&#45;&gt;reshape0 -->\n",
       "<g id=\"edge6\" class=\"edge\"><title>convolution1&#45;&gt;reshape0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M266.724,-275.196C274.481,-264.702 282.85,-253.38 290.078,-243.6\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"260.668,-283.39 262.994,-272.673 263.64,-279.369 266.612,-275.348 266.612,-275.348 266.612,-275.348 263.64,-279.369 270.231,-278.023 260.668,-283.39 260.668,-283.39\"/>\n",
       "</g>\n",
       "<!-- activation1 -->\n",
       "<g id=\"node8\" class=\"node\"><title>activation1</title>\n",
       "<ellipse fill=\"#ffffb3\" stroke=\"black\" cx=\"231\" cy=\"-405\" rx=\"48.4635\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"231\" y=\"-408.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"231\" y=\"-393.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- activation1&#45;&gt;convolution1 -->\n",
       "<g id=\"edge7\" class=\"edge\"><title>activation1&#45;&gt;convolution1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M235.158,-365.744C236.086,-357.204 237.055,-348.298 237.93,-340.248\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"234.055,-375.897 230.662,-365.469 234.595,-370.926 235.135,-365.956 235.135,-365.956 235.135,-365.956 234.595,-370.926 239.609,-366.442 234.055,-375.897 234.055,-375.897\"/>\n",
       "</g>\n",
       "<!-- pooling1 -->\n",
       "<g id=\"node9\" class=\"node\"><title>pooling1</title>\n",
       "<ellipse fill=\"#80b1d3\" stroke=\"black\" cx=\"228\" cy=\"-499\" rx=\"70.2314\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"228\" y=\"-502.8\" font-family=\"Times,serif\" font-size=\"14.00\">Pooling</text>\n",
       "<text text-anchor=\"middle\" x=\"228\" y=\"-487.8\" font-family=\"Times,serif\" font-size=\"14.00\">max, 110x1/1x1</text>\n",
       "</g>\n",
       "<!-- pooling1&#45;&gt;activation1 -->\n",
       "<g id=\"edge8\" class=\"edge\"><title>pooling1&#45;&gt;activation1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M229.247,-459.744C229.526,-451.204 229.816,-442.298 230.079,-434.248\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"228.916,-469.897 224.745,-459.756 229.079,-464.9 229.242,-459.902 229.242,-459.902 229.242,-459.902 229.079,-464.9 233.74,-460.049 228.916,-469.897 228.916,-469.897\"/>\n",
       "</g>\n",
       "<!-- convolution2 -->\n",
       "<g id=\"node10\" class=\"node\"><title>convolution2</title>\n",
       "<ellipse fill=\"#fb8072\" stroke=\"black\" cx=\"378\" cy=\"-311\" rx=\"58.3554\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"378\" y=\"-314.8\" font-family=\"Times,serif\" font-size=\"14.00\">Convolution</text>\n",
       "<text text-anchor=\"middle\" x=\"378\" y=\"-299.8\" font-family=\"Times,serif\" font-size=\"14.00\">4x200/1, 100</text>\n",
       "</g>\n",
       "<!-- convolution2&#45;&gt;reshape0 -->\n",
       "<g id=\"edge9\" class=\"edge\"><title>convolution2&#45;&gt;reshape0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M351.897,-275.196C344.026,-264.702 335.535,-253.38 328.2,-243.6\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"358.042,-283.39 348.442,-278.09 355.042,-279.39 352.042,-275.39 352.042,-275.39 352.042,-275.39 355.042,-279.39 355.642,-272.69 358.042,-283.39 358.042,-283.39\"/>\n",
       "</g>\n",
       "<!-- activation2 -->\n",
       "<g id=\"node11\" class=\"node\"><title>activation2</title>\n",
       "<ellipse fill=\"#ffffb3\" stroke=\"black\" cx=\"384\" cy=\"-405\" rx=\"48.4635\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"384\" y=\"-408.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"384\" y=\"-393.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- activation2&#45;&gt;convolution2 -->\n",
       "<g id=\"edge10\" class=\"edge\"><title>activation2&#45;&gt;convolution2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M381.505,-365.744C380.948,-357.204 380.367,-348.298 379.842,-340.248\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"382.167,-375.897 377.026,-366.211 381.842,-370.908 381.516,-365.918 381.516,-365.918 381.516,-365.918 381.842,-370.908 386.007,-365.625 382.167,-375.897 382.167,-375.897\"/>\n",
       "</g>\n",
       "<!-- pooling2 -->\n",
       "<g id=\"node12\" class=\"node\"><title>pooling2</title>\n",
       "<ellipse fill=\"#80b1d3\" stroke=\"black\" cx=\"386\" cy=\"-499\" rx=\"70.2314\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"386\" y=\"-502.8\" font-family=\"Times,serif\" font-size=\"14.00\">Pooling</text>\n",
       "<text text-anchor=\"middle\" x=\"386\" y=\"-487.8\" font-family=\"Times,serif\" font-size=\"14.00\">max, 109x1/1x1</text>\n",
       "</g>\n",
       "<!-- pooling2&#45;&gt;activation2 -->\n",
       "<g id=\"edge11\" class=\"edge\"><title>pooling2&#45;&gt;activation2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M385.168,-459.744C384.983,-451.204 384.789,-442.298 384.614,-434.248\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"385.389,-469.897 380.673,-459.997 385.28,-464.898 385.172,-459.899 385.172,-459.899 385.172,-459.899 385.28,-464.898 389.671,-459.802 385.389,-469.897 385.389,-469.897\"/>\n",
       "</g>\n",
       "<!-- convolution3 -->\n",
       "<g id=\"node13\" class=\"node\"><title>convolution3</title>\n",
       "<ellipse fill=\"#fb8072\" stroke=\"black\" cx=\"520\" cy=\"-311\" rx=\"58.3554\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"520\" y=\"-314.8\" font-family=\"Times,serif\" font-size=\"14.00\">Convolution</text>\n",
       "<text text-anchor=\"middle\" x=\"520\" y=\"-299.8\" font-family=\"Times,serif\" font-size=\"14.00\">5x200/1, 100</text>\n",
       "</g>\n",
       "<!-- convolution3&#45;&gt;reshape0 -->\n",
       "<g id=\"edge12\" class=\"edge\"><title>convolution3&#45;&gt;reshape0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M467.798,-287.239C430.321,-270.898 380.674,-249.251 346.938,-234.542\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"477.047,-291.272 466.082,-291.4 472.463,-289.273 467.88,-287.275 467.88,-287.275 467.88,-287.275 472.463,-289.273 469.679,-283.15 477.047,-291.272 477.047,-291.272\"/>\n",
       "</g>\n",
       "<!-- activation3 -->\n",
       "<g id=\"node14\" class=\"node\"><title>activation3</title>\n",
       "<ellipse fill=\"#ffffb3\" stroke=\"black\" cx=\"537\" cy=\"-405\" rx=\"48.4635\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"537\" y=\"-408.8\" font-family=\"Times,serif\" font-size=\"14.00\">Activation</text>\n",
       "<text text-anchor=\"middle\" x=\"537\" y=\"-393.8\" font-family=\"Times,serif\" font-size=\"14.00\">relu</text>\n",
       "</g>\n",
       "<!-- activation3&#45;&gt;convolution3 -->\n",
       "<g id=\"edge13\" class=\"edge\"><title>activation3&#45;&gt;convolution3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M529.966,-365.931C528.356,-357.222 526.675,-348.122 525.161,-339.931\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"531.807,-375.897 525.565,-366.881 530.898,-370.98 529.99,-366.064 529.99,-366.064 529.99,-366.064 530.898,-370.98 534.415,-365.246 531.807,-375.897 531.807,-375.897\"/>\n",
       "</g>\n",
       "<!-- pooling3 -->\n",
       "<g id=\"node15\" class=\"node\"><title>pooling3</title>\n",
       "<ellipse fill=\"#80b1d3\" stroke=\"black\" cx=\"544\" cy=\"-499\" rx=\"70.2314\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"544\" y=\"-502.8\" font-family=\"Times,serif\" font-size=\"14.00\">Pooling</text>\n",
       "<text text-anchor=\"middle\" x=\"544\" y=\"-487.8\" font-family=\"Times,serif\" font-size=\"14.00\">max, 108x1/1x1</text>\n",
       "</g>\n",
       "<!-- pooling3&#45;&gt;activation3 -->\n",
       "<g id=\"edge14\" class=\"edge\"><title>pooling3&#45;&gt;activation3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M541.089,-459.744C540.439,-451.204 539.762,-442.298 539.149,-434.248\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"541.862,-469.897 536.616,-460.267 541.482,-464.911 541.103,-459.926 541.103,-459.926 541.103,-459.926 541.482,-464.911 545.59,-459.584 541.862,-469.897 541.862,-469.897\"/>\n",
       "</g>\n",
       "<!-- concat0 -->\n",
       "<g id=\"node16\" class=\"node\"><title>concat0</title>\n",
       "<ellipse fill=\"#fdb462\" stroke=\"black\" cx=\"307\" cy=\"-593\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"307\" y=\"-589.3\" font-family=\"Times,serif\" font-size=\"14.00\">concat0</text>\n",
       "</g>\n",
       "<!-- concat0&#45;&gt;pooling0 -->\n",
       "<g id=\"edge15\" class=\"edge\"><title>concat0&#45;&gt;pooling0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M258.487,-573.168C218.069,-557.478 160.722,-535.217 119.918,-519.377\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"267.864,-576.808 256.914,-577.384 263.203,-574.999 258.542,-573.189 258.542,-573.189 258.542,-573.189 263.203,-574.999 260.17,-568.994 267.864,-576.808 267.864,-576.808\"/>\n",
       "</g>\n",
       "<!-- concat0&#45;&gt;pooling1 -->\n",
       "<g id=\"edge16\" class=\"edge\"><title>concat0&#45;&gt;pooling1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M278.659,-558.996C269.562,-548.401 259.604,-536.805 250.943,-526.718\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"285.413,-566.861 275.484,-562.205 282.156,-563.067 278.898,-559.274 278.898,-559.274 278.898,-559.274 282.156,-563.067 282.312,-556.342 285.413,-566.861 285.413,-566.861\"/>\n",
       "</g>\n",
       "<!-- concat0&#45;&gt;pooling2 -->\n",
       "<g id=\"edge17\" class=\"edge\"><title>concat0&#45;&gt;pooling2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M335.341,-558.996C344.438,-548.401 354.396,-536.805 363.057,-526.718\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"328.587,-566.861 331.688,-556.342 331.844,-563.067 335.102,-559.274 335.102,-559.274 335.102,-559.274 331.844,-563.067 338.516,-562.205 328.587,-566.861 328.587,-566.861\"/>\n",
       "</g>\n",
       "<!-- concat0&#45;&gt;pooling3 -->\n",
       "<g id=\"edge18\" class=\"edge\"><title>concat0&#45;&gt;pooling3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M355.513,-573.168C395.931,-557.478 453.278,-535.217 494.082,-519.377\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"346.136,-576.808 353.83,-568.994 350.797,-574.999 355.458,-573.189 355.458,-573.189 355.458,-573.189 350.797,-574.999 357.086,-577.384 346.136,-576.808 346.136,-576.808\"/>\n",
       "</g>\n",
       "<!-- reshape1 -->\n",
       "<g id=\"node17\" class=\"node\"><title>reshape1</title>\n",
       "<ellipse fill=\"#fdb462\" stroke=\"black\" cx=\"307\" cy=\"-687\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"307\" y=\"-683.3\" font-family=\"Times,serif\" font-size=\"14.00\">reshape1</text>\n",
       "</g>\n",
       "<!-- reshape1&#45;&gt;concat0 -->\n",
       "<g id=\"edge19\" class=\"edge\"><title>reshape1&#45;&gt;concat0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M307,-647.744C307,-639.204 307,-630.298 307,-622.248\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"307,-657.897 302.5,-647.897 307,-652.897 307,-647.897 307,-647.897 307,-647.897 307,-652.897 311.5,-647.897 307,-657.897 307,-657.897\"/>\n",
       "</g>\n",
       "<!-- fullyconnected0 -->\n",
       "<g id=\"node18\" class=\"node\"><title>fullyconnected0</title>\n",
       "<ellipse fill=\"#fb8072\" stroke=\"black\" cx=\"307\" cy=\"-781\" rx=\"68.2532\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"307\" y=\"-784.8\" font-family=\"Times,serif\" font-size=\"14.00\">FullyConnected</text>\n",
       "<text text-anchor=\"middle\" x=\"307\" y=\"-769.8\" font-family=\"Times,serif\" font-size=\"14.00\">2</text>\n",
       "</g>\n",
       "<!-- fullyconnected0&#45;&gt;reshape1 -->\n",
       "<g id=\"edge20\" class=\"edge\"><title>fullyconnected0&#45;&gt;reshape1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M307,-741.744C307,-733.204 307,-724.298 307,-716.248\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"307,-751.897 302.5,-741.897 307,-746.897 307,-741.897 307,-741.897 307,-741.897 307,-746.897 311.5,-741.897 307,-751.897 307,-751.897\"/>\n",
       "</g>\n",
       "<!-- softmax_label -->\n",
       "<g id=\"node19\" class=\"node\"><title>softmax_label</title>\n",
       "<ellipse fill=\"#8dd3c7\" stroke=\"black\" cx=\"444\" cy=\"-781\" rx=\"51.2967\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"444\" y=\"-777.3\" font-family=\"Times,serif\" font-size=\"14.00\">softmax_label</text>\n",
       "</g>\n",
       "<!-- softmax -->\n",
       "<g id=\"node20\" class=\"node\"><title>softmax</title>\n",
       "<ellipse fill=\"#fccde5\" stroke=\"black\" cx=\"375\" cy=\"-875\" rx=\"47\" ry=\"29\"/>\n",
       "<text text-anchor=\"middle\" x=\"375\" y=\"-871.3\" font-family=\"Times,serif\" font-size=\"14.00\">softmax</text>\n",
       "</g>\n",
       "<!-- softmax&#45;&gt;fullyconnected0 -->\n",
       "<g id=\"edge21\" class=\"edge\"><title>softmax&#45;&gt;fullyconnected0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M350.08,-840.285C342.407,-829.903 334.068,-818.621 326.788,-808.772\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"356.058,-848.373 346.495,-843.006 353.086,-844.352 350.114,-840.331 350.114,-840.331 350.114,-840.331 353.086,-844.352 353.733,-837.656 356.058,-848.373 356.058,-848.373\"/>\n",
       "</g>\n",
       "<!-- softmax&#45;&gt;softmax_label -->\n",
       "<g id=\"edge22\" class=\"edge\"><title>softmax&#45;&gt;softmax_label</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M400.394,-840.141C408.321,-829.572 416.938,-818.082 424.403,-808.129\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"394.22,-848.373 396.62,-837.673 397.22,-844.373 400.22,-840.373 400.22,-840.373 400.22,-840.373 397.22,-844.373 403.82,-843.073 394.22,-848.373 394.22,-848.373\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7f4f3a6d57f0>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mx.viz.plot_network(cnn, node_attrs={\"shape\":\"oval\",\"fixedsize\":\"false\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "***\n",
    "\n",
    "### Declaring the custom Training Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_fn(symbol,ctx,\n",
    "             x_train,x_test,y_train,y_test,\n",
    "             optimizer='rmsprop',max_grad_norm=2.0,display_step=1):\n",
    "    '''\n",
    "    Custom training function:\n",
    "    Params:\n",
    "        symbol : outermost layer of the network\n",
    "        x_train,y_train: training data\n",
    "        x_test, y_test: test data\n",
    "        \n",
    "    Returns:\n",
    "        train_accs: the train accuracies obtained for EACH BATCH (numpy array: n_epochs)\n",
    "        test_accs: the test accuracies obtained for EACH BATCH (numpy array: n_epochs)\n",
    "        train_preds: the predictions on the training data for EACH BATCH (numpy array: n_epochs X n_samples)\n",
    "        test_preds: the predictions on the test data for EACH BATCH (numpy array: n_epochs X n_samples)\n",
    "        \n",
    "    (Prints the status of training and test metrics)\n",
    "    '''\n",
    "    ##################################################################\n",
    "    test_accs=np.zeros(n_epochs)\n",
    "    train_accs=np.zeros(n_epochs)\n",
    "    train_preds=np.zeros((n_epochs,x_train.shape[0]))\n",
    "    test_preds=np.zeros((n_epochs,x_test.shape[0]))\n",
    "    \n",
    "    arg_names = symbol.list_arguments()\n",
    "    input_shapes = {}\n",
    "    input_shapes['data'] = (batch_size,)+x_train.shape[1:]\n",
    "    arg_shape, out_shape, aux_shape = symbol.infer_shape(**input_shapes)\n",
    "    \n",
    "    # declaring all weights/biases\n",
    "    arg_arrays = [mx.nd.zeros(s, ctx) for s in arg_shape]\n",
    "    \n",
    "    # gradients dictionary - declaring all gradients\n",
    "    args_grad = {}\n",
    "    for shape, name in zip(arg_shape, arg_names):\n",
    "        if name in ['softmax_label', 'data']: # input, output\n",
    "            continue\n",
    "        args_grad[name] = mx.nd.zeros(shape, ctx)\n",
    "\n",
    "    # binding the executor\n",
    "    cnn_exec = symbol.bind(ctx=ctx, args=arg_arrays, args_grad=args_grad, grad_req='add')\n",
    "    param_blocks = []\n",
    "    arg_dict = dict(zip(arg_names, cnn_exec.arg_arrays))\n",
    "    \n",
    "    #initializing - Uniform initialization of all variables\n",
    "    initializer = mx.initializer.Uniform(0.01)\n",
    "    for i, name in enumerate(arg_names):\n",
    "        if name in ['softmax_label', 'data']: # leave aside input, output\n",
    "            continue\n",
    "        initializer(mx.init.InitDesc(name), arg_dict[name])\n",
    "\n",
    "        param_blocks.append( (i, arg_dict[name], args_grad[name], name) )\n",
    "    \n",
    "    \n",
    "    \n",
    "    #################################################################\n",
    "    # creating and setting-up optimizer, updater\n",
    "    opt = mx.optimizer.create(optimizer)\n",
    "    opt.lr = learn_rate\n",
    "    updater = mx.optimizer.get_updater(opt)\n",
    "\n",
    "    # For each training epoch\n",
    "    for iteration in range(n_epochs):\n",
    "        num_correct = 0\n",
    "        num_total = 0\n",
    "\n",
    "        # Over each batch of training data\n",
    "        for begin in range(0, x_train.shape[0], batch_size):\n",
    "            batchX = x_train[begin:begin+batch_size]\n",
    "            batchY = y_train[begin:begin+batch_size]\n",
    "            if batchX.shape[0] != batch_size:\n",
    "                continue\n",
    "\n",
    "            cnn_exec.arg_dict['data'][:] = batchX\n",
    "            cnn_exec.arg_dict['softmax_label'][:] = batchY\n",
    "\n",
    "            # forward pass\n",
    "            cnn_exec.forward(is_train=True)\n",
    "\n",
    "            # backward pass - computes gradients\n",
    "            cnn_exec.backward()\n",
    "\n",
    "            # generated outputs from forward pass accessible through cnn_exec.outputs\n",
    "            # eval on training data\n",
    "            train_preds[iteration,begin:begin+batch_size]=np.argmax(cnn_exec.outputs[0].asnumpy(), axis=1)\n",
    "            num_correct += sum(batchY == np.argmax(cnn_exec.outputs[0].asnumpy(), axis=1))\n",
    "            num_total += len(batchY)\n",
    "\n",
    "            # updating weights\n",
    "            # the args_grad list contains the gradients and is bound to the executor...\n",
    "            # ... it, therefore gives access to the generated gradients\n",
    "            norm = 0\n",
    "            for idx, weight, grad, name in param_blocks:\n",
    "                grad /= batch_size\n",
    "                l2_norm = mx.nd.norm(grad).asscalar()\n",
    "                norm += l2_norm * l2_norm\n",
    "\n",
    "            norm = np.sqrt(norm)\n",
    "            for idx, weight, grad, name in param_blocks:\n",
    "                if norm > max_grad_norm:\n",
    "                    grad *= (max_grad_norm / norm)\n",
    "\n",
    "                updater(idx, grad, weight)\n",
    "\n",
    "                # resetting gradients to zero\n",
    "                grad[:] = 0.0\n",
    "\n",
    "        # End of training loop for this epoch\n",
    "        train_acc = num_correct * 100 / float(num_total)\n",
    "    \n",
    "        # Evaluate model after this epoch on dev (test) set\n",
    "        num_correct = 0\n",
    "        num_total = 0\n",
    "\n",
    "        # For each test batch\n",
    "        for begin in range(0, x_test.shape[0], batch_size):\n",
    "            batchX = x_test[begin:begin+batch_size]\n",
    "            batchY = y_test[begin:begin+batch_size]\n",
    "\n",
    "            if batchX.shape[0] != batch_size:\n",
    "                continue\n",
    "\n",
    "            cnn_exec.arg_dict['data'][:] = batchX\n",
    "            cnn_exec.forward(is_train=False)\n",
    "            test_preds[iteration,begin:begin+batch_size]=np.argmax(cnn_exec.outputs[0].asnumpy(), axis=1)\n",
    "            num_correct += sum(batchY == np.argmax(cnn_exec.outputs[0].asnumpy(), axis=1))\n",
    "            num_total += len(batchY)\n",
    "\n",
    "        test_acc = num_correct * 100 / float(num_total)\n",
    "        test_accs[iteration]=test_acc\n",
    "        train_accs[iteration]=train_acc\n",
    "        if (iteration%display_step)==0:\n",
    "            print('Iter [%d] Train: Training Accuracy: %.3f \\\n",
    "                    --- Test Accuracy : %.3f' % (iteration, train_acc, test_acc))\n",
    "    return train_accs,test_accs,train_preds,test_preds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training the Model now"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter [0] Train: Training Accuracy: 86.167                     --- Test Accuracy : 85.667\n",
      "Iter [1] Train: Training Accuracy: 90.417                     --- Test Accuracy : 84.000\n",
      "Iter [2] Train: Training Accuracy: 94.833                     --- Test Accuracy : 89.333\n",
      "Iter [3] Train: Training Accuracy: 97.917                     --- Test Accuracy : 89.667\n",
      "Iter [4] Train: Training Accuracy: 99.167                     --- Test Accuracy : 90.000\n",
      "Iter [5] Train: Training Accuracy: 99.500                     --- Test Accuracy : 90.667\n",
      "Iter [6] Train: Training Accuracy: 99.667                     --- Test Accuracy : 90.000\n",
      "Iter [7] Train: Training Accuracy: 99.667                     --- Test Accuracy : 89.667\n",
      "Iter [8] Train: Training Accuracy: 99.667                     --- Test Accuracy : 90.667\n",
      "Iter [9] Train: Training Accuracy: 99.750                     --- Test Accuracy : 89.333\n"
     ]
    }
   ],
   "source": [
    "#the context - CPU used here\n",
    "ctx = mx.cpu()\n",
    "train_accs,test_accs,train_preds,test_preds = train_fn(cnn,ctx,x_train,x_test,y_train,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extending the Process for Further Experimentation:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Experimenting by introducing FastText model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#using Gensim for FastText embeddings creation and usage\n",
    "from gensim.models import FastText\n",
    "ft_model = FastText(size=size_embed, window=3, min_count=1)  # instantiating FastText model\n",
    "#padded_sentences=pad_sentences(sentences)\n",
    "corpus=sentences\n",
    "ft_model.build_vocab(sentences=corpus)\n",
    "ft_model.train(sentences=corpus, total_examples=len(corpus), epochs=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Some nice examples proving that it did indeed learn something!!!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ai\n",
      "[('fights', 0.9892082214355469), ('praise', 0.9811339378356934), ('fighters', 0.9776649475097656), ('funds', 0.9769025444984436), ('please', 0.9765199422836304), ('fish', 0.9749014377593994), ('needs', 0.9748384356498718), ('rainbow', 0.9742629528045654), ('five', 0.9740978479385376), ('finish', 0.9724087119102478)]\n"
     ]
    }
   ],
   "source": [
    "ft_embeddings=ft_model.wv\n",
    "print (ft_embeddings.doesnt_match(\"usa france ai politics\".split()))\n",
    "print(ft_embeddings.similar_by_word(\"fight\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1506, 112, 200)\n"
     ]
    }
   ],
   "source": [
    "padded_sentences=pad_sentences(sentences)\n",
    "new_sent_vectors = np.array([\n",
    "            [ft_embeddings[word] if word!='</s>' else np.zeros(200) for word in sentence]\n",
    "            for sentence in padded_sentences])\n",
    "print(new_sent_vectors.shape)\n",
    "#this value ought to be same as the output of the embedding layer of the previous model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining & Training the new model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter [0] Train: Training Accuracy: 85.250                     --- Test Accuracy : 86.000\n",
      "Iter [1] Train: Training Accuracy: 85.333                     --- Test Accuracy : 86.667\n",
      "Iter [2] Train: Training Accuracy: 85.333                     --- Test Accuracy : 87.000\n",
      "Iter [3] Train: Training Accuracy: 86.250                     --- Test Accuracy : 87.333\n",
      "Iter [4] Train: Training Accuracy: 86.250                     --- Test Accuracy : 86.000\n",
      "Iter [5] Train: Training Accuracy: 86.500                     --- Test Accuracy : 86.000\n",
      "Iter [6] Train: Training Accuracy: 86.333                     --- Test Accuracy : 87.000\n",
      "Iter [7] Train: Training Accuracy: 86.750                     --- Test Accuracy : 85.667\n",
      "Iter [8] Train: Training Accuracy: 87.250                     --- Test Accuracy : 84.667\n",
      "Iter [9] Train: Training Accuracy: 87.833                     --- Test Accuracy : 84.667\n"
     ]
    }
   ],
   "source": [
    "new_x_train,new_y_train,new_x_test,new_y_test=create_shuffle(new_sent_vectors,y)\n",
    "\n",
    "\n",
    "# NOTE: It is imperative, at this stage that we DO NOT change the names of the ...\n",
    "#... the input and output placeholders, as the training function above uses them...\n",
    "#... Otherwise, the content of the function could be copied directly and used.\n",
    "\n",
    "\n",
    "# creating a CNN new model - almost same as before - without the embeddings layer\n",
    "new_input_x = mx.sym.Variable('data')\n",
    "new_input_y = mx.sym.Variable('softmax_label')\n",
    "new_conv_input = mx.sym.Reshape(data=new_input_x, shape=(batch_size, 1, sentence_len, size_embed))\n",
    "new_pooled_outputs = []\n",
    "for filter_size in filters:\n",
    "    new_convi = mx.sym.Convolution(data=new_conv_input, kernel=(filter_size, size_embed), num_filter=num_filter)\n",
    "    new_relui = mx.sym.Activation(data=new_convi, act_type='relu')\n",
    "    new_pooli = mx.sym.Pooling(data=new_relui, pool_type='max', kernel=(sentence_len - filter_size + 1, 1), stride=(1, 1))\n",
    "    new_pooled_outputs.append(new_pooli)\n",
    "\n",
    "total_filters = num_filter * len(filters)\n",
    "new_concat = mx.sym.Concat(*new_pooled_outputs, dim=1)  # ('*' unpacks the list)\n",
    "\n",
    "new_h_pool = mx.sym.Reshape(data=new_concat, shape=(batch_size, total_filters))\n",
    "\n",
    "# fully connected layer\n",
    "new_cls_weight = mx.sym.Variable('cls_weight')\n",
    "new_cls_bias = mx.sym.Variable('cls_bias')\n",
    "\n",
    "new_fc = mx.sym.FullyConnected(data=new_h_pool, weight=new_cls_weight, bias=new_cls_bias, num_hidden=2)\n",
    "\n",
    "# softmax output\n",
    "new_sm = mx.sym.SoftmaxOutput(data=new_fc, label=new_input_y, name='softmax')\n",
    "new_cnn = new_sm\n",
    "\n",
    "# calling the custom training function\n",
    "ctx=mx.cpu()\n",
    "new_train_accs,new_test_accs,new_train_preds,new_test_preds = train_fn(new_cnn,ctx,new_x_train,new_x_test,new_y_train,new_y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "**Therefore, with the parameters unchanged, the introduction of the FastText embeddings to this model does not add to accuracies. Several reasons could be given for the observation:**\n",
    "- Hyper-parameter tuning might change this\n",
    "- The corpus on which embeddings have been trained is not vast enough to capture much context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "***\n",
    "### Alternative Metrics\n",
    "- Accuracy is just the ratio of correct outcomes to all outcomes and hence is a poor indicator in this case because of data skew - a naive model throwing out just the result of the majority class could still achieve 6/7 = ~85% accuracy\n",
    "- Precision and Recall are better indicators, especially since we know what class concerns here more (the +ve ones)\n",
    "- Plotting the confusion matrix is a good way to visualize the distributions\n",
    "- Since we have the predictions of the models after each epoch, we could directly use them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function for calculating precision and recall\n",
    "def prec_recall(y_true,y_pred,ref_class=1):\n",
    "    '''\n",
    "    params:\n",
    "        y_true,y_pred = np array of same shape : different epochs along different rows\n",
    "        ref_class = the class wrt which precision/recall is to be calculated\n",
    "    return:\n",
    "        prec\n",
    "        recall\n",
    "    '''\n",
    "    tp=((y_pred==1)*(y_true==1)).sum(axis=1) #true positives\n",
    "    fp=((y_pred==1)*(y_true==0)).sum(axis=1) #false positives\n",
    "    tn=((y_pred==0)*(y_true==0)).sum(axis=1) #true negatives\n",
    "    fn=((y_pred==0)*(y_true==1)).sum(axis=1) #false positive\n",
    "    prec = np.nan_to_num(tp/(tp+fp))\n",
    "    rec = np.nan_to_num(tp/(tp+fn))\n",
    "    return prec,rec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "p1_train,r1_train=prec_recall(y_train,train_preds)\n",
    "p1_test,r1_test=prec_recall(y_test,test_preds)\n",
    "\n",
    "p2_train,r2_train=prec_recall(new_y_train,new_train_preds)\n",
    "p2_test,r2_test=prec_recall(new_y_test,new_test_preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model 1(TRAIN): best precision of positive class = 0.988, best recall= 0.994\n",
      "Model 1(TEST): best precision of positive class = 0.929, best recall= 0.548\n",
      "Model 2(TRAIN): best precision of positive class = 0.604, best recall= 0.331\n",
      "Model 2(TEST): best precision of positive class = 0.750, best recall= 0.357\n"
     ]
    }
   ],
   "source": [
    "print(\"Model 1(TRAIN): best precision of positive class = %.3f, best recall= %.3f\" %(p1_train.max(),r1_train.max()))\n",
    "print(\"Model 1(TEST): best precision of positive class = %.3f, best recall= %.3f\" %(p1_test.max(),r1_test.max()))\n",
    "print(\"Model 2(TRAIN): best precision of positive class = %.3f, best recall= %.3f\" %(p2_train.max(),r2_train.max()))\n",
    "print(\"Model 2(TEST): best precision of positive class = %.3f, best recall= %.3f\" %(p2_test.max(),r2_test.max()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Clearly, FastText has not performed too well for the minority class.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "### Tinkering with the hyper-parameters\n",
    "We could notice the following important points before going after the hyper-paramters:\n",
    "- In real world application, the performance on the TEST data matters more\n",
    "- Since it is much more important to capture all positive cases, even at the expense of some false positives (as while going through them manually, one could always leave them out. **We do not want to miss out the positive cases**. Therefore, **We must consider RECALL as the metric of choice**\n",
    "- Model 2 (with FastText) appears to perform much worse compared to its potential. I'll therefore try to spend more effort on it.\n",
    "- Model 1 has already achieved near-perfect performance on train data and improving performance in that case would involve some regularization, etc, for which the network will have to be edited"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### EXP-1: Smaller embeddings, more epochs on FastText model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter [0] Train: Training Accuracy: 86.167                     --- Test Accuracy : 86.000\n",
      "Iter [5] Train: Training Accuracy: 87.250                     --- Test Accuracy : 84.667\n",
      "Iter [10] Train: Training Accuracy: 89.833                     --- Test Accuracy : 81.333\n",
      "Iter [15] Train: Training Accuracy: 93.667                     --- Test Accuracy : 79.333\n",
      "Iter [20] Train: Training Accuracy: 95.667                     --- Test Accuracy : 85.667\n",
      "Iter [25] Train: Training Accuracy: 96.833                     --- Test Accuracy : 86.333\n",
      "Iter [30] Train: Training Accuracy: 98.167                     --- Test Accuracy : 83.333\n",
      "Iter [35] Train: Training Accuracy: 98.583                     --- Test Accuracy : 81.667\n",
      "Iter [40] Train: Training Accuracy: 98.167                     --- Test Accuracy : 76.333\n",
      "Iter [45] Train: Training Accuracy: 99.083                     --- Test Accuracy : 83.333\n",
      "\n",
      "\n",
      "Best recall on test data:  0.761904761905\n"
     ]
    }
   ],
   "source": [
    "n_epochs=50\n",
    "batch_size=10\n",
    "size_embed=100\n",
    "learn_rate=0.002\n",
    "ft_model = FastText(size=size_embed, window=3, min_count=1)  # instantiating FastText model\n",
    "corpus=sentences\n",
    "ft_model.build_vocab(sentences=corpus)\n",
    "ft_model.train(sentences=corpus, total_examples=len(corpus), epochs=20)\n",
    "ft_embeddings=ft_model.wv\n",
    "new_sent_vectors = np.array([\n",
    "            [ft_embeddings[word] if word!='</s>' else np.zeros(size_embed) for word in sentence]\n",
    "            for sentence in padded_sentences])\n",
    "new_x_train,new_y_train,new_x_test,new_y_test=create_shuffle(new_sent_vectors,y)\n",
    "new_input_x = mx.sym.Variable('data')\n",
    "new_input_y = mx.sym.Variable('softmax_label')\n",
    "new_conv_input = mx.sym.Reshape(data=new_input_x, shape=(batch_size, 1, sentence_len, size_embed))\n",
    "new_pooled_outputs = []\n",
    "for filter_size in filters:\n",
    "    new_convi = mx.sym.Convolution(data=new_conv_input, kernel=(filter_size, size_embed), num_filter=num_filter)\n",
    "    new_relui = mx.sym.Activation(data=new_convi, act_type='relu')\n",
    "    new_pooli = mx.sym.Pooling(data=new_relui, pool_type='max', kernel=(sentence_len - filter_size + 1, 1), stride=(1, 1))\n",
    "    new_pooled_outputs.append(new_pooli)\n",
    "\n",
    "total_filters = num_filter * len(filters)\n",
    "new_concat = mx.sym.Concat(*new_pooled_outputs, dim=1)  # ('*' unpacks the list)\n",
    "\n",
    "new_h_pool = mx.sym.Reshape(data=new_concat, shape=(batch_size, total_filters))\n",
    "new_cls_weight = mx.sym.Variable('cls_weight')\n",
    "new_cls_bias = mx.sym.Variable('cls_bias')\n",
    "\n",
    "new_fc = mx.sym.FullyConnected(data=new_h_pool, weight=new_cls_weight, bias=new_cls_bias, num_hidden=2)\n",
    "new_sm = mx.sym.SoftmaxOutput(data=new_fc, label=new_input_y, name='softmax')\n",
    "new_cnn = new_sm\n",
    "ctx=mx.cpu()\n",
    "new_train_accs,new_test_accs,new_train_preds,new_test_preds = train_fn(new_cnn,ctx,new_x_train,new_x_test,new_y_train,new_y_test,display_step=5)\n",
    "_,r=prec_recall(new_y_test,new_test_preds)\n",
    "print(\"\\n\\nBest recall on test data: \",r.max())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**=> Much better recall at low learn_rate, small embedding size, large no. of epochs**\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### EXP-2: Dropouts in Model 1 + larger no. of epochs + higher number of filters : 2,3,4,5,6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input shape:  (1204, 112)\n",
      "Shape of embed layer o/p:  [(1204, 112, 100)]\n",
      "Iter [0] Train: Training Accuracy: 86.167                     --- Test Accuracy : 86.667\n",
      "Iter [5] Train: Training Accuracy: 98.417                     --- Test Accuracy : 90.667\n",
      "Iter [10] Train: Training Accuracy: 99.750                     --- Test Accuracy : 89.667\n",
      "Iter [15] Train: Training Accuracy: 99.667                     --- Test Accuracy : 90.000\n",
      "Iter [20] Train: Training Accuracy: 99.500                     --- Test Accuracy : 89.667\n",
      "Iter [25] Train: Training Accuracy: 99.750                     --- Test Accuracy : 90.000\n",
      "Iter [30] Train: Training Accuracy: 99.750                     --- Test Accuracy : 89.667\n",
      "Iter [35] Train: Training Accuracy: 99.667                     --- Test Accuracy : 90.000\n",
      "Iter [40] Train: Training Accuracy: 99.833                     --- Test Accuracy : 89.667\n",
      "Iter [45] Train: Training Accuracy: 99.750                     --- Test Accuracy : 90.000\n",
      "\n",
      "\n",
      "Best recall on test data:  0.761904761905\n"
     ]
    }
   ],
   "source": [
    "n_epochs=50\n",
    "batch_size=10\n",
    "size_embed=100\n",
    "learn_rate=0.002\n",
    "dropout = 0.5\n",
    "input_x = mx.sym.Variable('data')\n",
    "input_y = mx.sym.Variable('softmax_label')\n",
    "print (\"input shape: \",x_train.shape)\n",
    "\n",
    "#embeddings layer - for converting to n-dimension vector representations while training\n",
    "embed = mx.sym.Embedding(data=input_x, input_dim=vocab_size, output_dim=size_embed, name='embed_layer')\n",
    "#inferring shape of the output:\n",
    "embed_op_shape=embed.infer_shape(data=x_train.shape)[1]\n",
    "print(\"Shape of embed layer o/p: \",embed_op_shape)\n",
    "\n",
    "#reshaping for further layers:\n",
    "conv_input = mx.sym.Reshape(data=embed, shape=(batch_size, 1, sentence_len, size_embed))\n",
    "filters=[2,3,4,5,6]\n",
    "pooled_outputs = []\n",
    "for filter_size in filters:\n",
    "    convi = mx.sym.Convolution(data=conv_input, kernel=(filter_size, size_embed), num_filter=num_filter)\n",
    "    relui = mx.sym.Activation(data=convi, act_type='relu')\n",
    "    pooli = mx.sym.Pooling(data=relui, pool_type='max', kernel=(sentence_len - filter_size + 1, 1), stride=(1, 1))\n",
    "    pooled_outputs.append(pooli)\n",
    "total_filters = num_filter * len(filters)\n",
    "concat = mx.sym.Concat(*pooled_outputs, dim=1)  # ('*' unpacks the list)\n",
    "h_pool = mx.sym.Reshape(data=concat, shape=(batch_size, total_filters))\n",
    "h_drop = mx.sym.Dropout(data=h_pool, p=dropout)\n",
    "cls_weight = mx.sym.Variable('cls_weight')\n",
    "cls_bias = mx.sym.Variable('cls_bias')\n",
    "fc = mx.sym.FullyConnected(data=h_drop, weight=cls_weight, bias=cls_bias, num_hidden=2)\n",
    "sm = mx.sym.SoftmaxOutput(data=fc, label=input_y, name='softmax')\n",
    "cnn = sm\n",
    "train_accs,test_accs,train_preds,test_preds = train_fn(cnn,ctx,x_train,x_test,y_train,y_test,display_step=5)\n",
    "_,r=prec_recall(new_y_test,new_test_preds)\n",
    "print(\"\\n\\nBest recall on test data: \",r.max())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "# Conclusion\n",
    "\n",
    "**Following were the outcomes of the experimentations:**\n",
    "- Model 1 performs much better than the one with FastText with the given hyper-paramters\n",
    "    - Model 1: Best Test Accuracy: 91%; Best Recall on Test data: 55%\n",
    "    - Model 2: Best Test Accuracy: 87%; Best Recall on Test data: 36%\n",
    "    \n",
    "**Results of tuning hyper-parameters**\n",
    "- Model 2, however shows marked improvement by:\n",
    "    1. Reducing embedding size (lesser paramters to train)\n",
    "    2. Lower learning rate and larger no. of epochs \n",
    "    3. The new performance: Best Test Accuracy: ~90%; Best Recall on Test data: 75%\n",
    "    \n",
    "    \n",
    "- Model 1 was subjected to following new conditions:\n",
    "    1. Dropouts introduced after convolutions (d=0.5)\n",
    "    2. Low learning rate; larger number of epochs\n",
    "    3. More filters: of sizes : 2,3,4,5,6 (100 each)\n",
    "    4. Result: Significant improvement on test recall : ~75%"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}