RCNN performance analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we will test a neural network model with a recurrent layer and a convolution layer.  \n",
    "  \n",
    "This model scored a public leaderboard score (AUC) of 0.9721 on Kaggle. It is higher than some of the most popular baseline models and thus should be considered a decent starting point. By examining its weakness, we would hopefully learn what should we focus on in the future."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dXsRFoKJSNrH"
   },
   "source": [
    "# 1. Preparation\n",
    "We need to first import the required library, download the data, and load the data into the memory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "swLpnk4u4XKo"
   },
   "source": [
    "## 1.1 Import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 394,
     "output_extras": [
      {
       "item_id": 5
      },
      {
       "item_id": 6
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 4509,
     "status": "ok",
     "timestamp": 1518570583550,
     "user": {
      "displayName": "Chuan Li",
      "photoUrl": "//lh3.googleusercontent.com/-ry1V_A03qTg/AAAAAAAAAAI/AAAAAAAAAh4/Ea41Em8n49A/s50-c-k-no/photo.jpg",
      "userId": "106476928809257656982"
     },
     "user_tz": 480
    },
    "id": "9_NbEq2FSFLe",
    "outputId": "58bb96d9-3ffa-4ccf-b90a-c2db87f537b0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Importing required packages...\n",
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     C:\\Users\\ChuanLi\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n",
      "[nltk_data] Downloading package wordnet to\n",
      "[nltk_data]     C:\\Users\\ChuanLi\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "print('Importing required packages...')\n",
    "\n",
    "from IPython.display import clear_output\n",
    "import re\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "np.random.seed()\n",
    "import matplotlib.pyplot as plt\n",
    "import nltk\n",
    "from nltk.tokenize import TweetTokenizer\n",
    "nltk.download('stopwords')\n",
    "from nltk.stem.wordnet import WordNetLemmatizer \n",
    "nltk.download('wordnet')\n",
    "from keras.preprocessing import sequence\n",
    "from keras.preprocessing import text as ktxt\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Embedding, GRU, Dropout\n",
    "from keras.layers.convolutional import Conv1D, MaxPooling1D\n",
    "from keras.callbacks import EarlyStopping\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import metrics\n",
    "\n",
    "\n",
    "def hint(message):\n",
    "    \"\"\"\n",
    "    erase previous ipynb output and show new message\n",
    "    \"\"\"\n",
    "    clear_output()\n",
    "    print(message)\n",
    "\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0oMhFbcOS7ch"
   },
   "source": [
    "## 1.2 Loading the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 153,
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 128,
     "status": "ok",
     "timestamp": 1518570590592,
     "user": {
      "displayName": "Chuan Li",
      "photoUrl": "//lh3.googleusercontent.com/-ry1V_A03qTg/AAAAAAAAAAI/AAAAAAAAAh4/Ea41Em8n49A/s50-c-k-no/photo.jpg",
      "userId": "106476928809257656982"
     },
     "user_tz": 480
    },
    "id": "C1nwDbfiS-vX",
    "outputId": "d69716f3-d89c-49a1-91d3-7cfe42286b4b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label distribution between training and validation set:\n",
      "           label     train  validation\n",
      "0          toxic  0.095170    0.098543\n",
      "1   severe_toxic  0.009988    0.010027\n",
      "2        obscene  0.052916    0.053078\n",
      "3         threat  0.003016    0.002914\n",
      "4         insult  0.049210    0.049977\n",
      "5  identity_hate  0.008875    0.008523\n"
     ]
    }
   ],
   "source": [
    "hint('loading data...')\n",
    "train = pd.read_csv('data/train.csv')\n",
    "train, valid = train_test_split(train, test_size=0.2)\n",
    "\n",
    "labels = [\n",
    "    'toxic', \n",
    "    'severe_toxic', \n",
    "    'obscene', \n",
    "    'threat', \n",
    "    'insult', \n",
    "    'identity_hate'\n",
    "]\n",
    "\n",
    "Ytr = train[labels].values\n",
    "Yva = valid[labels].values\n",
    "\n",
    "hint('Label distribution between training and validation set:')\n",
    "print(pd.DataFrame({\n",
    "    'label': labels,\n",
    "    'train': [np.mean(train[label]) for label in labels],\n",
    "    'validation' : [np.mean(valid[label]) for label in labels],\n",
    "}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KA8OAJOJWAoe"
   },
   "source": [
    "# 2. Pre-processing the Input\n",
    "There are many ways to pre-process the raw strings into valid input for the model. Here we will do it by building a dictionary with all the comments from the training set, mapping the words to their index in the dictionary, and pad/crop the resulting sequences so that they have the same length."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZmEOAxm4wdNn"
   },
   "source": [
    "## 2.1 Cleaning Input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 34,
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 127,
     "status": "ok",
     "timestamp": 1518570683382,
     "user": {
      "displayName": "Chuan Li",
      "photoUrl": "//lh3.googleusercontent.com/-ry1V_A03qTg/AAAAAAAAAAI/AAAAAAAAAh4/Ea41Em8n49A/s50-c-k-no/photo.jpg",
      "userId": "106476928809257656982"
     },
     "user_tz": 480
    },
    "id": "Q5jQUn7MxSud",
    "outputId": "73b337ab-1654-422d-cad2-a82158bd8880"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done\n"
     ]
    }
   ],
   "source": [
    "tkzr = TweetTokenizer(preserve_case=False)\n",
    "eng_stopwords = (\n",
    "    'what', 'which', 'who', 'whom', \n",
    "    'this', 'that', 'these', 'those', \n",
    "    'am', 'is', 'are', 'was', 'were', \n",
    "    'be', 'been', 'being', \n",
    "    'have', 'has', 'had', 'having', \n",
    "    'do', 'does', 'did', 'doing', \n",
    "    'a', 'an', 'the', \n",
    "    'and', 'but', 'if', 'or', \n",
    "    'because', 'as', 'until', 'while', \n",
    "    'of', 'at', 'by', 'for', 'with', \n",
    "    'about', 'against', 'between', \n",
    "    'into', 'through', 'during', 'before', 'after', \n",
    "    'above', 'below', 'to', 'from', \n",
    "    'up', 'down', 'in', 'out', 'on', 'off', \n",
    "    'over', 'under', 'again', 'further', \n",
    "    'then', 'once', 'here', \n",
    "    'there', 'when', 'where', 'why', \n",
    "    'how', 'all', 'any', 'both', 'each', \n",
    "    'few', 'more', 'most', 'other', 'some', \n",
    "    'such', 'no', 'nor', 'not', 'only', \n",
    "    'own', 'same', 'so', 'than', 'too', 'very', \n",
    "    'can', 'will', 'just', 'don', 'should', 'now'\n",
    ")\n",
    "lmtzr = WordNetLemmatizer()\n",
    "appos = {\n",
    "    \"aren't\" : \"are not\",\n",
    "    \"can't\" : \"cannot\",\n",
    "    \"couldn't\" : \"could not\",\n",
    "    \"didn't\" : \"did not\",\n",
    "    \"doesn't\" : \"does not\",\n",
    "    \"don't\" : \"do not\",\n",
    "    \"hadn't\" : \"had not\",\n",
    "    \"hasn't\" : \"has not\",\n",
    "    \"haven't\" : \"have not\",\n",
    "    \"he'd\" : \"he would\",\n",
    "    \"he'll\" : \"he will\",\n",
    "    \"he's\" : \"he is\",\n",
    "    \"i'd\" : \"I would\",\n",
    "    \"i'd\" : \"I had\",\n",
    "    \"i'll\" : \"I will\",\n",
    "    \"i'm\" : \"I am\",\n",
    "    \"isn't\" : \"is not\",\n",
    "    \"it's\" : \"it is\",\n",
    "    \"it'll\":\"it will\",\n",
    "    \"i've\" : \"I have\",\n",
    "    \"let's\" : \"let us\",\n",
    "    \"mightn't\" : \"might not\",\n",
    "    \"mustn't\" : \"must not\",\n",
    "    \"shan't\" : \"shall not\",\n",
    "    \"she'd\" : \"she would\",\n",
    "    \"she'll\" : \"she will\",\n",
    "    \"she's\" : \"she is\",\n",
    "    \"shouldn't\" : \"should not\",\n",
    "    \"that's\" : \"that is\",\n",
    "    \"there's\" : \"there is\",\n",
    "    \"they'd\" : \"they would\",\n",
    "    \"they'll\" : \"they will\",\n",
    "    \"they're\" : \"they are\",\n",
    "    \"they've\" : \"they have\",\n",
    "    \"we'd\" : \"we would\",\n",
    "    \"we're\" : \"we are\",\n",
    "    \"weren't\" : \"were not\",\n",
    "    \"we've\" : \"we have\",\n",
    "    \"what'll\" : \"what will\",\n",
    "    \"what're\" : \"what are\",\n",
    "    \"what's\" : \"what is\",\n",
    "    \"what've\" : \"what have\",\n",
    "    \"where's\" : \"where is\",\n",
    "    \"who'd\" : \"who would\",\n",
    "    \"who'll\" : \"who will\",\n",
    "    \"who're\" : \"who are\",\n",
    "    \"who's\" : \"who is\",\n",
    "    \"who've\" : \"who have\",\n",
    "    \"won't\" : \"will not\",\n",
    "    \"wouldn't\" : \"would not\",\n",
    "    \"you'd\" : \"you would\",\n",
    "    \"you'll\" : \"you will\",\n",
    "    \"you're\" : \"you are\",\n",
    "    \"you've\" : \"you have\",\n",
    "    \"'re\": \" are\",\n",
    "    \"wasn't\": \"was not\",\n",
    "    \"we'll\":\" will\",\n",
    "    \"didn't\": \"did not\"\n",
    "}\n",
    "\n",
    "def preprocess(comment):\n",
    "  \n",
    "    # credit to the author of this post:\n",
    "    # https://www.kaggle.com/jagangupta/stop-the-s-toxic-comments-eda\n",
    "\n",
    "    # remove special format\n",
    "    comment = re.sub('\\n\\t', '', comment)\n",
    "\n",
    "    # remove IP addresses\n",
    "    comment = re.sub('\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}', ' specipaddress ', comment)\n",
    "\n",
    "    # remove username\n",
    "    comment = re.sub(\"\\[\\[User.*\\]\", ' specusername ', comment)\n",
    "    comment = re.sub(\"\\[\\[User.*\\|\", ' specusername ', comment)\n",
    "\n",
    "    # tokenization \n",
    "    tokens = tkzr.tokenize(comment)\n",
    "\n",
    "    # aphostophe replacement\n",
    "    tokens = [ appos[token] if token in appos else token for token in tokens]\n",
    "\n",
    "    # remove stopwords\n",
    "    tokens = [ token for token in tokens if not token in eng_stopwords ]\n",
    "\n",
    "    # stemming\n",
    "    tokens = [ lmtzr.lemmatize(token, 'v') for token in tokens]\n",
    "\n",
    "    return \" \".join(tokens)\n",
    "  \n",
    "\n",
    "hint('Cleaning train set...')\n",
    "Xtr = train['comment_text'].apply(lambda c: preprocess(c))\n",
    "hint('Cleaning test set...')\n",
    "Xva = valid['comment_text'].apply(lambda c: preprocess(c))\n",
    "hint('Done')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "btLphIdFwmts"
   },
   "source": [
    "## 2.2 Transforming Comments to Sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 34,
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 144,
     "status": "ok",
     "timestamp": 1518570698059,
     "user": {
      "displayName": "Chuan Li",
      "photoUrl": "//lh3.googleusercontent.com/-ry1V_A03qTg/AAAAAAAAAAI/AAAAAAAAAh4/Ea41Em8n49A/s50-c-k-no/photo.jpg",
      "userId": "106476928809257656982"
     },
     "user_tz": 480
    },
    "id": "JmEQQZs-kHK2",
    "outputId": "073af561-39f5-4c18-acee-ab2794e9e3a6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done\n"
     ]
    }
   ],
   "source": [
    "vocab_max = 20000\n",
    "\n",
    "hint('Fitting the tokenizer...')\n",
    "tokenizer = ktxt.Tokenizer(num_words=vocab_max)\n",
    "tokenizer.fit_on_texts(Xtr)\n",
    "\n",
    "hint('Tokenizing...')\n",
    "Xtr = tokenizer.texts_to_sequences(Xtr)\n",
    "Xva = tokenizer.texts_to_sequences(Xva)\n",
    "\n",
    "hint('padding the sequences...')\n",
    "max_comment_length = 200  # padded/cropped comment length\n",
    "Xtr = sequence.pad_sequences(Xtr, maxlen=max_comment_length)\n",
    "Xva = sequence.pad_sequences(Xva, maxlen=max_comment_length)\n",
    "\n",
    "hint('Done')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OWHeGxuz3Wfn"
   },
   "source": [
    "# 3. Training Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 374,
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 389,
     "status": "ok",
     "timestamp": 1518570698678,
     "user": {
      "displayName": "Chuan Li",
      "photoUrl": "//lh3.googleusercontent.com/-ry1V_A03qTg/AAAAAAAAAAI/AAAAAAAAAh4/Ea41Em8n49A/s50-c-k-no/photo.jpg",
      "userId": "106476928809257656982"
     },
     "user_tz": 480
    },
    "id": "GoawhjBA3nyc",
    "outputId": "e814cec6-a98e-4cce-da53-d2579963c70f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, 200, 100)          2000000   \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 200, 100)          0         \n",
      "_________________________________________________________________\n",
      "conv1d_1 (Conv1D)            (None, 200, 64)           19264     \n",
      "_________________________________________________________________\n",
      "max_pooling1d_1 (MaxPooling1 (None, 100, 64)           0         \n",
      "_________________________________________________________________\n",
      "gru_1 (GRU)                  (None, 32)                9312      \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 16)                528       \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 6)                 102       \n",
      "=================================================================\n",
      "Total params: 2,029,206\n",
      "Trainable params: 2,029,206\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()\n",
    "model.add(Embedding(vocab_max, 100, input_length=max_comment_length))\n",
    "model.add(Dropout(0.2))\n",
    "model.add(Conv1D(filters=64, kernel_size=3, padding='same', activation='relu'))\n",
    "model.add(MaxPooling1D(pool_size=2))\n",
    "model.add(GRU(units=32))\n",
    "model.add(Dense(16, activation='relu'))\n",
    "model.add(Dense(len(labels), activation='sigmoid'))\n",
    "model.compile(\n",
    "    optimizer='adam', \n",
    "    loss='binary_crossentropy', \n",
    "    metrics=['accuracy']\n",
    ")\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1S6kDCqDFGUN"
   },
   "source": [
    "Now training the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 122,
     "output_extras": [
      {
       "item_id": 373
      },
      {
       "item_id": 1155
      },
      {
       "item_id": 1418
      },
      {
       "item_id": 2339
      },
      {
       "item_id": 2340
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 630277,
     "status": "ok",
     "timestamp": 1518571329571,
     "user": {
      "displayName": "Chuan Li",
      "photoUrl": "//lh3.googleusercontent.com/-ry1V_A03qTg/AAAAAAAAAAI/AAAAAAAAAh4/Ea41Em8n49A/s50-c-k-no/photo.jpg",
      "userId": "106476928809257656982"
     },
     "user_tz": 480
    },
    "id": "QVU9nuDzFtlf",
    "outputId": "e0ea2b97-fe62-42bf-a1b2-46c7d5f08532"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 127656 samples, validate on 31915 samples\n",
      "Epoch 1/2\n",
      "127656/127656 [==============================] - 164s 1ms/step - loss: 0.0674 - acc: 0.9765 - val_loss: 0.0493 - val_acc: 0.9819\n",
      "Epoch 2/2\n",
      "127656/127656 [==============================] - 160s 1ms/step - loss: 0.0427 - acc: 0.9838 - val_loss: 0.0469 - val_acc: 0.9826\n"
     ]
    }
   ],
   "source": [
    "epochs = 2\n",
    "batch_size = 64\n",
    "\n",
    "history = model.fit(\n",
    "    Xtr, Ytr, \n",
    "    epochs=epochs, \n",
    "    batch_size=batch_size,\n",
    "    validation_data=(Xva, Yva)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nAFiq9QuFI7R"
   },
   "source": [
    "Making prediction on the validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 34,
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 121,
     "status": "ok",
     "timestamp": 1518571356080,
     "user": {
      "displayName": "Chuan Li",
      "photoUrl": "//lh3.googleusercontent.com/-ry1V_A03qTg/AAAAAAAAAAI/AAAAAAAAAh4/Ea41Em8n49A/s50-c-k-no/photo.jpg",
      "userId": "106476928809257656982"
     },
     "user_tz": 480
    },
    "id": "hgiIMLjZE3Rr",
    "outputId": "f00c0add-a7ab-43f7-b75d-c47a54e12890"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Done\n"
     ]
    }
   ],
   "source": [
    "hint(\"Making prediction...\")\n",
    "Yva_ = model.predict(Xva)\n",
    "hint(\"Done\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RfRNkLv7v04I"
   },
   "source": [
    "# 4. Result Analysis\n",
    "## 4.1 Global Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 255,
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 275,
     "status": "ok",
     "timestamp": 1518571356419,
     "user": {
      "displayName": "Chuan Li",
      "photoUrl": "//lh3.googleusercontent.com/-ry1V_A03qTg/AAAAAAAAAAI/AAAAAAAAAh4/Ea41Em8n49A/s50-c-k-no/photo.jpg",
      "userId": "106476928809257656982"
     },
     "user_tz": 480
    },
    "id": "vakvDUO31-oo",
    "outputId": "5b7acf10-a987-41f7-801c-0f291245b687"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "validation set sample count: 31915\n",
      "\n",
      "accuracy for threshold 0.1: 96.55%\n",
      "accuracy for threshold 0.2: 97.59%\n",
      "accuracy for threshold 0.3: 98.00%\n",
      "accuracy for threshold 0.4: 98.19%\n",
      "accuracy for threshold 0.5: 98.26%\n",
      "accuracy for threshold 0.6: 98.26%\n",
      "accuracy for threshold 0.7: 98.17%\n",
      "accuracy for threshold 0.8: 98.00%\n",
      "accuracy for threshold 0.9: 97.66%\n",
      "\n",
      "best threshold: 0.6\n",
      "best accuracy: 98.26%\n"
     ]
    }
   ],
   "source": [
    "total_sample = Xva.shape[0]\n",
    "print(\"validation set sample count: %d\\n\" % total_sample)\n",
    "prediction_total = total_sample*Yva.shape[1]\n",
    "best_t = None\n",
    "best_accuracy = 0\n",
    "for t in [i*0.1 for i in range(1, 10)]:\n",
    "    accuracy = np.sum(Yva == (Yva_ >= t))/prediction_total\n",
    "    if accuracy > best_accuracy: \n",
    "        best_t = t\n",
    "        best_accuracy = accuracy\n",
    "    print(\"accuracy for threshold %.1f: %.2f%%\" % (t, accuracy*100))\n",
    "Yva_T = Yva_ >= best_t\n",
    "correct = Yva == Yva_T\n",
    "print(\"\\nbest threshold: %.1f\" % best_t)\n",
    "print(\"best accuracy: %.2f%%\" % (best_accuracy*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YSptfDNc58dZ"
   },
   "source": [
    "## 4.2 Accuracy by Classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "gLRH9AayowXA"
   },
   "outputs": [],
   "source": [
    "overview = pd.DataFrame(index=[\n",
    "    'label‰ of all',\n",
    "    'total wrong', \n",
    "    'P->N', \n",
    "    'N->P', \n",
    "    'P->N %', \n",
    "    'N->P %',\n",
    "    'avg len',\n",
    "])\n",
    "\n",
    "def analyze_class(i):\n",
    "    wrong = valid[correct[:, i] != 1]\n",
    "    total_class_error = len(wrong)\n",
    "    print(\"%d predicted incorrectly (%.2f%% of all samples)\" % (\n",
    "        total_class_error, \n",
    "        100*total_class_error/total_sample\n",
    "    ))\n",
    "        \n",
    "    wrong_seqs = Xva[correct[:, i] != 1]\n",
    "    lens = [ len(seq[seq != 0]) for seq in wrong_seqs]\n",
    "    avg_len = np.mean(lens)\n",
    "    print(\"Falsely predicted sequences have an average length of %d\" % avg_len)\n",
    "\n",
    "    PpN = valid[(Yva[:, i] == 1) & (Yva_T[:, i] == 0)]\n",
    "    PpN_count = len(PpN)\n",
    "    print(\"\\n%d (%.2f%%) positive label were predicted to be negative\" % (\n",
    "        PpN_count, \n",
    "        100*PpN_count/total_class_error \n",
    "    ))\n",
    "    if PpN_count > 4:\n",
    "        print(\"Samples:\")\n",
    "        for sample in PpN.sample(5)['comment_text']:\n",
    "            display(sample)\n",
    "  \n",
    "    NpP = valid[(Yva[:, i] == 0) & (Yva_T[:, i] == 1)]\n",
    "    NpP_count = len(NpP)\n",
    "    print(\"\\n%d (%.2f%%) negative label were predicted to be positive\" % (\n",
    "        NpP_count, \n",
    "        100*NpP_count/total_class_error \n",
    "    ))\n",
    "    if NpP_count > 4:\n",
    "        print(\"Samples:\")\n",
    "        for sample in NpP.sample(5)['comment_text']:\n",
    "            display(sample)\n",
    "  \n",
    "    overview[labels[i]] = [\n",
    "        np.mean(Yva[:, i]*1000),\n",
    "        total_class_error, \n",
    "        PpN_count,  \n",
    "        NpP_count,\n",
    "        100*PpN_count/total_class_error,\n",
    "        100*NpP_count/total_class_error,\n",
    "        avg_len\n",
    "    ]\n",
    "  \n",
    "    print('\\n')\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SSY9H7knpE5U"
   },
   "source": [
    "### 4.2.1 Toxic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 360,
     "output_extras": [
      {
       "item_id": 1
      },
      {
       "item_id": 2
      },
      {
       "item_id": 3
      },
      {
       "item_id": 4
      },
      {
       "item_id": 5
      },
      {
       "item_id": 6
      },
      {
       "item_id": 7
      },
      {
       "item_id": 8
      },
      {
       "item_id": 9
      },
      {
       "item_id": 10
      },
      {
       "item_id": 11
      },
      {
       "item_id": 12
      },
      {
       "item_id": 13
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 251,
     "status": "ok",
     "timestamp": 1518573461778,
     "user": {
      "displayName": "Chuan Li",
      "photoUrl": "//lh3.googleusercontent.com/-ry1V_A03qTg/AAAAAAAAAAI/AAAAAAAAAh4/Ea41Em8n49A/s50-c-k-no/photo.jpg",
      "userId": "106476928809257656982"
     },
     "user_tz": 480
    },
    "id": "NNGw690DpLMy",
    "outputId": "18e4b55e-63d1-42b2-9c93-4b75a8d5ad39"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1208 predicted incorrectly (3.79% of all samples)\n",
      "Falsely predicted sequences have an average length of 34\n",
      "\n",
      "899 (74.42%) positive label were predicted to be negative\n",
      "Samples:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"Prior to Quickpolls, he would have been perma-blocked by now. Guess I'll remove the block again. As soon as I get time, which will probably be about November. If you've got any brains, you will let that particular troll sleep on. Best.\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'Your merge was worthless; You basically removed everything in the article about the actual games.'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'u accuse me of that goodfellas thing again and i will hunt you down!'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'It also could forward to male homosexual orgies, of which there surely are several relevant articles. Of course, if someone finds and RS to figure out which is most relevant, that would help, i.e., all male Daisy chains, Bukkake, Gang bang, etc. per various sources better than urban dictionary or lots of porn sites. Jolly times!   (Talkie-Talkie)'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'Tony, you are unclear, unless your opinion is that the RfC was rubbish, and you have no faults what so ever.  The short quips and arguments from you on the RfC are hardly helpful.  Why is it so damn hard for you to simply answer a freaking question straight?  Are you really that incapable of dialog?'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "309 (25.58%) negative label were predicted to be positive\n",
      "Samples:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'\"\\n\\nLucky...\\nI\\'m going through the images, and you are damn lucky this image does not say: \"\"DVD still\"\" like  does...\"'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'\"\\n\\nMessage\\n\\nHey Coolrunner87, you tedious little shit. Why don\\'t you just grow a pair and stop wasting everyone\\'s time with your whiny posturing about \"\"vandalism\"\"? Fact is, RD Reynolds is a vile mound of goo and puss and is undeserving of a wikipedia entry. Maybe you should go and start an \"\"IWC wiki\"\" or some suchlike, and put all your overweight friends in there? Wikipedia is supposed to be about serious issues like politics, science, geography etc, not some morbidly obese basement dweller who makes up crap about professional wrestling! His books have no sources and contain more errors per page than there are zits on your face.   \"'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'I do the cha cha like a faggot.'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'my dear winner, you lunatic remarks ab the serious points I was discussing makes you worthless as an editor\\nand esp to call them vandalism ... you have some kind of moronic deviance that ill suits you to do anything on Wikipedia\\nmany thanks for your leaving all edits alone in future with such idiotic diatribes ... thanks again'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'\"\\n\\n Abusive Admins \\n\\nI was recently banned from wikipedia by  Crum falsely claimed multiple times that I was banned for using the \"\"epithet\"\" \"\"liar\"\".  This is verifiably false.  In none\\nof\\nmy\\nposts did I use the \"\"epithet\"\" \"\"liar\"\".  \\n\\nYes, I did say that SlimVirgin lies.  My defense for making such an accusation is that in the immediate moments before I made that post I saw her lie about me.  But I didn\\'t call her a liar.  I merely said she lies.  This happens to be true and is one of the many reasons SlimVirgin was  voted \"\"wikipedia\\'s most abusive admin\"\".  Crum\\'s accusation against me was false and his ban was baseless and abusive.   \"'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "analyze_class(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3cxSNKFepEg1"
   },
   "source": [
    "### 4.2.2 Severe Toxic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 360,
     "output_extras": [
      {
       "item_id": 1
      },
      {
       "item_id": 2
      },
      {
       "item_id": 3
      },
      {
       "item_id": 4
      },
      {
       "item_id": 5
      },
      {
       "item_id": 6
      },
      {
       "item_id": 7
      },
      {
       "item_id": 8
      },
      {
       "item_id": 9
      },
      {
       "item_id": 10
      },
      {
       "item_id": 11
      },
      {
       "item_id": 12
      },
      {
       "item_id": 13
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 254,
     "status": "ok",
     "timestamp": 1518573463811,
     "user": {
      "displayName": "Chuan Li",
      "photoUrl": "//lh3.googleusercontent.com/-ry1V_A03qTg/AAAAAAAAAAI/AAAAAAAAAh4/Ea41Em8n49A/s50-c-k-no/photo.jpg",
      "userId": "106476928809257656982"
     },
     "user_tz": 480
    },
    "id": "1cCGMbZKpV1H",
    "outputId": "428d69ea-64a9-4f28-e09c-f9ddd6d7516c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "303 predicted incorrectly (0.95% of all samples)\n",
      "Falsely predicted sequences have an average length of 33\n",
      "\n",
      "279 (92.08%) positive label were predicted to be negative\n",
      "Samples:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'fuk u retard i hate u i will kill u bettr watch ur bak cuz im teamin up wit my mothafuka boothy and we gonna rise the south on yo ass bitch'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'About fucking time you retarded little bitch!'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'YOURE A FRREAKING JEW, AND PLEAZE COMMIT SUICIDE.'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'fag \\n\\nyou fucking suck. people need to know the true meaning of thumbing'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'Fuck off!!!!- () Wolfkeeper (Talk)'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "24 (7.92%) negative label were predicted to be positive\n",
      "Samples:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'FUCK YOU I NEVER STOP FUCKING YOU HARRY POTTER FANS!!!!!!! HARRY POTTER SUCKZ! this fucking porn face!!!'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'I should try to track down your IP so I can burn your house down you irritating and loathesome little shit. Who the fuck who dyou think you are? I wish I knew who yo uwere so I could beat the everliving shit out of you, you fucking whiny, annoying loser. Stop watching STar Trek re-runs and get laid you fucking pussy ass pansy. I hope you fucking get AIDS and cancer at the same time.'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'Get off my fucking back.. You God-damn little Indian RETARD \\n\\nGet off my fucking back.. You God-damn little Indian RETARD. Go and bully around someone else. AND you can shove up your SPI up you Indian ASS.'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'Fuck off Mexican bastard! I will work endelessly to have your Wikipedia account banned indefinitely- you have made my life a misery and I hope you die.'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'Hey fucking retard?==\\n\\nI was deleting my own post. Quit sending me fucking sandbox warnings you fucking retard.\\n\\n=='"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "analyze_class(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "s0_zIe7m1A79"
   },
   "source": [
    "### 4.2.3 Obscene"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 360,
     "output_extras": [
      {
       "item_id": 1
      },
      {
       "item_id": 2
      },
      {
       "item_id": 3
      },
      {
       "item_id": 4
      },
      {
       "item_id": 5
      },
      {
       "item_id": 6
      },
      {
       "item_id": 7
      },
      {
       "item_id": 8
      },
      {
       "item_id": 9
      },
      {
       "item_id": 10
      },
      {
       "item_id": 11
      },
      {
       "item_id": 12
      },
      {
       "item_id": 13
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 258,
     "status": "ok",
     "timestamp": 1518573465290,
     "user": {
      "displayName": "Chuan Li",
      "photoUrl": "//lh3.googleusercontent.com/-ry1V_A03qTg/AAAAAAAAAAI/AAAAAAAAAh4/Ea41Em8n49A/s50-c-k-no/photo.jpg",
      "userId": "106476928809257656982"
     },
     "user_tz": 480
    },
    "id": "Qt9vOoNK1EFl",
    "outputId": "cfeec26f-71ad-4668-840e-c183a862ba74"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "568 predicted incorrectly (1.78% of all samples)\n",
      "Falsely predicted sequences have an average length of 36\n",
      "\n",
      "408 (71.83%) positive label were predicted to be negative\n",
      "Samples:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'George Remus \\n\\nGeorge Remus is not a fictional character. Do some FUCKING research and you will see he was a real person. A fictional book was written about him but that does not make Remus, himself, fictional.'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'Nipple \\n\\nNipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipple Nipp'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'I DID F*** ALL HERE SO F*** OF WIKIPEDIA YOU ARROGANT SELF RIGHTEOUS C***S'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "\"Ja, final warnings. Lived with 'em for decades. You're like the Rainbow Likudis at the Pentagon. Go get a colonoscopy. Your crap stinks.\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "\"Go to Hell, fatso \\n\\nHey, Dickwad! If you ignore me, I'll poke a hole in you and release all the fat! I WILL NOT hesitate to roast you in an oven like the little piggy you are! Unblock everyone you've blocked or suffer the consequences!!!\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "160 (28.17%) negative label were predicted to be positive\n",
      "Samples:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"Yes you stupid asshole, it does count. Just because it's not mentioned on the screen does not mean it's canon. Like WyrmKing said earlier, Nathan Petrelli's sons' names weren't mentioned on the script but the credits showed their names. Does that mean, they didn't have names? 800 series is T-800. Merchan-fucking-dise says it's T-800. The writers says it's T-800. The creature desginer, Stan Winston says it's T-800. It's fucking T-800. T-fucking-800.\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'I agree \\n\\nDayewalker is just another republicant whore who worships Palin McCain'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'that dumbass you got there didnt improve anythign all he did was try and steal my work and block my from it because hekept sayign i had fake stuff which all came from other places and wiki its self so you tell him to go to hell'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "\"Absurd claims \\n\\nStop, you dimwit. Your accusations and insinuations may be well-intentioned but you are clueless. Sure, be bold but don't be fucking stupid. -\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "\"SHUT UP AND LEAVE ME ALONE  \\n\\nFAGGOT! SHUT UP. GO AHEAD BAN ME I DON'T CARE. JUST SHUT UP AND LEAVE MY ALONE.\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "analyze_class(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Lwpco7aB1GrZ"
   },
   "source": [
    "### 4.2.4 Threat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 258,
     "output_extras": [
      {
       "item_id": 1
      },
      {
       "item_id": 2
      },
      {
       "item_id": 3
      },
      {
       "item_id": 4
      },
      {
       "item_id": 5
      },
      {
       "item_id": 6
      },
      {
       "item_id": 7
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 257,
     "status": "ok",
     "timestamp": 1518573466922,
     "user": {
      "displayName": "Chuan Li",
      "photoUrl": "//lh3.googleusercontent.com/-ry1V_A03qTg/AAAAAAAAAAI/AAAAAAAAAh4/Ea41Em8n49A/s50-c-k-no/photo.jpg",
      "userId": "106476928809257656982"
     },
     "user_tz": 480
    },
    "id": "gQgcRgOq1GLO",
    "outputId": "69893ed6-1588-41db-eff3-e86766fac8e6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "93 predicted incorrectly (0.29% of all samples)\n",
      "Falsely predicted sequences have an average length of 29\n",
      "\n",
      "93 (100.00%) positive label were predicted to be negative\n",
      "Samples:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'HOPE YOUR HEAD GETS CUT OFF AND SOMEONE WIPS THERE ASS WITH IT AND THEN STABS YOU IN YOUR HEART'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "\"you are part of isis \\n\\nCouple of days ago u threaten me to give u Ebola and if I don't you were going to destroy the white house\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'you just wait \\n\\nyour death is near'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'I am going to kill you I am going to murder you'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "\"Who gives a shit? Anthony Bourdain is a tool. I just wish he had children so I could stomp on there testicals. Just kidding but he is a massive tool. He couldn't carry Emerils jockstrap. Next time I see him I'll knock his teeth out....\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "0 (0.00%) negative label were predicted to be positive\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "analyze_class(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6lkRScLJ1fGN"
   },
   "source": [
    "### 4.2.5 Insult"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 360,
     "output_extras": [
      {
       "item_id": 1
      },
      {
       "item_id": 2
      },
      {
       "item_id": 3
      },
      {
       "item_id": 4
      },
      {
       "item_id": 5
      },
      {
       "item_id": 6
      },
      {
       "item_id": 7
      },
      {
       "item_id": 8
      },
      {
       "item_id": 9
      },
      {
       "item_id": 10
      },
      {
       "item_id": 11
      },
      {
       "item_id": 12
      },
      {
       "item_id": 13
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 270,
     "status": "ok",
     "timestamp": 1518573480301,
     "user": {
      "displayName": "Chuan Li",
      "photoUrl": "//lh3.googleusercontent.com/-ry1V_A03qTg/AAAAAAAAAAI/AAAAAAAAAh4/Ea41Em8n49A/s50-c-k-no/photo.jpg",
      "userId": "106476928809257656982"
     },
     "user_tz": 480
    },
    "id": "zJ2VKxgK1hRC",
    "outputId": "68a3f353-831b-426f-85dc-9d65a39561cd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "890 predicted incorrectly (2.79% of all samples)\n",
      "Falsely predicted sequences have an average length of 29\n",
      "\n",
      "664 (74.61%) positive label were predicted to be negative\n",
      "Samples:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"oh my bolagne has a first name it's d-u-m-b, my bbolange has a second name it's s-h-i-t.\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "\"Oh cool, you deleted other part, what is next? You will delete the whole article too? Good job! Can block me again, it's just that you can do. I-D-I-O-T\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'In German and European law you are the fool. Private datas are the property of the individual person.\\nYou want to force somebody to open his personal datas. That is a violation of german data protection law. MP'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'hey jerk \\n\\ngo write about gardening or britney spears or toby keith and FUCKING stop ruining articles on topics you have no damn clue about.'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'what up, my comma deleting nigga?'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "226 (25.39%) negative label were predicted to be positive\n",
      "Samples:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'GO F**K YOURSELF TOO! LIKE I GIVE A SHIT ABOUT A BAN!'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "\"By the way, I see you can't help but whine to others about how my words are affecting the last few brain cells you seem to have held on to. Fuckin' pussy.\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'\"\\n\\nYou are an open source patent buster. You attacked the Collins patents FIRST, right? I give you facts and you spew politics. I don\\'t have to stand for that. You are being stupid. Stupidity destroys. You are one idiot of many. You are worthless. Apologize for that and I will for mine. But I still hate you. That is a solid fact. There is something wrong with people like you, but that\\'s your problem.   \\n\\nAlso, unlike Hod Lipson\\'s bigoted crazy comments on how \"\"legal immigrants doing all the work\"\" (meaning dirty work as he had referred to) you, Feitas, Merkle Roco (who RUNS the ENTIRE nano department at NSF and he\\'s an X communist scientist!) are taking our GOOD jobs, in fact STEALING THEM as they were stolen from me and as they are stealing the Collins patent.   \\n\\n... Oh, and you said you came here \"\"15 years ago to do work that no American was able to do.\"\" What a pompous ass! You think you are BETTER than ANY American living here? Go to hell!   \"'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'\"\\nFuck Wikipedia and fuck Wales. The article cannot be more of a perfect case of violating WP:BLPCRIME. If I were a defense attorney I would be wetting my tiny white pants. Wales is a media whore. It suits him to have that article here. Chan \"'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "\"Absurd claims \\n\\nStop, you dimwit. Your accusations and insinuations may be well-intentioned but you are clueless. Sure, be bold but don't be fucking stupid. -\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "analyze_class(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ybjmae651jxY"
   },
   "source": [
    "### 4.2.6 Identity Hate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 258,
     "output_extras": [
      {
       "item_id": 1
      },
      {
       "item_id": 2
      },
      {
       "item_id": 3
      },
      {
       "item_id": 4
      },
      {
       "item_id": 5
      },
      {
       "item_id": 6
      },
      {
       "item_id": 7
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 440,
     "status": "ok",
     "timestamp": 1518573482230,
     "user": {
      "displayName": "Chuan Li",
      "photoUrl": "//lh3.googleusercontent.com/-ry1V_A03qTg/AAAAAAAAAAI/AAAAAAAAAh4/Ea41Em8n49A/s50-c-k-no/photo.jpg",
      "userId": "106476928809257656982"
     },
     "user_tz": 480
    },
    "id": "6ojSnXP51jPm",
    "outputId": "d1ee8ad8-5407-43c8-cab2-a5f77112fa2c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "272 predicted incorrectly (0.85% of all samples)\n",
      "Falsely predicted sequences have an average length of 26\n",
      "\n",
      "272 (100.00%) positive label were predicted to be negative\n",
      "Samples:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'\"\\n\\nGreat, the Ukrainian nationalists again. go back to your filthy hole, UPA dog! Fedor grew up in Russia, he only was in Ukraine for 2 yrs, he has a Russian passport. end of story. stop claiming what doesn\\'t belong to you! now you bastards say that ancient RUS dukes were Ukrainian! You say that ancient Rus itself was Ukraine! oh, yeah, the whole world is Ukraine! keep telling yourself that, fucking Orange asshole! I am half-Ukrainian myself, people like you just piss me off! oh, and no, his surname is NOT \"\"100% Ukrainian\"\"! You know how many people in Krasnodar, Stavropolie, and other areas of RUSSIA, have last names that end with \"\"-enko\"\"? millions! myself included (Levchenko). fucking uneducated banderovitz nazi, you have no place here, go, save your antisemitic, suppremacist theories for others (hint, maybe your brothers in Poltava will listen to your preachings). You people are the reason for all the hatred between Russians and Ukrainians these days! its people like you that should have perished in the Holodomor, seriously!  \"'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'HOE \\n\\nYour A slut hoe basstatard homo rapeist'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'\"HOLY SHIT I\"\"M GAY AS HELL.\\n\\n\"'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "\"Jews and Jews \\n\\nCan we (meaning male whites) truely co-exist with Jews? I mean niggers and spics are bad enough but the fucking jews go too far. I think there should be a genocide, and there should be an article on wikipedia that encourages the death of jews, for those who agree please message here, for those that don't, eat me. We Came In?\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'\"\\n\\nThat is not a insult. That is a reality, that link is a lint to a racist group. And armenian grow up  their children with racism. They send their children to \"\"genocide camps\"\" and teach them how barbarian Turks are making genocide(!). If you dont believe me please ask an Armenian user.  \"'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "0 (0.00%) negative label were predicted to be positive\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "analyze_class(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GYOBKlWc12Rc"
   },
   "source": [
    "### 4.2.7 Overview"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "base_uri": "https://localhost:8080/",
     "height": 235,
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 247,
     "status": "ok",
     "timestamp": 1518573484262,
     "user": {
      "displayName": "Chuan Li",
      "photoUrl": "//lh3.googleusercontent.com/-ry1V_A03qTg/AAAAAAAAAAI/AAAAAAAAAh4/Ea41Em8n49A/s50-c-k-no/photo.jpg",
      "userId": "106476928809257656982"
     },
     "user_tz": 480
    },
    "id": "w9g39VJ4n00f",
    "outputId": "8e4f9efb-795c-4253-de18-d7b2be471bb0"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>toxic</th>\n",
       "      <th>severe_toxic</th>\n",
       "      <th>obscene</th>\n",
       "      <th>threat</th>\n",
       "      <th>insult</th>\n",
       "      <th>identity_hate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>label‰ of all</th>\n",
       "      <td>98</td>\n",
       "      <td>10</td>\n",
       "      <td>53</td>\n",
       "      <td>2</td>\n",
       "      <td>49</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>total wrong</th>\n",
       "      <td>1208</td>\n",
       "      <td>303</td>\n",
       "      <td>568</td>\n",
       "      <td>93</td>\n",
       "      <td>890</td>\n",
       "      <td>272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P-&gt;N</th>\n",
       "      <td>899</td>\n",
       "      <td>279</td>\n",
       "      <td>408</td>\n",
       "      <td>93</td>\n",
       "      <td>664</td>\n",
       "      <td>272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N-&gt;P</th>\n",
       "      <td>309</td>\n",
       "      <td>24</td>\n",
       "      <td>160</td>\n",
       "      <td>0</td>\n",
       "      <td>226</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>P-&gt;N %</th>\n",
       "      <td>74</td>\n",
       "      <td>92</td>\n",
       "      <td>71</td>\n",
       "      <td>100</td>\n",
       "      <td>74</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N-&gt;P %</th>\n",
       "      <td>25</td>\n",
       "      <td>7</td>\n",
       "      <td>28</td>\n",
       "      <td>0</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>avg len</th>\n",
       "      <td>34</td>\n",
       "      <td>33</td>\n",
       "      <td>36</td>\n",
       "      <td>29</td>\n",
       "      <td>29</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               toxic  severe_toxic  obscene  threat  insult  identity_hate\n",
       "label‰ of all     98            10       53       2      49              8\n",
       "total wrong     1208           303      568      93     890            272\n",
       "P->N             899           279      408      93     664            272\n",
       "N->P             309            24      160       0     226              0\n",
       "P->N %            74            92       71     100      74            100\n",
       "N->P %            25             7       28       0      25              0\n",
       "avg len           34            33       36      29      29             26"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "overview.astype(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EdjD2HB1263p"
   },
   "source": [
    "# 5. Observation and Conclusion\n",
    "\n",
    "\n",
    "*   The classes that have more positive labels performed better. Giving these rare labels more weight may solve this.\n",
    "*   False negative is more common than false positive, although the severity of this problem varies among the classes.\n",
    "*   Quotes could be a potential cause of error (need further test).\n",
    "*   It seems that the model cannot differentiate those comments that are filled with toxic content and those that have a small portion of it. Utilizing TFIDF may solve this.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "default_view": {},
   "name": "RCNN performance analysis.ipynb",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}