3. Mood Detection of Tweets - Word Embeddings and LSTM.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Mood Detection of Tweets - Word Embeddings and LSTM\n",
    "\n",
    "Now that we have our cleaned dataset, we will now continue to train a neural network to classify a tweet's mood."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing for CUDA-enabled TF with GPU Support\n",
    "\n",
    "**What this means:**\n",
    "Deep Learning requires a lot of matrix calculations and your CPU is not really meant for that sort of task. However, graphics cards (the hardware that your computer uses to make games run well) are able to perform these sort of tasks very well, so we need to check if the Jupyter Notebook has access to your (Nvidia) GPU, in order to train the model much more efficiently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Default GPU Device: /device:GPU:0\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "if tf.test.gpu_device_name():\n",
    "    print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))\n",
    "else:\n",
    "    print(\"Please install GPU version of TF\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "2.2.4-tf\n",
      "1.13.1\n"
     ]
    }
   ],
   "source": [
    "print(tf.test.is_built_with_cuda())\n",
    "print(tf.keras.__version__)\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Processing our mood tweets dataset\n",
    "\n",
    "Dataset can also be accessed at: [future Kaggle link for my dataset] \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading the CSV dataset with pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>tweet</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>suicidal</td>\n",
       "      <td>\"He was lost &amp; scared\" says a Newport woman ab...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>suicidal</td>\n",
       "      <td>@TheDumbMedico Ameen. Or ye suicide wali bat a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>suicidal</td>\n",
       "      <td>You showed the leak in @HouseofCommons today @...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>suicidal</td>\n",
       "      <td>The number in my bio is for a Suicide hotline ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>suicidal</td>\n",
       "      <td>cw: suicidal ideation it's unsurprisingly hard...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      class                                              tweet\n",
       "0  suicidal  \"He was lost & scared\" says a Newport woman ab...\n",
       "1  suicidal  @TheDumbMedico Ameen. Or ye suicide wali bat a...\n",
       "2  suicidal  You showed the leak in @HouseofCommons today @...\n",
       "3  suicidal  The number in my bio is for a Suicide hotline ...\n",
       "4  suicidal  cw: suicidal ideation it's unsurprisingly hard..."
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(\"datasets/mood_tweets.csv\")\n",
    "df.drop(df.columns[0], inplace=True, axis=1)\n",
    "df[\"tweet\"] = df[\"tweet\"].astype(str)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>tweet</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>25758</td>\n",
       "      <td>25758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>6</td>\n",
       "      <td>25758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>cheerful</td>\n",
       "      <td>@JackieDP @LandSheriffs Have you done your acc...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>5885</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           class                                              tweet\n",
       "count      25758                                              25758\n",
       "unique         6                                              25758\n",
       "top     cheerful  @JackieDP @LandSheriffs Have you done your acc...\n",
       "freq        5885                                                  1"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's analyse how the dataset is lain out and roughly how the data looks like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>tweet</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>18115</th>\n",
       "      <td>cheerful</td>\n",
       "      <td>You know what's awesome? After a month of doing yoga I'm getting most of my range of motion and flexibility back in the knee that I blew out playing rugby in college &amp; university. I honestly wish I had started this years ago.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4418</th>\n",
       "      <td>cheerful</td>\n",
       "      <td>Just rolled over 10k awesome miles in @corbett and my @Tesla 3. @elonmusk one suggestion: can I plz haz headlights on by default? Every time I drive during the day they start off and I think they improve visibility for other drivers as I pass them.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4417</th>\n",
       "      <td>cheerful</td>\n",
       "      <td>Glad to see that Craig continues to keep the orientalism industry alive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4416</th>\n",
       "      <td>cheerful</td>\n",
       "      <td>@_Kazma8 No problem dude. Glad she was found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4415</th>\n",
       "      <td>cheerful</td>\n",
       "      <td>Sign up for this awesome contest!</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          class  \\\n",
       "18115  cheerful   \n",
       "4418   cheerful   \n",
       "4417   cheerful   \n",
       "4416   cheerful   \n",
       "4415   cheerful   \n",
       "\n",
       "                                                                                                                                                                                                                                                          tweet  \n",
       "18115  You know what's awesome? After a month of doing yoga I'm getting most of my range of motion and flexibility back in the knee that I blew out playing rugby in college & university. I honestly wish I had started this years ago.                         \n",
       "4418   Just rolled over 10k awesome miles in @corbett and my @Tesla 3. @elonmusk one suggestion: can I plz haz headlights on by default? Every time I drive during the day they start off and I think they improve visibility for other drivers as I pass them.  \n",
       "4417   Glad to see that Craig continues to keep the orientalism industry alive                                                                                                                                                                                   \n",
       "4416   @_Kazma8 No problem dude. Glad she was found                                                                                                                                                                                                              \n",
       "4415   Sign up for this awesome contest!                                                                                                                                                                                                                         "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>tweet</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>14794</th>\n",
       "      <td>suicidal</td>\n",
       "      <td>But why does my client commit suicide if I press the ranked ladder tab?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14793</th>\n",
       "      <td>suicidal</td>\n",
       "      <td>Ngl I don’t think I can do this education thing anymore. I’m moving to Naij this summer to sell pure water. I can’t come and kill myself abeg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14792</th>\n",
       "      <td>suicidal</td>\n",
       "      <td>The Island's suicide rate likely to rise following change in law.Read the next part in a special investigation into suicide on the Isle of Wight by @megbaynesLDR .#isleofwight</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8106</th>\n",
       "      <td>suicidal</td>\n",
       "      <td>@liangweihan4 my dad fucked up and married a white woman and had me. How do I keep the \"species\" strong? Do I date white women and marry out the the Asian or the other way around? Or should I just kill myself?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>suicidal</td>\n",
       "      <td>\"He was lost &amp; scared\" says a Newport woman about a 14y/o boy on the street today. Teen tells police he's Timmothy Pitzen. Pitzen disappeared in 2011 after being taken by his mother. In her suicide note she wrote he was safe but would never be found. @Local12 #TimmothyPitzen</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          class  \\\n",
       "14794  suicidal   \n",
       "14793  suicidal   \n",
       "14792  suicidal   \n",
       "8106   suicidal   \n",
       "0      suicidal   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                      tweet  \n",
       "14794  But why does my client commit suicide if I press the ranked ladder tab?                                                                                                                                                                                                               \n",
       "14793  Ngl I don’t think I can do this education thing anymore. I’m moving to Naij this summer to sell pure water. I can’t come and kill myself abeg                                                                                                                                         \n",
       "14792  The Island's suicide rate likely to rise following change in law.Read the next part in a special investigation into suicide on the Isle of Wight by @megbaynesLDR .#isleofwight                                                                                                       \n",
       "8106   @liangweihan4 my dad fucked up and married a white woman and had me. How do I keep the \"species\" strong? Do I date white women and marry out the the Asian or the other way around? Or should I just kill myself?                                                                     \n",
       "0      \"He was lost & scared\" says a Newport woman about a 14y/o boy on the street today. Teen tells police he's Timmothy Pitzen. Pitzen disappeared in 2011 after being taken by his mother. In her suicide note she wrote he was safe but would never be found. @Local12 #TimmothyPitzen   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "array(['cheerful', 'depressed', 'happy', 'overjoyed', 'sad', 'suicidal'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.set_option('display.max_colwidth', -1) # option to be set so that the tweet's texts won't be truncated\n",
    "df.sort_values(\"class\", inplace=True)\n",
    "display(df.head())\n",
    "display(df.tail())\n",
    "df[\"class\"].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now change the classes into a numerical representation to make it easier for deep learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>tweet</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>18115</th>\n",
       "      <td>4</td>\n",
       "      <td>You know what's awesome? After a month of doing yoga I'm getting most of my range of motion and flexibility back in the knee that I blew out playing rugby in college &amp; university. I honestly wish I had started this years ago.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4418</th>\n",
       "      <td>4</td>\n",
       "      <td>Just rolled over 10k awesome miles in @corbett and my @Tesla 3. @elonmusk one suggestion: can I plz haz headlights on by default? Every time I drive during the day they start off and I think they improve visibility for other drivers as I pass them.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4417</th>\n",
       "      <td>4</td>\n",
       "      <td>Glad to see that Craig continues to keep the orientalism industry alive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4416</th>\n",
       "      <td>4</td>\n",
       "      <td>@_Kazma8 No problem dude. Glad she was found</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4415</th>\n",
       "      <td>4</td>\n",
       "      <td>Sign up for this awesome contest!</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      class  \\\n",
       "18115  4      \n",
       "4418   4      \n",
       "4417   4      \n",
       "4416   4      \n",
       "4415   4      \n",
       "\n",
       "                                                                                                                                                                                                                                                          tweet  \n",
       "18115  You know what's awesome? After a month of doing yoga I'm getting most of my range of motion and flexibility back in the knee that I blew out playing rugby in college & university. I honestly wish I had started this years ago.                         \n",
       "4418   Just rolled over 10k awesome miles in @corbett and my @Tesla 3. @elonmusk one suggestion: can I plz haz headlights on by default? Every time I drive during the day they start off and I think they improve visibility for other drivers as I pass them.  \n",
       "4417   Glad to see that Craig continues to keep the orientalism industry alive                                                                                                                                                                                   \n",
       "4416   @_Kazma8 No problem dude. Glad she was found                                                                                                                                                                                                              \n",
       "4415   Sign up for this awesome contest!                                                                                                                                                                                                                         "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class</th>\n",
       "      <th>tweet</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>14794</th>\n",
       "      <td>0</td>\n",
       "      <td>But why does my client commit suicide if I press the ranked ladder tab?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14793</th>\n",
       "      <td>0</td>\n",
       "      <td>Ngl I don’t think I can do this education thing anymore. I’m moving to Naij this summer to sell pure water. I can’t come and kill myself abeg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14792</th>\n",
       "      <td>0</td>\n",
       "      <td>The Island's suicide rate likely to rise following change in law.Read the next part in a special investigation into suicide on the Isle of Wight by @megbaynesLDR .#isleofwight</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8106</th>\n",
       "      <td>0</td>\n",
       "      <td>@liangweihan4 my dad fucked up and married a white woman and had me. How do I keep the \"species\" strong? Do I date white women and marry out the the Asian or the other way around? Or should I just kill myself?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>\"He was lost &amp; scared\" says a Newport woman about a 14y/o boy on the street today. Teen tells police he's Timmothy Pitzen. Pitzen disappeared in 2011 after being taken by his mother. In her suicide note she wrote he was safe but would never be found. @Local12 #TimmothyPitzen</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      class  \\\n",
       "14794  0      \n",
       "14793  0      \n",
       "14792  0      \n",
       "8106   0      \n",
       "0      0      \n",
       "\n",
       "                                                                                                                                                                                                                                                                                      tweet  \n",
       "14794  But why does my client commit suicide if I press the ranked ladder tab?                                                                                                                                                                                                               \n",
       "14793  Ngl I don’t think I can do this education thing anymore. I’m moving to Naij this summer to sell pure water. I can’t come and kill myself abeg                                                                                                                                         \n",
       "14792  The Island's suicide rate likely to rise following change in law.Read the next part in a special investigation into suicide on the Isle of Wight by @megbaynesLDR .#isleofwight                                                                                                       \n",
       "8106   @liangweihan4 my dad fucked up and married a white woman and had me. How do I keep the \"species\" strong? Do I date white women and marry out the the Asian or the other way around? Or should I just kill myself?                                                                     \n",
       "0      \"He was lost & scared\" says a Newport woman about a 14y/o boy on the street today. Teen tells police he's Timmothy Pitzen. Pitzen disappeared in 2011 after being taken by his mother. In her suicide note she wrote he was safe but would never be found. @Local12 #TimmothyPitzen   "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "classes_index = {\n",
    "    0 : \"suicidal\",\n",
    "    1 : \"depressed\",\n",
    "    2 : \"sad\",\n",
    "    3 : \"happy\",\n",
    "    4 : \"cheerful\",\n",
    "    5 : \"overjoyed\",\n",
    "}\n",
    "\n",
    "for val, class_name in classes_index.items():\n",
    "    df.loc[(df[\"class\"] == class_name), \"class\"] = val\n",
    "\n",
    "display(df.head())\n",
    "display(df.tail())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Glove word embeddings.\n",
    "\n",
    "Right now, we have our tweets in text form. In order for our neural network to take them in as inputs, they have to be numerical. Therefore, we will be performing 2 tasks:\n",
    "1. Tokenising the text\n",
    "2. Adding a second layer in the neural network to map the text to their respective word embeddings\n",
    "\n",
    "Word embeddings are vector representations of distinct words in a vector space of many dimensions (in this case we are using 100 dimensions). This gives every word a distinct vector, and we can perform some very interesting vector operations on them.\n",
    "\n",
    "A lot of the code here is referenced from: https://github.com/keras-team/keras/blob/master/examples/pretrained_word_embeddings.py\n",
    "\n",
    "Note: Please run the 2 cells below in a Linux environment / Google Colab so that the commands work properly. Windows might not have the proper binaries to perform these commands."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget -qq http://nlp.stanford.edu/data/glove.6B.zip\n",
    "\n",
    "!unzip glove.6B.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Glove word embeddings should be downloaded and unzipped. Check the output of the cell below to see if they are reflected correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'1. Proof of Concept - Sentiment Analysis on Airline Tweets.ipynb'\r\n",
      "'2. Creating Mood Tweets Dataset.ipynb'\r\n",
      "'3. Mood Detection of Tweets - Word Embeddings and LSTM.ipynb'\r\n",
      " api-keys\r\n",
      " best_weights.hdf5\r\n",
      " datasets\r\n",
      " glove.6B.100d.txt\r\n",
      " glove.6B.200d.txt\r\n",
      " glove.6B.300d.txt\r\n",
      " glove.6B.50d.txt\r\n",
      " glove.6B.zip\r\n",
      " LICENSE\r\n",
      " mood_detection_model.h5\r\n",
      " mood_tweets.xlsx\r\n",
      " my_api_keys.py\r\n",
      " my_api_keys_SAMPLE.py\r\n",
      " __pycache__\r\n",
      " README.md\r\n"
     ]
    }
   ],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a new dictionary to map the words in the embeddings to their 100-dimension embedding vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 400000 word vectors.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "# NumPy is a very widely-used library in Python for mathematical operations.\n",
    "# It supports many complex mathematical functions and can work on large arrays and matrices.\n",
    "\n",
    "embeddings_index = {}\n",
    "with open(os.path.join('glove.6B.100d.txt')) as f:\n",
    "    for line in f:\n",
    "        values = line.split()\n",
    "        word = values[0]\n",
    "        coefs = np.asarray(values[1:], dtype='float32')\n",
    "        embeddings_index[word] = coefs\n",
    "\n",
    "print(\"Found {} word vectors.\".format(len(embeddings_index)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.37058 , -0.67807 ,  0.68365 , -1.8257  ,  0.033376, -0.19271 ,\n",
       "        0.23817 ,  0.20734 , -0.59292 , -0.68866 ,  0.056996, -0.45526 ,\n",
       "        0.050337,  0.38388 ,  0.5716  , -0.58939 , -0.065345, -0.15682 ,\n",
       "       -0.71076 ,  0.80948 ,  0.28472 ,  0.33693 ,  0.16452 , -0.10708 ,\n",
       "        0.93074 ,  0.63066 ,  0.31136 , -0.47565 ,  0.26755 ,  0.43212 ,\n",
       "       -0.54809 ,  0.45665 , -0.13095 ,  0.41672 ,  0.07025 , -1.0302  ,\n",
       "       -0.21576 ,  0.34737 , -0.52304 ,  0.43099 , -0.39802 , -0.25244 ,\n",
       "        0.39093 ,  0.58856 , -1.1839  ,  0.31923 , -1.0152  , -0.32431 ,\n",
       "       -0.099747, -0.12774 ,  0.20104 ,  0.16763 , -0.59854 , -0.25134 ,\n",
       "        0.49871 , -2.4691  , -0.88555 , -0.14335 , -0.037509,  0.87851 ,\n",
       "        0.086941, -0.097006, -0.59662 ,  0.63888 ,  0.72839 , -0.87794 ,\n",
       "        0.71174 ,  1.1118  , -0.26572 ,  0.3617  , -0.43472 ,  0.17808 ,\n",
       "       -0.57865 , -0.20282 ,  0.3917  , -0.37085 , -0.37463 ,  0.024523,\n",
       "        0.017991, -0.43885 ,  0.093244, -0.019675,  0.20245 , -0.0862  ,\n",
       "       -0.30843 , -0.20882 , -0.15978 , -0.37059 , -0.3803  , -0.33614 ,\n",
       "        0.24756 , -0.06655 , -0.015687, -0.18252 , -0.70065 ,  0.29659 ,\n",
       "        0.077268, -0.016513, -0.027181, -0.13    ], dtype=float32)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# to see what word embeddings look like\n",
    "embeddings_index[\"chris\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This array of numbers make up the representation of the word \"chris\" in 100d-space. This is beneficial as it gives every trained word a distinct vector representation, and we can calculate the cosine similarity between 2 different words to determine how similar / different the words are, in terms of contextual information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Processing them for TensorFlow\n",
    "\n",
    "TensorFlow (& Keras) is the library/framework which we will use to train our neural network.\n",
    "\n",
    "We will process our dataframe above into TF-ready numpy arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "from tensorflow.keras.utils import to_categorical\n",
    "from tensorflow.keras.layers import Dense, Input, LSTM, Embedding, Bidirectional, Dropout\n",
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.initializers import Constant"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setting some parameters\n",
    "\n",
    "MAX_SEQUENCE_LENGTH = 40 # determines how many words of every tweet will be processed\n",
    "# if the tweet length is smaller than this, the tweet will be padded until it has a length equivalent to this\n",
    "\n",
    "MAX_NUM_WORDS = 70000 # how many unique words in the training text to be tokenised\n",
    "EMBEDDING_DIM = 100 # number of dimensions in the word embeddings\n",
    "TRAINING_SPLIT = 0.8 # proportion of data to be used to training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Converting the dataframe into lists\n",
    "\n",
    "texts = df[\"tweet\"].tolist()\n",
    "labels = df[\"class\"].tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenising the Tweets into a 2D integer tensor\n",
    "\n",
    "Tokenising tweets will break up the text into individual words, and replace them with a numeric representation.\n",
    "\n",
    "e.g. \n",
    "- \"I am a human\" > \"i\", \"am\", \"a\", \"human\" > 1, 2, 3, 4\n",
    "- \"I am a robot\" > \"i\", \"am\", \"a\", \"robot\" > 1, 2, 3, 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "67094 distinct words in original text\n"
     ]
    }
   ],
   "source": [
    "tokenizer = Tokenizer(num_words=MAX_NUM_WORDS)\n",
    "tokenizer.fit_on_texts(texts)\n",
    "sequences = tokenizer.texts_to_sequences(texts)\n",
    "word_index = tokenizer.word_index\n",
    "\n",
    "print(\"{} distinct words in original text\".format(len(word_index)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All our tensors have to be of the same length, so we need to set a maximum length, and if a certain tweet has fewer than the maximum length of words, it will be padded until it is the same size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "117"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XeYXGXd//H3N9n0HlIIKWwgoYQgBBYMAqFDKBpEFPQR8hM0j4+oKChEQLoUURAUkCBIQITQgwRSCSAJCWwa6YWQRtqml02y7f79MWd3Z3an1z2zn9d17bUzZ065z5wz33Ofux1zziEiIvmrSa4TICIimaVALyKS5xToRUTynAK9iEieU6AXEclzCvQiInlOgV5EJM8p0IuI5DkFehGRPFeQ6wQAdOnSxRUWFuY6GSIivjJr1qwtzrmuseZrEIG+sLCQ4uLiXCdDRMRXzGx1PPPFLLoxs2fNbLOZLQia9pCZLTGzz83sTTPrGPTZ78xshZktNbMLkku+iIikSzxl9M8BQ+tMmwQMdM59DVgG/A7AzAYAVwLHeMs8YWZN05ZaERFJWMxA75z7CNhWZ9pE51yF93YG0Mt7PQx42Tl3wDn3JbACODmN6RURkQSlo9XNNcB73uuewNqgz9Z50+oxsxFmVmxmxSUlJWlIhoiIhJNSoDezW4EK4MXqSWFmCzvgvXNulHOuyDlX1LVrzEpjERFJUtKtbsxsOHAJcI6rfXrJOqB30Gy9gPXJJ09ERFKVVI7ezIYCNwPfcs6VBn30NnClmbUws75Af+DT1JMpIiLJiqd55UvAJ8CRZrbOzK4F/ga0AyaZ2Vwz+zuAc24h8AqwCBgPXOecq8xY6kVEwvhwWQlrt5XGnrGRsIbwzNiioiKnDlMiki6FI8fRtInxxX0X5TopGWVms5xzRbHm01g3IpKXKqtyn4ltKBToRUTynAK9iEieU6AXEclzCvQiInlOgV5EJM8p0IuI5DkFehGRPKdALyKS5xToRUTynAK9iEieU6AXEclzCvTSqN3/3mIG3T0x18kQyaikHzwikg+e+nBlrpMgknHK0YuI5DkFehHxlY079/Psx1/mOhm+oqIbEfGV/32hmHnrdnLu0d3pc1DrXCfHF5SjFxFf2bmvHIDKBvB0PL9QoBcRX9leWp7rJPiOAr2I+Ep1jl7ip0AvImHNW7uDmSu3ZmTdu/aX8/Kna3AqfskKVcaKSFjDHp8GwKoHLk77un/3+nzGzd/AUT3ac3zvjmlfv4RSjl5Esq5k9wEADpRX5jgljUNeBvp9ZZVUVumWsK49BypynQQRyYGYgd7MnjWzzWa2IGhaZzObZGbLvf+dvOlmZo+Z2Qoz+9zMTshk4iM5+vbx/OKl2bnYdIM1Z812Bt4xgQkLN+Y6KSJpofL9+MWTo38OGFpn2khginOuPzDFew9wIdDf+xsBPJmeZCbu3fkKaMHmrd0BwPQVW3KcEhHJtpiB3jn3EbCtzuRhwGjv9Wjg0qDpz7uAGUBHM+uRrsSKiEjiki2j7+6c2wDg/e/mTe8JrA2ab503TUSkhkPFLtmU7spYCzMt7BE1sxFmVmxmxSUlJWlOhoj4gVm4kCHplmyg31RdJOP93+xNXwf0DpqvF7A+3Aqcc6Occ0XOuaKuXbsmmQwREYkl2UD/NjDcez0cGBs0/Wqv9c1gYGd1EY+IiORGzJ6xZvYScCbQxczWAXcADwCvmNm1wBrgu97s7wIXASuAUuBHGUiziIhK+RMQM9A7574f4aNzwszrgOtSTZSIiKRPXvaMlciUCxJpfBToRSTr1Kk1uxToGxk1ZpOGRK0rs0OBXkQkzynQi4jkOQV6EZE8p0APvDhzNQNuH0+VxrAX8Q1V6MZPjxIE7hi7kIoqR6VzNFF1pUjGKUZnl3L0IpIzylZlhwJ9I6OclEjjo0DfSGg4WJHGS4G+kdDzNUUalp++MIu/vb88K9tSoG9kGlq+vrSsgtKyilwnQxqBqirHtr1luU5GjfELN/Knicuysi0FesmpAbdP4Jg7JuQ6GdIIPDJ5GSfcM4mS3QdynZSsU6CXnFOpUuOTnqLExNYxceEmALbuVaAXEcm46hCtNgLZoUAvIjmkSJ8NeRHoq6oc5ZVVuU6GiOQh5xxlFf6OL3kR6K9+9lP63/perpPhCyoOF0nMQxOWcsRt77G/vDLXSUlaXgT6j1dsyXUSGjw/dZjavreM6V/omErq0lHp+/JnawHYe8C/zYDzItBLfrn62U/5wdMzfX+7LJFlq6XVB8tKsrOhBk6BvpHwU8/YpRt3A+BU0JT3Mn2jub0BdZDKJQV6Gle5tX8KcESi81HeJecU6IMoCArAjtIyCkeO498z1+Q6KSJpoUAvUse67fsA+NeM1TlOiUh6pBTozezXZrbQzBaY2Utm1tLM+prZTDNbbmZjzKx5uhIrjYtuzSWdUq3z8fPpmHSgN7OewC+BIufcQKApcCXwIPCIc64/sB24Nh0JDWfPgQrueWdRplYvuZKjMrSqKscD7y1h8+79uUmANEj5UKSbatFNAdDKzAqA1sAG4GzgNe/z0cClKW4jor+9v4JnPv4yU6uXRubTVdv4+4df8NtXP891UvJeLnLHlhchOzlJB3rn3FfAn4A1BAL8TmAWsMM5V92zYB3QM9VERlKhYQ8S5ufbz0yr8sqKynReZU3jDb3ZlUrRTSdgGNAXOARoA1wYZtawscXMRphZsZkVl5TkR6eGLXsONNjbfj/1jBWR9Eql6OZc4EvnXIlzrhx4A/gG0NErygHoBawPt7BzbpRzrsg5V9S1a9cUktFwFN07mZP/MCXXyZBU6bbHF3SY4pdKoF8DDDaz1hbILp4DLAKmApd78wwHxqaWxMjSfaDz+cTxU8/YhkI3Qf6nYxiQShn9TAKVrrOB+d66RgE3AzeY2QrgIOCZNKQzoxrTudCY9jVhuhZKFH7OK6XU6sY5d4dz7ijn3EDn3FXOuQPOuZXOuZOdc/2cc991zvn2uV1LN+7miFvf46sd+2LO+/qsdQmt+5FJy/je3z9JNmmSSboaZp6fo6YPqWdsFP+euZqyyiomLdwYc96RbyTWJO/RKcv5dNW2ZJMmkheiNRLYV1bJtiQGJdu4MzMNIvxcDKRALyIN0neenM4J90xKaJnNu/cz+P7aBhG6cQhQoI9g1urtFK/enutkNEo5zzgpODQIizbsSniZ7XvLM5AS/yuIPUv+C/e7/s6T07OejmxQDIsi51cYyaTG/HyDRp2j37mvnAse+YjKqsAJkMrvPBu3iHeMXcBjU5Yntaw6TMUhj+PADa/M5Z/T8mu4kGR/c8kOheDnYqBGHeg/WlbC0k27c52MuI3+ZDUPT1qW62Q0Gvl0bXxj9lfc9R8NAAiJ5+zz4Txo1IHeT74o2ZPrJGSdn3NQEp0GNcsuXwf6VAOBn+LIis2pBXo/9YzNhxyUxCeVQx3uPKk7TedSgK8Dfbb4J0TG5ofz3kfXJBFfaNSBPp25XMWm9FNuLH/pYp5djTrQx0vxpnFQ7Mk+Xcyzw9eBPpmTZPqKLRSOHMfn63Ykvd1731lE4chxWS33bow5oLr7fPmT0ykcOS6pdW3fW0bhyHF518RQQmXyuuHndvi+DvTJBL+pSzcDMGPl1qS3+48G8PjCj5dvoawi8Sch+eFUjXQBT6Wn8vqdgYHpXimOPPicMpfZk46gmb3MT/gzo6Kyig+X+eOhSb4O9A1JNnP389bu4IfPzOT+9xbHvYw6TElDlOkmjxsyNMAZwBMffMHwZz/lAy/z2JAp0KdJNnPK20oDI/qtLNmbxa1m1vtLNvHvmWsSXu7TL7cx6qMvUt7+5EWbUl6HpGbSok2M+SzxcyCa0gOVcc1XUVnFHWMXsGFnuCHJw/+6V28tBaBkd8Mfib3RBnrn0tAOP6vlIH4odEneNc8Vc8ub8xNe7ntPfcJ97y6Je/5Id14/fr444W1Lev3k+WJufj3xcyBe0YqLZqzcxuhPVnPTa5GHG/dzh6tGF+jTUYSRi1KQRlkZm8aLWzw/0nBbGz19FYUjx7G/PL6coaTfNc99xqkPvA/At5+YVu/zV4vXUjhyHDv3lcf12wx3LlSfa8n8zvzw0/R1oM9WLXjwVpZt2u3LoBtvHUJllWPe2uRbJKVDznNOQZt/4oMVAOwo1fC3ufL+ks01T3mbs6b+ufmM1zhifRxPggunssrx+bqdCS/np2ovXwf6XDj/kY/CTs9F8E9mk7HOzUcnL2PY49OYn8SJnzd8eCH3m1xmlupmEB+fuoKHJixNeDk/aRSBfvveMh6euJSqqtoD5Yj/wDW0C3cm07NgfeBhD5t2Za61AkBZRRV/HB++bD3WcZkQx6Md0yH4zsLPP/Jce2HGahatD/8QkUznioOPW6Q7xUhpq9XQIkDifB3o473Fv+2tBTz2/go+XFaS1kOWzZ9+tsNMpn+AL3+2hic+iN5aJtLx/d8XZmUiSTWCv+ucFyPlgd+/tYCLHvtv2tebSrl6mhPS4Pk60MeruiKtsir0iNQ9QXxwvCIaeMcEnouj12esfcxWf4B4OnultTI2jnj9ZJQLT86DSRotWr+LwpHjwvbuHvF8MT/xaQukwDHO3oW5ekt+uNvzdaDP1Be8c198FW+795cn1IZ286797DlQEXWeisoq1njtc2NZvXUvlVWOlSV72HOggjujPFgi3tZG1d9oPLNv3rWf3fvTX0m5vzxwEYj1XcVjZ2k5W/fEd4w+XrGl3rTq76Gsoop12+M7Lg3dG7NrewfXzfxMXLSJSWnoU7Bue2nWWipFiwOJXqDjOe+dC/zmYs1bXlnFR8tKGsQQ4b4O9KmI9t2fGOeT54+/exIn/WFy3Ns8+b4pXBChMrfafe8uYchDU9kYR4++1VtL+cHTMzj7zx/GnYZ4xVNkcfJ9UyJWTidrS1BQvuSxj1Ne33F3T+TEe+M/RhA+T3jjq/M47cGpamYZh4rKKk57cCrXvzwnq9vNVjHbmM/WcvafP2Tml9uAyLHkiqc+4epnP+XB8bErejOt8QX6OM6FiqrMXYG/2rGPF2eujvj59C8CucrtXu/XasEnU3DuvPpkS5dEMx/BXcxXbN7Nf+atT2n7wc0YN2egx2GymatZ3jg7sc6NGSu3Mj3MnUG2vTF7Hau25KbndIl3sZ66JPPjwPx3eQnLNqX2UJ5E66PmeQMiro5x5z3bawo6dUnuh0goSGVhM+sI/AMYSOCu/xpgKTAGKARWAd9zziU/GlUa5f4GKuDWNxckvEzW+wwkkTk69+FA7v6bxx2StvSkSyI/5tDK2MRcOWoGAKseuDjBJdPrhlfm0bp5UxbdPTTqfJk4q375UiAnX1YZuR4mXaUZVz3zac3rdDQgiJouF8c8YRfLfeRJNUf/KDDeOXcUcBywGBgJTHHO9QemeO+zYt7aHZzx0NR60yOdAPEesEQP0w1j5vLj0cUU3Tsp480U06m6TBEyX6WV7UHWHp6Y+EPV53/l774EpWX1i5lmr9me1tFX35rzFRc9GtqiZnuUzmW795cz+L4pLNoQq0ljcuqeVi7kdXIBN9VTddmmPfw0wy3FYkk60JtZe2AI8AyAc67MObcDGAaM9mYbDVyaaiLjddd/FobcTm2JUgm3ryz1ir5I3pjzFZMXb2LLnrK423zvK6tkz4GKuC4+yaQ9uEJof3ll2ErUZAdnivQ97y+vZFed7VRUVrF9b1nY+aOpW2m4fW8ZFUE5xi17DkSt9JroVTBGy2WGU/eCtGd/RcLl9NHOw2gqq1xS31U0j0wKveBVF5WVV1axozTytkrLKthbp3K8rKKKX42Zy6INu0LOqbrHKticNTvYmKbMT93DvSvORhSGsXNfOQcqIh/HWJ+HpCPodaRjPT5LfT8iSSVHfxhQAvzTzOaY2T/MrA3Q3Tm3AcD73y0N6UxK0b2T6407X11h89j7K+K+vmcj7zn4/ikMvGNC7TajbPSn/5qd9HYMOOOhqRx758To8yWQjSm6d3LYTiff/OvHfK3Odm59cwGD7plEeYIB9553QlsUDbpnEre9VVsEVnTvZMZ8tjbmer5Msdx68P1TuPDR+NuEf7ZqG0X3TuadzxOvu7j7PwsZdM+kegE2naobE/z21Xkcf3fkRgjH3jmRY4LOT4Crn51Z8/qiR/9bc06l+h3Hq+5Ik5f//ZN680Q6i4+7ayI//Ecg/cGnevXr4+6ayP979rOwy0bKT3y+bgdF907m9VmRn3mQK6kE+gLgBOBJ59wgYC8JFNOY2QgzKzaz4pKSzFXavL9kM5MXx18ZkmplYrKqm3RWB8CpS0oYO/ermunpbKG1aVf9XMfmXftD7j4Svbgt37w75P2araUs31y/kuxt7/utSDDQvzt/Q71pL9cJ7NO+CFzUq6ocYz5bE/FiMmHhxqhFarGadSYSyBZ6xT+fJVFpPs7b53BFMOHsLC1n7NyvEt4OwFtzo5/34XLpM1bW7tPKCN/JrBQeFhNLPGPNO2DMZ2v4y+RlNZW21cfvs1XR0/aJl0ms+9ubvDi0+Wn150s27g5ZriFJpTJ2HbDOOVd9WX+NQKDfZGY9nHMbzKwHEDbKOudGAaMAioqKMlZbMeqjlXHPu3Hnfn7xUv0mYdmsSqn+wTzoDQ9w3oDuPH11UdrWH2lfrnx6Rkrj29cdS35ImLqS+CT/bVcX3bw2ax03vz6fLXvCF0X87wuzOPSg1nz427OS3lZDdP2YOXywtISv9eqY66TU+M6T00PeZ/q3VDeDMu7zDazZFto65qf/SrK83Fv5Vq84rYlBlYu/7H/u2h0c3zs3xybpQO+c22hma83sSOfcUuAcYJH3Nxx4wPs/Ni0pjcPa7cmNXlftQBKP5ovH395fHvGzqipHkyaR88+b01SeWV0U8/wntU07yyqqaF4QuKlL9SEmwZ3MHp1cf3+rqhw/fr6YfV759t44c6k1y8dxS+OAO8YuYLS3j9HKnFdvLY3r+bNfxTki4o7SMq4dXdujdPf+ctq1bBZ1mX9O+5JNuw4w8sKj6n22v7yy5kJ10h8mM/mGM/jta/O445vHcPd/FtK3S1uO79ORqwYfWrPMhh37a7Zd7bInpjF04MFx7UOwpz6s7SW8ZGNtsVz1dzbv9vMTXuedby/k9P5dQqbNWLmVyirHHyfEfqbA0o2hd43xtF6rG+TrCncOBN/t3fn2wtoP6pyCZhaS3X9zdu3d1LYwdSulGawXjCWl5pXAL4AXzaw5sBL4EYHioFfM7FpgDfDdFLcRUd3ffqzKxIrKqqR6qZWWVVJZ5WgaJSBHYsCforT42L2/gg6toweEVEWqVFq2aTcDe3YI+1lFVeC7qr5AOOcoq6yiRUHTmNt7ZHL9/d25r5z3g9oTT/8isdvbSLnzukYHXciy6T/z1ocUU7y/ZDPDju9Z876iylFaVkHr5rU/ubu8nsw3XXAkO/eV06p5U1o2C3y/s9eEFitcOeoTtuwp49LHp3mf7+D12eu4avChlFdW0cSspqI5uL5k9podNe25Ib4LJsD979UG3lveqP8wkLHzwhcRRft9PTd9FX06tw6Zdu+4xfTr1pYVYYr56rrtrdgPJUm0sj2cf05bVfP6uemr6BT0+4y2f8FFNsG9j6uVV7qajF28Fb3pklLzSufcXOdckXPua865S51z251zW51z5zjn+nv/09ujJwX/9+JsnkqgKKfaQxOWctUzM2PPmAE7qsvok1x+6pLNHHnb+LBjzEf70V/zXHHIoGPPTlvFkbeNZ/Pu8HcYDaCXNxMWhLZsaEjPyX1x5hoG3D4hbJn1Ybe8y6B7JnHU78dHfGh9tAtd/1vf45K/fhxX3cFXYe56Y2V+ZocZA/72sQvDzAl/GBf9OcbhthRPkIfYZeoAQ/+S/sHTgpuLjp6+quZ1pLPLOXjn8/p1SsOf/ZR7xi1if3klR942Ps2pjK7x9YxNUqI50GrBFVbJWL21NOEHgSzftJtlmwK3udVPqa+bQ4RA+WI0b835inXbS5mzZjtve5V8o6evYs+BinrjoSzdtDvcKgAYv2Bj/WZnEYLLl1uSH0+mbq/VuVl4gMqyTbtZvml3vQC21QvMdS82s1dvp6KyivELwje3m+mdL5/GWXlbfVwXB7VLXxWlx2a4z9LZETwdbfQnLtzI5+ty8/CbaSu28NKnkZ9b+5+gAF59aJ0jpKWNw0U89/45bVVOBo1Ltegmr4xfUP8qnKpxYVqLJGrY49N49Mrj457/PG/8mVi9M+O5jT/twUCl6nG9AkU8j0/9gsenJvYw7p/+axZtW8R3qkX6EdQpDo1LvMEyFdVj/dw97JiQ6Xe/s4hrTutbP02rtrHnQAWPTolcbwPwlzD1HOFc9sT0etP+/mFixyfe4pxsGZHDzkVVLv5BDQNNtQPf3Y2vzot7G/9dnv0hMhTog8RbDlzXDWPmpjkl9V3/cnzbCB7+9953FvGcd6sZ7qdcUem45c35ETuaBDePnJfiE6fqNlkM18QznO899UlWAnaiyiurYraNv/7lOSFl8gCTYowO+cjkZfzynH5pSWO8Trl/Sta2Vbc/hN+EFL1FKLt5Y3ZyTVwzSYE+Dd6Y03AO7McravskxLqNXrpxV71mkdkSb+/IhhjkIdAR6tdjanNx4TLFY2O0TY8k3otguiSbwWnsasejb/gadRn9P/6bWMXsypLURsnLpXC9K38foUJNYvvB06GV8+nsaBdtdFPJnZPvC73zqW6O/fu3Eh+kMNsadaBfUqdd7oIYg1h976kZGUlH3fbBqZgSoRewcm2ZVZzGHqDPBTXvE0mHRh3o64r1Y012cKpYLvhL+h7e8WKOimIkfXZncGwbaZwU6HOt4TT1FpEcWBuj9246KNCLiOTQ/72Y+eakCvQ5dtxd0YcLFpH8Vl6R+XY7CvQiInlOgV5EJIey8UxZXwf6ZEaiFBFpSKofiJJJvg7005IcaExEpDHxdaAvy9CDQkRE8omvA72IiMTm60DfgJ4rISLSYPk60IuISGy+DvRqdCMiEpuvA72KbkREYvN1oBcRkdh8HehLyypznQQRkQbP14F+z36N2y0iEouvA72IiMSmQC8ikudSDvRm1tTM5pjZO977vmY208yWm9kYM2ueejLD69a+RaZWLSKSN9KRo78eWBz0/kHgEedcf2A7cG0athHWSYWdM7VqEZG8kVKgN7NewMXAP7z3BpwNvObNMhq4NJVtiIhIalLN0f8FuAmoHkbyIGCHc666Ocw6oGe4Bc1shJkVm1lxSUlJUhtXfykRkdiSDvRmdgmw2TkX/GTbcLE37EAFzrlRzrki51xR165dk0pDE3WNFRGJqSCFZU8FvmVmFwEtgfYEcvgdzazAy9X3AtannkwREUlW0jl659zvnHO9nHOFwJXA+865/wGmApd7sw0HxqacShERSVom2tHfDNxgZisIlNk/k4FtABrUTEQkHqkU3dRwzn0AfOC9XgmcnI71xqJALyISm697xmo8ehGR2Hwd6EVEJDYFehGRPOfrQK8yehGR2Hwd6EVEJDYFehGRPOfzQK+yGxGRWHwd6FVGLyISm68DvdrRi4jE5utALyIisfk60KvoRkQkNl8HehERiU2BXkQkzynQi4jkOV8HehXRi4jE5utALyIisSnQi4jkOQV6EZE8p0AvIpLnFOhFRPKcrwO9esaKiMTm70CvBpYiIjH5OtCLiEhsvg70Do1TLCISS9KB3sx6m9lUM1tsZgvN7Hpvemczm2Rmy73/ndKXXBERSVQqOfoK4Ebn3NHAYOA6MxsAjASmOOf6A1O89xmhMnoRkdiSDvTOuQ3Oudne693AYqAnMAwY7c02Grg01USKiEjy0lJGb2aFwCBgJtDdObcBAhcDoFs6tiEiIslJOdCbWVvgdeBXzrldCSw3wsyKzay4pKQk1WSIiEgEKQV6M2tGIMi/6Jx7w5u8ycx6eJ/3ADaHW9Y5N8o5V+ScK+ratWuS209qMRGRRiWVVjcGPAMsds49HPTR28Bw7/VwYGzyyRMRkVQVpLDsqcBVwHwzm+tNuwV4AHjFzK4F1gDfTS2JIiKSiqQDvXPuYyI/5OmcZNebCJXciIjE5uuesaZCehGRmHwd6J3TEAgiIrH4O9DnOgEiIj7g60AvIiKxKdCLiOQ5Xwd6VcWKiMTm60AvIiKxKdCLiOQ5BXoRkTzn60Cv5pUiIrH5OtCLiEhsCvQiInlOgV5EJIeOOrhdxrehQC8ikkOHd22b8W0o0IuI5DkFehGRXMpCF38FehGRHMrGUC6+DvQajl5EJDZfB3oREb/LRn7V14FeTxIUqTX4sM65ToIkQUU3IhK32y85JtdJkCQoRy/SSEz69ZCY83Rs3Yy//WBQzfu5t5/HDwf3qXl/6EGtM5I2ySzl6GMYOvDgXCdBJC36d4/dO/LlEYNp1axpzfuOrZvTpW2LTCZL8oSvA32PDq1ynYRG5+RClQPHY+Kvh7DqgYvTtr5VD1zMUQe3p0VB04jzNAlTabXqgYv5+Oaz0paOYJm6g/jj5V/LyHobqoImmc/T+zrQZ8ophx2U0PS6DmrTnL//8MR0JilhXdo2D3l/9lHdos4fbzD40amFUT8/5pD2HNerQ837m4YeGXa+u751TEixA0Cn1s04qbATLZuFnpYdWjWreT38lEM5ukf7uNJ697Dsl1m/dd2p3PHNAfTvFujW/s8fnRTy+en9u9S8LqwTKIceU/8O9c2ffYOnry6qeX9qv4M4b0B3/vr9QSHzdWvXglbNw18EenWq3c7NQ49i8g1D+O0F4Y/L5BuG0LNjfBmooccczKXHHxLXvOF8e1BPnvifE0KmPfidY6Mu06xpaFD8yel9k97+we1bxj1v13b175zuuXQgt18ygL5d2sS1jtsuPjrs9Hju5lKVsUBvZkPNbKmZrTCzkZnYRvuWBTWvH7gs+gkC8MgVx8W13mFhTt4WBU14acRgnr/m5KjL/ujUQmb9/ryQYqXLTujJPVkOOk/8z4nMvOUcIBAcH/5e5H2//ZIB9OrUOuTiECk3eliMcTne+cVpjP35aax64GJWPXAxPzuzH2/+7Bv15hv+jULuGTaw5v20kWcz5/bzefWn3+A4K8R/AAALv0lEQVQHJx9aM338r06vCYAPXHYsdw0byHvXn15zYfrxaX0j5iyvPqWQ+XeeD8Blg3rWTI91DGMZ1KcjAO1aFPDzs/qFfHZIh5b86NS+mJe7PuvIbiHBKHj+X57TP2TZY4MukAD3fftYBvXpxHkDutdMMzOevrqIbx4XOEerB8S681vRz6+BPQMXx/8783D6dWvHdUHp+MXZgdc9OrSkX7d2TP3NmfWWv/G8I2pen++lZ8Ah7fnLlYOYd8f5Ebd7gvddVTvzyK41rx+54nguOrYHzwwPXMgevfJ4rjipT8TAeXLfziEXPYBbLx7AGUcE1rnqgYv583dDz/NHrzw+Ytpm3HJOvbuHy0/sFXbeH59W/4Jy1eBDuea0vvxjeFGYJer71nHJXxRTVRB7lsSZWVPgceA8YB3wmZm97ZxblM7tHNS2BW9ddyrtWxbQt0sbDu7QkrXbSmnVvIBTDj+I+et2UlpWwQ2vzAPg2J61P6Txvzqd1VtLOeaQ9jRr2oT563Yy8o35bNlzIGQbY687lTYtmtK1beDqP+SIrvz1+4No27KAz9fu5IKB3enUujktmzVl654D9O5cG3Tm3n4e7y3YyHdO6EWzpsbAnh1o1rQJ63fsY8QLswC47qzDGXH64Uz7YgutmjWlf/e2fLx8C6f268KBiirOffjDmvX996azKNlzgPnrdnLH2wtD0jl95Nn89f0VXDjwYPaXV3Jy30ARy9TfnEmvTq1o1rQJH/zmTM780wc1yyy46wI27txXM6hS786t2bKnjFFXhd6NLLzrArbuKaOiqipmoA9nUJ9OjPvlabRpXsCeAxU1OXQLKmoIzkXectFRvDhzNQcqqiJ2iuvVqTXv33gGfTq35vpz+7N88x4ue2I6EDi2HVsFLlrtWjbjw9+eycEdWvKrc49g1/5yBvbswPs3nkHvzq1ZvXUvLQqasresgo6tmuNwbN51gGGPTwPg2f9XRNMmTejYqhkHtW3Otr1lHNG9HQvX76Rf13a0bVnApYN6hhynum4eehRvzllfc24tuOsCZq/ezun9u3Dmkd2oqrOTw085lLOP7s6QoNx/JEMH9mDSr4fUyxWeeGinkNzymBGnsGNfecg8xbedSxMzOrRqRlFh55piueYFTfj3T77O4V3bMmfNdgb06ECvTq248NgeOOfo160tKzbvqdlmh1bN+OA3Z7J1bxm79pXT56DW7N5fwZ79FXz9sM6s3lpKh1bNaGLQpkUBq7bupX3L2ru0c47uHrIPJxV2ZvINQ+jYujlNzCirqGLb3jL6dmlDq+ZNmfTrIfTq1Lrm+3zqqhMp2R14fdkJPenctjlf79uZr7bvo3/3dhxzSPua8+i8Rz4CIFxpyX9vOovu7Vvy2qx1Yb/rWbedG/hu750cMj3aoGSTbziDHh1asm1vWcj0+y87lqlLNjNx0aaIy6ZTRgI9cDKwwjm3EsDMXgaGAWkN9ADH967NMZx5ZGjxRHXw+GzVduau3cEh3vvHvj+Iow5uz1EH1xYBdB/QkvucY8QLs2rW89fvD+K43qE5EqAmN3VWne0FFzFAoLLs+yfXFk8M6tMJgIE9O/Dn7x7Hja/O4xdn96dls6ZcdGyPmvmuDFrm1ouO5g/vLqZ186b07tya3p1b07NjK+54eyHtWhSw+0AFAId0bMX9Ye5qgnNHhV3acMtFR3Hfu0u4Z9gxtG1RQL9utQHiZ2f24yfPF/N1r4iqTfOm/Pzs/rRpUUCbFrWnyi/P6c+/Z67mpMLObNtbxswvtwHQNEpZ4zGHdAg7vXlBE26qU4xQ0LQJj1xxPD97cTa9O7fmipN7M6Z4Laf2Cw181Reddk2bcEKfTlw48GB2lJaHHFeAQw8KfAd9gnL+1csG73+1Hh1a0bNjKy45rgdnH9U95LPqYpATD62tq+jXrS1/vPxr3PTa53RoHXoOVO/Pg985lmtHF3Pkwe1o26KAIV4utHOb2ruoC445mIcmLOWKk/ow4JD4iqcg9NZ/xJDDGPXRSn51bn+6BxVN1D2GQEhFbnWuuNo3Dg9810MH1p6X/brVBrS6F5bCLm0ojJATD14OqHd8wq2v7nE5uEPLevNWZ6paNmta89rMan6X1fMFr+voHu1ZvGEXt18yAKDmnOrduVXNOn597hE8MnkZ/bq1ZcOOfewtq+TcAd05yPu+vnXcIWzYuS8kfUWHdqJ49faQabddfHTNvrdpUcCBikogUAfxvaLeHN+7IxMXbeKCMEV26WYuA+MImNnlwFDn3I+991cBX3fO/Tzc/EVFRa64uDjt6RARyaaZK7dyxagZnHFEV0anWEQYDzOb5ZyLWXaUqRx9uKxdyBXFzEYAIwD69OkTZnYREX85qbAzvzi7H1edcmjsmbMoU5Wx64DeQe97AeuDZ3DOjXLOFTnnirp2Db1tFBHxoyZNjBvPP5Ju7eJv0ZMNmQr0nwH9zayvmTUHrgTeztC2REQkiowU3TjnKszs58AEoCnwrHNuYYzFREQkAzJVRo9z7l3g3UytX0RE4qOesSIieU6BXkQkzynQi4jkOQV6EZE8p0AvIpLnMjIEQsKJMCsBVie5eBdgSxqT09A1pv3VvuYn7Wv6HOqci9njtEEE+lSYWXE8Yz3ki8a0v9rX/KR9zT4V3YiI5DkFehGRPJcPgX5UrhOQZY1pf7Wv+Un7mmW+L6MXEZHo8iFHLyIiUfg60GfjAeTZYGarzGy+mc01s2JvWmczm2Rmy73/nbzpZmaPefv8uZmdELSe4d78y81seK72J5iZPWtmm81sQdC0tO2bmZ3ofXcrvGUjP88wwyLs651m9pV3bOea2UVBn/3OS/dSM7sgaHrY89ob9num9x2M8YYAzwkz621mU81ssZktNLPrvel5d2yj7Kt/jq1zzpd/BIY//gI4DGgOzAMG5DpdSe7LKqBLnWl/BEZ6r0cCD3qvLwLeI/AUr8HATG96Z2Cl97+T97pTA9i3IcAJwIJM7BvwKXCKt8x7wIUNbF/vBH4TZt4B3jnbAujrnctNo53XwCvAld7rvwP/l8N97QGc4L1uByzz9invjm2UffXNsfVzjr7mAeTOuTKg+gHk+WIYMNp7PRq4NGj68y5gBtDRzHoAFwCTnHPbnHPbgUnA0Gwnui7n3EfAtjqT07Jv3mftnXOfuMAv5PmgdWVdhH2NZBjwsnPugHPuS2AFgXM67Hnt5WbPBl7zlg/+3rLOObfBOTfbe70bWAz0JA+PbZR9jaTBHVs/B/qewNqg9+uI/uU3ZA6YaGazLPAsXYDuzrkNEDjRgG7e9Ej77afvI1371tN7XXd6Q/Nzr7ji2eqiDBLf14OAHc65ijrTc87MCoFBwEzy/NjW2VfwybH1c6CP+QByHznVOXcCcCFwnZkNiTJvpP3Oh+8j0X3zwz4/CRwOHA9sAP7sTc+LfTWztsDrwK+cc7uizRpmmq/2N8y++ubY+jnQx3wAuV8459Z7/zcDbxK4xdvk3b7i/d/szR5pv/30faRr39Z5r+tObzCcc5ucc5XOuSrgaQLHFhLf1y0EijsK6kzPGTNrRiDwveice8ObnJfHNty++unY+jnQ58UDyM2sjZm1q34NnA8sILAv1S0QhgNjvddvA1d7rRgGAzu9W+QJwPlm1sm7hTzfm9YQpWXfvM92m9lgr5zz6qB1NQjVQc/zbQLHFgL7eqWZtTCzvkB/ApWPYc9rr5x6KnC5t3zw95Z13vf9DLDYOfdw0Ed5d2wj7auvjm02aq0z9UegJn8ZgZrsW3OdniT34TACte/zgIXV+0Gg3G4KsNz739mbbsDj3j7PB4qC1nUNgYqfFcCPcr1vXppeInBbW04gR3NtOvcNKCLwA/sC+BteJ8AGtK8vePvyOYEA0CNo/lu9dC8lqEVJpPPaO1c+9b6DV4EWOdzX0wgUL3wOzPX+LsrHYxtlX31zbNUzVkQkz/m56EZEROKgQC8ikucU6EVE8pwCvYhInlOgFxHJcwr0IiJ5ToFeRCTPKdCLiOS5/w9S7DZmejWltwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "seq_lengths = [len(x) for x in sequences]\n",
    "plt.plot(seq_lengths)\n",
    "max(seq_lengths)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow, the longest tweet is 117 words long... I'm quite curious about what the tweet is about so let's see."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"@AmyWillRaceAmy1 @alphafemale777 @Paul_Vindici @ncanarchist @Trump_guilty @Nina82997364 @_Ourlittlesatan @iamamber7____ @JillGregory15 @huff_angie @Szyszk_e @itstimetoriseX @hotstreek420 @TuesdayTMoney @TheMJJDynasty @Marjala2 @jacarpb @zacmacme @KellyBu77326182 @Randomacc531 @John_Hamilton_C @WnJSupporter @RealMJFacts @krosodgo @leavingmyseIf @Joquin67208074 @CharlieCatlife @evelyne1370 @Hammertonhal @jak_jeremy @annettaaa @LilMissK111 @ml_roussea @ObserverReport @CherylDiamond18 @averyroseaxl @JpGoLuz @GRosario_pr77 @0pinion8d @_MJBeLike @finehats1 @I_am_DirtyHarry @AntonTerry85 @Michelleheald6 @betrayed_bitch @dash4442 @anthonyking1 @akaurer @Cougarchic I have genius 😉Just because I don't take sad music as concrete proof does not mean I didn't see it.The point was: The COURT already ruled and the man is guilty. And with all the whole that has been shot in it with all those lies. My guess is the appeal won't go far either.\""
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "texts[np.argmax(seq_lengths)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay back to the task, as seen from the graph, the tweets vary quite a lot in length, and maxes out at 117. We will now pad all of them to 117."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_SEQUENCE_LENGTH = max(seq_lengths)\n",
    "\n",
    "data = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating One Hot Encoding Labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of data tensor: (25758, 117)\n",
      "Shape of label tensor: (25758, 6)\n"
     ]
    }
   ],
   "source": [
    "labels = to_categorical(np.asarray(labels))\n",
    "print('Shape of data tensor:', data.shape)\n",
    "print('Shape of label tensor:', labels.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Splitting the data into a train test split\n",
    "\n",
    "What this means is that we will allocate 80% of our cleaned up dataset for training the neural network. After every epoch (cycle) of training, the model will be \"examined\" against the other 20%, named the validation set, to see how well our model generalises (whether our model is 'memorising' its training set, which is undesirable)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we shuffle our sorted data, to make it random first\n",
    "indices = np.arange(data.shape[0])\n",
    "np.random.shuffle(indices)\n",
    "data = data[indices]\n",
    "labels = labels[indices]\n",
    "\n",
    "# before we begin to split them\n",
    "num_training_samples = int(TRAINING_SPLIT * data.shape[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = data[:num_training_samples]\n",
    "y_train = labels[:num_training_samples]\n",
    "x_val = data[num_training_samples:]\n",
    "y_val = labels[num_training_samples:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing Embedding Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_words = min(MAX_NUM_WORDS, len(word_index)) + 1\n",
    "embedding_matrix = np.zeros((num_words, EMBEDDING_DIM))\n",
    "for word, i in word_index.items():\n",
    "    if i > MAX_NUM_WORDS:\n",
    "        continue\n",
    "    embedding_vector = embeddings_index.get(word)\n",
    "    if embedding_vector is not None:\n",
    "        # words not found in embedding index will be all-zeros.\n",
    "        embedding_matrix[i] = embedding_vector\n",
    "\n",
    "# load pre-trained word embeddings into an Embedding layer\n",
    "# note that we set trainable = False so as to keep the embeddings fixed\n",
    "embedding_layer = Embedding(num_words,\n",
    "                            EMBEDDING_DIM,\n",
    "                            embeddings_initializer=Constant(embedding_matrix),\n",
    "                            input_length=MAX_SEQUENCE_LENGTH,\n",
    "                            trainable=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining and creating our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /opt/anaconda/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "WARNING:tensorflow:From /opt/anaconda/lib/python3.7/site-packages/tensorflow/python/keras/layers/core.py:143: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n"
     ]
    }
   ],
   "source": [
    "sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')\n",
    "embedded_sequences = embedding_layer(sequence_input)\n",
    "x = Bidirectional(LSTM(256))(embedded_sequences)\n",
    "y = Dropout(0.5)(x)\n",
    "\n",
    "preds = Dense(labels.shape[1], activation='softmax')(y)\n",
    "model = Model(sequence_input, preds)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         (None, 117)               0         \n",
      "_________________________________________________________________\n",
      "embedding (Embedding)        (None, 117, 100)          6709500   \n",
      "_________________________________________________________________\n",
      "bidirectional (Bidirectional (None, 512)               731136    \n",
      "_________________________________________________________________\n",
      "dropout (Dropout)            (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 6)                 3078      \n",
      "=================================================================\n",
      "Total params: 7,443,714\n",
      "Trainable params: 734,214\n",
      "Non-trainable params: 6,709,500\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='rmsprop',\n",
    "              metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20606 samples, validate on 5152 samples\n",
      "WARNING:tensorflow:From /opt/anaconda/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "Epoch 1/50\n",
      "\n",
      "Epoch 00001: val_acc improved from -inf to 0.93109, saving model to best_weights.hdf5\n",
      " - 193s - loss: 0.6475 - acc: 0.7846 - val_loss: 0.2567 - val_acc: 0.9311\n",
      "Epoch 2/50\n",
      "\n",
      "Epoch 00002: val_acc improved from 0.93109 to 0.97186, saving model to best_weights.hdf5\n",
      " - 193s - loss: 0.1517 - acc: 0.9600 - val_loss: 0.1042 - val_acc: 0.9719\n",
      "Epoch 3/50\n",
      "\n",
      "Epoch 00003: val_acc did not improve from 0.97186\n",
      " - 191s - loss: 0.1092 - acc: 0.9697 - val_loss: 0.0961 - val_acc: 0.9705\n",
      "Epoch 4/50\n",
      "\n",
      "Epoch 00004: val_acc improved from 0.97186 to 0.97651, saving model to best_weights.hdf5\n",
      " - 192s - loss: 0.1118 - acc: 0.9709 - val_loss: 0.0812 - val_acc: 0.9765\n",
      "Epoch 5/50\n",
      "\n",
      "Epoch 00005: val_acc did not improve from 0.97651\n",
      " - 191s - loss: 0.0832 - acc: 0.9765 - val_loss: 0.1011 - val_acc: 0.9742\n",
      "Epoch 6/50\n",
      "\n",
      "Epoch 00006: val_acc did not improve from 0.97651\n",
      " - 190s - loss: 0.0735 - acc: 0.9783 - val_loss: 0.0879 - val_acc: 0.9746\n",
      "Epoch 7/50\n",
      "\n",
      "Epoch 00007: val_acc did not improve from 0.97651\n",
      " - 190s - loss: 0.0666 - acc: 0.9804 - val_loss: 0.0811 - val_acc: 0.9728\n",
      "Epoch 8/50\n",
      "\n",
      "Epoch 00008: val_acc did not improve from 0.97651\n",
      " - 189s - loss: 0.0579 - acc: 0.9826 - val_loss: 0.0847 - val_acc: 0.9759\n",
      "Epoch 9/50\n",
      "\n",
      "Epoch 00009: val_acc improved from 0.97651 to 0.97710, saving model to best_weights.hdf5\n",
      " - 193s - loss: 0.0525 - acc: 0.9845 - val_loss: 0.0942 - val_acc: 0.9771\n",
      "Epoch 10/50\n",
      "\n",
      "Epoch 00010: val_acc did not improve from 0.97710\n",
      " - 189s - loss: 0.0455 - acc: 0.9872 - val_loss: 0.1170 - val_acc: 0.9744\n",
      "Epoch 11/50\n",
      "\n",
      "Epoch 00011: val_acc did not improve from 0.97710\n",
      " - 190s - loss: 0.0430 - acc: 0.9884 - val_loss: 0.1036 - val_acc: 0.9761\n",
      "Epoch 12/50\n",
      "\n",
      "Epoch 00012: val_acc did not improve from 0.97710\n",
      " - 192s - loss: 0.0322 - acc: 0.9915 - val_loss: 0.1079 - val_acc: 0.9759\n",
      "Epoch 13/50\n",
      "\n",
      "Epoch 00013: val_acc did not improve from 0.97710\n",
      " - 189s - loss: 0.0256 - acc: 0.9924 - val_loss: 0.1145 - val_acc: 0.9771\n",
      "Epoch 14/50\n",
      "\n",
      "Epoch 00014: val_acc did not improve from 0.97710\n",
      " - 188s - loss: 0.0245 - acc: 0.9926 - val_loss: 0.1276 - val_acc: 0.9750\n",
      "Epoch 15/50\n",
      "\n",
      "Epoch 00015: val_acc did not improve from 0.97710\n",
      " - 189s - loss: 0.0201 - acc: 0.9942 - val_loss: 0.1459 - val_acc: 0.9748\n",
      "Epoch 16/50\n",
      "\n",
      "Epoch 00016: val_acc did not improve from 0.97710\n",
      " - 188s - loss: 0.0834 - acc: 0.9859 - val_loss: 0.1369 - val_acc: 0.9744\n",
      "Epoch 17/50\n",
      "\n",
      "Epoch 00017: val_acc did not improve from 0.97710\n",
      " - 189s - loss: 0.0152 - acc: 0.9955 - val_loss: 0.1466 - val_acc: 0.9759\n",
      "Epoch 18/50\n",
      "\n",
      "Epoch 00018: val_acc did not improve from 0.97710\n",
      " - 189s - loss: 0.0138 - acc: 0.9961 - val_loss: 0.1523 - val_acc: 0.9755\n",
      "Epoch 19/50\n",
      "\n",
      "Epoch 00019: val_acc did not improve from 0.97710\n",
      " - 190s - loss: 0.0159 - acc: 0.9959 - val_loss: 0.1523 - val_acc: 0.9752\n",
      "Epoch 20/50\n",
      "\n",
      "Epoch 00020: val_acc did not improve from 0.97710\n",
      " - 191s - loss: 0.0125 - acc: 0.9967 - val_loss: 0.1728 - val_acc: 0.9730\n",
      "Epoch 21/50\n",
      "\n",
      "Epoch 00021: val_acc did not improve from 0.97710\n",
      " - 191s - loss: 0.0112 - acc: 0.9968 - val_loss: 0.1787 - val_acc: 0.9738\n",
      "Epoch 22/50\n",
      "\n",
      "Epoch 00022: val_acc did not improve from 0.97710\n",
      " - 190s - loss: 0.0094 - acc: 0.9970 - val_loss: 0.1727 - val_acc: 0.9753\n",
      "Epoch 23/50\n",
      "\n",
      "Epoch 00023: val_acc did not improve from 0.97710\n",
      " - 191s - loss: 0.0118 - acc: 0.9970 - val_loss: 0.1729 - val_acc: 0.9755\n",
      "Epoch 24/50\n",
      "\n",
      "Epoch 00024: val_acc did not improve from 0.97710\n",
      " - 192s - loss: 0.0096 - acc: 0.9978 - val_loss: 0.1869 - val_acc: 0.9767\n",
      "Epoch 25/50\n",
      "\n",
      "Epoch 00025: val_acc did not improve from 0.97710\n",
      " - 191s - loss: 0.0119 - acc: 0.9973 - val_loss: 0.1867 - val_acc: 0.9750\n",
      "Epoch 26/50\n",
      "\n",
      "Epoch 00026: val_acc did not improve from 0.97710\n",
      " - 191s - loss: 0.0074 - acc: 0.9978 - val_loss: 0.1818 - val_acc: 0.9759\n",
      "Epoch 27/50\n",
      "\n",
      "Epoch 00027: val_acc did not improve from 0.97710\n",
      " - 191s - loss: 0.0060 - acc: 0.9985 - val_loss: 0.2048 - val_acc: 0.9765\n",
      "Epoch 28/50\n",
      "\n",
      "Epoch 00028: val_acc did not improve from 0.97710\n",
      " - 190s - loss: 0.0080 - acc: 0.9977 - val_loss: 0.2016 - val_acc: 0.9728\n",
      "Epoch 29/50\n",
      "\n",
      "Epoch 00029: val_acc did not improve from 0.97710\n",
      " - 190s - loss: 0.0079 - acc: 0.9981 - val_loss: 0.2010 - val_acc: 0.9750\n",
      "Epoch 30/50\n",
      "\n",
      "Epoch 00030: val_acc did not improve from 0.97710\n",
      " - 190s - loss: 0.0076 - acc: 0.9985 - val_loss: 0.2037 - val_acc: 0.9750\n",
      "Epoch 31/50\n",
      "\n",
      "Epoch 00031: val_acc did not improve from 0.97710\n",
      " - 191s - loss: 0.0093 - acc: 0.9977 - val_loss: 0.2089 - val_acc: 0.9761\n",
      "DONE\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint\n",
    "\n",
    "checkpoint = ModelCheckpoint(\"best_weights.hdf5\", monitor='val_acc', verbose=1, save_best_only=True, mode='max')\n",
    "earlystopping = EarlyStopping(patience=4, monitor='acc')\n",
    "callbacks = [earlystopping, checkpoint]\n",
    "\n",
    "model.fit(x_train, y_train,\n",
    "          batch_size=32,\n",
    "          epochs=50,\n",
    "          validation_data=(x_val, y_val),\n",
    "          callbacks=callbacks,\n",
    "          verbose=2)\n",
    "\n",
    "model.load_weights(\"best_weights.hdf5\")\n",
    "\n",
    "print(\"DONE\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Making predictions with the model\n",
    "\n",
    "Our model is now trained! We have come to the exciting part where we try to make predictions with our model and see how well it predicts whether a tweet sounds happy or sad. Give it a try with your own input!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_tweet(text_to_predict):\n",
    "    test_sequence = tokenizer.texts_to_sequences([text_to_predict])\n",
    "    test_data = pad_sequences(test_sequence, maxlen=MAX_SEQUENCE_LENGTH)\n",
    "    predictions = model.predict(test_data)\n",
    "    prediction = np.argmax(predictions)\n",
    "    print(\"The person who tweeted this is {}\\n\".format(classes_index[prediction]))\n",
    "    ranking = predictions.argsort()[-6:][::-1]\n",
    "    for i in reversed(range(0, 6)):\n",
    "        rank = ranking[0][i]\n",
    "        print(\"{:>10} : {:.5f}%\".format(classes_index[rank], predictions[0][rank]*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The person who tweeted this is suicidal\n",
      "\n",
      "  suicidal : 63.40141%\n",
      " depressed : 18.13954%\n",
      "  cheerful : 6.18711%\n",
      "       sad : 6.14780%\n",
      "     happy : 5.57230%\n",
      " overjoyed : 0.55182%\n"
     ]
    }
   ],
   "source": [
    "predict_tweet(\"If I just disappear from the world right now, will anyone even care?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The person who tweeted this is depressed\n",
      "\n",
      " depressed : 50.97240%\n",
      "  suicidal : 47.91130%\n",
      "  cheerful : 0.52024%\n",
      "       sad : 0.37566%\n",
      "     happy : 0.12476%\n",
      " overjoyed : 0.09564%\n"
     ]
    }
   ],
   "source": [
    "predict_tweet(\"Nobody can understand the stress and anxiety that exists within me right now. Nobody cares.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The person who tweeted this is sad\n",
      "\n",
      "       sad : 45.75953%\n",
      "     happy : 30.72130%\n",
      " overjoyed : 15.37705%\n",
      "  cheerful : 7.13586%\n",
      " depressed : 0.59173%\n",
      "  suicidal : 0.41452%\n"
     ]
    }
   ],
   "source": [
    "predict_tweet(\"When I'm training my neural network, I can't watch netflix because it lags like crazy. Sigh\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The person who tweeted this is happy\n",
      "\n",
      "     happy : 49.70885%\n",
      "  cheerful : 21.92664%\n",
      "  suicidal : 19.15106%\n",
      "       sad : 4.73569%\n",
      " depressed : 3.61531%\n",
      " overjoyed : 0.86246%\n"
     ]
    }
   ],
   "source": [
    "predict_tweet(\"Today seems like a fine day. I can't wait to go to school today.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The person who tweeted this is cheerful\n",
      "\n",
      "  cheerful : 73.66031%\n",
      "     happy : 16.02193%\n",
      "       sad : 4.32910%\n",
      " depressed : 3.08085%\n",
      "  suicidal : 1.86836%\n",
      " overjoyed : 1.03945%\n"
     ]
    }
   ],
   "source": [
    "predict_tweet(\"Wow your scrambled eggs are delicious!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The person who tweeted this is overjoyed\n",
      "\n",
      " overjoyed : 44.64616%\n",
      "  suicidal : 29.85086%\n",
      "     happy : 9.41580%\n",
      "  cheerful : 7.27963%\n",
      " depressed : 4.47687%\n",
      "       sad : 4.33067%\n"
     ]
    }
   ],
   "source": [
    "predict_tweet(\"I am so excited for Avengers Endgame actually\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(\"mood_detection_model.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}