A Forum-based Chatbot for Parents of Autistic Children.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A Forum-based Chatbot for Parents of Autistic Children\n",
    "\n",
    "**[Khalil Mrini](https://goo.gl/7MCYvq)**\n",
    "\n",
    "# Table of Contents\n",
    "\n",
    "### [1. Scraping the Online Forum](#1)\n",
    "[1.1. Functions](#11) <br/>\n",
    "[1.2. Collected Data](#12) <br/>\n",
    "[1.2.1. Parents' Discussion](#121) <br/>\n",
    "[1.2.2. General Autism Discussion](#122) <br/>\n",
    "[1.3. Filtering Data](#13) <br/>\n",
    "\n",
    "### [2. Amazon Mechanical Turk](#2)\n",
    "[2.1. Sampling Threads](#21) <br/>\n",
    "[2.2. Using sent2vec to Cluster Samples](#22) <br/>\n",
    "[2.3. Checking Amazon Mechanical Turk Results](#23) <br/>\n",
    "[2.4. Using NLTK to Separate Text Data into Sentences](#24) <br/>\n",
    "[2.4.1. For the Samples](#241) <br/>\n",
    "[2.4.2. For All Collected Data](#242) <br/>\n",
    "\n",
    "### [3. Machine Learning Features](#3)\n",
    "[3.1. MTurk DataFrame](#31) <br/>\n",
    "[3.2. Adding Reply Vectors](#32) <br/>\n",
    "[3.3. Adding First Post Vectors](#33) <br/>\n",
    "[3.4. Adding Title Vectors](#34) <br/>\n",
    "[3.5. Computing sent2vec Similarity](#35) <br/>\n",
    "[3.6. Adding Features from Forum Threads](#36) <br/>\n",
    "[3.7. Standardization of Features](#37) <br/>\n",
    "[3.8. Train and Test Data Sets](#38) <br/>\n",
    "[3.9. PCA Visualization](#39) <br/>\n",
    "\n",
    "### [4. Ensemble Classifiers for Multi-Class Labels](#4)\n",
    "[4.1. Random Forest Classifier](#41) <br/>\n",
    "[4.2. Bagging Classifier](#42) <br/>\n",
    "[4.3. Ada Boost Classifier](#43) <br/>\n",
    "[4.4. Extra Trees Classifier](#44) <br/>\n",
    "[4.5. Gradient Boosting Classifier](#45) <br/>\n",
    "[4.6. SVM](#46) <br/>\n",
    "[4.7. Decision Tree Classifier](#47) <br/>\n",
    "[4.8. K-Neighbours Classifier](#48) <br/>\n",
    "[4.9. Radius Neighbours Classifier](#49) <br/>\n",
    "\n",
    "### [5. Building the Chatbot](#5)\n",
    "[5.1. Applying the Model to All Unlabelled Replies](#51) <br/>\n",
    "[5.2. Precomputing word2vec](#52) <br/>\n",
    "[5.3. Dot Product Similarity Functions](#53) <br/>\n",
    "[5.4. Chatbot Functions](#54) <br/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Scraping the Online Forum <a class=\"anchor\" id=\"1\"></a>\n",
    "\n",
    "In this section, we scrape the online forum [Wrong Planet](https://wrongplanet.net/forums/) for posts in the *General Autism Discussion* and the *Parents' Discussion*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import nltk\n",
    "import gensim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1. Functions <a class=\"anchor\" id=\"11\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def find_next(soup):\n",
    "    \"\"\"\n",
    "    Tries to find next page, or returns an empty string if there is no next page.\n",
    "    :param soup: BeautifulSoup\n",
    "    :return: string\n",
    "    \"\"\"\n",
    "    try:\n",
    "        return soup.find_all('a', string=\"Next\")[0]['href'][1:]\n",
    "    except:\n",
    "        return \"\"\n",
    "\n",
    "def find_all_rows(soup, type_name):\n",
    "    \"\"\"\n",
    "    Finds all rows of a soup with the given type name.\n",
    "    :param soup: BeautifulSoup\n",
    "    :param type_name: string\n",
    "    :return: list\n",
    "    \"\"\"\n",
    "    try:\n",
    "        return soup.find_all(type_name, {'class': 'row'})\n",
    "    except:\n",
    "        return []\n",
    "\n",
    "def convert_to_int(text):\n",
    "    \"\"\"\n",
    "    Converts string to integer by eliminating non-digit characters.\n",
    "    :param text: string\n",
    "    :return: int\n",
    "    \"\"\"\n",
    "    return int(''.join([char for char in text if char.isdigit()]))\n",
    "\n",
    "def transform_datetime(datetime):\n",
    "    \"\"\"\n",
    "    Removes mentions of Yesterday and Today.\n",
    "    \"\"\"\n",
    "    try:\n",
    "        return datetime.replace('Yesterday', '24 Oct 2017').replace('Today', '25 Oct 2017')\n",
    "    except:\n",
    "        return datetime\n",
    "\n",
    "def collect_forum_data(soup_row, link=\"\"):\n",
    "    \"\"\"\n",
    "    Collects all relevant data (title, link, text, number of replies and views) of a thread.\n",
    "    :param soup_row: BeautifulSoup\n",
    "    :param link: string\n",
    "    :return: dict\n",
    "    \"\"\"\n",
    "    data = {}\n",
    "    try:\n",
    "        title_soup = soup_row.find('h4').find('a')\n",
    "        data['Title'] = title_soup.text.strip()\n",
    "        data['Link'] = title_soup['href'][1:]\n",
    "        info_soup = soup_row.find_all('td', {'class': 'count-col'})\n",
    "        for info in info_soup:\n",
    "            info_text = info.text.strip()\n",
    "            info_number = convert_to_int(info_text)\n",
    "            if 'repl' in info_text:\n",
    "                data['Replies'] = info_number\n",
    "            if 'view' in info_text:\n",
    "                data['Views'] = info_number\n",
    "    except:\n",
    "        pass\n",
    "    return data\n",
    "\n",
    "def collect_post_data(soup_row, link=\"\"):\n",
    "    \"\"\"\n",
    "    Collects all relevant data of a post.\n",
    "    :param soup_row: BeautifulSoup\n",
    "    :param link: string\n",
    "    :return: dict\n",
    "    \"\"\"\n",
    "    data = {}\n",
    "    try:\n",
    "        data['Link'] = link.replace(PREFIX_URL, '')\n",
    "        user_soup = soup_row.find('div', class_='user-col').find_all('p')\n",
    "        for element in str(user_soup[-1])[3:-4].strip().split('<br/>'):\n",
    "            elements = element.split(': ')\n",
    "            if len(elements) == 2:\n",
    "                if elements[0] == \"Joined\":\n",
    "                    data[elements[0]] = transform_datetime(elements[1])\n",
    "                else:\n",
    "                    data[elements[0]] = elements[1]\n",
    "        try:\n",
    "            data['Username'] = user_soup[0].text.strip().split('\\n')[0]\n",
    "        except:\n",
    "            pass\n",
    "        try:\n",
    "            data['User Level'] = convert_to_int(user_soup[0].find('img')['src'])\n",
    "        except:\n",
    "            pass\n",
    "        message_soup = soup_row.find('div', class_='message-col')\n",
    "        data['Timestamp'] = transform_datetime(message_soup.find('div', class_='timestamp').text.strip())\n",
    "        data['Message'] = message_soup.find('div', class_='message-content').text.strip()\n",
    "    except:\n",
    "        pass\n",
    "    return data\n",
    "\n",
    "def collect_recursively(data, url, type_name, data_function, index=\"\"):\n",
    "    \"\"\"\n",
    "    Collects data recursively using a collection function from the two above.\n",
    "    :param data: dict\n",
    "    :param url: string\n",
    "    :param type_name: string\n",
    "    :param data_function: function\n",
    "    :param index: string\n",
    "    :return: dict\n",
    "    \"\"\"\n",
    "    try:\n",
    "        if index:\n",
    "            print(index, end='\\r', flush=True)\n",
    "        soup = BeautifulSoup(requests.get(url).text, 'html.parser')\n",
    "        soup_rows = find_all_rows(soup, type_name)\n",
    "        data.extend([data_function(soup_row, link=url) for soup_row in soup_rows])\n",
    "        next_url = find_next(soup)\n",
    "        if next_url:\n",
    "            if index:\n",
    "                return collect_recursively(data, PREFIX_URL + next_url, type_name, data_function, index+1)\n",
    "            else:\n",
    "                return collect_recursively(data, PREFIX_URL + next_url, type_name, data_function)\n",
    "        else:\n",
    "            return data\n",
    "    except:\n",
    "        return data\n",
    "\n",
    "def get_forum_dataframe(url, type_name, data_function):\n",
    "    \"\"\"\n",
    "    Launches process to collect a dataframe for all threads of a forum.\n",
    "    :param url: string\n",
    "    :param type_name: string\n",
    "    :param data_function: function\n",
    "    :return: DataFrame\n",
    "    \"\"\"\n",
    "    data = collect_recursively([], url, type_name, data_function, 1)\n",
    "    return pd.DataFrame(data)\n",
    "\n",
    "def get_thread_dataframe(forum_df, type_name, data_function):\n",
    "    \"\"\"\n",
    "    Launches process to collect a dataframe for all posts of a thread.\n",
    "    :param forum_df: DataFrame\n",
    "    :param type_name: string\n",
    "    :param data_function: function\n",
    "    :return: DataFrame\n",
    "    \"\"\"\n",
    "    data = []\n",
    "    total = len(forum_df['Link'])\n",
    "    index = 0\n",
    "    for url in forum_df['Link']:\n",
    "        index += 1\n",
    "        print('{} out of {}'.format(index, total), end='\\r', flush=True)\n",
    "        data.extend(collect_recursively([], PREFIX_URL + url, type_name, data_function))\n",
    "    return pd.DataFrame(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2. Collected Data <a class=\"anchor\" id=\"12\"></a>\n",
    "\n",
    "### 1.2.1. Parents' Discussion <a class=\"anchor\" id=\"121\"></a>\n",
    "\n",
    "We collect the data in DataFrames:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "PREFIX_URL = 'http://wrongplanet.net/forums'\n",
    "START_URL = PREFIX_URL + '/viewforum.php?f=19'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "forum_df = get_forum_dataframe(START_URL, 'tr', collect_forum_data)\n",
    "forum_df.to_json('Forum Data/parents_forum.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "thread_df = get_thread_dataframe(forum_df, 'div', collect_post_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We merge the thread titles with the corresponding posts and save them in *json* format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "parents_df = pd.merge(thread_df, forum_df, on='Link')\n",
    "parents_df.to_json('Forum Data/parents_forum_threads.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2.2. General Autism Discussion <a class=\"anchor\" id=\"122\"></a>\n",
    "\n",
    "We do the same for the *General Autism Discussion*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1462\r"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Link</th>\n",
       "      <th>Replies</th>\n",
       "      <th>Title</th>\n",
       "      <th>Views</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>/viewtopic.php?t=73832&amp;sid=90b20b008fa46184b87...</td>\n",
       "      <td>0</td>\n",
       "      <td>WrongPlanet Rules. You must read this before p...</td>\n",
       "      <td>24985</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>/viewtopic.php?t=102721&amp;sid=90b20b008fa46184b8...</td>\n",
       "      <td>2398</td>\n",
       "      <td>First time in history!! !! The NT/AS open hotl...</td>\n",
       "      <td>670346</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>/viewtopic.php?t=114120&amp;sid=90b20b008fa46184b8...</td>\n",
       "      <td>281</td>\n",
       "      <td>autism reality documentary</td>\n",
       "      <td>590556</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>/viewtopic.php?t=112787&amp;sid=90b20b008fa46184b8...</td>\n",
       "      <td>655</td>\n",
       "      <td>greenturtle74’s Cartoon Guide to Asperger’s</td>\n",
       "      <td>667480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>/viewtopic.php?t=187047&amp;sid=90b20b008fa46184b8...</td>\n",
       "      <td>224</td>\n",
       "      <td>Autism in France: Psychoanalysis, Packing, Oth...</td>\n",
       "      <td>210364</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                Link  Replies  \\\n",
       "0  /viewtopic.php?t=73832&sid=90b20b008fa46184b87...        0   \n",
       "1  /viewtopic.php?t=102721&sid=90b20b008fa46184b8...     2398   \n",
       "2  /viewtopic.php?t=114120&sid=90b20b008fa46184b8...      281   \n",
       "3  /viewtopic.php?t=112787&sid=90b20b008fa46184b8...      655   \n",
       "4  /viewtopic.php?t=187047&sid=90b20b008fa46184b8...      224   \n",
       "\n",
       "                                               Title   Views  \n",
       "0  WrongPlanet Rules. You must read this before p...   24985  \n",
       "1  First time in history!! !! The NT/AS open hotl...  670346  \n",
       "2                         autism reality documentary  590556  \n",
       "3        greenturtle74’s Cartoon Guide to Asperger’s  667480  \n",
       "4  Autism in France: Psychoanalysis, Packing, Oth...  210364  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "GENERAL_URL = 'http://wrongplanet.net/forums/viewforum.php?f=3'\n",
    "general_forum_df = get_forum_dataframe(GENERAL_URL, 'tr', collect_forum_data)\n",
    "general_forum_df.to_json('Forum Data/general_forum.json')\n",
    "general_forum_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "248 out of 248\r"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Joined</th>\n",
       "      <th>Link</th>\n",
       "      <th>Location</th>\n",
       "      <th>Message</th>\n",
       "      <th>Posts</th>\n",
       "      <th>Timestamp</th>\n",
       "      <th>User Level</th>\n",
       "      <th>Username</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>33</td>\n",
       "      <td>Male</td>\n",
       "      <td>27 Sep 2011</td>\n",
       "      <td>/viewtopic.php?t=297204&amp;sid=90b20b008fa46184b8...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>One common mistake people make, even on Wrong ...</td>\n",
       "      <td>724</td>\n",
       "      <td>01 Nov 2015, 12:00 pm</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Comp_Geek_573</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Female</td>\n",
       "      <td>16 Mar 2013</td>\n",
       "      <td>/viewtopic.php?t=297204&amp;sid=90b20b008fa46184b8...</td>\n",
       "      <td>Earth</td>\n",
       "      <td>http://www.cdc.gov/ncbddd/autism/hcp-dsm.html</td>\n",
       "      <td>2,173</td>\n",
       "      <td>01 Nov 2015, 12:03 pm</td>\n",
       "      <td>5.0</td>\n",
       "      <td>iliketrees</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>49</td>\n",
       "      <td>Female</td>\n",
       "      <td>1 Jul 2009</td>\n",
       "      <td>/viewtopic.php?t=297204&amp;sid=90b20b008fa46184b8...</td>\n",
       "      <td>Los Angeles, CA</td>\n",
       "      <td>How would any of us define autism outside of t...</td>\n",
       "      <td>499</td>\n",
       "      <td>01 Nov 2015, 2:01 pm</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NowhereWoman</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>26</td>\n",
       "      <td>NaN</td>\n",
       "      <td>19 Jun 2015</td>\n",
       "      <td>/viewtopic.php?t=297204&amp;sid=90b20b008fa46184b8...</td>\n",
       "      <td>Wyoming</td>\n",
       "      <td>That is a good point it is a spectum on severa...</td>\n",
       "      <td>81</td>\n",
       "      <td>01 Nov 2015, 2:24 pm</td>\n",
       "      <td>1.0</td>\n",
       "      <td>GreenPandaLord</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>28</td>\n",
       "      <td>Female</td>\n",
       "      <td>14 Oct 2008</td>\n",
       "      <td>/viewtopic.php?t=85651&amp;sid=90b20b008fa46184b87...</td>\n",
       "      <td>Nowhere</td>\n",
       "      <td>No this post is not bashing all neurotypicals,...</td>\n",
       "      <td>894</td>\n",
       "      <td>14 Dec 2008, 9:17 pm</td>\n",
       "      <td>5.0</td>\n",
       "      <td>timeisdead</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Age  Gender       Joined  \\\n",
       "0   33    Male  27 Sep 2011   \n",
       "1  NaN  Female  16 Mar 2013   \n",
       "2   49  Female   1 Jul 2009   \n",
       "3   26     NaN  19 Jun 2015   \n",
       "4   28  Female  14 Oct 2008   \n",
       "\n",
       "                                                Link         Location  \\\n",
       "0  /viewtopic.php?t=297204&sid=90b20b008fa46184b8...              NaN   \n",
       "1  /viewtopic.php?t=297204&sid=90b20b008fa46184b8...            Earth   \n",
       "2  /viewtopic.php?t=297204&sid=90b20b008fa46184b8...  Los Angeles, CA   \n",
       "3  /viewtopic.php?t=297204&sid=90b20b008fa46184b8...          Wyoming   \n",
       "4  /viewtopic.php?t=85651&sid=90b20b008fa46184b87...          Nowhere   \n",
       "\n",
       "                                             Message  Posts  \\\n",
       "0  One common mistake people make, even on Wrong ...    724   \n",
       "1      http://www.cdc.gov/ncbddd/autism/hcp-dsm.html  2,173   \n",
       "2  How would any of us define autism outside of t...    499   \n",
       "3  That is a good point it is a spectum on severa...     81   \n",
       "4  No this post is not bashing all neurotypicals,...    894   \n",
       "\n",
       "               Timestamp  User Level        Username  \n",
       "0  01 Nov 2015, 12:00 pm         5.0   Comp_Geek_573  \n",
       "1  01 Nov 2015, 12:03 pm         5.0      iliketrees  \n",
       "2   01 Nov 2015, 2:01 pm         4.0    NowhereWoman  \n",
       "3   01 Nov 2015, 2:24 pm         1.0  GreenPandaLord  \n",
       "4   14 Dec 2008, 9:17 pm         5.0      timeisdead  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "general_thread_df = get_thread_dataframe(general_forum_df, 'div', collect_post_data)\n",
    "general_thread_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "general_merged_df = pd.merge(general_thread_df, general_forum_df, on='Link')\n",
    "general_merged_df.to_json('general_forum_threads.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3. Filtering Data <a class=\"anchor\" id=\"13\"></a>\n",
    "\n",
    "In this subsection, we filter the threads to only keep those having questions as titles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "forum_threads = pd.concat([pd.read_json('general_forum_threads.json'), \n",
    "                           pd.read_json('parents_forum_threads.json')]).reset_index(drop=True)\n",
    "forum_subjects = pd.concat([pd.read_json('general_forum.json'), \n",
    "                            pd.read_json('parents_forum.json')]).reset_index(drop=True)\n",
    "forum_threads.Link = forum_threads.Link.apply(lambda row: row.split('t=')[1].split('&')[0])\n",
    "forum_threads.Message = forum_threads.Message.apply(lambda msg: msg.split('_________________')[0])\n",
    "forum_subjects.Link = forum_subjects.Link.apply(lambda row: row.split('t=')[1].split('&')[0])\n",
    "forum_subjects = forum_subjects.drop_duplicates(subset=['Link'], keep='first')\n",
    "forum_threads = forum_threads.drop_duplicates(subset=['Link', 'Message'], keep='first')\n",
    "forum_threads.to_json('forum_threads.json')\n",
    "forum_subjects.to_json('forum_subjects.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The total number of posts is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "701337"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(forum_threads)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The total number of threads is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "82383"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(forum_subjects)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function defines what is a question:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def is_question(sentence):\n",
    "    \"\"\"\n",
    "    Returns true if a sentence is a question.\n",
    "    :param sentence: list of strings, the sentence is tokenized\n",
    "    :return: boolean\n",
    "    \"\"\"\n",
    "    new_sentence = (''.join([c for c in sentence if c.isalnum() or c in '?!./ \"\\''])).replace('/', ' ')\n",
    "    tokens = nltk.word_tokenize(new_sentence)\n",
    "    tokens = [word[:1].lower() + word[1:] for word in tokens]\n",
    "    try:\n",
    "        question_mark_index = tokens.index('?')\n",
    "        if question_mark_index == len(tokens) - 1:\n",
    "            tags = nltk.pos_tag(tokens[:question_mark_index + 1])\n",
    "            if tags[0][1] in 'MD VB VBD VBN VBP VBZ WRB WP WDT'.split(' ') and tags[0][0] not in 'am'.split(' '):\n",
    "                for tag in tags:\n",
    "                    if tag[1].startswith('VB'):\n",
    "                        return True\n",
    "    except:\n",
    "        pass\n",
    "    return False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15708"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filtered_forum_subjects = forum_subjects[forum_subjects['Title'].map(lambda x: is_question(x))]\n",
    "len(filtered_forum_subjects)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Link</th>\n",
       "      <th>Replies</th>\n",
       "      <th>Title</th>\n",
       "      <th>Views</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>287074</td>\n",
       "      <td>18</td>\n",
       "      <td>Are you a multipotentialite?</td>\n",
       "      <td>1176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>293067</td>\n",
       "      <td>9</td>\n",
       "      <td>Why is my performance IQ so low?</td>\n",
       "      <td>979</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>293065</td>\n",
       "      <td>13</td>\n",
       "      <td>Are individuals with Autism often taller than ...</td>\n",
       "      <td>572</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>293055</td>\n",
       "      <td>8</td>\n",
       "      <td>Do you ever wonder if NT people are missing out?</td>\n",
       "      <td>356</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>292656</td>\n",
       "      <td>33</td>\n",
       "      <td>Does your diagnosis make you feel like a speci...</td>\n",
       "      <td>1365</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Link  Replies                                              Title  Views\n",
       "9   287074       18                       Are you a multipotentialite?   1176\n",
       "11  293067        9                   Why is my performance IQ so low?    979\n",
       "17  293065       13  Are individuals with Autism often taller than ...    572\n",
       "19  293055        8   Do you ever wonder if NT people are missing out?    356\n",
       "20  292656       33  Does your diagnosis make you feel like a speci...   1365"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filtered_forum_subjects.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Joined</th>\n",
       "      <th>Link</th>\n",
       "      <th>Location</th>\n",
       "      <th>Message</th>\n",
       "      <th>Posts</th>\n",
       "      <th>Timestamp</th>\n",
       "      <th>User Level</th>\n",
       "      <th>Username</th>\n",
       "      <th>Replies</th>\n",
       "      <th>Title</th>\n",
       "      <th>Views</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>17.0</td>\n",
       "      <td>Female</td>\n",
       "      <td>1.33989e+12</td>\n",
       "      <td>287074</td>\n",
       "      <td>None</td>\n",
       "      <td>Can anyone else relate to this video?\\n</td>\n",
       "      <td>346</td>\n",
       "      <td>1432979760000</td>\n",
       "      <td>4.0</td>\n",
       "      <td>physicsnut42</td>\n",
       "      <td>18</td>\n",
       "      <td>Are you a multipotentialite?</td>\n",
       "      <td>1176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Male</td>\n",
       "      <td>1.39147e+12</td>\n",
       "      <td>287074</td>\n",
       "      <td>Queens, NYC</td>\n",
       "      <td>Yep...I believe people should make use of all ...</td>\n",
       "      <td>37,484</td>\n",
       "      <td>1432981020000</td>\n",
       "      <td>5.0</td>\n",
       "      <td>kraftiekortie</td>\n",
       "      <td>18</td>\n",
       "      <td>Are you a multipotentialite?</td>\n",
       "      <td>1176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>34.0</td>\n",
       "      <td>Male</td>\n",
       "      <td>1.27293e+12</td>\n",
       "      <td>287074</td>\n",
       "      <td>was Louisiana but now Vermont</td>\n",
       "      <td>I'm not. I never had many interests.\\n</td>\n",
       "      <td>17,363</td>\n",
       "      <td>1432999500000</td>\n",
       "      <td>5.0</td>\n",
       "      <td>nick007</td>\n",
       "      <td>18</td>\n",
       "      <td>Are you a multipotentialite?</td>\n",
       "      <td>1176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Male</td>\n",
       "      <td>1.42033e+12</td>\n",
       "      <td>287074</td>\n",
       "      <td>Europe</td>\n",
       "      <td>Like physicsnut I can relate to this video. I'...</td>\n",
       "      <td>956</td>\n",
       "      <td>1433163360000</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Hyperborean</td>\n",
       "      <td>18</td>\n",
       "      <td>Are you a multipotentialite?</td>\n",
       "      <td>1176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>18.0</td>\n",
       "      <td>Male</td>\n",
       "      <td>1.39242e+12</td>\n",
       "      <td>287074</td>\n",
       "      <td>Australia</td>\n",
       "      <td>To an extent...I have 3 main interests, music,...</td>\n",
       "      <td>2,879</td>\n",
       "      <td>1433057160000</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Outrider</td>\n",
       "      <td>18</td>\n",
       "      <td>Are you a multipotentialite?</td>\n",
       "      <td>1176</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Age  Gender       Joined    Link                       Location  \\\n",
       "0  17.0  Female  1.33989e+12  287074                           None   \n",
       "1   NaN    Male  1.39147e+12  287074                    Queens, NYC   \n",
       "2  34.0    Male  1.27293e+12  287074  was Louisiana but now Vermont   \n",
       "3   NaN    Male  1.42033e+12  287074                         Europe   \n",
       "4  18.0    Male  1.39242e+12  287074                      Australia   \n",
       "\n",
       "                                             Message   Posts      Timestamp  \\\n",
       "0            Can anyone else relate to this video?\\n     346  1432979760000   \n",
       "1  Yep...I believe people should make use of all ...  37,484  1432981020000   \n",
       "2             I'm not. I never had many interests.\\n  17,363  1432999500000   \n",
       "3  Like physicsnut I can relate to this video. I'...     956  1433163360000   \n",
       "4  To an extent...I have 3 main interests, music,...   2,879  1433057160000   \n",
       "\n",
       "   User Level       Username  Replies                         Title  Views  \n",
       "0         4.0   physicsnut42       18  Are you a multipotentialite?   1176  \n",
       "1         5.0  kraftiekortie       18  Are you a multipotentialite?   1176  \n",
       "2         5.0        nick007       18  Are you a multipotentialite?   1176  \n",
       "3         5.0    Hyperborean       18  Are you a multipotentialite?   1176  \n",
       "4         5.0       Outrider       18  Are you a multipotentialite?   1176  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "merged_forum_threads = pd.merge(forum_threads.drop(['Views', 'Replies', 'Title'], axis=1), forum_subjects, \n",
    "                                on='Link', how='inner')\n",
    "merged_forum_threads.to_json('merged_forum_threads.json')\n",
    "merged_forum_threads.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "701337"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(merged_forum_threads)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Link</th>\n",
       "      <th>Title</th>\n",
       "      <th>Seeker</th>\n",
       "      <th>First Post</th>\n",
       "      <th>Replies</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100003</td>\n",
       "      <td>Okay, so what's up with this no gluten or milk...</td>\n",
       "      <td>wigglyspider</td>\n",
       "      <td>I can't find any info about it for adults, it'...</td>\n",
       "      <td>45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>100005</td>\n",
       "      <td>Who has DESPERATELY tried to fit in at times i...</td>\n",
       "      <td>millie</td>\n",
       "      <td>Pondering this came out of a consideration of ...</td>\n",
       "      <td>55</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>100007</td>\n",
       "      <td>Getting Diagnosed?</td>\n",
       "      <td>Toco</td>\n",
       "      <td>Hey guys, I've just joined, and posted an intr...</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>100020</td>\n",
       "      <td>Stimming, nervous habit, or other?</td>\n",
       "      <td>fiddlerpianist</td>\n",
       "      <td>So I've done this \"thing\" since I was fairly l...</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>100044</td>\n",
       "      <td>AS / HFA / PDD-NOS ???</td>\n",
       "      <td>littlemissfickle</td>\n",
       "      <td>i have always known there is something wrong w...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Link                                              Title  \\\n",
       "0  100003  Okay, so what's up with this no gluten or milk...   \n",
       "1  100005  Who has DESPERATELY tried to fit in at times i...   \n",
       "2  100007                                 Getting Diagnosed?   \n",
       "3  100020                 Stimming, nervous habit, or other?   \n",
       "4  100044                             AS / HFA / PDD-NOS ???   \n",
       "\n",
       "             Seeker                                         First Post  \\\n",
       "0      wigglyspider  I can't find any info about it for adults, it'...   \n",
       "1            millie  Pondering this came out of a consideration of ...   \n",
       "2              Toco  Hey guys, I've just joined, and posted an intr...   \n",
       "3    fiddlerpianist  So I've done this \"thing\" since I was fairly l...   \n",
       "4  littlemissfickle  i have always known there is something wrong w...   \n",
       "\n",
       "   Replies  \n",
       "0       45  \n",
       "1       55  \n",
       "2        8  \n",
       "3       16  \n",
       "4        3  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles_with_first_post = merged_forum_threads.groupby('Link').first().reset_index()[['Link', 'Title', 'Username', 'Message', 'Replies']]\n",
    "titles_with_first_post.columns = ['Link', 'Title', 'Seeker', 'First Post', 'Replies']\n",
    "titles_with_first_post.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "36659"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(titles_with_first_post)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "36659"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(merged_forum_threads.Link.unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "36659"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(forum_threads.Link.unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "45724"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "links = list(forum_subjects.Link)\n",
    "other = set(forum_threads.Link.unique())\n",
    "excluded_titles = [link for link in links if link not in other]\n",
    "len(excluded_titles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15684"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles = set(filtered_forum_subjects['Title'])\n",
    "filter_titles_with_first_post = titles_with_first_post[titles_with_first_post['Title'].map(lambda x: x in titles)]\n",
    "len(filter_titles_with_first_post)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    15684.000000\n",
       "mean       854.645754\n",
       "std       1101.757246\n",
       "min          0.000000\n",
       "25%        261.000000\n",
       "50%        547.000000\n",
       "75%       1052.250000\n",
       "max      33687.000000\n",
       "Name: First Post, dtype: float64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first_post_lengths = filter_titles_with_first_post['First Post'].map(lambda x: len(x))\n",
    "first_post_lengths.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    15708.000000\n",
       "mean        23.662592\n",
       "std         26.113242\n",
       "min          0.000000\n",
       "25%          8.000000\n",
       "50%         17.000000\n",
       "75%         31.000000\n",
       "max        883.000000\n",
       "Name: Replies, dtype: float64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filtered_forum_subjects.Replies.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9347"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_len = 1053\n",
    "min_reply = 8\n",
    "filter_titles_with_first_post = filter_titles_with_first_post[filter_titles_with_first_post['Replies'].map(lambda x: x >= min_reply)]\n",
    "filter_titles_with_first_post = filter_titles_with_first_post[filter_titles_with_first_post['First Post'].map(lambda x: len(x) <= max_len)]\n",
    "len(filter_titles_with_first_post)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Link</th>\n",
       "      <th>Title</th>\n",
       "      <th>Seeker</th>\n",
       "      <th>First Post</th>\n",
       "      <th>Replier</th>\n",
       "      <th>Message</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>100058</td>\n",
       "      <td>How Do YOU Hold Your Glass?</td>\n",
       "      <td>KarmicPyxis</td>\n",
       "      <td>What do you think of http://news.bbc.co.uk/2/h...</td>\n",
       "      <td>Justin6378</td>\n",
       "      <td>I hold my glass with both hands, for fear of s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>100058</td>\n",
       "      <td>How Do YOU Hold Your Glass?</td>\n",
       "      <td>KarmicPyxis</td>\n",
       "      <td>What do you think of http://news.bbc.co.uk/2/h...</td>\n",
       "      <td>typ3</td>\n",
       "      <td>Whut. I occasionally do all of those positions...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>100058</td>\n",
       "      <td>How Do YOU Hold Your Glass?</td>\n",
       "      <td>KarmicPyxis</td>\n",
       "      <td>What do you think of http://news.bbc.co.uk/2/h...</td>\n",
       "      <td>KingdomOfRats</td>\n",
       "      <td>do not use a glass,am hold cups/cans/bottles i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>100058</td>\n",
       "      <td>How Do YOU Hold Your Glass?</td>\n",
       "      <td>KarmicPyxis</td>\n",
       "      <td>What do you think of http://news.bbc.co.uk/2/h...</td>\n",
       "      <td>1234</td>\n",
       "      <td>my pinky sticks out, like those posh people ho...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>100058</td>\n",
       "      <td>How Do YOU Hold Your Glass?</td>\n",
       "      <td>KarmicPyxis</td>\n",
       "      <td>What do you think of http://news.bbc.co.uk/2/h...</td>\n",
       "      <td>Lecks</td>\n",
       "      <td>1234 wrote:my pinky sticks out, like those pos...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Link                        Title       Seeker  \\\n",
       "2  100058  How Do YOU Hold Your Glass?  KarmicPyxis   \n",
       "3  100058  How Do YOU Hold Your Glass?  KarmicPyxis   \n",
       "5  100058  How Do YOU Hold Your Glass?  KarmicPyxis   \n",
       "6  100058  How Do YOU Hold Your Glass?  KarmicPyxis   \n",
       "7  100058  How Do YOU Hold Your Glass?  KarmicPyxis   \n",
       "\n",
       "                                          First Post        Replier  \\\n",
       "2  What do you think of http://news.bbc.co.uk/2/h...     Justin6378   \n",
       "3  What do you think of http://news.bbc.co.uk/2/h...           typ3   \n",
       "5  What do you think of http://news.bbc.co.uk/2/h...  KingdomOfRats   \n",
       "6  What do you think of http://news.bbc.co.uk/2/h...           1234   \n",
       "7  What do you think of http://news.bbc.co.uk/2/h...          Lecks   \n",
       "\n",
       "                                             Message  \n",
       "2  I hold my glass with both hands, for fear of s...  \n",
       "3  Whut. I occasionally do all of those positions...  \n",
       "5  do not use a glass,am hold cups/cans/bottles i...  \n",
       "6  my pinky sticks out, like those posh people ho...  \n",
       "7  1234 wrote:my pinky sticks out, like those pos...  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles_with_messages = pd.merge(filter_titles_with_first_post.drop(['Replies'], axis=1), merged_forum_threads[['Username', 'Message', 'Link']], on='Link')\n",
    "titles_with_messages.rename(columns={'Username':'Replier'}, inplace=True)\n",
    "titles_with_messages = titles_with_messages[titles_with_messages.apply(lambda row: not row['First Post'] == row['Message'] and not row['Seeker'] == row['Replier'], axis=1)]\n",
    "titles_with_messages.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "260880"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(titles_with_messages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9346"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(titles_with_messages.Link.unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "file = open('text.txt', 'w')\n",
    "[print(title, file=file) for title in filter_titles_with_first_post.Title]\n",
    "file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Amazon Mechanical Turk <a class=\"anchor\" id=\"2\"></a>\n",
    "\n",
    "## 2.1. Sampling Threads <a class=\"anchor\" id=\"21\"></a>\n",
    "\n",
    "In this subsection, we sample threads which replies will be labelled in Amazon Mechanical Turk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Link</th>\n",
       "      <th>Title</th>\n",
       "      <th>Seeker</th>\n",
       "      <th>First Post</th>\n",
       "      <th>Replier</th>\n",
       "      <th>Message</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>348</th>\n",
       "      <td>100346</td>\n",
       "      <td>What are your favorites foods - how often do y...</td>\n",
       "      <td>AspieCard</td>\n",
       "      <td>I have several favorite meals.\\n\\nMy favorite ...</td>\n",
       "      <td>Spokane_Girl</td>\n",
       "      <td>Macaroni and Cheese. I will cook it for dinner...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>349</th>\n",
       "      <td>100346</td>\n",
       "      <td>What are your favorites foods - how often do y...</td>\n",
       "      <td>AspieCard</td>\n",
       "      <td>I have several favorite meals.\\n\\nMy favorite ...</td>\n",
       "      <td>Xs142</td>\n",
       "      <td>Pancakes,\\ndaily,\\ngirlfriend threw me out of ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>350</th>\n",
       "      <td>100346</td>\n",
       "      <td>What are your favorites foods - how often do y...</td>\n",
       "      <td>AspieCard</td>\n",
       "      <td>I have several favorite meals.\\n\\nMy favorite ...</td>\n",
       "      <td>AJCoyne</td>\n",
       "      <td>CHOCOLATE! !! !\\n    \\nI am a chocolate MONSTE...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>352</th>\n",
       "      <td>100346</td>\n",
       "      <td>What are your favorites foods - how often do y...</td>\n",
       "      <td>AspieCard</td>\n",
       "      <td>I have several favorite meals.\\n\\nMy favorite ...</td>\n",
       "      <td>millie</td>\n",
       "      <td>Many people with ASD's tend towards a restrict...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>353</th>\n",
       "      <td>100346</td>\n",
       "      <td>What are your favorites foods - how often do y...</td>\n",
       "      <td>AspieCard</td>\n",
       "      <td>I have several favorite meals.\\n\\nMy favorite ...</td>\n",
       "      <td>wigglyspider</td>\n",
       "      <td>I liiiike...\\n\\nCalamari with garlic aioli sau...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Link                                              Title     Seeker  \\\n",
       "348  100346  What are your favorites foods - how often do y...  AspieCard   \n",
       "349  100346  What are your favorites foods - how often do y...  AspieCard   \n",
       "350  100346  What are your favorites foods - how often do y...  AspieCard   \n",
       "352  100346  What are your favorites foods - how often do y...  AspieCard   \n",
       "353  100346  What are your favorites foods - how often do y...  AspieCard   \n",
       "\n",
       "                                            First Post       Replier  \\\n",
       "348  I have several favorite meals.\\n\\nMy favorite ...  Spokane_Girl   \n",
       "349  I have several favorite meals.\\n\\nMy favorite ...         Xs142   \n",
       "350  I have several favorite meals.\\n\\nMy favorite ...       AJCoyne   \n",
       "352  I have several favorite meals.\\n\\nMy favorite ...        millie   \n",
       "353  I have several favorite meals.\\n\\nMy favorite ...  wigglyspider   \n",
       "\n",
       "                                               Message  \n",
       "348  Macaroni and Cheese. I will cook it for dinner...  \n",
       "349  Pancakes,\\ndaily,\\ngirlfriend threw me out of ...  \n",
       "350  CHOCOLATE! !! !\\n    \\nI am a chocolate MONSTE...  \n",
       "352  Many people with ASD's tend towards a restrict...  \n",
       "353  I liiiike...\\n\\nCalamari with garlic aioli sau...  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_titles = set(filter_titles_with_first_post.sample(frac=0.04)['Title'])\n",
    "sample_titles_with_messages = titles_with_messages[titles_with_messages['Title'].map(lambda x: x in sample_titles)]\n",
    "sample_titles_with_messages = sample_titles_with_messages[sample_titles_with_messages.Message.map(lambda x: len(x) <= max_len)]\n",
    "sample_titles_with_messages.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9288"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_titles_with_messages.to_csv('amazon_mechanical_turk.csv', index=False)\n",
    "len(sample_titles_with_messages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    260880.000000\n",
       "mean        520.717537\n",
       "std         687.418261\n",
       "min           0.000000\n",
       "25%         133.000000\n",
       "50%         307.000000\n",
       "75%         655.000000\n",
       "max       34845.000000\n",
       "Name: Message, dtype: float64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles_with_messages.Message.map(len).describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2. Using sent2vec to Cluster Samples <a class=\"anchor\" id=\"22\"></a>\n",
    "\n",
    "Fetching vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 9347 vectors with 700 dimensions.\n"
     ]
    }
   ],
   "source": [
    "vectors = [[float(number) for number in line.strip().split(' ')] for line in open('vectors.txt', 'r').readlines()]\n",
    "print(\"There are {} vectors with {} dimensions.\".format(len(vectors), len(vectors[0])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fetching corresponding sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9347"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentences = [line.strip() for line in open('text.txt', 'r').readlines()]\n",
    "len(sentences)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "K-means clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.cluster import KMeans\n",
    "n_clusters = int(0.04 * len(sentences))\n",
    "kmeans = KMeans(n_clusters=n_clusters, max_iter=10000, tol=0.0001).fit_predict(np.array(vectors), sentences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9347"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(kmeans)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "373"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(np.unique(kmeans))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cluster_dict = {}\n",
    "for index in range(n_clusters):\n",
    "    cluster_dict[index] = []\n",
    "for index in range(len(kmeans)):\n",
    "    cluster_dict[kmeans[index]].append(index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    373.000000\n",
       "mean      25.058981\n",
       "std       31.814766\n",
       "min        1.000000\n",
       "25%        5.000000\n",
       "50%       14.000000\n",
       "75%       36.000000\n",
       "max      373.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series([len(cluster_dict[cluster]) for cluster in cluster_dict]).describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Does your social ineptness make you feel like a jerk?',\n",
       " 'Do you resent not living up to your parents expectations?',\n",
       " 'Can Anyone Explain this Phenomenon?',\n",
       " 'What, exactly, is empathy?',\n",
       " 'How much/how little do you sleep?']"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "samples = [sentences[np.random.choice(cluster_dict[cluster])] for cluster in cluster_dict]\n",
    "samples[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Link</th>\n",
       "      <th>Title</th>\n",
       "      <th>Seeker</th>\n",
       "      <th>First Post</th>\n",
       "      <th>Replier</th>\n",
       "      <th>Message</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>100116</td>\n",
       "      <td>Do you practice facial expressions?</td>\n",
       "      <td>Justin6378</td>\n",
       "      <td>I often sit in my room and practise smiles, fr...</td>\n",
       "      <td>Fo-Rum</td>\n",
       "      <td>I did this on days I knew I was going to have ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>100116</td>\n",
       "      <td>Do you practice facial expressions?</td>\n",
       "      <td>Justin6378</td>\n",
       "      <td>I often sit in my room and practise smiles, fr...</td>\n",
       "      <td>Sora</td>\n",
       "      <td>Sure! Helped plenty for me.\\n\\nI practised in ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66</th>\n",
       "      <td>100116</td>\n",
       "      <td>Do you practice facial expressions?</td>\n",
       "      <td>Justin6378</td>\n",
       "      <td>I often sit in my room and practise smiles, fr...</td>\n",
       "      <td>outlier</td>\n",
       "      <td>I don't do this, but maybe I should try. My fa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>100116</td>\n",
       "      <td>Do you practice facial expressions?</td>\n",
       "      <td>Justin6378</td>\n",
       "      <td>I often sit in my room and practise smiles, fr...</td>\n",
       "      <td>marshall</td>\n",
       "      <td>I have trouble smiling in pictures. I look cre...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>100116</td>\n",
       "      <td>Do you practice facial expressions?</td>\n",
       "      <td>Justin6378</td>\n",
       "      <td>I often sit in my room and practise smiles, fr...</td>\n",
       "      <td>the_phoenix</td>\n",
       "      <td>This reminds me of a scene from the Addams Fam...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Link                                Title      Seeker  \\\n",
       "64  100116  Do you practice facial expressions?  Justin6378   \n",
       "65  100116  Do you practice facial expressions?  Justin6378   \n",
       "66  100116  Do you practice facial expressions?  Justin6378   \n",
       "67  100116  Do you practice facial expressions?  Justin6378   \n",
       "68  100116  Do you practice facial expressions?  Justin6378   \n",
       "\n",
       "                                           First Post      Replier  \\\n",
       "64  I often sit in my room and practise smiles, fr...       Fo-Rum   \n",
       "65  I often sit in my room and practise smiles, fr...         Sora   \n",
       "66  I often sit in my room and practise smiles, fr...      outlier   \n",
       "67  I often sit in my room and practise smiles, fr...     marshall   \n",
       "68  I often sit in my room and practise smiles, fr...  the_phoenix   \n",
       "\n",
       "                                              Message  \n",
       "64  I did this on days I knew I was going to have ...  \n",
       "65  Sure! Helped plenty for me.\\n\\nI practised in ...  \n",
       "66  I don't do this, but maybe I should try. My fa...  \n",
       "67  I have trouble smiling in pictures. I look cre...  \n",
       "68  This reminds me of a scene from the Addams Fam...  "
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_len = 1053\n",
    "sample_titles_with_messages = titles_with_messages[titles_with_messages['Title'].map(lambda x: x in samples)]\n",
    "sample_titles_with_messages = sample_titles_with_messages[sample_titles_with_messages.Message.map(lambda x: len(x) <= max_len)]\n",
    "sample_titles_with_messages.to_csv('amazon_mechanical_turk.csv', index=False)\n",
    "sample_titles_with_messages.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9758"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(sample_titles_with_messages)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3. Checking Amazon Mechanical Turk Results <a class=\"anchor\" id=\"23\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>HITID</th>\n",
       "      <th>Link</th>\n",
       "      <th>Title</th>\n",
       "      <th>Seeker</th>\n",
       "      <th>First_Post</th>\n",
       "      <th>Replier</th>\n",
       "      <th>Message</th>\n",
       "      <th>Worker</th>\n",
       "      <th>Answer</th>\n",
       "      <th>Date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3K3G488TR2G3Y0HX29ACGBE4YKZQ5Z</td>\n",
       "      <td>160755</td>\n",
       "      <td>Are Simon Baron-Cohen and Tony Attwood respons...</td>\n",
       "      <td>simfish</td>\n",
       "      <td>What about other top autism researchers?</td>\n",
       "      <td>Indy</td>\n",
       "      <td>Don't know about emails, but there are lots of...</td>\n",
       "      <td>A2QLSHXNCHBRN4</td>\n",
       "      <td>Somewhat Useful</td>\n",
       "      <td>2017-11-09 21:41:01 UTC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3OLZC0DJ8JN7FW8TVSNGB7HHKYKIVV</td>\n",
       "      <td>165438</td>\n",
       "      <td>Is stimming involuntary?</td>\n",
       "      <td>cnidocyte</td>\n",
       "      <td>For the past 2 months I've got into the habit ...</td>\n",
       "      <td>Todesking</td>\n",
       "      <td>In second grade I use to rock back and forth u...</td>\n",
       "      <td>A2M5VW97GIYLHB</td>\n",
       "      <td>Neutral</td>\n",
       "      <td>2017-11-09 21:41:23 UTC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>36AZSFEYZ48XSOUNPIVZPKD4P80VBZ</td>\n",
       "      <td>161882</td>\n",
       "      <td>How toughen up?</td>\n",
       "      <td>Jacs</td>\n",
       "      <td>How do you toughen up? I have always been a ve...</td>\n",
       "      <td>CockneyRebel</td>\n",
       "      <td>I'm also a very sensitive person. I can't real...</td>\n",
       "      <td>A166A2M31CW2C7</td>\n",
       "      <td>Somewhat Useless</td>\n",
       "      <td>2017-11-09 21:40:56 UTC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3WGZLY9VCHUSW9WUVLBVSWHWHVBD8J</td>\n",
       "      <td>106087</td>\n",
       "      <td>Why are we bad at sports?</td>\n",
       "      <td>bluerose</td>\n",
       "      <td>I mean, what is the mechanism behind it. Somet...</td>\n",
       "      <td>Kaysea</td>\n",
       "      <td>Wolfen wrote:Wow, interesting somehow.For me i...</td>\n",
       "      <td>A2CK0OXMPOR9LE</td>\n",
       "      <td>Useful</td>\n",
       "      <td>2017-11-09 21:41:56 UTC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3IKMEYR0LW3PO00F7LOHMIZJF4H2KJ</td>\n",
       "      <td>106087</td>\n",
       "      <td>Why are we bad at sports?</td>\n",
       "      <td>bluerose</td>\n",
       "      <td>I mean, what is the mechanism behind it. Somet...</td>\n",
       "      <td>Wolfen</td>\n",
       "      <td>Wow, interesting somehow.\\n\\nFor me it's reall...</td>\n",
       "      <td>A2CK0OXMPOR9LE</td>\n",
       "      <td>Useful</td>\n",
       "      <td>2017-11-09 21:41:37 UTC</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            HITID    Link  \\\n",
       "0  3K3G488TR2G3Y0HX29ACGBE4YKZQ5Z  160755   \n",
       "1  3OLZC0DJ8JN7FW8TVSNGB7HHKYKIVV  165438   \n",
       "2  36AZSFEYZ48XSOUNPIVZPKD4P80VBZ  161882   \n",
       "3  3WGZLY9VCHUSW9WUVLBVSWHWHVBD8J  106087   \n",
       "4  3IKMEYR0LW3PO00F7LOHMIZJF4H2KJ  106087   \n",
       "\n",
       "                                               Title     Seeker  \\\n",
       "0  Are Simon Baron-Cohen and Tony Attwood respons...    simfish   \n",
       "1                           Is stimming involuntary?  cnidocyte   \n",
       "2                                    How toughen up?       Jacs   \n",
       "3                          Why are we bad at sports?   bluerose   \n",
       "4                          Why are we bad at sports?   bluerose   \n",
       "\n",
       "                                          First_Post       Replier  \\\n",
       "0           What about other top autism researchers?          Indy   \n",
       "1  For the past 2 months I've got into the habit ...     Todesking   \n",
       "2  How do you toughen up? I have always been a ve...  CockneyRebel   \n",
       "3  I mean, what is the mechanism behind it. Somet...        Kaysea   \n",
       "4  I mean, what is the mechanism behind it. Somet...        Wolfen   \n",
       "\n",
       "                                             Message          Worker  \\\n",
       "0  Don't know about emails, but there are lots of...  A2QLSHXNCHBRN4   \n",
       "1  In second grade I use to rock back and forth u...  A2M5VW97GIYLHB   \n",
       "2  I'm also a very sensitive person. I can't real...  A166A2M31CW2C7   \n",
       "3  Wolfen wrote:Wow, interesting somehow.For me i...  A2CK0OXMPOR9LE   \n",
       "4  Wow, interesting somehow.\\n\\nFor me it's reall...  A2CK0OXMPOR9LE   \n",
       "\n",
       "             Answer                     Date  \n",
       "0   Somewhat Useful  2017-11-09 21:41:01 UTC  \n",
       "1           Neutral  2017-11-09 21:41:23 UTC  \n",
       "2  Somewhat Useless  2017-11-09 21:40:56 UTC  \n",
       "3            Useful  2017-11-09 21:41:56 UTC  \n",
       "4            Useful  2017-11-09 21:41:37 UTC  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "amt = pd.read_csv('Batch_3003757_batch_results.csv')\n",
    "amt.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count     94.000000\n",
       "mean     103.808511\n",
       "std      190.931912\n",
       "min        1.000000\n",
       "25%        2.250000\n",
       "50%       10.000000\n",
       "75%      102.750000\n",
       "max      733.000000\n",
       "Name: Worker, dtype: float64"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "amt.Worker.value_counts().describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Useful              4127\n",
       "Somewhat Useful     3411\n",
       "Neutral             1143\n",
       "Somewhat Useless     670\n",
       "Useless              407\n",
       "Name: Answer, dtype: int64"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "amt.Answer.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9758"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(amt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Useful              436\n",
       "Somewhat Useful     291\n",
       "Neutral               5\n",
       "Somewhat Useless      1\n",
       "Name: Answer, dtype: int64"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "amt[amt.Worker == 'A3HAPMZV7CYDY8'].Answer.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.4. Using NLTK to Separate Text Data into Sentences <a class=\"anchor\" id=\"24\"></a>\n",
    "\n",
    "This will be used to build sent2vec vectors for sentences.\n",
    "\n",
    "### 2.4.1. For the Samples <a class=\"anchor\" id=\"241\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9734"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(amt[(amt.Message.notnull()) & (amt.First_Post.notnull())])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "373"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "amt[(amt.First_Post.notnull()) & (amt.Message.notnull())].drop_duplicates(subset='Title')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = amt.Message.apply(lambda y: [1,2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    [1, 2]\n",
       "1    [1, 2]\n",
       "2    [1, 2]\n",
       "3    [1, 2]\n",
       "4    [1, 2]\n",
       "Name: Message, dtype: object"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import nltk.data\n",
    "import subprocess\n",
    "\n",
    "tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')\n",
    "PREFIX = r'./fastText/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "msg_file = open('plain_message_sentences.txt', 'w')\n",
    "msg_indexes = open('plain_message_indexes.txt', 'w')\n",
    "title_hit_map = open('plain_hit_to_title.txt', 'w')\n",
    "for index, row in amt[(amt.Message.notnull()) & (amt.First_Post.notnull())].iterrows():\n",
    "    msg_sentences = tokenizer.tokenize(row['Message'])\n",
    "    print('\\n'.join(msg_sentences), file=msg_file)\n",
    "    print(\"{}\\t{}\".format(row['HITID'], len(msg_sentences)), file=msg_indexes)\n",
    "    print('{}\\t{}'.format(row['HITID'], row['Title']), file=title_hit_map)\n",
    "msg_file.close()\n",
    "msg_indexes.close()\n",
    "title_hit_map.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fp_file = open('plain_first_post_sentences.txt', 'w')\n",
    "fp_indexes = open('plain_first_post_indexes.txt', 'w')\n",
    "for index, row in amt[(amt.First_Post.notnull())].drop_duplicates(subset='Title').iterrows():\n",
    "    fp_sentences = tokenizer.tokenize(row['First_Post'])\n",
    "    print('\\n'.join(fp_sentences), file=fp_file)\n",
    "    print(\"{}\\t{}\".format(row['Title'], len(fp_sentences)), file=fp_indexes)\n",
    "fp_file.close()\n",
    "fp_indexes.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The command used in Terminal to get sentence vectors and print them to a file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'./fastText/fasttext print-sentence-vectors ./fastText/wiki_bigrams.bin < ./fastText/sentences.txt'"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r\"{}fasttext print-sentence-vectors {}wiki_bigrams.bin < {}sentences.txt\".format(PREFIX, PREFIX, PREFIX)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4.2. For All Collected Data <a class=\"anchor\" id=\"242\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import nltk.data\n",
    "import subprocess\n",
    "\n",
    "tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import nltk\n",
    "import gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Joined</th>\n",
       "      <th>Link</th>\n",
       "      <th>Location</th>\n",
       "      <th>Message</th>\n",
       "      <th>Posts</th>\n",
       "      <th>Replies</th>\n",
       "      <th>Timestamp</th>\n",
       "      <th>Title</th>\n",
       "      <th>User Level</th>\n",
       "      <th>Username</th>\n",
       "      <th>Views</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>17.0</td>\n",
       "      <td>Female</td>\n",
       "      <td>1.33989e+12</td>\n",
       "      <td>287074</td>\n",
       "      <td>None</td>\n",
       "      <td>Can anyone else relate to this video?\\n</td>\n",
       "      <td>346</td>\n",
       "      <td>18</td>\n",
       "      <td>1432979760000</td>\n",
       "      <td>Are you a multipotentialite?</td>\n",
       "      <td>4.0</td>\n",
       "      <td>physicsnut42</td>\n",
       "      <td>1176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Male</td>\n",
       "      <td>1.39147e+12</td>\n",
       "      <td>287074</td>\n",
       "      <td>Queens, NYC</td>\n",
       "      <td>Yep...I believe people should make use of all ...</td>\n",
       "      <td>37,484</td>\n",
       "      <td>18</td>\n",
       "      <td>1432981020000</td>\n",
       "      <td>Are you a multipotentialite?</td>\n",
       "      <td>5.0</td>\n",
       "      <td>kraftiekortie</td>\n",
       "      <td>1176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>54.0</td>\n",
       "      <td>Male</td>\n",
       "      <td>1.2801e+12</td>\n",
       "      <td>287074</td>\n",
       "      <td>None</td>\n",
       "      <td>Yes, that what society expects--throw away the...</td>\n",
       "      <td>3,504</td>\n",
       "      <td>18</td>\n",
       "      <td>1433161080000</td>\n",
       "      <td>Are you a multipotentialite?</td>\n",
       "      <td>5.0</td>\n",
       "      <td>BTDT</td>\n",
       "      <td>1176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>47.0</td>\n",
       "      <td>Male</td>\n",
       "      <td>1.27189e+12</td>\n",
       "      <td>173480</td>\n",
       "      <td>Depew NY</td>\n",
       "      <td>silver22 wrote:I also become more active (almo...</td>\n",
       "      <td>3,088</td>\n",
       "      <td>114</td>\n",
       "      <td>1316043300000</td>\n",
       "      <td>Is it alright to Smoke weed, if you have Autism?</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Todesking</td>\n",
       "      <td>19304</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1000</th>\n",
       "      <td>24.0</td>\n",
       "      <td>Female</td>\n",
       "      <td>1.33557e+12</td>\n",
       "      <td>203305</td>\n",
       "      <td>Pontiac, MI</td>\n",
       "      <td>Well not in that particular thing, but I think...</td>\n",
       "      <td>266</td>\n",
       "      <td>15</td>\n",
       "      <td>1341820740000</td>\n",
       "      <td>Symptom descriptions often too drastic?</td>\n",
       "      <td>3.0</td>\n",
       "      <td>corvuscorax</td>\n",
       "      <td>2521</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Age  Gender       Joined    Link     Location  \\\n",
       "0     17.0  Female  1.33989e+12  287074         None   \n",
       "1      NaN    Male  1.39147e+12  287074  Queens, NYC   \n",
       "10    54.0    Male   1.2801e+12  287074         None   \n",
       "100   47.0    Male  1.27189e+12  173480     Depew NY   \n",
       "1000  24.0  Female  1.33557e+12  203305  Pontiac, MI   \n",
       "\n",
       "                                                Message   Posts  Replies  \\\n",
       "0               Can anyone else relate to this video?\\n     346       18   \n",
       "1     Yep...I believe people should make use of all ...  37,484       18   \n",
       "10    Yes, that what society expects--throw away the...   3,504       18   \n",
       "100   silver22 wrote:I also become more active (almo...   3,088      114   \n",
       "1000  Well not in that particular thing, but I think...     266       15   \n",
       "\n",
       "          Timestamp                                             Title  \\\n",
       "0     1432979760000                      Are you a multipotentialite?   \n",
       "1     1432981020000                      Are you a multipotentialite?   \n",
       "10    1433161080000                      Are you a multipotentialite?   \n",
       "100   1316043300000  Is it alright to Smoke weed, if you have Autism?   \n",
       "1000  1341820740000           Symptom descriptions often too drastic?   \n",
       "\n",
       "      User Level       Username  Views  \n",
       "0            4.0   physicsnut42   1176  \n",
       "1            5.0  kraftiekortie   1176  \n",
       "10           5.0           BTDT   1176  \n",
       "100          5.0      Todesking  19304  \n",
       "1000         3.0    corvuscorax   2521  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_json('merged_forum_threads.json')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Link</th>\n",
       "      <th>Title</th>\n",
       "      <th>Seeker</th>\n",
       "      <th>First_Post</th>\n",
       "      <th>Replies</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>60</td>\n",
       "      <td>Is this an asperger's trait?</td>\n",
       "      <td>animallover</td>\n",
       "      <td>All of you that don't have jobs and the abilit...</td>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>61</td>\n",
       "      <td>Online Social Skills Education?</td>\n",
       "      <td>TyroneShoelaces</td>\n",
       "      <td>Hello Everyone\\n\\nAt present, I am in the proc...</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>63</td>\n",
       "      <td>Chat?</td>\n",
       "      <td>IrishWolf</td>\n",
       "      <td>Are there set times for chat? It's just everyt...</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>66</td>\n",
       "      <td>Do you like Insects?</td>\n",
       "      <td>TyroneShoelaces</td>\n",
       "      <td>To tell you the truth - I'm not much an insect...</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>99</td>\n",
       "      <td>Asperger's? Only in my vocabulary for 3 days. ...</td>\n",
       "      <td>focused</td>\n",
       "      <td>I am 28 years old and just got the news. How c...</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Link                                              Title           Seeker  \\\n",
       "0    60                       Is this an asperger's trait?      animallover   \n",
       "1    61                    Online Social Skills Education?  TyroneShoelaces   \n",
       "2    63                                              Chat?        IrishWolf   \n",
       "3    66                               Do you like Insects?  TyroneShoelaces   \n",
       "4    99  Asperger's? Only in my vocabulary for 3 days. ...          focused   \n",
       "\n",
       "                                          First_Post  Replies  \n",
       "0  All of you that don't have jobs and the abilit...       27  \n",
       "1  Hello Everyone\\n\\nAt present, I am in the proc...       22  \n",
       "2  Are there set times for chat? It's just everyt...        8  \n",
       "3  To tell you the truth - I'm not much an insect...       47  \n",
       "4  I am 28 years old and just got the news. How c...       25  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles_with_first_post = df.groupby('Link').first().reset_index()[['Link', 'Title', 'Username', 'Message', 'Replies']]\n",
    "titles_with_first_post.columns = ['Link', 'Title', 'Seeker', 'First_Post', 'Replies']\n",
    "titles_with_first_post.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Link</th>\n",
       "      <th>Title</th>\n",
       "      <th>Seeker</th>\n",
       "      <th>First_Post</th>\n",
       "      <th>Replies</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>60</td>\n",
       "      <td>[Is this an asperger's trait?]</td>\n",
       "      <td>animallover</td>\n",
       "      <td>[All of you that don't have jobs and the abili...</td>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>61</td>\n",
       "      <td>[Online Social Skills Education?]</td>\n",
       "      <td>TyroneShoelaces</td>\n",
       "      <td>[Hello Everyone., At present, I am in the proc...</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>63</td>\n",
       "      <td>[Chat?]</td>\n",
       "      <td>IrishWolf</td>\n",
       "      <td>[Are there set times for chat?, It's just ever...</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>66</td>\n",
       "      <td>[Do you like Insects?]</td>\n",
       "      <td>TyroneShoelaces</td>\n",
       "      <td>[To tell you the truth - I'm not much an insec...</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>99</td>\n",
       "      <td>[Asperger's?, Only in my vocabulary for 3 days...</td>\n",
       "      <td>focused</td>\n",
       "      <td>[I am 28 years old and just got the news., How...</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Link                                              Title           Seeker  \\\n",
       "0    60                     [Is this an asperger's trait?]      animallover   \n",
       "1    61                  [Online Social Skills Education?]  TyroneShoelaces   \n",
       "2    63                                            [Chat?]        IrishWolf   \n",
       "3    66                             [Do you like Insects?]  TyroneShoelaces   \n",
       "4    99  [Asperger's?, Only in my vocabulary for 3 days...          focused   \n",
       "\n",
       "                                          First_Post  Replies  \n",
       "0  [All of you that don't have jobs and the abili...       27  \n",
       "1  [Hello Everyone., At present, I am in the proc...       22  \n",
       "2  [Are there set times for chat?, It's just ever...        8  \n",
       "3  [To tell you the truth - I'm not much an insec...       47  \n",
       "4  [I am 28 years old and just got the news., How...       25  "
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def tokenize_properly(text):\n",
    "    return [sent for sent in tokenizer.tokenize(text.replace('\\n', '. ')) \n",
    "            if len(sent.replace('.', '').replace(' ', '')) >= 2]\n",
    "\n",
    "titles_with_first_post.Title = titles_with_first_post.Title.apply(tokenize_properly)\n",
    "titles_with_first_post.First_Post = titles_with_first_post.First_Post.apply(tokenize_properly)\n",
    "titles_with_first_post.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Link</th>\n",
       "      <th>Title</th>\n",
       "      <th>Seeker</th>\n",
       "      <th>First_Post</th>\n",
       "      <th>Replies</th>\n",
       "      <th>Title_sent_count</th>\n",
       "      <th>FP_sent_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>60</td>\n",
       "      <td>[Is this an asperger's trait?]</td>\n",
       "      <td>animallover</td>\n",
       "      <td>[All of you that don't have jobs and the abili...</td>\n",
       "      <td>27</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>61</td>\n",
       "      <td>[Online Social Skills Education?]</td>\n",
       "      <td>TyroneShoelaces</td>\n",
       "      <td>[Hello Everyone., At present, I am in the proc...</td>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>63</td>\n",
       "      <td>[Chat?]</td>\n",
       "      <td>IrishWolf</td>\n",
       "      <td>[Are there set times for chat?, It's just ever...</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>66</td>\n",
       "      <td>[Do you like Insects?]</td>\n",
       "      <td>TyroneShoelaces</td>\n",
       "      <td>[To tell you the truth - I'm not much an insec...</td>\n",
       "      <td>47</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>99</td>\n",
       "      <td>[Asperger's?, Only in my vocabulary for 3 days...</td>\n",
       "      <td>focused</td>\n",
       "      <td>[I am 28 years old and just got the news., How...</td>\n",
       "      <td>25</td>\n",
       "      <td>3</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Link                                              Title           Seeker  \\\n",
       "0    60                     [Is this an asperger's trait?]      animallover   \n",
       "1    61                  [Online Social Skills Education?]  TyroneShoelaces   \n",
       "2    63                                            [Chat?]        IrishWolf   \n",
       "3    66                             [Do you like Insects?]  TyroneShoelaces   \n",
       "4    99  [Asperger's?, Only in my vocabulary for 3 days...          focused   \n",
       "\n",
       "                                          First_Post  Replies  \\\n",
       "0  [All of you that don't have jobs and the abili...       27   \n",
       "1  [Hello Everyone., At present, I am in the proc...       22   \n",
       "2  [Are there set times for chat?, It's just ever...        8   \n",
       "3  [To tell you the truth - I'm not much an insec...       47   \n",
       "4  [I am 28 years old and just got the news., How...       25   \n",
       "\n",
       "   Title_sent_count  FP_sent_count  \n",
       "0                 1              4  \n",
       "1                 1              8  \n",
       "2                 1              2  \n",
       "3                 1              2  \n",
       "4                 3             18  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles_with_first_post['Title_sent_count'] = titles_with_first_post.Title.apply(len)\n",
    "titles_with_first_post['FP_sent_count'] = titles_with_first_post.First_Post.apply(len)\n",
    "titles_with_first_post.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tfp_file = open('all_title_fp_sentences.txt', 'w')\n",
    "for index, row in titles_with_first_post.iterrows():\n",
    "    print('\\n'.join(row['Title']), file=tfp_file)\n",
    "    print('\\n'.join(row['First_Post']), file=tfp_file)\n",
    "tfp_file.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sent2vecs = [[float(number) for number in line.strip().split(' ')] \n",
    "                for line in open('all_title_fp_sent2vec.txt', 'r').readlines()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "title_vecs = {}\n",
    "fp_vecs = {}\n",
    "current = 0\n",
    "for index, row in titles_with_first_post.iterrows():\n",
    "    title_sent_count = row['Title_sent_count']\n",
    "    fp_sent_count = row['FP_sent_count']\n",
    "    title_vecs[index] = sent2vecs[current:current+title_sent_count]\n",
    "    current += title_sent_count\n",
    "    fp_vecs[index] = sent2vecs[current:current+fp_sent_count]\n",
    "    current += fp_sent_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Link</th>\n",
       "      <th>Title</th>\n",
       "      <th>Seeker</th>\n",
       "      <th>First_Post</th>\n",
       "      <th>Replies</th>\n",
       "      <th>Title_sent_count</th>\n",
       "      <th>FP_sent_count</th>\n",
       "      <th>Title_sent2vec</th>\n",
       "      <th>FP_sent2vec</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>60</td>\n",
       "      <td>[Is this an asperger's trait?]</td>\n",
       "      <td>animallover</td>\n",
       "      <td>[All of you that don't have jobs and the abili...</td>\n",
       "      <td>27</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>[[-0.026278, -0.018632, -0.010761, 0.020232, -...</td>\n",
       "      <td>[[0.0012997, 0.0078113, -0.002381, 0.0056695, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>61</td>\n",
       "      <td>[Online Social Skills Education?]</td>\n",
       "      <td>TyroneShoelaces</td>\n",
       "      <td>[Hello Everyone., At present, I am in the proc...</td>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>[[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,...</td>\n",
       "      <td>[[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>63</td>\n",
       "      <td>[Chat?]</td>\n",
       "      <td>IrishWolf</td>\n",
       "      <td>[Are there set times for chat?, It's just ever...</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>[[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,...</td>\n",
       "      <td>[[-0.0080967, -0.046131, 0.014837, -0.018551, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>66</td>\n",
       "      <td>[Do you like Insects?]</td>\n",
       "      <td>TyroneShoelaces</td>\n",
       "      <td>[To tell you the truth - I'm not much an insec...</td>\n",
       "      <td>47</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>[[-0.017121, 0.0075421, -0.021786, 0.016274, 0...</td>\n",
       "      <td>[[-0.011399, 0.002861, 0.0030487, 0.010823, -0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>99</td>\n",
       "      <td>[Asperger's?, Only in my vocabulary for 3 days...</td>\n",
       "      <td>focused</td>\n",
       "      <td>[I am 28 years old and just got the news., How...</td>\n",
       "      <td>25</td>\n",
       "      <td>3</td>\n",
       "      <td>18</td>\n",
       "      <td>[[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,...</td>\n",
       "      <td>[[-0.011724, -0.016215, -0.012698, -0.0047712,...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Link                                              Title           Seeker  \\\n",
       "0    60                     [Is this an asperger's trait?]      animallover   \n",
       "1    61                  [Online Social Skills Education?]  TyroneShoelaces   \n",
       "2    63                                            [Chat?]        IrishWolf   \n",
       "3    66                             [Do you like Insects?]  TyroneShoelaces   \n",
       "4    99  [Asperger's?, Only in my vocabulary for 3 days...          focused   \n",
       "\n",
       "                                          First_Post  Replies  \\\n",
       "0  [All of you that don't have jobs and the abili...       27   \n",
       "1  [Hello Everyone., At present, I am in the proc...       22   \n",
       "2  [Are there set times for chat?, It's just ever...        8   \n",
       "3  [To tell you the truth - I'm not much an insec...       47   \n",
       "4  [I am 28 years old and just got the news., How...       25   \n",
       "\n",
       "   Title_sent_count  FP_sent_count  \\\n",
       "0                 1              4   \n",
       "1                 1              8   \n",
       "2                 1              2   \n",
       "3                 1              2   \n",
       "4                 3             18   \n",
       "\n",
       "                                      Title_sent2vec  \\\n",
       "0  [[-0.026278, -0.018632, -0.010761, 0.020232, -...   \n",
       "1  [[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,...   \n",
       "2  [[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,...   \n",
       "3  [[-0.017121, 0.0075421, -0.021786, 0.016274, 0...   \n",
       "4  [[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,...   \n",
       "\n",
       "                                         FP_sent2vec  \n",
       "0  [[0.0012997, 0.0078113, -0.002381, 0.0056695, ...  \n",
       "1  [[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,...  \n",
       "2  [[-0.0080967, -0.046131, 0.014837, -0.018551, ...  \n",
       "3  [[-0.011399, 0.002861, 0.0030487, 0.010823, -0...  \n",
       "4  [[-0.011724, -0.016215, -0.012698, -0.0047712,...  "
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles_with_first_post['Title_sent2vec'] = pd.Series(titles_with_first_post.index.values).map(title_vecs)\n",
    "titles_with_first_post['FP_sent2vec'] = pd.Series(titles_with_first_post.index.values).map(fp_vecs)\n",
    "titles_with_first_post.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Link</th>\n",
       "      <th>Title</th>\n",
       "      <th>Seeker</th>\n",
       "      <th>First_Post</th>\n",
       "      <th>Title_sent_count</th>\n",
       "      <th>FP_sent_count</th>\n",
       "      <th>Title_sent2vec</th>\n",
       "      <th>FP_sent2vec</th>\n",
       "      <th>Replier</th>\n",
       "      <th>Reply</th>\n",
       "      <th>User Level</th>\n",
       "      <th>User Posts</th>\n",
       "      <th>Reply_sent_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>60</td>\n",
       "      <td>[Is this an asperger's trait?]</td>\n",
       "      <td>animallover</td>\n",
       "      <td>[All of you that don't have jobs and the abili...</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>[[-0.026278, -0.018632, -0.010761, 0.020232, -...</td>\n",
       "      <td>[[0.0012997, 0.0078113, -0.002381, 0.0056695, ...</td>\n",
       "      <td>Nuttdan</td>\n",
       "      <td>[I hate mirrors., They remind me how crummy I ...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>323</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>60</td>\n",
       "      <td>[Is this an asperger's trait?]</td>\n",
       "      <td>animallover</td>\n",
       "      <td>[All of you that don't have jobs and the abili...</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>[[-0.026278, -0.018632, -0.010761, 0.020232, -...</td>\n",
       "      <td>[[0.0012997, 0.0078113, -0.002381, 0.0056695, ...</td>\n",
       "      <td>Rogue</td>\n",
       "      <td>[Quote:I've found that I tend to have about a ...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>32</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>60</td>\n",
       "      <td>[Is this an asperger's trait?]</td>\n",
       "      <td>animallover</td>\n",
       "      <td>[All of you that don't have jobs and the abili...</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>[[-0.026278, -0.018632, -0.010761, 0.020232, -...</td>\n",
       "      <td>[[0.0012997, 0.0078113, -0.002381, 0.0056695, ...</td>\n",
       "      <td>TyroneShoelaces</td>\n",
       "      <td>[Like you, I have a lot of trouble staying in ...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>150</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>60</td>\n",
       "      <td>[Is this an asperger's trait?]</td>\n",
       "      <td>animallover</td>\n",
       "      <td>[All of you that don't have jobs and the abili...</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>[[-0.026278, -0.018632, -0.010761, 0.020232, -...</td>\n",
       "      <td>[[0.0012997, 0.0078113, -0.002381, 0.0056695, ...</td>\n",
       "      <td>alex</td>\n",
       "      <td>[Rogue wrote:My guess would be that it's not a...</td>\n",
       "      <td>51.0</td>\n",
       "      <td>9,923</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>60</td>\n",
       "      <td>[Is this an asperger's trait?]</td>\n",
       "      <td>animallover</td>\n",
       "      <td>[All of you that don't have jobs and the abili...</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>[[-0.026278, -0.018632, -0.010761, 0.020232, -...</td>\n",
       "      <td>[[0.0012997, 0.0078113, -0.002381, 0.0056695, ...</td>\n",
       "      <td>Mich</td>\n",
       "      <td>[animallover wrote:All of you that don't have ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>508</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Link                           Title       Seeker  \\\n",
       "1    60  [Is this an asperger's trait?]  animallover   \n",
       "2    60  [Is this an asperger's trait?]  animallover   \n",
       "3    60  [Is this an asperger's trait?]  animallover   \n",
       "4    60  [Is this an asperger's trait?]  animallover   \n",
       "5    60  [Is this an asperger's trait?]  animallover   \n",
       "\n",
       "                                          First_Post  Title_sent_count  \\\n",
       "1  [All of you that don't have jobs and the abili...                 1   \n",
       "2  [All of you that don't have jobs and the abili...                 1   \n",
       "3  [All of you that don't have jobs and the abili...                 1   \n",
       "4  [All of you that don't have jobs and the abili...                 1   \n",
       "5  [All of you that don't have jobs and the abili...                 1   \n",
       "\n",
       "   FP_sent_count                                     Title_sent2vec  \\\n",
       "1              4  [[-0.026278, -0.018632, -0.010761, 0.020232, -...   \n",
       "2              4  [[-0.026278, -0.018632, -0.010761, 0.020232, -...   \n",
       "3              4  [[-0.026278, -0.018632, -0.010761, 0.020232, -...   \n",
       "4              4  [[-0.026278, -0.018632, -0.010761, 0.020232, -...   \n",
       "5              4  [[-0.026278, -0.018632, -0.010761, 0.020232, -...   \n",
       "\n",
       "                                         FP_sent2vec          Replier  \\\n",
       "1  [[0.0012997, 0.0078113, -0.002381, 0.0056695, ...          Nuttdan   \n",
       "2  [[0.0012997, 0.0078113, -0.002381, 0.0056695, ...            Rogue   \n",
       "3  [[0.0012997, 0.0078113, -0.002381, 0.0056695, ...  TyroneShoelaces   \n",
       "4  [[0.0012997, 0.0078113, -0.002381, 0.0056695, ...             alex   \n",
       "5  [[0.0012997, 0.0078113, -0.002381, 0.0056695, ...             Mich   \n",
       "\n",
       "                                               Reply  User Level User Posts  \\\n",
       "1  [I hate mirrors., They remind me how crummy I ...         5.0        323   \n",
       "2  [Quote:I've found that I tend to have about a ...         1.0         32   \n",
       "3  [Like you, I have a lot of trouble staying in ...         2.0        150   \n",
       "4  [Rogue wrote:My guess would be that it's not a...        51.0      9,923   \n",
       "5  [animallover wrote:All of you that don't have ...         NaN        508   \n",
       "\n",
       "   Reply_sent_count  \n",
       "1                 3  \n",
       "2                 3  \n",
       "3                 9  \n",
       "4                 2  \n",
       "5                 8  "
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles_with_messages = pd.merge(titles_with_first_post.drop(['Replies'], axis=1), \n",
    "                                df[['Username', 'Message', 'Link', 'User Level', 'Posts']], on='Link')\n",
    "titles_with_messages.rename(columns={'Username':'Replier', 'Posts': 'User Posts'}, inplace=True)\n",
    "titles_with_messages = titles_with_messages[titles_with_messages.apply(\n",
    "    lambda row: not row['First_Post'] == row['Message'] and not row['Seeker'] == row['Replier'], axis=1)]\n",
    "titles_with_messages.Message = titles_with_messages.Message.apply(tokenize_properly)\n",
    "titles_with_messages.rename(columns={'Message': 'Reply'}, inplace=True)\n",
    "titles_with_messages['Reply_sent_count'] = titles_with_messages.Reply.apply(len)\n",
    "titles_with_messages.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tfp_file = open('all_reply_sentences.txt', 'w')\n",
    "for index, row in titles_with_messages.iterrows():\n",
    "    print('\\n'.join(row['Reply']), file=tfp_file)\n",
    "tfp_file.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sent2vecs = [[float(number) for number in line.strip().split(' ')] \n",
    "                for line in open('all_reply_sent2vec.txt', 'r').readlines()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Link</th>\n",
       "      <th>Title</th>\n",
       "      <th>Seeker</th>\n",
       "      <th>First_Post</th>\n",
       "      <th>Title_sent_count</th>\n",
       "      <th>FP_sent_count</th>\n",
       "      <th>Title_sent2vec</th>\n",
       "      <th>FP_sent2vec</th>\n",
       "      <th>Replier</th>\n",
       "      <th>Reply</th>\n",
       "      <th>User Level</th>\n",
       "      <th>User Posts</th>\n",
       "      <th>Reply_sent_count</th>\n",
       "      <th>Reply_sent2vec</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>60</td>\n",
       "      <td>[Is this an asperger's trait?]</td>\n",
       "      <td>animallover</td>\n",
       "      <td>[All of you that don't have jobs and the abili...</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>[[-0.026278, -0.018632, -0.010761, 0.020232, -...</td>\n",
       "      <td>[[0.0012997, 0.0078113, -0.002381, 0.0056695, ...</td>\n",
       "      <td>Nuttdan</td>\n",
       "      <td>[I hate mirrors., They remind me how crummy I ...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>323</td>\n",
       "      <td>3</td>\n",
       "      <td>[[0.020335, -0.045285, 0.0068728, 0.0064149, -...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>60</td>\n",
       "      <td>[Is this an asperger's trait?]</td>\n",
       "      <td>animallover</td>\n",
       "      <td>[All of you that don't have jobs and the abili...</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>[[-0.026278, -0.018632, -0.010761, 0.020232, -...</td>\n",
       "      <td>[[0.0012997, 0.0078113, -0.002381, 0.0056695, ...</td>\n",
       "      <td>Rogue</td>\n",
       "      <td>[Quote:I've found that I tend to have about a ...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>32</td>\n",
       "      <td>3</td>\n",
       "      <td>[[0.0053239, 0.0092477, -0.012076, -0.0088438,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>60</td>\n",
       "      <td>[Is this an asperger's trait?]</td>\n",
       "      <td>animallover</td>\n",
       "      <td>[All of you that don't have jobs and the abili...</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>[[-0.026278, -0.018632, -0.010761, 0.020232, -...</td>\n",
       "      <td>[[0.0012997, 0.0078113, -0.002381, 0.0056695, ...</td>\n",
       "      <td>TyroneShoelaces</td>\n",
       "      <td>[Like you, I have a lot of trouble staying in ...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>150</td>\n",
       "      <td>9</td>\n",
       "      <td>[[0.0101, -0.013583, 0.0044502, -0.02776, 0.01...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>60</td>\n",
       "      <td>[Is this an asperger's trait?]</td>\n",
       "      <td>animallover</td>\n",
       "      <td>[All of you that don't have jobs and the abili...</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>[[-0.026278, -0.018632, -0.010761, 0.020232, -...</td>\n",
       "      <td>[[0.0012997, 0.0078113, -0.002381, 0.0056695, ...</td>\n",
       "      <td>alex</td>\n",
       "      <td>[Rogue wrote:My guess would be that it's not a...</td>\n",
       "      <td>51.0</td>\n",
       "      <td>9,923</td>\n",
       "      <td>2</td>\n",
       "      <td>[[-0.0075652, -0.0032049, -0.0053176, -0.00444...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>60</td>\n",
       "      <td>[Is this an asperger's trait?]</td>\n",
       "      <td>animallover</td>\n",
       "      <td>[All of you that don't have jobs and the abili...</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>[[-0.026278, -0.018632, -0.010761, 0.020232, -...</td>\n",
       "      <td>[[0.0012997, 0.0078113, -0.002381, 0.0056695, ...</td>\n",
       "      <td>Mich</td>\n",
       "      <td>[animallover wrote:All of you that don't have ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>508</td>\n",
       "      <td>8</td>\n",
       "      <td>[[0.0012997, 0.0078113, -0.002381, 0.0056695, ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Link                           Title       Seeker  \\\n",
       "1    60  [Is this an asperger's trait?]  animallover   \n",
       "2    60  [Is this an asperger's trait?]  animallover   \n",
       "3    60  [Is this an asperger's trait?]  animallover   \n",
       "4    60  [Is this an asperger's trait?]  animallover   \n",
       "5    60  [Is this an asperger's trait?]  animallover   \n",
       "\n",
       "                                          First_Post  Title_sent_count  \\\n",
       "1  [All of you that don't have jobs and the abili...                 1   \n",
       "2  [All of you that don't have jobs and the abili...                 1   \n",
       "3  [All of you that don't have jobs and the abili...                 1   \n",
       "4  [All of you that don't have jobs and the abili...                 1   \n",
       "5  [All of you that don't have jobs and the abili...                 1   \n",
       "\n",
       "   FP_sent_count                                     Title_sent2vec  \\\n",
       "1              4  [[-0.026278, -0.018632, -0.010761, 0.020232, -...   \n",
       "2              4  [[-0.026278, -0.018632, -0.010761, 0.020232, -...   \n",
       "3              4  [[-0.026278, -0.018632, -0.010761, 0.020232, -...   \n",
       "4              4  [[-0.026278, -0.018632, -0.010761, 0.020232, -...   \n",
       "5              4  [[-0.026278, -0.018632, -0.010761, 0.020232, -...   \n",
       "\n",
       "                                         FP_sent2vec          Replier  \\\n",
       "1  [[0.0012997, 0.0078113, -0.002381, 0.0056695, ...          Nuttdan   \n",
       "2  [[0.0012997, 0.0078113, -0.002381, 0.0056695, ...            Rogue   \n",
       "3  [[0.0012997, 0.0078113, -0.002381, 0.0056695, ...  TyroneShoelaces   \n",
       "4  [[0.0012997, 0.0078113, -0.002381, 0.0056695, ...             alex   \n",
       "5  [[0.0012997, 0.0078113, -0.002381, 0.0056695, ...             Mich   \n",
       "\n",
       "                                               Reply  User Level User Posts  \\\n",
       "1  [I hate mirrors., They remind me how crummy I ...         5.0        323   \n",
       "2  [Quote:I've found that I tend to have about a ...         1.0         32   \n",
       "3  [Like you, I have a lot of trouble staying in ...         2.0        150   \n",
       "4  [Rogue wrote:My guess would be that it's not a...        51.0      9,923   \n",
       "5  [animallover wrote:All of you that don't have ...         NaN        508   \n",
       "\n",
       "   Reply_sent_count                                     Reply_sent2vec  \n",
       "1                 3  [[0.020335, -0.045285, 0.0068728, 0.0064149, -...  \n",
       "2                 3  [[0.0053239, 0.0092477, -0.012076, -0.0088438,...  \n",
       "3                 9  [[0.0101, -0.013583, 0.0044502, -0.02776, 0.01...  \n",
       "4                 2  [[-0.0075652, -0.0032049, -0.0053176, -0.00444...  \n",
       "5                 8  [[0.0012997, 0.0078113, -0.002381, 0.0056695, ...  "
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reply_vecs = []\n",
    "current = 0\n",
    "for index, row in titles_with_messages.iterrows():\n",
    "    reply_sent_count = row['Reply_sent_count']\n",
    "    reply_vecs.append(sent2vecs[current:current+reply_sent_count])\n",
    "    current += reply_sent_count\n",
    "titles_with_messages['Reply_sent2vec'] = reply_vecs\n",
    "titles_with_messages.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "titles_with_messages.to_json('titles_with_messages.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Machine Learning Features <a class=\"anchor\" id=\"3\"></a>\n",
    "\n",
    "In this section, we collect the features from the labels needed for a Machine Learning model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import sklearn\n",
    "import nltk\n",
    "import gensim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1. MTurk DataFrame <a class=\"anchor\" id=\"31\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>HITID</th>\n",
       "      <th>Link</th>\n",
       "      <th>Title</th>\n",
       "      <th>Seeker</th>\n",
       "      <th>First_Post</th>\n",
       "      <th>Replier</th>\n",
       "      <th>Message</th>\n",
       "      <th>Worker</th>\n",
       "      <th>Answer</th>\n",
       "      <th>Date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3K3G488TR2G3Y0HX29ACGBE4YKZQ5Z</td>\n",
       "      <td>160755</td>\n",
       "      <td>Are Simon Baron-Cohen and Tony Attwood respons...</td>\n",
       "      <td>simfish</td>\n",
       "      <td>What about other top autism researchers?</td>\n",
       "      <td>Indy</td>\n",
       "      <td>Don't know about emails, but there are lots of...</td>\n",
       "      <td>A2QLSHXNCHBRN4</td>\n",
       "      <td>Somewhat Useful</td>\n",
       "      <td>2017-11-09 21:41:01 UTC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3OLZC0DJ8JN7FW8TVSNGB7HHKYKIVV</td>\n",
       "      <td>165438</td>\n",
       "      <td>Is stimming involuntary?</td>\n",
       "      <td>cnidocyte</td>\n",
       "      <td>For the past 2 months I've got into the habit ...</td>\n",
       "      <td>Todesking</td>\n",
       "      <td>In second grade I use to rock back and forth u...</td>\n",
       "      <td>A2M5VW97GIYLHB</td>\n",
       "      <td>Neutral</td>\n",
       "      <td>2017-11-09 21:41:23 UTC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>36AZSFEYZ48XSOUNPIVZPKD4P80VBZ</td>\n",
       "      <td>161882</td>\n",
       "      <td>How toughen up?</td>\n",
       "      <td>Jacs</td>\n",
       "      <td>How do you toughen up? I have always been a ve...</td>\n",
       "      <td>CockneyRebel</td>\n",
       "      <td>I'm also a very sensitive person. I can't real...</td>\n",
       "      <td>A166A2M31CW2C7</td>\n",
       "      <td>Somewhat Useless</td>\n",
       "      <td>2017-11-09 21:40:56 UTC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3WGZLY9VCHUSW9WUVLBVSWHWHVBD8J</td>\n",
       "      <td>106087</td>\n",
       "      <td>Why are we bad at sports?</td>\n",
       "      <td>bluerose</td>\n",
       "      <td>I mean, what is the mechanism behind it. Somet...</td>\n",
       "      <td>Kaysea</td>\n",
       "      <td>Wolfen wrote:Wow, interesting somehow.For me i...</td>\n",
       "      <td>A2CK0OXMPOR9LE</td>\n",
       "      <td>Useful</td>\n",
       "      <td>2017-11-09 21:41:56 UTC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3IKMEYR0LW3PO00F7LOHMIZJF4H2KJ</td>\n",
       "      <td>106087</td>\n",
       "      <td>Why are we bad at sports?</td>\n",
       "      <td>bluerose</td>\n",
       "      <td>I mean, what is the mechanism behind it. Somet...</td>\n",
       "      <td>Wolfen</td>\n",
       "      <td>Wow, interesting somehow.\\n\\nFor me it's reall...</td>\n",
       "      <td>A2CK0OXMPOR9LE</td>\n",
       "      <td>Useful</td>\n",
       "      <td>2017-11-09 21:41:37 UTC</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            HITID    Link  \\\n",
       "0  3K3G488TR2G3Y0HX29ACGBE4YKZQ5Z  160755   \n",
       "1  3OLZC0DJ8JN7FW8TVSNGB7HHKYKIVV  165438   \n",
       "2  36AZSFEYZ48XSOUNPIVZPKD4P80VBZ  161882   \n",
       "3  3WGZLY9VCHUSW9WUVLBVSWHWHVBD8J  106087   \n",
       "4  3IKMEYR0LW3PO00F7LOHMIZJF4H2KJ  106087   \n",
       "\n",
       "                                               Title     Seeker  \\\n",
       "0  Are Simon Baron-Cohen and Tony Attwood respons...    simfish   \n",
       "1                           Is stimming involuntary?  cnidocyte   \n",
       "2                                    How toughen up?       Jacs   \n",
       "3                          Why are we bad at sports?   bluerose   \n",
       "4                          Why are we bad at sports?   bluerose   \n",
       "\n",
       "                                          First_Post       Replier  \\\n",
       "0           What about other top autism researchers?          Indy   \n",
       "1  For the past 2 months I've got into the habit ...     Todesking   \n",
       "2  How do you toughen up? I have always been a ve...  CockneyRebel   \n",
       "3  I mean, what is the mechanism behind it. Somet...        Kaysea   \n",
       "4  I mean, what is the mechanism behind it. Somet...        Wolfen   \n",
       "\n",
       "                                             Message          Worker  \\\n",
       "0  Don't know about emails, but there are lots of...  A2QLSHXNCHBRN4   \n",
       "1  In second grade I use to rock back and forth u...  A2M5VW97GIYLHB   \n",
       "2  I'm also a very sensitive person. I can't real...  A166A2M31CW2C7   \n",
       "3  Wolfen wrote:Wow, interesting somehow.For me i...  A2CK0OXMPOR9LE   \n",
       "4  Wow, interesting somehow.\\n\\nFor me it's reall...  A2CK0OXMPOR9LE   \n",
       "\n",
       "             Answer                     Date  \n",
       "0   Somewhat Useful  2017-11-09 21:41:01 UTC  \n",
       "1           Neutral  2017-11-09 21:41:23 UTC  \n",
       "2  Somewhat Useless  2017-11-09 21:40:56 UTC  \n",
       "3            Useful  2017-11-09 21:41:56 UTC  \n",
       "4            Useful  2017-11-09 21:41:37 UTC  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "amt = pd.read_csv('Batch_3003757_batch_results.csv')\n",
    "amt.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2. Adding Reply Vectors <a class=\"anchor\" id=\"32\"></a>\n",
    "\n",
    "Replies are also referred to as Messages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9734"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "indexes = dict((line.strip().split('\\t')[0], int(line.strip().split('\\t')[1])) \n",
    "                for line in open('plain_amt_message_indexes.txt', 'r').readlines())\n",
    "vector_lines = [[float(number) for number in line.strip().split(' ')] \n",
    "                for line in open('sent2vec_amt_messages.txt', 'r').readlines()]\n",
    "sentences = [line.strip() for line in open('plain_amt_message_sentences.txt', 'r').readlines()]\n",
    "vector_index = 0\n",
    "vector_dict = dict()\n",
    "sent_dict = dict()\n",
    "for (key, value) in indexes.items():\n",
    "    vector_dict[key] = vector_lines[vector_index : vector_index + value]\n",
    "    sent_dict[key] = sentences[vector_index : vector_index + value]\n",
    "    vector_index += value\n",
    "len(vector_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "amt['Message_sent2vec'] = amt.HITID.map(vector_dict)\n",
    "amt['Message'] = amt.HITID.map(sent_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3. Adding First Post Vectors <a class=\"anchor\" id=\"33\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "373"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "indexes = dict((line.strip().split('\\t')[0], int(line.strip().split('\\t')[1])) \n",
    "                for line in open('plain_amt_fp_indexes.txt', 'r').readlines())\n",
    "vector_dict = dict()\n",
    "vector_lines = [[float(number) for number in line.strip().split(' ')] \n",
    "                for line in open('sent2vec_amt_fp.txt', 'r').readlines()]\n",
    "sentences = [line.strip() for line in open('plain_amt_fp_sentences.txt', 'r').readlines()]\n",
    "vector_index = 0\n",
    "for (key, value) in indexes.items():\n",
    "    vector_dict[key] = vector_lines[vector_index : vector_index + value]\n",
    "    sent_dict[key] = sentences[vector_index : vector_index + value]\n",
    "    vector_index += value\n",
    "len(vector_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "amt['First_Post_sent2vec'] = amt.Title.map(vector_dict)\n",
    "amt['First_Post'] = amt.Title.map(sent_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.4. Adding Title Vectors <a class=\"anchor\" id=\"34\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9227"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vector_lines = [[float(number) for number in line.strip().split(' ')] \n",
    "                for line in open('sent2vec_titles.txt', 'r').readlines()]\n",
    "indexes = dict((key.strip(), value) for (key, value) in zip(open('plain_titles.txt', 'r').readlines(), vector_lines))\n",
    "len(indexes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "amt['Title_sent2vec'] = amt.Title.map(indexes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>HITID</th>\n",
       "      <th>Link</th>\n",
       "      <th>Title</th>\n",
       "      <th>Seeker</th>\n",
       "      <th>First_Post</th>\n",
       "      <th>Replier</th>\n",
       "      <th>Message</th>\n",
       "      <th>Worker</th>\n",
       "      <th>Answer</th>\n",
       "      <th>Date</th>\n",
       "      <th>Message_sent2vec</th>\n",
       "      <th>First_Post_sent2vec</th>\n",
       "      <th>Title_sent2vec</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3K3G488TR2G3Y0HX29ACGBE4YKZQ5Z</td>\n",
       "      <td>160755</td>\n",
       "      <td>Are Simon Baron-Cohen and Tony Attwood respons...</td>\n",
       "      <td>simfish</td>\n",
       "      <td>[Can anyone speak to the AS perspective of the...</td>\n",
       "      <td>Indy</td>\n",
       "      <td>[Your psychiatrist is a quack.]</td>\n",
       "      <td>A2QLSHXNCHBRN4</td>\n",
       "      <td>Somewhat Useful</td>\n",
       "      <td>2017-11-09 21:41:01 UTC</td>\n",
       "      <td>[[0.027164, -0.021549, -0.011077, 0.011312, -0...</td>\n",
       "      <td>[[-0.011538, -0.0063551, 0.0076127, 0.011633, ...</td>\n",
       "      <td>[0.042822, 0.036247, 0.023017, 0.010165, 0.001...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3OLZC0DJ8JN7FW8TVSNGB7HHKYKIVV</td>\n",
       "      <td>165438</td>\n",
       "      <td>Is stimming involuntary?</td>\n",
       "      <td>cnidocyte</td>\n",
       "      <td>[Shall we cite here our most favorite ones?, M...</td>\n",
       "      <td>Todesking</td>\n",
       "      <td>[Panic., Maybe if you gave an example of the a...</td>\n",
       "      <td>A2M5VW97GIYLHB</td>\n",
       "      <td>Neutral</td>\n",
       "      <td>2017-11-09 21:41:23 UTC</td>\n",
       "      <td>[[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,...</td>\n",
       "      <td>[[0.0030118, -0.024251, 0.0055384, -0.0015611,...</td>\n",
       "      <td>[0.046063, -0.019432, -0.052288, 0.11009, -0.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>36AZSFEYZ48XSOUNPIVZPKD4P80VBZ</td>\n",
       "      <td>161882</td>\n",
       "      <td>How toughen up?</td>\n",
       "      <td>Jacs</td>\n",
       "      <td>[I think I get the basic idea of what empathy ...</td>\n",
       "      <td>CockneyRebel</td>\n",
       "      <td>[I find it hard to believe that anybody could ...</td>\n",
       "      <td>A166A2M31CW2C7</td>\n",
       "      <td>Somewhat Useless</td>\n",
       "      <td>2017-11-09 21:40:56 UTC</td>\n",
       "      <td>[[0.0017417, 0.0070132, 0.002501, -0.00036597,...</td>\n",
       "      <td>[[-0.017079, -0.010601, 0.00086344, 0.011627, ...</td>\n",
       "      <td>[-0.019942, -0.0084813, -0.063804, -0.00046972...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3WGZLY9VCHUSW9WUVLBVSWHWHVBD8J</td>\n",
       "      <td>106087</td>\n",
       "      <td>Why are we bad at sports?</td>\n",
       "      <td>bluerose</td>\n",
       "      <td>[I'm actually happy that I'm autistic, because...</td>\n",
       "      <td>Kaysea</td>\n",
       "      <td>[But I also shake my feet and legs too., Heck ...</td>\n",
       "      <td>A2CK0OXMPOR9LE</td>\n",
       "      <td>Useful</td>\n",
       "      <td>2017-11-09 21:41:56 UTC</td>\n",
       "      <td>[[0.010428, 0.00098291, 0.0041583, 0.030868, 0...</td>\n",
       "      <td>[[0.0017658, 0.0020074, -0.0073089, -0.0080459...</td>\n",
       "      <td>[0.00049173, -0.019857, 0.0039392, -0.027633, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3IKMEYR0LW3PO00F7LOHMIZJF4H2KJ</td>\n",
       "      <td>106087</td>\n",
       "      <td>Why are we bad at sports?</td>\n",
       "      <td>bluerose</td>\n",
       "      <td>[I'm actually happy that I'm autistic, because...</td>\n",
       "      <td>Wolfen</td>\n",
       "      <td>[People throw the word crazy around like the w...</td>\n",
       "      <td>A2CK0OXMPOR9LE</td>\n",
       "      <td>Useful</td>\n",
       "      <td>2017-11-09 21:41:37 UTC</td>\n",
       "      <td>[[-0.013568, -0.013771, -0.0062426, -0.0050828...</td>\n",
       "      <td>[[0.0017658, 0.0020074, -0.0073089, -0.0080459...</td>\n",
       "      <td>[0.00049173, -0.019857, 0.0039392, -0.027633, ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            HITID    Link  \\\n",
       "0  3K3G488TR2G3Y0HX29ACGBE4YKZQ5Z  160755   \n",
       "1  3OLZC0DJ8JN7FW8TVSNGB7HHKYKIVV  165438   \n",
       "2  36AZSFEYZ48XSOUNPIVZPKD4P80VBZ  161882   \n",
       "3  3WGZLY9VCHUSW9WUVLBVSWHWHVBD8J  106087   \n",
       "4  3IKMEYR0LW3PO00F7LOHMIZJF4H2KJ  106087   \n",
       "\n",
       "                                               Title     Seeker  \\\n",
       "0  Are Simon Baron-Cohen and Tony Attwood respons...    simfish   \n",
       "1                           Is stimming involuntary?  cnidocyte   \n",
       "2                                    How toughen up?       Jacs   \n",
       "3                          Why are we bad at sports?   bluerose   \n",
       "4                          Why are we bad at sports?   bluerose   \n",
       "\n",
       "                                          First_Post       Replier  \\\n",
       "0  [Can anyone speak to the AS perspective of the...          Indy   \n",
       "1  [Shall we cite here our most favorite ones?, M...     Todesking   \n",
       "2  [I think I get the basic idea of what empathy ...  CockneyRebel   \n",
       "3  [I'm actually happy that I'm autistic, because...        Kaysea   \n",
       "4  [I'm actually happy that I'm autistic, because...        Wolfen   \n",
       "\n",
       "                                             Message          Worker  \\\n",
       "0                    [Your psychiatrist is a quack.]  A2QLSHXNCHBRN4   \n",
       "1  [Panic., Maybe if you gave an example of the a...  A2M5VW97GIYLHB   \n",
       "2  [I find it hard to believe that anybody could ...  A166A2M31CW2C7   \n",
       "3  [But I also shake my feet and legs too., Heck ...  A2CK0OXMPOR9LE   \n",
       "4  [People throw the word crazy around like the w...  A2CK0OXMPOR9LE   \n",
       "\n",
       "             Answer                     Date  \\\n",
       "0   Somewhat Useful  2017-11-09 21:41:01 UTC   \n",
       "1           Neutral  2017-11-09 21:41:23 UTC   \n",
       "2  Somewhat Useless  2017-11-09 21:40:56 UTC   \n",
       "3            Useful  2017-11-09 21:41:56 UTC   \n",
       "4            Useful  2017-11-09 21:41:37 UTC   \n",
       "\n",
       "                                    Message_sent2vec  \\\n",
       "0  [[0.027164, -0.021549, -0.011077, 0.011312, -0...   \n",
       "1  [[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,...   \n",
       "2  [[0.0017417, 0.0070132, 0.002501, -0.00036597,...   \n",
       "3  [[0.010428, 0.00098291, 0.0041583, 0.030868, 0...   \n",
       "4  [[-0.013568, -0.013771, -0.0062426, -0.0050828...   \n",
       "\n",
       "                                 First_Post_sent2vec  \\\n",
       "0  [[-0.011538, -0.0063551, 0.0076127, 0.011633, ...   \n",
       "1  [[0.0030118, -0.024251, 0.0055384, -0.0015611,...   \n",
       "2  [[-0.017079, -0.010601, 0.00086344, 0.011627, ...   \n",
       "3  [[0.0017658, 0.0020074, -0.0073089, -0.0080459...   \n",
       "4  [[0.0017658, 0.0020074, -0.0073089, -0.0080459...   \n",
       "\n",
       "                                      Title_sent2vec  \n",
       "0  [0.042822, 0.036247, 0.023017, 0.010165, 0.001...  \n",
       "1  [0.046063, -0.019432, -0.052288, 0.11009, -0.0...  \n",
       "2  [-0.019942, -0.0084813, -0.063804, -0.00046972...  \n",
       "3  [0.00049173, -0.019857, 0.0039392, -0.027633, ...  \n",
       "4  [0.00049173, -0.019857, 0.0039392, -0.027633, ...  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "amt.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.5. Computing sent2vec Similarity <a class=\"anchor\" id=\"35\"></a>\n",
    "\n",
    "Our features are composed of dot product-based conceptual similarity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def is_not_null(sent_vec):\n",
    "    for element in sent_vec:\n",
    "        if not element == 0.0:\n",
    "            return True\n",
    "    return False\n",
    "\n",
    "def sent_to_text_similarity(sent_vec, text_vec):\n",
    "    similarities = []\n",
    "    for vec in text_vec:\n",
    "        if is_not_null(vec):\n",
    "            similarities.append(np.dot(sent_vec, vec))\n",
    "    if similarities:\n",
    "        return np.mean(similarities)\n",
    "    else:\n",
    "        return np.nan\n",
    "\n",
    "def text_to_text_similarity(sent_vecs1, sent_vecs2):\n",
    "    similarities = []\n",
    "    for v1 in sent_vecs1:\n",
    "        if is_not_null(v1):\n",
    "            similarity = sent_to_text_similarity(v1, sent_vecs2)\n",
    "            if not np.isnan(similarity):\n",
    "                similarities.append(similarity)\n",
    "    if similarities:\n",
    "        return np.mean(similarities)\n",
    "    else:\n",
    "        return np.nan\n",
    "\n",
    "def text_to_corpus_similarity(text, corpus):\n",
    "    sent_vecs = text_to_sent_vec(text)\n",
    "    corpus_vecs = [text_to_sent_vec(other_text) for other_text in corpus]\n",
    "    max_sim = 0\n",
    "    index = -1\n",
    "    for text_index in range(len(corpus_vecs)):\n",
    "        similarity = text_to_text_similarity(sent_vecs, corpus_vecs[text_index])\n",
    "        if not np.isnan(similarity) and max_sim < similarity:\n",
    "            max_sim = similarity\n",
    "            index = text_index\n",
    "    if index >= 0:\n",
    "        return corpus[index]\n",
    "    else:\n",
    "        return None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "amt['Sim_Title_Message'] = amt.apply(\n",
    "    lambda row: sent_to_text_similarity(row['Title_sent2vec'],row['Message_sent2vec']) \n",
    "    if type(row['Message_sent2vec']) == type([1]) else np.nan, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "amt['Sim_Title_FP_Message'] = amt.apply(\n",
    "    lambda row: text_to_text_similarity(row['First_Post_sent2vec'] + [row['Title_sent2vec']],row['Message_sent2vec']) \n",
    "    if type(row['Message_sent2vec']) == type([1]) else np.nan, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "amt['Sim_FP_Message'] = amt.apply(\n",
    "    lambda row: text_to_text_similarity(row['First_Post_sent2vec'],row['Message_sent2vec']) \n",
    "    if type(row['Message_sent2vec']) == type([1]) else np.nan, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "amt['Binary_Answer'] = amt.Answer.apply(lambda row: 1 if 'Useful' in row else 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "amt['Cont_Answer'] = amt.Answer.map({'Useless': 0, \n",
    "                                     'Somewhat Useless': 0.25, \n",
    "                                     'Neutral': 0.5, \n",
    "                                     'Somewhat Useful': 0.75,\n",
    "                                     'Useful': 1})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    7538\n",
       "0    2220\n",
       "Name: Binary_Answer, dtype: int64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "amt.Binary_Answer.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    0.772494\n",
       "0    0.227506\n",
       "Name: Binary_Answer, dtype: float64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "amt.Binary_Answer.value_counts().apply(lambda x: x / len(amt))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.00    4127\n",
       "0.75    3411\n",
       "0.50    1143\n",
       "0.25     670\n",
       "0.00     407\n",
       "Name: Cont_Answer, dtype: int64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "amt.Cont_Answer.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.00    0.422935\n",
       "0.75    0.349559\n",
       "0.50    0.117135\n",
       "0.25    0.068662\n",
       "0.00    0.041709\n",
       "Name: Cont_Answer, dtype: float64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "amt.Cont_Answer.value_counts().apply(lambda x: x/len(amt))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "amt['Message_Length'] = amt.Message.apply(lambda message: len(' '.join(message)) if not type(message) == float else 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "amt['Message_Sent_Count'] = amt.Message.apply(lambda message: len(message) if not type(message) == float else 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.6. Adding Features from Forum Threads <a class=\"anchor\" id=\"36\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ft = pd.read_json('forum_threads.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "user_level = ft.set_index('Username')['User Level'].to_dict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "user_posts = ft.set_index('Username')['Posts'].apply(\n",
    "    lambda row: float(row.replace(',', '')) if type(row) == str else 1 if row == None else row).to_dict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "amt['User Level'] = amt.Replier.map(user_level)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "amt['User Posts'] = amt.Replier.map(user_posts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>HITID</th>\n",
       "      <th>Link</th>\n",
       "      <th>Title</th>\n",
       "      <th>Seeker</th>\n",
       "      <th>First_Post</th>\n",
       "      <th>Replier</th>\n",
       "      <th>Message</th>\n",
       "      <th>Worker</th>\n",
       "      <th>Answer</th>\n",
       "      <th>Date</th>\n",
       "      <th>...</th>\n",
       "      <th>Title_sent2vec</th>\n",
       "      <th>Sim_Title_Message</th>\n",
       "      <th>Sim_Title_FP_Message</th>\n",
       "      <th>Sim_FP_Message</th>\n",
       "      <th>Binary_Answer</th>\n",
       "      <th>Cont_Answer</th>\n",
       "      <th>Message_Length</th>\n",
       "      <th>Message_Sent_Count</th>\n",
       "      <th>User Level</th>\n",
       "      <th>User Posts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3K3G488TR2G3Y0HX29ACGBE4YKZQ5Z</td>\n",
       "      <td>160755</td>\n",
       "      <td>Are Simon Baron-Cohen and Tony Attwood respons...</td>\n",
       "      <td>simfish</td>\n",
       "      <td>[Can anyone speak to the AS perspective of the...</td>\n",
       "      <td>Indy</td>\n",
       "      <td>[Your psychiatrist is a quack.]</td>\n",
       "      <td>A2QLSHXNCHBRN4</td>\n",
       "      <td>Somewhat Useful</td>\n",
       "      <td>2017-11-09 21:41:01 UTC</td>\n",
       "      <td>...</td>\n",
       "      <td>[0.042822, 0.036247, 0.023017, 0.010165, 0.001...</td>\n",
       "      <td>-0.034820</td>\n",
       "      <td>-0.004958</td>\n",
       "      <td>0.024903</td>\n",
       "      <td>1</td>\n",
       "      <td>0.75</td>\n",
       "      <td>29</td>\n",
       "      <td>1</td>\n",
       "      <td>5.0</td>\n",
       "      <td>950.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3OLZC0DJ8JN7FW8TVSNGB7HHKYKIVV</td>\n",
       "      <td>165438</td>\n",
       "      <td>Is stimming involuntary?</td>\n",
       "      <td>cnidocyte</td>\n",
       "      <td>[Shall we cite here our most favorite ones?, M...</td>\n",
       "      <td>Todesking</td>\n",
       "      <td>[Panic., Maybe if you gave an example of the a...</td>\n",
       "      <td>A2M5VW97GIYLHB</td>\n",
       "      <td>Neutral</td>\n",
       "      <td>2017-11-09 21:41:23 UTC</td>\n",
       "      <td>...</td>\n",
       "      <td>[0.046063, -0.019432, -0.052288, 0.11009, -0.0...</td>\n",
       "      <td>-0.007079</td>\n",
       "      <td>0.022013</td>\n",
       "      <td>0.027831</td>\n",
       "      <td>0</td>\n",
       "      <td>0.50</td>\n",
       "      <td>434</td>\n",
       "      <td>6</td>\n",
       "      <td>5.0</td>\n",
       "      <td>3088.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>36AZSFEYZ48XSOUNPIVZPKD4P80VBZ</td>\n",
       "      <td>161882</td>\n",
       "      <td>How toughen up?</td>\n",
       "      <td>Jacs</td>\n",
       "      <td>[I think I get the basic idea of what empathy ...</td>\n",
       "      <td>CockneyRebel</td>\n",
       "      <td>[I find it hard to believe that anybody could ...</td>\n",
       "      <td>A166A2M31CW2C7</td>\n",
       "      <td>Somewhat Useless</td>\n",
       "      <td>2017-11-09 21:40:56 UTC</td>\n",
       "      <td>...</td>\n",
       "      <td>[-0.019942, -0.0084813, -0.063804, -0.00046972...</td>\n",
       "      <td>-0.007033</td>\n",
       "      <td>0.030711</td>\n",
       "      <td>0.038260</td>\n",
       "      <td>0</td>\n",
       "      <td>0.25</td>\n",
       "      <td>222</td>\n",
       "      <td>2</td>\n",
       "      <td>5.0</td>\n",
       "      <td>98804.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3WGZLY9VCHUSW9WUVLBVSWHWHVBD8J</td>\n",
       "      <td>106087</td>\n",
       "      <td>Why are we bad at sports?</td>\n",
       "      <td>bluerose</td>\n",
       "      <td>[I'm actually happy that I'm autistic, because...</td>\n",
       "      <td>Kaysea</td>\n",
       "      <td>[But I also shake my feet and legs too., Heck ...</td>\n",
       "      <td>A2CK0OXMPOR9LE</td>\n",
       "      <td>Useful</td>\n",
       "      <td>2017-11-09 21:41:56 UTC</td>\n",
       "      <td>...</td>\n",
       "      <td>[0.00049173, -0.019857, 0.0039392, -0.027633, ...</td>\n",
       "      <td>0.016929</td>\n",
       "      <td>0.019164</td>\n",
       "      <td>0.019443</td>\n",
       "      <td>1</td>\n",
       "      <td>1.00</td>\n",
       "      <td>650</td>\n",
       "      <td>8</td>\n",
       "      <td>5.0</td>\n",
       "      <td>688.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3IKMEYR0LW3PO00F7LOHMIZJF4H2KJ</td>\n",
       "      <td>106087</td>\n",
       "      <td>Why are we bad at sports?</td>\n",
       "      <td>bluerose</td>\n",
       "      <td>[I'm actually happy that I'm autistic, because...</td>\n",
       "      <td>Wolfen</td>\n",
       "      <td>[People throw the word crazy around like the w...</td>\n",
       "      <td>A2CK0OXMPOR9LE</td>\n",
       "      <td>Useful</td>\n",
       "      <td>2017-11-09 21:41:37 UTC</td>\n",
       "      <td>...</td>\n",
       "      <td>[0.00049173, -0.019857, 0.0039392, -0.027633, ...</td>\n",
       "      <td>0.043337</td>\n",
       "      <td>0.028253</td>\n",
       "      <td>0.026368</td>\n",
       "      <td>1</td>\n",
       "      <td>1.00</td>\n",
       "      <td>480</td>\n",
       "      <td>6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                            HITID    Link  \\\n",
       "0  3K3G488TR2G3Y0HX29ACGBE4YKZQ5Z  160755   \n",
       "1  3OLZC0DJ8JN7FW8TVSNGB7HHKYKIVV  165438   \n",
       "2  36AZSFEYZ48XSOUNPIVZPKD4P80VBZ  161882   \n",
       "3  3WGZLY9VCHUSW9WUVLBVSWHWHVBD8J  106087   \n",
       "4  3IKMEYR0LW3PO00F7LOHMIZJF4H2KJ  106087   \n",
       "\n",
       "                                               Title     Seeker  \\\n",
       "0  Are Simon Baron-Cohen and Tony Attwood respons...    simfish   \n",
       "1                           Is stimming involuntary?  cnidocyte   \n",
       "2                                    How toughen up?       Jacs   \n",
       "3                          Why are we bad at sports?   bluerose   \n",
       "4                          Why are we bad at sports?   bluerose   \n",
       "\n",
       "                                          First_Post       Replier  \\\n",
       "0  [Can anyone speak to the AS perspective of the...          Indy   \n",
       "1  [Shall we cite here our most favorite ones?, M...     Todesking   \n",
       "2  [I think I get the basic idea of what empathy ...  CockneyRebel   \n",
       "3  [I'm actually happy that I'm autistic, because...        Kaysea   \n",
       "4  [I'm actually happy that I'm autistic, because...        Wolfen   \n",
       "\n",
       "                                             Message          Worker  \\\n",
       "0                    [Your psychiatrist is a quack.]  A2QLSHXNCHBRN4   \n",
       "1  [Panic., Maybe if you gave an example of the a...  A2M5VW97GIYLHB   \n",
       "2  [I find it hard to believe that anybody could ...  A166A2M31CW2C7   \n",
       "3  [But I also shake my feet and legs too., Heck ...  A2CK0OXMPOR9LE   \n",
       "4  [People throw the word crazy around like the w...  A2CK0OXMPOR9LE   \n",
       "\n",
       "             Answer                     Date     ...      \\\n",
       "0   Somewhat Useful  2017-11-09 21:41:01 UTC     ...       \n",
       "1           Neutral  2017-11-09 21:41:23 UTC     ...       \n",
       "2  Somewhat Useless  2017-11-09 21:40:56 UTC     ...       \n",
       "3            Useful  2017-11-09 21:41:56 UTC     ...       \n",
       "4            Useful  2017-11-09 21:41:37 UTC     ...       \n",
       "\n",
       "                                      Title_sent2vec Sim_Title_Message  \\\n",
       "0  [0.042822, 0.036247, 0.023017, 0.010165, 0.001...         -0.034820   \n",
       "1  [0.046063, -0.019432, -0.052288, 0.11009, -0.0...         -0.007079   \n",
       "2  [-0.019942, -0.0084813, -0.063804, -0.00046972...         -0.007033   \n",
       "3  [0.00049173, -0.019857, 0.0039392, -0.027633, ...          0.016929   \n",
       "4  [0.00049173, -0.019857, 0.0039392, -0.027633, ...          0.043337   \n",
       "\n",
       "  Sim_Title_FP_Message  Sim_FP_Message  Binary_Answer  Cont_Answer  \\\n",
       "0            -0.004958        0.024903              1         0.75   \n",
       "1             0.022013        0.027831              0         0.50   \n",
       "2             0.030711        0.038260              0         0.25   \n",
       "3             0.019164        0.019443              1         1.00   \n",
       "4             0.028253        0.026368              1         1.00   \n",
       "\n",
       "   Message_Length  Message_Sent_Count  User Level  User Posts  \n",
       "0              29                   1         5.0       950.0  \n",
       "1             434                   6         5.0      3088.0  \n",
       "2             222                   2         5.0     98804.0  \n",
       "3             650                   8         5.0       688.0  \n",
       "4             480                   6         0.0        19.0  \n",
       "\n",
       "[5 rows x 22 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "amt.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.7. Standardization of Features <a class=\"anchor\" id=\"37\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def standardize(column):\n",
    "    x_mean = column.mean()\n",
    "    x_std = column.std()\n",
    "    return (column.fillna(value=x_mean) - x_mean)/x_std"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "FEATURES = ['Sim_Title_Message', 'Sim_Title_FP_Message', \n",
    "            'Sim_FP_Message', 'Message_Length', 'Message_Sent_Count', \n",
    "            'User Level', 'User Posts']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for feature in FEATURES:\n",
    "    amt[feature] = standardize(amt[feature])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sim_Title_Message</th>\n",
       "      <th>Sim_Title_FP_Message</th>\n",
       "      <th>Sim_FP_Message</th>\n",
       "      <th>Message_Length</th>\n",
       "      <th>Message_Sent_Count</th>\n",
       "      <th>User Level</th>\n",
       "      <th>User Posts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-2.887282</td>\n",
       "      <td>-2.969596</td>\n",
       "      <td>-0.175195</td>\n",
       "      <td>-0.994035</td>\n",
       "      <td>-0.996321</td>\n",
       "      <td>0.370038</td>\n",
       "      <td>-0.369337</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.621780</td>\n",
       "      <td>-0.489299</td>\n",
       "      <td>0.090470</td>\n",
       "      <td>0.598028</td>\n",
       "      <td>0.749872</td>\n",
       "      <td>0.370038</td>\n",
       "      <td>-0.225151</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-1.619683</td>\n",
       "      <td>0.310552</td>\n",
       "      <td>1.036519</td>\n",
       "      <td>-0.235348</td>\n",
       "      <td>-0.647083</td>\n",
       "      <td>0.370038</td>\n",
       "      <td>6.229892</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.526552</td>\n",
       "      <td>-0.751286</td>\n",
       "      <td>-0.670490</td>\n",
       "      <td>1.447129</td>\n",
       "      <td>1.448349</td>\n",
       "      <td>0.370038</td>\n",
       "      <td>-0.387006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.678109</td>\n",
       "      <td>0.084563</td>\n",
       "      <td>-0.042296</td>\n",
       "      <td>0.778855</td>\n",
       "      <td>0.749872</td>\n",
       "      <td>-2.190046</td>\n",
       "      <td>-0.432123</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Sim_Title_Message  Sim_Title_FP_Message  Sim_FP_Message  Message_Length  \\\n",
       "0          -2.887282             -2.969596       -0.175195       -0.994035   \n",
       "1          -1.621780             -0.489299        0.090470        0.598028   \n",
       "2          -1.619683              0.310552        1.036519       -0.235348   \n",
       "3          -0.526552             -0.751286       -0.670490        1.447129   \n",
       "4           0.678109              0.084563       -0.042296        0.778855   \n",
       "\n",
       "   Message_Sent_Count  User Level  User Posts  \n",
       "0           -0.996321    0.370038   -0.369337  \n",
       "1            0.749872    0.370038   -0.225151  \n",
       "2           -0.647083    0.370038    6.229892  \n",
       "3            1.448349    0.370038   -0.387006  \n",
       "4            0.749872   -2.190046   -0.432123  "
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "amt[FEATURES].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "amt.to_json('amt_processed.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.8. Train and Test Data Sets <a class=\"anchor\" id=\"38\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn import preprocessing\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.linear_model import *\n",
    "from sklearn.metrics import precision_recall_fscore_support\n",
    "from sklearn.ensemble import *\n",
    "from sklearn.kernel_ridge import KernelRidge\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.gaussian_process import GaussianProcessRegressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = amt[FEATURES].as_matrix()\n",
    "y = amt.Cont_Answer.as_matrix()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_size = int(len(y)/5)\n",
    "X_train, _, y_train, _ = train_test_split(X, y, test_size=test_size)\n",
    "_, X_test, _, y_test = train_test_split(X, y, test_size=test_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9758"
      ]
     },
     "execution_count": 164,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7807"
      ]
     },
     "execution_count": 162,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1951"
      ]
     },
     "execution_count": 163,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.9 PCA Visualisation <a class=\"anchor\" id=\"39\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_new = PCA(n_components=2, svd_solver='full').fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "colormap = {0: 'r', 0.25: 'm', 0.5: 'y', 0.75: 'g', 1: 'b'}\n",
    "colors = [colormap[x] for x in y]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAEACAYAAACnJV25AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXd4FVX6xz9n5vabSkgCBAxFkC5FFBURQQTBtoiKir2h\noKisAuou6K5t7aCsP8ta1wqKddUVwXXtINhAQIr0LiQhyc0t7++PdwJB0QUMRML5PM88uXfumZkz\nc2++c+Y93/MeIyJYLBaLpfbg1HQFLBaLxVK9WGG3WCyWWoYVdovFYqllWGG3WCyWWoYVdovFYqll\nWGG3WCyWWsYOC7sx5lFjzGpjzFdV1mUbY94xxsw1xrxtjMncPdW0WCwWy46yMy32x4A+P1k3CnhX\nRA4A3gNGV1fFLBaLxbJrmJ0ZoGSMKQReE5H23vvvgCNFZLUxph4wTURa7p6qWiwWi2VH+K0x9jwR\nWQ0gIquAvN9eJYvFYrH8Fqq789TmJ7BYLJYaxvcbt19tjMmvEopZ80sFjTFW9C0Wi2UXEBGzM+V3\ntsVuvKWSV4FzvdfnAK/82sYiUmuXMWPG1Hgd7PnZc7PnV/uWXWFn7I7PAB8BLYwxS4wx5wG3Ab2N\nMXOBXt57i8VisdQgOxyKEZEzfuGjo6upLhaLxWKpBuzI02qiR48eNV2F3UptPr/afG5gz29fZKd8\n7L/pQMbInjqWxWKx1BaMMchu7jy1WCwWy+8cK+wWi8VSy7DCbrFYLLUMK+wWi8VSy7DCbrFYLLUM\nK+wWi8VSy7DCbrFYLLUMK+wWi8VSy7DCbrFYLLUMK+wWi8VSy7DCbrFYLLUMK+wWi8VSy7DCbrFY\nLLUMK+wWi8VSy6gWYTfGXGWM+cYY85Ux5p/GmEB17NdisVgsO89vFnZjTAPgcqCTiLRHZ2Ua9Fv3\na7Hs65x44olkZhbQuHFTvvrqq5qujmUvYoenxvsfuEDUGJMCIsCKatqvxbJP0qxZWxYuLAKuoajo\ncw488HC+/34WzZo1q+mqWfYCqmUGJWPMFcDNQCnwjoictZ0ydgYli2UHSCaT+HwRYC7Q2Ft7FD17\nOkyZMqXmKmapEXZlBqXf3GI3xmQBJwKFwCZgojHmDBF55qdlx44du+V1jx497FyFFst2SCaTQBKo\nW2VtPqWlP9RQjSx7kmnTpjFt2rTftI/f3GI3xgwE+ojIRd77s4BDRGTYT8rZFrvFsoOEQnnEYocC\nfwFmAUOYOPEpTj755BqumWVPUyMtdmAJ0NUYEwJiQC/g82rYr8WyzzJv3nTatj2M4uLuGOMwZsxI\nK+qWHaa6YuxjUCdMHJgJXCgi8Z+UsS12i8Vi2Ul2pcVeLcK+Qweywm6xWCw7za4Iux15arFYLLUM\nK+wWi8VSy7DCbrHsAslkki5dDsPnyycUymf8+PE1XSWLZQvVNfLUYtmnOOCAA1mwoBz4O8nkIq64\nYiTZ2dkMHjy4pqtmsdjOU4tlVzAmHfgIaOetuZwmTd5k4cIFNVgrS23Edp5a9lnat++MMRGMCZKe\n3oC1a9fWdJUslhrDCrtlr2fgwIF8/fUqYDqwlJKS1rRs2WW3HrNp00JgADAJuAt4lJtuunG3HtNi\n2VFsKMay15OZWUBR0bXAcG/NF0AvRH7cbcdMJpN07XoEM2cuwOeDO+64gcsvv3y3Hc+y71JTKQUs\nlholPT1AUdHXVdZ8hzG792HUdV0+//yj3XoMi2VXsaEYy17PxInPAs+hSUYvAi7kmmsu3GPHLysr\no3fv3rRs2ZLhw4f/7w0slt2MDcVYagWzZs3i4osvpqysjBEjRnDuuefukeOWlZVRr14GeXkJ2reH\nKVOgWbN2zJhhZzyyVA82V4zFshtYsGAB1113HalUiptvvpkWLVoAKurNmjWjpGQlI0fC4YfDypVw\n1lmwfv1Gzj33XObPn0/37t2ZMGFCDZ+FZW/FCrvFUs18+OGH9O3WjVbo/I9fA5Peeoujjz6aSHaE\nikgF7A+h+fCHY+Ci86F3b8jICJKWFqNTJ/jPfyAvrwnffbewhs/GsjdiO08tlmrm1H79OAe433t/\nLXDugAFcePXVVEiFhvR9UH4YvDgO4uXg8wHEePhhCAa1BT9o0CK++eYb2rZtW1OnYtmHsJ2nFsuv\nkCgt5agq73sAyfJyli5dChlsbRpFIGHgzTchGq1LXp6KOkBODoTDMHfu3D1ad8u+ixV2i+VXyGnY\nkHuBzUAZcA+QkZ/PZZddBmuAL4ES4D9ACg47rDcbNgiLFoX44AMoK4Pnn4dEAvr161dzJ2LZp6iu\nGZQygUeAtkAKOF9EPv1JGRtjt+x1bNq0iRb16/NjWRkAWcEg361YQZ06dbjyyiu5b9xDIAZwOfro\nQ3j33U+Bs4HWhEK3UVGxnGBQeOihJ22CMMsuUWOdp8aYx4H3ReQxY4wPiIhI0U/KWGG37LUsWLCA\nOXPmMHDwQGJlMXDAqcgileoD3AZ8A5wGBIHjgA+B04G6RKO3U1KysuYqb9mrqRFhN8ZkADNFpNn/\nKGeF3bJX44v6SDZPwpHAMuAlF/gRSPdKnA20AkYD64EWQCmXXXY+DzzwQE1U2VILqKnsjk2AdcaY\nx4wxXxhjHjLGhKthvxbL74a1a9eSLEvCCUAdoD1g/MAPXgkBFgMF3vscIBeIM2HCQ0Sj9SgpKdnD\ntbbsq1SH3dEHdAKGish0Y8y9wChgzE8Ljh07dsvrHj160KNHj2o4vMWy+8nMzAQDFANZqI6nx6Co\nB3AZMMNbDgTOAl4FlgNLgXRKS/ty4IGHsGDBtzVRfctexLRp05g2bdpv2kd1hGLygY9FpKn3vhsw\nUkSO/0k5G4qx7NW0aNWC+cvnQxdgibckgaiBAwUaoVl8YwaIADcBV3tbT8Lnu4x4fHVNVN2yF1Mj\noRgRWQ0sNca08Fb1Amb/1v1aLL835s2Zx2n9TiP/u3zahduRkZ4BYeBMgTZAFOgKGaayATOzytbv\nE4nY8YCWPUN1uWIORO2OfmAhcJ6IbPpJGdtit9QqcurlsKFsA6ShJnef/m1V4jCHM9HmewsgAHzL\n+++/Sffu3Wuwxpa9kRqbGk9EvhSRLiLSQUQG/FTULZbayI033KiDk8LAMDTU3gbm+wGuAC4FmgNz\nGTbsPCvqlj2GHXlqsfwP7r//fvr370/r1q3x++sRCtVj9OjRDBs2DCfo6LA8F+1cbQMJJxc4CCgF\nOgKtePzxJ1m1alUNnoVlX8Jmd7RYfoWDux7M5zM/h0xgbR3gXtQacw04pdo0auiDwQkV99eAr7tA\nvDfwD+BvaOu9E47zBSUlKwiHrRvYsuPYtL0WSzXyzTff0K5DOw2xvJgBq/6JjioFuBPa/AkKymGq\nC44DrgPxOGnxXEopJkUp0Bh4HOgGNGXUqDO49dZba+R8LHsnNRZjt1hqI59++imE0LFGAtv+u7ja\nQu8IJEMQC2jkJR6hF0fQmTZo8H0BOlTVBTIoKSnhww8/JD0nHTfNpW6DuixZsmSPnpel9mNb7BbL\nL7Bq1SrqN6oPA4A48GoeJO8HisF3BZyzGT4Mw3cDgcfQHJDdaEmCZjTlDf4NnIcmbX8TuJ1PPplC\n125d1fPeAvgB3CUusaIYruvWzIlaftfYFrvFUo2kpaVBwgcvAW+A5um9CBgO0c3wLTA3AFxJZYsc\nhjDX+YF3eIfRo0cQCk0GeuP3j2fixCc4+tij1f2YBnwM1IdkIsm//vWvmjhFSy3FjpiwWH7Ciy++\nyN133838+fOBKCS6gmkIJgEsgQOmwvoAfBJGVfrfaFaNFJg3kfRyEkXCfffdSjwOrmto0+ZA/H6/\n5osZjgp7MTBBN3Mc28ayVB82FGPZJ/jrX//Kww8/iM8XZMKECfTp02fLZ0uWLOGQboewbuM6AiZA\naVEK6AnMATag4n0z8D7wbyhcA0uPhNS7aBKwQ4FCYCPGXYEkMwkEYhhTRjJ5EolEM+A+6tdPY2Vs\npZpkKhkHZrMhvjluQzGW7WJDMRbLdrj44ou5+eY/0b37clq3XsiJJ/bl9ddfB6CiooImLZuwyreK\nRDhIaZkfTBbwAjAPzfnSEPgzMFGXJRFIHY8+8DZDPY7fcPzx39OkMAVspKKiiIqKP5BIPAP8BXiZ\nlSs3QxFQOUPeXKAIxAiBzABnnHHGnrsollqNFXZLref55x9h1Ci44AIYMQKOOw6GDLkAgAkTJpAy\nKVgZhjVjIP4GyIHopBk+oC+hjJkYs4pAIAb8CHI06lHfhNplnsL1O0QifpYv76ZlGMG2UxTsB6S4\n5qpr4EU0P9iL6MPAAEj1S/HsS89y8cUX76GrYqnN2Bi7pVZQUlLCXXfdRUVFBSNGjKBOnTpbPkul\nhLy8rWXr1YNEohzQFjspIHU4cI1X4mW0I7Q1odBcTh0AZ50Jc+bANdecTCwmEPJBvB6YAGQnkM2l\nfPFFOrHYMHQWpRO9pQcq6hfiOJCRkUGiPMHixYvZv93+cBLqjgHYDE9PfJqHHnpo910oyz6BbbFb\nftcMGzaMunWD1K0b4IILLthumXnz5pGRm8HYu8dyy4RbyGmQw4wZM7Z8npdXwL33wuLF8NVX8NRT\n0K/fyQA6KXUCSBRX2eNmND/ABOJx4dyzweeDdu2gU6eEFvGlYHg5nFsECUiVpjN/fgq4BJgOdAU6\nACeB6QDup6Qcw5/+9DcOPbQ7zZo10w7TeJXDxsEx9l/SUg2IyB5Z9FAWy44zdOhQCYeRESOQa65B\nolHknHPO+Vm53Aa5QhuEPyOMQeiMpOWkbfm8tLRU9tsvTyIR3cdRRx25zfavv/66QETgXIFHBFoL\njBCIi8/nk8ceQ6ZORd55B6lf3wicIbiHS9gfFUgT2F/gUoEFAk96+zpAIE8gLDRAOAfhKARfloBP\n1qxZI2eccYYQROiH0BvBj/zlL3/ZzVfVsrfhaedO6a11xVh+t+TkBDj//Dj9++v7f/8b/v53Hxs2\nxLcp58/wk+ib0JzoAN+DedmQKknt8LFmzJjBsccOYOPGMuLxIjSX+tEY8yOhUBldu8L8+YYVK7JJ\npQSooBPt+JJZJHHQXtFKV8vRqFPmOuDfkD0YhntPBI9HYXGCqVPfokePHgwZMoQnX3gSxziMumoU\nN9xww65dLEutxbpiLLWOSnt3PA4ffQSJRIKCghy++uqrLWXysvNgFjqbUQr4EjKjmdvd34IFC8gr\nyMOf4Se/IJ9FixYB0LlzZx544E5uvXUkPl8Y9aWDmPMoi7lMfR+WrfCRSuUDnwEzmc1mQqR7B17r\nHSEFrAIOQ1MKtIey5NYKxGMAdOzYkfvvv58uXbpQvLaYkvUlVtQt1cfONvF3dcGGYiw7yPfffy9D\nhw6Vdu3aSSSCjB6NtGmDdOyIjB2L9O2LRCLImjVrRERk5cqV4kZcIYQQRkzIyNy5c7fsL5FIyDHH\nHCMF9QoEF6EVwqn61wk78n//93+CExZIF2johVIaCFwhpEWEq7wwTzRDYLKAeMtE2Z/O4pIu0Fjg\nrwI9vf2cJVBfIEfwBYQ/IHTScMsFF1wgxmQI5AvUEdfNlpUrV9bU5bb8zqEmQzHGGAftNVomIids\n53OprmNZai+TJk1i8MDB1Kc+m9lMCSU4aaUkEvDKKxAIqKSecw707XsRffr0oXnz5rRp04Znn32W\nWCzGWWedRSAQ2LLPepn1CBYF6UlPPuADlh2wjNigmDoV70KTdzkHQPJzIB24HbgNOACO+lRzeAE8\nEYZFI4A/oT7Fv9KAJ0hRyhocUhwOHAI8i7bgp6ADnPrh+NZzuBzOjOQXlAJqp3wYbe2fQGHhAhYv\nnre7L69lL6RG0/YaY64COgMZVtgtu0qdQB1Oip/E2ZxNkiTXcz3LGyxn3YZlvPaaulNE4PzzYfli\nP+AjgYvfTbJ+4xrN74KOJv3Pf/5DPB7nsvMvYyITiRKlggpO8Z9C0YVFkIsKewKI9UWticehydeb\nA0lonIIjkvBBFJYKmFIQF5JHAJ+i7pkEmsWxoXcWNwArgUe99+Pow3v0pxcjGEOcJPCKdzyAp/H7\n/0hFhZ2Iw/JzaizGboxpCPRD5z21WHaZeDxOV7oC4OJyKIeybsU6XNcwdix89hncfz+sWmnIoB6G\nYxHupSLZgczMAgDOOussCpsVctYlZ3H+RecTIUKUKAABAmQkMuArYDJQgYbFW74NXW4A/8HAreD6\n6XRQHP/qJDyVDovvguQdYKJwXBL804Be4HYHfxzoj3oXVwJPoLl+K5nLVN7kakYT5zH0RvAi+siQ\nBF4kLy9t915Yyz5FdU1m/SKaTCMTGGFb7JZdpW64LoeVH8aVXEmMGMO5mvnkAjNJT4/hugn8/gjr\n15YRYH/KmYO2T8qAXMaNu5Urrr4CzkbHBS2C4BMhTud0+tKH//JfHjYPE/PHwEB+nXxW56/WgUIA\ns4HJBgYK7osBkvFc4FbgLK/Ag9Dij1BeCit8cFAc8oH3DRT59SYREEj6IXkxhjVk8TKPUMZ5BCii\nEz7fdyQSSSAbqMBxylmy5BsKCgr24JW27C3sSov9N488Ncb0B1aLyCxjTA/02XS7jB07dsvrHj16\n0KNHj996eEst483336Rn155MlanESSKcgg7f/4Ti4j6IVAAQMtpTuvWhMwgEeP/993XQ6H7e6iYQ\ny/DxXNGrPMszOBkOsbNjGkp/GFavXa1zllZSB/AJtICkLwjxJt6+2XqchIGVAvvHoa+3ulDggTg4\nfjg3Dmvi+CffywjgKiAPmEkFtzKDpUuXEA6HGTduHMFgkCuvvHKbPgHLvs20adOYNm3ab9rHb26x\nG2NuAQajz5dh9F/mJRE5+yflbIvdskNUVFRQUFDAunXHoRNYgOZfqYeI2gWPP/54Xn99KnA5cALw\nf8CLTJ78DCcNPEmns6uD9l1OCEEiF3xLNXV6ZdTjBWB2FCLlMDip6ycBK9DJNd5Nh/VD0Vj5BCAJ\n5hIIbIJ4GNqWaTnQFLz3OtAuBRvCBFcnCCfivJhUVzvAIOC93FxWrlxpMzladpgaibGLyHUisp+I\nNMX77f5U1C2WneHll19m3bpiNAj+CbARuBK/P2NLmVAoBPU3Q8E4yOwLbV8AfymhUIjOnTvD34H/\nM/B3HyQLcFiv44e+8XYQA+alA3dCzMBzwINoaz+FCvyPxcB44CBgGIYhIJsh1l+TPn6PWtoXAc8Z\nwFE//ZKWZGcHufJPcHoQRhk4GXgVl7VrU0Qi+ZSVle3mq2jZl7FJwCy/O847bzgwGmgNnIpaBwNM\nn/7BljItW7aEfwFXlGqkZD0wB9q2bcv0T6bz1FNPccHZZ9OcJBewgHbACTEon4LOXFQGJI3uOz0C\nVxZtrcAPQFEU/ffYDLwNpHTaUz+QeAdaJlTvp6IdsauE5qkEy4AymlBY+D1HHgn5+fDJxzD5aUil\n5gEFVFR0p1evXnz00Ue77Rpa9m1sSgHL7w7XzSOVegQNsQC8jOsOIZFYvaVMMpkku142xRXF6jJc\nAF06dOGzTz4DYMOGDeTl5FDO1tbLIOD7zp055ZRTaNy4MYPOGgSpdDAxOLtiS2crT6SjkfGO6NPC\nKPD/CIUV0AT4AihHJ8wIArOg2ZvwfQU8DlxOmHgQ7ryzjFat4Jln4Jln/JQnDZAOyXbkZM9j3brl\nu/MyWmoJNepj/58HssJuQYf09+nbh7Ub13JQ+4N45513tsSbb7zxRm664yZSiZTGxJMz0SbyMfiD\nc4mjqXY7tOnAzBkzSSaTDB06lG+//ZbjjjuOkSNHbjlOMpkk4vMxC2iFmgo7As0HDGDSpEkAmLDR\nlYtDsLYc/CEo86Nxmkbe8iMwH3LKYJioNaAEuFur5qQ5mE0ppic0l+NKoClQTjqBQAnxuBAM1qfc\n/REGl+td5nlDuDxEaXHpbr7altqAzRVj+V2zdu1amrdpzgLfAooOKuK9L9+jQWEDAJ5++mnG/nUs\nqaNT2hVfsBZ8jYACXN93xMPlamEcDLO+n0XPXj1xXZcHH3yQBg0acOedY8nPz+TRR3VQkOu6HHLo\noRyGGlfyCDEHl0MOOYS8enkYNwDlrsbII+UgISjrgyZHzwGORSfSiAKd4ceQajxofN2B/GQ+Q9YP\nIT0RoQB1pd+Pand/ulNREUVkM+WpTnBMOdRHB0X1FcqlfA9cccu+im2xW/YYgwYN4vlpz8MQtOW7\nGbgT1qxeQ69evfja/Xqrn7wEuAcSsQT+TD/STTTkkge86sK3aYBLdraQSv3IBRfA+vXw3HPwyCNP\nMXjwYAAyMupSXBwGRqEZG5+B9mVQF/jAy0/gj+tApaRBk3c5QAPU5PUPr7I3Q+6tcOpm+BCYDWMq\nxjCZd5jDKmARISCJQwUN8dOKUj4A+oFbBt3ehKO83//n4HvfR7x42yyVFsv2qBEfu8WyoxQXF2sD\nuPInGtLXRUVFBINBtQxWUq6fVVRUIKVheDcMvmyQZZBoCDwFFPHjj4O45BI4/njdrLQUrrtuJIMH\nD2bTpk0UF29Ge0ubawGzDArfhvbAxxVqV9kfTcj4aAjis4Ej0Kb8DVUq2wPW3UHgScjMgA1JuJG7\ncWlDknuAcxB8pGhAks+JE0AHNT0HSQf+E4ZNpXrO06FZs6rT5lks1YsNxVj2GCNHjoQlqGauAiaD\nG3Jp1qwZDz74ICwEXgc+B56EFs1b6CA2aQvJ5RCbDxWNIPUQOkPRMcBfmDEjvOUYkQgkEnEGDhxI\nYWEhGl2vU6UWuTriogjN47W/t7oekBMAbkFdMCtQq2MRmipgHEh7+vQIcNtNkHQBt5xkYCZwOpAi\nznqSHIr2qE5CnxCWA6VgBsLiCM4MuOAcWLZsLldffXX1XmCLxcMKu2WP0b17d8bdPQ5nmgOPQ2RZ\nhK+nfw1oPvSp70wlb0UekY8j9D2sL3Nnz2XhwuXoKKDK0Z95bM19DrCaOXOSzJihE3E8/zwUlZYy\n6d1JbGq2CVwXnIHo3eJxkInaH1uKhoJWo9r/DbCuDNxnoSAG/WOoUT0HSCPAvwizkGQywWVXAt2B\nq+PQNwa+OHA92hP7LOqX/BA4F8034IPUaMwmh7/eCIMHw2mnwfPP/3P3XGjLPo+NsVt+13Tv3p0P\nPtgEfIT2Wp6JNuuvQ62I4ygoyKC4eA3GGPr2PYXnJz4PV6Mh8ieAHwIQCEFuClqXwJdov2gCjbQE\n0BBJyIE1KbgAWA9pr8I1FTDcO3JP4EMDEmHrvNcAD2TA2smosT3f27Efzff7hneQR8nMvJLJk0sA\nuOMOmDOnMQsXLtpt185SO7CuGEut49133yUSWY3GSgrQ0UBPo63iexg7diTLlq1i06YUGzcmOe20\n01RTI2iH6EFAVgV0KoKLSuBw1NDuQ9MLON5uhwEXpzST7hS8gUhwCprZzg/8AUgLoW7ISqdiHCgp\nQpObPglb8tf8EW3xtwN/L3AuZ/PmEv7xD7j9dnj7bVi0aC3GhIlG67N8ufW0W6oPK+yWGmfkyJG4\nbl2MqUNe3n7bDLcPBAJs3ryKbt06oNkQZ6LO9HOBoKYPqEJ2draK7e1R+JsfXvJBiav3g5no/eAF\ntHP2rhCkfNqv6gAbIbA4QGRhBOaCGJ0KQ1CTzuMOFJeBE0Z9jc+gKQvip6CxnL8AZeAEgGVAY/DN\nhSbvgVNGIvEATz99Am+95ZBMZgPTgMWUlnaiZcuDd8OVteyz7OyUS7u6YKfGs2yHe+65x5uK7nGB\nKQJtBNIkFottU27mzJkCYYEe3pRyrbztQuI4deTBBx+Ubt26edPS1dFp7UiKzo6RL4T8QpYRQkZw\nswTO9Mo2EvIQLkcC4YCcyZkyhjFSSKE4xpGIQepo9EVCUQSD0CgiuPuL+hfTBW4XSAlcLgR8gg+B\nQjEmLC1aI+mZCLQQ6C3wpEBHgRurTLE3TyCthr4By+8danJqvP+FjbFbtsd++zVm6dJTgb95a74B\njqBXr868++6725Rt3bo1c+YkgRloKsYJqM98EDDGK3UNcB8wBw3fgHrYF6BR8nbA8WimL6+M+wdI\nvcrhcjh/5a8ArGUtZ3AGCXxAF+BwjHkRkUXgHAKpD9CsYs+BuRicUyE5FxDwzdSZlgJ+KItr2gL+\njiY+vQooA3M4yIte/d4ATkekSr4ai8XDxtgtex2aTqCqoG0GIsydu+38n/fffz9z581FRzBV5t09\nBfVI/hGXbFSAxwKFqCsFIImPaQR4D51FYzA4ZWDao4OQHEi+ApKGQ2RrvXARBO1V/Ri4C5FifZ/q\nhYo6QDeQBDiPeuvegmS5xuwTKUhFgZvQTt8TgAeAMvC/CU4/YJjeFPxVTfwWy2/DCrulRrn77jvR\nTscb0JkVTweEJk0aU1Ghk2rce++9XH7V5aQOSIEzia03gmeBNsAkks5KtFcTtCV/KdAbaInLt+RQ\nguZVXwFuDGQW2rkJ6lsP8Clf8jT/5GM+ZiQjSZkUKtZ10ZtAAM0I8w/dDylw/gbpDvj8qAf+ewJB\n8P8fBJ0katOsqHLGMYxJ4a+I0ya1hE58TDCVJK9Ontbk7bepW68ukToRjjvuuOq5yJZ9DhuKsdQ4\nl156KQ8++BTqKCnGwQeU4eDQ9dCuzJo3i5IOJdANeC0Is/yQykY9i510/tFCYEEE5BzUZH4zOPPI\nJsrZqbNZwEL+zZfEzCZwyiHZHFiKhmsqzexxggZ8AZey3DJSywKoZfFV1EZzGfAyOAmQIjAu1PVB\nfDNsciF1Bf7ABNq0i3HdSCguhuHDoaQkG/gzGor5Izl1NtJtw4lcyZUAfMiH3M7t/Ou//6LbUd2g\nE5pT5j/QqrAVs7+avWe+CMvvEhuKseyV/P3vfyc3Nxf1pRv8uDzJkzzKo8z6eBZlpWWq+QZNqn5C\nCfiXUlCQDu40OB9NHHZ5KQQeBS4G/3zcVIo7U3eyivVsIEZdDEhzT9Q7A/9Gk341B0eAKDFTzuaA\nkFqVhrbQT/D+Omg4xUDqDXAM5MSgaLNGj1IXAh+AVPDtVwEGDXIYPz7CqaeC6zqoRfM6oBUVFenU\n2xL/h1xySZHivPPO0+oci9o0z4Q538351Wu3fPly0tMbYEwaxmRy0kkn/Wp5y77BbxZ2Y0xDY8x7\nxphvjTGG8phGAAAgAElEQVRfG2OuqI6KWfYdmjdvztq1RdrZ2DlFrKCMC/2XUJ/6dKc7GaEMeBeY\nBywG/gOEHJYv36BiX9/bUR2gaQU4xRB3EWAsd/ASho84kxVEge+AA4Fv0ZQE7YFbIHUIUK4x8eKh\nkHgbjeH/BQ2lCDARNbD3pm62j+svhn9MgCMOcdCY/jfE43WJx78mlSrl229P5rXXoiSTTdGwz8XA\nU5SWJXmCJ3iSJ5nNbO7iLmLA/Pk/wOqAHgo0vC9QVla2JSz1U1q27EJJSQfvfF7hlVemcv3111fP\nF2PZa6mOOU/rAfVEJ7NOQy0LJ4rIdz8pZ0MxtYgNGzYAUKdOnf9RcluWLFnCo48+SpMmTTj33HPp\n2PEQZs2aBT4D58c0lJ0CHobRK0fzHM/R8tiWvPX2W4jPy4eeAJIhfL5yEg6al7cTmn/mH0A8ANIQ\nHy1IsYQU36AbtgNuBY5D1fNoVLyHoFMqpaHZ1Od7tU2hKQxcNDbiovPn9eTuu8vp2BGSSTjvPFi6\nEkjURcX7HDS8kw0cgibIeQt4GQ39PA7uAZAMAF/h0pwkn6Nu+UOg83y997wDrDRe1kmHBg0KWbJk\n7jbzpeq/3LdoLApgNIWFz7F4sR3RWlvYlVDM7vCrTwZ6bWd9Nbs7LTVBcXGxRLOj6uc2SDQ7KsXF\nxTu07YMPPiihENKgARKNIrm5aQKuOPh0f9cjjPWWA5FMMiXDzZBo1EjTpshBXZBgBCESFr//YEmr\ng9ANIQvBj+AicLLnUc8VmCxwcBW/eJ7AsirvRwuM8V5vEnAFGggkvHVlAhkCzQQ+FSj31jeVUCgk\nxx4blCZNIhIMO8IgBMcR3IZCMCzUzRR8YYGoQGOB+vraFxWci35ShwFV3t8qOD4haAQnKnC+54Ef\nKtBcunU7cptrqvWbWmX7AeL3++WNN97YDd++pSZgF3zs1S3qjdGH5bTtfLZ7z34PEYvF5LDDDpM6\neXXk4IMPltLS0pqu0h6l8f6NhUYIo72lEdJ4/8Y7tG1ampErrkCmTkXefBNp2NAT4yBiAggFCKMQ\nzlOhzsrKknA4LAcdhEyZottddx0SCiEh1y9REHMswp8RrkVohcDDAq0F/AIFAlmeeE4TaC5wnkBM\nYK5AjhhOEHhGoLN0oZsYMgX+4O3naO8GkSXwmSec/xUdKDVc4Hotkx3Rm1EAIQ1hpHdzGoTgdwS+\nFZjh3TQyBJ6rIsRvC+zvvY4LdJOmTZt6N4T13voNAnUFbpFwuP421/TEE0/09jlK4A8SiYSlb18k\nGEReeeWV3fETsOxhdkXYqy0fuxeGmQgMF5GS6tpvTXL77bfzwgsv0LBhQ5577jkCgQBZeVmURcrg\nQPhs9mdk5WdR+mPpNo/HtZmlq5ZqJKMy2eLhsOT1Jb9Yfvny5aSlpZGZmUksJvTooevDYXCD6OQZ\nR4OsBt4ycJsD/iTEfWzc5MNxNnLggeB4vUFNm6pb8RTiHAnc/TbMXQoVJ6AORIaCrwICUShdA44f\n5F6Qt4Cj0DBLGHW5hDG8Rx5zWG0WMDPoIOVBNLRRhE6ZFEGTyvRCg97FEK6A+vepqSZeqJq7Bk1l\n0MLbPd7ruHgvfMAt4F4G5m5I9PfW3QOsAn9zkBgk15Od3QoNFVWGubK9C/Up6en+ba7v5MmTuf76\n67njjlvo2ROGDYO0NIhG4bLLLuKEE07Asu9RLcJujPGhov6UiLzyS+XGjh275XWPHj001/bvlKOP\n7suUKR8Df+CLL2aQkdGQhx66k7LyMhiKGiUOhoq7K/BFfCDgBlwmPze5VvuPQ4EQm5duhpbeiqUQ\nDoR/Vm7RokW0aNeCRFkCBOrk1sHngylTYOBAKCmBHxahubIi6PSii4PwTQX+uEvcnEIw/XkCxvDK\nK9C3r5CdDfffr+3xx7zjnJiC3G/QjtXE/sAiCEZh82nAXZD6BjgWnHWQOg84DU2xmw+MJ8WPrGY+\nHA6JvBR86sLydcAGNOd7Q+AhIJ98vmd1BhpGnxqEoiCsWwMry7SI+GB2EN4qh55JHdjqz4B45b/Z\nQihMQvQr+DZbk9HQEOQuSBsB5SWk+YIsmjGDIGFiPA2cCryo2zKP116b+rNrffPNNzN+/C2ceKKK\nOkC9epBI2On39kamTZvGtGnTfttOdraJv70FHWFy9/8osxsfVqqXRCIhEBSYVeURubW0b99eyKgS\nBx6LkI1wKMJwhCP0cXxHY857I2+88YbGsxt7i5/txnPT66ZraOQGhGsQMvXahEJIQQESiaA5VYZX\nuZbNIqK+QyOuPyTZOUjr1lFxnJPFdX3iuq74fPWlx9Y4hpSB+DAa0gmEvLCJK1BSJdxxseD4ve/0\nANGcNPleCOZlgTwxBiEHwecKTBe4XzTnzBni0EHS8EldvHBRfkRwBwm8KXCmEIh4IZsjBB4T6C+4\nEb1OPgQuEZxLBBPSPoGxCNchHITgDhF4RHDSpdsR3cQPMh9kJkhjogJGICpNmzaV2bNn/+L30qxZ\nQ2nRAvnHP5B770UyMpDzzjtvd/4ULHsIaiLGjiZCTQKz0Px5XwB9t1Nut1+A6mLjxo2ewCSqiMNp\n0rlzZ42jHokwDKGnxoe3dPqNQYggTzzxRE2fwm7lyy+/lH79+knfvn3lyy+/3G4ZQggXVRHt7trZ\nesghyMMPI336eTeGLITjEDr7vM7GiJCNuEHEcYJiTMgT5IcENgscIWECcjOOvA/SF7+ECQlZIYFe\nAp8L5Hh/RTQ511ECPsEEBBoKDBaYUOW7fUWMP0O/2xxH60ChQMjb12kSCrliDOL3GzEmXTTBmOhf\n08CLiVd2riYEp0A4xzvHAoSjvD4An09oFRD29zp8+aNASFw3W+rWbSgOSLLKjasfSDAUlClTpvzq\nd1JaWiqFhfUkEtGO6Z49j9odX72lBqgRYd/hA+1Fwi4iYkym1yEVE/hYIE3Gjx8vr7zyirhprhBA\nTNgIYa9VOtbr+PPxs3/CV155RRwnRyAqxmTLhAkTauis9hxu1BWO9a7LJQi+iIBPXDckd96J3H47\nEshC6IIQdQXXJxAWLva2GYzgN4LjqkvE5xM4yBPQP0uUvpJJGwlwmnYeGp/Aj54ePieQLTDME/uO\nAgGvA7KzqMvl3irC/oJQkCH0Rgg4op2sbUQ7YJ8Vvz9Hevc28vbb2iJOS0PgP1tFnHyBzCpin9Jj\nnOPd0Bqj+/Yh1IkIdYL6lNca7+bR3LtxnS+GiFwGsgFkEojxe52yfiSQHpCNGzfW9Fdr2cNYYa9G\n3nrrLTEmy3sUDsuAAQN+ViaRSIg/za8tsp4IdZHM3MxtyhQXFwukidrqfvAe8SO/+lhdWloqjfdv\nLCZixE1z5aabbqr28/spiURC9t9/PwmHkXAYCQQCO+SqmD59unTq1EnatGkjjz322Jb148ePVyFr\nimDCAv/0BG+quG5YunVDMrIqQxUhbSU39ET9ei8s0sBz3tyAtnDTEA5D8DUVWOW1kE9SsTcBgYVV\nxLqrJ+aNvf0j0FbUDnm4d4O4TwXVZKqD5WL0RsJpAmM9sW4t0FnS08Py5JPqzDn5ZLz1E8XhFMmk\nssV+oahr5lp9f5FXZ8KCL13P9ZIqT3f74d2QVm2ptzGHSzCMBFzEF/bE/0/eNSlEmrVotht/BZbf\nI1bYdwM/zQv+U4qLi+XQQw+V/Pr5cswxx0gikdjm88cee0w0/pqqIjrtZMiQIdKhQweJZEekSbMm\nsmzZsi3b1G1QV6iPtviOU/HblfDO3LlzJZQZ0kf+IHLhhRf+YtkDD2wl++2HPPYY8tBDSG6uT+DX\n/dDvv/++hAjJERwhx3GcBAnKqFGjtnw+duxYCYfR1rN7vhDoK5j7BToI7fRGqIJ4hgp/wBGuROjl\n9V0MqBLKORsh6KhQpvkFfKKx9HTRuLpPoJF347zQ269fRbVeQMCIz4fG0vELvC5wjsDpgpOnwh6M\niIZp7vW+sz6yNRx3l+Tnp8k99yCtW4dFc6vXkUKaywDnJDme/uIjTdS62EKP6yCuaSXwiEA9rWvQ\ne0ppUinsjmzbH9BffH6fmBz0afCcKtdgAOLP8O/078Cyd2OF/XfIlClT9J98S5igXCBP3LCrLdL+\nXmu0SqfrzzoVD0Jatmy508f2p/uFtqhYnqGP878UBsrKMnLbbdoirfSLh8MNpXHjpr+4/0b1Gsnx\nHC/v8Z5MYYpcx3WSYTJERGT27NkSCnmtWz/CYa4Kdd2I4AsKlyFcUSnsCYGRgmmqn0VdoRlCZ69l\nWxmj9x8qmAyN3+ciuA0EvhCYLTr4aITAJQI3iHrMGwncKdBWQiHkkUeQiRMRx4n85EZ7pCf2Paqs\nO1vgnirvvxKcdDE5iAmGRT3mjeSkk5CzzkTSAgHpQFAyjCuNCclykGKQQ4mIw988YW8vOsFGPaGh\nEUYg+MMCxwh8JDBeNKzTWPLzvRDMYVVa+G2R/Ib5O/07sOzd7Iqw2yRgu5mePXuSlZUDHAzcBM6h\n4C8iGU/q7G5dgDOAsKFLly66kUE90ZVUgM+3c87UZDJJvCSuc0pkoVbq1nDDDTdst7wxLqtWbX2/\nYoVDMplFeXl8u+UByovLWcoajqE/x3AsU/iIpCQBuO2229hvP2jSBHz7A8ckNS3L2aVATEfoh0Dz\nA5QBz4O8CYkFsPlyKHV1JP7jaJqAj4MQv0GjKycDm9MgeSvQEZ0qrwGaIuB+/P5VuO6t4F8K4Wug\n4TdUCKxZA3XqQIMGAKNR8/mLqC8x29tHJYeiHsaNQArMOGiRRIaCFKbAnEEksp5AANauh44HVxDz\nx0hIks2+cka7OmXIRZQSZgrqiV+PeuIfh2UZuvsGZWCmoTNovwO8B5zB6jUw7s5xMB3NQjwezHzD\n1Hem/uL3YbFUYoV9D/Dpp1PAzIeDx8LxM+Hick07UqnVDhAMsHjxMgBatmgJ/0Q9Rv8G5sB99923\nU8d0XVePsdFbIUAxbNi4gffee+9n5a++egwTJsD48XDXXS7PPBOkomIhQ4Zc8IvH2CzlzKKIBAtJ\nsprPSG7JiB6NRikt1VwqbrDKRpXjuIqALwFj0Bmkk6jwFQDdYVWGvl0OLM+D+AtgRsPB5TpHadME\nOitSJe1wnAtx3VNo2vRxbr45RTAdnSHvQkgNhptu0bb3ueeWomrZErgZTcvbAs3l8iRq8JqMjkBq\nCGRCnWfhhFL9rg6Igf9dwuFSysuhbVtYthwWOVAagrWHwpNHQU8/PIKhHG+2Jd9hQAaaEKwDbO4E\na4HCCgh8D+YEdPD26yBhHnnkKX6Y/wOX/OEShp0xjHUr1tGqVatf/D4sli3sbBN/Vxf20VCMiHbE\nEqgSVviz577o4NMOth6u4A9K8+bNt2zTq3cvCWYFJTM3U1566aVdOm7T/ZtqnPZIhJZo3L7eL/ub\nx48fLz5fQNRemPGrPuh+/Y73OhefqhKueFcgR0RE1q9fL9GokYMPRoJRhKPV6RIqQHL9SNjnhRo6\nI3Q0ggmKulaOE+3wvEVw+wiRiECahMMZ4mT59dpVxtwJCVwkMFyMCUvr1upYufVW5PrrkXB7th1z\n4EMOOggJBBDcqKjqrxB4UePfpqngpgvBDKFhyMs9E9I5Utt539tor1PXh7RttzV0NWkS4rqorbHy\neCcjJmg0vDQUIeoT7eyNCNwh+Pxqmx2L/nWNqHf+XIEFAkGZPn36Ln33ltoDNsb++yQWi6mI9UBz\nmpyK4KCDYYIRwZ8hEJWFCxdW63FnzZqlxzkYoS/CVQgh5MwzzxTXzRHIlEAgV/773//KddddJ07U\nEcJIg8IGvzjIauLEiernJ+R1el5dRdj/JrDVFTRo0CBxAoi/KeJkIDkh5M8OUgHSA4QDqojgmXgd\noD6B5bLFNmjaCY4OanKC3rn0wnPaHOTdXHIFjpVoNF0KCoycd57aEoMZbBXOAXruWu823s0rzRPS\nOgLPa1+Ib6DgO1UIhoSWRrSD9mDB304HQPmM4I8IhKVhw63C/uabiOMi9KtyTmchhCJCo3R9X4hA\nZb/CfoLpIITCwqVe+SiiA6Yq4//19wlrrOXXscL+O+all14SJ+xsSXo1atQomThxonTr1k369+8v\nK1eu3O52X375pdRvVF/C2WE5uOvBP3PdbI8ffvhBAukB7+bhdV62QkhDsvKyPGG5SkKh7hKN5onj\n+LReJyGcj9AQyc7P/tl+r732Wq8juJdogq2VAk28VuiJooN50uSee+6R/v37SzSKNGzqtdYH6mjT\nAwzSCiSA9xRRKYJpYVF3i190pG/lzaKPGJ92loYM0txFfCZNdNToN9rCbotwoCvhsGaMDASMdO0a\nksZNAnpeIe86EBAYJ/CSwGvedcj1lvayNdOjCOZmIRrRTI287YntctHkYH8QHZ0akCGXIuPHIx07\nIk4AIYK27s/yOnj9CLkRoXnQu5mEBDrI1sFMDwl5acLplXV821v/kkB4G7eUZd/ECnst44cfftCW\nfnuE41UonIAj69ev/1nZ008/XSJExI9fXL8jtEAHTA3VlmBubq4MHz5cLr30UoEDJBBIlyFDjNxy\nC1JYiLgNq4jsJfpE0bt3b2nfvr106tRJpk6d6gnvCwLFnoj/VzTz4M2eSM4Q6CGhtJCm4Q0i4Wxv\nkE2WJ3gBRO2A+6noRfG83gFP1PqK2hXniYZ5wkKaEXxRySRdngVJIyTqegkKJiqEI0JzxGSo/149\n7Dmi1sXB3s0i6tW/ncCx+t4fFrIiQtD1BkA9U+WG8qo6cJoGBXOZJ+wp0eyQ12gZN0OCzZFIIz1X\nWhlhIHrtowgdEAItvZtgU4EiUcfOn6ocZ7meox8dtBQICk5QICwnnHDCTv1eLrnkEsnMbCB5eY1k\n4sSJ1fUztNQwVthrGQMGDNB47hhPcK9FMIgxmdukC+7atatEw450CreUsYxVcRtaRaiPRurWqysi\nItddd51Aphx7bGBLGOGf/0SCYa/sZZ4oNUTIQaJZyBFHaBpYFcfKQUD/Ek0X29D7+4wnxiFttUYR\nTvSWIMIJCIcjuAEt4/qERq4e74xKwX9PdETnKZ4wZ2iLO80ImRHB30BCZEomPu+Y33l1mSCEo+p9\nz/dCHl6+eIJeS9gxAoeKWitnCYGA0MdszduSa7yW+w+iOdvbSVaOK/hCoqNYW3hPJy1Ec7dPFkIh\nHTx0CZpDqDL+/yfv/H0hgXdE860PrtISbylql0yJDoTKFup4+zjCux4GefLJJ0VEB4/16nW0hMP1\nJDe3kUydOvVnv5X+/Y/zblx3iobHwrvcN2P5fbErwm5dMb9jSktL1RJYOXdKADA+RNIYM2YMABdd\ndBFffvkJlw5N0WvYd9wXuZXMVKY6+SpZAVnpWQBce+21QBmJxFYbYzKJThb0LzRHZ3fgQmAobM6B\nUAhOPdUzsDAGiKH2EBdY5r2/HGilE0VnolluO3pLP9Td81EEkh8CZeAE4aSkTlDUAnWDcgz4jgfn\nZTBl4CuCJqhOXVkKHVdQHihmEw7QHzjAO4MhUBbTKfLOAYqB/YC2bHUfdRbdP67WNxXXwaOV17W1\n0LTpOvz+FkBTcnIWsOnHECSu8bbp6p3ICtS5czq0L9fvZqVXDVPlrwCJp9GZjd5F0wUPQdP0rkPT\nWRZ6F/weiGfqx58Bk4BcOPv8sxk+fDgtWrRnypT5lJXdyNq1x3PUUf2ZMWPGNr+VN974EHgBGAHc\nBVzIFVdchWXfpNrysVuqnzFjxvBW97fgI1QHPgiBeySk1rJ+/XoAXnrpCS6/HI49VrdxnBivjouw\ndjIEFkF8A8hSmDR9ErNmzaJTpx5AmPffr6B+fSgshEefgEQXVNyLUDEF1e5mMG0a/HEEBAKbiSUn\nQuIlIApOCCQCbik4MfWpz0Und64UuXVoSl0BpB06SzNgsqFkM+R45TYB9ILEAHD+CBSrnrZlqym3\nDfBDClbH0Vxzm7UefALGhUYJ+ATVyxO9baajc6TGADMeJA/4SvO0f10B3bzPZkODA4RTT43x/POw\ncmUSSUWARehN68/eDo+CwPnQokRdkZ975+oG4dWY1vEr1J7P2RiSZBBlE7NRe+WNwLPA297SHHgD\nYo5eKz9wmbf9ezBu/DgQP+q1b+bVYSnDhg3j+eefZ9SoUcTjcdQuWrfKryef8vIkln0TK+y/Y7p2\n7cqD4x9kyOWXgZMOid6QPAi4iauuesIrJVsmoQCdkGKZWUqz/aDoB0hPh4VxaNmyJdnZhYgMBB6i\nouI7nnvuSHz+dZQdBnIYqg0zHM15drxAOQRnQDgFTzwBiYQDdeLw47GQeBlSDnA+JJ6Do8rxfwfh\nICTKoPQtdC6t2UAHtHH97WeQuBKfeY1gRRnxJwJUHFKhU33O9ia08N0KGXEdwLQA9bpXivtX4KwH\nTJiUZKJ3klbAVGgYU62vj1rPK8kDiiPw7VEgvdDW7Gqo6ApTpsOnMYilIBXmv/8dzkcfvQnMI5Xq\ngT72fIX3OOFRB1wHBqJaOwlIpEEqB75uC7PnQ2oRmDhZrnB+chjjeQJIR0daOahvvxDNKt8GGAH1\nSvR8G6KD0x4MQ2wQSF3gAXTyj0phz+LHH+dR2LxQ5+LwoTeE+JnoiK6VwO1ceOHQHfqd/T97Zx6n\nVVn+//fZn232GRj2fZN9E0RFYhEE13BDUUNFxUjF1EhNUdxDLSRScsMlcU+tVFzAb5lRJhL+1JIs\niwwXDAUGBmbm8/vjOjOMqQjDpnDer9e85Hme+5xzn8Xrvs91X9fnStgN2VrfTUP/SHzsDeaFF15Q\nGJYJ8uS6Jbr11lvrfjvxxBOVzZoEwIUXWhy366IxY9BZZ5lMbnFxJEmysL4bhD9BON8VXGd1OvMd\nS13vj6CPCPoLPxSup8DzFdWF6EXCydOnS7s9IygQzZFbjDr3sNC/TAFyUvGib62vf18U+ZH6uf10\nBVeoBR21SfMlKwtDzIj9g02x6gEWFpqyBd2SPJMpKCnJxNuFglEi9OOokkAUOhba+X0sdp+9tCmE\n8HhBf8FcWbx453gfbwiukMnoFsmhWPmU2HoAOZkkwKkymQLXwhoHI5MymCWTM2gmi4u/RNBMnZy9\nFEYZu5YUCzbEfagSlMt1UiKbZ/sL7PxIIfr4wjmn3jW+T1ZW70XBLYK0coU50ZtPSy6ErqBQjlOs\n8ePH76rHNWE7Q7J4umcyfvx4FRX5Kiryddxxx+nyyy9Xfr6rXA4VFEQqbVyqRk0amQFL+xbTPsCL\ni0oUy8L8DjKjyGmxEfwwNlKBLF69XBbB0UOWRLRRJlP7LTN8JWkTtHICDRjgqawc5ZXxaRGrkSh0\nQj3N05rEtxUxKF6IbGmLkUiwQgQl4pR4Wx/R3BV5CN9VEHRSJrO3CgrSOvZY5LpBbHhzNsAwQrg/\nEG5o/XEdbdKAWRUPIKtVFydPt3jb1rJkp8tVKyoWkiePjrKapa/LFk5LZTVUe8Xb/b6eAZ4ouFYw\nXjgdbCH3aMR+jiUjMVqmcnmIWrXK6DvfsWSp448/XvPnz9fKlSvVca+OVpCDmfX2+7u4T8XyvBJd\nddVVCvKDz4ikORlnVz+KCTuAxLAnfIprrrnGDON+8YwuwuLUa41Bb8zYhGOEf5wZLjdPVrh5jWCC\nyc26mdiIHSuLY99bFkHSVhZa+GdZlMoCwT/keU1VUoKaNEduU8R3EGdYWGPommEfwHDBPbLiGaE+\nLcp1hDzPs6zZY+K+Xowo9WSJRJLrXqmCgpSsIMYlsuSoEsE18aBTI5t9d46/ny34jWwGfk/cZ8mq\nHoXxAFV7/NoKSyNkWam13z8afyeZTn9TwU/r/f7deKBobxE559e71u0ysnj/QcpmM3rySYtIGjwY\nDRo0qO6etWvXIb7GZTKf2JuCvsrlSj91bzt37Wxy0bWyxh1RSZOSL3wWqqqq1K/fPnKcUnlemaZM\nmbLDnruE7csuM+zAKOBNbJnse1/QZgeffsL/EhVEYmQ945JjUybmNExBMedbYtJQx2K5HTc2LJ6I\nssIfLPhHbGSKZJWllsSGb4ksSWlg/FtOcEZsXEPbj9dbBHkWM+6hqCBSb6+3BjBYDqfIZv0tBY/F\nxvE9RTTTD/iBuSam1utvf0+muniYYLgdL4hEt7RokhNeLu67I3OZNBMcI8dJKZ0O5DiR7A3lIJkL\nZbRsBu/L4sslWCTH6SLfd+U4+fG51BruHwrG1X126S9oEs+o75XnZeQ6yKVQgGgSiY4ZC1/skJHN\nwrtr6FB02WXoBz9A3bqhIAxU0rhE5eVN43790DTiyZO9hfQXFKt79z5197aiokLZoqxdIw8FuWCz\nyUzdu/eRvZX8Ula+L7NTdP4Ttp1dYtix1aBl2GpQgMUKdP6cdjv8AiR8Gj/PF2PrGcYuWFz8qYij\nPmcGvw9WrejbWEx4Ok/wWj3DdlU8kzw+NjrPxoburHiG/LFggCzeu5ksxn2D4K7YcDYSLioAFeLI\nIRT0VhB0iw1yWwUU6ERO1QIWqCQssQImlyLORWSd2PDdJ7jVKiydEPf9LISTkc3Kq2KDnJPvu2rX\nztOZZyLXzQnei8/l74KUVXn6FvGbyk2xob9AcL48Ly3LFj1VDmcI0vF//yO4U6b54gsK1Lp1nq68\nEkWeJ9dzhZuWubhuFm6JzeBxRZCR62UUBP3leUPsvNtg0s1kBAtF0FH0ckTYUvZmUNvf4DPJacuW\nLdts0ZZazJXz23r3crqKi5vtqEcvYTuyqwz7QODJep+nft6sPTHsO59hI4aZb3pCbMDzkJ/xbYEu\nhbk6Tqtn2PdHlHn2716YG4bH6xmDEwX5CklrL/ZSQH5sMJbWa3OjSol0EKl4tvkNEeTMn++aUNZD\noKdBzUjJYZjCMNR556EUkU7ndF3GZRrIMOXRxBYVQ2yR1y2KZ/YbZW8GoQhSYkg8gLkj6vWjRhDJ\ncdDjj1uB52x2r3q/S9DUEqSmYRmj5MuqKm06lzDMKEO+hjBEAaVyyJNLRg758lp4JlLmRPKCtFq3\nRkt0m64AACAASURBVK7vinJH5pap3c9zdt2cULQLhFPf7XOVgiCr4cNRYSHy/fE2uB6FCL/xP+eT\n1qJFixr0LNgb1XP19nehSkubb+cnLmFH0BDDvj0SlJph+qa1LI+/S9jFPDv/Wfp16Wdh0z+HHu16\nsP6T9WidGHnASGv0MJY78wrwEqabvi7+rmw1MA44DzgGh0foR1f8VCX/CP4K7hoiqoBjMJnbalye\n4CAq+R7racFaCP4A3TfAhdVwHlQXQAWWKnQr68nnfao2jmD2j6CGKm7hDi7lan5PitUcCBtbwIb2\nUFMMNd2wgPirsBfDd2Dj/4MXW1ioYM2bUCcc/FebJAO+b/H61dV/B16If38C+C+8HX8sAwtIr//o\nNkPyOYgR/IHX2MhAFK6mxq1E3T+hekUOKmaAplK9sYwVK2DokBqaByKKnsASAwBygBAh/C0LOqDe\nMfalvNznoovgttvAde8xu5sB9HvgKSxe/3LAp2/fvl9636urqznhhBMYMGAAV1xxBQCDB/eO7+Wd\nwLXAjVx++YVfuq+ErylbOxL87x9W9mBOvc/jgZmf024Hj2sJW8OqVavUrFUz89FGNoN3QakiTEq3\nMXW6MW4Gefiay1z1SHXQsceaeqLnZWS+9uGxCyNfDjnl2E8hhXJwTfukvrzBgegUz6aND4AKCRXQ\nQ9MgLi33TYsg6ROI7lH81pAR+/pxfdASmf/8+Xqzzzvj4xfJQhvHKaJMZbRRGA5Wz55pzZiBDjwQ\nc604aUFKRx2FSpvGWjYucpxa3/wiwe/lus3l+U5ciPxQEaZNGsFzhJuLZ/hFsbsmT6lUTvfdh+bP\nR+XlCL4neFa47WN5gfMEJ8j0aj4SrBOM0mGHRXXyDqWNsDeRiLherIV1Ok6hvvnNb2rChAmblfKt\nqqpSSbpETWmq4QxXlqz2HbivJOmwww5XGJYrkynX9ddfv7MetYRthF3oinmq3ucvdMVceumldX+f\np3eRsOOpqKhQQVlBnZZKtiir//znPzruuOOUTaflYd+TiiMupqGwHJVQqgUsUNYP9dhjaODAUBYC\n+VFsXBfHBu5nsgXFTlYMPHTFaGeTDn2powz5KiM/jo83l03j2hqloWN+79qBYD9X+K4pJeYwDRtq\n/eG1hv1cWcTJ2YLOKqRQc5ijwRwkh6ny/fOVzfZSKtVKpkzpClLy/aw8L1IQpAUtFYZo1ChX2Wye\nwjBPvu+a5ksnzCXUFdPB8TIyP39aFpXzr9hVMl0tWuS0YIHlD0BKuAWipSvcIXFfz9em2PlI0Fz7\n7+/p2WfR1KkoKooXtVO2KDpq1CitWrXKiqbnUacFP2PGjM+9v2effbZKKdVTPKUFLNDd3C0PT6tW\nrdrJT1pCQ1mwYMGnbOWuMuwemxZPQ+wducvntNvhFyThy+m0VydL2vleHHHSErVo06Lu98rKSr39\n9tvm1y5E9DNffESkczlXrdKNdMUVqEuXSKbEWN9nnRUMFO5QEfUTweHyvKZxLHrt4mcTWVjk84JC\n4R4pE/66WFAkL/LE+HqG/Yj4jeJ4LOmoM7IFyzLBKbLF2wJZ4s7jgiJFbqRJTNIsZilNsWxBd2B8\n7AI7Lo/Gff593O+XFEVO3cx5wQLUt29tUY4CQSjXcwTIQiP/Gs/W6ycSfSLX9TVzJopS8QAZxOce\nesK7RDBU8JRsUXSd4H45Tp4cB6XK7Q2JttiaQVcURKEgZ7Ht3VLiIsShyEl/fsz6wQcfrH700wIW\n6Cf8RIdyqFKkNHHixJ31iCVsZ3aJYbfjMgpTCXkLmPoFbXbw6SdsCX5evDBXaziPR27W/Uy7yspK\nFRYXisCqAJWXlytDSmBKjy1aEBvE2oXTeYKM8ALROmXGebAn/Kxatdo0M6+NQ7folHx9Wnu9l5yU\nY26gszEXTiFyWtZTYTyBeKZ8mCxR6FiZO+gdwXBRFAg3Upo2isgqore5m5r6tojLlbKwv/oDUh/5\nhAoDRzfeuKkiUn4+8rxessiev9hxnVpFy1/Fx62vrf6YIKswwqpW+VgE0jTEeYhUINhPMEWbFkRP\ntutSGIemlseL2mPjxWwKZOGUy4U/UvRMWYRQYP8/VVRUaMqUKTrppJO0aNEi3X333YqIdDZnq4AC\nTWKSTuM0RUS68MILd/bjlrAdaIhhd2y7HY/jONpZx0r4YvJL81ndcTXEa6cshNTiFOtWrdui7aur\nq+nfvz9LliympgbsJS2FLRRWmbrj1PhrgNtd/HehqqoluB9CzXXAJKy4c3NMjyULCNzOcNxfTXfr\nj9i7YCP7iUOBe+JDrQKqPKhKkcNlPetxccnDZaVbBTUOFnnbEtz/QN9VsBcwtxhbPBwfH6Aj8CEB\n7XiIT7gbeDyA0jL46CMzvZWVxeBshKAQalZC1cy4v2dgmi8LsMXRdpiwzVpo78CwKjvU9+tdvDuA\nd3LxCbUFImzV917gNuA5CGug6ydWyHuNA5UXYwunWNt0T+i/FvdPLh/88wMaNWpPdXU+JgD2OrAO\nL+2RWpdiEpMYwxgAHuVR7onuYeX6lVt0nxO+OjiOgyTny1tuIpHt3cN4+L6HTfFwLnA38CLcc9s9\nW7y953m88sorVFeLefPmYQb9MmAWsATkbAoGAaiuoaqqBhOmWoNF2FwOzMQev6GYBTwG8pdbLedh\nQEcH/Azke7ACqzE9ELOn3wWKq4F1rGEdTfF5nkr2oxJq0sBEYCSEf4XgYysY/RcX6InpEddgKpPD\ngE5Us5EjyPEwWTZszPLuu8WsXx9RWZkPbj8oWg+H/Qv2Wwf+d4H+4HWB8DGIPsEUwc4DfgWEsOw0\nuC2NU+3YeyyYcuN7QNk6gjAf130N132ZMKzCxMH+CTwFG+bC0qyFDW0Qm3YAsAzWObAIHpv3GCNG\njKC6urt9z8vAD6E8R3WTahwccuTqtsyRM5G3hD2DrZ3iN/SPxBXzlWHJkiUaPny4hg0b1uC46CVL\nlpjrhe6Ck2SLqC9Z9mrTOC68vyOCUJYJ+lP5fmBuEdKx37u1bAE2J5yc6IklI3VPCae3bCH2m3ac\nMPax17qQDiB2x8yQRwdFuObHjlpbWTs3YwlXo7DknxIEHWRaONWyyJfmssXPqwUHyqQSfhS7Sd4V\nNBKe9+ls3e6R8PuJNr7FwB8T+9EZYa4lr5Xg7/LYX+MYZ3Ht2Xid4DDkjkDt26PHHkNPPIH22suR\n54WCV2XJYNNtX73d2J+fjc/zXEFO+++/v+6//3716NEj3u779VxKb4lM1vIWfFRMsa7lWl3N1Sqg\nQIcfdvh2fpISdgYkrpiEncUhhxzCL3/5CfA4Ngv+JZACtxoarTZ9dRcLwSbEXBXXABdi8eJLN+3M\nLyFb9REVAcgH1oXYFDcPc1u0gXA57F8N+2Oh6nNcWNkB2Bd4APw15l7KA54KYHUNBBEogOrV4NbA\nhnzgt0D3+MA/wuLEy7A3hzMwF1HGfnZOB3+OFR7ZP97kUd80wc6utmOBTdT/CXQA5x3Qux5edX8G\nEbGIP7Mh+i+cA6Qh8yCcfyQMGWKbLloE06cHrF07A3uT+Zb1wbkPJ1zPn1/+MyeccAJr1qzh5JNP\npri4mDMmn2Fqv9XAmy5U/REoBWc6tL0POqzF/63Pfn32Y/FvFiPEwBEDefrppxt4txN2JQ1xxSR6\n7AkNwnVdzKVRAJyAZTfdYOuB750OQz6GA7DCF09FmLtlb0xDvRvmYugEPEO66iOuAw7YCAduDHmX\nCEjHR1oIrIQNR8H/PQp/2GgBJTVlmEa5C97jMGAN9I83ydtorpuDK0zb/ElIeSnWR4LKZzDDvhyC\nS6BqNTh+nNzUFvg55jdfBnqHrhu78tZzb7Eh3GDFL96osnFpNZsM+yfgboBOL8Gx1XA/1bzNH/k9\n1Ti4ZGug4negPrCxAl5/fZNhf+MNqKrKxypT/Rg40X5QIYFuY+LEiXiex/XXX8+hhx5KVBDBN7Dx\nDODpGlKv9sV1oKoKWAUbnoEbf3QjkydP3tbbnPB1ZWun+A39I3HF7Fa8/vrrsZtgiqyW6L31XAJz\nRbs8c12MI45+qa9TMkqQUkBbhQQaXC9E5QqQqSaeZK4d2mpTaOJ6mXBXP5k+jQT/MnfJoHruklOw\n+qG1nw9Cffw+au+1jxN++pl42KA4tv7M2FVCQXxO3xP8Rg7HqzN99W2+rbSbtjDE46ygOBnEMKxA\nd4CgiTyyegG0BpQPghZyaaYsaTUuitsF3RVFJerdO6d+/RylI2Qx7Y1kYZ+1l+Knsnj9w2QhnWnN\nnDlTbs41UbH43KJ2aPRo9NxzpoPfoQPq2LH9rn48ErYj7CJJgYQ9kC5duvDcc4+TlzcPm31vrPfr\nBlv4ew+YD+l0NXA0FtZyHfACWdZzF29zHxtZzab11gmAuUOewEJhPsCqJIFFkeyLzdTvAl4Dqs2F\n/yrwIubheRAor9cdQYFTwFXVVxFoI/BnqKq0oBYXi7zZC2wm3xFzGe2HmMvbvEMxxbg511xAD6Wp\n2XADVEyDBW1gaQQb3wH+TTVTOIh8skAhAfAINd4RrHXW897H2ME2VlFZWcbixRlee81jY2UaK5nX\nHQuheQtbCJ2G+YB+gb1FXMO5506nXYt28BwWGfQReO/DYYdZ5ax0GkaPhlWrVnDttdeSyzmEoUN+\nvsf999+/tbc44evM1o4EDf0jmbHvtpx66qnx7PLmeKaZsxjuELVu31pVVVUaOXKUgqCxUqlGKi0t\nVQ50E2gDaAhoJOiHoA4gH0+WhNRBFv8+Wlb4Y4mgVGVemfy6zE1HbuApbIJojSjGFivTcXLTwSgK\nIv2En+h2bldEoaBGBMWm6jgN8QMsdj6FoI0cZilHY2UpVkSodl47+T1826fTo96suiaeaf89/vy6\nIE9XgTKUCH4iSjPiAuo00/FzIhgo/Ixl2eJoUwGPwfGCbnks1zC33rGel+OUqrKyUo2aNhIewkOZ\nDDrlFIu9f/ZZy3ht3bqFosgyWX/xC3TqqSidRqtXr97Vj0pCA2BXJSht0YESw75bM2nSJIVhY0VR\nY5111ll131dUVOjyyy/XBRdcoNdff12OUyA4QPAtpUGDQO0dFHgocFBQiiIH4RcInhZ8N3afmL77\nWI7SAhboMR5TGWU6n/PVmQ7a33XUJUReYyyjdpxFw7iBq33ZT2dxloooEhwtS1K6SgS+vM6eKI0N\n+4WI/FCN8fRH0Jug7qCo1LU2bRFBU20qcbdKFuHzr/jz5YJ85QjtOMEx4uD/cRGlXNErtAifY4ij\nhL4jeEDQJTbwcbUrOsi08N8XDFKzZh0kSUuXLrXBk0jgK4os0qZJE5TLuRo/frzat+dTWbQFBeju\nu+/eVY9HwjaQGPaErxTLly+Xm47L2hXX+qK7aVO1pN8JXLkBooPNKi920E9AOJEsq7RacJQo8OTh\naQEL6v6GMUxTmarbuE1NSesakFce+82nxVmfIfIoUcB4mZTAYJneektBI7m4ZlzPsW0yndEt9Xz+\nCzH9eFxHIZFomhHOIME1CsPOcpycTJisXeyfR/hpQZGJjXVjU13SA60//CD+3Cc27J4jJ3DlMV4m\nlTBTEZEGcYD8WEPHdfLqZtx2zAJZicJmghYqKirRlClTtHr1arVr10Gum69GjfJ08smeHn3UpBFe\neumlXfxEJDSExLAn7HCWLVumadOmad68eV/atnXb1uZ+uCQ2boMQYdN67oVKgavzz0dRBu0Nuhk0\nFUcjIZ65tjQj6aMoivQDfqAFLNA85qmUUs1hjn7ID5XzMsp5yAlc08LpFbtjOiOYJnhC0FNWhEOC\nv8nquUbmgumFGIrCVuiCeob9TlAv2uo5ntNJnKSIKJ5NhwJXjuPq299GjoPShWjgYNRnb+SXxQNZ\nCaIVJuzlhIJ80ToUh8Wun7Ps2rgHuHICT9BULoU6mZO1gAV6nuc1i1nKkdOyZcvUuHGLeKb+/+Iu\nfiAok+8XSpJOOukkWZWo+YI/yPM6KZv11KpV4x39aCTsIBpi2JM49oQt5oYbbuC7U79rGfRroai4\niA+Wf4DneZ/bPl2UZv3g9dAn/uJfwL0OrP810Bu4GLifVGo1xaXw8fIMojcVjCDFbaznIyxOvgzf\nP54qlhJWheQ5WVZrDZ3pwgAG8HN+zlp3HXgObFxEwCBa0oK/+X+zdcnXe0FNSyzw+5dxZ2qwePVD\nsYXaEkyv/HFSvMVxiEJgDi7XMJOudKWaaoZzGBaSOAvLpu1HJvMeFRUFkK6Bzp8QvWUmd4OPJcHO\nA5ZngNGYnMF9wKPQZ4MdHmADcDWgNOByFKM5kzMBeIVXuIRLWIuLpd8ujY9dy774/p8IwyIqKtYB\n3wLHtdgcegKnIH20dTc74StDQ+LYkxl7whZTmz3JNEwdsgCNHz/+C9t36tLJZqsXx7P2fuZyIT8r\nq25UoJdfflm33HJLXDqubb0Z9Yp4Rp0RdIp9yqhTJzR3LjrnHBT6zeRxjuCF2GWTFdwquE8RaRVQ\nELt/onhfxfHM/QPBZEEPmUZ6SptK5q2P3SFePCv3Ncwdpju5U0dzdPzd87JF4jtkGastZesBDwgn\nT3RFXojJDOdh6o5Oo7iPiv9bZlLEF8fX8wSs1B9/FYyWS6STOdmyV0PXFktpK1OEbCp4JN7Xn+Jr\nlJJlzXYTkW8lBXv7wrcqVwlfX0hcMQk7Epx4gbF2MbA/6tKlyxe2r9MRj7C47zxM9XF/c1O8/fbb\ndW1HjhwZ+79Vz/jly9Qj/yCYLciquNjV00+jO+5AYZhfzyC/JoukqTVygcBK8aVSgSxqZ4GsEEdh\n7Js+Ih4w4kgZNgpulk+j2LjfK9xvx/tLCY4yd47viK4p0SYbb1u/6McNojhlA0pBHKkzCeH9r2Fv\nazrzhfGirE/ct3MFt8ij0Nw+PqbGObr2d8nkEJrF5xTJFnDPFvxOhM3FifXuUS+Uy+W0fPlyzZgx\nQ0899dQ2PwerV6/WX/7yl822qaio0NChQ9WuXTudfPLJ23zMPZmGGPYkjj1hywmx0HGw8nl/ZbOl\n2goKCli3ah0P3fsQl33vMtJeGh6GcEnIE488QZs2beraTp06FVNcvBt4B8vBz2BSBB2AW4BK1q11\nmDgRHn0UHKcKC0DfD7wBMKYav1MFrZ31FLERBkJ1dZr160/D0kZXYTHwv4v//RT0WQOFG4DpZBjB\nAM5iOu/ThTVEPAs1V2DZtWlgPjgfwBjBUevhpLWQqsHEzWr5BD5yrVstgGJMsaB4Debq+SUWrZ+G\nqsa4n7hWnq8qBI7HguW/z6EcSC96mShaV8yNFf4DnKmY32Y/oIrrrpuOBeMvAk4G3t+UEYt13XEd\nmrdpznmXn8eoQ0bRtHXTzd7mzdG37z7k5RXTqVN3HKeAX//6159ps2HDBgrKCnj+jef5W+O/cfsD\nt9OuU7sGHzOhAWztSNDQP5IZ+9ee66+/3maiRRbdUdKkRFVVVdtt/9OnT49n6RmbndNMLqFcQh3A\nSIWEGspQHcIhsUukl+A40da1RchpiO8Sz2A9WaWkA+JZ7o3x9yWqXfjEdy3s8BxEaVrlcVy9QB+D\n0kSC42VFvKsEawR7i32cTTPi3sSz9tmCKwT58ojUj35qTGNzoYzHtOWDePHUCa24N/PlOGmF4VDZ\nAu8p8ZtHR7WmvTwnDsW8BNESE1brlBGlBaJlSoSmjb9J332j8FqIZvFbwvHY21JYz4V2vrmITj/9\n9M3ei9WrV6u8vK0s+qZI48aN0wUXXCBzZy2L33Aukut+1s0zefJke1u5OI5MOhzhoHfeeWe7PSt7\nEiSumIQdzdtvv63p06froYce2u77vvXWWxWSVU/6KaJUHjn1YbAe5EHdx30KCTWRicr6WStK0Sy0\nykLNYxfReYjOtZWd1gh+IiuFV+smeVPgWKQOLUQ4UBzgWsTOOLRXvWiYGlAJKWvHS/X28TORzVis\n/FmIfCd207Sx8MYQBSWBogj16YMaNUJRGuEEZuzOiyOEBvoi6Bgbzk8EM+JBrVyWpHS1LDErX7R0\n7By7xUb+IhR0QCefiqZNq3VJrYv7d43w3Lo6tmPHjhXU8+XHg1HPnj03ey8aNWpjgxiLZUVF8lRa\nWiqYVO9arBF4n9n2m9/8pq2tdE2LoLGIBgrSGjt27HZ/ZvYEdrphx/LD38ASuh8G8jfTdkeff8LX\nlJUrVyovr6kgUoq0QgoFvxT8Ww6nqTVd1Y7u8glVRJHSTdM2G83G/vrIEwTCqfV5L4oNz8rYV36x\nGSd/kEkCt0XgKOIAEfhxHVOUBl0PegsLecziyvdzcpzRMjnd3wgOFa4Xx5+7wikU3CZLekqJiShV\niq67zhKD5s9HLVvGbwhD6hnXc+JwSNI2WJCTLX7WT3iqEfSJB5dSEWRFFIgQte+CZs60YxQVBbKs\n11WCrjrggAM+dX0J49l+7aJ3PpowYcJm74kNMq/VM+LTFIYZ2YJzbYLWM4LcZ7adP39+fH3aCSri\ntr8SZLfnY7PHsCsM+3DAjf99DXD1Ztru4NNP+LpSWtpaMEhwkRynQC6DhXO8cFwziB5yyFO7dnvJ\nw7NF2GY2y+YkhF8sixJ5V9A1nqnXGqRBZiy9fNHON833AM0B3Qv6McgHwT2CXytHsXKgAnwFASpv\ngWiDabtHjqwcXka2qNtDFpFTe6zzxCBXroeeempT1ufo0ZYhSnNvU3LS4Yiwle0vlZUthK6zASrc\nX0TthH+KTATsjNjQ9rIZfohS3VCqBI06GAUB8e8pFRQ0+4x77PLLL7dBpNRcM8XlxV/qQrMB8tl6\n53aaWrVqLd8vliVjHSzIfOHCaKNGjWRl/2q3Xy9wtqvrbk9hl7pigMOBuzfz+w49+YSvL1Bgr+xd\nQ5vV+o4ZKu9bIugmooxwUH5+vs0EU64lH52G6OzGRnGAzP87PZ71HiMYpQyB/BAxHNECc8OEqBHo\nQCJlQeZzP0owS/htRRalSKmt09p825fGxvg0hB/KIlJKbBZd93YgwSWil6dUG3TSt9Dzz6N770V5\neSmBp1SQNXdM8ygOQ7zQBokoa4afp6xQyajYN90xEEEgS6Y6QRC/KbTNiLIC0cuSq4LAUY8ePbT3\n3ntr8eLFn3uNly5dqgsuuEB33HHHFt2TMWMOjo371YIzBRnNnz9flZWVmjBhgoYPH77ZJLUpU6bE\n1+efqosWIr8hj8cez6427I8Dx23m9x168glfXyBlRmxabES9sJ4bYKPwO8WGMB3Hcx8r3DGikS/c\ntCz2WzKRsPzY6N4h+LnSdFWqvoTvNFv83Zd95RPIIysf5FJgrpwAOYWOfMeXA5b2X7vdVGJtl0w8\nmBTLYsqfjI9XaG6aNEqlbSbteY4cx1dHeulwjtSJnCgPX7XaNyYRfJt8mppxb1dvYfZiW3TEceLj\n5cfbXShYKPxDRVirozM8/svoiSee+NT1nTt3rvJK8hQVRBo9evQW35fJkyeruLiZmjVr1aAwyU6d\netobSCyHPHfu3K3eR0LDDPuXZp46jvMM0Lj+V4CAiyQ9Ebe5COgjaexm9qNLL7207vOQIUMYUltt\nIGGPpqioiFWdVsFBWFTilR6oEqtmDbjfBGcZVB8MXA9cAFwK7nioeY1NMZgALTEJ4TexMMXJOMFP\nYAioO1bv+VmHR6oe5mqu42VeISSgktuAt/H5IS6yEtnRektQHYeFLD4N7l+gZmMWK8hdDkwHirDi\nIVPBuQn2vh/6YvWpG4HzLxOwcfkbWe5hHSupogYrUHJn3O9XcRlITdNKy1Z1sOpTNwBtgHe6wMaf\nYVmvE7Cs3Y1APrAegr1g40PAXLLZu1iz5l0AvvOd7zBrziy7rjnbpPdevXnl5Ve2/cZtAX/72994\n4403GDZsGOl0+ss3SGDhwoUsXLiw7vNll12GdnbmKVbL60Ug+pJ2O25IS/hac+2115oP+DjEdxBh\nVnB+7HP+jSyj9EnBTJHOCD9PtnBX+9tSbUrcKRJ8W7Zo2lGQthl5iPCRH4a6hmv0LM+qOc3lk45n\nlYEsGiUryImerqkxjonDBUNEl1rXz7n13C+/jl0WZwhGiVRaHItFsQyqPZc/1rX3mGDrBJTF/azd\nz1/leWkFYRz9MgZRjtgvjoTJywlelmW4dlBdVIoT2kLsaEcEjQTz5HllkixkkQCLILoI61fGzqU+\nL7zwgsaMGaNx48bp/fff3xWPQMJmYBcsno7CMj5KtqDtjj37hK81J510kolihSjKRbHrwYsN7aUy\nUasiC+MbiyjME6lcbJBzsRFPCQ4RrBXcrwi0L+hC0DRQClcFFGgsY9WJjiogJavG9LAsEuV94fom\nCBYg9ooXHOtn3PYLBNfUM8h/iF0NrQQ9TRYgskHEyat127xVr/3ZsZa6F7tibhU8ryjqoVGjAqXT\nqLDIs2txaOyauhRRlCdTw7wrHhROV5omCgrT4uy4b/k5QTc1adJGkvTAAw/Yfuq7oVpb32q58cYb\nLbu1LaIRciJni+PNKyoqdOSRR2rQoEG6+eabd8hzkbBrDPtbWJrgK/Hf7M203dHnn7AbsWzZMqUz\ntYlG+bGhd+WEnmmwtK8NF+wjC2t8ValUD7VokZXresrlIjlklKWnQk5XhiL18h0NHGjFKXI+akKB\n8imXy8Wxce8puoUWEggmNXw0drzTYsM4gbgvj8vi2/dS48YtVVxcHL8tPCeTJhgn05DpIegtKw14\nt/DSwouEky9aooKCnNq0KdCECb4uvBC1amUFM9p2wpKfTkYMRAQFgksEGbmEakxTTWWqjnOOU5gK\n4wQoZKGRKV1zzTV69tlnbRA5N+77hYgsyi/atIjpZlx7O6hd3+iEOnToUPf75MmT1bNnT40fP/5T\nES0VFRXyvCKZzMHRgqyOPvronfqM7CnsdMO+VQdKDHtCA3jnnXf0yCOP1GmRV1VV6YILLtDo0aPV\nsmVLwfi62bDvn6b99zfDeMUVxC6LjXWuDodAl1xi+jGNU6iMEt3HfRrOIerJ/kq5OcvYHIMZrp3R\nTQAAGm5JREFUxG/EBm+sGUT6o6A5auShvciXQ77Ky9uosrJSJSVNBdfVm5m/KQsL/Ch+6ygQbqEZ\ndk63hdpjUDqD+vZFI0daFunVV1uI5OOPoyYtqHuL8TxXvh8Ix1dEpDu5c5MuvTNMpBzhnRkf+zY5\nTpF69+4t3LYiCkU6T7j5wom0fPnyuutLhJ1z7Yx+JCooK5AktWzZWRbeeYqgtfLzm9VtN2bMmHjQ\nqr2+vxVkdu7DsYfQEMOeaMUkfKVp2bIlRxxxBLlcDgDP87j22mv51a9+xY9+9CMsL24WsJCqqsX8\n8RWHsePgquvA6pf68Z7aIcS6dRCG8N56+CBcyYRwAv/136UAB1RF8BA4z0B+Nh/+gL2PdgVagfsn\nl97L+1Jc3YJWrMfnE4YNG8Sjjz5KNhsAz2NxBQC/BZpiC6u/BTZCTVeoHo7Pz3FSFfAMrG8Mi1+F\nhQvBcWDpUjOTH34IK1YABwPnQPX+DlVOM1AjAELCumuUVhp3fQlUz4q/2RdpI4tfXww1faAyB+tu\nhprnQP3Za6+9Afj973+Pg2Pdq8YWa/8I/Xv258UXX+Sf//wHlns4AbiCTz5Zw0033QTAP/7xD6At\nsCw+537Aeqqrq7f5nidsB7Z2JGjoH8mMPWEHcP3118v3S+U4JcrLa2zhiH3jhUsyskXWTwTnKwiy\nsbYKynNRAPId5JXGBTqYqjBsJy/wzK3Rkjp/OQG6ndu1gAV6iqdUSKF8fHWms1KkFPmRbPG1t2BE\nfOy9ZGGQx8olrZ+CuuGrI83Ug+7xIiqxuymtFrRSQSqldNqX63ommXBJPTdJlBaMlk9HtaOLZjBD\nZ3O2UkTyXWRhn1WCE0XrjDgkXh+gdiYvwXJByrJDyVh/A1+4jrX1Aq1cuVI333yzoLHSHK0mZDWY\nfKVx1KlTJz333HNKp1FpqcXoR9EIwXly3aJd/TjslpC4YhL2dAoKsgrSKMyLY87DrC2INs6aa8U3\noz7JRdWg5aAmFt+oIDhGcIsZdQerfnScGfUs2U+V5WtPB32P72kBC3QP98jFk0XyPCGYJ8tITSuK\nyuW6vgpC9BCoKl7IzRDoeq5Xa1ornS4UuIpIqYwy2SLw4eaGcbBM2yMQbq2m/B/kcbWy9Faen9NJ\nJ6Eoqi2X54vGKaU7ovwMcQLWgfUM+xJBRiUlzQVnyZKIFseDX4Wgt8aMGaP3339fEKkNoSrijZ8H\nZUDFxaGOOcYSsObPR927I8cJNX/+/F19+3dLGmLYE1dMwteWv//976xZs+ZT361c+TG9u/clwrEq\nQvuuhUuqYNJaaAauA54PU2tM7LYZcFoNODVw5JH3g386jMHCxIcCj4OTDz4b+Dk/ZzWreY7nWM6/\n2Id9qKGGfPLJJw9zCVViLqCplJQ04s9/fgHXreLqH8HkfBMivpGIgziUNaxhBSu4+eYfU1m5jn+9\n/0+GHjMUnPWQ9wsYEPfjSEztV1XA3kB/qpnKWm7HzdYwb16GysqbMMnjcvzV6+n8V4fhFQHfBNLM\nx9wpM4FhQMDKlWuAxcB/MY3hPEyauDsrVqygrKyMli0bsw8bqI0+H4ypNVdWbmDoUHMdBQEMHw6l\npT4jRozYAXc5oUFs7UjQ0D+SGXvCduKll16yItnxTLvjXh0/t10mkzE3ykhMmyVjtUmb5qF58Sy0\nGnQAqKDI0a23Ir/4f8IDm9hC6iLQXqTl4ytNRj6+Tud0RV4kx0NOgEow4TDIU+PGbbV69WpVVlbK\n99HPfmYz3F69bEHU7e6KfOSFn1VHzBXn7NwurdePDlZUOyITu3fmy/M6qlOnUCZAVjsjfyKOnOkX\nu1p8pUmLzk4sh5BSEPRTOr2/HGdfWejkWYLVsryAnGbOnClJmjZtmopAy+Kd3wDKA5WVoWOP3TRj\n79kTde3aeYfe8z0ZEldMwp6An/PF3nHizhQLR4wii3ZJp9GkSZPq2kZRJAJXhObPfuihh+T7KM9H\nYwLUJXYvXH01evBB5EaYZvk0LOwxRMWxYVuKGVdJGjRokLlsTo3bHoYa++h0UON0uu74K1eu1EEH\njVQ6jQYMiP31teGHUxFpdOONN37q/FauXGmG/dtxux+YW8gJHOVAqSBPjRsXKJ321LatH4dB1hr2\nyYK+sX//HZni4zDRNhIOiiJPRx/t6fvfR02bZuV5Qd0AAFkdeeSRn+rLPn37KsBcOjlQKoVuvBG1\nbWuhmUVFds1XrVq1A+/4nk1DDHviikn42lFVWWUFhFxMNaA3dOgAjz0Gl14Kt9/+Ux5++GEA1q9f\njzZUo8oqJDF27FjmzLmdtYKnajzeICIqAteFl1/GUvl/Cu6j4M+GoNoS/98DzgXyCgsBaNKkiQW9\nNI871Rs+ca1c9cZ16zjzzDNxQoeSkhKefPZpDjnkaD7+uDNEmAoAQMr+fcEFUygry3HXXXcBUFxc\nzKBBg+BW4FHqpAl0nlgTQo2zms6dP6ZXr2r+8U4V8ENgBjAXuB2TGpiMySsUAFfC2xG+77P33jVM\nmlTNgQfCD3+4FtfdCHjMm3cP0hoefPDBT13r3738MtOuvJK1pFhDIRs25PHaa3DzzXDxxdC9OzRq\n1IKCgoLteIcTtpXEsCd8/XCB5fG/VwNvweDBkM3CgAHQsyf87Gc/+8LNJ0yYQHWVqKqqoqpqLa1b\nd+eqqxxumgU13YFDoGYtVK2FjRH8zIFWwNJMhj++8QYA3bt3hw+A9fFOP4CaangWcIKAn875KZRk\nzM9fE/HAQw9aGbkNWCpfDfBX4CM45RQ46KC1TJx4Evfffz8AL/7mRQb2Hgh/AfYHjsIGhQwcdNBh\nrFnTl8rKPtx5x1307t0Zz7uSILiAnj07xDt+td4ZLwYcOnfqTBSp7tsoAtPjuZBjjz2ZI488knHj\nxvGNb3yD4cOH8+qrr/Laa69x0UWXASXArdTUzOa223LMmAEPPwyLFsFdd93DnXfeycEHH8xFF10E\nQJfuXXACB8dzyJXk+OCDD7b+Pic0nK2d4jf0j8QVk7CdOP/88y1lv8xRSKjGNFZeFOqaa9Azz6Cm\nTWlQAeV7773XNGFS5lcPs6HatWunwYMH1yVI1aewUeGmLNjQfOwZ4mgcPyO8E2R6N28ISjR8+HDN\nnDkzzhBFBK7OOmuTbvvYsahVqxZ1+3///fet7TBMD2aEReh8Wcp/XUIUw2TyxQUKggI98sgjiiI0\neTKaMQN16hTK98+Ifetpwf6CZiKTNg36gDgRqVzwWD13zyxz33gorzhPXbt2l8k6HChoJN/PiXzE\n5Nit1cbKKCY0DBIfe8Kewvnnn68UKd3FXVrAAs1kpiInVF6er1TKUWVlZYP2W1FRoUceeUSvv/76\nFrWfMmWKhg4dapmqAzCxrTOIDeXyesbwQpWXN5Uk5eXnCaep0ummmjNnk2E/4QTUrFnjT+1/3rx5\nlvYfICftfKn0raX61+rsOCKIhIeat2wuybRh8vIcZbOBPO8iwc0yKYRfxP2sFEFXq7g0CAW5QCaD\n/FC9c/mRcDKmcb8vMtngJfFvHws/EgfVW/g9zQbLhIbREMOeuGISvpasXLmSVrSiBS0A6E530spn\n9epzWL++hGHDDmzQftPpNEcccQRdunTZovY33HADt956qyVfjgICTM038NjkDhGwiKZNywG44/Y7\nQKvYsKGE6dMjXnkFnnkGHnwQvv3tsz+1/2OOOYbqtdVog6ipqOHEE0+s+2369Om0KC2lZePG3Hbb\nbQD06bMP1dUdMD/Rx6D+0Ctg+b+X88Ybb3DOOefw3/9upKqqkOrq24CrMH/SN+K9hlA9BD4GmsPG\nqo04zjrgNOAOYDZwIRxQYesc+wCOD/SIt8+H6sawot5JvA+um5iancrWjgQN/SOZsSdsR+bNm6cU\nKd3LvVrAAs1mtgIKZQWWFwnyvnDbmTNnqnPnzurbt6+WLl26zX2pqqqyGfvp8Qz1olhbhoxMDGyA\nHCf/Uxots2fPFqTlupFyOVRQ4Oqss87a4mOeffbZyoIuB03F6rXedNNN8v3Ggvvrza6fEk0KRA7N\nmTOnbvvKyso42ekxwQGCK2Q1VpeLoEwci6ImlplbUVGhsrIyQb5KyaqthzihXsROlI5DMCV41c47\nsBBNelkk0MUXX7zN13lPhcQVk7AnMaDPAEVEak5zhWTruRP+oS8SpJo4cWLsgz5TJvGb/cJyclvD\nmDFjzD/fCVGA0oVpzZkzR/3799fIkSO3u855medp7ibrrStBTXI55ec3FUzZZNidaaKFxfPXH1gk\nyffLBHfGYZE9ZLrygRw8eQ46xUMOaOXKlZKkffbZR07d+gBxrVYnHsCi+L+hRo06WEuWLNHee++t\nHj166O67796u576nkRj2hD2KxYsX20x5KLFq4iOxr3ewCgqaf+42ZrzqLwSOU/v2HT637dYyZ84c\nDRs2TJMmTWpQ0ebVq1ervEW5SCMn42jKlClf2LbUcfRkPcN+G6g8ivTcc8/FA9cQmWaNlRM8++yz\nP7OPyZMnx22vFVwmSCkLegu0EfTTOCFJkl5++WWLwT8Wk/+NMvEi6gZZUlRas2bNqltknjZtmlq1\naq1u3bppyZIlW30tEjaRGPaEPYoZM2ZYcehpiBMRZXnCzVMUFdfNMv8X03Z/o55hv1KNGn3+ILCz\nKW5cbJWXTsWkggNzr0jS2LFj1ayoSB3attXSpUu1d69eagd6KdZwKaudRbvm+mjUqJEGDRqk6dOn\nf+G1kKSLL75YBQVNVFjYVDNmzFD/Hj0UggrjpKTZs2dLigeBxvG1PgsRFNe7hhL01YQJEyRJRx55\npEyz/gLB2O32VrSnkhj2hD2K119/3WaRp8QG5wwzbpsTo8rLayYYJVgh+JOgRBMnTtyJvf5iCNhU\nDWkaYiBq3769unfsqGag60CHxgZ32bJl6tm5swpBRSDHwwpvT8WKgYRoxowZDerH22+/rUceeUQV\nFRV1382ePdvCQKfGGbluWC/qZ42gRFdddZWdBwUyVc1ao3+0unTpsl2uUUNZtmyZunTprKZNG2nc\nuHG7tC9bS0MM+5cWs94SHMf5Lpb+Virpoy9oo+1xrISE+hxz7DE88PADpq5VAQcOO5Cnn3r6C9v/\n+9//pmPHvlRUrAJ89tmnD7/73f/ttP5uDidyrIJw0/iLh6FLVRf+9sYbLATOTME7DqzZCFEqZPXq\nSgCqq6vxAx++D3Uy7Y9Bz5qevPrqq/97mAZTXF7Mf9f914prv+5DdSG4Y6BmIfA+VRtW43kejpMH\nLMH02gEupnnze/jXv/6x3fqyNdg9b07v3tClCzz6KDRr1pklS97YJf3ZWhzHQVtZzHqbY5Acx2kO\njMBKEiQk7FTun3c/SxcvZcYlM1j8x8WbNeoAzZo1Y+3aFUjrkdZ8ZYw6QN9efeFeYBHwJPAmzJo1\ni2rgBynofjg8/Cu4/wFIpzdw7LHHAlZ8BA/4MN6RgPehrKxsu/bvg39/wMlHnkxvtzdRGB+wZi64\n7/DCgqesH0Aulw+cCfwT+A0wi3Hjjt6ufdkaTj75ZNq2henT4fjjYeZMeP31N3froiDbPGN3HOdB\n4HLgcaBvMmNPSGg4Rx11FE8teIpMmOGeufcwYsQIyqKICjZw18+hpMTa3Xor/Pa3reJKRjBy5Ejm\nvzDfwslXgPOBw4f//pDi4uId1tfXXnuN//znPwwdOrTOqAOsWLGCDh36sGbNx4DH6NGD+dWvfrnD\n+vFlDBo0iFTqJS65xD5XVsLo0bBmTQXpdHrzG38F2OkzdsdxDgX+JWnptuwnISHBePDBB1n94Wre\ne/e9On3zP7z5Jo4Hr7xibaqq4E9/srKBtTz99NNM+/40uqzpwuA2g3nvn+/tUKMO0K1bN0aMGPEp\now5QXl7O6tXvIq1F+mSLjPrhhx9BFDUhm21SV35vSzjzzDMpLU1RVpbm3HPP/dw2Z599Nr/7HTz/\nPCxfDtdeC/n5wdfCqDeUL52xO47zDNC4/lfYy97FwIXACEmrHcf5O9BP0sov2I8uvfTSus9Dhgxh\nyJAh29b7hIQ9hOnTp3PllZfQoQN88AGsW+fz73//t64W7NeZ/fcfwm9/+wZwLZayejlz5vyYiRMn\nbna7iRMn8vOf38rpp0NNDcyZAxMnnsWPf/zjz7T93ve+x+zZP6S6WmSzaV588VU6duy4Q85nW1m4\ncCELFy6s+3zZZZdt9Yy9wa4Yx3G6YWJ2FZixbw78G9hb0vuf0z5xxSQkbAP/93//x+zZsykvL+e6\n664jDMMv3+hrgOMUY/rEB8TffJ9Gje7lvff+udntiosDTj+9ipEj7fMTT8DcuSEffli5I7u702mI\nK8b/8iafj6TXMFWM2oP/Hegj6b8N3WdCQsIXM3jwYAYPHryru7EDEKZJXEuKmpqaLdouCDZ9CgJI\nJo9Ggw375yBs5p6QkJCwxXTp0po33jgeuAkraXId55578Zdu941vHMaPf/wInmeumJ/8BA499Kgd\n3d2vBdsljn2LDpS4YhISEj6H6upq9t57XxYvfhvPg+98Zzw33HDDFm17+OGH88ILTwAwfPg3P1MB\nanegIa6YxLAnJCQkfIXZJQlKCQkJCQlfLRLDnpCQkLCbkRj2hISEhN2MxLAnJCQk7GYkhj0hISFh\nNyMx7AkJCQlbyJ///GeKi1vi+41p2rQ9K1as+PKNdgFJuGNCQkLCFvDRRx9RUtIaGAscCvwM3/89\n69d/8BkhtO1JEu6YkJCQsIO44oorMBWV24EjgF9QVbWRJ598ctd27HNIDHtCQkLCFvBZ/RoHcLZQ\n12bnkrhiEhISEraADz74gEaN2gHHAYcAP8Pz/o/KysQVk5CQkPC1pKysjEWLniU//wlcdwJlZa/y\n9tuv7lCj3lCSGXtCQkLCV5hkxp6QkJCQkBj2hISEhN2NxLAnJCR8LTnjjDNwUg5O4BDmh/zpT3/a\n1V36yrDNPnbHcb4DnAlUAb+SNPUL2iU+9oSEhO3CnXfeyYTTJsBRQBNgAXhvelStrdrVXdvu7NSa\np/EBh2BxP90lVTmOU7ot+0tISEjYEm655RboBHSMvxgN1YurWbNmDblcbld27SvBtrpiJgHXSKoC\nkPThtncpISEhYfOUlJTASqzSMsAqwIV0Or0Le/XVYZtcMY7jLAYeA0YB64DzJb38BW0TV0xCQsJ2\n4aOPPqK0WSkqFzQHXoG+3fry8qLPNT9fa3aIK8ZxnGeAxvW/wsbJi+PtiyQNdBynP/AA0PaL9jVt\n2rS6fw8ZMoQhQ4ZsTV8TEhISACguLubdv7/LoYceyvvL3+fgCQcza9asXd2t7cLChQtZuHDhNu1j\nW2fsvwaulfRC/HkZMEDSys9pm8zYExISEraSXZGg9AtgaHzwjkDweUY9ISEhIWHnsU1RMcAdwO2O\n4ywFKoETt71LCQkJCQnbQqIVk5CQkPAVJtGKSUhISEhIDHtCQkLC7kZi2BMSEhJ2MxLDnpCQkLCb\nkRj2hISEhN2MxLAnJCQk7GYkhj0hISFhNyMx7AkJCQm7GYlhT0hISNjNSAx7QkJCwm5GYtgTEhIS\ndjMSw56QkJCwm5EY9oSEhP/f3v2EaFHHcRx/fyo8VKCRuUFlf5DKOpQSHirKqExPWkjYqShMECE6\nlXRYkA5h5KFiL2XhoZA6aH8OuUYhBEHCZiWabsVabrZ5WIOIyPTTYcaYtn1Wn52ZZ5yf3xcsOzP7\nh8+X3/6+O8888yckJhp7CCEkJhp7CCEkplRjl3SLpM8lfSnpC0m3VRUshBDC9JTdY98I9NteAPQD\nL5aP1E5lHz57tku5vpRrg6jvXFS2sZ8EZubLs4DRkr+vtVL/40q5vpRrg6jvXFT2madPAzskvQQI\nuL18pBBCCGWctrFL2gn0FTcBBp4D7gOesr1d0krgDeD+OoKGEEI4M6UeZi3pmO1ZhfXfbM/s8L3x\nJOsQQpiGbh9mXfZQzKiku23vknQvcLCqYCGEEKanbGNfDbws6XzgT+DJ8pFCCCGUUepQTAghhLNP\nrVeeSlopaa+kE5IWFrZfLekPSUP5x0CdOerSqb78a+slDUvaL2lJUxmrIqlf0uHCmC1tOlMVJC2V\n9K2kg5KeaTpP1SSNSPrq1EWETecpS9JmSWOSvi5su0TSoKQDknZImvR9vjboUF/Xc6/uWwp8AzwI\n7Jrka9/ZXph/rK05R10mrU/SfOBhYD6wDBiQlMJ7DJsKY/ZR02HKknQe8CrwAHAz8IikG5tNVbmT\nwGLbC2wvajpMBd4kG6+iZ4GPbd8AfAKs73mq6kxWH3Q592pt7LYP2B4mO0VyotY3uinqWw5stf23\n7RFgGEhhUrV+zCZYBAzbPmT7OLCVbOxSIhK6J5Ttz4DxCZuXA1vy5S3Aip6GqlCH+qDLudfkgF+T\nv6z4VNKdDeaowxXAT4X10Xxb262TtEfS621+uVswcZwOk8Y4FRnYKWm3pNVNh6nJHNtjALZ/AeY0\nnKcOXc29smfFTHkBk+0POvzYz8Bc2+P5sentkm6y/XvZPFWbZn2tdJqL0QaADbYt6XlgE/BE71OG\nLt1h+4iky8ga/P58rzBlqZ0R0vXcK93YbXd9pWn+snc8Xx6S9D1wPTBUNk/VplMf2R76VYX1K2nB\nfXS6qPU1IIV/aqPA3MJ6K8apG7aP5J+PStpGdvgptcY+JqnP9piky4Ffmw5UJdtHC6tnNPd6eSjm\n32NEkmbnb1wh6TpgHvBDD7PUoXgM7H1glaQZkq4lq6/VZyTkE+aUh4C9TWWp0G5gXn6W1gxgFdnY\nJUHShZIuzpcvApaQxriJ/8+3x/LlR4H3eh2oYv+pbzpzr/Qe+1QkrQBeAWYDH0raY3sZcBewQdJf\nZO/ar7F9rM4sdehUn+19kt4B9gHHgbVu/wUDGyXdSjZeI8CaZuOUZ/uEpHXAINlOzmbb+xuOVaU+\nYFt+O48LgLdsDzacqRRJbwOLgUsl/Uh2u/AXgHclPQ4cIjsjrZU61HdPt3MvLlAKIYTEJHMaVAgh\nhEw09hBCSEw09hBCSEw09hBCSEw09hBCSEw09hBCSEw09hBCSEw09hBCSMw/E0QYx3vHNDMAAAAA\nSUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x28c72ac88>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(X_new[:,0], X_new[:,1], c=colors)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Ensemble Classifiers for Multi-Class Answers <a class=\"anchor\" id=\"4\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def test_sklearn_model(model):\n",
    "    model.fit(X_train, y_train * 4)\n",
    "    print('Model Score:', model.score(X_test, y_test * 4))\n",
    "    table = precision_recall_fscore_support(y_test * 4, model.predict(X_test))\n",
    "    print('Precision:', table[0], np.mean(table[0]))\n",
    "    print('Recall:   ', table[1], np.mean(table[1]))\n",
    "    print('F1 Score: ', table[2], np.mean(table[2]))\n",
    "    print('Support:  ', table[3])\n",
    "    print('Confusion Matrix:')\n",
    "    print(confusion_matrix(y_test * 4, model.predict(X_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1. Random Forest Classifier <a class=\"anchor\" id=\"41\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Score: 0.708354689903\n",
      "Precision: [ 0.88        0.95384615  0.85416667  0.61781609  0.73180077] 0.80752593575\n",
      "Recall:    [ 0.52380952  0.49206349  0.42051282  0.73504274  0.7950052 ] 0.593286754868\n",
      "F1 Score:  [ 0.65671642  0.64921466  0.56357388  0.67135051  0.76209476] 0.660590046253\n",
      "Support:   [ 84 126 195 585 961]\n",
      "Confusion Matrix:\n",
      "[[ 44   0   0  18  22]\n",
      " [  0  62   4  19  41]\n",
      " [  0   0  82  45  68]\n",
      " [  3   1   2 430 149]\n",
      " [  3   2   8 184 764]]\n"
     ]
    }
   ],
   "source": [
    "test_sklearn_model(RandomForestClassifier(n_estimators=1000, max_depth=1000))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Saving Best Model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pickle\n",
    "clf = RandomForestClassifier(n_estimators=1000, max_depth=1000)\n",
    "clf.fit(X_train, y_train * 4)\n",
    "pickle.dump(clf, open('model.pkl', 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2. Bagging Classifier <a class=\"anchor\" id=\"42\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Score: 0.707329574577\n",
      "Precision: [ 0.88        0.96825397  0.82653061  0.61794501  0.73117255] 0.804780426603\n",
      "Recall:    [ 0.52380952  0.48412698  0.41538462  0.72991453  0.79812695] 0.590272520866\n",
      "F1 Score:  [ 0.65671642  0.64550265  0.55290102  0.669279    0.76318408] 0.657516632754\n",
      "Support:   [ 84 126 195 585 961]\n",
      "Confusion Matrix:\n",
      "[[ 44   0   0  21  19]\n",
      " [  0  61   5  17  43]\n",
      " [  0   1  81  44  69]\n",
      " [  3   0   4 427 151]\n",
      " [  3   1   8 182 767]]\n"
     ]
    }
   ],
   "source": [
    "test_sklearn_model(BaggingClassifier(n_estimators=1000))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.3. Ada Boost Classifier <a class=\"anchor\" id=\"43\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Score: 0.443362378268\n",
      "Precision: [ 0.15384615  0.07692308  0.25714286  0.3279483   0.51140834] 0.265453746304\n",
      "Recall:    [ 0.02380952  0.00793651  0.04615385  0.34700855  0.67637877] 0.220257439404\n",
      "F1 Score:  [ 0.04123711  0.01438849  0.07826087  0.3372093   0.58243728] 0.210706610097\n",
      "Support:   [ 84 126 195 585 961]\n",
      "Confusion Matrix:\n",
      "[[  2   0   1  33  48]\n",
      " [  1   1   0  40  84]\n",
      " [  2   1   9  62 121]\n",
      " [  3   1  10 203 368]\n",
      " [  5  10  15 281 650]]\n"
     ]
    }
   ],
   "source": [
    "test_sklearn_model(AdaBoostClassifier(n_estimators=1000, learning_rate=0.9))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.4. Extra Trees Classifier <a class=\"anchor\" id=\"44\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Score: 0.703229113275\n",
      "Precision: [ 0.8627451   0.88732394  0.74545455  0.61730205  0.73481196] 0.769527519502\n",
      "Recall:    [ 0.52380952  0.5         0.42051282  0.71965812  0.79292404] 0.591380900288\n",
      "F1 Score:  [ 0.65185185  0.63959391  0.53770492  0.66456196  0.76276276] 0.651295079731\n",
      "Support:   [ 84 126 195 585 961]\n",
      "Confusion Matrix:\n",
      "[[ 44   0   0  19  21]\n",
      " [  0  63   5  20  38]\n",
      " [  0   1  82  48  64]\n",
      " [  3   1   8 421 152]\n",
      " [  4   6  15 174 762]]\n"
     ]
    }
   ],
   "source": [
    "test_sklearn_model(ExtraTreesClassifier(n_estimators=1000))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.5. Gradient Boosting Classifier <a class=\"anchor\" id=\"45\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Score: 0.691440287032\n",
      "Precision: [ 0.74576271  0.7804878   0.6         0.61158192  0.75259875] 0.698086238049\n",
      "Recall:    [ 0.52380952  0.50793651  0.43076923  0.74017094  0.75338189] 0.591213619309\n",
      "F1 Score:  [ 0.61538462  0.61538462  0.50149254  0.66976025  0.75299012] 0.631002427035\n",
      "Support:   [ 84 126 195 585 961]\n",
      "Confusion Matrix:\n",
      "[[ 44   2   0  15  23]\n",
      " [  0  64   8  21  33]\n",
      " [  1   3  84  49  58]\n",
      " [  7   5  16 433 124]\n",
      " [  7   8  32 190 724]]\n"
     ]
    }
   ],
   "source": [
    "test_sklearn_model(GradientBoostingClassifier(n_estimators=1000, max_depth=1000))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.6. SVM <a class=\"anchor\" id=\"46\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Score: 0.662224500256\n",
      "Precision: [ 0.92        1.          0.96296296  0.60355987  0.66282895] 0.829870356176\n",
      "Recall:    [ 0.27380952  0.3015873   0.26666667  0.63760684  0.83870968] 0.463676001418\n",
      "F1 Score:  [ 0.42201835  0.46341463  0.41767068  0.62011638  0.74046853] 0.532737715182\n",
      "Support:   [ 84 126 195 585 961]\n",
      "Confusion Matrix:\n",
      "[[ 23   0   0  24  37]\n",
      " [  0  38   1  26  61]\n",
      " [  0   0  52  42 101]\n",
      " [  0   0   1 373 211]\n",
      " [  2   0   0 153 806]]\n"
     ]
    }
   ],
   "source": [
    "test_sklearn_model(sklearn.svm.SVC(gamma=10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.7. Decision Tree Classifier <a class=\"anchor\" id=\"47\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Score: 0.657611481292\n",
      "Precision: [ 0.52222222  0.48550725  0.5         0.59071118  0.78356481] 0.576401091806\n",
      "Recall:    [ 0.55952381  0.53174603  0.43589744  0.6957265   0.70447451] 0.585473655723\n",
      "F1 Score:  [ 0.54022989  0.50757576  0.46575342  0.6389325   0.74191781] 0.578881874317\n",
      "Support:   [ 84 126 195 585 961]\n",
      "Confusion Matrix:\n",
      "[[ 47   3   4  18  12]\n",
      " [  2  67   6  23  28]\n",
      " [  5  10  85  52  43]\n",
      " [ 18  27  29 407 104]\n",
      " [ 18  31  46 189 677]]\n"
     ]
    }
   ],
   "source": [
    "test_sklearn_model(sklearn.tree.DecisionTreeClassifier())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.8. K-Neighbours Classifier <a class=\"anchor\" id=\"48\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Score: 0.44951307022\n",
      "Precision: [ 0.29545455  0.2         0.19428571  0.39479905  0.5955335 ] 0.336014562575\n",
      "Recall:    [ 0.1547619   0.12698413  0.17435897  0.57094017  0.49947971] 0.305304977136\n",
      "F1 Score:  [ 0.203125    0.15533981  0.18378378  0.46680643  0.54329372] 0.310469747369\n",
      "Support:   [ 84 126 195 585 961]\n",
      "Confusion Matrix:\n",
      "[[ 13   1   6  33  31]\n",
      " [  2  16  14  41  53]\n",
      " [  1   7  34  84  69]\n",
      " [ 14  22  42 334 173]\n",
      " [ 14  34  79 354 480]]\n"
     ]
    }
   ],
   "source": [
    "test_sklearn_model(KNeighborsClassifier(n_neighbors=5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.9. Radius Neighbours Classifier <a class=\"anchor\" id=\"49\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Score: 0.471040492055\n",
      "Precision: [ 0.54545455  0.26315789  0.26666667  0.36534447  0.51345609] 0.39081593303\n",
      "Recall:    [ 0.07142857  0.03968254  0.04102564  0.2991453   0.75442248] 0.241140905574\n",
      "F1 Score:  [ 0.12631579  0.06896552  0.07111111  0.32894737  0.61104088] 0.241276132555\n",
      "Support:   [ 84 126 195 585 961]\n",
      "Confusion Matrix:\n",
      "[[  6   0   2  21  55]\n",
      " [  0   5   5  29  87]\n",
      " [  0   3   8  42 142]\n",
      " [  3   2   2 175 403]\n",
      " [  2   9  13 212 725]]\n"
     ]
    }
   ],
   "source": [
    "test_sklearn_model(sklearn.neighbors.RadiusNeighborsClassifier(outlier_label=4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Building the Chatbot <a class=\"anchor\" id=\"5\"></a>\n",
    "\n",
    "## 5.1. Applying the Model to All Unlabelled Replies <a class=\"anchor\" id=\"49\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading Merged DF...\n",
      "Loaded Merged DF...\n",
      "Tokenizing...\n",
      "Tokenized. Loading Sent2Vec...\n",
      "Loaded Sent2Vec...\n",
      "Added Sent2Vec. Adding Replies...\n",
      "Added Replies. Loading Sent2Vec...\n",
      "Loaded, Adding Sent2Vec...\n",
      "Added Sent2Vec...\n",
      "Computing Features...\n"
     ]
    },
    {
     "ename": "KeyError",
     "evalue": "('First_Post_sent2vec', 'occurred at index 1')",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m~/anaconda/lib/python3.5/site-packages/pandas/core/indexes/base.py\u001b[0m in \u001b[0;36mget_value\u001b[0;34m(self, series, key)\u001b[0m\n\u001b[1;32m   2565\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2566\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mlibts\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_value_box\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2567\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mIndexError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/tslib.pyx\u001b[0m in \u001b[0;36mpandas._libs.tslib.get_value_box\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/tslib.pyx\u001b[0m in \u001b[0;36mpandas._libs.tslib.get_value_box\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: 'str' object cannot be interpreted as an integer",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-9abb1754b295>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m    132\u001b[0m titles_with_messages['Sim_Title_FP_Reply'] = titles_with_messages.apply(\n\u001b[1;32m    133\u001b[0m     \u001b[0;32mlambda\u001b[0m \u001b[0mrow\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mtext_to_text_similarity\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrow\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'First_Post_sent2vec'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mrow\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'Title_sent2vec'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mrow\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'Reply_sent2vec'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 134\u001b[0;31m     if type(row['Reply_sent2vec']) == type([1]) else np.nan, axis=1)\n\u001b[0m\u001b[1;32m    135\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    136\u001b[0m titles_with_messages['Sim_FP_Reply'] = titles_with_messages.apply(\n",
      "\u001b[0;32m~/anaconda/lib/python3.5/site-packages/pandas/core/frame.py\u001b[0m in \u001b[0;36mapply\u001b[0;34m(self, func, axis, broadcast, raw, reduce, args, **kwds)\u001b[0m\n\u001b[1;32m   4875\u001b[0m                         \u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   4876\u001b[0m                         \u001b[0mreduce\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mreduce\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 4877\u001b[0;31m                         ignore_failures=ignore_failures)\n\u001b[0m\u001b[1;32m   4878\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   4879\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply_broadcast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda/lib/python3.5/site-packages/pandas/core/frame.py\u001b[0m in \u001b[0;36m_apply_standard\u001b[0;34m(self, func, axis, ignore_failures, reduce)\u001b[0m\n\u001b[1;32m   4971\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   4972\u001b[0m                 \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mv\u001b[0m \u001b[0;32min\u001b[0m \u001b[0menumerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mseries_gen\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 4973\u001b[0;31m                     \u001b[0mresults\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mv\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   4974\u001b[0m                     \u001b[0mkeys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   4975\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-1-9abb1754b295>\u001b[0m in \u001b[0;36m<lambda>\u001b[0;34m(row)\u001b[0m\n\u001b[1;32m    132\u001b[0m titles_with_messages['Sim_Title_FP_Reply'] = titles_with_messages.apply(\n\u001b[1;32m    133\u001b[0m     \u001b[0;32mlambda\u001b[0m \u001b[0mrow\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mtext_to_text_similarity\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrow\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'First_Post_sent2vec'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mrow\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'Title_sent2vec'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mrow\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'Reply_sent2vec'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 134\u001b[0;31m     if type(row['Reply_sent2vec']) == type([1]) else np.nan, axis=1)\n\u001b[0m\u001b[1;32m    135\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    136\u001b[0m titles_with_messages['Sim_FP_Reply'] = titles_with_messages.apply(\n",
      "\u001b[0;32m~/anaconda/lib/python3.5/site-packages/pandas/core/series.py\u001b[0m in \u001b[0;36m__getitem__\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m    621\u001b[0m         \u001b[0mkey\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcom\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_apply_if_callable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    622\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 623\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_value\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    624\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    625\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mis_scalar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda/lib/python3.5/site-packages/pandas/core/indexes/base.py\u001b[0m in \u001b[0;36mget_value\u001b[0;34m(self, series, key)\u001b[0m\n\u001b[1;32m   2572\u001b[0m                     \u001b[0;32mraise\u001b[0m \u001b[0mInvalidIndexError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2573\u001b[0m                 \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2574\u001b[0;31m                     \u001b[0;32mraise\u001b[0m \u001b[0me1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2575\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m:\u001b[0m  \u001b[0;31m# pragma: no cover\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2576\u001b[0m                 \u001b[0;32mraise\u001b[0m \u001b[0me1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda/lib/python3.5/site-packages/pandas/core/indexes/base.py\u001b[0m in \u001b[0;36mget_value\u001b[0;34m(self, series, key)\u001b[0m\n\u001b[1;32m   2558\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2559\u001b[0m             return self._engine.get_value(s, k,\n\u001b[0;32m-> 2560\u001b[0;31m                                           tz=getattr(series.dtype, 'tz', None))\n\u001b[0m\u001b[1;32m   2561\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mKeyError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2562\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m0\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minferred_type\u001b[0m \u001b[0;32min\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m'integer'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'boolean'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_value\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_value\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/hashtable_class_helper.pxi\u001b[0m in \u001b[0;36mpandas._libs.hashtable.PyObjectHashTable.get_item\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/hashtable_class_helper.pxi\u001b[0m in \u001b[0;36mpandas._libs.hashtable.PyObjectHashTable.get_item\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mKeyError\u001b[0m: ('First_Post_sent2vec', 'occurred at index 1')"
     ]
    }
   ],
   "source": [
    "import nltk.data\n",
    "import subprocess\n",
    "\n",
    "tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')\n",
    "\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import nltk\n",
    "import gensim\n",
    "\n",
    "print('Loading Merged DF...')\n",
    "\n",
    "df = pd.read_json('merged_forum_threads.json')\n",
    "\n",
    "print('Loaded Merged DF...')\n",
    "\n",
    "titles_with_first_post = df.groupby('Link').first().reset_index()[['Link', 'Title', 'Username', 'Message', 'Replies']]\n",
    "titles_with_first_post.columns = ['Link', 'Title', 'Seeker', 'First_Post', 'Replies']\n",
    "\n",
    "def tokenize_properly(text):\n",
    "    return [sent for sent in tokenizer.tokenize(text.replace('\\n', '. '))\n",
    "            if len(sent.replace('.', '').replace(' ', '')) >= 2]\n",
    "\n",
    "print('Tokenizing...')\n",
    "\n",
    "titles_with_first_post.Title = titles_with_first_post.Title.apply(tokenize_properly)\n",
    "titles_with_first_post.First_Post = titles_with_first_post.First_Post.apply(tokenize_properly)\n",
    "\n",
    "titles_with_first_post['Title_sent_count'] = titles_with_first_post.Title.apply(len)\n",
    "titles_with_first_post['FP_sent_count'] = titles_with_first_post.First_Post.apply(len)\n",
    "\n",
    "print('Tokenized. Loading Sent2Vec...')\n",
    "\n",
    "sent2vecs = [[float(number) for number in line.strip().split(' ')] for line in open('all_title_fp_sent2vec.txt', 'r').readlines()]\n",
    "\n",
    "print('Loaded Sent2Vec...')\n",
    "\n",
    "title_vecs = {}\n",
    "fp_vecs = {}\n",
    "current = 0\n",
    "for index, row in titles_with_first_post.iterrows():\n",
    "    title_sent_count = row['Title_sent_count']\n",
    "    fp_sent_count = row['FP_sent_count']\n",
    "    title_vecs[index] = sent2vecs[current:current+title_sent_count]\n",
    "    current += title_sent_count\n",
    "    fp_vecs[index] = sent2vecs[current:current+fp_sent_count]\n",
    "    current += fp_sent_count\n",
    "\n",
    "titles_with_first_post['Title_sent2vec'] = pd.Series(titles_with_first_post.index.values).map(title_vecs)\n",
    "titles_with_first_post['FP_sent2vec'] = pd.Series(titles_with_first_post.index.values).map(fp_vecs)\n",
    "\n",
    "print('Added Sent2Vec. Adding Replies...')\n",
    "\n",
    "titles_with_messages = pd.merge(titles_with_first_post.drop(['Replies'], axis=1), df[['Username', 'Message', 'Link', 'User Level', 'Posts']], on='Link')\n",
    "titles_with_messages.rename(columns={'Username':'Replier', 'Posts': 'User Posts'}, inplace=True)\n",
    "titles_with_messages = titles_with_messages[titles_with_messages.apply(lambda row: not row['First_Post'] == row['Message'] and not row['Seeker'] == row['Replier'], axis=1)]\n",
    "titles_with_messages.Message = titles_with_messages.Message.apply(tokenize_properly)\n",
    "titles_with_messages.rename(columns={'Message': 'Reply'}, inplace=True)\n",
    "titles_with_messages['Reply_sent_count'] = titles_with_messages.Reply.apply(len)\n",
    "\n",
    "print('Added Replies. Loading Sent2Vec...')\n",
    "\n",
    "sent2vecs = [[float(number) for number in line.strip().split(' ')] for line in open('all_reply_sent2vec.txt', 'r').readlines()]\n",
    "\n",
    "print('Loaded, Adding Sent2Vec...')\n",
    "\n",
    "reply_vecs = []\n",
    "current = 0\n",
    "for index, row in titles_with_messages.iterrows():\n",
    "    reply_sent_count = row['Reply_sent_count']\n",
    "    reply_vecs.append(sent2vecs[current:current+reply_sent_count])\n",
    "    current += reply_sent_count\n",
    "titles_with_messages['Reply_sent2vec'] = reply_vecs\n",
    "\n",
    "print('Added Sent2Vec...')\n",
    "\n",
    "def is_not_null(sent_vec):\n",
    "    for element in sent_vec:\n",
    "        if not element == 0.0:\n",
    "            return True\n",
    "    return False\n",
    "\n",
    "def sent_to_text_similarity(sent_vec, text_vec):\n",
    "    similarities = []\n",
    "    for vec in text_vec:\n",
    "        if is_not_null(vec):\n",
    "            similarities.append(np.dot(sent_vec, vec))\n",
    "    if similarities:\n",
    "        return np.mean(similarities)\n",
    "    else:\n",
    "        return np.nan\n",
    "\n",
    "def text_to_text_similarity(sent_vecs1, sent_vecs2):\n",
    "    similarities = []\n",
    "    for v1 in sent_vecs1:\n",
    "        if is_not_null(v1):\n",
    "            similarity = sent_to_text_similarity(v1, sent_vecs2)\n",
    "            if not np.isnan(similarity):\n",
    "                similarities.append(similarity)\n",
    "    if similarities:\n",
    "        return np.mean(similarities)\n",
    "    else:\n",
    "        return np.nan\n",
    "\n",
    "def text_to_corpus_similarity(text, corpus):\n",
    "    sent_vecs = text_to_sent_vec(text)\n",
    "    corpus_vecs = [text_to_sent_vec(other_text) for other_text in corpus]\n",
    "    max_sim = 0\n",
    "    index = -1\n",
    "    for text_index in range(len(corpus_vecs)):\n",
    "        similarity = text_to_text_similarity(sent_vecs, corpus_vecs[text_index])\n",
    "        if not np.isnan(similarity) and max_sim < similarity:\n",
    "            max_sim = similarity\n",
    "            index = text_index\n",
    "    if index >= 0:\n",
    "        return corpus[index]\n",
    "    else:\n",
    "        return None\n",
    "\n",
    "#print('Saving to JSON...')\n",
    "\n",
    "#titles_with_messages.to_json('titles_with_messages.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing Features...\n",
      "Standardising Features...\n",
      "Predicting with Model...\n",
      "Predicted with Model.\n"
     ]
    }
   ],
   "source": [
    "print('Computing Features...')\n",
    "\n",
    "titles_with_messages['Sim_Title_Reply'] = titles_with_messages.apply(\n",
    "    lambda row: text_to_text_similarity(row['Title_sent2vec'],row['Reply_sent2vec'])\n",
    "    if type(row['Reply_sent2vec']) == type([1]) else np.nan, axis=1)\n",
    "\n",
    "titles_with_messages['Sim_Title_FP_Reply'] = titles_with_messages.apply(\n",
    "    lambda row: text_to_text_similarity(row['FP_sent2vec'] + [row['Title_sent2vec']],row['Reply_sent2vec'])\n",
    "    if type(row['Reply_sent2vec']) == type([1]) else np.nan, axis=1)\n",
    "\n",
    "titles_with_messages['Sim_FP_Reply'] = titles_with_messages.apply(\n",
    "    lambda row: text_to_text_similarity(row['FP_sent2vec'],row['Reply_sent2vec'])\n",
    "    if type(row['Reply_sent2vec']) == type([1]) else np.nan, axis=1)\n",
    "\n",
    "titles_with_messages['Reply_Length'] = titles_with_messages.Reply.apply(lambda message: len(' '.join(message)) if not type(message) == float else 0)\n",
    "\n",
    "titles_with_messages['User Posts'] = titles_with_messages['User Posts'].apply(lambda x: float(x.replace(',','')) if type(x) == str else 1)\n",
    "\n",
    "print('Standardising Features...')\n",
    "\n",
    "user_level_mean = titles_with_messages['User Level'].mean()\n",
    "\n",
    "def standardize(column):\n",
    "    x_mean = column.mean()\n",
    "    x_std = column.std()\n",
    "    return (column.fillna(value=x_mean) - x_mean)/x_std\n",
    "\n",
    "FEATURES = ['Sim_Title_Reply', 'Sim_Title_FP_Reply',\n",
    "            'Sim_FP_Reply', 'Reply_Length', 'Reply_sent_count',\n",
    "            'User Level', 'User Posts']\n",
    "\n",
    "for feature in FEATURES:\n",
    "    titles_with_messages[feature] = standardize(titles_with_messages[feature])\n",
    "\n",
    "print('Predicting with Model...')\n",
    "\n",
    "import pickle\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "clf = pickle.load(open('model.pkl', 'rb'))\n",
    "X = titles_with_messages[FEATURES]\n",
    "titles_with_messages['Usefulness'] = clf.predict(X)\n",
    "\n",
    "print('Predicted with Model.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Link</th>\n",
       "      <th>Title</th>\n",
       "      <th>Seeker</th>\n",
       "      <th>First_Post</th>\n",
       "      <th>Title_sent_count</th>\n",
       "      <th>FP_sent_count</th>\n",
       "      <th>Title_sent2vec</th>\n",
       "      <th>FP_sent2vec</th>\n",
       "      <th>Replier</th>\n",
       "      <th>Reply</th>\n",
       "      <th>User Level</th>\n",
       "      <th>User Posts</th>\n",
       "      <th>Reply_sent_count</th>\n",
       "      <th>Reply_sent2vec</th>\n",
       "      <th>Sim_Title_Reply</th>\n",
       "      <th>Sim_Title_FP_Reply</th>\n",
       "      <th>Sim_FP_Reply</th>\n",
       "      <th>Reply_Length</th>\n",
       "      <th>Usefulness</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>60</td>\n",
       "      <td>[Is this an asperger's trait?]</td>\n",
       "      <td>animallover</td>\n",
       "      <td>[All of you that don't have jobs and the abili...</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>[[-0.026278, -0.018632, -0.010761, 0.020232, -...</td>\n",
       "      <td>[[0.0012997, 0.0078113, -0.002381, 0.0056695, ...</td>\n",
       "      <td>Nuttdan</td>\n",
       "      <td>[I hate mirrors., They remind me how crummy I ...</td>\n",
       "      <td>0.367785</td>\n",
       "      <td>-0.387636</td>\n",
       "      <td>-0.446793</td>\n",
       "      <td>[[0.020335, -0.045285, 0.0068728, 0.0064149, -...</td>\n",
       "      <td>0.111477</td>\n",
       "      <td>1.661545</td>\n",
       "      <td>1.660984</td>\n",
       "      <td>-0.538998</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>60</td>\n",
       "      <td>[Is this an asperger's trait?]</td>\n",
       "      <td>animallover</td>\n",
       "      <td>[All of you that don't have jobs and the abili...</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>[[-0.026278, -0.018632, -0.010761, 0.020232, -...</td>\n",
       "      <td>[[0.0012997, 0.0078113, -0.002381, 0.0056695, ...</td>\n",
       "      <td>Rogue</td>\n",
       "      <td>[Quote:I've found that I tend to have about a ...</td>\n",
       "      <td>-1.661479</td>\n",
       "      <td>-0.410385</td>\n",
       "      <td>-0.446793</td>\n",
       "      <td>[[0.0053239, 0.0092477, -0.012076, -0.0088438,...</td>\n",
       "      <td>1.989944</td>\n",
       "      <td>0.751713</td>\n",
       "      <td>0.117104</td>\n",
       "      <td>-0.264240</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>60</td>\n",
       "      <td>[Is this an asperger's trait?]</td>\n",
       "      <td>animallover</td>\n",
       "      <td>[All of you that don't have jobs and the abili...</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>[[-0.026278, -0.018632, -0.010761, 0.020232, -...</td>\n",
       "      <td>[[0.0012997, 0.0078113, -0.002381, 0.0056695, ...</td>\n",
       "      <td>TyroneShoelaces</td>\n",
       "      <td>[Like you, I have a lot of trouble staying in ...</td>\n",
       "      <td>-1.154163</td>\n",
       "      <td>-0.401160</td>\n",
       "      <td>0.347739</td>\n",
       "      <td>[[0.0101, -0.013583, 0.0044502, -0.02776, 0.01...</td>\n",
       "      <td>0.739014</td>\n",
       "      <td>0.291466</td>\n",
       "      <td>0.026274</td>\n",
       "      <td>0.046737</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>60</td>\n",
       "      <td>[Is this an asperger's trait?]</td>\n",
       "      <td>animallover</td>\n",
       "      <td>[All of you that don't have jobs and the abili...</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>[[-0.026278, -0.018632, -0.010761, 0.020232, -...</td>\n",
       "      <td>[[0.0012997, 0.0078113, -0.002381, 0.0056695, ...</td>\n",
       "      <td>alex</td>\n",
       "      <td>[Rogue wrote:My guess would be that it's not a...</td>\n",
       "      <td>23.704330</td>\n",
       "      <td>0.362878</td>\n",
       "      <td>-0.579215</td>\n",
       "      <td>[[-0.0075652, -0.0032049, -0.0053176, -0.00444...</td>\n",
       "      <td>3.838123</td>\n",
       "      <td>1.246859</td>\n",
       "      <td>0.057298</td>\n",
       "      <td>-0.332929</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>60</td>\n",
       "      <td>[Is this an asperger's trait?]</td>\n",
       "      <td>animallover</td>\n",
       "      <td>[All of you that don't have jobs and the abili...</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>[[-0.026278, -0.018632, -0.010761, 0.020232, -...</td>\n",
       "      <td>[[0.0012997, 0.0078113, -0.002381, 0.0056695, ...</td>\n",
       "      <td>Mich</td>\n",
       "      <td>[animallover wrote:All of you that don't have ...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-0.373173</td>\n",
       "      <td>0.215317</td>\n",
       "      <td>[[0.0012997, 0.0078113, -0.002381, 0.0056695, ...</td>\n",
       "      <td>0.933444</td>\n",
       "      <td>1.159144</td>\n",
       "      <td>0.875974</td>\n",
       "      <td>0.264046</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Link                           Title       Seeker  \\\n",
       "1    60  [Is this an asperger's trait?]  animallover   \n",
       "2    60  [Is this an asperger's trait?]  animallover   \n",
       "3    60  [Is this an asperger's trait?]  animallover   \n",
       "4    60  [Is this an asperger's trait?]  animallover   \n",
       "5    60  [Is this an asperger's trait?]  animallover   \n",
       "\n",
       "                                          First_Post  Title_sent_count  \\\n",
       "1  [All of you that don't have jobs and the abili...                 1   \n",
       "2  [All of you that don't have jobs and the abili...                 1   \n",
       "3  [All of you that don't have jobs and the abili...                 1   \n",
       "4  [All of you that don't have jobs and the abili...                 1   \n",
       "5  [All of you that don't have jobs and the abili...                 1   \n",
       "\n",
       "   FP_sent_count                                     Title_sent2vec  \\\n",
       "1              4  [[-0.026278, -0.018632, -0.010761, 0.020232, -...   \n",
       "2              4  [[-0.026278, -0.018632, -0.010761, 0.020232, -...   \n",
       "3              4  [[-0.026278, -0.018632, -0.010761, 0.020232, -...   \n",
       "4              4  [[-0.026278, -0.018632, -0.010761, 0.020232, -...   \n",
       "5              4  [[-0.026278, -0.018632, -0.010761, 0.020232, -...   \n",
       "\n",
       "                                         FP_sent2vec          Replier  \\\n",
       "1  [[0.0012997, 0.0078113, -0.002381, 0.0056695, ...          Nuttdan   \n",
       "2  [[0.0012997, 0.0078113, -0.002381, 0.0056695, ...            Rogue   \n",
       "3  [[0.0012997, 0.0078113, -0.002381, 0.0056695, ...  TyroneShoelaces   \n",
       "4  [[0.0012997, 0.0078113, -0.002381, 0.0056695, ...             alex   \n",
       "5  [[0.0012997, 0.0078113, -0.002381, 0.0056695, ...             Mich   \n",
       "\n",
       "                                               Reply  User Level  User Posts  \\\n",
       "1  [I hate mirrors., They remind me how crummy I ...    0.367785   -0.387636   \n",
       "2  [Quote:I've found that I tend to have about a ...   -1.661479   -0.410385   \n",
       "3  [Like you, I have a lot of trouble staying in ...   -1.154163   -0.401160   \n",
       "4  [Rogue wrote:My guess would be that it's not a...   23.704330    0.362878   \n",
       "5  [animallover wrote:All of you that don't have ...    0.000000   -0.373173   \n",
       "\n",
       "   Reply_sent_count                                     Reply_sent2vec  \\\n",
       "1         -0.446793  [[0.020335, -0.045285, 0.0068728, 0.0064149, -...   \n",
       "2         -0.446793  [[0.0053239, 0.0092477, -0.012076, -0.0088438,...   \n",
       "3          0.347739  [[0.0101, -0.013583, 0.0044502, -0.02776, 0.01...   \n",
       "4         -0.579215  [[-0.0075652, -0.0032049, -0.0053176, -0.00444...   \n",
       "5          0.215317  [[0.0012997, 0.0078113, -0.002381, 0.0056695, ...   \n",
       "\n",
       "   Sim_Title_Reply  Sim_Title_FP_Reply  Sim_FP_Reply  Reply_Length  Usefulness  \n",
       "1         0.111477            1.661545      1.660984     -0.538998         4.0  \n",
       "2         1.989944            0.751713      0.117104     -0.264240         3.0  \n",
       "3         0.739014            0.291466      0.026274      0.046737         4.0  \n",
       "4         3.838123            1.246859      0.057298     -0.332929         4.0  \n",
       "5         0.933444            1.159144      0.875974      0.264046         4.0  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles_with_messages.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.0    347248\n",
       "3.0    251856\n",
       "1.0      2844\n",
       "2.0       962\n",
       "0.0       275\n",
       "Name: Usefulness, dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles_with_messages.Usefulness.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Link', 'Title', 'Seeker', 'First_Post', 'Title_sent_count',\n",
       "       'FP_sent_count', 'Title_sent2vec', 'FP_sent2vec', 'Replier', 'Reply',\n",
       "       'User Level', 'User Posts', 'Reply_sent_count', 'Reply_sent2vec',\n",
       "       'Sim_Title_Reply', 'Sim_Title_FP_Reply', 'Sim_FP_Reply', 'Reply_Length',\n",
       "       'Usefulness'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles_with_messages.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "603185"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(titles_with_messages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Link</th>\n",
       "      <th>Reply</th>\n",
       "      <th>Usefulness</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>60</td>\n",
       "      <td>[I hate mirrors., They remind me how crummy I ...</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>60</td>\n",
       "      <td>[Quote:I've found that I tend to have about a ...</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>60</td>\n",
       "      <td>[Like you, I have a lot of trouble staying in ...</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>60</td>\n",
       "      <td>[Rogue wrote:My guess would be that it's not a...</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>60</td>\n",
       "      <td>[animallover wrote:All of you that don't have ...</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Link                                              Reply  Usefulness\n",
       "1    60  [I hate mirrors., They remind me how crummy I ...         4.0\n",
       "2    60  [Quote:I've found that I tend to have about a ...         3.0\n",
       "3    60  [Like you, I have a lot of trouble staying in ...         4.0\n",
       "4    60  [Rogue wrote:My guess would be that it's not a...         4.0\n",
       "5    60  [animallover wrote:All of you that don't have ...         4.0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CHOSEN_COLUMNS = ['Link', 'Reply', 'Usefulness']\n",
    "titles_with_messages[CHOSEN_COLUMNS].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "titles_with_messages[CHOSEN_COLUMNS].to_csv('all_replies.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Link</th>\n",
       "      <th>Reply</th>\n",
       "      <th>Usefulness</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>60</td>\n",
       "      <td>[I hate mirrors., They remind me how crummy I ...</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>60</td>\n",
       "      <td>[Quote:I've found that I tend to have about a ...</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>60</td>\n",
       "      <td>[Like you, I have a lot of trouble staying in ...</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>60</td>\n",
       "      <td>[Rogue wrote:My guess would be that it's not a...</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>60</td>\n",
       "      <td>[animallover wrote:All of you that don't have ...</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Link                                              Reply  Usefulness\n",
       "1    60  [I hate mirrors., They remind me how crummy I ...         4.0\n",
       "2    60  [Quote:I've found that I tend to have about a ...         3.0\n",
       "3    60  [Like you, I have a lot of trouble staying in ...         4.0\n",
       "4    60  [Rogue wrote:My guess would be that it's not a...         4.0\n",
       "5    60  [animallover wrote:All of you that don't have ...         4.0"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "msg_df = titles_with_messages[CHOSEN_COLUMNS]\n",
    "msg_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Link</th>\n",
       "      <th>Title</th>\n",
       "      <th>Seeker</th>\n",
       "      <th>First_Post</th>\n",
       "      <th>Replies</th>\n",
       "      <th>Title_sent_count</th>\n",
       "      <th>FP_sent_count</th>\n",
       "      <th>Title_sent2vec</th>\n",
       "      <th>FP_sent2vec</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>60</td>\n",
       "      <td>[Is this an asperger's trait?]</td>\n",
       "      <td>animallover</td>\n",
       "      <td>[All of you that don't have jobs and the abili...</td>\n",
       "      <td>27</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>[[-0.026278, -0.018632, -0.010761, 0.020232, -...</td>\n",
       "      <td>[[0.0012997, 0.0078113, -0.002381, 0.0056695, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>61</td>\n",
       "      <td>[Online Social Skills Education?]</td>\n",
       "      <td>TyroneShoelaces</td>\n",
       "      <td>[Hello Everyone., At present, I am in the proc...</td>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>[[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,...</td>\n",
       "      <td>[[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>63</td>\n",
       "      <td>[Chat?]</td>\n",
       "      <td>IrishWolf</td>\n",
       "      <td>[Are there set times for chat?, It's just ever...</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>[[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,...</td>\n",
       "      <td>[[-0.0080967, -0.046131, 0.014837, -0.018551, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>66</td>\n",
       "      <td>[Do you like Insects?]</td>\n",
       "      <td>TyroneShoelaces</td>\n",
       "      <td>[To tell you the truth - I'm not much an insec...</td>\n",
       "      <td>47</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>[[-0.017121, 0.0075421, -0.021786, 0.016274, 0...</td>\n",
       "      <td>[[-0.011399, 0.002861, 0.0030487, 0.010823, -0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>99</td>\n",
       "      <td>[Asperger's?, Only in my vocabulary for 3 days...</td>\n",
       "      <td>focused</td>\n",
       "      <td>[I am 28 years old and just got the news., How...</td>\n",
       "      <td>25</td>\n",
       "      <td>3</td>\n",
       "      <td>18</td>\n",
       "      <td>[[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,...</td>\n",
       "      <td>[[-0.011724, -0.016215, -0.012698, -0.0047712,...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Link                                              Title           Seeker  \\\n",
       "0    60                     [Is this an asperger's trait?]      animallover   \n",
       "1    61                  [Online Social Skills Education?]  TyroneShoelaces   \n",
       "2    63                                            [Chat?]        IrishWolf   \n",
       "3    66                             [Do you like Insects?]  TyroneShoelaces   \n",
       "4    99  [Asperger's?, Only in my vocabulary for 3 days...          focused   \n",
       "\n",
       "                                          First_Post  Replies  \\\n",
       "0  [All of you that don't have jobs and the abili...       27   \n",
       "1  [Hello Everyone., At present, I am in the proc...       22   \n",
       "2  [Are there set times for chat?, It's just ever...        8   \n",
       "3  [To tell you the truth - I'm not much an insec...       47   \n",
       "4  [I am 28 years old and just got the news., How...       25   \n",
       "\n",
       "   Title_sent_count  FP_sent_count  \\\n",
       "0                 1              4   \n",
       "1                 1              8   \n",
       "2                 1              2   \n",
       "3                 1              2   \n",
       "4                 3             18   \n",
       "\n",
       "                                      Title_sent2vec  \\\n",
       "0  [[-0.026278, -0.018632, -0.010761, 0.020232, -...   \n",
       "1  [[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,...   \n",
       "2  [[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,...   \n",
       "3  [[-0.017121, 0.0075421, -0.021786, 0.016274, 0...   \n",
       "4  [[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,...   \n",
       "\n",
       "                                         FP_sent2vec  \n",
       "0  [[0.0012997, 0.0078113, -0.002381, 0.0056695, ...  \n",
       "1  [[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,...  \n",
       "2  [[-0.0080967, -0.046131, 0.014837, -0.018551, ...  \n",
       "3  [[-0.011399, 0.002861, 0.0030487, 0.010823, -0...  \n",
       "4  [[-0.011724, -0.016215, -0.012698, -0.0047712,...  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titles_with_first_post.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "titles_with_first_post[['Link', 'Title', 'First_Post']].to_csv('all_titles_fp.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Link</th>\n",
       "      <th>Title</th>\n",
       "      <th>First_Post</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>60</td>\n",
       "      <td>[Is this an asperger's trait?]</td>\n",
       "      <td>[All of you that don't have jobs and the abili...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>61</td>\n",
       "      <td>[Online Social Skills Education?]</td>\n",
       "      <td>[Hello Everyone., At present, I am in the proc...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>63</td>\n",
       "      <td>[Chat?]</td>\n",
       "      <td>[Are there set times for chat?, It's just ever...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>66</td>\n",
       "      <td>[Do you like Insects?]</td>\n",
       "      <td>[To tell you the truth - I'm not much an insec...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>99</td>\n",
       "      <td>[Asperger's?, Only in my vocabulary for 3 days...</td>\n",
       "      <td>[I am 28 years old and just got the news., How...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Link                                              Title  \\\n",
       "0    60                     [Is this an asperger's trait?]   \n",
       "1    61                  [Online Social Skills Education?]   \n",
       "2    63                                            [Chat?]   \n",
       "3    66                             [Do you like Insects?]   \n",
       "4    99  [Asperger's?, Only in my vocabulary for 3 days...   \n",
       "\n",
       "                                          First_Post  \n",
       "0  [All of you that don't have jobs and the abili...  \n",
       "1  [Hello Everyone., At present, I am in the proc...  \n",
       "2  [Are there set times for chat?, It's just ever...  \n",
       "3  [To tell you the truth - I'm not much an insec...  \n",
       "4  [I am 28 years old and just got the news., How...  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfp_df = titles_with_first_post[['Link', 'Title', 'First_Post']]\n",
    "tfp_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.2. Precomputing word2vec <a class=\"anchor\" id=\"52\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import gensim\n",
    "model = gensim.models.KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)\n",
    "from nltk.corpus import stopwords\n",
    "stopwords = set(nltk.corpus.stopwords.words('english'))\n",
    "\n",
    "def get_sentence_vector(sentence):\n",
    "    tokens = [token for token in nltk.word_tokenize(sentence) if token not in stopwords]\n",
    "    vectors = []\n",
    "    for token in tokens:\n",
    "        try:\n",
    "            word_vec = model.wv[token]\n",
    "            vectors.append(word_vec)\n",
    "        except:\n",
    "            pass\n",
    "    if vectors:\n",
    "        return np.mean(vectors, axis=0)\n",
    "    else:\n",
    "        return []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/Khalil/anaconda/lib/python3.5/site-packages/ipykernel/__main__.py:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  if __name__ == '__main__':\n",
      "/Users/Khalil/anaconda/lib/python3.5/site-packages/ipykernel/__main__.py:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  from ipykernel import kernelapp as app\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Link</th>\n",
       "      <th>Title</th>\n",
       "      <th>First_Post</th>\n",
       "      <th>Title_word2vec</th>\n",
       "      <th>First_Post_word2vec</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>60</td>\n",
       "      <td>[Is this an asperger's trait?]</td>\n",
       "      <td>[All of you that don't have jobs and the abili...</td>\n",
       "      <td>[[0.121663, 0.0303548, -0.0671387, 0.249674, -...</td>\n",
       "      <td>[[0.064051, 0.074655, -0.0577895, 0.138198, -0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>61</td>\n",
       "      <td>[Online Social Skills Education?]</td>\n",
       "      <td>[Hello Everyone., At present, I am in the proc...</td>\n",
       "      <td>[[0.0338745, -0.101135, 0.145386, -0.20874, 0....</td>\n",
       "      <td>[[-0.0499268, 0.0224609, -0.0239868, 0.0745239...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>63</td>\n",
       "      <td>[Chat?]</td>\n",
       "      <td>[Are there set times for chat?, It's just ever...</td>\n",
       "      <td>[[-0.0854492, 0.123047, -0.0412598, 0.145508, ...</td>\n",
       "      <td>[[0.0261917, 0.158264, -0.0190582, 0.170044, 0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>66</td>\n",
       "      <td>[Do you like Insects?]</td>\n",
       "      <td>[To tell you the truth - I'm not much an insec...</td>\n",
       "      <td>[[0.191081, 0.122233, 0.00389099, 0.156097, -0...</td>\n",
       "      <td>[[0.120117, 0.0454521, 0.0491943, 0.0600586, -...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>99</td>\n",
       "      <td>[Asperger's?, Only in my vocabulary for 3 days...</td>\n",
       "      <td>[I am 28 years old and just got the news., How...</td>\n",
       "      <td>[[0.170898, -0.283203, -0.296875, 0.392578, -0...</td>\n",
       "      <td>[[0.0266602, 0.0945496, -0.00478516, -0.001123...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Link                                              Title  \\\n",
       "0    60                     [Is this an asperger's trait?]   \n",
       "1    61                  [Online Social Skills Education?]   \n",
       "2    63                                            [Chat?]   \n",
       "3    66                             [Do you like Insects?]   \n",
       "4    99  [Asperger's?, Only in my vocabulary for 3 days...   \n",
       "\n",
       "                                          First_Post  \\\n",
       "0  [All of you that don't have jobs and the abili...   \n",
       "1  [Hello Everyone., At present, I am in the proc...   \n",
       "2  [Are there set times for chat?, It's just ever...   \n",
       "3  [To tell you the truth - I'm not much an insec...   \n",
       "4  [I am 28 years old and just got the news., How...   \n",
       "\n",
       "                                      Title_word2vec  \\\n",
       "0  [[0.121663, 0.0303548, -0.0671387, 0.249674, -...   \n",
       "1  [[0.0338745, -0.101135, 0.145386, -0.20874, 0....   \n",
       "2  [[-0.0854492, 0.123047, -0.0412598, 0.145508, ...   \n",
       "3  [[0.191081, 0.122233, 0.00389099, 0.156097, -0...   \n",
       "4  [[0.170898, -0.283203, -0.296875, 0.392578, -0...   \n",
       "\n",
       "                                 First_Post_word2vec  \n",
       "0  [[0.064051, 0.074655, -0.0577895, 0.138198, -0...  \n",
       "1  [[-0.0499268, 0.0224609, -0.0239868, 0.0745239...  \n",
       "2  [[0.0261917, 0.158264, -0.0190582, 0.170044, 0...  \n",
       "3  [[0.120117, 0.0454521, 0.0491943, 0.0600586, -...  \n",
       "4  [[0.0266602, 0.0945496, -0.00478516, -0.001123...  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfp_df['Title_word2vec'] = tfp_df.Title.apply(lambda sents: [get_sentence_vector(sent) for sent in sents])\n",
    "tfp_df['First_Post_word2vec'] = tfp_df.First_Post.apply(lambda sents: [get_sentence_vector(sent) for sent in sents])\n",
    "tfp_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/Khalil/anaconda/lib/python3.5/site-packages/ipykernel/__main__.py:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  if __name__ == '__main__':\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Link</th>\n",
       "      <th>Reply</th>\n",
       "      <th>Usefulness</th>\n",
       "      <th>Reply_word2vec</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>60</td>\n",
       "      <td>[I hate mirrors., They remind me how crummy I ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>[[0.136719, 0.0665182, 0.149251, 0.098348, -0....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>60</td>\n",
       "      <td>[Quote:I've found that I tend to have about a ...</td>\n",
       "      <td>3.0</td>\n",
       "      <td>[[0.0297873, 0.0276773, 0.00529322, 0.110268, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>60</td>\n",
       "      <td>[Like you, I have a lot of trouble staying in ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>[[0.0877686, 0.0608025, -0.000738144, 0.129211...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>60</td>\n",
       "      <td>[Rogue wrote:My guess would be that it's not a...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>[[0.0807059, 0.012704, 0.0712891, 0.11363, -0....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>60</td>\n",
       "      <td>[animallover wrote:All of you that don't have ...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>[[0.065701, 0.0700633, -0.0568848, 0.126912, -...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Link                                              Reply  Usefulness  \\\n",
       "1    60  [I hate mirrors., They remind me how crummy I ...         4.0   \n",
       "2    60  [Quote:I've found that I tend to have about a ...         3.0   \n",
       "3    60  [Like you, I have a lot of trouble staying in ...         4.0   \n",
       "4    60  [Rogue wrote:My guess would be that it's not a...         4.0   \n",
       "5    60  [animallover wrote:All of you that don't have ...         4.0   \n",
       "\n",
       "                                      Reply_word2vec  \n",
       "1  [[0.136719, 0.0665182, 0.149251, 0.098348, -0....  \n",
       "2  [[0.0297873, 0.0276773, 0.00529322, 0.110268, ...  \n",
       "3  [[0.0877686, 0.0608025, -0.000738144, 0.129211...  \n",
       "4  [[0.0807059, 0.012704, 0.0712891, 0.11363, -0....  \n",
       "5  [[0.065701, 0.0700633, -0.0568848, 0.126912, -...  "
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "msg_df['Reply_word2vec'] = msg_df.Reply.apply(lambda sents: [get_sentence_vector(sent) for sent in sents])\n",
    "msg_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.3. Dot Product Similarity Functions <a class=\"anchor\" id=\"53\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\gensim\\utils.py:860: UserWarning: detected Windows; aliasing chunkize to chunkize_serial\n",
      "  warnings.warn(\"detected Windows; aliasing chunkize to chunkize_serial\")\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from os import listdir\n",
    "import gensim\n",
    "import numpy as np\n",
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "import ast\n",
    "stopwords = set(nltk.corpus.stopwords.words('english'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = gensim.models.KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Link</th>\n",
       "      <th>Title</th>\n",
       "      <th>First_Post</th>\n",
       "      <th>Title_word2vec</th>\n",
       "      <th>First_Post_word2vec</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>60</td>\n",
       "      <td>Is this an asperger's trait?</td>\n",
       "      <td>[All of you that don't have jobs and the abili...</td>\n",
       "      <td>[0.121663411458, 0.0303548177083, -0.067138671...</td>\n",
       "      <td>[[0.0640510110294, 0.074654971852, -0.05778952...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>61</td>\n",
       "      <td>Online Social Skills Education?</td>\n",
       "      <td>[Hello Everyone., At present, I am in the proc...</td>\n",
       "      <td>[0.0338745117188, -0.101135253906, 0.145385742...</td>\n",
       "      <td>[[-0.0499267578125, 0.0224609375, -0.023986816...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>63</td>\n",
       "      <td>Chat?</td>\n",
       "      <td>[Are there set times for chat?, It's just ever...</td>\n",
       "      <td>[-0.08544921875, 0.123046875, -0.041259765625,...</td>\n",
       "      <td>[[0.0261917114258, 0.158264160156, -0.01905822...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>66</td>\n",
       "      <td>Do you like Insects?</td>\n",
       "      <td>[To tell you the truth - I'm not much an insec...</td>\n",
       "      <td>[0.191080729167, 0.122233072917, 0.00389099121...</td>\n",
       "      <td>[[0.1201171875, 0.0454521179199, 0.04919433593...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>99</td>\n",
       "      <td>Asperger's? Only in my vocabulary for 3 days. ...</td>\n",
       "      <td>[I am 28 years old and just got the news., How...</td>\n",
       "      <td>[0.0485432942708, 0.0144856770833, 0.053985595...</td>\n",
       "      <td>[[0.02666015625, 0.0945495605469, -0.004785156...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Link                                              Title  \\\n",
       "0    60                       Is this an asperger's trait?   \n",
       "1    61                    Online Social Skills Education?   \n",
       "2    63                                              Chat?   \n",
       "3    66                               Do you like Insects?   \n",
       "4    99  Asperger's? Only in my vocabulary for 3 days. ...   \n",
       "\n",
       "                                          First_Post  \\\n",
       "0  [All of you that don't have jobs and the abili...   \n",
       "1  [Hello Everyone., At present, I am in the proc...   \n",
       "2  [Are there set times for chat?, It's just ever...   \n",
       "3  [To tell you the truth - I'm not much an insec...   \n",
       "4  [I am 28 years old and just got the news., How...   \n",
       "\n",
       "                                      Title_word2vec  \\\n",
       "0  [0.121663411458, 0.0303548177083, -0.067138671...   \n",
       "1  [0.0338745117188, -0.101135253906, 0.145385742...   \n",
       "2  [-0.08544921875, 0.123046875, -0.041259765625,...   \n",
       "3  [0.191080729167, 0.122233072917, 0.00389099121...   \n",
       "4  [0.0485432942708, 0.0144856770833, 0.053985595...   \n",
       "\n",
       "                                 First_Post_word2vec  \n",
       "0  [[0.0640510110294, 0.074654971852, -0.05778952...  \n",
       "1  [[-0.0499267578125, 0.0224609375, -0.023986816...  \n",
       "2  [[0.0261917114258, 0.158264160156, -0.01905822...  \n",
       "3  [[0.1201171875, 0.0454521179199, 0.04919433593...  \n",
       "4  [[0.02666015625, 0.0945495605469, -0.004785156...  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfp_df = pd.read_pickle('tfp_df.pkl')\n",
    "tfp_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def cosine_similarity(vec1, vec2):\n",
    "    return np.dot(vec1, vec2)/(np.linalg.norm(vec1) * np.linalg.norm(vec2))\n",
    "\n",
    "def get_sentence_vector(sentence):\n",
    "    tokens = [token for token in nltk.word_tokenize(sentence) if token not in stopwords]\n",
    "    vectors = []\n",
    "    for token in tokens:\n",
    "        try:\n",
    "            word_vec = model.wv[token]\n",
    "            vectors.append(word_vec)\n",
    "        except:\n",
    "            pass\n",
    "    if vectors:\n",
    "        return np.mean(vectors, axis=0)\n",
    "    else:\n",
    "        return []\n",
    "\n",
    "def is_not_null(sent_vec):\n",
    "    for element in sent_vec:\n",
    "        if not element == 0.0:\n",
    "            return True\n",
    "    return False\n",
    "\n",
    "def sent_to_text_similarity(sent_vec, text_vec):\n",
    "    similarities = []\n",
    "    for vec in text_vec:\n",
    "        if is_not_null(vec):\n",
    "            similarities.append(np.dot(sent_vec, vec)/(np.linalg.norm(sent_vec) * np.linalg.norm(vec)))\n",
    "    if similarities:\n",
    "        return np.mean(similarities)\n",
    "    else:\n",
    "        return np.nan\n",
    "\n",
    "def text_to_text_similarity(sent_vecs1, sent_vecs2):\n",
    "    similarities = []\n",
    "    for v1 in sent_vecs1:\n",
    "        if is_not_null(v1):\n",
    "            similarity = sent_to_text_similarity(v1, sent_vecs2)\n",
    "            if not np.isnan(similarity):\n",
    "                similarities.append(similarity)\n",
    "    if similarities:\n",
    "        return np.mean(similarities)\n",
    "    else:\n",
    "        return np.nan\n",
    "\n",
    "def text_to_corpus_similarity(text, corpus):\n",
    "    sent_vecs = text_to_sent_vec(text)\n",
    "    corpus_vecs = [text_to_sent_vec(other_text) for other_text in corpus]\n",
    "    max_sim = 0\n",
    "    index = -1\n",
    "    for text_index in range(len(corpus_vecs)):\n",
    "        similarity = text_to_text_similarity(sent_vecs, corpus_vecs[text_index])\n",
    "        if not np.isnan(similarity) and max_sim < similarity:\n",
    "            max_sim = similarity\n",
    "            index = text_index\n",
    "    if index >= 0:\n",
    "        return corpus[index]\n",
    "    else:\n",
    "        return None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.4. Chatbot Functions <a class=\"anchor\" id=\"54\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def compute_similarity(row, sent_vec):\n",
    "    title_sim = 0\n",
    "    title_word2vec = row['Title_word2vec']\n",
    "    if len(title_word2vec) > 0:\n",
    "        title_sim = cosine_similarity(sent_vec, title_word2vec)\n",
    "    return title_sim\n",
    "\n",
    "def compute_separate_similarity(row, sent_vecs):\n",
    "    title_sim = 0\n",
    "    title_word2vec = row['Title_word2vec']\n",
    "    if len(title_word2vec) > 0:\n",
    "        title_sim = np.dot(sent_vecs[0], title_word2vec)/(np.linalg.norm(sent_vecs[0])*np.linalg.norm(title_word2vec))\n",
    "    fp_sim = text_to_text_similarity(sent_vecs[1:], row['First_Post_word2vec'])\n",
    "    return title_sim + fp_sim\n",
    "\n",
    "def compute_separate_similarity_no_question(row, sent_vecs):\n",
    "    fp_sim = text_to_text_similarity(sent_vecs, row['First_Post_word2vec'])\n",
    "    return fp_sim\n",
    "\n",
    "def get_most_similar_title(sentences, sent_vecs):\n",
    "    if sentences == 0:\n",
    "        raise ValueError('Write something!')\n",
    "    elif len(sentences) == 1:\n",
    "        title_fp_sim = tfp_df.apply(lambda row: compute_similarity(row, sent_vecs[0]), axis=1)\n",
    "    elif sentences[0].endswith('?'):\n",
    "        title_fp_sim = tfp_df.apply(lambda row: compute_separate_similarity(row, sent_vecs), axis=1)\n",
    "    else:\n",
    "        title_fp_sim = tfp_df.apply(lambda row: compute_separate_similarity_no_question(row, sent_vecs), axis=1)\n",
    "    return tfp_df.loc[title_fp_sim.idxmax()]\n",
    "\n",
    "def get_response_sentences(sentences, sent_vecs, link, max_sentences):\n",
    "    answer_df = pd.read_pickle('msg_df/msg_df_{}.pkl'.format(link))\n",
    "    best_answer = answer_df.loc[answer_df[answer_df.Usefulness == max(answer_df.Usefulness)]\n",
    "                                .Reply_word2vec.apply(lambda other_vecs: \n",
    "                                                      text_to_text_similarity(sent_vecs, other_vecs)).idxmax()]\n",
    "    best_sentence_idx = np.argmax([sent_to_text_similarity(sent_vec, sent_vecs) for sent_vec in best_answer.Reply_word2vec])\n",
    "    reply_sentences = ast.literal_eval(best_answer.Reply)\n",
    "    if max_sentences <= 1:\n",
    "        return reply_sentences[best_sentence_idx]\n",
    "    else:\n",
    "        context_sent_count = int((max_sentences - 1)/2)\n",
    "        sent_count = len(reply_sentences)\n",
    "        lower_bound = best_sentence_idx - context_sent_count\n",
    "        upper_bound = best_sentence_idx + context_sent_count + 1\n",
    "        return ' '.join(reply_sentences[max(0, lower_bound - max(0, upper_bound - sent_count)): \n",
    "                                        min(upper_bound + max(0, 0 - lower_bound) + ((max_sentences - 1) % 2), sent_count)])\n",
    "\n",
    "def chatbot_answer(question, max_sentences=1):\n",
    "    sentences = tokenizer.tokenize(question)\n",
    "    sent_vecs = [get_sentence_vector(sent) for sent in sentences]\n",
    "    most_similar_title = get_most_similar_title(sentences, sent_vecs)\n",
    "    return get_response_sentences(sentences, sent_vecs, most_similar_title.Link, max_sentences)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An example..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"there have been several times when we've seriously considered doing this......the stress of having a child that is out of control one minute, and passive the next is quite taxing...........don't feel bad for doing what you feel is best for your child~ if you truly cannot handle him at home, then the best thing for him would be a different environment that's medically supervised and they have round-the-clock supervision......my heart aches for you.....best of luck\""
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chatbot_answer('My parents beat me! They are abusive to me.', max_sentences=3)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}