Reddit_Machine_Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dCbSWCYSmTRN"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Adrien\\Anaconda3\\lib\\site-packages\\IPython\\core\\interactiveshell.py:3058: DtypeWarning: Columns (0,22,24) have mixed types. Specify dtype option on import or set low_memory=False.\n",
      "  interactivity=interactivity, compiler=compiler, result=result)\n"
     ]
    }
   ],
   "source": [
    "df = pd.read_csv('Reddit_Comments_7M_2019.csv')\n",
    "df = df[df['body'].notnull()]\n",
    "\n",
    "# I sample 200,000 comments for testing out KNN\n",
    "df_sample = df.sample(n=200000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# K-Nearest Neighbors for Classification\n",
    "\n",
    "I will first be using KNN. I expect this method to be slow and innacurate, but it gives us a good baseline. I will use an out of the box model at first and then do some slight tuning, but nothing too intense because I want to focus on other models. KNN is very slow so I am only using 150,000 comments to train and 50,000 to test. If I want to keep my sanity while doing some slight parameter checks this is necessary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.compose import make_column_transformer\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(df_sample[['body']], df_sample['subreddit'], train_size=.75, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because of my abundance of data, I use train_test_split to split my data into training and testing data. I will train only on my training data and calculate my metrics using test data to see how my machine learning methods do."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def machine_learn(model, X_train, y_train, n_unigrams=500, n_bigrams=500):\n",
    "    ct = make_column_transformer(\n",
    "        (TfidfVectorizer(max_features=n_unigrams), 'body'),\n",
    "        (TfidfVectorizer(max_features=n_bigrams, ngram_range=(2,2)), 'body'),\n",
    "        remainder='passthrough'\n",
    "    )\n",
    "\n",
    "    pipeline = make_pipeline(\n",
    "        ct,\n",
    "        StandardScaler(with_mean=False),\n",
    "        model\n",
    "    )\n",
    "\n",
    "    return pipeline.fit(X_train, y_train)\n",
    "\n",
    "def get_scores(y_test, ypred, labels):\n",
    "    print(\"Accuracy:\", accuracy_score(y_test, ypred))\n",
    "    for label in labels:\n",
    "        print(label + ':')\n",
    "        print(\"\\tPrecision for\", label + \":\", precision_score(y_test, ypred, labels=[label], average='micro'))\n",
    "        print(\"\\tRecall for\", label + \":\", recall_score(y_test, ypred, labels=[label], average='micro'))\n",
    "        print(\"\\tF1 Score for\", label + \":\", f1_score(y_test, ypred, labels=[label], average='micro'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I decide to run just a 5 Nearest Neighbors for now. I will tune this parameter later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "pipeline = machine_learn(KNeighborsClassifier(n_neighbors=5, n_jobs=-1), X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "ypred = pipeline.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.2954\n",
      "GlobalOffensive:\n",
      "\tPrecision for GlobalOffensive: 0.15628415300546447\n",
      "\tRecall for GlobalOffensive: 0.28671679197994987\n",
      "\tF1 Score for GlobalOffensive: 0.20229885057471267\n",
      "leagueoflegends:\n",
      "\tPrecision for leagueoflegends: 0.4880888507350574\n",
      "\tRecall for leagueoflegends: 0.4605609558525719\n",
      "\tF1 Score for leagueoflegends: 0.47392550143266476\n",
      "frugalmalefashion:\n",
      "\tPrecision for frugalmalefashion: 0.1261682242990654\n",
      "\tRecall for frugalmalefashion: 0.028784648187633263\n",
      "\tF1 Score for frugalmalefashion: 0.046875\n",
      "cscareerquestions:\n",
      "\tPrecision for cscareerquestions: 0.24752475247524752\n",
      "\tRecall for cscareerquestions: 0.04054054054054054\n",
      "\tF1 Score for cscareerquestions: 0.06967022758941012\n",
      "malefashionadvice:\n",
      "\tPrecision for malefashionadvice: 0.35545023696682465\n",
      "\tRecall for malefashionadvice: 0.09900990099009901\n",
      "\tF1 Score for malefashionadvice: 0.15487867836861124\n",
      "Android:\n",
      "\tPrecision for Android: 0.16845714285714286\n",
      "\tRecall for Android: 0.19533527696793002\n",
      "\tF1 Score for Android: 0.1809032891507118\n",
      "MechanicalKeyboards:\n",
      "\tPrecision for MechanicalKeyboards: 0.19217081850533807\n",
      "\tRecall for MechanicalKeyboards: 0.1524347212420607\n",
      "\tF1 Score for MechanicalKeyboards: 0.17001180637544275\n",
      "hiphopheads:\n",
      "\tPrecision for hiphopheads: 0.2439440464005459\n",
      "\tRecall for hiphopheads: 0.2864583333333333\n",
      "\tF1 Score for hiphopheads: 0.26349732817394506\n",
      "indieheads:\n",
      "\tPrecision for indieheads: 0.06463878326996197\n",
      "\tRecall for indieheads: 0.012762762762762763\n",
      "\tF1 Score for indieheads: 0.021316614420062694\n",
      "Kanye:\n",
      "\tPrecision for Kanye: 0.08108108108108109\n",
      "\tRecall for Kanye: 0.0803727431566686\n",
      "\tF1 Score for Kanye: 0.08072535829189822\n",
      "ProgrammerHumor:\n",
      "\tPrecision for ProgrammerHumor: 0.11913357400722022\n",
      "\tRecall for ProgrammerHumor: 0.09083759339362957\n",
      "\tF1 Score for ProgrammerHumor: 0.10307898259705488\n",
      "fantanoforever:\n",
      "\tPrecision for fantanoforever: 0.07692307692307693\n",
      "\tRecall for fantanoforever: 0.020512820512820513\n",
      "\tF1 Score for fantanoforever: 0.032388663967611336\n",
      "CalPoly:\n",
      "\tPrecision for CalPoly: 0.014084507042253521\n",
      "\tRecall for CalPoly: 0.01282051282051282\n",
      "\tF1 Score for CalPoly: 0.01342281879194631\n"
     ]
    }
   ],
   "source": [
    "get_scores(y_test, ypred, y_test.unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results\n",
    "\n",
    "Predicting with KNN takes a loooong time. This is understandable because for each sample we are trying to predict for, we must compare to all the points to see which one is closest.\n",
    "\n",
    "I printed all the scores above and you can see our model is quite terrible. Our overall accuracy is 29%, which isn't fantastic, but you must remember we are trying to predict 13 class, which is no easy feat. The precision, recall, and f1 scores are also quite terrible for many of the classes (like /r/CalPoly). Let's take a quick look at what happens when I try to predict between two subreddits with low cosine similarity, which we determined in the Data Exploration Notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Binary Classification\n",
    "\n",
    "I want to see how KNN does for binary classification of my data. I want to test with two different pairs of classes. First with **/r/fantanoforever** (a subreddit dedicated to a well known online music reviewer) and **/r/malefashionadvice** (a subreddit dedicated to male fashion.\n",
    "\n",
    "Below is the heatmap from the Data Exploration notebook where I compared the subreddits. As you can see on the heatmap, /r/malefashionadvice is very different from /r/fantanoforever\n",
    "![heatmap](./images/heatmap4.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_scores_binary(y_test, ypred, labels):\n",
    "    print(\"Accuracy:\", accuracy_score(y_test, ypred))\n",
    "    for label in labels:\n",
    "        print(label + ':')\n",
    "        print(\"\\tPrecision for\", label + \":\", precision_score(y_test, ypred, pos_label=label))\n",
    "        print(\"\\tRecall for\", label + \":\", recall_score(y_test, ypred, pos_label=label))\n",
    "        print(\"\\tF1 Score for\", label + \":\", f1_score(y_test, ypred, pos_label=label))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "260040"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "two_subs = df[(df['subreddit'] == 'fantanoforever') | (df['subreddit'] == 'malefashionadvice')]\n",
    "len(two_subs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 260 thousand+ comments from these two subreddits. To speed up the process of training and predicting, I will thin this down to 100 thousand."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_sample = two_subs.sample(n=100000)\n",
    "X_train, X_test, y_train, y_test = train_test_split(df_sample[['body']], df_sample['subreddit'], train_size=.75, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = machine_learn(KNeighborsClassifier(n_neighbors=5, n_jobs=-1), X_train, y_train)\n",
    "ypred = pipeline.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.83916\n",
      "malefashionadvice:\n",
      "\tPrecision for malefashionadvice: 0.9216756056963196\n",
      "\tRecall for malefashionadvice: 0.8954674093706482\n",
      "\tF1 Score for malefashionadvice: 0.9083825104240241\n",
      "fantanoforever:\n",
      "\tPrecision for fantanoforever: 0.3099051008303677\n",
      "\tRecall for fantanoforever: 0.3815261044176707\n",
      "\tF1 Score for fantanoforever: 0.34200621829487804\n"
     ]
    }
   ],
   "source": [
    "get_scores_binary(y_test, ypred, y_test.unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So our overall accuracy is decent at 83.9%. Unforunately, if you look at the recall and precision, this model does terribly with the /r/fantanoforever subreddit. The accuracy is so high because it predicts malefashionadvice very well, which has much more comments overall."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1d0a4621108>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAFNCAYAAADihgPWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAViUlEQVR4nO3df7RdZX3n8ffHRItVEJCADEHij6xaRECIGBczrT9aCGiFdrRKUVJKmy4X/mDGNTMwHScW6xI7bWeKy6GL1kCwVmRGHagG0zRLS6tASQQDiDQpoqQgRMOvkUEFvvPHfq4ew0luSLh3X+5+v9Y665zzPfuc+z2LSz53P89+9k5VIUkatqf13YAkqX+GgSTJMJAkGQaSJAwDSRKGgSQJmNt3A7tqv/32qwULFvTdhiQ9paxfv/67VTVv2/pTNgwWLFjAunXr+m5Dkp5SknxrXN1hIkmSYSBJMgwkSRgGkiQMA0kShoEkCcNAkoRhIEniKbzo7Kliwdmf77uFWeP2817fdwvSrOWegSTJMJAkGQaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIktiJMEhycJIvJrklyc1J3tPq+yZZk2Rju9+n1ZPk/CSbkmxIctTIZy1t229MsnSkfnSSG9t7zk+SqfiykqTxdmbP4BHgvVX188Bi4MwkhwJnA2uraiGwtj0HOAFY2G7LgAugCw9gOfBK4Bhg+USAtG2Wjbxvye5/NUnSzpo0DKrqrqr6anv8IHALcBBwErCybbYSOLk9Pgm4pDrXAHsnORA4HlhTVVur6l5gDbCkvbZXVV1dVQVcMvJZkqRp8ITmDJIsAF4OXAscUFV3QRcYwP5ts4OAO0betrnVdlTfPKYuSZomOx0GSZ4NfBo4q6oe2NGmY2q1C/VxPSxLsi7Jui1btkzWsiRpJ+1UGCR5Ol0QfKKqPtPKd7chHtr9Pa2+GTh45O3zgTsnqc8fU3+cqrqwqhZV1aJ58+btTOuSpJ2wM0cTBfgYcEtV/cnIS1cAE0cELQUuH6mf1o4qWgzc34aRVgPHJdmnTRwfB6xurz2YZHH7WaeNfJYkaRrM3YltjgXeDtyY5IZW+8/AecBlSc4Avg28ub22CjgR2AQ8BJwOUFVbk3wAuK5td25VbW2P3wFcDDwTuLLdJEnTZNIwqKp/YPy4PsDrxmxfwJnb+awVwIox9XXAYZP1IkmaGq5AliQZBpIkw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAksRNhkGRFknuS3DRSe3+Sf0lyQ7udOPLaOUk2Jbk1yfEj9SWttinJ2SP1FyS5NsnGJJ9K8own8wtKkia3M3sGFwNLxtT/e1Ud2W6rAJIcCrwVeGl7z/9MMifJHOCjwAnAocApbVuAD7fPWgjcC5yxO19IkvTETRoGVXUVsHUnP+8k4NKq+kFVfRPYBBzTbpuq6raq+iFwKXBSkgCvBf53e/9K4OQn+B0kSbtpd+YM3plkQxtG2qfVDgLuGNlmc6ttr/5c4L6qemSbuiRpGu1qGFwAvAg4ErgL+ONWz5htaxfqYyVZlmRdknVbtmx5Yh1LkrZrl8Kgqu6uqker6jHgz+mGgaD7y/7gkU3nA3fuoP5dYO8kc7epb+/nXlhVi6pq0bx583aldUnSGLsUBkkOHHn6q8DEkUZXAG9N8jNJXgAsBP4RuA5Y2I4cegbdJPMVVVXAF4E3tfcvBS7flZ4kSbtu7mQbJPkk8GpgvySbgeXAq5McSTekczvwuwBVdXOSy4CvA48AZ1bVo+1z3gmsBuYAK6rq5vYj/hNwaZI/AK4HPvakfTtJ0k6ZNAyq6pQx5e3+g11VHwQ+OKa+Clg1pn4bPxlmkiT1wBXIkiTDQJJkGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEjsRBklWJLknyU0jtX2TrEmysd3v0+pJcn6STUk2JDlq5D1L2/YbkywdqR+d5Mb2nvOT5Mn+kpKkHduZPYOLgSXb1M4G1lbVQmBtew5wArCw3ZYBF0AXHsBy4JXAMcDyiQBp2ywbed+2P0uSNMUmDYOqugrYuk35JGBle7wSOHmkfkl1rgH2TnIgcDywpqq2VtW9wBpgSXttr6q6uqoKuGTksyRJ02RX5wwOqKq7ANr9/q1+EHDHyHabW21H9c1j6pKkafRkTyCPG++vXaiP//BkWZJ1SdZt2bJlF1uUJG1rV8Pg7jbEQ7u/p9U3AwePbDcfuHOS+vwx9bGq6sKqWlRVi+bNm7eLrUuStrWrYXAFMHFE0FLg8pH6ae2oosXA/W0YaTVwXJJ92sTxccDq9tqDSRa3o4hOG/ksSdI0mTvZBkk+Cbwa2C/JZrqjgs4DLktyBvBt4M1t81XAicAm4CHgdICq2prkA8B1bbtzq2piUvoddEcsPRO4st0kSdNo0jCoqlO289LrxmxbwJnb+ZwVwIox9XXAYZP1IUmaOq5AliQZBpIkw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAksZthkOT2JDcmuSHJulbbN8maJBvb/T6tniTnJ9mUZEOSo0Y+Z2nbfmOSpbv3lSRJT9STsWfwmqo6sqoWtednA2uraiGwtj0HOAFY2G7LgAugCw9gOfBK4Bhg+USASJKmx1QME50ErGyPVwInj9Qvqc41wN5JDgSOB9ZU1daquhdYAyyZgr4kSduxu2FQwN8kWZ9kWasdUFV3AbT7/Vv9IOCOkfdubrXt1SVJ02Tubr7/2Kq6M8n+wJok39jBthlTqx3UH/8BXeAsA3j+85//RHuVJG3Hbu0ZVNWd7f4e4LN0Y/53t+Ef2v09bfPNwMEjb58P3LmD+rifd2FVLaqqRfPmzdud1iVJI3Y5DJI8K8meE4+B44CbgCuAiSOClgKXt8dXAKe1o4oWA/e3YaTVwHFJ9mkTx8e1miRpmuzOMNEBwGeTTHzOX1XVF5JcB1yW5Azg28Cb2/argBOBTcBDwOkAVbU1yQeA69p251bV1t3oS5L0BO1yGFTVbcARY+rfA143pl7Amdv5rBXAil3tRZK0e1yBLEkyDCRJhoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJ7N6VziQ9hS04+/N9tzCr3H7e6/tuYbe4ZyBJMgwkSYaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEliBoVBkiVJbk2yKcnZffcjSUMyI8IgyRzgo8AJwKHAKUkO7bcrSRqOGREGwDHApqq6rap+CFwKnNRzT5I0GDMlDA4C7hh5vrnVJEnTYG7fDTQZU6vHbZQsA5a1p/83ya1T2tVw7Ad8t+8mJpMP992BeuLv55PrkHHFmRIGm4GDR57PB+7cdqOquhC4cLqaGook66pqUd99SOP4+zk9Zsow0XXAwiQvSPIM4K3AFT33JEmDMSP2DKrqkSTvBFYDc4AVVXVzz21J0mDMiDAAqKpVwKq++xgoh940k/n7OQ1S9bh5WknSwMyUOQNJUo8MA0mSYSBp5kgyJ8m/67uPITIMBirJzyZ5X5I/b88XJnlD331p2KrqUTwVTS+cQB6oJJ8C1gOnVdVhSZ4JXF1VR/bcmgYuyQeB5wCfAr4/Ua+qr/bW1AAYBgM1saozyfVV9fJW+1pVHdF3bxq2JF8cU66qeu20NzMgM2adgabdD9veQAEkeRHwg35bkqCqXtN3D0PknMFwLQe+AByc5BPAWuA/9tuSBEkOSPKxJFe254cmOaPvvmY7h4kGLMlzgcV0Z429pqpm/JkhNfu1ELgI+L2qOiLJXOD6qnpZz63Nau4ZDFSSXwUeqarPV9XngEeSnNx3XxKwX1VdBjwG3bnLgEf7bWn2MwyGa3lV3T/xpKruoxs6kvr2/bbXOjGftRi4f8dv0e5yAnm4xv0h4O+DZoL30p3C/kVJvgzMA97Ub0uzn3MGA5VkBXAf8FG6v8DeBexTVb/ZZ18SQJsn+Dm6+axbq+pHPbc06zlMNFzvAn5It7DnfwEPA2f22pFEt96F7si2h6vqJoNgerhnIGlGSXII8JZ2e4zuD5bLqurbvTY2yxkGA5Pkf1TVWUn+mjZBN6qq3thDW9JYSRYC7wNOrao5ffczmzlhODwfb/d/1GsX0g4kWQD8Ot3ewaO4IHLKGQYDU1Xr28N9gVVV5SkoNKMkuRZ4Ot1c1pur6raeWxoEh4kGKslFwGuBq4BLgdVtcY/UqyQvqapv9N3H0Hg00UBV1enAi+n++voN4J+T/EW/XUkA3Ou5iaafYTBg7ZC9K+n2DNbjRUU0M1wMrAb+VXv+T8BZvXUzEIbBQCVZkuRiYBPd6s6/AA7stSmp47mJeuAE8nD9Jt0ewe86iawZxnMT9cAJZEkzSpKjgI8AhwE30c5NVFUbem1slnPPYGCSPMiYxWYTqmqvaWxH+ilJngbsAfwinptoWhkGA1NVewIkORf4Dt0itACnAnv22JpEVT2W5I+r6lXAzX33MyQOEw1Ukmur6pWT1aTpluT3gQ3AZ8p/oKaNewbD9WiSU+kmkQs4BY/Y0Mzw74Fn0f2O/j+6PddyCHNquWcwUO3cL38KHEsXBl8Gzqqq2/vrSlJfDANJM06SNwK/0J5+qV2nW1PIMBioJHsAZwAvpTt6A4Cq+q3empKAJOcBrwA+0UqnAOur6uz+upr9XIE8XB8HngccD/wdMB94sNeOpM6JwC9X1YqqWgEsaTVNIcNguF5cVe8Dvl9VK4HXAy/ruSdpwt4jj5/TWxcD4tFEwzWxiOe+JIfRrTlY0F870o99CLg+yRfpjiT6BeCcflua/ZwzGKgkvw18GjgcuAh4NvBfq+rPem1Mg5Xk2Kr6cpKfobv40ivowuDaqvpOv93NfoaBpBkhyfqqOjrJV6vqqL77GRqHiQaq/fX1b+mGhn78e1BV5/bVkwbvR+0KfAclOX/bF6vq3T30NBiGwXBdTnda4PWAp7DWTPAG4JfoLse6fpJt9SRzmGigktxUVYf13Ye0rSRHVNXX+u5jaDy0dLi+ksRDSTUTfS/JZ5Pck+TuJJ9OMr/vpmY79wwGKsnXgRcD36QbJpo4GdjhvTamwUuyBvgruoWRAG8DTq2qX+6vq9nPMBioJIeMq1fVt6a7F2lUkq9V1RHb1G6oqiP76mkIHCYaqPaP/t7Ar7Tb3gaBZogtSd6WZE67vQ34Xt9NzXaGwUAleQ/dicD2b7e/TPKufruSAPgt4NfpVsXfBbyp1TSFHCYaqCQbgFdV1ffb82cBVztnIA2T6wyGK/z0lc0ebTWpV0nmAb/D4xdEuncwhQyD4boIuDbJZ9vzk4GP9diPNOFy4O+Bv8VLsU4bh4kGLMlRwL+m2yO4qqqu77klySOHemIYDEySvarqgST7jnu9qrZOd0/SqCR/AHylqlb13cuQGAYDk+RzVfWGJN8ERv/jTyw6e2FPrUkAJHkQeBbdYsgf8ZPfzb16bWyWMwwkSU4gD1mSg4BD+OkjNq7qryOpk2QfYCGwx0TN382pZRgMVJIPA28Bvs5PjtgowP/h1Kt2Fb73APOBG4DFwNV0p7bWFHGYaKCS3AocXlVey0AzSpIb6S55eU1VHZnkJcDvV9Vbem5tVvN0FMN1G/D0vpuQxni4qh6G7op8VfUN4Od67mnWc5hoYJJ8hG446CHghiRrGbnSmZcW1AywOcnewP8B1iS5F7iz555mPYeJBibJ0h29XlUrp6sXaTJJfhF4DnBlVf2o735mM8NAE0duHFxVG/ruRUry8ap6+2Q1PbmcMxioJF9Ksldbifw14KIkf9J3XxLw0tEnSeYAR/fUy2AYBsP1nKp6APg14KKqOhr4pZ570oAlOaetPj48yQPt9iBwD93J6zSFDIPhmpvkQLqLiHyu72akqvpQVe0J/Leq2qvd9qyq51bVOX33N9t5NNFwnQusBv6hqq5L8kJgY889SVTVOa6On35OIEuaUZKcB7yVbVbHV9Ub++tq9jMMBirJHsAZdJN1o+d/8WpS6pWr4/vhnMFwfRx4HnA88Hd054F5sNeOpI6r43vgnsFAJbm+ql6eZENVHZ7k6cDqqvJkYOpVkk8DRwCujp9GTiAP18RqzvuSHAZ8h+4C5FLfrmg3TSPDYLgubCuP/wvd/3jPBt7Xb0uSp0Tpi8NEA5PkPVX1p0mOraov992PtK0kC4EPAYfy0wc3eEnWKeQE8vCc3u4/0msX0vZdBFwAPAK8BriE7oAHTSH3DAYmySeBVwHzgH8efYnuWO7De2lMapKsr6qjk9xYVS9rtb+vqn/Td2+zmXMGA1NVpyR5Ht3qYxfxaCZ6OMnTgI1J3gn8C7B/zz3Neg4TDVBVfaeqjqA7AdgeVfWtiVvfvWm4kkwMBV0O/Czwbrqzlb4d2OF1OLT7DIOBSvIrdBcb/0J7fmQSD+dTn45OcghwKt2is4eA9wK/DfxTn40NgcNEw/V+4BjgSwBVdUOSBf21I/FndH+cvBBYT5vHGrn3aKIp5J7BcD1SVff33YQ0oarOr6qfB1ZU1Qur6gWj9333N9u5ZzBcNyX5DWBOO6773cBXeu5Joqre0XcPQ+SewXC9i+6MpT8APgk8AJzVa0eSeuM6A0mSw0RDk+Sv6SbjxvICItIwGQbD80d9NyBp5nGYSJLknsFQeWZISaM8mmi4PDOkpB8zDIbrmVW1lm6o8FtV9X7AS15KA+Uw0XB5ZkhJP+YE8kAleQVwC7A38AFgL+APq+raXhuT1AvDYKCSLAJ+DziE7gyR4MVtpMEyDAYqya3AfwBuBB6bqHtNA2mYnDMYri1V5fULJAHuGQxWktcBpwBr6U5WB0BVfaa3piT1xj2D4TodeAndfMHEMFEBhoE0QIbBcB1RVS/ruwlJM4OLzobrmiSH9t2EpJnBOYOBSnIL8CLgm3RzBsFDS6XBMgwGKskh4+oeWioNk2EgSXLOQJJkGEiSMAwkSRgGkiQMA0kS8P8B846MwisTz+EAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Here is the spread\n",
    "y_test.value_counts().plot.bar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's try to see what we can improve about our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "def run_grid_search(model, X_train, y_train, n_unigrams=500, n_bigrams=500):\n",
    "    ct = make_column_transformer(\n",
    "        (TfidfVectorizer(max_features=n_unigrams), 'body'),\n",
    "        (TfidfVectorizer(max_features=n_bigrams, ngram_range=(2,2)), 'body'),\n",
    "        remainder='passthrough'\n",
    "    )\n",
    "\n",
    "    pipeline = make_pipeline(\n",
    "        ct,\n",
    "        StandardScaler(with_mean=False),\n",
    "        model\n",
    "    )\n",
    "    grid_search = GridSearchCV(pipeline,\n",
    "                              param_grid={\n",
    "                                  \"kneighborsclassifier__n_neighbors\": range(5,26)\n",
    "                              },\n",
    "                              scoring='f1_micro', #I use f1 because I want to see how I can improve for all the classes\n",
    "                              cv=4, n_jobs=8)\n",
    "\n",
    "    return grid_search.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "gs = run_grid_search(KNeighborsClassifier(n_jobs=-1), X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('columntransformer',\n",
       "                 ColumnTransformer(n_jobs=None, remainder='passthrough',\n",
       "                                   sparse_threshold=0.3,\n",
       "                                   transformer_weights=None,\n",
       "                                   transformers=[('tfidfvectorizer-1',\n",
       "                                                  TfidfVectorizer(analyzer='word',\n",
       "                                                                  binary=False,\n",
       "                                                                  decode_error='strict',\n",
       "                                                                  dtype=<class 'numpy.float64'>,\n",
       "                                                                  encoding='utf-8',\n",
       "                                                                  input='content',\n",
       "                                                                  lowercase=True,\n",
       "                                                                  max_df=1.0,\n",
       "                                                                  max_feature...\n",
       "                                                                  token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                                                                  tokenizer=None,\n",
       "                                                                  use_idf=True,\n",
       "                                                                  vocabulary=None),\n",
       "                                                  'body')],\n",
       "                                   verbose=False)),\n",
       "                ('standardscaler',\n",
       "                 StandardScaler(copy=True, with_mean=False, with_std=True)),\n",
       "                ('kneighborsclassifier',\n",
       "                 KNeighborsClassifier(algorithm='auto', leaf_size=30,\n",
       "                                      metric='minkowski', metric_params=None,\n",
       "                                      n_jobs=-1, n_neighbors=25, p=2,\n",
       "                                      weights='uniform'))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see from this GridSearch that k should be set to 25 for best results. Let's see how that does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = machine_learn(KNeighborsClassifier(n_neighbors=25, n_jobs=-1), X_train, y_train)\n",
    "ypred = pipeline.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.8872\n",
      "malefashionadvice:\n",
      "\tPrecision for malefashionadvice: 0.9037926307481411\n",
      "\tRecall for malefashionadvice: 0.9773595076591348\n",
      "\tF1 Score for malefashionadvice: 0.9391375663659516\n",
      "fantanoforever:\n",
      "\tPrecision for fantanoforever: 0.4563106796116505\n",
      "\tRecall for fantanoforever: 0.15443592552026286\n",
      "\tF1 Score for fantanoforever: 0.23076923076923075\n"
     ]
    }
   ],
   "source": [
    "get_scores_binary(y_test, ypred, y_test.unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With k = 19, we get a slightly better accuracy of 88.7%. The precision for /r/malefashionadvice drops, but the recall is great. On the other hand the precision for /r/fantanoforever rises but the recall now drops to an abysmal 15.4%.\n",
    "\n",
    "For now, we will use this as our baseline, but let's hope our other models do better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Random Forest\n",
    "## How Does Random Forest Perform On Our Data?\n",
    "### All Subreddits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reminder of what our code looks like\n",
    "# Also, I removed scaling because it is unnecessary for Random Forest\n",
    "def machine_learn(model, X_train, y_train, n_unigrams=500, n_bigrams=500):\n",
    "    ct = make_column_transformer(\n",
    "        (TfidfVectorizer(max_features=n_unigrams), 'body'),\n",
    "        (TfidfVectorizer(max_features=n_bigrams, ngram_range=(2,2)), 'body'),\n",
    "        remainder='passthrough'\n",
    "    )\n",
    "\n",
    "    pipeline = make_pipeline(\n",
    "        ct,\n",
    "        model\n",
    "    )\n",
    "\n",
    "    return pipeline.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_sample = df.sample(n=200000)\n",
    "X_train, X_test, y_train, y_test = train_test_split(df_sample[['body']], df_sample['subreddit'], train_size=.75, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "pipeline = machine_learn(RandomForestClassifier(n_jobs=-1), X_train, y_train)\n",
    "ypred = pipeline.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.48375\n",
      "Kanye:\n",
      "\tPrecision for Kanye: 0.3834355828220859\n",
      "\tRecall for Kanye: 0.0716743119266055\n",
      "\tF1 Score for Kanye: 0.12077294685990339\n",
      "hiphopheads:\n",
      "\tPrecision for hiphopheads: 0.47529191966370854\n",
      "\tRecall for hiphopheads: 0.33908697100966345\n",
      "\tF1 Score for hiphopheads: 0.3957992998833139\n",
      "leagueoflegends:\n",
      "\tPrecision for leagueoflegends: 0.48270725127833475\n",
      "\tRecall for leagueoflegends: 0.9124511644426404\n",
      "\tF1 Score for leagueoflegends: 0.6313932361382967\n",
      "malefashionadvice:\n",
      "\tPrecision for malefashionadvice: 0.5248508946322068\n",
      "\tRecall for malefashionadvice: 0.18169304886441845\n",
      "\tF1 Score for malefashionadvice: 0.2699386503067485\n",
      "cscareerquestions:\n",
      "\tPrecision for cscareerquestions: 0.5775542615140286\n",
      "\tRecall for cscareerquestions: 0.29534380075798594\n",
      "\tF1 Score for cscareerquestions: 0.3908293032419846\n",
      "MechanicalKeyboards:\n",
      "\tPrecision for MechanicalKeyboards: 0.47108345534407026\n",
      "\tRecall for MechanicalKeyboards: 0.22254885007781428\n",
      "\tF1 Score for MechanicalKeyboards: 0.3022900763358779\n",
      "GlobalOffensive:\n",
      "\tPrecision for GlobalOffensive: 0.3421595598349381\n",
      "\tRecall for GlobalOffensive: 0.08187952600394997\n",
      "\tF1 Score for GlobalOffensive: 0.13213811420982738\n",
      "ProgrammerHumor:\n",
      "\tPrecision for ProgrammerHumor: 0.41151919866444076\n",
      "\tRecall for ProgrammerHumor: 0.09867894315452362\n",
      "\tF1 Score for ProgrammerHumor: 0.15918630933161124\n",
      "Android:\n",
      "\tPrecision for Android: 0.6485801217038539\n",
      "\tRecall for Android: 0.34349402443937155\n",
      "\tF1 Score for Android: 0.44912650337986126\n",
      "indieheads:\n",
      "\tPrecision for indieheads: 0.17142857142857143\n",
      "\tRecall for indieheads: 0.010877447425670777\n",
      "\tF1 Score for indieheads: 0.020456870098874872\n",
      "frugalmalefashion:\n",
      "\tPrecision for frugalmalefashion: 0.3425196850393701\n",
      "\tRecall for frugalmalefashion: 0.048013245033112585\n",
      "\tF1 Score for frugalmalefashion: 0.08422071636011616\n",
      "CalPoly:\n",
      "\tPrecision for CalPoly: 0.0\n",
      "\tRecall for CalPoly: 0.0\n",
      "\tF1 Score for CalPoly: 0.0\n",
      "fantanoforever:\n",
      "\tPrecision for fantanoforever: 0.07142857142857142\n",
      "\tRecall for fantanoforever: 0.002638522427440633\n",
      "\tF1 Score for fantanoforever: 0.005089058524173028\n"
     ]
    }
   ],
   "source": [
    "get_scores(y_test, ypred, y_test.unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow! We did a lot better than our previous accuracy of 29.53% for classification of all 13 classes. While not amazing, 48.4% is a good start especially given that I set no parameters. Let's see how it does on binary, hypertune some parameters, and then come back and see how the new parameters do on the full classification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Binary Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "two_subs = df[(df['subreddit'] == 'fantanoforever') | (df['subreddit'] == 'malefashionadvice')]\n",
    "df_sample = two_subs.sample(n=100000)\n",
    "X_train, X_test, y_train, y_test = train_test_split(df_sample[['body']], df_sample['subreddit'], train_size=.75, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.90468\n",
      "malefashionadvice:\n",
      "\tPrecision for malefashionadvice: 0.9140988949006664\n",
      "\tRecall for malefashionadvice: 0.9852739186237643\n",
      "\tF1 Score for malefashionadvice: 0.9483528391850888\n",
      "fantanoforever:\n",
      "\tPrecision for fantanoforever: 0.6931018301267011\n",
      "\tRecall for fantanoforever: 0.2642691000178923\n",
      "\tF1 Score for fantanoforever: 0.3826424870466321\n"
     ]
    }
   ],
   "source": [
    "pipeline = machine_learn(RandomForestClassifier(n_jobs=-1), X_train, y_train)\n",
    "ypred = pipeline.predict(X_test)\n",
    "get_scores_binary(y_test, ypred, y_test.unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Out of the box our accuracy is a lot better with Random Forest with no tuning than the slightly tuned KNN model. Our accuracy here is 90% while for our b|est KNN we got 87.7%. The precision, recall, and F1 score are also a fair bit better with our Random Forest classifier which indicates that it is improving our predictions quite a bit.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now I am going to tune the hyperparamters quite a bit. This will take a very long time to run so I cut down the number of cross validations to 4 to save time as well as my sanity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_grid_search(model, X_train, y_train, n_unigrams=500, n_bigrams=500):\n",
    "    ct = make_column_transformer(\n",
    "        (TfidfVectorizer(max_features=n_unigrams), 'body'),\n",
    "        (TfidfVectorizer(max_features=n_bigrams, ngram_range=(2,2)), 'body'),\n",
    "        remainder='passthrough'\n",
    "    )\n",
    "\n",
    "    pipeline = make_pipeline(\n",
    "        ct,\n",
    "        model\n",
    "    )\n",
    "    grid_search = GridSearchCV(pipeline,\n",
    "                                param_grid={\n",
    "                                    \"randomforestclassifier__max_features\": [None, 'sqrt', 'log2'],\n",
    "                                    \"randomforestclassifier__criterion\": ['gini', 'entropy'],\n",
    "                                    \"randomforestclassifier__bootstrap\": [True, False]\n",
    "                                },\n",
    "                                scoring='accuracy',\n",
    "                                cv=4, n_jobs=-1)\n",
    "\n",
    "    return grid_search.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "gs = run_grid_search(RandomForestClassifier(n_jobs=-1), X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('columntransformer',\n",
       "                 ColumnTransformer(n_jobs=None, remainder='passthrough',\n",
       "                                   sparse_threshold=0.3,\n",
       "                                   transformer_weights=None,\n",
       "                                   transformers=[('tfidfvectorizer-1',\n",
       "                                                  TfidfVectorizer(analyzer='word',\n",
       "                                                                  binary=False,\n",
       "                                                                  decode_error='strict',\n",
       "                                                                  dtype=<class 'numpy.float64'>,\n",
       "                                                                  encoding='utf-8',\n",
       "                                                                  input='content',\n",
       "                                                                  lowercase=True,\n",
       "                                                                  max_df=1.0,\n",
       "                                                                  max_feature...\n",
       "                 RandomForestClassifier(bootstrap=True, ccp_alpha=0.0,\n",
       "                                        class_weight=None, criterion='entropy',\n",
       "                                        max_depth=None, max_features='sqrt',\n",
       "                                        max_leaf_nodes=None, max_samples=None,\n",
       "                                        min_impurity_decrease=0.0,\n",
       "                                        min_impurity_split=None,\n",
       "                                        min_samples_leaf=1, min_samples_split=2,\n",
       "                                        min_weight_fraction_leaf=0.0,\n",
       "                                        n_estimators=100, n_jobs=-1,\n",
       "                                        oob_score=False, random_state=None,\n",
       "                                        verbose=0, warm_start=False))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The best estimator from this GridSearch use sqrt(n_features) for each tree, entropy as its split metric, and bootstrap sampled the training data for each tree. I will now run another GridSearch exploring other parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_grid_search(model, X_train, y_train, n_unigrams=500, n_bigrams=500):\n",
    "    ct = make_column_transformer(\n",
    "        (TfidfVectorizer(max_features=n_unigrams), 'body'),\n",
    "        (TfidfVectorizer(max_features=n_bigrams, ngram_range=(2,2)), 'body'),\n",
    "        remainder='passthrough'\n",
    "    )\n",
    "\n",
    "    pipeline = make_pipeline(\n",
    "        ct,\n",
    "        model\n",
    "    )\n",
    "    grid_search = GridSearchCV(pipeline,\n",
    "                                param_grid={\n",
    "                                    \"randomforestclassifier__min_samples_split\": range(2, 11, 2),\n",
    "                                    \"randomforestclassifier__min_samples_leaf\": [1,2,4,8]\n",
    "                                },\n",
    "                                scoring='accuracy',\n",
    "                                cv=4, n_jobs=-1)\n",
    "    \n",
    "    return grid_search.fit(X_train, y_train)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('columntransformer',\n",
       "                 ColumnTransformer(n_jobs=None, remainder='passthrough',\n",
       "                                   sparse_threshold=0.3,\n",
       "                                   transformer_weights=None,\n",
       "                                   transformers=[('tfidfvectorizer-1',\n",
       "                                                  TfidfVectorizer(analyzer='word',\n",
       "                                                                  binary=False,\n",
       "                                                                  decode_error='strict',\n",
       "                                                                  dtype=<class 'numpy.float64'>,\n",
       "                                                                  encoding='utf-8',\n",
       "                                                                  input='content',\n",
       "                                                                  lowercase=True,\n",
       "                                                                  max_df=1.0,\n",
       "                                                                  max_feature...\n",
       "                 RandomForestClassifier(bootstrap=True, ccp_alpha=0.0,\n",
       "                                        class_weight=None, criterion='entropy',\n",
       "                                        max_depth=None, max_features='sqrt',\n",
       "                                        max_leaf_nodes=None, max_samples=None,\n",
       "                                        min_impurity_decrease=0.0,\n",
       "                                        min_impurity_split=None,\n",
       "                                        min_samples_leaf=2, min_samples_split=4,\n",
       "                                        min_weight_fraction_leaf=0.0,\n",
       "                                        n_estimators=100, n_jobs=-1,\n",
       "                                        oob_score=False, random_state=None,\n",
       "                                        verbose=0, warm_start=False))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs = run_grid_search(RandomForestClassifier(n_jobs=-1, max_features='sqrt',\n",
    "                                            bootstrap=True,\n",
    "                                            criterion='entropy'\n",
    "                                           ),\n",
    "                     X_train, y_train)\n",
    "gs.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like the min_samples_leaf should be left at 2 (the default) but min_samples_split should be set to 4 instead of 1. These parameters are thresholds for deciding whether to create children nodes during the decision tree training process. They help with mitigating the overfitting of our data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tuning the number of unigrams and bigrams\n",
    "\n",
    "To tune the number of unigrams and bigrams, I took our RF model that we tuned and sampled 200 thousand comments from all subreddits. I then tested the combination of 100, 250, 500, and 1000 unigrams with 100, 250, and 500 bigrams."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "# to speed this up I will only use a sample of 200 thousand \n",
    "\n",
    "df_sample = df.sample(n=200000)\n",
    "X_train, X_test, y_train, y_test = train_test_split(df_sample[['body']], df_sample['subreddit'], train_size=.75, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "tuned_rf = RandomForestClassifier(n_jobs=-1, max_features='sqrt',\n",
    "                                  bootstrap=True, criterion='entropy',\n",
    "                                  min_samples_leaf=2, min_samples_split=4)\n",
    "\n",
    "def run_grid_search(model, X_train, y_train, X_test, y_test, n_unigrams=500, n_bigrams=500):\n",
    "    ct = make_column_transformer(\n",
    "        (TfidfVectorizer(max_features=n_unigrams), 'body'),\n",
    "        (TfidfVectorizer(max_features=n_bigrams, ngram_range=(2,2)), 'body'),\n",
    "        remainder='passthrough'\n",
    "    )\n",
    "\n",
    "    pipeline = make_pipeline(\n",
    "        ct,\n",
    "        model\n",
    "    )\n",
    "    \n",
    "    pipeline.fit(X_train, y_train)\n",
    "    ypred = pipeline.predict(X_test)\n",
    "    return accuracy_score(y_test, ypred), f1_score(y_test, ypred, average='micro')\n",
    "\n",
    "unigram_list = [100, 250, 500, 1000]\n",
    "bigram_list = [100, 250, 500]\n",
    "\n",
    "\n",
    "results = pd.DataFrame(columns=['unigram', 'bigram', 'Accuracy', 'F1'])\n",
    "\n",
    "for u in unigram_list:\n",
    "    for b in bigram_list:\n",
    "        row = {}\n",
    "        row['unigram'] = u\n",
    "        row['bigram'] = b\n",
    "        row[\"Accuracy\"], row[\"F1\"] = run_grid_search(tuned_rf, X_train, y_train, X_test, y_test, u, b)\n",
    "        results = results.append(row, ignore_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>unigram</th>\n",
       "      <th>bigram</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>F1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>0.40966</td>\n",
       "      <td>0.40966</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>100.0</td>\n",
       "      <td>250.0</td>\n",
       "      <td>0.41244</td>\n",
       "      <td>0.41244</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>100.0</td>\n",
       "      <td>500.0</td>\n",
       "      <td>0.41580</td>\n",
       "      <td>0.41580</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>250.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>0.44980</td>\n",
       "      <td>0.44980</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>250.0</td>\n",
       "      <td>250.0</td>\n",
       "      <td>0.44988</td>\n",
       "      <td>0.44988</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>250.0</td>\n",
       "      <td>500.0</td>\n",
       "      <td>0.45104</td>\n",
       "      <td>0.45104</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>500.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>0.47842</td>\n",
       "      <td>0.47842</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>500.0</td>\n",
       "      <td>250.0</td>\n",
       "      <td>0.47896</td>\n",
       "      <td>0.47896</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>500.0</td>\n",
       "      <td>500.0</td>\n",
       "      <td>0.47970</td>\n",
       "      <td>0.47970</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1000.0</td>\n",
       "      <td>250.0</td>\n",
       "      <td>0.50982</td>\n",
       "      <td>0.50982</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1000.0</td>\n",
       "      <td>100.0</td>\n",
       "      <td>0.51044</td>\n",
       "      <td>0.51044</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1000.0</td>\n",
       "      <td>500.0</td>\n",
       "      <td>0.51058</td>\n",
       "      <td>0.51058</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    unigram  bigram  Accuracy       F1\n",
       "0     100.0   100.0   0.40966  0.40966\n",
       "1     100.0   250.0   0.41244  0.41244\n",
       "2     100.0   500.0   0.41580  0.41580\n",
       "3     250.0   100.0   0.44980  0.44980\n",
       "4     250.0   250.0   0.44988  0.44988\n",
       "5     250.0   500.0   0.45104  0.45104\n",
       "6     500.0   100.0   0.47842  0.47842\n",
       "7     500.0   250.0   0.47896  0.47896\n",
       "8     500.0   500.0   0.47970  0.47970\n",
       "10   1000.0   250.0   0.50982  0.50982\n",
       "9    1000.0   100.0   0.51044  0.51044\n",
       "11   1000.0   500.0   0.51058  0.51058"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.sort_values('Accuracy')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hm it appears that the micro-weighted F1 score is the same as accuracy. The best I can find about this on short notice is [here.](https://stackoverflow.com/questions/51815299/same-accuracy-and-f1-score-while-doing-multi-label-classification). I should have set the average to 'weighted', but this took forever to run and we can still use the results of accuracy to make our decision.\n",
    "\n",
    "It appears that 1000 unigrams and 500 bigrams is what gets us both the best accuracy and F1 score. Let's put this all together and predit for a different number of classes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Best RF Model For 2 Classes\n",
    "Let's revist a binary classification but with 2 random subreddits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3            Android\n",
       "1    GlobalOffensive\n",
       "dtype: object"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(df['subreddit'].unique()).sample(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "two_subs = df[(df['subreddit'] == 'Android') | (df['subreddit'] == 'GlobalOffensive')]\n",
    "df_sample = two_subs.sample(n=200000)\n",
    "X_train, X_test, y_train, y_test = train_test_split(df_sample[['body']], df_sample['subreddit'], train_size=.75, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "def machine_learn(model, X_train, y_train, n_unigrams=500, n_bigrams=500):\n",
    "    ct = make_column_transformer(\n",
    "        (TfidfVectorizer(max_features=n_unigrams), 'body'),\n",
    "        (TfidfVectorizer(max_features=n_bigrams, ngram_range=(2,2)), 'body'),\n",
    "        remainder='passthrough'\n",
    "    )\n",
    "\n",
    "    pipeline = make_pipeline(\n",
    "        ct,\n",
    "        model\n",
    "    )\n",
    "\n",
    "    return pipeline.fit(X_train, y_train)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = machine_learn(tuned_rf, X_train, y_train, n_unigrams=1000, n_bigrams=100)\n",
    "ypred = pipeline.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.847232\n",
      "Android:\n",
      "\tPrecision for Android: 0.9003893026404874\n",
      "\tRecall for Android: 0.672191567330778\n",
      "\tF1 Score for Android: 0.7697335101893162\n",
      "GlobalOffensive:\n",
      "\tPrecision for GlobalOffensive: 0.8261903698409863\n",
      "\tRecall for GlobalOffensive: 0.9544492891973477\n",
      "\tF1 Score for GlobalOffensive: 0.8857006045370205\n"
     ]
    }
   ],
   "source": [
    "get_scores_binary(y_test, ypred, labels=y_test.unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since this is a completely different set of data than what we tuned on. I am not surprised by the drop of accuracy given how this is a completely different dataset than what we tested the hyperparametrs on. However, it appears that it did a pretty decent job with the F1 scores meaning it is fairly balanced. The accuracy did drop to 84% but it did a good job balancing precision and recall between the two classes. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How About 5 Random Subreddits?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1       GlobalOffensive\n",
       "10    cscareerquestions\n",
       "11    frugalmalefashion\n",
       "5               CalPoly\n",
       "8                 Kanye\n",
       "dtype: object"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(df['subreddit'].unique()).sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "five_subs = df[(df['subreddit'] == 'GlobalOffensive') | (df['subreddit'] == 'cscareerquestions')\n",
    "             | (df['subreddit'] == 'frugalmalefashion') | (df['subreddit'] == 'CalPoly') | (df['subreddit'] == 'Kanye')]\n",
    "df_sample = five_subs.sample(n=500000)\n",
    "X_train, X_test, y_train, y_test = train_test_split(df_sample[['body']], df_sample['subreddit'], train_size=.75, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = machine_learn(tuned_rf, X_train, y_train, n_unigrams=1000, n_bigrams=500)\n",
    "ypred = pipeline.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.733608\n",
      "GlobalOffensive:\n",
      "\tPrecision for GlobalOffensive: 0.6988297279497442\n",
      "\tRecall for GlobalOffensive: 0.9697016562157298\n",
      "\tF1 Score for GlobalOffensive: 0.8122788112188717\n",
      "frugalmalefashion:\n",
      "\tPrecision for frugalmalefashion: 0.862278978388998\n",
      "\tRecall for frugalmalefashion: 0.3967278315104402\n",
      "\tF1 Score for frugalmalefashion: 0.5434284653005633\n",
      "cscareerquestions:\n",
      "\tPrecision for cscareerquestions: 0.8788718730510706\n",
      "\tRecall for cscareerquestions: 0.584577802359882\n",
      "\tF1 Score for cscareerquestions: 0.7021341379023999\n",
      "Kanye:\n",
      "\tPrecision for Kanye: 0.8338988051335006\n",
      "\tRecall for Kanye: 0.2788437823706408\n",
      "\tF1 Score for Kanye: 0.41793582729557893\n",
      "CalPoly:\n",
      "\tPrecision for CalPoly: 1.0\n",
      "\tRecall for CalPoly: 0.0059382422802850355\n",
      "\tF1 Score for CalPoly: 0.01180637544273908\n"
     ]
    }
   ],
   "source": [
    "get_scores(y_test, ypred, y_test.unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For 5 subreddits we get a decent accuracy of 73%. Precision scores across the board are actually quite good, however recall scores are low for most. I'd say this right here is the best we could ask for using such an unbalanced dataset, and I'm really happy with the results for the more moderate classification of 5 classes as opposed to 13, and it shows how our tree still works well when we stray from binary classification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## All 13 Subreddits\n",
    "\n",
    "I will use the majority of my data for this, because this is the ultimate test of my models. Let's see how my more refined Random Forest does on all the subreddits when fed 3 million comments to train and 1 million comments to test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_sample = df.sample(n=4000000)\n",
    "X_train, X_test, y_train, y_test = train_test_split(df_sample[['body']], df_sample['subreddit'], train_size=.75, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = machine_learn(tuned_rf, X_train, y_train, n_unigrams=1000, n_bigrams=500)\n",
    "ypred = pipeline.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.545213\n",
      "leagueoflegends:\n",
      "\tPrecision for leagueoflegends: 0.504429780940166\n",
      "\tRecall for leagueoflegends: 0.9534603898185222\n",
      "\tF1 Score for leagueoflegends: 0.6597943044241712\n",
      "malefashionadvice:\n",
      "\tPrecision for malefashionadvice: 0.641080646242542\n",
      "\tRecall for malefashionadvice: 0.3196296667669374\n",
      "\tF1 Score for malefashionadvice: 0.42657685788205907\n",
      "hiphopheads:\n",
      "\tPrecision for hiphopheads: 0.5906851243547631\n",
      "\tRecall for hiphopheads: 0.40199600798403196\n",
      "\tF1 Score for hiphopheads: 0.478407525298114\n",
      "ProgrammerHumor:\n",
      "\tPrecision for ProgrammerHumor: 0.5880821790462025\n",
      "\tRecall for ProgrammerHumor: 0.12597578461048273\n",
      "\tF1 Score for ProgrammerHumor: 0.20750168107194988\n",
      "Android:\n",
      "\tPrecision for Android: 0.7921448311396319\n",
      "\tRecall for Android: 0.45882225266899085\n",
      "\tF1 Score for Android: 0.5810763217798394\n",
      "GlobalOffensive:\n",
      "\tPrecision for GlobalOffensive: 0.7788479521226856\n",
      "\tRecall for GlobalOffensive: 0.1372237278920531\n",
      "\tF1 Score for GlobalOffensive: 0.23333636828429555\n",
      "indieheads:\n",
      "\tPrecision for indieheads: 0.8161764705882353\n",
      "\tRecall for indieheads: 0.024085057138724143\n",
      "\tF1 Score for indieheads: 0.04678937754671912\n",
      "MechanicalKeyboards:\n",
      "\tPrecision for MechanicalKeyboards: 0.6731423621852379\n",
      "\tRecall for MechanicalKeyboards: 0.3748492678725237\n",
      "\tF1 Score for MechanicalKeyboards: 0.48154377268301324\n",
      "Kanye:\n",
      "\tPrecision for Kanye: 0.6289635589209654\n",
      "\tRecall for Kanye: 0.1164257555847569\n",
      "\tF1 Score for Kanye: 0.196481371969249\n",
      "cscareerquestions:\n",
      "\tPrecision for cscareerquestions: 0.6849148418491484\n",
      "\tRecall for cscareerquestions: 0.3935898470059961\n",
      "\tF1 Score for cscareerquestions: 0.49990608404624076\n",
      "frugalmalefashion:\n",
      "\tPrecision for frugalmalefashion: 0.6296296296296297\n",
      "\tRecall for frugalmalefashion: 0.13341110345193277\n",
      "\tF1 Score for frugalmalefashion: 0.22017064099759354\n",
      "fantanoforever:\n",
      "\tPrecision for fantanoforever: 0.6\n",
      "\tRecall for fantanoforever: 0.006907137375287797\n",
      "\tF1 Score for fantanoforever: 0.013657056145675266\n",
      "CalPoly:\n",
      "\tPrecision for CalPoly: 0.9473684210526315\n",
      "\tRecall for CalPoly: 0.012474012474012475\n",
      "\tF1 Score for CalPoly: 0.024623803009575923\n"
     ]
    }
   ],
   "source": [
    "get_scores(y_test, ypred, y_test.unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After runnning the code for 6+ hours, we finally get our results. Our accuracy at 54.5% isn't amazing, but I am actually quite happy with it. Our recall, precision, and F1 scores have something to be desired for many of the classes. /r/leagueoflegends, our most represented class, has the highest F1 score of 66%. Not bad, but not great. Our lowest F1 score belongs to /r/fantanoforever, which is understandable given how underrepresented the class is and how it is similar to other music subreddits.\n",
    "\n",
    "Some other interesting metrics to note are that I suspect some of the classes are conflicting with each other such as all of the music subreddits which seem to share similar frequency of words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Final Thoughts\n",
    "I wish I had more time to fully explore the parameters for binary vs full classification and try to make models specific to those use cases. I wish I had made more use of all my data but there was so much and everything took a very long time to run which was unfortunate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "name": "Reddit Data Exploration.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}