Final_Project_Emre_Okcular.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Click-Prediction | Final Project MSDS 699 - Machine Learning Laboratory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Author : _Emre Okcular_\n",
    "\n",
    "Date : 03.12.2021"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "* Introduction\n",
    "* Dataset\n",
    "* Preprocessing and Exploratory Data Analysis\n",
    "* Training Models\n",
    "* Summary "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this project, the predictions of whether or not people clicked on these ads will be made with machine learning techniques. The goal is to find more accurate predictions. This information will be used in optimizing the revenue from ads and ad auction systems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Author: Tencent Inc.\n",
    "Source: [KDD Cup](https://www.kddcup2012.org/) - 2012\n",
    "Please cite:\n",
    "\n",
    "This data is derived from the 2012 KDD Cup. The data is subsampled to 1% of the original number of instances, downsampling the majority class (click=0) so that the target feature is reasonably balanced (5 to 1).\n",
    "\n",
    "The data is about advertisements shown alongside search results in a search engine, and whether or not people clicked on these ads.\n",
    "The task is to build the best possible model to predict whether a user will click on a given ad.\n",
    "\n",
    "A search session contains information on user id, the query issued by the user, ads displayed to the user, and target feature indicating whether a user clicked at least one of the ads in this session. The number of ads displayed to a user in a session is called ‘depth’. The order of an ad in the displayed list is called ‘position’. An ad is displayed as a short text called ‘title’, followed by a slightly longer text called ’description’, and a URL called ‘display URL’.\n",
    "To construct this dataset each session was split into multiple instances. Each instance describes an ad displayed under a certain setting (‘depth’, ‘position’). Instances with the same user id, ad id, query, and setting are merged. Each ad and each user have some additional properties located in separate data files that can be looked up using ids in the instances.\n",
    "\n",
    "The dataset has the following features:\n",
    "* Click – binary variable indicating whether a user clicked on at least one ad.\n",
    "* Impression - the number of search sessions in which AdID was impressed by UserID who issued Query.\n",
    "* Url_hash - URL is hashed for anonymity\n",
    "* AdID\n",
    "* AdvertiserID - some advertisers consistently optimize their ads, so the title and description of their ads are more attractive than those of others’ ads.\n",
    "* Depth - number of ads displayed to a user in a session\n",
    "* Position - order of an ad in the displayed list\n",
    "* QueryID - is the key of the data file 'queryid_tokensid.txt'. \n",
    "* KeywordID - is the key of 'purchasedkeyword_tokensid.txt' \n",
    "* TitleID - is the key of 'titleid_tokensid.txt'\n",
    "* DescriptionID - is the key of 'descriptionid_tokensid.txt' \n",
    "* UserID – is also the key of 'userid_profile.txt' . 0 is a special value denoting that the user could be identified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://raw.githubusercontent.com/emreokcular/click-prediction/main/requirements.txt\n",
    "!pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from scipy.io.arff import loadarff # For loading .arff data file.\n",
    "\n",
    "from sklearn.compose            import *\n",
    "from sklearn.ensemble           import RandomForestClassifier, ExtraTreesClassifier\n",
    "from sklearn.experimental       import enable_iterative_imputer\n",
    "from sklearn.impute             import *\n",
    "from sklearn.linear_model       import LogisticRegression, PassiveAggressiveClassifier, RidgeClassifier, SGDClassifier\n",
    "from sklearn.metrics            import accuracy_score, balanced_accuracy_score, f1_score, precision_score, average_precision_score , make_scorer, recall_score\n",
    "from sklearn.pipeline           import Pipeline\n",
    "from sklearn.preprocessing      import *\n",
    "from sklearn.tree               import DecisionTreeClassifier, ExtraTreeClassifier\n",
    "from sklearn.model_selection    import train_test_split\n",
    "from sklearn.preprocessing      import LabelEncoder\n",
    "from sklearn.model_selection    import RandomizedSearchCV\n",
    "from sklearn.model_selection    import cross_validate, KFold\n",
    "from sklearn.base               import BaseEstimator\n",
    "\n",
    "from pandas_profiling import ProfileReport # For EDA\n",
    "\n",
    "from sklearn import set_config # For visualizing pipelines\n",
    "set_config(display='diagram')\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Helper Functions and Classes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_scores(y_test,y_pred):\n",
    "    \"\"\"\n",
    "    Prints Accuracy and F1 Scores\n",
    "    \"\"\"\n",
    "    score = accuracy_score(y_test, y_pred)\n",
    "    print(f\"The Accuracy score for is: %.{3}f\" % score)\n",
    "    score = f1_score(y_pred, y_test, average=\"weighted\")\n",
    "    print(f\"The F1 score for is: %.{3}f\" % score)\n",
    "\n",
    "    \n",
    "class DummyEstimator(BaseEstimator):\n",
    "    def fit(self): pass\n",
    "    def score(self): pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing and Exploratory Data Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_data_arff = loadarff('phpfGCaQC.arff')\n",
    "df = pd.DataFrame(raw_data_arff[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"click\"] = df['click'].str.decode(\"utf-8\").astype(int)\n",
    "df[\"ad_id\"] = df[\"ad_id\"].astype(int)\n",
    "df[\"advertiser_id\"] = df[\"advertiser_id\"].astype(int)\n",
    "df[\"depth\"] = df[\"depth\"].astype(int)\n",
    "df[\"position\"] = df[\"position\"].astype(int)\n",
    "df[\"query_id\"] = df[\"query_id\"].astype(int)\n",
    "df[\"keyword_id\"] = df[\"keyword_id\"].astype(int)\n",
    "df[\"title_id\"] = df[\"title_id\"].astype(int)\n",
    "df[\"description_id\"] = df[\"description_id\"].astype(int)\n",
    "df[\"user_id\"] = df[\"user_id\"].astype(int)\n",
    "df[\"impression\"] = df[\"impression\"].astype(int)\n",
    "df[\"url_hash\"] = df[\"url_hash\"].astype(float)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>click</th>\n",
       "      <th>impression</th>\n",
       "      <th>url_hash</th>\n",
       "      <th>ad_id</th>\n",
       "      <th>advertiser_id</th>\n",
       "      <th>depth</th>\n",
       "      <th>position</th>\n",
       "      <th>query_id</th>\n",
       "      <th>keyword_id</th>\n",
       "      <th>title_id</th>\n",
       "      <th>description_id</th>\n",
       "      <th>user_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1.071003e+19</td>\n",
       "      <td>8343295</td>\n",
       "      <td>11700</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>7702266</td>\n",
       "      <td>21264</td>\n",
       "      <td>27892</td>\n",
       "      <td>1559</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.736385e+19</td>\n",
       "      <td>20017077</td>\n",
       "      <td>23798</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>93079</td>\n",
       "      <td>35498</td>\n",
       "      <td>4</td>\n",
       "      <td>36476</td>\n",
       "      <td>562934</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>8.915473e+18</td>\n",
       "      <td>21348354</td>\n",
       "      <td>36654</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>10981</td>\n",
       "      <td>19975</td>\n",
       "      <td>36105</td>\n",
       "      <td>33292</td>\n",
       "      <td>11621116</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4.426693e+18</td>\n",
       "      <td>20366086</td>\n",
       "      <td>33280</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>5942</td>\n",
       "      <td>4057</td>\n",
       "      <td>4390</td>\n",
       "      <td>8778348</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1.157260e+19</td>\n",
       "      <td>6803526</td>\n",
       "      <td>10790</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>9881978</td>\n",
       "      <td>60593</td>\n",
       "      <td>25242</td>\n",
       "      <td>1679</td>\n",
       "      <td>12118311</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   click  impression      url_hash     ad_id  advertiser_id  depth  position  \\\n",
       "0      0           1  1.071003e+19   8343295          11700      3         3   \n",
       "1      1           1  1.736385e+19  20017077          23798      1         1   \n",
       "2      0           1  8.915473e+18  21348354          36654      1         1   \n",
       "3      0           1  4.426693e+18  20366086          33280      3         3   \n",
       "4      0           1  1.157260e+19   6803526          10790      2         1   \n",
       "\n",
       "   query_id  keyword_id  title_id  description_id   user_id  \n",
       "0   7702266       21264     27892            1559         0  \n",
       "1     93079       35498         4           36476    562934  \n",
       "2     10981       19975     36105           33292  11621116  \n",
       "3         0        5942      4057            4390   8778348  \n",
       "4   9881978       60593     25242            1679  12118311  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 39948 entries, 0 to 39947\n",
      "Data columns (total 12 columns):\n",
      " #   Column          Non-Null Count  Dtype  \n",
      "---  ------          --------------  -----  \n",
      " 0   click           39948 non-null  int64  \n",
      " 1   impression      39948 non-null  int64  \n",
      " 2   url_hash        39948 non-null  float64\n",
      " 3   ad_id           39948 non-null  int64  \n",
      " 4   advertiser_id   39948 non-null  int64  \n",
      " 5   depth           39948 non-null  int64  \n",
      " 6   position        39948 non-null  int64  \n",
      " 7   query_id        39948 non-null  int64  \n",
      " 8   keyword_id      39948 non-null  int64  \n",
      " 9   title_id        39948 non-null  int64  \n",
      " 10  description_id  39948 non-null  int64  \n",
      " 11  user_id         39948 non-null  int64  \n",
      "dtypes: float64(1), int64(11)\n",
      "memory usage: 3.7 MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a95411a5588c4508bb8f7c2e3922fa00",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Summarize dataset', max=27.0, style=ProgressStyle(descrip…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8d823610738942949ee2731a98446143",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Generate report structure', max=1.0, style=ProgressStyle(…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Render widgets', max=1.0, style=ProgressStyle(description…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ce8062e177b24737a072d2118e930a6c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Tab(children=(Tab(children=(GridBox(children=(VBox(children=(GridspecLayout(children=(HTML(valu…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "88db6396d9fa49ebb68c2b958f480038",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Render HTML', max=1.0, style=ProgressStyle(description_wi…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "532014a911784542935932b40423c207",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Export report to file', max=1.0, style=ProgressStyle(desc…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "profile = ProfileReport(df, title=\"Pandas Profiling Report\")\n",
    "profile.to_widgets()\n",
    "profile.to_file(\"Pandas Profiling Report.html\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the pandas profiling warning report it says that there are 22 duplicate rows in dataset. These rows are dropped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.drop_duplicates()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6941"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"url_hash\"].nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have one categorical variable which has 6941 unique values. Type of this variable is float for numeric calculations in models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "click             0\n",
       "impression        0\n",
       "url_hash          0\n",
       "ad_id             0\n",
       "advertiser_id     0\n",
       "depth             0\n",
       "position          0\n",
       "query_id          0\n",
       "keyword_id        0\n",
       "title_id          0\n",
       "description_id    0\n",
       "user_id           0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have no null values but null values encoded as 0 in the dataset so we have zeros to impute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "click             33201\n",
       "impression            0\n",
       "url_hash              0\n",
       "ad_id                 0\n",
       "advertiser_id         0\n",
       "depth                 0\n",
       "position              0\n",
       "query_id            570\n",
       "keyword_id          569\n",
       "title_id            355\n",
       "description_id      355\n",
       "user_id            9627\n",
       "dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.sum(df==0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Percentage of ones in total : 16.84\n"
     ]
    }
   ],
   "source": [
    "print(\"Percentage of ones in total :\",round(len(df[df[\"click\"]==1])/len(df)*100,2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = df[\"click\"]\n",
    "X = df.iloc[:,1:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train test split is applied with 20:80 ratio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size = 0.2, shuffle=True , random_state=99)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Firstly, LogisticRegression, DecisionTreeClassifier and RandomForestClassifier models are used for algorithm search with default parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a pipeline\n",
    "pipe_dummy = Pipeline([('clf', DummyEstimator())]) # Placeholder Estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create space of candidate learning algorithms and their hyperparameters\n",
    "search_space = [{'clf': [LogisticRegression()]},\n",
    "                {'clf': [DecisionTreeClassifier()]},\n",
    "                {'clf': [RandomForestClassifier()]}]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Random search for default models\n",
    "clf_algos_rand = RandomizedSearchCV(estimator=pipe_dummy, \n",
    "                                    param_distributions=search_space, \n",
    "                                    n_iter=25,\n",
    "                                    cv=5, \n",
    "                                    n_jobs=-1,\n",
    "                                    verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 3 candidates, totalling 15 fits\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>div.sk-top-container {color: black;background-color: white;}div.sk-toggleable {background-color: white;}label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.2em 0.3em;box-sizing: border-box;text-align: center;}div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}div.sk-estimator {font-family: monospace;background-color: #f0f8ff;margin: 0.25em 0.25em;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;}div.sk-estimator:hover {background-color: #d4ebff;}div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;}div.sk-item {z-index: 1;}div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}div.sk-parallel-item:only-child::after {width: 0;}div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0.2em;box-sizing: border-box;padding-bottom: 0.1em;background-color: white;position: relative;}div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}div.sk-label-container {position: relative;z-index: 2;text-align: center;}div.sk-container {display: inline-block;position: relative;}</style><div class=\"sk-top-container\"><div class=\"sk-container\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"dce69314-d648-4cad-a64e-a6c1a29a01e3\" type=\"checkbox\" checked><label class=\"sk-toggleable__label\" for=\"dce69314-d648-4cad-a64e-a6c1a29a01e3\">RandomForestClassifier</label><div class=\"sk-toggleable__content\"><pre>RandomForestClassifier()</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "RandomForestClassifier()"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#  Fit  search\n",
    "best_model = clf_algos_rand.fit(X_train, y_train);\n",
    "\n",
    "# View best model\n",
    "best_model.best_estimator_.get_params()['clf']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Accuracy score for is: 0.833\n",
      "The F1 score for is: 0.881\n"
     ]
    }
   ],
   "source": [
    "y_pred = best_model.predict(X_test)\n",
    "print_scores(y_test,y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the default parameter algorithm search gives Logistic Regression and Decision Tree classifiers as the best estimator randomly, we will examine these two algorithm closely by applyting hyperparameter grid search."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Logistic Regression Classifier with Grid Search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we will try Logistic Regression with grid search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>div.sk-top-container {color: black;background-color: white;}div.sk-toggleable {background-color: white;}label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.2em 0.3em;box-sizing: border-box;text-align: center;}div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}div.sk-estimator {font-family: monospace;background-color: #f0f8ff;margin: 0.25em 0.25em;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;}div.sk-estimator:hover {background-color: #d4ebff;}div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;}div.sk-item {z-index: 1;}div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}div.sk-parallel-item:only-child::after {width: 0;}div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0.2em;box-sizing: border-box;padding-bottom: 0.1em;background-color: white;position: relative;}div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}div.sk-label-container {position: relative;z-index: 2;text-align: center;}div.sk-container {display: inline-block;position: relative;}</style><div class=\"sk-top-container\"><div class=\"sk-container\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"782e377e-fc90-44b3-af2d-fd1a0f82d2ce\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"782e377e-fc90-44b3-af2d-fd1a0f82d2ce\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[('preprocessing',\n",
       "                 ColumnTransformer(transformers=[('categorical',\n",
       "                                                  Pipeline(steps=[('imputer',\n",
       "                                                                   SimpleImputer(missing_values=0,\n",
       "                                                                                 strategy='most_frequent')),\n",
       "                                                                  ('encoder',\n",
       "                                                                   OneHotEncoder(handle_unknown='ignore'))]),\n",
       "                                                  impression        False\n",
       "url_hash          False\n",
       "ad_id             False\n",
       "advertiser_id     False\n",
       "depth             False\n",
       "position          False\n",
       "query_id          False\n",
       "keyword_id        False\n",
       "title_id          False\n",
       "description_id    False\n",
       "user_id           False\n",
       "dtype: bool),\n",
       "                                                 ('continuous',\n",
       "                                                  Pipeline(steps=[('imputer',\n",
       "                                                                   SimpleImputer(add_indicator=True,\n",
       "                                                                                 missing_values=0)),\n",
       "                                                                  ('scaler',\n",
       "                                                                   MinMaxScaler())]),\n",
       "                                                  impression        True\n",
       "url_hash          True\n",
       "ad_id             True\n",
       "advertiser_id     True\n",
       "depth             True\n",
       "position          True\n",
       "query_id          True\n",
       "keyword_id        True\n",
       "title_id          True\n",
       "description_id    True\n",
       "user_id           True\n",
       "dtype: bool)])),\n",
       "                ('lm', LogisticRegression())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"cf4043b9-f2ac-40c7-9b83-c48414408de8\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"cf4043b9-f2ac-40c7-9b83-c48414408de8\">preprocessing: ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[('categorical',\n",
       "                                 Pipeline(steps=[('imputer',\n",
       "                                                  SimpleImputer(missing_values=0,\n",
       "                                                                strategy='most_frequent')),\n",
       "                                                 ('encoder',\n",
       "                                                  OneHotEncoder(handle_unknown='ignore'))]),\n",
       "                                 impression        False\n",
       "url_hash          False\n",
       "ad_id             False\n",
       "advertiser_id     False\n",
       "depth             False\n",
       "position          False\n",
       "query_id          False\n",
       "keyword_id        False\n",
       "title_id          False\n",
       "description_id    False\n",
       "user_id           False\n",
       "dtype: bool),\n",
       "                                ('continuous',\n",
       "                                 Pipeline(steps=[('imputer',\n",
       "                                                  SimpleImputer(add_indicator=True,\n",
       "                                                                missing_values=0)),\n",
       "                                                 ('scaler', MinMaxScaler())]),\n",
       "                                 impression        True\n",
       "url_hash          True\n",
       "ad_id             True\n",
       "advertiser_id     True\n",
       "depth             True\n",
       "position          True\n",
       "query_id          True\n",
       "keyword_id        True\n",
       "title_id          True\n",
       "description_id    True\n",
       "user_id           True\n",
       "dtype: bool)])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"8e5cc0ba-933a-4d3a-83a1-567ae9be63fa\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"8e5cc0ba-933a-4d3a-83a1-567ae9be63fa\">categorical</label><div class=\"sk-toggleable__content\"><pre>impression        False\n",
       "url_hash          False\n",
       "ad_id             False\n",
       "advertiser_id     False\n",
       "depth             False\n",
       "position          False\n",
       "query_id          False\n",
       "keyword_id        False\n",
       "title_id          False\n",
       "description_id    False\n",
       "user_id           False\n",
       "dtype: bool</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"a053322b-abfc-442a-9620-2b942612d397\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"a053322b-abfc-442a-9620-2b942612d397\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(missing_values=0, strategy='most_frequent')</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"e8280856-b8b6-41e1-8ece-ec0b990b1bc8\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"e8280856-b8b6-41e1-8ece-ec0b990b1bc8\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown='ignore')</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"85d599f1-21ae-47a4-bce3-ca22bd639bae\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"85d599f1-21ae-47a4-bce3-ca22bd639bae\">continuous</label><div class=\"sk-toggleable__content\"><pre>impression        True\n",
       "url_hash          True\n",
       "ad_id             True\n",
       "advertiser_id     True\n",
       "depth             True\n",
       "position          True\n",
       "query_id          True\n",
       "keyword_id        True\n",
       "title_id          True\n",
       "description_id    True\n",
       "user_id           True\n",
       "dtype: bool</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"68b921a7-1358-4616-833f-66ea7fd98cee\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"68b921a7-1358-4616-833f-66ea7fd98cee\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(add_indicator=True, missing_values=0)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"dd586e39-63c2-4686-a8c9-a925acabfbe0\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"dd586e39-63c2-4686-a8c9-a925acabfbe0\">MinMaxScaler</label><div class=\"sk-toggleable__content\"><pre>MinMaxScaler()</pre></div></div></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"07c4d180-0baf-48e4-92eb-7859e59d3d79\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"07c4d180-0baf-48e4-92eb-7859e59d3d79\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression()</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('preprocessing',\n",
       "                 ColumnTransformer(transformers=[('categorical',\n",
       "                                                  Pipeline(steps=[('imputer',\n",
       "                                                                   SimpleImputer(missing_values=0,\n",
       "                                                                                 strategy='most_frequent')),\n",
       "                                                                  ('encoder',\n",
       "                                                                   OneHotEncoder(handle_unknown='ignore'))]),\n",
       "                                                  impression        False\n",
       "url_hash          False\n",
       "ad_id             False\n",
       "advertiser_id     False\n",
       "depth             False\n",
       "position          False\n",
       "query_id          False\n",
       "keyword_id        False\n",
       "title_id          False\n",
       "description_id    False\n",
       "user_id           False\n",
       "dtype: bool),\n",
       "                                                 ('continuous',\n",
       "                                                  Pipeline(steps=[('imputer',\n",
       "                                                                   SimpleImputer(add_indicator=True,\n",
       "                                                                                 missing_values=0)),\n",
       "                                                                  ('scaler',\n",
       "                                                                   MinMaxScaler())]),\n",
       "                                                  impression        True\n",
       "url_hash          True\n",
       "ad_id             True\n",
       "advertiser_id     True\n",
       "depth             True\n",
       "position          True\n",
       "query_id          True\n",
       "keyword_id        True\n",
       "title_id          True\n",
       "description_id    True\n",
       "user_id           True\n",
       "dtype: bool)])),\n",
       "                ('lm', LogisticRegression())])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "con_pipe = Pipeline([('imputer', SimpleImputer(missing_values=0 ,strategy='mean', add_indicator=True)), \n",
    "                        ('scaler', MinMaxScaler())])\n",
    "\n",
    "cat_pipe = Pipeline([('imputer', SimpleImputer(missing_values=0 , strategy='most_frequent')),\n",
    "                        ('encoder', OneHotEncoder(handle_unknown='ignore'))])\n",
    "\n",
    "preprocessing = ColumnTransformer([('categorical', cat_pipe,  (X_train.dtypes == object)), \n",
    "                                    ('continuous',  con_pipe, ~(X_train.dtypes == object))])\n",
    "\n",
    "pipe_lm = Pipeline([('preprocessing', preprocessing), \n",
    "                    ('lm', LogisticRegression())])\n",
    "\n",
    "pipe_lm.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will continue to increase this score with hyperparameter search and CV techniques."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "hyperparameters = {\"n_jobs\": -1, \"class_weight\":\"balanced_subsample\", \"max_depth\":2, \"max_leaf_nodes\":10 }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_grid_lr = {'lm__penalty': [\"l1\", \"l2\"], # Type of regularization, for penalizing outlier features\n",
    "              'lm__C': [0.1, 0.2, 0.3, 0.5, 0.6, 0.7, 0.9, 1], # The amount of regularization\n",
    "              'lm__solver': [\"newton-cg\", \"lbfgs\", \"liblinear\", \"sag\"] # Solver for finding the parameters\n",
    "             }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>div.sk-top-container {color: black;background-color: white;}div.sk-toggleable {background-color: white;}label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.2em 0.3em;box-sizing: border-box;text-align: center;}div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}div.sk-estimator {font-family: monospace;background-color: #f0f8ff;margin: 0.25em 0.25em;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;}div.sk-estimator:hover {background-color: #d4ebff;}div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;}div.sk-item {z-index: 1;}div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}div.sk-parallel-item:only-child::after {width: 0;}div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0.2em;box-sizing: border-box;padding-bottom: 0.1em;background-color: white;position: relative;}div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}div.sk-label-container {position: relative;z-index: 2;text-align: center;}div.sk-container {display: inline-block;position: relative;}</style><div class=\"sk-top-container\"><div class=\"sk-container\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"4d60d814-e10d-4e59-857d-747dd9b09036\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"4d60d814-e10d-4e59-857d-747dd9b09036\">RandomizedSearchCV</label><div class=\"sk-toggleable__content\"><pre>RandomizedSearchCV(cv=5,\n",
       "                   estimator=Pipeline(steps=[('preprocessing',\n",
       "                                              ColumnTransformer(transformers=[('categorical',\n",
       "                                                                               Pipeline(steps=[('imputer',\n",
       "                                                                                                SimpleImputer(missing_values=0,\n",
       "                                                                                                              strategy='most_frequent')),\n",
       "                                                                                               ('encoder',\n",
       "                                                                                                OneHotEncoder(handle_unknown='ignore'))]),\n",
       "                                                                               impression        False\n",
       "url_hash          False\n",
       "ad_id             False\n",
       "advertiser_id     False\n",
       "depth             False\n",
       "position          False\n",
       "query_id          False\n",
       "keywor...\n",
       "                                                                                                MinMaxScaler())]),\n",
       "                                                                               impression        True\n",
       "url_hash          True\n",
       "ad_id             True\n",
       "advertiser_id     True\n",
       "depth             True\n",
       "position          True\n",
       "query_id          True\n",
       "keyword_id        True\n",
       "title_id          True\n",
       "description_id    True\n",
       "user_id           True\n",
       "dtype: bool)])),\n",
       "                                             ('lm', LogisticRegression())]),\n",
       "                   n_iter=50, n_jobs=-1,\n",
       "                   param_distributions={'lm__C': [0.1, 0.2, 0.3, 0.5, 0.6, 0.7,\n",
       "                                                  0.9, 1],\n",
       "                                        'lm__penalty': ['l1', 'l2'],\n",
       "                                        'lm__solver': ['newton-cg', 'lbfgs',\n",
       "                                                       'liblinear', 'sag']})</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"92c7a051-b2f8-45a3-8312-aaab805197ac\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"92c7a051-b2f8-45a3-8312-aaab805197ac\">preprocessing: ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[('categorical',\n",
       "                                 Pipeline(steps=[('imputer',\n",
       "                                                  SimpleImputer(missing_values=0,\n",
       "                                                                strategy='most_frequent')),\n",
       "                                                 ('encoder',\n",
       "                                                  OneHotEncoder(handle_unknown='ignore'))]),\n",
       "                                 impression        False\n",
       "url_hash          False\n",
       "ad_id             False\n",
       "advertiser_id     False\n",
       "depth             False\n",
       "position          False\n",
       "query_id          False\n",
       "keyword_id        False\n",
       "title_id          False\n",
       "description_id    False\n",
       "user_id           False\n",
       "dtype: bool),\n",
       "                                ('continuous',\n",
       "                                 Pipeline(steps=[('imputer',\n",
       "                                                  SimpleImputer(add_indicator=True,\n",
       "                                                                missing_values=0)),\n",
       "                                                 ('scaler', MinMaxScaler())]),\n",
       "                                 impression        True\n",
       "url_hash          True\n",
       "ad_id             True\n",
       "advertiser_id     True\n",
       "depth             True\n",
       "position          True\n",
       "query_id          True\n",
       "keyword_id        True\n",
       "title_id          True\n",
       "description_id    True\n",
       "user_id           True\n",
       "dtype: bool)])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"b7f338d9-2a3d-4885-99e6-1091e0479b99\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"b7f338d9-2a3d-4885-99e6-1091e0479b99\">categorical</label><div class=\"sk-toggleable__content\"><pre>impression        False\n",
       "url_hash          False\n",
       "ad_id             False\n",
       "advertiser_id     False\n",
       "depth             False\n",
       "position          False\n",
       "query_id          False\n",
       "keyword_id        False\n",
       "title_id          False\n",
       "description_id    False\n",
       "user_id           False\n",
       "dtype: bool</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"d0346b6e-9e20-4c71-8add-42a384426763\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"d0346b6e-9e20-4c71-8add-42a384426763\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(missing_values=0, strategy='most_frequent')</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"d08b5b4c-c3d9-4a6f-b4b3-24c9875b79b8\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"d08b5b4c-c3d9-4a6f-b4b3-24c9875b79b8\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown='ignore')</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"346732aa-dbc7-4ef7-8bcb-521fe06473d3\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"346732aa-dbc7-4ef7-8bcb-521fe06473d3\">continuous</label><div class=\"sk-toggleable__content\"><pre>impression        True\n",
       "url_hash          True\n",
       "ad_id             True\n",
       "advertiser_id     True\n",
       "depth             True\n",
       "position          True\n",
       "query_id          True\n",
       "keyword_id        True\n",
       "title_id          True\n",
       "description_id    True\n",
       "user_id           True\n",
       "dtype: bool</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"32448a76-e332-4fcc-8a89-2a43e98ec9e4\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"32448a76-e332-4fcc-8a89-2a43e98ec9e4\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(add_indicator=True, missing_values=0)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"bd6a7673-4537-4fc3-94af-507634e8f11b\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"bd6a7673-4537-4fc3-94af-507634e8f11b\">MinMaxScaler</label><div class=\"sk-toggleable__content\"><pre>MinMaxScaler()</pre></div></div></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"36e67e4d-eddc-4f34-bddf-bc15f35cd4cc\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"36e67e4d-eddc-4f34-bddf-bc15f35cd4cc\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression()</pre></div></div></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "RandomizedSearchCV(cv=5,\n",
       "                   estimator=Pipeline(steps=[('preprocessing',\n",
       "                                              ColumnTransformer(transformers=[('categorical',\n",
       "                                                                               Pipeline(steps=[('imputer',\n",
       "                                                                                                SimpleImputer(missing_values=0,\n",
       "                                                                                                              strategy='most_frequent')),\n",
       "                                                                                               ('encoder',\n",
       "                                                                                                OneHotEncoder(handle_unknown='ignore'))]),\n",
       "                                                                               impression        False\n",
       "url_hash          False\n",
       "ad_id             False\n",
       "advertiser_id     False\n",
       "depth             False\n",
       "position          False\n",
       "query_id          False\n",
       "keywor...\n",
       "                                                                                                MinMaxScaler())]),\n",
       "                                                                               impression        True\n",
       "url_hash          True\n",
       "ad_id             True\n",
       "advertiser_id     True\n",
       "depth             True\n",
       "position          True\n",
       "query_id          True\n",
       "keyword_id        True\n",
       "title_id          True\n",
       "description_id    True\n",
       "user_id           True\n",
       "dtype: bool)])),\n",
       "                                             ('lm', LogisticRegression())]),\n",
       "                   n_iter=50, n_jobs=-1,\n",
       "                   param_distributions={'lm__C': [0.1, 0.2, 0.3, 0.5, 0.6, 0.7,\n",
       "                                                  0.9, 1],\n",
       "                                        'lm__penalty': ['l1', 'l2'],\n",
       "                                        'lm__solver': ['newton-cg', 'lbfgs',\n",
       "                                                       'liblinear', 'sag']})"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_rand = RandomizedSearchCV(estimator = pipe_lm,\n",
    "                                   param_distributions=param_grid_lr,\n",
    "                                   n_iter = 50,\n",
    "                                   cv=5,\n",
    "                                   n_jobs=-1)\n",
    "clf_rand.fit(X_train, y_train) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('lm__C', 0.1),\n",
       " ('lm__class_weight', None),\n",
       " ('lm__dual', False),\n",
       " ('lm__fit_intercept', True),\n",
       " ('lm__intercept_scaling', 1),\n",
       " ('lm__l1_ratio', None),\n",
       " ('lm__max_iter', 100),\n",
       " ('lm__multi_class', 'auto'),\n",
       " ('lm__n_jobs', None),\n",
       " ('lm__penalty', 'l1'),\n",
       " ('lm__random_state', None),\n",
       " ('lm__solver', 'liblinear'),\n",
       " ('lm__tol', 0.0001),\n",
       " ('lm__verbose', 0)]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(clf_rand.best_estimator_.get_params().items())[-15:-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Accuracy score for is: 0.832\n",
      "The F1 score for is: 0.908\n"
     ]
    }
   ],
   "source": [
    "y_pred = clf_rand.predict(X_test)\n",
    "print_scores(y_test,y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Random Forest Classifier with Grid Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>div.sk-top-container {color: black;background-color: white;}div.sk-toggleable {background-color: white;}label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.2em 0.3em;box-sizing: border-box;text-align: center;}div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}div.sk-estimator {font-family: monospace;background-color: #f0f8ff;margin: 0.25em 0.25em;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;}div.sk-estimator:hover {background-color: #d4ebff;}div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;}div.sk-item {z-index: 1;}div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}div.sk-parallel-item:only-child::after {width: 0;}div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0.2em;box-sizing: border-box;padding-bottom: 0.1em;background-color: white;position: relative;}div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}div.sk-label-container {position: relative;z-index: 2;text-align: center;}div.sk-container {display: inline-block;position: relative;}</style><div class=\"sk-top-container\"><div class=\"sk-container\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"f08ec643-3f87-45dc-84da-2474a48a76e9\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"f08ec643-3f87-45dc-84da-2474a48a76e9\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[('preprocessing',\n",
       "                 ColumnTransformer(transformers=[('categorical',\n",
       "                                                  Pipeline(steps=[('imputer',\n",
       "                                                                   SimpleImputer(missing_values=0,\n",
       "                                                                                 strategy='most_frequent')),\n",
       "                                                                  ('encoder',\n",
       "                                                                   OneHotEncoder(handle_unknown='ignore'))]),\n",
       "                                                  impression        False\n",
       "url_hash          False\n",
       "ad_id             False\n",
       "advertiser_id     False\n",
       "depth             False\n",
       "position          False\n",
       "query_id          False\n",
       "keyword_id        False\n",
       "title_id          False\n",
       "description_id    False\n",
       "user_id           False\n",
       "dtype: bool),\n",
       "                                                 ('continuous',\n",
       "                                                  Pipeline(steps=[('imputer',\n",
       "                                                                   SimpleImputer(add_indicator=True,\n",
       "                                                                                 missing_values=0)),\n",
       "                                                                  ('scaler',\n",
       "                                                                   MinMaxScaler())]),\n",
       "                                                  impression        True\n",
       "url_hash          True\n",
       "ad_id             True\n",
       "advertiser_id     True\n",
       "depth             True\n",
       "position          True\n",
       "query_id          True\n",
       "keyword_id        True\n",
       "title_id          True\n",
       "description_id    True\n",
       "user_id           True\n",
       "dtype: bool)])),\n",
       "                ('rf', RandomForestClassifier())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"64dc7454-c302-4947-ae1b-04faae998ba5\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"64dc7454-c302-4947-ae1b-04faae998ba5\">preprocessing: ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[('categorical',\n",
       "                                 Pipeline(steps=[('imputer',\n",
       "                                                  SimpleImputer(missing_values=0,\n",
       "                                                                strategy='most_frequent')),\n",
       "                                                 ('encoder',\n",
       "                                                  OneHotEncoder(handle_unknown='ignore'))]),\n",
       "                                 impression        False\n",
       "url_hash          False\n",
       "ad_id             False\n",
       "advertiser_id     False\n",
       "depth             False\n",
       "position          False\n",
       "query_id          False\n",
       "keyword_id        False\n",
       "title_id          False\n",
       "description_id    False\n",
       "user_id           False\n",
       "dtype: bool),\n",
       "                                ('continuous',\n",
       "                                 Pipeline(steps=[('imputer',\n",
       "                                                  SimpleImputer(add_indicator=True,\n",
       "                                                                missing_values=0)),\n",
       "                                                 ('scaler', MinMaxScaler())]),\n",
       "                                 impression        True\n",
       "url_hash          True\n",
       "ad_id             True\n",
       "advertiser_id     True\n",
       "depth             True\n",
       "position          True\n",
       "query_id          True\n",
       "keyword_id        True\n",
       "title_id          True\n",
       "description_id    True\n",
       "user_id           True\n",
       "dtype: bool)])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"eea2780b-2e98-427f-b083-28593283ff0b\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"eea2780b-2e98-427f-b083-28593283ff0b\">categorical</label><div class=\"sk-toggleable__content\"><pre>impression        False\n",
       "url_hash          False\n",
       "ad_id             False\n",
       "advertiser_id     False\n",
       "depth             False\n",
       "position          False\n",
       "query_id          False\n",
       "keyword_id        False\n",
       "title_id          False\n",
       "description_id    False\n",
       "user_id           False\n",
       "dtype: bool</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"94438aad-b484-43e6-95a4-5778c856814d\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"94438aad-b484-43e6-95a4-5778c856814d\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(missing_values=0, strategy='most_frequent')</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"5fbc71de-b8f0-4272-a371-9f785f58933e\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"5fbc71de-b8f0-4272-a371-9f785f58933e\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown='ignore')</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"05aab2f8-8e69-4248-9d0c-a393723a2956\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"05aab2f8-8e69-4248-9d0c-a393723a2956\">continuous</label><div class=\"sk-toggleable__content\"><pre>impression        True\n",
       "url_hash          True\n",
       "ad_id             True\n",
       "advertiser_id     True\n",
       "depth             True\n",
       "position          True\n",
       "query_id          True\n",
       "keyword_id        True\n",
       "title_id          True\n",
       "description_id    True\n",
       "user_id           True\n",
       "dtype: bool</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"a409973f-eb9e-47a2-bf26-5ca23a85e3b2\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"a409973f-eb9e-47a2-bf26-5ca23a85e3b2\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(add_indicator=True, missing_values=0)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"9aab657d-fa96-4ffe-b034-5ab0e554ea97\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"9aab657d-fa96-4ffe-b034-5ab0e554ea97\">MinMaxScaler</label><div class=\"sk-toggleable__content\"><pre>MinMaxScaler()</pre></div></div></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"7bd77ab8-b464-45e4-9840-2dca2993e00d\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"7bd77ab8-b464-45e4-9840-2dca2993e00d\">RandomForestClassifier</label><div class=\"sk-toggleable__content\"><pre>RandomForestClassifier()</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('preprocessing',\n",
       "                 ColumnTransformer(transformers=[('categorical',\n",
       "                                                  Pipeline(steps=[('imputer',\n",
       "                                                                   SimpleImputer(missing_values=0,\n",
       "                                                                                 strategy='most_frequent')),\n",
       "                                                                  ('encoder',\n",
       "                                                                   OneHotEncoder(handle_unknown='ignore'))]),\n",
       "                                                  impression        False\n",
       "url_hash          False\n",
       "ad_id             False\n",
       "advertiser_id     False\n",
       "depth             False\n",
       "position          False\n",
       "query_id          False\n",
       "keyword_id        False\n",
       "title_id          False\n",
       "description_id    False\n",
       "user_id           False\n",
       "dtype: bool),\n",
       "                                                 ('continuous',\n",
       "                                                  Pipeline(steps=[('imputer',\n",
       "                                                                   SimpleImputer(add_indicator=True,\n",
       "                                                                                 missing_values=0)),\n",
       "                                                                  ('scaler',\n",
       "                                                                   MinMaxScaler())]),\n",
       "                                                  impression        True\n",
       "url_hash          True\n",
       "ad_id             True\n",
       "advertiser_id     True\n",
       "depth             True\n",
       "position          True\n",
       "query_id          True\n",
       "keyword_id        True\n",
       "title_id          True\n",
       "description_id    True\n",
       "user_id           True\n",
       "dtype: bool)])),\n",
       "                ('rf', RandomForestClassifier())])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "con_pipe = Pipeline([('imputer', SimpleImputer(missing_values=0 ,strategy='mean', add_indicator=True)), \n",
    "                        ('scaler', MinMaxScaler())])\n",
    "\n",
    "cat_pipe = Pipeline([('imputer', SimpleImputer(missing_values=0 , strategy='most_frequent')),\n",
    "                        ('encoder', OneHotEncoder(handle_unknown='ignore'))])\n",
    "\n",
    "preprocessing = ColumnTransformer([('categorical', cat_pipe,  (X_train.dtypes == object)), \n",
    "                                    ('continuous',  con_pipe, ~(X_train.dtypes == object))])\n",
    "\n",
    "pipe_rf = Pipeline([('preprocessing', preprocessing), \n",
    "                    ('rf', RandomForestClassifier())])\n",
    "\n",
    "pipe_rf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_grid_rf = {'rf__max_features': [\"sqrt\", \"log2\"], # Define the number of features to be used\n",
    "              'rf__min_samples_leaf': [3, 4, 5, 6, 7, 8, 9, 10], # Number of samples in leaf\n",
    "              'rf__max_depth': [None, 1, 2, 3, 5, 7, 8, 9, 10], # Tree depth\n",
    "              'rf__n_estimators': [100, 120, 140, 160, 180] # Number of sub trees\n",
    "             }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>div.sk-top-container {color: black;background-color: white;}div.sk-toggleable {background-color: white;}label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.2em 0.3em;box-sizing: border-box;text-align: center;}div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}div.sk-estimator {font-family: monospace;background-color: #f0f8ff;margin: 0.25em 0.25em;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;}div.sk-estimator:hover {background-color: #d4ebff;}div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;}div.sk-item {z-index: 1;}div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}div.sk-parallel-item:only-child::after {width: 0;}div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0.2em;box-sizing: border-box;padding-bottom: 0.1em;background-color: white;position: relative;}div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}div.sk-label-container {position: relative;z-index: 2;text-align: center;}div.sk-container {display: inline-block;position: relative;}</style><div class=\"sk-top-container\"><div class=\"sk-container\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"6bbc263b-ec9c-4aff-8831-905a4a5f04f7\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"6bbc263b-ec9c-4aff-8831-905a4a5f04f7\">RandomizedSearchCV</label><div class=\"sk-toggleable__content\"><pre>RandomizedSearchCV(cv=5,\n",
       "                   estimator=Pipeline(steps=[('preprocessing',\n",
       "                                              ColumnTransformer(transformers=[('categorical',\n",
       "                                                                               Pipeline(steps=[('imputer',\n",
       "                                                                                                SimpleImputer(missing_values=0,\n",
       "                                                                                                              strategy='most_frequent')),\n",
       "                                                                                               ('encoder',\n",
       "                                                                                                OneHotEncoder(handle_unknown='ignore'))]),\n",
       "                                                                               impression        False\n",
       "url_hash          False\n",
       "ad_id             False\n",
       "advertiser_id     False\n",
       "depth             False\n",
       "position          False\n",
       "query_id          False\n",
       "keywor...\n",
       "advertiser_id     True\n",
       "depth             True\n",
       "position          True\n",
       "query_id          True\n",
       "keyword_id        True\n",
       "title_id          True\n",
       "description_id    True\n",
       "user_id           True\n",
       "dtype: bool)])),\n",
       "                                             ('rf', RandomForestClassifier())]),\n",
       "                   n_iter=50, n_jobs=-1,\n",
       "                   param_distributions={'rf__max_depth': [None, 1, 2, 3, 5, 7,\n",
       "                                                          8, 9, 10],\n",
       "                                        'rf__max_features': ['sqrt', 'log2'],\n",
       "                                        'rf__min_samples_leaf': [3, 4, 5, 6, 7,\n",
       "                                                                 8, 9, 10],\n",
       "                                        'rf__n_estimators': [100, 120, 140, 160,\n",
       "                                                             180]})</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"18ba0127-dc13-40ee-874f-e84f14431cdf\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"18ba0127-dc13-40ee-874f-e84f14431cdf\">preprocessing: ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[('categorical',\n",
       "                                 Pipeline(steps=[('imputer',\n",
       "                                                  SimpleImputer(missing_values=0,\n",
       "                                                                strategy='most_frequent')),\n",
       "                                                 ('encoder',\n",
       "                                                  OneHotEncoder(handle_unknown='ignore'))]),\n",
       "                                 impression        False\n",
       "url_hash          False\n",
       "ad_id             False\n",
       "advertiser_id     False\n",
       "depth             False\n",
       "position          False\n",
       "query_id          False\n",
       "keyword_id        False\n",
       "title_id          False\n",
       "description_id    False\n",
       "user_id           False\n",
       "dtype: bool),\n",
       "                                ('continuous',\n",
       "                                 Pipeline(steps=[('imputer',\n",
       "                                                  SimpleImputer(add_indicator=True,\n",
       "                                                                missing_values=0)),\n",
       "                                                 ('scaler', MinMaxScaler())]),\n",
       "                                 impression        True\n",
       "url_hash          True\n",
       "ad_id             True\n",
       "advertiser_id     True\n",
       "depth             True\n",
       "position          True\n",
       "query_id          True\n",
       "keyword_id        True\n",
       "title_id          True\n",
       "description_id    True\n",
       "user_id           True\n",
       "dtype: bool)])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"216e6402-d7f6-418e-9d85-e3331be5916f\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"216e6402-d7f6-418e-9d85-e3331be5916f\">categorical</label><div class=\"sk-toggleable__content\"><pre>impression        False\n",
       "url_hash          False\n",
       "ad_id             False\n",
       "advertiser_id     False\n",
       "depth             False\n",
       "position          False\n",
       "query_id          False\n",
       "keyword_id        False\n",
       "title_id          False\n",
       "description_id    False\n",
       "user_id           False\n",
       "dtype: bool</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"fda59dfc-683e-437b-bfa2-49c367d2ef5d\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"fda59dfc-683e-437b-bfa2-49c367d2ef5d\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(missing_values=0, strategy='most_frequent')</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"44b2851f-72da-4f2e-a4de-63a330da4b87\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"44b2851f-72da-4f2e-a4de-63a330da4b87\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown='ignore')</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"b2e72fec-a4e8-4507-93d4-551354c40237\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"b2e72fec-a4e8-4507-93d4-551354c40237\">continuous</label><div class=\"sk-toggleable__content\"><pre>impression        True\n",
       "url_hash          True\n",
       "ad_id             True\n",
       "advertiser_id     True\n",
       "depth             True\n",
       "position          True\n",
       "query_id          True\n",
       "keyword_id        True\n",
       "title_id          True\n",
       "description_id    True\n",
       "user_id           True\n",
       "dtype: bool</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"853fde26-3896-420d-8293-a4deeb0ee734\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"853fde26-3896-420d-8293-a4deeb0ee734\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(add_indicator=True, missing_values=0)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"4d19e657-fc42-4aa6-a00d-d12d803c8263\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"4d19e657-fc42-4aa6-a00d-d12d803c8263\">MinMaxScaler</label><div class=\"sk-toggleable__content\"><pre>MinMaxScaler()</pre></div></div></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"f328f9b6-4215-40a3-883f-882157122ed2\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"f328f9b6-4215-40a3-883f-882157122ed2\">RandomForestClassifier</label><div class=\"sk-toggleable__content\"><pre>RandomForestClassifier()</pre></div></div></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "RandomizedSearchCV(cv=5,\n",
       "                   estimator=Pipeline(steps=[('preprocessing',\n",
       "                                              ColumnTransformer(transformers=[('categorical',\n",
       "                                                                               Pipeline(steps=[('imputer',\n",
       "                                                                                                SimpleImputer(missing_values=0,\n",
       "                                                                                                              strategy='most_frequent')),\n",
       "                                                                                               ('encoder',\n",
       "                                                                                                OneHotEncoder(handle_unknown='ignore'))]),\n",
       "                                                                               impression        False\n",
       "url_hash          False\n",
       "ad_id             False\n",
       "advertiser_id     False\n",
       "depth             False\n",
       "position          False\n",
       "query_id          False\n",
       "keywor...\n",
       "advertiser_id     True\n",
       "depth             True\n",
       "position          True\n",
       "query_id          True\n",
       "keyword_id        True\n",
       "title_id          True\n",
       "description_id    True\n",
       "user_id           True\n",
       "dtype: bool)])),\n",
       "                                             ('rf', RandomForestClassifier())]),\n",
       "                   n_iter=50, n_jobs=-1,\n",
       "                   param_distributions={'rf__max_depth': [None, 1, 2, 3, 5, 7,\n",
       "                                                          8, 9, 10],\n",
       "                                        'rf__max_features': ['sqrt', 'log2'],\n",
       "                                        'rf__min_samples_leaf': [3, 4, 5, 6, 7,\n",
       "                                                                 8, 9, 10],\n",
       "                                        'rf__n_estimators': [100, 120, 140, 160,\n",
       "                                                             180]})"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf_rand = RandomizedSearchCV(estimator = pipe_rf,\n",
    "                                   param_distributions=param_grid_rf,\n",
    "                                   n_iter = 50,\n",
    "                                   cv=5,\n",
    "                                   n_jobs=-1)\n",
    "best_model = clf_rand.fit(X_train, y_train) \n",
    "best_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>div.sk-top-container {color: black;background-color: white;}div.sk-toggleable {background-color: white;}label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.2em 0.3em;box-sizing: border-box;text-align: center;}div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}div.sk-estimator {font-family: monospace;background-color: #f0f8ff;margin: 0.25em 0.25em;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;}div.sk-estimator:hover {background-color: #d4ebff;}div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;}div.sk-item {z-index: 1;}div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}div.sk-parallel-item:only-child::after {width: 0;}div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0.2em;box-sizing: border-box;padding-bottom: 0.1em;background-color: white;position: relative;}div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}div.sk-label-container {position: relative;z-index: 2;text-align: center;}div.sk-container {display: inline-block;position: relative;}</style><div class=\"sk-top-container\"><div class=\"sk-container\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"126bceb5-0d53-4cdb-b4d9-4116dd27dab2\" type=\"checkbox\" checked><label class=\"sk-toggleable__label\" for=\"126bceb5-0d53-4cdb-b4d9-4116dd27dab2\">RandomForestClassifier</label><div class=\"sk-toggleable__content\"><pre>RandomForestClassifier(max_features='sqrt', min_samples_leaf=7,\n",
       "                       n_estimators=160)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "RandomForestClassifier(max_features='sqrt', min_samples_leaf=7,\n",
       "                       n_estimators=160)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_model.best_estimator_.get_params()['rf']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Accuracy score for is: 0.839\n",
      "The F1 score for is: 0.898\n"
     ]
    }
   ],
   "source": [
    "y_pred = clf_rand.predict(X_test)\n",
    "print_scores(y_test,y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Extra Tree Classifier with Grid Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "kfold = KFold(n_splits=10, shuffle=True, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>div.sk-top-container {color: black;background-color: white;}div.sk-toggleable {background-color: white;}label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.2em 0.3em;box-sizing: border-box;text-align: center;}div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}div.sk-estimator {font-family: monospace;background-color: #f0f8ff;margin: 0.25em 0.25em;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;}div.sk-estimator:hover {background-color: #d4ebff;}div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;}div.sk-item {z-index: 1;}div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}div.sk-parallel-item:only-child::after {width: 0;}div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0.2em;box-sizing: border-box;padding-bottom: 0.1em;background-color: white;position: relative;}div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}div.sk-label-container {position: relative;z-index: 2;text-align: center;}div.sk-container {display: inline-block;position: relative;}</style><div class=\"sk-top-container\"><div class=\"sk-container\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"89a7a8f9-0d6f-460f-9665-bfc6f1b51b6f\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"89a7a8f9-0d6f-460f-9665-bfc6f1b51b6f\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[('preprocessing',\n",
       "                 ColumnTransformer(transformers=[('categorical',\n",
       "                                                  Pipeline(steps=[('imputer',\n",
       "                                                                   SimpleImputer(missing_values=0,\n",
       "                                                                                 strategy='most_frequent')),\n",
       "                                                                  ('encoder',\n",
       "                                                                   OneHotEncoder(handle_unknown='ignore'))]),\n",
       "                                                  impression        False\n",
       "url_hash          False\n",
       "ad_id             False\n",
       "advertiser_id     False\n",
       "depth             False\n",
       "position          False\n",
       "query_id          False\n",
       "keyword_id        False\n",
       "title_id          False\n",
       "description_...e\n",
       "user_id           False\n",
       "dtype: bool),\n",
       "                                                 ('continuous',\n",
       "                                                  Pipeline(steps=[('imputer',\n",
       "                                                                   SimpleImputer(add_indicator=True,\n",
       "                                                                                 missing_values=0,\n",
       "                                                                                 strategy='median')),\n",
       "                                                                  ('scaler',\n",
       "                                                                   MinMaxScaler())]),\n",
       "                                                  impression        True\n",
       "url_hash          True\n",
       "ad_id             True\n",
       "advertiser_id     True\n",
       "depth             True\n",
       "position          True\n",
       "query_id          True\n",
       "keyword_id        True\n",
       "title_id          True\n",
       "description_id    True\n",
       "user_id           True\n",
       "dtype: bool)])),\n",
       "                ('xt', ExtraTreesClassifier())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"28404a90-089d-49d5-86d6-d27c8dec03ae\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"28404a90-089d-49d5-86d6-d27c8dec03ae\">preprocessing: ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[('categorical',\n",
       "                                 Pipeline(steps=[('imputer',\n",
       "                                                  SimpleImputer(missing_values=0,\n",
       "                                                                strategy='most_frequent')),\n",
       "                                                 ('encoder',\n",
       "                                                  OneHotEncoder(handle_unknown='ignore'))]),\n",
       "                                 impression        False\n",
       "url_hash          False\n",
       "ad_id             False\n",
       "advertiser_id     False\n",
       "depth             False\n",
       "position          False\n",
       "query_id          False\n",
       "keyword_id        False\n",
       "title_id          False\n",
       "description_id    False\n",
       "user_id           False\n",
       "dtype: bool),\n",
       "                                ('continuous',\n",
       "                                 Pipeline(steps=[('imputer',\n",
       "                                                  SimpleImputer(add_indicator=True,\n",
       "                                                                missing_values=0,\n",
       "                                                                strategy='median')),\n",
       "                                                 ('scaler', MinMaxScaler())]),\n",
       "                                 impression        True\n",
       "url_hash          True\n",
       "ad_id             True\n",
       "advertiser_id     True\n",
       "depth             True\n",
       "position          True\n",
       "query_id          True\n",
       "keyword_id        True\n",
       "title_id          True\n",
       "description_id    True\n",
       "user_id           True\n",
       "dtype: bool)])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"cfa1f05f-e574-4aca-8a13-b19a97dbe6dd\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"cfa1f05f-e574-4aca-8a13-b19a97dbe6dd\">categorical</label><div class=\"sk-toggleable__content\"><pre>impression        False\n",
       "url_hash          False\n",
       "ad_id             False\n",
       "advertiser_id     False\n",
       "depth             False\n",
       "position          False\n",
       "query_id          False\n",
       "keyword_id        False\n",
       "title_id          False\n",
       "description_id    False\n",
       "user_id           False\n",
       "dtype: bool</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"920668a5-36de-47c9-a279-c889e1cad12d\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"920668a5-36de-47c9-a279-c889e1cad12d\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(missing_values=0, strategy='most_frequent')</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"3944f2c7-0542-4ac9-9c7d-63715d7d87cd\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"3944f2c7-0542-4ac9-9c7d-63715d7d87cd\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown='ignore')</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"106966e3-c6c7-41f7-8cc8-740f6e7185b5\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"106966e3-c6c7-41f7-8cc8-740f6e7185b5\">continuous</label><div class=\"sk-toggleable__content\"><pre>impression        True\n",
       "url_hash          True\n",
       "ad_id             True\n",
       "advertiser_id     True\n",
       "depth             True\n",
       "position          True\n",
       "query_id          True\n",
       "keyword_id        True\n",
       "title_id          True\n",
       "description_id    True\n",
       "user_id           True\n",
       "dtype: bool</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"0e587c34-f53a-45e1-b64e-fcafbb29557b\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"0e587c34-f53a-45e1-b64e-fcafbb29557b\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(add_indicator=True, missing_values=0, strategy='median')</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"8d479efc-5c5e-4fa6-8435-148e5b4cc959\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"8d479efc-5c5e-4fa6-8435-148e5b4cc959\">MinMaxScaler</label><div class=\"sk-toggleable__content\"><pre>MinMaxScaler()</pre></div></div></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"d38a6aea-6a46-4d45-bd5f-0bbe8eeb7a91\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"d38a6aea-6a46-4d45-bd5f-0bbe8eeb7a91\">ExtraTreesClassifier</label><div class=\"sk-toggleable__content\"><pre>ExtraTreesClassifier()</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('preprocessing',\n",
       "                 ColumnTransformer(transformers=[('categorical',\n",
       "                                                  Pipeline(steps=[('imputer',\n",
       "                                                                   SimpleImputer(missing_values=0,\n",
       "                                                                                 strategy='most_frequent')),\n",
       "                                                                  ('encoder',\n",
       "                                                                   OneHotEncoder(handle_unknown='ignore'))]),\n",
       "                                                  impression        False\n",
       "url_hash          False\n",
       "ad_id             False\n",
       "advertiser_id     False\n",
       "depth             False\n",
       "position          False\n",
       "query_id          False\n",
       "keyword_id        False\n",
       "title_id          False\n",
       "description_...e\n",
       "user_id           False\n",
       "dtype: bool),\n",
       "                                                 ('continuous',\n",
       "                                                  Pipeline(steps=[('imputer',\n",
       "                                                                   SimpleImputer(add_indicator=True,\n",
       "                                                                                 missing_values=0,\n",
       "                                                                                 strategy='median')),\n",
       "                                                                  ('scaler',\n",
       "                                                                   MinMaxScaler())]),\n",
       "                                                  impression        True\n",
       "url_hash          True\n",
       "ad_id             True\n",
       "advertiser_id     True\n",
       "depth             True\n",
       "position          True\n",
       "query_id          True\n",
       "keyword_id        True\n",
       "title_id          True\n",
       "description_id    True\n",
       "user_id           True\n",
       "dtype: bool)])),\n",
       "                ('xt', ExtraTreesClassifier())])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "con_pipe = Pipeline([('imputer', SimpleImputer(missing_values=0 ,strategy='median', add_indicator=True)), \n",
    "                        ('scaler', MinMaxScaler())])\n",
    "\n",
    "cat_pipe = Pipeline([('imputer', SimpleImputer(missing_values=0 , strategy='most_frequent')),\n",
    "                        ('encoder', OneHotEncoder(handle_unknown='ignore'))])\n",
    "\n",
    "preprocessing = ColumnTransformer([('categorical', cat_pipe,  (X_train.dtypes == object)), \n",
    "                                    ('continuous',  con_pipe, ~(X_train.dtypes == object))])\n",
    "\n",
    "pipe_xt = Pipeline([('preprocessing', preprocessing), \n",
    "                    ('xt', ExtraTreesClassifier())])\n",
    "\n",
    "pipe_xt.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "scoring = {'acc': 'accuracy',\n",
    "           'prec_macro': 'precision_macro',\n",
    "           'rec_micro': 'recall_macro',\n",
    "           \"f1\":\"f1_macro\"}\n",
    "scores = cross_validate(pipe_xt, \n",
    "                          X_train,\n",
    "                          y_train, \n",
    "                          cv=kfold,\n",
    "                          scoring=scoring)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The mean training test accuracy - 0.825\n",
      "The mean training test F1 Score - 0.536\n"
     ]
    }
   ],
   "source": [
    "sc_acc = scores[\"test_acc\"].mean()\n",
    "print(f\"The mean training test accuracy - {sc_acc:.3f}\")\n",
    "sc_f = scores[\"test_f1\"].mean()\n",
    "print(f\"The mean training test F1 Score - {sc_f:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To wrap up, Decision Tree model with random grid search has the highest accuracy score. We wanted to deterimne how accurate are our predictions by using accuracy metric and also f1 score is used which is a combined mesaure of precision and recall. We increased our scores slightly. For future work boosting and deep learning methods could be used for enhancement with more cpu power."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Models**|**Hyperparameters**|**accuracy\\_score**|**f1\\_score**\n",
    ":-----:|:-----:|:-----:|:-----:\n",
    "RandomForestClassifier|sklearn default|0.833|0.881\n",
    "LogisticRegression with Randomized Grid Search|See notebook|0.832|0.908\n",
    "RandomForestClassifier with Randomized Grid Search|See notebook|0.839|0.898\n",
    "ExtraTreesClassifier with 5 fold CV|sklearn default|0.825|0.536"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}