Cost-Sensitive Multilabel Classification .ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is multilabel classification?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Imagine a paragraph from any article and one want to associate a tag to it for example `Science and Tech`, `crime`\n",
    "`Economics` etc. \n",
    "Now, one article can be assigned multilabels and each label is correct in some sense. We can also weight how each label is correct.\n",
    "\n",
    "This task of predicting most suitable label fall under multilabel classification task. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We are going to use Vowpal Wabbit Library because of it's online learning and high speed functionality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from vowpalwabbit import pyvw\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.preprocessing import MultiLabelBinarizer\n",
    "import re\n",
    "import random"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use `BibTex` data for this specific task which has `159` tags available for each dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download BibTex data from http://manikvarma.org/downloads/XC/XMLRepository.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_data(file_name):\n",
    "    features = list()\n",
    "    labels = list()\n",
    "    with open(file_name, 'rt') as f:\n",
    "        f.readline()\n",
    "        for l in f:\n",
    "            if bool(re.search(\"^[0-9]\", l)):\n",
    "                g = re.search(\"^(([0-9]{1,2},?)+)\\s(.*)$\", l)\n",
    "                labels.append([int(i) for i in g.group(1).split(\",\")])\n",
    "                features.append(eval(\"{\" + re.sub(\"\\s\", \",\", g.group(3)) + \"}\"))\n",
    "            else:\n",
    "                l = l.strip()\n",
    "                labels.append([])\n",
    "                features.append(eval(\"{\" + re.sub(\"\\s\", \",\", l) + \"}\"))\n",
    "#     features_ = pd.DataFrame.from_dict(features).fillna(0)\n",
    "    df = pd.DataFrame.from_dict(features).fillna(0)\n",
    "    mlb = MultiLabelBinarizer()\n",
    "    y = mlb.fit_transform(labels)\n",
    "    return df, y\n",
    "\n",
    "df, y = parse_data(\"data/Bibtex/Bibtex_data.txt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this data, there are total 159 possible labels with total data size of 7394. Total 1836 number of binary features are present in this data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train, y_train = df[:-1000], y[:-1000]\n",
    "df_test, y_test = df[-1000:], y[-1000:]\n",
    "# splitting data manually into train and test split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "actions = np.arange(1, y.shape[1]+1, 1) # Number of actions/label you can choose for one data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert to Vowpal Wabbit Format\n",
    "def make_example(index):\n",
    "    example_string = \"\"\n",
    "    for i in range(y.shape[1]):\n",
    "        # cost for corresponding label(1-label)\n",
    "        example_string += str(i+1) + \":\" + str(1.0 - y[index][i]) + \" \"\n",
    "    return example_string[:-1]\n",
    "def make_example_(index):\n",
    "    example_string = \"\"\n",
    "    # features for corresponding example\n",
    "    for feature in np.array(np.where(df.iloc[index] == 1))[0]:\n",
    "        example_string += str(feature) + \":1 \"\n",
    "    return example_string[:-1]\n",
    "def make_example_vw(index):\n",
    "    return make_example(index) + \" | \" + make_example_(index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining model to be learned\n",
    "model = pyvw.vw(\"--csoaa 159 -q UA\")\n",
    "# Here we are using csoaa, which means \"cost-sensitive one against all\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can look more about Vowpal Wabbit input format and model information on [webpage](https://vowpalwabbit.org/tutorials.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Online Learning loop\n",
    "for i in range(df_train.shape[0]):\n",
    "    i = i%df_train.shape[0]\n",
    "    train_example = make_example_vw(i)\n",
    "    model.learn(train_example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def prediction_accuracy(model):\n",
    "    correct = 0\n",
    "    for i in range(df_train.shape[0], df.shape[0]): # here iterating over test examples\n",
    "        test_example = make_example_vw(i)\n",
    "        ans = model.predict(test_example)\n",
    "        if ans-1 in np.array(np.where(y[i] == 1))[0]:\n",
    "            # as model predict 1 indexing classes, for check ans -1\n",
    "            correct += 1\n",
    "    return correct/df_test.shape[0]*100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "60.8"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prediction_accuracy(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's decent accuracy considering we just iterated over every example once that too in online fashion. Also possible label for each example from 159 dimensional label space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's save data in VW format for future easy use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# saving data in VW format\n",
    "with open(\"data/bibtex_data.vw\", \"w\") as f:\n",
    "    for i in range(df.shape[0]):\n",
    "        train_example = make_example(i) + \" | \" + make_example_(i)\n",
    "        f.write(train_example)\n",
    "        f.write(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's try command line tool for proper diagnosis and see blazing speed for training.\n",
    "You need to install vowpalwabbit command line tool for following section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/bin/sh: 1: vw: not found\r\n"
     ]
    }
   ],
   "source": [
    "!~/vowpal_wabbit/build/vowpalwabbit/vw -d \"data/bibtex_data.vw\" --csoaa 159 -f model.vw"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "predictions = predictions.txt\n",
      "Num weight bits = 18\n",
      "learning rate = 0.5\n",
      "initial_t = 0\n",
      "power_t = 0.5\n",
      "using no cache\n",
      "Reading datafile = data/bibtex_data.vw\n",
      "num sources = 1\n",
      "average  since         example        example  current  current  current\n",
      "loss     last          counter         weight    label  predict features\n",
      "0.000000 0.000000            1            1.0    known       76       73\n",
      "0.000000 0.000000            2            2.0    known      132        7\n",
      "0.250000 0.500000            4            4.0    known       15       59\n",
      "0.125000 0.000000            8            8.0    known      130       46\n",
      "0.187500 0.250000           16           16.0    known       53       85\n",
      "0.093750 0.000000           32           32.0    known      157       90\n",
      "0.109375 0.125000           64           64.0    known       36       98\n",
      "0.117188 0.125000          128          128.0    known      123       18\n",
      "0.093750 0.070312          256          256.0    known      156       82\n",
      "0.115234 0.136719          512          512.0    known       99       14\n",
      "0.115234 0.115234         1024         1024.0    known      112       54\n",
      "0.123047 0.130859         2048         2048.0    known       53       72\n",
      "0.146973 0.170898         4096         4096.0    known       45       60\n",
      "\n",
      "finished run\n",
      "number of examples = 7394\n",
      "weighted example sum = 7394.000000\n",
      "weighted label sum = 0.000000\n",
      "average loss = 0.163376\n",
      "total feature number = 515048\n"
     ]
    }
   ],
   "source": [
    "!~/vowpal_wabbit/build/vowpalwabbit/vw -d data/bibtex_data.vw -i model.vw -p predictions.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "76\r\n",
      "132\r\n",
      "27\r\n"
     ]
    }
   ],
   "source": [
    "!cat predictions.txt | head -n3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## You can play with different arguments in command line mode\n",
    "Go to [this](https://github.com/VowpalWabbit/vowpal_wabbit/wiki/Command-line-arguments) for more help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let us cbify this problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is specific feature of Vowpal Wabbit which allow reductions on complex problem.\n",
    "To cbify any problem, we need cost for chosen action and probablities to chose an action.\n",
    "In Contextual Bandits, labels are represented by actions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "CORRECT_TAG = 0.0\n",
    "INCORRECT_TAG = -1.0\n",
    "\n",
    "def get_cost(cont_ind, action):\n",
    "    correct_tags = np.array(np.where(y[cont_ind] == 1)[0])\n",
    "    if(action in correct_tags):\n",
    "        return CORRECT_TAG\n",
    "    else:\n",
    "        return INCORRECT_TAG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To return the action with max confidence score\n",
    "def sample_custom_pmf(pmf):\n",
    "    total = sum(pmf)\n",
    "    scale = 1 / total\n",
    "    pmf = [x * scale for x in pmf]\n",
    "    draw = random.random()\n",
    "    sum_prob = 0.0\n",
    "    for index, prob in enumerate(pmf):\n",
    "        sum_prob += prob\n",
    "        if(sum_prob > draw):\n",
    "            return index, prob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get action prediction from CB algorithm\n",
    "def get_action(vw, cont_ind, actions):\n",
    "    vw_text_example = make_example_vw(cont_ind)\n",
    "    pmf = vw.predict(vw_text_example)\n",
    "    chosen_action_index, prob = sample_custom_pmf(pmf)\n",
    "    return actions[chosen_action_index], prob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert to Vowpal Wabbit Format\n",
    "def cbmake_example(index, chosen_action, prob):\n",
    "    example_string = \"\"\n",
    "    for i in range(y.shape[1]):\n",
    "        if( (i+1) == chosen_action):\n",
    "            example_string += str(i+1) + \":\" + str(1.0 - y[index][i]) + \":\" + str(prob) + \" \"\n",
    "        else:\n",
    "            example_string += str(i+1) + \":\" + str(1.0 - y[index][i]) + \" \"\n",
    "    return example_string[:-1]\n",
    "def cbmake_example_(index):\n",
    "    example_string = \"\"\n",
    "    for feature in np.array(np.where(df.iloc[index] == 1))[0]:\n",
    "        example_string += str(feature) + \":1 \"\n",
    "    return example_string[:-1]\n",
    "def cbmake_example_vw(index, chosen_action, prob):\n",
    "    return cbmake_example(index, chosen_action, prob) + \" | \" + cbmake_example_(index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "def to_vw_example_format(cont_ind, actions, cb_label=None):\n",
    "    if cb_label:\n",
    "        action, cost, prob = cb_label\n",
    "        return cbmake_example_vw(cont_ind, action, prob)\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_simulation(vw, num_iterations, actions, df, do_learn = True):\n",
    "    cost_sum = 0.\n",
    "    ctr = []\n",
    "\n",
    "    for i in range(num_iterations):\n",
    "        \n",
    "        # 1. Pass context to vw to get an action\n",
    "        cont_ind = i\n",
    "        action, prob = get_action(vw, cont_ind, actions)\n",
    "        # 2. Get cost of the action we chose\n",
    "        cost = get_cost(cont_ind, action)\n",
    "        cost_sum += cost\n",
    "\n",
    "        if do_learn:\n",
    "            # 3. Inform VW of what happened so we can learn from it\n",
    "            vw_format = vw.parse(to_vw_example_format(cont_ind, actions, (action, cost, prob)),pyvw.vw.lContextualBandit)\n",
    "#             print(to_vw_example_format(cont_ind, actions, (action, cost, prob)))\n",
    "            vw.learn(vw_format)\n",
    "\n",
    "        # We negate this so that on the plot instead of minimizing cost, we are maximizing reward\n",
    "        ctr.append(-1*cost_sum/(i+1))\n",
    "\n",
    "    return ctr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_ctr(num_iterations, ctr):\n",
    "    plt.plot(range(1,num_iterations+1), ctr)\n",
    "    plt.xlabel('num_iterations', fontsize=14)\n",
    "    plt.ylabel('ctr', fontsize=14)\n",
    "    plt.ylim([0,1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEOCAYAAAB8aOvdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAGhpJREFUeJzt3X2UZHdd5/H3p6ofZjIzSTCZxJjEJEgwRA8r2TGCIrCgmKCb7FldSVZWRJbsg/HAEdeTLCtqRFEBFRYQoouwqETEXR3ZsIE1IGddwEyAQB5MGGIgEyEzk0Am89hP3/3j3p6p6fTN9EB3dQ3zfp1Tp+/93V/d+62q7vrUvbf6d1NVSJK0mN5qFyBJGl2GhCSpkyEhSepkSEiSOhkSkqROhoQkqdPQQiLJO5JsT3J7x/IkeVOSrUk+k+SiYdUmSVrcMPck3glc8jjLLwXOb29XAb83hJokSY9jaCFRVR8FHn6cLpcD/70aHwdOTnLGcKqTJC1mbLULGHAmcP/A/La27UsLOya5imZvg3Xr1v3TCy644Kg3tnP3Ab78yP7O5fP/h37q+kkAzjhpzVFvQ5JG1a233rqzqjYeqd8ohcSSVdX1wPUAmzZtqi1btiz7Nn7/o/fyazfedXB+J/CGf/VP+OLDe7l35x6+smeKX7n8O/i2jeuXfduStNKSfGEp/UYpJB4Azh6YP6ttWxUve9YT+dff863s2j/Ns1/3EaZm5njln91GAt9y0loe+Oo+nveGv+H/XfNcxnrhkX3TnH/6hqPezv7pWSb6PXq9rMCjkI5f8+PSJSv7t1VVVMFcFXPtz0PzTVsN/Jw/SjFfVRIC9HphcqxHAr2kvXXXP6zHN0ohsRm4OskNwPcAj1TVYw41DdO6yTHWTY7x2V9+Ph+9ZydnnLSGb9u4nvF+eNKrPgDA9/7GzYfd5ylnnMgF37yBU9ZN8LJnPZGt23dzz4OP8rntu/niQ3u5/yt7mZ0rTj5hnC99dT8P7ZkC4JxTTmDNWJ8HH91PP+HfPOMcfuafPYnxfnPaqKrYtX+G7bv2s/3RAzy4az87dx9g7cQY33bqOr6yd5o14z16CWsn+uzaN83aiT57p2aZmytOPmGCibEej+yb4qt7p5mZLSbGekzNzDE53mO832Pd5BgnrR2nn/Do/ml27Z9mz4FZTt0wyXg/VMFYL+w+MHPwtm9qlvWTYyQwM1fMzBbTs3NMzc4xPVPMzM0xO9f8gaxptzM7V8zMznHSCRNs3NAczls30Wd6tpianePA9Gz7s1nP1MwcJ0z0AZidKxJYO97nwMwc0+325rdZBWvGeky325iebf5Q102O0QuM93uM9cLU/P1m5u97aD1zBaesmwBgerapf349p504yclrJ5rHOzt36DHPzTEzW4z1mz/Y+cc+X9/M7BzTc8X0THOf+W01951vP7SOXprnuzj0htPvhTXjzfMw/7YwOdaHwNxcMVvFXPtcN895Mdu2j/d6rJ3oMzHWY+14/3HfyCbHegeXLdaHhMl+D9LUMdYPY73eoe0NbHv+TbOrfbzfO2w7s3OHb3N2br6uIgn9XtrH2K5jYHq26uBjmJ1r3kB7gX6veS7h0GHkwYFNJ8Z6hDD/9r2wL8BEW+fC52q+9pUy0e8x1s9jnpf24fGaf/GdvOjp56xcAUCGNQpskvcAzwFOBR4EfgkYB6iqt6WJwzfTfANqL/CSqjricaSVOtx0JA/tPsAbPnQP+6dn+eYT1/DWj3z+cfuftHacc09dx4OP7OfEtWN8y8lrOeOktezaN83ff3kX69eMc/qGSe7duYet23cfvN+p6ydZN9nnwV372T89t9IPa9kl0E8oDv3hroR+L4z3w8xsMTNXjLdvXGP9QMG+6VmgCbJ5E/0e4/0wPtaE1/z83qlZ9hyYYawNlPk3wenZOXbsPvCYN4VeYKzfY7wXDszM0evl4B/3eNs+Nr+tw9oHpgfqPTAzR9rnLoRer/k51YbN4Ob3Ts2S9vH3e4c+fR6abtqnZuc4MDPH/unZpsaDn1abT6Lz8wAHZuboJ4c+0faan/P9ZtuQa0KsCcvZuaLXC/3k4M9+b366qSc5vL2XJoQHPzX3cmjZwun5cOn3Bj9pt8t7Ofi7Ntg2Xyft8wnNczk/XwUHZmYPfho/+Jl8oG9V8zt18DkaeO4Gn5fDn8/B5RxeJ4NhdSiwZgY+jMyH0J4DM1Rb62LbfO4Fp/HUs07+mv5mktxaVZuO2O9YHyp8tUKiy8N7ppiZneN1N93Nhd9yIk8+fQPnn76ejesnl7xb+Mi+aa77qzv5wkN76PfC6Seu4bQNk83PEycPzp+yfpLPbnuEfi9MjIX903PsOTDDuslmB3Guig1rxnl0/zSP7p/hhIk+J58wwUlrx9lzYIa1E332t2+ej+6fYff+GfZMzbB2vM+Ja8fZsGaMKnhw137WjDd914z3Wd/uYW1YM0a/F/ZNzR58kxrrNW+6zRtuj/7AYbR9U7Mkzd5Ivxe+8NBeZuaK/e2ew+RYr701n3jnp/u9sOfADP1+s/4D03PNp9BFtnOkXfADM7P00qzna9lNryp2H5hpH2sTIh4q1LHIkJAkdVpqSDgshySpkyEhSepkSEiSOhkSkqROhoQkqZMhIUnqZEhIkjoZEpKkToaEJKmTISFJ6mRISJI6GRKSpE6GhCSpkyEhSepkSEiSOhkSkqROhoQkqZMhIUnqZEhIkjoZEpKkToaEJKmTISFJ6mRISJI6GRKSpE6GhCSpkyEhSepkSEiSOhkSkqROhoQkqZMhIUnqZEhIkjoZEpKkToaEJKmTISFJ6jTUkEhySZK7k2xNcs0iy781yYeTfCrJZ5K8YJj1SZION7SQSNIH3gJcClwIXJnkwgXd/gvw3qp6GnAF8NZh1SdJeqxh7klcDGytqnuragq4Abh8QZ8CTmynTwL+cYj1SZIWGGZInAncPzC/rW0b9MvAi5JsA24EfnaxFSW5KsmWJFt27NixErVKkhi9E9dXAu+sqrOAFwDvTvKYGqvq+qraVFWbNm7cOPQiJel4McyQeAA4e2D+rLZt0EuB9wJU1ceANcCpQ6lOkvQYwwyJW4Dzk5yXZILmxPTmBX2+CDwPIMlTaELC40mStEqGFhJVNQNcDdwE3EXzLaY7klyX5LK22yuBlyW5DXgP8FNVVcOqUZJ0uLFhbqyqbqQ5IT3Y9uqB6TuB7xtmTZKkbqN24lqSNEIMCUlSJ0NCktTJkJAkdTIkJEmdDAlJUidDQpLUyZCQJHUyJCRJnQwJSVInQ0KS1MmQkCR1MiQkSZ0MCUlSJ0NCktTJkJAkdTIkJEmdDAlJUidDQpLUyZCQJHUyJCRJnQwJSVInQ0KS1MmQkCR1MiQkSZ0MCUlSJ0NCktTJkJAkdTIkJEmdDAlJUidDQpLUyZCQJHUyJCRJnQwJSVKnoYZEkkuS3J1ka5JrOvr8eJI7k9yR5E+GWZ8k6XBjw9pQkj7wFuAHgW3ALUk2V9WdA33OB64Fvq+qvpLktGHVJ0l6rGHuSVwMbK2qe6tqCrgBuHxBn5cBb6mqrwBU1fYh1idJWmCYIXEmcP/A/La2bdCTgScn+dskH09yyWIrSnJVki1JtuzYsWOFypUkjdqJ6zHgfOA5wJXA7yc5eWGnqrq+qjZV1aaNGzcOuURJOn4MMyQeAM4emD+rbRu0DdhcVdNV9Q/APTShIUlaBcMMiVuA85Ocl2QCuALYvKDPX9DsRZDkVJrDT/cOsUZJ0oChhURVzQBXAzcBdwHvrao7klyX5LK2203AQ0nuBD4M/KeqemhYNUqSDpeqWu0avi6bNm2qLVu2rHYZknRMSXJrVW06Ur9RO3EtSRohhoQkqZMhIUnqZEhIkjoZEpKkToaEJKmTISFJ6mRISJI6GRKSpE5LCokk40m+nOQ7VrogSdLoWFJIVNU0MA0c22N4SJKOytEcbvqvwLVJhnbJU0nS6jqaN/zvB54NPJDkdmDP4MKqumzRe0mSjllHExI7gT9fqUIkSaNnySFRVS9ZyUIkSaNnyeckkty82PWmk5yY5OblLUuSNAqO5sT1c4CJRdrX0JyvkCR9gzni4aYkFw3MPjXJwwPzfeCHgAeWuzBJ0upbyjmJLTT/H1HABxdZvg/42eUsSpI0GpYSEucBAe4FvpvmW07zpoDtVTW7ArVJklbZEUOiqr4AkOQ3gO+uqrcNLk/y75OcWVW/uEI1SpJWydGcuH4R8KlF2j8J/OTylCNJGiVHExKnATsWad8JnL485UiSRsnRhMQXWfyrrs8Cti1POZKkUXI0w3K8HfidJBPA/D/PPQ94LfCby12YJGn1Hc2wHG9IcirwJg79U90U8Maq+q2VKE6StLqOatjvqro2yWuAC9umu6pq9/KXJUkaBUd9bYiq2gPcsgK1SJJGjNe4liR1MiQkSZ0MCUlSJ0NCktTJkJAkdTIkJEmdDAlJUqehhkSSS5LcnWRrkmsep9+PJqkkm4ZZnyTpcEMLiSR94C3ApTT/sX1lkgsX6bcBeDnwiWHVJkla3DD3JC4GtlbVvVU1BdwAXL5Iv1+lGTBw/xBrkyQtYpghcSZw/8D8trbtoCQXAWdX1f96vBUluSrJliRbduxY7BIXkqTlMDInrpP0gN8GXnmkvlV1fVVtqqpNGzduXPniJOk4NcyQeAA4e2D+rLZt3gbgO4GPJLkPeDqw2ZPXkrR6hhkStwDnJzmvvXDRFcDm+YVV9UhVnVpV51bVucDHgcuqassQa5QkDRhaSFTVDHA1cBNwF/DeqrojyXVJLhtWHZKkpTvq60l8ParqRuDGBW2v7uj7nGHUJEnqNjInriVJo8eQkCR1MiQkSZ0MCUlSJ0NCktTJkJAkdTIkJEmdDAlJUidDQpLUyZCQJHUyJCRJnQwJSVInQ0KS1MmQkCR1MiQkSZ0MCUlSJ0NCktTJkJAkdTIkJEmdDAlJUidDQpLUyZCQJHUyJCRJnQwJSVInQ0KS1MmQkCR1MiQkSZ0MCUlSJ0NCktTJkJAkdTIkJEmdDAlJUidDQpLUyZCQJHUaakgkuSTJ3Um2JrlmkeU/l+TOJJ9J8tdJzhlmfZKkww0tJJL0gbcAlwIXAlcmuXBBt08Bm6rqqcD7gN8aVn2SpMca5p7ExcDWqrq3qqaAG4DLBztU1Yeram87+3HgrCHWJ0laYJghcSZw/8D8traty0uBDyy2IMlVSbYk2bJjx45lLFGSNGgkT1wneRGwCXjdYsur6vqq2lRVmzZu3Djc4iTpODI2xG09AJw9MH9W23aYJD8AvAp4dlUdGFJtkqRFDHNP4hbg/CTnJZkArgA2D3ZI8jTg7cBlVbV9iLVJkhYxtJCoqhngauAm4C7gvVV1R5LrklzWdnsdsB74sySfTrK5Y3WSpCEY5uEmqupG4MYFba8emP6BYdYjSXp8I3niWpI0GgwJSVInQ0KS1MmQkCR1MiQkSZ0MCUlSJ0NCktTJkJAkdTIkJEmdDAlJUidDQpLUyZCQJHUyJCRJnQwJSVInQ0KS1MmQkCR1MiQkSZ0MCUlSJ0NCktTJkJAkdTIkJEmdDAlJUidDQpLUyZCQJHUyJCRJnQwJSVInQ0KS1MmQkCR1MiQkSZ0MCUlSJ0NCktTJkJAkdTIkJEmdDAlJUidDQpLUaaghkeSSJHcn2ZrkmkWWTyb503b5J5KcO8z6JEmHG1pIJOkDbwEuBS4Erkxy4YJuLwW+UlVPAn4H+M1h1SdJeqxh7klcDGytqnuragq4Abh8QZ/LgXe10+8DnpckQ6xRkjRgbIjbOhO4f2B+G/A9XX2qaibJI8ApwM7BTkmuAq5qZ3cnuftrrOnUheseQcdCjXBs1GmNy8Mal8dq13jOUjoNMySWTVVdD1z/9a4nyZaq2rQMJa2YY6FGODbqtMblYY3L41ioEYZ7uOkB4OyB+bPatkX7JBkDTgIeGkp1kqTHGGZI3AKcn+S8JBPAFcDmBX02Ay9up38MuLmqaog1SpIGDO1wU3uO4WrgJqAPvKOq7khyHbClqjYD/w14d5KtwMM0QbKSvu5DVkNwLNQIx0ad1rg8rHF5HAs1Ej+oS5K6+B/XkqROhoQkqdNxGxJHGiJkhbf9jiTbk9w+0PZNST6U5HPtzye07UnyprbOzyS5aOA+L277fy7Jixfb1tdR49lJPpzkziR3JHn5qNWZZE2Sv0tyW1vjr7Tt57XDumxth3mZaNs7h31Jcm3bfneSH1quGgfW30/yqSTvH8Uak9yX5LNJPp1kS9s2Mq91u+6Tk7wvyd8nuSvJM0awxm9vn8P5264krxi1Oo9KVR13N5oT558HnghMALcBFw5x+88CLgJuH2j7LeCadvoa4Dfb6RcAHwACPB34RNv+TcC97c8ntNNPWMYazwAuaqc3APfQDKcyMnW221rfTo8Dn2i3/V7girb9bcB/aKf/I/C2dvoK4E/b6Qvb34FJ4Lz2d6O/zK/5zwF/Ary/nR+pGoH7gFMXtI3Ma92u/13Av22nJ4CTR63GBfX2gS/T/NPayNZ5xMexGhtd7RvwDOCmgflrgWuHXMO5HB4SdwNntNNnAHe3028HrlzYD7gSePtA+2H9VqDevwR+cFTrBE4APknzX/w7gbGFrzXNN+ue0U6Ptf2y8PUf7LdMtZ0F/DXwXOD97TZHrcb7eGxIjMxrTfM/U/9A+2WbUaxxkZqfD/ztqNd5pNvxerhpsSFCzlylWuadXlVfaqe/DJzeTnfVOrTH0B7yeBrNJ/WRqrM9jPNpYDvwIZpP2F+tqplFtnfYsC/A/LAvK/1c/i7wC8BcO3/KCNZYwAeT3Jpm2BsYrdf6PGAH8IftYbs/SLJuxGpc6ArgPe30KNf5uI7XkBhp1Xx0GInvJidZD/w58Iqq2jW4bBTqrKrZqvoumk/rFwMXrGY9CyX5EWB7Vd262rUcwTOr6iKaUZp/JsmzBheOwGs9RnOI9veq6mnAHprDNgeNQI0HteeYLgP+bOGyUapzKY7XkFjKECHD9mCSMwDan9vb9q5aV/wxJBmnCYg/rqr/Map1AlTVV4EP0xy6OTnNsC4Lt9c17MtK1vh9wGVJ7qMZ+fi5wBtHrEaq6oH253bgf9IE7ii91tuAbVX1iXb+fTShMUo1DroU+GRVPdjOj2qdR3S8hsRShggZtsEhSV5Mcw5gvv0n229BPB14pN1tvQl4fpIntN+UeH7btiyShOY/4O+qqt8exTqTbExycju9luacyV00YfFjHTUuNuzLZuCK9ptF5wHnA3+3HDVW1bVVdVZVnUvze3ZzVf3EKNWYZF2SDfPTNK/R7YzQa11VXwbuT/LtbdPzgDtHqcYFruTQoab5ekaxziNbjRMho3Cj+VbBPTTHsF815G2/B/gSME3zCemlNMed/xr4HPB/gG9q+4bmYk2fBz4LbBpYz08DW9vbS5a5xmfS7BJ/Bvh0e3vBKNUJPBX4VFvj7cCr2/Yn0ryBbqXZ3Z9s29e081vb5U8cWNer2trvBi5dodf9ORz6dtPI1NjWclt7u2P+72GUXut23d8FbGlf77+g+dbPSNXYrn8dzd7fSQNtI1fnUm8OyyFJ6nS8Hm6SJC2BISFJ6mRISJI6GRKSpE6GhCSpkyEhHUGSc5NUkpG6aP2o1qVvLIaEdGT30wy69mmAJM9p35xPHVYBST6S5M2PV5e0EoZ2jWvpWFVVszSDsi27JONVNf213Hcl65LmuSehkdJ+Yn5rkl9PsjPNxZlen6TXLr8vyc8vcp83D8zfl+TVSd6Z5NEk9yd5YZqL1tyQZHd7IZfnL7Gmg4d12hFxP9wu2tG2v7PtlyS/kOTzSfaluYjPixZZz5VJbk6yD/h3SU5J8p4k29r73ZHkJQP3eyfwbJqB96q9nbvY4aYkz0pzsaL9SR5M8jvt0DNLen7bPv8yzQVw9iV5OMnfJJkftVTHGUNCo+gngBnge4GrgVcALzzKdbyCZliLi2gu8PMumov+3EgzvMNHgT9KsuYo13s/8KPt9HfQHO55eTv/GpohVn6G5iJBrwXenuSHF6zjtcBb2z5/QTMUxyeBH2nX+cb2fs9r+78c+Bjwh+32zuDwYaQBSHImzQVsPkUztPtLacYQeu2Crp3Pb5JvphmI8F3AU2gukPXuJTwv+ka1WuOBePO22A34CPCxBW0fAv6gnb4P+PlF7vPmgfn7gPcMzK+nGYfqTQNt57Ztm5ZQ02F9acZgKgYu0kMzXs8+4PsX3Pd3gRsXrOeVS9jmDfOPebHH2FHXr9GMDdQb6PNTwAHghCU+vxe16zxntX8XvI3GzXMSGkWfWTD/j8BpX+s6qmp3kr00A6jNmx/C+WjX2+VCmj2C/51kcEC0cZrQGrRlcCZJn+baCC+kubDMJM3lOT9ylDU8Bfh4Vc0NtP3fdl1P4tBz8njP7200A9DdnuSD7fT7qmrHUdaibxCGhEbRwhO5xaFDo3M0I2cOGl/iOqYXzMPyHXKdX88/B754hFr2LJj/eeCVNIeVPgvsBn6d5QswOPwiN53Pb1XNtudqnk4zPPVLgdcmeXZV3baM9egYYUjoWLOD5pg8AO05hQtojsMPy1T7sz/QdifNYZ1zqurmo1zfM4G/qqp3w8FreTwZ+OqCbfYXue+gu4AfT9Ib2Jt4Znvfzy+1mKoqmnMgH0tyHc3w4S+k2cvQccaQ0LHmZuCnk2ymCYxXMfzf4y/QfPr+4SR/BeyrqkeTvB54ffsm/1GacyFPB+aq6vrHWd89wAuTPBPYCfwszTWdB4PvPuDi9ttVu4GHF1nPW2lOQr81yRtprhPxGzTnMvYu5YG1F775AZoL3DxIcwL8bJoQ1HHIbzfpWPNamqD4S+CDNMfch7kXQTWX+vwlmhPFDwLzX7/9ReCXaQ4f3UFzQvhHgX84wipfQ/NNrA/QhMse4I8X9Hk9zR7BnTTh+K0ddV1K88b+aeAdNBe4+s9H8fAeobnk6vtpToK/AfjVqvqjo1iHvoF40SFJUif3JCRJnQwJHfeSvK39L+zFbm9b7fqk1eThJh33kpwGnNixeFdVbR9mPdIoMSQkSZ083CRJ6mRISJI6GRKSpE6GhCSp0/8HlAymwOifJ7kAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Instantiate learner in VW\n",
    "vw = pyvw.vw(\"--cb_explore 159 --epsilon 0.2 --cb_type ips\")\n",
    "\n",
    "num_iterations = df.shape[0] - 1\n",
    "ctr = run_simulation(vw, num_iterations, actions, df)\n",
    "\n",
    "plot_ctr(num_iterations, ctr)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.6 64-bit ('base': conda)",
   "language": "python",
   "name": "python37664bitbasecondae216400be78948018946a12046ff31a0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}