Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting a customer's next purchase using automated feature engineering\n",
    "\n",
    "<p style=\"margin:30px\">\n",
    "    <img width=50% src=\"https://www.featuretools.com/wp-content/uploads/2017/12/FeatureLabs-Logo-Tangerine-800.png\" alt=\"Featuretools\" />\n",
    "</p>\n",
    "\n",
    "**As customers use your product, they leave behind a trail of behaviors that indicate how they will act in the future. Through automated feature engineering we can identify the predictive patterns in granular customer behavioral data that can be used to improve the customer's experience and generate additional revenue for your business.**\n",
    "\n",
    "In this tutorial, we show how [Featuretools](http://www.featuretools.com) can be used to perform feature engineering on a multi-table dataset of 3 million online grocery orders provided by Instacart to train an accurate machine learning model to predict what product a customer buys next.\n",
    "\n",
    "*Note: If you are running this notebook yourself, refer to the [read me on Github](https://github.com/featuretools/predict_next_purchase#running-the-tutorial) for instructions to download the Instacart dataset*\n",
    "\n",
    "## Highlights\n",
    "\n",
    "* We automatically generate 150+ features using Deep Feature Synthesis and select the 20 most important features for predictive modeling\n",
    "* We build a pipeline that it can be reused for numerous prediction problems (you can try this yourself!)\n",
    "* We quickly develop a model on a subset of the data and validate on the entire dataset in a scalable manner using [Dask](http://dask.pydata.org/en/latest/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.6.0'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import featuretools as ft\n",
    "from dask import bag\n",
    "from dask.diagnostics import ProgressBar\n",
    "import pandas as pd\n",
    "import utils\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import cross_val_score\n",
    "import os\n",
    "ft.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1. Load data\n",
    "\n",
    "We start by loading in just one partion of our datast. In this case, a partition of our dataset contains the complete purchase history for each user within it. At the end of the tutorial, we will run the pipeline on every partiton to generate a final model. To learn more about loading data into Featuretools, read the guide [here](https://docs.featuretools.com/loading_data/using_entitysets.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Entityset: instacart\n",
       "  Entities:\n",
       "    order_products [Rows: 156605, Columns: 7]\n",
       "    orders [Rows: 16328, Columns: 3]\n",
       "    users [Rows: 1000, Columns: 2]\n",
       "  Relationships:\n",
       "    order_products.order_id -> orders.order_id\n",
       "    orders.user_id -> users.user_id"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "es = utils.load_entityset(\"partitioned_data/part_1/\")\n",
    "es"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize EntitySet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: instacart Pages: 1 -->\n",
       "<svg width=\"249pt\" height=\"356pt\"\n",
       " viewBox=\"0.00 0.00 249.00 356.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 352)\">\n",
       "<title>instacart</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-352 245,-352 245,4 -4,4\"/>\n",
       "<!-- order_products -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>order_products</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"17,-211.5 17,-347.5 224,-347.5 224,-211.5 17,-211.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"120.5\" y=\"-332.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">order_products</text>\n",
       "<polyline fill=\"none\" stroke=\"#000000\" points=\"17,-324.5 224,-324.5 \"/>\n",
       "<text text-anchor=\"start\" x=\"25\" y=\"-309.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">order_product_id : index</text>\n",
       "<text text-anchor=\"start\" x=\"25\" y=\"-294.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">order_id : id</text>\n",
       "<text text-anchor=\"start\" x=\"25\" y=\"-279.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">product_name : categorical</text>\n",
       "<text text-anchor=\"start\" x=\"25\" y=\"-264.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">department : categorical</text>\n",
       "<text text-anchor=\"start\" x=\"25\" y=\"-249.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">order_time : datetime_time_index</text>\n",
       "<text text-anchor=\"start\" x=\"25\" y=\"-234.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">aisle_id : categorical</text>\n",
       "<text text-anchor=\"start\" x=\"25\" y=\"-219.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">reordered : boolean</text>\n",
       "</g>\n",
       "<!-- orders -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>orders</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"17,-98.5 17,-174.5 224,-174.5 224,-98.5 17,-98.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"120.5\" y=\"-159.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">orders</text>\n",
       "<polyline fill=\"none\" stroke=\"#000000\" points=\"17,-151.5 224,-151.5 \"/>\n",
       "<text text-anchor=\"start\" x=\"25\" y=\"-136.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">order_id : index</text>\n",
       "<text text-anchor=\"start\" x=\"25\" y=\"-121.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">user_id : id</text>\n",
       "<text text-anchor=\"start\" x=\"25\" y=\"-106.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">order_time : datetime_time_index</text>\n",
       "</g>\n",
       "<!-- order_products&#45;&gt;orders -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>order_products&#45;&gt;orders</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M120.5,-211.3505C120.5,-211.3505 120.5,-184.5397 120.5,-184.5397\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"124.0001,-184.5397 120.5,-174.5397 117.0001,-184.5397 124.0001,-184.5397\"/>\n",
       "<text text-anchor=\"middle\" x=\"96.5\" y=\"-186.7451\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">order_id</text>\n",
       "</g>\n",
       "<!-- users -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>users</title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"0,-.5 0,-61.5 241,-61.5 241,-.5 0,-.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"120.5\" y=\"-46.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">users</text>\n",
       "<polyline fill=\"none\" stroke=\"#000000\" points=\"0,-38.5 241,-38.5 \"/>\n",
       "<text text-anchor=\"start\" x=\"8\" y=\"-23.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">user_id : index</text>\n",
       "<text text-anchor=\"start\" x=\"8\" y=\"-8.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">first_orders_time : datetime_time_index</text>\n",
       "</g>\n",
       "<!-- orders&#45;&gt;users -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>orders&#45;&gt;users</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M120.5,-98.4121C120.5,-98.4121 120.5,-71.7631 120.5,-71.7631\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"124.0001,-71.7631 120.5,-61.7631 117.0001,-71.7631 124.0001,-71.7631\"/>\n",
       "<text text-anchor=\"middle\" x=\"99.5\" y=\"-73.8876\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">user_id</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x120df8128>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "es.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2. Make Labels\n",
    "\n",
    "For supervised machine learning, we need labels. These labels define what our predictive model will be used for. In this tutorial, we will predict if a customer will buy Bananas in the next 4 weeks.\n",
    "\n",
    "We generate training examples by selecting a `cutoff_time` in the past to make our labels. Using users who had acivity during `training_window` days before the `cutoff_time`, we look to see if they purchase the product in the `prediction_window`. \n",
    "\n",
    "If you are running this code yourself, feel free to experiment with any of these parameters! For example, try to predict if a customer will buy \"Limes\" instead of \"Bananas\" or increase the size of your `prediction_window`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>time</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1001</td>\n",
       "      <td>2015-03-15</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1002</td>\n",
       "      <td>2015-03-15</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1003</td>\n",
       "      <td>2015-03-15</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1004</td>\n",
       "      <td>2015-03-15</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1005</td>\n",
       "      <td>2015-03-15</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id       time  label\n",
       "0     1001 2015-03-15  False\n",
       "1     1002 2015-03-15  False\n",
       "2     1003 2015-03-15  False\n",
       "3     1004 2015-03-15  False\n",
       "4     1005 2015-03-15  False"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "label_times = utils.make_labels(es=es,\n",
    "                                product_name = \"Banana\",\n",
    "                                cutoff_time = pd.Timestamp('March 15, 2015'),\n",
    "                                prediction_window = ft.Timedelta(\"4 weeks\"),\n",
    "                                training_window = ft.Timedelta(\"60 days\"))\n",
    "label_times.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see above the our training examples contain three pieces of information: a user id, the last time we can use data before feature engineering (called the \"cutoff time\"), and the label to predict. These are called our \"label times\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The distribution of the labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False    654\n",
       "True     128\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "label_times[\"label\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Automated Feature Engineering\n",
    "With our label times in hand, we can use Deep Feature Synthesis to automatically generate features.\n",
    "\n",
    "When we use DFS, we specify\n",
    "\n",
    "* `target_entity` - the table to build feature for\n",
    "* `cutoff_time` the point in time to calculate the features\n",
    "* `training_window` - the amount of historical data we want to use when calculating features\n",
    "\n",
    "A good way to think of the `cutoff_time` is that it let's us \"pretend\" we are at an earlier point in time when generating our features so we can simulate making predictions. We get this time for each customer from the label times we generated above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Built 76 features\n",
      "Elapsed: 00:20 | Remaining: 00:00 | Progress: 100%|██████████| Calculated: 11/11 chunks\n",
      "Number of features 161\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>COUNT(orders)</th>\n",
       "      <th>COUNT(order_products)</th>\n",
       "      <th>PERCENT_TRUE(order_products.reordered)</th>\n",
       "      <th>NUM_UNIQUE(order_products.product_name)</th>\n",
       "      <th>NUM_UNIQUE(order_products.department)</th>\n",
       "      <th>NUM_UNIQUE(order_products.aisle_id)</th>\n",
       "      <th>MODE(order_products.product_name) = Banana</th>\n",
       "      <th>MODE(order_products.product_name) = Bag of Organic Bananas</th>\n",
       "      <th>MODE(order_products.product_name) = Organic Baby Spinach</th>\n",
       "      <th>MODE(order_products.product_name) = Michigan Organic Kale</th>\n",
       "      <th>...</th>\n",
       "      <th>COUNT(order_products WHERE department = produce)</th>\n",
       "      <th>COUNT(order_products WHERE department = bakery)</th>\n",
       "      <th>COUNT(order_products WHERE product_name = Banana)</th>\n",
       "      <th>COUNT(order_products WHERE product_name = Limes)</th>\n",
       "      <th>COUNT(order_products WHERE product_name = Organic Hass Avocado)</th>\n",
       "      <th>COUNT(order_products WHERE department = pantry)</th>\n",
       "      <th>COUNT(order_products WHERE department = frozen)</th>\n",
       "      <th>COUNT(order_products WHERE product_name = Organic Whole Milk)</th>\n",
       "      <th>COUNT(order_products WHERE department = snacks)</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1001</th>\n",
       "      <td>2</td>\n",
       "      <td>33</td>\n",
       "      <td>0.424242</td>\n",
       "      <td>26</td>\n",
       "      <td>11</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1002</th>\n",
       "      <td>5</td>\n",
       "      <td>49</td>\n",
       "      <td>0.244898</td>\n",
       "      <td>40</td>\n",
       "      <td>8</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>25.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1003</th>\n",
       "      <td>6</td>\n",
       "      <td>79</td>\n",
       "      <td>0.670886</td>\n",
       "      <td>41</td>\n",
       "      <td>12</td>\n",
       "      <td>26</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>9.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1004</th>\n",
       "      <td>8</td>\n",
       "      <td>169</td>\n",
       "      <td>0.692308</td>\n",
       "      <td>76</td>\n",
       "      <td>11</td>\n",
       "      <td>19</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>126.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1005</th>\n",
       "      <td>5</td>\n",
       "      <td>54</td>\n",
       "      <td>0.518519</td>\n",
       "      <td>38</td>\n",
       "      <td>12</td>\n",
       "      <td>28</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1006</th>\n",
       "      <td>2</td>\n",
       "      <td>19</td>\n",
       "      <td>0.105263</td>\n",
       "      <td>18</td>\n",
       "      <td>9</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1007</th>\n",
       "      <td>6</td>\n",
       "      <td>46</td>\n",
       "      <td>0.478261</td>\n",
       "      <td>31</td>\n",
       "      <td>8</td>\n",
       "      <td>19</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1008</th>\n",
       "      <td>2</td>\n",
       "      <td>32</td>\n",
       "      <td>0.031250</td>\n",
       "      <td>31</td>\n",
       "      <td>7</td>\n",
       "      <td>19</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1009</th>\n",
       "      <td>2</td>\n",
       "      <td>14</td>\n",
       "      <td>0.428571</td>\n",
       "      <td>10</td>\n",
       "      <td>5</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1011</th>\n",
       "      <td>2</td>\n",
       "      <td>38</td>\n",
       "      <td>0.342105</td>\n",
       "      <td>34</td>\n",
       "      <td>12</td>\n",
       "      <td>19</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>12.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 162 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         COUNT(orders)  COUNT(order_products)  \\\n",
       "user_id                                         \n",
       "1001                 2                     33   \n",
       "1002                 5                     49   \n",
       "1003                 6                     79   \n",
       "1004                 8                    169   \n",
       "1005                 5                     54   \n",
       "1006                 2                     19   \n",
       "1007                 6                     46   \n",
       "1008                 2                     32   \n",
       "1009                 2                     14   \n",
       "1011                 2                     38   \n",
       "\n",
       "         PERCENT_TRUE(order_products.reordered)  \\\n",
       "user_id                                           \n",
       "1001                                   0.424242   \n",
       "1002                                   0.244898   \n",
       "1003                                   0.670886   \n",
       "1004                                   0.692308   \n",
       "1005                                   0.518519   \n",
       "1006                                   0.105263   \n",
       "1007                                   0.478261   \n",
       "1008                                   0.031250   \n",
       "1009                                   0.428571   \n",
       "1011                                   0.342105   \n",
       "\n",
       "         NUM_UNIQUE(order_products.product_name)  \\\n",
       "user_id                                            \n",
       "1001                                          26   \n",
       "1002                                          40   \n",
       "1003                                          41   \n",
       "1004                                          76   \n",
       "1005                                          38   \n",
       "1006                                          18   \n",
       "1007                                          31   \n",
       "1008                                          31   \n",
       "1009                                          10   \n",
       "1011                                          34   \n",
       "\n",
       "         NUM_UNIQUE(order_products.department)  \\\n",
       "user_id                                          \n",
       "1001                                        11   \n",
       "1002                                         8   \n",
       "1003                                        12   \n",
       "1004                                        11   \n",
       "1005                                        12   \n",
       "1006                                         9   \n",
       "1007                                         8   \n",
       "1008                                         7   \n",
       "1009                                         5   \n",
       "1011                                        12   \n",
       "\n",
       "         NUM_UNIQUE(order_products.aisle_id)  \\\n",
       "user_id                                        \n",
       "1001                                      20   \n",
       "1002                                      16   \n",
       "1003                                      26   \n",
       "1004                                      19   \n",
       "1005                                      28   \n",
       "1006                                      12   \n",
       "1007                                      19   \n",
       "1008                                      19   \n",
       "1009                                       9   \n",
       "1011                                      19   \n",
       "\n",
       "         MODE(order_products.product_name) = Banana  \\\n",
       "user_id                                               \n",
       "1001                                              0   \n",
       "1002                                              0   \n",
       "1003                                              0   \n",
       "1004                                              0   \n",
       "1005                                              0   \n",
       "1006                                              0   \n",
       "1007                                              0   \n",
       "1008                                              0   \n",
       "1009                                              0   \n",
       "1011                                              0   \n",
       "\n",
       "         MODE(order_products.product_name) = Bag of Organic Bananas  \\\n",
       "user_id                                                               \n",
       "1001                                                     0            \n",
       "1002                                                     0            \n",
       "1003                                                     0            \n",
       "1004                                                     0            \n",
       "1005                                                     0            \n",
       "1006                                                     0            \n",
       "1007                                                     0            \n",
       "1008                                                     0            \n",
       "1009                                                     0            \n",
       "1011                                                     0            \n",
       "\n",
       "         MODE(order_products.product_name) = Organic Baby Spinach  \\\n",
       "user_id                                                             \n",
       "1001                                                     0          \n",
       "1002                                                     0          \n",
       "1003                                                     0          \n",
       "1004                                                     0          \n",
       "1005                                                     0          \n",
       "1006                                                     0          \n",
       "1007                                                     0          \n",
       "1008                                                     0          \n",
       "1009                                                     0          \n",
       "1011                                                     0          \n",
       "\n",
       "         MODE(order_products.product_name) = Michigan Organic Kale  ...    \\\n",
       "user_id                                                             ...     \n",
       "1001                                                     0          ...     \n",
       "1002                                                     0          ...     \n",
       "1003                                                     0          ...     \n",
       "1004                                                     0          ...     \n",
       "1005                                                     0          ...     \n",
       "1006                                                     0          ...     \n",
       "1007                                                     0          ...     \n",
       "1008                                                     0          ...     \n",
       "1009                                                     0          ...     \n",
       "1011                                                     0          ...     \n",
       "\n",
       "         COUNT(order_products WHERE department = produce)  \\\n",
       "user_id                                                     \n",
       "1001                                                  1.0   \n",
       "1002                                                  1.0   \n",
       "1003                                                  9.0   \n",
       "1004                                                126.0   \n",
       "1005                                                  6.0   \n",
       "1006                                                  5.0   \n",
       "1007                                                 12.0   \n",
       "1008                                                  5.0   \n",
       "1009                                                  5.0   \n",
       "1011                                                 12.0   \n",
       "\n",
       "         COUNT(order_products WHERE department = bakery)  \\\n",
       "user_id                                                    \n",
       "1001                                                 2.0   \n",
       "1002                                                 0.0   \n",
       "1003                                                 3.0   \n",
       "1004                                                 7.0   \n",
       "1005                                                 0.0   \n",
       "1006                                                 1.0   \n",
       "1007                                                 0.0   \n",
       "1008                                                 1.0   \n",
       "1009                                                 0.0   \n",
       "1011                                                 1.0   \n",
       "\n",
       "         COUNT(order_products WHERE product_name = Banana)  \\\n",
       "user_id                                                      \n",
       "1001                                                   0.0   \n",
       "1002                                                   1.0   \n",
       "1003                                                   0.0   \n",
       "1004                                                   0.0   \n",
       "1005                                                   3.0   \n",
       "1006                                                   0.0   \n",
       "1007                                                   0.0   \n",
       "1008                                                   0.0   \n",
       "1009                                                   0.0   \n",
       "1011                                                   1.0   \n",
       "\n",
       "         COUNT(order_products WHERE product_name = Limes)  \\\n",
       "user_id                                                     \n",
       "1001                                                  0.0   \n",
       "1002                                                  0.0   \n",
       "1003                                                  0.0   \n",
       "1004                                                  4.0   \n",
       "1005                                                  0.0   \n",
       "1006                                                  0.0   \n",
       "1007                                                  0.0   \n",
       "1008                                                  0.0   \n",
       "1009                                                  0.0   \n",
       "1011                                                  1.0   \n",
       "\n",
       "         COUNT(order_products WHERE product_name = Organic Hass Avocado)  \\\n",
       "user_id                                                                    \n",
       "1001                                                   0.0                 \n",
       "1002                                                   0.0                 \n",
       "1003                                                   0.0                 \n",
       "1004                                                   0.0                 \n",
       "1005                                                   0.0                 \n",
       "1006                                                   0.0                 \n",
       "1007                                                   1.0                 \n",
       "1008                                                   0.0                 \n",
       "1009                                                   0.0                 \n",
       "1011                                                   0.0                 \n",
       "\n",
       "         COUNT(order_products WHERE department = pantry)  \\\n",
       "user_id                                                    \n",
       "1001                                                 0.0   \n",
       "1002                                                 9.0   \n",
       "1003                                                 4.0   \n",
       "1004                                                 2.0   \n",
       "1005                                                 2.0   \n",
       "1006                                                 0.0   \n",
       "1007                                                 5.0   \n",
       "1008                                                 1.0   \n",
       "1009                                                 2.0   \n",
       "1011                                                 1.0   \n",
       "\n",
       "         COUNT(order_products WHERE department = frozen)  \\\n",
       "user_id                                                    \n",
       "1001                                                 2.0   \n",
       "1002                                                 0.0   \n",
       "1003                                                 4.0   \n",
       "1004                                                 4.0   \n",
       "1005                                                 5.0   \n",
       "1006                                                 4.0   \n",
       "1007                                                 0.0   \n",
       "1008                                                 6.0   \n",
       "1009                                                 0.0   \n",
       "1011                                                 2.0   \n",
       "\n",
       "         COUNT(order_products WHERE product_name = Organic Whole Milk)  \\\n",
       "user_id                                                                  \n",
       "1001                                                   0.0               \n",
       "1002                                                   0.0               \n",
       "1003                                                   0.0               \n",
       "1004                                                   0.0               \n",
       "1005                                                   0.0               \n",
       "1006                                                   0.0               \n",
       "1007                                                   0.0               \n",
       "1008                                                   0.0               \n",
       "1009                                                   0.0               \n",
       "1011                                                   0.0               \n",
       "\n",
       "         COUNT(order_products WHERE department = snacks)  label  \n",
       "user_id                                                          \n",
       "1001                                                 9.0  False  \n",
       "1002                                                25.0  False  \n",
       "1003                                                 6.0  False  \n",
       "1004                                                 1.0  False  \n",
       "1005                                                 6.0  False  \n",
       "1006                                                 0.0  False  \n",
       "1007                                                 5.0  False  \n",
       "1008                                                 8.0   True  \n",
       "1009                                                 2.0  False  \n",
       "1011                                                 5.0  False  \n",
       "\n",
       "[10 rows x 162 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_matrix, features = ft.dfs(target_entity=\"users\", \n",
    "                                  cutoff_time=label_times,\n",
    "                                  training_window=ft.Timedelta(\"60 days\"), # same as above\n",
    "                                  entityset=es,\n",
    "                                  verbose=True)\n",
    "# encode categorical values\n",
    "fm_encoded, features_encoded = ft.encode_features(feature_matrix,\n",
    "                                                  features)\n",
    "\n",
    "print(\"Number of features %s\" % len(features_encoded))\n",
    "fm_encoded.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4. Machine Learning\n",
    "\n",
    "Using the default parameters, we generated 160 potential features for our prediction problem. With a few simple commands, this feature matrix can be used for machine learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = utils.merge_features_labels(fm_encoded, label_times)\n",
    "X.drop([\"user_id\", \"time\"], axis=1, inplace=True)\n",
    "X = X.fillna(0)\n",
    "y = X.pop(\"label\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's train a Random Forest and validate using 3-fold cross validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   3 out of   3 | elapsed:    6.7s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'AUC 0.84 +/- 0.03'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = RandomForestClassifier(n_estimators=400, n_jobs=-1)\n",
    "scores = cross_val_score(estimator=clf,X=X, y=y, cv=3,\n",
    "                         scoring=\"roc_auc\", verbose=True)\n",
    "\n",
    "\"AUC %.2f +/- %.2f\" % (scores.mean(), scores.std())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see we perform noticably better than guessing! However, we have a pretty high difference in performance accross folds.\n",
    "\n",
    "To improve this, let's identify the top 20 features using a Random Forest and then perform machine learning on the whole dataset (all of the partions)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1: Feature: COUNT(order_products WHERE product_name = Banana), 0.124\n",
      "2: Feature: MODE(order_products.product_name) = Banana, 0.035\n",
      "3: Feature: MODE(orders.MODE(order_products.product_name)) = Banana, 0.027\n",
      "4: Feature: COUNT(order_products WHERE department = produce), 0.020\n",
      "5: Feature: STD(orders.PERCENT_TRUE(order_products.reordered)), 0.019\n",
      "6: Feature: COUNT(order_products WHERE department = dairy eggs), 0.019\n",
      "7: Feature: MEAN(orders.NUM_UNIQUE(order_products.aisle_id)), 0.019\n",
      "8: Feature: MEAN(orders.PERCENT_TRUE(order_products.reordered)), 0.017\n",
      "9: Feature: COUNT(order_products), 0.016\n",
      "10: Feature: PERCENT_TRUE(order_products.reordered), 0.016\n",
      "11: Feature: STD(orders.NUM_UNIQUE(order_products.department)), 0.016\n",
      "12: Feature: NUM_UNIQUE(order_products.product_name), 0.016\n",
      "13: Feature: SUM(orders.NUM_UNIQUE(order_products.product_name)), 0.016\n",
      "14: Feature: MEAN(orders.NUM_UNIQUE(order_products.product_name)), 0.016\n",
      "15: Feature: MEAN(orders.COUNT(order_products)), 0.016\n",
      "16: Feature: SUM(orders.NUM_UNIQUE(order_products.department)), 0.015\n",
      "17: Feature: MEAN(orders.NUM_UNIQUE(order_products.department)), 0.015\n",
      "18: Feature: STD(orders.COUNT(order_products)), 0.015\n",
      "19: Feature: SUM(orders.PERCENT_TRUE(order_products.reordered)), 0.015\n",
      "20: Feature: STD(orders.NUM_UNIQUE(order_products.product_name)), 0.015\n"
     ]
    }
   ],
   "source": [
    "clf.fit(X, y)\n",
    "top_features = utils.feature_importances(clf, features_encoded, n=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To persist this features, we can save them to disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "ft.save_features(top_features, \"top_features\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Understanding feature engineering in Featuretools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before moving forward, take a look at the feature we created. You will see that they are more than just simple transformations of columns in our raw data. Instead, they aggregations (and sometimes stacking of aggregations) across the relationships in our dataset. If you're curious how this works, learn about the Deep Feature Synthesis algorithm in our documentation [here](https://docs.featuretools.com/automated_feature_engineering/afe.html).\n",
    "\n",
    "DFS is so powerful because with no manual work, the library figured out that historical purchases of bananas are important for predicting future purchases. Additionally, it surfaces that purchasing dairy or eggs and reordering behavior are important features. \n",
    "\n",
    "Even though these features are intuitive, Deep Feature Synthesis will automatically adapt as we change the prediction problem, saving us the time of manually brainstorming and implementing these data transformation. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scaling to full dataset\n",
    "\n",
    "Once we have written the pipeline for one partition, we can easily scale it out to the full dataset using [Dask](dask.pydata.org). A similar pipeline could also be built using [Spark](http://spark.apache.org/docs/2.2.0/api/python/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "pbar = ProgressBar()\n",
    "pbar.register()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we assemble our partitions and map them to entity sets using the function from before. A single partition contains all the data for each user within it, so this computation is easily parallelized. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = \"partitioned_data/\"\n",
    "#_, dirnames, _ = os.walk(path).next()\n",
    "dirnames = [os.path.join(path, d) for d in os.listdir(path)]\n",
    "b = bag.from_sequence(dirnames)\n",
    "entity_sets = b.map(utils.load_entityset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we create label times for each entity set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dask.bag<map-das..., npartitions=104>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "label_times = entity_sets.map(utils.dask_make_labels,\n",
    "                              product_name = \"Banana\",\n",
    "                              cutoff_time = pd.Timestamp('March 1, 2015'),\n",
    "                              prediction_window = ft.Timedelta(\"4 weeks\"),\n",
    "                              training_window = ft.Timedelta(\"60 days\"))\n",
    "label_times"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load in the features from before\n",
    "top_features = ft.load_features(\"top_features\")\n",
    "feature_matrices = label_times.map(utils.calculate_feature_matrix, features=top_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we compute with Dask. Running on a Macbook Pro with a 2.2 GHz Intel Core i7 and 16gb of ram, this takes about 20 minutes to run. The compute method can take an argument `num_workers` which defaults to using all of the cores on your computer. If you don't want it to do that, you can explicitly specify a number of workers with `feature_matrices.compute(num_workers=2)` where you can replace 2 with the number of cores you want to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[########################################] | 100% Completed | 52min 38.1s\n"
     ]
    }
   ],
   "source": [
    "fms_out = feature_matrices.compute()\n",
    "X = pd.concat(fms_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we repeat the same machine learning steps from the sample dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.drop([\"user_id\", \"time\"], axis=1, inplace=True)\n",
    "X = X.fillna(0)\n",
    "y = X.pop(\"label\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   3 out of   3 | elapsed:  4.5min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'AUC 0.87 +/- 0.00'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = RandomForestClassifier(n_estimators=400, n_jobs=-1)\n",
    "scores = cross_val_score(estimator=clf,X=X, y=y, cv=3,\n",
    "                         scoring=\"roc_auc\", verbose=True)\n",
    "\n",
    "\"AUC %.2f +/- %.2f\" % (scores.mean(), scores.std())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now we that our accuracy has stabalized across folds, giving us much more confidence in our model.\n",
    "\n",
    "Now, let's look at the top features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1: Feature: COUNT(order_products WHERE product_name = Banana), 0.263\n",
      "2: Feature: MODE(order_products.product_name) = Banana, 0.069\n",
      "3: Feature: MODE(orders.MODE(order_products.product_name)) = Banana, 0.048\n",
      "4: Feature: COUNT(order_products WHERE department = produce), 0.045\n",
      "5: Feature: STD(orders.PERCENT_TRUE(order_products.reordered)), 0.043\n",
      "6: Feature: MEAN(orders.PERCENT_TRUE(order_products.reordered)), 0.042\n",
      "7: Feature: PERCENT_TRUE(order_products.reordered), 0.041\n",
      "8: Feature: SUM(orders.PERCENT_TRUE(order_products.reordered)), 0.041\n",
      "9: Feature: MEAN(orders.NUM_UNIQUE(order_products.aisle_id)), 0.040\n",
      "10: Feature: STD(orders.NUM_UNIQUE(order_products.product_name)), 0.038\n",
      "11: Feature: STD(orders.COUNT(order_products)), 0.038\n",
      "12: Feature: STD(orders.NUM_UNIQUE(order_products.department)), 0.038\n",
      "13: Feature: COUNT(order_products WHERE department = dairy eggs), 0.035\n",
      "14: Feature: MEAN(orders.NUM_UNIQUE(order_products.product_name)), 0.034\n",
      "15: Feature: MEAN(orders.NUM_UNIQUE(order_products.department)), 0.034\n",
      "16: Feature: MEAN(orders.COUNT(order_products)), 0.033\n",
      "17: Feature: NUM_UNIQUE(order_products.product_name), 0.031\n",
      "18: Feature: SUM(orders.NUM_UNIQUE(order_products.department)), 0.030\n",
      "19: Feature: SUM(orders.NUM_UNIQUE(order_products.product_name)), 0.029\n",
      "20: Feature: COUNT(order_products), 0.028\n"
     ]
    }
   ],
   "source": [
    "clf.fit(X, y)\n",
    "top_features = utils.feature_importances(clf, top_features, n=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the top features shifted around a bit, but mostly stayed the same."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next Steps\n",
    "\n",
    "While this is an end-to-end example of going from raw data to a trained machine learning model, it is necessary to do further exploration before claiming we've built something impact full.\n",
    "\n",
    "Fortunately, Featuretools makes it easy to build structured data science pipeline. As a next steps, you could\n",
    "* Further validate these results by creating feature vectors at different cutoff times\n",
    "* Define other prediction problems for this dataset (you can even change the entity you are making predictions on!)\n",
    "* Save feature matrices to disk as CSVs so they can be reused with different problems without recalculating\n",
    "* Experiment with parameters to Deep Feature Synthesis\n",
    "* Create custom primitives for DFS. More info [here](https://docs.featuretools.com/automated_feature_engineering/primitives.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>\n",
    "    <img src=\"https://www.featurelabs.com/wp-content/uploads/2017/12/logo.png\" alt=\"Featuretools\" />\n",
    "</p>\n",
    "\n",
    "\n",
    "Featuretools was created by the developers at [Feature Labs](https://www.featurelabs.com/). If building impactful data science pipelines is important to you or your business, please [get in touch](https://www.featurelabs.com/contact)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}