Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting a customer's next purchase using automated feature engineering\n",
    "\n",
    "<p style=\"margin:30px\">\n",
    "    <img width=50% src=\"https://www.featuretools.com/wp-content/uploads/2017/12/FeatureLabs-Logo-Tangerine-800.png\" alt=\"Featuretools\" />\n",
    "</p>\n",
    "\n",
    "**As customers use your product, they leave behind a trail of behaviors that indicate how they will act in the future. Through automated feature engineering we can identify the predictive patterns in granular customer behavioral data that can be used to improve the customer's experience and generate additional revenue for your business.**\n",
    "\n",
    "In this tutorial, we show how [Featuretools](http://www.featuretools.com) can be used to perform feature engineering on a multi-table dataset of 3 million online grocery orders provided by Instacart to train an accurate machine learning model to predict what product a customer buys next.\n",
    "\n",
    "*Note: If you are running this notebook yourself, refer to the [read me on Github](https://github.com/featuretools/predict_next_purchase#running-the-tutorial) for instructions to download the Instacart dataset*\n",
    "\n",
    "## Highlights\n",
    "\n",
    "* We automatically generate 150+ features using Deep Feature Synthesis and select the 20 most important features for predictive modeling\n",
    "* We automatically generate label times using [Compose](https://github.com/FeatureLabs/compose) which can be reused for numerous prediction problems (you can try this yourself!)\n",
    "* We quickly develop a model on a subset of the data and validate on the entire dataset in a scalable manner using [Dask](http://dask.pydata.org/en/latest/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.16.0'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import composeml as cp\n",
    "import featuretools as ft\n",
    "from dask import bag\n",
    "from dask.diagnostics import ProgressBar\n",
    "import pandas as pd\n",
    "import utils\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import cross_val_score\n",
    "import os\n",
    "ft.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1. Load data\n",
    "\n",
    "We start by loading in just one partion of our datast. In this case, a partition of our dataset contains the complete purchase history for each user within it. At the end of the tutorial, we will run the pipeline on every partiton to generate a final model. To learn more about loading data into Featuretools, read the guide [here](https://docs.featuretools.com/en/stable/loading_data/using_entitysets.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Entityset: instacart\n",
       "  Entities:\n",
       "    order_products [Rows: 156605, Columns: 7]\n",
       "    orders [Rows: 16328, Columns: 3]\n",
       "    users [Rows: 1000, Columns: 2]\n",
       "  Relationships:\n",
       "    order_products.order_id -> orders.order_id\n",
       "    orders.user_id -> users.user_id"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "es = utils.load_entityset(\"partitioned_data/part_1/\")\n",
    "es"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize EntitySet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.43.0 (0)\n",
       " -->\n",
       "<!-- Title: instacart Pages: 1 -->\n",
       "<svg width=\"249pt\" height=\"341pt\"\n",
       " viewBox=\"0.00 0.00 248.73 341.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 337)\">\n",
       "<title>instacart</title>\n",
       "<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-337 244.73,-337 244.73,4 -4,4\"/>\n",
       "<!-- order_products -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>order_products</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"17.12,-204.5 17.12,-332.5 223.61,-332.5 223.61,-204.5 17.12,-204.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"120.37\" y=\"-317.3\" font-family=\"Times,serif\" font-size=\"14.00\">order_products (156605 rows)</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"17.12,-310.5 223.61,-310.5 \"/>\n",
       "<text text-anchor=\"start\" x=\"25.12\" y=\"-295.3\" font-family=\"Times,serif\" font-size=\"14.00\">order_product_id : index</text>\n",
       "<text text-anchor=\"start\" x=\"25.12\" y=\"-281.3\" font-family=\"Times,serif\" font-size=\"14.00\">order_id : id</text>\n",
       "<text text-anchor=\"start\" x=\"25.12\" y=\"-267.3\" font-family=\"Times,serif\" font-size=\"14.00\">product_name : categorical</text>\n",
       "<text text-anchor=\"start\" x=\"25.12\" y=\"-253.3\" font-family=\"Times,serif\" font-size=\"14.00\">department : categorical</text>\n",
       "<text text-anchor=\"start\" x=\"25.12\" y=\"-239.3\" font-family=\"Times,serif\" font-size=\"14.00\">order_time : datetime_time_index</text>\n",
       "<text text-anchor=\"start\" x=\"25.12\" y=\"-225.3\" font-family=\"Times,serif\" font-size=\"14.00\">aisle_id : categorical</text>\n",
       "<text text-anchor=\"start\" x=\"25.12\" y=\"-211.3\" font-family=\"Times,serif\" font-size=\"14.00\">reordered : boolean</text>\n",
       "</g>\n",
       "<!-- orders -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>orders</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"17.12,-95.5 17.12,-167.5 223.61,-167.5 223.61,-95.5 17.12,-95.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"120.37\" y=\"-152.3\" font-family=\"Times,serif\" font-size=\"14.00\">orders (16328 rows)</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"17.12,-145.5 223.61,-145.5 \"/>\n",
       "<text text-anchor=\"start\" x=\"25.12\" y=\"-130.3\" font-family=\"Times,serif\" font-size=\"14.00\">order_id : index</text>\n",
       "<text text-anchor=\"start\" x=\"25.12\" y=\"-116.3\" font-family=\"Times,serif\" font-size=\"14.00\">user_id : id</text>\n",
       "<text text-anchor=\"start\" x=\"25.12\" y=\"-102.3\" font-family=\"Times,serif\" font-size=\"14.00\">order_time : datetime_time_index</text>\n",
       "</g>\n",
       "<!-- order_products&#45;&gt;orders -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>order_products&#45;&gt;orders</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M120.37,-204.41C120.37,-204.41 120.37,-177.53 120.37,-177.53\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"123.87,-177.53 120.37,-167.53 116.87,-177.53 123.87,-177.53\"/>\n",
       "<text text-anchor=\"middle\" x=\"96.65\" y=\"-193.77\" font-family=\"Times,serif\" font-size=\"14.00\">order_id</text>\n",
       "</g>\n",
       "<!-- users -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>users</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"0,-0.5 0,-58.5 240.73,-58.5 240.73,-0.5 0,-0.5\"/>\n",
       "<text text-anchor=\"middle\" x=\"120.37\" y=\"-43.3\" font-family=\"Times,serif\" font-size=\"14.00\">users (1000 rows)</text>\n",
       "<polyline fill=\"none\" stroke=\"black\" points=\"0,-36.5 240.73,-36.5 \"/>\n",
       "<text text-anchor=\"start\" x=\"8\" y=\"-21.3\" font-family=\"Times,serif\" font-size=\"14.00\">user_id : index</text>\n",
       "<text text-anchor=\"start\" x=\"8\" y=\"-7.3\" font-family=\"Times,serif\" font-size=\"14.00\">first_orders_time : datetime_time_index</text>\n",
       "</g>\n",
       "<!-- orders&#45;&gt;users -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>orders&#45;&gt;users</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M120.37,-95.25C120.37,-95.25 120.37,-68.59 120.37,-68.59\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"123.87,-68.59 120.37,-58.59 116.87,-68.59 123.87,-68.59\"/>\n",
       "<text text-anchor=\"middle\" x=\"99.76\" y=\"-84.72\" font-family=\"Times,serif\" font-size=\"14.00\">user_id</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x13919b410>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "es.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2. Make Labels\n",
    "\n",
    "For supervised machine learning, we need labels. These labels define what our predictive model will be used for. In this tutorial, we will predict if a customer will buy Bananas in the next 4 weeks. To structure the labeling process, we will use [Compose](https://github.com/FeatureLabs/compose) which is an open source project for automatically generating labels with cutoff times.\n",
    "\n",
    "To get started, we define the labeling function that will return whether a customer bought a product given purchases within a time frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bought_product(df, product_name):\n",
    "    purchased = df.product_name.str.contains(product_name).any()\n",
    "    return purchased"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the labeling function, we create and configure the label maker for our specific prediction problem. To process 4 weeks of purchases for each customer, we set the `target_entity` to the customer ID and the `window_size` to 4 weeks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "lm = cp.LabelMaker(\n",
    "    target_entity='user_id',\n",
    "    time_index='order_time',\n",
    "    labeling_function=bought_product,\n",
    "    window_size='4w',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This helper function will create a single table from the entity set which is needed for generating label times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def denormalize(es):\n",
    "    df = es['order_products'].df.merge(es['orders'].df).merge(es['users'].df)\n",
    "    return df\n",
    "\n",
    "df = denormalize(es)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, the label maker will continuously search until there is no data to label whether a customer bought bananas given 4 weeks of purchases. This happens when we use `search` with no limit for `num_examples_per_instance` and when we set the `product_name` to bananas. Also, the first cutoff time is set to `2015-03-05` by using the `minimum_data` parameter.\n",
    "\n",
    "If you are running this code yourself, feel free to experiment with any of these parameters! For example, try to predict if a customer will buy \"Limes\" instead of \"Bananas\" or increase your `window_size`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Elapsed: 00:02 | Remaining: 00:00 | Progress: 100%|██████████| user_id: 2000/2000 \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>time</th>\n",
       "      <th>bought_product</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1001</td>\n",
       "      <td>2015-03-15</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1001</td>\n",
       "      <td>2015-04-12</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1002</td>\n",
       "      <td>2015-03-15</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1002</td>\n",
       "      <td>2015-04-12</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1003</td>\n",
       "      <td>2015-03-15</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    user_id       time  bought_product\n",
       "id                                    \n",
       "0      1001 2015-03-15           False\n",
       "1      1001 2015-04-12           False\n",
       "2      1002 2015-03-15           False\n",
       "3      1002 2015-04-12           False\n",
       "4      1003 2015-03-15           False"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "label_times = lm.search(\n",
    "    df.sort_values('order_time'),\n",
    "    minimum_data='2015-03-15',\n",
    "    num_examples_per_instance=2,\n",
    "    product_name='Banana',\n",
    "    verbose=True,\n",
    ")\n",
    "\n",
    "label_times.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see above the our training examples contain three pieces of information: a user id, the last time we can use data before feature engineering (called the \"cutoff time\"), and the label to predict. These are called our \"label times\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use `describe` to print out the distribution with the settings and transforms that were used to make these labels. This is useful as a reference for understanding how the labels were generated from raw data. Also, the label distribution is helpful for determining if we have imbalanced labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label Distribution\n",
      "------------------\n",
      "False      951\n",
      "True       499\n",
      "Total:    1450\n",
      "\n",
      "\n",
      "Settings\n",
      "--------\n",
      "gap                          <4 * Weeks: weekday=6>\n",
      "label_type                                 discrete\n",
      "labeling_function                    bought_product\n",
      "minimum_data                             2015-03-15\n",
      "num_examples_per_instance                         2\n",
      "target_entity                               user_id\n",
      "window_size                  <4 * Weeks: weekday=6>\n",
      "\n",
      "\n",
      "Transforms\n",
      "----------\n",
      "No transforms applied\n",
      "\n"
     ]
    }
   ],
   "source": [
    "label_times.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Automated Feature Engineering\n",
    "With our label times in hand, we can use Deep Feature Synthesis to automatically generate features.\n",
    "\n",
    "When we use DFS, we specify\n",
    "\n",
    "* `target_entity` - the table to build feature for\n",
    "* `cutoff_time` the point in time to calculate the features\n",
    "* `training_window` - the amount of historical data we want to use when calculating features\n",
    "\n",
    "A good way to think of the `cutoff_time` is that it let's us \"pretend\" we are at an earlier point in time when generating our features so we can simulate making predictions. We get this time for each customer from the label times we generated above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Built 78 features\n",
      "Elapsed: 00:10 | Progress: 100%|██████████\n",
      "Number of features 174\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>COUNT(orders)</th>\n",
       "      <th>COUNT(order_products)</th>\n",
       "      <th>PERCENT_TRUE(order_products.reordered)</th>\n",
       "      <th>NUM_UNIQUE(order_products.department)</th>\n",
       "      <th>NUM_UNIQUE(order_products.aisle_id)</th>\n",
       "      <th>NUM_UNIQUE(order_products.product_name)</th>\n",
       "      <th>MODE(order_products.department) = produce</th>\n",
       "      <th>MODE(order_products.department) = dairy eggs</th>\n",
       "      <th>MODE(order_products.department) = beverages</th>\n",
       "      <th>MODE(order_products.department) = snacks</th>\n",
       "      <th>...</th>\n",
       "      <th>MODE(order_products.orders.user_id) = 1994</th>\n",
       "      <th>MODE(order_products.orders.user_id) = 1993</th>\n",
       "      <th>MODE(order_products.orders.user_id) = 1992</th>\n",
       "      <th>MODE(order_products.orders.user_id) = 1991</th>\n",
       "      <th>MODE(order_products.orders.user_id) = 1990</th>\n",
       "      <th>MODE(order_products.orders.user_id) = 1988</th>\n",
       "      <th>MODE(order_products.orders.user_id) = 1987</th>\n",
       "      <th>MODE(order_products.orders.user_id) = 1985</th>\n",
       "      <th>MODE(order_products.orders.user_id) is unknown</th>\n",
       "      <th>bought_product</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1001</th>\n",
       "      <td>2</td>\n",
       "      <td>33</td>\n",
       "      <td>0.424242</td>\n",
       "      <td>11</td>\n",
       "      <td>20</td>\n",
       "      <td>26</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1001</th>\n",
       "      <td>2</td>\n",
       "      <td>33</td>\n",
       "      <td>0.606061</td>\n",
       "      <td>12</td>\n",
       "      <td>20</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1002</th>\n",
       "      <td>5</td>\n",
       "      <td>49</td>\n",
       "      <td>0.244898</td>\n",
       "      <td>8</td>\n",
       "      <td>16</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1002</th>\n",
       "      <td>7</td>\n",
       "      <td>52</td>\n",
       "      <td>0.307692</td>\n",
       "      <td>8</td>\n",
       "      <td>17</td>\n",
       "      <td>44</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1003</th>\n",
       "      <td>6</td>\n",
       "      <td>79</td>\n",
       "      <td>0.670886</td>\n",
       "      <td>12</td>\n",
       "      <td>26</td>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1004</th>\n",
       "      <td>8</td>\n",
       "      <td>169</td>\n",
       "      <td>0.692308</td>\n",
       "      <td>11</td>\n",
       "      <td>19</td>\n",
       "      <td>76</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1004</th>\n",
       "      <td>8</td>\n",
       "      <td>140</td>\n",
       "      <td>0.900000</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>58</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1005</th>\n",
       "      <td>5</td>\n",
       "      <td>54</td>\n",
       "      <td>0.518519</td>\n",
       "      <td>12</td>\n",
       "      <td>28</td>\n",
       "      <td>38</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1005</th>\n",
       "      <td>6</td>\n",
       "      <td>59</td>\n",
       "      <td>0.576271</td>\n",
       "      <td>13</td>\n",
       "      <td>31</td>\n",
       "      <td>37</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1006</th>\n",
       "      <td>2</td>\n",
       "      <td>19</td>\n",
       "      <td>0.105263</td>\n",
       "      <td>9</td>\n",
       "      <td>12</td>\n",
       "      <td>18</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 175 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         COUNT(orders)  COUNT(order_products)  \\\n",
       "user_id                                         \n",
       "1001                 2                     33   \n",
       "1001                 2                     33   \n",
       "1002                 5                     49   \n",
       "1002                 7                     52   \n",
       "1003                 6                     79   \n",
       "1004                 8                    169   \n",
       "1004                 8                    140   \n",
       "1005                 5                     54   \n",
       "1005                 6                     59   \n",
       "1006                 2                     19   \n",
       "\n",
       "         PERCENT_TRUE(order_products.reordered)  \\\n",
       "user_id                                           \n",
       "1001                                   0.424242   \n",
       "1001                                   0.606061   \n",
       "1002                                   0.244898   \n",
       "1002                                   0.307692   \n",
       "1003                                   0.670886   \n",
       "1004                                   0.692308   \n",
       "1004                                   0.900000   \n",
       "1005                                   0.518519   \n",
       "1005                                   0.576271   \n",
       "1006                                   0.105263   \n",
       "\n",
       "         NUM_UNIQUE(order_products.department)  \\\n",
       "user_id                                          \n",
       "1001                                        11   \n",
       "1001                                        12   \n",
       "1002                                         8   \n",
       "1002                                         8   \n",
       "1003                                        12   \n",
       "1004                                        11   \n",
       "1004                                         9   \n",
       "1005                                        12   \n",
       "1005                                        13   \n",
       "1006                                         9   \n",
       "\n",
       "         NUM_UNIQUE(order_products.aisle_id)  \\\n",
       "user_id                                        \n",
       "1001                                      20   \n",
       "1001                                      20   \n",
       "1002                                      16   \n",
       "1002                                      17   \n",
       "1003                                      26   \n",
       "1004                                      19   \n",
       "1004                                      16   \n",
       "1005                                      28   \n",
       "1005                                      31   \n",
       "1006                                      12   \n",
       "\n",
       "         NUM_UNIQUE(order_products.product_name)  \\\n",
       "user_id                                            \n",
       "1001                                          26   \n",
       "1001                                          25   \n",
       "1002                                          40   \n",
       "1002                                          44   \n",
       "1003                                          41   \n",
       "1004                                          76   \n",
       "1004                                          58   \n",
       "1005                                          38   \n",
       "1005                                          37   \n",
       "1006                                          18   \n",
       "\n",
       "         MODE(order_products.department) = produce  \\\n",
       "user_id                                              \n",
       "1001                                             0   \n",
       "1001                                             0   \n",
       "1002                                             0   \n",
       "1002                                             0   \n",
       "1003                                             0   \n",
       "1004                                             1   \n",
       "1004                                             1   \n",
       "1005                                             0   \n",
       "1005                                             0   \n",
       "1006                                             1   \n",
       "\n",
       "         MODE(order_products.department) = dairy eggs  \\\n",
       "user_id                                                 \n",
       "1001                                                0   \n",
       "1001                                                0   \n",
       "1002                                                0   \n",
       "1002                                                0   \n",
       "1003                                                1   \n",
       "1004                                                0   \n",
       "1004                                                0   \n",
       "1005                                                1   \n",
       "1005                                                1   \n",
       "1006                                                0   \n",
       "\n",
       "         MODE(order_products.department) = beverages  \\\n",
       "user_id                                                \n",
       "1001                                               0   \n",
       "1001                                               0   \n",
       "1002                                               0   \n",
       "1002                                               0   \n",
       "1003                                               0   \n",
       "1004                                               0   \n",
       "1004                                               0   \n",
       "1005                                               0   \n",
       "1005                                               0   \n",
       "1006                                               0   \n",
       "\n",
       "         MODE(order_products.department) = snacks  ...  \\\n",
       "user_id                                            ...   \n",
       "1001                                            1  ...   \n",
       "1001                                            1  ...   \n",
       "1002                                            1  ...   \n",
       "1002                                            1  ...   \n",
       "1003                                            0  ...   \n",
       "1004                                            0  ...   \n",
       "1004                                            0  ...   \n",
       "1005                                            0  ...   \n",
       "1005                                            0  ...   \n",
       "1006                                            0  ...   \n",
       "\n",
       "         MODE(order_products.orders.user_id) = 1994  \\\n",
       "user_id                                               \n",
       "1001                                              0   \n",
       "1001                                              0   \n",
       "1002                                              0   \n",
       "1002                                              0   \n",
       "1003                                              0   \n",
       "1004                                              0   \n",
       "1004                                              0   \n",
       "1005                                              0   \n",
       "1005                                              0   \n",
       "1006                                              0   \n",
       "\n",
       "         MODE(order_products.orders.user_id) = 1993  \\\n",
       "user_id                                               \n",
       "1001                                              0   \n",
       "1001                                              0   \n",
       "1002                                              0   \n",
       "1002                                              0   \n",
       "1003                                              0   \n",
       "1004                                              0   \n",
       "1004                                              0   \n",
       "1005                                              0   \n",
       "1005                                              0   \n",
       "1006                                              0   \n",
       "\n",
       "         MODE(order_products.orders.user_id) = 1992  \\\n",
       "user_id                                               \n",
       "1001                                              0   \n",
       "1001                                              0   \n",
       "1002                                              0   \n",
       "1002                                              0   \n",
       "1003                                              0   \n",
       "1004                                              0   \n",
       "1004                                              0   \n",
       "1005                                              0   \n",
       "1005                                              0   \n",
       "1006                                              0   \n",
       "\n",
       "         MODE(order_products.orders.user_id) = 1991  \\\n",
       "user_id                                               \n",
       "1001                                              0   \n",
       "1001                                              0   \n",
       "1002                                              0   \n",
       "1002                                              0   \n",
       "1003                                              0   \n",
       "1004                                              0   \n",
       "1004                                              0   \n",
       "1005                                              0   \n",
       "1005                                              0   \n",
       "1006                                              0   \n",
       "\n",
       "         MODE(order_products.orders.user_id) = 1990  \\\n",
       "user_id                                               \n",
       "1001                                              0   \n",
       "1001                                              0   \n",
       "1002                                              0   \n",
       "1002                                              0   \n",
       "1003                                              0   \n",
       "1004                                              0   \n",
       "1004                                              0   \n",
       "1005                                              0   \n",
       "1005                                              0   \n",
       "1006                                              0   \n",
       "\n",
       "         MODE(order_products.orders.user_id) = 1988  \\\n",
       "user_id                                               \n",
       "1001                                              0   \n",
       "1001                                              0   \n",
       "1002                                              0   \n",
       "1002                                              0   \n",
       "1003                                              0   \n",
       "1004                                              0   \n",
       "1004                                              0   \n",
       "1005                                              0   \n",
       "1005                                              0   \n",
       "1006                                              0   \n",
       "\n",
       "         MODE(order_products.orders.user_id) = 1987  \\\n",
       "user_id                                               \n",
       "1001                                              0   \n",
       "1001                                              0   \n",
       "1002                                              0   \n",
       "1002                                              0   \n",
       "1003                                              0   \n",
       "1004                                              0   \n",
       "1004                                              0   \n",
       "1005                                              0   \n",
       "1005                                              0   \n",
       "1006                                              0   \n",
       "\n",
       "         MODE(order_products.orders.user_id) = 1985  \\\n",
       "user_id                                               \n",
       "1001                                              0   \n",
       "1001                                              0   \n",
       "1002                                              0   \n",
       "1002                                              0   \n",
       "1003                                              0   \n",
       "1004                                              0   \n",
       "1004                                              0   \n",
       "1005                                              0   \n",
       "1005                                              0   \n",
       "1006                                              0   \n",
       "\n",
       "         MODE(order_products.orders.user_id) is unknown  bought_product  \n",
       "user_id                                                                  \n",
       "1001                                                  1           False  \n",
       "1001                                                  1           False  \n",
       "1002                                                  1           False  \n",
       "1002                                                  1           False  \n",
       "1003                                                  1           False  \n",
       "1004                                                  1            True  \n",
       "1004                                                  1            True  \n",
       "1005                                                  1           False  \n",
       "1005                                                  1            True  \n",
       "1006                                                  1           False  \n",
       "\n",
       "[10 rows x 175 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_matrix, features = ft.dfs(target_entity=\"users\", \n",
    "                                  cutoff_time=label_times,\n",
    "                                  training_window=ft.Timedelta(\"60 days\"),\n",
    "                                  entityset=es,\n",
    "                                  verbose=True)\n",
    "# encode categorical values\n",
    "fm_encoded, features_encoded = ft.encode_features(feature_matrix,\n",
    "                                                  features)\n",
    "\n",
    "print(\"Number of features %s\" % len(features_encoded))\n",
    "fm_encoded.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4. Machine Learning\n",
    "\n",
    "Using the default parameters, we generated 78 potential features for our prediction problem. With a few simple commands, this feature matrix can be used for machine learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = utils.merge_features_labels(fm_encoded, label_times)\n",
    "X.drop([\"user_id\", \"time\"], axis=1, inplace=True)\n",
    "X = X.fillna(0)\n",
    "y = X.pop(\"bought_product\").astype('bool')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's train a Random Forest and validate using 3-fold cross validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   3 out of   3 | elapsed:    3.6s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'AUC 0.83 +/- 0.01'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = RandomForestClassifier(n_estimators=400, n_jobs=-1)\n",
    "scores = cross_val_score(estimator=clf,X=X, y=y, cv=3,\n",
    "                         scoring=\"roc_auc\", verbose=True)\n",
    "\n",
    "\"AUC %.2f +/- %.2f\" % (scores.mean(), scores.std())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see we perform noticably better than guessing! However, we have a pretty high difference in performance accross folds.\n",
    "\n",
    "To improve this, let's identify the top 20 features using a Random Forest and then perform machine learning on the whole dataset (all of the partions)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1: Feature: COUNT(order_products WHERE product_name = Banana), 0.064\n",
      "2: Feature: COUNT(order_products WHERE product_name = Bag of Organic Bananas), 0.056\n",
      "3: Feature: COUNT(order_products WHERE department = produce), 0.033\n",
      "4: Feature: MODE(orders.MODE(order_products.product_name)) is unknown, 0.023\n",
      "5: Feature: MEAN(orders.COUNT(order_products)), 0.020\n",
      "6: Feature: COUNT(order_products WHERE department = dairy eggs), 0.020\n",
      "7: Feature: SUM(orders.NUM_UNIQUE(order_products.product_name)), 0.019\n",
      "8: Feature: MEAN(orders.NUM_UNIQUE(order_products.product_name)), 0.019\n",
      "9: Feature: COUNT(order_products), 0.018\n",
      "10: Feature: NUM_UNIQUE(order_products.product_name), 0.018\n",
      "11: Feature: SUM(orders.NUM_UNIQUE(order_products.aisle_id)), 0.017\n",
      "12: Feature: MEAN(orders.NUM_UNIQUE(order_products.aisle_id)), 0.017\n",
      "13: Feature: STD(orders.PERCENT_TRUE(order_products.reordered)), 0.017\n",
      "14: Feature: MAX(orders.COUNT(order_products)), 0.016\n",
      "15: Feature: MODE(order_products.product_name) is unknown, 0.016\n",
      "16: Feature: MEAN(orders.NUM_UNIQUE(order_products.department)), 0.016\n",
      "17: Feature: MAX(orders.NUM_UNIQUE(order_products.product_name)), 0.016\n",
      "18: Feature: MEAN(orders.PERCENT_TRUE(order_products.reordered)), 0.015\n",
      "19: Feature: MODE(orders.MODE(order_products.aisle_id)) = 24, 0.015\n",
      "20: Feature: PERCENT_TRUE(order_products.reordered), 0.015\n"
     ]
    }
   ],
   "source": [
    "clf.fit(X, y)\n",
    "top_features = utils.feature_importances(clf, features_encoded, n=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To persist this features, we can save them to disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "ft.save_features(top_features, \"top_features\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Understanding feature engineering in Featuretools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before moving forward, take a look at the feature we created. You will see that they are more than just simple transformations of columns in our raw data. Instead, they aggregations (and sometimes stacking of aggregations) across the relationships in our dataset. If you're curious how this works, learn about the Deep Feature Synthesis algorithm in our documentation [here](https://docs.featuretools.com/en/stable/automated_feature_engineering/afe.html).\n",
    "\n",
    "DFS is so powerful because with no manual work, the library figured out that historical purchases of bananas are important for predicting future purchases. Additionally, it surfaces that purchasing dairy or eggs and reordering behavior are important features. \n",
    "\n",
    "Even though these features are intuitive, Deep Feature Synthesis will automatically adapt as we change the prediction problem, saving us the time of manually brainstorming and implementing these data transformation. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scaling to full dataset\n",
    "\n",
    "Once we have written the pipeline for one partition, we can easily scale it out to the full dataset using [Dask](https://dask.pydata.org). A similar pipeline could also be built using [Spark](https://spark.apache.org/docs/2.2.0/api/python/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "pbar = ProgressBar()\n",
    "pbar.register()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we assemble our partitions and map them to entity sets using the function from before. A single partition contains all the data for each user within it, so this computation is easily parallelized. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = \"partitioned_data/\"\n",
    "#_, dirnames, _ = os.walk(path).next()\n",
    "dirnames = [os.path.join(path, d) for d in os.listdir(path)]\n",
    "b = bag.from_sequence(dirnames)\n",
    "entity_sets = b.map(utils.load_entityset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we create label times for each entity set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dask.bag<add_label_times, npartitions=104>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def add_label_times(es):\n",
    "    df = denormalize(es)\n",
    "    lt = lm.search(\n",
    "        df.sort_values('order_time'),\n",
    "        minimum_data='2015-03-01',\n",
    "        num_examples_per_instance=2,\n",
    "        product_name='Banana',\n",
    "        verbose=False,\n",
    "    )\n",
    "    return lt, es\n",
    "\n",
    "label_times = entity_sets.map(add_label_times)\n",
    "label_times"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load in the features from before\n",
    "top_features = ft.load_features(\"top_features\")\n",
    "feature_matrices = label_times.map(utils.calculate_feature_matrix, features=top_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The compute method can take an argument `num_workers` which defaults to using all of the cores on your computer. If you don't want it to do that, you can explicitly specify a number of workers with `feature_matrices.compute(num_workers=2)` where you can replace 2 with the number of cores you want to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[########################################] | 100% Completed | 12min 59.4s\n"
     ]
    }
   ],
   "source": [
    "fms_out = feature_matrices.compute()\n",
    "X = pd.concat(fms_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we repeat the same machine learning steps from the sample dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.drop([\"user_id\", \"time\"], axis=1, inplace=True)\n",
    "X = X.fillna(0)\n",
    "y = X.pop(\"bought_product\").astype('bool')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   3 out of   3 | elapsed:  1.8min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'AUC 0.82 +/- 0.00'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = RandomForestClassifier(n_estimators=400, n_jobs=-1)\n",
    "scores = cross_val_score(estimator=clf,X=X, y=y, cv=3,\n",
    "                         scoring=\"roc_auc\", verbose=True)\n",
    "\n",
    "\"AUC %.2f +/- %.2f\" % (scores.mean(), scores.std())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now we that our accuracy has stabalized across folds, giving us much more confidence in our model.\n",
    "\n",
    "Now, let's look at the top features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1: Feature: COUNT(order_products WHERE product_name = Banana), 0.117\n",
      "2: Feature: COUNT(order_products WHERE product_name = Bag of Organic Bananas), 0.081\n",
      "3: Feature: COUNT(order_products WHERE department = produce), 0.065\n",
      "4: Feature: STD(orders.PERCENT_TRUE(order_products.reordered)), 0.061\n",
      "5: Feature: MEAN(orders.PERCENT_TRUE(order_products.reordered)), 0.060\n",
      "6: Feature: PERCENT_TRUE(order_products.reordered), 0.056\n",
      "7: Feature: MEAN(orders.NUM_UNIQUE(order_products.department)), 0.051\n",
      "8: Feature: MEAN(orders.NUM_UNIQUE(order_products.aisle_id)), 0.049\n",
      "9: Feature: COUNT(order_products WHERE department = dairy eggs), 0.047\n",
      "10: Feature: MEAN(orders.NUM_UNIQUE(order_products.product_name)), 0.046\n",
      "11: Feature: COUNT(order_products), 0.045\n",
      "12: Feature: SUM(orders.NUM_UNIQUE(order_products.aisle_id)), 0.045\n",
      "13: Feature: MEAN(orders.COUNT(order_products)), 0.045\n",
      "14: Feature: SUM(orders.NUM_UNIQUE(order_products.product_name)), 0.042\n",
      "15: Feature: NUM_UNIQUE(order_products.product_name), 0.042\n",
      "16: Feature: MAX(orders.COUNT(order_products)), 0.035\n",
      "17: Feature: MAX(orders.NUM_UNIQUE(order_products.product_name)), 0.035\n",
      "18: Feature: MODE(orders.MODE(order_products.product_name)) is unknown, 0.034\n",
      "19: Feature: MODE(order_products.product_name) is unknown, 0.030\n",
      "20: Feature: MODE(orders.MODE(order_products.aisle_id)) = 24, 0.014\n"
     ]
    }
   ],
   "source": [
    "clf.fit(X, y)\n",
    "top_features = utils.feature_importances(clf, top_features, n=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the top features shifted around a bit, but mostly stayed the same."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next Steps\n",
    "\n",
    "While this is an end-to-end example of going from raw data to a trained machine learning model, it is necessary to do further exploration before claiming we've built something impact full.\n",
    "\n",
    "Fortunately, Featuretools makes it easy to build structured data science pipeline. As a next steps, you could\n",
    "* Further validate these results by creating feature vectors at different cutoff times\n",
    "* Define other prediction problems for this dataset (you can even change the entity you are making predictions on!)\n",
    "* Save feature matrices to disk as CSVs so they can be reused with different problems without recalculating\n",
    "* Experiment with parameters to Deep Feature Synthesis\n",
    "* Create custom primitives for DFS. More info [here](https://docs.featuretools.com/en/stable/automated_feature_engineering/primitives.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p>\n",
    "    <img src=\"https://www.featurelabs.com/wp-content/uploads/2017/12/logo.png\" alt=\"Featuretools\" />\n",
    "</p>\n",
    "\n",
    "\n",
    "Featuretools was created by the developers at [Feature Labs](https://www.featurelabs.com/). If building impactful data science pipelines is important to you or your business, please [get in touch](https://www.featurelabs.com/contact)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}