tutorial/tutorial03-Shapash-overview-model-in-production.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Shapash model in production - Overview\n",
    "\n",
    "<b>With this tutorial you:</b><br />\n",
    "Understand how to create a Shapash SmartPredictor to make prediction and have local explanation in production\n",
    "with a simple use case.<br />\n",
    "\n",
    "This tutorial describes the different steps from training the model to Shapash SmartPredictor deployment.\n",
    "A more detailed tutorial allows you to know more about the SmartPredictor Object.\n",
    "\n",
    "Contents:\n",
    "- Build a Regressor\n",
    "- Compile Shapash SmartExplainer\n",
    "- From Shapash SmartExplainer to SmartPredictor\n",
    "- Save Shapash Smartpredictor Object in pickle file\n",
    "- Make a prediction\n",
    "\n",
    "Data from Kaggle [House Prices](https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from category_encoders import OrdinalEncoder\n",
    "from lightgbm import LGBMRegressor\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1 : Exploration and training of the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building Supervized Model "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we train a Machine Learning supervized model with our data House Prices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from shapash.data.data_loader import data_loading\n",
    "house_df, house_dict = data_loading('house_prices')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_df=house_df['SalePrice'].to_frame()\n",
    "X_df=house_df[house_df.columns.difference(['SalePrice'])]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Preprocessing step "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Encoding Categorical Features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from category_encoders import OrdinalEncoder\n",
    "\n",
    "categorical_features = [col for col in X_df.columns if X_df[col].dtype == 'object']\n",
    "\n",
    "encoder = OrdinalEncoder(cols=categorical_features,\n",
    "                         handle_unknown='ignore',\n",
    "                         return_df=True).fit(X_df)\n",
    "\n",
    "X_encoded=encoder.transform(X_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train / Test Split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "Xtrain, Xtest, ytrain, ytest = train_test_split(X_encoded, y_df, train_size=0.75, random_state=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model Fitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "regressor = LGBMRegressor(n_estimators=200).fit(Xtrain, ytrain)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = pd.DataFrame(regressor.predict(Xtest), columns=['pred'], index=Xtest.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Understand my model with shapash"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we use the SmartExplainer Object from shapash.\n",
    "- It allows users to understand how the model works with the specified data. \n",
    "- This object must be used only for data mining step. Shapash provides another object for deployment.\n",
    "- In this tutorial, we are not exploring possibilites of the SmartExplainer but others will. (see other tutorials)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Declare and Compile SmartExplainer "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from shapash.explainer.smart_explainer import SmartExplainer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use wording on features names to better understanding results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we use a wording to rename our features label with more understandable terms. It's usefull to make our local explainability more operational and understandable for users.\n",
    "- To do this, we use the house_dict dictionary which maps a description to each features.\n",
    "- We can then use it features_dict as a parameter of the SmartExplainer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "xpl = SmartExplainer(features_dict=house_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**compile()<br />** This method is the first step to understand model and prediction.<br /> It performs the sorting\n",
    "of contributions, the reverse preprocessing steps and all the calculations necessary for\n",
    "a quick display of plots and efficient summary of explanation. (see SmartExplainer documentation and tutorials)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Backend: Shap TreeExplainer\n"
     ]
    }
   ],
   "source": [
    "xpl.compile(\n",
    "            x=Xtest,\n",
    "            model=regressor,\n",
    "            preprocessing=encoder, # Optional: compile step can use inverse_transform method\n",
    "            y_pred=y_pred # Optional\n",
    "            )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Understand results of your trained model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we can easily get a first summary of the explanation of the model results.\n",
    "- Here, we chose to get the 3 most contributive features for each prediction.\n",
    "- We used a wording to get features names more understandable in operationnal case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pred</th>\n",
       "      <th>feature_1</th>\n",
       "      <th>value_1</th>\n",
       "      <th>contribution_1</th>\n",
       "      <th>feature_2</th>\n",
       "      <th>value_2</th>\n",
       "      <th>contribution_2</th>\n",
       "      <th>feature_3</th>\n",
       "      <th>value_3</th>\n",
       "      <th>contribution_3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>259</th>\n",
       "      <td>209141.256921</td>\n",
       "      <td>Ground living area square feet</td>\n",
       "      <td>1792</td>\n",
       "      <td>13710.4</td>\n",
       "      <td>Overall material and finish of the house</td>\n",
       "      <td>7</td>\n",
       "      <td>12776.3</td>\n",
       "      <td>Total square feet of basement area</td>\n",
       "      <td>963</td>\n",
       "      <td>-5103.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>268</th>\n",
       "      <td>178734.474531</td>\n",
       "      <td>Ground living area square feet</td>\n",
       "      <td>2192</td>\n",
       "      <td>29747</td>\n",
       "      <td>Overall material and finish of the house</td>\n",
       "      <td>5</td>\n",
       "      <td>-26151.3</td>\n",
       "      <td>Overall condition of the house</td>\n",
       "      <td>8</td>\n",
       "      <td>9190.84</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>289</th>\n",
       "      <td>113950.844570</td>\n",
       "      <td>Overall material and finish of the house</td>\n",
       "      <td>5</td>\n",
       "      <td>-24730</td>\n",
       "      <td>Ground living area square feet</td>\n",
       "      <td>900</td>\n",
       "      <td>-16342.6</td>\n",
       "      <td>Total square feet of basement area</td>\n",
       "      <td>882</td>\n",
       "      <td>-5922.64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>650</th>\n",
       "      <td>74957.162142</td>\n",
       "      <td>Overall material and finish of the house</td>\n",
       "      <td>4</td>\n",
       "      <td>-33927.7</td>\n",
       "      <td>Ground living area square feet</td>\n",
       "      <td>630</td>\n",
       "      <td>-23234.4</td>\n",
       "      <td>Total square feet of basement area</td>\n",
       "      <td>630</td>\n",
       "      <td>-11687.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1234</th>\n",
       "      <td>135305.243500</td>\n",
       "      <td>Overall material and finish of the house</td>\n",
       "      <td>5</td>\n",
       "      <td>-25445.7</td>\n",
       "      <td>Ground living area square feet</td>\n",
       "      <td>1188</td>\n",
       "      <td>-11476.6</td>\n",
       "      <td>Condition of sale</td>\n",
       "      <td>Abnormal Sale</td>\n",
       "      <td>-5071.82</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               pred                                 feature_1 value_1  \\\n",
       "259   209141.256921            Ground living area square feet    1792   \n",
       "268   178734.474531            Ground living area square feet    2192   \n",
       "289   113950.844570  Overall material and finish of the house       5   \n",
       "650    74957.162142  Overall material and finish of the house       4   \n",
       "1234  135305.243500  Overall material and finish of the house       5   \n",
       "\n",
       "     contribution_1                                 feature_2 value_2  \\\n",
       "259         13710.4  Overall material and finish of the house       7   \n",
       "268           29747  Overall material and finish of the house       5   \n",
       "289          -24730            Ground living area square feet     900   \n",
       "650        -33927.7            Ground living area square feet     630   \n",
       "1234       -25445.7            Ground living area square feet    1188   \n",
       "\n",
       "     contribution_2                           feature_3        value_3  \\\n",
       "259         12776.3  Total square feet of basement area            963   \n",
       "268        -26151.3      Overall condition of the house              8   \n",
       "289        -16342.6  Total square feet of basement area            882   \n",
       "650        -23234.4  Total square feet of basement area            630   \n",
       "1234       -11476.6                   Condition of sale  Abnormal Sale   \n",
       "\n",
       "     contribution_3  \n",
       "259        -5103.03  \n",
       "268         9190.84  \n",
       "289        -5922.64  \n",
       "650        -11687.9  \n",
       "1234       -5071.82  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xpl.to_pandas(max_contrib=3).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2 : SmartPredictor in production"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Switch from SmartExplainer to SmartPredictor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you are satisfied by your results and the explainablity given by Shapash, you can use the SmartPredictor object for deployment. \n",
    "- In this section, we learn how to easily switch from SmartExplainer to a SmartPredictor.\n",
    "- SmartPredictor allows you to make predictions, detail and summarize contributions on new data automatically.\n",
    "- It only keeps the attributes needed for deployment to be lighter than the SmartExplainer object. \n",
    "- SmartPredictor performs additional consistency checks before deployment.\n",
    "- SmartPredictor allows you to configure the way of summary to suit your use cases.\n",
    "- It can be used with API or in batch mode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor = xpl.to_smartpredictor()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save and Load your SmartPredictor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can easily save and load your SmartPredictor Object in pickle."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save your SmartPredictor in Pickle File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor.save('./predictor.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load your SmartPredictor in Pickle File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from shapash.utils.load_smartpredictor import load_smartpredictor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor_load = load_smartpredictor('./predictor.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make a prediction with your SmartPredictor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to make new predictions and summarize local explainability of your model on new datasets, you can use the method add_input of the SmartPredictor.\n",
    "- The add_input method is the first step to add a dataset for prediction and explainability.\n",
    "- It checks the structure of the dataset, the prediction and the contribution if specified. \n",
    "- It applies the preprocessing specified in the initialisation and reorder the features with the order used by the model. (see the documentation of this method)\n",
    "- In API mode, this method can handle dictionnaries data which can be received from a GET or a POST request."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Add data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The x input in add_input method doesn't have to be encoded, add_input applies preprocessing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor_load.add_input(x=X_df, ypred=y_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Make prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we can see ypred is the one given in add_input method by checking the attribute data[\"ypred\"]. If not specified, it will automatically be computed in the method. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Id</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>208500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>181500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>223500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>140000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>250000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    SalePrice\n",
       "Id           \n",
       "1      208500\n",
       "2      181500\n",
       "3      223500\n",
       "4      140000\n",
       "5      250000"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictor_load.data[\"ypred\"].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get detailed explanability associated to the prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use the method detail_contributions to see the detailed contributions of each of your features for each row of your new dataset.\n",
    "- For classification problems, it automatically associates contributions with the right predicted label. \n",
    "- The predicted label can be computed automatically in the method or you can specify an ypred with add_input method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "detailed_contributions = predictor_load.detail_contributions()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SalePrice</th>\n",
       "      <th>1stFlrSF</th>\n",
       "      <th>2ndFlrSF</th>\n",
       "      <th>3SsnPorch</th>\n",
       "      <th>BedroomAbvGr</th>\n",
       "      <th>BldgType</th>\n",
       "      <th>BsmtCond</th>\n",
       "      <th>BsmtExposure</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>BsmtFinSF2</th>\n",
       "      <th>...</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>Street</th>\n",
       "      <th>TotRmsAbvGrd</th>\n",
       "      <th>TotalBsmtSF</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>WoodDeckSF</th>\n",
       "      <th>YearBuilt</th>\n",
       "      <th>YearRemodAdd</th>\n",
       "      <th>YrSold</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>208500</td>\n",
       "      <td>-1104.994176</td>\n",
       "      <td>1281.445856</td>\n",
       "      <td>0.0</td>\n",
       "      <td>375.679661</td>\n",
       "      <td>12.259902</td>\n",
       "      <td>157.224629</td>\n",
       "      <td>-233.025420</td>\n",
       "      <td>-738.445396</td>\n",
       "      <td>-59.294761</td>\n",
       "      <td>...</td>\n",
       "      <td>-104.645827</td>\n",
       "      <td>-351.621116</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-498.228775</td>\n",
       "      <td>-5165.503476</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-944.040092</td>\n",
       "      <td>3870.961681</td>\n",
       "      <td>2219.313761</td>\n",
       "      <td>17.478037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>181500</td>\n",
       "      <td>2249.403962</td>\n",
       "      <td>-655.861167</td>\n",
       "      <td>0.0</td>\n",
       "      <td>123.907278</td>\n",
       "      <td>-9.270166</td>\n",
       "      <td>139.431860</td>\n",
       "      <td>2699.247506</td>\n",
       "      <td>5102.469936</td>\n",
       "      <td>-84.771341</td>\n",
       "      <td>...</td>\n",
       "      <td>-153.842142</td>\n",
       "      <td>-236.526862</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-705.112993</td>\n",
       "      <td>2988.981279</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2090.785074</td>\n",
       "      <td>323.902986</td>\n",
       "      <td>-3861.776078</td>\n",
       "      <td>424.382977</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>223500</td>\n",
       "      <td>-1426.795115</td>\n",
       "      <td>-616.113112</td>\n",
       "      <td>0.0</td>\n",
       "      <td>369.536957</td>\n",
       "      <td>9.210944</td>\n",
       "      <td>199.213726</td>\n",
       "      <td>1032.288162</td>\n",
       "      <td>-92.179454</td>\n",
       "      <td>-93.169310</td>\n",
       "      <td>...</td>\n",
       "      <td>-91.178667</td>\n",
       "      <td>-280.832451</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-324.734175</td>\n",
       "      <td>-5338.340597</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-777.746743</td>\n",
       "      <td>3837.761102</td>\n",
       "      <td>2192.921648</td>\n",
       "      <td>-98.965041</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>140000</td>\n",
       "      <td>-653.873832</td>\n",
       "      <td>121.459865</td>\n",
       "      <td>0.0</td>\n",
       "      <td>307.677892</td>\n",
       "      <td>9.720006</td>\n",
       "      <td>252.786934</td>\n",
       "      <td>-530.156452</td>\n",
       "      <td>-2987.649814</td>\n",
       "      <td>-77.039912</td>\n",
       "      <td>...</td>\n",
       "      <td>-114.608224</td>\n",
       "      <td>-338.435699</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-635.065828</td>\n",
       "      <td>-6548.453864</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-974.503140</td>\n",
       "      <td>-3386.361210</td>\n",
       "      <td>-5232.537839</td>\n",
       "      <td>1633.763619</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>250000</td>\n",
       "      <td>-9531.577733</td>\n",
       "      <td>-1097.620788</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1574.988323</td>\n",
       "      <td>7.453569</td>\n",
       "      <td>130.470247</td>\n",
       "      <td>623.939546</td>\n",
       "      <td>-2396.572526</td>\n",
       "      <td>-92.929525</td>\n",
       "      <td>...</td>\n",
       "      <td>-481.118248</td>\n",
       "      <td>-366.250007</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-4733.603060</td>\n",
       "      <td>-4675.706762</td>\n",
       "      <td>0.0</td>\n",
       "      <td>165.653455</td>\n",
       "      <td>2334.652063</td>\n",
       "      <td>1355.358932</td>\n",
       "      <td>-395.126541</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 73 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    SalePrice     1stFlrSF     2ndFlrSF  3SsnPorch  BedroomAbvGr   BldgType  \\\n",
       "Id                                                                            \n",
       "1      208500 -1104.994176  1281.445856        0.0    375.679661  12.259902   \n",
       "2      181500  2249.403962  -655.861167        0.0    123.907278  -9.270166   \n",
       "3      223500 -1426.795115  -616.113112        0.0    369.536957   9.210944   \n",
       "4      140000  -653.873832   121.459865        0.0    307.677892   9.720006   \n",
       "5      250000 -9531.577733 -1097.620788        0.0  -1574.988323   7.453569   \n",
       "\n",
       "      BsmtCond  BsmtExposure   BsmtFinSF1  BsmtFinSF2  ...    SaleType  \\\n",
       "Id                                                     ...               \n",
       "1   157.224629   -233.025420  -738.445396  -59.294761  ... -104.645827   \n",
       "2   139.431860   2699.247506  5102.469936  -84.771341  ... -153.842142   \n",
       "3   199.213726   1032.288162   -92.179454  -93.169310  ...  -91.178667   \n",
       "4   252.786934   -530.156452 -2987.649814  -77.039912  ... -114.608224   \n",
       "5   130.470247    623.939546 -2396.572526  -92.929525  ... -481.118248   \n",
       "\n",
       "    ScreenPorch  Street  TotRmsAbvGrd  TotalBsmtSF  Utilities   WoodDeckSF  \\\n",
       "Id                                                                           \n",
       "1   -351.621116     0.0   -498.228775 -5165.503476        0.0  -944.040092   \n",
       "2   -236.526862     0.0   -705.112993  2988.981279        0.0  2090.785074   \n",
       "3   -280.832451     0.0   -324.734175 -5338.340597        0.0  -777.746743   \n",
       "4   -338.435699     0.0   -635.065828 -6548.453864        0.0  -974.503140   \n",
       "5   -366.250007     0.0  -4733.603060 -4675.706762        0.0   165.653455   \n",
       "\n",
       "      YearBuilt  YearRemodAdd       YrSold  \n",
       "Id                                          \n",
       "1   3870.961681   2219.313761    17.478037  \n",
       "2    323.902986  -3861.776078   424.382977  \n",
       "3   3837.761102   2192.921648   -98.965041  \n",
       "4  -3386.361210  -5232.537839  1633.763619  \n",
       "5   2334.652063   1355.358932  -395.126541  \n",
       "\n",
       "[5 rows x 73 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "detailed_contributions.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summarize explanability of the predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- You can use the summarize method to summarize your local explainability\n",
    "- This summary can be configured with modify_mask method so that you have explainability that meets your operational needs.\n",
    "- When you initialize the SmartPredictor, you can also specify  :\n",
    ">- postprocessing: to apply a wording to several values of your dataset.\n",
    ">- label_dict: to rename your label for classification problems.\n",
    ">- features_dict: to rename your features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor_load.modify_mask(max_contrib=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "explanation = predictor_load.summarize()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, here, we chose to build a summary with 3 most contributive features of your dataset.\n",
    "- As you can see below, the wording defined in the first step of this tutorial has been kept by the SmartPredictor and used in the summarize method. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SalePrice</th>\n",
       "      <th>feature_1</th>\n",
       "      <th>value_1</th>\n",
       "      <th>contribution_1</th>\n",
       "      <th>feature_2</th>\n",
       "      <th>value_2</th>\n",
       "      <th>contribution_2</th>\n",
       "      <th>feature_3</th>\n",
       "      <th>value_3</th>\n",
       "      <th>contribution_3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>208500</td>\n",
       "      <td>Overall material and finish of the house</td>\n",
       "      <td>7</td>\n",
       "      <td>8248.82</td>\n",
       "      <td>Total square feet of basement area</td>\n",
       "      <td>856</td>\n",
       "      <td>-5165.5</td>\n",
       "      <td>Original construction date</td>\n",
       "      <td>2003</td>\n",
       "      <td>3870.96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>181500</td>\n",
       "      <td>Overall material and finish of the house</td>\n",
       "      <td>6</td>\n",
       "      <td>-14419.4</td>\n",
       "      <td>Ground living area square feet</td>\n",
       "      <td>1262</td>\n",
       "      <td>-9238.07</td>\n",
       "      <td>Overall condition of the house</td>\n",
       "      <td>8</td>\n",
       "      <td>6371.61</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>223500</td>\n",
       "      <td>Ground living area square feet</td>\n",
       "      <td>1786</td>\n",
       "      <td>15880.4</td>\n",
       "      <td>Overall material and finish of the house</td>\n",
       "      <td>7</td>\n",
       "      <td>9651.28</td>\n",
       "      <td>Size of garage in square feet</td>\n",
       "      <td>608</td>\n",
       "      <td>6259.46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>140000</td>\n",
       "      <td>Total square feet of basement area</td>\n",
       "      <td>756</td>\n",
       "      <td>-6548.45</td>\n",
       "      <td>Remodel date</td>\n",
       "      <td>1970</td>\n",
       "      <td>-5232.54</td>\n",
       "      <td>Size of garage in square feet</td>\n",
       "      <td>642</td>\n",
       "      <td>4384.29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>250000</td>\n",
       "      <td>Overall material and finish of the house</td>\n",
       "      <td>8</td>\n",
       "      <td>55722.1</td>\n",
       "      <td>Ground living area square feet</td>\n",
       "      <td>2198</td>\n",
       "      <td>17176.5</td>\n",
       "      <td>Size of garage in square feet</td>\n",
       "      <td>836</td>\n",
       "      <td>14907.7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   SalePrice                                 feature_1 value_1 contribution_1  \\\n",
       "1     208500  Overall material and finish of the house       7        8248.82   \n",
       "2     181500  Overall material and finish of the house       6       -14419.4   \n",
       "3     223500            Ground living area square feet    1786        15880.4   \n",
       "4     140000        Total square feet of basement area     756       -6548.45   \n",
       "5     250000  Overall material and finish of the house       8        55722.1   \n",
       "\n",
       "                                  feature_2 value_2 contribution_2  \\\n",
       "1        Total square feet of basement area     856        -5165.5   \n",
       "2            Ground living area square feet    1262       -9238.07   \n",
       "3  Overall material and finish of the house       7        9651.28   \n",
       "4                              Remodel date    1970       -5232.54   \n",
       "5            Ground living area square feet    2198        17176.5   \n",
       "\n",
       "                        feature_3 value_3 contribution_3  \n",
       "1      Original construction date    2003        3870.96  \n",
       "2  Overall condition of the house       8        6371.61  \n",
       "3   Size of garage in square feet     608        6259.46  \n",
       "4   Size of garage in square feet     642        4384.29  \n",
       "5   Size of garage in square feet     836        14907.7  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "explanation.head()"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Aucun(e)",
  "kernelspec": {
   "display_name": "env_test_shapash_lot2_v36",
   "language": "python",
   "name": "env_test_shapash_lot2_v36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.12"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}