Ineuron_session_saturday_11_01_2020_pandas_handling_nan.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Ineuron_session_saturday_11_01_2020_pandas_handling_nan.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyMOXLrrNcQqev9l9R4hgg9x",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/plaban1981/INEURON/blob/master/Ineuron_session_saturday_11_01_2020_pandas_handling_nan.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z3FTDqvK8JYT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YOCiM8Un9FJ6",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 137
        },
        "outputId": "434df08d-461c-4fde-d5cb-fba6caf4dfa9"
      },
      "source": [
        "df =pd.DataFrame({'A':[1,2,np.nan],'B':[5,np.nan,np.nan],'C':[1,2,3]})\n",
        "df.head()"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>A</th>\n",
              "      <th>B</th>\n",
              "      <th>C</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     A    B  C\n",
              "0  1.0  5.0  1\n",
              "1  2.0  NaN  2\n",
              "2  NaN  NaN  3"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UK35bLtD9sBB",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 137
        },
        "outputId": "3bf298d5-4830-468a-fa97-113f6e773b24"
      },
      "source": [
        "df[\"Country\"] = \"India Australia USA\".split()\n",
        "df.head()"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>A</th>\n",
              "      <th>B</th>\n",
              "      <th>C</th>\n",
              "      <th>Country</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1</td>\n",
              "      <td>India</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2</td>\n",
              "      <td>Australia</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3</td>\n",
              "      <td>USA</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     A    B  C    Country\n",
              "0  1.0  5.0  1      India\n",
              "1  2.0  NaN  2  Australia\n",
              "2  NaN  NaN  3        USA"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zi5gE6DQ952v",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df.set_index(\"Country\",inplace=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Cy1n6t4G-OCh",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 167
        },
        "outputId": "6ba79f21-3930-4872-9f5f-fa85b9cfbe2d"
      },
      "source": [
        "df.head()"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>A</th>\n",
              "      <th>B</th>\n",
              "      <th>C</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Country</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>India</th>\n",
              "      <td>1.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Australia</th>\n",
              "      <td>2.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>USA</th>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "             A    B  C\n",
              "Country               \n",
              "India      1.0  5.0  1\n",
              "Australia  2.0  NaN  2\n",
              "USA        NaN  NaN  3"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7kb4L8QC-RKY",
        "colab_type": "text"
      },
      "source": [
        "##handle nan values"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z7dP3n5k-UJG",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 107
        },
        "outputId": "f3c35bbd-90ad-4d80-f3f7-df41415c4b01"
      },
      "source": [
        "df.dropna(axis=0)# drops a nan value found in any row"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>A</th>\n",
              "      <th>B</th>\n",
              "      <th>C</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Country</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>India</th>\n",
              "      <td>1.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "           A    B  C\n",
              "Country             \n",
              "India    1.0  5.0  1"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NuXo3iz7-p45",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 167
        },
        "outputId": "91335526-c0a1-483e-b881-88c27952fc32"
      },
      "source": [
        "df.dropna(axis = 1) #drop columns having nan values"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>C</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Country</th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>India</th>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Australia</th>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>USA</th>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "           C\n",
              "Country     \n",
              "India      1\n",
              "Australia  2\n",
              "USA        3"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "25vtN3Rg_HHN",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 137
        },
        "outputId": "f28a02be-3f53-43bc-8767-ab84d659337d"
      },
      "source": [
        "df.dropna(axis=0,thresh=2)#mininum number of nan need to be present in a row in order to drop those values"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>A</th>\n",
              "      <th>B</th>\n",
              "      <th>C</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Country</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>India</th>\n",
              "      <td>1.0</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Australia</th>\n",
              "      <td>2.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "             A    B  C\n",
              "Country               \n",
              "India      1.0  5.0  1\n",
              "Australia  2.0  NaN  2"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xTyJ0Lzo_jWg",
        "colab_type": "text"
      },
      "source": [
        "##Fill missing Values"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bQG8mwEb_mKx",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 167
        },
        "outputId": "73577c20-bb39-4616-8208-c380ad5e96f2"
      },
      "source": [
        "df.fillna(\"Missing Values\")"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>A</th>\n",
              "      <th>B</th>\n",
              "      <th>C</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Country</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>India</th>\n",
              "      <td>1</td>\n",
              "      <td>5</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Australia</th>\n",
              "      <td>2</td>\n",
              "      <td>Missing Values</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>USA</th>\n",
              "      <td>Missing Values</td>\n",
              "      <td>Missing Values</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                        A               B  C\n",
              "Country                                     \n",
              "India                   1               5  1\n",
              "Australia               2  Missing Values  2\n",
              "USA        Missing Values  Missing Values  3"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Tr6PXCBC_8AX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 167
        },
        "outputId": "95055a0b-4f41-4084-cd72-67f5de9b5148"
      },
      "source": [
        "df.isnull() ==True"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>A</th>\n",
              "      <th>B</th>\n",
              "      <th>C</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Country</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>India</th>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Australia</th>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>USA</th>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "               A      B      C\n",
              "Country                       \n",
              "India      False  False  False\n",
              "Australia  False   True  False\n",
              "USA         True   True  False"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s2X7_GBrBqMU",
        "colab_type": "text"
      },
      "source": [
        "## Count the number of null values for each row"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mPfNKTBAA-2L",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "3c24d8c8-2ccc-45f0-d3bb-db484a8ff80d"
      },
      "source": [
        "df.isnull().sum()"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "A    1\n",
              "B    2\n",
              "C    0\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WTzNNcHzBT9d",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "22f86b74-92db-4950-fda9-e0da91ce7b79"
      },
      "source": [
        "df.isna().sum()"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "A    1\n",
              "B    2\n",
              "C    0\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tzGhRlgIBd4R",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "outputId": "55a2b948-ce53-42e9-f117-4a8511b3ce79"
      },
      "source": [
        "np.isnan(df).sum()"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "A    1\n",
              "B    2\n",
              "C    0\n",
              "dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LP_LnN_4B7Wz",
        "colab_type": "text"
      },
      "source": [
        "## inplace = True ==> makes the changes permanent"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GGSGzYGFB6we",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 287
        },
        "outputId": "01b2b86d-3b12-46a4-c56d-d28a291de426"
      },
      "source": [
        "data = {'Company':['GOOG','GOOG','MSFT','MSFT','FB','FB'],\n",
        "        'Person':['Sam','Charlie','Amy','Vanessa','Carl','Sarah'],\n",
        "        'Sales': [200,120,340,124,243,350]}\n",
        "df = pd.DataFrame(data)\n",
        "df.describe()#only integer values"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Sales</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>6.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>229.500000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>100.899455</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>120.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>143.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>221.500000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>315.750000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>350.000000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "            Sales\n",
              "count    6.000000\n",
              "mean   229.500000\n",
              "std    100.899455\n",
              "min    120.000000\n",
              "25%    143.000000\n",
              "50%    221.500000\n",
              "75%    315.750000\n",
              "max    350.000000"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Bn4x0EweC9Ku",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "group_cmp = df.groupby('Company').sum()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rco_J1MTDJ9u",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 167
        },
        "outputId": "365b2868-58b4-444e-b343-b6d7050170c2"
      },
      "source": [
        "group_cmp"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Sales</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Company</th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>FB</th>\n",
              "      <td>593</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>GOOG</th>\n",
              "      <td>320</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>MSFT</th>\n",
              "      <td>464</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         Sales\n",
              "Company       \n",
              "FB         593\n",
              "GOOG       320\n",
              "MSFT       464"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a1cOdrjNDdKO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "grp = df.groupby('Company').describe().loc[['MSFT','GOOG']]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LCZi29d_EQX0",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "13b82b4d-71ef-4a94-d37c-b147485539e7"
      },
      "source": [
        "type(grp)"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "pandas.core.frame.DataFrame"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z-C2EBMBE4Cu",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 167
        },
        "outputId": "a27e60b2-ed96-41bc-9b47-67cab1df8ea5"
      },
      "source": [
        "grp"
      ],
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead tr th {\n",
              "        text-align: left;\n",
              "    }\n",
              "\n",
              "    .dataframe thead tr:last-of-type th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th></th>\n",
              "      <th colspan=\"8\" halign=\"left\">Sales</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th></th>\n",
              "      <th>count</th>\n",
              "      <th>mean</th>\n",
              "      <th>std</th>\n",
              "      <th>min</th>\n",
              "      <th>25%</th>\n",
              "      <th>50%</th>\n",
              "      <th>75%</th>\n",
              "      <th>max</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Company</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>MSFT</th>\n",
              "      <td>2.0</td>\n",
              "      <td>232.0</td>\n",
              "      <td>152.735065</td>\n",
              "      <td>124.0</td>\n",
              "      <td>178.0</td>\n",
              "      <td>232.0</td>\n",
              "      <td>286.0</td>\n",
              "      <td>340.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>GOOG</th>\n",
              "      <td>2.0</td>\n",
              "      <td>160.0</td>\n",
              "      <td>56.568542</td>\n",
              "      <td>120.0</td>\n",
              "      <td>140.0</td>\n",
              "      <td>160.0</td>\n",
              "      <td>180.0</td>\n",
              "      <td>200.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        Sales                                                      \n",
              "        count   mean         std    min    25%    50%    75%    max\n",
              "Company                                                            \n",
              "MSFT      2.0  232.0  152.735065  124.0  178.0  232.0  286.0  340.0\n",
              "GOOG      2.0  160.0   56.568542  120.0  140.0  160.0  180.0  200.0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kuLy0tD_ESgr",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 176
        },
        "outputId": "67ecb941-f9b9-4a4d-98fd-2cabb3f4229a"
      },
      "source": [
        "grp.columns"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "MultiIndex([('Sales', 'count'),\n",
              "            ('Sales',  'mean'),\n",
              "            ('Sales',   'std'),\n",
              "            ('Sales',   'min'),\n",
              "            ('Sales',   '25%'),\n",
              "            ('Sales',   '50%'),\n",
              "            ('Sales',   '75%'),\n",
              "            ('Sales',   'max')],\n",
              "           )"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dT9nugO1EYbR",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 137
        },
        "outputId": "43668e0e-5a34-4f87-feb4-4725dc5050e7"
      },
      "source": [
        "grp.iloc[:1,2:]"
      ],
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead tr th {\n",
              "        text-align: left;\n",
              "    }\n",
              "\n",
              "    .dataframe thead tr:last-of-type th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr>\n",
              "      <th></th>\n",
              "      <th colspan=\"6\" halign=\"left\">Sales</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th></th>\n",
              "      <th>std</th>\n",
              "      <th>min</th>\n",
              "      <th>25%</th>\n",
              "      <th>50%</th>\n",
              "      <th>75%</th>\n",
              "      <th>max</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Company</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>MSFT</th>\n",
              "      <td>152.735065</td>\n",
              "      <td>124.0</td>\n",
              "      <td>178.0</td>\n",
              "      <td>232.0</td>\n",
              "      <td>286.0</td>\n",
              "      <td>340.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "              Sales                                   \n",
              "                std    min    25%    50%    75%    max\n",
              "Company                                               \n",
              "MSFT     152.735065  124.0  178.0  232.0  286.0  340.0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 43
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dbHmnI6jGern",
        "colab_type": "text"
      },
      "source": [
        "##pd.concat([df1,df2,df3],axis=1) --> column wise concatenation , by default axis = 0, row wise"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "km_ooMdlINxm",
        "colab_type": "text"
      },
      "source": [
        "## df1 = df.copy() - separate memory location allocated\n",
        "## df1 = df ,same memory location is referenced"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PpX1eH7OQKwS",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 197
        },
        "outputId": "47279bed-6643-4a3a-d168-1e834c02729d"
      },
      "source": [
        "midx = pd.MultiIndex.from_tuples([(10,'Ten'),(10,'Twenty'),(20,'Ten'),(20,'Twenty')],names=['Num','Char'])\n",
        "midx.to_frame(index=True)"
      ],
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th>Num</th>\n",
              "      <th>Char</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Num</th>\n",
              "      <th>Char</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">10</th>\n",
              "      <th>Ten</th>\n",
              "      <td>10</td>\n",
              "      <td>Ten</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Twenty</th>\n",
              "      <td>10</td>\n",
              "      <td>Twenty</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th rowspan=\"2\" valign=\"top\">20</th>\n",
              "      <th>Ten</th>\n",
              "      <td>20</td>\n",
              "      <td>Ten</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Twenty</th>\n",
              "      <td>20</td>\n",
              "      <td>Twenty</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "            Num    Char\n",
              "Num Char               \n",
              "10  Ten      10     Ten\n",
              "    Twenty   10  Twenty\n",
              "20  Ten      20     Ten\n",
              "    Twenty   20  Twenty"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VuamebPjRDqQ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 304
        },
        "outputId": "95095ee4-2f50-4f84-ffec-1e31ee3ce1f2"
      },
      "source": [
        "# Creating the Tuple \n",
        "tuples =[('Physics', 85), ('Chemistry', 88), \n",
        "           ('Maths', 95), ('Computers', 99)] \n",
        "  \n",
        "# Print the Tuple \n",
        "print(tuples) \n",
        "# Creating the MultiIndex \n",
        "midx = pd.MultiIndex.from_tuples(tuples, names =('Subject', 'Marks')) \n",
        "  \n",
        "# Print the MultiIndex \n",
        "print(midx) \n",
        "\n",
        "midx.to_frame(index=True)"
      ],
      "execution_count": 50,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[('Physics', 85), ('Chemistry', 88), ('Maths', 95), ('Computers', 99)]\n",
            "MultiIndex([(  'Physics', 85),\n",
            "            ('Chemistry', 88),\n",
            "            (    'Maths', 95),\n",
            "            ('Computers', 99)],\n",
            "           names=['Subject', 'Marks'])\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th>Subject</th>\n",
              "      <th>Marks</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Subject</th>\n",
              "      <th>Marks</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>Physics</th>\n",
              "      <th>85</th>\n",
              "      <td>Physics</td>\n",
              "      <td>85</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Chemistry</th>\n",
              "      <th>88</th>\n",
              "      <td>Chemistry</td>\n",
              "      <td>88</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Maths</th>\n",
              "      <th>95</th>\n",
              "      <td>Maths</td>\n",
              "      <td>95</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Computers</th>\n",
              "      <th>99</th>\n",
              "      <td>Computers</td>\n",
              "      <td>99</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                   Subject  Marks\n",
              "Subject   Marks                  \n",
              "Physics   85       Physics     85\n",
              "Chemistry 88     Chemistry     88\n",
              "Maths     95         Maths     95\n",
              "Computers 99     Computers     99"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 50
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u6_eM2qPSc8B",
        "colab_type": "text"
      },
      "source": [
        "## .apply function is applicable to Series"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0mkqa-nGRvhG",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "fb2d0ff3-91b4-4bee-b7c3-85a4fcf7eddd"
      },
      "source": [
        "grp.index"
      ],
      "execution_count": 55,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Index(['MSFT', 'GOOG'], dtype='object', name='Company')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 55
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i7Wb8051U7Yp",
        "colab_type": "text"
      },
      "source": [
        "##How to reduce memory usage of data using Pandas?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_pGIPrZRVCAV",
        "colab_type": "text"
      },
      "source": [
        "#### 1. Read CSV file data in chunk size\n",
        "To be honest, I was baffled when I encountered an error and I couldn’t read the data from CSV file, only to realize that the memory of my local machine was too small for the data with 16GB of RAM.\n",
        "\n",
        "#### pandas.read_csv has a parameter called chunksize!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "79BRzK4SVXtn",
        "colab_type": "text"
      },
      "source": [
        "The parameter essentially means the number of rows to be read into a dataframe at any single time in order to fit into the local memory. Since the data consists of more than 70 millions of rows, I specified the chunksize as 1 million rows each time that broke the large data set into many smaller pieces."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tdClQ6XpU8ng",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "# read the large csv file with specified chunksize \n",
        "df_chunk = pd.read_csv(r'../input/data.csv', chunksize=1000000)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MtnEwVunVjsL",
        "colab_type": "text"
      },
      "source": [
        "The operation above resulted in a TextFileReader object for iteration. Strictly speaking, df_chunk is not a dataframe but an object for further operation in the next step."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x7kL3wshW8Y3",
        "colab_type": "text"
      },
      "source": [
        "By iterating each chunk, I performed data filtering/preprocessing using a function — chunk_preprocessing before appending each chunk to a list. And finally I concatenated the list into a final dataframe to fit into the local memory."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zwr-jeoAVoSc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "chunk_list = []  # append each chunk df here \n",
        "\n",
        "# Each chunk is in df format\n",
        "for chunk in df_chunk:  \n",
        "    # perform data filtering \n",
        "    chunk_filter = chunk_preprocessing(chunk)\n",
        "    \n",
        "    # Once the data filtering is done, append the chunk to list\n",
        "    chunk_list.append(chunk_filter)\n",
        "    \n",
        "# concat the list into dataframe \n",
        "df_concat = pd.concat(chunk_list)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z1ocCNLJXyRv",
        "colab_type": "text"
      },
      "source": [
        "2. Filter out unimportant columns to save memory\n",
        "Great. At this stage, I already had a dataframe to do all sorts of analysis required.\n",
        "To save more time for data manipulation and computation, I further filtered out some unimportant columns to save more memory.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-G1VT2zXX1uU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df = df[['col_1','col_2', 'col_3', 'col_4', 'col_5', 'col_6','col_7', 'col_8', 'col_9', 'col_10']]\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ezM-MC4MX6fp",
        "colab_type": "text"
      },
      "source": [
        "3. Change dtypes for columns\n",
        "The simplest way to convert a pandas column of data to a different type is to use astype().\n",
        "I can say that changing data types in Pandas is extremely helpful to save memory, especially if you have large data for intense analysis or computation (For example, feed data into your machine learning model for training)."
      ]
    }
  ]
}