machine_learning_exercises1_2020.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "machine_learning_exercises1.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ko7Hm5QWCWqX",
        "colab_type": "text"
      },
      "source": [
        "# Exercises for Machine Learning with Python, Lecture 1: *Linear Least Squares Regression*\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AXY2tywnCawC",
        "colab_type": "text"
      },
      "source": [
        "## Before you begin the exercises:\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5WFgpPP6mrpa",
        "colab_type": "text"
      },
      "source": [
        "The datafiles are text files stored here (also try and download them to your own computer, open and see what they contain):\n",
        "\n",
        "Features: https://www.dropbox.com/s/pf2sfiy9l86xhww/boston_features.txt\n",
        "\n",
        "Prices: https://www.dropbox.com/s/j7flze0oe86pr6o/boston_prices.txt\n",
        "\n",
        "*   Start by downloading the dataset:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nFtEVyvxmrwJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Download the features and target (the prices) to Google Colab using !wget\n",
        "!wget -O boston_features.txt https://www.dropbox.com/s/pf2sfiy9l86xhww/boston_features.txt\n",
        "!wget -O boston_prices.txt https://www.dropbox.com/s/j7flze0oe86pr6o/boston_prices.txt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oww6BHzPnN2K",
        "colab_type": "text"
      },
      "source": [
        "This time we will not be using a Pandas DataFrames, because we will code the Linear Least Squares Regression ourselves using Numpy.\n",
        "\n",
        "*   Now, load the txt files into NumPy arrays:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q8Eg67JbnMXs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "features = np.loadtxt(\"boston_features.txt\")\n",
        "prices = np.loadtxt(\"boston_prices.txt\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W-A0JFizohhK",
        "colab_type": "text"
      },
      "source": [
        "As you can see below, the features is a matrix contatining 506 rows (one for each house), each with 13 entries (one for each feature). Additionally, there are 506 prices (again, one for each house)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1Ksqhh8DodXx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(features.shape)\n",
        "print(prices.shape)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8uLLxkixpUWP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(features)\n",
        "print(prices)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rUeXGXjbGjnG",
        "colab_type": "text"
      },
      "source": [
        "Beside the housing prices, the dataset contains and information about location, etc (see below), and the prices are units of in USD $1000.\n",
        "\n",
        "```\n",
        "Index   Description\n",
        "-----------------------------------------------------------------------\n",
        "   0    per capita crime rate by town\n",
        "   1    proportion of residential land zoned for lots over 25,000 sq.ft.\n",
        "   2    proportion of non-retail business acres per town\n",
        "   3    Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)\n",
        "   4    nitric oxides concentration (parts per 10 million)\n",
        "   5    average number of rooms per dwelling\n",
        "   6    proportion of owner-occupied units built prior to 1940\n",
        "   7    weighted distances to five Boston employment centres\n",
        "   8    index of accessibility to radial highways\n",
        "   9    full-value property-tax rate per $10,000\n",
        "  10    pupil-teacher ratio by town\n",
        "  11    1000(Bk - 0.63)^2 where Bk is the proportion of African Americans by town\n",
        "  12    % lower status of the population\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gKW4IOjjpgpd",
        "colab_type": "text"
      },
      "source": [
        "## Exercise 1.1: Exploratory Data Analysis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RLxdW_pPpg5U",
        "colab_type": "text"
      },
      "source": [
        "Before we begin the regression, lets have a look at the data. As in the lecture, we will plot the correlation between our features using Seaborn.\n",
        "\n",
        "Since we loaded in the data as a Numpy Array, we have to create a Pandas DataFrame for Seaborn. This can be done with the code below:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-wyVjM5UqK5y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Pandas DataFrames\n",
        "import pandas as pd\n",
        "\n",
        "# Make a Pandas dataframe from the features\n",
        "df = pd.DataFrame(features[:,:5]) \n",
        "\n",
        "# NOTE: ONLY TAKES THE FIRST 5 COLUMNS/FEATURES\n",
        "# features[:,:5] is Numpy slice notation for \"take all rows, and first five colums\"\n",
        "\n",
        "# Print the type of the dataframe, just to make sure we coverted correctly\n",
        "print(type(df))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CrOYprZ5qLOe",
        "colab_type": "text"
      },
      "source": [
        "Note that since there are 13 features, the correlation plot would contain 169 plots. This would be quite slow in Google Colab. For speed reasons, we only loaded the first 5 featues into the pandas dataframe for plotting.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1Z-a3Etntajv",
        "colab_type": "text"
      },
      "source": [
        "### Question 1.1.1: "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dOPcP5g_tawa",
        "colab_type": "text"
      },
      "source": [
        "*   Just like in the lecture, make an `sns.pairplot()` using Seaborn of the dataframe we just made.\n",
        "*   (Optional) If you want, change the style of the plot as shown during the lecture. You can see some customization options in the documentation: https://seaborn.pydata.org/generated/seaborn.pairplot.html"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O3-7w7byp5AH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Plot the pairplot of the Pandas DataFrame"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nLy2rXKUtwKo",
        "colab_type": "text"
      },
      "source": [
        "### Question 1.1.2:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gOuaxzWsuEzc",
        "colab_type": "text"
      },
      "source": [
        "One thing you might note from the pair-correlation plot is that some of the features look odd. However, these features still work well for linear regression. \n",
        "\n",
        "*    Can you explain why the correlations between Feature 3 and everything else look odd?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7BUPWp-WuT13",
        "colab_type": "text"
      },
      "source": [
        "**Give your answer:** "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G_cWc1o-u-E7",
        "colab_type": "text"
      },
      "source": [
        "## Exercise 1.2: Training and Test sets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mRjjCAMmvFdZ",
        "colab_type": "text"
      },
      "source": [
        "In order to train and evaluate our linear regression model, we first have to divide the dataset into two parts: A training set and a test set.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "742iy28bzVIN",
        "colab_type": "text"
      },
      "source": [
        "### Question 1.2.1:\n",
        "A common practice is to have the training set be 70% of the full set and test set 30%.\n",
        "\n",
        "There are 506 houses in our dataset, so divide your data into 354 in the training set and 152 in the test set.\n",
        "\n",
        "You can use Numpy's slice notation for this, as introducted in the first lectures. For example, to take the first five *rows* from an array you can use the following notation:\n",
        "```\n",
        "data_first_five_rows = data[:5]\n",
        "```\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "To complete this Question, do the following task:\n",
        "*    Divide both of the numpy arrays `features` and `prices` into two parts.\n",
        "*    Use, for example, the Numpy slice notation to achieve this.\n",
        "*    Make sure that the same house in never in both the test and training set at the same time! Other than that, you are free to choose which rows to the test set and which go to the training set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f75qk7pww1K1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Make the two sets of features below\n",
        "features_train = ???\n",
        "features_test = ???\n",
        "\n",
        "# Make the two sets of prices below\n",
        "prices_train = ???\n",
        "prices_test = ???"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c0d3lJE_zc-4",
        "colab_type": "text"
      },
      "source": [
        "## Exercise 1.3: Regression Fit"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cad8uWOTzdB7",
        "colab_type": "text"
      },
      "source": [
        "In the linear least squares regression, we solve an equation that looks like this:\n",
        "\n",
        "$$ \\mathbf{y} = \\mathbf{X} \\mathbf{\\alpha}$$\n",
        "\n",
        "Where $\\mathbf{y}$ is a vector containing the target labels (for example housing prices), $\\mathbf{X}$ is a matrix where the each row contain features (in our case the features for a given house in Boston) for each house. Finally, $\\alpha is a vector containing the (unknown) regression coefficients.\n",
        "\n",
        "Fortunately, Numpy has built-in capabilites for solving linear least squares regression. This is done using the function which performs the \"least squares\" fit:\n",
        "\n",
        "```\n",
        "alpha, residual, rank, singular_values = np.linalg.lstsq(X, y)\n",
        "```\n",
        "\n",
        "As you can see, the function actually returns four items, the first one being the regression coefficients, and the next three are the residuals of the fit, the rank of the matrix problem, and finally the singular values of the matrix.\n",
        "\n",
        "For this exercise we only need to use the first one, but feel free to print and look at the three other items as well.\n",
        "\n",
        "You can see the documentation for `np.linalg.lstsq()` here: https://numpy.org/doc/stable/reference/generated/numpy.linalg.lstsq.html\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A2ORY5BZ2PPz",
        "colab_type": "text"
      },
      "source": [
        "### Question 1.3.1:\n",
        "\n",
        "*    Just like in the code above, use `np.linalg.lstsq()` to fit the alpha regression-coefficients. \n",
        "\n",
        "**Hint:** Use the `features_train` instead of `X` and `prices_train` instead of `y`.\n",
        "\n",
        "**Note:** *Google Colab uses a version of Numpy which might give you a warning abour \"rcond\", which is not a problem.*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DSxMpIjz15kA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Get the alpha coefficients \n",
        "\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gJylexpu9dTD",
        "colab_type": "text"
      },
      "source": [
        "## Exercise 1.4: Making predictions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gOLtQqUK9dee",
        "colab_type": "text"
      },
      "source": [
        "Now that you have obtained the alpha coefficients in the previous question, it is time to use them to make predictions.\n",
        "\n",
        "Remember how we are approximating the price as a linear sum of the weighted features for a given house:\n",
        "\n",
        "\\begin{equation}\n",
        "f(\\mathbf{x}) = x_1 \\alpha_1 + x_2 \\alpha_2 + \\dots + x_n \\alpha_n\n",
        "\\end{equation}\n",
        "or written in vector notation:\n",
        "\\begin{equation}\n",
        "f(\\mathbf{x}) = \\mathbf{x} \\cdot \\mathbf{\\alpha}\n",
        "\\end{equation}\n",
        "\n",
        "where $\\mathbf{x}$ is the vector containing all the features for the house and $\\mathbf{\\alpha}$ is the vector of regression coefficient you obtained by fitting the features to the prices of the training set."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7j9iFs8DEZcL",
        "colab_type": "text"
      },
      "source": [
        "### Question 1.4.1:\n",
        "\n",
        "With the alpha regression coefficients you found in the previous exercise, you can use your linear least squares \"machine\" to predict the price of house that are not in your training set.\n",
        "\n",
        "As in the equation above, this is done by taking the dot product between the feature vector for a given house and the regression coefficients.\n",
        "\n",
        "Remember, that the features for each house are stored in the rows of the \"features\" numpy arrays you created in Question 1.2.1.\n",
        "\n",
        "*    Calculate the price of all the houses in the test set. That is, take the dot product between the regression coefficients and every row-vector of the `features_test` array.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HirrxfiqGEzI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Calculate the prices of the houses in the test set\n",
        "#, i.e. the dot product between \"features_test\" and \"alpha\"\n",
        "\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jZ5ZhTqQMab6",
        "colab_type": "text"
      },
      "source": [
        "*Hint:* Save the predicted prices for the test set in a list or numpy array, which you can use in question 1.4.2."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-Qa_7aIKFVeF",
        "colab_type": "text"
      },
      "source": [
        "### Question 1.4.2:\n",
        "Since we already know the true price of the houses in ou dataset, we can now compare the true prices of the house \n",
        "\n",
        "Use for example `plt.scatter()` as we did in the introductory lectures.\n",
        "\n",
        "*    Plot the correlation between the true prices stored in the `prices_test` and the predicted prices.\n",
        "\n",
        "Does it look like there is some correlation?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QYU4QANnMurZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Write the code to plot the correlation between \"prices_test\" and the prices  \n",
        "# you predicted in Question 1.4.1.\n",
        "\n",
        "\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NDJsRFCxN3rx",
        "colab_type": "text"
      },
      "source": [
        "### Question 1.4.3:\n",
        "\n",
        "One common measure of the prediction error is the \"root mean squared error\" (RMSE) between the true values and the predicted values. The RMSE is calculated as follows:\n",
        "\n",
        "\\begin{equation}\n",
        "\\mathrm{RMSE} = \\sqrt{\\frac{1}{N}\\sum_{i=0}^{N} \\left( y_i^\\mathrm{true} - y_i^\\mathrm{predicted}\\right)^2}\n",
        "\\end{equation}\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7Ni70RBhZKyg",
        "colab_type": "text"
      },
      "source": [
        "*    In this question, calculate the RMSE for your test set, i.e. the RMSE between the true prices and the prices you predicted in Question 1.4.1."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eYO9U5-pZajH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Calculate the RMSE \n",
        "\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ml0aWTqdFOO4",
        "colab_type": "text"
      },
      "source": [
        "## Exercise 1.5: Linear Regression with Scikit-learn"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EeSKRpzUaxT9",
        "colab_type": "text"
      },
      "source": [
        "As we saw in the lecture, the Python library Scikit-learn has capabilities for almost any type of macine learning you can think of.\n",
        "\n",
        "It also does Linear Regression. Below is the a code example to use linear regression with sklearn.\n",
        "\n",
        "```\n",
        "# Import the machine\n",
        "from sklearn.linear_model import LinearRegression\n",
        "\n",
        "# Make a linear regression machine identically to our fitting in Numpy\n",
        "machine = LinearRegression(fit_intercept=False)\n",
        "\n",
        "# Fit the machine using the training features and training labels\n",
        "machine.fit(x_train, y_train)\n",
        "```\n",
        "\n",
        "After the machine has been fitted, you can use the built-in `predict()` method to make predictions on new feature maps:\n",
        "\n",
        "```\n",
        "# Predict y-values using x_test features\n",
        "y_test_predicted = machine.predict(x_test)\n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7yyahz3HhZVy",
        "colab_type": "text"
      },
      "source": [
        "### Question 1.5.1:\n",
        "\n",
        "*    In this question, use Scikit-learn to train a `LinearRegression` machine on the housing dataset. Use the same training and test data which you used in the previous example.\n",
        "\n",
        "**Hint:** You use the above code snippets to solve this question."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hE-cVxkIf9r1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Implement the linear regression with Scikit-learn\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tL9FTkdbiD4l",
        "colab_type": "text"
      },
      "source": [
        "### Question 1.5.2:\n",
        "\n",
        "*    In this question, make a scatter plot between the between housing prices you predicted with Scikit-learn and the true prices.\n",
        "*    Additionally, calculate the RMSE for prices you predicted with Scikit-learn.\n",
        "\n",
        "**Hint:** If everything went well, you should have gotten the exact same plot and RMSE as you found in Questions 1.4.2 and 1.4.3."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sX5YekdUPSXd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Make a scatter plot for  the housing prices you predicted with \n",
        "# Scikit-learn and the true prices for the test set\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2X662vC8iEAT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Calculate the RMSE for the housing prices you predicted with \n",
        "# Scikit-learn and the true prices for the test set\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}