EX_1_cost_function_regression_FOR_STUDENTS.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Simple programming excercise for both learning what a cost function is and how it behaves, as well as how to do matlab presentation\n",
    "\n",
    "Take a simple regression problem and provide a linear solution $y=a+bx$ with the quadratic cost function defined as $J(\\theta)=\\frac{1}{2N}\\sum_i(h(x^{(i)})-y^{(i)})^2$ where $D=\\{(x^{(i)}, y^{(i)})\\}_{i=1}^{N}$ is the training set.\n",
    "\n",
    "Now draw two figures one beside the other: on the left the examples together with the current solution (a straight line) while on the right a contour plot of the cost function with a cross where the current solution is. Make it possible to either move the solution or illustrate a gradient descent algorithm.\n",
    "\n",
    "**Questions:**\n",
    "* How does changes in step size in gradient decent influence the algorithm?\n",
    "* Is it better to have a big step size or small?\n",
    "* Is it important what are the inital values of $\\theta$ ?\n",
    "* Is it better to have more or less data points in the training set? Make experiments also with more complicated function.\n",
    "\n",
    "Students shall understand what are the isolines, the duality between the hypothesis and the solution space, and last, but not least, how to plot figures in matplotlib!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "% matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from itertools import izip\n",
    "from sklearn import preprocessing\n",
    "import time\n",
    "import math\n",
    "# feel free to add here anything you need"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "if not specified otherwise:\n",
    "* **X_real**, **X** are np.matrix shape (n_samples,)\n",
    "* **y_real**, **y** are np.matrix shape (n_samples,)\n",
    "* **theta** is a vector of scalar values (floats or ints) representing parameters of a function\n",
    "* **basis_function** is an instance of Function class implementing basis function\n",
    "* **theta** is a dictionary of parameters of a function, ex. f(x) = ax+b has two parameters: a and b (x is an argument), so the dictionary can look like this: {'a': 2, 'b': 3} (produces function f(x)=2x+3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class Function(object):\n",
    "    def __init__(self, function, first_derivatives):\n",
    "        self.function = function\n",
    "        self.derivatives = first_derivatives"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def polynomial_ord_1(theta, X):\n",
    "    # implement ax+b\n",
    "    # keep in mind that X and theta are matrices\n",
    "    # return matrix (one resulting value for each element of X)\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def polynomial_ord_1_derivatives(theta, X):\n",
    "    # return derivative over theta[0], derivative over theta[1]\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def cost_function(basis_function, theta, X, y):\n",
    "# return the the value of the cost function (scalar)\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def cost_function_derivatives(basis_function, theta, X, y):\n",
    "# return the the values of the derivatives of the cost function\n",
    "# in the form [der_over_theta_1, der_over_theta_2, ..., der_over_theta_n] (scalars)\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def cost_function_contour(basis_function, A, B, X, y):\n",
    "    # cost function for plotting contours.\n",
    "    results = []\n",
    "    for a_, b_ in np.nditer([A, B]):\n",
    "        results.append(cost_function(basis_function=basis_function, theta=np.array([a_, b_]), X=X, y=y))\n",
    "    return np.array(results).reshape(A.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def gradient_decent(basis_function, current_theta, X, y):\n",
    "    # based on current values of theta, the data and the cost you need to return the new values of theta\n",
    "    # use cost_function_gradient you've already written to get the gradient\n",
    "    # return new theta (same type as the old theta (a np.array))\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def simple_init():\n",
    "    # initalise data (feel free to play around with this function ;) )\n",
    "    f = Function(polynomial_ord_1, polynomial_ord_1_derivatives)\n",
    "    a_real = 2\n",
    "    b_real = 1\n",
    "    theta_real = np.array([a_real, b_real])\n",
    "    X = np.arange(0.0, 5.0, 0.1)\n",
    "    y = f.function(theta=theta_real, X = X)\n",
    "    # starting values for gradient decent\n",
    "    a = -1\n",
    "    b = -1\n",
    "    theta = np.array([a, b])\n",
    "    return X, y, theta_real, theta, f"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def draw(basis_function, X, y, theta_real, theta):\n",
    "    # draw left\n",
    "    x_reg = np.arange(0.0, 5.0, 0.01)\n",
    "    y_reg = basis_function.function(X=x_reg, theta=theta)\n",
    "\n",
    "    plt.subplot(121)\n",
    "    plt.scatter(x_reg, y_reg, lw=0, s=5)\n",
    "    plt.scatter(X, y, c='r', lw=0, s=5)\n",
    "\n",
    "    # draw right\n",
    "    delta = 0.25\n",
    "    a_cost = np.arange(-3.0, 3.0, delta)\n",
    "    b_cost = np.arange(-3.0, 3.0, delta)\n",
    "    A_cost, B_cost = np.meshgrid(a_cost, b_cost)\n",
    "    Z_cost = cost_function_contour(basis_function=basis_function, A=A_cost, B=B_cost, X=X, y=y)\n",
    "\n",
    "    plt.subplot(122)\n",
    "   \n",
    "    CS = plt.contour(A_cost, B_cost, Z_cost)\n",
    "    plt.clabel(CS, inline=1, fontsize=10)\n",
    "    \n",
    "    # the solution\n",
    "    plt.scatter(theta_real[0], theta_real[1], marker='+', s=30, c='r', lw=2)\n",
    "    # we're here\n",
    "    plt.scatter(theta[0], theta[1])\n",
    "    plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def draw_cost_function(basis_function, scale, X, y):\n",
    "    delta = float(scale)/5\n",
    "    a_cost = np.arange(-3*scale, 3*scale, delta)\n",
    "    b_cost = np.arange(-3*scale, 3*scale, delta)\n",
    "    A_cost, B_cost = np.meshgrid(a_cost, b_cost)\n",
    "    Z_cost = cost_function_contour(basis_function=basis_function, A=A_cost, B=B_cost, X=X, y=y)\n",
    "   \n",
    "    CS = plt.contour(A_cost, B_cost, Z_cost)\n",
    "    plt.clabel(CS, inline=1, fontsize=10)\n",
    "    \n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# prosty przykład bez szumu\n",
    "# this is playground for your experiments. Change the code when needed.\n",
    "n_steps = 100\n",
    "X, y, theta_real, theta, basis_function = simple_init()\n",
    "draw_cost_function(basis_function, scale = 10.**150, X=X, y=y)\n",
    "for i in xrange(n_steps):\n",
    "    if i%5 == 1:\n",
    "        print '\\n\\na:', theta[0], \"b:\", theta[1]\n",
    "        draw(basis_function, X, y, theta_real, theta)\n",
    "    print 'iteration', i, \":\\ta distance\", math.fabs(theta[0]-theta_real[0]), '\\tb distance', math.fabs(theta[1]-theta_real[1]), '\\tcost:', cost_function(basis_function, theta, X, y)\n",
    "    theta = gradient_decent(basis_function=basis_function, current_theta = theta, X=X, y=y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}