CM20315_Gradients_I.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyMDEfAZvjcjpvBNmdrYv3EW",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/udlbook/udlbook/blob/main/CM20315_Gradients_I.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# CM20315 Gradients I\n",
        "\n",
        "We're going to investigate how to take the derivatives of functions where one operation is composed with another, which is composed with a third and so on.    For example, consider the function:\n",
        "\n",
        "\\begin{equation}\n",
        "     y = \\beta_4+\\omega_4\\cdot \\log\\biggl[\\beta_3+\\omega_3\\cdot\\cos\\Bigl[\\beta_2+\\omega_2\\cdot\\exp\\bigl[\\beta_1+\\omega_1\\cdot\\sin[\\beta_0+\\omega_0x]\\bigr]\\Bigr]\\biggr],\n",
        "\\end{equation}\n",
        "\n",
        "which is a composition of the functions $\\log[\\bullet], \\cos[\\bullet],\\exp[\\bullet],\\sin[\\bullet]$.   I chose these just because you probably already know the derivatives of these functions:\n",
        "\n",
        "\\begin{eqnarray*}\n",
        "\\frac{\\partial \\log[z]}{\\partial z} = \\frac{1}{z}\\quad\\quad \\frac{\\partial \\cos[z]}{\\partial z} = -\\sin[z] \\quad\\quad \\frac{\\partial \\exp[z]}{\\partial z} = \\exp[z] \\quad\\quad \\frac{\\partial \\sin[z]}{\\partial z} = -\\cos[z].\n",
        "\\end{eqnarray*}\n",
        "\n",
        "Suppose that we know the current values of $\\beta_{0},\\beta_{1},\\beta_{2},\\beta_{3},\\beta_{4},\\omega_{0},\\omega_{1},\\omega_{2},\\omega_{3},\\omega_{4}$, and $x$. We could obviously calculate $y$.   But we also want to know how $y$ changes when we make a small change to $\\beta_{0},\\beta_{1},\\beta_{2},\\beta_{3},\\beta_{4},\\omega_{0},\\omega_{1},\\omega_{2},\\omega_{3}$, or $\\omega_{4}$.  In other words, we want to compute the ten derivatives:\n",
        "\n",
        "\\begin{eqnarray*}\n",
        "\\frac{\\partial y}{\\partial \\beta_{0}}, \\quad \\frac{\\partial y}{\\partial \\beta_{1}}, \\quad \\frac{\\partial y}{\\partial \\beta_{2}}, \\quad \\frac{\\partial y }{\\partial \\beta_{3}}, \\quad\n",
        "\\frac{\\partial y}{\\partial \\beta_{4}}, \\quad \\frac{\\partial y}{\\partial \\omega_{0}}, \\quad \\frac{\\partial y}{\\partial \\omega_{1}}, \\quad \\frac{\\partial y}{\\partial \\omega_{2}}, \\quad \\frac{\\partial y}{\\partial \\omega_{3}},  \\quad\\mbox{and} \\quad \\frac{\\partial y}{\\partial \\omega_{4}}.\n",
        "\\end{eqnarray*}"
      ],
      "metadata": {
        "id": "1DmMo2w63CmT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# import library\n",
        "import numpy as np"
      ],
      "metadata": {
        "id": "RIPaoVN834Lj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's first define the original function for $y$:"
      ],
      "metadata": {
        "id": "32-ufWhc3v2c"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AakK_qen3BpU"
      },
      "outputs": [],
      "source": [
        "def fn(x, beta0, beta1, beta2, beta3, beta4, omega0, omega1, omega2, omega3, omega4):\n",
        "  return beta4 + omega4 * np.log(beta3+omega3 * np.cos(beta2 + omega2 * np.exp(beta1 + omega1 * np.sin(beta0 + omega0 * x))))"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we'll choose some values for the betas and the omegas and x and compute the output of the function:"
      ],
      "metadata": {
        "id": "y7tf0ZMt5OXt"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "beta0 = 1.0; beta1 = 2.0; beta2 = -3.0; beta3 = 0.4; beta4 = -0.3\n",
        "omega0 = 0.1; omega1 = -0.4; omega2 = 2.0; omega3 = 3.0; omega4 = -0.5\n",
        "x = 2.3\n",
        "y_func = fn(x,beta0,beta1,beta2,beta3,beta4,omega0,omega1,omega2,omega3,omega4)\n",
        "print('y=%3.3f'%y_func)"
      ],
      "metadata": {
        "id": "pwvOcCxr41X_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Computing derivatives by hand\n",
        "\n",
        "We could compute expressions for the derivatives by hand and write code to compute them directly.  Some of them are easy. For example:\n",
        "\n",
        "\\begin{equation}\n",
        "\\frac{\\partial y}{\\partial \\beta_{4}}  = 1,\n",
        "\\end{equation}\n",
        "\n",
        "but some have very complex expressions, even for this relatively simple original equation. For example:\n",
        "\n",
        "\\begin{eqnarray*}\n",
        "\\frac{\\partial y}{\\partial \\omega_{0}}  &=& \n",
        "-\\frac{\\omega_{1}\\omega_{2}\\omega_{3}\\omega_{4} x \\cos[\\beta_{0}\\!+\\!\\omega_{0}x]\\cdot\\exp\\bigl[\\omega_{1}\\sin[\\beta_{0}\\!+\\!\\omega_{0}x]\\!+\\!\\beta_{1}\\bigr]\\cdot\\sin\\Bigl[\\omega_{2}\\exp\\bigl[\\omega_{1}\\sin[\\beta_{0}\\!+\\!\\omega_{0}x]\\!+\\!\\beta_{1}\\bigr]\\!+\\!\\beta_{2}\\Bigr]}\n",
        "{\\omega_{3}\\cos[\\omega_{2}\\exp[\\omega_{1}\\sin[\\beta_{0}\\!+\\!\\omega_{0}x]\\!+\\!\\beta_{1}]\\!+\\!\\beta_{2}]\\!+\\!\\beta_{3}}.\n",
        "\\end{eqnarray*}"
      ],
      "metadata": {
        "id": "u5w69NeT64yV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "dydbeta4_func = 1\n",
        "dydomega0_func = -omega1*omega2*omega3*omega4*x * np.cos(beta0+omega0*x) * \\\n",
        "              np.exp(omega1 * np.sin(beta0+omega0*x)+beta1) * \\\n",
        "              np.sin(omega2 * np.exp(omega1 * np.sin(beta0+omega0 *x)+beta1)+beta2)/ \\\n",
        "              (omega3 * np.cos(omega2 * np.exp(omega1 * np.sin(beta0+omega0*x)+beta1)+beta2)+beta3)"
      ],
      "metadata": {
        "id": "7t22hALp5zkq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's make sure these are correct using finite differences:"
      ],
      "metadata": {
        "id": "iRh4hnu3-H3n"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "dydbeta4_fd = (fn(x,beta0,beta1,beta2,beta3,beta4+0.0001,omega0,omega1,omega2,omega3,omega4)-fn(x,beta0,beta1,beta2,beta3,beta4,omega0,omega1,omega2,omega3,omega4))/0.0001\n",
        "dydomega0_fd = (fn(x,beta0,beta1,beta2,beta3,beta4,omega0+0.0001,omega1,omega2,omega3,omega4)-fn(x,beta0,beta1,beta2,beta3,beta4,omega0,omega1,omega2,omega3,omega4))/0.0001\n",
        "\n",
        "print('dydbeta4: Function value = %3.3f, Finite difference value = %3.3f'%(dydbeta4_func,dydbeta4_fd))\n",
        "print('dydomega0: Function value = %3.3f, Finite difference value = %3.3f'%(dydomega0_func,dydomega0_fd))"
      ],
      "metadata": {
        "id": "1O3XmXMx-HlZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The code to calculate $\\partial y/ \\partial \\omega_0$ is a bit of a nightmare.  It's easy to make mistakes, and you can see that some parts of it are repeated (for example, the $\\sin[\\bullet]$ term), which suggests some kind of redundancy in the calculations.  The goal of this practical is to compute the derivatives in a much simpler way.  There will be three steps:"
      ],
      "metadata": {
        "id": "wS4IPjZAKWTN"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Step 1:** Write the original equations as a series of intermediate calculations.  We change \n",
        "\n",
        "\\begin{equation}\n",
        "     y = \\beta_4+\\omega_4\\cdot \\log\\biggl[\\beta_3+\\omega_3\\cdot\\cos\\Bigl[\\beta_2+\\omega_2\\cdot\\exp\\bigl[\\beta_1+\\omega_1\\cdot\\sin[\\beta_0+\\omega_0x]\\bigr]\\Bigr]\\biggr]\n",
        "\\end{equation}\n",
        "\n",
        "to \n",
        "\n",
        "\\begin{eqnarray}\n",
        "f_{0} &=& \\beta_{0} + \\omega_{0} x\\nonumber\\\\\n",
        "h_{1} &=& \\sin[f_{0}]\\nonumber\\\\\n",
        "f_{1} &=& \\beta_{1} + \\omega_{1}h_{1}\\nonumber\\\\\n",
        "h_{2} &=& \\exp[f_{1}]\\nonumber\\\\\n",
        "f_{2} &=& \\beta_{2} + \\omega_{2} h_{2}\\nonumber\\\\\n",
        "h_{3} &=& \\cos[f_{2}]\\nonumber\\\\\n",
        "f_{3} &=& \\beta_{3} + \\omega_{3}h_{3}\\nonumber\\\\\n",
        "h_{4} &=& \\log[f_{3}]\\nonumber\\\\\n",
        "y &=& \\beta_{4} + \\omega_{4} h_{4}\n",
        "\\end{eqnarray}\n",
        "\n",
        "and compute and store the values of all of these intermediate values.  We'll need them to compute the derivatives.<br>  This is called the **forward pass**."
      ],
      "metadata": {
        "id": "8UWhvDeNDudz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# TODO compute all the f_k and h_k terms \n",
        "# Replace the code below\n",
        "\n",
        "f0 = 1\n",
        "h1 = 1\n",
        "f1 = 1\n",
        "h2 = 1\n",
        "f2 = 1\n",
        "h3 = 1\n",
        "f3 = 1\n",
        "h4 = 1\n",
        "y = 1"
      ],
      "metadata": {
        "id": "ZWKAq6HC90qV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Let's check we got that right:\n",
        "print(\"f0: true value = %3.3f, your value = %3.3f\"%(1.230, f0))\n",
        "print(\"h1: true value = %3.3f, your value = %3.3f\"%(0.942, h1))\n",
        "print(\"f1: true value = %3.3f, your value = %3.3f\"%(1.623, f1))\n",
        "print(\"h2: true value = %3.3f, your value = %3.3f\"%(5.068, h2))\n",
        "print(\"f2: true value = %3.3f, your value = %3.3f\"%(7.137, f2))\n",
        "print(\"h3: true value = %3.3f, your value = %3.3f\"%(0.657, h3))\n",
        "print(\"f3: true value = %3.3f, your value = %3.3f\"%(2.372, f3))\n",
        "print(\"h4: true value = %3.3f, your value = %3.3f\"%(0.864, h4))\n",
        "print(\"y_func = %3.3f, y = %3.3f\"%(y_func, y))\n"
      ],
      "metadata": {
        "id": "ibxXw7TUW4Sx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Step 2:** Compute the derivatives of $y$ with respect to the intermediate quantities that we just calculated, but in reverse order:\n",
        "\n",
        "\\begin{eqnarray}\n",
        "\\frac{\\partial y}{\\partial h_4}, \\quad \\frac{\\partial y}{\\partial f_3}, \\quad \\frac{\\partial y}{\\partial h_3}, \\quad \\frac{\\partial y}{\\partial f_2}, \\quad\n",
        "\\frac{\\partial y}{\\partial h_2}, \\quad \\frac{\\partial y}{\\partial f_1}, \\quad \\frac{\\partial y}{\\partial h_1},  \\quad\\mbox{and} \\quad \\frac{\\partial y}{\\partial f_0}.\n",
        "\\end{eqnarray}\n",
        "\n",
        "The first of these derivatives is straightforward:\n",
        "\n",
        "\\begin{equation}\n",
        "\\frac{\\partial y}{\\partial h_{4}} = \\frac{\\partial }{\\partial h_{4}} \\beta_{4} + \\omega_{4} h_{4} = \\omega_{4}.\n",
        "\\end{equation}\n",
        "\n",
        "The second derivative can be calculated using the chain rule:\n",
        "\n",
        "\\begin{equation}\n",
        "\\frac{\\partial y}{\\partial f_{3}} = \\frac{\\partial y}{\\partial h_{4}} \\frac{\\partial h_{4}}{\\partial f_{3}}.\n",
        "\\end{equation}\n",
        "\n",
        "The left-hand side asks how $y$ changes when $f_{3}$ changes.  The right-hand side says we can decompose this into (i) how $y$ changes when $h_{4}$ changes and how $h_{4}$ changes when $f_{4}$ changes.  So you get a chain of events happening:  $f_{3}$ changes $h_{4}$, which changes $y$, and the derivatives represent the effects of this chain.  Notice that we computed the first of these derivatives already and the other one is the derivative of $\\log[f_{3}]$ is simply $1/f_{3}$.  We calculated $f_{3}$ in step 1.\n",
        "\n",
        "We can continue in this way, computing the derivatives of the output with respect to these intermediate quantities:\n",
        "\n",
        "\\begin{eqnarray}\n",
        "\\frac{\\partial y}{\\partial h_{3}} &=& \\frac{\\partial y}{\\partial h_{4}} \\frac{\\partial h_{4}}{\\partial f_{3}}\\frac{\\partial f_{3}}{\\partial h_{3}}=\\frac{\\partial y}{\\partial f_{3}} \\frac{\\partial f_{3}}{\\partial h_{3}}\\nonumber \\\\\n",
        "\\frac{\\partial y}{\\partial f_{2}} &=& \\frac{\\partial y}{\\partial h_{4}} \\frac{\\partial h_{4}}{\\partial f_{3}}\\frac{\\partial f_{3}}{\\partial h_{3}}\\frac{\\partial h_{3}}{\\partial f_{2}} = \\frac{\\partial y}{\\partial h_{3}}\\frac{\\partial h_{3}}{\\partial f_{2}}\\nonumber \\\\\n",
        "\\frac{\\partial y}{\\partial h_{2}} &=& \\frac{\\partial y}{\\partial h_{4}} \\frac{\\partial h_{4}}{\\partial f_{3}}\\frac{\\partial f_{3}}{\\partial h_{3}}\\frac{\\partial h_{3}}{\\partial f_{2}}\\frac{\\partial f_{2}}{\\partial h_{2}}=\\frac{\\partial y}{\\partial f_{2}}\\frac{\\partial f_{2}}{\\partial h_{2}}\\nonumber \\\\\n",
        "\\frac{\\partial y}{\\partial f_{1}} &=& \\frac{\\partial y}{\\partial h_{4}} \\frac{\\partial h_{4}}{\\partial f_{3}}\\frac{\\partial f_{3}}{\\partial h_{3}}\\frac{\\partial h_{3}}{\\partial f_{2}}\\frac{\\partial f_{2}}{\\partial h_{2}}\\frac{\\partial h_{2}}{\\partial f_{1}}=\\frac{\\partial y}{\\partial h_{2}}\\frac{\\partial h_{2}}{\\partial f_{1}}\\nonumber \\\\\n",
        "\\frac{\\partial y}{\\partial h_{1}} &=& \\frac{\\partial y}{\\partial h_{4}} \\frac{\\partial h_{4}}{\\partial f_{3}}\\frac{\\partial f_{3}}{\\partial h_{3}}\\frac{\\partial h_{3}}{\\partial f_{2}}\\frac{\\partial f_{2}}{\\partial h_{2}}\\frac{\\partial h_{2}}{\\partial f_{1}}\\frac{\\partial f_{1}}{\\partial h_{1}}=\\frac{\\partial y}{\\partial f_{1}}\\frac{\\partial f_{1}}{\\partial h_{1}}\\nonumber \\\\\n",
        "\\frac{\\partial y}{\\partial f_{0}} &=& \\frac{\\partial y}{\\partial h_{4}} \\frac{\\partial h_{4}}{\\partial f_{3}}\\frac{\\partial f_{3}}{\\partial h_{3}}\\frac{\\partial h_{3}}{\\partial f_{2}}\\frac{\\partial f_{2}}{\\partial h_{2}}\\frac{\\partial h_{2}}{\\partial f_{1}}\\frac{\\partial f_{1}}{\\partial h_{1}}\\frac{\\partial h_{1}}{\\partial f_{0}}=\\frac{\\partial y}{\\partial h_{1}}\\frac{\\partial h_{1}}{\\partial f_{0}}.\n",
        "\\end{eqnarray}\n",
        "\n",
        "In each case, we have already computed all of the terms except the last one in the previous step, and the last term is simple to evaluate.  This is called the **backward pass**."
      ],
      "metadata": {
        "id": "jay8NYWdFHuZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# TODO -- Compute the derivatives of the output with respect\n",
        "# to the intermediate computations h_k and f_k (i.e, run the backward pass)\n",
        "# I've done the first two for you.  You replace the code below:\n",
        "dydh4 = omega4\n",
        "dydf3 = dydh4 * (1/f3)\n",
        "# Replace the code below\n",
        "dydh3 = 1\n",
        "dydf2 = 1\n",
        "dydh2 = 1\n",
        "dydf1 = 1\n",
        "dydh1 = 1\n",
        "dydf0 = 1 "
      ],
      "metadata": {
        "id": "gCQJeI--Egdl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Let's check we got that right\n",
        "print(\"dydh3: true value = %3.3f, your value = %3.3f\"%(-0.632, dydh3))\n",
        "print(\"dydf2: true value = %3.3f, your value = %3.3f\"%(0.476, dydf2))\n",
        "print(\"dydh2: true value = %3.3f, your value = %3.3f\"%(0.953, dydh2))\n",
        "print(\"dydf1: true value = %3.3f, your value = %3.3f\"%(4.830, dydf1))\n",
        "print(\"dydh1: true value = %3.3f, your value = %3.3f\"%(-1.932, dydh1))\n",
        "print(\"dydf0: true value = %3.3f, your value = %3.3f\"%(-0.646, dydf0))"
      ],
      "metadata": {
        "id": "dS1OrLtlaFr7"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Step 3:**  Now we will find how $y$ changes when we change the $\\beta$ and $\\omega$ terms. The first two are easy:\n",
        "\n",
        "\\begin{eqnarray}\n",
        "\\frac{\\partial y}{\\partial \\beta_{4}} &=& \\frac{\\partial }{\\partial \\beta_{4}}(\\beta_{4} + \\omega_{4} h_{4}) = 1\\nonumber \\\\\n",
        "\\frac{\\partial y}{\\partial \\omega_{4}} &=& \\frac{\\partial }{\\partial \\omega_{4}}(\\beta_{4} + \\omega_{4} h_{4}) = h_{4}.\n",
        "\\end{eqnarray}\n",
        "\n",
        "The remaining terms are calculated using the chain rule again:\n",
        "\n",
        "\\begin{eqnarray}\n",
        "\\frac{\\partial y}{\\partial \\beta_{3}} &=& \\frac{\\partial y}{\\partial f_{3}}\\frac{\\partial f_{3}}{\\partial \\beta_{3}}\\nonumber \\\\\n",
        "\\frac{\\partial y}{\\partial \\omega_{3}} &=& \\frac{\\partial y}{\\partial f_{3}}\\frac{\\partial f_{3}}{\\partial \\omega_{3}}\n",
        "\\end{eqnarray}\n",
        "\n",
        "where we already computed the first term of each right-hand side in Step 2, and the second terms are also easy to compute.  By the same logic, the other terms are:\n",
        "\n",
        "\\begin{eqnarray}\n",
        "\\frac{\\partial y}{\\partial \\beta_{k}} &=& \\frac{\\partial y}{\\partial f_{k}}\\frac{\\partial f_{k}}{\\partial \\beta_{k}}\\nonumber \\\\\n",
        "\\frac{\\partial y}{\\partial \\omega_{k}} &=& \\frac{\\partial y}{\\partial f_{k}}\\frac{\\partial f_{k}}{\\partial \\omega_{k}}\n",
        "\\end{eqnarray}\n",
        "\n",
        "for $k=2,1,0$."
      ],
      "metadata": {
        "id": "FlzlThQPGpkU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# TODO -- Calculate the final derivatives with respect to the beta and omega terms\n",
        "\n",
        "dydbeta4 = 1\n",
        "dydomega4 = 1\n",
        "dydbeta3 = 1\n",
        "dydomega3 = 1\n",
        "dydbeta2 = 1\n",
        "dydomega2 = 1\n",
        "dydbeta1 = 1\n",
        "dydomega1 = 1\n",
        "dydbeta0 = 1\n",
        "dydomega0 = 1\n"
      ],
      "metadata": {
        "id": "1I2BhqZhGMK6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Let's check we got them right\n",
        "print('dydbeta4: Your value = %3.3f, Function value = %3.3f, Finite difference value = %3.3f'%(dydbeta4, dydbeta4_func,dydbeta4_fd))\n",
        "print('dydomega4: Your value = %3.3f, True value = %3.3f'%(dydomega4, 0.864))\n",
        "print('dydbeta3: Your value = %3.3f, True value = %3.3f'%(dydbeta3, -0.211))\n",
        "print('dydomega3: Your value = %3.3f, True value = %3.3f'%(dydomega3, -0.139))\n",
        "print('dydbeta2: Your value = %3.3f, True value = %3.3f'%(dydbeta2, 0.476))\n",
        "print('dydomega2: Your value = %3.3f, True value = %3.3f'%(dydomega2, 2.415))\n",
        "print('dydbeta1: Your value = %3.3f, True value = %3.3f'%(dydbeta1, 4.830))\n",
        "print('dydomega1: Your value = %3.3f, True value = %3.3f'%(dydomega1, 4.552))\n",
        "print('dydbeta0: Your value = %3.3f, True value = %3.3f'%(dydbeta0, -0.646))\n",
        "print('dydomega0: Your value = %3.3f, Function value = %3.3f, Finite difference value = %3.3f'%(dydomega0, dydomega0_func,dydomega0_fd))"
      ],
      "metadata": {
        "id": "38eiOn2aHgHI"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Using this method, we can compute the derivatives quite easily without needing to compute very complicated expressions.  This is exactly the same way that the derivatives of the parameters are computed in the backpropagation algorithm.  In fact, this basically *is* the backpropagation algorithm."
      ],
      "metadata": {
        "id": "N2ZhrR-2fNa1"
      }
    }
  ]
}