Ch12_Optimization_Algorithms/Gradient_Descent.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gradient Descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section we are going to introduce the basic concepts underlying gradient descent. This is brief by necessity. See e.g. :cite:Boyd.Vandenberghe.2004 for an in-depth introduction to convex optimization. Although the latter is rarely used directly in deep learning, an understanding of gradient descent is key to understanding stochastic gradient descent algorithms. For instance, the optimization problem might diverge due to an overly large learning rate. This phenomenon can already be seen in gradient descent. Likewise, preconditioning is a common technique in gradient descent and carries over to more advanced algorithms. Let's start with a simple special case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient Descent in One Dimension\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gradient descent in one dimension is an excellent example to explain why the gradient descent algorithm may reduce the value of the objective function. Consider some continously differentiable real-valued function $f: \\mathbb{R} \\rightarrow \\mathbb{R}$. Using a Taylor expansion (:numref:chapter_math) we obtain that\n",
    "\n",
    "$$f(x + \\epsilon) = f(x) + \\epsilon f'(x) + O(\\epsilon^2).$$ :eqlabel:gd-taylor\n",
    "\n",
    "That is, in first approximation $f(x+\\epsilon)$ is given by the function value $f(x)$ and the first derivative $f'(x)$ at $x$. It is not unreasonable to assume that for small $\\epsilon$ moving in the direction of the negative gradient will decrease $f$. To keep things simple we pick a fixed step size $\\eta > 0$ and choose $\\epsilon = -\\eta f'(x)$. Plugging this into the Taylor expansion above we get\n",
    "\n",
    "$$f(x - \\eta f'(x)) = f(x) - \\eta f'^2(x) + O(\\eta^2 f'^2(x)).$$\n",
    "\n",
    "If the derivative $f'(x) \\neq 0$ does not vanish we make progress since $\\eta f'^2(x)>0$. Moreover, we can always choose $\\eta$ small enough for the higher order terms to become irrelevant. Hence we arrive at\n",
    "\n",
    "$$f(x - \\eta f'(x)) \\lessapprox f(x).$$\n",
    "\n",
    "This means that, if we use\n",
    "\n",
    "$$x \\leftarrow x - \\eta f'(x)$$\n",
    "\n",
    "to iterate $x$, the value of function $f(x)$ might decline. Therefore, in gradient descent we first choose an initial value $x$ and a constant $\\eta > 0$ and then use them to continuously iterate $x$ until the stop condition is reached, for example, when the magnitude of the gradient $|f'(x)|$ is small enough or the number of iterations has reached a certain value.\n",
    "\n",
    "For simplicity we choose the objective function $f(x)=x^2$ to illustrate how to implement gradient descent. Although we know that $x=0$ is the solution to minimize $f(x)$, we still use this simple function to observe how $x$ changes. As always, we begin by importing all required modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0, '..')\n",
    "%matplotlib inline\n",
    "import d2l\n",
    "import numpy as np\n",
    "import math\n",
    "\n",
    "def f(x):     return x**2  # objective function\n",
    "def gradf(x): return 2 * x # its derivative"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we use $x=10$ as the initial value and assume $\\eta=0.2$. Using gradient descent to iterate $x$ for 10 times we can see that, eventually, the value of $x$ approaches the optimal solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 10, x: 0.06046617599999997\n"
     ]
    }
   ],
   "source": [
    "def gd(eta):\n",
    "    x = 10\n",
    "    results = [x]\n",
    "    for i in range(10):\n",
    "        x -= eta * gradf(x)\n",
    "        results.append(x)\n",
    "    print('epoch 10, x:', x)\n",
    "    return results\n",
    "\n",
    "res = gd(0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The progress of optimizing over $x$ can be plotted as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\n",
       "<svg height=\"187pt\" version=\"1.1\" viewBox=\"0 0 253 187\" width=\"253pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;stroke-miterlimit:100000;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 187.095313 \n",
       "L 253.828125 187.095313 \n",
       "L 253.828125 0 \n",
       "L 0 0 \n",
       "L 0 187.095313 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 44.965625 151.539062 \n",
       "L 240.265625 151.539062 \n",
       "L 240.265625 12.039062 \n",
       "L 44.965625 12.039062 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_1\">\n",
       "    <path clip-path=\"url(#p3d33272580)\" d=\"M 44.965625 12.039062 \n",
       "L 49.750475 25.375123 \n",
       "L 54.437675 37.789507 \n",
       "L 59.027225 49.32239 \n",
       "L 63.519125 60.013112 \n",
       "L 67.913375 69.900175 \n",
       "L 72.209975 79.021243 \n",
       "L 76.408925 87.413144 \n",
       "L 80.510225 95.11187 \n",
       "L 84.513875 102.152575 \n",
       "L 88.419875 108.569575 \n",
       "L 92.228225 114.39635 \n",
       "L 95.938925 119.665544 \n",
       "L 99.551975 124.408963 \n",
       "L 103.067375 128.657575 \n",
       "L 106.485125 132.441512 \n",
       "L 109.805225 135.79007 \n",
       "L 113.027675 138.731707 \n",
       "L 116.152475 141.294043 \n",
       "L 119.277275 143.570683 \n",
       "L 122.304425 145.503734 \n",
       "L 125.233925 147.119144 \n",
       "L 128.163425 148.483454 \n",
       "L 130.995275 149.563603 \n",
       "L 133.827125 150.409112 \n",
       "L 136.658975 151.019983 \n",
       "L 139.393175 151.387147 \n",
       "L 142.127375 151.535575 \n",
       "L 144.861575 151.465267 \n",
       "L 147.595775 151.176223 \n",
       "L 150.329975 150.668443 \n",
       "L 153.161825 149.911935 \n",
       "L 155.993675 148.920787 \n",
       "L 158.825525 147.695001 \n",
       "L 161.755025 146.180031 \n",
       "L 164.684525 144.413961 \n",
       "L 167.711675 142.325227 \n",
       "L 170.738825 139.968375 \n",
       "L 173.863625 137.254263 \n",
       "L 177.086075 134.156107 \n",
       "L 180.406175 130.646287 \n",
       "L 183.726275 126.813943 \n",
       "L 187.144025 122.531991 \n",
       "L 190.659425 117.771135 \n",
       "L 194.272475 112.501243 \n",
       "L 197.983175 106.691347 \n",
       "L 201.791525 100.309641 \n",
       "L 205.697525 93.323481 \n",
       "L 209.701175 85.699387 \n",
       "L 213.802475 77.403043 \n",
       "L 218.001425 68.399295 \n",
       "L 222.298025 58.652151 \n",
       "L 226.692275 48.124783 \n",
       "L 231.184175 36.779527 \n",
       "L 235.773725 24.577881 \n",
       "L 240.167975 12.317923 \n",
       "L 240.167975 12.317923 \n",
       "\" style=\"fill:none;stroke:#0000ff;stroke-linecap:square;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_2\">\n",
       "    <path clip-path=\"url(#p3d33272580)\" d=\"M 240.265625 12.039062 \n",
       "L 201.205625 101.319063 \n",
       "L 177.769625 133.459863 \n",
       "L 163.708025 145.030551 \n",
       "L 155.271065 149.195998 \n",
       "L 150.208889 150.695559 \n",
       "L 147.171583 151.235401 \n",
       "L 145.3492 151.429744 \n",
       "L 144.25577 151.499708 \n",
       "L 143.599712 151.524895 \n",
       "L 143.206077 151.533962 \n",
       "\" style=\"fill:none;stroke:#008000;stroke-linecap:square;\"/>\n",
       "    <defs>\n",
       "     <path d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" id=\"m424ebfc181\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#p3d33272580)\">\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"240.26562500000003\" xlink:href=\"#m424ebfc181\" y=\"12.0390625\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"201.20562500000003\" xlink:href=\"#m424ebfc181\" y=\"101.3190625\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"177.76962500000002\" xlink:href=\"#m424ebfc181\" y=\"133.4598625\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"163.70802500000002\" xlink:href=\"#m424ebfc181\" y=\"145.0305505\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"155.27106500000002\" xlink:href=\"#m424ebfc181\" y=\"149.19599818\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"150.20888900000003\" xlink:href=\"#m424ebfc181\" y=\"150.6955593448\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"147.17158340000003\" xlink:href=\"#m424ebfc181\" y=\"151.235401364128\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"145.34920004000003\" xlink:href=\"#m424ebfc181\" y=\"151.42974449108607\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"144.25577002400001\" xlink:href=\"#m424ebfc181\" y=\"151.49970801679098\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"143.5997120144\" xlink:href=\"#m424ebfc181\" y=\"151.52489488604476\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"143.20607720864\" xlink:href=\"#m424ebfc181\" y=\"151.53396215897612\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 240.265625 151.539062 \n",
       "L 240.265625 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 44.965625 12.039062 \n",
       "L 240.265625 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 44.965625 151.539062 \n",
       "L 44.965625 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 44.965625 151.539062 \n",
       "L 240.265625 151.539062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path clip-path=\"url(#p3d33272580)\" d=\"M 44.965625 151.539062 \n",
       "L 44.965625 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 -4 \n",
       "\" id=\"m5e568fe270\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"44.96562500000002\" xlink:href=\"#m5e568fe270\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_5\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 4 \n",
       "\" id=\"ma4d020e9b9\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"44.96562500000002\" xlink:href=\"#ma4d020e9b9\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −10 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "\" id=\"BitstreamVeraSans-Roman-30\"/>\n",
       "       <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-31\"/>\n",
       "       <path d=\"M 10.59375 35.5 \n",
       "L 73.1875 35.5 \n",
       "L 73.1875 27.203125 \n",
       "L 10.59375 27.203125 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-2212\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(34.41328125000002 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-31\"/>\n",
       "       <use x=\"147.412109375\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <path clip-path=\"url(#p3d33272580)\" d=\"M 93.790625 151.539062 \n",
       "L 93.790625 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"93.79062500000002\" xlink:href=\"#m5e568fe270\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"93.79062500000002\" xlink:href=\"#ma4d020e9b9\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −5 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.796875 72.90625 \n",
       "L 49.515625 72.90625 \n",
       "L 49.515625 64.59375 \n",
       "L 19.828125 64.59375 \n",
       "L 19.828125 46.734375 \n",
       "Q 21.96875 47.46875 24.109375 47.828125 \n",
       "Q 26.265625 48.1875 28.421875 48.1875 \n",
       "Q 40.625 48.1875 47.75 41.5 \n",
       "Q 54.890625 34.8125 54.890625 23.390625 \n",
       "Q 54.890625 11.625 47.5625 5.09375 \n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \n",
       "Q 12.796875 0.140625 7.71875 1.703125 \n",
       "L 7.71875 11.625 \n",
       "Q 12.109375 9.234375 16.796875 8.0625 \n",
       "Q 21.484375 6.890625 26.703125 6.890625 \n",
       "Q 35.15625 6.890625 40.078125 11.328125 \n",
       "Q 45.015625 15.765625 45.015625 23.390625 \n",
       "Q 45.015625 31 40.078125 35.4375 \n",
       "Q 35.15625 39.890625 26.703125 39.890625 \n",
       "Q 22.75 39.890625 18.8125 39.015625 \n",
       "Q 14.890625 38.140625 10.796875 36.28125 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-35\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(86.41953125000002 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path clip-path=\"url(#p3d33272580)\" d=\"M 142.615625 151.539062 \n",
       "L 142.615625 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"142.61562500000002\" xlink:href=\"#m5e568fe270\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"142.61562500000002\" xlink:href=\"#ma4d020e9b9\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(139.43437500000002 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <path clip-path=\"url(#p3d33272580)\" d=\"M 191.440625 151.539062 \n",
       "L 191.440625 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"191.440625\" xlink:href=\"#m5e568fe270\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"191.440625\" xlink:href=\"#ma4d020e9b9\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(188.259375 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path clip-path=\"url(#p3d33272580)\" d=\"M 240.265625 151.539062 \n",
       "L 240.265625 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"240.26562500000003\" xlink:href=\"#m5e568fe270\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_17\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"240.26562500000003\" xlink:href=\"#ma4d020e9b9\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(233.90312500000002 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-31\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- x -->\n",
       "     <defs>\n",
       "      <path d=\"M 54.890625 54.6875 \n",
       "L 35.109375 28.078125 \n",
       "L 55.90625 0 \n",
       "L 45.3125 0 \n",
       "L 29.390625 21.484375 \n",
       "L 13.484375 0 \n",
       "L 2.875 0 \n",
       "L 24.125 28.609375 \n",
       "L 4.6875 54.6875 \n",
       "L 15.28125 54.6875 \n",
       "L 29.78125 35.203125 \n",
       "L 44.28125 54.6875 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-78\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(139.65625000000003 177.815625)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#BitstreamVeraSans-Roman-78\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_18\">\n",
       "      <path clip-path=\"url(#p3d33272580)\" d=\"M 44.965625 151.539062 \n",
       "L 240.265625 151.539062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_19\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 4 0 \n",
       "\" id=\"m0fb10f9a6b\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"44.965625\" xlink:href=\"#m0fb10f9a6b\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -4 0 \n",
       "\" id=\"mf6e8d54e04\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"240.265625\" xlink:href=\"#mf6e8d54e04\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(34.603125000000006 154.2984375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_21\">\n",
       "      <path clip-path=\"url(#p3d33272580)\" d=\"M 44.965625 123.639062 \n",
       "L 240.265625 123.639062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"44.965625\" xlink:href=\"#m0fb10f9a6b\" y=\"123.6390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_23\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"240.265625\" xlink:href=\"#mf6e8d54e04\" y=\"123.6390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 20 -->\n",
       "      <defs>\n",
       "       <path d=\"M 19.1875 8.296875 \n",
       "L 53.609375 8.296875 \n",
       "L 53.609375 0 \n",
       "L 7.328125 0 \n",
       "L 7.328125 8.296875 \n",
       "Q 12.9375 14.109375 22.625 23.890625 \n",
       "Q 32.328125 33.6875 34.8125 36.53125 \n",
       "Q 39.546875 41.84375 41.421875 45.53125 \n",
       "Q 43.3125 49.21875 43.3125 52.78125 \n",
       "Q 43.3125 58.59375 39.234375 62.25 \n",
       "Q 35.15625 65.921875 28.609375 65.921875 \n",
       "Q 23.96875 65.921875 18.8125 64.3125 \n",
       "Q 13.671875 62.703125 7.8125 59.421875 \n",
       "L 7.8125 69.390625 \n",
       "Q 13.765625 71.78125 18.9375 73 \n",
       "Q 24.125 74.21875 28.421875 74.21875 \n",
       "Q 39.75 74.21875 46.484375 68.546875 \n",
       "Q 53.21875 62.890625 53.21875 53.421875 \n",
       "Q 53.21875 48.921875 51.53125 44.890625 \n",
       "Q 49.859375 40.875 45.40625 35.40625 \n",
       "Q 44.1875 33.984375 37.640625 27.21875 \n",
       "Q 31.109375 20.453125 19.1875 8.296875 \n",
       "\" id=\"BitstreamVeraSans-Roman-32\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(28.240625 126.3984375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-32\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_24\">\n",
       "      <path clip-path=\"url(#p3d33272580)\" d=\"M 44.965625 95.739063 \n",
       "L 240.265625 95.739063 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_25\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"44.965625\" xlink:href=\"#m0fb10f9a6b\" y=\"95.7390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_26\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"240.265625\" xlink:href=\"#mf6e8d54e04\" y=\"95.7390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 40 -->\n",
       "      <defs>\n",
       "       <path d=\"M 37.796875 64.3125 \n",
       "L 12.890625 25.390625 \n",
       "L 37.796875 25.390625 \n",
       "z\n",
       "M 35.203125 72.90625 \n",
       "L 47.609375 72.90625 \n",
       "L 47.609375 25.390625 \n",
       "L 58.015625 25.390625 \n",
       "L 58.015625 17.1875 \n",
       "L 47.609375 17.1875 \n",
       "L 47.609375 0 \n",
       "L 37.796875 0 \n",
       "L 37.796875 17.1875 \n",
       "L 4.890625 17.1875 \n",
       "L 4.890625 26.703125 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-34\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(28.240625 98.49843750000001)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-34\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_27\">\n",
       "      <path clip-path=\"url(#p3d33272580)\" d=\"M 44.965625 67.839062 \n",
       "L 240.265625 67.839062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_28\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"44.965625\" xlink:href=\"#m0fb10f9a6b\" y=\"67.8390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_29\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"240.265625\" xlink:href=\"#mf6e8d54e04\" y=\"67.8390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 60 -->\n",
       "      <defs>\n",
       "       <path d=\"M 33.015625 40.375 \n",
       "Q 26.375 40.375 22.484375 35.828125 \n",
       "Q 18.609375 31.296875 18.609375 23.390625 \n",
       "Q 18.609375 15.53125 22.484375 10.953125 \n",
       "Q 26.375 6.390625 33.015625 6.390625 \n",
       "Q 39.65625 6.390625 43.53125 10.953125 \n",
       "Q 47.40625 15.53125 47.40625 23.390625 \n",
       "Q 47.40625 31.296875 43.53125 35.828125 \n",
       "Q 39.65625 40.375 33.015625 40.375 \n",
       "M 52.59375 71.296875 \n",
       "L 52.59375 62.3125 \n",
       "Q 48.875 64.0625 45.09375 64.984375 \n",
       "Q 41.3125 65.921875 37.59375 65.921875 \n",
       "Q 27.828125 65.921875 22.671875 59.328125 \n",
       "Q 17.53125 52.734375 16.796875 39.40625 \n",
       "Q 19.671875 43.65625 24.015625 45.921875 \n",
       "Q 28.375 48.1875 33.59375 48.1875 \n",
       "Q 44.578125 48.1875 50.953125 41.515625 \n",
       "Q 57.328125 34.859375 57.328125 23.390625 \n",
       "Q 57.328125 12.15625 50.6875 5.359375 \n",
       "Q 44.046875 -1.421875 33.015625 -1.421875 \n",
       "Q 20.359375 -1.421875 13.671875 8.265625 \n",
       "Q 6.984375 17.96875 6.984375 36.375 \n",
       "Q 6.984375 53.65625 15.1875 63.9375 \n",
       "Q 23.390625 74.21875 37.203125 74.21875 \n",
       "Q 40.921875 74.21875 44.703125 73.484375 \n",
       "Q 48.484375 72.75 52.59375 71.296875 \n",
       "\" id=\"BitstreamVeraSans-Roman-36\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(28.240625 70.5984375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-36\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_30\">\n",
       "      <path clip-path=\"url(#p3d33272580)\" d=\"M 44.965625 39.939063 \n",
       "L 240.265625 39.939063 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_31\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"44.965625\" xlink:href=\"#m0fb10f9a6b\" y=\"39.939062500000006\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_32\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"240.265625\" xlink:href=\"#mf6e8d54e04\" y=\"39.939062500000006\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 80 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 34.625 \n",
       "Q 24.75 34.625 20.71875 30.859375 \n",
       "Q 16.703125 27.09375 16.703125 20.515625 \n",
       "Q 16.703125 13.921875 20.71875 10.15625 \n",
       "Q 24.75 6.390625 31.78125 6.390625 \n",
       "Q 38.8125 6.390625 42.859375 10.171875 \n",
       "Q 46.921875 13.96875 46.921875 20.515625 \n",
       "Q 46.921875 27.09375 42.890625 30.859375 \n",
       "Q 38.875 34.625 31.78125 34.625 \n",
       "M 21.921875 38.8125 \n",
       "Q 15.578125 40.375 12.03125 44.71875 \n",
       "Q 8.5 49.078125 8.5 55.328125 \n",
       "Q 8.5 64.0625 14.71875 69.140625 \n",
       "Q 20.953125 74.21875 31.78125 74.21875 \n",
       "Q 42.671875 74.21875 48.875 69.140625 \n",
       "Q 55.078125 64.0625 55.078125 55.328125 \n",
       "Q 55.078125 49.078125 51.53125 44.71875 \n",
       "Q 48 40.375 41.703125 38.8125 \n",
       "Q 48.828125 37.15625 52.796875 32.3125 \n",
       "Q 56.78125 27.484375 56.78125 20.515625 \n",
       "Q 56.78125 9.90625 50.3125 4.234375 \n",
       "Q 43.84375 -1.421875 31.78125 -1.421875 \n",
       "Q 19.734375 -1.421875 13.25 4.234375 \n",
       "Q 6.78125 9.90625 6.78125 20.515625 \n",
       "Q 6.78125 27.484375 10.78125 32.3125 \n",
       "Q 14.796875 37.15625 21.921875 38.8125 \n",
       "M 18.3125 54.390625 \n",
       "Q 18.3125 48.734375 21.84375 45.5625 \n",
       "Q 25.390625 42.390625 31.78125 42.390625 \n",
       "Q 38.140625 42.390625 41.71875 45.5625 \n",
       "Q 45.3125 48.734375 45.3125 54.390625 \n",
       "Q 45.3125 60.0625 41.71875 63.234375 \n",
       "Q 38.140625 66.40625 31.78125 66.40625 \n",
       "Q 25.390625 66.40625 21.84375 63.234375 \n",
       "Q 18.3125 60.0625 18.3125 54.390625 \n",
       "\" id=\"BitstreamVeraSans-Roman-38\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(28.240625 42.69843750000001)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-38\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_33\">\n",
       "      <path clip-path=\"url(#p3d33272580)\" d=\"M 44.965625 12.039062 \n",
       "L 240.265625 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_34\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"44.965625\" xlink:href=\"#m0fb10f9a6b\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_35\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"240.265625\" xlink:href=\"#mf6e8d54e04\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(21.878125 14.798437500000006)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-31\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "       <use x=\"127.24609375\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- f(x) -->\n",
       "     <defs>\n",
       "      <path d=\"M 37.109375 75.984375 \n",
       "L 37.109375 68.5 \n",
       "L 28.515625 68.5 \n",
       "Q 23.6875 68.5 21.796875 66.546875 \n",
       "Q 19.921875 64.59375 19.921875 59.515625 \n",
       "L 19.921875 54.6875 \n",
       "L 34.71875 54.6875 \n",
       "L 34.71875 47.703125 \n",
       "L 19.921875 47.703125 \n",
       "L 19.921875 0 \n",
       "L 10.890625 0 \n",
       "L 10.890625 47.703125 \n",
       "L 2.296875 47.703125 \n",
       "L 2.296875 54.6875 \n",
       "L 10.890625 54.6875 \n",
       "L 10.890625 58.5 \n",
       "Q 10.890625 67.625 15.140625 71.796875 \n",
       "Q 19.390625 75.984375 28.609375 75.984375 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-66\"/>\n",
       "      <path d=\"M 31 75.875 \n",
       "Q 24.46875 64.65625 21.28125 53.65625 \n",
       "Q 18.109375 42.671875 18.109375 31.390625 \n",
       "Q 18.109375 20.125 21.3125 9.0625 \n",
       "Q 24.515625 -2 31 -13.1875 \n",
       "L 23.1875 -13.1875 \n",
       "Q 15.875 -1.703125 12.234375 9.375 \n",
       "Q 8.59375 20.453125 8.59375 31.390625 \n",
       "Q 8.59375 42.28125 12.203125 53.3125 \n",
       "Q 15.828125 64.359375 23.1875 75.875 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-28\"/>\n",
       "      <path d=\"M 8.015625 75.875 \n",
       "L 15.828125 75.875 \n",
       "Q 23.140625 64.359375 26.78125 53.3125 \n",
       "Q 30.421875 42.28125 30.421875 31.390625 \n",
       "Q 30.421875 20.453125 26.78125 9.375 \n",
       "Q 23.140625 -1.703125 15.828125 -13.1875 \n",
       "L 8.015625 -13.1875 \n",
       "Q 14.5 -2 17.703125 9.0625 \n",
       "Q 20.90625 20.125 20.90625 31.390625 \n",
       "Q 20.90625 42.671875 17.703125 53.65625 \n",
       "Q 14.5 64.65625 8.015625 75.875 \n",
       "\" id=\"BitstreamVeraSans-Roman-29\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(14.7984375 90.41015625)rotate(-90.0)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#BitstreamVeraSans-Roman-66\"/>\n",
       "      <use x=\"35.205078125\" xlink:href=\"#BitstreamVeraSans-Roman-28\"/>\n",
       "      <use x=\"74.21875\" xlink:href=\"#BitstreamVeraSans-Roman-78\"/>\n",
       "      <use x=\"133.3984375\" xlink:href=\"#BitstreamVeraSans-Roman-29\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p3d33272580\">\n",
       "   <rect height=\"139.5\" width=\"195.3\" x=\"44.965625\" y=\"12.0390625\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f234ef43470>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def show_trace(res):\n",
    "    n = max(abs(min(res)), abs(max(res)))\n",
    "    f_line = np.arange(-n, n, 0.01)\n",
    "    d2l.set_figsize((3.5, 2.5))\n",
    "    d2l.plot([f_line, res], [[f(x) for x in f_line], [f(x) for x in res]],'x', 'f(x)', fmts=['-', '-o'])\n",
    "\n",
    "show_trace(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning Rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The learning rate $\\eta$ can be set by the algorithm designer. If we use a learning rate that is too small, it will cause $x$ to update very slowly, requiring more iterations to get a better solution. To show what happens in such a case, consider the progress in the same optimization problem for $\\eta = 0.05$. As we can see, even after 10 steps we are still very far from the optimal solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 10, x: 3.4867844009999995\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\n",
       "<svg height=\"187pt\" version=\"1.1\" viewBox=\"0 0 253 187\" width=\"253pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;stroke-miterlimit:100000;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 187.095313 \n",
       "L 253.828125 187.095313 \n",
       "L 253.828125 0 \n",
       "L 0 0 \n",
       "L 0 187.095313 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 44.965625 151.539062 \n",
       "L 240.265625 151.539062 \n",
       "L 240.265625 12.039062 \n",
       "L 44.965625 12.039062 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_1\">\n",
       "    <path clip-path=\"url(#p068ea6c088)\" d=\"M 44.965625 12.039062 \n",
       "L 49.750475 25.375123 \n",
       "L 54.437675 37.789507 \n",
       "L 59.027225 49.32239 \n",
       "L 63.519125 60.013112 \n",
       "L 67.913375 69.900175 \n",
       "L 72.209975 79.021243 \n",
       "L 76.408925 87.413144 \n",
       "L 80.510225 95.11187 \n",
       "L 84.513875 102.152575 \n",
       "L 88.419875 108.569575 \n",
       "L 92.228225 114.39635 \n",
       "L 95.938925 119.665544 \n",
       "L 99.551975 124.408963 \n",
       "L 103.067375 128.657575 \n",
       "L 106.485125 132.441512 \n",
       "L 109.805225 135.79007 \n",
       "L 113.027675 138.731707 \n",
       "L 116.152475 141.294043 \n",
       "L 119.277275 143.570683 \n",
       "L 122.304425 145.503734 \n",
       "L 125.233925 147.119144 \n",
       "L 128.163425 148.483454 \n",
       "L 130.995275 149.563603 \n",
       "L 133.827125 150.409112 \n",
       "L 136.658975 151.019983 \n",
       "L 139.393175 151.387147 \n",
       "L 142.127375 151.535575 \n",
       "L 144.861575 151.465267 \n",
       "L 147.595775 151.176223 \n",
       "L 150.329975 150.668443 \n",
       "L 153.161825 149.911935 \n",
       "L 155.993675 148.920787 \n",
       "L 158.825525 147.695001 \n",
       "L 161.755025 146.180031 \n",
       "L 164.684525 144.413961 \n",
       "L 167.711675 142.325227 \n",
       "L 170.738825 139.968375 \n",
       "L 173.863625 137.254263 \n",
       "L 177.086075 134.156107 \n",
       "L 180.406175 130.646287 \n",
       "L 183.726275 126.813943 \n",
       "L 187.144025 122.531991 \n",
       "L 190.659425 117.771135 \n",
       "L 194.272475 112.501243 \n",
       "L 197.983175 106.691347 \n",
       "L 201.791525 100.309641 \n",
       "L 205.697525 93.323481 \n",
       "L 209.701175 85.699387 \n",
       "L 213.802475 77.403043 \n",
       "L 218.001425 68.399295 \n",
       "L 222.298025 58.652151 \n",
       "L 226.692275 48.124783 \n",
       "L 231.184175 36.779527 \n",
       "L 235.773725 24.577881 \n",
       "L 240.167975 12.317923 \n",
       "L 240.167975 12.317923 \n",
       "\" style=\"fill:none;stroke:#0000ff;stroke-linecap:square;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_2\">\n",
       "    <path clip-path=\"url(#p068ea6c088)\" d=\"M 240.265625 12.039062 \n",
       "L 230.500625 38.544062 \n",
       "L 221.712125 60.013113 \n",
       "L 213.802475 77.403043 \n",
       "L 206.68379 91.488887 \n",
       "L 200.276974 102.89842 \n",
       "L 194.510839 112.140142 \n",
       "L 189.321317 119.625937 \n",
       "L 184.650748 125.689431 \n",
       "L 180.447236 130.600861 \n",
       "L 176.664075 134.579119 \n",
       "\" style=\"fill:none;stroke:#008000;stroke-linecap:square;\"/>\n",
       "    <defs>\n",
       "     <path d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" id=\"mbf3c00d50d\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#p068ea6c088)\">\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"240.26562500000003\" xlink:href=\"#mbf3c00d50d\" y=\"12.0390625\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"230.500625\" xlink:href=\"#mbf3c00d50d\" y=\"38.544062499999995\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"221.71212500000001\" xlink:href=\"#mbf3c00d50d\" y=\"60.013112500000005\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"213.80247500000002\" xlink:href=\"#mbf3c00d50d\" y=\"77.40304300000001\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"206.68379000000002\" xlink:href=\"#mbf3c00d50d\" y=\"91.48888670500001\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"200.27697350000003\" xlink:href=\"#mbf3c00d50d\" y=\"102.89842010605001\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"194.51083865000004\" xlink:href=\"#mbf3c00d50d\" y=\"112.14014216090051\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"189.32131728500002\" xlink:href=\"#mbf3c00d50d\" y=\"119.6259370253294\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"184.65074805650002\" xlink:href=\"#mbf3c00d50d\" y=\"125.68943086551683\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"180.44723575085\" xlink:href=\"#mbf3c00d50d\" y=\"130.60086087606862\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"176.66407467576502\" xlink:href=\"#mbf3c00d50d\" y=\"134.5791191846156\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 240.265625 151.539062 \n",
       "L 240.265625 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 44.965625 12.039062 \n",
       "L 240.265625 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 44.965625 151.539062 \n",
       "L 44.965625 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 44.965625 151.539062 \n",
       "L 240.265625 151.539062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path clip-path=\"url(#p068ea6c088)\" d=\"M 44.965625 151.539062 \n",
       "L 44.965625 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 -4 \n",
       "\" id=\"m7ef6360cd5\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"44.96562500000002\" xlink:href=\"#m7ef6360cd5\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_5\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 4 \n",
       "\" id=\"m56528464b9\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"44.96562500000002\" xlink:href=\"#m56528464b9\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −10 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "\" id=\"BitstreamVeraSans-Roman-30\"/>\n",
       "       <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-31\"/>\n",
       "       <path d=\"M 10.59375 35.5 \n",
       "L 73.1875 35.5 \n",
       "L 73.1875 27.203125 \n",
       "L 10.59375 27.203125 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-2212\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(34.41328125000002 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-31\"/>\n",
       "       <use x=\"147.412109375\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <path clip-path=\"url(#p068ea6c088)\" d=\"M 93.790625 151.539062 \n",
       "L 93.790625 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"93.79062500000002\" xlink:href=\"#m7ef6360cd5\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"93.79062500000002\" xlink:href=\"#m56528464b9\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −5 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.796875 72.90625 \n",
       "L 49.515625 72.90625 \n",
       "L 49.515625 64.59375 \n",
       "L 19.828125 64.59375 \n",
       "L 19.828125 46.734375 \n",
       "Q 21.96875 47.46875 24.109375 47.828125 \n",
       "Q 26.265625 48.1875 28.421875 48.1875 \n",
       "Q 40.625 48.1875 47.75 41.5 \n",
       "Q 54.890625 34.8125 54.890625 23.390625 \n",
       "Q 54.890625 11.625 47.5625 5.09375 \n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \n",
       "Q 12.796875 0.140625 7.71875 1.703125 \n",
       "L 7.71875 11.625 \n",
       "Q 12.109375 9.234375 16.796875 8.0625 \n",
       "Q 21.484375 6.890625 26.703125 6.890625 \n",
       "Q 35.15625 6.890625 40.078125 11.328125 \n",
       "Q 45.015625 15.765625 45.015625 23.390625 \n",
       "Q 45.015625 31 40.078125 35.4375 \n",
       "Q 35.15625 39.890625 26.703125 39.890625 \n",
       "Q 22.75 39.890625 18.8125 39.015625 \n",
       "Q 14.890625 38.140625 10.796875 36.28125 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-35\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(86.41953125000002 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path clip-path=\"url(#p068ea6c088)\" d=\"M 142.615625 151.539062 \n",
       "L 142.615625 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"142.61562500000002\" xlink:href=\"#m7ef6360cd5\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"142.61562500000002\" xlink:href=\"#m56528464b9\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(139.43437500000002 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <path clip-path=\"url(#p068ea6c088)\" d=\"M 191.440625 151.539062 \n",
       "L 191.440625 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"191.440625\" xlink:href=\"#m7ef6360cd5\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"191.440625\" xlink:href=\"#m56528464b9\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(188.259375 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path clip-path=\"url(#p068ea6c088)\" d=\"M 240.265625 151.539062 \n",
       "L 240.265625 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"240.26562500000003\" xlink:href=\"#m7ef6360cd5\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_17\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"240.26562500000003\" xlink:href=\"#m56528464b9\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(233.90312500000002 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-31\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- x -->\n",
       "     <defs>\n",
       "      <path d=\"M 54.890625 54.6875 \n",
       "L 35.109375 28.078125 \n",
       "L 55.90625 0 \n",
       "L 45.3125 0 \n",
       "L 29.390625 21.484375 \n",
       "L 13.484375 0 \n",
       "L 2.875 0 \n",
       "L 24.125 28.609375 \n",
       "L 4.6875 54.6875 \n",
       "L 15.28125 54.6875 \n",
       "L 29.78125 35.203125 \n",
       "L 44.28125 54.6875 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-78\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(139.65625000000003 177.815625)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#BitstreamVeraSans-Roman-78\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_18\">\n",
       "      <path clip-path=\"url(#p068ea6c088)\" d=\"M 44.965625 151.539062 \n",
       "L 240.265625 151.539062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_19\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 4 0 \n",
       "\" id=\"mbaa037cb26\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"44.965625\" xlink:href=\"#mbaa037cb26\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -4 0 \n",
       "\" id=\"m8d01c6bf7b\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"240.265625\" xlink:href=\"#m8d01c6bf7b\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(34.603125000000006 154.2984375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_21\">\n",
       "      <path clip-path=\"url(#p068ea6c088)\" d=\"M 44.965625 123.639062 \n",
       "L 240.265625 123.639062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"44.965625\" xlink:href=\"#mbaa037cb26\" y=\"123.6390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_23\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"240.265625\" xlink:href=\"#m8d01c6bf7b\" y=\"123.6390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 20 -->\n",
       "      <defs>\n",
       "       <path d=\"M 19.1875 8.296875 \n",
       "L 53.609375 8.296875 \n",
       "L 53.609375 0 \n",
       "L 7.328125 0 \n",
       "L 7.328125 8.296875 \n",
       "Q 12.9375 14.109375 22.625 23.890625 \n",
       "Q 32.328125 33.6875 34.8125 36.53125 \n",
       "Q 39.546875 41.84375 41.421875 45.53125 \n",
       "Q 43.3125 49.21875 43.3125 52.78125 \n",
       "Q 43.3125 58.59375 39.234375 62.25 \n",
       "Q 35.15625 65.921875 28.609375 65.921875 \n",
       "Q 23.96875 65.921875 18.8125 64.3125 \n",
       "Q 13.671875 62.703125 7.8125 59.421875 \n",
       "L 7.8125 69.390625 \n",
       "Q 13.765625 71.78125 18.9375 73 \n",
       "Q 24.125 74.21875 28.421875 74.21875 \n",
       "Q 39.75 74.21875 46.484375 68.546875 \n",
       "Q 53.21875 62.890625 53.21875 53.421875 \n",
       "Q 53.21875 48.921875 51.53125 44.890625 \n",
       "Q 49.859375 40.875 45.40625 35.40625 \n",
       "Q 44.1875 33.984375 37.640625 27.21875 \n",
       "Q 31.109375 20.453125 19.1875 8.296875 \n",
       "\" id=\"BitstreamVeraSans-Roman-32\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(28.240625 126.3984375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-32\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_24\">\n",
       "      <path clip-path=\"url(#p068ea6c088)\" d=\"M 44.965625 95.739063 \n",
       "L 240.265625 95.739063 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_25\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"44.965625\" xlink:href=\"#mbaa037cb26\" y=\"95.7390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_26\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"240.265625\" xlink:href=\"#m8d01c6bf7b\" y=\"95.7390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 40 -->\n",
       "      <defs>\n",
       "       <path d=\"M 37.796875 64.3125 \n",
       "L 12.890625 25.390625 \n",
       "L 37.796875 25.390625 \n",
       "z\n",
       "M 35.203125 72.90625 \n",
       "L 47.609375 72.90625 \n",
       "L 47.609375 25.390625 \n",
       "L 58.015625 25.390625 \n",
       "L 58.015625 17.1875 \n",
       "L 47.609375 17.1875 \n",
       "L 47.609375 0 \n",
       "L 37.796875 0 \n",
       "L 37.796875 17.1875 \n",
       "L 4.890625 17.1875 \n",
       "L 4.890625 26.703125 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-34\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(28.240625 98.49843750000001)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-34\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_27\">\n",
       "      <path clip-path=\"url(#p068ea6c088)\" d=\"M 44.965625 67.839062 \n",
       "L 240.265625 67.839062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_28\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"44.965625\" xlink:href=\"#mbaa037cb26\" y=\"67.8390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_29\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"240.265625\" xlink:href=\"#m8d01c6bf7b\" y=\"67.8390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 60 -->\n",
       "      <defs>\n",
       "       <path d=\"M 33.015625 40.375 \n",
       "Q 26.375 40.375 22.484375 35.828125 \n",
       "Q 18.609375 31.296875 18.609375 23.390625 \n",
       "Q 18.609375 15.53125 22.484375 10.953125 \n",
       "Q 26.375 6.390625 33.015625 6.390625 \n",
       "Q 39.65625 6.390625 43.53125 10.953125 \n",
       "Q 47.40625 15.53125 47.40625 23.390625 \n",
       "Q 47.40625 31.296875 43.53125 35.828125 \n",
       "Q 39.65625 40.375 33.015625 40.375 \n",
       "M 52.59375 71.296875 \n",
       "L 52.59375 62.3125 \n",
       "Q 48.875 64.0625 45.09375 64.984375 \n",
       "Q 41.3125 65.921875 37.59375 65.921875 \n",
       "Q 27.828125 65.921875 22.671875 59.328125 \n",
       "Q 17.53125 52.734375 16.796875 39.40625 \n",
       "Q 19.671875 43.65625 24.015625 45.921875 \n",
       "Q 28.375 48.1875 33.59375 48.1875 \n",
       "Q 44.578125 48.1875 50.953125 41.515625 \n",
       "Q 57.328125 34.859375 57.328125 23.390625 \n",
       "Q 57.328125 12.15625 50.6875 5.359375 \n",
       "Q 44.046875 -1.421875 33.015625 -1.421875 \n",
       "Q 20.359375 -1.421875 13.671875 8.265625 \n",
       "Q 6.984375 17.96875 6.984375 36.375 \n",
       "Q 6.984375 53.65625 15.1875 63.9375 \n",
       "Q 23.390625 74.21875 37.203125 74.21875 \n",
       "Q 40.921875 74.21875 44.703125 73.484375 \n",
       "Q 48.484375 72.75 52.59375 71.296875 \n",
       "\" id=\"BitstreamVeraSans-Roman-36\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(28.240625 70.5984375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-36\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_30\">\n",
       "      <path clip-path=\"url(#p068ea6c088)\" d=\"M 44.965625 39.939063 \n",
       "L 240.265625 39.939063 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_31\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"44.965625\" xlink:href=\"#mbaa037cb26\" y=\"39.939062500000006\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_32\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"240.265625\" xlink:href=\"#m8d01c6bf7b\" y=\"39.939062500000006\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 80 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 34.625 \n",
       "Q 24.75 34.625 20.71875 30.859375 \n",
       "Q 16.703125 27.09375 16.703125 20.515625 \n",
       "Q 16.703125 13.921875 20.71875 10.15625 \n",
       "Q 24.75 6.390625 31.78125 6.390625 \n",
       "Q 38.8125 6.390625 42.859375 10.171875 \n",
       "Q 46.921875 13.96875 46.921875 20.515625 \n",
       "Q 46.921875 27.09375 42.890625 30.859375 \n",
       "Q 38.875 34.625 31.78125 34.625 \n",
       "M 21.921875 38.8125 \n",
       "Q 15.578125 40.375 12.03125 44.71875 \n",
       "Q 8.5 49.078125 8.5 55.328125 \n",
       "Q 8.5 64.0625 14.71875 69.140625 \n",
       "Q 20.953125 74.21875 31.78125 74.21875 \n",
       "Q 42.671875 74.21875 48.875 69.140625 \n",
       "Q 55.078125 64.0625 55.078125 55.328125 \n",
       "Q 55.078125 49.078125 51.53125 44.71875 \n",
       "Q 48 40.375 41.703125 38.8125 \n",
       "Q 48.828125 37.15625 52.796875 32.3125 \n",
       "Q 56.78125 27.484375 56.78125 20.515625 \n",
       "Q 56.78125 9.90625 50.3125 4.234375 \n",
       "Q 43.84375 -1.421875 31.78125 -1.421875 \n",
       "Q 19.734375 -1.421875 13.25 4.234375 \n",
       "Q 6.78125 9.90625 6.78125 20.515625 \n",
       "Q 6.78125 27.484375 10.78125 32.3125 \n",
       "Q 14.796875 37.15625 21.921875 38.8125 \n",
       "M 18.3125 54.390625 \n",
       "Q 18.3125 48.734375 21.84375 45.5625 \n",
       "Q 25.390625 42.390625 31.78125 42.390625 \n",
       "Q 38.140625 42.390625 41.71875 45.5625 \n",
       "Q 45.3125 48.734375 45.3125 54.390625 \n",
       "Q 45.3125 60.0625 41.71875 63.234375 \n",
       "Q 38.140625 66.40625 31.78125 66.40625 \n",
       "Q 25.390625 66.40625 21.84375 63.234375 \n",
       "Q 18.3125 60.0625 18.3125 54.390625 \n",
       "\" id=\"BitstreamVeraSans-Roman-38\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(28.240625 42.69843750000001)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-38\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_33\">\n",
       "      <path clip-path=\"url(#p068ea6c088)\" d=\"M 44.965625 12.039062 \n",
       "L 240.265625 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_34\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"44.965625\" xlink:href=\"#mbaa037cb26\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_35\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"240.265625\" xlink:href=\"#m8d01c6bf7b\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 100 -->\n",
       "      <g transform=\"translate(21.878125 14.798437500000006)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-31\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "       <use x=\"127.24609375\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- f(x) -->\n",
       "     <defs>\n",
       "      <path d=\"M 37.109375 75.984375 \n",
       "L 37.109375 68.5 \n",
       "L 28.515625 68.5 \n",
       "Q 23.6875 68.5 21.796875 66.546875 \n",
       "Q 19.921875 64.59375 19.921875 59.515625 \n",
       "L 19.921875 54.6875 \n",
       "L 34.71875 54.6875 \n",
       "L 34.71875 47.703125 \n",
       "L 19.921875 47.703125 \n",
       "L 19.921875 0 \n",
       "L 10.890625 0 \n",
       "L 10.890625 47.703125 \n",
       "L 2.296875 47.703125 \n",
       "L 2.296875 54.6875 \n",
       "L 10.890625 54.6875 \n",
       "L 10.890625 58.5 \n",
       "Q 10.890625 67.625 15.140625 71.796875 \n",
       "Q 19.390625 75.984375 28.609375 75.984375 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-66\"/>\n",
       "      <path d=\"M 31 75.875 \n",
       "Q 24.46875 64.65625 21.28125 53.65625 \n",
       "Q 18.109375 42.671875 18.109375 31.390625 \n",
       "Q 18.109375 20.125 21.3125 9.0625 \n",
       "Q 24.515625 -2 31 -13.1875 \n",
       "L 23.1875 -13.1875 \n",
       "Q 15.875 -1.703125 12.234375 9.375 \n",
       "Q 8.59375 20.453125 8.59375 31.390625 \n",
       "Q 8.59375 42.28125 12.203125 53.3125 \n",
       "Q 15.828125 64.359375 23.1875 75.875 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-28\"/>\n",
       "      <path d=\"M 8.015625 75.875 \n",
       "L 15.828125 75.875 \n",
       "Q 23.140625 64.359375 26.78125 53.3125 \n",
       "Q 30.421875 42.28125 30.421875 31.390625 \n",
       "Q 30.421875 20.453125 26.78125 9.375 \n",
       "Q 23.140625 -1.703125 15.828125 -13.1875 \n",
       "L 8.015625 -13.1875 \n",
       "Q 14.5 -2 17.703125 9.0625 \n",
       "Q 20.90625 20.125 20.90625 31.390625 \n",
       "Q 20.90625 42.671875 17.703125 53.65625 \n",
       "Q 14.5 64.65625 8.015625 75.875 \n",
       "\" id=\"BitstreamVeraSans-Roman-29\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(14.7984375 90.41015625)rotate(-90.0)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#BitstreamVeraSans-Roman-66\"/>\n",
       "      <use x=\"35.205078125\" xlink:href=\"#BitstreamVeraSans-Roman-28\"/>\n",
       "      <use x=\"74.21875\" xlink:href=\"#BitstreamVeraSans-Roman-78\"/>\n",
       "      <use x=\"133.3984375\" xlink:href=\"#BitstreamVeraSans-Roman-29\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p068ea6c088\">\n",
       "   <rect height=\"139.5\" width=\"195.3\" x=\"44.965625\" y=\"12.0390625\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f23b82f31d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_trace(gd(0.05))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Conversely, if we use an excessively high learning rate, $\\left|\\eta f'(x)\\right|$ might be too large for the first-order Taylor expansion formula. That is, the term $O(\\eta^2 f'^2(x))$ in :eqref:gd-taylor might become significant. In this case, we cannot guarantee that the iteration of $x$ will be able to lower the value of $f(x)$. For example, when we set the learning rate to $\\eta=1.1$, $x$ overshoots the optimal solution $x=0$ and gradually diverges.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 10, x: 61.917364224000096\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\n",
       "<svg height=\"187pt\" version=\"1.1\" viewBox=\"0 0 260 187\" width=\"260pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;stroke-miterlimit:100000;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 187.095313 \n",
       "L 260.190625 187.095313 \n",
       "L 260.190625 0 \n",
       "L 0 0 \n",
       "L 0 187.095313 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 51.328125 151.539062 \n",
       "L 246.628125 151.539062 \n",
       "L 246.628125 12.039062 \n",
       "L 51.328125 12.039062 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_1\">\n",
       "    <path clip-path=\"url(#pb8412aa6da)\" d=\"M 73.400242 17.836683 \n",
       "L 77.57478 32.198848 \n",
       "L 81.639461 45.39916 \n",
       "L 85.606492 57.536544 \n",
       "L 89.463667 68.631403 \n",
       "L 93.210986 78.743246 \n",
       "L 96.848449 87.929889 \n",
       "L 100.388261 96.275222 \n",
       "L 103.818217 103.80199 \n",
       "L 107.150524 110.587015 \n",
       "L 110.38518 116.675999 \n",
       "L 113.50998 122.093039 \n",
       "L 116.53713 126.904887 \n",
       "L 119.478836 131.169918 \n",
       "L 122.322892 134.908205 \n",
       "L 125.081505 138.17241 \n",
       "L 127.754674 140.995637 \n",
       "L 130.342399 143.409965 \n",
       "L 132.856886 145.45566 \n",
       "L 135.298136 147.158588 \n",
       "L 137.666149 148.543857 \n",
       "L 139.985336 149.646115 \n",
       "L 142.255699 150.481267 \n",
       "L 144.489442 151.067448 \n",
       "L 146.686567 151.416146 \n",
       "L 148.871486 151.538796 \n",
       "L 151.044199 151.439145 \n",
       "L 153.229117 151.116072 \n",
       "L 155.426242 150.565832 \n",
       "L 157.64778 149.779708 \n",
       "L 159.905936 148.743842 \n",
       "L 162.212917 147.439062 \n",
       "L 164.58093 145.840635 \n",
       "L 167.009974 143.928264 \n",
       "L 169.500049 141.68113 \n",
       "L 172.063361 139.064701 \n",
       "L 174.712117 136.037909 \n",
       "L 177.434111 132.585233 \n",
       "L 180.241549 128.660814 \n",
       "L 183.146636 124.211443 \n",
       "L 186.137167 119.218514 \n",
       "L 189.225349 113.623147 \n",
       "L 192.41118 107.382989 \n",
       "L 195.694661 100.454349 \n",
       "L 199.075792 92.792196 \n",
       "L 202.554574 84.350161 \n",
       "L 206.131005 75.080535 \n",
       "L 209.817292 64.899507 \n",
       "L 213.60123 53.787134 \n",
       "L 217.495024 41.652374 \n",
       "L 221.486467 28.476587 \n",
       "L 224.550236 17.857103 \n",
       "L 224.550236 17.857103 \n",
       "\" style=\"fill:none;stroke:#0000ff;stroke-linecap:square;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_2\">\n",
       "    <path clip-path=\"url(#pb8412aa6da)\" d=\"M 161.184375 148.051562 \n",
       "L 134.330625 146.517063 \n",
       "L 166.555125 144.307382 \n",
       "L 127.885725 141.125443 \n",
       "L 174.289005 136.543451 \n",
       "L 118.605069 129.945382 \n",
       "L 185.425792 120.444162 \n",
       "L 105.240924 106.762406 \n",
       "L 201.462766 87.060677 \n",
       "L 85.996556 58.690188 \n",
       "L 224.556008 17.836683 \n",
       "\" style=\"fill:none;stroke:#008000;stroke-linecap:square;\"/>\n",
       "    <defs>\n",
       "     <path d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" id=\"m0674ef5bce\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#pb8412aa6da)\">\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"161.18437500000002\" xlink:href=\"#m0674ef5bce\" y=\"148.0515625\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"134.330625\" xlink:href=\"#m0674ef5bce\" y=\"146.5170625\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"166.555125\" xlink:href=\"#m0674ef5bce\" y=\"144.3073825\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"127.885725\" xlink:href=\"#m0674ef5bce\" y=\"141.1254433\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"174.28900500000003\" xlink:href=\"#m0674ef5bce\" y=\"136.54345085199998\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"118.60506899999999\" xlink:href=\"#m0674ef5bce\" y=\"129.94538172688\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"185.42579220000005\" xlink:href=\"#m0674ef5bce\" y=\"120.44416218670715\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"105.24092435999995\" xlink:href=\"#m0674ef5bce\" y=\"106.76240604885827\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"201.46276576800008\" xlink:href=\"#m0674ef5bce\" y=\"87.06067721035589\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"85.99655607839992\" xlink:href=\"#m0674ef5bce\" y=\"58.69018768291245\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"224.55600770592014\" xlink:href=\"#m0674ef5bce\" y=\"17.836682763393867\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 246.628125 151.539062 \n",
       "L 246.628125 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 51.328125 12.039062 \n",
       "L 246.628125 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 51.328125 151.539062 \n",
       "L 51.328125 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 51.328125 151.539062 \n",
       "L 246.628125 151.539062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path clip-path=\"url(#pb8412aa6da)\" d=\"M 51.328125 151.539062 \n",
       "L 51.328125 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 -4 \n",
       "\" id=\"m1fd6d6b6fc\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"51.328125\" xlink:href=\"#m1fd6d6b6fc\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_5\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 4 \n",
       "\" id=\"md8e7f41bf5\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"51.328125\" xlink:href=\"#md8e7f41bf5\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −80 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "\" id=\"BitstreamVeraSans-Roman-30\"/>\n",
       "       <path d=\"M 31.78125 34.625 \n",
       "Q 24.75 34.625 20.71875 30.859375 \n",
       "Q 16.703125 27.09375 16.703125 20.515625 \n",
       "Q 16.703125 13.921875 20.71875 10.15625 \n",
       "Q 24.75 6.390625 31.78125 6.390625 \n",
       "Q 38.8125 6.390625 42.859375 10.171875 \n",
       "Q 46.921875 13.96875 46.921875 20.515625 \n",
       "Q 46.921875 27.09375 42.890625 30.859375 \n",
       "Q 38.875 34.625 31.78125 34.625 \n",
       "M 21.921875 38.8125 \n",
       "Q 15.578125 40.375 12.03125 44.71875 \n",
       "Q 8.5 49.078125 8.5 55.328125 \n",
       "Q 8.5 64.0625 14.71875 69.140625 \n",
       "Q 20.953125 74.21875 31.78125 74.21875 \n",
       "Q 42.671875 74.21875 48.875 69.140625 \n",
       "Q 55.078125 64.0625 55.078125 55.328125 \n",
       "Q 55.078125 49.078125 51.53125 44.71875 \n",
       "Q 48 40.375 41.703125 38.8125 \n",
       "Q 48.828125 37.15625 52.796875 32.3125 \n",
       "Q 56.78125 27.484375 56.78125 20.515625 \n",
       "Q 56.78125 9.90625 50.3125 4.234375 \n",
       "Q 43.84375 -1.421875 31.78125 -1.421875 \n",
       "Q 19.734375 -1.421875 13.25 4.234375 \n",
       "Q 6.78125 9.90625 6.78125 20.515625 \n",
       "Q 6.78125 27.484375 10.78125 32.3125 \n",
       "Q 14.796875 37.15625 21.921875 38.8125 \n",
       "M 18.3125 54.390625 \n",
       "Q 18.3125 48.734375 21.84375 45.5625 \n",
       "Q 25.390625 42.390625 31.78125 42.390625 \n",
       "Q 38.140625 42.390625 41.71875 45.5625 \n",
       "Q 45.3125 48.734375 45.3125 54.390625 \n",
       "Q 45.3125 60.0625 41.71875 63.234375 \n",
       "Q 38.140625 66.40625 31.78125 66.40625 \n",
       "Q 25.390625 66.40625 21.84375 63.234375 \n",
       "Q 18.3125 60.0625 18.3125 54.390625 \n",
       "\" id=\"BitstreamVeraSans-Roman-38\"/>\n",
       "       <path d=\"M 10.59375 35.5 \n",
       "L 73.1875 35.5 \n",
       "L 73.1875 27.203125 \n",
       "L 10.59375 27.203125 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-2212\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(40.77578125 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-38\"/>\n",
       "       <use x=\"147.412109375\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <path clip-path=\"url(#pb8412aa6da)\" d=\"M 75.740625 151.539062 \n",
       "L 75.740625 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"75.740625\" xlink:href=\"#m1fd6d6b6fc\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"75.740625\" xlink:href=\"#md8e7f41bf5\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −60 -->\n",
       "      <defs>\n",
       "       <path d=\"M 33.015625 40.375 \n",
       "Q 26.375 40.375 22.484375 35.828125 \n",
       "Q 18.609375 31.296875 18.609375 23.390625 \n",
       "Q 18.609375 15.53125 22.484375 10.953125 \n",
       "Q 26.375 6.390625 33.015625 6.390625 \n",
       "Q 39.65625 6.390625 43.53125 10.953125 \n",
       "Q 47.40625 15.53125 47.40625 23.390625 \n",
       "Q 47.40625 31.296875 43.53125 35.828125 \n",
       "Q 39.65625 40.375 33.015625 40.375 \n",
       "M 52.59375 71.296875 \n",
       "L 52.59375 62.3125 \n",
       "Q 48.875 64.0625 45.09375 64.984375 \n",
       "Q 41.3125 65.921875 37.59375 65.921875 \n",
       "Q 27.828125 65.921875 22.671875 59.328125 \n",
       "Q 17.53125 52.734375 16.796875 39.40625 \n",
       "Q 19.671875 43.65625 24.015625 45.921875 \n",
       "Q 28.375 48.1875 33.59375 48.1875 \n",
       "Q 44.578125 48.1875 50.953125 41.515625 \n",
       "Q 57.328125 34.859375 57.328125 23.390625 \n",
       "Q 57.328125 12.15625 50.6875 5.359375 \n",
       "Q 44.046875 -1.421875 33.015625 -1.421875 \n",
       "Q 20.359375 -1.421875 13.671875 8.265625 \n",
       "Q 6.984375 17.96875 6.984375 36.375 \n",
       "Q 6.984375 53.65625 15.1875 63.9375 \n",
       "Q 23.390625 74.21875 37.203125 74.21875 \n",
       "Q 40.921875 74.21875 44.703125 73.484375 \n",
       "Q 48.484375 72.75 52.59375 71.296875 \n",
       "\" id=\"BitstreamVeraSans-Roman-36\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(65.18828124999999 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-36\"/>\n",
       "       <use x=\"147.412109375\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path clip-path=\"url(#pb8412aa6da)\" d=\"M 100.153125 151.539062 \n",
       "L 100.153125 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"100.153125\" xlink:href=\"#m1fd6d6b6fc\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"100.153125\" xlink:href=\"#md8e7f41bf5\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- −40 -->\n",
       "      <defs>\n",
       "       <path d=\"M 37.796875 64.3125 \n",
       "L 12.890625 25.390625 \n",
       "L 37.796875 25.390625 \n",
       "z\n",
       "M 35.203125 72.90625 \n",
       "L 47.609375 72.90625 \n",
       "L 47.609375 25.390625 \n",
       "L 58.015625 25.390625 \n",
       "L 58.015625 17.1875 \n",
       "L 47.609375 17.1875 \n",
       "L 47.609375 0 \n",
       "L 37.796875 0 \n",
       "L 37.796875 17.1875 \n",
       "L 4.890625 17.1875 \n",
       "L 4.890625 26.703125 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-34\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(89.60078125 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-34\"/>\n",
       "       <use x=\"147.412109375\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <path clip-path=\"url(#pb8412aa6da)\" d=\"M 124.565625 151.539062 \n",
       "L 124.565625 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"124.56562500000001\" xlink:href=\"#m1fd6d6b6fc\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"124.56562500000001\" xlink:href=\"#md8e7f41bf5\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- −20 -->\n",
       "      <defs>\n",
       "       <path d=\"M 19.1875 8.296875 \n",
       "L 53.609375 8.296875 \n",
       "L 53.609375 0 \n",
       "L 7.328125 0 \n",
       "L 7.328125 8.296875 \n",
       "Q 12.9375 14.109375 22.625 23.890625 \n",
       "Q 32.328125 33.6875 34.8125 36.53125 \n",
       "Q 39.546875 41.84375 41.421875 45.53125 \n",
       "Q 43.3125 49.21875 43.3125 52.78125 \n",
       "Q 43.3125 58.59375 39.234375 62.25 \n",
       "Q 35.15625 65.921875 28.609375 65.921875 \n",
       "Q 23.96875 65.921875 18.8125 64.3125 \n",
       "Q 13.671875 62.703125 7.8125 59.421875 \n",
       "L 7.8125 69.390625 \n",
       "Q 13.765625 71.78125 18.9375 73 \n",
       "Q 24.125 74.21875 28.421875 74.21875 \n",
       "Q 39.75 74.21875 46.484375 68.546875 \n",
       "Q 53.21875 62.890625 53.21875 53.421875 \n",
       "Q 53.21875 48.921875 51.53125 44.890625 \n",
       "Q 49.859375 40.875 45.40625 35.40625 \n",
       "Q 44.1875 33.984375 37.640625 27.21875 \n",
       "Q 31.109375 20.453125 19.1875 8.296875 \n",
       "\" id=\"BitstreamVeraSans-Roman-32\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(114.01328125 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-32\"/>\n",
       "       <use x=\"147.412109375\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path clip-path=\"url(#pb8412aa6da)\" d=\"M 148.978125 151.539062 \n",
       "L 148.978125 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"148.978125\" xlink:href=\"#m1fd6d6b6fc\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_17\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"148.978125\" xlink:href=\"#md8e7f41bf5\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(145.796875 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_18\">\n",
       "      <path clip-path=\"url(#pb8412aa6da)\" d=\"M 173.390625 151.539062 \n",
       "L 173.390625 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_19\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"173.390625\" xlink:href=\"#m1fd6d6b6fc\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"173.390625\" xlink:href=\"#md8e7f41bf5\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(167.028125 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-32\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"line2d_21\">\n",
       "      <path clip-path=\"url(#pb8412aa6da)\" d=\"M 197.803125 151.539062 \n",
       "L 197.803125 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"197.80312500000002\" xlink:href=\"#m1fd6d6b6fc\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_23\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"197.80312500000002\" xlink:href=\"#md8e7f41bf5\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 40 -->\n",
       "      <g transform=\"translate(191.440625 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-34\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_8\">\n",
       "     <g id=\"line2d_24\">\n",
       "      <path clip-path=\"url(#pb8412aa6da)\" d=\"M 222.215625 151.539062 \n",
       "L 222.215625 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_25\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"222.21562500000002\" xlink:href=\"#m1fd6d6b6fc\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_26\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"222.21562500000002\" xlink:href=\"#md8e7f41bf5\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 60 -->\n",
       "      <g transform=\"translate(215.853125 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-36\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_9\">\n",
       "     <g id=\"line2d_27\">\n",
       "      <path clip-path=\"url(#pb8412aa6da)\" d=\"M 246.628125 151.539062 \n",
       "L 246.628125 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_28\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"246.628125\" xlink:href=\"#m1fd6d6b6fc\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_29\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"246.628125\" xlink:href=\"#md8e7f41bf5\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 80 -->\n",
       "      <g transform=\"translate(240.265625 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-38\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- x -->\n",
       "     <defs>\n",
       "      <path d=\"M 54.890625 54.6875 \n",
       "L 35.109375 28.078125 \n",
       "L 55.90625 0 \n",
       "L 45.3125 0 \n",
       "L 29.390625 21.484375 \n",
       "L 13.484375 0 \n",
       "L 2.875 0 \n",
       "L 24.125 28.609375 \n",
       "L 4.6875 54.6875 \n",
       "L 15.28125 54.6875 \n",
       "L 29.78125 35.203125 \n",
       "L 44.28125 54.6875 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-78\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(146.01875 177.815625)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#BitstreamVeraSans-Roman-78\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_30\">\n",
       "      <path clip-path=\"url(#pb8412aa6da)\" d=\"M 51.328125 151.539062 \n",
       "L 246.628125 151.539062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_31\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 4 0 \n",
       "\" id=\"m9f1ecc07d0\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"51.328125\" xlink:href=\"#m9f1ecc07d0\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_32\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -4 0 \n",
       "\" id=\"m85caf5eb8b\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"246.628125\" xlink:href=\"#m85caf5eb8b\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(40.965625 154.2984375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_33\">\n",
       "      <path clip-path=\"url(#pb8412aa6da)\" d=\"M 51.328125 134.101562 \n",
       "L 246.628125 134.101562 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_34\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"51.328125\" xlink:href=\"#m9f1ecc07d0\" y=\"134.1015625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_35\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"246.628125\" xlink:href=\"#m85caf5eb8b\" y=\"134.1015625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 500 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.796875 72.90625 \n",
       "L 49.515625 72.90625 \n",
       "L 49.515625 64.59375 \n",
       "L 19.828125 64.59375 \n",
       "L 19.828125 46.734375 \n",
       "Q 21.96875 47.46875 24.109375 47.828125 \n",
       "Q 26.265625 48.1875 28.421875 48.1875 \n",
       "Q 40.625 48.1875 47.75 41.5 \n",
       "Q 54.890625 34.8125 54.890625 23.390625 \n",
       "Q 54.890625 11.625 47.5625 5.09375 \n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \n",
       "Q 12.796875 0.140625 7.71875 1.703125 \n",
       "L 7.71875 11.625 \n",
       "Q 12.109375 9.234375 16.796875 8.0625 \n",
       "Q 21.484375 6.890625 26.703125 6.890625 \n",
       "Q 35.15625 6.890625 40.078125 11.328125 \n",
       "Q 45.015625 15.765625 45.015625 23.390625 \n",
       "Q 45.015625 31 40.078125 35.4375 \n",
       "Q 35.15625 39.890625 26.703125 39.890625 \n",
       "Q 22.75 39.890625 18.8125 39.015625 \n",
       "Q 14.890625 38.140625 10.796875 36.28125 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-35\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(28.240624999999998 136.8609375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-35\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "       <use x=\"127.24609375\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_36\">\n",
       "      <path clip-path=\"url(#pb8412aa6da)\" d=\"M 51.328125 116.664062 \n",
       "L 246.628125 116.664062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_37\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"51.328125\" xlink:href=\"#m9f1ecc07d0\" y=\"116.6640625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_38\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"246.628125\" xlink:href=\"#m85caf5eb8b\" y=\"116.6640625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 1000 -->\n",
       "      <defs>\n",
       "       <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-31\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(21.878124999999997 119.4234375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-31\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "       <use x=\"127.24609375\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "       <use x=\"190.869140625\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_39\">\n",
       "      <path clip-path=\"url(#pb8412aa6da)\" d=\"M 51.328125 99.226562 \n",
       "L 246.628125 99.226562 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_40\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"51.328125\" xlink:href=\"#m9f1ecc07d0\" y=\"99.2265625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_41\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"246.628125\" xlink:href=\"#m85caf5eb8b\" y=\"99.2265625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_14\">\n",
       "      <!-- 1500 -->\n",
       "      <g transform=\"translate(21.878124999999997 101.9859375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-31\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-35\"/>\n",
       "       <use x=\"127.24609375\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "       <use x=\"190.869140625\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_42\">\n",
       "      <path clip-path=\"url(#pb8412aa6da)\" d=\"M 51.328125 81.789062 \n",
       "L 246.628125 81.789062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_43\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"51.328125\" xlink:href=\"#m9f1ecc07d0\" y=\"81.7890625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_44\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"246.628125\" xlink:href=\"#m85caf5eb8b\" y=\"81.7890625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_15\">\n",
       "      <!-- 2000 -->\n",
       "      <g transform=\"translate(21.878124999999997 84.5484375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-32\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "       <use x=\"127.24609375\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "       <use x=\"190.869140625\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_45\">\n",
       "      <path clip-path=\"url(#pb8412aa6da)\" d=\"M 51.328125 64.351562 \n",
       "L 246.628125 64.351562 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_46\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"51.328125\" xlink:href=\"#m9f1ecc07d0\" y=\"64.35156249999999\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_47\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"246.628125\" xlink:href=\"#m85caf5eb8b\" y=\"64.35156249999999\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_16\">\n",
       "      <!-- 2500 -->\n",
       "      <g transform=\"translate(21.878124999999997 67.11093749999999)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-32\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-35\"/>\n",
       "       <use x=\"127.24609375\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "       <use x=\"190.869140625\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_7\">\n",
       "     <g id=\"line2d_48\">\n",
       "      <path clip-path=\"url(#pb8412aa6da)\" d=\"M 51.328125 46.914062 \n",
       "L 246.628125 46.914062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_49\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"51.328125\" xlink:href=\"#m9f1ecc07d0\" y=\"46.914062499999986\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_50\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"246.628125\" xlink:href=\"#m85caf5eb8b\" y=\"46.914062499999986\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_17\">\n",
       "      <!-- 3000 -->\n",
       "      <defs>\n",
       "       <path d=\"M 40.578125 39.3125 \n",
       "Q 47.65625 37.796875 51.625 33 \n",
       "Q 55.609375 28.21875 55.609375 21.1875 \n",
       "Q 55.609375 10.40625 48.1875 4.484375 \n",
       "Q 40.765625 -1.421875 27.09375 -1.421875 \n",
       "Q 22.515625 -1.421875 17.65625 -0.515625 \n",
       "Q 12.796875 0.390625 7.625 2.203125 \n",
       "L 7.625 11.71875 \n",
       "Q 11.71875 9.328125 16.59375 8.109375 \n",
       "Q 21.484375 6.890625 26.8125 6.890625 \n",
       "Q 36.078125 6.890625 40.9375 10.546875 \n",
       "Q 45.796875 14.203125 45.796875 21.1875 \n",
       "Q 45.796875 27.640625 41.28125 31.265625 \n",
       "Q 36.765625 34.90625 28.71875 34.90625 \n",
       "L 20.21875 34.90625 \n",
       "L 20.21875 43.015625 \n",
       "L 29.109375 43.015625 \n",
       "Q 36.375 43.015625 40.234375 45.921875 \n",
       "Q 44.09375 48.828125 44.09375 54.296875 \n",
       "Q 44.09375 59.90625 40.109375 62.90625 \n",
       "Q 36.140625 65.921875 28.71875 65.921875 \n",
       "Q 24.65625 65.921875 20.015625 65.03125 \n",
       "Q 15.375 64.15625 9.8125 62.3125 \n",
       "L 9.8125 71.09375 \n",
       "Q 15.4375 72.65625 20.34375 73.4375 \n",
       "Q 25.25 74.21875 29.59375 74.21875 \n",
       "Q 40.828125 74.21875 47.359375 69.109375 \n",
       "Q 53.90625 64.015625 53.90625 55.328125 \n",
       "Q 53.90625 49.265625 50.4375 45.09375 \n",
       "Q 46.96875 40.921875 40.578125 39.3125 \n",
       "\" id=\"BitstreamVeraSans-Roman-33\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(21.878124999999997 49.67343749999998)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-33\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "       <use x=\"127.24609375\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "       <use x=\"190.869140625\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_8\">\n",
       "     <g id=\"line2d_51\">\n",
       "      <path clip-path=\"url(#pb8412aa6da)\" d=\"M 51.328125 29.476562 \n",
       "L 246.628125 29.476562 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_52\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"51.328125\" xlink:href=\"#m9f1ecc07d0\" y=\"29.476562499999986\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_53\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"246.628125\" xlink:href=\"#m85caf5eb8b\" y=\"29.476562499999986\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_18\">\n",
       "      <!-- 3500 -->\n",
       "      <g transform=\"translate(21.878124999999997 32.23593749999998)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-33\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-35\"/>\n",
       "       <use x=\"127.24609375\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "       <use x=\"190.869140625\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_9\">\n",
       "     <g id=\"line2d_54\">\n",
       "      <path clip-path=\"url(#pb8412aa6da)\" d=\"M 51.328125 12.039062 \n",
       "L 246.628125 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_55\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"51.328125\" xlink:href=\"#m9f1ecc07d0\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_56\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"246.628125\" xlink:href=\"#m85caf5eb8b\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_19\">\n",
       "      <!-- 4000 -->\n",
       "      <g transform=\"translate(21.878124999999997 14.798437500000006)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-34\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "       <use x=\"127.24609375\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "       <use x=\"190.869140625\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_20\">\n",
       "     <!-- f(x) -->\n",
       "     <defs>\n",
       "      <path d=\"M 37.109375 75.984375 \n",
       "L 37.109375 68.5 \n",
       "L 28.515625 68.5 \n",
       "Q 23.6875 68.5 21.796875 66.546875 \n",
       "Q 19.921875 64.59375 19.921875 59.515625 \n",
       "L 19.921875 54.6875 \n",
       "L 34.71875 54.6875 \n",
       "L 34.71875 47.703125 \n",
       "L 19.921875 47.703125 \n",
       "L 19.921875 0 \n",
       "L 10.890625 0 \n",
       "L 10.890625 47.703125 \n",
       "L 2.296875 47.703125 \n",
       "L 2.296875 54.6875 \n",
       "L 10.890625 54.6875 \n",
       "L 10.890625 58.5 \n",
       "Q 10.890625 67.625 15.140625 71.796875 \n",
       "Q 19.390625 75.984375 28.609375 75.984375 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-66\"/>\n",
       "      <path d=\"M 31 75.875 \n",
       "Q 24.46875 64.65625 21.28125 53.65625 \n",
       "Q 18.109375 42.671875 18.109375 31.390625 \n",
       "Q 18.109375 20.125 21.3125 9.0625 \n",
       "Q 24.515625 -2 31 -13.1875 \n",
       "L 23.1875 -13.1875 \n",
       "Q 15.875 -1.703125 12.234375 9.375 \n",
       "Q 8.59375 20.453125 8.59375 31.390625 \n",
       "Q 8.59375 42.28125 12.203125 53.3125 \n",
       "Q 15.828125 64.359375 23.1875 75.875 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-28\"/>\n",
       "      <path d=\"M 8.015625 75.875 \n",
       "L 15.828125 75.875 \n",
       "Q 23.140625 64.359375 26.78125 53.3125 \n",
       "Q 30.421875 42.28125 30.421875 31.390625 \n",
       "Q 30.421875 20.453125 26.78125 9.375 \n",
       "Q 23.140625 -1.703125 15.828125 -13.1875 \n",
       "L 8.015625 -13.1875 \n",
       "Q 14.5 -2 17.703125 9.0625 \n",
       "Q 20.90625 20.125 20.90625 31.390625 \n",
       "Q 20.90625 42.671875 17.703125 53.65625 \n",
       "Q 14.5 64.65625 8.015625 75.875 \n",
       "\" id=\"BitstreamVeraSans-Roman-29\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(14.798437499999997 90.41015625)rotate(-90.0)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#BitstreamVeraSans-Roman-66\"/>\n",
       "      <use x=\"35.205078125\" xlink:href=\"#BitstreamVeraSans-Roman-28\"/>\n",
       "      <use x=\"74.21875\" xlink:href=\"#BitstreamVeraSans-Roman-78\"/>\n",
       "      <use x=\"133.3984375\" xlink:href=\"#BitstreamVeraSans-Roman-29\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pb8412aa6da\">\n",
       "   <rect height=\"139.5\" width=\"195.3\" x=\"51.328125\" y=\"12.0390625\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f234ef43c18>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_trace(gd(1.1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Local Minima"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To illustrate what happens for nonconvex functions consider the case of $f(x) = x \\cdot \\cos c x$. This function has infinitely many local minima. Depending on our choice of learning rate and depending on how well conditioned the problem is, we may end up with one of many solutions. The example below illustrates how an (unrealistically) high learning rate will lead to a poor local minimum."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 10, x: -1.528165927635083\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\n",
       "<svg height=\"187pt\" version=\"1.1\" viewBox=\"0 0 249 187\" width=\"249pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;stroke-miterlimit:100000;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 187.095313 \n",
       "L 249.482813 187.095313 \n",
       "L 249.482813 0 \n",
       "L 0 0 \n",
       "L 0 187.095313 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 40.620313 151.539062 \n",
       "L 235.920313 151.539062 \n",
       "L 235.920313 12.039062 \n",
       "L 40.620313 12.039062 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_1\">\n",
       "    <path clip-path=\"url(#p11e68f3dfb)\" d=\"M 40.620313 81.789062 \n",
       "L 43.745112 69.110398 \n",
       "L 46.479312 58.923998 \n",
       "L 48.920562 50.681972 \n",
       "L 51.068862 44.173348 \n",
       "L 53.119512 38.661022 \n",
       "L 54.974862 34.293289 \n",
       "L 56.732562 30.716479 \n",
       "L 58.392612 27.849479 \n",
       "L 59.857362 25.735929 \n",
       "L 61.322112 24.013016 \n",
       "L 62.689212 22.755575 \n",
       "L 63.958662 21.887863 \n",
       "L 65.130462 21.339458 \n",
       "L 66.302262 21.029048 \n",
       "L 67.474062 20.951365 \n",
       "L 68.645862 21.100276 \n",
       "L 69.817662 21.46882 \n",
       "L 71.087112 22.106926 \n",
       "L 72.454212 23.059773 \n",
       "L 73.918962 24.369994 \n",
       "L 75.481362 26.07579 \n",
       "L 77.239062 28.343927 \n",
       "L 79.192062 31.252214 \n",
       "L 81.340362 34.85859 \n",
       "L 83.879262 39.567587 \n",
       "L 87.101712 46.04456 \n",
       "L 93.741912 60.065789 \n",
       "L 97.843212 68.396008 \n",
       "L 100.772712 73.871076 \n",
       "L 103.311612 78.169047 \n",
       "L 105.557562 81.559667 \n",
       "L 107.608212 84.279816 \n",
       "L 109.561212 86.513105 \n",
       "L 111.416562 88.297271 \n",
       "L 113.174262 89.676505 \n",
       "L 114.834312 90.698494 \n",
       "L 116.396712 91.412027 \n",
       "L 117.959112 91.888002 \n",
       "L 119.521512 92.132032 \n",
       "L 121.083912 92.152044 \n",
       "L 122.646312 91.958175 \n",
       "L 124.306362 91.531555 \n",
       "L 126.064062 90.850782 \n",
       "L 128.017062 89.845767 \n",
       "L 130.165362 88.479116 \n",
       "L 132.704262 86.580546 \n",
       "L 135.926712 83.868194 \n",
       "L 144.715212 76.310767 \n",
       "L 147.254112 74.509903 \n",
       "L 149.402412 73.249762 \n",
       "L 151.355412 72.358836 \n",
       "L 153.113112 71.793346 \n",
       "L 154.773162 71.485255 \n",
       "L 156.335562 71.41032 \n",
       "L 157.897962 71.555235 \n",
       "L 159.460362 71.928905 \n",
       "L 161.022762 72.537967 \n",
       "L 162.585162 73.3867 \n",
       "L 164.245212 74.553121 \n",
       "L 165.905262 75.991337 \n",
       "L 167.662962 77.805876 \n",
       "L 169.518312 80.037207 \n",
       "L 171.568962 82.862958 \n",
       "L 173.814912 86.359425 \n",
       "L 176.256162 90.584305 \n",
       "L 179.088012 95.947001 \n",
       "L 182.701062 103.307888 \n",
       "L 193.247262 125.133946 \n",
       "L 195.883812 129.908888 \n",
       "L 198.129762 133.534956 \n",
       "L 200.082762 136.285645 \n",
       "L 201.840462 138.393513 \n",
       "L 203.402862 139.943053 \n",
       "L 204.867612 141.095779 \n",
       "L 206.234712 141.892738 \n",
       "L 207.504162 142.379038 \n",
       "L 208.675962 142.601973 \n",
       "L 209.847762 142.600545 \n",
       "L 211.019562 142.368339 \n",
       "L 212.191362 141.899796 \n",
       "L 213.363162 141.190248 \n",
       "L 214.632612 140.145287 \n",
       "L 215.902062 138.809619 \n",
       "L 217.269162 137.043688 \n",
       "L 218.733912 134.773849 \n",
       "L 220.296312 131.923581 \n",
       "L 221.956362 128.415342 \n",
       "L 223.714062 124.172995 \n",
       "L 225.667062 118.843442 \n",
       "L 227.717712 112.583246 \n",
       "L 229.963662 105.000028 \n",
       "L 232.502562 95.603077 \n",
       "L 235.334412 84.239114 \n",
       "L 235.822662 82.199512 \n",
       "L 235.822662 82.199512 \n",
       "\" style=\"fill:none;stroke:#0000ff;stroke-linecap:square;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_2\">\n",
       "    <path clip-path=\"url(#p11e68f3dfb)\" d=\"M 235.920313 81.789063 \n",
       "L 143.887356 76.956972 \n",
       "L 126.487982 90.653798 \n",
       "L 116.010037 91.257724 \n",
       "L 125.151854 91.232186 \n",
       "L 116.72121 91.530278 \n",
       "L 124.34748 91.518241 \n",
       "L 117.230311 91.695236 \n",
       "L 123.780264 91.690406 \n",
       "L 117.62365 91.805587 \n",
       "L 123.347772 91.804741 \n",
       "\" style=\"fill:none;stroke:#008000;stroke-linecap:square;\"/>\n",
       "    <defs>\n",
       "     <path d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" id=\"m2716788d3f\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#p11e68f3dfb)\">\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"235.92031250000002\" xlink:href=\"#m2716788d3f\" y=\"81.78906250000001\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"143.88735571308703\" xlink:href=\"#m2716788d3f\" y=\"76.95697226159365\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"126.4879821118244\" xlink:href=\"#m2716788d3f\" y=\"90.65379764470188\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"116.01003671805991\" xlink:href=\"#m2716788d3f\" y=\"91.25772352579611\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"125.15185427236358\" xlink:href=\"#m2716788d3f\" y=\"91.23218640774472\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"116.7212097712562\" xlink:href=\"#m2716788d3f\" y=\"91.53027817978236\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"124.34748023060575\" xlink:href=\"#m2716788d3f\" y=\"91.51824125227972\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"117.23031107593881\" xlink:href=\"#m2716788d3f\" y=\"91.69523646426423\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"123.78026436487431\" xlink:href=\"#m2716788d3f\" y=\"91.69040570441636\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"117.62364954467836\" xlink:href=\"#m2716788d3f\" y=\"91.80558677837348\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"123.34777221664343\" xlink:href=\"#m2716788d3f\" y=\"91.80474101851385\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 235.920313 151.539062 \n",
       "L 235.920313 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 40.620313 12.039062 \n",
       "L 235.920313 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 40.620313 151.539062 \n",
       "L 40.620313 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 40.620313 151.539062 \n",
       "L 235.920313 151.539062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path clip-path=\"url(#p11e68f3dfb)\" d=\"M 40.620313 151.539062 \n",
       "L 40.620313 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 -4 \n",
       "\" id=\"m2a28af0777\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"40.62031250000001\" xlink:href=\"#m2a28af0777\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_5\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 4 \n",
       "\" id=\"m15aa11808e\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"40.62031250000001\" xlink:href=\"#m15aa11808e\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −10 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "\" id=\"BitstreamVeraSans-Roman-30\"/>\n",
       "       <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-31\"/>\n",
       "       <path d=\"M 10.59375 35.5 \n",
       "L 73.1875 35.5 \n",
       "L 73.1875 27.203125 \n",
       "L 10.59375 27.203125 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-2212\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(30.067968750000013 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-31\"/>\n",
       "       <use x=\"147.412109375\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <path clip-path=\"url(#p11e68f3dfb)\" d=\"M 89.445313 151.539062 \n",
       "L 89.445313 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"89.44531250000001\" xlink:href=\"#m2a28af0777\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"89.44531250000001\" xlink:href=\"#m15aa11808e\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −5 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.796875 72.90625 \n",
       "L 49.515625 72.90625 \n",
       "L 49.515625 64.59375 \n",
       "L 19.828125 64.59375 \n",
       "L 19.828125 46.734375 \n",
       "Q 21.96875 47.46875 24.109375 47.828125 \n",
       "Q 26.265625 48.1875 28.421875 48.1875 \n",
       "Q 40.625 48.1875 47.75 41.5 \n",
       "Q 54.890625 34.8125 54.890625 23.390625 \n",
       "Q 54.890625 11.625 47.5625 5.09375 \n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \n",
       "Q 12.796875 0.140625 7.71875 1.703125 \n",
       "L 7.71875 11.625 \n",
       "Q 12.109375 9.234375 16.796875 8.0625 \n",
       "Q 21.484375 6.890625 26.703125 6.890625 \n",
       "Q 35.15625 6.890625 40.078125 11.328125 \n",
       "Q 45.015625 15.765625 45.015625 23.390625 \n",
       "Q 45.015625 31 40.078125 35.4375 \n",
       "Q 35.15625 39.890625 26.703125 39.890625 \n",
       "Q 22.75 39.890625 18.8125 39.015625 \n",
       "Q 14.890625 38.140625 10.796875 36.28125 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-35\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(82.07421875000001 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path clip-path=\"url(#p11e68f3dfb)\" d=\"M 138.270313 151.539062 \n",
       "L 138.270313 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"138.27031250000002\" xlink:href=\"#m2a28af0777\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"138.27031250000002\" xlink:href=\"#m15aa11808e\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(135.0890625 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <path clip-path=\"url(#p11e68f3dfb)\" d=\"M 187.095313 151.539062 \n",
       "L 187.095313 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"187.09531250000003\" xlink:href=\"#m2a28af0777\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"187.09531250000003\" xlink:href=\"#m15aa11808e\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(183.91406250000003 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path clip-path=\"url(#p11e68f3dfb)\" d=\"M 235.920313 151.539062 \n",
       "L 235.920313 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"235.92031250000002\" xlink:href=\"#m2a28af0777\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_17\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"235.92031250000002\" xlink:href=\"#m15aa11808e\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(229.5578125 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-31\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- x -->\n",
       "     <defs>\n",
       "      <path d=\"M 54.890625 54.6875 \n",
       "L 35.109375 28.078125 \n",
       "L 55.90625 0 \n",
       "L 45.3125 0 \n",
       "L 29.390625 21.484375 \n",
       "L 13.484375 0 \n",
       "L 2.875 0 \n",
       "L 24.125 28.609375 \n",
       "L 4.6875 54.6875 \n",
       "L 15.28125 54.6875 \n",
       "L 29.78125 35.203125 \n",
       "L 44.28125 54.6875 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-78\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(135.31093750000002 177.815625)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#BitstreamVeraSans-Roman-78\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_18\">\n",
       "      <path clip-path=\"url(#p11e68f3dfb)\" d=\"M 40.620313 151.539062 \n",
       "L 235.920313 151.539062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_19\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 4 0 \n",
       "\" id=\"mc550008ac1\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"40.620312500000004\" xlink:href=\"#mc550008ac1\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -4 0 \n",
       "\" id=\"m368833af5b\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"235.92031250000002\" xlink:href=\"#m368833af5b\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- −8 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 34.625 \n",
       "Q 24.75 34.625 20.71875 30.859375 \n",
       "Q 16.703125 27.09375 16.703125 20.515625 \n",
       "Q 16.703125 13.921875 20.71875 10.15625 \n",
       "Q 24.75 6.390625 31.78125 6.390625 \n",
       "Q 38.8125 6.390625 42.859375 10.171875 \n",
       "Q 46.921875 13.96875 46.921875 20.515625 \n",
       "Q 46.921875 27.09375 42.890625 30.859375 \n",
       "Q 38.875 34.625 31.78125 34.625 \n",
       "M 21.921875 38.8125 \n",
       "Q 15.578125 40.375 12.03125 44.71875 \n",
       "Q 8.5 49.078125 8.5 55.328125 \n",
       "Q 8.5 64.0625 14.71875 69.140625 \n",
       "Q 20.953125 74.21875 31.78125 74.21875 \n",
       "Q 42.671875 74.21875 48.875 69.140625 \n",
       "Q 55.078125 64.0625 55.078125 55.328125 \n",
       "Q 55.078125 49.078125 51.53125 44.71875 \n",
       "Q 48 40.375 41.703125 38.8125 \n",
       "Q 48.828125 37.15625 52.796875 32.3125 \n",
       "Q 56.78125 27.484375 56.78125 20.515625 \n",
       "Q 56.78125 9.90625 50.3125 4.234375 \n",
       "Q 43.84375 -1.421875 31.78125 -1.421875 \n",
       "Q 19.734375 -1.421875 13.25 4.234375 \n",
       "Q 6.78125 9.90625 6.78125 20.515625 \n",
       "Q 6.78125 27.484375 10.78125 32.3125 \n",
       "Q 14.796875 37.15625 21.921875 38.8125 \n",
       "M 18.3125 54.390625 \n",
       "Q 18.3125 48.734375 21.84375 45.5625 \n",
       "Q 25.390625 42.390625 31.78125 42.390625 \n",
       "Q 38.140625 42.390625 41.71875 45.5625 \n",
       "Q 45.3125 48.734375 45.3125 54.390625 \n",
       "Q 45.3125 60.0625 41.71875 63.234375 \n",
       "Q 38.140625 66.40625 31.78125 66.40625 \n",
       "Q 25.390625 66.40625 21.84375 63.234375 \n",
       "Q 18.3125 60.0625 18.3125 54.390625 \n",
       "\" id=\"BitstreamVeraSans-Roman-38\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(21.878125000000004 154.2984375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-38\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_21\">\n",
       "      <path clip-path=\"url(#p11e68f3dfb)\" d=\"M 40.620313 134.101562 \n",
       "L 235.920313 134.101562 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"40.620312500000004\" xlink:href=\"#mc550008ac1\" y=\"134.1015625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_23\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"235.92031250000002\" xlink:href=\"#m368833af5b\" y=\"134.1015625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- −6 -->\n",
       "      <defs>\n",
       "       <path d=\"M 33.015625 40.375 \n",
       "Q 26.375 40.375 22.484375 35.828125 \n",
       "Q 18.609375 31.296875 18.609375 23.390625 \n",
       "Q 18.609375 15.53125 22.484375 10.953125 \n",
       "Q 26.375 6.390625 33.015625 6.390625 \n",
       "Q 39.65625 6.390625 43.53125 10.953125 \n",
       "Q 47.40625 15.53125 47.40625 23.390625 \n",
       "Q 47.40625 31.296875 43.53125 35.828125 \n",
       "Q 39.65625 40.375 33.015625 40.375 \n",
       "M 52.59375 71.296875 \n",
       "L 52.59375 62.3125 \n",
       "Q 48.875 64.0625 45.09375 64.984375 \n",
       "Q 41.3125 65.921875 37.59375 65.921875 \n",
       "Q 27.828125 65.921875 22.671875 59.328125 \n",
       "Q 17.53125 52.734375 16.796875 39.40625 \n",
       "Q 19.671875 43.65625 24.015625 45.921875 \n",
       "Q 28.375 48.1875 33.59375 48.1875 \n",
       "Q 44.578125 48.1875 50.953125 41.515625 \n",
       "Q 57.328125 34.859375 57.328125 23.390625 \n",
       "Q 57.328125 12.15625 50.6875 5.359375 \n",
       "Q 44.046875 -1.421875 33.015625 -1.421875 \n",
       "Q 20.359375 -1.421875 13.671875 8.265625 \n",
       "Q 6.984375 17.96875 6.984375 36.375 \n",
       "Q 6.984375 53.65625 15.1875 63.9375 \n",
       "Q 23.390625 74.21875 37.203125 74.21875 \n",
       "Q 40.921875 74.21875 44.703125 73.484375 \n",
       "Q 48.484375 72.75 52.59375 71.296875 \n",
       "\" id=\"BitstreamVeraSans-Roman-36\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(21.878125000000004 136.8609375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-36\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_24\">\n",
       "      <path clip-path=\"url(#p11e68f3dfb)\" d=\"M 40.620313 116.664062 \n",
       "L 235.920313 116.664062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_25\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"40.620312500000004\" xlink:href=\"#mc550008ac1\" y=\"116.6640625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_26\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"235.92031250000002\" xlink:href=\"#m368833af5b\" y=\"116.6640625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- −4 -->\n",
       "      <defs>\n",
       "       <path d=\"M 37.796875 64.3125 \n",
       "L 12.890625 25.390625 \n",
       "L 37.796875 25.390625 \n",
       "z\n",
       "M 35.203125 72.90625 \n",
       "L 47.609375 72.90625 \n",
       "L 47.609375 25.390625 \n",
       "L 58.015625 25.390625 \n",
       "L 58.015625 17.1875 \n",
       "L 47.609375 17.1875 \n",
       "L 47.609375 0 \n",
       "L 37.796875 0 \n",
       "L 37.796875 17.1875 \n",
       "L 4.890625 17.1875 \n",
       "L 4.890625 26.703125 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-34\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(21.878125000000004 119.4234375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-34\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_27\">\n",
       "      <path clip-path=\"url(#p11e68f3dfb)\" d=\"M 40.620313 99.226562 \n",
       "L 235.920313 99.226562 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_28\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"40.620312500000004\" xlink:href=\"#mc550008ac1\" y=\"99.2265625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_29\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"235.92031250000002\" xlink:href=\"#m368833af5b\" y=\"99.2265625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- −2 -->\n",
       "      <defs>\n",
       "       <path d=\"M 19.1875 8.296875 \n",
       "L 53.609375 8.296875 \n",
       "L 53.609375 0 \n",
       "L 7.328125 0 \n",
       "L 7.328125 8.296875 \n",
       "Q 12.9375 14.109375 22.625 23.890625 \n",
       "Q 32.328125 33.6875 34.8125 36.53125 \n",
       "Q 39.546875 41.84375 41.421875 45.53125 \n",
       "Q 43.3125 49.21875 43.3125 52.78125 \n",
       "Q 43.3125 58.59375 39.234375 62.25 \n",
       "Q 35.15625 65.921875 28.609375 65.921875 \n",
       "Q 23.96875 65.921875 18.8125 64.3125 \n",
       "Q 13.671875 62.703125 7.8125 59.421875 \n",
       "L 7.8125 69.390625 \n",
       "Q 13.765625 71.78125 18.9375 73 \n",
       "Q 24.125 74.21875 28.421875 74.21875 \n",
       "Q 39.75 74.21875 46.484375 68.546875 \n",
       "Q 53.21875 62.890625 53.21875 53.421875 \n",
       "Q 53.21875 48.921875 51.53125 44.890625 \n",
       "Q 49.859375 40.875 45.40625 35.40625 \n",
       "Q 44.1875 33.984375 37.640625 27.21875 \n",
       "Q 31.109375 20.453125 19.1875 8.296875 \n",
       "\" id=\"BitstreamVeraSans-Roman-32\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(21.878125000000004 101.9859375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_30\">\n",
       "      <path clip-path=\"url(#p11e68f3dfb)\" d=\"M 40.620313 81.789062 \n",
       "L 235.920313 81.789062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_31\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"40.620312500000004\" xlink:href=\"#mc550008ac1\" y=\"81.7890625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_32\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"235.92031250000002\" xlink:href=\"#m368833af5b\" y=\"81.7890625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(30.257812500000004 84.5484375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_33\">\n",
       "      <path clip-path=\"url(#p11e68f3dfb)\" d=\"M 40.620313 64.351562 \n",
       "L 235.920313 64.351562 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_34\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"40.620312500000004\" xlink:href=\"#mc550008ac1\" y=\"64.3515625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_35\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"235.92031250000002\" xlink:href=\"#m368833af5b\" y=\"64.3515625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(30.257812500000004 67.1109375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_7\">\n",
       "     <g id=\"line2d_36\">\n",
       "      <path clip-path=\"url(#p11e68f3dfb)\" d=\"M 40.620313 46.914062 \n",
       "L 235.920313 46.914062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_37\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"40.620312500000004\" xlink:href=\"#mc550008ac1\" y=\"46.9140625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_38\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"235.92031250000002\" xlink:href=\"#m368833af5b\" y=\"46.9140625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 4 -->\n",
       "      <g transform=\"translate(30.257812500000004 49.673437500000006)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-34\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_8\">\n",
       "     <g id=\"line2d_39\">\n",
       "      <path clip-path=\"url(#p11e68f3dfb)\" d=\"M 40.620313 29.476562 \n",
       "L 235.920313 29.476562 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_40\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"40.620312500000004\" xlink:href=\"#mc550008ac1\" y=\"29.4765625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_41\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"235.92031250000002\" xlink:href=\"#m368833af5b\" y=\"29.4765625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_14\">\n",
       "      <!-- 6 -->\n",
       "      <g transform=\"translate(30.257812500000004 32.235937500000006)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-36\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_9\">\n",
       "     <g id=\"line2d_42\">\n",
       "      <path clip-path=\"url(#p11e68f3dfb)\" d=\"M 40.620313 12.039062 \n",
       "L 235.920313 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_43\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"40.620312500000004\" xlink:href=\"#mc550008ac1\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_44\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"235.92031250000002\" xlink:href=\"#m368833af5b\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_15\">\n",
       "      <!-- 8 -->\n",
       "      <g transform=\"translate(30.257812500000004 14.798437500000006)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-38\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_16\">\n",
       "     <!-- f(x) -->\n",
       "     <defs>\n",
       "      <path d=\"M 37.109375 75.984375 \n",
       "L 37.109375 68.5 \n",
       "L 28.515625 68.5 \n",
       "Q 23.6875 68.5 21.796875 66.546875 \n",
       "Q 19.921875 64.59375 19.921875 59.515625 \n",
       "L 19.921875 54.6875 \n",
       "L 34.71875 54.6875 \n",
       "L 34.71875 47.703125 \n",
       "L 19.921875 47.703125 \n",
       "L 19.921875 0 \n",
       "L 10.890625 0 \n",
       "L 10.890625 47.703125 \n",
       "L 2.296875 47.703125 \n",
       "L 2.296875 54.6875 \n",
       "L 10.890625 54.6875 \n",
       "L 10.890625 58.5 \n",
       "Q 10.890625 67.625 15.140625 71.796875 \n",
       "Q 19.390625 75.984375 28.609375 75.984375 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-66\"/>\n",
       "      <path d=\"M 31 75.875 \n",
       "Q 24.46875 64.65625 21.28125 53.65625 \n",
       "Q 18.109375 42.671875 18.109375 31.390625 \n",
       "Q 18.109375 20.125 21.3125 9.0625 \n",
       "Q 24.515625 -2 31 -13.1875 \n",
       "L 23.1875 -13.1875 \n",
       "Q 15.875 -1.703125 12.234375 9.375 \n",
       "Q 8.59375 20.453125 8.59375 31.390625 \n",
       "Q 8.59375 42.28125 12.203125 53.3125 \n",
       "Q 15.828125 64.359375 23.1875 75.875 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-28\"/>\n",
       "      <path d=\"M 8.015625 75.875 \n",
       "L 15.828125 75.875 \n",
       "Q 23.140625 64.359375 26.78125 53.3125 \n",
       "Q 30.421875 42.28125 30.421875 31.390625 \n",
       "Q 30.421875 20.453125 26.78125 9.375 \n",
       "Q 23.140625 -1.703125 15.828125 -13.1875 \n",
       "L 8.015625 -13.1875 \n",
       "Q 14.5 -2 17.703125 9.0625 \n",
       "Q 20.90625 20.125 20.90625 31.390625 \n",
       "Q 20.90625 42.671875 17.703125 53.65625 \n",
       "Q 14.5 64.65625 8.015625 75.875 \n",
       "\" id=\"BitstreamVeraSans-Roman-29\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(14.798437500000004 90.41015625)rotate(-90.0)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#BitstreamVeraSans-Roman-66\"/>\n",
       "      <use x=\"35.205078125\" xlink:href=\"#BitstreamVeraSans-Roman-28\"/>\n",
       "      <use x=\"74.21875\" xlink:href=\"#BitstreamVeraSans-Roman-78\"/>\n",
       "      <use x=\"133.3984375\" xlink:href=\"#BitstreamVeraSans-Roman-29\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p11e68f3dfb\">\n",
       "   <rect height=\"139.5\" width=\"195.3\" x=\"40.620312500000004\" y=\"12.0390625\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f234cda9ac8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "c = 0.15 * math.pi\n",
    "def f(x):     return x*math.cos(c * x) \n",
    "def gradf(x): return math.cos(c * x) - c * x * math.sin(c * x)\n",
    "\n",
    "show_trace(gd(2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multivariate Gradient Descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that have a better intuition of the univariate case, let us consider the situation where $\\mathbf{x} \\in \\mathbb{R}^d$. That is, the objective function $f: \\mathbb{R}^d \\to \\mathbb{R}$ maps vectors into scalars. Correspondingly its gradient is multivariate, too. It is a vector consisting of $d$ partial derivatives:\n",
    "\n",
    "$$\\nabla f(\\mathbf{x}) = \\bigg[\\frac{\\partial f(\\mathbf{x})}{\\partial x_1}, \\frac{\\partial f(\\mathbf{x})}{\\partial x_2}, \\ldots, \\frac{\\partial f(\\mathbf{x})}{\\partial x_d}\\bigg]^\\top.$$\n",
    "\n",
    "Each partial derivative element $\\partial f(\\mathbf{x})/\\partial x_i$ in the gradient indicates the rate of change of $f$ at $\\mathbf{x}$ with respect to the input $x_i$. As before in the univariate case we can use the corresponding Taylor approximation for multivariate functions to get some idea of what we should do. In particular, we have that\n",
    "\n",
    "$$f(\\mathbf{x} + \\mathbf{\\epsilon}) = f(\\mathbf{x}) + \\mathbf{\\epsilon}^\\top \\nabla f(\\mathbf{x}) + O(|\\mathbf{\\epsilon}|^2).$$ :eqlabel:gd-multi-taylor\n",
    "\n",
    "In other words, up to second order terms in $\\mathbf{epsilon}$ the direction of steepest descent is given by the negative gradient $-\\nabla f(\\mathbf{x})$. Choosing a suitable learning rate $\\eta > 0$ yields the prototypical gradient descent algorithm:\n",
    "\n",
    "$\\mathbf{x} \\leftarrow \\mathbf{x} - \\eta \\nabla f(\\mathbf{x}).$\n",
    "\n",
    "To see how the algorithm behaves in practice let's construct an objective function $f(\\mathbf{x})=x_1^2+2x_2^2$ with a two-dimensional vector $\\mathbf{x} = [x_1, x_2]^\\top$ as input and a scalar as output. The gradient is given by $\\nabla f(\\mathbf{x}) = [2x_1, 4x_2]^\\top$. We will observe the trajectory of $\\mathbf{x}$ by gradient descent from the initial position $[-5,-2]$. We need two more helper functions. The first uses an update function and applies it $20$ times to the initial value. The second helper visualizes the trajectory of $\\mathbf{x}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we observe the trajectory of the optimization variable $\\mathbf{x}$ for learning rate $\\eta = 0.1$. We can see that after 20 steps the value of $\\mathbf{x}$ approaches its minimum at $[0, 0]$. Progress is fairly well-behaved albeit rather slow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 20, x1 -0.057646, x2 -0.000073\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\n",
       "<svg height=\"182pt\" version=\"1.1\" viewBox=\"0 0 252 182\" width=\"252pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;stroke-miterlimit:100000;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 182.25625 \n",
       "L 252.660938 182.25625 \n",
       "L 252.660938 -0 \n",
       "L 0 -0 \n",
       "L 0 182.25625 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 50.160938 146.7 \n",
       "L 245.460938 146.7 \n",
       "L 245.460938 7.2 \n",
       "L 50.160938 7.2 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_1\">\n",
       "    <path clip-path=\"url(#p3cde0fc7ae)\" d=\"M 245.460938 107.208836 \n",
       "L 244.922426 107.353846 \n",
       "L 242.409375 107.995858 \n",
       "L 239.357813 108.683728 \n",
       "L 236.30625 109.279882 \n",
       "L 233.254688 109.78432 \n",
       "L 230.203125 110.197041 \n",
       "L 227.151563 110.518047 \n",
       "L 224.1 110.747337 \n",
       "L 221.048438 110.884911 \n",
       "L 217.996875 110.930769 \n",
       "L 217.996875 110.930769 \n",
       "L 217.996875 110.930769 \n",
       "L 214.945313 110.884911 \n",
       "L 211.89375 110.747337 \n",
       "L 208.842188 110.518047 \n",
       "L 205.790625 110.197041 \n",
       "L 202.739063 109.78432 \n",
       "L 199.6875 109.279882 \n",
       "L 196.635938 108.683728 \n",
       "L 193.584375 107.995858 \n",
       "L 191.071324 107.353846 \n",
       "L 190.532813 107.208836 \n",
       "L 187.48125 106.290437 \n",
       "L 184.429688 105.275364 \n",
       "L 181.378125 104.163617 \n",
       "L 180.401625 103.776923 \n",
       "L 178.326563 102.908242 \n",
       "L 175.275 101.528571 \n",
       "L 172.539116 100.2 \n",
       "L 172.223438 100.037413 \n",
       "L 169.171875 98.357343 \n",
       "L 166.212784 96.623077 \n",
       "L 166.120313 96.565385 \n",
       "L 163.06875 94.546154 \n",
       "L 160.924409 93.046154 \n",
       "L 160.017188 92.367772 \n",
       "L 156.965625 89.962599 \n",
       "L 156.370198 89.469231 \n",
       "L 153.914062 87.283333 \n",
       "L 152.423765 85.892308 \n",
       "L 150.8625 84.318462 \n",
       "L 148.96375 82.315385 \n",
       "L 147.810938 80.993478 \n",
       "L 145.928059 78.738462 \n",
       "L 144.759375 77.205495 \n",
       "L 143.264732 75.161538 \n",
       "L 141.707813 72.8083 \n",
       "L 140.929963 71.584615 \n",
       "L 138.895588 68.007692 \n",
       "L 138.65625 67.530769 \n",
       "L 137.159257 64.430769 \n",
       "L 135.662264 60.853846 \n",
       "L 135.604688 60.691259 \n",
       "L 134.439545 57.276923 \n",
       "L 133.440852 53.7 \n",
       "L 132.664091 50.123077 \n",
       "L 132.553125 49.407692 \n",
       "L 132.124836 46.546154 \n",
       "L 131.803618 42.969231 \n",
       "L 131.696546 39.392308 \n",
       "L 131.803618 35.815385 \n",
       "L 132.124836 32.238462 \n",
       "L 132.553125 29.376923 \n",
       "L 132.664091 28.661538 \n",
       "L 133.440852 25.084615 \n",
       "L 134.439545 21.507692 \n",
       "L 135.604688 18.093357 \n",
       "L 135.662264 17.930769 \n",
       "L 137.159257 14.353846 \n",
       "L 138.65625 11.253846 \n",
       "L 138.895588 10.776923 \n",
       "L 140.929963 7.2 \n",
       "\" style=\"fill:none;stroke:#1f77b4;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_2\">\n",
       "    <path clip-path=\"url(#p3cde0fc7ae)\" d=\"M 245.460938 137.952797 \n",
       "L 242.409375 138.505594 \n",
       "L 239.357813 138.993357 \n",
       "L 236.30625 139.416084 \n",
       "L 235.196591 139.546154 \n",
       "L 233.254688 139.765789 \n",
       "L 230.203125 140.048178 \n",
       "L 227.151563 140.267814 \n",
       "L 224.1 140.424696 \n",
       "L 221.048438 140.518826 \n",
       "L 217.996875 140.550202 \n",
       "L 214.945313 140.518826 \n",
       "L 211.89375 140.424696 \n",
       "L 208.842188 140.267814 \n",
       "L 205.790625 140.048178 \n",
       "L 202.739063 139.765789 \n",
       "L 200.797159 139.546154 \n",
       "L 199.6875 139.416084 \n",
       "L 196.635938 138.993357 \n",
       "L 193.584375 138.505594 \n",
       "L 190.532813 137.952797 \n",
       "L 187.48125 137.334965 \n",
       "L 184.429688 136.652098 \n",
       "L 181.643478 135.969231 \n",
       "L 181.378125 135.901742 \n",
       "L 178.326563 135.058128 \n",
       "L 175.275 134.147025 \n",
       "L 172.223438 133.168433 \n",
       "L 169.959375 132.392308 \n",
       "L 169.171875 132.111765 \n",
       "L 166.120313 130.954525 \n",
       "L 163.06875 129.727149 \n",
       "L 160.924409 128.815385 \n",
       "L 160.017188 128.413893 \n",
       "L 156.965625 126.990424 \n",
       "L 153.914062 125.493956 \n",
       "L 153.417297 125.238462 \n",
       "L 150.8625 123.868576 \n",
       "L 147.810938 122.156219 \n",
       "L 146.966888 121.661538 \n",
       "L 144.759375 120.310256 \n",
       "L 141.707813 118.362821 \n",
       "L 141.288971 118.084615 \n",
       "L 138.65625 116.254562 \n",
       "L 136.238031 114.507692 \n",
       "L 135.604688 114.027861 \n",
       "L 132.553125 111.628705 \n",
       "L 131.696546 110.930769 \n",
       "L 129.501563 109.050592 \n",
       "L 127.587871 107.353846 \n",
       "L 126.45 106.290437 \n",
       "L 123.848668 103.776923 \n",
       "L 123.398438 103.317033 \n",
       "L 120.44375 100.2 \n",
       "L 120.346875 100.091608 \n",
       "L 117.34226 96.623077 \n",
       "L 117.295313 96.565385 \n",
       "L 114.517024 93.046154 \n",
       "L 114.24375 92.676127 \n",
       "L 111.944022 89.469231 \n",
       "L 111.192188 88.343162 \n",
       "L 109.601937 85.892308 \n",
       "L 108.140625 83.46 \n",
       "L 107.471789 82.315385 \n",
       "L 105.548887 78.738462 \n",
       "L 105.089062 77.801648 \n",
       "L 103.82775 75.161538 \n",
       "L 102.281625 71.584615 \n",
       "L 102.0375 70.953394 \n",
       "L 100.927841 68.007692 \n",
       "L 99.73892 64.430769 \n",
       "L 98.985938 61.816864 \n",
       "L 98.715546 60.853846 \n",
       "L 97.865744 57.276923 \n",
       "L 97.170451 53.7 \n",
       "L 96.629668 50.123077 \n",
       "L 96.243394 46.546154 \n",
       "L 96.01163 42.969231 \n",
       "L 95.934375 39.392308 \n",
       "L 96.01163 35.815385 \n",
       "L 96.243394 32.238462 \n",
       "L 96.629668 28.661538 \n",
       "L 97.170451 25.084615 \n",
       "L 97.865744 21.507692 \n",
       "L 98.715546 17.930769 \n",
       "L 98.985938 16.967751 \n",
       "L 99.73892 14.353846 \n",
       "L 100.927841 10.776923 \n",
       "L 102.0375 7.831222 \n",
       "L 102.281625 7.2 \n",
       "\" style=\"fill:none;stroke:#1f77b4;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_3\">\n",
       "    <path clip-path=\"url(#p3cde0fc7ae)\" d=\"M 143.264732 146.7 \n",
       "L 141.707813 145.942177 \n",
       "L 138.65625 144.396219 \n",
       "L 136.238031 143.123077 \n",
       "L 135.604688 142.777935 \n",
       "L 132.553125 141.052227 \n",
       "L 129.983388 139.546154 \n",
       "L 129.501563 139.253497 \n",
       "L 126.45 137.334965 \n",
       "L 124.348924 135.969231 \n",
       "L 123.398438 135.328084 \n",
       "L 120.346875 133.202177 \n",
       "L 119.220144 132.392308 \n",
       "L 117.295313 130.954525 \n",
       "L 114.517024 128.815385 \n",
       "L 114.24375 128.596389 \n",
       "L 111.192188 126.077943 \n",
       "L 110.203653 125.238462 \n",
       "L 108.140625 123.411948 \n",
       "L 106.217723 121.661538 \n",
       "L 105.089063 120.588462 \n",
       "L 102.52575 118.084615 \n",
       "L 102.0375 117.58551 \n",
       "L 99.10483 114.507692 \n",
       "L 98.985938 114.376829 \n",
       "L 95.934375 110.930769 \n",
       "L 95.934375 110.930769 \n",
       "L 92.995833 107.353846 \n",
       "L 92.882813 107.208836 \n",
       "L 90.27244 103.776923 \n",
       "L 89.83125 103.163736 \n",
       "L 87.749007 100.2 \n",
       "L 86.779688 98.736713 \n",
       "L 85.411746 96.623077 \n",
       "L 83.728125 93.853846 \n",
       "L 83.248104 93.046154 \n",
       "L 81.259445 89.469231 \n",
       "L 80.676563 88.343162 \n",
       "L 79.435817 85.892308 \n",
       "L 77.759135 82.315385 \n",
       "L 77.625 82.004348 \n",
       "L 76.246875 78.738462 \n",
       "L 74.86875 75.161538 \n",
       "L 74.573438 74.314372 \n",
       "L 73.641908 71.584615 \n",
       "L 72.54977 68.007692 \n",
       "L 71.586118 64.430769 \n",
       "L 71.521875 64.155621 \n",
       "L 70.766849 60.853846 \n",
       "L 70.074742 57.276923 \n",
       "L 69.508473 53.7 \n",
       "L 69.068041 50.123077 \n",
       "L 68.753447 46.546154 \n",
       "L 68.564691 42.969231 \n",
       "L 68.501772 39.392308 \n",
       "L 68.564691 35.815385 \n",
       "L 68.753447 32.238462 \n",
       "L 69.068041 28.661538 \n",
       "L 69.508473 25.084615 \n",
       "L 70.074742 21.507692 \n",
       "L 70.766849 17.930769 \n",
       "L 71.521875 14.628994 \n",
       "L 71.586118 14.353846 \n",
       "L 72.54977 10.776923 \n",
       "L 73.641908 7.2 \n",
       "\" style=\"fill:none;stroke:#1f77b4;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_4\">\n",
       "    <path clip-path=\"url(#p3cde0fc7ae)\" d=\"M 103.82775 146.7 \n",
       "L 102.0375 145.366232 \n",
       "L 99.10483 143.123077 \n",
       "L 98.985938 143.028947 \n",
       "L 95.934375 140.550202 \n",
       "L 94.728819 139.546154 \n",
       "L 92.882813 137.952797 \n",
       "L 90.640098 135.969231 \n",
       "L 89.83125 135.226851 \n",
       "L 86.815588 132.392308 \n",
       "L 86.779688 132.35724 \n",
       "L 83.728125 129.306335 \n",
       "L 83.248104 128.815385 \n",
       "L 80.676563 126.077943 \n",
       "L 79.905288 125.238462 \n",
       "L 77.625 122.6509 \n",
       "L 76.771875 121.661538 \n",
       "L 74.573438 118.998718 \n",
       "L 73.834638 118.084615 \n",
       "L 71.521875 115.089982 \n",
       "L 71.081443 114.507692 \n",
       "L 68.501772 110.930769 \n",
       "L 68.470312 110.884911 \n",
       "L 66.096875 107.353846 \n",
       "L 65.41875 106.290437 \n",
       "L 63.847649 103.776923 \n",
       "L 62.367188 101.273077 \n",
       "L 61.745024 100.2 \n",
       "L 59.789654 96.623077 \n",
       "L 59.315625 95.7 \n",
       "L 57.97875 93.046154 \n",
       "L 56.293125 89.469231 \n",
       "L 56.264062 89.402991 \n",
       "L 54.752541 85.892308 \n",
       "L 53.326577 82.315385 \n",
       "L 53.2125 82.004348 \n",
       "L 52.036669 78.738462 \n",
       "L 50.860837 75.161538 \n",
       "L 50.160938 72.8083 \n",
       "\" style=\"fill:none;stroke:#1f77b4;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_5\">\n",
       "    <path clip-path=\"url(#p3cde0fc7ae)\" d=\"M 74.86875 146.7 \n",
       "L 74.573438 146.427184 \n",
       "L 71.521875 143.547458 \n",
       "L 71.081443 143.123077 \n",
       "L 68.470313 140.518826 \n",
       "L 67.514773 139.546154 \n",
       "L 65.41875 137.334965 \n",
       "L 64.149783 135.969231 \n",
       "L 62.367188 133.978302 \n",
       "L 60.974727 132.392308 \n",
       "L 59.315625 130.428507 \n",
       "L 57.97875 128.815385 \n",
       "L 56.264062 126.661931 \n",
       "L 55.151811 125.238462 \n",
       "L 53.2125 122.6509 \n",
       "L 52.484604 121.661538 \n",
       "L 50.160938 118.362821 \n",
       "\" style=\"fill:none;stroke:#1f77b4;\"/>\n",
       "   </g>\n",
       "   <g id=\"LineCollection_6\">\n",
       "    <path clip-path=\"url(#p3cde0fc7ae)\" d=\"M 50.860837 146.7 \n",
       "L 50.160938 145.942177 \n",
       "\" style=\"fill:none;stroke:#1f77b4;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_1\">\n",
       "    <path clip-path=\"url(#p3cde0fc7ae)\" d=\"M 65.41875 110.930769 \n",
       "L 95.934375 82.315385 \n",
       "L 120.346875 65.146154 \n",
       "L 139.876875 54.844615 \n",
       "L 155.500875 48.663692 \n",
       "L 168.000075 44.955138 \n",
       "L 177.999435 42.730006 \n",
       "L 185.998923 41.394927 \n",
       "L 192.398513 40.593879 \n",
       "L 197.518186 40.113251 \n",
       "L 201.613924 39.824873 \n",
       "L 204.890514 39.651847 \n",
       "L 207.511786 39.548031 \n",
       "L 209.608804 39.485742 \n",
       "L 211.286418 39.448368 \n",
       "L 212.628509 39.425944 \n",
       "L 213.702183 39.412489 \n",
       "L 214.561121 39.404417 \n",
       "L 215.248272 39.399573 \n",
       "L 215.797992 39.396667 \n",
       "L 216.237769 39.394923 \n",
       "\" style=\"fill:none;stroke:#ff7f0e;stroke-linecap:square;\"/>\n",
       "    <defs>\n",
       "     <path d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" id=\"m0cbd199217\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#p3cde0fc7ae)\">\n",
       "     <use style=\"fill:#ff7f0e;stroke:#000000;stroke-width:0.5;\" x=\"65.41875000000007\" xlink:href=\"#m0cbd199217\" y=\"110.93076923076926\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#000000;stroke-width:0.5;\" x=\"95.93437500000017\" xlink:href=\"#m0cbd199217\" y=\"82.31538461538466\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#000000;stroke-width:0.5;\" x=\"120.34687500000025\" xlink:href=\"#m0cbd199217\" y=\"65.14615384615391\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#000000;stroke-width:0.5;\" x=\"139.87687500000033\" xlink:href=\"#m0cbd199217\" y=\"54.84461538461546\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#000000;stroke-width:0.5;\" x=\"155.50087500000038\" xlink:href=\"#m0cbd199217\" y=\"48.663692307692386\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#000000;stroke-width:0.5;\" x=\"168.00007500000044\" xlink:href=\"#m0cbd199217\" y=\"44.955138461538546\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#000000;stroke-width:0.5;\" x=\"177.99943500000046\" xlink:href=\"#m0cbd199217\" y=\"42.73000615384624\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#000000;stroke-width:0.5;\" x=\"185.9989230000005\" xlink:href=\"#m0cbd199217\" y=\"41.39492676923086\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#000000;stroke-width:0.5;\" x=\"192.39851340000052\" xlink:href=\"#m0cbd199217\" y=\"40.593879138461624\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#000000;stroke-width:0.5;\" x=\"197.51818572000053\" xlink:href=\"#m0cbd199217\" y=\"40.11325056000009\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#000000;stroke-width:0.5;\" x=\"201.61392357600056\" xlink:href=\"#m0cbd199217\" y=\"39.82487341292317\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#000000;stroke-width:0.5;\" x=\"204.89051386080055\" xlink:href=\"#m0cbd199217\" y=\"39.65184712467701\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#000000;stroke-width:0.5;\" x=\"207.51178608864058\" xlink:href=\"#m0cbd199217\" y=\"39.54803135172932\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#000000;stroke-width:0.5;\" x=\"209.6088038709126\" xlink:href=\"#m0cbd199217\" y=\"39.48574188796071\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#000000;stroke-width:0.5;\" x=\"211.2864180967302\" xlink:href=\"#m0cbd199217\" y=\"39.44836820969954\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#000000;stroke-width:0.5;\" x=\"212.62850947738428\" xlink:href=\"#m0cbd199217\" y=\"39.42594400274283\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#000000;stroke-width:0.5;\" x=\"213.70218258190755\" xlink:href=\"#m0cbd199217\" y=\"39.412489478568816\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#000000;stroke-width:0.5;\" x=\"214.56112106552615\" xlink:href=\"#m0cbd199217\" y=\"39.4044167640644\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#000000;stroke-width:0.5;\" x=\"215.24827185242106\" xlink:href=\"#m0cbd199217\" y=\"39.399573135361756\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#000000;stroke-width:0.5;\" x=\"215.79799248193697\" xlink:href=\"#m0cbd199217\" y=\"39.396666958140166\"/>\n",
       "     <use style=\"fill:#ff7f0e;stroke:#000000;stroke-width:0.5;\" x=\"216.23776898554968\" xlink:href=\"#m0cbd199217\" y=\"39.394923251807214\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 245.460938 146.7 \n",
       "L 245.460938 7.2 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 50.160938 7.2 \n",
       "L 245.460938 7.2 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 50.160938 146.7 \n",
       "L 50.160938 7.2 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 50.160938 146.7 \n",
       "L 245.460938 146.7 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 -4 \n",
       "\" id=\"mdbecd66cb6\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"65.41875000000007\" xlink:href=\"#mdbecd66cb6\" y=\"146.7\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_3\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 4 \n",
       "\" id=\"m7af75d2afd\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"65.41875000000007\" xlink:href=\"#m7af75d2afd\" y=\"7.199999999999989\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −5 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.796875 72.90625 \n",
       "L 49.515625 72.90625 \n",
       "L 49.515625 64.59375 \n",
       "L 19.828125 64.59375 \n",
       "L 19.828125 46.734375 \n",
       "Q 21.96875 47.46875 24.109375 47.828125 \n",
       "Q 26.265625 48.1875 28.421875 48.1875 \n",
       "Q 40.625 48.1875 47.75 41.5 \n",
       "Q 54.890625 34.8125 54.890625 23.390625 \n",
       "Q 54.890625 11.625 47.5625 5.09375 \n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \n",
       "Q 12.796875 0.140625 7.71875 1.703125 \n",
       "L 7.71875 11.625 \n",
       "Q 12.109375 9.234375 16.796875 8.0625 \n",
       "Q 21.484375 6.890625 26.703125 6.890625 \n",
       "Q 35.15625 6.890625 40.078125 11.328125 \n",
       "Q 45.015625 15.765625 45.015625 23.390625 \n",
       "Q 45.015625 31 40.078125 35.4375 \n",
       "Q 35.15625 39.890625 26.703125 39.890625 \n",
       "Q 22.75 39.890625 18.8125 39.015625 \n",
       "Q 14.890625 38.140625 10.796875 36.28125 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-35\"/>\n",
       "       <path d=\"M 10.59375 35.5 \n",
       "L 73.1875 35.5 \n",
       "L 73.1875 27.203125 \n",
       "L 10.59375 27.203125 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-2212\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(58.047656250000074 158.29843749999998)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"95.93437500000017\" xlink:href=\"#mdbecd66cb6\" y=\"146.7\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"95.93437500000017\" xlink:href=\"#m7af75d2afd\" y=\"7.199999999999989\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −4 -->\n",
       "      <defs>\n",
       "       <path d=\"M 37.796875 64.3125 \n",
       "L 12.890625 25.390625 \n",
       "L 37.796875 25.390625 \n",
       "z\n",
       "M 35.203125 72.90625 \n",
       "L 47.609375 72.90625 \n",
       "L 47.609375 25.390625 \n",
       "L 58.015625 25.390625 \n",
       "L 58.015625 17.1875 \n",
       "L 47.609375 17.1875 \n",
       "L 47.609375 0 \n",
       "L 37.796875 0 \n",
       "L 37.796875 17.1875 \n",
       "L 4.890625 17.1875 \n",
       "L 4.890625 26.703125 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-34\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(88.56328125000017 158.29843749999998)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-34\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"126.45000000000029\" xlink:href=\"#mdbecd66cb6\" y=\"146.7\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"126.45000000000029\" xlink:href=\"#m7af75d2afd\" y=\"7.199999999999989\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- −3 -->\n",
       "      <defs>\n",
       "       <path d=\"M 40.578125 39.3125 \n",
       "Q 47.65625 37.796875 51.625 33 \n",
       "Q 55.609375 28.21875 55.609375 21.1875 \n",
       "Q 55.609375 10.40625 48.1875 4.484375 \n",
       "Q 40.765625 -1.421875 27.09375 -1.421875 \n",
       "Q 22.515625 -1.421875 17.65625 -0.515625 \n",
       "Q 12.796875 0.390625 7.625 2.203125 \n",
       "L 7.625 11.71875 \n",
       "Q 11.71875 9.328125 16.59375 8.109375 \n",
       "Q 21.484375 6.890625 26.8125 6.890625 \n",
       "Q 36.078125 6.890625 40.9375 10.546875 \n",
       "Q 45.796875 14.203125 45.796875 21.1875 \n",
       "Q 45.796875 27.640625 41.28125 31.265625 \n",
       "Q 36.765625 34.90625 28.71875 34.90625 \n",
       "L 20.21875 34.90625 \n",
       "L 20.21875 43.015625 \n",
       "L 29.109375 43.015625 \n",
       "Q 36.375 43.015625 40.234375 45.921875 \n",
       "Q 44.09375 48.828125 44.09375 54.296875 \n",
       "Q 44.09375 59.90625 40.109375 62.90625 \n",
       "Q 36.140625 65.921875 28.71875 65.921875 \n",
       "Q 24.65625 65.921875 20.015625 65.03125 \n",
       "Q 15.375 64.15625 9.8125 62.3125 \n",
       "L 9.8125 71.09375 \n",
       "Q 15.4375 72.65625 20.34375 73.4375 \n",
       "Q 25.25 74.21875 29.59375 74.21875 \n",
       "Q 40.828125 74.21875 47.359375 69.109375 \n",
       "Q 53.90625 64.015625 53.90625 55.328125 \n",
       "Q 53.90625 49.265625 50.4375 45.09375 \n",
       "Q 46.96875 40.921875 40.578125 39.3125 \n",
       "\" id=\"BitstreamVeraSans-Roman-33\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(119.07890625000029 158.29843749999998)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-33\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"156.9656250000004\" xlink:href=\"#mdbecd66cb6\" y=\"146.7\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"156.9656250000004\" xlink:href=\"#m7af75d2afd\" y=\"7.199999999999989\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- −2 -->\n",
       "      <defs>\n",
       "       <path d=\"M 19.1875 8.296875 \n",
       "L 53.609375 8.296875 \n",
       "L 53.609375 0 \n",
       "L 7.328125 0 \n",
       "L 7.328125 8.296875 \n",
       "Q 12.9375 14.109375 22.625 23.890625 \n",
       "Q 32.328125 33.6875 34.8125 36.53125 \n",
       "Q 39.546875 41.84375 41.421875 45.53125 \n",
       "Q 43.3125 49.21875 43.3125 52.78125 \n",
       "Q 43.3125 58.59375 39.234375 62.25 \n",
       "Q 35.15625 65.921875 28.609375 65.921875 \n",
       "Q 23.96875 65.921875 18.8125 64.3125 \n",
       "Q 13.671875 62.703125 7.8125 59.421875 \n",
       "L 7.8125 69.390625 \n",
       "Q 13.765625 71.78125 18.9375 73 \n",
       "Q 24.125 74.21875 28.421875 74.21875 \n",
       "Q 39.75 74.21875 46.484375 68.546875 \n",
       "Q 53.21875 62.890625 53.21875 53.421875 \n",
       "Q 53.21875 48.921875 51.53125 44.890625 \n",
       "Q 49.859375 40.875 45.40625 35.40625 \n",
       "Q 44.1875 33.984375 37.640625 27.21875 \n",
       "Q 31.109375 20.453125 19.1875 8.296875 \n",
       "\" id=\"BitstreamVeraSans-Roman-32\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(149.5945312500004 158.29843749999998)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"187.4812500000005\" xlink:href=\"#mdbecd66cb6\" y=\"146.7\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"187.4812500000005\" xlink:href=\"#m7af75d2afd\" y=\"7.199999999999989\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- −1 -->\n",
       "      <defs>\n",
       "       <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-31\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(180.1101562500005 158.29843749999998)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-31\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"217.9968750000006\" xlink:href=\"#mdbecd66cb6\" y=\"146.7\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"217.9968750000006\" xlink:href=\"#m7af75d2afd\" y=\"7.199999999999989\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- 0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "\" id=\"BitstreamVeraSans-Roman-30\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(214.8156250000006 158.29843749999998)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_7\">\n",
       "     <!-- x1 -->\n",
       "     <defs>\n",
       "      <path d=\"M 54.890625 54.6875 \n",
       "L 35.109375 28.078125 \n",
       "L 55.90625 0 \n",
       "L 45.3125 0 \n",
       "L 29.390625 21.484375 \n",
       "L 13.484375 0 \n",
       "L 2.875 0 \n",
       "L 24.125 28.609375 \n",
       "L 4.6875 54.6875 \n",
       "L 15.28125 54.6875 \n",
       "L 29.78125 35.203125 \n",
       "L 44.28125 54.6875 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-78\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(141.67031250000002 172.9765625)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#BitstreamVeraSans-Roman-78\"/>\n",
       "      <use x=\"59.1796875\" xlink:href=\"#BitstreamVeraSans-Roman-31\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_14\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 4 0 \n",
       "\" id=\"m7920931895\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"50.1609375\" xlink:href=\"#m7920931895\" y=\"146.7\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_15\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -4 0 \n",
       "\" id=\"me443581992\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"245.4609375\" xlink:href=\"#me443581992\" y=\"146.7\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- −3.0 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.6875 12.40625 \n",
       "L 21 12.40625 \n",
       "L 21 0 \n",
       "L 10.6875 0 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-2e\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(21.878125 149.459375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-33\"/>\n",
       "       <use x=\"147.412109375\" xlink:href=\"#BitstreamVeraSans-Roman-2e\"/>\n",
       "       <use x=\"179.19921875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"50.1609375\" xlink:href=\"#m7920931895\" y=\"128.81538461538463\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_17\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"245.4609375\" xlink:href=\"#me443581992\" y=\"128.81538461538463\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- −2.5 -->\n",
       "      <g transform=\"translate(21.878125 131.57475961538464)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-32\"/>\n",
       "       <use x=\"147.412109375\" xlink:href=\"#BitstreamVeraSans-Roman-2e\"/>\n",
       "       <use x=\"179.19921875\" xlink:href=\"#BitstreamVeraSans-Roman-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"50.1609375\" xlink:href=\"#m7920931895\" y=\"110.93076923076926\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_19\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"245.4609375\" xlink:href=\"#me443581992\" y=\"110.93076923076926\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- −2.0 -->\n",
       "      <g transform=\"translate(21.878125 113.69014423076926)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-32\"/>\n",
       "       <use x=\"147.412109375\" xlink:href=\"#BitstreamVeraSans-Roman-2e\"/>\n",
       "       <use x=\"179.19921875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"50.1609375\" xlink:href=\"#m7920931895\" y=\"93.04615384615389\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_21\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"245.4609375\" xlink:href=\"#me443581992\" y=\"93.04615384615389\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- −1.5 -->\n",
       "      <g transform=\"translate(21.878125 95.80552884615389)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-31\"/>\n",
       "       <use x=\"147.412109375\" xlink:href=\"#BitstreamVeraSans-Roman-2e\"/>\n",
       "       <use x=\"179.19921875\" xlink:href=\"#BitstreamVeraSans-Roman-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"50.1609375\" xlink:href=\"#m7920931895\" y=\"75.16153846153853\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_23\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"245.4609375\" xlink:href=\"#me443581992\" y=\"75.16153846153853\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- −1.0 -->\n",
       "      <g transform=\"translate(21.878125 77.92091346153853)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-31\"/>\n",
       "       <use x=\"147.412109375\" xlink:href=\"#BitstreamVeraSans-Roman-2e\"/>\n",
       "       <use x=\"179.19921875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_24\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"50.1609375\" xlink:href=\"#m7920931895\" y=\"57.276923076923154\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_25\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"245.4609375\" xlink:href=\"#me443581992\" y=\"57.276923076923154\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- −0.5 -->\n",
       "      <g transform=\"translate(21.878125 60.03629807692316)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "       <use x=\"147.412109375\" xlink:href=\"#BitstreamVeraSans-Roman-2e\"/>\n",
       "       <use x=\"179.19921875\" xlink:href=\"#BitstreamVeraSans-Roman-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_7\">\n",
       "     <g id=\"line2d_26\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"50.1609375\" xlink:href=\"#m7920931895\" y=\"39.39230769230778\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_27\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"245.4609375\" xlink:href=\"#me443581992\" y=\"39.39230769230778\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_14\">\n",
       "      <!-- 0.0 -->\n",
       "      <g transform=\"translate(30.2578125 42.15168269230779)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-2e\"/>\n",
       "       <use x=\"95.41015625\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_8\">\n",
       "     <g id=\"line2d_28\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"50.1609375\" xlink:href=\"#m7920931895\" y=\"21.507692307692412\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_29\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"245.4609375\" xlink:href=\"#me443581992\" y=\"21.507692307692412\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_15\">\n",
       "      <!-- 0.5 -->\n",
       "      <g transform=\"translate(30.2578125 24.26706730769243)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-2e\"/>\n",
       "       <use x=\"95.41015625\" xlink:href=\"#BitstreamVeraSans-Roman-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_16\">\n",
       "     <!-- x2 -->\n",
       "     <g transform=\"translate(14.7984375 83.09062499999999)rotate(-90.0)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#BitstreamVeraSans-Roman-78\"/>\n",
       "      <use x=\"59.1796875\" xlink:href=\"#BitstreamVeraSans-Roman-32\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p3cde0fc7ae\">\n",
       "   <rect height=\"139.5\" width=\"195.3\" x=\"50.1609375\" y=\"7.199999999999989\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f23b8b6e4e0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def f(x1, x2): return x1 ** 2 + 2 * x2 ** 2   # objective\n",
    "def gradf(x1, x2): return (2 * x1, 4 * x2)    # gradient \n",
    "def gd(x1, x2, s1, s2): \n",
    "    (g1, g2) = gradf(x1, x2)                  # compute gradient\n",
    "    return (x1 -eta * g1, x2 -eta * g2, 0, 0) # update variables\n",
    "\n",
    "eta = 0.1\n",
    "d2l.show_trace_2d(f, d2l.train_2d(gd))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adaptive Methods\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we could see in :numref:section_gd-learningrate, getting the learning rate $\\eta$ 'just right' is tricky. If we pick it too small, we make no progress. If we pick it too large, the solution oscillates and in the worst case it might even diverge. What if we could determine $\\eta$ automatically or get rid of having to select a step size at all? Second order methods that look not only at the value and gradient of the objective but also at its curvature can help in this case. While these methods cannot be applied to deep learning directly due to the computational cost, they provide useful intuition into how to design advanced optimization algorithms that mimic many of the desirable properties of the algorithms outlined below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Newton's Method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reviewing the Taylor expansion of $f$ there's no need to stop after the first term. In fact, we can write it as\n",
    "\n",
    "$$f(\\mathbf{x} + \\mathbf{\\epsilon}) = f(\\mathbf{x}) + \\mathbf{\\epsilon}^\\top \\nabla f(\\mathbf{x}) + \\frac{1}{2} \\mathbf{\\epsilon}^\\top \\nabla \\nabla^\\top f(\\mathbf{x}) \\mathbf{\\epsilon} + O(|\\mathbf{\\epsilon}|^3)$$ :eqlabel:gd-hot-taylor\n",
    "\n",
    "To avoid cumbersome notation we define $H_f := \\nabla \\nabla^\\top f(\\mathbf{x})$ to be the Hessian of $f$. This is a $d \\times d$ matrix. For small $d$ and simple problems $H_f$ is easy to compute. For deep networks, on the other hand, $H_f$ may be prohibitively large, due to the cost of storing $O(d^2)$ entries. Furthermore it may be too expensive to compute via backprop as we would need to apply backprop to the backpropagation call graph. For now let us ignore such considerations and look at what algorithm we'd get.\n",
    "\n",
    "After all, the minimum of $f$ satisfies $\\nabla f(\\mathbf{x}) = 0$. Taking derivatives of :eqref:gd-hot-taylor with regard to $\\mathbf{\\epsilon}$ and ignoring higher order terms we arrive at\n",
    "\n",
    "$$\\nabla f(\\mathbf{x}) + H_f \\mathbf{\\epsilon} = 0 \\text{ and hence } \\mathbf{\\epsilon} = -H_f^{-1} \\nabla f(\\mathbf{x}).$$\n",
    "\n",
    "That is, we need to invert the Hessian $H_f$ as part of the optimization problem.\n",
    "\n",
    "For $f(x) = \\frac{1}{2} x^2$ we have $\\nabla f(x) = x$ and $H_f = 1$. Hence for any $x$ we obtain $\\epsilon = -x$. In other words, a single step is sufficient to converge perfectly without the need for any adjustment! Alas, we got a bit lucky here since the Taylor expansion was exact. Let's see what happens in other problems"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 10, x: 0.0\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\n",
       "<svg height=\"187pt\" version=\"1.1\" viewBox=\"0 0 247 187\" width=\"247pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;stroke-miterlimit:100000;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 187.095313 \n",
       "L 247.465625 187.095313 \n",
       "L 247.465625 0 \n",
       "L 0 0 \n",
       "L 0 187.095313 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 38.603125 151.539062 \n",
       "L 233.903125 151.539062 \n",
       "L 233.903125 12.039062 \n",
       "L 38.603125 12.039062 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_1\">\n",
       "    <path clip-path=\"url(#pd0488643bb)\" d=\"M 38.603125 22.135465 \n",
       "L 40.751425 35.613607 \n",
       "L 42.899725 47.687637 \n",
       "L 45.048025 58.503797 \n",
       "L 47.196325 68.193094 \n",
       "L 49.344625 76.872888 \n",
       "L 51.492925 84.64831 \n",
       "L 53.641225 91.613538 \n",
       "L 55.789525 97.852935 \n",
       "L 57.937825 103.442075 \n",
       "L 60.183775 108.663392 \n",
       "L 62.429725 113.317053 \n",
       "L 64.675675 117.46467 \n",
       "L 66.921625 121.161157 \n",
       "L 69.265225 124.590252 \n",
       "L 71.608825 127.630819 \n",
       "L 74.050075 130.432131 \n",
       "L 76.491325 132.903218 \n",
       "L 79.030225 135.164354 \n",
       "L 81.666775 137.219611 \n",
       "L 84.400975 139.075361 \n",
       "L 87.330475 140.792999 \n",
       "L 90.455275 142.358986 \n",
       "L 93.775375 143.765605 \n",
       "L 97.290775 145.010274 \n",
       "L 101.099125 146.120406 \n",
       "L 105.298075 147.106241 \n",
       "L 109.887625 147.949845 \n",
       "L 114.965425 148.652729 \n",
       "L 120.629125 149.20691 \n",
       "L 126.878725 149.590546 \n",
       "L 133.616575 149.779398 \n",
       "L 140.549725 149.752943 \n",
       "L 147.287575 149.509505 \n",
       "L 153.537175 149.066688 \n",
       "L 159.200875 148.446555 \n",
       "L 164.278675 147.66989 \n",
       "L 168.868225 146.743388 \n",
       "L 172.969525 145.692186 \n",
       "L 176.777875 144.485504 \n",
       "L 180.195625 143.175048 \n",
       "L 183.418075 141.705064 \n",
       "L 186.445225 140.080567 \n",
       "L 189.277075 138.312008 \n",
       "L 192.011275 136.340544 \n",
       "L 194.647825 134.157389 \n",
       "L 197.186725 131.755751 \n",
       "L 199.627975 129.131283 \n",
       "L 201.971575 126.282536 \n",
       "L 204.217525 123.211382 \n",
       "L 206.463475 119.765182 \n",
       "L 208.709425 115.898309 \n",
       "L 210.857725 111.758775 \n",
       "L 213.006025 107.137415 \n",
       "L 215.154325 101.978252 \n",
       "L 217.302625 96.218799 \n",
       "L 219.450925 89.789295 \n",
       "L 221.501575 82.955531 \n",
       "L 223.552225 75.364939 \n",
       "L 225.602875 66.933755 \n",
       "L 227.653525 57.56894 \n",
       "L 229.704175 47.167153 \n",
       "L 231.754825 35.613607 \n",
       "L 233.805475 22.780809 \n",
       "L 233.805475 22.780809 \n",
       "\" style=\"fill:none;stroke:#0000ff;stroke-linecap:square;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_2\">\n",
       "    <path clip-path=\"url(#pd0488643bb)\" d=\"M 233.903125 22.135465 \n",
       "L 214.374898 103.916011 \n",
       "L 194.857995 133.970272 \n",
       "L 175.424429 144.942584 \n",
       "L 156.589149 148.761425 \n",
       "L 141.387202 149.734712 \n",
       "L 136.368211 149.795282 \n",
       "L 136.253126 149.795312 \n",
       "L 136.253125 149.795312 \n",
       "L 136.253125 149.795312 \n",
       "L 136.253125 149.795312 \n",
       "\" style=\"fill:none;stroke:#008000;stroke-linecap:square;\"/>\n",
       "    <defs>\n",
       "     <path d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" id=\"mfcb8aad43c\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#pd0488643bb)\">\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"233.903125\" xlink:href=\"#mfcb8aad43c\" y=\"22.13546475990117\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"214.37489824075152\" xlink:href=\"#mfcb8aad43c\" y=\"103.9160106179377\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"194.85799463923487\" xlink:href=\"#mfcb8aad43c\" y=\"133.970271610016\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"175.42442870640843\" xlink:href=\"#mfcb8aad43c\" y=\"144.94258390330285\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"156.58914942097402\" xlink:href=\"#mfcb8aad43c\" y=\"148.7614246506668\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"141.3872022497521\" xlink:href=\"#mfcb8aad43c\" y=\"149.73471228662\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"136.36821117364372\" xlink:href=\"#mfcb8aad43c\" y=\"149.79528222408362\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"136.2531263321026\" xlink:href=\"#mfcb8aad43c\" y=\"149.7953125\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"136.25312499999998\" xlink:href=\"#mfcb8aad43c\" y=\"149.7953125\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"136.25312499999998\" xlink:href=\"#mfcb8aad43c\" y=\"149.7953125\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"136.25312499999998\" xlink:href=\"#mfcb8aad43c\" y=\"149.7953125\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 233.903125 151.539062 \n",
       "L 233.903125 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 38.603125 12.039062 \n",
       "L 233.903125 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 38.603125 151.539062 \n",
       "L 38.603125 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 38.603125 151.539062 \n",
       "L 233.903125 151.539062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path clip-path=\"url(#pd0488643bb)\" d=\"M 38.603125 151.539062 \n",
       "L 38.603125 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 -4 \n",
       "\" id=\"m1f92203fc3\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"38.60312499999998\" xlink:href=\"#m1f92203fc3\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_5\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 4 \n",
       "\" id=\"mdcffb8352e\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"38.60312499999998\" xlink:href=\"#mdcffb8352e\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −10 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "\" id=\"BitstreamVeraSans-Roman-30\"/>\n",
       "       <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-31\"/>\n",
       "       <path d=\"M 10.59375 35.5 \n",
       "L 73.1875 35.5 \n",
       "L 73.1875 27.203125 \n",
       "L 10.59375 27.203125 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-2212\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(28.05078124999998 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-31\"/>\n",
       "       <use x=\"147.412109375\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <path clip-path=\"url(#pd0488643bb)\" d=\"M 87.428125 151.539062 \n",
       "L 87.428125 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"87.42812499999998\" xlink:href=\"#m1f92203fc3\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"87.42812499999998\" xlink:href=\"#mdcffb8352e\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −5 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.796875 72.90625 \n",
       "L 49.515625 72.90625 \n",
       "L 49.515625 64.59375 \n",
       "L 19.828125 64.59375 \n",
       "L 19.828125 46.734375 \n",
       "Q 21.96875 47.46875 24.109375 47.828125 \n",
       "Q 26.265625 48.1875 28.421875 48.1875 \n",
       "Q 40.625 48.1875 47.75 41.5 \n",
       "Q 54.890625 34.8125 54.890625 23.390625 \n",
       "Q 54.890625 11.625 47.5625 5.09375 \n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \n",
       "Q 12.796875 0.140625 7.71875 1.703125 \n",
       "L 7.71875 11.625 \n",
       "Q 12.109375 9.234375 16.796875 8.0625 \n",
       "Q 21.484375 6.890625 26.703125 6.890625 \n",
       "Q 35.15625 6.890625 40.078125 11.328125 \n",
       "Q 45.015625 15.765625 45.015625 23.390625 \n",
       "Q 45.015625 31 40.078125 35.4375 \n",
       "Q 35.15625 39.890625 26.703125 39.890625 \n",
       "Q 22.75 39.890625 18.8125 39.015625 \n",
       "Q 14.890625 38.140625 10.796875 36.28125 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-35\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(80.05703124999998 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path clip-path=\"url(#pd0488643bb)\" d=\"M 136.253125 151.539062 \n",
       "L 136.253125 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"136.25312499999998\" xlink:href=\"#m1f92203fc3\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"136.25312499999998\" xlink:href=\"#mdcffb8352e\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(133.07187499999998 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <path clip-path=\"url(#pd0488643bb)\" d=\"M 185.078125 151.539062 \n",
       "L 185.078125 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"185.078125\" xlink:href=\"#m1f92203fc3\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"185.078125\" xlink:href=\"#mdcffb8352e\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(181.896875 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path clip-path=\"url(#pd0488643bb)\" d=\"M 233.903125 151.539062 \n",
       "L 233.903125 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"233.903125\" xlink:href=\"#m1f92203fc3\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_17\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"233.903125\" xlink:href=\"#mdcffb8352e\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(227.54062499999998 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-31\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- x -->\n",
       "     <defs>\n",
       "      <path d=\"M 54.890625 54.6875 \n",
       "L 35.109375 28.078125 \n",
       "L 55.90625 0 \n",
       "L 45.3125 0 \n",
       "L 29.390625 21.484375 \n",
       "L 13.484375 0 \n",
       "L 2.875 0 \n",
       "L 24.125 28.609375 \n",
       "L 4.6875 54.6875 \n",
       "L 15.28125 54.6875 \n",
       "L 29.78125 35.203125 \n",
       "L 44.28125 54.6875 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-78\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(133.29375 177.815625)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#BitstreamVeraSans-Roman-78\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_18\">\n",
       "      <path clip-path=\"url(#pd0488643bb)\" d=\"M 38.603125 151.539062 \n",
       "L 233.903125 151.539062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_19\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 4 0 \n",
       "\" id=\"m92065022d4\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"38.603125\" xlink:href=\"#m92065022d4\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -4 0 \n",
       "\" id=\"m99c08219a3\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"233.903125\" xlink:href=\"#m99c08219a3\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(28.240624999999998 154.2984375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_21\">\n",
       "      <path clip-path=\"url(#pd0488643bb)\" d=\"M 38.603125 134.101562 \n",
       "L 233.903125 134.101562 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"38.603125\" xlink:href=\"#m92065022d4\" y=\"134.1015625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_23\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"233.903125\" xlink:href=\"#m99c08219a3\" y=\"134.1015625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(21.878124999999997 136.8609375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-31\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_24\">\n",
       "      <path clip-path=\"url(#pd0488643bb)\" d=\"M 38.603125 116.664062 \n",
       "L 233.903125 116.664062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_25\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"38.603125\" xlink:href=\"#m92065022d4\" y=\"116.6640625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_26\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"233.903125\" xlink:href=\"#m99c08219a3\" y=\"116.6640625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 20 -->\n",
       "      <defs>\n",
       "       <path d=\"M 19.1875 8.296875 \n",
       "L 53.609375 8.296875 \n",
       "L 53.609375 0 \n",
       "L 7.328125 0 \n",
       "L 7.328125 8.296875 \n",
       "Q 12.9375 14.109375 22.625 23.890625 \n",
       "Q 32.328125 33.6875 34.8125 36.53125 \n",
       "Q 39.546875 41.84375 41.421875 45.53125 \n",
       "Q 43.3125 49.21875 43.3125 52.78125 \n",
       "Q 43.3125 58.59375 39.234375 62.25 \n",
       "Q 35.15625 65.921875 28.609375 65.921875 \n",
       "Q 23.96875 65.921875 18.8125 64.3125 \n",
       "Q 13.671875 62.703125 7.8125 59.421875 \n",
       "L 7.8125 69.390625 \n",
       "Q 13.765625 71.78125 18.9375 73 \n",
       "Q 24.125 74.21875 28.421875 74.21875 \n",
       "Q 39.75 74.21875 46.484375 68.546875 \n",
       "Q 53.21875 62.890625 53.21875 53.421875 \n",
       "Q 53.21875 48.921875 51.53125 44.890625 \n",
       "Q 49.859375 40.875 45.40625 35.40625 \n",
       "Q 44.1875 33.984375 37.640625 27.21875 \n",
       "Q 31.109375 20.453125 19.1875 8.296875 \n",
       "\" id=\"BitstreamVeraSans-Roman-32\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(21.878124999999997 119.4234375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-32\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_27\">\n",
       "      <path clip-path=\"url(#pd0488643bb)\" d=\"M 38.603125 99.226562 \n",
       "L 233.903125 99.226562 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_28\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"38.603125\" xlink:href=\"#m92065022d4\" y=\"99.2265625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_29\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"233.903125\" xlink:href=\"#m99c08219a3\" y=\"99.2265625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 30 -->\n",
       "      <defs>\n",
       "       <path d=\"M 40.578125 39.3125 \n",
       "Q 47.65625 37.796875 51.625 33 \n",
       "Q 55.609375 28.21875 55.609375 21.1875 \n",
       "Q 55.609375 10.40625 48.1875 4.484375 \n",
       "Q 40.765625 -1.421875 27.09375 -1.421875 \n",
       "Q 22.515625 -1.421875 17.65625 -0.515625 \n",
       "Q 12.796875 0.390625 7.625 2.203125 \n",
       "L 7.625 11.71875 \n",
       "Q 11.71875 9.328125 16.59375 8.109375 \n",
       "Q 21.484375 6.890625 26.8125 6.890625 \n",
       "Q 36.078125 6.890625 40.9375 10.546875 \n",
       "Q 45.796875 14.203125 45.796875 21.1875 \n",
       "Q 45.796875 27.640625 41.28125 31.265625 \n",
       "Q 36.765625 34.90625 28.71875 34.90625 \n",
       "L 20.21875 34.90625 \n",
       "L 20.21875 43.015625 \n",
       "L 29.109375 43.015625 \n",
       "Q 36.375 43.015625 40.234375 45.921875 \n",
       "Q 44.09375 48.828125 44.09375 54.296875 \n",
       "Q 44.09375 59.90625 40.109375 62.90625 \n",
       "Q 36.140625 65.921875 28.71875 65.921875 \n",
       "Q 24.65625 65.921875 20.015625 65.03125 \n",
       "Q 15.375 64.15625 9.8125 62.3125 \n",
       "L 9.8125 71.09375 \n",
       "Q 15.4375 72.65625 20.34375 73.4375 \n",
       "Q 25.25 74.21875 29.59375 74.21875 \n",
       "Q 40.828125 74.21875 47.359375 69.109375 \n",
       "Q 53.90625 64.015625 53.90625 55.328125 \n",
       "Q 53.90625 49.265625 50.4375 45.09375 \n",
       "Q 46.96875 40.921875 40.578125 39.3125 \n",
       "\" id=\"BitstreamVeraSans-Roman-33\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(21.878124999999997 101.9859375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-33\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_30\">\n",
       "      <path clip-path=\"url(#pd0488643bb)\" d=\"M 38.603125 81.789062 \n",
       "L 233.903125 81.789062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_31\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"38.603125\" xlink:href=\"#m92065022d4\" y=\"81.7890625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_32\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"233.903125\" xlink:href=\"#m99c08219a3\" y=\"81.7890625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 40 -->\n",
       "      <defs>\n",
       "       <path d=\"M 37.796875 64.3125 \n",
       "L 12.890625 25.390625 \n",
       "L 37.796875 25.390625 \n",
       "z\n",
       "M 35.203125 72.90625 \n",
       "L 47.609375 72.90625 \n",
       "L 47.609375 25.390625 \n",
       "L 58.015625 25.390625 \n",
       "L 58.015625 17.1875 \n",
       "L 47.609375 17.1875 \n",
       "L 47.609375 0 \n",
       "L 37.796875 0 \n",
       "L 37.796875 17.1875 \n",
       "L 4.890625 17.1875 \n",
       "L 4.890625 26.703125 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-34\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(21.878124999999997 84.5484375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-34\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_33\">\n",
       "      <path clip-path=\"url(#pd0488643bb)\" d=\"M 38.603125 64.351562 \n",
       "L 233.903125 64.351562 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_34\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"38.603125\" xlink:href=\"#m92065022d4\" y=\"64.3515625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_35\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"233.903125\" xlink:href=\"#m99c08219a3\" y=\"64.3515625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 50 -->\n",
       "      <g transform=\"translate(21.878124999999997 67.1109375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-35\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_7\">\n",
       "     <g id=\"line2d_36\">\n",
       "      <path clip-path=\"url(#pd0488643bb)\" d=\"M 38.603125 46.914062 \n",
       "L 233.903125 46.914062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_37\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"38.603125\" xlink:href=\"#m92065022d4\" y=\"46.914062499999986\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_38\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"233.903125\" xlink:href=\"#m99c08219a3\" y=\"46.914062499999986\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 60 -->\n",
       "      <defs>\n",
       "       <path d=\"M 33.015625 40.375 \n",
       "Q 26.375 40.375 22.484375 35.828125 \n",
       "Q 18.609375 31.296875 18.609375 23.390625 \n",
       "Q 18.609375 15.53125 22.484375 10.953125 \n",
       "Q 26.375 6.390625 33.015625 6.390625 \n",
       "Q 39.65625 6.390625 43.53125 10.953125 \n",
       "Q 47.40625 15.53125 47.40625 23.390625 \n",
       "Q 47.40625 31.296875 43.53125 35.828125 \n",
       "Q 39.65625 40.375 33.015625 40.375 \n",
       "M 52.59375 71.296875 \n",
       "L 52.59375 62.3125 \n",
       "Q 48.875 64.0625 45.09375 64.984375 \n",
       "Q 41.3125 65.921875 37.59375 65.921875 \n",
       "Q 27.828125 65.921875 22.671875 59.328125 \n",
       "Q 17.53125 52.734375 16.796875 39.40625 \n",
       "Q 19.671875 43.65625 24.015625 45.921875 \n",
       "Q 28.375 48.1875 33.59375 48.1875 \n",
       "Q 44.578125 48.1875 50.953125 41.515625 \n",
       "Q 57.328125 34.859375 57.328125 23.390625 \n",
       "Q 57.328125 12.15625 50.6875 5.359375 \n",
       "Q 44.046875 -1.421875 33.015625 -1.421875 \n",
       "Q 20.359375 -1.421875 13.671875 8.265625 \n",
       "Q 6.984375 17.96875 6.984375 36.375 \n",
       "Q 6.984375 53.65625 15.1875 63.9375 \n",
       "Q 23.390625 74.21875 37.203125 74.21875 \n",
       "Q 40.921875 74.21875 44.703125 73.484375 \n",
       "Q 48.484375 72.75 52.59375 71.296875 \n",
       "\" id=\"BitstreamVeraSans-Roman-36\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(21.878124999999997 49.67343749999998)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-36\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_8\">\n",
       "     <g id=\"line2d_39\">\n",
       "      <path clip-path=\"url(#pd0488643bb)\" d=\"M 38.603125 29.476562 \n",
       "L 233.903125 29.476562 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_40\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"38.603125\" xlink:href=\"#m92065022d4\" y=\"29.476562499999986\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_41\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"233.903125\" xlink:href=\"#m99c08219a3\" y=\"29.476562499999986\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_14\">\n",
       "      <!-- 70 -->\n",
       "      <defs>\n",
       "       <path d=\"M 8.203125 72.90625 \n",
       "L 55.078125 72.90625 \n",
       "L 55.078125 68.703125 \n",
       "L 28.609375 0 \n",
       "L 18.3125 0 \n",
       "L 43.21875 64.59375 \n",
       "L 8.203125 64.59375 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-37\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(21.878124999999997 32.23593749999998)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-37\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_9\">\n",
       "     <g id=\"line2d_42\">\n",
       "      <path clip-path=\"url(#pd0488643bb)\" d=\"M 38.603125 12.039062 \n",
       "L 233.903125 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_43\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"38.603125\" xlink:href=\"#m92065022d4\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_44\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"233.903125\" xlink:href=\"#m99c08219a3\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_15\">\n",
       "      <!-- 80 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 34.625 \n",
       "Q 24.75 34.625 20.71875 30.859375 \n",
       "Q 16.703125 27.09375 16.703125 20.515625 \n",
       "Q 16.703125 13.921875 20.71875 10.15625 \n",
       "Q 24.75 6.390625 31.78125 6.390625 \n",
       "Q 38.8125 6.390625 42.859375 10.171875 \n",
       "Q 46.921875 13.96875 46.921875 20.515625 \n",
       "Q 46.921875 27.09375 42.890625 30.859375 \n",
       "Q 38.875 34.625 31.78125 34.625 \n",
       "M 21.921875 38.8125 \n",
       "Q 15.578125 40.375 12.03125 44.71875 \n",
       "Q 8.5 49.078125 8.5 55.328125 \n",
       "Q 8.5 64.0625 14.71875 69.140625 \n",
       "Q 20.953125 74.21875 31.78125 74.21875 \n",
       "Q 42.671875 74.21875 48.875 69.140625 \n",
       "Q 55.078125 64.0625 55.078125 55.328125 \n",
       "Q 55.078125 49.078125 51.53125 44.71875 \n",
       "Q 48 40.375 41.703125 38.8125 \n",
       "Q 48.828125 37.15625 52.796875 32.3125 \n",
       "Q 56.78125 27.484375 56.78125 20.515625 \n",
       "Q 56.78125 9.90625 50.3125 4.234375 \n",
       "Q 43.84375 -1.421875 31.78125 -1.421875 \n",
       "Q 19.734375 -1.421875 13.25 4.234375 \n",
       "Q 6.78125 9.90625 6.78125 20.515625 \n",
       "Q 6.78125 27.484375 10.78125 32.3125 \n",
       "Q 14.796875 37.15625 21.921875 38.8125 \n",
       "M 18.3125 54.390625 \n",
       "Q 18.3125 48.734375 21.84375 45.5625 \n",
       "Q 25.390625 42.390625 31.78125 42.390625 \n",
       "Q 38.140625 42.390625 41.71875 45.5625 \n",
       "Q 45.3125 48.734375 45.3125 54.390625 \n",
       "Q 45.3125 60.0625 41.71875 63.234375 \n",
       "Q 38.140625 66.40625 31.78125 66.40625 \n",
       "Q 25.390625 66.40625 21.84375 63.234375 \n",
       "Q 18.3125 60.0625 18.3125 54.390625 \n",
       "\" id=\"BitstreamVeraSans-Roman-38\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(21.878124999999997 14.798437500000006)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-38\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_16\">\n",
       "     <!-- f(x) -->\n",
       "     <defs>\n",
       "      <path d=\"M 37.109375 75.984375 \n",
       "L 37.109375 68.5 \n",
       "L 28.515625 68.5 \n",
       "Q 23.6875 68.5 21.796875 66.546875 \n",
       "Q 19.921875 64.59375 19.921875 59.515625 \n",
       "L 19.921875 54.6875 \n",
       "L 34.71875 54.6875 \n",
       "L 34.71875 47.703125 \n",
       "L 19.921875 47.703125 \n",
       "L 19.921875 0 \n",
       "L 10.890625 0 \n",
       "L 10.890625 47.703125 \n",
       "L 2.296875 47.703125 \n",
       "L 2.296875 54.6875 \n",
       "L 10.890625 54.6875 \n",
       "L 10.890625 58.5 \n",
       "Q 10.890625 67.625 15.140625 71.796875 \n",
       "Q 19.390625 75.984375 28.609375 75.984375 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-66\"/>\n",
       "      <path d=\"M 31 75.875 \n",
       "Q 24.46875 64.65625 21.28125 53.65625 \n",
       "Q 18.109375 42.671875 18.109375 31.390625 \n",
       "Q 18.109375 20.125 21.3125 9.0625 \n",
       "Q 24.515625 -2 31 -13.1875 \n",
       "L 23.1875 -13.1875 \n",
       "Q 15.875 -1.703125 12.234375 9.375 \n",
       "Q 8.59375 20.453125 8.59375 31.390625 \n",
       "Q 8.59375 42.28125 12.203125 53.3125 \n",
       "Q 15.828125 64.359375 23.1875 75.875 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-28\"/>\n",
       "      <path d=\"M 8.015625 75.875 \n",
       "L 15.828125 75.875 \n",
       "Q 23.140625 64.359375 26.78125 53.3125 \n",
       "Q 30.421875 42.28125 30.421875 31.390625 \n",
       "Q 30.421875 20.453125 26.78125 9.375 \n",
       "Q 23.140625 -1.703125 15.828125 -13.1875 \n",
       "L 8.015625 -13.1875 \n",
       "Q 14.5 -2 17.703125 9.0625 \n",
       "Q 20.90625 20.125 20.90625 31.390625 \n",
       "Q 20.90625 42.671875 17.703125 53.65625 \n",
       "Q 14.5 64.65625 8.015625 75.875 \n",
       "\" id=\"BitstreamVeraSans-Roman-29\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(14.798437499999997 90.41015625)rotate(-90.0)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#BitstreamVeraSans-Roman-66\"/>\n",
       "      <use x=\"35.205078125\" xlink:href=\"#BitstreamVeraSans-Roman-28\"/>\n",
       "      <use x=\"74.21875\" xlink:href=\"#BitstreamVeraSans-Roman-78\"/>\n",
       "      <use x=\"133.3984375\" xlink:href=\"#BitstreamVeraSans-Roman-29\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pd0488643bb\">\n",
       "   <rect height=\"139.5\" width=\"195.29999999999998\" x=\"38.603125\" y=\"12.0390625\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f234cdf7ef0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "c = 0.5\n",
    "def f(x):     return math.cosh(c * x)        # objective\n",
    "def gradf(x): return c * math.sinh(c * x)    # derivative\n",
    "def hessf(x): return c**2 * math.cosh(c * x) # hessian\n",
    "\n",
    "# hide learning rate for now\n",
    "def newton(eta = 1):\n",
    "    x = 10\n",
    "    results = [x]\n",
    "    for i in range(10):\n",
    "        x -= eta * gradf(x) / hessf(x)\n",
    "        results.append(x)\n",
    "    print('epoch 10, x:', x)\n",
    "    return results\n",
    "\n",
    "show_trace(newton())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's see what happens when we have a nonconvex function, such as $f(x) = x \\cos(c x)$. After all, note that in Newton's method we end up dividing by the Hessian. This means that if the second derivative is negative we would walk into the direction of increasing $f$. That is a fatal flaw of the algorithm. Let's see what happens in practice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 10, x: 26.83413291324767\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\n",
       "<svg height=\"187pt\" version=\"1.1\" viewBox=\"0 0 255 187\" width=\"255pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;stroke-miterlimit:100000;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M -0 187.095313 \n",
       "L 255.845313 187.095313 \n",
       "L 255.845313 0 \n",
       "L -0 0 \n",
       "L -0 187.095313 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 46.982813 151.539062 \n",
       "L 242.282813 151.539062 \n",
       "L 242.282813 12.039062 \n",
       "L 46.982813 12.039062 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_1\">\n",
       "    <path clip-path=\"url(#pf143bf2fbf)\" d=\"M 54.198802 137.649326 \n",
       "L 55.045102 140.639103 \n",
       "L 55.761202 142.435152 \n",
       "L 56.379652 143.437155 \n",
       "L 56.900452 143.885014 \n",
       "L 57.323602 143.983397 \n",
       "L 57.746752 143.845717 \n",
       "L 58.202452 143.436656 \n",
       "L 58.723252 142.643854 \n",
       "L 59.341702 141.263468 \n",
       "L 60.057802 139.092636 \n",
       "L 60.904102 135.779996 \n",
       "L 61.913152 130.861688 \n",
       "L 63.117502 123.788285 \n",
       "L 64.582252 113.774605 \n",
       "L 66.665452 97.823971 \n",
       "L 70.831852 65.651448 \n",
       "L 72.459352 54.986476 \n",
       "L 73.761352 47.833279 \n",
       "L 74.868052 42.895339 \n",
       "L 75.844552 39.494279 \n",
       "L 76.690852 37.309147 \n",
       "L 77.406952 36.026893 \n",
       "L 78.025402 35.340794 \n",
       "L 78.546202 35.065057 \n",
       "L 79.034452 35.054994 \n",
       "L 79.522702 35.281891 \n",
       "L 80.043502 35.779879 \n",
       "L 80.661952 36.704753 \n",
       "L 81.378052 38.209734 \n",
       "L 82.224352 40.553274 \n",
       "L 83.200852 43.952121 \n",
       "L 84.372652 48.886909 \n",
       "L 85.837402 56.104692 \n",
       "L 87.888052 67.466092 \n",
       "L 92.087002 90.955555 \n",
       "L 93.681952 98.518232 \n",
       "L 95.016502 103.820191 \n",
       "L 96.155752 107.475451 \n",
       "L 97.132252 109.912864 \n",
       "L 97.978552 111.483005 \n",
       "L 98.727202 112.445014 \n",
       "L 99.378202 112.955594 \n",
       "L 99.964102 113.158587 \n",
       "L 100.550002 113.123061 \n",
       "L 101.135902 112.855226 \n",
       "L 101.786902 112.294869 \n",
       "L 102.535552 111.325317 \n",
       "L 103.381852 109.839416 \n",
       "L 104.390902 107.580911 \n",
       "L 105.595252 104.294304 \n",
       "L 107.092552 99.519636 \n",
       "L 109.436152 91.188075 \n",
       "L 112.593502 80.098595 \n",
       "L 114.253552 75.081346 \n",
       "L 115.588102 71.706991 \n",
       "L 116.759902 69.314914 \n",
       "L 117.801502 67.67342 \n",
       "L 118.712902 66.623816 \n",
       "L 119.559202 65.973349 \n",
       "L 120.340402 65.645145 \n",
       "L 121.089052 65.567653 \n",
       "L 121.837702 65.711048 \n",
       "L 122.651452 66.101115 \n",
       "L 123.562852 66.801655 \n",
       "L 124.604452 67.899992 \n",
       "L 125.873902 69.586237 \n",
       "L 127.566502 72.240443 \n",
       "L 132.416452 80.074185 \n",
       "L 133.848652 81.862485 \n",
       "L 135.085552 83.067622 \n",
       "L 136.192252 83.849721 \n",
       "L 137.233852 84.321052 \n",
       "L 138.242902 84.535634 \n",
       "L 139.251952 84.524119 \n",
       "L 140.326102 84.286463 \n",
       "L 141.530452 83.785255 \n",
       "L 143.027752 82.90472 \n",
       "L 148.431052 79.475977 \n",
       "L 149.602852 79.119035 \n",
       "L 150.677002 79.021835 \n",
       "L 151.686052 79.15716 \n",
       "L 152.662552 79.511773 \n",
       "L 153.671602 80.115458 \n",
       "L 154.745752 81.019723 \n",
       "L 155.950102 82.335524 \n",
       "L 157.349752 84.214412 \n",
       "L 159.140002 87.02276 \n",
       "L 163.924852 94.740202 \n",
       "L 165.226852 96.310937 \n",
       "L 166.268452 97.24242 \n",
       "L 167.179852 97.77315 \n",
       "L 167.993602 97.995336 \n",
       "L 168.742252 97.97296 \n",
       "L 169.490902 97.721112 \n",
       "L 170.239552 97.231217 \n",
       "L 171.053302 96.42277 \n",
       "L 171.932152 95.224943 \n",
       "L 172.908652 93.504292 \n",
       "L 174.015352 91.079447 \n",
       "L 175.284802 87.729798 \n",
       "L 176.782102 83.120498 \n",
       "L 178.800202 76.126504 \n",
       "L 183.129352 60.934064 \n",
       "L 184.594102 56.709976 \n",
       "L 185.765902 53.976408 \n",
       "L 186.742402 52.234138 \n",
       "L 187.556152 51.202507 \n",
       "L 188.272252 50.63676 \n",
       "L 188.890702 50.419795 \n",
       "L 189.476602 50.454973 \n",
       "L 190.062502 50.730179 \n",
       "L 190.680952 51.285292 \n",
       "L 191.364502 52.217543 \n",
       "L 192.145702 53.691894 \n",
       "L 193.024552 55.863526 \n",
       "L 194.033602 59.00204 \n",
       "L 195.172852 63.318448 \n",
       "L 196.507402 69.296227 \n",
       "L 198.167452 77.835897 \n",
       "L 200.738902 92.428826 \n",
       "L 203.668402 108.722615 \n",
       "L 205.230802 116.200468 \n",
       "L 206.467702 121.11345 \n",
       "L 207.509302 124.393907 \n",
       "L 208.388152 126.468406 \n",
       "L 209.104252 127.649552 \n",
       "L 209.722702 128.281791 \n",
       "L 210.243502 128.52581 \n",
       "L 210.731752 128.509714 \n",
       "L 211.220002 128.253176 \n",
       "L 211.740802 127.712124 \n",
       "L 212.326702 126.772344 \n",
       "L 213.010252 125.234473 \n",
       "L 213.791452 122.902979 \n",
       "L 214.702852 119.432407 \n",
       "L 215.744452 114.529653 \n",
       "L 216.948802 107.728441 \n",
       "L 218.381002 98.297783 \n",
       "L 220.268902 84.243471 \n",
       "L 225.802402 42.029339 \n",
       "L 227.202052 33.532572 \n",
       "L 228.341302 27.886685 \n",
       "L 229.317802 24.113354 \n",
       "L 230.131552 21.799035 \n",
       "L 230.815102 20.476275 \n",
       "L 231.368452 19.837723 \n",
       "L 231.824152 19.609462 \n",
       "L 232.247302 19.641861 \n",
       "L 232.670452 19.911773 \n",
       "L 233.158702 20.519907 \n",
       "L 233.712052 21.593725 \n",
       "L 234.363052 23.377292 \n",
       "L 235.046602 25.846591 \n",
       "L 235.046602 25.846591 \n",
       "\" style=\"fill:none;stroke:#0000ff;stroke-linecap:square;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_2\">\n",
       "    <path clip-path=\"url(#pf143bf2fbf)\" d=\"M 177.182813 81.789062 \n",
       "L 160.907813 90.009179 \n",
       "L 225.991145 40.796173 \n",
       "L 235.066823 25.928799 \n",
       "L 231.867136 19.60197 \n",
       "L 231.978064 19.593878 \n",
       "L 231.977915 19.593878 \n",
       "L 231.977915 19.593878 \n",
       "L 231.977915 19.593878 \n",
       "L 231.977915 19.593878 \n",
       "L 231.977915 19.593878 \n",
       "\" style=\"fill:none;stroke:#008000;stroke-linecap:square;\"/>\n",
       "    <defs>\n",
       "     <path d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" id=\"m02e84655e9\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#pf143bf2fbf)\">\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"177.1828125\" xlink:href=\"#m02e84655e9\" y=\"81.7890625\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"160.9078125\" xlink:href=\"#m02e84655e9\" y=\"90.00917883129362\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"225.99114525360153\" xlink:href=\"#m02e84655e9\" y=\"40.79617323777093\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"235.06682252063317\" xlink:href=\"#m02e84655e9\" y=\"25.928798551634443\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"231.86713565421297\" xlink:href=\"#m02e84655e9\" y=\"19.601969793847424\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"231.97806388755873\" xlink:href=\"#m02e84655e9\" y=\"19.59387788617324\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"231.97791513287137\" xlink:href=\"#m02e84655e9\" y=\"19.593877871570008\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"231.97791513262118\" xlink:href=\"#m02e84655e9\" y=\"19.593877871570008\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"231.97791513262118\" xlink:href=\"#m02e84655e9\" y=\"19.593877871570008\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"231.97791513262118\" xlink:href=\"#m02e84655e9\" y=\"19.593877871570008\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"231.97791513262118\" xlink:href=\"#m02e84655e9\" y=\"19.593877871570008\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 242.282813 151.539062 \n",
       "L 242.282813 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 46.982812 12.039062 \n",
       "L 242.282813 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 46.982813 151.539062 \n",
       "L 46.982813 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 46.982812 151.539062 \n",
       "L 242.282813 151.539062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path clip-path=\"url(#pf143bf2fbf)\" d=\"M 46.982812 151.539062 \n",
       "L 46.982812 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 -4 \n",
       "\" id=\"m267ee26b35\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"46.982812499999994\" xlink:href=\"#m267ee26b35\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_5\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 4 \n",
       "\" id=\"mf67cc4f398\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"46.982812499999994\" xlink:href=\"#mf67cc4f398\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −30 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "\" id=\"BitstreamVeraSans-Roman-30\"/>\n",
       "       <path d=\"M 10.59375 35.5 \n",
       "L 73.1875 35.5 \n",
       "L 73.1875 27.203125 \n",
       "L 10.59375 27.203125 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-2212\"/>\n",
       "       <path d=\"M 40.578125 39.3125 \n",
       "Q 47.65625 37.796875 51.625 33 \n",
       "Q 55.609375 28.21875 55.609375 21.1875 \n",
       "Q 55.609375 10.40625 48.1875 4.484375 \n",
       "Q 40.765625 -1.421875 27.09375 -1.421875 \n",
       "Q 22.515625 -1.421875 17.65625 -0.515625 \n",
       "Q 12.796875 0.390625 7.625 2.203125 \n",
       "L 7.625 11.71875 \n",
       "Q 11.71875 9.328125 16.59375 8.109375 \n",
       "Q 21.484375 6.890625 26.8125 6.890625 \n",
       "Q 36.078125 6.890625 40.9375 10.546875 \n",
       "Q 45.796875 14.203125 45.796875 21.1875 \n",
       "Q 45.796875 27.640625 41.28125 31.265625 \n",
       "Q 36.765625 34.90625 28.71875 34.90625 \n",
       "L 20.21875 34.90625 \n",
       "L 20.21875 43.015625 \n",
       "L 29.109375 43.015625 \n",
       "Q 36.375 43.015625 40.234375 45.921875 \n",
       "Q 44.09375 48.828125 44.09375 54.296875 \n",
       "Q 44.09375 59.90625 40.109375 62.90625 \n",
       "Q 36.140625 65.921875 28.71875 65.921875 \n",
       "Q 24.65625 65.921875 20.015625 65.03125 \n",
       "Q 15.375 64.15625 9.8125 62.3125 \n",
       "L 9.8125 71.09375 \n",
       "Q 15.4375 72.65625 20.34375 73.4375 \n",
       "Q 25.25 74.21875 29.59375 74.21875 \n",
       "Q 40.828125 74.21875 47.359375 69.109375 \n",
       "Q 53.90625 64.015625 53.90625 55.328125 \n",
       "Q 53.90625 49.265625 50.4375 45.09375 \n",
       "Q 46.96875 40.921875 40.578125 39.3125 \n",
       "\" id=\"BitstreamVeraSans-Roman-33\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(36.430468749999996 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-33\"/>\n",
       "       <use x=\"147.412109375\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <path clip-path=\"url(#pf143bf2fbf)\" d=\"M 79.532812 151.539062 \n",
       "L 79.532812 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"79.53281249999999\" xlink:href=\"#m267ee26b35\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"79.53281249999999\" xlink:href=\"#mf67cc4f398\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −20 -->\n",
       "      <defs>\n",
       "       <path d=\"M 19.1875 8.296875 \n",
       "L 53.609375 8.296875 \n",
       "L 53.609375 0 \n",
       "L 7.328125 0 \n",
       "L 7.328125 8.296875 \n",
       "Q 12.9375 14.109375 22.625 23.890625 \n",
       "Q 32.328125 33.6875 34.8125 36.53125 \n",
       "Q 39.546875 41.84375 41.421875 45.53125 \n",
       "Q 43.3125 49.21875 43.3125 52.78125 \n",
       "Q 43.3125 58.59375 39.234375 62.25 \n",
       "Q 35.15625 65.921875 28.609375 65.921875 \n",
       "Q 23.96875 65.921875 18.8125 64.3125 \n",
       "Q 13.671875 62.703125 7.8125 59.421875 \n",
       "L 7.8125 69.390625 \n",
       "Q 13.765625 71.78125 18.9375 73 \n",
       "Q 24.125 74.21875 28.421875 74.21875 \n",
       "Q 39.75 74.21875 46.484375 68.546875 \n",
       "Q 53.21875 62.890625 53.21875 53.421875 \n",
       "Q 53.21875 48.921875 51.53125 44.890625 \n",
       "Q 49.859375 40.875 45.40625 35.40625 \n",
       "Q 44.1875 33.984375 37.640625 27.21875 \n",
       "Q 31.109375 20.453125 19.1875 8.296875 \n",
       "\" id=\"BitstreamVeraSans-Roman-32\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(68.98046874999999 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-32\"/>\n",
       "       <use x=\"147.412109375\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path clip-path=\"url(#pf143bf2fbf)\" d=\"M 112.082812 151.539062 \n",
       "L 112.082812 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"112.08281249999999\" xlink:href=\"#m267ee26b35\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"112.08281249999999\" xlink:href=\"#mf67cc4f398\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- −10 -->\n",
       "      <defs>\n",
       "       <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-31\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(101.53046874999998 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-31\"/>\n",
       "       <use x=\"147.412109375\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <path clip-path=\"url(#pf143bf2fbf)\" d=\"M 144.632812 151.539062 \n",
       "L 144.632812 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"144.6328125\" xlink:href=\"#m267ee26b35\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"144.6328125\" xlink:href=\"#mf67cc4f398\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(141.4515625 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path clip-path=\"url(#pf143bf2fbf)\" d=\"M 177.182813 151.539062 \n",
       "L 177.182813 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"177.1828125\" xlink:href=\"#m267ee26b35\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_17\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"177.1828125\" xlink:href=\"#mf67cc4f398\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(170.8203125 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-31\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_18\">\n",
       "      <path clip-path=\"url(#pf143bf2fbf)\" d=\"M 209.732813 151.539062 \n",
       "L 209.732813 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_19\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"209.73281250000002\" xlink:href=\"#m267ee26b35\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"209.73281250000002\" xlink:href=\"#mf67cc4f398\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(203.3703125 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-32\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"line2d_21\">\n",
       "      <path clip-path=\"url(#pf143bf2fbf)\" d=\"M 242.282813 151.539062 \n",
       "L 242.282813 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"242.2828125\" xlink:href=\"#m267ee26b35\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_23\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"242.2828125\" xlink:href=\"#mf67cc4f398\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 30 -->\n",
       "      <g transform=\"translate(235.9203125 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-33\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_8\">\n",
       "     <!-- x -->\n",
       "     <defs>\n",
       "      <path d=\"M 54.890625 54.6875 \n",
       "L 35.109375 28.078125 \n",
       "L 55.90625 0 \n",
       "L 45.3125 0 \n",
       "L 29.390625 21.484375 \n",
       "L 13.484375 0 \n",
       "L 2.875 0 \n",
       "L 24.125 28.609375 \n",
       "L 4.6875 54.6875 \n",
       "L 15.28125 54.6875 \n",
       "L 29.78125 35.203125 \n",
       "L 44.28125 54.6875 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-78\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(141.6734375 177.815625)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#BitstreamVeraSans-Roman-78\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_24\">\n",
       "      <path clip-path=\"url(#pf143bf2fbf)\" d=\"M 46.982813 151.539062 \n",
       "L 242.282813 151.539062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_25\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 4 0 \n",
       "\" id=\"m587733297e\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"46.98281250000001\" xlink:href=\"#m587733297e\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_26\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -4 0 \n",
       "\" id=\"me39dbfc48b\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"242.28281250000003\" xlink:href=\"#me39dbfc48b\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- −30 -->\n",
       "      <g transform=\"translate(21.878125000000008 154.2984375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-33\"/>\n",
       "       <use x=\"147.412109375\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_27\">\n",
       "      <path clip-path=\"url(#pf143bf2fbf)\" d=\"M 46.982813 128.289062 \n",
       "L 242.282813 128.289062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_28\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"46.98281250000001\" xlink:href=\"#m587733297e\" y=\"128.2890625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_29\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"242.28281250000003\" xlink:href=\"#me39dbfc48b\" y=\"128.2890625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- −20 -->\n",
       "      <g transform=\"translate(21.878125000000008 131.0484375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-32\"/>\n",
       "       <use x=\"147.412109375\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_30\">\n",
       "      <path clip-path=\"url(#pf143bf2fbf)\" d=\"M 46.982813 105.039062 \n",
       "L 242.282813 105.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_31\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"46.98281250000001\" xlink:href=\"#m587733297e\" y=\"105.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_32\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"242.28281250000003\" xlink:href=\"#me39dbfc48b\" y=\"105.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- −10 -->\n",
       "      <g transform=\"translate(21.878125000000008 107.7984375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-31\"/>\n",
       "       <use x=\"147.412109375\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_33\">\n",
       "      <path clip-path=\"url(#pf143bf2fbf)\" d=\"M 46.982813 81.789062 \n",
       "L 242.282813 81.789062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_34\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"46.98281250000001\" xlink:href=\"#m587733297e\" y=\"81.7890625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_35\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"242.28281250000003\" xlink:href=\"#me39dbfc48b\" y=\"81.7890625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(36.62031250000001 84.5484375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_36\">\n",
       "      <path clip-path=\"url(#pf143bf2fbf)\" d=\"M 46.982813 58.539062 \n",
       "L 242.282813 58.539062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_37\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"46.98281250000001\" xlink:href=\"#m587733297e\" y=\"58.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_38\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"242.28281250000003\" xlink:href=\"#me39dbfc48b\" y=\"58.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(30.257812500000007 61.298437500000006)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-31\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_39\">\n",
       "      <path clip-path=\"url(#pf143bf2fbf)\" d=\"M 46.982813 35.289062 \n",
       "L 242.282813 35.289062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_40\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"46.98281250000001\" xlink:href=\"#m587733297e\" y=\"35.2890625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_41\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"242.28281250000003\" xlink:href=\"#me39dbfc48b\" y=\"35.2890625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_14\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(30.257812500000007 38.048437500000006)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-32\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_7\">\n",
       "     <g id=\"line2d_42\">\n",
       "      <path clip-path=\"url(#pf143bf2fbf)\" d=\"M 46.982813 12.039062 \n",
       "L 242.282813 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_43\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"46.98281250000001\" xlink:href=\"#m587733297e\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_44\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"242.28281250000003\" xlink:href=\"#me39dbfc48b\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_15\">\n",
       "      <!-- 30 -->\n",
       "      <g transform=\"translate(30.257812500000007 14.798437500000006)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-33\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_16\">\n",
       "     <!-- f(x) -->\n",
       "     <defs>\n",
       "      <path d=\"M 37.109375 75.984375 \n",
       "L 37.109375 68.5 \n",
       "L 28.515625 68.5 \n",
       "Q 23.6875 68.5 21.796875 66.546875 \n",
       "Q 19.921875 64.59375 19.921875 59.515625 \n",
       "L 19.921875 54.6875 \n",
       "L 34.71875 54.6875 \n",
       "L 34.71875 47.703125 \n",
       "L 19.921875 47.703125 \n",
       "L 19.921875 0 \n",
       "L 10.890625 0 \n",
       "L 10.890625 47.703125 \n",
       "L 2.296875 47.703125 \n",
       "L 2.296875 54.6875 \n",
       "L 10.890625 54.6875 \n",
       "L 10.890625 58.5 \n",
       "Q 10.890625 67.625 15.140625 71.796875 \n",
       "Q 19.390625 75.984375 28.609375 75.984375 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-66\"/>\n",
       "      <path d=\"M 31 75.875 \n",
       "Q 24.46875 64.65625 21.28125 53.65625 \n",
       "Q 18.109375 42.671875 18.109375 31.390625 \n",
       "Q 18.109375 20.125 21.3125 9.0625 \n",
       "Q 24.515625 -2 31 -13.1875 \n",
       "L 23.1875 -13.1875 \n",
       "Q 15.875 -1.703125 12.234375 9.375 \n",
       "Q 8.59375 20.453125 8.59375 31.390625 \n",
       "Q 8.59375 42.28125 12.203125 53.3125 \n",
       "Q 15.828125 64.359375 23.1875 75.875 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-28\"/>\n",
       "      <path d=\"M 8.015625 75.875 \n",
       "L 15.828125 75.875 \n",
       "Q 23.140625 64.359375 26.78125 53.3125 \n",
       "Q 30.421875 42.28125 30.421875 31.390625 \n",
       "Q 30.421875 20.453125 26.78125 9.375 \n",
       "Q 23.140625 -1.703125 15.828125 -13.1875 \n",
       "L 8.015625 -13.1875 \n",
       "Q 14.5 -2 17.703125 9.0625 \n",
       "Q 20.90625 20.125 20.90625 31.390625 \n",
       "Q 20.90625 42.671875 17.703125 53.65625 \n",
       "Q 14.5 64.65625 8.015625 75.875 \n",
       "\" id=\"BitstreamVeraSans-Roman-29\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(14.798437500000007 90.41015625)rotate(-90.0)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#BitstreamVeraSans-Roman-66\"/>\n",
       "      <use x=\"35.205078125\" xlink:href=\"#BitstreamVeraSans-Roman-28\"/>\n",
       "      <use x=\"74.21875\" xlink:href=\"#BitstreamVeraSans-Roman-78\"/>\n",
       "      <use x=\"133.3984375\" xlink:href=\"#BitstreamVeraSans-Roman-29\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pf143bf2fbf\">\n",
       "   <rect height=\"139.5\" width=\"195.3\" x=\"46.98281250000001\" y=\"12.0390625\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f234cc5fb70>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "c = 0.15 * math.pi\n",
    "def f(x):     return x*math.cos(c * x) \n",
    "def gradf(x): return math.cos(c * x) - c * x * math.sin(c * x)\n",
    "def hessf(x): return - 2 * c * math.sin(c * x) - x * c**2 * math.cos(c * x)\n",
    "\n",
    "show_trace(newton())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This went spectacularly wrong. How can we fix it? One way would be to 'fix' the Hessian by taking its absolute value instead. Another strategy is to bring back the learning rate. This seems to defeat the purpose, but not quite. Having second order information allows us to be cautious whenever the curvature is large and to take longer steps whenever the objective is flat. Let's see how this works with a slightly smaller learning rate, say $\\eta = 0.5$. As we can see, we have quite an efficient algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 10, x: 7.269860168684531\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Created with matplotlib (http://matplotlib.org/) -->\n",
       "<svg height=\"187pt\" version=\"1.1\" viewBox=\"0 0 249 187\" width=\"249pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       " <defs>\n",
       "  <style type=\"text/css\">\n",
       "*{stroke-linecap:butt;stroke-linejoin:round;stroke-miterlimit:100000;}\n",
       "  </style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 187.095313 \n",
       "L 249.482813 187.095313 \n",
       "L 249.482813 0 \n",
       "L 0 0 \n",
       "L 0 187.095313 \n",
       "z\n",
       "\" style=\"fill:none;\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 40.620313 151.539062 \n",
       "L 235.920313 151.539062 \n",
       "L 235.920313 12.039062 \n",
       "L 40.620313 12.039062 \n",
       "z\n",
       "\" style=\"fill:#ffffff;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_1\">\n",
       "    <path clip-path=\"url(#pa6d8f14215)\" d=\"M 40.620313 81.789062 \n",
       "L 43.745112 69.110398 \n",
       "L 46.479312 58.923998 \n",
       "L 48.920562 50.681972 \n",
       "L 51.068862 44.173348 \n",
       "L 53.119512 38.661022 \n",
       "L 54.974862 34.293289 \n",
       "L 56.732562 30.716479 \n",
       "L 58.392612 27.849479 \n",
       "L 59.857362 25.735929 \n",
       "L 61.322112 24.013016 \n",
       "L 62.689212 22.755575 \n",
       "L 63.958662 21.887863 \n",
       "L 65.130462 21.339458 \n",
       "L 66.302262 21.029048 \n",
       "L 67.474062 20.951365 \n",
       "L 68.645862 21.100276 \n",
       "L 69.817662 21.46882 \n",
       "L 71.087112 22.106926 \n",
       "L 72.454212 23.059773 \n",
       "L 73.918962 24.369994 \n",
       "L 75.481362 26.07579 \n",
       "L 77.239062 28.343927 \n",
       "L 79.192062 31.252214 \n",
       "L 81.340362 34.85859 \n",
       "L 83.879262 39.567587 \n",
       "L 87.101712 46.04456 \n",
       "L 93.741912 60.065789 \n",
       "L 97.843212 68.396008 \n",
       "L 100.772712 73.871076 \n",
       "L 103.311612 78.169047 \n",
       "L 105.557562 81.559667 \n",
       "L 107.608212 84.279816 \n",
       "L 109.561212 86.513105 \n",
       "L 111.416562 88.297271 \n",
       "L 113.174262 89.676505 \n",
       "L 114.834312 90.698494 \n",
       "L 116.396712 91.412027 \n",
       "L 117.959112 91.888002 \n",
       "L 119.521512 92.132032 \n",
       "L 121.083912 92.152044 \n",
       "L 122.646312 91.958175 \n",
       "L 124.306362 91.531555 \n",
       "L 126.064062 90.850782 \n",
       "L 128.017062 89.845767 \n",
       "L 130.165362 88.479116 \n",
       "L 132.704262 86.580546 \n",
       "L 135.926712 83.868194 \n",
       "L 144.715212 76.310767 \n",
       "L 147.254112 74.509903 \n",
       "L 149.402412 73.249762 \n",
       "L 151.355412 72.358836 \n",
       "L 153.113112 71.793346 \n",
       "L 154.773162 71.485255 \n",
       "L 156.335562 71.41032 \n",
       "L 157.897962 71.555235 \n",
       "L 159.460362 71.928905 \n",
       "L 161.022762 72.537967 \n",
       "L 162.585162 73.3867 \n",
       "L 164.245212 74.553121 \n",
       "L 165.905262 75.991337 \n",
       "L 167.662962 77.805876 \n",
       "L 169.518312 80.037207 \n",
       "L 171.568962 82.862958 \n",
       "L 173.814912 86.359425 \n",
       "L 176.256162 90.584305 \n",
       "L 179.088012 95.947001 \n",
       "L 182.701062 103.307888 \n",
       "L 193.247262 125.133946 \n",
       "L 195.883812 129.908888 \n",
       "L 198.129762 133.534956 \n",
       "L 200.082762 136.285645 \n",
       "L 201.840462 138.393513 \n",
       "L 203.402862 139.943053 \n",
       "L 204.867612 141.095779 \n",
       "L 206.234712 141.892738 \n",
       "L 207.504162 142.379038 \n",
       "L 208.675962 142.601973 \n",
       "L 209.847762 142.600545 \n",
       "L 211.019562 142.368339 \n",
       "L 212.191362 141.899796 \n",
       "L 213.363162 141.190248 \n",
       "L 214.632612 140.145287 \n",
       "L 215.902062 138.809619 \n",
       "L 217.269162 137.043688 \n",
       "L 218.733912 134.773849 \n",
       "L 220.296312 131.923581 \n",
       "L 221.956362 128.415342 \n",
       "L 223.714062 124.172995 \n",
       "L 225.667062 118.843442 \n",
       "L 227.717712 112.583246 \n",
       "L 229.963662 105.000028 \n",
       "L 232.502562 95.603077 \n",
       "L 235.334412 84.239114 \n",
       "L 235.822662 82.199512 \n",
       "L 235.822662 82.199512 \n",
       "\" style=\"fill:none;stroke:#0000ff;stroke-linecap:square;\"/>\n",
       "   </g>\n",
       "   <g id=\"line2d_2\">\n",
       "    <path clip-path=\"url(#pa6d8f14215)\" d=\"M 235.920313 81.789063 \n",
       "L 211.507813 142.202123 \n",
       "L 210.405978 142.519072 \n",
       "L 209.838021 142.6015 \n",
       "L 209.548879 142.622593 \n",
       "L 209.402885 142.627934 \n",
       "L 209.329515 142.629278 \n",
       "L 209.292734 142.629615 \n",
       "L 209.274319 142.6297 \n",
       "L 209.265105 142.629721 \n",
       "L 209.260497 142.629726 \n",
       "\" style=\"fill:none;stroke:#008000;stroke-linecap:square;\"/>\n",
       "    <defs>\n",
       "     <path d=\"M 0 3 \n",
       "C 0.795609 3 1.55874 2.683901 2.12132 2.12132 \n",
       "C 2.683901 1.55874 3 0.795609 3 0 \n",
       "C 3 -0.795609 2.683901 -1.55874 2.12132 -2.12132 \n",
       "C 1.55874 -2.683901 0.795609 -3 0 -3 \n",
       "C -0.795609 -3 -1.55874 -2.683901 -2.12132 -2.12132 \n",
       "C -2.683901 -1.55874 -3 -0.795609 -3 0 \n",
       "C -3 0.795609 -2.683901 1.55874 -2.12132 2.12132 \n",
       "C -1.55874 2.683901 -0.795609 3 0 3 \n",
       "z\n",
       "\" id=\"mdf08acd7f3\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "    </defs>\n",
       "    <g clip-path=\"url(#pa6d8f14215)\">\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"235.92031250000002\" xlink:href=\"#mdf08acd7f3\" y=\"81.78906250000001\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"211.50781250000006\" xlink:href=\"#mdf08acd7f3\" y=\"142.20212255562086\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"210.40597755576317\" xlink:href=\"#mdf08acd7f3\" y=\"142.51907231662437\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"209.83802074798524\" xlink:href=\"#mdf08acd7f3\" y=\"142.60149964141493\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"209.54887901728105\" xlink:href=\"#mdf08acd7f3\" y=\"142.62259328249706\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"209.40288527588513\" xlink:href=\"#mdf08acd7f3\" y=\"142.6279340545916\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"209.32951468581427\" xlink:href=\"#mdf08acd7f3\" y=\"142.62927811435782\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"209.29273360566083\" xlink:href=\"#mdf08acd7f3\" y=\"142.6296152669484\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"209.2743188179012\" xlink:href=\"#mdf08acd7f3\" y=\"142.6296996991804\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"209.26510532397418\" xlink:href=\"#mdf08acd7f3\" y=\"142.62972082536783\"/>\n",
       "     <use style=\"fill:#008000;stroke:#000000;stroke-width:0.5;\" x=\"209.26049704720447\" xlink:href=\"#mdf08acd7f3\" y=\"142.62972610918837\"/>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 235.920313 151.539062 \n",
       "L 235.920313 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 40.620313 12.039062 \n",
       "L 235.920313 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 40.620313 151.539062 \n",
       "L 40.620313 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 40.620313 151.539062 \n",
       "L 235.920313 151.539062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-linecap:square;stroke-linejoin:miter;\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <path clip-path=\"url(#pa6d8f14215)\" d=\"M 40.620313 151.539062 \n",
       "L 40.620313 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_4\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 -4 \n",
       "\" id=\"ma32d63e438\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"40.62031250000001\" xlink:href=\"#ma32d63e438\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_5\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 0 4 \n",
       "\" id=\"mff9bb9708c\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"40.62031250000001\" xlink:href=\"#mff9bb9708c\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- −10 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 66.40625 \n",
       "Q 24.171875 66.40625 20.328125 58.90625 \n",
       "Q 16.5 51.421875 16.5 36.375 \n",
       "Q 16.5 21.390625 20.328125 13.890625 \n",
       "Q 24.171875 6.390625 31.78125 6.390625 \n",
       "Q 39.453125 6.390625 43.28125 13.890625 \n",
       "Q 47.125 21.390625 47.125 36.375 \n",
       "Q 47.125 51.421875 43.28125 58.90625 \n",
       "Q 39.453125 66.40625 31.78125 66.40625 \n",
       "M 31.78125 74.21875 \n",
       "Q 44.046875 74.21875 50.515625 64.515625 \n",
       "Q 56.984375 54.828125 56.984375 36.375 \n",
       "Q 56.984375 17.96875 50.515625 8.265625 \n",
       "Q 44.046875 -1.421875 31.78125 -1.421875 \n",
       "Q 19.53125 -1.421875 13.0625 8.265625 \n",
       "Q 6.59375 17.96875 6.59375 36.375 \n",
       "Q 6.59375 54.828125 13.0625 64.515625 \n",
       "Q 19.53125 74.21875 31.78125 74.21875 \n",
       "\" id=\"BitstreamVeraSans-Roman-30\"/>\n",
       "       <path d=\"M 12.40625 8.296875 \n",
       "L 28.515625 8.296875 \n",
       "L 28.515625 63.921875 \n",
       "L 10.984375 60.40625 \n",
       "L 10.984375 69.390625 \n",
       "L 28.421875 72.90625 \n",
       "L 38.28125 72.90625 \n",
       "L 38.28125 8.296875 \n",
       "L 54.390625 8.296875 \n",
       "L 54.390625 0 \n",
       "L 12.40625 0 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-31\"/>\n",
       "       <path d=\"M 10.59375 35.5 \n",
       "L 73.1875 35.5 \n",
       "L 73.1875 27.203125 \n",
       "L 10.59375 27.203125 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-2212\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(30.067968750000013 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-31\"/>\n",
       "       <use x=\"147.412109375\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <path clip-path=\"url(#pa6d8f14215)\" d=\"M 89.445313 151.539062 \n",
       "L 89.445313 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"89.44531250000001\" xlink:href=\"#ma32d63e438\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"89.44531250000001\" xlink:href=\"#mff9bb9708c\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- −5 -->\n",
       "      <defs>\n",
       "       <path d=\"M 10.796875 72.90625 \n",
       "L 49.515625 72.90625 \n",
       "L 49.515625 64.59375 \n",
       "L 19.828125 64.59375 \n",
       "L 19.828125 46.734375 \n",
       "Q 21.96875 47.46875 24.109375 47.828125 \n",
       "Q 26.265625 48.1875 28.421875 48.1875 \n",
       "Q 40.625 48.1875 47.75 41.5 \n",
       "Q 54.890625 34.8125 54.890625 23.390625 \n",
       "Q 54.890625 11.625 47.5625 5.09375 \n",
       "Q 40.234375 -1.421875 26.90625 -1.421875 \n",
       "Q 22.3125 -1.421875 17.546875 -0.640625 \n",
       "Q 12.796875 0.140625 7.71875 1.703125 \n",
       "L 7.71875 11.625 \n",
       "Q 12.109375 9.234375 16.796875 8.0625 \n",
       "Q 21.484375 6.890625 26.703125 6.890625 \n",
       "Q 35.15625 6.890625 40.078125 11.328125 \n",
       "Q 45.015625 15.765625 45.015625 23.390625 \n",
       "Q 45.015625 31 40.078125 35.4375 \n",
       "Q 35.15625 39.890625 26.703125 39.890625 \n",
       "Q 22.75 39.890625 18.8125 39.015625 \n",
       "Q 14.890625 38.140625 10.796875 36.28125 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-35\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(82.07421875000001 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <path clip-path=\"url(#pa6d8f14215)\" d=\"M 138.270313 151.539062 \n",
       "L 138.270313 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"138.27031250000002\" xlink:href=\"#ma32d63e438\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"138.27031250000002\" xlink:href=\"#mff9bb9708c\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(135.0890625 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <path clip-path=\"url(#pa6d8f14215)\" d=\"M 187.095313 151.539062 \n",
       "L 187.095313 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"187.09531250000003\" xlink:href=\"#ma32d63e438\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"187.09531250000003\" xlink:href=\"#mff9bb9708c\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 5 -->\n",
       "      <g transform=\"translate(183.91406250000003 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-35\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <path clip-path=\"url(#pa6d8f14215)\" d=\"M 235.920313 151.539062 \n",
       "L 235.920313 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"235.92031250000002\" xlink:href=\"#ma32d63e438\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_17\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"235.92031250000002\" xlink:href=\"#mff9bb9708c\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(229.5578125 163.1375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-31\"/>\n",
       "       <use x=\"63.623046875\" xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_6\">\n",
       "     <!-- x -->\n",
       "     <defs>\n",
       "      <path d=\"M 54.890625 54.6875 \n",
       "L 35.109375 28.078125 \n",
       "L 55.90625 0 \n",
       "L 45.3125 0 \n",
       "L 29.390625 21.484375 \n",
       "L 13.484375 0 \n",
       "L 2.875 0 \n",
       "L 24.125 28.609375 \n",
       "L 4.6875 54.6875 \n",
       "L 15.28125 54.6875 \n",
       "L 29.78125 35.203125 \n",
       "L 44.28125 54.6875 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-78\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(135.31093750000002 177.815625)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#BitstreamVeraSans-Roman-78\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_18\">\n",
       "      <path clip-path=\"url(#pa6d8f14215)\" d=\"M 40.620313 151.539062 \n",
       "L 235.920313 151.539062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_19\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L 4 0 \n",
       "\" id=\"m8edfce477b\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"40.620312500000004\" xlink:href=\"#m8edfce477b\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_20\">\n",
       "      <defs>\n",
       "       <path d=\"M 0 0 \n",
       "L -4 0 \n",
       "\" id=\"ma326e612c6\" style=\"stroke:#000000;stroke-width:0.5;\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"235.92031250000002\" xlink:href=\"#ma326e612c6\" y=\"151.5390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- −8 -->\n",
       "      <defs>\n",
       "       <path d=\"M 31.78125 34.625 \n",
       "Q 24.75 34.625 20.71875 30.859375 \n",
       "Q 16.703125 27.09375 16.703125 20.515625 \n",
       "Q 16.703125 13.921875 20.71875 10.15625 \n",
       "Q 24.75 6.390625 31.78125 6.390625 \n",
       "Q 38.8125 6.390625 42.859375 10.171875 \n",
       "Q 46.921875 13.96875 46.921875 20.515625 \n",
       "Q 46.921875 27.09375 42.890625 30.859375 \n",
       "Q 38.875 34.625 31.78125 34.625 \n",
       "M 21.921875 38.8125 \n",
       "Q 15.578125 40.375 12.03125 44.71875 \n",
       "Q 8.5 49.078125 8.5 55.328125 \n",
       "Q 8.5 64.0625 14.71875 69.140625 \n",
       "Q 20.953125 74.21875 31.78125 74.21875 \n",
       "Q 42.671875 74.21875 48.875 69.140625 \n",
       "Q 55.078125 64.0625 55.078125 55.328125 \n",
       "Q 55.078125 49.078125 51.53125 44.71875 \n",
       "Q 48 40.375 41.703125 38.8125 \n",
       "Q 48.828125 37.15625 52.796875 32.3125 \n",
       "Q 56.78125 27.484375 56.78125 20.515625 \n",
       "Q 56.78125 9.90625 50.3125 4.234375 \n",
       "Q 43.84375 -1.421875 31.78125 -1.421875 \n",
       "Q 19.734375 -1.421875 13.25 4.234375 \n",
       "Q 6.78125 9.90625 6.78125 20.515625 \n",
       "Q 6.78125 27.484375 10.78125 32.3125 \n",
       "Q 14.796875 37.15625 21.921875 38.8125 \n",
       "M 18.3125 54.390625 \n",
       "Q 18.3125 48.734375 21.84375 45.5625 \n",
       "Q 25.390625 42.390625 31.78125 42.390625 \n",
       "Q 38.140625 42.390625 41.71875 45.5625 \n",
       "Q 45.3125 48.734375 45.3125 54.390625 \n",
       "Q 45.3125 60.0625 41.71875 63.234375 \n",
       "Q 38.140625 66.40625 31.78125 66.40625 \n",
       "Q 25.390625 66.40625 21.84375 63.234375 \n",
       "Q 18.3125 60.0625 18.3125 54.390625 \n",
       "\" id=\"BitstreamVeraSans-Roman-38\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(21.878125000000004 154.2984375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-38\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_21\">\n",
       "      <path clip-path=\"url(#pa6d8f14215)\" d=\"M 40.620313 134.101562 \n",
       "L 235.920313 134.101562 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"40.620312500000004\" xlink:href=\"#m8edfce477b\" y=\"134.1015625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_23\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"235.92031250000002\" xlink:href=\"#ma326e612c6\" y=\"134.1015625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- −6 -->\n",
       "      <defs>\n",
       "       <path d=\"M 33.015625 40.375 \n",
       "Q 26.375 40.375 22.484375 35.828125 \n",
       "Q 18.609375 31.296875 18.609375 23.390625 \n",
       "Q 18.609375 15.53125 22.484375 10.953125 \n",
       "Q 26.375 6.390625 33.015625 6.390625 \n",
       "Q 39.65625 6.390625 43.53125 10.953125 \n",
       "Q 47.40625 15.53125 47.40625 23.390625 \n",
       "Q 47.40625 31.296875 43.53125 35.828125 \n",
       "Q 39.65625 40.375 33.015625 40.375 \n",
       "M 52.59375 71.296875 \n",
       "L 52.59375 62.3125 \n",
       "Q 48.875 64.0625 45.09375 64.984375 \n",
       "Q 41.3125 65.921875 37.59375 65.921875 \n",
       "Q 27.828125 65.921875 22.671875 59.328125 \n",
       "Q 17.53125 52.734375 16.796875 39.40625 \n",
       "Q 19.671875 43.65625 24.015625 45.921875 \n",
       "Q 28.375 48.1875 33.59375 48.1875 \n",
       "Q 44.578125 48.1875 50.953125 41.515625 \n",
       "Q 57.328125 34.859375 57.328125 23.390625 \n",
       "Q 57.328125 12.15625 50.6875 5.359375 \n",
       "Q 44.046875 -1.421875 33.015625 -1.421875 \n",
       "Q 20.359375 -1.421875 13.671875 8.265625 \n",
       "Q 6.984375 17.96875 6.984375 36.375 \n",
       "Q 6.984375 53.65625 15.1875 63.9375 \n",
       "Q 23.390625 74.21875 37.203125 74.21875 \n",
       "Q 40.921875 74.21875 44.703125 73.484375 \n",
       "Q 48.484375 72.75 52.59375 71.296875 \n",
       "\" id=\"BitstreamVeraSans-Roman-36\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(21.878125000000004 136.8609375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-36\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_24\">\n",
       "      <path clip-path=\"url(#pa6d8f14215)\" d=\"M 40.620313 116.664062 \n",
       "L 235.920313 116.664062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_25\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"40.620312500000004\" xlink:href=\"#m8edfce477b\" y=\"116.6640625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_26\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"235.92031250000002\" xlink:href=\"#ma326e612c6\" y=\"116.6640625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- −4 -->\n",
       "      <defs>\n",
       "       <path d=\"M 37.796875 64.3125 \n",
       "L 12.890625 25.390625 \n",
       "L 37.796875 25.390625 \n",
       "z\n",
       "M 35.203125 72.90625 \n",
       "L 47.609375 72.90625 \n",
       "L 47.609375 25.390625 \n",
       "L 58.015625 25.390625 \n",
       "L 58.015625 17.1875 \n",
       "L 47.609375 17.1875 \n",
       "L 47.609375 0 \n",
       "L 37.796875 0 \n",
       "L 37.796875 17.1875 \n",
       "L 4.890625 17.1875 \n",
       "L 4.890625 26.703125 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-34\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(21.878125000000004 119.4234375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-34\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_27\">\n",
       "      <path clip-path=\"url(#pa6d8f14215)\" d=\"M 40.620313 99.226562 \n",
       "L 235.920313 99.226562 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_28\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"40.620312500000004\" xlink:href=\"#m8edfce477b\" y=\"99.2265625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_29\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"235.92031250000002\" xlink:href=\"#ma326e612c6\" y=\"99.2265625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- −2 -->\n",
       "      <defs>\n",
       "       <path d=\"M 19.1875 8.296875 \n",
       "L 53.609375 8.296875 \n",
       "L 53.609375 0 \n",
       "L 7.328125 0 \n",
       "L 7.328125 8.296875 \n",
       "Q 12.9375 14.109375 22.625 23.890625 \n",
       "Q 32.328125 33.6875 34.8125 36.53125 \n",
       "Q 39.546875 41.84375 41.421875 45.53125 \n",
       "Q 43.3125 49.21875 43.3125 52.78125 \n",
       "Q 43.3125 58.59375 39.234375 62.25 \n",
       "Q 35.15625 65.921875 28.609375 65.921875 \n",
       "Q 23.96875 65.921875 18.8125 64.3125 \n",
       "Q 13.671875 62.703125 7.8125 59.421875 \n",
       "L 7.8125 69.390625 \n",
       "Q 13.765625 71.78125 18.9375 73 \n",
       "Q 24.125 74.21875 28.421875 74.21875 \n",
       "Q 39.75 74.21875 46.484375 68.546875 \n",
       "Q 53.21875 62.890625 53.21875 53.421875 \n",
       "Q 53.21875 48.921875 51.53125 44.890625 \n",
       "Q 49.859375 40.875 45.40625 35.40625 \n",
       "Q 44.1875 33.984375 37.640625 27.21875 \n",
       "Q 31.109375 20.453125 19.1875 8.296875 \n",
       "\" id=\"BitstreamVeraSans-Roman-32\"/>\n",
       "      </defs>\n",
       "      <g transform=\"translate(21.878125000000004 101.9859375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-2212\"/>\n",
       "       <use x=\"83.7890625\" xlink:href=\"#BitstreamVeraSans-Roman-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_30\">\n",
       "      <path clip-path=\"url(#pa6d8f14215)\" d=\"M 40.620313 81.789062 \n",
       "L 235.920313 81.789062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_31\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"40.620312500000004\" xlink:href=\"#m8edfce477b\" y=\"81.7890625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_32\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"235.92031250000002\" xlink:href=\"#ma326e612c6\" y=\"81.7890625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(30.257812500000004 84.5484375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_33\">\n",
       "      <path clip-path=\"url(#pa6d8f14215)\" d=\"M 40.620313 64.351562 \n",
       "L 235.920313 64.351562 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_34\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"40.620312500000004\" xlink:href=\"#m8edfce477b\" y=\"64.3515625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_35\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"235.92031250000002\" xlink:href=\"#ma326e612c6\" y=\"64.3515625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(30.257812500000004 67.1109375)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_7\">\n",
       "     <g id=\"line2d_36\">\n",
       "      <path clip-path=\"url(#pa6d8f14215)\" d=\"M 40.620313 46.914062 \n",
       "L 235.920313 46.914062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_37\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"40.620312500000004\" xlink:href=\"#m8edfce477b\" y=\"46.9140625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_38\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"235.92031250000002\" xlink:href=\"#ma326e612c6\" y=\"46.9140625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 4 -->\n",
       "      <g transform=\"translate(30.257812500000004 49.673437500000006)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-34\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_8\">\n",
       "     <g id=\"line2d_39\">\n",
       "      <path clip-path=\"url(#pa6d8f14215)\" d=\"M 40.620313 29.476562 \n",
       "L 235.920313 29.476562 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_40\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"40.620312500000004\" xlink:href=\"#m8edfce477b\" y=\"29.4765625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_41\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"235.92031250000002\" xlink:href=\"#ma326e612c6\" y=\"29.4765625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_14\">\n",
       "      <!-- 6 -->\n",
       "      <g transform=\"translate(30.257812500000004 32.235937500000006)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-36\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_9\">\n",
       "     <g id=\"line2d_42\">\n",
       "      <path clip-path=\"url(#pa6d8f14215)\" d=\"M 40.620313 12.039062 \n",
       "L 235.920313 12.039062 \n",
       "\" style=\"fill:none;stroke:#000000;stroke-dasharray:1.000000,3.000000;stroke-dashoffset:0.0;stroke-width:0.5;\"/>\n",
       "     </g>\n",
       "     <g id=\"line2d_43\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"40.620312500000004\" xlink:href=\"#m8edfce477b\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"line2d_44\">\n",
       "      <g>\n",
       "       <use style=\"stroke:#000000;stroke-width:0.5;\" x=\"235.92031250000002\" xlink:href=\"#ma326e612c6\" y=\"12.0390625\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_15\">\n",
       "      <!-- 8 -->\n",
       "      <g transform=\"translate(30.257812500000004 14.798437500000006)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#BitstreamVeraSans-Roman-38\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_16\">\n",
       "     <!-- f(x) -->\n",
       "     <defs>\n",
       "      <path d=\"M 37.109375 75.984375 \n",
       "L 37.109375 68.5 \n",
       "L 28.515625 68.5 \n",
       "Q 23.6875 68.5 21.796875 66.546875 \n",
       "Q 19.921875 64.59375 19.921875 59.515625 \n",
       "L 19.921875 54.6875 \n",
       "L 34.71875 54.6875 \n",
       "L 34.71875 47.703125 \n",
       "L 19.921875 47.703125 \n",
       "L 19.921875 0 \n",
       "L 10.890625 0 \n",
       "L 10.890625 47.703125 \n",
       "L 2.296875 47.703125 \n",
       "L 2.296875 54.6875 \n",
       "L 10.890625 54.6875 \n",
       "L 10.890625 58.5 \n",
       "Q 10.890625 67.625 15.140625 71.796875 \n",
       "Q 19.390625 75.984375 28.609375 75.984375 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-66\"/>\n",
       "      <path d=\"M 31 75.875 \n",
       "Q 24.46875 64.65625 21.28125 53.65625 \n",
       "Q 18.109375 42.671875 18.109375 31.390625 \n",
       "Q 18.109375 20.125 21.3125 9.0625 \n",
       "Q 24.515625 -2 31 -13.1875 \n",
       "L 23.1875 -13.1875 \n",
       "Q 15.875 -1.703125 12.234375 9.375 \n",
       "Q 8.59375 20.453125 8.59375 31.390625 \n",
       "Q 8.59375 42.28125 12.203125 53.3125 \n",
       "Q 15.828125 64.359375 23.1875 75.875 \n",
       "z\n",
       "\" id=\"BitstreamVeraSans-Roman-28\"/>\n",
       "      <path d=\"M 8.015625 75.875 \n",
       "L 15.828125 75.875 \n",
       "Q 23.140625 64.359375 26.78125 53.3125 \n",
       "Q 30.421875 42.28125 30.421875 31.390625 \n",
       "Q 30.421875 20.453125 26.78125 9.375 \n",
       "Q 23.140625 -1.703125 15.828125 -13.1875 \n",
       "L 8.015625 -13.1875 \n",
       "Q 14.5 -2 17.703125 9.0625 \n",
       "Q 20.90625 20.125 20.90625 31.390625 \n",
       "Q 20.90625 42.671875 17.703125 53.65625 \n",
       "Q 14.5 64.65625 8.015625 75.875 \n",
       "\" id=\"BitstreamVeraSans-Roman-29\"/>\n",
       "     </defs>\n",
       "     <g transform=\"translate(14.798437500000004 90.41015625)rotate(-90.0)scale(0.1 -0.1)\">\n",
       "      <use xlink:href=\"#BitstreamVeraSans-Roman-66\"/>\n",
       "      <use x=\"35.205078125\" xlink:href=\"#BitstreamVeraSans-Roman-28\"/>\n",
       "      <use x=\"74.21875\" xlink:href=\"#BitstreamVeraSans-Roman-78\"/>\n",
       "      <use x=\"133.3984375\" xlink:href=\"#BitstreamVeraSans-Roman-29\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pa6d8f14215\">\n",
       "   <rect height=\"139.5\" width=\"195.3\" x=\"40.620312500000004\" y=\"12.0390625\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f234cc53240>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_trace(newton(0.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convergence Analysis\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We only analyze the convergence rate for convex and three times differentiable $f$, where at its minimum $x^$ the second derivative is nonzero, i.e. where $f''(x^) > 0$. The multivariate proof is a straightforward extension of the argument below and omitted since it doesn't help us much in terms of intuition.\n",
    "\n",
    "Denote by $x_k$ the value of $x$ at the $k$-th iteration and let $e_k := x_k - x^$ be the distance from optimality. By Taylor series expansion we have that the condition $f'(x^) = 0$ can be written as\n",
    "\n",
    "$$0 = f'(x_k - e_k) = f'(x_k) - e_k f''(x_k) + \\frac{1}{2} e_k^2 f'''(\\xi_k).$$\n",
    "\n",
    "This holds for some $\\xi_k \\in [x_k - e_k, x_k]$. Recall that we have the update $x_{k+1} = x_k - f'(x_k) / f''(x_k)$. Dividing the above expansion by $f''(x_k)$ yields\n",
    "\n",
    "$$e_k - f'(x_k) / f''(x_k) = \\frac{1}{2} e_k^2 f'''(\\xi_k) / f'(x_k)$$\n",
    "\n",
    "Plugging in the update equations leads to the following bound $e_{k+1} \\leq e_k^2 f'''(\\xi_k) / f'(x_k)$. Consequently, whenever we are in a region of bounded $f'''(\\xi_k) / f'(x_k) \\leq c$, we have a quadratically decreasing error $e_{k+1} \\leq c e_k^2$.\n",
    "\n",
    "As an aside, optimization researchers call this linear convergence, whereas a condition such as $e_{k+1} \\leq \\alpha e_k$ would be called a constant rate of convergence. Note that this analysis comes with a number of caveats: We don't really have much of a guarantee when we will reach the region of rapid convergence. Instead, we only know that once we reach it, convergence will be very quick. Second, this requires that $f$ is well-behaved up to higher order derivatives. It comes down to ensuring that $f$ doesn't have any 'surprising' properties in terms of how it might change its values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preconditioning\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Quite unsurprisingly computing and storing the full Hessian is very expensive. It is thus desirable to find alternatives. One way to improve matters is by avoiding to compute the Hessian in its entirety but only compute the diagonal entries. While this isn't quite as good as the full Newton method, it is still much better than not using it. Moreover, estimates for the main diagonal elements are what drives some of the innovation in stochastic gradient descent optimization algorithms. This leads to update algorithms of the form\n",
    "\n",
    "$$\\mathbf{x} \\leftarrow \\mathbf{x} - \\eta \\mathrm{diag}(H_f)^{-1} \\nabla \\mathbf{x}.$$\n",
    "\n",
    "To see why this might be a good idea consider a situation where one variable denotes height in milimeters and the other one denotes height in kilometers. Assuming that for both the natural scale is in meters we have a terrible mismatch in parametrizations. Using preconditioning removes this. Effectively preconditioning with gradient descent amounts to selecting a different learning rate for each coordinate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient Descent with Line Search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the key problems in gradient descent was that we might overshoot the goal or make insufficient progress. A simple fix for the problem is to use line search in conjunction with gradient descent. That is, we use the direction given by $\\nabla f(\\mathbf{x})$ and then perform binary search as to which steplength $\\eta$ minimizes $f(x - \\eta \\nabla f(\\mathbf{x}))$.\n",
    "\n",
    "This algorithm converges rapidly (for an analysis and proof see e.g. :cite:Boyd.Vandenberghe.2004). However, for the purpose of deep learning this isn't quite so feasible, since each step of the line search would require us to evaluate the objective function on the entire dataset. This is way too costly to accomplish."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Experiment with different learning rates and objective functions for gradient descent.\n",
    "2. Implement line search to minimize a convex function in the interval $[a, b]$.\n",
    "   - Do you need derivatives for binary search, i.e. to decide whether to pick $[a, (a+b)/2]$ or $[(a+b)/2, b]$.\n",
    "   - How rapid is the rate of convergence for the algorithm?\n",
    "   - Implement the algorithm and apply it to minimizing $\\log (\\exp(x) + \\exp(-2*x -3))$.\n",
    "3. Design an objective function defined on $\\mathbb{R}^2$ where gradient descent is exceedingly slow. Hint - scale different coordinates differently.\n",
    "4. Implement the lightweight version of Newton's method using preconditioning:\n",
    "   - Use diagonal Hessian as preconditioner.\n",
    "   - Use the absolute values of that rather than the actual (possibly signed) values.\n",
    "   - Apply this to the problem above.\n",
    "5. Apply the algorithm above to a number of objective functions (convex or not). What happens if you rotate coordinates by $45$ degrees?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}