Ch06_Multilayer_Perceptrons/Multilayer_Perceptron.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multilayer Perceptron\n",
    "\n",
    "In the previous chapters, we showed how you could implement multiclass logistic regression (also called softmax regression)\n",
    "for classifying images of clothing into the 10 possible categories.\n",
    "To get there, we had to learn how to wrangle data,\n",
    "coerce our outputs into a valid probability distribution (via `softmax`),\n",
    "how to apply an appropriate loss function,\n",
    "and how to optimize over our parameters.\n",
    "Now that we’ve covered these preliminaries,\n",
    "we are free to focus our attention on\n",
    "the more exciting enterprise of designing powerful models\n",
    "using deep neural networks.\n",
    "\n",
    "## Hidden Layers\n",
    "\n",
    "Recall that for linear regression and softmax regression,\n",
    "we mapped our inputs directly to our outputs\n",
    "via a single linear transformation:\n",
    "\n",
    "$$\n",
    "\\hat{\\mathbf{o}} = \\mathrm{softmax}(\\mathbf{W} \\mathbf{x} + \\mathbf{b})\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"88pt\" version=\"1.1\" viewBox=\"0 0 275 88\" width=\"275pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<defs>\n",
       "<g>\n",
       "<symbol id=\"glyph0-0\" overflow=\"visible\">\n",
       "<path d=\"M 1.125 0 L 1.125 -5.625 L 5.625 -5.625 L 5.625 0 Z M 1.265625 -0.140625 L 5.484375 -0.140625 L 5.484375 -5.484375 L 1.265625 -5.484375 Z M 1.265625 -0.140625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-1\" overflow=\"visible\">\n",
       "<path d=\"M -0.015625 0 L 2.015625 -2.375 L 0.859375 -4.671875 L 1.734375 -4.671875 L 2.125 -3.84375 C 2.269531 -3.53125 2.398438 -3.226562 2.515625 -2.9375 L 3.875 -4.671875 L 4.84375 -4.671875 L 2.875 -2.3125 L 4.0625 0 L 3.171875 0 L 2.71875 -0.953125 C 2.613281 -1.148438 2.5 -1.398438 2.375 -1.703125 L 0.984375 0 Z M -0.015625 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-2\" overflow=\"visible\">\n",
       "<path d=\"M 0.4375 -1.765625 C 0.4375 -2.679688 0.707031 -3.4375 1.25 -4.03125 C 1.6875 -4.519531 2.265625 -4.765625 2.984375 -4.765625 C 3.546875 -4.765625 4 -4.585938 4.34375 -4.234375 C 4.6875 -3.890625 4.859375 -3.421875 4.859375 -2.828125 C 4.859375 -2.285156 4.75 -1.785156 4.53125 -1.328125 C 4.3125 -0.867188 4.003906 -0.515625 3.609375 -0.265625 C 3.210938 -0.015625 2.789062 0.109375 2.34375 0.109375 C 1.976562 0.109375 1.644531 0.03125 1.34375 -0.125 C 1.050781 -0.28125 0.828125 -0.5 0.671875 -0.78125 C 0.515625 -1.070312 0.4375 -1.398438 0.4375 -1.765625 Z M 1.234375 -1.84375 C 1.234375 -1.40625 1.335938 -1.070312 1.546875 -0.84375 C 1.765625 -0.625 2.035156 -0.515625 2.359375 -0.515625 C 2.523438 -0.515625 2.691406 -0.546875 2.859375 -0.609375 C 3.023438 -0.679688 3.179688 -0.785156 3.328125 -0.921875 C 3.472656 -1.066406 3.59375 -1.226562 3.6875 -1.40625 C 3.789062 -1.582031 3.875 -1.773438 3.9375 -1.984375 C 4.03125 -2.273438 4.078125 -2.554688 4.078125 -2.828125 C 4.078125 -3.242188 3.96875 -3.566406 3.75 -3.796875 C 3.539062 -4.035156 3.273438 -4.15625 2.953125 -4.15625 C 2.703125 -4.15625 2.472656 -4.09375 2.265625 -3.96875 C 2.066406 -3.851562 1.882812 -3.679688 1.71875 -3.453125 C 1.550781 -3.222656 1.425781 -2.957031 1.34375 -2.65625 C 1.269531 -2.351562 1.234375 -2.082031 1.234375 -1.84375 Z M 1.234375 -1.84375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-0\" overflow=\"visible\">\n",
       "<path d=\"M 0.875 0 L 0.875 -4.375 L 4.375 -4.375 L 4.375 0 Z M 0.984375 -0.109375 L 4.265625 -0.109375 L 4.265625 -4.265625 L 0.984375 -4.265625 Z M 0.984375 -0.109375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-1\" overflow=\"visible\">\n",
       "<path d=\"M 1.6875 0 L 2.484375 -3.78125 C 2.140625 -3.507812 1.65625 -3.296875 1.03125 -3.140625 L 1.15625 -3.703125 C 1.457031 -3.828125 1.757812 -3.988281 2.0625 -4.1875 C 2.363281 -4.382812 2.585938 -4.554688 2.734375 -4.703125 C 2.828125 -4.796875 2.914062 -4.90625 3 -5.03125 L 3.359375 -5.03125 L 2.3125 0 Z M 1.6875 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-2\" overflow=\"visible\">\n",
       "<path d=\"M 0.40625 0 C 0.46875 -0.300781 0.554688 -0.550781 0.671875 -0.75 C 0.785156 -0.945312 0.9375 -1.132812 1.125 -1.3125 C 1.3125 -1.5 1.671875 -1.804688 2.203125 -2.234375 C 2.523438 -2.492188 2.75 -2.6875 2.875 -2.8125 C 3.039062 -2.988281 3.160156 -3.160156 3.234375 -3.328125 C 3.285156 -3.441406 3.3125 -3.566406 3.3125 -3.703125 C 3.3125 -3.929688 3.226562 -4.125 3.0625 -4.28125 C 2.90625 -4.445312 2.707031 -4.53125 2.46875 -4.53125 C 2.238281 -4.53125 2.035156 -4.445312 1.859375 -4.28125 C 1.679688 -4.125 1.554688 -3.863281 1.484375 -3.5 L 0.875 -3.59375 C 0.9375 -4.039062 1.109375 -4.390625 1.390625 -4.640625 C 1.679688 -4.898438 2.039062 -5.03125 2.46875 -5.03125 C 2.75 -5.03125 3.003906 -4.96875 3.234375 -4.84375 C 3.460938 -4.726562 3.632812 -4.5625 3.75 -4.34375 C 3.875 -4.132812 3.9375 -3.914062 3.9375 -3.6875 C 3.9375 -3.351562 3.816406 -3.035156 3.578125 -2.734375 C 3.429688 -2.535156 3.003906 -2.15625 2.296875 -1.59375 C 1.984375 -1.351562 1.753906 -1.15625 1.609375 -1 C 1.460938 -0.84375 1.351562 -0.695312 1.28125 -0.5625 L 3.515625 -0.5625 L 3.390625 0 Z M 0.40625 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-3\" overflow=\"visible\">\n",
       "<path d=\"M 0.390625 -1.3125 L 0.984375 -1.390625 C 1.023438 -1.035156 1.125 -0.78125 1.28125 -0.625 C 1.4375 -0.476562 1.644531 -0.40625 1.90625 -0.40625 C 2.207031 -0.40625 2.46875 -0.515625 2.6875 -0.734375 C 2.914062 -0.953125 3.03125 -1.203125 3.03125 -1.484375 C 3.03125 -1.734375 2.945312 -1.9375 2.78125 -2.09375 C 2.613281 -2.257812 2.390625 -2.34375 2.109375 -2.34375 C 2.078125 -2.34375 2.007812 -2.335938 1.90625 -2.328125 L 2.015625 -2.84375 C 2.078125 -2.832031 2.132812 -2.828125 2.1875 -2.828125 C 2.539062 -2.828125 2.8125 -2.910156 3 -3.078125 C 3.1875 -3.253906 3.28125 -3.46875 3.28125 -3.71875 C 3.28125 -3.945312 3.203125 -4.140625 3.046875 -4.296875 C 2.890625 -4.453125 2.703125 -4.53125 2.484375 -4.53125 C 2.253906 -4.53125 2.050781 -4.445312 1.875 -4.28125 C 1.695312 -4.125 1.585938 -3.898438 1.546875 -3.609375 L 0.9375 -3.734375 C 1.03125 -4.148438 1.21875 -4.46875 1.5 -4.6875 C 1.789062 -4.914062 2.128906 -5.03125 2.515625 -5.03125 C 2.929688 -5.03125 3.265625 -4.90625 3.515625 -4.65625 C 3.773438 -4.40625 3.90625 -4.101562 3.90625 -3.75 C 3.90625 -3.476562 3.832031 -3.242188 3.6875 -3.046875 C 3.550781 -2.847656 3.347656 -2.6875 3.078125 -2.5625 C 3.265625 -2.445312 3.40625 -2.304688 3.5 -2.140625 C 3.601562 -1.972656 3.65625 -1.785156 3.65625 -1.578125 C 3.65625 -1.128906 3.488281 -0.738281 3.15625 -0.40625 C 2.820312 -0.0820312 2.421875 0.078125 1.953125 0.078125 C 1.492188 0.078125 1.125 -0.046875 0.84375 -0.296875 C 0.570312 -0.546875 0.421875 -0.882812 0.390625 -1.3125 Z M 0.390625 -1.3125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-4\" overflow=\"visible\">\n",
       "<path d=\"M 2.09375 0 L 2.359375 -1.28125 L 0.3125 -1.28125 L 0.453125 -1.890625 L 3.25 -5.015625 L 3.765625 -5.015625 L 3.09375 -1.828125 L 3.796875 -1.828125 L 3.6875 -1.28125 L 2.984375 -1.28125 L 2.703125 0 Z M 2.46875 -1.828125 L 2.90625 -3.921875 L 1.046875 -1.828125 Z M 2.46875 -1.828125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-5\" overflow=\"visible\">\n",
       "<path d=\"M 0.484375 -1.4375 L 1.125 -1.5 C 1.113281 -1.40625 1.109375 -1.347656 1.109375 -1.328125 C 1.109375 -1.179688 1.144531 -1.03125 1.21875 -0.875 C 1.300781 -0.726562 1.40625 -0.613281 1.53125 -0.53125 C 1.664062 -0.445312 1.804688 -0.40625 1.953125 -0.40625 C 2.140625 -0.40625 2.332031 -0.46875 2.53125 -0.59375 C 2.726562 -0.726562 2.890625 -0.921875 3.015625 -1.171875 C 3.140625 -1.429688 3.203125 -1.6875 3.203125 -1.9375 C 3.203125 -2.21875 3.117188 -2.441406 2.953125 -2.609375 C 2.785156 -2.773438 2.566406 -2.859375 2.296875 -2.859375 C 2.117188 -2.859375 1.945312 -2.8125 1.78125 -2.71875 C 1.625 -2.632812 1.476562 -2.507812 1.34375 -2.34375 L 0.796875 -2.375 L 1.5625 -4.9375 L 4 -4.9375 L 3.890625 -4.375 L 1.984375 -4.375 L 1.609375 -3.09375 C 1.742188 -3.195312 1.882812 -3.273438 2.03125 -3.328125 C 2.1875 -3.378906 2.34375 -3.40625 2.5 -3.40625 C 2.882812 -3.40625 3.195312 -3.28125 3.4375 -3.03125 C 3.6875 -2.78125 3.8125 -2.429688 3.8125 -1.984375 C 3.8125 -1.597656 3.726562 -1.242188 3.5625 -0.921875 C 3.394531 -0.597656 3.160156 -0.347656 2.859375 -0.171875 C 2.566406 -0.00390625 2.25 0.078125 1.90625 0.078125 C 1.625 0.078125 1.367188 0.015625 1.140625 -0.109375 C 0.921875 -0.234375 0.753906 -0.410156 0.640625 -0.640625 C 0.535156 -0.867188 0.484375 -1.097656 0.484375 -1.328125 C 0.484375 -1.347656 0.484375 -1.382812 0.484375 -1.4375 Z M 0.484375 -1.4375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-0\" overflow=\"visible\">\n",
       "<path d=\"M 1.125 0 L 1.125 -5.625 L 5.625 -5.625 L 5.625 0 Z M 1.265625 -0.140625 L 5.484375 -0.140625 L 5.484375 -5.484375 L 1.265625 -5.484375 Z M 1.265625 -0.140625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-1\" overflow=\"visible\">\n",
       "<path d=\"M 0.84375 0 L 0.84375 -6.4375 L 1.6875 -6.4375 L 1.6875 0 Z M 0.84375 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-2\" overflow=\"visible\">\n",
       "<path d=\"M 0.59375 0 L 0.59375 -4.671875 L 1.3125 -4.671875 L 1.3125 -4 C 1.644531 -4.507812 2.140625 -4.765625 2.796875 -4.765625 C 3.078125 -4.765625 3.332031 -4.710938 3.5625 -4.609375 C 3.800781 -4.515625 3.976562 -4.382812 4.09375 -4.21875 C 4.207031 -4.0625 4.289062 -3.867188 4.34375 -3.640625 C 4.375 -3.492188 4.390625 -3.238281 4.390625 -2.875 L 4.390625 0 L 3.59375 0 L 3.59375 -2.84375 C 3.59375 -3.164062 3.5625 -3.40625 3.5 -3.5625 C 3.4375 -3.71875 3.328125 -3.84375 3.171875 -3.9375 C 3.015625 -4.039062 2.832031 -4.09375 2.625 -4.09375 C 2.289062 -4.09375 2 -3.984375 1.75 -3.765625 C 1.507812 -3.554688 1.390625 -3.148438 1.390625 -2.546875 L 1.390625 0 Z M 0.59375 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-3\" overflow=\"visible\">\n",
       "<path d=\"M 0.59375 1.78125 L 0.59375 -4.671875 L 1.3125 -4.671875 L 1.3125 -4.0625 C 1.476562 -4.300781 1.664062 -4.476562 1.875 -4.59375 C 2.09375 -4.707031 2.359375 -4.765625 2.671875 -4.765625 C 3.066406 -4.765625 3.414062 -4.660156 3.71875 -4.453125 C 4.019531 -4.253906 4.25 -3.96875 4.40625 -3.59375 C 4.5625 -3.21875 4.640625 -2.8125 4.640625 -2.375 C 4.640625 -1.894531 4.550781 -1.460938 4.375 -1.078125 C 4.207031 -0.691406 3.960938 -0.394531 3.640625 -0.1875 C 3.316406 0.0078125 2.972656 0.109375 2.609375 0.109375 C 2.347656 0.109375 2.113281 0.0507812 1.90625 -0.0625 C 1.695312 -0.175781 1.523438 -0.316406 1.390625 -0.484375 L 1.390625 1.78125 Z M 1.3125 -2.3125 C 1.3125 -1.707031 1.429688 -1.257812 1.671875 -0.96875 C 1.921875 -0.6875 2.21875 -0.546875 2.5625 -0.546875 C 2.90625 -0.546875 3.203125 -0.691406 3.453125 -0.984375 C 3.710938 -1.285156 3.84375 -1.75 3.84375 -2.375 C 3.84375 -2.96875 3.71875 -3.410156 3.46875 -3.703125 C 3.226562 -4.003906 2.9375 -4.15625 2.59375 -4.15625 C 2.257812 -4.15625 1.960938 -3.992188 1.703125 -3.671875 C 1.441406 -3.359375 1.3125 -2.90625 1.3125 -2.3125 Z M 1.3125 -2.3125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-4\" overflow=\"visible\">\n",
       "<path d=\"M 3.65625 0 L 3.65625 -0.6875 C 3.289062 -0.15625 2.796875 0.109375 2.171875 0.109375 C 1.898438 0.109375 1.644531 0.0546875 1.40625 -0.046875 C 1.164062 -0.160156 0.984375 -0.296875 0.859375 -0.453125 C 0.742188 -0.609375 0.664062 -0.800781 0.625 -1.03125 C 0.59375 -1.1875 0.578125 -1.4375 0.578125 -1.78125 L 0.578125 -4.671875 L 1.359375 -4.671875 L 1.359375 -2.078125 C 1.359375 -1.660156 1.378906 -1.382812 1.421875 -1.25 C 1.460938 -1.039062 1.5625 -0.875 1.71875 -0.75 C 1.882812 -0.632812 2.085938 -0.578125 2.328125 -0.578125 C 2.566406 -0.578125 2.789062 -0.632812 3 -0.75 C 3.207031 -0.875 3.351562 -1.039062 3.4375 -1.25 C 3.519531 -1.457031 3.5625 -1.765625 3.5625 -2.171875 L 3.5625 -4.671875 L 4.359375 -4.671875 L 4.359375 0 Z M 3.65625 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-5\" overflow=\"visible\">\n",
       "<path d=\"M 2.328125 -0.703125 L 2.4375 -0.015625 C 2.207031 0.0351562 2.007812 0.0625 1.84375 0.0625 C 1.550781 0.0625 1.328125 0.015625 1.171875 -0.078125 C 1.015625 -0.171875 0.898438 -0.289062 0.828125 -0.4375 C 0.765625 -0.582031 0.734375 -0.890625 0.734375 -1.359375 L 0.734375 -4.046875 L 0.15625 -4.046875 L 0.15625 -4.671875 L 0.734375 -4.671875 L 0.734375 -5.828125 L 1.53125 -6.296875 L 1.53125 -4.671875 L 2.328125 -4.671875 L 2.328125 -4.046875 L 1.53125 -4.046875 L 1.53125 -1.328125 C 1.53125 -1.097656 1.539062 -0.953125 1.5625 -0.890625 C 1.59375 -0.828125 1.640625 -0.773438 1.703125 -0.734375 C 1.765625 -0.691406 1.851562 -0.671875 1.96875 -0.671875 C 2.0625 -0.671875 2.179688 -0.679688 2.328125 -0.703125 Z M 2.328125 -0.703125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-6\" overflow=\"visible\">\n",
       "<path d=\"\" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-7\" overflow=\"visible\">\n",
       "<path d=\"M 0.578125 0 L 0.578125 -6.4375 L 1.359375 -6.4375 L 1.359375 0 Z M 0.578125 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-8\" overflow=\"visible\">\n",
       "<path d=\"M 3.640625 -0.578125 C 3.347656 -0.328125 3.066406 -0.148438 2.796875 -0.046875 C 2.523438 0.0546875 2.234375 0.109375 1.921875 0.109375 C 1.410156 0.109375 1.015625 -0.015625 0.734375 -0.265625 C 0.460938 -0.515625 0.328125 -0.835938 0.328125 -1.234375 C 0.328125 -1.460938 0.378906 -1.671875 0.484375 -1.859375 C 0.585938 -2.046875 0.722656 -2.195312 0.890625 -2.3125 C 1.054688 -2.425781 1.242188 -2.515625 1.453125 -2.578125 C 1.609375 -2.609375 1.84375 -2.644531 2.15625 -2.6875 C 2.800781 -2.757812 3.273438 -2.851562 3.578125 -2.96875 C 3.578125 -3.070312 3.578125 -3.140625 3.578125 -3.171875 C 3.578125 -3.492188 3.503906 -3.71875 3.359375 -3.84375 C 3.148438 -4.03125 2.847656 -4.125 2.453125 -4.125 C 2.078125 -4.125 1.800781 -4.054688 1.625 -3.921875 C 1.445312 -3.796875 1.316406 -3.566406 1.234375 -3.234375 L 0.46875 -3.328125 C 0.53125 -3.660156 0.640625 -3.925781 0.796875 -4.125 C 0.960938 -4.332031 1.195312 -4.488281 1.5 -4.59375 C 1.8125 -4.707031 2.164062 -4.765625 2.5625 -4.765625 C 2.96875 -4.765625 3.289062 -4.71875 3.53125 -4.625 C 3.78125 -4.53125 3.960938 -4.410156 4.078125 -4.265625 C 4.203125 -4.128906 4.285156 -3.953125 4.328125 -3.734375 C 4.359375 -3.597656 4.375 -3.359375 4.375 -3.015625 L 4.375 -1.953125 C 4.375 -1.222656 4.390625 -0.757812 4.421875 -0.5625 C 4.453125 -0.363281 4.519531 -0.175781 4.625 0 L 3.796875 0 C 3.710938 -0.164062 3.660156 -0.359375 3.640625 -0.578125 Z M 3.578125 -2.34375 C 3.285156 -2.226562 2.851562 -2.128906 2.28125 -2.046875 C 1.957031 -1.992188 1.726562 -1.9375 1.59375 -1.875 C 1.457031 -1.820312 1.351562 -1.738281 1.28125 -1.625 C 1.207031 -1.507812 1.171875 -1.382812 1.171875 -1.25 C 1.171875 -1.039062 1.25 -0.863281 1.40625 -0.71875 C 1.5625 -0.582031 1.796875 -0.515625 2.109375 -0.515625 C 2.410156 -0.515625 2.679688 -0.582031 2.921875 -0.71875 C 3.160156 -0.851562 3.335938 -1.035156 3.453125 -1.265625 C 3.535156 -1.441406 3.578125 -1.703125 3.578125 -2.046875 Z M 3.578125 -2.34375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-9\" overflow=\"visible\">\n",
       "<path d=\"M 0.5625 1.796875 L 0.46875 1.0625 C 0.644531 1.101562 0.796875 1.125 0.921875 1.125 C 1.097656 1.125 1.238281 1.09375 1.34375 1.03125 C 1.445312 0.976562 1.535156 0.898438 1.609375 0.796875 C 1.648438 0.710938 1.726562 0.515625 1.84375 0.203125 C 1.863281 0.160156 1.890625 0.0976562 1.921875 0.015625 L 0.140625 -4.671875 L 1 -4.671875 L 1.96875 -1.96875 C 2.09375 -1.625 2.207031 -1.265625 2.3125 -0.890625 C 2.394531 -1.242188 2.5 -1.597656 2.625 -1.953125 L 3.625 -4.671875 L 4.421875 -4.671875 L 2.640625 0.078125 C 2.453125 0.585938 2.304688 0.941406 2.203125 1.140625 C 2.054688 1.398438 1.894531 1.585938 1.71875 1.703125 C 1.539062 1.828125 1.320312 1.890625 1.0625 1.890625 C 0.914062 1.890625 0.75 1.859375 0.5625 1.796875 Z M 0.5625 1.796875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-10\" overflow=\"visible\">\n",
       "<path d=\"M 3.78125 -1.5 L 4.609375 -1.40625 C 4.472656 -0.925781 4.226562 -0.550781 3.875 -0.28125 C 3.53125 -0.0195312 3.085938 0.109375 2.546875 0.109375 C 1.867188 0.109375 1.328125 -0.0976562 0.921875 -0.515625 C 0.523438 -0.941406 0.328125 -1.535156 0.328125 -2.296875 C 0.328125 -3.078125 0.53125 -3.679688 0.9375 -4.109375 C 1.34375 -4.546875 1.867188 -4.765625 2.515625 -4.765625 C 3.140625 -4.765625 3.644531 -4.550781 4.03125 -4.125 C 4.425781 -3.707031 4.625 -3.113281 4.625 -2.34375 C 4.625 -2.289062 4.625 -2.21875 4.625 -2.125 L 1.140625 -2.125 C 1.171875 -1.613281 1.316406 -1.222656 1.578125 -0.953125 C 1.835938 -0.679688 2.164062 -0.546875 2.5625 -0.546875 C 2.851562 -0.546875 3.097656 -0.617188 3.296875 -0.765625 C 3.503906 -0.921875 3.664062 -1.164062 3.78125 -1.5 Z M 1.1875 -2.78125 L 3.796875 -2.78125 C 3.765625 -3.175781 3.664062 -3.472656 3.5 -3.671875 C 3.25 -3.972656 2.921875 -4.125 2.515625 -4.125 C 2.148438 -4.125 1.84375 -4 1.59375 -3.75 C 1.351562 -3.507812 1.21875 -3.1875 1.1875 -2.78125 Z M 1.1875 -2.78125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-11\" overflow=\"visible\">\n",
       "<path d=\"M 0.578125 0 L 0.578125 -4.671875 L 1.296875 -4.671875 L 1.296875 -3.953125 C 1.472656 -4.285156 1.640625 -4.503906 1.796875 -4.609375 C 1.953125 -4.710938 2.125 -4.765625 2.3125 -4.765625 C 2.570312 -4.765625 2.84375 -4.679688 3.125 -4.515625 L 2.84375 -3.78125 C 2.65625 -3.894531 2.460938 -3.953125 2.265625 -3.953125 C 2.097656 -3.953125 1.941406 -3.898438 1.796875 -3.796875 C 1.660156 -3.691406 1.5625 -3.546875 1.5 -3.359375 C 1.414062 -3.078125 1.375 -2.769531 1.375 -2.4375 L 1.375 0 Z M 0.578125 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-12\" overflow=\"visible\">\n",
       "<path d=\"M 0.4375 -3.140625 C 0.4375 -4.203125 0.722656 -5.035156 1.296875 -5.640625 C 1.867188 -6.253906 2.609375 -6.5625 3.515625 -6.5625 C 4.109375 -6.5625 4.644531 -6.414062 5.125 -6.125 C 5.601562 -5.84375 5.96875 -5.445312 6.21875 -4.9375 C 6.46875 -4.425781 6.59375 -3.851562 6.59375 -3.21875 C 6.59375 -2.5625 6.460938 -1.972656 6.203125 -1.453125 C 5.941406 -0.941406 5.566406 -0.550781 5.078125 -0.28125 C 4.597656 -0.0195312 4.078125 0.109375 3.515625 0.109375 C 2.910156 0.109375 2.367188 -0.0351562 1.890625 -0.328125 C 1.410156 -0.617188 1.046875 -1.019531 0.796875 -1.53125 C 0.554688 -2.039062 0.4375 -2.578125 0.4375 -3.140625 Z M 1.3125 -3.125 C 1.3125 -2.34375 1.519531 -1.726562 1.9375 -1.28125 C 2.351562 -0.84375 2.878906 -0.625 3.515625 -0.625 C 4.148438 -0.625 4.675781 -0.847656 5.09375 -1.296875 C 5.507812 -1.742188 5.71875 -2.382812 5.71875 -3.21875 C 5.71875 -3.738281 5.628906 -4.191406 5.453125 -4.578125 C 5.273438 -4.972656 5.015625 -5.28125 4.671875 -5.5 C 4.328125 -5.71875 3.945312 -5.828125 3.53125 -5.828125 C 2.925781 -5.828125 2.40625 -5.617188 1.96875 -5.203125 C 1.53125 -4.785156 1.3125 -4.09375 1.3125 -3.125 Z M 1.3125 -3.125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "</g>\n",
       "</defs>\n",
       "<g id=\"surface1\">\n",
       "<path d=\"M 124.015625 139.234375 C 128.996094 144.214844 128.996094 152.285156 124.015625 157.265625 C 119.035156 162.246094 110.964844 162.246094 105.984375 157.265625 C 101.003906 152.285156 101.003906 144.214844 105.984375 139.234375 C 110.964844 134.253906 119.035156 134.253906 124.015625 139.234375 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"86.803467\" xlink:href=\"#glyph0-1\" y=\"76.635498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"91.303467\" xlink:href=\"#glyph1-1\" y=\"78.635498\"/>\n",
       "</g>\n",
       "<path d=\"M 176.890625 139.234375 C 181.871094 144.214844 181.871094 152.285156 176.890625 157.265625 C 171.910156 162.246094 163.839844 162.246094 158.859375 157.265625 C 153.878906 152.285156 153.878906 144.214844 158.859375 139.234375 C 163.839844 134.253906 171.910156 134.253906 176.890625 139.234375 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"139.678467\" xlink:href=\"#glyph0-1\" y=\"76.635498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"144.178467\" xlink:href=\"#glyph1-2\" y=\"78.635498\"/>\n",
       "</g>\n",
       "<path d=\"M 157.015625 78.234375 C 161.996094 83.214844 161.996094 91.285156 157.015625 96.265625 C 152.035156 101.246094 143.964844 101.246094 138.984375 96.265625 C 134.003906 91.285156 134.003906 83.214844 138.984375 78.234375 C 143.964844 73.253906 152.035156 73.253906 157.015625 78.234375 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.902344%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"119.550781\" xlink:href=\"#glyph0-2\" y=\"15.635498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"124.556152\" xlink:href=\"#glyph1-2\" y=\"17.635498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"8.98755\" xlink:href=\"#glyph2-1\" y=\"76.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"11.48775\" xlink:href=\"#glyph2-2\" y=\"76.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"16.49355\" xlink:href=\"#glyph2-3\" y=\"76.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"21.49935\" xlink:href=\"#glyph2-4\" y=\"76.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"26.50515\" xlink:href=\"#glyph2-5\" y=\"76.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"29.00535\" xlink:href=\"#glyph2-6\" y=\"76.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"31.50555\" xlink:href=\"#glyph2-7\" y=\"76.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"33.50535\" xlink:href=\"#glyph2-8\" y=\"76.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"38.51115\" xlink:href=\"#glyph2-9\" y=\"76.352783\"/>\n",
       "  <use x=\"43.01115\" xlink:href=\"#glyph2-10\" y=\"76.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"48.01695\" xlink:href=\"#glyph2-11\" y=\"76.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"5.4873\" xlink:href=\"#glyph2-12\" y=\"16.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"12.4875\" xlink:href=\"#glyph2-4\" y=\"16.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"17.4933\" xlink:href=\"#glyph2-5\" y=\"16.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"19.9935\" xlink:href=\"#glyph2-3\" y=\"16.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"24.9993\" xlink:href=\"#glyph2-4\" y=\"16.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"30.0051\" xlink:href=\"#glyph2-5\" y=\"16.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"32.5053\" xlink:href=\"#glyph2-6\" y=\"16.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"35.0055\" xlink:href=\"#glyph2-7\" y=\"16.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"37.0053\" xlink:href=\"#glyph2-8\" y=\"16.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"42.0111\" xlink:href=\"#glyph2-9\" y=\"16.102783\"/>\n",
       "  <use x=\"46.5111\" xlink:href=\"#glyph2-10\" y=\"16.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"51.5169\" xlink:href=\"#glyph2-11\" y=\"16.102783\"/>\n",
       "</g>\n",
       "<path d=\"M 229.765625 139.234375 C 234.746094 144.214844 234.746094 152.285156 229.765625 157.265625 C 224.785156 162.246094 216.714844 162.246094 211.734375 157.265625 C 206.753906 152.285156 206.753906 144.214844 211.734375 139.234375 C 216.714844 134.253906 224.785156 134.253906 229.765625 139.234375 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"192.553467\" xlink:href=\"#glyph0-1\" y=\"76.635498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"197.053467\" xlink:href=\"#glyph1-3\" y=\"78.635498\"/>\n",
       "</g>\n",
       "<path d=\"M 282.640625 138.484375 C 287.621094 143.464844 287.621094 151.535156 282.640625 156.515625 C 277.660156 161.496094 269.589844 161.496094 264.609375 156.515625 C 259.628906 151.535156 259.628906 143.464844 264.609375 138.484375 C 269.589844 133.503906 277.660156 133.503906 282.640625 138.484375 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"245.428467\" xlink:href=\"#glyph0-1\" y=\"75.885498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"249.928467\" xlink:href=\"#glyph1-4\" y=\"77.885498\"/>\n",
       "</g>\n",
       "<path d=\"M 248.265625 78.234375 C 253.246094 83.214844 253.246094 91.285156 248.265625 96.265625 C 243.285156 101.246094 235.214844 101.246094 230.234375 96.265625 C 225.253906 91.285156 225.253906 83.214844 230.234375 78.234375 C 235.214844 73.253906 243.285156 73.253906 248.265625 78.234375 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.902344%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"210.800781\" xlink:href=\"#glyph0-2\" y=\"15.635498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"215.806152\" xlink:href=\"#glyph1-4\" y=\"17.635498\"/>\n",
       "</g>\n",
       "<path d=\"M 202.640625 78.234375 C 207.621094 83.214844 207.621094 91.285156 202.640625 96.265625 C 197.660156 101.246094 189.589844 101.246094 184.609375 96.265625 C 179.628906 91.285156 179.628906 83.214844 184.609375 78.234375 C 189.589844 73.253906 197.660156 73.253906 202.640625 78.234375 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.902344%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"165.175781\" xlink:href=\"#glyph0-2\" y=\"15.635498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"170.181152\" xlink:href=\"#glyph1-3\" y=\"17.635498\"/>\n",
       "</g>\n",
       "<path d=\"M 111.390625 78.234375 C 116.371094 83.214844 116.371094 91.285156 111.390625 96.265625 C 106.410156 101.246094 98.339844 101.246094 93.359375 96.265625 C 88.378906 91.285156 88.378906 83.214844 93.359375 78.234375 C 98.339844 73.253906 106.410156 73.253906 111.390625 78.234375 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.902344%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"73.925781\" xlink:href=\"#glyph0-2\" y=\"15.635498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"78.931152\" xlink:href=\"#glyph1-1\" y=\"17.635498\"/>\n",
       "</g>\n",
       "<path d=\"M 293.890625 78.234375 C 298.871094 83.214844 298.871094 91.285156 293.890625 96.265625 C 288.910156 101.246094 280.839844 101.246094 275.859375 96.265625 C 270.878906 91.285156 270.878906 83.214844 275.859375 78.234375 C 280.839844 73.253906 288.910156 73.253906 293.890625 78.234375 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.902344%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"256.425781\" xlink:href=\"#glyph0-2\" y=\"15.635498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"261.431152\" xlink:href=\"#glyph1-5\" y=\"17.635498\"/>\n",
       "</g>\n",
       "<path d=\"M 112.414062 135.761719 L 106.15625 105.515625 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 105.34375 101.597656 L 106.15625 105.515625 M 104.6875 105.820312 L 105.34375 101.597656 L 107.625 105.210938 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 121.066406 137.03125 L 139.125 103.65625 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 141.027344 100.136719 L 139.125 103.65625 M 137.804688 102.941406 L 141.027344 100.136719 L 140.445312 104.371094 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 125.074219 140.433594 L 178.890625 98.683594 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 182.050781 96.230469 L 178.890625 98.683594 M 177.96875 97.496094 L 182.050781 96.230469 L 179.808594 99.867188 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 126.449219 142.628906 L 222.503906 95.46875 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 226.097656 93.707031 L 222.503906 95.46875 M 221.84375 94.125 L 226.097656 93.707031 L 223.167969 96.816406 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 127.003906 143.941406 L 267.320312 93.554688 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 271.085938 92.203125 L 267.320312 93.554688 M 266.8125 92.140625 L 271.085938 92.203125 L 267.828125 94.964844 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 158.542969 139.558594 L 116.023438 99.960938 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 113.097656 97.234375 L 116.023438 99.960938 M 115 101.058594 L 113.097656 97.234375 L 117.046875 98.863281 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 163.925781 136.125 L 153.777344 104.984375 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 152.539062 101.183594 L 153.777344 104.984375 M 152.351562 105.449219 L 152.539062 101.183594 L 155.203125 104.519531 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 172.835938 136.5 L 186.371094 104.433594 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 187.925781 100.75 L 186.371094 104.433594 M 184.988281 103.851562 L 187.925781 100.75 L 187.753906 105.019531 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 177.566406 139.964844 L 225.070312 99.367188 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 228.113281 96.769531 L 225.070312 99.367188 M 224.097656 98.226562 L 228.113281 96.769531 L 226.046875 100.507812 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 179.183594 142.355469 L 268.335938 95.875 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 271.882812 94.023438 L 268.335938 95.875 M 267.640625 94.542969 L 271.882812 94.023438 L 269.027344 97.203125 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 209.414062 142.40625 L 118.957031 95.792969 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 115.402344 93.960938 L 118.957031 95.792969 M 118.269531 97.128906 L 115.402344 93.960938 L 119.644531 94.460938 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 210.980469 140.058594 L 162.292969 99.234375 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 159.226562 96.664062 L 162.292969 99.234375 M 161.328125 100.382812 L 159.226562 96.664062 L 163.253906 98.082031 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 215.566406 136.597656 L 201.203125 104.292969 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 199.578125 100.640625 L 201.203125 104.292969 M 199.832031 104.902344 L 199.578125 100.640625 L 202.574219 103.683594 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 224.449219 136.046875 L 233.835938 105.101562 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 234.996094 101.273438 L 233.835938 105.101562 M 232.402344 104.664062 L 234.996094 101.273438 L 235.273438 105.535156 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 229.988281 139.460938 L 271.363281 100.105469 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 274.261719 97.347656 L 271.363281 100.105469 M 270.328125 99.015625 L 274.261719 97.347656 L 272.394531 101.191406 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 261.59375 143.265625 L 119.96875 93.441406 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 116.199219 92.113281 L 119.96875 93.441406 M 119.472656 94.855469 L 116.199219 92.113281 L 120.46875 92.027344 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 262.125 141.984375 L 164.820312 95.316406 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 161.210938 93.585938 L 164.820312 95.316406 M 164.171875 96.667969 L 161.210938 93.585938 L 165.46875 93.964844 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 263.4375 139.828125 L 208.523438 98.472656 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 205.328125 96.0625 L 208.523438 98.472656 M 207.621094 99.667969 L 205.328125 96.0625 L 209.425781 97.273438 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 267.304688 136.421875 L 248.492188 103.453125 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 246.511719 99.976562 L 248.492188 103.453125 M 247.191406 104.195312 L 246.511719 99.976562 L 249.796875 102.707031 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 275.964844 134.964844 L 281.453125 105.585938 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "<path d=\"M 282.1875 101.652344 L 281.453125 105.585938 M 279.976562 105.308594 L 282.1875 101.652344 L 282.925781 105.859375 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-24,-74)\"/>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import SVG\n",
    "SVG(filename='../img/singlelayer.svg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If our labels really were related to our input data\n",
    "by an approximately linear function, then this approach would be perfect.\n",
    "But linearity is a *strong assumption*.\n",
    "Linearity implies that for whatever target value we are trying to predict,\n",
    "increasing the value of each of our inputs\n",
    "should either drive the value of the output up or drive it down,\n",
    "irrespective of the value of the other inputs.\n",
    "\n",
    "Sometimes this makes sense!\n",
    "Say we are trying to predict whether an individual\n",
    "will or will not repay a loan.\n",
    "We might reasonably imagine that all else being equal,\n",
    "an applicant with a higher income\n",
    "would be more likely to repay than one with a lower income.\n",
    "In these cases, linear models might perform well,\n",
    "and they might even be hard to beat.\n",
    "\n",
    "But what about classifying images in FashionMNIST?\n",
    "Should increasing the intensity of the pixel at location (13,17)\n",
    "always increase the likelihood that the image depicts a pocketbook?\n",
    "That seems ridiculous because we all know\n",
    "that you cannot make sense out of an image\n",
    "without accounting for the interactions among pixels.\n",
    "\n",
    "\n",
    "\n",
    "### From one to many\n",
    "\n",
    "As another case, consider trying to classify images\n",
    "based on whether they depict *cats* or *dogs* given black-and-white images.\n",
    "\n",
    "If we use a linear model, we'd basically be saying that\n",
    "for each pixel, increasing its value (making it more white)\n",
    "must always increase the probability that the image depicts a dog\n",
    "or must always increase the probability that the image depicts a cat.\n",
    "We would be making the absurd assumption that the only requirement\n",
    "for differentiating cats vs. dogs is to assess how bright they are.\n",
    "That approach is doomed to fail in a work\n",
    "that contains both black dogs and black cats,\n",
    "and both white dogs and white cats.\n",
    "\n",
    "Teasing out what is depicted in an image generally requires\n",
    "allowing more complex relationships between our inputs and outputs.\n",
    "Thus we need models capable of discovering patterns\n",
    "that might be characterized by interactions among the many features.\n",
    "We can over come these limitations of linear models\n",
    "and handle a more general class of functions\n",
    "by incorporating one or more hidden layers.\n",
    "The easiest way to do this is to stack\n",
    "many layers of neurons on top of each other.\n",
    "Each layer feeds into the layer above it, until we generate an output.\n",
    "This architecture is commonly called a *multilayer perceptron*,\n",
    "often abbreviated as *MLP*.\n",
    "The neural network diagram for an MLP looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"150pt\" version=\"1.1\" viewBox=\"0 0 276 150\" width=\"276pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<defs>\n",
       "<g>\n",
       "<symbol id=\"glyph0-0\" overflow=\"visible\">\n",
       "<path d=\"M 1.125 0 L 1.125 -5.625 L 5.625 -5.625 L 5.625 0 Z M 1.265625 -0.140625 L 5.484375 -0.140625 L 5.484375 -5.484375 L 1.265625 -5.484375 Z M 1.265625 -0.140625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-1\" overflow=\"visible\">\n",
       "<path d=\"M -0.015625 0 L 2.015625 -2.375 L 0.859375 -4.671875 L 1.734375 -4.671875 L 2.125 -3.84375 C 2.269531 -3.53125 2.398438 -3.226562 2.515625 -2.9375 L 3.875 -4.671875 L 4.84375 -4.671875 L 2.875 -2.3125 L 4.0625 0 L 3.171875 0 L 2.71875 -0.953125 C 2.613281 -1.148438 2.5 -1.398438 2.375 -1.703125 L 0.984375 0 Z M -0.015625 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-2\" overflow=\"visible\">\n",
       "<path d=\"M 0.296875 0 L 1.640625 -6.4375 L 2.4375 -6.4375 L 1.921875 -3.96875 C 2.222656 -4.257812 2.5 -4.460938 2.75 -4.578125 C 3.007812 -4.703125 3.269531 -4.765625 3.53125 -4.765625 C 3.925781 -4.765625 4.226562 -4.660156 4.4375 -4.453125 C 4.65625 -4.253906 4.765625 -3.988281 4.765625 -3.65625 C 4.765625 -3.5 4.71875 -3.195312 4.625 -2.75 L 4.046875 0 L 3.25 0 L 3.84375 -2.828125 C 3.925781 -3.234375 3.96875 -3.488281 3.96875 -3.59375 C 3.96875 -3.75 3.914062 -3.875 3.8125 -3.96875 C 3.707031 -4.070312 3.554688 -4.125 3.359375 -4.125 C 3.066406 -4.125 2.789062 -4.046875 2.53125 -3.890625 C 2.269531 -3.742188 2.066406 -3.535156 1.921875 -3.265625 C 1.773438 -3.003906 1.640625 -2.582031 1.515625 -2 L 1.09375 0 Z M 0.296875 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-3\" overflow=\"visible\">\n",
       "<path d=\"M 0.4375 -1.765625 C 0.4375 -2.679688 0.707031 -3.4375 1.25 -4.03125 C 1.6875 -4.519531 2.265625 -4.765625 2.984375 -4.765625 C 3.546875 -4.765625 4 -4.585938 4.34375 -4.234375 C 4.6875 -3.890625 4.859375 -3.421875 4.859375 -2.828125 C 4.859375 -2.285156 4.75 -1.785156 4.53125 -1.328125 C 4.3125 -0.867188 4.003906 -0.515625 3.609375 -0.265625 C 3.210938 -0.015625 2.789062 0.109375 2.34375 0.109375 C 1.976562 0.109375 1.644531 0.03125 1.34375 -0.125 C 1.050781 -0.28125 0.828125 -0.5 0.671875 -0.78125 C 0.515625 -1.070312 0.4375 -1.398438 0.4375 -1.765625 Z M 1.234375 -1.84375 C 1.234375 -1.40625 1.335938 -1.070312 1.546875 -0.84375 C 1.765625 -0.625 2.035156 -0.515625 2.359375 -0.515625 C 2.523438 -0.515625 2.691406 -0.546875 2.859375 -0.609375 C 3.023438 -0.679688 3.179688 -0.785156 3.328125 -0.921875 C 3.472656 -1.066406 3.59375 -1.226562 3.6875 -1.40625 C 3.789062 -1.582031 3.875 -1.773438 3.9375 -1.984375 C 4.03125 -2.273438 4.078125 -2.554688 4.078125 -2.828125 C 4.078125 -3.242188 3.96875 -3.566406 3.75 -3.796875 C 3.539062 -4.035156 3.273438 -4.15625 2.953125 -4.15625 C 2.703125 -4.15625 2.472656 -4.09375 2.265625 -3.96875 C 2.066406 -3.851562 1.882812 -3.679688 1.71875 -3.453125 C 1.550781 -3.222656 1.425781 -2.957031 1.34375 -2.65625 C 1.269531 -2.351562 1.234375 -2.082031 1.234375 -1.84375 Z M 1.234375 -1.84375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-0\" overflow=\"visible\">\n",
       "<path d=\"M 0.875 0 L 0.875 -4.375 L 4.375 -4.375 L 4.375 0 Z M 0.984375 -0.109375 L 4.265625 -0.109375 L 4.265625 -4.265625 L 0.984375 -4.265625 Z M 0.984375 -0.109375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-1\" overflow=\"visible\">\n",
       "<path d=\"M 1.6875 0 L 2.484375 -3.78125 C 2.140625 -3.507812 1.65625 -3.296875 1.03125 -3.140625 L 1.15625 -3.703125 C 1.457031 -3.828125 1.757812 -3.988281 2.0625 -4.1875 C 2.363281 -4.382812 2.585938 -4.554688 2.734375 -4.703125 C 2.828125 -4.796875 2.914062 -4.90625 3 -5.03125 L 3.359375 -5.03125 L 2.3125 0 Z M 1.6875 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-2\" overflow=\"visible\">\n",
       "<path d=\"M 0.40625 0 C 0.46875 -0.300781 0.554688 -0.550781 0.671875 -0.75 C 0.785156 -0.945312 0.9375 -1.132812 1.125 -1.3125 C 1.3125 -1.5 1.671875 -1.804688 2.203125 -2.234375 C 2.523438 -2.492188 2.75 -2.6875 2.875 -2.8125 C 3.039062 -2.988281 3.160156 -3.160156 3.234375 -3.328125 C 3.285156 -3.441406 3.3125 -3.566406 3.3125 -3.703125 C 3.3125 -3.929688 3.226562 -4.125 3.0625 -4.28125 C 2.90625 -4.445312 2.707031 -4.53125 2.46875 -4.53125 C 2.238281 -4.53125 2.035156 -4.445312 1.859375 -4.28125 C 1.679688 -4.125 1.554688 -3.863281 1.484375 -3.5 L 0.875 -3.59375 C 0.9375 -4.039062 1.109375 -4.390625 1.390625 -4.640625 C 1.679688 -4.898438 2.039062 -5.03125 2.46875 -5.03125 C 2.75 -5.03125 3.003906 -4.96875 3.234375 -4.84375 C 3.460938 -4.726562 3.632812 -4.5625 3.75 -4.34375 C 3.875 -4.132812 3.9375 -3.914062 3.9375 -3.6875 C 3.9375 -3.351562 3.816406 -3.035156 3.578125 -2.734375 C 3.429688 -2.535156 3.003906 -2.15625 2.296875 -1.59375 C 1.984375 -1.351562 1.753906 -1.15625 1.609375 -1 C 1.460938 -0.84375 1.351562 -0.695312 1.28125 -0.5625 L 3.515625 -0.5625 L 3.390625 0 Z M 0.40625 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-3\" overflow=\"visible\">\n",
       "<path d=\"M 0.390625 -1.3125 L 0.984375 -1.390625 C 1.023438 -1.035156 1.125 -0.78125 1.28125 -0.625 C 1.4375 -0.476562 1.644531 -0.40625 1.90625 -0.40625 C 2.207031 -0.40625 2.46875 -0.515625 2.6875 -0.734375 C 2.914062 -0.953125 3.03125 -1.203125 3.03125 -1.484375 C 3.03125 -1.734375 2.945312 -1.9375 2.78125 -2.09375 C 2.613281 -2.257812 2.390625 -2.34375 2.109375 -2.34375 C 2.078125 -2.34375 2.007812 -2.335938 1.90625 -2.328125 L 2.015625 -2.84375 C 2.078125 -2.832031 2.132812 -2.828125 2.1875 -2.828125 C 2.539062 -2.828125 2.8125 -2.910156 3 -3.078125 C 3.1875 -3.253906 3.28125 -3.46875 3.28125 -3.71875 C 3.28125 -3.945312 3.203125 -4.140625 3.046875 -4.296875 C 2.890625 -4.453125 2.703125 -4.53125 2.484375 -4.53125 C 2.253906 -4.53125 2.050781 -4.445312 1.875 -4.28125 C 1.695312 -4.125 1.585938 -3.898438 1.546875 -3.609375 L 0.9375 -3.734375 C 1.03125 -4.148438 1.21875 -4.46875 1.5 -4.6875 C 1.789062 -4.914062 2.128906 -5.03125 2.515625 -5.03125 C 2.929688 -5.03125 3.265625 -4.90625 3.515625 -4.65625 C 3.773438 -4.40625 3.90625 -4.101562 3.90625 -3.75 C 3.90625 -3.476562 3.832031 -3.242188 3.6875 -3.046875 C 3.550781 -2.847656 3.347656 -2.6875 3.078125 -2.5625 C 3.265625 -2.445312 3.40625 -2.304688 3.5 -2.140625 C 3.601562 -1.972656 3.65625 -1.785156 3.65625 -1.578125 C 3.65625 -1.128906 3.488281 -0.738281 3.15625 -0.40625 C 2.820312 -0.0820312 2.421875 0.078125 1.953125 0.078125 C 1.492188 0.078125 1.125 -0.046875 0.84375 -0.296875 C 0.570312 -0.546875 0.421875 -0.882812 0.390625 -1.3125 Z M 0.390625 -1.3125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-4\" overflow=\"visible\">\n",
       "<path d=\"M 2.09375 0 L 2.359375 -1.28125 L 0.3125 -1.28125 L 0.453125 -1.890625 L 3.25 -5.015625 L 3.765625 -5.015625 L 3.09375 -1.828125 L 3.796875 -1.828125 L 3.6875 -1.28125 L 2.984375 -1.28125 L 2.703125 0 Z M 2.46875 -1.828125 L 2.90625 -3.921875 L 1.046875 -1.828125 Z M 2.46875 -1.828125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-5\" overflow=\"visible\">\n",
       "<path d=\"M 0.484375 -1.4375 L 1.125 -1.5 C 1.113281 -1.40625 1.109375 -1.347656 1.109375 -1.328125 C 1.109375 -1.179688 1.144531 -1.03125 1.21875 -0.875 C 1.300781 -0.726562 1.40625 -0.613281 1.53125 -0.53125 C 1.664062 -0.445312 1.804688 -0.40625 1.953125 -0.40625 C 2.140625 -0.40625 2.332031 -0.46875 2.53125 -0.59375 C 2.726562 -0.726562 2.890625 -0.921875 3.015625 -1.171875 C 3.140625 -1.429688 3.203125 -1.6875 3.203125 -1.9375 C 3.203125 -2.21875 3.117188 -2.441406 2.953125 -2.609375 C 2.785156 -2.773438 2.566406 -2.859375 2.296875 -2.859375 C 2.117188 -2.859375 1.945312 -2.8125 1.78125 -2.71875 C 1.625 -2.632812 1.476562 -2.507812 1.34375 -2.34375 L 0.796875 -2.375 L 1.5625 -4.9375 L 4 -4.9375 L 3.890625 -4.375 L 1.984375 -4.375 L 1.609375 -3.09375 C 1.742188 -3.195312 1.882812 -3.273438 2.03125 -3.328125 C 2.1875 -3.378906 2.34375 -3.40625 2.5 -3.40625 C 2.882812 -3.40625 3.195312 -3.28125 3.4375 -3.03125 C 3.6875 -2.78125 3.8125 -2.429688 3.8125 -1.984375 C 3.8125 -1.597656 3.726562 -1.242188 3.5625 -0.921875 C 3.394531 -0.597656 3.160156 -0.347656 2.859375 -0.171875 C 2.566406 -0.00390625 2.25 0.078125 1.90625 0.078125 C 1.625 0.078125 1.367188 0.015625 1.140625 -0.109375 C 0.921875 -0.234375 0.753906 -0.410156 0.640625 -0.640625 C 0.535156 -0.867188 0.484375 -1.097656 0.484375 -1.328125 C 0.484375 -1.347656 0.484375 -1.382812 0.484375 -1.4375 Z M 0.484375 -1.4375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-0\" overflow=\"visible\">\n",
       "<path d=\"M 1.125 0 L 1.125 -5.625 L 5.625 -5.625 L 5.625 0 Z M 1.265625 -0.140625 L 5.484375 -0.140625 L 5.484375 -5.484375 L 1.265625 -5.484375 Z M 1.265625 -0.140625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-1\" overflow=\"visible\">\n",
       "<path d=\"M 0.84375 0 L 0.84375 -6.4375 L 1.6875 -6.4375 L 1.6875 0 Z M 0.84375 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-2\" overflow=\"visible\">\n",
       "<path d=\"M 0.59375 0 L 0.59375 -4.671875 L 1.3125 -4.671875 L 1.3125 -4 C 1.644531 -4.507812 2.140625 -4.765625 2.796875 -4.765625 C 3.078125 -4.765625 3.332031 -4.710938 3.5625 -4.609375 C 3.800781 -4.515625 3.976562 -4.382812 4.09375 -4.21875 C 4.207031 -4.0625 4.289062 -3.867188 4.34375 -3.640625 C 4.375 -3.492188 4.390625 -3.238281 4.390625 -2.875 L 4.390625 0 L 3.59375 0 L 3.59375 -2.84375 C 3.59375 -3.164062 3.5625 -3.40625 3.5 -3.5625 C 3.4375 -3.71875 3.328125 -3.84375 3.171875 -3.9375 C 3.015625 -4.039062 2.832031 -4.09375 2.625 -4.09375 C 2.289062 -4.09375 2 -3.984375 1.75 -3.765625 C 1.507812 -3.554688 1.390625 -3.148438 1.390625 -2.546875 L 1.390625 0 Z M 0.59375 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-3\" overflow=\"visible\">\n",
       "<path d=\"M 0.59375 1.78125 L 0.59375 -4.671875 L 1.3125 -4.671875 L 1.3125 -4.0625 C 1.476562 -4.300781 1.664062 -4.476562 1.875 -4.59375 C 2.09375 -4.707031 2.359375 -4.765625 2.671875 -4.765625 C 3.066406 -4.765625 3.414062 -4.660156 3.71875 -4.453125 C 4.019531 -4.253906 4.25 -3.96875 4.40625 -3.59375 C 4.5625 -3.21875 4.640625 -2.8125 4.640625 -2.375 C 4.640625 -1.894531 4.550781 -1.460938 4.375 -1.078125 C 4.207031 -0.691406 3.960938 -0.394531 3.640625 -0.1875 C 3.316406 0.0078125 2.972656 0.109375 2.609375 0.109375 C 2.347656 0.109375 2.113281 0.0507812 1.90625 -0.0625 C 1.695312 -0.175781 1.523438 -0.316406 1.390625 -0.484375 L 1.390625 1.78125 Z M 1.3125 -2.3125 C 1.3125 -1.707031 1.429688 -1.257812 1.671875 -0.96875 C 1.921875 -0.6875 2.21875 -0.546875 2.5625 -0.546875 C 2.90625 -0.546875 3.203125 -0.691406 3.453125 -0.984375 C 3.710938 -1.285156 3.84375 -1.75 3.84375 -2.375 C 3.84375 -2.96875 3.71875 -3.410156 3.46875 -3.703125 C 3.226562 -4.003906 2.9375 -4.15625 2.59375 -4.15625 C 2.257812 -4.15625 1.960938 -3.992188 1.703125 -3.671875 C 1.441406 -3.359375 1.3125 -2.90625 1.3125 -2.3125 Z M 1.3125 -2.3125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-4\" overflow=\"visible\">\n",
       "<path d=\"M 3.65625 0 L 3.65625 -0.6875 C 3.289062 -0.15625 2.796875 0.109375 2.171875 0.109375 C 1.898438 0.109375 1.644531 0.0546875 1.40625 -0.046875 C 1.164062 -0.160156 0.984375 -0.296875 0.859375 -0.453125 C 0.742188 -0.609375 0.664062 -0.800781 0.625 -1.03125 C 0.59375 -1.1875 0.578125 -1.4375 0.578125 -1.78125 L 0.578125 -4.671875 L 1.359375 -4.671875 L 1.359375 -2.078125 C 1.359375 -1.660156 1.378906 -1.382812 1.421875 -1.25 C 1.460938 -1.039062 1.5625 -0.875 1.71875 -0.75 C 1.882812 -0.632812 2.085938 -0.578125 2.328125 -0.578125 C 2.566406 -0.578125 2.789062 -0.632812 3 -0.75 C 3.207031 -0.875 3.351562 -1.039062 3.4375 -1.25 C 3.519531 -1.457031 3.5625 -1.765625 3.5625 -2.171875 L 3.5625 -4.671875 L 4.359375 -4.671875 L 4.359375 0 Z M 3.65625 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-5\" overflow=\"visible\">\n",
       "<path d=\"M 2.328125 -0.703125 L 2.4375 -0.015625 C 2.207031 0.0351562 2.007812 0.0625 1.84375 0.0625 C 1.550781 0.0625 1.328125 0.015625 1.171875 -0.078125 C 1.015625 -0.171875 0.898438 -0.289062 0.828125 -0.4375 C 0.765625 -0.582031 0.734375 -0.890625 0.734375 -1.359375 L 0.734375 -4.046875 L 0.15625 -4.046875 L 0.15625 -4.671875 L 0.734375 -4.671875 L 0.734375 -5.828125 L 1.53125 -6.296875 L 1.53125 -4.671875 L 2.328125 -4.671875 L 2.328125 -4.046875 L 1.53125 -4.046875 L 1.53125 -1.328125 C 1.53125 -1.097656 1.539062 -0.953125 1.5625 -0.890625 C 1.59375 -0.828125 1.640625 -0.773438 1.703125 -0.734375 C 1.765625 -0.691406 1.851562 -0.671875 1.96875 -0.671875 C 2.0625 -0.671875 2.179688 -0.679688 2.328125 -0.703125 Z M 2.328125 -0.703125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-6\" overflow=\"visible\">\n",
       "<path d=\"\" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-7\" overflow=\"visible\">\n",
       "<path d=\"M 0.578125 0 L 0.578125 -6.4375 L 1.359375 -6.4375 L 1.359375 0 Z M 0.578125 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-8\" overflow=\"visible\">\n",
       "<path d=\"M 3.640625 -0.578125 C 3.347656 -0.328125 3.066406 -0.148438 2.796875 -0.046875 C 2.523438 0.0546875 2.234375 0.109375 1.921875 0.109375 C 1.410156 0.109375 1.015625 -0.015625 0.734375 -0.265625 C 0.460938 -0.515625 0.328125 -0.835938 0.328125 -1.234375 C 0.328125 -1.460938 0.378906 -1.671875 0.484375 -1.859375 C 0.585938 -2.046875 0.722656 -2.195312 0.890625 -2.3125 C 1.054688 -2.425781 1.242188 -2.515625 1.453125 -2.578125 C 1.609375 -2.609375 1.84375 -2.644531 2.15625 -2.6875 C 2.800781 -2.757812 3.273438 -2.851562 3.578125 -2.96875 C 3.578125 -3.070312 3.578125 -3.140625 3.578125 -3.171875 C 3.578125 -3.492188 3.503906 -3.71875 3.359375 -3.84375 C 3.148438 -4.03125 2.847656 -4.125 2.453125 -4.125 C 2.078125 -4.125 1.800781 -4.054688 1.625 -3.921875 C 1.445312 -3.796875 1.316406 -3.566406 1.234375 -3.234375 L 0.46875 -3.328125 C 0.53125 -3.660156 0.640625 -3.925781 0.796875 -4.125 C 0.960938 -4.332031 1.195312 -4.488281 1.5 -4.59375 C 1.8125 -4.707031 2.164062 -4.765625 2.5625 -4.765625 C 2.96875 -4.765625 3.289062 -4.71875 3.53125 -4.625 C 3.78125 -4.53125 3.960938 -4.410156 4.078125 -4.265625 C 4.203125 -4.128906 4.285156 -3.953125 4.328125 -3.734375 C 4.359375 -3.597656 4.375 -3.359375 4.375 -3.015625 L 4.375 -1.953125 C 4.375 -1.222656 4.390625 -0.757812 4.421875 -0.5625 C 4.453125 -0.363281 4.519531 -0.175781 4.625 0 L 3.796875 0 C 3.710938 -0.164062 3.660156 -0.359375 3.640625 -0.578125 Z M 3.578125 -2.34375 C 3.285156 -2.226562 2.851562 -2.128906 2.28125 -2.046875 C 1.957031 -1.992188 1.726562 -1.9375 1.59375 -1.875 C 1.457031 -1.820312 1.351562 -1.738281 1.28125 -1.625 C 1.207031 -1.507812 1.171875 -1.382812 1.171875 -1.25 C 1.171875 -1.039062 1.25 -0.863281 1.40625 -0.71875 C 1.5625 -0.582031 1.796875 -0.515625 2.109375 -0.515625 C 2.410156 -0.515625 2.679688 -0.582031 2.921875 -0.71875 C 3.160156 -0.851562 3.335938 -1.035156 3.453125 -1.265625 C 3.535156 -1.441406 3.578125 -1.703125 3.578125 -2.046875 Z M 3.578125 -2.34375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-9\" overflow=\"visible\">\n",
       "<path d=\"M 0.5625 1.796875 L 0.46875 1.0625 C 0.644531 1.101562 0.796875 1.125 0.921875 1.125 C 1.097656 1.125 1.238281 1.09375 1.34375 1.03125 C 1.445312 0.976562 1.535156 0.898438 1.609375 0.796875 C 1.648438 0.710938 1.726562 0.515625 1.84375 0.203125 C 1.863281 0.160156 1.890625 0.0976562 1.921875 0.015625 L 0.140625 -4.671875 L 1 -4.671875 L 1.96875 -1.96875 C 2.09375 -1.625 2.207031 -1.265625 2.3125 -0.890625 C 2.394531 -1.242188 2.5 -1.597656 2.625 -1.953125 L 3.625 -4.671875 L 4.421875 -4.671875 L 2.640625 0.078125 C 2.453125 0.585938 2.304688 0.941406 2.203125 1.140625 C 2.054688 1.398438 1.894531 1.585938 1.71875 1.703125 C 1.539062 1.828125 1.320312 1.890625 1.0625 1.890625 C 0.914062 1.890625 0.75 1.859375 0.5625 1.796875 Z M 0.5625 1.796875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-10\" overflow=\"visible\">\n",
       "<path d=\"M 3.78125 -1.5 L 4.609375 -1.40625 C 4.472656 -0.925781 4.226562 -0.550781 3.875 -0.28125 C 3.53125 -0.0195312 3.085938 0.109375 2.546875 0.109375 C 1.867188 0.109375 1.328125 -0.0976562 0.921875 -0.515625 C 0.523438 -0.941406 0.328125 -1.535156 0.328125 -2.296875 C 0.328125 -3.078125 0.53125 -3.679688 0.9375 -4.109375 C 1.34375 -4.546875 1.867188 -4.765625 2.515625 -4.765625 C 3.140625 -4.765625 3.644531 -4.550781 4.03125 -4.125 C 4.425781 -3.707031 4.625 -3.113281 4.625 -2.34375 C 4.625 -2.289062 4.625 -2.21875 4.625 -2.125 L 1.140625 -2.125 C 1.171875 -1.613281 1.316406 -1.222656 1.578125 -0.953125 C 1.835938 -0.679688 2.164062 -0.546875 2.5625 -0.546875 C 2.851562 -0.546875 3.097656 -0.617188 3.296875 -0.765625 C 3.503906 -0.921875 3.664062 -1.164062 3.78125 -1.5 Z M 1.1875 -2.78125 L 3.796875 -2.78125 C 3.765625 -3.175781 3.664062 -3.472656 3.5 -3.671875 C 3.25 -3.972656 2.921875 -4.125 2.515625 -4.125 C 2.148438 -4.125 1.84375 -4 1.59375 -3.75 C 1.351562 -3.507812 1.21875 -3.1875 1.1875 -2.78125 Z M 1.1875 -2.78125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-11\" overflow=\"visible\">\n",
       "<path d=\"M 0.578125 0 L 0.578125 -4.671875 L 1.296875 -4.671875 L 1.296875 -3.953125 C 1.472656 -4.285156 1.640625 -4.503906 1.796875 -4.609375 C 1.953125 -4.710938 2.125 -4.765625 2.3125 -4.765625 C 2.570312 -4.765625 2.84375 -4.679688 3.125 -4.515625 L 2.84375 -3.78125 C 2.65625 -3.894531 2.460938 -3.953125 2.265625 -3.953125 C 2.097656 -3.953125 1.941406 -3.898438 1.796875 -3.796875 C 1.660156 -3.691406 1.5625 -3.546875 1.5 -3.359375 C 1.414062 -3.078125 1.375 -2.769531 1.375 -2.4375 L 1.375 0 Z M 0.578125 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-12\" overflow=\"visible\">\n",
       "<path d=\"M 0.71875 0 L 0.71875 -6.4375 L 1.578125 -6.4375 L 1.578125 -3.796875 L 4.921875 -3.796875 L 4.921875 -6.4375 L 5.78125 -6.4375 L 5.78125 0 L 4.921875 0 L 4.921875 -3.03125 L 1.578125 -3.03125 L 1.578125 0 Z M 0.71875 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-13\" overflow=\"visible\">\n",
       "<path d=\"M 0.59375 -5.53125 L 0.59375 -6.4375 L 1.390625 -6.4375 L 1.390625 -5.53125 Z M 0.59375 0 L 0.59375 -4.671875 L 1.390625 -4.671875 L 1.390625 0 Z M 0.59375 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-14\" overflow=\"visible\">\n",
       "<path d=\"M 3.625 0 L 3.625 -0.59375 C 3.320312 -0.125 2.882812 0.109375 2.3125 0.109375 C 1.945312 0.109375 1.609375 0.00390625 1.296875 -0.203125 C 0.984375 -0.410156 0.738281 -0.695312 0.5625 -1.0625 C 0.394531 -1.425781 0.3125 -1.847656 0.3125 -2.328125 C 0.3125 -2.796875 0.390625 -3.21875 0.546875 -3.59375 C 0.703125 -3.976562 0.929688 -4.269531 1.234375 -4.46875 C 1.546875 -4.664062 1.894531 -4.765625 2.28125 -4.765625 C 2.5625 -4.765625 2.8125 -4.707031 3.03125 -4.59375 C 3.25 -4.476562 3.425781 -4.320312 3.5625 -4.125 L 3.5625 -6.4375 L 4.359375 -6.4375 L 4.359375 0 Z M 1.125 -2.328125 C 1.125 -1.734375 1.25 -1.285156 1.5 -0.984375 C 1.75 -0.691406 2.046875 -0.546875 2.390625 -0.546875 C 2.734375 -0.546875 3.023438 -0.6875 3.265625 -0.96875 C 3.515625 -1.25 3.640625 -1.679688 3.640625 -2.265625 C 3.640625 -2.898438 3.515625 -3.367188 3.265625 -3.671875 C 3.015625 -3.972656 2.710938 -4.125 2.359375 -4.125 C 2.003906 -4.125 1.707031 -3.976562 1.46875 -3.6875 C 1.238281 -3.394531 1.125 -2.941406 1.125 -2.328125 Z M 1.125 -2.328125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-15\" overflow=\"visible\">\n",
       "<path d=\"M 0.4375 -3.140625 C 0.4375 -4.203125 0.722656 -5.035156 1.296875 -5.640625 C 1.867188 -6.253906 2.609375 -6.5625 3.515625 -6.5625 C 4.109375 -6.5625 4.644531 -6.414062 5.125 -6.125 C 5.601562 -5.84375 5.96875 -5.445312 6.21875 -4.9375 C 6.46875 -4.425781 6.59375 -3.851562 6.59375 -3.21875 C 6.59375 -2.5625 6.460938 -1.972656 6.203125 -1.453125 C 5.941406 -0.941406 5.566406 -0.550781 5.078125 -0.28125 C 4.597656 -0.0195312 4.078125 0.109375 3.515625 0.109375 C 2.910156 0.109375 2.367188 -0.0351562 1.890625 -0.328125 C 1.410156 -0.617188 1.046875 -1.019531 0.796875 -1.53125 C 0.554688 -2.039062 0.4375 -2.578125 0.4375 -3.140625 Z M 1.3125 -3.125 C 1.3125 -2.34375 1.519531 -1.726562 1.9375 -1.28125 C 2.351562 -0.84375 2.878906 -0.625 3.515625 -0.625 C 4.148438 -0.625 4.675781 -0.847656 5.09375 -1.296875 C 5.507812 -1.742188 5.71875 -2.382812 5.71875 -3.21875 C 5.71875 -3.738281 5.628906 -4.191406 5.453125 -4.578125 C 5.273438 -4.972656 5.015625 -5.28125 4.671875 -5.5 C 4.328125 -5.71875 3.945312 -5.828125 3.53125 -5.828125 C 2.925781 -5.828125 2.40625 -5.617188 1.96875 -5.203125 C 1.53125 -4.785156 1.3125 -4.09375 1.3125 -3.125 Z M 1.3125 -3.125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "</g>\n",
       "</defs>\n",
       "<g id=\"surface1\">\n",
       "<path d=\"M 124.015625 139.234375 C 128.996094 144.214844 128.996094 152.285156 124.015625 157.265625 C 119.035156 162.246094 110.964844 162.246094 105.984375 157.265625 C 101.003906 152.285156 101.003906 144.214844 105.984375 139.234375 C 110.964844 134.253906 119.035156 134.253906 124.015625 139.234375 \" style=\"fill-rule:nonzero;fill:rgb(69.802856%,85.096741%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"87.803467\" xlink:href=\"#glyph0-1\" y=\"138.635498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"92.303467\" xlink:href=\"#glyph1-1\" y=\"140.635498\"/>\n",
       "</g>\n",
       "<path d=\"M 176.890625 139.234375 C 181.871094 144.214844 181.871094 152.285156 176.890625 157.265625 C 171.910156 162.246094 163.839844 162.246094 158.859375 157.265625 C 153.878906 152.285156 153.878906 144.214844 158.859375 139.234375 C 163.839844 134.253906 171.910156 134.253906 176.890625 139.234375 \" style=\"fill-rule:nonzero;fill:rgb(69.802856%,85.096741%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"140.678467\" xlink:href=\"#glyph0-1\" y=\"138.635498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"145.178467\" xlink:href=\"#glyph1-2\" y=\"140.635498\"/>\n",
       "</g>\n",
       "<path d=\"M 157.015625 78.234375 C 161.996094 83.214844 161.996094 91.285156 157.015625 96.265625 C 152.035156 101.246094 143.964844 101.246094 138.984375 96.265625 C 134.003906 91.285156 134.003906 83.214844 138.984375 78.234375 C 143.964844 73.253906 152.035156 73.253906 157.015625 78.234375 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.900818%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"120.550781\" xlink:href=\"#glyph0-2\" y=\"77.635498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"125.556152\" xlink:href=\"#glyph1-2\" y=\"79.635498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"9.98755\" xlink:href=\"#glyph2-1\" y=\"138.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"12.48775\" xlink:href=\"#glyph2-2\" y=\"138.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"17.49355\" xlink:href=\"#glyph2-3\" y=\"138.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"22.49935\" xlink:href=\"#glyph2-4\" y=\"138.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"27.50515\" xlink:href=\"#glyph2-5\" y=\"138.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"30.00535\" xlink:href=\"#glyph2-6\" y=\"138.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"32.50555\" xlink:href=\"#glyph2-7\" y=\"138.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"34.50535\" xlink:href=\"#glyph2-8\" y=\"138.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"39.51115\" xlink:href=\"#glyph2-9\" y=\"138.352783\"/>\n",
       "  <use x=\"44.01115\" xlink:href=\"#glyph2-10\" y=\"138.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"49.01695\" xlink:href=\"#glyph2-11\" y=\"138.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"5.73584\" xlink:href=\"#glyph2-12\" y=\"78.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"12.23564\" xlink:href=\"#glyph2-13\" y=\"78.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"14.23544\" xlink:href=\"#glyph2-14\" y=\"78.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"19.24124\" xlink:href=\"#glyph2-14\" y=\"78.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"24.24704\" xlink:href=\"#glyph2-10\" y=\"78.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"29.25284\" xlink:href=\"#glyph2-2\" y=\"78.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"34.25864\" xlink:href=\"#glyph2-6\" y=\"78.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"36.75884\" xlink:href=\"#glyph2-7\" y=\"78.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"38.75864\" xlink:href=\"#glyph2-8\" y=\"78.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"43.76444\" xlink:href=\"#glyph2-9\" y=\"78.102783\"/>\n",
       "  <use x=\"48.26444\" xlink:href=\"#glyph2-10\" y=\"78.102783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"53.27024\" xlink:href=\"#glyph2-11\" y=\"78.102783\"/>\n",
       "</g>\n",
       "<path d=\"M 229.765625 139.234375 C 234.746094 144.214844 234.746094 152.285156 229.765625 157.265625 C 224.785156 162.246094 216.714844 162.246094 211.734375 157.265625 C 206.753906 152.285156 206.753906 144.214844 211.734375 139.234375 C 216.714844 134.253906 224.785156 134.253906 229.765625 139.234375 \" style=\"fill-rule:nonzero;fill:rgb(69.802856%,85.096741%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"193.553467\" xlink:href=\"#glyph0-1\" y=\"138.635498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"198.053467\" xlink:href=\"#glyph1-3\" y=\"140.635498\"/>\n",
       "</g>\n",
       "<path d=\"M 282.640625 138.484375 C 287.621094 143.464844 287.621094 151.535156 282.640625 156.515625 C 277.660156 161.496094 269.589844 161.496094 264.609375 156.515625 C 259.628906 151.535156 259.628906 143.464844 264.609375 138.484375 C 269.589844 133.503906 277.660156 133.503906 282.640625 138.484375 \" style=\"fill-rule:nonzero;fill:rgb(69.802856%,85.096741%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"246.428467\" xlink:href=\"#glyph0-1\" y=\"137.885498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"250.928467\" xlink:href=\"#glyph1-4\" y=\"139.885498\"/>\n",
       "</g>\n",
       "<path d=\"M 248.265625 78.234375 C 253.246094 83.214844 253.246094 91.285156 248.265625 96.265625 C 243.285156 101.246094 235.214844 101.246094 230.234375 96.265625 C 225.253906 91.285156 225.253906 83.214844 230.234375 78.234375 C 235.214844 73.253906 243.285156 73.253906 248.265625 78.234375 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.900818%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"211.800781\" xlink:href=\"#glyph0-2\" y=\"77.635498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"216.806152\" xlink:href=\"#glyph1-4\" y=\"79.635498\"/>\n",
       "</g>\n",
       "<path d=\"M 202.640625 78.234375 C 207.621094 83.214844 207.621094 91.285156 202.640625 96.265625 C 197.660156 101.246094 189.589844 101.246094 184.609375 96.265625 C 179.628906 91.285156 179.628906 83.214844 184.609375 78.234375 C 189.589844 73.253906 197.660156 73.253906 202.640625 78.234375 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.900818%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"166.175781\" xlink:href=\"#glyph0-2\" y=\"77.635498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"171.181152\" xlink:href=\"#glyph1-3\" y=\"79.635498\"/>\n",
       "</g>\n",
       "<path d=\"M 149.765625 17.234375 C 154.746094 22.214844 154.746094 30.285156 149.765625 35.265625 C 144.785156 40.246094 136.714844 40.246094 131.734375 35.265625 C 126.753906 30.285156 126.753906 22.214844 131.734375 17.234375 C 136.714844 12.253906 144.785156 12.253906 149.765625 17.234375 \" style=\"fill-rule:nonzero;fill:rgb(69.802856%,85.096741%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"113.300781\" xlink:href=\"#glyph0-3\" y=\"16.635498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"118.306152\" xlink:href=\"#glyph1-1\" y=\"18.635498\"/>\n",
       "</g>\n",
       "<path d=\"M 202.640625 17.234375 C 207.621094 22.214844 207.621094 30.285156 202.640625 35.265625 C 197.660156 40.246094 189.589844 40.246094 184.609375 35.265625 C 179.628906 30.285156 179.628906 22.214844 184.609375 17.234375 C 189.589844 12.253906 197.660156 12.253906 202.640625 17.234375 \" style=\"fill-rule:nonzero;fill:rgb(69.802856%,85.096741%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"166.175781\" xlink:href=\"#glyph0-3\" y=\"16.635498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"171.181152\" xlink:href=\"#glyph1-2\" y=\"18.635498\"/>\n",
       "</g>\n",
       "<path d=\"M 255.515625 16.484375 C 260.496094 21.464844 260.496094 29.535156 255.515625 34.515625 C 250.535156 39.496094 242.464844 39.496094 237.484375 34.515625 C 232.503906 29.535156 232.503906 21.464844 237.484375 16.484375 C 242.464844 11.503906 250.535156 11.503906 255.515625 16.484375 \" style=\"fill-rule:nonzero;fill:rgb(69.802856%,85.096741%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"219.050781\" xlink:href=\"#glyph0-3\" y=\"15.885498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"224.056152\" xlink:href=\"#glyph1-3\" y=\"17.885498\"/>\n",
       "</g>\n",
       "<path d=\"M 111.390625 78.234375 C 116.371094 83.214844 116.371094 91.285156 111.390625 96.265625 C 106.410156 101.246094 98.339844 101.246094 93.359375 96.265625 C 88.378906 91.285156 88.378906 83.214844 93.359375 78.234375 C 98.339844 73.253906 106.410156 73.253906 111.390625 78.234375 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.900818%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"74.925781\" xlink:href=\"#glyph0-2\" y=\"77.635498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"79.931152\" xlink:href=\"#glyph1-1\" y=\"79.635498\"/>\n",
       "</g>\n",
       "<path d=\"M 293.890625 78.234375 C 298.871094 83.214844 298.871094 91.285156 293.890625 96.265625 C 288.910156 101.246094 280.839844 101.246094 275.859375 96.265625 C 270.878906 91.285156 270.878906 83.214844 275.859375 78.234375 C 280.839844 73.253906 288.910156 73.253906 293.890625 78.234375 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.900818%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"257.425781\" xlink:href=\"#glyph0-2\" y=\"77.635498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"262.431152\" xlink:href=\"#glyph1-5\" y=\"79.635498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"6.4873\" xlink:href=\"#glyph2-15\" y=\"16.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"13.4875\" xlink:href=\"#glyph2-4\" y=\"16.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"18.4933\" xlink:href=\"#glyph2-5\" y=\"16.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"20.9935\" xlink:href=\"#glyph2-3\" y=\"16.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"25.9993\" xlink:href=\"#glyph2-4\" y=\"16.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"31.0051\" xlink:href=\"#glyph2-5\" y=\"16.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"33.5053\" xlink:href=\"#glyph2-6\" y=\"16.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"36.0055\" xlink:href=\"#glyph2-7\" y=\"16.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"38.0053\" xlink:href=\"#glyph2-8\" y=\"16.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"43.0111\" xlink:href=\"#glyph2-9\" y=\"16.352783\"/>\n",
       "  <use x=\"47.5111\" xlink:href=\"#glyph2-10\" y=\"16.352783\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"52.5169\" xlink:href=\"#glyph2-11\" y=\"16.352783\"/>\n",
       "</g>\n",
       "<path d=\"M 109.164062 76.457031 L 130.816406 42.039062 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 132.949219 38.652344 L 130.816406 42.039062 M 129.546875 41.238281 L 132.949219 38.652344 L 132.085938 42.835938 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 112.976562 80.164062 L 178.117188 36.617188 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 181.445312 34.394531 L 178.117188 36.617188 M 177.285156 35.367188 L 181.445312 34.394531 L 178.953125 37.863281 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 114.097656 82.226562 L 229.355469 32.847656 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 233.03125 31.269531 L 229.355469 32.847656 M 228.761719 31.46875 L 233.03125 31.269531 L 229.945312 34.226562 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 146.496094 74.589844 L 142.953125 44.769531 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 142.480469 40.796875 L 142.953125 44.769531 M 141.460938 44.945312 L 142.480469 40.796875 L 144.441406 44.59375 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 155.636719 77.039062 L 182.453125 41.1875 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 184.847656 37.984375 L 182.453125 41.1875 M 181.253906 40.289062 L 184.847656 37.984375 L 183.65625 42.085938 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 158.804688 80.476562 L 230.695312 35.40625 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 234.085938 33.28125 L 230.695312 35.40625 M 229.898438 34.136719 L 234.085938 33.28125 L 231.492188 36.679688 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 185.273438 77.613281 L 152.964844 40.34375 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 150.347656 37.320312 L 152.964844 40.34375 M 151.832031 41.324219 L 150.347656 37.320312 L 154.097656 39.359375 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 193.625 74.5 L 193.625 44.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 193.625 40.898438 L 193.625 44.898438 M 192.125 44.898438 L 193.625 40.898438 L 195.125 44.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 201.917969 77.566406 L 234.371094 39.667969 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 236.972656 36.628906 L 234.371094 39.667969 M 233.230469 38.691406 L 236.972656 36.628906 L 235.507812 40.640625 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 228.40625 80.535156 L 156.609375 36.070312 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 153.207031 33.964844 L 156.609375 36.070312 M 155.820312 37.347656 L 153.207031 33.964844 L 157.398438 34.796875 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 231.613281 77.039062 L 204.796875 41.1875 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 202.402344 37.984375 L 204.796875 41.1875 M 203.59375 42.085938 L 202.402344 37.984375 L 205.996094 40.289062 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 240.738281 74.585938 L 244.324219 44.023438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 244.792969 40.050781 L 244.324219 44.023438 M 242.835938 43.847656 L 244.792969 40.050781 L 245.816406 44.199219 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 273.128906 82.277344 L 157.929688 33.519531 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 154.246094 31.960938 L 157.929688 33.519531 M 157.34375 34.902344 L 154.246094 31.960938 L 158.511719 32.140625 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 274.273438 80.164062 L 209.132812 36.617188 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 205.804688 34.394531 L 209.132812 36.617188 M 208.296875 37.863281 L 205.804688 34.394531 L 209.964844 35.367188 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 278.144531 76.417969 L 256.34375 41.34375 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 254.234375 37.945312 L 256.34375 41.34375 M 255.070312 42.132812 L 254.234375 37.945312 L 257.621094 40.550781 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 112.414062 135.761719 L 106.15625 105.515625 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 105.34375 101.597656 L 106.15625 105.515625 M 104.6875 105.820312 L 105.34375 101.597656 L 107.625 105.210938 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 121.066406 137.03125 L 139.125 103.65625 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 141.027344 100.136719 L 139.125 103.65625 M 137.804688 102.941406 L 141.027344 100.136719 L 140.445312 104.371094 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 125.074219 140.433594 L 178.890625 98.683594 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 182.050781 96.230469 L 178.890625 98.683594 M 177.96875 97.496094 L 182.050781 96.230469 L 179.808594 99.867188 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 126.449219 142.628906 L 222.503906 95.46875 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 226.097656 93.707031 L 222.503906 95.46875 M 221.84375 94.125 L 226.097656 93.707031 L 223.167969 96.816406 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 127.003906 143.941406 L 267.320312 93.554688 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 271.085938 92.203125 L 267.320312 93.554688 M 266.8125 92.140625 L 271.085938 92.203125 L 267.828125 94.964844 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 158.542969 139.558594 L 116.023438 99.960938 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 113.097656 97.234375 L 116.023438 99.960938 M 115 101.058594 L 113.097656 97.234375 L 117.046875 98.863281 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 163.925781 136.125 L 153.777344 104.984375 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 152.539062 101.183594 L 153.777344 104.984375 M 152.351562 105.449219 L 152.539062 101.183594 L 155.203125 104.519531 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 172.835938 136.5 L 186.371094 104.433594 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 187.925781 100.75 L 186.371094 104.433594 M 184.988281 103.851562 L 187.925781 100.75 L 187.753906 105.019531 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 177.566406 139.964844 L 225.070312 99.367188 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 228.113281 96.769531 L 225.070312 99.367188 M 224.097656 98.226562 L 228.113281 96.769531 L 226.046875 100.507812 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 179.183594 142.355469 L 268.335938 95.875 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 271.882812 94.023438 L 268.335938 95.875 M 267.640625 94.542969 L 271.882812 94.023438 L 269.027344 97.203125 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 209.414062 142.40625 L 118.957031 95.792969 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 115.402344 93.960938 L 118.957031 95.792969 M 118.269531 97.128906 L 115.402344 93.960938 L 119.644531 94.460938 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 210.980469 140.058594 L 162.292969 99.234375 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 159.226562 96.664062 L 162.292969 99.234375 M 161.328125 100.382812 L 159.226562 96.664062 L 163.253906 98.082031 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 215.566406 136.597656 L 201.203125 104.292969 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 199.578125 100.640625 L 201.203125 104.292969 M 199.832031 104.902344 L 199.578125 100.640625 L 202.574219 103.683594 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 224.449219 136.046875 L 233.835938 105.101562 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 234.996094 101.273438 L 233.835938 105.101562 M 232.402344 104.664062 L 234.996094 101.273438 L 235.273438 105.535156 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 229.988281 139.460938 L 271.363281 100.105469 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 274.261719 97.347656 L 271.363281 100.105469 M 270.328125 99.015625 L 274.261719 97.347656 L 272.394531 101.191406 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 261.59375 143.265625 L 119.96875 93.441406 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 116.199219 92.113281 L 119.96875 93.441406 M 119.472656 94.855469 L 116.199219 92.113281 L 120.46875 92.027344 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 262.125 141.984375 L 164.820312 95.316406 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 161.210938 93.585938 L 164.820312 95.316406 M 164.171875 96.667969 L 161.210938 93.585938 L 165.46875 93.964844 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 263.4375 139.828125 L 208.523438 98.472656 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 205.328125 96.0625 L 208.523438 98.472656 M 207.621094 99.667969 L 205.328125 96.0625 L 209.425781 97.273438 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 267.304688 136.421875 L 248.492188 103.453125 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 246.511719 99.976562 L 248.492188 103.453125 M 247.191406 104.195312 L 246.511719 99.976562 L 249.796875 102.707031 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 275.964844 134.964844 L 281.453125 105.585938 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "<path d=\"M 282.1875 101.652344 L 281.453125 105.585938 M 279.976562 105.308594 L 282.1875 101.652344 L 282.925781 105.859375 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-23,-12)\"/>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SVG(filename='../img/mlp.svg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The multilayer perceptron above has 4 inputs and 3 outputs,\n",
    "and the hidden layer in the middle contains 5 hidden units.\n",
    "Since the input layer does not involve any calculations,\n",
    "building this network would consist of\n",
    "implementing 2 layers of computation.\n",
    "The neurons in the input layer are fully connected\n",
    "to the inputs in the hidden layer.\n",
    "Likewise, the neurons in the hidden layer\n",
    "are fully connected to the neurons in the output layer.\n",
    "\n",
    "\n",
    "### From linear to nonlinear\n",
    "\n",
    "We can write out the calculations that define this one-hidden-layer MLP in mathematical notation as follows:\n",
    "$$\n",
    "\\begin{aligned}\n",
    "    \\mathbf{h} & = \\mathbf{W}_1 \\mathbf{x} + \\mathbf{b}_1 \\\\\n",
    "    \\mathbf{o} & = \\mathbf{W}_2 \\mathbf{h} + \\mathbf{b}_2 \\\\\n",
    "    \\hat{\\mathbf{y}} & = \\mathrm{softmax}(\\mathbf{o})\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "By adding another layer, we have added two new sets of parameters,\n",
    "but what have we gained in exchange?\n",
    "In the model defined above, we do not achieve anything for our troubles!\n",
    "\n",
    "That's because our hidden units are just a linear function of the inputs\n",
    "and the outputs (pre-softmax) are just a linear function of the hidden units.\n",
    "A linear function of a linear function is itself a linear function.\n",
    "That means that for any values of the weights,\n",
    "we could just collapse out the hidden layer\n",
    "yielding an equivalent single-layer model using\n",
    "$\\mathbf{W} = \\mathbf{W}_2 \\mathbf{W}_1$ and $\\mathbf{b} = \\mathbf{W}_2 \\mathbf{b}_1 + \\mathbf{b}_2$.\n",
    "\n",
    "$$\\mathbf{o} = \\mathbf{W}_2 \\mathbf{h} + \\mathbf{b}_2 = \\mathbf{W}_2 (\\mathbf{W}_1 \\mathbf{x} + \\mathbf{b}_1) + \\mathbf{b}_2 = (\\mathbf{W}_2 \\mathbf{W}_1) \\mathbf{x} + (\\mathbf{W}_2 \\mathbf{b}_1 + \\mathbf{b}_2) = \\mathbf{W} \\mathbf{x} + \\mathbf{b}$$\n",
    "\n",
    "In order to get a benefit from multilayer architectures,\n",
    "we need another key ingredient—a nonlinearity $\\sigma$ to be applied to each of the hidden units after each layer's linear transformation.\n",
    "The most popular choice for the nonlinearity these days is the rectified linear unit (ReLU) $\\mathrm{max}(x,0)$.\n",
    "After incorporating these non-linearities\n",
    "it becomes impossible to merge layers.\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "    \\mathbf{h} & = \\sigma(\\mathbf{W}_1 \\mathbf{x} + \\mathbf{b}_1) \\\\\n",
    "    \\mathbf{o} & = \\mathbf{W}_2 \\mathbf{h} + \\mathbf{b}_2 \\\\\n",
    "    \\hat{\\mathbf{y}} & = \\mathrm{softmax}(\\mathbf{o})\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "Clearly, we could continue stacking such hidden layers,\n",
    "e.g. $\\mathbf{h}_1 = \\sigma(\\mathbf{W}_1 \\mathbf{x} + \\mathbf{b}_1)$\n",
    "and $\\mathbf{h}_2 = \\sigma(\\mathbf{W}_2 \\mathbf{h}_1 + \\mathbf{b}_2)$\n",
    "on top of each other to obtain a true multilayer perceptron.\n",
    "\n",
    "Multilayer perceptrons can account for complex interactions in the inputs\n",
    "because the hidden neurons depend on the values of each of the inputs.\n",
    "It’s easy to design a hidden node that does arbitrary computation,\n",
    "such as, for instance, logical operations on its inputs.\n",
    "Moreover, for certain choices of the activation function\n",
    "it’s widely known that multilayer perceptrons are universal approximators.\n",
    "That means that even for a single-hidden-layer neural network,\n",
    "with enough nodes, and the right set of weights,\n",
    "we can model any function at all!\n",
    "*Actually learning that function is the hard part.*\n",
    "\n",
    "Moreover, just because a single-layer network *can* learn any function\n",
    "doesn't mean that you should try to solve all of your problems with single-layer networks.\n",
    "It turns out that we can approximate many functions\n",
    "much more compactly if we use deeper (vs wider) neural networks.\n",
    "We’ll get more into the math in a subsequent chapter,\n",
    "but for now let’s actually build an MLP.\n",
    "In this example, we’ll implement a multilayer perceptron\n",
    "with two hidden layers and one output layer.\n",
    "\n",
    "### Vectorization and mini-batch\n",
    "\n",
    "As before, by the matrix $\\mathbf{X}$, we denote a mini-batch of inputs.\n",
    "The calculations to produce outputs from an MLP with two hidden layers\n",
    "can thus be expressed:\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "    \\mathbf{H}_1 & = \\sigma(\\mathbf{W}_1 \\mathbf{X} + \\mathbf{b}_1) \\\\\n",
    "    \\mathbf{H}_2 & = \\sigma(\\mathbf{W}_2 \\mathbf{H}_1 + \\mathbf{b}_2) \\\\\n",
    "    \\mathbf{O} & = \\mathrm{softmax}(\\mathbf{W}_3 \\mathbf{H}_2 + \\mathbf{b}_3)\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "With some abuse of notation, we define the nonlinearity $\\sigma$\n",
    "to apply to its inputs on a row-wise fashion, i.e. one observation at a time.\n",
    "Note that we are also using the notation for *softmax* in the same way to denote a row-wise operation.\n",
    "Often, as in this chapter, the activation functions that we apply to hidden layers are not merely row-wise, but component wise.\n",
    "That means that after computing the linear portion of the layer,\n",
    "we can calculate each nodes activation without looking at the values taken by the other hidden units.\n",
    "This is true for most activation functions\n",
    "(the batch normalization operation will be introduced in :numref:`chapter_batch_norm` is a notable exception to that rule).\n",
    "\n",
    "## Activation Functions\n",
    "\n",
    "Because they are so fundamental to deep learning, before going further,\n",
    "let's take a brief look at some common activation functions.\n",
    "\n",
    "### ReLU Function\n",
    "\n",
    "As stated above, the most popular choice,\n",
    "due to its simplicity of implementation\n",
    "and its efficacy in training is the rectified linear unit (ReLU).\n",
    "ReLUs provide a very simple nonlinear transformation.\n",
    "Given the element $z$, the function is defined\n",
    "as the maximum of that element and 0.\n",
    "\n",
    "$$\\mathrm{ReLU}(z) = \\max(z, 0).$$\n",
    "\n",
    "It can be understood that the ReLU function retains only positive elements and discards negative elements (setting those nodes to 0).\n",
    "To get a better idea of what it looks like, we can plot it.\n",
    "For convenience, we define a plotting function `xyplot`\n",
    "to take care of the groundwork."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0, '..')\n",
    "\n",
    "import numpy\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.autograd import Variable\n",
    "def xyplot(x_vals,y_vals,name):\n",
    "    x_vals=x_vals.detach().numpy() # we can't directly use var.numpy() because varibles might \n",
    "    y_vals=y_vals.detach().numpy() # already required grad.,thus using var.detach().numpy() \n",
    "    plt.plot(x_vals,y_vals) \n",
    "    plt.xlabel('x')\n",
    "    plt.ylabel(name+'(x)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since relu is commomly used as activation function, PyTorch supports\n",
    "the `relu` function as a basic native operator.\n",
    "As you can see, the activation function is piece-wise linear.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEGCAYAAABvtY4XAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAd90lEQVR4nO3dd5xU9dn+8c8tRaQJyAJKkSIIipRlRbBFbLFFkxgLiokxEUNR9DGxRZ/8NCbxsRNbHhI1hQVERE1ssdcouo3eexMWkF522b1/f8ysz4qU3dk9c2bOXO/Xixc7zOx8L2G99+x3zpzL3B0REYmeg8IOICIiwdCAFxGJKA14EZGI0oAXEYkoDXgRkYiqG3aAylq2bOkdO3YMO4aISNrIz89f5+5Ze7svpQZ8x44dycvLCzuGiEjaMLOl+7pPWzQiIhGlAS8iElEa8CIiERXogDezm8xsppnNMLPxZtYgyPVEROT/BDbgzawtcAOQ4+49gTrA5UGtJyIi3xT0Fk1d4BAzqws0BFYFvJ6IiMQFNuDdfSXwILAMWA1scvc393ycmQ01szwzyysuLg4qjohIxglyi6Y5cBHQCTgCaGRmQ/Z8nLuPcfccd8/JytrrufoiIpE1ZdF6nv54MUFcuj3ILZozgcXuXuzupcBk4MQA1xMRSStrt+xk5PhCxn62lB2lZbX+/EEO+GXAADNraGYGnAHMDnA9EZG0sbusnFHji9iys5SnhmTTsH7tX1ggyD34KcAkoACYHl9rTFDriYikk0fenseni9bz24t60r1N00DWCPRaNO7+G+A3Qa4hIpJu3p2zhifeW8hlOe25JKd9YOvonawiIkm04qvt3PTcVHoc3pS7Lzo20LU04EVEkmTX7jJG5BZQXu48dWU2DerVCXS9lLpcsIhIlP3u1dlMXbGJPw3pR8eWjQJfT0fwIiJJ8HLRSv7+6VKuPaUT5/Rsk5Q1NeBFRAK2YO0Wbp88nZwjm3PLOd2Ttq4GvIhIgLaX7GbY2AIOqVeHx6/Ipl6d5I1d7cGLiATE3blj8nQWFG/lH9ecQJtDk3vFdB3Bi4gEZNzny3ipaBU3ndmNk7u2TPr6GvAiIgGYvmITd/9zFqd2y2LkoKNCyaABLyJSyzZtL2VYbj4tG9fn0cv6cNBBFkoO7cGLiNSi8nLn5ueLWLN5J89dN5AWjeqHlkVH8CIitWjMR4t4e/Za7jivB9kdmoeaRQNeRKSWfLZoPQ/8ey7nH3c4V5/YMew4GvAiIrVh7ZadXD++kCNbNOS+i48jVoMRLu3Bi4jU0O6ycm4YX8iWnaX842f9adKgXtiRAA14EZEae/iteXy2aAMPXdI7sPKORARZun20mRVV+rXZzG4Maj0RkTC8M3sNT76/kMH923Nxv3Zhx/mGwI7g3X0u0AfAzOoAK4EXg1pPRCTZlm/Yzn9NnMqxRzTlN98LtrwjEcl6kfUMYKG7L03SeiIigdq1u4wR4wood+fJJJR3JCJZA/5yYPze7jCzoWaWZ2Z5xcXFSYojIlIz974ym2krNvHgJb058rDgyzsSEfiAN7P6wIXA83u7393HuHuOu+dkZWUFHUdEpMZeLlrJPz5bytBTO/PdY5NT3pGIZBzBnwsUuPuaJKwlIhKo+Wti5R3Hd2zOr757dNhx9isZA34w+9ieERFJJ9t27WZYbgEN6ye/vCMRgaYzs4bAWcDkINcREQmau3PHi9NZVLyV0Zf3pXXT5JZ3JCLQNzq5+3bgsCDXEBFJhtwpy3i5aBU3n9WNk45KfnlHIlL75wsRkRQwbcVG7vnXLE47OosRIZV3JEIDXkRkPzZuL2HY2AJaNq7PI5eGV96RCF2LRkRkH8rLnZsnTmXtlp1MvG4gzUMs70iEjuBFRPbhTx8u5J05a7nz/GPoG3J5RyI04EVE9uLThet58N9zuaDX4fx44JFhx0mIBryIyB7Wbo6Vd3Rs2Yj7Lu6VEuUdidAevIhIJbvLyrl+fCFbd5WS+/MTaHxw+o7J9E0uIhKAh96ax5TFG3j40t4c3aZJ2HFqRFs0IiJxb89aw1PvL2Rw/w78MDu1yjsSoQEvIkJFeUdRvLzjmLDj1AoNeBHJeDtLyxiWm48DT13ZLyXLOxKhPXgRyXi/fWUWM1Zu5s8/zqHDYQ3DjlNrdAQvIhntpcKV5E5ZxnWnduasY1qHHadWacCLSMaqKO/o37EFv0zx8o5EaMCLSEbatms3vxibT6OD6/DYFX1TvrwjEdqDF5GM4+7cNnk6i9dtY+zPT0iL8o5EBN3o1MzMJpnZHDObbWYDg1xPRKQqxn62lH9NXcXNZx/NiV3So7wjEUEfwY8G3nD3H5lZfSA6L0+LSFqaunwj97wyi0FHZzHsO13CjhOowAa8mTUFTgWuBnD3EqAkqPVERA5k4/YShucW0KpJAx65LL3KOxIR5BZNZ6AYeNbMCs3sL2bWaM8HmdlQM8szs7zi4uIA44hIJisvd/4rXt7xxJXZNGuYXuUdiQhywNcFsoGn3L0vsA24bc8HufsYd89x95ysrKwA44hIJnvqg4W8O2ctd11wDH3aNws7TlIEOeBXACvcfUr89iRiA19EJKn+s3AdD705l+/1PoKrBqRneUciAhvw7v4lsNzMKt49cAYwK6j1RET2Zs3mndwwvpBOLRvxhx8el7blHYkI+iya64Hc+Bk0i4CfBryeiMjXKso7tu0qY9y1A9K6vCMRgf7XunsRkBPkGiIi+/Lgm/P4fPEGHr2sD91ap3d5RyKi995cERHgrVlr+NMHC7nyhA58v2/bsOOEQgNeRCJn2frt3DyxiJ5tm3LXBdEo70iEBryIRMrO0jKGj8sHolXekYjMesVBRCLvnkrlHe1bZPbVUXQELyKR8WLhCsZNWcYvvtMlcuUdidCAF5FImLdmC3dMnkH/Ti345dndwo6TEjTgRSTtbf26vKMujw/uS90IlnckQn8LIpLW3J3bJ09nybptPDa4L60iWt6RCA14EUlr/6hU3jGwy2Fhx0kpGvAikraKlm/kt6/M4vTurSJf3pEIDXgRSUtfbSthRLy84+FLe0e+vCMROg9eRNJOeblz08QiirfsYtKwgRlR3pEIHcGLSNp56oOFvD+3mLu+dwy92mVGeUciNOBFJK38Z0GsvOOiPkcw5IQOYcdJaRrwIpI21mzeyQ0TCumc1Zjf/yCzyjsSoT14EUkLpWXljBxXwLZdZYy/NptGGVbekYhA/4bMbAmwBSgDdru7yj9EJCEP/nsuXyz5itGX96FrBpZ3JCIZ3wIHufu6JKwjIhH15swv+d8PFzFkQAcu6pOZ5R2J0B68iKS0Zeu3c/PzU+nV7tCMLu9IRNAD3oE3zSzfzIbu7QFmNtTM8swsr7i4OOA4IpJOdpaWMSw3n4PMeOKKbA6um7nlHYkIesCf5O7ZwLnACDM7dc8HuPsYd89x95ysrKyA44hIOrn7X7OYuWozD1/aO+PLOxIR6IB391Xx39cCLwL9g1xPRKLjhfwVjP98GcNO68IZPVTekYjABryZNTKzJhUfA2cDM4JaT0SiY86Xm/n1S9M5oVMLbj5L5R2JCvIsmtbAi/E3ItQFxrn7GwGuJyIRsHXXbobnFtCkQT0eu0LlHTUR2IB390VA76CeX0Six9259YVpLFm3jXHXDqBVE5V31IS+NYpIyvj7p0t5ddpqfvXd7gzorPKOmtKAF5GUULjsK+59dRZn9mjFdad2DjtOJGjAi0joKso7WjdtwEOX9FF5Ry3R1XpEJFTl5c6NzxWxbmsJk4YN5NCG9cKOFBk6gheRUD3x3gI+mFfMf6u8o9ZpwItIaD5ZsI6H357H9/scwZUq76h1Vd6iMbODiJ32eASwA5jp7muCCiYi0fblpp2MmlDIUVmN+Z3KOwJxwAFvZl2AW4EzgflAMdAA6GZm24H/Bf7m7uVBBhWR6CgtK+f68QVsLyljwlCVdwSlKn+r9wJPAde5u1e+w8xaAVcAVwF/q/14IhJFD8TLO/44uC9HtVJ5R1AOOODdffB+7lsLPFqriUQk0t6Y8SVjPlzEVQOO5MLeR4QdJ9Kq/CKrmf3WzOpWut3UzJ4NJpaIRNHS9dv4Vby8484LeoQdJ/KqcxZNXWCKmfUys7OBL4D8YGKJSNTsLC1j2NgCDjpI5R3JUuVXNtz9djN7B5gCfAWc6u4LAksmIpFy979mMmv1Zp65OkflHUlSnS2aU4HRwD3A+8DjZqYNNBE5oEn5Kxj/+XJGDOrC6d1V3pEs1Tk36UHgEnefBWBmPwTeBboHEUxEomHOl5u586XpDOx8GDedqfKOZKrOgB/o7mUVN9x9spl9EEAmEYmILTtLGTY2Vt4xenAflXck2QH/ts1siJkdVHm4V3D39WbWxcxO3s/n1zGzQjN7paZhRSR9uDu3vTCdZRu28/jgvirvCEFVjuAPAwrNLJ/YWTMV72Q9CvgOsA64bT+fPwqYDTStWVQRSSd//c8SXp2+mtvO7c4JKu8IxQGP4N19NJANjAeygDPit1cCV7n7xe4+f2+fa2btgPOBv9RaYhFJeQXLvuL3r83mzB6tGXqKyjvCUqU9+Pj2zFvxX9XxKHALsM/3IpvZUGAoQIcOupqcSLrbsK2EkbkFtDm0AQ9d0lvlHSGqztUknwV8zz9392v28fgLgLXunm9mp+3red19DDAGICcn51vPLyLp4+vyjm0lTB52oso7Qlads2gqv0jaAPgBsGo/jz8JuNDMzos/vqmZjXX3IdWPKSLp4PH3FvDhvGJ+/4Pj6Nn20LDjZLzqvJP1hcq3zWw88PZ+Hn87cHv8sacBv9RwF4muj+ev45G35/GDvm0Z3L992HGEmjU6dQW0aS4irN60gxsmFNK1VWN+94OeKu9IEdXZg99CbA/e4r9/SawI5IDc/X1ilzcQkYgpLStn5LhCdpaW8eSV/WhYX+UdqaI6WzS6Kr+IfMv9b8whf2lFeUfjsONIJVWp7Mve3/3uXlB7cUQknbwxYzV//mgxPxmo8o5UVJUj+If2c58Dp9dSFhFJI0vWbeNXz0+jd/tm3HG+yjtSUVUq+wYlI4iIpI+dpWUMy60o7+ir8o4UVZ3rwTc0szvNbEz8dtf4m5lEJMP85uWZzF69mUcv60O75irvSFXVOU3yWaAEODF+ewVwb60nEpGU9nzecp7LW87IQUcxqHursOPIflRnwHdx9/uBUgB330HslEkRyRCzV2/mrpdncGKXw7jpLJV3pLrqDPgSMzuE+PVozKwLsCuQVCKScrbsLGV4bgFNG9Rj9OV9qaOLiKW8Kp0Hb7G3pf0JeANob2a5xK41c3Vw0UQkVbg7t74wjWUbtjP+2gFkNTk47EhSBVW9XLCb2SjgbGAAsa2ZUe6+LshwIpIanv1kCa9N/5I7zutO/04two4jVVSd9xR/BnR291eDCiMiqSd/aay846xjWnOtyjvSSnUG/CDgOjNbCmwjfk0ad+8VSDIRCd2GbSWMHFfA4c0a8OAlvXURsTRTnQF/bmApRCTllJU7oyYUsr6ivOMQlXekm+pcbGxpkEFEJLU8/u4CPpq/jj/8UOUd6aom14MXkYj6aH4xj74zjx9mt+Xy41Xeka404EXkG1Zv2sGoCUV0a9WEe7+v8o50FtiAN7MGZva5mU01s5lmdndQa4lI7SgtK2dEbgG7Sst4cki2yjvSXJD/eruA0919q5nVAz42s9fd/bMA1xSRGrjv9TkULNvIY4P70iVL5R3pLrAB7+4ObI3frBf/5UGtJyI18/r01Tz98WKuPrEj31N5RyQEugdvZnXMrAhYC7zl7lP28pihZpZnZnnFxcVBxhGRfVi8bhu3TJpGn/bNuOM8lXdERaAD3t3L3L0P0A7ob2Y99/KYMe6e4+45WVlZQcYRkb3YWVrGsLH51KljPHFlNvXr6tyLqEjKv6S7bwTeB85JxnoiUnX//fIM5q7ZwqOX9aFts0PCjiO1KMizaLLMrFn840OAM4E5Qa0nItU38YvlTMxbwfWDjuK0o1XeETVBnkVzOPA3M6tD7BvJRHd/JcD1RKQaZq2KlXecdNRhjDpT5R1RFORZNNOAvkE9v4gkbvPOUobn5tOsoco7okzvYhDJMO7OrZOmsfyrHUwYOoCWjVXeEVV6uVwkwzzzyRJen/Elt53TneM7qrwjyjTgRTJI/tIN/OG12Zx9TGt+fkqnsONIwDTgRTLE+q27GJFbSNvmh/CAyjsygvbgRTJAWblz43NFbNhewovDVd6RKXQEL5IB/vjOfD6av457LjyWY49QeUem0IAXibgP5xXzx3fnc3F2Oy5TeUdG0YAXibBVG3cwakIhR7dWeUcm0oAXiaiS3eWMHFdAaZnz5JXZHFK/TtiRJMn0IqtIRFWUdzxxRTadVd6RkXQELxJBr01fzTOfLOanJ3Xk/F6Hhx1HQqIBLxIxi4q3csukafTt0Izbz1V5RybTgBeJkB0lZQzPLaBeHePxK1Tekem0By8SIRXlHc9efbzKO0RH8CJRMfGL5Tyfv4LrT++q8g4BNOBFImHmqk3c9fIMTj6qJaPO6Bp2HEkRQVb2tTez98xstpnNNLNRQa0lksli5R0FNG9Yn9GX91F5h3wtyD343cDN7l5gZk2AfDN7y91nBbimSEZxd371/FRWxss7DlN5h1QS2BG8u69294L4x1uA2UDboNYTyURPf7yYf89cw23ndidH5R2yh6TswZtZR2L9rFP2ct9QM8szs7zi4uJkxBGJhLwlG7jv9Tl899jW/OxklXfItwU+4M2sMfACcKO7b97zfncf4+457p6TlZUVdByRSFi3dRcjx6m8Q/Yv0PPgzaweseGe6+6Tg1xLJFOUlTs3Tijiq+0lTB5+Ik0bqLxD9i6wAW+xQ4qngdnu/nBQ64hkmtHvzOfjBeu4/+JeKu+Q/Qpyi+Yk4CrgdDMriv86L8D1RCLv/blreezd+VzSrx2XqrxDDiCwI3h3/xjQxqBILVm1cQc3PVfE0a2bcM9FPcOOI2lA72QVSQMlu8sZofIOqSZdbEwkDfzh9dkULtvIk1eqvEOqTkfwIinu1WmrefaTJVxzUifOO07lHVJ1GvAiKWxh8VZumTSV7A7NuO3c7mHHkTSjAS+SonaUlDF8bAEH16uj8g5JiPbgRVKQu3PnSzOYt3YLf/tpf45QeYckQIcEIiloYt5yXihYwQ2nd+XUbrqEhyRGA14kxcTKO2ZySteW3KDyDqkBDXiRFLJpR6y8o0XD+jx6mco7pGa0By+SIiqXdzx3nco7pOZ0BC+SIv7y0WLenLWG28/rQb8jVd4hNacBL5ICvliygfvemMO5PdtwzUkdw44jEaEBLxKyWHlHAR1aNOT+H/VSeYfUGg14kRCVlTujJhSycXspT16ZTROVd0gt0ousIiEa/fY8Plmwnvt/1IsehzcNO45EjI7gRULy3ty1/PHdBVya045Lc1TeIbUvsAFvZs+Y2VozmxHUGiLpamW8vKN7G5V3SHCCPIL/K3BOgM8vkpZKdpczPLeAsjLnqSH9aFBP5R0SjMAGvLt/CGwI6vlF0tXvX5vN1OUbeeCSXnRq2SjsOBJhoe/Bm9lQM8szs7zi4uKw44gE6pVpq/jrf5bws5M7cU5PlXdIsEIf8O4+xt1z3D0nK0tXzZPoWli8lVsnTaPfkc1V3iFJEfqAF8kE20t2M2xsfry8oy/16uh/PQmezoMXCZi7c+eLM5i/dit/v6Y/hx+q8g5JjiBPkxwPfAocbWYrzOxnQa0lksomfLGcyYUrufGMbpzSVduQkjyBHcG7++CgnlskXcxYuYnf/HMmp3bL4vrTjwo7jmQYbQSKBKSivOOwRrHyjoNU3iFJpj14kQC4O798fiqrNu7guesG0qJR/bAjSQbSEbxIAP780SLemrWGO87rQb8jm4cdRzKUBrxILft88Qb+5425nHdcG36q8g4JkQa8SC0q3vJ/5R3/c7HKOyRcGvAitaSs3LlhfCGbd5by1BCVd0j49CKrSC155K15fLpoPQ9e0pvubVTeIeHTEbxILXhv7loef28Bl+W050f92oUdRwTQgBepsRVfbeem54rocXhT7r7o2LDjiHxNA16kBnbtLmPEuMJYeceV2SrvkJSiPXiRGvj9q7Hyjj8N6UdHlXdIitERvEiC/jl1FX/7dCnXntKJc3q2CTuOyLdowIskYMHardz2wjRyjmzOLeeovENSkwa8SDVtL9nN8Nx8DqlXh8evyFZ5h6Qs7cGLVIO78+t4ecc/rjmBNoc2CDuSyD7p0EOkGsZ/vpwXC1dy05ndOLlry7DjiOxXoAPezM4xs7lmtsDMbgtyLZGgTV+xif8XL+8YOUjlHZL6gqzsqwM8AZwLHAMMNrNjglpPJEibtpcyfFw+LRurvEPSR5B78P2BBe6+CMDMJgAXAbNqe6HvPfYxO0vLavtpRb62aUcpX20vUXmHpJUgB3xbYHml2yuAE/Z8kJkNBYYCdOjQIaGFumQ1oqSsPKHPFamqC3sfQXYHlXdI+ghywO/tZ1j/1h+4jwHGAOTk5Hzr/qp49PK+iXyaiEikBfki6wqgfaXb7YBVAa4nIiKVBDngvwC6mlknM6sPXA78M8D1RESkksC2aNx9t5mNBP4N1AGecfeZQa0nIiLfFOg7Wd39NeC1INcQEZG90ztZRUQiSgNeRCSiNOBFRCJKA15EJKLMPaH3FgXCzIqBpQl+ektgXS3GqS3KVX2pmk25qi9Vs0Up15HunrW3O1JqwNeEmeW5e07YOfakXNWXqtmUq/pSNVum5NIWjYhIRGnAi4hEVJQG/JiwA+yDclVfqmZTrupL1WwZkSsye/AiIvJNUTqCFxGRSjTgRUQiKlID3sz6mNlnZlZkZnlm1j/sTBXM7Pp4AflMM7s/7DyVmdkvzczNrGXYWQDM7AEzm2Nm08zsRTNrFnKelCyPN7P2Zvaemc2Of12NCjtTZWZWx8wKzeyVsLNUMLNmZjYp/vU128wGhp2pgpndFP93nGFm482sQU2fM1IDHrgfuNvd+wD/Hb8dOjMbRKyPtpe7Hws8GHKkr5lZe+AsYFnYWSp5C+jp7r2AecDtYQVJ8fL43cDN7t4DGACMSKFsAKOA2WGH2MNo4A137w70JkXymVlb4AYgx917ErvE+uU1fd6oDXgHmsY/PpTUaZAaBtzn7rsA3H1tyHkqewS4hb3UKYbF3d90993xm58RawMLy9fl8e5eAlSUx4fO3Ve7e0H84y3EhlXbcFPFmFk74HzgL2FnqWBmTYFTgacB3L3E3TeGm+ob6gKHmFldoCG1ML+iNuBvBB4ws+XEjpJDO/LbQzfgFDObYmYfmNnxYQcCMLMLgZXuPjXsLPtxDfB6iOvvrTw+JYZoZWbWEegLTAk3ydceJXbgUB52kEo6A8XAs/Gto7+YWaOwQwG4+0piM2sZsBrY5O5v1vR5Ay38CIKZvQ202ctdvwbOAG5y9xfM7FJi36nPTIFcdYHmxH6MPh6YaGadPQnnqB4g1x3A2UFn2Jv95XL3l+OP+TWxbYjcZGbbQ5XK48NkZo2BF4Ab3X1zCuS5AFjr7vlmdlrYeSqpC2QD17v7FDMbDdwG3BVuLDCz5sR+MuwEbASeN7Mh7j62Js+bdgPe3fc5sM3s78T2/QCeJ4k/Hh4g1zBgcnygf25m5cQuKlQcVi4zO47YF9NUM4PYNkiBmfV39y/DylUp30+AC4AzkvGNcD9SujzezOoRG+657j457DxxJwEXmtl5QAOgqZmNdfchIedaAaxw94qfciYRG/Cp4ExgsbsXA5jZZOBEoEYDPmpbNKuA78Q/Ph2YH2KWyl4ilgcz6wbUJ+Qr2bn7dHdv5e4d3b0jsS/+7GQM9wMxs3OAW4EL3X17yHFStjzeYt+ZnwZmu/vDYeep4O63u3u7+NfV5cC7KTDciX9tLzezo+N/dAYwK8RIlS0DBphZw/i/6xnUwgvAaXcEfwDXAqPjL1LsBIaGnKfCM8AzZjYDKAF+EvJRaap7HDgYeCv+08Vn7v6LMIKkeHn8ScBVwHQzK4r/2R3xLmTZu+uB3Pg360XAT0POA0B8y2gSUEBsW7KQWrhsgS5VICISUVHbohERkTgNeBGRiNKAFxGJKA14EZGI0oAXEYkoDXgRkYjSgBcRiSgNeJF9MLPj49ekb2BmjeLX6u4Zdi6RqtIbnUT2w8zuJXY9lUOIXcfkDyFHEqkyDXiR/Yi/pf0LYpe+ONHdy0KOJFJl2qIR2b8WQGOgCbEjeZG0oSN4kf0ws38Sa3HqBBzu7iNDjiRSZVG7mqRIrTGzHwO73X1cvJv1P2Z2uru/G3Y2karQEbyISERpD15EJKI04EVEIkoDXkQkojTgRUQiSgNeRCSiNOBFRCJKA15EJKL+P7+oay1v1UzwAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x=Variable(torch.arange(-8.0,8.0,0.1,dtype=torch.float32).reshape(int(16/0.1),1),requires_grad=True)\n",
    "y=torch.nn.functional.relu(x)\n",
    "xyplot(x,y,'relu')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the input is negative, the derivative of ReLU function is 0\n",
    "and when the input is positive, the derivative of ReLU function is 1.\n",
    "Note that the ReLU function is not differentiable\n",
    "when the input takes value precisely equal to  0.\n",
    "In these cases, we go with the left-hand-side (LHS) derivative\n",
    "and say that the derivative is 0 when the input is 0.\n",
    "We can get away with this because the input may never actually be zero.\n",
    "There's an old adage that if subtle boundary conditions matter,\n",
    "we are probably doing (*real*) mathematics, not engineering.\n",
    "That conventional wisdom may apply here.\n",
    "See the derivative of the ReLU function plotted below.\n",
    "\n",
    "When we use .backward(), by default it is .backward(torch.Tensor([1])).This is useful when we are dealing with single scalar input.But here we are dealing with a vector input so we have to use this snippet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAX30lEQVR4nO3df7RdZX3n8fcniYiIgJKgTH6YsBqnRsYqc4lWW0BBF6gNXWtsSzq0trrM2CWWqjMWq6JiZ5xirWNHnGlUptUiFPFHM65Y6lR0pnVCk4CiCVKzqJIUHGJRarUKOeczf+x9w+nl/tg37H33fXI+r7WycvY5+577zcq993P383z388g2ERExvpb0XUBERPQrQRARMeYSBBERYy5BEBEx5hIEERFjblnfBczX8uXLvXbt2r7LiIgoyu7du79te8V0rxUXBGvXrmXXrl19lxERURRJ35zptQwNRUSMuQRBRMSYSxBERIy5BEFExJhLEEREjLnOgkDS1ZLulfTVGV6XpN+XtE/SbZLO6KqWiIiYWZdXBH8InD/L6xcA6+s/W4D/1mEtERExg87uI7D9vyWtneWUC4EPu1oHe4ekkySdavuermqKWAj3ff8BrtnxTR4cDPsuJY4y5z71ifzE6pNaf98+byhbCewfOT5QP/ewIJC0heqqgTVr1ixIcRFH6sY93+Ldn/0bAKSei4mjyiknHHvUBcF03yLT7pJjeyuwFWBiYiI76cSiNnklsPvN53Hy8Y/uuZqIufXZNXQAWD1yvAq4u6daIlozHFa/qyzJ5UAUos8g2Ab8ct099Gzg/swPxNFgUF+zJgiiFJ0NDUm6FjgHWC7pAPBW4FEAtv87sB14EbAP+AHwq13VErGQJvcBX5K7dKIQXXYNbZ7jdQOv7urzR/RlkKGhKEx+Z4loWZ0DLF2SIIgyJAgiWjash4ZyQRClSBBEtGyya2hpkiAKkSCIaNnAmSOIsiQIIlo2OUewJHMEUYgEQUTLhkOTDIiSJAgiWja00zEURUkQRLRsYKPMD0RBEgQRLbPTMRRlSRBEtGyQOYIoTIIgomVDOx1DUZQEQUTLqq6hBEGUI0EQ0bKhs85QlCVBENGygTNHEGVJEES0zM7QUJQlQRDRskHmCKIwCYKIlmWOIEqTIIho2XDo7EUQRUkQRLQsaw1FaRIEES0bOnsRRFkSBBEtqxad67uKiOYSBBEts51F56IoCYKIlqV9NEqTIIho2dDZpjLKkiCIaFm2qozSJAgiWpb20ShNgiCiZQOTrSqjKAmCiJZVXUN9VxHRXIIgomXpGorSJAgiWpatKqM0CYKIlg2HpGsoitJpEEg6X9IdkvZJumya19dIuknSrZJuk/SiLuuJWAjpGorSdBYEkpYCVwEXABuAzZI2TDntzcD1tp8JXAS8v6t6IhbKIDuURWG6vCLYCOyzfaftB4DrgAunnGPghPrxicDdHdYTsSCy+miUpssgWAnsHzk+UD836m3AxZIOANuB10z3RpK2SNoladfBgwe7qDWiNbmzOErTZRBM963gKcebgT+0vQp4EfARSQ+ryfZW2xO2J1asWNFBqRHtyRxBlKbLIDgArB45XsXDh35eAVwPYPv/AscCyzusKaJzg6FzZ3EUpcsg2Amsl7RO0jFUk8HbppxzF3AugKSnUgVBxn6iaDbZjyCK0lkQ2D4EXALcCNxO1R20R9IVkjbVp70eeKWkLwPXAr9ie+rwUURRBjZLcodOFGRZl29uezvVJPDoc5ePPN4LPLfLGiIW2jDto1GY/N4S0bJh1hqKwiQIIlo2NOkaiqIkCCJaVnUN9V1FRHMJgoiWOXMEUZgEQUTLBnbaR6MoCYKIlg1N2kejKPlyjWhZuoaiNAmCiJblPoIoTYIgomVpH43SJAgiWjZM+2gUJkEQ0bJhuoaiMAmCiJZVi84lCKIcCYKIlmWryihNgiCiZdmqMkqTIIhoWbaqjNIkCCJaZJuhyVaVUZQEQUSLJvfXS9dQlCRBENGiQZ0EGRmKkiQIIlo0nAyCJEEUpNGexZIeD/wL4J+Ab9gedlpVRKGG9XdG2kejJDMGgaQTgVcDm4FjgIPAscATJe0A3m/7pgWpMqIQk1cES3OtHQWZ7YrgBuDDwE/b/u7oC5L+NfBLkk6z/aEuC4woyUNzBLkiiHLMGAS2XzDLa7uB3Z1UFFEwZ2goCjTnBaykV0w5Xirprd2VFFGudA1FiZqMZJ4rabukUyWdDuwAHtdxXRFFemiOIEkQ5Ziza8j2L0r6BeArwA+Azbb/qvPKIgo0HFZBkDuLoyRNhobWA5cCHwe+QTVJfFzHdUUUqc6BzBFEUZoMDf1P4C22/x1wNvB1YGenVUUUapD20ShQkxvKNtr+BwDbBt4taVu3ZUWUKUNDUaIZf2+R9FMAkyEwyvbXJZ1QTx5HRO3wZHGCIAoy2wXsv5H0RUmXS3qxpI2SzpL0ckkfAT4NPGa2N5d0vqQ7JO2TdNkM5/y8pL2S9kj66CP4t0T07vAcQYaGoiCz3VD22nqNoZcCPwecSrXW0O3AH9j+y9neWNJS4CrgBcABYKekbbb3jpyzHngj8Fzb35F0yiP9B0X0aTDMncVRnlnnCGx/B/hA/We+NgL7bN8JIOk64EJg78g5rwSuqj8Ptu89gs8TsWg4S0xEgeacLJZ0+XTP275ijg9dCewfOT4APGvKOU+pP8dfAUuBt9n+s2lq2AJsAVizZs1cJUf0ZpAbyqJATUYyvz/yZwBcAKxt8HHTfSd4yvEyYD1wDtUqpx+UdNLDPsjeanvC9sSKFSsafOqIfjy0DHW/dUTMR5M7i989eizpd4Em7aMHgNUjx6uAu6c5Z4ftB4G/lXQHVTDkPoUo0jBDQ1GgI+ltOA44rcF5O4H1ktZJOga4iIcHyKeA5wFIWk41VHTnEdQUsSgkCKJETeYIvsJDQzpLgRXAXPMD2D4k6RLgxvrjrra9R9IVwC7b2+rXXihpL9Ww03+w/fdH9k+J6N9k+2jmCKIkTe4sfsnI40PA/7N9qMmb294ObJ/y3OUjjw28rv4TUbzB4TuLey4kYh5m26ryCfXD70156QRJ2L6vu7IiyuR0DUWBZrsi2E01JDRT90+TeYKIsZIbyqJEs91ZvG4hC4k4GmQZ6ihRk/0IJOliSW+pj9dI2th9aRHlGWaryihQk/bR9wM/Cfxiffw9qjWEImKKbFUZJWrSNfQs22dIuhWq9Yfq+wIiYopB9iOIAjW5IniwXknUAJJWAMNOq4oolHMfQRSoSRD8PvBJ4BRJ/xH4S+A/dVpVRKEe6hrquZCIeWiy1tA1knYD51K1kv6s7ds7ryyiQFliIko0axBIWgLcZvt04GsLU1JEuRIEUaJZh4ZsD4EvS8omABENZKvKKFGTrqFTgT2S/ppqTwIAbG/qrKqIQk3OEWTz+ihJkyB4e+dVRBwlJoeG0j4aJWkyWfyFhSgk4miQG8qiRBnJjGhRtqqMEiUIIlo0SNdQFGjGIJD0F/Xfv7Nw5USUbXI/giW5JIiCzDZHcKqks4FNkq5jyr4Etm/ptLKIAg3qoaF0DUVJZguCy4HLgFXA7015zcDzuyoqolRZhjpKNNvGNDcAN0h6i+13LGBNEcUaZmgoCtSkffQdkjYBZ9VPfd72p7stK6JMw2xVGQVqskPZO4FLgb31n0vr5yJiisHkMtQJgihIkzuLXww8o153CEl/BNwKvLHLwiJKNNk1pDRmR0GafrmeNPL4xC4KiTgaZK2hKFGTK4J3ArdKuomqhfQscjUQMa3Dq48mCKIgTSaLr5X0eeBMqiD4Tdvf6rqwiBI91DXUcyER89DkigDb9wDbOq4lonjpGooS5feWiBZNrjWUOYIoSYIgokWTcwTJgSjJjENDkp4w2wfavq/9ciLKZpslysY0UZbZ5gh2U60pJGAN8J368UnAXcC6zquLKMxg6MwPRHFmHBqyvc72acCNwM/YXm77ZOAlwCeavLmk8yXdIWmfpMtmOe+lkixpYr7/gIjFZOhMFEd5mswRnGl7++SB7c8AZ8/1QZKWAlcBFwAbgM2SNkxz3uOAXwdublp0xGI1tNM6GsVp8iX7bUlvlrRW0pMlvQn4+wYftxHYZ/tO2w8A1wEXTnPeO4ArgR82rjpikRpmaCgK1CQINgMrgE8CnwJOqZ+by0pg/8jxgfq5wyQ9E1g912qmkrZI2iVp18GDBxt86oh+DOy0jkZxmtxZfB/V6qPzNd13gw+/KC0B3gP8SoMatgJbASYmJjzH6RG9sdM6GuWZMwgkrQDeADwNOHbyedtz7VB2AFg9crwKuHvk+HHA6cDn61a7JwHbJG2yvatR9RGLzGBolmZTmihMk6Gha4CvUbWLvh34BrCzwcftBNZLWifpGOAiRpapsH1/3Ym01vZaYAeQEIiiDZ05gihPkyA42faHgAdtf8H2y4Fnz/VBtg8Bl1C1n94OXG97j6Qr6h3PIo46VddQgiDK0mTRuQfrv++R9GKq4Z1VTd68bjvdPuW5y2c495wm7xmxmA2H2bg+ytMkCH5b0onA64H/CpwAvLbTqiIKla6hKNGsQVDfFLa+bu+8H3jeglQVUaihnXWGojizzhHYHgAZz49oaJiuoShQk6GhL0p6H/AnwPcnn7R9S2dVRRSqWmuo7yoi5qdJEDyn/vuKkecMzHUfQcTYGaRrKArU5M7izAtENOTcRxAFanJn8eumefp+YLftL7VfUkS5BsN0DUV5mtxQNgG8imrBuJXAFuAc4AOS3tBdaRHlGWatoShQkzmCk4EzbP8jgKS3AjcAZ1HtYnZld+VFlCVdQ1GiJlcEa4AHRo4fBJ5s+5+AH3VSVUShstZQlKjJFcFHgR2S/rQ+/hngWkmPBfZ2VllEgQYmXUNRnCZdQ++QtB34Kao9Bl41skLov+2yuIjSVF1DfVcRMT9NrgiwvZtqPiAiZpGuoShRttmOaFHmCKJECYKIFg2HaR+N8iQIIlo0dNpHozwJgogWDTI0FAVKEES0aJj20ShQgiCiRWkfjRIlCCJalPbRKFGCIKJF1aJzCYIoS4IgokXVonN9VxExP/mSjWhRbiiLEiUIIlqUrSqjRAmCiBbZ5IogipMgiGhR1TXUdxUR85MgiGhR5giiRAmCiBYNh5kjiPIkCCJaNDS5sziKkyCIaNEgq49GgToNAknnS7pD0j5Jl03z+usk7ZV0m6S/kPTkLuuJ6Jrt3FkcxeksCCQtBa4CLgA2AJslbZhy2q3AhO2nAzcAV3ZVT8RCyFpDUaIurwg2Avts32n7AeA64MLRE2zfZPsH9eEOYFWH9UR0LnMEUaIug2AlsH/k+ED93ExeAXxmuhckbZG0S9KugwcPtlhiRLvSNRQl6jIIpvtu8LQnShcDE8C7pnvd9lbbE7YnVqxY0WKJEe3KfQRRomUdvvcBYPXI8Srg7qknSToPeBNwtu0fdVhPROfSNRQl6vKKYCewXtI6SccAFwHbRk+Q9EzgD4BNtu/tsJaIBVHtR9B3FRHz01kQ2D4EXALcCNwOXG97j6QrJG2qT3sXcDzwMUlfkrRthreLKMJwmKGhKE+XQ0PY3g5sn/Lc5SOPz+vy80cstKHTPhrlyZ3FES2xnfbRKFKCIKIlrnvi0j4apUkQRLRkUCdB5giiNAmCiJYM6yBI+2iUJkEQ0ZLhsPo7FwRRmgRBREsOXxEkCaIwCYKIlmSOIEqVIIhoieuhoXQNRWkSBBEtGR6+Iui5kIh5ShBEtGSQrqEoVIIgoiWTVwTZqjJKkyCIaMlk+2i6hqI0CYKIlmSOIEqVIIhoyWBYB0GSIAqTIIhoyeFF5zI0FIVJEES05KGuoZ4LiZinfMlGtGSYO4ujUAmCiJYMhwmCKFOCIKIlw8wRRKESBBEtmewayhxBlCZfshEtyZ3FUaoEQURLMlkcpUoQRLRkco4gQ0NRmnzJRrRkco4gQ0NRmgRBREucrSqjUAmCiJYMch9BFCpBENGSw/cR5LsqCpMv2YiWpGsoSpUgiGjJMFtVRqESBBEteWiOoOdCIuYpQRDRkuxHEKXqNAgknS/pDkn7JF02zeuPlvQn9es3S1rbZT0RXUrXUJSqsyCQtBS4CrgA2ABslrRhymmvAL5j+8eA9wC/01U9EV3LHEGUalmH770R2Gf7TgBJ1wEXAntHzrkQeFv9+AbgfZLkyTtzWnT9zv184P/c2fbbRhz2jz86BEAuCKI0XQbBSmD/yPEB4FkznWP7kKT7gZOBb4+eJGkLsAVgzZo1R1TMScc9ivVPPP6IPjaiqXMe8yh+7JR8nUVZugyC6X4vmvqbfpNzsL0V2AowMTFxRFcLL3zak3jh0550JB8aEXFU63Ky+ACweuR4FXD3TOdIWgacCNzXYU0RETFFl0GwE1gvaZ2kY4CLgG1TztkGvKx+/FLgc13MD0RExMw6Gxqqx/wvAW4ElgJX294j6Qpgl+1twIeAj0jaR3UlcFFX9URExPS6nCPA9nZg+5TnLh95/EPg57qsISIiZpc7iyMixlyCICJizCUIIiLGXIIgImLMqbRuTUkHgW8e4YcvZ8pdy4vIYq0tdc3PYq0LFm9tqWv+jqS2J9teMd0LxQXBIyFpl+2JvuuYzmKtLXXNz2KtCxZvbalr/tquLUNDERFjLkEQETHmxi0ItvZdwCwWa22pa34Wa12weGtLXfPXam1jNUcQEREPN25XBBERMUWCICJizI1dEEh6hqQdkr4kaZekjX3XNEnSayTdIWmPpCv7rmcqSf9ekiUt77sWAEnvkvQ1SbdJ+qSkk3qu5/z6/2+fpMv6rGWSpNWSbpJ0e/11dWnfNY2StFTSrZI+3XctoySdJOmG+uvrdkk/2XdNAJJeW/8/flXStZKObeN9xy4IgCuBt9t+BnB5fdw7Sc+j2sP56bafBvxuzyX9M5JWAy8A7uq7lhGfBU63/XTgb4A39lWIpKXAVcAFwAZgs6QNfdUz4hDwettPBZ4NvHqR1DXpUuD2vouYxnuBP7P948BPsAhqlLQS+HVgwvbpVMv7t7J0/zgGgYET6scn8vBd0/rya8B/tv0jANv39lzPVO8B3sA0W4n2xfaf2z5UH+6g2gWvLxuBfbbvtP0AcB1VsPfK9j22b6kff4/qB9rKfquqSFoFvBj4YN+1jJJ0AnAW1X4p2H7A9nf7reqwZcBj6h0dj6Oln1/jGAS/AbxL0n6q37p7+y1yiqcAPy3pZklfkHRm3wVNkrQJ+DvbX+67llm8HPhMj59/JbB/5PgAi+QH7iRJa4FnAjf3W8lh/4Xql4th34VMcRpwEPgf9bDVByU9tu+ibP8d1c+su4B7gPtt/3kb793pxjR9kfS/gOl2qn8TcC7wWtsfl/TzVKl/3iKoaxnweKrL9zOB6yWdtlBbd85R228BL1yIOqaarS7bf1qf8yaqIZBrFrK2KTTNc4vm6knS8cDHgd+w/Q+LoJ6XAPfa3i3pnL7rmWIZcAbwGts3S3ovcBnwlj6LkvR4qqvMdcB3gY9Jutj2Hz/S9z4qg8D2jD/YJX2YalwS4GMs4GXpHHX9GvCJ+gf/X0saUi0sdbDP2iT9K6ovvC9Lgmr45RZJG21/q6+6Rup7GfAS4Nye97s+AKweOV7FIhl2lPQoqhC4xvYn+q6n9lxgk6QXAccCJ0j6Y9sX91wXVP+XB2xPXjndQBUEfTsP+FvbBwEkfQJ4DvCIg2Ach4buBs6uHz8f+HqPtYz6FFU9SHoKcAyLYOVD21+xfYrttbbXUn2TnLEQITAXSecDvwlssv2DnsvZCayXtE7SMVSTeNt6rglV6f0h4Hbbv9d3PZNsv9H2qvpr6iLgc4skBKi/tvdL+pf1U+cCe3ssadJdwLMlHVf/v55LS5PYR+UVwRxeCby3nmz5IbCl53omXQ1cLemrwAPAy3r+DbcE7wMeDXy2vlrZYftVfRRi+5CkS4Abqbo5rra9p49apngu8EvAVyR9qX7ut+r9xGNmrwGuqUP9TuBXe66HepjqBuAWqqHQW2lpqYksMRERMebGcWgoIiJGJAgiIsZcgiAiYswlCCIixlyCICJizCUIIiLGXIIgImLMJQgiHiFJZ9Z7Ihwr6bH1evGn911XRFO5oSyiBZJ+m2rNnMdQrVPzzp5LimgsQRDRgnopgp1Uy5Y8x/ag55IiGsvQUEQ7ngAcDzyO6sogohi5IohogaRtVLuSrQNOtX1JzyVFNDaOq49GtErSLwOHbH+03rv4i5Keb/tzfdcW0USuCCIixlzmCCIixlyCICJizCUIIiLGXIIgImLMJQgiIsZcgiAiYswlCCIixtz/B0lhayVd94bJAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y.backward(torch.ones_like(x),retain_graph=True)\n",
    "xyplot(x,x.grad,\"grad of relu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that there are many variants to the ReLU function, such as the parameterized ReLU (pReLU) of [He et al., 2015](https://arxiv.org/abs/1502.01852). This variation adds a linear term to the ReLU, so some information still gets through, even when the argument is negative.\n",
    "\n",
    "$$\\mathrm{pReLU}(x) = \\max(0, x) + \\alpha \\min(0, x)$$\n",
    "\n",
    "The reason for using the ReLU is that its derivatives are particularly well behaved - either they vanish or they just let the argument through. This makes optimization better behaved and it reduces the issue of the vanishing gradient problem (more on this later).\n",
    "\n",
    "### Sigmoid Function\n",
    "\n",
    "The sigmoid function transforms its inputs which take values in $\\mathbb{R}$ to the interval $(0,1)$.\n",
    "For that reason, the sigmoid is often called a *squashing* function:\n",
    "it squashes any input in the range (-inf, inf)\n",
    "to some value in the range (0,1).\n",
    "\n",
    "$$\\mathrm{sigmoid}(x) = \\frac{1}{1 + \\exp(-x)}.$$\n",
    "\n",
    "In the earliest neural networks, scientists\n",
    "were interested in modeling biological neurons\n",
    "which either *fire* or *don't fire*.\n",
    "Thus the pioneers of this field, going all the way back to McCulloch and Pitts in the 1940s, were focused on thresholding units.\n",
    "A thresholding function takes either value $0$\n",
    "(if the input is below the threshold)\n",
    "or value $1$ (if the input exceeds the threshold)\n",
    "\n",
    "\n",
    "When attention shifted to gradient based learning,\n",
    "the sigmoid function was a natural choice\n",
    "because it is a smooth, differentiable approximation to a thresholding unit.\n",
    "Sigmoids are still common as activation functions on the output units,\n",
    "when we want to interpret the outputs as probabilities\n",
    "for binary classification problems\n",
    "(you can think of the sigmoid as a special case of the softmax)\n",
    "but the sigmoid has mostly been replaced by the simpler and easier to train ReLU for most use in hidden layers.\n",
    "In the \"Recurrent Neural Network\" chapter, we will describe\n",
    "how sigmoid units can be used to control\n",
    "the flow of information in a neural network\n",
    "thanks to its capacity to transform the value range between 0 and 1.\n",
    "\n",
    "See the sigmoid function plotted below.\n",
    "When the input is close to 0, the sigmoid function\n",
    "approaches a linear transformation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xl8VPW9//HXZ7ITEhDCTiCgLAKK2rC4VsV97XJV3Op2pb23WrW9vbXtrW29v/7aa2u3W7t4lWrdAJda2ouKVm3dgIDgwiaRLWGHAAGyZz73jxnSGAMZMJMzy/v5eOQxc86cSd6Qybzne1Zzd0RERABCQQcQEZHEoVIQEZEWKgUREWmhUhARkRYqBRERaaFSEBGRFioFERFpoVIQEZEWKgUREWmRGXSAQ1VUVOQlJSVBxxARSSqLFi3a7u59Olou6UqhpKSEhQsXBh1DRCSpmNm6WJbT6iMREWmhUhARkRYqBRERaaFSEBGRFioFERFpEbdSMLPpZrbVzN4/wONmZr80s3Ize9fMTohXFhERiU08RwoPAecd5PHzgRHRr2nAb+KYRUREYhC34xTc/e9mVnKQRS4F/uCR64HOM7OeZjbA3TfFK5OIyMG4O43NTkNzmPrGZhqawzQ0Rb+awzSH/eNf7jSFnXA4ctvRMu5O2MGjP88dwu440VtvPR+cyPK4M+Xofowv7hnX/4MgD14bBFS0mq6MzvtYKZjZNCKjCYYMGdIl4UQkOdQ2NLOzpoGqfQ3sqmmkqqaB6tpG9tU3sa+hmX31TdQ0NLG3PnI/Mr+Jmvpm6pvC1DdFbve/8SfqZevNoF+P3JQuBWtnXru/Dne/H7gfoLS0NEF/ZSLSmeoam6moqmHj7jo2765l8+56NlfXsnl3HVuq69lZ08DOmgbqGsMH/T752Rl0y8mke04m+TkZdMvOpG9BLnm9M8jNzCAnK0R2RoiczMhXdvQrJzMjcj8jRE5WiMxQiMyQkZFhZJiRGTJCoTa3ZmRmtLofChEK0XKbYZH5ZmDR25AZBq3mg2GEosvsv+0qQZZCJVDcanowsDGgLCISkK3VdSzdWE351r2s2bGPtdsjX5uq6z72qb2oezb9e+QyoEcuYwYW0is/m57dsujVLZue3bLplZ/NEd2y6JGXRX5OJnlZGYRCXfeGmgqCLIXZwC1mNgOYBOzW9gSR1LZxVy2L1+9i6cbdLN1YzdKN1WzfW9/yeM9uWZT0zmfS8N6U9M6npKgbg3rm0a8wl36FuWRnai/6eItbKZjZE8DpQJGZVQLfBbIA3P23wBzgAqAcqAFuiFcWEQnGpt21vPbBduavqWL+mh1U7qwFIDNkjOhXwOmj+jB2YCFjB/ZgZL/u9OyWHXBiiefeR1d28LgDX47XzxeRrhcOO0sqd/Hy8q38dcVWlm+qBqBXfjYTS3px48nDKC05glH9C8jJzAg4rbQn6U6dLSKJZ+XmPTy7ZAOzl2xkw65aMkLGp4YewZ3nj+b0UX0Y1a+gSzeWyuFTKYjIYaltaObZJRt45K11LNtUTUbIOOWoIr52zkimjO5Hj25ZQUeUw6BSEJFDUlFVwyPz1jGzrILdtY2M7l/A9y4ew4XHDqRPQU7Q8eQTUimISEw27KrlVy+v4smFlThw3tj+XHdSCRNKjtCqoRSiUhCRg9paXcevXilnxoLICQiumjSEL336SAb2zAs4mcSDSkFE2tXUHObht9bxsxc/oK6xmctKB3PLmSMYpDJIaSoFEfmYReuq+PYf32fF5j18emQfvn/JWEqK8oOOJV1ApSAiLeoam/nRcyt46M21DOiRy2+vOYFzx/bXNoM0olIQEQBWbdnDrU8sZsXmPVx/UglfP3cU+Tl6i0g3+o2LpDl3Z0ZZBd//81LyszP5/Q0TOGNU36BjSUBUCiJprKEpzLf/+B5PLqrk1BFF3Hv5ePoW5AYdSwKkUhBJU7trGvnSo4t4a/UOvjJlBLdPGaHTTItKQSQdrd9Rww0PLWB9VQ0/vXw8nzthcNCRJEGoFETSzPsbdvOF6QsIu/PoTZOYNLx30JEkgagURNLI0o27uebB+eRnZ/LITRMZ3qd70JEkwagURNLEso3VXP3AfLplZfDEzZMZ0rtb0JEkAenadiJpIFII88jLyuCJaSoEOTCVgkiKW7t9H9c8OJ/crAxmTJvM0N46XYUcmEpBJIXtqmngxofKcHcev1mFIB3TNgWRFNXQFOZLjy6icmctj/7zJIbphHYSA5WCSApyd775zHvMW13Fz64Yz8RhvYKOJElCq49EUtCvX/2Qp9+u5LYpI/js8TowTWKnUhBJMW+Ub+cnc1dyyfiB3H7WiKDjSJJRKYikkG176rl95hKGF+Xzo88fo+sgyCHTNgWRFBEOO1+dtYTq2kb+cONEumXrz1sOnUYKIinit3//kNdWbee7F4/l6AGFQceRJKVSEEkBi9ZVce/cD7jw2AFcObE46DiSxFQKIkmutqGZO2a+w6Ceefzwc9qOIJ+MVjqKJLl7565kfVUNM6ZNpjA3K+g4kuQ0UhBJYovX72T6G2u4etIQJuu6CNIJ4loKZnaema00s3Izu7Odx4eY2StmttjM3jWzC+KZRySVNDSF+cbT79KvMJc7zx8ddBxJEXErBTPLAO4DzgfGAFea2Zg2i/0HMMvdjwemAr+OVx6RVPPrV8v5YMtefvDZcRRotZF0kniOFCYC5e6+2t0bgBnApW2WcWD/vnM9gI1xzCOSMlZu3sN9r5TzmeMGcubofkHHkRQSzw3Ng4CKVtOVwKQ2y3wPmGtmtwL5wFlxzCOSEtyd7/zpfbrnZHLXxWODjiMpJp4jhfb2i/M201cCD7n7YOAC4BEz+1gmM5tmZgvNbOG2bdviEFUkeTz3/mYWrKnia+eMold+dtBxJMXEsxQqgdZH0Qzm46uHbgJmAbj7W0AuUNT2G7n7/e5e6u6lffr0iVNckcRX19jM/5+znNH9C5g6QQepSeeLZymUASPMbJiZZRPZkDy7zTLrgSkAZnY0kVLQUEDkAB58fQ2VO2u566IxZGZoj3LpfHF7Vbl7E3AL8AKwnMheRkvN7G4zuyS62NeAm83sHeAJ4Hp3b7uKSUSALdV13PdKOeeO7cdJR31sQC3SKeJ6RLO7zwHmtJl3V6v7y4CT45lBJFXc8/xKmpqdb1/Qds9ukc6j8adIEnivcjdPv13JjacMY0jvbkHHkRSmUhBJAve8sIJe+dl8+Ywjg44iKU6lIJLg5q/ewWurtvMvnz5SRy5L3KkURBKYu3Pv3A/oW5DDNZOHBh1H0oBKQSSBvV6+nQVrq7jlzKPIy84IOo6kAZWCSIJyd37ywkoG9czjCh2oJl1EpSCSoF5avpV3Kndz25QR5GRqlCBdQ6UgkoDCYefeuSsZVpTP504YFHQcSSMqBZEENHfZFlZs3sPtZ43Q6SykS+nVJpJg3J3fvFrO0N7duPCYAUHHkTSjUhBJMG9+uIN3KnfzxdOO1ChBupxecSIJ5jevfkifghxtS5BAqBREEsi7lbt4vXw7/3zKMHKztMeRdD2VgkgC+fUrH1KYm8lVk4YEHUXSlEpBJEGUb93LC8s284UTS3SOIwmMSkEkQfzubx+SnRHi+pNLgo4iaUylIJIAtlTX8eySDVwxoZii7jlBx5E0plIQSQCPzltHU9i56ZRhQUeRNKdSEAlYXWMzj81fz5TR/RjaOz/oOJLmVAoiAfvTkg1U7WvgxlNKgo4iolIQCZK7M/31tYzuX8CJw3sHHUdEpSASpLc+3MHKLXu48ZRhmFnQcURUCiJBmv7GGnrnZ3PJ+IFBRxEBVAoigVmzfR9/XbGVqycN0SktJGGoFEQC8vCba8kMGddMHhp0FJEWKgWRAOytb+KpRZVcdOxA+hbmBh1HpIVKQSQAzy7ewN76Jq49UaMESSwqBZEu5u48Om8dYwYUcnxxz6DjiHyESkGkiy1at5MVm/dw7YlDtRuqJByVgkgXe3TeOgpyMrn0OO2GKoknM9YFzewIYCBQC6x193DcUomkqB1765nz3maumjSEbtkx//mJdJmDjhTMrIeZfcvM3gPmAb8DZgHrzOxJMzujg+efZ2YrzazczO48wDKXm9kyM1tqZo8f7j9EJBnMWlhJQ3OYq3VlNUlQHX1UeQr4A3Cqu+9q/YCZfQq41syGu/uDbZ9oZhnAfcDZQCVQZmaz3X1Zq2VGAN8ETnb3nWbW95P9c0QSV3PYeXzBOiYP78WIfgVBxxFp10FLwd3PPshji4BFB3n6RKDc3VcDmNkM4FJgWatlbgbuc/ed0e+5NcbcIknn7x9so6Kqlm+cNzroKCIHFNOGZjO7qc10hpl9t4OnDQIqWk1XRue1NhIYaWZvmNk8MzvvAD9/mpktNLOF27ZtiyWySMJ5bP46irrncM6Y/kFHETmgWPc+mmJmc8xsgJmNI7J9oaPxb3v72nmb6UxgBHA6cCXwgJl9bMdtd7/f3UvdvbRPnz4xRhZJHJt31/Hyiq1cXjqY7Ezt9CeJK6bdH9z9KjO7AngPqAGudPc3OnhaJVDcanowsLGdZea5eyOwxsxWEimJslhyiSSLp9+uJOxweWlxxwuLBCjW1UcjgNuAp4G1RDYwd+vgaWXACDMbZmbZwFRgdptlngXOiP6MIiKrk1bHnF4kCYTDzsyyCk4c3puSIl1uUxJbrOPYPwPfcfcvAp8GVtHBp3l3bwJuAV4AlgOz3H2pmd1tZpdEF3sB2GFmy4BXgK+7+47D+HeIJKx5q3ewvqqGqRM1SpDEF+vRMxPdvRrA3R2418zafur/GHefA8xpM++uVvcd+Gr0SyQlzSiroEdeFueO1QZmSXwdHbx2CsD+QmjN3VeZWWF0w7OItGPnvgaef38znz1+kC6kI0mho5HC583sHuB5IsckbANygaOIbAsYCnwtrglFktizSzbQ0BzmigladSTJoaOD1+6InvPon4DLgAFEzn20HPidu78e/4giycndmbGggvHFPTl6QGHQcURi0uE2hejRxv8T/RKRGC2p2MXKLXv44eeOCTqKSMwOWgpmdtANwO7+086NI5I6ZpZV0C07g4vH6xTZkjw6GinsP2p5FDCBfxxncDHw93iFEkl2e+ubmP3ORi46dgDdc3SKbEkeHW1T+D6Amc0FTnD3PdHp7wFPxj2dSJL633c3UtPQzBUTdIpsSS6xHrw2BGhoNd0AlHR6GpEUMaOsghF9u3PCEF2DWZJLrOPaR4AFZvZHIie1+yyR6yyISBsrN+9h8fpd/MeFR+sazJJ0Yj0h3g/M7Dng1OisG9x9cfxiiSSvmWUVZGUYnzthcNBRRA5ZR3sfFbp7tZn1InIivLWtHuvl7lXxjSeSXOqbmnlmcSXnjO1Pr/zsoOOIHLKORgqPAxcROZrZ+eg1EhwYHqdcIklp7tIt7KppZKqOYJYk1dHeRxdFb4d1TRyR5DazrIJBPfM4+ciioKOIHJaYd6COnu76tOjkq+7+l/hEEklO63fU8Hr5dr569khCIW1gluQU60V2fkTkIjvLol+3mdkP4xlMJNnMWlhByOCyUm1gluQV60jhAuA4dw8DmNnDwGLgm/EKJpJMmprDPLmogtNH9WVAj7yg44gctkO5gnjro3B6dHYQkWT2tw+2saW6XqfIlqQX60jhh8BiM3uFyB5Ip6FRgkiLGWUVFHXP4czRfYOOIvKJxHrw2hNm9iqRk+IZ8A133xzPYCLJYmt1HS+v2MrNpw4nK+NQBt8iiedQXsF9orcZwElm9rk45BFJOk+9XUlz2LXqSFJCTCMFM5sOHAssBcLR2Q48E6dcIknB3ZlZVsGkYb0YVpQfdByRTyzWbQqT3X1MXJOIJKF5q6tYt6OG288aEXQUkU4R6+qjt8xMpSDSxsyy9RTkZnL+uAFBRxHpFLGOFB4mUgybgXoiG5vd3Y+NWzKRBLerpoE5729m6oRicrMygo4j0iliLYXpwLXAe/xjm4JIWnvm7Q00NIW1gVlSSqylsN7dZ3e8mEh6cHeeWLCe8cU9GTtQx3JK6oi1FFaY2ePAn4msPgLA3bX3kaSlRet2smrrXu75vNagSmqJtRTyiJTBOa3maZdUSVuPz19P95xMLhqvDcySWmI9ovmGeAcRSRa7ahr4y3ubuLx0MN2yYz77vEhSiPXgtV+2M3s3sNDd/9S5kUQS2x8XRzYwXzVxaNBRRDpdrMcp5ALHAauiX8cCvYCbzOznB3qSmZ1nZivNrNzM7jzIcv9kZm5mpYeQXaTLuTuPz49sYB4zsDDoOCKdLtax71HAme7eBGBmvwHmAmcT2U31Y8wsA7gvukwlUGZms919WZvlCoCvAPMP618g0oX2b2D+r88fE3QUkbiIdaQwCGh9Ypd8YKC7N9Nqb6Q2JgLl7r7a3RuAGcCl7Sz3n8A9QF2MWUQC8/iCyAbmi8cPDDqKSFzEWgr3AEvM7Pdm9hCRq679xMzygZcO8JxBQEWr6crovBZmdjxQrOs9SzLYXdPI/767ic8cP1AbmCVlxbr30YNmNofIp38DvuXuG6MPf/0AT2vvyuXe8qBZCPgZcH1HP9/MpgHTAIYMGRJLZJFO98ziSuqbwlw5Ua9BSV0HHSmY2ejo7QnAACKf/NcD/aPzDqYSaH38/2BgY6vpAmAc8KqZrQUmA7Pb29js7ve7e6m7l/bp06ftwyJxpyOYJV10NFL4KpFP6Pe2muet7p95kOeWASPMbBiwAZgKXNXyTdx3A0X7p6NXdvs3d18YU3KRLrRo3U4+2KINzJL6DjpScPdp0bu/AS519zOAV4gco/BvHTy3CbgFeAFYDsxy96VmdreZXfKJk4t0of0bmC86VhuYJbXFurXsP9x9lpmdQmQX03uJFMWkgz3J3ecAc9rMu+sAy54eYxaRLrVzXwP/++4mLisdTH6ONjBLaot176Pm6O2FwG+jRzFnxyeSSGKZUVZBfVOYayeXBB1FJO5iLYUNZvY74HJgjpnlHMJzRZJWc9h5dN46Thzem1H9C4KOIxJ3sb6xX05k28B57r6LyCkuDrQrqkjK+OvyLWzYVct1J+k8R5IeYj1OoYZWp8l2903ApniFEkkUD7+1loE9cjnr6H5BRxHpEloFJHIA5Vv38Eb5Dq6ePJTMDP2pSHrQK13kAB5+cx3ZmSGm6hrMkkZUCiLtqK5r5Om3K7n42IH07p4TdByRLqNSEGnHM4sqqWlo1gZmSTsqBZE2msPOQ2+u5bjinhw7uGfQcUS6lEpBpI0Xl21h7Y4abj51eNBRRLqcSkGkjf95bTXFvfI4d6x2Q5X0o1IQaWXRup0sWreTm04ept1QJS3pVS/SygOvraZHXhaXlWo3VElPKgWRqHU79vH80s1cPWmIzoYqaUulIBL14OtryAwZ159UEnQUkcCoFESIXDNh1sIKPnPcIPoW5gYdRyQwKgUR4JF566hrDHPzadoNVdKbSkHS3t76Jh58fQ1TRvdlZD9dM0HSm0pB0t4f3lrL7tpGvjJlRNBRRAKnUpC0tq++iQdeW8Ppo/owvlintBBRKUhae2z+Oqr2NXDrmRoliIBKQdJYbUMz9/99NaeOKOJTQ48IOo5IQlApSNp6fMF6tu9t0LYEkVZUCpKW6hqb+e3fPuTE4b2ZUNIr6DgiCUOlIGnpsfnr2bannlunHBV0FJGEolKQtFNd18ivXl7FKUcVcdKRRUHHEUkoKgVJO7999UN21jRy5/mjg44iknBUCpJWNu2u5cHX1/CZ4wYyblCPoOOIJByVgqSVn734Ae7wtXNGBR1FJCGpFCRtfLBlD08tquQLJw6luFe3oOOIJCSVgqSN/3puBfk5mXz5DO1xJHIgcS0FMzvPzFaaWbmZ3dnO4181s2Vm9q6Z/dXMhsYzj6Sv11dt568rtvKvpx/FEfnZQccRSVhxKwUzywDuA84HxgBXmtmYNostBkrd/VjgKeCeeOWR9FXf1Mxdf3qfkt7duOHkkqDjiCS0eI4UJgLl7r7a3RuAGcClrRdw91fcvSY6OQ8YHMc8kqb+5++rWb19H3dfOo7crIyg44gktHiWwiCgotV0ZXTegdwEPNfeA2Y2zcwWmtnCbdu2dWJESXUVVTX898vlXHjMAE4b2SfoOCIJL56lYO3M83YXNLsGKAV+3N7j7n6/u5e6e2mfPvrDlti4O9+dvZTMkPGdi9quuRSR9sSzFCqB4lbTg4GNbRcys7OAbwOXuHt9HPNImnlx2RZeXrGVO84eSf8euUHHEUkK8SyFMmCEmQ0zs2xgKjC79QJmdjzwOyKFsDWOWSTN7Klr5Pt/Xsbo/gVcd1JJ0HFEkkbcSsHdm4BbgBeA5cAsd19qZneb2SXRxX4MdAeeNLMlZjb7AN9O5JD851+WsWl3LT/47DFkZehwHJFYZcbzm7v7HGBOm3l3tbp/Vjx/vqSnuUs3M2thJV8+40hdUU3kEOkjlKSU7Xvr+eYz7zF2YCG3TRkZdByRpBPXkYJIV3J37nz6PfbUN/HEFceRnanPPCKHSn81kjKeXFjJS8u38O/njmJkv4Kg44gkJZWCpIQVm6v57uylTB7eixtPHhZ0HJGkpVKQpLe7ppEvPrKIgtxMfjn1eEKh9o6bFJFYaJuCJLXmsHPbzMVs3FXLjGmT6Vuog9REPgmNFCSp/fylD3h15Ta+e/FYPjW0V9BxRJKeSkGS1vPvb+a/Xy7n8tLBXD1pSNBxRFKCSkGSUtnaKm6bsZjxxT25+9JxmGk7gkhnUClI0lm+qZobHypjUM88pl9XqmskiHQilYIklXU79vGF6QvonpPJI/88id7dc4KOJJJSVAqSNLZW13HtgwtobA7zyE0TGdQzL+hIIilHpSBJoaKqhst/9xbb99bz++sncFRfHbEsEg86TkES3qote7jmwfnUNjTzyE2TOH6IznwqEi8qBUloSyp2cf3vF5CVEWLWl05kdP/CoCOJpDSVgiSsl1ds4dbHF9OrezaP3jSJob3zg44kkvJUCpJwwmHnly+v4ucvrWLswEKmXz+Bfjp9hUiXUClIQtld28gdM5fw8oqtfP6Ewfzgs+N0HIJIF1IpSMJ4e/1O7pi5hA07a/nPS8dyzeShOlJZpIupFCRwdY3N/PTFD3jgtdUM6JHHjGmTKS3Rye1EgqBSkEAtWlfF1596l9Xb9nHVpCF864Kj6Z6jl6VIUPTXJ4HYsKuWHz+/gmeXbGRQzzwevWkSp4woCjqWSNpTKUiX2lvfxG9eLeeB19bgwL+efiT/esZRGh2IJAj9JUqXqNrXwMNvruXht9ayq6aRzxw3kK+fN1rnLxJJMCoFiauKqhoefH0NM8sqqG1s5qyj+3HrmUcxvrhn0NFEpB0qBel0dY3NzF22hZll63mjfAeZIeMzxw/ii6cNZ0Q/nchOJJGpFKRTNDaHefPDHTz//iaee38zu2oaGdQzj6+ePZLLSgczoIdWE4kkA5WCHLbte+t588MdvLpyKy8t20J1XRP52RlMObofl5cWc9KRvQmFdPCZSDJRKUjMtlbX8fb6XSxaV8Ub5TtYtqkagB55WZw9pj/nj+vPKSOKdFoKkSSmUpCPCYedip01rNy8h5Wb97Biyx6WrN/Fhl21AGRnhDhhaE++fu4oTjmqiHGDepChEYFISohrKZjZecAvgAzgAXf/UZvHc4A/AJ8CdgBXuPvaeGaSiHDY2bKnjoqqWiqqalhfVUPFzho+3LaPVVv2UNPQ3LJsca88jivuyQ0nl3D8kCMYN6iQnEyNBkRSUdxKwcwygPuAs4FKoMzMZrv7slaL3QTsdPejzGwq8F/AFfHKlMrcnfqmMNW1jVTXNbK7tpFtexrYvreebXvq2bb/ttVXQ3O45flmMKAwl5KifK6YUMyofgWM6l/AyH4F5OvAMpG0Ec+/9olAubuvBjCzGcClQOtSuBT4XvT+U8CvzMzc3eOYq9O5O2GHsDsevW0OO43NYRqbI7dNzU5jONxyv2H/vOZwy3JNzWEaovdrG5upbWiitiH8j/uNzdQ2hlvu761vZk+0BKprmz7yJt+aGfTqlk2fghz6FOQwvCifPoU5FB/RjeJe3RjSqxsDe+bq07+IxLUUBgEVraYrgUkHWsbdm8xsN9Ab2N7ZYWaVVXD/a6s/8sYddicc/uibetj3TztOZDWLt3rsY8/vgvrKzgiRl51BXlbGR24LczMpPiKPwrwsCnOzKMzLjN5mUZibSVH3HPoW5NArP5vMjFD8g4pI0otnKbS35bHtW2gsy2Bm04BpAEOGDDmsMEfkZzOqXwFmEDIjFL21VvdDIVqmjeit2T+WD9lBn2/8Y5kMM7IyQmRlRG4zW92PTBvZGSEyQ0ZWZoisUIisTCMzFCI7I0Rudohu2ZnkZob0hi4iXSaepVAJFLeaHgxsPMAylWaWCfQAqtp+I3e/H7gfoLS09LA+m589ph9nj+l3OE8VEUkb8fwIWgaMMLNhZpYNTAVmt1lmNnBd9P4/AS8n2/YEEZFUEreRQnQbwS3AC0R2SZ3u7kvN7G5gobvPBh4EHjGzciIjhKnxyiMiIh2L676G7j4HmNNm3l2t7tcBl8Uzg4iIxE5bMEVEpIVKQUREWqgURESkhUpBRERaqBRERKSFJdthAWa2DVh3mE8vIg6n0OgkiZpNuQ5NouaCxM2mXIfucLINdfc+HS2UdKXwSZjZQncvDTpHexI1m3IdmkTNBYmbTbkOXTyzafWRiIi0UCmIiEiLdCuF+4MOcBCJmk25Dk2i5oLEzaZchy5u2dJqm4KIiBxcuo0URETkINKuFMzsODObZ2ZLzGyhmU0MOtN+Znarma00s6Vmdk/Qedoys38zMzezoqCzAJjZj81shZm9a2Z/NLOeAec5L/r7KzezO4PMsp+ZFZvZK2a2PPq6ui3oTK2ZWYaZLTazvwSdpTUz62lmT0VfX8vN7MSgMwGY2R3R3+P7ZvaEmeV29s9Iu1IA7gG+7+7HAXdFpwNnZmcQuWb1se4+FvhJwJE+wsyKgbOB9UFnaeVFYJy7Hwt8AHwzqCBmlgHcB5wPjAGuNLMxQeVppQn4mrsfDUwGvpwgufa7DVgedIh2/AJ43t1HA+NJgIxmNgj4ClCJycoQAAAD3ElEQVTq7uOIXJKg0y83kI6l4EBh9H4PPn41uKD8C/Ajd68HcPetAedp62fAv9PO5VKD4u5z3b0pOjmPyNX9gjIRKHf31e7eAMwgUvKBcvdN7v529P4eIm9ug4JNFWFmg4ELgQeCztKamRUCpxG53gvu3uDuu4JN1SITyIteqbIbcXj/SsdSuB34sZlVEPk0HtinyzZGAqea2Xwz+5uZTQg60H5mdgmwwd3fCTrLQdwIPBfgzx8EVLSariRB3nz3M7MS4HhgfrBJWvycyAeNcNBB2hgObAN+H1219YCZ5Qcdyt03EHnPWg9sAna7+9zO/jlxvchOUMzsJaB/Ow99G5gC3OHuT5vZ5UQ+DZyVALkygSOIDPEnALPMbHhXXZ60g2zfAs7pihxtHSyXu/8pusy3iawmeawrs7Vh7cxLmFGVmXUHngZud/fqBMhzEbDV3ReZ2elB52kjEzgBuNXd55vZL4A7ge8EGcrMjiAy+hwG7AKeNLNr3P3Rzvw5KVkK7n7AN3kz+wOR9ZgAT9KFQ9cOcv0L8Ey0BBaYWZjI+U22BZnNzI4h8iJ8x8wgsormbTOb6O6bg8rVKt91wEXAlICv710JFLeaHkyCrJo0sywihfCYuz8TdJ6ok4FLzOwCIBcoNLNH3f2agHNB5HdZ6e77R1RPESmFoJ0FrHH3bQBm9gxwEtCppZCOq482Ap+O3j8TWBVgltaeJZIHMxsJZJMAJ+Ny9/fcva+7l7h7CZE/mBO6ohA6YmbnAd8ALnH3moDjlAEjzGyYmWUT2QA4O+BMWKTJHwSWu/tPg86zn7t/090HR19TU4GXE6QQiL62K8xsVHTWFGBZgJH2Ww9MNrNu0d/rFOKwATwlRwoduBn4RXRDTR0wLeA8+00HppvZ+0ADcF3An3yTwa+AHODF6Chmnrt/KYgg7t5kZrcALxDZK2S6uy8NIksbJwPXAu+Z2ZLovG9Fr58uB3Yr8Fi04FcDNwSch+iqrKeAt4msLl1MHI5s1hHNIiLSIh1XH4mIyAGoFEREpIVKQUREWqgURESkhUpBRERaqBRERKSFSkFERFqoFEQ+ITObEL2mQ66Z5UfPdz8u6Fwih0MHr4l0AjP7f0TO4ZNH5Lw5Pww4kshhUSmIdILo6RDKiJw65SR3bw44kshh0eojkc7RC+gOFBAZMYgkJY0URDqBmc0mcrW1YcAAd78l4EgihyUdz5Iq0qnM7AtAk7s/Hr1W85tmdqa7vxx0NpFDpZGCiIi00DYFERFpoVIQEZEWKgUREWmhUhARkRYqBRERaaFSEBGRFioFERFpoVIQEZEW/wfvtLs9oLV34QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x=Variable(torch.arange(-8.0,8.0,0.1,dtype=torch.float32).reshape(int(16/0.1),1),requires_grad=True)\n",
    "y=torch.sigmoid(x)\n",
    "xyplot(x,y,'sigmoid')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The derivative of sigmoid function is given by the following equation:\n",
    "\n",
    "$$\\frac{d}{dx} \\mathrm{sigmoid}(x) = \\frac{\\exp(-x)}{(1 + \\exp(-x))^2} = \\mathrm{sigmoid}(x)\\left(1-\\mathrm{sigmoid}(x)\\right).$$\n",
    "\n",
    "\n",
    "The derivative of sigmoid function is plotted below.\n",
    "Note that when the input is 0, the derivative of the sigmoid function\n",
    "reaches a maximum of 0.25. As the input diverges from 0 in either direction, the derivative approaches 0.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xd8XNWZ//HPI416sVVdZMmWLBsXim3kCrYx1TSTsCGUkEBCYMmGJJtkNz+ySWBD9heSsMsmu2E3IYHfhoQayIIJJGDAYBsXLOOCuyW5y5ZVrGKrzzy/P2ZkBlnSjKQZ3ZH0vF+veWnmzr0zX14Weu49555zRFUxxhhjehLldABjjDGRz4qFMcaYgKxYGGOMCciKhTHGmICsWBhjjAnIioUxxpiArFgYY4wJyIqFMcaYgKxYGGOMCcjldIBQyczM1AkTJjgdwxhjBpVNmzZVqWpWoP2GTLGYMGECxcXFTscwxphBRUQOBrOfNUMZY4wJyIqFMcaYgKxYGGOMCciKhTHGmICsWBhjjAkorMVCRJaKyB4RKRGR+7t4/1sislNEtonI2yIy3u89t4hs8T2WhzOnMcaYnoXt1lkRiQYeA64AjgAbRWS5qu70220zUKSqjSLyFeBnwM2+95pUdUa48hljjAleOMdZzAFKVLUMQESeA24AzhQLVV3pt/964PYw5jFmwLk9yq5j9WzYX0NGUizzJ2YwKjXe6VjG9Fo4i0UOcNjv9RFgbg/73wX8xe91vIgUA+3AT1T15c4HiMg9wD0AeXl5/Q5sTCjtOlbPV5/5kLLK05/YfnNRLj+8YTrxMdEOJTOm98JZLKSLbdrljiK3A0XAYr/NeapaLiIFwDsi8pGqln7iw1QfBx4HKCoq6vKzjXHC8q3lfOfFraTGx/CvN13ARYUZVJ9q5eXNR/ntmv18dLSOJ+4sYsyIBKejGhOUcBaLI0Cu3+txQHnnnUTkcuB7wGJVbenYrqrlvp9lIvIuMBMo7Xy8MZGm+EAN33p+C7Py0njsc7PISokDYMyIBM7NGcFFhZl87dnN3Pv7TTz/t/PtCsMMCuG8G2ojMElE8kUkFrgF+MRdTSIyE/g1sExVT/htTxORON/zTOAi/Po6jIlUFfXNfOXpD8lNT+S3dxadKRT+lkzJ5tHPXsDWI3U8+MoOVO2i2ES+sBULVW0H7gPeAHYBL6jqDhF5SESW+XZ7BEgG/tjpFtmpQLGIbAVW4u2zsGJhIpqq8g9/3Mqp5nZ+dfuFpMbHdLvvldNHc9+SQp4vPsxrHx0bwJTG9E1YZ51V1deB1ztte8Dv+eXdHLcWOC+c2YwJtbd2nWD1vioeuG4a54xOCbj/N6+YzNu7T/Dw67u5fOooa44yEc1GcBsTAi3tbv7vazspzE7m8/PHBz4AiI4SHrx+Gkdrm3h8VVmYExrTP1YsjAmBp9Ye5EB1Iz+4bhox0cH/bzWvIINrzhvNf71bQkV9cxgTGtM/ViyM6aeWdjePry7j4sJMFk8OuODYWe5fOpXWdg9PrtkfhnTGhIYVC2P66eXNR6lsaOErl0zs0/F5GYlce/5Ynt5wiPrmthCnMyY0rFgY0w8ej/LrVWVMH5vKgokZff6cv11UwKmWdp7ZcCiE6YwJHSsWxvTDW7sqKKs8zd8unohIV5MWBMc7WC+DJ9fsp6XdHcKExoSGFQtj+uEPGw4xdkQ815w7ut+f9eWFBZxoaOGtnScC72zMALNiYUwfHTnZyOp9ldxUlIurF3dAdWfRpCzGjojn+eLDgXc2ZoBZsTCmj17cdASAm4rGheTzoqOEzxTlsnpfJUdONobkM40JFSsWxvSB26P8sfgIFxdmMi4tMWSfe9OF3sLTUYiMiRRWLIzpg/dLqjha28TNs3MD79wLuemJXFyYyR+Lj+Dx2ASDJnJYsTCmD17ecpTUeBdXTBsV8s/+zIXjOFrbRPHBkyH/bGP6yoqFMb3U3ObmzR0VLD13NHGu0E/+551UMIpXt561/IsxjrFiYUwvvbe3klMt7Vx3/tiwfH5SnIvLpozi9Y+O0e72hOU7jOktKxbG9NKrW8tJT4rt14jtQK6/YAzVp1tZX1YTtu8wpjesWBjTC42t7by96wRXnzs6JGMrunPJOdkkxUZbU5SJGFYsjOmFd3afoKnNHbYmqA7xMdFcOX00f9l+jDZrijIRwIqFMb3w5o4KMpJimZOfHvbvumr6aOqb2/lgvzVFGedZsTAmSG1uDyv3nODSKdlER/V90sBgLZqcSZwrihU7K8L+XcYEYsXCmCBtKKuhobmdK6f3f9LAYCTGulg4KZMVOytQtQF6xllWLIwJ0ps7jxMfE8XFhZkD9p1XTBvF0domdh6rH7DvNKYrViyMCYKq8tbOChZNyiIhNvQD8bpz2dRRiGBNUcZxViyMCcKO8nrK65rDMr1HTzKT47gwL82KhXGcFQtjgvDuHu+CREumZA/4d186NZsd5fWcaGge8O82poMVC2OCsGpvFefmpJKZHDfg371oUhYAq/dWDfh3G9PBioUxAdQ3t7Hp0EkWT85y5PunjfEWqff2Vjry/caAFQtjAlpbUo3bo2fO8AdaVJSwaHImq/dV4rY1LoxDrFgYE8B7eytJjnMxa3yaYxkWT87iZGMb24/WOZbBDG9WLIzpgaqyam8lCyZmEBPGiQMDubgwExFYZU1RxiFWLIzpQWnlaY7WNrH4HGeaoDpkJMdxXs4I67cwjglrsRCRpSKyR0RKROT+Lt7/lojsFJFtIvK2iIz3e+8OEdnne9wRzpzGdKfjTN6p/gp/iydnsflwLXVNbU5HMcNQ2IqFiEQDjwFXA9OAW0VkWqfdNgNFqno+8CLwM9+x6cCDwFxgDvCgiDjXYGyGrff2VlKQlURueqLTUVg8OQu3R1lbYrfQmoEXziuLOUCJqpapaivwHHCD/w6qulJVG30v1wPjfM+vAlaoao2qngRWAEvDmNWYszS3udmwvzoirioAZuSOJCXexap91hRlBl44i0UOcNjv9RHftu7cBfylj8caE3If7K+huc3jeH9FB1e0dxLD9/ZU2iy0ZsCFs1h0NeF/l7/hInI7UAQ80ptjReQeESkWkeLKSjvbMqH13t5KYl1RzMsP31rbvbVochbldc2UnDjldBQzzISzWBwBcv1ejwPOWlBYRC4HvgcsU9WW3hyrqo+rapGqFmVlRcbZnxk6Vu2tZG5++oDOMhvIIt8ocrsrygy0cBaLjcAkEckXkVjgFmC5/w4iMhP4Nd5CccLvrTeAK0UkzdexfaVvmzED4lhdE/tOnIqY/ooOOSMTKMxOZtU+6+Q2AytsxUJV24H78P6R3wW8oKo7ROQhEVnm2+0RIBn4o4hsEZHlvmNrgB/hLTgbgYd824wZEOtKqwFYUBg5TVAdLi7MZOP+GlrbPU5HMcOIK5wfrqqvA6932vaA3/PLezj2SeDJ8KUzpnvrSqsZmRjD1NGpTkc5y/yJGfzP2gNsPVLL7AnpTscxw4SN4DamC+vKqpmbn05UVFf3WjhrXn4GIt4JDo0ZKFYsjOnkcE0jR042Mb8g8pqgAEYkxnDu2BGsLbV+CzNwrFgY08nH/RWZDifp3oKJGWw+VEtTq9vpKGaYsGJhTCfryqrJTI5lUnay01G6NX9iBq1uD5sOnnQ6ihkmrFgY40dVWVdazdyCDEQir7+iw+wJ6biihPetKcoMECsWxvg5UN3I8frmiO2v6JAU52Jm3kjWllontxkYViyM8dPRaTx/YmQXC4D5EzP56Egt9c02ZbkJPysWxvhZV1rNqNQ4CjKTnI4S0IKJGXgUPiiz8aom/KxYGOOjqqwvq2F+hPdXdJiZN5I4V5Q1RZkBEXSxEJEk34JGxgxJJSdOUXWqZVA0QQHEuaKZPSHdxluYAdFtsRCRKBG5TUReE5ETwG7gmIjsEJFHRGTSwMU0JvzWlXnP0OcXRO74is7mT8xg9/EGqk61BN7ZmH7o6cpiJTAR+C4wWlVzVTUbWIh3Vbuf+NahMGZIWFtSTc7IBHLTE5yOErQFvqug9WXWFGXCq6eJBC9X1bNus/DN/voS8JKIxIQtmTEDyONR1u+v5vKpowZFf0WH83JGkBznYm1pNdedP9bpOGYI6/bKoqNQ+BYn+gQRucN/H2MGu93HG6htbIv48RWduaKjmJOfznrr5DZhFkwH9wMi8t++Du5RIvIqcH24gxkzkM70VwySzm1/8wsyKKs6TUV9s9NRzBAWTLFYDJQCW4A1wDOq+pmwpjJmgK0rrWZ8RiJjRw6e/ooOHQVunV1dmDAKplikAXPxFowWYLwMpkZdYwJwe5QN+6sHXRNUh6ljUkmNd1mxMGEVTLFYD/xFVZcCs4GxwPthTWXMANpRXkdDc/ugbIICiI4S5hZksH6/FQsTPsEsq3q5qh4CUNUm4Osisii8sYwZOB1n5IP1ygJgXkEGK3ZWUF7bNCib0kzk62lQ3gSAjkLhT1VXide48EUzZmCsK6tmYlYS2anxTkfps45CZ01RJlx6aoZ6REReEpEviMh0EckWkTwRuVREfoS3KWrqAOU0Jiza3B427q8ZtE1QHaaMTiEtMebMXV3GhFq3zVCqepOITAM+B3wJGAM0AbuA14D/q6p2r54Z1LYdqeN0q5sFEwfPFB9diYoS5uZn2JWFCZse+yxUdSfwvQHKYsyA65gmY94g7q/oMH9iBn/dcZzDNY3kpic6HccMMd0WCxG5sacDVfVPoY9jzMBaV1rNlNEppCfFOh2l3/zHW1ixMKHW05VFxyjtbGAB8I7v9RLgXcCKhRnUWtrdFB+s4ZbZeU5HCYlJ2clkJseyrqyaz87OdTqOGWJ66rP4IoCI/BmYpqrHfK/HAI8NTDxjwmfLoVqa2zyDvnO7g4h3vMW60mpUdVBNiGgiXzCD8iZ0FAqfCmBymPIYM2DWlVUjAvPyh0axAO8ttMfrmzlQ3eh0FDPEBDMo710ReQN4FlDgFrxrXRgzqK0rrWb62FRGJA6dmfb9+y3yB8E64mbwCHhloar3Ab8GLgBmAI+r6tfCHcyYcGpuc7P5UO2gHrXdlYLMJLJT4my8hQm5YK4sOu58sg5tM2R8ePAkre6h01/RQUSYPzGD90us38KEVk/Tfazx/WwQkXq/R4OI1A9cRGNCb21pNdFRwuwJ6U5HCbn5BRlUnWqhtPK001HMENLTSnkX+36mqGqq3yNFVVOD+XARWSoie0SkRETu7+L9RSLyoYi0i8hnOr3nFpEtvsfy3v6HGdOTdWXVnJczgpT4odNf0eFMv4U1RZkQCuZuKETkAhG5z/c4P8hjovHeYns1MA241Td9iL9DwJ3AM118RJOqzvA9lgXzncYE43RLO1sP1w65JqgOeemJjB0Rb0utmpAKWCxE5BvA03gH52UDT4tIMB3cc4ASVS1T1VbgOeAG/x1U9YCqbgM8vU5uTB8VHzxJu0eHXOd2BxFh3sQM1pd5+y2MCYVgrizuAuaq6gOq+gAwD7g7iONygMN+r4/4tgUrXkSKRWS9iHyqqx1E5B7fPsWVlZW9+GgznK0rrSYmWiiakOZ0lLCZX5BB9elW9laccjqKGSKCKRYCuP1eu33bgjmus96c5uSpahFwG/BzEZl41oepPq6qRapalJWV1YuPNsPZ2tIqZuSOJDE2qJsBB6WPx1tUOZzEDBXBFIv/B2wQkX8WkX/Gu8zqE0EcdwTwn6BmHFAebDBVLff9LMM7F9XMYI81pjt1jW1sP1o36KckD2RcWiK56QnWyW1CJphBeY8CXwRqgJPAF1X150F89kZgkojki0gs3pHfQd3VJCJpIhLne54JXATsDOZYY3qyfn81HoUFQ7Rz29/8ggw27K/B47F+C9N/Qd0NBezHe3a/GhARmRXoAFVtB+4D3sC7YNILqrpDRB4SkWV4P2i2iBwBbgJ+LSI7fIdPBYpFZCveqUV+4ltbw5h+WVtSRUJMNDPzhm5/RYf5EzOobWxj13EbFmX6L2CjrW8J1TuBUj7uc1Dg0kDHqurrwOudtj3g93wj3uapzsetBc4L9PnG9Nb7pdXMzk8n1hXsedLgNc9vXe7pY0c4nMYMdsH08H0WmOi7/dWYQauivpmSE6e46cKzzk+GpDEjEpiQkcj6smq+vLDA6ThmkAvm9Go7MDLcQYwJt471qYd657a/+RO9/RZu67cw/RRMsXgY2Cwib4jI8o5HuIMZE2rvl1QxIiGGaWODmq1mSJg/MZOG5na2H61zOooZ5IJphvod8FPgI2yktRmkVJW1pdXML8ggOmr4zMTacdfXmpIqLsi1BgLTd8FcWVSp6n+o6kpVfa/jEfZkxoTQwepGjtY2cVHh0L9l1l9mchxTx6TyfokNzjP9E0yx2CQiD4vIfBGZ1fEIezJjQuh930jmBYXDp7+iw8WFGRQfOElTqzvwzsZ0I5hmqI6R0/P8tgV166wxkWJtaTWjUuMoGIZLjV48KYvfrN7PBwdqWDzZpsUxfROwWKjqkoEIYky4eDzKutJqLpmcNSxXjpszIZ3Y6CjeL6myYmH6LJhBed/qYnMdsElVt4Q+kjGhtft4AzWnW4dlExRAQmw0F45PY/U+67cwfRdMn0URcC/e6cVzgHuAS4DfiMh3whfNmNBY29FfMQzmg+rOxZMy2XWsnqpTLU5HMYNUMMUiA5ilqt9W1W/jLR5ZwCK804AYE9FW7atiYlYSY0cmOB3FMRf7rqrsrijTV8EUizzAf6qPNmC8qjYBdppiIlpzm5sNZdUsGuZt9efmjGBEQgxrrCnK9FEwd0M9A6wXkVd8r68HnhWRJGzacBPhPthfQ0u7Z9gXi+goYcHEDN4vqUJVh2VHv+mfYNaz+BHeZVRr8XZs36uqD6nqaVX9XLgDGtMf7+2tJNYVxbz84dtf0eHiSZmU1zVTVnXa6ShmEOr2ykJEUlW1XkTS8a5nsd/vvXRVrRmIgMb0x6q9lcyZkE5CbLTTURy3sNB7dbVmXxUTs5IdTmMGm56uLJ7x/dwEFPs9Ol4bE9HKa5vYd+IUiyYPz1tmO8vL8C61usY6uU0fdHtloarX+X7mD1wcY0Jn9b5KgGHfX+Hv4sIs/ry1nDa3h5joob8AlAmdgL8tInKRrzMbEbldRB4VkbzwRzOmf1btrWJUahznjEpxOkrEWDw5k4aWdj48eNLpKGaQCebU4r+BRhG5APgOcBD4fVhTGdNPbo+ypqSKhZOG5xQf3bmoMBNXlLByT6XTUcwgE0yxaFdVBW4AfqGqvwDsVM1EtK1HaqlrarMmqE5S4mOYPSGdd/eccDqKGWSCKRYNIvJd4HbgNRGJBmLCG8uY/lm1txIRWDhM54PqyZIpWew+3kB5bZPTUcwgEkyxuBnvSO27VPU43vmhHglrKmP6adXeSs7PGUFaUqzTUSLOknOyAXjXmqJMLwQzKO+4qj6qqqt9rw+p6lPhj2ZM39Q1trHlcK01QXWjMDuZnJEJrLSmKNMLdu+cGXLWlFThUbtltjsiwpIpWbxfUkVLu62eZ4JjxcIMOW/vqmBkYgwzc0c6HSViLTknm8ZWNxv32y20JjjdFgsRedv386cDF8eY/ml3e1i55wRLzsnGZYPOujV/YgaxrihrijJB6+n/pjEishhYJiIzRWSW/2OgAhrTGx8equVkYxuXTx3ldJSIlhjrYl5BhhULE7Sepih/ALgfGAc82uk9BS4NVyhj+urtXRXERIvNBxWEJedk8cNXd3Kw+jTjM5KcjmMiXLdXFqr6oqpeDfxMVZd0elihMBHprV0VzM3PICXehgIFYrfQmt4Iaj0LEVkmIv/qe1w3EMGM6a39VacprTzN5VOznY4yKEzITKIgM4l3dltTlAksmIkEHwa+gXdVvJ3AN3zbAhKRpSKyR0RKROT+Lt5fJCIfiki7iHym03t3iMg+3+OO4P5zzHD29q4KAC6z/oqgLZmSzbrSak61tDsdxUS4YG4XuRa4QlWfVNUngaW+bT3yTQvyGHA1MA24VUSmddrtEHAnH6+d0XFsOvAgMBeYAzwoImlBZDXD2IqdFUwZnUJueqLTUQaNpeeOptXtsbmiTEDB3lvof8P6iCCPmQOUqGqZqrYCz+GdjPAMVT2gqtsAT6djrwJWqGqNqp4EVuAtUsZ0qa6xjeKDJ7nMmqB6ZVZeGpnJsfx1+3Gno5gI19PdUB0eBjaLyEpAgEXAd4M4Lgc47Pf6CN4rhWB0dWxOkMeaYejdvSdwe9SaoHopOkq4Ytoolm8pp7nNTXyMLT9ruhZMB/ezwDzgT77HfFV9LojP7moRAQ0yV1DHisg9IlIsIsWVlXZHx3C2YmcFmcmxzBhno7Z766rpoznd6mZtqS23aroXVDOUqh5T1eWq+opv5tlgHAFy/V6PA8pDeayqPq6qRapalJVl8wANV21uD+/treTSKdlERdlCR721YGImKXEua4oyPQrnfAgbgUkiki8iscAtwPIgj30DuFJE0nwd21f6thlzlrWl1TQ0t9uo7T6KdUVx6dRs3tp1gnZ35+5DY7zCVixUtR24D+8f+V3AC6q6Q0QeEpFlACIyW0SOADcBvxaRHb5ja4Af4S04G4GHfNuMOctr28pJjnPZLLP9cNX00dScbmXjAZtY0HSt2w5u3+2r3Qrmj7eqvg683mnbA37PN+JtYurq2CeBJwN9hxneWts9vLGjgiumjbLO2X5YPDmLOFcUb+w4zvyJGU7HMRGopyuLTUCx72clsBfY53u+KfzRjAns/dIq6prauO78MU5HGdSS4lwsnJTFmzuOoxrsfShmOOlpbqh8VS3A24x0vapmqmoGcB3eu6KMcdxr246REu/i4kk2cWB/LT13NOV1zXx0tM7pKCYCBdNnMdvXnASAqv4FWBy+SMYEp6XdzRs7jnPltNHEuawJqr8un5pNdJTYXVGmS8EUiyoR+b6ITBCR8SLyPaA63MGMCWTNvioamtutCSpERibGsmBiBn/edsyaosxZgikWtwJZwP8CLwPZvm3GOOq1bcdIjXdxUaE1QYXKDTNyOFTTyIeHap2OYiJMMCO4a1T1G6o60/f4ht3GapzW3OZmxc4Krpo+mliXLZ8aKldNH0WcK4pXthx1OoqJMMFMUZ4lIo+IyOsi8k7HYyDCGdOd1fuqaGhp51prggqplPgYrpg2ij9vO0abDdAzfoI5JXsa2A3kAz8EDuAdKGeMY/68rZwRCTHWBBUGn5qRQ83pVlbvs/nWzMeCKRYZqvoE0Kaq76nql/BOLGiMI5pa3by1s4Kl00cTE21NUKG2aHIWIxNjeHlzsFO5meEgmP/T2nw/j4nItSIyk25GXRszEN7ceZzTrW4+NdNmrQ+HWFcU150/hjd3HrcV9MwZwRSLfxGREcC3gX8Afgt8M6ypjOnBSx8eJWdkAnPze5yRxvTDp2bk0Nzm4c0dNubCePVYLHxLo05S1TpV3a6qS1T1QlUNdvZYY0Kqor6ZNfsq+fTMHJuOPIwuHJ/GuLQEXt5iTVHGq8dioapuYNkAZTEmoJc3H8WjcOMsa4IKJxHhUzNyWLOvkhMNzU7HMREgmGaotSLySxFZKCKzOh5hT2ZMJ6rKSx8eYWbeSAqykp2OM+R9auZYPAqvbj3mdBQTAYJZg3uB7+dDftsUuDT0cYzp3pbDteytOMWPP32e01GGhcLsFC4YN4IXNh7mSxdNQMSa/YazgMVCVZcMRBBjAnnug8MkxkazbMZYp6MMG7fMyeO7f/qIzYdrmZWX5nQc46CAxUJEvtXF5jpgk6puCX0kY852qqWdV7eVc935Y0iOC+aC2ITC9ReM5Ud/3slzHxyyYjHMBdNnUQTcC+T4HvcAlwC/EZHvhC+aMR97dWs5ja1ubpmT53SUYSU5zsX154/l1a3HaGhuC3yAGbKCGsENzFLVb6vqt/EWjyxgEXBnGLMZc8azHxzinFEpzMwd6XSUYeeWObk0tbl5xW6jHdaCKRZ5QKvf6zZgvKo2AS1hSWWMny2Ha9l2pI7PzcuzTlYHzMgdybQxqfx+3UFb52IYC6ZYPAOsF5EHReRB4H3gWRFJAnaGNZ0xwFNrD5Ac5+LGWTbLjBNEhDsWjGdPRQMf7LfVCYarYNaz+BFwN1CLt2P7XlV9SFVPq+rnwh3QDG9Vp1r487Zj/M2sHOvYdtCyC3IYkRDDU+sOOh3FOCSo//tUdROwKcxZjDnL8xsP0+r28Pn5E5yOMqwlxEZz8+xcnlizn+N1zYweEe90JDPAbH5nE7Fa2z08te4AFxdmUphtI7addvvc8XhUeWrdAaejGAdYsTAR69Wt5VTUt/DlhflORzFAXkYiS6eP5g/rD3Lapi4fdqxYmIikqvxmdRmTRyWzeHKW03GMz5cXFlDf3M4fiw87HcUMMCsWJiK9X1LN7uMNfHlhgd0uG0EuHJ/GhePTeOL9/bTbGt3DihULE5F+9V4pWSlx3GDzQEWcuxcWcLimidc+stlohxMrFibifHjoJGtKqrhnYQFxrmin45hOrpw2isLsZP5rZSkejw3SGy6sWJiI859v7yMtMYbb5to8UJEoKkq4b0kheyoaeHNnhdNxzACxYmEiyvajdazcU8mXFxaQZIPwItZ1549hQkYiv1y5z6YAGSbCWixEZKmI7BGREhG5v4v340Tked/7G0Rkgm/7BBFpEpEtvsevwpnTRI6fv7WXlHgXn58/3ukopgeu6Cj+bkkh24/Ws8KuLoaFsBULEYkGHgOuBqYBt4rItE673QWcVNVC4N+Bn/q9V6qqM3yPe8OV00SOTQdP8tauE9y7eCKp8TFOxzEB3Dgzh/zMJP7tzb3WdzEMhPPKYg5QoqplqtoKPAfc0GmfG4Df+Z6/CFwmdp/ksKSqPPLGbjKTY/niRROcjmOC4IqO4ltXTGZPRQPLt9r05UNdOItFDuA/cueIb1uX+6hqO96JCjN87+WLyGYReU9EFoYxp4kAq/dVsb6shvuWFJIYa30Vg8W1541h2phUHl2xl9Z2G3cxlIWzWHR1hdD5WrW7fY4Beao6E/gW8IyIpJ71BSL3iEixiBRXVlb2O7Bxhtuj/Pj1XYxLS+BWuwNqUImKEv5x6Tkcqmm0OaOGuHAWiyOQN/AJAAAQxklEQVRArt/rcUDna9Uz+4iICxgB1Khqi6pWw5kZb0uByZ2/QFUfV9UiVS3KyrIpIQar5zceZvfxBv7pmqk2rmIQumRyFosmZ/Efb+/j5OnWwAeYQSmcxWIjMElE8kUkFrgFWN5pn+XAHb7nnwHeUVUVkSxfBzkiUgBMAsrCmNU4pL65jX97cw9zJqRz9bmjnY5j+kBE+P61Uznd6ubnb+11Oo4Jk7AVC18fxH3AG8Au4AVV3SEiD4nIMt9uTwAZIlKCt7mp4/baRcA2EdmKt+P7XlW1JbqGoJ+v2EdNYyvfv26qzQE1iE0elcJtc/L4w4ZD7CyvdzqOCQMZKgNqioqKtLi42OkYphe2H61j2S/XcMucPH786fOcjmP6qbaxlcv+7T3yMhJ56d4FREVZ8R8MRGSTqhYF2s9GcBtHeDzK91/eTlpiLP/nqilOxzEhMDIxln+6ZiqbD9Xy3EabwnyosWJhHPHUugNsOVzL966dyohEG4A3VNw4K4e5+ek8/JddHK9rdjqOCSErFmbAHaw+zU//uodLzsni0zM7D70xg5mI8JO/OZ82t4fv/mmbzRs1hFixMAPK41H+8cVtuKKFh288zzq1h6D8zCS+c9UUVu6p5MVNR5yOY0LEioUZUI+vLuOD/TX84LppjBmR4HQcEyZ3LpjAnAnpPPTqTg5VNzodx4SAFQszYLYcruVf39jDNeeN5qYLxzkdx4RRVJTw6M0XgMDXn9tMmy3BOuhZsTADoq6pja8/u5lRqfE8/OnzrflpGBiXlshPbjyfLYdr+bc3bbDeYGfFwoSdx6N86/ktlNc28R+3zrC7n4aRa88fw21z8/jVe6X8dftxp+OYfrBiYcLuP98p4e3dJ/jBddO4cHy603HMAHvw+mlckDuSb7+whZITDU7HMX1kxcKE1WvbjvHvb+3lxlk5fMFWvxuW4lzR/Or2WSTERvPl3xVTY5MNDkpWLEzYbDp4km++sIWi8Wn8+NN2m+xwNmZEAo9/oYhjdc3c/VQxzW1upyOZXrJiYcJiX0UDdz9VzNgR8Tz+hSLiY2zq8eFuVl4a/37zDDYdPMk3nttMu90hNahYsTAhd6i6kduf2EB0lPA/X5xDelKs05FMhLjmvDE8eP003thRwXde2mZrdw8itn6lCanDNY3c9tv1tLR7eP6e+UzITHI6kokwX7won4bmdh5dsZfY6Ch+/OnzbIbaQcCKhQmZ/VWnue0362lsdfP7u+ZwzugUpyOZCPW1SwtpaXfz2MpSWt0eHvnMBURbwYhoVixMSGw7UsuX/mcjHoVn757HtLFnLZluzBkiwj9eNYU4VzSPrthLXWMbv7h1Jslx9icpUlmfhem3lbtPcMvj64lzRfPC3863QmGC9vXLJvGjG6bz7t5KbvrVOsprm5yOZLphxcL0mary3++W8qXfbSQ/M4n//eoCCrOTnY5lBpnPz5/Ak3fO5khNIzc89j5bD9c6Hcl0wYqF6ZOTp1u59w+b+Olfd3PteWN48d4FZKfEOx3LDFKLJ2fx0t8tIM4VxWd/vY6nNxy0tTAijBUL02ur9lZy1c9X8c7uE3zvmqn8560zSYi1cRSmfyaPSuHlr17EnPx0vve/27n7qWKqTrU4Hcv4WLEwQWtuc/PDV3fwhSc/IDUhhpe/ehF3LyqwkdkmZDKT4/jdF+fwg+umsWpfFUt/vop3dlc4Hctgd0OZIKgqf91+nH95bRdHa5u4c8EE7r96io3KNmERFSXcdXE+FxVm8PfPbeFL/1PMVdNH8f1rp5Gbnuh0vGFLhkq7YFFRkRYXFzsdY8jZW9HAPy/fwdrSas4ZlcKDy6axYGKm07HMMNHc5uaJNfv55TsluFW5d1EBX7mk0Jo9Q0hENqlqUcD9rFiYrpScOMV/vVvCK1vKSY5z8e0rJ3PbnDxc0dZyaQbesbomHn59N8u3lpOdEsc9iwq4bW4eibHWONJfVixMn+wsr+exd0t4/aNjxLuiuW1uHl9dUmjzO5mIsPFADY++uZd1ZdWkJ8Vy18X5fH7+eFLjbUGtvrJiYYLW3ObmtW3HePaDQxQfPElynIs7FoznSxflk5Ec53Q8Y85SfKCGX64s4d09lSTERLPsgrHcOjePC8aNsBsuesmKhemR26MUH6jhtY+O8cqWcuqa2sjPTOLWObncXJRnS5+aQWH70Tp+v+4gy7eW09TmZtqYVG6clcPV540hZ2SC0/EGBSsW5izNbW4+2F/Dip0V/HXHcSobWohzRXHFtFHcNjeP+QUZdlZmBqWG5jZe2VLOcxsPsf1oPQAX5I7kmnNHc+mUbAqzk+13uxtWLAyt7R52Hatnw/5qVu+rYsP+GlrbPcTHRHHplGyuPncMS6Zk2+RtZkg5UHWa17cf4y8fHeejo3UAjEqNY+GkLBZOyuTC8WnkjEyw4uFjxWKYaXN7KKs8ze7j9ewsr+fDQyfZdqSOlnbvamSTRyVzcWEWCydnMjc/3e4iMcPC0domVu+tZPW+Kt4vraK2sQ2A7JQ4ZuWlMSNvJFNGpzBldCqjUuOGZQGxYjEEqSonG9s4XNPIId9jb0UDe443UFp5ija3998yJlqYPnYEF45PY1ZeGkUT0hiVavM2meHN7VF2HfOeSG06eJIPD53kcM3Hs9yOSIjhnNEpTBmdQmF2MrlpieSmJzAuLXFID0CNiGIhIkuBXwDRwG9V9Sed3o8DngIuBKqBm1X1gO+97wJ3AW7g66r6Rk/fNZiLhapyutVNZUMLVadaqGxoOev50domDtc0crr1kwvd54xM4JzRKd7HKO/Pgqwk4lxD95fbmFA5ebqVPb4Trt3HG9hzvJ69Fac41dL+if2yU+LITU9kdGo8WSlxZKXEkZkc632e7N2WkRxLzCAchxRssQhbW4SIRAOPAVcAR4CNIrJcVXf67XYXcFJVC0XkFuCnwM0iMg24BZgOjAXeEpHJqvrJv5QDRFVpcyvtHg9t7Uqbx0O7W2lze2hze2hp99DU5qap1fdoc3/82u9nQ3Mb9U3t1De3Ud/URn1zu+9n25mrAn9RAhnJcWQmxzEuLYF5BRnkpieSl+4948lNSyTJ+huM6bO0pFjmFWQwryDjzDZVpfJUC4drGjlc03TmSv7wyUZ2Ha9n1b4WGprbu/y8xNhoUuNjSE1w+X7GkBrvIjUhhqQ4F4kx0STERhMfE01ibDQJMdHE+34m+rbHRkcR44oiJkqI8T13RQmx0VGOLj8bzr80c4ASVS0DEJHngBsA/2JxA/DPvucvAr8Ub6PhDcBzqtoC7BeREt/nrQt1yJOnW/nsr9fR7lFa2z3eguBXCNrdSnsIFpWPj4n6xC9PWlIseRlJZ36RRibE+M5W4s6cuaQlxtpSk8YMMBEhOyWe7JR4Lhzf9T7NbZ1aAk61UH2q9czJX8dJYWVDC6WVp6hvauNUS3uXJ4W9ER0lZwpHRxGJiY7ivJwR/OrzF/brswMJZ7HIAQ77vT4CzO1uH1VtF5E6IMO3fX2nY3M6f4GI3APcA5CXl9enkDGuKAqzk4mJjsIV7f1HcEX7Knp0FDHRgisqili/f5gY3/su3/M4VxQJsS4SYrxnCAmxn3wd53L2jMAYE1rxMdHkpif2emLDNreHZl/LQ3Orh8a29jMtD81tbhpb3b4TVd8Ja7vveUerhu8k1v+Ets2tjEsL/5iScBaLrv46di6r3e0TzLGo6uPA4+Dts+htQIDkOBf/fXt4K7IxxgBnTkJTBuH0JOHsjTkC5Pq9HgeUd7ePiLiAEUBNkMcaY4wZIOEsFhuBSSKSLyKxeDusl3faZzlwh+/5Z4B31Ht71nLgFhGJE5F8YBLwQRizGmOM6UHYmqF8fRD3AW/gvXX2SVXdISIPAcWquhx4Avi9rwO7Bm9BwbffC3g7w9uBrzp1J5QxxhgblGeMMcNasOMsBt8IEmOMMQPOioUxxpiArFgYY4wJyIqFMcaYgIZMB7eIVAIH+/ERmUBViOKEkuXqvUjNZrl6J1JzQeRm60uu8aqaFWinIVMs+ktEioO5I2CgWa7ei9Rslqt3IjUXRG62cOayZihjjDEBWbEwxhgTkBWLjz3udIBuWK7ei9Rslqt3IjUXRG62sOWyPgtjjDEB2ZWFMcaYgKxY+IjIDBFZLyJbRKRYROY4ncmfiHxNRPaIyA4R+ZnTefyJyD+IiIpIptNZAETkERHZLSLbROR/RWSkw3mW+v7tSkTkfiez+BORXBFZKSK7fL9X33A6kz8RiRaRzSLyZ6ezdBCRkSLyou/3a5eIzHc6E4CIfNP3b7hdRJ4VkfhQf4cVi4/9DPihqs4AHvC9jggisgTvUrPnq+p04F8djnSGiOTiXWf9kNNZ/KwAzlXV84G9wHedCuK3Fv3VwDTgVt8a85GgHfi2qk4F5gFfjaBsAN8AdjkdopNfAH9V1SnABURAPhHJAb4OFKnquXhn+b4l1N9jxeJjCqT6no8gshZb+grwE9+a5KjqCYfz+Pt34Dt0sZKhU1T1TVVt971cj3fxLKecWYteVVuBjrXoHaeqx1T1Q9/zBrx/+M5avtgJIjIOuBb4rdNZOohIKrAI79IKqGqrqtY6m+oMF5DgW0QukTD8/bJi8bG/Bx4RkcN4z9wdOxvtwmRgoYhsEJH3RGS204EARGQZcFRVtzqdpQdfAv7i4Pd3tRZ9RPxB9iciE4CZwAZnk5zxc7wnIR6ng/gpACqB/+drHvutiCQ5HUpVj+L9m3UIOAbUqeqbof6ecK7BHXFE5C1gdBdvfQ+4DPimqr4kIp/Fe/ZweYRkcwFpeJsKZgMviEiBDsCtbAFy/RNwZbgzdKWnXKr6im+f7+Ftanl6ILN1EtR68k4SkWTgJeDvVbU+AvJcB5xQ1U0iconTefy4gFnA11R1g4j8Argf+IGToUQkDe/Vaj5QC/xRRG5X1T+E8nuGVbFQ1W7/+IvIU3jbSAH+yABf/gbI9hXgT77i8IGIePDOAVPpVC4ROQ/vL+dWEQFvU8+HIjJHVY87lcsv3x3AdcBlA1FUexDR68mLSAzeQvG0qv7J6Tw+FwHLROQaIB5IFZE/qOrtDuc6AhxR1Y6rrxfxFgunXQ7sV9VKABH5E7AACGmxsGaoj5UDi33PLwX2OZils5fxZkJEJgOxODyJmap+pKrZqjpBVSfg/R9p1kAUikBEZCnwf4BlqtrocJxg1qJ3hHir/BPALlV91Ok8HVT1u6o6zvd7dQvwTgQUCny/24dF5BzfpsvwLv3stEPAPBFJ9P2bXkYYOt6H1ZVFAHcDv/B1EDUD9zicx9+TwJMish1oBe5w+Gw50v0SiANW+K561qvqvU4E6W4teieydOEi4PPARyKyxbftn1T1dQczRbqvAU/7Cn8Z8EWH8+BrEnsR+BBvs+tmwjCS20ZwG2OMCciaoYwxxgRkxcIYY0xAViyMMcYEZMXCGGNMQFYsjDHGBGTFwhhjTEBWLIwxxgRkxcKYMBGR2b41NeJFJMm33sC5Tucypi9sUJ4xYSQi/4J3fqMEvPMKPexwJGP6xIqFMWHkmxZiI94pZBaoqtvhSMb0iTVDGRNe6UAykIL3CsOYQcmuLIwJIxFZjnd1vHxgjKre53AkY/rEZp01JkxE5AtAu6o+41uLe62IXKqq7zidzZjesisLY4wxAVmfhTHGmICsWBhjjAnIioUxxpiArFgYY4wJyIqFMcaYgKxYGGOMCciKhTHGmICsWBhjjAno/wPDYQvJgedJHgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y.backward(torch.ones_like(x),retain_graph=True)\n",
    "xyplot(x,x.grad,'grad of sigmoid')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tanh Function\n",
    "\n",
    "Like the sigmoid function, the tanh (Hyperbolic Tangent)\n",
    "function also squashes its inputs,\n",
    "transforms them into elements on the interval between -1 and 1:\n",
    "\n",
    "$$\\text{tanh}(x) = \\frac{1 - \\exp(-2x)}{1 + \\exp(-2x)}.$$\n",
    "\n",
    "We plot the tanh function blow. Note that as the input nears 0, the tanh function approaches a linear transformation. Although the shape of the function is similar to the sigmoid function, the tanh function exhibits point symmetry about the origin of the coordinate system.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAEKCAYAAAA1qaOTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xl8XPV97vHPI8mS993YxjvEbMZgg3DANAuLE0N7IW020iZ1mqROc0O2Nm0g3Ca5pGlIc3tp2qZNXEJLlkIKSYpvgBDWpImBWjbGK44XjCXLYHmRvMjW+r1/zHEyFpIt2zM6M5rn/XrNa+ac8ztnHtkjfeec3znnp4jAzMzsdJWlHcDMzPoHFxQzM8sJFxQzM8sJFxQzM8sJFxQzM8sJFxQzM8sJFxQzM8sJFxQzM8sJFxQzM8uJirQD9KWxY8fG9OnT045hZlZUVqxYsTsixp2oXUkVlOnTp1NTU5N2DDOzoiLp5d608yEvMzPLCRcUMzPLCRcUMzPLCRcUMzPLCRcUMzPLiVQLiqS7Je2StLaH5ZL095I2S1ot6ZKsZYskbUoei/outZmZdSftPZR/AxYeZ/l1wMzksRj4ZwBJo4HPA68H5gGflzQqr0nNzOy4Ur0OJSJ+Lmn6cZrcCHw7MuMUPytppKSJwJuBxyJiL4Ckx8gUpnvzm9jMTkdE0NLeSXNrB82t7Rxu7aC5tYMjbR10dAZtnUFHZyftHUFHZ9DeGbR3me7oDDojiIBItpnZdvIeZC/LTB+zPGtdstpkz+ujf4y+fDcWzZ/OmKFVeX2PQr+wcRJQmzVdl8zraf5rSFpMZu+GqVOn5ielWQmLCBoOtrB9TzPb9jTz6v4j7DnYyp5DLew52Mrugy3sPdTKoZZ2Drd10Nm3f0cLmtR373XDnEklX1C6++eO48x/7cyIJcASgOrqan+UzU5DZ2ewueEgL9Q2smZHEy/UNbHp1QM0t3Yc025IZTljhlYxZmglk0cN4qLJIxhaNYDBleUMqixncPIYVFnB4AGZeRVloqJclJeV/fp1Rdmx0+VlolyZZwAhUOYPswDp6Pyj83TMH+3sed2278u/8P1QoReUOmBK1vRkoD6Z/+Yu85/us1RmJeRwawe/3LybJ158lcc37KLhQAuQKRoXThrBu6qnMH3MYKaNHcK00YM5c+QgBg4oTzm1paHQC8pS4GZJ95HpgG+KiJ2SHgX+Oqsj/i3ArWmFNOuPVm7fx/ee3c5Da+o50tbJ0KoK3nTOON587jjmTh3FWWOHUFbmb/T2G6kWFEn3ktnTGCupjsyZWwMAIuIbwMPA9cBmoBn4o2TZXklfBJYnm7r9aAe9mZ2eX2zazd8/sYn/3raXIZXl/O7cyVw/ewKvnzGGyoq0Twy1Qqbo4zMN0lRdXR2+27BZ97btPsTtP17Pky/uYsLwgXz4TWfxzuopDK0q9AMZlm+SVkRE9Yna+ZNiVuIignuWbeOvH3mRAWXis9efx6L506mqcD+InRwXFLMStv9IG5+8bxVPvriLq887gzt+bzZnDB+YdiwrUi4oZiWqdm8zH7xnOVsbDnH7jbN43+XTfNqsnRYXFLMS9NLuQ7z7m89wpK2Db39wHvPPHpt2JOsHXFDMSsz2Pc38/r88S0dn8MBH5nPO+GFpR7J+wgXFrITsO9TK++5+jsNtHdz7x5e7mFhO+aRysxLR1tHJR/99JTsbj/CtRZdx/sThaUeyfsZ7KGYl4iuPvMiyLXv46jsu4tJpHu3Bcs97KGYl4Jebd3PXL17ifZdP453VU068gtkpcEEx6+eaDrfx6ftf4KxxQ/js9eenHcf6MR/yMuvnvvzwBnYdaOGHH5nPoEpf/W754z0Us35sdV0j36+p5QNXTufiKSPTjmP9nAuKWT/V2Rl8Yek6xgyp4uPXzEw7jpUAFxSzfmrpC/Ws3N7IZxaey7CBA9KOYyXABcWsH2rv6OTOx3/FrDOH8/ZLJqcdx0pEqgVF0kJJGyVtlnRLN8vvlLQqefxKUmPWso6sZUv7NrlZYXtwVT0v72nmE9fM9KiK1mdSO8tLUjnwdWABmTHil0taGhHrj7aJiE9ltf8YMDdrE4cjYk5f5TUrFu0dnfzjU5u5YOJwFlwwPu04VkLS3EOZB2yOiK0R0QrcB9x4nPbvAe7tk2RmReyhNTt5afchPnHtTN+O3vpUmgVlElCbNV2XzHsNSdOAGcCTWbMHSqqR9Kykt+UvplnxiAi+9YuXOHvcEBac770T61tpFpTuvjr1NMD9TcADEdGRNW9qMsbx7wN/J+nsbt9EWpwUnpqGhobTS2xW4J6vbWR1XRPvnz/dfSfW59IsKHVA9k2FJgP1PbS9iS6HuyKiPnneCjzNsf0r2e2WRER1RFSPGzfudDObFbR7lm1jWFUFv+czuywFaRaU5cBMSTMkVZIpGq85W0vSucAo4JmseaMkVSWvxwJXAuu7rmtWSnbtP8JDq3fyzuopDKnyXZWs76X2qYuIdkk3A48C5cDdEbFO0u1ATUQcLS7vAe6LiOzDYecD35TUSaYo3pF9dphZKbp/RR3tncH7rpiWdhQrUal+jYmIh4GHu8z7XJfpL3Sz3jJgdl7DmRWRiOAHK+uYN300M8YOSTuOlShfKW/WD6yqbWRrwyHefmm3J0qa9QkXFLN+4IcrdzBwQBnXz56YdhQrYS4oZkWupb2DpS/U89ZZE3wTSEuVC4pZkfvZxgaaDrf5VGFLnQuKWZF7ZO0rjBw8gPlnj0k7ipU4FxSzItba3snjG15lwfnjGVDuX2dLlz+BZkXsl1t2c+BIO9fNnpB2FDMXFLNi9sianQyrquDK141NO4qZC4pZsWrv6OSx9a9y9flnUFVRnnYcMxcUs2JV8/I+9jW38dZZPtxlhcEFxaxIPbVxFxVl4g0zfbjLCoMLilmR+tnGBi6bPtoXM1rBcEExK0L1jYd58ZUDXHWex/ixwuGCYlaEnt6YGX30zeeekXISs99wQTErQk9v3MWkkYOYecbQtKOY/ZoLilmRaW3v5Jebd/Pmc8chedx4KxypFhRJCyVtlLRZ0i3dLH+/pAZJq5LHh7KWLZK0KXks6tvkZulZVdvIodYO3niO+0+ssKQ2YqOkcuDrwAKgDlguaWk3Q/l+PyJu7rLuaODzQDUQwIpk3X19EN0sVcu27KZMcPlZvhmkFZY091DmAZsjYmtEtAL3ATf2ct23Ao9FxN6kiDwGLMxTTrOCsmzLHi6cNIIRg3y6sBWWNAvKJKA2a7oumdfV2yWtlvSApCknua5Zv9Lc2s7z2/dxhW9VbwUozYLSXW9idJn+f8D0iLgIeBy45yTWzTSUFkuqkVTT0NBwymHNCkHNtn20dQTzz/bV8VZ40iwodcCUrOnJQH12g4jYExEtyeS/AJf2dt2sbSyJiOqIqB43zp2YVtyWbdnDgHJx2fRRaUcxe400C8pyYKakGZIqgZuApdkNJE3MmrwB2JC8fhR4i6RRkkYBb0nmmfVrz2zZzdwpoxhcmdr5NGY9Su1TGRHtkm4mUwjKgbsjYp2k24GaiFgKfFzSDUA7sBd4f7LuXklfJFOUAG6PiL19/kOY9aEDR9pYs6OJm6+emXYUs26l+jUnIh4GHu4y73NZr28Fbu1h3buBu/Ma0KyArHh5H50B86aPTjuKWbd8pbxZkajZto/yMjF36si0o5h1ywXFrEgs37aXWWcOZ0iV+0+sMLmgmBWBlvYOVtU2Uj3Nh7uscLmgmBWBtTv209LeybwZPl3YCpcLilkRqNmWOYnxUu+hWAFzQTErAsu37WXG2CGMG1aVdhSzHrmgmBW4iGDFy/uonubDXVbYXFDMCtz2vc3sa25j7lQXFCtsLihmBW5VbSMAc6b4+hMrbC4oZgVuVW0jAweUcc54jx9vhc0FxazAvVDbyOxJI6go96+rFTZ/Qs0KWFtHJ2vr93PxZB/ussLngmJWwF7ceYDW9k7m+P5dVgRcUMwK2Kq6TIe891CsGLigmBWwF2obGTOkksmjBqUdxeyEXFDMCtgLtY3MmTISSWlHMTuhVAuKpIWSNkraLOmWbpb/qaT1klZLekLStKxlHZJWJY+lXdc1K3YHjrSxueEgF/v6EysSqQ2sIKkc+DqwAKgDlktaGhHrs5o9D1RHRLOkjwB/A7w7WXY4Iub0aWizPrSmrokIXFCsaKS5hzIP2BwRWyOiFbgPuDG7QUQ8FRHNyeSzwOQ+zmiWmt90yI9IOYlZ76RZUCYBtVnTdcm8nnwQeCRreqCkGknPSnpbTytJWpy0q2loaDi9xGZ9aNX2RmaMHcLIwZVpRzHrlTTHEu2ulzG6bSi9F6gG3pQ1e2pE1Es6C3hS0pqI2PKaDUYsAZYAVFdXd7t9s0L0Ql0jV5w1Ju0YZr2W5h5KHTAla3oyUN+1kaRrgduAGyKi5ej8iKhPnrcCTwNz8xnWrC+90nSEV/e3uP/EikqaBWU5MFPSDEmVwE3AMWdrSZoLfJNMMdmVNX+UpKrk9VjgSiC7M9+sqPkOw1aMUjvkFRHtkm4GHgXKgbsjYp2k24GaiFgKfBUYCtyfnIe/PSJuAM4Hvimpk0xRvKPL2WFmRW1VbSMDysX5E4enHcWs19LsQyEiHgYe7jLvc1mvr+1hvWXA7PymM0vP6rpGzpswnIEDytOOYtZrvlLerMBEBGt3NDHbpwtbkXFBMSswtXsPs/9IOxee6YJixcUFxazArK1vAmD2JBcUKy4uKGYFZu2OJirKxDkTPOSvFZdedcpLqgbeAJwJHAbWAo9HxN48ZjMrSWvr93PO+GFUVbhD3orLcfdQJL1f0krgVmAQsBHYBfwW8JikeyRNzX9Ms9IQEazb0cSFk3y6sBWfE+2hDAGujIjD3S2UNAeYCWzPdTCzUvTK/iPsOdTKhe4/sSJ03IISEV/vaZmkyohYlftIZqVr7Y79AMzyGV5WhHrVKS/paUnTs6bnkbl1ipnl0NodTZQJzp84LO0oZiett1fKfxn4iaS/J3OL+euAP8pbKrMSta6+ibPHDWVwZao3sTA7Jb361EbEo5L+BHgM2A3MjYhX8prMrASt2dHE/LPHph3D7JT09pDXXwL/ALwR+ALwtKTfzmMus5Kz60DmlvWzzvQZXlacertfPRaYl5zt9YyknwB3AQ/lLZlZiVlXn+mQ9xleVqx6e8jrE12mXwYW5CWRWYlatyNzy5ULvIdiRepEFzYukdTtbeIlDZH0AUl/kJ9oZqVl7Y79TB8zmOEDB6QdxeyUnKgP5Z+Av5S0QdL9kv5J0t2S/gtYBgwDHjjVN5e0UNJGSZsl3dLN8ipJ30+WP9fl1OVbk/kbJb31VDOYFYq19U0+3GVF7UQXNq4C3iVpKFANTCRzL68NEbHxdN5YUjnwdTKHzuqA5ZKWdhl58YPAvoh4naSbgK8A75Z0AZkhg2eRub/Y45LOiYiO08lklpbG5lbq9h3mvZdPSzuK2SnrbR/KQeDpHL/3PGBzRGwFkHQfcCPHjg1/I5mzyiCzJ/SPyowFfCNwX0S0AC9J2pxs75kcZzTrE7/ukPcV8lbEenva8JWSHpP0K0lbJb0kaetpvvckoDZrui6Z122biGgHmoAxvVzXrGisTTrkfcqwFbPenjb8LeBTwAogV4eV1M286GWb3qyb2YC0GFgMMHWqb4xshWlt/X4mjRzEqCGVaUcxO2W9HWCrKSIeiYhdEbHn6OM037sOmJI1PRmo76mNpApgBLC3l+sCEBFLIqI6IqrHjRt3mpHN8sO3rLf+4ESnDV8i6RLgKUlflXTF0XnJ/NOxHJgpaYakSjKd7Eu7tFkKLEpevwN4MiIimX9TchbYDDK30P/v08xjlooDR9rYuvuQ+0+s6J3okNffdpmuznodwNWn+sYR0S7pZuBRoBy4OyLWSbodqImIpWQOtX0n6XTfS6bokLT7DzId+O3AR32GlxWr9b5C3vqJE502fFU+3zwiHgYe7jLvc1mvjwDv7GHdLwFfymc+s76wNikos3zIy4pcb8eUrwLeDkzPXicibs9PLLPSsW5HE2cMq+KMYQPTjmJ2Wnp7lteDZE7ZXQG05C+OWenxFfLWX/S2oEyOiIV5TWJWgg63drB510EWzpqQdhSz09bb04aX9XSTSDM7dRte2U9nwCzvoVg/0Ns9lN8C3i/pJTKHvARERFyUt2RmJeDoLetnu6BYP9DbgnJdXlOYlajVdU2MHlLJxBHukLfi19ubQ74MIOkMwJ98sxxZs6OJ2ZNGkLnnqVlx6+3NIW+QtAl4CfgZsA14JI+5zPq9w60dbNp1kIsm+3CX9Q+97ZT/InA58KuImAFcA/wyb6nMSsD6nfvp6Az3n1i/0duC0pbcDLJMUllEPAXMyWMus37v6C3rZ3sPxfqJ3nbKNyajNv4c+J6kXUBb/mKZ9X+r65oYO7SKCcPdLWn9Q28LygtAM5kxUf6AzG3kh+YrlFkpWLOjkYsmu0Pe+o/eFpSrIqIT6ATuAZC0Om+pzPq55tZ2Nu86yHUXTkw7ilnOHLegSPoI8D+Bs7sUkGG4U97slK2vz1wh7w55609OtIfy72ROD/4ycEvW/AMRsTdvqcz6udV17pC3/udE46E0kbnL8Hv6Jo5ZaVizo4nxw6sY7w5560d6e9pwTkkaLekxSZuS51HdtJkj6RlJ6yStlvTurGX/JuklSauSh09htqKSuUJ+ZNoxzHIqlYJC5vDZExExE3iCYw+nHdUM/GFEzAIWAn8nKfs38M8jYk7yWJX/yGa5cbClnS0NB91/Yv1OWgXlRpKzxZLnt3VtEBG/iohNyet6YBcwrs8SmuXJuh1NROBbrli/k1ZBGR8ROwGS5zOO11jSPKAS2JI1+0vJobA7kyGKzYrCmuQKeY/SaP1Nb69DOWmSHge6G4butpPczkTgO8Ci5FoYgFuBV8gUmSXAZ4Bux7eXtBhYDDB16tSTeWuzvFizo4kzRwxk3DB/D7L+JW8FJSKu7WmZpFclTYyInUnB2NVDu+HAQ8D/iohns7a9M3nZIulfgU8fJ8cSMkWH6urqOPmfxCy31tQ1+XRh65fSOuS1FFiUvF4EPNi1gaRK4EfAtyPi/i7LJibPItP/sjavac1yZP+RNrbuPuQOeeuX0ioodwALkjFWFiTTSKqWdFfS5l3AG8kMPdz19ODvSVoDrAHGAn/Vt/HNTs26HfsBmD3Zpwxb/5O3Q17Hk9wK/5pu5tcAH0pefxf4bg/rX53XgGZ5sqq2EYCLvIdi/VBaeyhmJen57fs4a+wQRg2pTDuKWc65oJj1kYjg+dpG5kz14S7rn1xQzPrIjsbDNBxoYe4UFxTrn1xQzPrI89sz/Sdzp77m1nVm/YILilkfeX57IwMHlHHuhGFpRzHLCxcUsz7yfO0+Lpo0kgHl/rWz/smfbLM+0NLewbod+5nrDnnrx1xQzPrAuvr9tHZ0Mscd8taPuaCY9YGabZkRsy+d7g55679cUMz6wPJt+5g+ZjBnDPOQv9Z/uaCY5VlnZ1CzbS/V00enHcUsr1xQzPJs6+6D7GtuY54LivVzLihmebZ82z4Aqt1/Yv2cC4pZni1/aS9jh1YyY+yQtKOY5ZULilmeLX95L9XTRpMZD86s/3JBMcujV5qOULv3sA93WUlIpaBIGi3pMUmbkuduf9skdWSN1rg0a/4MSc8l638/GS7YrOA8s3U3AJefNSblJGb5l9Yeyi3AExExE3gime7O4YiYkzxuyJr/FeDOZP19wAfzG9fs1CzbvIcRgwZwwcThaUcxy7u0CsqNwD3J63uAt/V2RWUORF8NPHAq65v1lYhg2ZY9XHHWGMrK3H9i/V9aBWV8ROwESJ7P6KHdQEk1kp6VdLRojAEaI6I9ma4DJvX0RpIWJ9uoaWhoyFV+sxOq3XuYHY2Hmf86H+6y0lCRrw1LehyY0M2i205iM1Mjol7SWcCTktYA+7tpFz1tICKWAEsAqqure2xnlmu/3JLpP5l/tguKlYa8FZSIuLanZZJelTQxInZKmgjs6mEb9cnzVklPA3OBHwAjJVUkeymTgfqc/wBmp2nZlj2cMayKs8cNTTuKWZ9I65DXUmBR8noR8GDXBpJGSapKXo8FrgTWR0QATwHvON76ZmmKCJ7Zspv5Z4/x9SdWMtIqKHcACyRtAhYk00iqlnRX0uZ8oEbSC2QKyB0RsT5Z9hngTyVtJtOn8q0+TW92Aht2HmD3wVbmv25s2lHM+kzeDnkdT0TsAa7pZn4N8KHk9TJgdg/rbwXm5TOj2el4amPmKO6bzxmXchKzvuMr5c3y4OmNu5h15nDOGO7xT6x0uKCY5VhTcxsrtzdy1bk9nQ1v1j+5oJjl2H9tbqCjM7jqPB/ustLigmKWY09vbGDEoAHMmeIbQlppcUExy6HOzuDpjQ288ZxxlPt2K1ZiXFDMcmjl9n3sPtjCtee7/8RKjwuKWQ49svYVKsvLuPo8FxQrPS4oZjkSEfxk7Su8YeZYhg0ckHYcsz7ngmKWI6vrmtjReJiFF3Z3T1Sz/s8FxSxHHln7ChVlYsEF49OOYpYKFxSzHIgIHlm7kyvOHsPIwR6R2kqTC4pZDqzcvo+X9zRzw8Vnph3FLDUuKGY58MCKHQwaUM51syemHcUsNS4oZqfpSFsHP15dz8ILJzC0KpUbeJsVBBcUs9P02PpXOXCknbdfMjntKGapckExO00PrKhj4oiBXOGx463EpVJQJI2W9JikTcnza+6iJ+kqSauyHkckvS1Z9m+SXspaNqfvfwoz2Lb7ED/f1MA7q6f43l1W8tLaQ7kFeCIiZgJPJNPHiIinImJORMwBrgaagZ9mNfnzo8sjYlWfpDbr4tvPvEy5xHtfPzXtKGapS6ug3Ajck7y+B3jbCdq/A3gkIprzmsrsJBxqaef+mlqunz3RIzOakV5BGR8ROwGS5xPdSe8m4N4u874kabWkOyVV9bSipMWSaiTVNDQ0nF5qsyw/XFnHgZZ2Fs2fnnYUs4KQt4Ii6XFJa7t53HiS25kIzAYezZp9K3AecBkwGvhMT+tHxJKIqI6I6nHjPIKe5UZ7Ryd3/eIlLpo8gkumjkw7jllByNtJ8xFxbU/LJL0qaWJE7EwKxq7jbOpdwI8ioi1r2zuTly2S/hX4dE5Cm/XSf66q5+U9zSx536VI7ow3g/QOeS0FFiWvFwEPHqfte+hyuCspQijzm/w2YG0eMpp1q72jk394chMXTBzuG0GaZUmroNwBLJC0CViQTCOpWtJdRxtJmg5MAX7WZf3vSVoDrAHGAn/VB5nNAHgw2Tv55LUzvXdiliWV+0RExB7gmm7m1wAfypreBkzqpt3V+cxn1pPm1nb+9qcbuXCS907MuvKV8mYn4RtPb6G+6Qif/x+zvHdi1oULilkv1e5t5ps/38oNF5/JZdNHpx3HrOC4oJj1QkRw23+upUzi1uvPSzuOWUFyQTHrhe8+t52f/6qBz15/HhNHDEo7jllBckExO4GtDQf564c28IaZY3nv5dPSjmNWsFxQzI7jwJE2PvydFVQNKOOr77jYHfFmx+Hh5cx60NkZfOr7q9i6+xDf+cA8JozwDSDNjsd7KGbdiAg+v3Qdj2/Yxed+5wLmv25s2pHMCp4LilkXEcHtP17Pd559mQ+/8Sz+8Ar3m5j1hg95mWVpbe/ksz9awwMr6vjAlTO45brz3G9i1ksuKGaJhgMtfOzelTy7dS8fv2Ymn/K9usxOiguKGfDEhlf5iwdWc7ClnTvffTG/O3dy2pHMio4LipW0un3NfOmhDTyy9hXOmzCMexdfzjnjh6Udy6wouaBYSdrReJhvPL2F79fUUib4swXnsPhNZ1FVUZ52NLOi5YJiJeNwawc/39TA/TV1PPniq5SXiXdcOpmbr57JpJG+nYrZ6UqloEh6J/AF4HxgXjIOSnftFgJfA8qBuyLi6EBcM4D7yIwnvxJ4X0S09kF0KyIRQe3ew/xi826e2PAqv9i8m5b2TsYOreTDbzqb914+zYXELIfS2kNZC/we8M2eGkgqB75OZkTHOmC5pKURsR74CnBnRNwn6RvAB4F/zn9sK0QRwb7mNrbtOcT2Pc1s2nWA1XVNrNnRRGNzGwCTRw3iPfOmsuCC8cybMZoB5b4EyyzX0hqxcQNwolMy5wGbI2Jr0vY+4EZJG4Crgd9P2t1DZm/HBaVIRATtnUF7R9DW2Ul7R9De0UlbZ/LcERxp6+BgSzsHj7RzqLX9N69b2jnQ0s7ug600HDhCw4EWdu1v4UBL+6+3X14mzh0/jIWzJnDR5JFcOm0U54wf6lOAzfKskPtQJgG1WdN1wOuBMUBjRLRnzX/NMMG5dNuP1vDcS3uBzB/DoyK7UXT78pj2r12WPT+6n3/s6t1ut6dtHne7Pa7Tm/a9/Jm6ZGzvCNo7MwXjVEkwtLKCMUMrGTesinMnDOO3XjeWqWOGMG30YKaNGcyU0YMZOMCd62Z9LW8FRdLjwIRuFt0WEQ/2ZhPdzIvjzO8px2JgMcDUqVN78bavdebIQZybfSqpun15zDfgY+d3ydSLdY59j6w2Pb539+1fs6yHNznZ7fb2Z8o2oFxUlJcxoCzzXFEuKsvLqEimB5SLirLM/EEDyhlaVcGQqgqGDqxgaFXmMWhAOWVl3tMwK0R5KygRce1pbqIOmJI1PRmoB3YDIyVVJHspR+f3lGMJsASgurr6lL4af/Sq153KamZmJaWQeyaXAzMlzZBUCdwELI3McZSngHck7RYBvdnjMTOzPEqloEj6XUl1wBXAQ5IeTeafKelhgGTv42bgUWAD8B8RsS7ZxGeAP5W0mUyfyrf6+mcwM7NjqWsHa39WXV0dNTXdXvJiZmY9kLQiIqpP1K6QD3mZmVkRcUExM7OccEExM7OccEExM7OccEExM7OcKKmzvCQ1AC+f4upjyVxUWWgKNRcUbjbnOjmFmgsKN1t/yzUtIsadqFFJFZTTIammN6fN9bVCzQWFm825Tk6h5oLCzVaquXzIy8zMcsIFxczMcsIFpfeWpB2gB4WaCwo3m3OdnELNBYWbrSRzuQ/FzMxywnsoZmaWEy4oJ0HSHEnPSlolqUbSvLQzHSXpY5I2Slon6W/SzpP+2OAMAAAEoklEQVRN0qclhaSxaWc5StJXJb0oabWkH0kamXKehcn/32ZJt6SZ5ShJUyQ9JWlD8rn6RNqZskkql/S8pB+nnSWbpJGSHkg+XxskXZF2JgBJn0r+H9dKulfSwFy/hwvKyfkb4H9HxBzgc8l06iRdBdwIXBQRs4D/k3KkX5M0BVgAbE87SxePARdGxEXAr4Bb0woiqRz4OnAdcAHwHkkXpJUnSzvwZxFxPnA58NECyXXUJ8gMbVFovgb8JCLOAy6mADJKmgR8HKiOiAuBcjJjTOWUC8rJCWB48noExxkpso99BLgjIloAImJXynmy3Qn8BccZpjkNEfHTZMwdgGfJjPyZlnnA5ojYGhGtwH1kviCkKiJ2RsTK5PUBMn8YJ6WbKkPSZOC3gbvSzpJN0nDgjSRjNEVEa0Q0ppvq1yqAQZIqgMHk4e+XC8rJ+STwVUm1ZPYCUvtW28U5wBskPSfpZ5IuSzsQgKQbgB0R8ULaWU7gA8AjKb7/JKA2a7qOAvnDfZSk6cBc4Ll0k/za35H5otKZdpAuzgIagH9NDsfdJWlI2qEiYgeZv1nbgZ1AU0T8NNfvk7cx5YuVpMeBCd0sug24BvhURPxA0rvIfAu5tgByVQCjyByWuAz4D0lnRR+cwneCXJ8F3pLvDD05XraIeDBpcxuZQzvf68tsXaibeQWzRydpKPAD4JMRsb8A8vwOsCsiVkh6c9p5uqgALgE+FhHPSfoacAvwl2mGkjSKzF7vDKARuF/SeyPiu7l8HxeULiKixwIh6dtkjtsC3E8f7m6fINdHgB8mBeS/JXWSuWdPQ1q5JM0m8+F9QRJkDimtlDQvIl7Jd67jZTtK0iLgd4Br+qL4HkcdMCVrejIFcjhV0gAyxeR7EfHDtPMkrgRukHQ9MBAYLum7EfHelHNB5v+yLiKO7sk9QKagpO1a4KWIaACQ9ENgPpDTguJDXienHnhT8vpqYFOKWbL9J5k8SDoHqCTlG9NFxJqIOCMipkfEdDK/aJf0VTE5EUkLgc8AN0REc8pxlgMzJc2QVEmms3RpyplQ5pvAt4ANEfF/085zVETcGhGTk8/VTcCTBVJMSD7ftZLOTWZdA6xPMdJR24HLJQ1O/l+vIQ8nC3gP5eT8MfC1pFPrCLA45TxH3Q3cLWkt0AosSvkbdzH4R6AKeCzZg3o2Iv4kjSAR0S7pZuBRMmff3B0R69LI0sWVwPuANZJWJfM+GxEPp5ipGHwM+F7y5WAr8Ecp5yE5/PYAsJLMId7nycNV875S3szMcsKHvMzMLCdcUMzMLCdcUMzMLCdcUMzMLCdcUMzMLCdcUMzMLCdcUMzMLCdcUMxSJOmyZEyWgZKGJONVXJh2LrNT4QsbzVIm6a/I3JNqEJn7QH055Uhmp8QFxSxlyS06lpO5nc/8iOhIOZLZKfEhL7P0jQaGAsPI7KmYFSXvoZilTNJSMqM0zgAmRsTNKUcyOyW+27BZiiT9IdAeEf+ejC2/TNLVEfFk2tnMTpb3UMzMLCfch2JmZjnhgmJmZjnhgmJmZjnhgmJmZjnhgmJmZjnhgmJmZjnhgmJmZjnhgmJmZjnx/wFn6/J6kxmUSQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x=Variable(torch.arange(-8.0,8.0,0.1,dtype=torch.float32).reshape(int(16/0.1),1),requires_grad=True)\n",
    "y=torch.tanh(x)\n",
    "xyplot(x,y,\"tanh\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The derivative of the Tanh function is:\n",
    "\n",
    "$$\\frac{d}{dx} \\mathrm{tanh}(x) = 1 - \\mathrm{tanh}^2(x).$$\n",
    "\n",
    "The derivative of tanh function is plotted below.\n",
    "As the input nears 0,\n",
    "the derivative of the tanh function approaches a maximum of 1.\n",
    "And as we saw with the sigmoid function,\n",
    "as the input moves away from 0 in either direction,\n",
    "the derivative of the tanh function approaches 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xl4nHd57vHvM1qszdbuVbLlRYnjJM7mhCUhLCEkJCGhhdIECNCFnJQGKJTSUFpKoedAw+lCIfQQIFCWhpKwudQhFAhNaDbL2RzbMVYc25I3yZIsW/syz/ljZpSJomUkzzvvjHR/rstY886rmcdE0q3fbu6OiIgIQCTsAkREJHsoFEREZIxCQURExigURERkjEJBRETGKBRERGSMQkFERMYoFEREZIxCQURExuSHXcBM1dTUeENDQ9hliIjklG3bth1z99rp7su5UGhoaKCpqSnsMkREcoqZ7U/lPnUfiYjIGIWCiIiMUSiIiMgYhYKIiIxRKIiIyJjAQsHM7jSzNjN7ZpLnzcz+2cyazexpMzs/qFpERCQ1QbYUvgFcOcXzbwQa439uAv4lwFpERCQFgYWCuz8AdE5xy3XANz3mEaDCzJYFVY9I0Lp6h/jOo/sZGB4NuxSRWQtzTGEF0JL0uDV+7SXM7CYzazKzpvb29owUJzIT3X3DvOOrj/LxHz7DH317G4MjCgbJTWGGgk1wzSe60d3vcPdN7r6ptnbaVdoiGdU/NMq77nyU5rYe3vWKVdy/u533/9sTRKMTfjmLZLUwt7loBeqTHtcBh0KqRWTWfvL0IZ5q7eb2t5/P1RuXsbS8iNt+upttB7q4sKEq7PJEZiTMlsJm4F3xWUgvB7rd/XCI9YjMyn9uP8yKimKuOnspAO96RQOF+RH+82l9OUvuCXJK6l3Aw8DpZtZqZn9gZjeb2c3xW7YAe4Fm4CvA+4KqRSQo3X3D/E/zMa7euAyzWI9o2YJ8XnNaLfc+c1hdSJJzAus+cvcbpnnegT8O6v1FMuFnO48wPOpcdfaLJ85dvXEZP9t5VF1IknO0olnkFCS6js6pK3/R9cvOWKIuJMlJCgWRWeruf2nXUUKiC2nL9sPEGsUiuUGhIDJLTxzoYnjUec3pE0+Tfu36xbSdHGRfR1+GKxOZPYWCyCw91dKNGWysq5jw+XPrK+L3Hc9kWSKnRKEgMktPtR6ncXEZZQsmnq/RuLiM4oI8nlQoSA5RKIjMgrvzVMtxzpmklQCQnxfh7BXlPNWqUJDcoVAQmYXWrn46eoc4p37yUAA4p76cHYdOMDQSzVBlIqdGoSAyC4nf/s+dNhQqGBqJsvvIyUyUJXLKFAois/BUy3EK8yOcvnThlPclupeeVBeS5AiFgsgsPNXSzVnLF1GQN/W3UF1lMdWlhZqBJDlDoSAyQyOjUbYf7J52PAHAzDinvkIzkCRnKBREZuhAZx/9w6Ocubx8+puBs5YvYm97jw7ekZygUBCZoT1tPUBsHUIq1i4uI+rw/LHeIMsSSQuFgsgMNcdDYW2KodC4ODYYvedoT2A1iaSLQkFkhprbelhRUTzpSubx1tSWErEXwkQkmykURGZoT9vJlFsJAEUFedRXlSgUJCcoFERmIBp1mtt6Uh5PSGhcXMaeNi1gk+ynUBCZgYPH+xkYjs44FNYtXsjzx3oZGdV2F5LdFAoiM5DoAlo341AoY3jU2d+psxUkuykURGYg0QU001BItCw0A0mynUJBZAaa23qoXbiAipLCGX1eYmD6uXaFgmQ3hYLIDOxp62Fd7cxaCRA7s3l5eRF7jmqwWbKbQkEkRe6xmUcz7TpKWLdk4dhqaJFspVAQSVFn7xAnB0ZYU1s6q89fU1PK/o4+3D3NlYmkj0JBJEX7OmIzhxqqZxcKq6pL6BkcoaN3KJ1liaSVQkEkRfs7YhvaraoumdXnJ8Ik8Toi2UihIJKifR19RAzqKmcXCokw2XdMaxUkeykURFK0v6OX5RXFFObP7tumrrKEiKmlINlNoSCSon0dfbMeTwAozI+worJYq5olqykURFK0v6N31uMJCauqSscGrEWyUaChYGZXmtluM2s2s1sneH6lmd1vZk+Y2dNmdlWQ9YjMVnffMMf7hk+ppQCxcQV1H0k2CywUzCwPuB14I7ABuMHMNoy77S+B77n7ecD1wJeCqkfkVOzvjP0gX3mKLYWG6lKO9w1zvE/TUiU7BdlSuAhodve97j4EfBe4btw9DiyKf1wOHAqwHpFZO9U1CgmJ7qf96kKSLBVkKKwAWpIet8avJfsk8E4zawW2AO8PsB6RWdt/LN5SqDrFlkJNfK2CBpslSwUZCjbBtfHr+28AvuHudcBVwLfM7CU1mdlNZtZkZk3t7e0BlCoytX0dfSxdVERxYd4pvU4iVBIhI5JtggyFVqA+6XEdL+0e+gPgewDu/jBQBNSMfyF3v8PdN7n7ptra2oDKFZlcOmYeQey85qWLijQDSbJWkKGwFWg0s9VmVkhsIHnzuHsOAJcBmNkZxEJBTQHJOvs7+9ISCqAZSJLdAgsFdx8BbgHuA3YRm2W0w8w+ZWbXxm/7U+C9ZvYUcBfwHtcWkpJlBoZHaT85SP0st7cYr76qhJYutRQkO+UH+eLuvoXYAHLytU8kfbwTuDjIGkROVWv8B3j9KQ4yJ9RXlnD0xCADw6MUFZzaGIVIumlFs8g0Wrr6AairLE7L69VXxV7n4PH+tLyeSDopFESm0dqZ3pZCYgZSi6alShZSKIhMo6Wrn8L8CLVlC9LyeolwSbRARLKJQkFkGi2dfdRVFBOJTLT0ZuZqyxZQmB8Za4GIZBOFgsg0Wrv6qUtT1xFAJGLUVRZrBpJkJYWCyDRauvqoT9Mgc0J9ZQkH1FKQLKRQEJnCyYHYltmzPYJzMvVVxbR0akxBso9CQWQKrfHB4MQ00nSpryyhu3+YEwPDaX1dkVOlUBCZQmLaaLpWMyfUa1qqZCmFgsgUWsZaCmkOhcpEKKgLSbKLQkFkCq1dfZQU5lFZUpDW1010R7VqBpJkGYWCyBRaOvupryzBLD1rFBLKiwtYWJSv7iPJOgoFkSm0dvWlfZAZwMyoryzRqmbJOgoFkUm4e2zhWpoHmRNi01LVUpDsolAQmcTxvmF6BkfStjvqeLGWQh86QkSyiUJBZBItaT5HYbz6qhIGhqO09wwG8vois6FQEJlEa5rPURgvMVahaamSTRQKIpNoSfM5CuMl1ipoWqpkk2mP4zSzxcSOzFwO9APPAE3uHg24NpFQtXT1UV5cwKKi9K5RSKir1KpmyT6ThoKZvRa4FagCngDagCLgzcBaM7sH+Ht3P5GJQkUyLTbzKJiuI4Diwjxqyhao+0iyylQthauA97r7gfFPmFk+cA1wOfD9gGoTCVVLZx+NixcG+h71VTpXQbLLpKHg7n82xXMjwI8CqUgkCyTWKLxu/eJA32dlVQmPH+gK9D1EZmLagWYz+5aZlSc9bjCzXwRblki42nsGGRyJBrZwLaG+soRDxwcYGdUQnWSHVGYf/Rp41MyuMrP3Aj8D/inYskTClejnD2KLi2T1VcWMRp3D3QOBvo9IqqadfeTuXzazHcD9wDHgPHc/EnhlIiFKTBNN9zkK49UnzUAKauqryEyk0n10I3An8C7gG8AWMzsn4LpEQpVYuLYiwNlHkHTYjgabJUtM21IA3gJc4u5twF1m9kNi4XBekIWJhKmls4+askJKClP5Fpm9ZeVF5EVM01Ila6TSffTmcY8fM7OXBVeSSPhauvoCH2QGyM+LsKy8SC0FyRqTdh+Z2V+aWdVEz7n7kJm9zsyuCa40kfC0dPZnrI+/vrJEq5ola0zVUtgO/IeZDQCPA+3EVjQ3AucCPwf+T+AVimTYyGiUg8f7edM5yzLyfvVVxdy/uz0j7yUynakWr/0Y+LGZNRLb+2gZcAL4NnCTu6sTVOakw90DjEadlRlsKbSfHGRgeJSigryMvKfIZFIZU9gD7JnNi5vZlcDngTzgq+7+2QnueRvwScCBp9z97bN5L5F0GdsdNQNjCgArq1/YLXVdwNtqiEwnlV1STwM+AjQk3+/ur5vm8/KA24ntj9QKbDWzze6+M+meRuBjwMXu3hXfkVUkVAcC3jJ7vBd2S+1XKEjoUplvdzfw/4CvAqMzeO2LgGZ33wtgZt8FrgN2Jt3zXuB2d+8CiE97FQlVS1cfeRFjWXlRRt5v7LAdzUCSLJBKKIy4+7/M4rVXAC1Jj1uB8VNZTwMws/8h1sX0SXf/6fgXMrObgJsAVq5cOYtSRFJ3oLOfFRXF5Odl5gyq2rIFFBVEONChUJDwTTUltSo+JfU/zOx9ZrYscW2yqarjX2KCa+NPKM8nNpvpNcANwFfNrOIln+R+h7tvcvdNtbW1Kby1yOzFtpwIdiVzMjOjrrJELQXJClO1FLYR+yGe+OGevJW2A2umee1WoD7pcR1waIJ7HnH3YeB5M9tNLCS2TvPaIoFp6ezjDWcuyeh71lcWa1WzZIWppqSuPsXX3go0mtlq4CBwPTB+ZtGPiLUQvmFmNcS6k/ae4vuKzFrv4AgdvUMZWc2crL6qhKb9OldBwpfSxi5m9kpeOvvom1N9jruPmNktwH3ExgvudPcdZvYpYmc8b44/9wYz20lsEPvP3L1jVv8SkTRIdOFkao1CQn1lCScHRujuG6a8JJgzoUVSkcqU1G8Ba4EneWH2kQNThgKAu28Btoy79omkjx34cPyPSOgSXTgZD4WkGUjlJeXT3C0SnFRaCpuADfEf4CJzWqbXKCTUJZ2rcNYKhYKEJ5U5d88AS4MuRCQbtHT2UbYgn8oMd+EkVjVrBpKELZWWQg2w08weAwYTF9392sCqEglJS2cfdZXFmE00ozo4i4oKKC8u0AwkCV0qofDJoIsQyRYHOvtYXVMaynvXVxWPdV+JhCWVDfH+OxOFiITN3Wnp6uPS08JZIFlfWcLuoydDeW+RhFTOaH65mW01sx4zGzKzUTM7kYniRDKpvWeQgeFoxmceJdRXldDa1U80qjkdEp5UBpq/SGyB2R6gGPjD+DWROSWs6agJ9ZXFDI1Eae8ZnP5mkYCktOOXuzcDee4+6u5fJ7ZXkcicMnaOQgb3PUpWV/XCtFSRsKQSCn1mVgg8aWa3mdmHgHBG4kQClPhhnOktLhISh/poWqqEKZVQuDF+3y1AL7FN7n47yKJEwnCgs4/FCxeEdiRmXWV8VbOmpUqIUgmFN7v7gLufcPe/cfcPA9cEXZhIprV09YU2ngBQVJDHkkUL1H0koUolFN49wbX3pLkOkdC1dPZnfHuL8ep1roKEbNJ1CmZ2A7Gtrleb2eakpxYC2slU5pShkSiHu7MgFKpKeOz5zlBrkPltqsVrDwGHiW1z8fdJ108CTwdZlEimHTreT9TDm46aUF9ZzI+f7Gd4NEpBho4DFUk21SE7+4H9wCsyV45IOMZ2R60MZzpqQl1VCVGPhdSqak3yk8zTryIiJB2uUx12SyGxVkEzkCQcCgURYi2FwrwISxYWhVpHYuGcNsaTsEwaCmb2i/jff5e5ckTCse9YL/VVxUQimd0ye7xl5cUU5kXY39Ebah0yf0010LzMzF4NXGtm3wVe9N3i7o8HWplIBu07Ft6W2cnyIsbK6hKeP6ZQkHBMFQqfAG4F6oB/GPecA68LqiiRTIpGnX0dvbyqsSbsUgBoqC5ln1oKEpKpZh/dA9xjZn/l7p/OYE0iGXXkxACDI1EasqClALC6poQH97QTjXro3Vky/6RyyM6nzexa4NL4pV+5+0+CLUskc/bFu2qyofsIoKGmlMGRKIdPDLCiItwpsjL/pHLIzmeADwI7438+GL8mMic8H++qyZqWQnx9wj6NK0gIUpmSejVwubvf6e53AlfGr4nMCfuO9bIgP8KyReFOR01IhJMGmyUMqa5TqEj6uDyIQkTC8vyxPlZVl2RN//3SRUUsyI+opSChmHZMAfgM8ISZ3U9sWuqlwMcCrUokg/Z19LImS7qOACIR0wwkCU0qA813mdmvgAuJhcKfu/uRoAsTyYTRqHOgo4/L1i8Ou5QXaagpobmtJ+wyZB5KpaWAux8GNk97o0iOOXS8n6HR7JmOmtBQU8r9z7YzGnXysqRbS+YH7X0k81qii6Yhy3YkXV1dytBolEPHtTGeZJZCQea1bFujkJBouezVYLNk2FQb4lVN9SeVFzezK81st5k1m9mtU9z3VjNzM9s0m3+EyGw9195LaWHsbORssqY2HgrtGleQzJpqTGEbsT2ODFgJdMU/rgAOAKunemEzywNuBy4HWoGtZrbZ3XeOu28h8AHg0Vn+G0Rmrbmth7WLyzDLrn772rIFLCrK12CzZNykLQV3X+3ua4D7gDe5e427VwPXAD9I4bUvAprdfa+7DwHfBa6b4L5PA7cBAzOuXuQUPdfew7rasrDLeAkzY93iMp5TS0EyLJUxhQvdfUvigbvfC7w6hc9bAbQkPW6NXxtjZucB9dpLScLQMzjC4e4B1i7OvlAAWFtbRnObxhQks1IJhWNm9pdm1mBmq8zs40BHCp83UXvcx540iwD/CPzptC9kdpOZNZlZU3t7ewpvLTK95+JdM2uzsKUAsG5xGcd6BunuGw67FJlHUgmFG4Ba4IfAj4DF8WvTaQXqkx7XAYeSHi8EzgJ+ZWb7gJcDmycabHb3O9x9k7tvqq2tTeGtRaaX6K9fl6UthURdze0nQ65E5pNUVjR3Etsldaa2Ao1mtho4CFwPvD3pdbuBsVNN4qumP+LuTbN4L5EZe669h/yIsaq6JOxSJpQIhefaerlgVUoT/kRO2bShYGa1wEeBM4GxbSTdfcqT19x9xMxuITZQnQfc6e47zOxTQJO7a4W0hKq5rYeGmlIK8rJzuU5dZQmF+RGaNdgsGZTKNhffAf6d2Kyjm4F3Ayl17McHqLeMu/aJSe59TSqvKZIuze09nLZ4YdhlTCovYqypKdW0VMmoVH5Fqnb3rwHD7v7f7v77xPr/RXLW0EiU/R19rF2cXSuZx1u7uEyhIBmVSigkpj4cNrOr49NI6wKsSSRwBzp7GY161g4yJ6yrLaOlq4+B4dGwS5F5IpXuo781s3JiU0e/ACwCPhRoVSIBG5t5VJu93UcQG2x2j53CdsayRWGXI/PAlKEQ36qiMb64rBt4bUaqEgnY7iM9mJH13UenL42F1m+OnlQoSEZM2X3k7qPAtRmqRSRjnj1ygobqUkoKUzpSJDSra0opyDN2HdZaBcmMVL4jHjKzLxKbgTS25t7dHw+sKpGAPXvkJOuXZnfXEUBBXoR1ixfy7JETYZci80QqofDK+N+fSrrmwJTrFESyVd/QCPs6ennzuSumvzkLnLF0IQ89l8rOMiKnLpUVzRpHkDllz9Ee3F/or89265ct5AdPHOR43xAVJYVhlyNzXCormj88weVuYJu7P5n+kkSCleiKOWNZboTC6UtjA8zPHjnJy9dUh1yNzHWprFPYRGwl84r4n5uA1wBfMbOPBleaSDB2HT5JSWEe9ZXZuefReGfEWzTPHta4ggQvlTGFauB8d+8BMLO/Bu4BLiV2OtttwZUnkn7PHjnBaUsWEolk12lrk6lduICq0kKePaIZSBK8VFoKK4GhpMfDwCp37wcGA6lKJCDuzrNHTuZM1xHETmFbv3QhuxQKkgGptBT+DXjEzH4cf/wm4C4zKwV2Tv5pItnn6IlBjvcNs35pbi0EW790EXc9doBo1HOmhSO5KZXZR582sy3AJcROU7s56cyDdwRZnEi67YoPMufKzKOE9csW0j88yr6OXtZk6UlxMjektJzT3bcRGz8QyWnbW7sBOHN5brUUzlpeDsD2g90KBQlUdp4uIhKQp1u7WVNbysKigrBLmZHGJWUsyI+MhZpIUBQKMq9sP3icjSvKwy5jxgryImxYvoinDyoUJFgKBZk3jp4Y4OiJQc6uqwi7lFnZuKKcHQe7GY162KXIHKZQkHkj0fWysS73WgoAZ9dV0Ds0yvPHdBKbBEehIPPG0we7iRhsyNFzCRJh9rTGFSRACgWZN7a3Hmfd4jJKF2T3GQqTWVtbRklhnkJBAqVQkHnB3dl+sJuNOTqeAJAXMc5aXs52DTZLgBQKMi8c6h7gWM9Qzo4nJJxdV86OQ92MjEbDLkXmKIWCzAtPHOgC4JwcbikAnFNfwcBwVJvjSWAUCjIvNO3rorggjw05tpJ5vAsbKgHYuq8z5EpkrlIoyLywdV8n562soCAvt7/kl5UXs6KimKZ9XWGXInNUbn+HiKTg5MAwuw6f4MKGqrBLSYsLGyrZuq8Tdy1ik/RTKMic98SB40SdORMKmxqqaDs5SEtnf9ilyBykUJA5r2lfJ3kR49yVuT3InJAIN40rSBAUCjLnbd3XxYZliyjL0UVr4zUuLmNRUT5N+xUKkn4KBZnThkejPNHSxab4rJ25IBIxNjVUsVWDzRKAQEPBzK40s91m1mxmt07w/IfNbKeZPW1mvzCzVUHWI/PPUy3HGRiOctEcGU9IuGh1Fc1tPbSdGAi7FJljAgsFM8sDbgfeCGwAbjCzDeNuewLY5O4bgXuA24KqR+anB/YcI2LwyrU1YZeSVpesi/17ft18LORKZK4JsqVwEdDs7nvdfQj4LnBd8g3ufr+798UfPgLUBViPzEMP7mnnnPoKykty66S16WxYtojq0kIe3KNQkPQKMhRWAC1Jj1vj1ybzB8C9Ez1hZjeZWZOZNbW3t6exRJnLuvuGearlOK9qrA27lLSLRIxLGmt4cM8xojp0R9IoyFCwCa5N+NVrZu8ENgGfm+h5d7/D3Te5+6ba2rn3DS7BeOi5Y0QdLm2cW11HCa9qrOVYz6D2QZK0CjIUWoH6pMd1wKHxN5nZ64GPA9e6+2CA9cg888CeYyxckM859XNjfcJ4r4qH3YN71HqW9AkyFLYCjWa22swKgeuBzck3mNl5wJeJBUJbgLXIPOPuPLinnVesrc75/Y4ms2RREacvWcgDCgVJo8C+W9x9BLgFuA/YBXzP3XeY2afM7Nr4bZ8DyoC7zexJM9s8ycuJzMieth5au/q59LS53d346tNr2fp8FycGhsMuReaIQJd4uvsWYMu4a59I+vj1Qb6/zF8/feYIZvCGDUvCLiVQV5y5hDse2Mv9z7Zx3blTzeMQSc3cbFfLvPfTZ45wwcpKFi8qCruUQJ1XX0ntwgX89JkjYZcic4RCQeacAx197Dx8givPWhp2KYGLRIwrzlzCr3a30z80GnY5MgcoFGTOuW9H7LfmK86c+6EAcOWZy+gfHtWAs6SFQkHmnHufOcxZKxZRX1USdikZ8bI1VVSUFKgLSdJCoSBzyoGOPh4/cJw3nrUs7FIypiAvwhs2LOFnO47QNzQSdjmS4xQKMqfcs62FiMFvnz+/ZuK89YJ6eodG2bJdrQU5NQoFmTNGo84921q5pLGWZeXFYZeTURc2VNJQXcLdTS3T3ywyBYWCzBkPPXeMQ90D/M4F82+zXTPjrRfU8ejznRzo6Jv+E0QmoVCQOePuplbKiwu4fI4vWJvMWy6owwzu3qbWgsyeQkHmhLYTA9z7zGF+67wVFBXkhV1OKJaVF/Pq02q567EWBke0ZkFmR6Egc8K/PryPkajzexc3hF1KqP7wkjUc6xnkx0+8ZENikZQoFCTn9Q2N8O1HDnDFhqWsqi4Nu5xQXbyumvVLF/KVB/firsN3ZOYUCpLz7m5qpbt/mPdeuibsUkJnZtx06Rr2tPXwq99ohbPMnEJBctrA8Ch3PLCX81dWcMGqyrDLyQrXbFzO0kVFfOEXe9RakBlTKEhO+9bD+zl4vJ+PvOH0sEvJGoX5ET5wWSOPHzjOf+08GnY5kmMUCpKzuvuH+eL9zVx6Wi2vXDc3z2GerbdtqmNNbSm33bebkdFo2OVIDlEoSM760v3NnBgY5tYr14ddStbJz4vw0SvW09zWw/eaWsMuR3KIQkFy0o5D3Xzt18/zlvPr2LB8UdjlZKUrzlzCRQ1VfPbeXbSdHAi7HMkRCgXJOSOjUf78+09TUVLAX159RtjlZC0z4zNvOZuBkSh//eMdYZcjOUKhIDnnyw/s5ZmDJ/iba8+ioqQw7HKy2traMj54WSP3PnOEnzytBW0yPYWC5JSHn+vg73+2m6s3LuOqs+fHyWqn6qZL13BufQW3fn87z7X3hF2OZDmFguSMI90DvP+ux1ldU8rfvWUjZhZ2STmhIC/Cl95xPoX5EW7+1jZ6B3UQj0xOoSA54XjfEO/5+mP0DY3y5RsvoGxBftgl5ZTlFcV84YbzeK69h5u/vU0b5smkFAqS9XoGR3jP17eyt72XO27cxLrFC8MuKSddvK6Gz75lIw/uOcYH7npC6xdkQgoFyWptJwe44Y5H2H6wmy+8/TwuadQitVPxtk31fOKaDdy34yg3qStJJqBQkKy16/AJfvtLD9Hc1sMdN17AFWdqYDkdfv+S1fzv3zqLX+1u4/o7HuHg8f6wS5IsolCQrBONOv/60D6uu/1/GByJ8u//6+Vcdsb8PE0tKO942Sq+8q5N7G3v4Y3/9ABbth8OuyTJEgoFySo7D53gbV9+mL/evINL1tXw0w++io11FWGXNSdddsYS/vMDr2J1TSnv+87j/OG/NtHSqfOd5zvLta11N23a5E1NTWGXIWn2m6Mn+eIvm/mPpw9RWVLIrVeu53c21WnaaQYMjUT52q+f559/sYfRqPO7F9Zz82vWsqKiOOzSJI3MbJu7b5r2PoWChOXEwDD3P9vGdx9r4eG9HZQU5nHjK1bxvlevo7ykIOzy5p1Dx/v5wi/3cHdTKw5ctn4xb9tUzyWNNfP23Ou5JCtCwcyuBD4P5AFfdffPjnt+AfBN4AKgA/hdd9831WsqFHJXNOrs6+jlgd+08/NdbTyyt4ORqLOiopgbLqrn7S9bRVWptq0IW2tXH99+5AB3N7XQ0TtEcUEelzTW8PozFnPxuhpWVBSrBZeDQg8FM8sDfgNcDrQCW4Eb3H1n0j3vAza6+81mdj3wW+7+u1O9rkIh+7k7Hb1D7O/oZX9HH7uPnmR7azfbD3ZzciA2BXJtbSmv37CE15+xhPNXVpIX0Q8s3CZWAAAIqklEQVSZbDM0EuWRvR38fNdRfr7zKIe6YzutVpcWcnZdORvrKli3uIxVVSU0VJeqdZflsiEUXgF80t2viD/+GIC7fybpnvvi9zxsZvnAEaDWpyhKoZB+7s5o1BmJOsOjUUZGY38PR52R0SjDo07/0Cg9gyP0Do7QE//TG/9zYmCEYz2DtJ8cpL1nkKPdA/QOvbBitiDPWL90ERvrytlYV85Fq6tZXVMa4r9YZsrd2XX4JNv2d/JUazfbW7vZ03aSaNJ36qKifBYvKqKqtJCaskKqSxdQVVpIVWkhxYV5lMT/FBfkx/4uzKO4II8FBREKIhHy8oz8iJEXMfIjESKGWiRplGooBLlXwAqgJelxK/Cyye5x9xEz6waqgWPpLuZ7W1u448G9xN8r9nfyDT7hhxPe6y+61196bZJIm9VrTXIv09479XtF3RkZdUaisR/6s2UGZYX51CxcQG3ZAtYvXciljbWsqi5hVXUJK6tKqa8qZkG++qRzmZmxYfkiNixfxI3xa/1Do+zvjLUGD3T0caCzj2M9g3T0DrH7yEk6ezvo6hs+pfd9ISTif+dFMGJfd2Dx2hi7ZhiJHDFeCJWxa0n3vOj5+P+kO4LSHWofvKyRN52zPK2vOV6QoTDR/xvjf/qkcg9mdhNwE8DKlStnVUxlaSGnL0naHsFe9FfifSYszFK9N+kGS3pgL7o+0bWX3vvir6VTfK1J6i7Ii32TFcS/2fLzjIJI7O/k6wV5RnFBHmVF+ZQtyKd0QezvsgX5FBfkEVHXz7xUXJjH+qWLWL908kOORkajdPcP0zc0Sv/wKH1Do/QNjdA/FPu4f2iUwdEoo6NRRqIvtFhf+Dt+fTT2eCQaxf2FHxKxX3Q8ds1f/IuVk3Qt/gmxaz723PhraRVAJ0x5cfBddEGGQitQn/S4Dhi/oXvintZ491E50Dn+hdz9DuAOiHUfzaaYyzcs4fINWgAlkkn5eRGqyxZQHXYhkrIgF69tBRrNbLWZFQLXA5vH3bMZeHf847cCv5xqPEFERIIVWEshPkZwC3AfsSmpd7r7DjP7FNDk7puBrwHfMrNmYi2E64OqR0REphfopvTuvgXYMu7aJ5I+HgB+J8gaREQkddr7SERExigURERkjEJBRETGKBRERGSMQkFERMbk3NbZZtYO7J/lp9cQwBYaaZKttamumcnWuiB7a1NdMzeb2la5e+10N+VcKJwKM2tKZUOoMGRrbaprZrK1Lsje2lTXzAVZm7qPRERkjEJBRETGzLdQuCPsAqaQrbWprpnJ1roge2tTXTMXWG3zakxBRESmNt9aCiIiMoV5Fwpmdq6ZPWJmT5pZk5ldFHZNCWb2fjPbbWY7zOy2sOsZz8w+YmZuZjVh1wJgZp8zs2fN7Gkz+6GZVYRcz5Xx/37NZnZrmLUkmFm9md1vZrviX1cfDLumZGaWZ2ZPmNlPwq4lmZlVmNk98a+vXfHjhUNnZh+K/3d8xszuMrOidL/HvAsF4Dbgb9z9XOAT8cehM7PXAtcBG939TOD/hlzSi5hZPXA5cCDsWpL8F3CWu28EfgN8LKxCzCwPuB14I7ABuMHMNoRVT5IR4E/d/Qzg5cAfZ0ldCR8EdoVdxAQ+D/zU3dcD55AFNZrZCuADwCZ3P4vYkQRpP25gPoaCA4nzA8t56WlwYfkj4LPuPgjg7m0h1zPePwIfJZBDBmfH3X/m7iPxh48QO90vLBcBze6+192HgO8SC/lQufthd388/vFJYj/cVoRbVYyZ1QFXA18Nu5ZkZrYIuJTYeS+4+5C7Hw+3qjH5QHH8pMoSAvj5NR9D4U+Az5lZC7HfxkP77XKc04BXmdmjZvbfZnZh2AUlmNm1wEF3fyrsWqbw+8C9Ib7/CqAl6XErWfLDN8HMGoDzgEfDrWTMPxH7RSMadiHjrAHaga/Hu7a+amalYRfl7geJ/cw6ABwGut39Z+l+n0AP2QmLmf0cWDrBUx8HLgM+5O7fN7O3Eftt4PVZUFc+UEmsiX8h8D0zW5Op40mnqe0vgDdkoo7xpqrL3X8cv+fjxLpJvpPJ2saxCa5lTavKzMqA7wN/4u4nsqCea4A2d99mZq8Ju55x8oHzgfe7+6Nm9nngVuCvwizKzCqJtT5XA8eBu83sne7+7XS+z5wMBXef9Ie8mX2TWD8mwN1ksOk6TV1/BPwgHgKPmVmU2P4m7WHWZmZnE/sifMrMINZF87iZXeTuR8KqK6m+dwPXAJeFfL53K1Cf9LiOLOmaNLMCYoHwHXf/Qdj1xF0MXGtmVwFFwCIz+7a7vzPkuiD237LV3RMtqnuIhULYXg887+7tAGb2A+CVQFpDYT52Hx0CXh3/+HXAnhBrSfYjYvVgZqcBhWTBZlzuvt3dF7t7g7s3EPuGOT8TgTAdM7sS+HPgWnfvC7mcrUCjma02s0JiA4CbQ64JiyX514Bd7v4PYdeT4O4fc/e6+NfU9cAvsyQQiH9tt5jZ6fFLlwE7Qywp4QDwcjMrif93vYwABsDnZEthGu8FPh8fqBkAbgq5noQ7gTvN7BlgCHh3yL/55oIvAguA/4q3Yh5x95vDKMTdR8zsFuA+YrNC7nT3HWHUMs7FwI3AdjN7Mn7tL+Lnp8vk3g98Jx7we4HfC7ke4l1Z9wCPE+sufYIAVjZrRbOIiIyZj91HIiIyCYWCiIiMUSiIiMgYhYKIiIxRKIiIyBiFgoiIjFEoiIjIGIWCyCkyswvjZzoUmVlpfL/7s8KuS2Q2tHhNJA3M7G+J7eFTTGzfnM+EXJLIrCgURNIgvh3CVmJbp7zS3UdDLklkVtR9JJIeVUAZsJBYi0EkJ6mlIJIGZraZ2Glrq4Fl7n5LyCWJzMp83CVVJK3M7F3AiLv/W/ys5ofM7HXu/suwaxOZKbUURERkjMYURERkjEJBRETGKBRERGSMQkFERMYoFEREZIxCQURExigURERkjEJBRETG/H+HMIGuxZTIPwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y.backward(torch.ones_like(x),retain_graph=True)\n",
    "xyplot(x,x.grad,\"grad of tanh\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In summary, we now know how to incorporate nonlinearities\n",
    "to build expressive multilayer neural network architectures.\n",
    "As a side note, your knowledge now already\n",
    "puts you in command of the state of the art in deep learning, circa 1990.\n",
    "In fact, you have an advantage over anyone working the 1990s,\n",
    "because you can leverage powerful open-source deep learning frameworks\n",
    "to build models rapidly, using only a few lines of code.\n",
    "Previously, getting these nets training\n",
    "required researchers to code up thousands of lines of C and Fortran.\n",
    "\n",
    "## Summary\n",
    "\n",
    "* The multilayer perceptron adds one or multiple fully-connected hidden layers between the output and input layers and transforms the output of the hidden layer via an activation function.\n",
    "* Commonly-used activation functions include the ReLU function, the sigmoid function, and the tanh function.\n",
    "\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Compute the derivative of the tanh and the pReLU activation function.\n",
    "1. Show that a multilayer perceptron using only ReLU (or pReLU) constructs a continuous piecewise linear function.\n",
    "1. Show that $\\mathrm{tanh}(x) + 1 = 2 \\mathrm{sigmoid}(2x)$.\n",
    "1. Assume we have a multilayer perceptron *without* nonlinearities between the layers. In particular, assume that we have $d$ input dimensions, $d$ output dimensions and that one of the layers had only $d/2$ dimensions. Show that this network is less expressive (powerful) than a single layer perceptron.\n",
    "1. Assume that we have a nonlinearity that applies to one minibatch at a time. What kinds of problems to you expect this to cause?\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}