Ch06_Multilayer_Perceptrons/Model_Selection_Underfitting_and_Overfitting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Underfitting or Overfitting?\n",
    "When we compare the training and validation errors, we want to be mindful of two common situations: First, we want\n",
    "to watch out for cases when our training error and validation error are both substantial but there is a little gap between\n",
    "them. If the model is unable to reduce the training error, that could mean that our model is too simple (i.e., insufficiently\n",
    "expressive) to capture the pattern that we are trying to model. Moreover, since the generalization gap between our training\n",
    "and validation errors is small, we have reason to believe that we could get away with a more complex model. This\n",
    "phenomenon is known as underfitting.\n",
    "\n",
    "On the other hand, as we discussed above, we want to watch out for the cases when our training error is significantly lower\n",
    "than our validation error, indicating severe overfitting. Note that overfitting is not always a bad thing. With deep learning\n",
    "especially, it’s well known that the best predictive models often perform far better on training data than on holdout data.\n",
    "\n",
    "Ultimately, we usually care more about the validation error than about the gap between the training and validation errors.\n",
    "Whether we overfit or underfit can depend both on the complexity of our model and the size of the available training\n",
    "datasets, two topics that we discuss below.\n",
    "\n",
    "## Model Complexity \n",
    "\n",
    "To illustrate some classical intuition about overfitting and model complexity, we given an example using polynomials.\n",
    "Given training data consisting of a single feature x and a corresponding real-valued label y, we try to find the polynomial\n",
    "of degree d\n",
    "\n",
    "$$y=\\sum_{i=0}^d\\ W^ix^i$$\n",
    "\n",
    "to estimate the labels y. This is just a linear regression problem where our features are given by the powers of x, the wi\n",
    "given the model’s weights, and the bias is given by w0 since x\n",
    "0 = 1 for all x. Since this is just a linear regression problem,\n",
    "we can use the squared error as our loss function.\n",
    "A higher-order polynomial function is more complex than a lower order polynomial function, since the higher-order polynomial has more parameters and the model function’s selection range is wider. Fixing the training data set, higher-order\n",
    "polynomial functions should always achieve lower (at worst, equal) training error relative to lower degree polynomials.\n",
    "In fact, whenever the data points each have a distinct value of x, a polynomial function with degree equal to the number\n",
    "of data points can fit the training set perfectly. We visualize the relationship between polynomial degree and under- vs\n",
    "over-fitting below.\n",
    "\n",
    "## Data Set Size\n",
    "\n",
    "The other big consideration to bear in mind is the dataset size. Fixing our model, the fewer samples we have in the\n",
    "training dataset, the more likely (and more severely) we are to encounter overfitting. As we increase the amount of\n",
    "training data, the generalization error typically decreases. Moreover, in general, more data never hurts. For a fixed task\n",
    "and data distribution, there is typically a relationship between model complexity and dataset size. Given more data, we\n",
    "might profitably attempt to fit a more complex model. Absent sufficient data, simpler models may be difficult to beat. For\n",
    "many tasks, deep learning only outperforms linear models when many thousands of training examples are available. In part, the current success of deep learning owes to the current abundance of massive datasets due to internet companies,\n",
    "cheap storage, connected devices, and the broad digitization of the economy.\n",
    "\n",
    "## Polynomial Regression\n",
    "We can now explore these concepts interactively by fitting polynomials to data. To get started we’ll import our usual\n",
    "packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.utils.data as Data\n",
    "from torch.utils.data import TensorDataset\n",
    "from torch.utils.data import DataLoader\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating Data Sets\n",
    "\n",
    "First we need data. Given x, we will use the following cubic polynomial to generate the labels on training and test data:\n",
    "\n",
    "$$y=5+1.2x-3.4\\frac{x^2}{2!}+5.6\\frac{x^3}{3!}+ E where E-N(0,0.1)$$\n",
    "\n",
    "The noise term ϵ obeys a normal distribution with a mean of 0 and a standard deviation of 0.1. We’ll synthesize 100\n",
    "samples each for the training set and test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_degree=20\n",
    "n_train,n_test=100,100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-0.6245])\n",
      "tensor([ 0.0000e+00, -6.2449e-01,  3.8999e-01, -2.4354e-01,  1.5209e-01,\n",
      "        -9.4978e-02,  5.9313e-02, -3.7040e-02,  2.3131e-02, -1.4445e-02,\n",
      "         9.0209e-03, -5.6334e-03,  3.5180e-03, -2.1970e-03,  1.3720e-03,\n",
      "        -8.5679e-04,  5.3505e-04, -3.3413e-04,  2.0866e-04, -1.3031e-04])\n"
     ]
    }
   ],
   "source": [
    "poly_features=torch.zeros(20,200)\n",
    "true_w=torch.zeros(max_degree)\n",
    "true_w[0:4] = torch.tensor([5, 1.2, -3.4, 5.6])\n",
    "features = torch.randn(size=(n_train + n_test, 1))\n",
    "x_list=torch.arange(max_degree)\n",
    "x_list.float()\n",
    "features=features.reshape(1,-1)\n",
    "\n",
    "for i in range(1,max_degree):\n",
    "    \n",
    "    poly_features[i] = torch.pow(features,i)\n",
    "    \n",
    "print(features[:,4])\n",
    "print(poly_features[:,4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For optimization, we typically want to avoid very large values of gradients, losses, etc. This is why the monomials stored\n",
    "in poly_features are rescaled from x\n",
    "\n",
    "It allows us to avoid very large values for large exponents i. Factorials\n",
    "are implemented in Gluon using the Gamma function, where n! = Γ(n + b 1).\n",
    "Take a look at the first 2 samples from the generated data set. The value 1 is technically a feature, namely the constant\n",
    "feature corresponding to the bias"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.special import factorial\n",
    "ok=torch.arange(1,(max_degree) + 1).reshape((1, -1))\n",
    "\n",
    "dr=factorial(ok)\n",
    "\n",
    "dr2=torch.from_numpy(dr)\n",
    "\n",
    "poly_features = poly_features.double() /dr2.t()\n",
    "\n",
    "labels = torch.matmul(true_w.double(),poly_features)\n",
    "\n",
    "poly_features = poly_features.type(torch.FloatTensor)\n",
    "\n",
    "labels = labels.type(torch.FloatTensor)\n",
    "\n",
    "labels += torch.randn(200)*0.5\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining, Training and Testing Model\n",
    "We first define the plotting functionsemilogy, where the y axis makes use of the logarithmic scale\n",
    "\n",
    "Since we will be attempting to fit the generated dataset using models of varying complexity, we insert the model definition\n",
    "into the fit_and_plot function. The training and testing steps involved in polynomial function fitting are similar to\n",
    "those previously described in softmax regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def semilogy(x_vals, y_vals, x_label, y_label, x2_vals=None, y2_vals=None,\n",
    "legend=None, figsize=(3.5, 2.5)):\n",
    "   \n",
    "    \n",
    "    plt.xlabel(x_label)\n",
    "    plt.ylabel(y_label)\n",
    "    plt.semilogy(x_vals, y_vals)\n",
    "    if x2_vals and y2_vals:\n",
    "        plt.semilogy(x2_vals, y2_vals, linestyle=':')\n",
    "        plt.legend(legend)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit_and_plot(train_features,train_labels,test_features,test_labels,no_inputs):\n",
    "    class LinearRegressionModel(torch.nn.Module): \n",
    "  \n",
    "        def __init__(self): \n",
    "            super(LinearRegressionModel, self).__init__() \n",
    "            self.linear = torch.nn.Linear(no_inputs, 1)  \n",
    "  \n",
    "        def forward(self, x): \n",
    "            y_pred = self.linear(x) \n",
    "            return y_pred \n",
    "    \n",
    "    model = LinearRegressionModel() \n",
    "    criterion = torch.nn.MSELoss(reduction='sum') \n",
    "    optimizer = torch.optim.SGD(model.parameters(), lr = 0.01)\n",
    "    train_ls,test_ls=[],[]\n",
    "\n",
    "   \n",
    "   \n",
    "   \n",
    "    train_labels=train_labels.reshape(-1,1)\n",
    "    train_ds=TensorDataset(train_features,train_labels)\n",
    "    batch_size=10\n",
    "    train_dl=DataLoader(train_ds,batch_size,shuffle=True)\n",
    "\n",
    "\n",
    "    \n",
    "    test_labels=test_labels.reshape(-1,1)\n",
    "    for ep in range(100):\n",
    "        for xb,yb in train_dl: \n",
    "  \n",
    "        \n",
    "            pred_y = model(xb) \n",
    "  \n",
    "    \n",
    "            loss = criterion(pred_y, yb) \n",
    "            \n",
    "            optimizer.zero_grad() \n",
    "            loss.backward() \n",
    "            optimizer.step()\n",
    "           \n",
    "            \n",
    "        \n",
    "        predytr=model(train_features)\n",
    "        train_ls.append((criterion(predytr,train_labels)).mean())\n",
    "        predyts=model(test_features)\n",
    "        test_ls.append((criterion(predyts,test_labels)).mean())    \n",
    "    \n",
    "    \n",
    "    print('final epoch:train loss',train_ls[-1],'test Loss',test_ls[-1])\n",
    "    semilogy(range(1,ep+2), train_ls,'epoch','loss',range(1,ep+2),test_ls,['train','test'])\n",
    "  \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Third-order Polynomial Function Fitting (Normal)\n",
    "\n",
    "We will begin by first using a third-order polynomial function with the same order as the data generation function. The\n",
    "results show that this model’s training error rate when using the testing data set is low. The trained model parameters are\n",
    "also close to the true values w = [5, 1.2, −3.4, 5.6].\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "final epoch:train loss tensor(25.5834, grad_fn=<MeanBackward1>) test Loss tensor(28.1122, grad_fn=<MeanBackward1>)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZkAAAELCAYAAAALC/uGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd4HNXV+PHvkbTqvdlWseXee8HGFFNd6AFMDy1A3oT6pkEaKeQNSX4kgYSSAKYEMMVAaAZMcaHaWO7dxlWyLMmyepf2/v64K6tYstVGu5LO53n8WDM7O3OGRXt8y5wrxhiUUkopJ/h5OwCllFI9lyYZpZRSjtEko5RSyjGaZJRSSjlGk4xSSinHaJJRSinlGE0ySimlHKNJRimllGN6ZJIRkYtF5EkReUtEzvV2PEop1Vs5lmREJFVElorIVhHZLCJ3deBcC0QkR0Q2NfPaHBHZLiK7ROReAGPMf40xtwA3AFe0+yaUUkp1iDhVVkZE+gH9jDFrRCQCSAcuNsZsaXBMIlBujClusG+IMWZXk3OdBpQAzxtjxjTY7w/sAM4BMoBvgKvqriEiDwEvGmPWtBRnfHy8SUtL6/D9KqVUb5Kenn7YGJNwouMCnArAGJMFZHl+LhaRrUAysKXBYacD/yMi84wxFSJyC3AJMK/JuVaISFozl5kG7DLG7AYQkZeBizzXehB4v6UEIyIXABcMGTKE1atXd+BOlVKq9xGRfa05rkvGZDwJYiKwsuF+Y8xrwAfAyyJyDXATML8Np04GDjTYzvDsuwM4G7hMRL7f3BuNMe8YY26Niopqw+WUUkq1hWMtmToiEg68DtxtjClq+rox5s+eFsjjwGBjTElbTt/MPmOMeQR4pF0BK6WU6jSOtmRExIVNMC8aY95o4ZhTgTHAm8D9bbxEBpDaYDsFONiOUJVSSjnAsZaMiAjwNLDVGPPXFo6ZCDwJnAfsAV4QkQeMMb9s5WW+AYaKyEAgE7gSuLrDwSul1HFUV1eTkZFBRUWFt0NxXHBwMCkpKbhcrna938nuspnAdcBGEVnn2fdzY8ziBseEApcbY74FEJHrsdOOGxGRhcAsIF5EMoD7jTFPG2NqROR24EPAH1hgjNns1A0ppRRARkYGERERpKWlYf893TMZY8jLyyMjI4OBAwe26xxOzi77nObHTBoe80WT7Wpsy6bpcVcd5xyLgcUtva6UUp2toqKixycYABEhLi6O3Nzcdp+jRz7xr5RSTuvpCaZOR+9Tk4xSSinHaJJpr8//Bq9c6+0olFK9VEFBAY899lib3zdv3jwKCgociKh5mmTaS/zAz/HHjJRSqlktJZna2trjvm/x4sVER0c7FdYx9FuyvWa2u96nUkp12L333su3337LhAkTcLlchIeH069fP9atW8eWLVu4+OKLOXDgABUVFdx1113ceuutAKSlpbF69WpKSkqYO3cup5xyCl9++SXJycm89dZbhISEdGqcmmSUUqoDfvvOZrYcPKaYSYeMSork/gtGH/eYBx98kE2bNrFu3TqWLVvGeeedx6ZNm45ONV6wYAGxsbGUl5czdepULr30UuLi4hqdY+fOnSxcuJAnn3yS+fPn8/rrr3PttZ07DKDdZe21exk8cSoc2ePtSJRSimnTpjV6luWRRx5h/PjxTJ8+nQMHDrBz585j3jNw4EAmTJgAwOTJk9m7d2+nx6UtmfZyhUFEPzBub0eilPKiE7U4ukpYWNjRn5ctW8bHH3/MV199RWhoKLNmzWq2OkFQUNDRn/39/SkvL+/0uDTJtFfqVLjmVW9HoZTqpSIiIiguLm72tcLCQmJiYggNDWXbtm18/fXXXRxdPU0ySinVDcXFxTFz5kzGjBlDSEgIffr0OfranDlzeOKJJxg3bhzDhw9n+vTpXovTsZUxu4spU6aYdi1aVlUGT50NJ90Kk2/o9LiUUr5r69atjBw50tthdJnm7ldE0o0xU070Xh34by9XCMQOhJAYb0eilFI+S7vL2ksErnzR21EopZRP05aMUkopx2iS6Yg3btP6ZUopdRzaXdYRiSOhptLbUSillM/SJNMRp9zt7QiUUsqnaXeZUkp1Q+0t9Q/w97//nbKysk6OqHmaZDoi/Vn400CoLPF2JEqpXqa7JBntLuuI2EEw+hIwx1+/QSmlOlvDUv/nnHMOiYmJvPrqq1RWVnLJJZfw29/+ltLSUubPn09GRga1tbX86le/Ijs7m4MHD3LGGWcQHx/P0qVLHY1TWzIdMfA0OP+vEBzl7UiUUt70zHmw1vPcXG213V7/it2uKrPbm1632xWFdnvL23a7NM9ub3/fbhdnt+qSDz74IIMHD2bdunWcc8457Ny5k1WrVrFu3TrS09NZsWIFH3zwAUlJSaxfv55NmzYxZ84c7rzzTpKSkli6dKnjCQY0ySilVLe3ZMkSlixZwsSJE5k0aRLbtm1j586djB07lo8//pif/exnfPbZZ0RFdf0/iLW7rCPyvoUnz4Dz/wZjLvV2NEopb7nxvfqf/V2NtwNDG28HRzXeDotrvB1RX+iytYwx3Hfffdx2223HvJaens7ixYu57777OPfcc/n1r3/d5vN3hLZk2ul/X13HNS/tgnFXQHSat8NRSvUyDUv9z549mwULFlBSYichZWZmkpOTw8GDBwkNDeXaa6/lxz/+MWvWrDnmvU7Tlkw7CcLeUhfM+4u3Q1FK9UINS/3PnTuXq6++mhkzZgAQHh7OCy+8wK5du/jJT36Cn58fLpeLxx9/HIBbb72VuXPn0q9fP8fHZbTUfztL/f/+3S28vGo/m383B9xu8NNGoVK9hZb611L/josOcVFaVYv7sRnw5q3eDkcppXySdpe1U3SoC4CyUVcSHpfs5WiUUso3aZJpp6jQQAAOjbqJIYkRXo5GKdXVjDGIiLfDcFxHh1S0u6ydokJsS6awvForMSvVywQHB5OXl9fhL2BfZ4whLy+P4ODgdp9DWzLtFO1JMvFf/A72vQ73HfByREqprpKSkkJGRga5ubneDsVxwcHBpKSktPv9mmTaqW5MZn/sTAak9gdj7JLMSqkez+VyMXDgQG+H0S1od1k7RYfYMZkd4VPg1P/VBKOUUs3QJNNOEcEBiEBhaSVUFtuieEoppRrRJNNOfn5CVIiL2Jyv4I8pkPGNt0NSSimfo0mmA6JDXOw2SXDO7yEq1dvhKKWUz9GB/w6ICg1kX00YzLzT26EopZRP0pZMB0SFuCgoq4LyfCgv8HY4SinlczTJdEB0iIvCsir4f8Ph8796OxyllPI52l3WAdGhLgoqamDeg5A42tvhKKWUz9Ek0wHRIS4Ky6txT7oRPz99TkYppZrS7rIOiAoNxBgoLjgMBVpWRimlmtIk0wF19csC3rsLXrjUy9EopZTv0e6yDqirX5Y17FqGRNZ6ORqllPI9mmQ6oC7JHIyZypBhCV6ORimlfI92l3VA3ZoyRSUlkL3Z1jBTSil1lCaZDojyVGIOOLQWHj8Z9q/0ckRKKeVbNMl0QF1LZp9/Gly2APqO9W5ASinlY3RMpgMCA/wIC/QnpzoYxujsMqWUakpbMh0UHRpIQVk15H0LWRu8HY5SSvkUbcl0UFSIi8LyKnj7TnDXwM0fejskpZTyGZpkOig61JaW4bzfgL/L2+EopZRP0STTQVEhLnbllEDqyd4ORSmlfI6OyXRQdKiLgvJqKDsCOz/SdWWUUqoBTTIdFBUSSGFZNebQBnjxMjikg/9KKVVHk0wHRYe6qKp1U54wDm5aAkkTvR2SUkr5DB2T6aC6SswFtSGE9j/Jy9EopZRv0ZZMB9UVySwoq4a9n8OeFV6OSCmlfIe2ZDqorn5ZYXk1LP0tBATBwNO8HJVSSvkGTTIdVNeSKSyvgosfh8AwL0eklFK+o0clGRG5GDgPSAQeNcYscfqadUUyC8qqIX6I05dTSqluxefHZERkgYjkiMimJvvniMh2EdklIvcCGGP+a4y5BbgBuKIr4js6JlNeDQUHYPUzUJ7fFZdWSimf5/NJBngWmNNwh4j4A48Cc4FRwFUiMqrBIb/0vO64EJc/gf5+tiWTuw3evRtyd3TFpZVSyuf5fJIxxqwAjjTZPQ3YZYzZbYypAl4GLhLrT8D7xpg1LZ1TRG4VkdUisjo3N7dD8YkIUaGeIpkDZsI9myFlSofOqZRSPYXPJ5kWJAMHGmxnePbdAZwNXCYi32/pzcaYfxtjphhjpiQkJHQ4mOgQl23JBIZCVAr4+Xf4nEop1RN014F/aWafMcY8AjzS1cEcrcQMsOE1O8NsxLyuDkMppXxOd23JZACpDbZTgINeioWoEM/CZQBf/RNWL/BWKEop5VO6a0vmG2CoiAwEMoErgau9FUxUiIstBwvtxnVvQlCkt0JRSimf4vMtGRFZCHwFDBeRDBG52RhTA9wOfAhsBV41xmz2Vox9IoPIKa6k1m0gNBb8u2vuVkqpzuXz34bGmKta2L8YWNzF4TQrOSaEGrchp7iCfrVZsPYFmHoLRPbzdmhKKeVVPt+S6Q5SYkIByMgvh9LD8Pnf4fB2L0ellFLe5/Mtme4gOToEgMz8cqaOmwS/yLKFMpVSqpfTJNMJUmJsksnIL/OMx+h/VqWUAu0u6xTBLn/iwwPJLCi3Oza9AR/+wrtBKaWUD+i1SUZELhCRfxcWFnbK+ZJjQu2YDED2ZtjxIbjdnXJupZTqrnptkjHGvGOMuTUqKqpTzpcSHUJmXZI585dwx2rw67X/eZVSCujFSaazpcSEkFFQjtttQJqreqOUUr2PJplOkhwTQlWNm8OllXbH23fCqie9G5RSSnmZJplOUj/DzNNllr8XSrK9F5BSSvkAnWvbSZKj7QOZmfnlTOofA9e/7eWIlFLK+7Ql00mSm7ZklFJKaZLpLOFBAUSHusgsKLM7sjfDgjmQme7dwJRSyos0yXSilJiQ+pZMXbn/mkrvBaSUUl6mYzKdKDk6hN25pXYjOhVu+sC7ASmllJdpS6YTpXie+jfGeDsUpZTyCb02yXR2WRmwLZny6lqOlFbZHV89Cv+cBpp0lFK9VK9NMp1dVgbqn5U5WigzLAH6T9dxGaVUr9Vrk4wTjpnGPG4+XPgIuIK9GJVSSnmPJplOVLdCZmbTZ2Xy92mXmVKqV9Ik04miQlxEBAXYxcvqbFsMD4+DjNXeC0wppbxEk0wnS44JqR+TAUibCWf/BqL7eyskpZTyGn1OppM1eiATIDgKTrnHewEppZQXaUumkzX7rIy7Fr5dCtlbvBeYUkp5gSaZTjasTwQllTXsy2swLlNbBa9cB6v+7b3AlFLKC1qVZETkLhGJFOtpEVkjIuc6HVx3NHlADADp+/Lrd7pCbOn/OX/0UlRKKeUdrW3J3GSMKQLOBRKAG4EHHYuqGxuaGE5EUADp+/Mbv5A8ySYbpZTqRVqbZOoWrZ8HPGOMWd9gn2rAz0+YOCCGNfvyj31x4yL49A9dH5RSSnlJa5NMuogswSaZD0UkAnA7F1b3Nql/NNuziymuqG78QmY67HjfTgRQSqleoLVTmG8GJgC7jTFlIhKL7TJTzZg8IAZjYN2BAk4dmlD/wtm/Af9AEG0EKqV6h9a2ZGYA240xBSJyLfBLoPPKF3uBE1WY60xIjUakyeA/QECQTTBaYkYp1Uu0Nsk8DpSJyHjgp8A+4HnHouoCTlRhrhMR7GJ4n4hjkwzA7uXwtzG2nplSSvVwrU0yNcY+XXgR8LAx5mEgwrmwur/JA2JYt78At7tJqyUmDfqMhuqyZt+nlFI9SWuTTLGI3AdcB7wnIv6Ay7mwur9J/WMorqxhZ05J4xdiBsA1r0LiSO8EppRSXai1SeYKoBL7vMwhIBn4i2NR9QDNPpTZUHk+FB3swoiUUqrrtSrJeBLLi0CUiJwPVBhjuvWYjNMGxIUSFxbImqYPZQLUVsM/p8KnD3R9YEop1YVaW1ZmPrAKuByYD6wUkcucDKy7ExEmtfRQpr8LZv8Rpv9P1wemlFJdqLXPyfwCmGqMyQEQkQTgY2CRU4H1BJMHxPDRlmxyiitIjGiyBPO4y70TlFJKdaHWjsn41SUYj7w2vLfXOmVIPACf7zzc/AH5e2Hp/2kFAKVUj9XaRPGBiHwoIjeIyA3Ae8Bi58LqGUb1iyQ+PIjlO3KbP+DgWvjsITi0oWsDU0qpLtKq7jJjzE9E5FJgJrYw5r+NMW86GlkP4OcnnDY0nqXbc6h1G/z9mpSTGXE+3LMZIvp6J0CllHJYq5dfNsa8DrzuYCw90unDE3hjbSabMgsZnxrd+EV/V32Cqa2220op1YMct7tMRIpFpKiZP8UiUtRVQbaViAzyLK7m9YkJpwyJR4SWu8wA3r0HXruhy2JSSqmuctwkY4yJMMZENvMnwhgTeaKTi0i0iCwSkW0islVEZrQnSBFZICI5IrKpmdfmiMh2EdklIvd64t5tjLm5PdfqbHHhQYxLjjp+kokdDAnDwa2rJyilehanZ4g9DHxgjBkBjAe2NnxRRBI9a9M03DekmfM8C8xputNT3uZRYC4wCrhKREZ1Tuid57RhCazdn09hWXXzB5x8O5z1a/DTCXtKqZ7FsW81EYkETgOeBjDGVBljCpocdjrwlogEe95zC/BI03MZY1YAR5q5zDRgl6flUgW8jC3i2Zr4HCv139TpwxJwG/ji2xamMtfJTIet7zgej1JKdRUn/+k8CMgFnhGRtSLylIiENTzAGPMa8AHwsohcA9yErSjQWsnAgQbbGUCyiMSJyBPARE9hz2M4Weq/qQmp0UQEB7B8+3G6zAA++Z0tNaPdZkqpHqLVs8vaee5JwB3GmJUi8jBwL/CrhgcZY/4sIi9j16wZbIwpOfZULWpuiUljjMkDvt/OuDtdgL8fpw6NZ/mOXIwxSEsrY170KARHabeZUqrHcPLbLAPIMMas9GwvwiadRkTkVGAM8CZwfzuukdpgOwXwydLGs4Ylcqiogg0Zx+mei0qBoAi7cqZWAVBK9QCOJRlP5eYDIjLcs+ssYEvDY0RkIvAkdhzlRiBWRNpSmvgbYKiIDBSRQOBK4O0OB++A2WP6EhTgx6L0jOMfWFkMT58DK//VNYEppZSDnO6XuQN4UUQ2ABOA/2vyeihwuTHmW2OMG7geu7RzIyKyEPgKGC4iGSJyM4Axpga4HfgQO3PtVWPMZsfupgOiQlzMHdOXt9ZlUlF9nFZKUATEDYGwhK4LTimlHCJ2VeXea8qUKWb16tVdcq0vdx3m6qdW8vCVE7hoQnKXXFMppZwgIunGmCknOk5HmLvQ9EFxpMSE8NrqE3SZgR2T2bHEjs8opVQ3pUmmC/n5CZdPTuWLbw9z4EjZ8Q/e+ja8dLl9dkYppbopTTJd7NLJtpvs9TUnaM2MOB8ueASSJ3dBVEop5QxNMl0sJSaUmYPjeW11Bm73cbrC/F0w+XoQgYIDsH9ly8cqpZSP0iTjBfOnppJZUM5HW7Nb94Z374ZFN0FNpd3WcRqlVDehScYL5o7py8D4MP720Y7jt2bqXPhPuGohBATZRPPQCFj9jPOBKqVUB2mS8QKXvx93nz2UbYeKeWdDKwoURPaDfuPsz5UlMGAGJE1wNkillOoEmmS85IJxSYzoG8HfPtpBdW0bCmKGxcHlz0LSRLut5WeUUj5Mk4yX+PkJPzp3OHvzynj9RKVmWvLFI/DchXbpZqWU8kGaZLzo7JGJTEiN5pFPdlJZ044WSUQ/iEzSJKOU8lmaZLxIRPjp7OEcLKzguS/3tv0E4y6HS5+EwNBOj00ppTqDJhkvO3lIPGcMT+Afn+4ir6SyfScpOgivfheKWzklWimlukivTTJdufzyifzivJGUVdXy8Cc723eCiiLY+wVkb+zcwJRSqoN6bZLpyuWXT2RIYgTXnNSfF1fuZ2d2cdtPkDgC7t4IQ87u/OCUUqoDem2S8TV3nTWU0EB//m/x1vadoG5cZvN/YfFPdWqzUsonaJLxEXHhQdxx5hCWbs/lrXWZ7T/RoY1wcG3bZpy5a+uP15I1SqlOpEnGh1x/chpTBsTwo1fX89GWdg7in/UruP4dcAVD9hZYMNcmneP5+jF44lR4eAJ89lD7rquUUs3QJONDggL8eebGqYxOjuKHL65h2fac9p3IFWz/ri4D44bAcLu940N49nwobNJSih0EqdNg4GkQldr+G1BKqSZ0+eUuXH65tQrLqrnqya/5NreEl26ZzuQBMZ1z4i1vw5ePwA2LISDQLiEQmQR+/p1zfqVUr6HLL3djUaEuXvjeScSHB/Gz1zdQVdOG2mbHM+pC+N7HNsG4a+GZufDpA43HYWproLIdM9yUUqoZmmR8VGxYIL+/eDS7ckp46vPdnX8B8YPZfwBXSH2SqamC/zcUPvtr519PKdUrBXg7ANWyM0f0YfboPjzyyU4uGJdEamwnlo8RgVEXNd4XEAgz76yv8KyUUh2kLRkfd/8Fo/ET4Tdvb6ZLxs9OuQcGzXL+OkqpXkGTjI9Lig7hnrOH8cm2HBZvPNQ1Fy3MhMJ2Lj+glFINaJLpBm6Ymcb41Gh+umg9O9pTdqYtaqrgH5Phy386ex2lVK+gSaYbcPn78a9rJxMaFMAtz6+moKzKuYsFBMIlj8OEq+GLh2H3cueupZTq8TTJdBN9o4J54trJZBVUcMfCtdS0Zcnmthp9CSSMgFVPwa6PoGA/vH2nTm1WSrWZJpluZPKAGB64ZAyf7TzMvW9sdDbRBATCbcvh3AegJBc2vgaZ6c5dTynVI+kU5m5m/pRUsgoq+NvHOygoq+YfV00kJNChJ/ZDY+3fKZPhf7dCSPSJ35P3LYTE1L9XKdWraUumG7rr7KH8/uIxfLItm2ufXunsGE2dugSTv6/xfmPg7Ttgw6t2+8Ofw1Nn1b9eelgrOyvVi2mS6aaumz6AR6+exMaMQi785xdsyuyCFT7XvgCPTIT8vfX7qkrh8E4oPGC3Z90Lcx6sf33BbHjzNudjU0r5pB6ZZERkkIg8LSKLvB2Lk+aN7cfCW0+iutbNdx77kv98tdfZBzaHzYVpt0L0gPp9QeFww3sw8267nTQRhs22P7vdMOOHMHa+3a4uh/TnnItPKeVzHE8yIuIvImtF5N0OnGOBiOSIyKZmXpsjIttFZJeI3AtgjNltjLm5I3F3F5MHxPLenacyc0gcv3prM99/IZ3DJZXOXCwsDuY+aEvSlObZ9WcqimwV5+YqOfv5wZSbYKhnWeg1z8OSX9nWj1KqV+iKlsxdQLNrCotIoohENNk3pJlDnwXmNPN+f+BRYC4wCrhKREZ1NODuJjYskKevn8p9c0ewdFsu5/5tBYs3Zjl70bydNtns/bz170k7Fa58AVwt1GBb/wq8dIVtASmlegRHk4yIpADnAU+1cMjpwFsiEuw5/hbgkaYHGWNWAEeaef80YJen5VIFvAxc1MxxzcV2gYj8u7CwC8YyuoCfn3Db6YN5985TSI4O4QcvruGHL64hp7jCmQv2nw53roUR81r/nj6j7MJoIvX7CjPsktEAVSVQmmtbQEqpHsHp3+a/Az8Fmv2nqTHmNeAD4GURuQa4CZjfhvMnAwcabGcAySISJyJPABNF5L4Wrv2OMebWqKioNlzO9w3rE8GbPziZn8wezkdbsznroeW8tHI/brePzPCqKIIVf4Gs9XZ7+Z/g6dlQUQhTb4ZbPrX7C/bDf3+gXWtKdXOOJRkROR/IMcYc9wk+Y8yfgQrgceBCY0xJWy7T/ClNnjHm+8aYwcaYP7bhfD1CgL8fPzxjCB/cdSqjkyL5+Zsb+c7jX/LZztyuqeR8Ip//HXYvsz+f8zuY/zwEN0n2Gath+/t29U6lVLfl2PLLIvJH4DqgBggGIoE3jDHXNjnuVGyCSQeKjTG3t3C+NOBdY8yYBvtmAL8xxsz2bN8H0JbE4ovLL3cmYwyvr8nkoSXbySqsYMqAGO46eyinDIlHpLkc3QXKjtgHNqFx11lTFYXHJh+llE/w+vLLxpj7jDEpxpg04Erg02YSzETgSew4yo1ArIg80IbLfAMMFZGBIhLouc7bnXIDPYSIcNnkFJb9ZBa/v2g0GfnlXPf0Ki785xe8tyGL8qpavtx1mD99sI17XllHYXm180GFxsLWd+C5C2zJmpYER9kHOdOf1ZI2SnVT3i4rEwpcboz5FkBErgduaHqQiCwEZgHxIpIB3G+MedoYUyMitwMfAv7AAmPM5q4KvjsJCvDnuhlpzJ+ayhtrMvn3it388KU1iNjv8QA/wQC5xZU8c+NUXP4OD9flbrPjLnUtmpZUlcDyP8PQcyF58vGPLTtSX84mMx1qKiH1pOanV9dZ9SQEhtmq00qpTudYd1l30dO7y1pS6zZ8uPkQ6zMKmJYWy0mD4nh/YxY/WbSBK6em8sfvjHW2O62y2Ga34MgTH1twACKTIXsTvPUDuOTfdqbazo+hshBGfwc2vQ7v3G0nDiQMg1eus5ML7lxnZ6tVldpk0tTaF+GD++CejfVdc+5am5jctfa8Yy7TGW9KNdHa7jJvt2SUl/j7CfPG9mPe2H5H910+JZW9eaU8uvRb0uLD+P7pg50LICjixMfUiU61f0cmgX+gHasBSH8GjuyGkRfZZ3DGzYfwRPvavL94HhT1s8nsX6fBsDkw+w9QlAWf/xXO/QMMPQeSJ9UnmOpyePY8mHyjTUpv3AIhsfUPlCql2kSTjGrkR+cMZ19eGQ++v42PtmRz1bT+nD+uH8Euhyo9t0VYfP0UZ7Cz0kqywT8AIvrA+X+tfy2ir/0DUFsF46+ExNF2e+/nsG4hTPou9B1bn5iKDoIrBMIS7HuHnG278waf0TX3p1QPpN1lvbS77Hgqa2r5z1f7eGnlfnYfLiUiOIDpg+I4aWAs0wfFMaJvBAFOj9k4rSQXwhPqtze8ZrvivveJTTxNuwqzt8CRb2HkBV0bp1I+SrvLVLsFBfjzvVMHcfMpA/l69xHeXJvB17uP8NGWbABCXP6MS4li0oAY5k9JZWB8M2Mdvq5hggEYchZM/wHEDW5+WvVHv7aldIbNtS0npVSraEtGWzKtllVYzqo9R1i7v4C1BwrYcrAQY+CKqancedZQ+kQGeztE5xRm2JpruhibUkCwVvSaAAAZYElEQVTrWzKaZDTJtFtucSX/+HQnL63cT4C/MG9MP84d3YfThiUQGthD/7VvjJ0qHRbXeJ8I7P3C/j3g5PrXSvNsYvLWg69KOUSTTCtpkum4/XllPLZsFx9sPkRBWTVBAX6cPDiO04clMGt4ImndsTutJe/cBRtfh3v3g6mFt++0XWyn3AOPzbCTCG7wrGpRnm9ntY3+DpzzWzsTLqKfnVzQWbI2ND+GVKfu91uTnOpkOiajukz/uFAevHQcD1w8hlV7j7BkczbLtuewdHsuvLOFtLhQZg1PZNbwBAbGh1Fda6hxu0mKDiEy2OXt8Ntm2m0QO8jz3Iyfnblm3Pa5mqsWQnif+mODomD81fZB0toau4xBWIJd5K0zvvT3fmGnW8/7i12tdNhsW+W6TtYGWHQTfOdfJ36QVSmHaEtGWzKO2Xu4lOU7clm+I5cvvz1MRXXjYtwRwQHcddZQvjsjjcCAbjpbra6rrKHaGjiwEtJmNt7/7VIIioSUTvrCd7th5eMw8Vp49CSYcTuc7Cn9l7Ueqspg2R/hjF9A/5M655pKeWh3WStpkukaFdW1rNxzhMPFlQT4C/5+wmurM1i+I5dBCWHcfsYQpgyIJTU2xHuFOzvL4p/Cqn/Bj7bXP6vT1KonbQvopNvafv41z8OI8xtPQqiphICg+u0X59sKCXdvPH5ZHaXaSbvLlE8Jdvlz+rDG04bPG9uPpdtz+P27W/nfV+36MtGhLqYMiOXiiUmcPbKPbzwE2laz7oXUaRCW2PzrxsCeFeCugUnXg6uZWXnPzIOYgXDxo3Y7ezMkjrLdYu/9GAoz4YwGSyXVJZjaGjvF+pInIG+XTTDV5VBeAJH9jrmMUk7Tloy2ZLyuptbNtkPFbMgoZENGAcu253KoqIKI4ADOGdWHCanRjE6KZGS/yJ4za83ttt1sLbXalv/ZTiKYfAMc2QP/mAxn3w8z77JdYX3GHNtC+fi3sGd546oIxsA/JkHfcTD/OcduR/U+2l3WSppkfE+t2/D17jxeT89g2Y5cjpRWAeDyF04eHM/s0X05e2QiiT3huZzSPPjkt3bxtq8fg8BwOPmOxsmnthrWvWhrr7XU/Qaw/mVbfbr4kJ3tljzJ7t/wmm3FpJ3i7L2oXkW7y1S35e8nzBwSz8wh8RhjOFRUwabMIlbtyWPJlmx+/uZGfv4mJEUFMzo5ijFJUUweEMOE/tGEB3Wz/6WP7IaNi2y5mtxtdmJAU/4u26I5kfFXQp/R8J/vQGVR/f5xl3dauEq1lbZktCXTrRhj2JFdwvIdOWzKLGLTwUL2HC7FGPATGNkvkkn9Y5jYP5qJ/WNIiwv1/YkEdevguN322Rv/Dk7rrqmy52h434UZdhxI181RnURbMqpHEhGG941geN/6pQKKKqpZu7+A9L1HWL0vnzfWZPCfr/cBEB8eyLSBsUxNs398srhn3SyxumdvOiog8Nh9G1+DT35vu8yi+3f8Gkq1krZktCXT49S6DbtySlizP59v9hxh5Z4jZBaUA/XFPaNDXbiNbRmNT4nmimmpJEb0gDGellQUQkkOxA9tvP/TB+wstonXeCeuOnULxaluQwf+W0mTTO+QWVDOmn35pO/LZ92BAsqrahEBt6f7zeUvzBnTj6GJ4WQVlpNVWEFCeBAXTkhixqA432v9dETdA6S11fD8RXam2tw/QXXZsauHHtpo/+471v695FcQlVL/fE/Th1GrK46dkr1nhV1srv/05uNZeJWtnHDt682fsy2qSuHtO2Dq9xrXkGvx+DLY9RGMuqh91+tIrN1cr+4uE5FBwC+AKGPMZd6OR3lfcnQIydEhXDA+6ZjXdueW8MLX+3kt/QDvrD9IfHggfSKDSd+bz2vpGcSHB3H+uH58Z1IyY5OjfH+M53g+/IVt0Vz6pB23ueE9+yDn6zfb1s41ixp/ab51u51Kfc1rdjt7k00IYMeQHhlvl0iY/j+w70tYdDNc/TL0G19/jv0rYekD8LO9dhE4gMoSm9BEYPCZ9sFUsONTz10As/8PBp3ejhsU2P6BfUapNUlm9dOw5Jfw/S+g75jWXeKT39sF7QbMgJevsbXoLnmi42NpPZRjSUZEgoEVQJDnOouMMfe381wLgPOBHGPMmCavzQEeBvyBp4wxDxpjdgM3i8iijtyD6h0GJYTz6wtGce/cERgMQQG226aiupZl23N4a91BXlq1n2e/3MvghDDmjOnLuJRoxiZH0S8quHslneAomyQy19ius6AI2/JIO8U+tFmWBx/dD+f+3o4VnfeQXeKgznVv1v9cXWrX14nwPOQZmWRbPJEpdrs4265YOvFa24qpSzDuWnhpPsSkwcWPwbRb6s9ZlmdbPWFN1vtprcBQ+NmextUPjmf6D2xLzl0Da1+wsR5P2RFY+x8ICrdJJnYgpJzwH/O9mmPdZWJ/88KMMSUi4gI+B+4yxnzd4JhEoNwYU9xg3xBjzK4m5zoNKAGeb5hkRMQf2AGcA2QA3wBXGWO2eF5fdKKWjHaXqdYoLK/m/Y1ZvLE2k9V7j+D2/NpEBgeQEhNKckwIA2JDGZsSxdjkKNLiwvDz89HkU1MFD4+HfuPg6lcav5a5xnahXbYAhp7T/msc2mRbJDe+D4kj6vcX7LdJ6Kt/2mKi46849r0Nu6CW/cl+kY+bf+Jr7v3ctpCGnmsnURzaaJPWoFm2tbZ7uS0g6gq2K6O6gm2SBVtNe/tiuGfziRNUdYVteQWGHv+4Hs7r3WXGZq8Sz6bL86dpRjsd+B8RmWeMqRCRW4BLgHlNzrVCRNKaucw0YJen5YKIvAxcBGzprPtQCiAqxMWV0/pz5bT+lFfVsvVQEZszC9mRXUJmQTn788pYsSOXyhrb7RMW6E9SdAh9o4JJiQllYv9opqXFMsAXplQHBMIV/2m+eyd5kq13FhLdsWtkrYMR59llEOrsX2mrRl/6JMy8s+X31v33qa2G3ctsYhpzmWf23XF8/bgtvzP0HJuo3rnLJtTvfwZL/2Drxd2XaY/96NeQvRFuXW4nHJz5Szj3gfoEU5RlWyyn/rj+utUV9vWmY06lebDtXZhwje+tmlpXZsiLHL26p6WRDgwBHjXGrGz4ujHmNREZCLwsIq8BN2FbJa2VDBxosJ0BnCQiccAfgIkicp8x5o/NxHYBcMGQIUPadE9KhQT6M6l/DJP6xzTaX1PrZmdOCRsyCtiaVXx0AsH6AwdZuGo/APHhQYxNjmR0UhTD+kZQXePmSGkVxRXVTB8Ux/RBcV3TAjpeF09HEwzYbqemXU/Jk21ZnIGtHGvxd9m1ecTv2MH10jybLIPqp7Jz+bM2IdXNUvvOk7aLTsTWiBs0q/48yZNg5Pn1x4Y3qDNnDGxaBF88DGMutV2MVaWw8l9w4Gu4aUnjL+59n8M7d9rux9aMA3UVY+CZufa+z/yF18LoktllIhINvAncYYzZ1MzrL2NbL4ONMbktnCMNeLdJd9nlwGxjzPc829cB04wxd7Q2Nu0uU05zuw27ckv4Zu8R0vfms/lgEbtyS6h1H/u71z82lCumpjJ9UCwD4sKICwv0fsvHFxQfgjdugTl/sqV1HpkIU26Es39jE0DRwWOnZ7dVaR4sugEmXgdjL7cJK7o/PHU21FTYpRTydsFZv2r8vqoyW7mhz2jfmmlWXQEf3w9JEyFpkm1VduI0ca93lzVkjCkQkWXAHKBRkhGRU4Ex2CR0P3B7G06dAaQ22E4BDnYoWKU6mZ+fMKxPBMP6RHDNSQMAO6lgz+FSQgP9iQkLJNDfjw83H2Lhqv385cPtR98bERTAwIQwBsWHMTA+nJgwFyKCAIdLKtmdW8qew6X0iwrmp3NGMCQx/Oh7K6prqXGbY0rtFFVUsy2rmGkDY+k23DW28nRxFvQZBaf/1M5KA1j9DCz5BdyytL5eW3uExNjutdA4myxi7GfFnD/aVkFLa/IEhrZ+Zlpr7FkB8cNsMj3RFOnszbYUUXTqsa+5gu3U9P0r4dFpcNnTtmXWxZwc+E8Aqj0JJgRYAvzJGPNug2MmAguB84A9wAvAbmPML5s5XxrHtmQCsAP/ZwGZ2IH/q40xm1sbp7ZklK/JLChnx6Fi9hwuZW+eTSK7c0s5WFhOw19XETs1e2B82NFnf747I42ZQ+J4b2MWSzZnU13r5o4zh3DLaYMICvBn+Y5cfrZoA4eKKvjBrMH8ZPbw7tNSaml8oSQXtr9nx22Cwo99vS0KDtiEFjuwbe8rzYMv/maX2u5Ioqup9EzKGG9bUxtegSsXHnvf7lrbjfjEqfZ/hNtWNE5GWevBL8C2rtxu+PpRu0prWFz7Y2vC6w9jisg44Dns1GI/4FVjzO+aHDMTKDLGbPRsu4AbjDFPNjluITALiAeygfuNMU97XpsH/N1znQXGmD+0JU5NMqq7qKiupbSyBrexD5FGhbiOrreTV1LJQx/t4OVV+3Ebu+ro3DF9KSqv4YPNhxicEMb4lGjeWJvJkMRwRidF8ta6g1w7vT+/u3AMfn5CSWUN+/JKGdE3En9fnRnnqyqL4aGRcO7vYORF9gs/NNYmjeIsO127tXK32wkG+1dC+rNw5YuNF6jb8jZ8/le4+lWoKIKqEkiaYKdXVxbZa71wKeRsg7vWOzbw7/Uk011oklE9yY7sYg4WlDNjcNzR532Wbs/h129tIiO/nFtPHcQ95wwjKMCPBz/Yxr+W72bmkDjKqmrZkFFIrduQFhfKzacM5NLJKT1n/Z6uUF1uWxh/HwujL4Hz/2q78t77EXz/c9vN15DbXT9z7chuO+171IXNH1O3yN3gM2HHh7DyCdvCaTjT7atH4cOf29mBgeH2nA0neGSm22MufdomwYbXbwdNMq2kSUb1BhXVteQWV5Ia2/jZjseXfctjS3cxtE84MwbHkRoTysvfHGDdgQIiggMYnRRJWlwYafFhDIgNpX9cKP1jQ4kI1qfbW5T+nH34NGG4nQq9aZGdNCBSX3anJBcWXgmn/q+d6r3+ZXjzNrjiBbvsQ1P/ucS2Wm75xG43N1ZTcMBO+Z50XfNxrXke3r8Xfp7pKSvUsenNmmRaSZOM6u2MMY3GZYwxpO/L57XVGezKLWHv4VLyPAvH1YkICiAxMog+kcEkRYeQEhNCakwo4cEB+Ivg7yf0iQxmaJ9wXD2p7ltHFByABbNtyZxhs+HFy23FgRHzbLda2RE72N/cGFlRlp3h1taxooaMsV13kUn12x0Yj/Op2WVKKd/VdOBfRJiSFsuUtPpxgKKKavbnlbH/SBn78srILqogu6iCQ0UVfLYzl+yiymbPHRjgx8i+ESREBFNV66a6xk1kSADjUqKZkBrN0MRwIkNcBAX4dZ8JCO0VGGafFeoz2tY7u/6d+i/5gCC7emlLjvdaa4nUJ5i67S6gLRltySjVYRXVtWQVVngmJhhq3IYDR8rYlFnIpswiCsqrCQzwI8jfj9ySSvYcLm30fpe/EB0aSP/YUAbEhpISG0pCRBAJ4UGEBPqzP6+U3YdLOVxSxcTUaE4bFs/ghPCen5h8mHaXtZImGaW6XmFZNeszCth3pIziimqKK2o4XFzJ/iO2tZRVWHHMe0ID/YkOcXHQ81p8eBChgf4YT7WqqBAXcWFBxIcHMbJfBBP7RzM6KeroDLyWNO0uVK2j3WVKKZ8VFeritGEtV1qurrXldnKLKymrqmVAXCiJEUGICAeOlPH5rsOk78unptaNnwgGKCir4khpFdsPFfP6mgzAtpASwoOICg0kKiSAmNBAYsICiQl1caS0im2HitmZXUJEcABnjUzk7JF9mJgaQ1iQf7NrCNXUutmVW0JOUSX9ooJJjgnRGXgnoC0Zbcko1ePkFFewZl8B6zMKyCmqpLC8msLyKvLLqskvrSK/rIrIEBfD+9ilvLOLKlix4zDl1bVHzxEU4EdkiIuYUBcxoYFU1brZmlVERbW70bViQl3EhwcR51mHaHxKNFPSYhjVzz5vVFnjJq+0imXbc1iyOZtVe44wfVAst5w2iBmD4rptK0q7y1pJk4xSvY/bbRBpPOmhorqWr3bn8W1OCaWVtZRW1VBUXk1+mU1OAoxOimJsSiRJUSEcKqogI7+crMJyDhdXkVda6dm23Xn+foLbmEZVGgbEhTItLZZPt+WQV1rF6KRIJvaPJjokkKgQF32jgkmNrZsmHoCfCH5y7OQMX6DdZUop1YLmKl0Hu/w5Y3giZwxPbOYdrXewoJz0fflszSoiwN+PYJcf4UEBTB8Ux9BEO1mhorqWN9dm8uLKfby3IYvC8mqaqZd6VKC/H2FB/oQFBRAd6iIxIpiE8CBiwgKJCA4gIjiAEJe/nVwR4EdljZuMfLsERUlVDeNTopg8IJYxyZFHH9LtKtqS0ZaMUsrL3G5DcWUNWYU2MRzIL6e8ypYQqnUbKmvclFbWUFpZQ35ZFTnFleQWV5JfVkV1bcvf4QkRQQQF+JGRXw7YWcvhgQGEBQUQHhzAo1dPYnjfiBbffzzaklFKqW7Cz0+ICnERFeJiRN/INr23orqWksoayiprqap1U1XjJsBfSGkwKcGOUeWzJauY4opqSipqKK2qITTQ+VaNJhmllOrGgl3+dpr2cQpQJ0YEM2dMP+aM6YSHOttI6z0opZRyjCYZpZRSjtEko5RSyjGaZJRSSjlGk4xSSinHaJJRSinlGE0ySimlHKNJRimllGN6fVkZEckF9rXhLfHAYYfC8VW98Z6hd953b7xn6J333dF7HmCMaXm9Bo9en2TaSkRWt6ZeT0/SG+8Zeud998Z7ht553111z9pdppRSyjGaZJRSSjlGk0zb/dvbAXhBb7xn6J333RvvGXrnfXfJPeuYjFJKKcdoS0YppZRjNMm0gYjMEZHtIrJLRO71djxOEJFUEVkqIltFZLOI3OXZHysiH4nITs/fMd6OtbOJiL+IrBWRdz3bA0VkpeeeXxGRQG/H2NlEJFpEFonINs9nPqOnf9Yico/n/+1NIrJQRIJ74mctIgtEJEdENjXY1+xnK9Yjnu+2DSIyqbPi0CTTSiLiDzwKzAVGAVeJyCjvRuWIGuBHxpiRwHTgh577vBf4xBgzFPjEs93T3AVsbbD9J+BvnnvOB272SlTOehj4wBgzAhiPvf8e+1mLSDJwJzDFGDMG8AeupGd+1s8Cc5rsa+mznQsM9fy5FXi8s4LQJNN604Bdxpjdxpgq4GXgIi/H1OmMMVnGmDWen4uxXzrJ2Ht9znPYc8DF3onQGSKSApwHPOXZFuBMYJHnkJ54z5HAacDTAMaYKmNMAT38s8auCBwiIgFAKJBFD/ysjTErgCNNdrf02V4EPG+sr4FoEemUZTQ1ybReMnCgwXaGZ1+PJSJpwERgJdDHGJMFNhEBid6LzBF/B34KuD3bcUCBMabGs90TP+9BQC7wjKeb8CkRCaMHf9bGmEzg/wH7scmlEEin53/WdVr6bB37ftMk03rSzL4eOzVPRMKB14G7jTFF3o7HSSJyPpBjjElvuLuZQ3va5x0ATAIeN8ZMBErpQV1jzfGMQVwEDASSgDBsV1FTPe2zPhHH/n/XJNN6GUBqg+0U4KCXYnGUiLiwCeZFY8wbnt3Zdc1nz9853orPATOBC0VkL7Yb9Exsyyba06UCPfPzzgAyjDErPduLsEmnJ3/WZwN7jDG5xphq4A3gZHr+Z12npc/Wse83TTKt9w0w1DMLJRA7WPi2l2PqdJ6xiKeBrcaYvzZ46W3ges/P1wNvdXVsTjHG3GeMSTHGpGE/10+NMdcAS4HLPIf1qHsGMMYcAg6IyHDPrrOALfTgzxrbTTZdREI9/6/X3XOP/qwbaOmzfRv4rmeW2XSgsK5braP0Ycw2EJF52H/h+gMLjDF/8HJInU5ETgE+AzZSPz7xc+y4zKtAf+wv6uXGmKaDit2eiMwCfmyMOV9EBmFbNrHAWuBaY0ylN+PrbCIyATvZIRDYDdyI/cdnj/2sReS3wBXYmZRrge9hxx961GctIguBWdhqy9nA/cB/aeaz9STcf2Jno5UBNxpjVndKHJpklFJKOUW7y5RSSjlGk4xSSinHaJJRSinlGE0ySimlHKNJRimllGM0ySjVjYnIrLqq0Ur5Ik0ySimlHKNJRqkuICLXisgqEVknIv/yrF1TIiIPicgaEflERBI8x04Qka8963q82WDNjyEi8rGIrPe8Z7Dn9OEN1oR50fNgnVI+QZOMUg4TkZHYJ8xnGmMmALXANdjijGuMMZOA5dgnsgGeB35mjBmHrbxQt/9F4FFjzHhsva26sh8Tgbux6xwNwtZiU8onBJz4EKVUB50FTAa+8TQyQrCFCd3AK55jXgDeEJEoINoYs9yz/zngNRGJAJKNMW8CGGMqADznW2WMyfBsrwPSgM+dvy2lTkyTjFLOE+A5Y8x9jXaK/KrJccer8XS8LrCGNbZq0d9r5UO0u0wp530CXCYiiXB0nfUB2N+/usq/VwOfG2MKgXwROdWz/zpguWdNnwwRudhzjiARCe3Su1CqHfRfPEo5zBizRUR+CSwRET+gGvghdpGw0SKSjl2h8QrPW64HnvAkkbrKyGATzr9E5Heec1zehbehVLtoFWalvERESowx4d6OQyknaXeZUkopx2hLRimllGO0JaOUUsoxmmSUUko5RpOMUkopx2iSUUop5RhNMkoppRyjSUYppZRj/j+K28lhPvwjgwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "poly_features_t=poly_features.t()\n",
    "fit_and_plot(train_features=poly_features_t[:100,0:4],train_labels=labels[:100],test_features=poly_features_t[100:,0:4],test_labels=labels[100:],no_inputs=4)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear Function Fitting\n",
    "Let’s take another look at linear function fitting. After the decline in the early epoch, it becomes difficult to further\n",
    "decrease this model’s training error rate. After the last epoch iteration has been completed, the training error rate is\n",
    "still high. When used to fit non-linear patterns (like the third-order polynomial function here) linear models are liable to\n",
    "underfit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "final epoch:train loss tensor(46.7403, grad_fn=<MeanBackward1>) test Loss tensor(54.2278, grad_fn=<MeanBackward1>)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZkAAAEKCAYAAADAVygjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd4HOW1+PHvu1pJq95sS7bcCzbGuAOmhF5sSAKkQCiBJPwu5N6QhORCgJSbkNwQckMSQgi9JCFAcOgYA8bEdIN7w93GRbZVLNvq0mp3z++Pd2VJtmSvtDs7K+l8nkePNXXPaNZz5i3zjhERlFJKKSd43A5AKaVU76VJRimllGM0ySillHKMJhmllFKO0SSjlFLKMZpklFJKOUaTjFJKKcdoklFKKeUYTTJKKaUc43U7ALf169dPhg8f7nYYSinVoyxdunSviPQ/2np9PskMHz6cJUuWuB2GUkr1KMaY7ZGsp9VlSimlHKNJRimllGM0ySillHJMn2+TUUqprmpubqakpITGxka3Q3Gcz+dj8ODBJCcnd2t7TTJKKdVFJSUlZGVlMXz4cIwxbofjGBGhsrKSkpISRowY0a19aHWZUkp1UWNjIwUFBb06wQAYYygoKIiqxKZJRimluqG3J5gW0R6nJhmllFKO0STTXe/dDbOvcTsKpVQfdeDAAe6///4ub3fhhRdy4MABByLqmCaZ7vJ47Y9SSrmgsyQTDAaPuN3cuXPJzc11KqzD6FWyu067ye0IlFJ92G233caWLVuYPHkyycnJZGZmMnDgQFasWMHatWu55JJL2LlzJ42NjXz/+9/n+uuvB1qH0qqtrWXWrFmcdtppfPTRRxQXF/Pyyy+TlpYW0zg1ySilVBTuePVT1u6ujuk+xw/K5udfOO6I69x1112sWbOGFStW8M4773DRRRexZs2ag12NH3/8cfLz82loaOCEE07gy1/+MgUFBe32sWnTJp555hkeeeQRLrvsMp5//nmuvvrqmB6LVpd11/q58NDpUFvhdiRKKcWJJ57Y7lmWe++9l0mTJjFjxgx27tzJpk2bDttmxIgRTJ48GYBp06axbdu2mMelJZnuSk6DzCIIBdyORCnloqOVOOIlIyPj4O/vvPMO8+fPZ+HChaSnp3PmmWd2+KxLamrqwd+TkpJoaGiIeVyaZLpr1Fn2RymlXJCVlUVNTU2Hy6qqqsjLyyM9PZ3169fz8ccfxzm6VppklFKqByooKODUU09lwoQJpKWlUVhYeHDZzJkzefDBB5k4cSJjx45lxowZrsVpRMS1D08E06dPl269tOzADnjmCjj7ZzB2ZuwDU0olrHXr1nHssce6HUbcdHS8xpilIjL9aNtqw393JadD3nBISXc7EqWUSlhaXdZdGf3ga0+5HYVSSiU0LckopZRyjCaZaDx2Acy/w+0olFIqYWmSicbAiZA3zO0olFIqYWmbTDQu/J3bESilVELTkoxSSvVA3R3qH+Cee+6hvr4+xhF1TJNMNObeAo/rMzJKqfjrKUlGq8uiUXgcpGS6HYVSqg9qO9T/eeedx4ABA5g9ezZNTU1ceuml3HHHHdTV1XHZZZdRUlJCMBjkZz/7GWVlZezevZuzzjqLfv36sWDBAkfj1JJMNKZ9A879udtRKKXc9sRFsDz83Fyw2U6vfNZO++vt9Jrn7XRjlZ1e+4qdrqu00xtet9M1ZRF95F133cWoUaNYsWIF5513Hps2bWLRokWsWLGCpUuX8t577/HGG28waNAgVq5cyZo1a5g5cybf+973GDRoEAsWLHA8wYAmGaWU6vHmzZvHvHnzmDJlClOnTmX9+vVs2rSJ448/nvnz53Prrbfy/vvvk5OTE/fYtLosGiuehjdug+8usyMAKKX6pm++1vp7UnL76ZT09tO+nPbTGQXtp7NaB7qMlIhw++23c8MNNxy2bOnSpcydO5fbb7+d888/n//5n//p8v6joSWZaOSPhIlfA6N/RqVUfLUd6v+CCy7g8ccfp7a2FoBdu3ZRXl7O7t27SU9P5+qrr+bmm29m2bJlh23rNC3JRGPoDPujlFJx1nao/1mzZnHllVdy8sknA5CZmck//vEPNm/ezC233ILH4yE5OZkHHngAgOuvv55Zs2YxcOBAx9tldKj/7g71r5Tqs3Sofx3qPz4qNsCvB8Hal92ORCmlEpImmWhk9LfdmPOGux2JUkolJG2TiUZ6Psy80+0olFIuEBGMMW6H4bhom1S0JBMLoZDbESil4sjn81FZWRn1BTjRiQiVlZX4fL5u70NLMtG6eyyM/6KOyKxUHzJ48GBKSkqoqKhwOxTH+Xw+Bg8e3O3tNcl0U4M/SENzkPyTboD+49wORykVR8nJyYwYMcLtMHoETTLddNOzy9m2t543f/BDt0NRSqmEpW0y3ZSZmkxtUwBEoLnR7XCUUiohaZLppiyfl5rGZpj9dXjkbLfDUUqphKTVZd2U5fNS2xRAJnwF07DP7XCUUiohaZLppsxULyGB+tGfJyNV/4xKKdURrS7rpixfMgC1DU3QsN+2zSillGpHk0w3Zfps6cWz+GH47XBoPOBuQEoplYA0yXRTVriKrLLfCXDBneBJdjkipZRKPNqY0E1Z4ZJMReZYxk05zeVolFIqMWlJpptaqstqGpqgbi80N7gckVJKJR5NMt2UGa4u85avhd+Ngs1vuxyRUkolHk0y3dTSu6wsqRBm/Q4Kx7sckVJKJR5tk+mmlpJMZTANTrre5WiUUioxaUmmm5I8hoyUJGobA1BTCnWVboeklFIJR5NMFDJ9XmoaA3DfCfD+3W6Ho5RSCUery6KQmWrHL+PCu6FglNvhKKVUwtEkE4UsXzI1TQGYdLnboSilVELS6rIoHBzuv6YM9n3mdjhKKZVwNMlEITPVaxv+X/2efa+MUkqpdrS6LAot75Th5Bsh0OR2OEoplXA0yUQhMzXZ9i4bcY7boSilVELS6rIoZIZLMqGGKti1DAJ+t0NSSqmEokkmCtnhQTKbPp0Dj5wFB3a4HJFSSiUWTTJRaBlapqpwBnztGcgc4HJESimVWLRNJgotw/1XpwygaIg+jKmUUofSkkwUWkZirmnwQ8lSqNzickRKKZVYNMlEoaW6rKYxAH+9CJY87nJESimVWLS6LAotr2Cu9Qfhimcgb5jLESmlVGLRJBOFliRT0xiAiWe5HI1SSiUerS6LQkt1WW1jwLbH6CuYlVKqHU0yUchI8WIMdiTmxY/C7GvcDkkppRKKVpdFweMxZKaER2I+9XqYeDmIgDFuh6aUUglBk0yUMn3hkZjzR7gdilJKJRytLovSwbdjNlbB2legapfbISmlVMLQJBMl++KyANSU2nfKbP/I7ZCUUiphaHVZlDJ9yVQ1NEPecLjhfa02U0qpNrQkE6Usn5faxmbwpsLAiZCa5XZISimVMDTJRCkrNVxdBrB5vv1RSikFaHVZ1A42/AO893swHhh9rrtBKaVUgtAkE6UsXzL1/iCBYAjvlx6C5Ay3Q1JKqYShSSZKLe+UqWsKkpM71OVolFIqsWibTJSyWob7b2qGvZvg4wfBX+9yVEoplRg0yUSp3UjMu5bBG7dCtT6QqZRSoNVlUWupLqttCsC4i+CWrZCe73JUSimVGDTJRKndcP+p+ZCa6XJESimVOLS6LEpZvmQAqhubIRSEhffDtg9djkoppRKDJpkoZbWtLjMe+PevYMNcl6NSSqnEoNVlUWpXXWYM/HAd+HJcjkoppRJDryrJGGMuMcY8Yox52Rhzfjw+Mz0lCY+hdWiZtFx9aZlSSoUlfJIxxjxujCk3xqw5ZP5MY8wGY8xmY8xtACLykoj8B/AN4PI4xdd+aJlNb8H7f4jHRyulVMJL+CQD/BWY2XaGMSYJ+AswCxgPXGGMGd9mlZ+Gl8dFli/ZNvwDbFkAix+zr2FWSqk+LuGTjIi8B+w7ZPaJwGYR2SoifuCfwMXG+i3wuogs62yfxpjrjTFLjDFLKioqoo4xq+UVzABn/AhuWq1VZkopRQ9IMp0oBna2mS4Jz/sucC7wFWPMtzvbWEQeFpHpIjK9f//+UQfTrrosLRc8PfXPqpRSsdVTe5d1VEwQEbkXuDfewWT6vFTW+ltnLH4UKrfCzDvjHYpSSiWUnnrLXQIMaTM9GNjtUixk+ZJbSzIA+z6DstXaLqOU6vN6aklmMTDGGDMC2AV8DbjSrWCyfF6qGppbZ5z/v9omo5RS9ICSjDHmGWAhMNYYU2KMuU5EAsCNwJvAOmC2iHzqVowDslLZV+fHHwi1BO1WKEoplVASviQjIld0Mn8ukBDjtxRl+wAor2lkcF66nfnOXbBzEXz9BRcjU0opdyV8SaYnKMqxSaa0qrF1Zlo+ZA+EUMilqJRSyn0JX5LpCQ4mmeo2Seak612KRimlEoeWZGKgpbqsXUmmhb6KWSnVh2mSiYGctGRSvR7Kqg9JMq/fCg+c4k5QSimVACJKMsaY7xtjssPDtjxmjFkWr1GOewJjDANzfOw5tCQz6myYfBUEmzveUCmlerlISzLfEpFq4HygP/BN4C7HouqBCrN9h5dkjrkAzrgFkpLdCUoppVwWaZJpefDjQuAJEVlJx0O79FlFOb72Df8tAk1QVxn/gJRSKgFEmmSWGmPmYZPMm8aYLED75rZRlO2jrLoJOXQomfumw5u3uxOUUkq5LNIuzNcBk4GtIlJvjMnHVpmpsKIcH/5AiP31zeRnpLQuOOM2yCpyLzCllHJRpEnmZGCFiNQZY64GpgJ/ci6snqelG/Oeqob2SWbKVS5FpJRS7ou0uuwBoN4YMwn4EbAd+LtjUcWBMeYLxpiHq6qqYrK/wvADmYc1/odCULkFGg7E5HOUUqoniTTJBMQ2NlwM/ElE/gRkOReW80TkVRG5PicnJyb7a30gs6n9gop18OepsOmtmHyOUkr1JJFWl9UYY24Hvg58zhiTBGi/3Db6Z6XiMRzew6xgDHzxPhg6w53AlFLKRZGWZC4HmrDPy5RiX3X8O8ei6oGSkzz0y0yltKqh/QJvCkz9OuQO6XhDpZTqxSJKMuHE8hSQY4z5PNAoIj26TcYJ9lmZpsMX1O2FLQviH5BSSrks0mFlLgMWAV8FLgM+McZ8xcnAeqLCbB9lHQ2SueIpePISqN8X/6CUUspFkbbJ/AQ4QUTKAYwx/YH5wHNOBdYTDczxseizDhLJcZdC8XRIyYx/UEop5aJIk4ynJcGEVaIjOB+mMNtHVUMzDf4gaSlJrQtyh9ofpZTqYyJNMm8YY94EnglPX06CvPo4kRzsxlzdyIh+Ge0X7lwEzQ0w8gwXIlNKKXdElGRE5BZjzJeBU7EDYz4sIi86GlkP1PY1zIclmbd/GU4yb7sQmVJKuSPi1y+LyPPA8w7G0uMVdfbUP8BFf4DUHv38qlJKddkRk4wxpgaQjhYBIiLZjkTVQ7WtLjtM/2PiHI1SSrnviElGRPTWuwsyUr1kpXop7agbswiseBoyB8CY8+IfnFJKuUB7iMVYYY6v4yRjDHzwR5tolFKqj4i4TUZFZmBnb8gE+MZrtiSjlFJ9hJZkYqww29dxwz9AVqEt0SilVB+hSSbGBuXYJNMUCHa8wju/hY/ui29QSinlEk0yMTa6MIuQwNaKuo5X2L0cyj6Nb1BKKeWSPtsmY4z5AvCF0aNHx3S/44psh7yNZTUcO7CDHt5fexo8mtuVUn1Dn73axfrNmC2GF2SQnGTYUFrT8QqaYJRSfYhe8WIsxethZL9MNpZ1kmQA5v4IXvx2/IJSSimX9NnqMicdU5TF8h37O18hLQ88+qdXSvV+eqVzwNjCTF5duZvapgCZqR38ic+6Pf5BKaWUC7S6zAFji2yD/6YjVZkB1JTFIRqllHKPJhkHjC1s7WHWqffuhj9NgqajJCKllOrBtLrMAYPz0khLTmJ9Zz3MwA6SmZwWv6CUUsoFmmQc4PEYjik8Sg+zgZPsj1JK9WJaXeaQsUVZbCitPfJKwQBsnAc1pfEJSiml4kyTjEOOKcxib20TlbVNna9UtROe/iqsejZ+gSmlVBxpknHI2IPDyxyhNJM/Aq6dAyf9Z5yiUkqp+NIk45CWHmYbSquPvOKIz4E3JQ4RKaVU/GmScUj/rFRy05PZcKSSTIvFj8KSJ5wPSiml4kyTjEOMMYwtzDpyD7MWG16HjW86H5RSSsWZdmF20NiiLF5ctgsRwRzpjZiXPQkp6fELTCml4kRLMg4aV5RNTVOAnfsajryiJhilVC+lScZBEwfbd9Ws2nXg6CsvegSeuAhEHI5KKaXiR5OMg8YWZZHi9bCqpOroKyenQ3o++Dt5bbNSSvVA2ibjoOQkD+MHZrNyZwQlmSlX2R+llOpF+mxJxhjzBWPMw1VVEZQyojBxcA5rdlURDEVYDdYQQUJSSqkeos8mGRF5VUSuz8nJcfRzJg7Opc4fZGtFBM/LrH8NfjcKytc5GpNSSsVLn00y8TIp3Pi/MpJ2meLpcOL1kJbvcFRKKRUfmmQcNrJ/JhkpSawqiaAaLKsQZv7G/quUUr2AJhmHJXkME4pzIivJtChdDevnOheUUkrFiSaZOJg0JJd1e6rxB0KRbTD/DnjzxxCKcH2llEpQ2oU5DiYOzsEfCLGxrIYJxRF0NLjobvDlgEfvAZRSPZtexeJg0uBcAFZG0i4DkDcc0vKcC0gppeJEk0wcDM5LIy89mVU7u9AuU1sOf78E1r3qXGBKKeUwTTJxYIzh+MG5kZdkwHZjbm6wP0op1UNpkomTSYNz2FReS4M/GNkGSV647k2YeJmzgSmllIM0ycTJ1GF5BEPC4m37urahCJQscSYopZRymCaZOJkxooAUr4d3N1Z0bcMVT8Gj50DJUmcCU0opB2kX5jhJS0nipBH5vLuxgp91ZcPxF9vSTNHxToWmlFKO0ZJMHJ05dgCby2sp2V8f+UapWTD161C1E2ZfCxUb7Pzq3fqCM6VUwtMkE0dnHNMfoOtVZgApmbBjIez7DHZ8AvdMhI1vxjhCpZSKLU0ycTSqfwbFuWm8u6EbSSarEH64HsbOhOKpcMqNWoWmlEp4mmTiyBjDGWP789GWysjHMWurZZiZpGQ49xeQUxzL8JRSKuY0ycTZmcf0p7YpwNLt+6PfWeUWeP1WCDZHvy+llHKAJpk4O2V0P7we0712mUPt3QjLnrSvBlBKqQTUK5OMMWakMeYxY8xzbsdyqMxUL9OH58UmyRwzE25abdtolFIqATmaZIwxucaY54wx640x64wxJ3dzP48bY8qNMWs6WDbTGLPBGLPZGHMbgIhsFZHroo3fKWccM4B1e6opq26MbkfGQEaB/b2xOvrAlFIqxpwuyfwJeENExgGTgHVtFxpjBhhjsg6ZN7qD/fwVmHnoTGNMEvAXYBYwHrjCGDM+NqE757zxAwB4bdWe2Oxw3s/godMhGIjN/pRSKkYcSzLGmGzgdOAxABHxi8ihwxCfAbxsjPGFt/kP4N5D9yUi7wEdDfp1IrA5XHLxA/8ELo7dUThj9IAsjhuUzcsrdsVmhyPOgClXQ6ibSSYYgPd/Dy99JzbxKKVUmJMlmZFABfCEMWa5MeZRY0xG2xVE5F/AG8A/jTFXAd8CujLscDGws810CVBsjCkwxjwITDHG3N7RhsaYLxhjHq6q6sI7XmLoksnFrCyp4rO9ddHvbMy5cPrNkOyLfJumWqjba1/xbDyw/jUYfqpdFgraH6WUipKTScYLTAUeEJEpQB1w26Ericj/AY3AA8AXRaS2C59hOpgnIlIpIt8WkVEi8puONhSRV0Xk+pycCF6H7IAvTBqEMfDS8hiVZkRg67udv+Rs67tw/ylQU2qn1zwHvxsFNbvt8zfXvAKTr7TL3v89PHmpTURKKRUFJ5NMCVAiIp+Ep5/DJp12jDGfAyYALwI/78ZnDGkzPRjY3fVQ468ox8fJIwt4ecUuJBZjkInAG7fDpy/Z6VAI/v1r2LPSTqflQWZ/aAjXWA6ZAbP+DzLsUDekZrbuK6sICieAJwHGT/XXgb8LY70ppRKKY0lGREqBncaYseFZ5wBr265jjJkCPIJtR/kmkG+M+d8ufMxiYIwxZoQxJgX4GvBK1MHHySWTi9lWWc/KkhhU2Xk88I05cPZP7HT9XvjwHtgVfkXAwIlwzcswYJydHjAOTroBvKmH72vqNTDzzq5Vvzmhbi/8cQIsfqTzdZb9Hf7xFWiqiV9cSqmIOd277LvAU8aYVcBk4M5DlqcDXxWRLSISAq4Fth+6E2PMM8BCYKwxpsQYcx2AiASAG4E3sT3XZovIp44dTYzNPL6IFK8ndlVm6fmQP9L+npoFN2+E6d/q/v72boInvwS15bGJrytqymDVbDjuUhh2WufriYAE7QCiSqmE42iSEZEVIjJdRCaKyCUisv+Q5R+KyOo2080icthtq4hcISIDRSRZRAaLyGNtls0VkWPC7S+/dvJ4Yi3bl8w54wYwZ9VuAsFujGV2JMlptoosGqGAfbXAvq2xiakrtr0Pb94OJ1wHg6d1vt60a+Gcn9sSjVtE7PA+2xe6F0NPF4rx978v2/4RvPU/CfMqkF75xH9PcvHkYvbW+nmnOyMzO23AsfC95TB0Rvw/+/ivwPdXwoDxttrswz8d3uOtsdr+R1r7Esy9BZq78HCrCDx7Naz8J6x5Hja91f1Ya0rh0xdh5yfw4n/a/+Qqcgv/As9c3rXzpzq38U37/6W2zO1IAE0yrjvn2AEMzPHx2AefuR1Kx7wp9uK+f1v8PztvuB3VYPuH8NbPYeei9sufv872gjv5Rrh5Q9fakBqrbCeIphp47/ew9K/djzN7oB3eZ+o1ULIISg8bmKJrytbCoiO0Q/U2yWmQkgGfPADv/8HtaNzX3GBvoJobYc+qrm9/9k/hZ5W2A08CMDHp2dSDTZ8+XZYsWeJqDA+9u4XfvL6eOd89jQnF7nSpPqLnrrMX+JtW2Yu+07Z9AGtegLN+YofNCQVtlV2/Me3XW/4PkJC9uHdHKGSPp2YPZBaCJ6nr+2huAK+v9e/S3Bhdh4mt78JH98K2D+G/10Nabvf3lehqyyHTjn6BiL1pCDbD5U+6G1c0tr4LGf2g8Lju72P1c/DC9TBosv3ef38l+CK8LlTtsj1GvSn2+91c377naAwZY5aKyPSjraclmQTwtROHkpGSxKPvu9D2EYlp34Dz7rAX9Bb1++CRs2HT/CNvu2sZVJV07fP2brTP+6SEn931JLUmmPI2IxNNubo1wax7Fd67++j7FoGF99uSjMdjk0P2IPsZzQ22w0FXvPljeOy81qq8lgSzewXsXt61fQHM/wUc2AE/XBv7BJNI7R4fPwj3TYf94X4+xsClD/fsBNPcAC/8B7z72+j2UzjBPlz9pUfg4vsjTzAAL/8XPH6BHcXjwVPh7TuiiyUGNMkkgJy0ZC4/YShzVu1hT1WD2+EcbsTnYMKX2t/p11XYC37eMDvt72DkgrJP4ZGz7F15V0z/lr2LP7REsH0hPHAKrHzWlqzafua2D2D5k0e/kFastx0KVs1uPz8Ussli9jVdazAdMgOOuaD93yYYgNlftwmjq65+Hi570vYUjKWF98PfvpAwjcGMnQlTr7UJvkVS+LksJ0abqN/XmtCcEGy2HWWufh4uefDw5c2NkSf5AePgrB9DwSgYd6Gdt/0jewxHc/KNcNpN9m854csw/HORH4NTRKRP/0ybNk0SwY7KOhlx2xy587W1bofSsZoykXVzOl4WDIo8fLbIaze3nx8KiXx4r0jDgfB6gY63P7AzshiCAZH3/2hj+dUAkbk/al3mr7efF4nyDSJNdYfP3zhPZMuCyPZRV3nkz9u9QqR+f2T76sj+HSJPXS6y/ePu76Ot2d8Quf/U2OxLxJ6DripbJ/Lu7468ztK/ifz+WHs+Y+mNH9vvTG2FyOLH7OdE+n2JxMa3RH7ZX2TnEjsd8Ivs325/X/OiyC9yRXYts9P+epGqXR3vp3avyO6V9v9Ui6Zakd+OEHn2663zYhl7NwFLJIJrrJZkEsSQ/HRmHT+QpxftoLYpAUdTXvI4/PMq29Nr7+b2pQgJwuhzYGj4TQ615fbHGDjlu7a4v/VdePA0W2fc1vaFcO+U1uFwnvsWPHIOBPyHx+BJsndp6QVw1b9gepu3OSSnRd5e1P8YSEk/fP6Y82Dkmfb3z97v/PUJezfDn6faNqHODJxkq7tEOi7ldeSjP9sx5MB2P6/cBLWlkW17NF99Aq5708bSnZJC+TrY8m/7e+lquOd426OuK1Y9C588BLVH6EmZPxJGndX1IY2qSmDBnZ2XFk7+Dnz+j/a7s+F1WDendVkkJYSjyR0KJ/5Ha1vMc9+yo6OHQjDidDjh/0HeCPt9eOgMeO3mjvez9iV46HPtHxtIyYAr/wXnh5/QeP/38OdpraXSlvg/fQmqDxnwpLnBdu8/sBPXRJKJevNPopRkRESW79gvw26dI3+av9HtUA53oERkzyp7B/Xg50QePb/zdefeKvKnKe1LLiVL7DaH3gH7G0Tm/UykocpOL3pE5N+/7l6MH95r71g78+pNR17eoqZc5H+LRF7+bsfLg0F7jOUbjryfUEjkH18R+efVdnrjPJFfDxLZ9mHr8pbSTjBo/2av/qD99rFUukbkN0NE1r3WOq9i09G389eL3HeiyJ9PaI359dtsaU5EZMFvbEmpMy3HEQyIVJdGFmtdpciOT46+XnOj/ffDP4vcOVikcsvRtwkGRBqr7e9Vu0R+kSey/Gk7HfB3XMrtqs/eF3nh2x0vW/Uvkc8+6HhZTZldfqRz/+nLInP+256XPavtcS961JaW3v5V+3W3f2y/cx/9pXvHcQREWJJx/SLv9k8iJRkRkW8/uUTG/nSu7Nof4+qCWAmFRLYvFNn8dufrLLhLZNP8jrcVsf/Jn/xS96pcjmTuj0SevsImxA/vbZ/kQiGROT8UefMnke1ry4LWxNdi7Su2OqMrFj5gq2dCIXtBe/UHItV77LLXbxe576TWC14wKNJY0377UKi1urG7Fj1qE2zAL/LK9+3Ngog9Rz/PEVn/uj3ejfM634e/3ibfjrx3t8gLN9jf6/eLVLS5SSpZKvLg6ZFd/NuCNKpVAAAU1klEQVR693ciP8+2VUUitqrp0OrWYEDk8Vn2xiEUaq12bXuB3rvZJsADJR1/Tm2FyDu/tVV5IiJb3xP5y4zOE03JElv9FWhunVe5VaR8fdeOL1aaakVe+Z79TlVsFKnaffg6B3a2Vr9t/1hkxTMx+WhNMj00yeyorJNjfjJXbnx6mduhHK5ik8hbvxBpbopuP/t3iNw7LbJSRXesft5eoErC9eMrn22tK++qQLPdT22FyP8ObF/SiNaWd2yp7Uh3rc9eYxNRZ+1ZkZj/S5G/fv7w+U219gLrb7Btao+ce3gsJUvbX1CPJBSy+7n/1Nb9bFkg8tAZXW+f2rtZZNNbtqRSuUXkl/1siamtgN+Wgg+9aC5/WuSPx9tEseZFkf8bHfkNzZYF9m/VNqHuWt6aOF/9gS05tG0zmfPfIr8q7FoJqK7SHk/bkl3lVpHVz7Um1ljz14v8cYLIExfFZHeaZHpokhER+f28DTLs1jnyydZKt0Npb+2r9uK9Z3X0+wqF2v9HjSV/Q2uja6DZVkM9fUX39vXWL2yD7v4d9m760NJNLIRCIg+dKbL074cv2/CGyJK/RpdkDlW1yybitqpLW4+t4YCNqabMVhu+fnvk+976bmsDd4tYVPste1Kkfp/9/fXbbfVYZxY/bkutLVqq1CLVNt6W6uH7T7Hf1+bG1mrSYNAmsardIhve7Npn7N3cvppOROSDe+z/r0irFLsj4I9ZNawmmR6cZOqbAnLynfNl1j3vSSDofi+SgwLNIp887MyF1kkNVd3/j1tbIbJydmzjOZS/wd5tr3y2a9t9+GeRsm70Rnzh2/Zi1lJNd6inv2bv5kMhkU9fsgm2q7Z9ZC/2TvSCeuoy2ybkpIYqm6iq99jqptI1h6+zcrb9O3ZUNRyJlh5mdZU2UQWabc+yHiLSJKNP/CfAE/8dmbNqNzc+vZwfXziO608f5XY4ymn+utaHTw8VCsGnL0Byun1mKdBk32Z6/8n2+aWZv7Hv3Ak02udrqkpsz6PcIfDu72Dbe/D1l1qf5fHXwY6PYdTZHffIW/G03Vc0I3i/8l1Y86IdCbyjnnyJrmKDfdj48/fAxK92vI6IHSfsmAuiGwnjiQvt+fzGnKOvm0AifeJfk0yCJhkR4T//sYx5a0v5+7dO4rQx/dwOSblFBB46HbKL7ft/ytbAf4Yfzsvob0diuH8GDD4BLr4P7p1qu2lf/Twsfsy+U+iS++Mbc8VG+7qJ7IHx/dxYqqu0wxo5beM8m6TGnOf8Z8WQJpkIJWqSAahtCvDl+z+itLqRV248lWEFndzpqt6vOjy+2u7ldtSCKVe1X774MTtq9rBT7IjSBaNa3y2klAM0yUQokZMMwPbKOr5434cUZqfywn+dSmZqArwSWfUMzY12qBOHBkhUfZsOkNlLDCvI4C9XTmVzeS0/eHYFoVDfvilQXfDXC+E3xXZcLaVcokmmBzhtTD9+etF43lpbxu/f2uB2OKqnuOgPcOlDkJTsdiSqD9O6lx7im6cOZ2NZDX9ZsIVjCrO4eHKx2yGpRDdosv1RykVakukhjDH88uIJnDgin1ueW8XS7fvdDkkppY5Kk0wPkuL18ODV0yjK9nHNY5/w4ea9boeklFJHpEmmh8nPSGH2DSdTnJfGN59YzNzVe9wOSSmlOtVnk4wx5gvGmIerqqrcDqXLinJ8zL7hZI4fnMN3nl7GXa+vZ3tlhO8sUUqpONLnZBL8OZkjafAH+dHzq5izajcicMLwPK47bQQXHFeEiWaYC6WUOgp9GDNCPTnJtNhT1cCLy3fxryUlfLa3jvPHF/KrSyZQmO1zOzSlVC+lSSZCvSHJtAgEQzz2wWf84a2NpHg93DZrHJdPH4I3qc/WiiqlHKJP/PdB3iQPN5wxijduOp3xA7P5yYtrOO+P7/Hyil06UkAftGLnAZ7+ZIfbYag+Tksyvagk05aI8Pa6cu6et4H1pTUU56YxdVgek4fkcnxxDqMHZJKfkXJw/WBIaA6G8CUnHXGfG8tqGTMgE49H23wS2e4DDVx47/scqG/mviun8PmJg9wOSfUykZZk9In/XsoYw7njCzl73ADmrN7D66v3sHTbPl5dufvgOvkZKQzISqWyzk9lbRMAxxfncOrofpw2uh8njMgnOVzVtvtAAz95cTULNlRw4fFF/OGyyUdMSLEQDAk1jc3kpqccfeVeLhgSHn5vK8V5aXxx0pEThj8Q4jtPLyMQFI4Nl2inD8unKEfb6FT8aUmml5ZkOlNe3cine6rZUl7LlopaKmr8FGSk0D8rFUH4ZOs+Vuw8QCAk5KYnc+6xhYzol8ED72whGBJmTSjiheW7OHFEPo9cM52cNGfGxdpYVsN/z17JhtIavn3GSP7rrNGHJbXapgD3vLWROav28OVpxfzXmaPJ6MIo1YFgiEWf7WNAdiqjB2R1O9aWUuPynfu56PhBjB+U3e19daQpEOSHz67ktfAzUV+dNpg7Lj6O9BQvB+r9vLR8F02BEDMnFDGsIINfvrqWxz/8jPuvmsq4oiwuuvcDpg/P42/fPDEhS6DBkLB0+3721/s5Z9wAbUOM0paKWrZX1nH6mP6O/i214T9CfS3JRKK2KcCHm/fy5ppS3lpXRk1jgFNHF/CbSycytCCdl1fs4uZ/rWREvwwunTKYjNQk0lO8BEMhGvxBGgMhMlO9FGX7KMz2kenz4vUYkpM8bCqv4cPNlXy0ZS/BkHDSiAJmjMxnbFEWyUkevB7D88t28ce3NpLp8zJ9WB7z1pYxrCCdWy4YS3FuGmkpSWworeHOuesor2li6tA8lm7fT1G2j1suGMu0YXnkZaSQ7fNS3RCgvKaRipomjDGkpSThMTB/bRmzl5RQWt0IwInD87nypKF8bkw/8tJT8HgMIsLuqkY2lFbjDwhji7IYmp9OUpsL9dLt+7jr9fUs3tY6zM+0YXlcfsIQJg7OYUS/DFK9nZf4GpuDbCyrYf2eGtbuqWbtnmo2ldUwtsiOT3fa6H7c8txKPt66j9tnjaOuKcCfF2xmdP9Mjh+cw2ur9tAUCB3c39jCLDaU1fCNU4bziy8eB8A/Pt7OT19aww/PO4azxw0AIMljKMhMoSAjlaTwsdb5g+yv81Na3cjuAw3sq/MzvF8GE4tzKMhMjei7U93YzCdb97FwSyW+ZA8zRhYwfXge6Sneduts21vHZ3vrWLxtH2+sKWNvuCQ9rCCd7549hksmD2p3gTxQ72d9aQ079tWTkeIlNz2ZnLRk+mWmkp+RQor38ItpY3OQmsYAjc1B/MEQzcEQGSle+mWmkpbS+TkJBEOs21PDku37aGgOMmVIHpOG5Bw8hmBI2H2ggWU79rNs+3627q0j25dMTnoy/TJSOK44h8lDctv17gyGBI+xNQzBkLBk2z7eWlvGe5sqCASFjFQvGalJNDSH2F/nZ3+9n+LctIO1CuMHZVOQkXLEpLG5vJY//3sTr6y0jzQMzU/nO2eN4tIpgzv8+0RLk0yENMkcmT8QYse+Okb1z2z37M1Hm/dy4zPL2Vfn7/I+k5MMU4bm4fUYlm7f3+4i2WLWhCJ+dckE+mWm8tHmvfz0pTVs3dv+gdMJxdn86uIJTBmax9Lt+/jFK2tZvav14Vpj7EslO2IMnHFMfy6bPoSd++p5ZtEOtlXWA+D1GPpnpVLbFKCmMdBuO1+yh/5ZqTQ1h2gIX8T6Z6Vy07ljOH98ES+v2MVTn+zgs3CsHgODctPwegwtoRjAYwyBkFCyv56WPhnpKUkcOzCbkf0yWLJ9/8F9eD2Gu786iUum2EFRP9i0l5ueXU5jc4hLpgziyhOHkZ3m5fXVpcxZvYdsn5fHrj3h4IVFRPjWXxezYEPFYX8Hj4HMVC91/iDBI3QOKcr24U0yNDaHaAoESfIYUr0eUrwevB4PBhBgx756giEh1eshGBICIcEbTmiN4b+Zv835TktO4qxx/Zk1YSApXg/3vr2JT3dXk5ueTEaKN/yZQcqqmzqNDSDL5z1YtduSMP0dfK9aZKQktUs0HmNvglK8HsqqG6n3B9utn+QxFGX7qG5sbvedyEhJYtSATOqaAlQ1NLOvzn/wfOZnpBASob7JJjmA1PA5aQqESEnycNLIfHLSkqlrClDXFMSXkkR+OIFuqahj0bZ9B4/DGCjISCEtJYlQyCYuQfAYgwFKqxtJ9SZxzSnDmFicy4PvbmH1rioyU70Hb648xtj1w78/cs10xhZ1rxSvSSZCmmS6T0RobA5R2xSg3h/Am+QhPTmJ1GQP1Q0ByqobKa1upN4foDkgNIdCDMpN48Th+QertZoCQVbsOHDw4hQICUPy0zl9TL92Sa0pEGTlzirq/AEa/UFSvB7OHDugXakiFBIWbq2ktKqR/fV+qhqayUlLZkC2j/7hO/HG5iCNzUEmDsmlODet3baffLaP9aXVlNc0UV7dRFqKh7FF2RxblIU3ycPG0ho2lNVQWduELzkJX3ISQ/LTueLEIe3u1EMhYUNZDZvKa9lcZu++Q2IvEi0X45DY34f3y+DYoizGDcxmWH76weosEWH1rirmryvn1FEFnDSy/WuAm4MhgiGJuF2ssTnIwi2VBMJXwOZgiMraJipqmqhqaCbT5yUnzV7cinLSGJTjIzc9hc3ltazZVcW60moAUr1JpHo9hETwB0I0BWwcIREEGF6Qzmmj+zN1WG74jn0/C7dWUlnbRHqKF19yErnpyQwvyGBEvwyGFaS3OwYR4a21Zby9rpzmUIhQSPAmeRgzIJOxRVmM6JdBQ3OQA/XNHKj3h9sT/eyr87dLkumpSWT7ksn22c9M8XpITvJQ2xRgb20Te2v8NAVaE4k9Htv5JS89mWnD85k+LI/0lCSW7zjA0u372V3VYPeZlsyArFSmDM1lbGFWu9JFY3OQtXuqWbnzABvLaklJMqSnevF5kwiGQjQFQwSDwpSheZwxtv9RX0LY2BxkafiGo7ymiYqaRhqbQ3iMIckDBoMghMTezFx78rCDpU4R4Z2NFby9rswmJGlJTPZ4ReCH5x3DkPz0iL5Dh9IkEyFNMkop1XX6nIxSSinXaZJRSinlGE0ySimlHKNJRimllGM0ySillHKMJhmllFKO0SSjlFLKMZpklFJKOabPP4xpjKkAtndhk37AXofCSVR98Zihbx53Xzxm6JvHHe0xDxOR/kdbqc8nma4yxiyJ5CnX3qQvHjP0zePui8cMffO443XMWl2mlFLKMZpklFJKOUaTTNc97HYALuiLxwx987j74jFD3zzuuByztskopZRyjJZklFJKOUaTTBcYY2YaYzYYYzYbY25zOx4nGGOGGGMWGGPWGWM+NcZ8Pzw/3xjzljFmU/jfPLdjjTVjTJIxZrkxZk54eoQx5pPwMT9rjElxO8ZYM8bkGmOeM8asD5/zk3v7uTbG/CD83V5jjHnGGOPrjefaGPO4MabcGLOmzbwOz62x7g1f21YZY6bGKg5NMhEyxiQBfwFmAeOBK4wx492NyhEB4L9F5FhgBvCd8HHeBrwtImOAt8PTvc33gXVtpn8L/DF8zPuB61yJyll/At4QkXHAJOzx99pzbYwpBr4HTBeRCUAS8DV657n+KzDzkHmdndtZwJjwz/XAA7EKQpNM5E4ENovIVhHxA/8ELnY5ppgTkT0isiz8ew32olOMPda/hVf7G3CJOxE6wxgzGLgIeDQ8bYCzgefCq/TGY84GTgceAxARv4gcoJefa8ALpBljvEA6sIdeeK5F5D1g3yGzOzu3FwN/F+tjINcYMzAWcWiSiVwxsLPNdEl4Xq9ljBkOTAE+AQpFZA/YRAQMcC8yR9wD/AgIhacLgAMiEghP98bzPRKoAJ4IVxM+aozJoBefaxHZBdwN7MAmlypgKb3/XLfo7Nw6dn3TJBM508G8Xts1zxiTCTwP3CQi1W7H4yRjzOeBchFZ2nZ2B6v2tvPtBaYCD4jIFKCOXlQ11pFwG8TFwAhgEJCBrSo6VG8710fj2Pddk0zkSoAhbaYHA7tdisVRxphkbIJ5SkReCM8uayk+h/8tdys+B5wKfNEYsw1bDXo2tmSTG65Sgd55vkuAEhH5JDz9HDbp9OZzfS7wmYhUiEgz8AJwCr3/XLfo7Nw6dn3TJBO5xcCYcC+UFGxj4SsuxxRz4baIx4B1IvKHNoteAa4N/34t8HK8Y3OKiNwuIoNFZDj2vP5bRK4CFgBfCa/Wq44ZQERKgZ3GmLHhWecAa+nF5xpbTTbDGJMe/q63HHOvPtdtdHZuXwGuCfcymwFUtVSrRUsfxuwCY8yF2DvcJOBxEfm1yyHFnDHmNOB9YDWt7RM/xrbLzAaGYv+jflVEDm1U7PGMMWcCN4vI540xI7Elm3xgOXC1iDS5GV+sGWMmYzs7pABbgW9ibz577bk2xtwBXI7tSbkc+H/Y9odeda6NMc8AZ2JHWy4Dfg68RAfnNpxw78P2RqsHvikiS2IShyYZpZRSTtHqMqWUUo7RJKOUUsoxmmSUUko5RpOMUkopx2iSUUop5RhNMkr1YMaYM1tGjVYqEWmSUUop5RhNMkrFgTHmamPMImPMCmPMQ+F319QaY35vjFlmjHnbGNM/vO5kY8zH4fd6vNjmnR+jjTHzjTErw9uMCu8+s807YZ4KP1inVELQJKOUw4wxx2KfMD9VRCYDQeAq7OCMy0RkKvAu9olsgL8Dt4rIROzICy3znwL+IiKTsONttQz7MQW4Cfueo5HYsdiUSgjeo6+ilIrSOcA0YHG4kJGGHZgwBDwbXucfwAvGmBwgV0TeDc//G/AvY0wWUCwiLwKISCNAeH+LRKQkPL0CGA584PxhKXV0mmSUcp4B/iYit7ebaczPDlnvSGM8HakKrO0YW0H0/7VKIFpdppTz3ga+YowZAAffsz4M+/+vZeTfK4EPRKQK2G+M+Vx4/teBd8Pv9CkxxlwS3keqMSY9rkehVDfoHY9SDhORtcaYnwLzjDEeoBn4DvYlYccZY5Zi39B4eXiTa4EHw0mkZWRksAnnIWPML8P7+GocD0OpbtFRmJVyiTGmVkQy3Y5DKSdpdZlSSinHaElGKaWUY7Qko5RSyjGaZJRSSjlGk4xSSinHaJJRSinlGE0ySimlHKNJRimllGP+P+buocItB7BwAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fit_and_plot(train_features=poly_features_t[:100,0:3],train_labels=labels[:100],test_features=poly_features_t[100:,0:3],test_labels=labels[100:],no_inputs=3)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insufficient Training(Overfitting)\n",
    "Now let’s try to train the model using a polynomial of too high degree. Here, there is insufficient data to learn that the\n",
    "higher-degree coefficients should have values close to zero. As a result, our overly-complex model is far too susceptible\n",
    "to being influenced by noise in the training data. Of course, our training error will now be low (even lower than if we had\n",
    "the right model!) but our test error will be high.\n",
    "Try out different model complexities (n_degree) and training set sizes (n_subset) to gain some intuition of what is\n",
    "happening.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "final epoch:train loss tensor(24.8952, grad_fn=<MeanBackward1>) test Loss tensor(28.2323, grad_fn=<MeanBackward1>)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZkAAAEKCAYAAADAVygjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd8VGX2+PHPycykTHoCoSRAQhEEpFexI4ptsbfVta26RdfdVXd1i3636rrlp+66uhYsq4u9rtgbuiodpBs6oQbSezLz/P54JmYIA6TM5E6S8369eMG9c+fOuQ7m8LTziDEGpZRSKhJinA5AKaVU16VJRimlVMRoklFKKRUxmmSUUkpFjCYZpZRSEaNJRimlVMRoklFKKRUxmmSUUkpFjCYZpZRSEeN2OoBIEJGzgTOALOABY8y7B7u2R48eJjc3t6NCU0qpLmHx4sV7jTE9D3ddxJKMiPQDngJ6A37gYWPMfW2812zgTGCPMWZks9dmAvcBLuBRY8zdxphXgVdFJB34C3DQJJObm8uiRYvaEpZSSnVbIrKlJddFsrusAbjZGHMkMAX4oYgMD75ARLJEJLnZucEh7vUEMLP5SRFxAQ8ApwHDgUuafcavAq8rpZRyQMSSjDFmpzFmSeDP5cAaILvZZccDr4lIPICIXAvcH+Je84CiEB8zCVhvjNlojKkDngVmifUn4K3GGJRSSnW8DhmTEZFcYCwwP/i8MeYFEckDnhWRF4CrgRmtuHU2sC3ouACYDNwInAykishgY8xDIWI6Czhr8OBQDSellFLhEPEkIyJJwEvAj40xZc1fN8bcIyLPAg8Cg4wxFa25fYhzxhhzPyFaRM0uegN4Y8KECde24vOUUor6+noKCgqoqalxOpSIi4+PJycnB4/H06b3RzTJiIgHm2CeMca8fJBrjgVGAq8AdwI3tOIjCoB+Qcc5wI62RauUUi1TUFBAcnIyubm5iIT6t27XYIxh3759FBQUkJeX16Z7RGxMRux/+ceANcaYvx3kmrHAI8As4CogQ0R+34qPWQgMEZE8EYkFLgZeb1/kSil1aDU1NWRmZnbpBAMgImRmZrarxRbJ2WXTgMuBk0RkWeDX6c2u8QIXGGM2GGP8wBXAAdPiRGQO8AUwVEQKROQaAGNMA7bl8w52YsHzxphVkXskpZSyunqCadTe54xYd5kx5jNCj5kEX/O/Zsf12JZN8+suOcQ95gJz2ximUkqpCNKyMm316d/gucudjkIp1U2VlJTwz3/+s9XvO/300ykpKYlARKFpkmmrGJf9pZRSDjhYkvH5fId839y5c0lLS4tUWAfokrXLOsS0m5yOQCnVjd12221s2LCBMWPG4PF4SEpKok+fPixbtozVq1dz9tlns23bNmpqarjpppu47rrrgKZSWhUVFZx22mkcc8wxfP7552RnZ/Paa6+RkJAQ1jg1ySilVDv85o1VrN5xwBLAdhneN4U7zxpxyGvuvvtuVq5cybJly/j4448544wzWLly5TdTjWfPnk1GRgbV1dVMnDiR8847j8zMzP3ukZ+fz5w5c3jkkUe48MILeemll7jsssvC+izaXdZWGz6Eh46F4hbViFNKqYiaNGnSfmtZ7r//fkaPHs2UKVPYtm0b+fn5B7wnLy+PMWPGADB+/Hg2b94c9ri0JdNWnkRI6QvG73QkSikHHa7F0VESExO/+fPHH3/M+++/zxdffIHX6+WEE04IudYlLi7umz+7XC6qq6vDHpcmmbbqPxkufc7pKJRS3VRycjLl5eUhXystLSU9PR2v18vatWv58ssvOzi6JppklFKqE8rMzGTatGmMHDmShIQEevXq9c1rM2fO5KGHHmLUqFEMHTqUKVOmOBanGGMc+/BoMGHCBNOmTctqy+GxU2HK92GcrpdRqjtZs2YNRx55pNNhdJhQzysii40xEw73Xh34byuPF9JzwZvhdCRKKRW1tLusrWJccMl/nI5CKaWimrZklFJKRYwmmfZ48Wr7SymlVEjaXdYeWcOhm5T7VkqpttAk0x7H3eJ0BEopFdW0u0wppTqhtpb6B7j33nupqqoKc0ShaZJpj/n/gnsGgq/B6UiUUt1MZ0ky2l3WHhmDYPjZ4KsDl/6nVEp1nOBS/zNmzCArK4vnn3+e2tpazjnnHH7zm99QWVnJhRdeSEFBAT6fj1//+tfs3r2bHTt2cOKJJ9KjRw8++uijiMapLZn2GHIynPk3iPU6HYlSykmPnwFLn7F/9tXb4+WB2oZ1VfZ45Uv2uKbUHq9+3R5X7rPH696yx+W7W/SRd999N4MGDWLZsmXMmDGD/Px8FixYwLJly1i8eDHz5s3j7bffpm/fvixfvpyVK1cyc+ZMfvSjH9G3b18++uijiCcY0CSjlFKd3rvvvsu7777L2LFjGTduHGvXriU/P5+jjjqK999/n5///Od8+umnpKamdnhs2sfTRrUNPmoLviJlziw492EYOtPpkJRSTrnqzaY/uzz7H8d69z+OT93/ODFz/+PkpkKXLWWM4fbbb+f6668/4LXFixczd+5cbr/9dk455RTuuOOOVt+/PbQl00Y3/Gcp1728FUZfbPeVUUqpDhRc6v/UU09l9uzZVFRUALB9+3b27NnDjh078Hq9XHbZZdxyyy0sWbLkgPdGmrZk2ijd62FFTRKcfo/ToSiluqHgUv+nnXYal156KVOnTgUgKSmJp59+mvXr13PrrbcSExODx+PhwQcfBOC6667jtNNOo0+fPhEfl9FS/20s9f/HuWt48vPNrPv9aWCMrvxXqhvRUv9a6j/i0rweahv8+O8dBW/+1OlwlFIqKml3WRule2MBqBh+KSk5wx2ORimlopMmmTZK93oA2Dby+4zo2/HTApVSzjLGIN2gm7y9QyraXdZGaYGWTElVPTTUOhyNUqojxcfHs2/fvnb/AI52xhj27dtHfHx8m++hLZk2auwu6/fpz+CNBfCTlQ5HpJTqKDk5ORQUFFBYWOh0KBEXHx9PTk5Om9+vSaaNGrvLNmQeT/9h4x2ORinVkTweD3l5eU6H0Slokmmjxu6ylYlHc+LUIQ5Ho5RS0UnHZNoo1h1DYqyLkqo6qC0Hv8/pkJRSKupokmmHNG8s/Xe9A3flwN6vnQ5HKaWijiaZdkjzelhDHsz4LXgznQ5HKaWijo7JtEO6N5Z1dVkw7RynQ1FKqaikLZl2SPN6KKmsg+piOy6jlFJqP5pk2iHdG0tNVRn8KRcWPuZ0OEopFXU0ybRDutfDrhoX/lP+CAOPdzocpZSKOjom0w5p3liMEUpHX0t6YqzT4SilVNTRlkw7pCfaVf+lRbuhbKfD0SilVPTRJNMOjav+M/97Nbx8rcPRKKVU9NHusnZIS7AtmY1HXMPofhkOR6OUUtFHWzLt0FiJeX3aMXDEKQ5Ho5RS0UeTTDs0Jpmy8jLYtQJ89Q5HpJRS0UWTTDskx7uJEehZ8B48dAzs2+B0SEopFVU0ybRDTIyQ5o1llWcknP84JPd2OiSllIoqOvDfTmleD1sbUmDkdKdDUUqpqKMtmXZK98ZSXFUHe/Nhl27BrJRSwTTJtFO610NxVT28fB28+0unw1FKqaii3WXtlOaNZfWOMjjnboj1Oh2OUkpFFU0y7ZSWEGjJ9J/sdChKKRV1tLusndITY6mu91FTXgRfvwNVRU6HpJRSUUOTTDuleW1pmcqCVfCfC6FgocMRKaVU9NDusnZqXPVfmDiEzGveg6zhDkeklFLRQ5NMOzW2ZIrq3TBoksPRKKVUdNHusnZqbMmUVNXDtgWwdq7DESmlVPTQlkw7NSaZ4qo6WPtP2Lkchp3ucFRKKRUdNMm0U2N3WUlVPcz4LbjiHI5IKaWiR5dKMiJyNnAGkAU8YIx5N9KfGe9xkeBxUVJVB2mDI/1xSinVqUT9mIyIzBaRPSKystn5mSKyTkTWi8htAMaYV40x1wJXAhd1VIxpjaVlqktg4WNQ+HVHfbRSSkW1qE8ywBPAzOATIuICHgBOA4YDl4hI8NzhXwVe7xBp3ljbkmmogTd/Cps+6aiPVkqpqBb13WXGmHkiktvs9CRgvTFmI4CIPAvMEpE1wN3AW8aYJR0V4zdFMpN6wU9WQ3KfjvpopZSKap2hJRNKNrAt6LggcO5G4GTgfBH53sHeLCLXicgiEVlUWFjY7mC+KfcvAqnZENNZ/7MqpVR4ddafhhLinDHG3G+MGW+M+Z4x5qGDvdkY87AxZoIxZkLPnj3bHUxmUix7y2vtwcaP4X/3tfueSinVFXTWJFMA9As6zgF2OBQL/dK9lNU0UFpVb5PMp38Dv8+pcJRSKmp01iSzEBgiInkiEgtcDLzuVDD9Muw+MluLquCYn8KtGyDG5VQ4SikVNaI+yYjIHOALYKiIFIjINcaYBuAG4B1gDfC8MWaVUzEOyLRJZktRJcSngCvq51MopVSHiPqfhsaYSw5yfi4QFYXC+gdaMlv2VdkTCx6xa2aOv9XBqJRSynlRn2Q6g8Q4Nz2SYtnamGS2L4aK3c4GpZRSUUCTTJj0z/DaMRmAWf/UacxKKUUnGJPpLAZkJjYlGU0wSikFaJIJm/4ZXnaUVlPbEJi6/MaP4eO7nQ1KKaUc1m2TjIicJSIPl5aWhuV+/TO8GAMFxdX2RH0V1FeH5d5KKdVZddskY4x5wxhzXWpqalju1ziN+Zsus3Mfhhm/Ccu9lVKqs+q2SSbc+jcmmcYZZkoppTTJhEvPpDgSPK6mtTK+enj4RPj0r84GppRSDtIkEyYisv80ZpcHeg2H5L7OBqaUUg7SdTJh1D/Ty5Z9lU0nZnXYvmlKKRWVtCUTRo0tGWNM00m/D+prnAtKKaUcpEkmjAZkeqmp97OncW+ZikL4Uy4s/bejcSmllFM0yYRR/4xm05gTe8D4K6DXCAejUkop52iSCaMBmYlAUDVmETjl9zDgaAejUkop52iSCaPstARiBLYGD/4DlO2EmjJnglJKKQdpkgmjWHcMfVITmrrLAPashb8Ng7VvOheYUko5pNsmmXDXLms0INPLluAk0+MImHk39J8c1s9RSqnOoNsmmXDXLmvUP8O7f2mZmBiY8n3IGBjWz1FKqc6g2yaZSBmclcS+yjoKG6cxg63GvGme3ZJZKaW6EU0yYTYqJw2ArwqCEsrOr+DJs2Dzpw5FpZRSztAkE2Yj+qYQI/BVQdBYT9+xcOnzkHe8c4EppZQDtHZZmCXGuRmclbR/S8YdC0ec6lxQSinlEG3JRMConDS+Kijdv4ZZ+S5Y+BjUVjgXmFJKdTBNMhEwOieVfZV1bC8J2n55z2p486dQsMC5wJRSqoO1KMmIyE0ikiLWYyKyREROiXRwndVRgcH/FcHjMv2PhhuXwMATHYpKKaU6XktbMlcbY8qAU4CewFXA3RGLqpM7sk8yHpewPDjJeOIhc5CtZ6aUUt1ES5NM40/G04HHjTHLg86pZuLcLob1Ttl/8B+gcB28dRvUhLfKgFJKRauWJpnFIvIuNsm8IyLJgD9yYXV+R+WksqKgFL8/aPC/YjcsftwmG6WU6gZammSuAW4DJhpjqgAPtstMHcTonFTKaxvYHFyRuf/RcNtW6DfJucCUUqoDtTTJTAXWGWNKROQy4FdAp+7ziVSBzEZNK/+D7u9ygzsuIp+nlFLRqKVJ5kGgSkRGAz8DtgBPRSyqDhCpApmNhmQlEe+JYXnzcZmt8+Hp86GqKCKfq5RS0aSlSabB2JWFs4D7jDH3AcmRC6vzc7tiGNE3df9pzADGDyVboWyHM4EppVQHammSKReR24HLgTdFxIUdl1GHMConlZU7SmnwBc2RGDAVblgAvUc6F5hSSnWQliaZi4Ba7HqZXUA28OeIRdVFjOufTk29n1U7dOtlpVT31KIkE0gszwCpInImUGOM6dRjMh1hcl4GAAs2NRt/+foduG8MVO51ICqllOo4LS0rcyGwALgAuBCYLyLnRzKwriArJZ68HonM37Rv/xeSekHPYVBb7kxgSinVQVpa6v+X2DUyewBEpCfwPvBipALrKiblZvDWyp34/AZXTKBIQt8xcOmzzgamlFIdoKVjMjGNCSZgXyve261NHphBWU0D63aFaLVUF+uWzEqpLq2lieJtEXlHRK4UkSuBN4G5kQur65g8MBPgwC6zyr3wl6Gw+ImOD0oppTpISwf+bwUeBkYBo4GHjTE/j2RgXUV2WgLZaQnM39hs8D+xB0y/Awaf7ExgSinVAVq8/bIx5iXgpQjGEjYiMhA7jpRqjHF8gsLkgRl8vK4QYwwSXOr/6BucC0oppTrAIVsyIlIuImUhfpWLyGEXf4hImoi8KCJrRWSNiExtS5AiMltE9ojIyhCvzRSRdSKyXkRuAzDGbDTGXNOWz4qEKXmZFFXWsX5PiK2Xd6+G/Pc6PiillOoAh0wyxphkY0xKiF/JxpiUFtz/PuBtY8wwbDfbmuAXRSQrsG1A8LnBIe7zBDCz+clA5YEHgNOA4cAlIjK8BXF1qEmB9TJfNl8vA/D+/8GbN4MxB76mlFKdXMRmiIlICnAc8BiAMabOGNN8KtXxwGsiEh94z7XA/c3vZYyZB4SqKDkJWB9oudQBz2Lrq0WVAZleeqXEMX/jvgNfPPWP8N0PdMdMpVSXFMlpyAOBQuBxEVkqIo+KSGLwBcaYF4C3gWdF5NvA1djFni2VDWwLOi4AskUkU0QeAsYGaq4dINKl/pt9FpPzMlmwqQjTvMXSYzAk9Yx4DEop5YRIJhk3MA540BgzFqjEbny2H2PMPUANdjuBbxljQgxcHFSof/4bY8w+Y8z3jDGDjDF3hXpjpEv9Nzd5YAZ7ymvZUFh54IsVe+A/F8H69zskFqWU6iiRTDIFQIExZn7g+EVs0tmPiBwLjAReAe5sw2f0CzrOAaKyhv7xR9jWykdr9xz4YnwalO+EyhDdaUop1YlFLMkEimpuE5GhgVPTgdXB14jIWOAR7DjKVUCGiPy+FR+zEBgiInkiEgtcDLze7uAjICfdy7DeyXywdveBL7pj4bpPYPRFHR+YUkpFUKRLw9wIPCMiXwFjgD82e90LXGCM2WCM8QNXYHfd3I+IzAG+AIaKSIGIXANgjGkAbgDewc5ce94YsypiT9NOJw3LYuHmYkqr6g98UcTOMNs6X2eaKaW6jIgmGWPMMmPMBGPMKGPM2caY4mav/88YsyLouN4Y80iI+1xijOljjPEYY3KMMY8FvTbXGHNEYPzlD5F8nvaafmQvfH7DJ/mFoS/Ifxdmn2K3AlBKqS5Ai1x2oDH90shIjOXDNSG6zAAGTYcz/qqlZpRSXYYmmQ7kihFOHJrFR+sK99+S+ZsL3DDxu/b32nLdb0Yp1elpkulg04/MorS6niVbD1Hiv6EOHjsF3rip4wJTSqkI0CTTwY4d0gOPS0LPMmvkjoWJ18D4KzssLqWUigRNMh0sOd7D5LxMPlgTYr1MsInfhbzjOiYopZSKEE0yDjhpWBbr91SweW+I1f/B6irh07/C1i87JjCllAozTTIOmDmyNyLwytLth75QYuCLf8KGjzomMKWUCjNNMg7om5bAMYN78NKSAvz+Qyy89CTADQvhxJA1PpVSKuppknHI+eNzKCiuZn6oPWaCee1eNPgaIh+UUkqFmSYZh5wyvDfJcW5eXFxw+IuXzYH7RkFtawpUK6WU8zTJOCQh1sWZo/vw1sqdVNYeppXSY4idaVanSUYp1bloknHQ+eNzqKrzMXfFzkNfmDMBznkIkntD8WZY+2aHxKeUUu3VbZNMR+6MeTDj+qeT1yOxZV1mjT7/B7z6fdi2IHKBKaVUmHTbJNPRO2OGIiKcPz6H+ZuK2LLvMGtmGs28C360DPpNimxwSikVBt02yUSLc8dl44oR/jN/a8ve4PI0zTirPkT9M6WUigKaZBzWJzWBU0f04tmF26iu87X8jW/cBLNP1Q3OlFJRTZNMFLhiai6l1fW8tuwwFQCCDZ4B474Dfl0/o5SKXm6nA1AwKS+DYb2TeeLzzVw0sR8icvg3HXlm5ANTSql20pZMFBARrjw6l7W7yllwuAoAwfx+WPcWFG20x2vfhOXPRSZIpZRqA00yUWLWmGzSvB6e/GJzy99UtQ/mXAy7V9vj4s0w9xaoKYtAhEop1XqaZKJEQqyLiyb2451Vu9lRUt2yNyX1hAuehIyB9njUxXD9JxCfErlAlVKqFTTJRJHLpwwgRuC+9/Nb/qYRZ0Ov4fbPiZlNCafsMFUElFKqA2iSiSI56V6umpbH84u3saKgHZUIFj4Kfx/XNFajlFIO0SQTZW44aTCZibH85o1VmLaugRl6Okz5AST3CW9wSinVSppkokxKvIdbTx3Koi3FvL58Rxtv0hem/9pueqaUUg7SJBOFLhjfj6OyU7n7rbVU1bVjseWetfDULCjfFb7glFKqFTTJRKGYGOHOs4azs7SGf3y4vh03csO+jXZqs1JKOUCTTJSakJvBeeNyeOTTjeTvLm/bTXoMhh8thf5TwhucUkq1kCaZKPaL04eRGOfml6+sxO9v4yQAV6By0Ht32F9aUFMp1YE0yUSxzKQ4bj9tGAs2F7VuY7PmjLFVAGrLoSV10ZRSKky6ZJIRkYEi8piIvOh0LO11wfh+TMxN549vraGosq5tNxGBs+6FM/5mj3evgo/v1laNUiriIp5kRMQlIktF5L/tuMdsEdkjIitDvDZTRNaJyHoRuQ3AGLPRGHNNe+KOFjExwh/OOYqKmgZueWE5vrZ2m0FTK2bVK7D4SajcG54glVLqIDqiJXMTsCbUCyKSJSLJzc4NDnHpE8DMEO93AQ8ApwHDgUtEZHh7A442R/RK5s6zhvPh2j3c8/ba9t/whF/A9fNs7bP66pa3aPx+WP8B1Ne0PwalVLcQ0SQjIjnAGcCjB7nkeOA1EYkPXH8tcH/zi4wx84BQNfAnAesDLZc64FlgVgtjO0tEHi4tbUf5lg50+dRcLp8ygH/N28gLi7a172YxMTbBAMz7MzwwCRpq7XHh13Z9TSiLZ8Mr10OtVnlWSrVMpFsy9wI/A/yhXjTGvAC8DTwrIt8GrgYubMX9s4Hgn7gFQLaIZIrIQ8BYEbn9IJ/9hjHmutTU1FZ8nLPuOGs40wZn8stXVrJwcyv2nTmUvuPgyLPAHWePX74W3v1V6Gv7TYajLoTEnk2tn/Jd8MFvwdcAK1+CfRvCE5dSqkuIWJIRkTOBPcaYxYe6zhhzD1ADPAh8yxhT0ZqPCX1Ls88Y8z1jzCBjzF2tuF9U87hieODSceSkJ3D1EwtZtSMMrbAjz4TpdzQdz7zLlqQB2PIFbJ3f9Frvo2DmH21L5j8XQv57sOYN+PIh2L4I3vgJfPlg+2NSSnUZkWzJTAO+JSKbsd1YJ4nI080vEpFjgZHAK8CdrfyMAqBf0HEO0MaCX51DmjeWf393Mslxbr7z2ALW72lNTm6BAUdDn9Hg98EbN9lWypYvbOumrtJeE+O2G6ZVFsKka+GGBXbB59Vvwcy7wxuPUqpTi1iSMcbcbozJMcbkAhcDHxpjLgu+RkTGAo9gx1GuAjJE5Pet+JiFwBARyROR2MDnvB6WB4hi2WkJPHPtFESEyx6dz7aiqvB/SIwLLpkDl/wHtn4Oq15rei02Ea55D8Zcao9Tc+zvvUbYxZ91VVBReOA9N3wEO5aGP1alVNRyep2MF7jAGLPBGOMHrgC2NL9IROYAXwBDRaRARK4BMMY0ADcA72BnsD1vjFnVYdE7KK9HIk9/dxI1DT4ueOgL1u1qY+mZQ8kcBPGpcOzN8IPPbXJpFOMK/R6/H2afAs+cZ//cqL4a/n02zPtL+ONUSkUtafOeJV3EhAkTzKJFi5wOo83W7irjitkLqKrz8eh3JjB5YKbTIcHWL6G2AoacbI+NsWt0dq+CjEHgiW+6tngz7PwKhn/LkVCVUm0jIouNMRMOd53TLRnVTsN6p/DyD6aRlRzH5bMXMHdFFGy73H9KU4L54Lfwzi/sn3uNsAmmYg988UDT62/ebMd7Sgt0DY5SXYwmmS4gOy2BF793NEdlp/KDZ5bwwEfr276rZjj5/bBrBRj//gs+lz5tk8u+DbbUzVVvga8eHj4R3gk549xqqLO/lFKdhnaXdfLusmA19T5+/tJXvLZsB+eOzeau844izn2QsRMn+X1QtMluRRBs4WOQdxz0GHLge+qq4InT7eszftsxcSqlDkq7y7qheI+Ley8aw80zjuDlpds5+4HP+Xjdnuho1QSLcR2YYAAmXtOUYNa9ZZNRfQ0sf9ZuJZ13PORM7NhYlVLt4nY6ABVeIsKN04dwRO9kfvff1Vz5+EIm5WXw85lDGT8gw+nwWmbrfJhzMZx5LxifHbPJHAIzfuN0ZJGzfQn0GgnuWKcjUSqstCXTRZ06ojcf3nwCv5s1gk17KznvwS/46XPL2FPWCQbW+0+Gi/8DYy+H8VfDlW9Czvim11e8CC9eY0vZBDPGdsO1tOW27i1YOzd8cbfVvg3wwhWwc7nTkSgVdppkurBYdwyXT83lk1tP4IYTB/Pfr3Zy0l8/4R8f5rOhsCL6utGCDTvDLuyMiYHcY/Z/raoISrcBzeLfsRTuHwNL/x36npV7YflzUF1sJyV8cg/Mfygi4bdKaYFNmN5O0tJUqhV04L8LDfwfzqa9lfz2jVV8tM6uxu+f4eWU4b24+pg8+qYlOBxdK9VV2sWhdZWw4gUYf6Vtwbx+A5z+FzuGAzaZ7MuHnkNtC+ila+C6j6HvWCjbCd5MKNkCS56EGb9zbudQvx+2L4aMPEjs4UwMSrWCDvyrA+T1SOTxqybx6c9O5Hdnj2RwVhJPfL6Z4//8Eb94ZUVkytNESmP1gc//AW/fDsVbbIKY9YBNML56W2XgywfgoWOgcB2MOAeu/xR6j7LvTeljx0C2fgGLn4J96w/9mcbA/Icjs9lb0QZ47GRY+XL4762Ug7Ql041aMqEUFFfx4McbeD6wR823Jw/gxpMGk5kU53BkLdRQC1v+B7nH2e41sGtpHj/NVo2efgcsnwNTfnDwVorfD9VFh29B7F4ND02D0/8ME7/btlhdsfvHsXM5vPoDmPUPKNkGecdCQnrr761UB2tpS0aTTDdRDRjSAAAZ0UlEQVRPMo12lFTz9w/zeW7hNryxbi6fOoCUeA/VdQ0gwkUT+5HdmbrUPvmznQ494uzWve/9/4Mhp9hq1KEUfg0FCyH/XbjgiZZ3r9VVwr/Ptet8Tvpl0/mCRXZh6jkPQUrf1sWqlINammR0CrMCoG9aAnedO4prjsnjnrfX8eDHTZuPicDD8zbwwxMGc+1xA4n3ROECz+aOv7X176ncZ6sReHscmGT8Pru+p+cRsOkTKN9p99WJb+Gmdx6vLavTq9nu4DkT4IpA4XBfA6x8EZJ7w8ATWh+/UlFIWzLakgmporYBd4wQ545he0k1f3hzDW+t3EX/DC/XHpvH2WOzSY73OB1m+DXUgb8BYr2Q/z4sexrOuh+ePte2cI7/WVPBz2ChzoHtDkvuA0lZ9ri6BN75pd0sbsgptuSOy9N0j3tH2QR37r8i+5xKtZO2ZFS7JMU1/dXISffy4GXj+Sx/L3e/vYZfv7aKu99ay7fG9CUn3QuAK0Y4aVgWR/RKdirk8HDHAoEFkcWboGijTTq9R0HaAHu+MZnUBSZKfPUsbJoH5z4KhWvgs/9nN2+LS4GnzrYtmCvesO+LTbKzyHofBdvmw9Pnw+Wv2LVBInbjt+Qu3G22/Fmo2A3TbnI6EtVBNMmoFjtmSA/eGHwMywtKefrLLby8ZDu1DU17xvzp7bWcMzabn5x8BP0yvA5GGiaTroXxV9kJBWf+bf/XynfDAxPhhNsD5W+qbXWC3atg2wLbOvHE20WlvYY3JSaXG77/P9v1tmctjL7YdsE1atwArnQ7pGYfOr7G/XpiOtEk0ZoyeO8OGH1JU+tOdWnaXabdZW3m8xsaAj/oyqobeOTTjTzx+WaMMZwwNIupAzM5enAmQ7KSccU4tP4kkj74HQw9zY6r+P1NP+wbasHdgtl55bvs+EtzO7+CR0+Gb91vk1CwqiI76eDIs2DOJTD4ZJj2o9D3X/smbPzYzrCLi5IWZk2ZHctqTKaq09LuMhVxrhjBFdghs2eyi1+cfiRXTcvlX59s5KN1e3hv9W4AYgR6JsfROyWeo3JSOWtUXybmZhDT2RPP9F83/Tm4NdGSBLPgEZh7C9y64cCp0z2OgKk/sGM2YLvs4tNsRYAvHoDP/w79p9rZaIea7lxaAItmw6l/PPC1/90HCx+FH6+wx1+/C/WVcOSsA1tGDbW2BE/jTL23boMxl0Cf0Yd/zkbVJfDJn+DoHzUlmNry6El+KmK0JaMtmYgpKK7iy41FbN5bye6yGnaV1bBoczHV9T76pMYzKS+DdG8s6d5YBmUlMn1YLxJiO8HMtXAo3Q5zLrLjOFnDDn3tY6dAjAeuetPOQNv1FWSPa9nn+BpsF11dFTw1y3bfJfWENW/A+g9sdQSX27aKirfYrrzmExg++TOse9Pu++OOhz/lwtjL4NQ/HPxz37zZJpT0wDjWurfgucvh2g+hzyj48Pe2AsMP57csKYfDrhWQkq3le8KkW7dkRGQg8Esg1RhzvtPxdFc56V7OH7//2ExlbQPvr9nNG8t3smRrMSWV9ZTX2kKX3lgXp47ozcyRvRnbL42slPhQt+0aUrPhe5+17NoTbuebOm0u94EJ5ut37ID6OYEZaS9dA5O/B7nTmhaorn/PrtUp2miTzJFn2V+NLnraTssWsWNM/oamH/5HnRcYYwqsk/rhAkjudfB4i7fYwqM5k2ySqa2w3Yo3r21qtQ2YZn83/oPfJ5xqSm3lhzGXwdkPdMxnKiCCLRkRiQfmAXHYZPaiMebONt5rNnAmsMcYM7LZazOB+wAX8Kgx5u6g1148XJLRlozz6n1+Fm0u5rVl23lzxU7Ka2zS6ZUSx7DeKfRMjqNHUhw56QlMPzKLPqmdaFFoR/jkHlj7X7h+nm0hPTULTrgNjmrDv698DfDvsyFzMJx1b9vi8ftg82e2XlzxFnj1+3Dlfw++2HTfBjtzz9WCf/NWF8OeNQdfLHswvgZY8bzdjyjUpniq1Rxf8S8iAiQaYypExAN8BtxkjPky6JosoNoYUx50brAxZn2zex0HVABPBScZEXEBXwMzgAJgIXCJMWZ14HVNMp1MbYOPrwpKWVFQyortpeTvKWdveR37Kmup99m/q2P7pzFjeC8G9UwiJz2BfhleUrrimp3WCF6n09KJBwfz0V2Q1t/OjtvwIZx2T1OtuEaLHretp6vfPnTVg9ICeO2HMObbMOrCA1+vrYC/j4dBJ8E5D9pk0FANnsT9x4by37dTyjfNs2WEfrLaxtcWZTtsq0zL97SL491lxmavisChJ/CreUY7Hvi+iJxujKkRkWuBc4DTm91rnojkhviYScB6Y8xGABF5FpgFrA7Xc6iOFed2MTE3g4m5+/ebG2PYUFjJO6t2MXfFTu55e91+rx/RK4kpAzOZmJtBr5R4UhM8pHs99EyOQ5yqrNyRgp+xvWMcJ95uf//sXltGxx3ih3lsku36qimFhLSm87tWQv47MOEaez41B77z2sE/Ky7Jju00ti4KFti6c5e/CoNObLpu9Su2BM95jwGmdQmmfBd89ZztKnO54Z9TYeR5B05Lb4/6anDF2WoQa9+09e26w9+7FojowH+gpbEYGAw8YIz5eYhrfgYcDbwA3ADMMMZUhLguF/hvs5bM+cBMY8x3A8eXA5OBO4E/YFs4jxpj7gpxv7OAswYPHnxtfn5+O59UdbTiyjoKiqspKK5iQ2EFCzYXs2hzEVV1vv2u65sazzFDenDMkJ6MyUkjJz2h889q60iNEwda6vN/wIe/g5vX7Z98Wqq0wFaiHnmuTVB719txnRg3VO1r2zYIy5+FV66HGxbZZLb0aTsmlJF36PeVbLVbQTRvxYXy2g12y4hB0+1+Rle/C4mZrY+1E3G8u6xZMGnAK8CNxpiVIV5/Ftt6GWSMKTzIPXI5MMlcAJzaLMlMMsbc2NLYtLus66j3+fl6dzlFlXWUVTdQWF7D/E1F/G/9XsoC4zxJcW6O6JVEUrznm03bRvRN5dxx2Z2/WoETKvfallPwVOTKfeH5AdvYlTbjN/uvFzIG3v2VrWh9cguHeUu22i7AYL4G+PKfdi+i+BQ72620AI75sX39oWMBY7eHOFyrZMm/bYvpmJ/YSRPBLa0tX8Dzl8OF/4YBU1sWb2vUlNnvoKNm6QU43l0WzBhTIiIfAzOB/ZKMiBwLjMQmoTuxrZmWKgD6BR3nADvaFazqtDyuGEb03b9g5ZXT8vD5Dat2lLJqRxlrd5axbnc5pdX1xIhdUPrIpxt56JMNjOibwrFDejKwZyIDeySSnhiLACJCcrybzMTY7tH11lIlW+H+cXYdzuTrms6H61/wcUkw84+w5XMYdVHTD3oRu6DTFRv6fb76pnpwjZonGLBjO+/9GtJzYfi3bFfXrpVw9I22IsPk79kuu0N951VFdmxn3OVN51xuG8OmeTB4ul1P5PGGXnh7KMWbbbHWuCQ7oaO+GnoM3v+a+mqYfSpkj7fbRQQvCm7uvTvseqtjf9q6ONopkgP/PYH6QIJJAN4F/mSM+W/QNWOBOcAZwCbgaWCjMeZXIe6Xy4EtGTd24H86sB078H+pMWZVS+PUlozaW1HLG8t38OrS7azeWfbNBIPmUhM8DOqZSP8MLz2T4+iZHEduZiLHHdGz3ZWp631+PK5OVB6m0Qe/s+V3knvDi1fbYqCHWj8TLsETHRY+BjuXwbf+bo/nXAr1VbYm3Od/t91Yp/8ldLLYswZ6DgskrgrbNRbquprSAytub/oUnj7Pjjk1b6HM+7NdC/SLnbbYakueZ8Ejdmr7sDPsVPO/j4dT/mAX5j5zgS1Z9KOltsVSvtt25bncdmFtnzGw8BGbRGb9I/T9X7zKJq1T/wCrXtk/cbdBNLRk+gBPBsZlYoDngxNMgBe4wBizAUBErgCubH4jEZkDnAD0EJEC4E5jzGPGmAYRuQF4BzuFeXZrEoxSAD2S4rhqWh5XTcujwedne0k1GwsrKaupxxgwGIoq69lYWMGGwgoWbi5mb0XtN3XbkuPcnDqyNzNH9Ca3RyI56QktTjrGGJ78fDN3vbWWK6flctvMYZ2rtRRc9QCxYycdIfi/ka8Oti9tSjyjL7LJQwQqC2031sH+m2Yd2fTnuKTQ18z9mW31XPexXQjr90O/iXa90sRrQq8Zmvx9WyDVV4f9MYcdXypYAGMuPfD6qiKY/yAMPd0mmYyBdlbfsDPs62f81e7c6o6D/PfgmfPtuE//yU3FRjd/dvBnELH7H/n9sHi2XSybOQRyxoe+Pox0xb+2ZFQbGGOoqG1g2bYSXlu2g7dX7qIisKgUCLRyvAzITKRvajwVtT5KquuoqfcxOieN447oSc/kOH7+4ld8sHYPAzK9bNlXxQ9PHMQtpwztXImmkd8HEhN9s6oOtg1DS617y7Yipt4AD061ExKueKP195n7M1j2H7h5TehyOnVVNonEHOYfKFVFsOQpOOqCwxdRBVvSx1fXVJDU74etn0PuMa1/hiBRNfAfzTTJqHCoqfexYnsp2wMz3rYWVbF5XxWb91ayp7yWpDg3qQke3C5hyz67RUCMgDsmhl+cPozvTM3ll6+uZM6Crdw0fQg/mXHEYT5ROWL3KkjtZycKtFZFYE5TUs+mc34fLHsGRl/aull8B2MM7F5pt5Jo9MHv7ASHH68M64y3aOguU6rbiPc0ru858DW/3+w3bXpXaQ3z8gtZvaOMCybkfDNZ4Q9nj8Tn93PfB/m8tKSAvqkJ9E6Np09aPH1TE+iblkBmUiypCR5S4u06IHdnHMfpzHqNaPt7G5NLXaWtctBnlC0J9PqNdnylsWusPeY/BG/fBjcttxMawG6rkNzbsSnV2pLRloyKIj6/4fH/bWLl9lJ2ltaws7SGXaU11PkOrPHljhGy0xPon+ElO5CAMhPj6JEcR5/UeHqnxNMrJZ5YtyaiqFFfY2vLFSyEW76257Z8bqtqh6ObsWyH3d5h0HRY+RIMO70p2YSZtmSU6oRcMcJ3jx243zm/37C3spYdJTUUV9ZRVlNPaXU9u0pr2FJUxdZ9VazdZdcH+fzmgPsNyPAyKCuJARlekuLdJMW5SYxzkxLvITneTZrXQ066l3Svp11jQWU19STGurvm3kHh4om3U4hrg9abt7YO26Gk9LUTC8p22LVEMS6YfH347t8GmmSUinIxMUJWcjxZyYcupeL3G8pq6tldVsuushp2lVaztaiKDXsqWV9Ywaf5hdTUH7zqcWKsi5x0L6leDynxNgklxbtJjneTFOchKc5FYpwbb6yLmno/ZTX1lFTVs25XOV9tL2FbUTX9M7z8+OQhzBqTfUCyMcawaEsx9T4/Uwdmds7JDeGQHfkZXaT0tdOdG7dacJB2l2l3mepGfH5DZV0DFTUNlNc0UF5TT1GgRM/Woip2lFRTWl1PeU0DZTX1VNTa65q3kILlpCcwKieVYb1TeHvlLlbvLGNwVhIXTejHiL4pHNknhaXbivnHh+tZsrUEgGmDM/nVGcM5sk8bBtBVVNDZZS2kSUapQzPGUF3vo6K2gapaH5V1DSR4XCTHe0hJcBPnbppy6/cb3lm1i3vfz2fd7vL97pOdlsD3ThiEz+fn3g/yKauu55ThvRnaO5mBPe36otQEDymBiQ1x7phvWjuNU8ar63zdp+hplNMk00KaZJSKjL0VtazZWcbqHWX0SonnjFF9vqlqUFJVx/0frOedVbvYUVpNqB9D7hjBG+si1h1DaXX9N5UYkuPcDO2dzJBeSSR43HhcgscVY7f4TrWTHcBOK6+u97GjpJqt+6ooKK5mcFYS54/PoV9GC1bhq0PSJNNCmmSUclZNvY9tRVVsD3TVldU0UFZdT2VtA1V1Pmob/KR5PWR4Y4l1x7ChsIK1O8vZUFhBbYOfep/9dYgePWJdMfROjWdbcRXGwNGD7LYQKQkeUhPseFOc20WcO4aEWDv2lBjnxucz7CmvYU95LT6/YXjfFPIyE0NW8q6p91FYXktOesIhW1o19bZSeHBLrTPS2WVKqU4h3uNiSK9khrSjCrbfbyiqqmNXaQ27y2oQgXi3iziPi96B6dyuGGF7STUvLS7g5SUF3PdB27b4SIx1MbhXMhleD2leW0R19c4y8vdU4PMbspLjOP6Inhw9OJN4t4sGv6GqroHlBaUs3lzM13vKMcbO/PPGupgyMJPzxuVw0rCsdk83L62uZ8GmIvZV1HL0oB70z3S+xaYtGW3JKNUt+fyG8sB08MpaH7UNttVUXWfHnSprGxAReqXE0yslLlDNu4xV20vZuLeSkqp6SqrrqG8wDOuTzMi+qfRKiePLTUV8+nXhN9tLNEqOdzOufzpj+qUR646hqq6B4qp63lu9m8LyWtK9HgZnJRHvcZHgcSGBKuENfkOCx0WPJFuUNSXeTazbdiPWNvjYFVhLtXZXOat2lO7XosvN9DJlYCZ9UhPomRxHr5Q48nrYIq/tXcir3WUtpElGKRVuDT4/GworMRjcMUKsy3XQDfMafH4+Xb+X15ftYHdZDdX1PqoDm++5YgRXjFBZ28DeijpKq+sPeH+M8E1F8CkDM5k6KJMeSbF8lr+Xefl7Wbq1mOKq/d/ncQkDMhO5/+KxDO/bthl+2l2mlFIOcbtiGNq7Zd1/blcMJw7N4sShWYe9trbBR2Wtj7oGP3UNfmLdMfRIig3ZKhmclcyV0+zun3UNfvZW1LKz1FYY31BYycbCCjISD7InTxhpklFKqU7CTk5o/d5Fse4Y+qbZ+nfjB2REILKD06JGSimlIkaTjFJKqYjRJKOUUipiNMkopZSKGE0ySimlIkaTjFJKqYjRJKOUUipiNMkopZSKmG5fVkZECoEtrXhLD2BvhMKJVt3xmaF7Pnd3fGbons/d3mceYIzpebiLun2SaS0RWdSSej1dSXd8Zuiez90dnxm653N31DNrd5lSSqmI0SSjlFIqYjTJtN7DTgfggO74zNA9n7s7PjN0z+fukGfWMRmllFIRoy0ZpZRSEaNJphVEZKaIrBOR9SJym9PxRIKI9BORj0RkjYisEpGbAuczROQ9EckP/J7udKzhJiIuEVkqIv8NHOeJyPzAMz8nIpHf4amDiUiaiLwoImsD3/nUrv5di8hPAn+3V4rIHBGJ74rftYjMFpE9IrIy6FzI71as+wM/274SkXHhikOTTAuJiAt4ADgNGA5cIiLDnY0qIhqAm40xRwJTgB8GnvM24ANjzBDgg8BxV3MTsCbo+E/A/ws8czFwjSNRRdZ9wNvGmGHAaOzzd9nvWkSygR8BE4wxIwEXcDFd87t+ApjZ7NzBvtvTgCGBX9cBD4YrCE0yLTcJWG+M2WiMqQOeBWY5HFPYGWN2GmOWBP5cjv2hk4191icDlz0JnO1MhJEhIjnAGcCjgWMBTgJeDFzSFZ85BTgOeAzAGFNnjCmhi3/X2B2BE0TEDXiBnXTB79oYMw8oanb6YN/tLOApY30JpIlIn3DEoUmm5bKBbUHHBYFzXZaI5AJjgflAL2PMTrCJCDj8huSdy73AzwB/4DgTKDHGNASOu+L3PRAoBB4PdBM+KiKJdOHv2hizHfgLsBWbXEqBxXT977rRwb7biP180yTTchLiXJedmiciScBLwI+NMWVOxxNJInImsMcYszj4dIhLu9r37QbGAQ8aY8YClXShrrFQAmMQs4A8oC+QiO0qaq6rfdeHE7G/75pkWq4A6Bd0nAPscCiWiBIRDzbBPGOMeTlwendj8znw+x6n4ouAacC3RGQzthv0JGzLJi3QpQJd8/suAAqMMfMDxy9ik05X/q5PBjYZYwqNMfXAy8DRdP3vutHBvtuI/XzTJNNyC4EhgVkosdjBwtcdjinsAmMRjwFrjDF/C3rpdeCKwJ+vAF7r6NgixRhzuzEmxxiTi/1ePzTGfBv4CDg/cFmXemYAY8wuYJuIDA2cmg6spgt/19husiki4g38XW985i79XQc52Hf7OvCdwCyzKUBpY7dae+lizFYQkdOx/8J1AbONMX9wOKSwE5FjgE+BFTSNT/wCOy7zPNAf+z/qBcaY5oOKnZ6InADcYow5U0QGYls2GcBS4DJjTK2T8YWbiIzBTnaIBTYCV2H/8dllv2sR+Q1wEXYm5VLgu9jxhy71XYvIHOAEbLXl3cCdwKuE+G4DCfcf2NloVcBVxphFYYlDk4xSSqlI0e4ypZRSEaNJRimlVMRoklFKKRUxmmSUUkpFjCYZpZRSEaNJRqlOTEROaKwarVQ00iSjlFIqYjTJKNUBROQyEVkgIstE5F+BvWsqROSvIrJERD4QkZ6Ba8eIyJeBfT1eCdrzY7CIvC8iywPvGRS4fVLQnjDPBBbWKRUVNMkoFWEiciR2hfk0Y8wYwAd8G1uccYkxZhzwCXZFNsBTwM+NMaOwlRcazz8DPGCMGY2tt9VY9mMs8GPsPkcDsbXYlIoK7sNfopRqp+nAeGBhoJGRgC1M6AeeC1zzNPCyiKQCacaYTwLnnwReEJFkINsY8wqAMaYGIHC/BcaYgsDxMiAX+Czyj6XU4WmSUSryBHjSGHP7fidFft3sukPVeDpUF1hwjS0f+v+1iiLaXaZU5H0AnC8iWfDNPusDsP//NVb+vRT4zBhTChSLyLGB85cDnwT29CkQkbMD94gTEW+HPoVSbaD/4lEqwowxq0XkV8C7IhID1AM/xG4SNkJEFmN3aLwo8JYrgIcCSaSxMjLYhPMvEflt4B4XdOBjKNUmWoVZKYeISIUxJsnpOJSKJO0uU0opFTHaklFKKRUx2pJRSikVMZpklFJKRYwmGaWUUhGjSUYppVTEaJJRSikVMZpklFJKRcz/B3Ra8FmNrbwTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fit_and_plot(train_features=poly_features_t[1:100,0:20],train_labels=labels[1:100],test_features=poly_features_t[100:,0:20],test_labels=labels[100:],no_inputs=20)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "• Since the generalization error rate cannot be estimated based on the training error rate, simply minimizing the\n",
    "training error rate will not necessarily mean a reduction in the generalization error rate. Machine learning models\n",
    "need to be careful to safeguard against overfitting such as to minimize the generalization error.\n",
    "\n",
    "• A validation set can be used for model selection (provided that it isn’t used too liberally).\n",
    "\n",
    "• Underfitting means that the model is not able to reduce the training error rate while overfitting is a result of the\n",
    "model training error rate being much lower than the testing data set rate.\n",
    "\n",
    "• We should choose an appropriately complex model and avoid using insufficient training samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises\n",
    "\n",
    "1. Can you solve the polynomial regression problem exactly? \n",
    "Hint - use linear algebra.\n",
    "\n",
    "2. Model selection for polynomials\n",
    "    \n",
    "    • Plot the training error vs. model complexity (degree of the polynomial). \n",
    "    What do you observe?\n",
    "    \n",
    "    • Plot the test error in this case.\n",
    "    \n",
    "    • Generate the same graph as a function of the amount of data?\n",
    "\n",
    "3. What happens if you drop the normalization of the polynomial features x\n",
    "i by 1/i!. \n",
    "Can you fix this in some other\n",
    "way?\n",
    "\n",
    "4. What degree of polynomial do you need to reduce the training error to 0?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}