b-vanilla-neural-network-from-scratch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0CggyzPOvQ8d"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/apssouza22/cnn-for-devs/blob/master/b-vanilla-neural-network-from-scratch.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2JTfVbH7FHUx"
   },
   "source": [
    "# Implementing a Backpropagation algorithm from scratch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wCsjDO-bvIqp"
   },
   "source": [
    "\n",
    "\n",
    "In this section, we will learn how to implement the backpropagation algorithm from scratch using Python. \n",
    "\n",
    "**What is Backpropagation?**\n",
    "Back-propagation is the essence of neural net training. It is the method of fine-tuning the weights of a neural net based on the error rate obtained in the previous epoch (i.e., iteration). Proper tuning of the weights allows you to reduce error rates and to make the model reliable by increasing its generalization.\n",
    "\n",
    "Backpropagation is a short form for \"backward propagation of errors.\" It is a standard method of training artificial neural networks. This method helps to calculate the gradient of a loss function with respects to all the weights in the network.\n",
    "\n",
    "The backpropagation algorithm consists of two phases:\n",
    "- 1. The forward pass where we pass our inputs through the network to obtain our output classifications.\n",
    "\n",
    "-  2. The backward pass (i.e., weight update phase) where we compute the gradient of the loss function and use this information to iteratively apply the chain rule to update the weights in our network.\n",
    "\n",
    "<img src=\"https://www.guru99.com/images/1/030819_0937_BackPropaga1.png\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "id": "2DrYmkG701Je"
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "import numpy as np\n",
    "from IPython.display import clear_output\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zFd2HULUWrDo"
   },
   "source": [
    "# Concepts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TFsuJylxWrDp"
   },
   "source": [
    "Before we dive into our neural network implementation, lets look at some important concepts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DSZzUBVPWrDp"
   },
   "source": [
    "## The dot product\n",
    "\n",
    "The dot product of two vectors tells you how similar they are in terms of direction and is scaled by the magnitude of the two vectors.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "8cE_RvheWrDq",
    "outputId": "543a3839-e578-4fc4-a38d-592f26070b67",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dot product 1 is: 2.1672\n",
      "The dot product 2 is: 4.1259999999999994\n",
      "The dot product 2 is closer: 4.1259999999999994 > 2.1672\n"
     ]
    }
   ],
   "source": [
    "input_vector = [1.72, 1.23]\n",
    "weights_1 = [1.26, 0]\n",
    "weights_2 = [2.17, 0.32]\n",
    "\n",
    "# Computing the dot product of input_vector and weights_1\n",
    "first_indexes_mult = input_vector[0] * weights_1[0]\n",
    "second_indexes_mult = input_vector[1] * weights_1[1]\n",
    "dot_product_1 = first_indexes_mult + second_indexes_mult\n",
    "\n",
    "print(f\"The dot product 1 is: {dot_product_1}\")\n",
    "\n",
    "#use np instead\n",
    "dot_product_2 = np.dot(input_vector, weights_2)\n",
    "print(f\"The dot product 2 is: {dot_product_2}\")\n",
    "print(f\"The dot product 2 is closer: {dot_product_2} > {dot_product_1}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1EuHSBJuWrDr"
   },
   "source": [
    "## Matrix multiplication\n",
    "\n",
    "The fundamental operations of any typical neural network can be reduced to a bunch of addition and multiplication operations. Neural networks can be expressed in terms of matrices. Matrix multiplication is one of the most important mathematical operations when it comes to deep neural networks.\n",
    "\n",
    "Below a very simple \"neural net\" for helping to understand matrix multiplication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ayYEuwR5WrDs",
    "outputId": "5421f15b-58e7-4b37-c5e5-426d368fe488",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 0]  ->  [[0.]]\n",
      "[1, 0]  ->  [[-2.]]\n",
      "[0, 1]  ->  [[2.]]\n",
      "[1, 1]  ->  [[0.]]\n"
     ]
    }
   ],
   "source": [
    "#weights for 2 neurons\n",
    "weights_for_layer1 = np.array([\n",
    "\t[1.5, -0.5],\n",
    "\t[-1, 1]\n",
    "])\n",
    "\n",
    "#weights for 1 neurons\n",
    "weights_for_layer2 = np.array([\n",
    "\t[-1],\n",
    "\t[1],\n",
    "])\n",
    "\n",
    "def neural_net(inputs, layer_weights):\n",
    "\toutputs = inputs\n",
    "\t#The output of a layer become the input for the subsequent layer\n",
    "\tfor layer_weight in layer_weights:\n",
    "\t\tinputs = outputs\n",
    "\t\toutputs = np.matmul(inputs, layer_weight)\n",
    "\treturn outputs\n",
    "\n",
    "\n",
    "inputs = [\n",
    "\t[0, 0],\n",
    "\t[1, 0],\n",
    "\t[0, 1],\n",
    "\t[1, 1]\n",
    "]\n",
    "\n",
    "for i in inputs:\n",
    "\tprint(\n",
    "\t\ti,\n",
    "\t\t\" -> \",\n",
    "\t\tneural_net(\n",
    "\t\t\tinputs=np.array([i]),\n",
    "\t\t\tlayer_weights=[weights_for_layer1, weights_for_layer2]\n",
    "\t\t)\n",
    "\t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZM9aKUkvWrDs"
   },
   "source": [
    "# Implementing a Backpropagation algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fQgU_Z8q1JeB"
   },
   "source": [
    "A neural network is compound by multiple layers.\n",
    "\n",
    "Every layer will have a forward pass and backpass implementation. Let's create a main class layer which can do a forward pass .forward() and Backward pass .backward().\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "id": "lD0VYK4I1nTN"
   },
   "outputs": [],
   "source": [
    "class Layer:\n",
    "\t\"\"\"\n",
    "    A building block. Each layer is capable of performing two things:\n",
    "\n",
    "    - Process input to get output:           output = layer.forward(input)\n",
    "    \n",
    "    - Propagate gradients through itself:    grad_input = layer.backward(input, grad_output)\n",
    "    \n",
    "    Some layers also have learnable parameters which they update during layer.backward.\n",
    "    \"\"\"\n",
    "\n",
    "\tdef __init__(self):\n",
    "\t\t\"\"\"Here we can initialize layer parameters (if any) and auxiliary stuff.\"\"\"\n",
    "\t\t# A dummy layer does nothing\n",
    "\t\tpass\n",
    "\n",
    "\tdef forward(self, input):\n",
    "\t\t\"\"\"\n",
    "        Takes input data of shape [batch, input_units], returns output data [batch, output_units]\n",
    "        \"\"\"\n",
    "\t\t# A dummy layer just returns whatever it gets as input.\n",
    "\t\treturn input\n",
    "\n",
    "\tdef backward(self, input, grad_output):\n",
    "\t\t\"\"\"\n",
    "        Performs a backpropagation step through the layer, with respect to the given input.\n",
    "        \n",
    "        To compute loss gradients w.r.t input, we need to apply chain rule (backprop):\n",
    "        \n",
    "        d loss / d x  = (d loss / d layer) * (d layer / d x)\n",
    "        \n",
    "        Luckily, we already receive d loss / d layer as input, so you only need to multiply it by d layer / d x.\n",
    "        \n",
    "        If our layer has parameters (e.g. linear layer), we also need to update them here using d loss / d layer\n",
    "        \"\"\"\n",
    "\t\treturn input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d7_1NJKN748m"
   },
   "source": [
    "### Linear layer\n",
    "\n",
    "A linear layer applies affine transformation. In a vectorized form, it can be described as:\n",
    "$$f(X)= W \\cdot X + \\vec b $$\n",
    "\n",
    "Both W and b are initialized during layer creation and updated each time backward is called. Note that we are using **Xavier initialization** which is a trick to train our model to converge faster [read more](http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization). Instead of initializing our weights with small numbers which are distributed randomly we initialize our weights with mean zero and variance of 2/(number of inputs + number of outputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-03T19:04:12.511355-05:00",
     "start_time": "2018-06-03T19:04:12.492305Z"
    },
    "id": "ksQ6fD_Z748m"
   },
   "outputs": [],
   "source": [
    "class Linear(Layer):\n",
    "\tdef __init__(self, input_size, output_size, learning_rate=0.1):\n",
    "\t\t\"\"\"\n",
    "        A linear layer is a layer which performs a learned affine transformation:\n",
    "        f(x) = <W*x> + b\n",
    "        \"\"\"\n",
    "\t\tself.learning_rate = learning_rate\n",
    "\t\tself.weights = np.random.normal(loc=0.0,\n",
    "\t\t\t\t\t\t\t\t\t\tscale=np.sqrt(2 / (input_size + output_size)),\n",
    "\t\t\t\t\t\t\t\t\t\tsize=(input_size, output_size))\n",
    "\t\tself.biases = np.zeros(output_size)\n",
    "\n",
    "\tdef forward(self, img):\n",
    "\t\t\"\"\"\n",
    "        Perform forward activation: f(x) = <W*x> + b\n",
    "        \n",
    "        input shape: [batch, input_size]\n",
    "        output shape: [batch, output_size]\n",
    "        \"\"\"\n",
    "\t\tinput = img.reshape([img.shape[0], -1])\n",
    "\t\treturn np.dot(input, self.weights) + self.biases\n",
    "\n",
    "\tdef backward(self, input, grad_output):\n",
    "\t\t# .T transposed weights\n",
    "\t\tgrad_input = np.dot(grad_output, self.weights.T)\n",
    "\n",
    "\t\t# compute gradient with respect to weights and biases\n",
    "\t\tgrad_weights = np.dot(input.T, grad_output)\n",
    "\t\tgrad_biases = grad_output.mean(axis=0) * input.shape[0]\n",
    "\n",
    "\t\tassert grad_weights.shape == self.weights.shape and grad_biases.shape == self.biases.shape\n",
    "\n",
    "\t\t# Here we perform a stochastic gradient descent step. Updating weights and biases to all layers in the network\n",
    "\t\tself.weights = self.weights - self.learning_rate * grad_weights\n",
    "\t\tself.biases = self.biases - self.learning_rate * grad_biases\n",
    "\n",
    "\t\treturn grad_input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6k4MyzE9748n"
   },
   "source": [
    "### The loss function\n",
    "\n",
    "Cross-entropy loss, or log loss, measures the performance of a classification model whose output is a probability value between 0 and 1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-03T19:04:47.947883-05:00",
     "start_time": "2018-06-03T19:04:47.939333Z"
    },
    "id": "5su5_htL748n"
   },
   "outputs": [],
   "source": [
    "def crossentropy_loss(predictions, labels):\n",
    "\t\"\"\"Calculate the softmax cross entropy loss for the predictions\"\"\"\n",
    "\tpredictions_for_answers = predictions[np.arange(len(predictions)), labels]\n",
    "\txentropy = - predictions_for_answers + np.log(np.sum(np.exp(predictions), axis=-1))\n",
    "\treturn xentropy\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GOqtHXxmIf98"
   },
   "source": [
    "## Why derivative/gradient is used ?\n",
    "\n",
    "When updating the curve, to know in which direction and how much to change or update the curve depending upon the slope.\n",
    "\n",
    "The error is given by the y-axis. If you’re in point A and want to reduce the error toward 0, then you need to bring the x value down. On the other hand, if you’re in point B and want to reduce the error, then you need to bring the x value up. To know which direction you should go to reduce the error, you’ll use the derivative. \n",
    "\n",
    "<img src=\"https://files.realpython.com/media/quatratic_function.002729dea332.png\" width=\"500px\">\n",
    "\n",
    "The gradient is a vector; it points in the direction and derivative is a rate of change of , which can be thought of the slope of the function at a point .\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "id": "eD8__mJnIiUC"
   },
   "outputs": [],
   "source": [
    "def crossentropy_loss_grad_derivative(predictions, labels):\n",
    "\t\"\"\"Calculate the direction rate of the slope\"\"\"\n",
    "\tones_for_answers = np.zeros_like(predictions)\n",
    "\tones_for_answers[np.arange(len(predictions)), labels] = 1\n",
    "\n",
    "\tsoftmax = np.exp(predictions) / np.exp(predictions).sum(axis=-1, keepdims=True)\n",
    "\n",
    "\treturn (- ones_for_answers + softmax) / predictions.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "shEEaQxS3tO8"
   },
   "source": [
    "## Learning rate\n",
    "\n",
    "Is the step size at each iteration while moving toward a minimum of a loss function. Learning rate can not be too small or too big. \n",
    "\n",
    "\n",
    "<img src=\"https://srdas.github.io/DLBook/DL_images/TNN2.png\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Activation function (Relu)\n",
    "In a neural network, the purpose of an activation function is to add non-linearity to the neural network.\n",
    "\n",
    "They allow backpropagation because now the derivative function would be related to the input, and it’s possible to go back and understand which weights in the input neurons can provide a better prediction.\n",
    "\n",
    "They allow the stacking of multiple layers of neurons as the output would now be a non-linear combination of input passed through multiple layers. Any output can be represented as a functional computation in a neural network.\n",
    "\n",
    "The rectified linear activation function is a simple calculation that returns the value provided as input directly, or the value 0.0 if the input is 0.0 or less.\n",
    "\n",
    "```\n",
    "if input > 0:\n",
    "\treturn input\n",
    "else:\n",
    "\treturn 0\n",
    "```\n",
    "\n",
    "<img src=\"https://machinelearningmastery.com/wp-content/uploads/2018/10/Line-Plot-of-Rectified-Linear-Activation-for-Negative-and-Positive-Inputs.png\" width=\"50%\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "id": "VKn2K0Ud0Qca"
   },
   "outputs": [],
   "source": [
    "class ReLU(Layer):\n",
    "\tdef __init__(self):\n",
    "\t\t\"\"\"ReLU layer simply applies elementwise rectified linear unit to all inputs\"\"\"\n",
    "\t\tpass\n",
    "\n",
    "\tdef forward(self, input):\n",
    "\t\t\"\"\"Apply elementwise ReLU to [batch, input_units] matrix\"\"\"\n",
    "\t\trelu_forward = np.maximum(0, input)\n",
    "\t\treturn relu_forward\n",
    "\n",
    "\tdef backward(self, input, grad_output):\n",
    "\t\t\"\"\"Compute gradient of loss with respect to ReLU input\"\"\"\n",
    "\t\trelu_grad = input > 0\n",
    "\t\treturn grad_output * relu_grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Y9hJvYimE6K_"
   },
   "source": [
    "# Train neural network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EyOk3mmp748n"
   },
   "source": [
    "## Loading the dataset\n",
    "Now let's combine what we've just built into a working neural network. As I have told earlier, we are going to use MNIST data of handwritten digit for our example.\n",
    "\n",
    "\n",
    "This subset of the MNIST dataset is built-into the scikit-learn library and includes 1,797 example digits, each of which are 8 × 8 grayscale images (the original images are 28 × 28. When flattened, these images are represented by an 8 × 8 = 64-dim vector.\n",
    "\n",
    "<img src=\"https://nvsyashwanth.github.io/machinelearningmaster/assets/images/digitsMNIST/samples.png\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "gpl9LZQ7Ej87",
    "outputId": "f6080595-5d12-4290-8f83-2e0712ffd153"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] loading MNIST (sample) dataset...\n",
      "[INFO] samples: 1797, dim: 64\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn import datasets\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def load_dataset(flatten=False):\n",
    "\t# load the MNIST dataset and apply min/max scaling to scale the\n",
    "\t# pixel intensity values to the range [0, 1] (each image is\n",
    "\t# represented by an 8 x 8 = 64-dim feature vector)\n",
    "\tprint(\"[INFO] loading MNIST (sample) dataset...\")\n",
    "\tdigits = datasets.load_digits()\n",
    "\tdata = digits.data.astype(\"float\")\n",
    "\tdata = (data - data.min()) / (data.max() - data.min())  # normalization\n",
    "\tprint(\"[INFO] samples: {}, dim: {}\".format(data.shape[0], data.shape[1]))\n",
    "\t#X = data, Y = label\n",
    "\tX_train, X_test, y_train, y_test = train_test_split(data, digits.target, test_size=0.25)\n",
    "\n",
    "\t# we reserve the last 300 training examples for validation\n",
    "\tX_train, X_val = X_train[:-300], X_train[-300:]\n",
    "\ty_train, y_val = y_train[:-300], y_train[-300:]\n",
    "\n",
    "\treturn X_train, y_train, X_val, y_val, X_test, y_test\n",
    "\n",
    "\n",
    "X_train, y_train, X_val, y_val, X_test, y_test = load_dataset(flatten=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GzZfmc9SkO0Z"
   },
   "source": [
    "# Training loop\n",
    "\n",
    "We split data into minibatches, feed each such minibatch into the network and update weights. This training method is called a mini-batch stochastic gradient descent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-03T18:44:01.727147-05:00",
     "start_time": "2018-06-03T18:44:01.719126Z"
    },
    "id": "-chb7zrr748p"
   },
   "outputs": [],
   "source": [
    "from tqdm import trange\n",
    "\n",
    "\n",
    "def iterate_minibatches(inputs, targets, batchsize, shuffle=False):\n",
    "\tassert len(inputs) == len(targets)\n",
    "\tif shuffle:\n",
    "\t\tindices = np.random.permutation(len(inputs))\n",
    "\n",
    "\tfor start_idx in trange(0, len(inputs) - batchsize + 1, batchsize):\n",
    "\t\tif shuffle:\n",
    "\t\t\texcerpt = indices[start_idx:start_idx + batchsize]\n",
    "\t\telse:\n",
    "\t\t\texcerpt = slice(start_idx, start_idx + batchsize)\n",
    "\t\tyield inputs[excerpt], targets[excerpt]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "duseU63wqyfc"
   },
   "source": [
    "## Creating our network model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "id": "qWWJM69RXDuN"
   },
   "outputs": [],
   "source": [
    "num_classes=10 # numbers from 0-9\n",
    "input_size = 8*8 # our image is size 8 x 8 \n",
    "class NeuralNetwork:\n",
    "\tdef __init__(self, learning_rate=0.1):\n",
    "\t\t# [input_size, 32, 16, 10]       \n",
    "\t\t# We'll define network as a list of layers, each applied on top of previous one. In this setting, computing predictions and training becomes trivial. \n",
    "\t\tself.network = []\n",
    "\t\tself.network.append(Linear(input_size, 100, learning_rate))\n",
    "\t\tself.network.append(ReLU())\n",
    "\t\tself.network.append(Linear(100, 200, learning_rate))\n",
    "\t\tself.network.append(ReLU())\n",
    "\t\tself.network.append(Linear(200, num_classes, learning_rate))\n",
    "\n",
    "\tdef forward(self, img):\n",
    "\t\t\"\"\"\n",
    "\t\tCompute activations of all network layers by applying them sequentially.\n",
    "\t\tReturn a list of activations for each layer. \n",
    "\t\t\"\"\"\n",
    "\t\tpredictions = []\n",
    "\t\tinput = img\n",
    "\n",
    "\t\t# Looping through each layer\n",
    "\t\tfor l in self.network:\n",
    "\t\t\tpredictions.append(l.forward(input))\n",
    "\t\t\tinput = predictions[-1] # Updating input to last layer output\n",
    "\n",
    "\t\tassert len(predictions) == len(self.network)\n",
    "\t\treturn predictions\n",
    "\n",
    "\tdef predict(self, img):\n",
    "\t\t\"\"\"\n",
    "\t\tCompute network predictions. Returning indices of largest probability\n",
    "\t\t\"\"\"\n",
    "\t\tprediction = self.forward(img)[-1]\n",
    "\t\treturn prediction.argmax(axis=-1)\n",
    "\n",
    "\tdef train(self, epoch, img_train, label_train, img_val, label_val):\n",
    "\t\t\"\"\"\n",
    "\t\tTrain our network on a given batch of X and y.\n",
    "\t\tWe first need to run forward to get all layer activations.\n",
    "\t\tThen we can run layer.backward going from last to first layer.\n",
    "\t\tAfter we have called backward for all layers, all Linear layers have already made one gradient step.\n",
    "\t\t\"\"\"\n",
    "\t\ttrain_logs = []\n",
    "\t\tval_logs = []\n",
    "\t\tfor epoch in range(epoch):\n",
    "\t\t\tfor x_batch, y_batch in iterate_minibatches(img_train, label_train, batchsize=32, shuffle=True):\n",
    "\t\t\t\tloss, loss_grad, layer_inputs = self.training_batch(x_batch, y_batch)\n",
    "\t\t\t\tself.backward(layer_inputs, loss_grad)\n",
    "\n",
    "\t\t\ttrain_log, val_log = self.validation_epoch_end(img_train, label_train, img_val, label_val)\n",
    "\t\t\ttrain_logs.append(loss)\n",
    "\t\t\tval_logs.append(val_log)\n",
    "\n",
    "\t\treturn train_logs, val_logs\n",
    "\n",
    "\tdef training_batch(self, img, label):\n",
    "\t\t# Get the layer activations\n",
    "\t\tlayer_predictions = self.forward(img)\n",
    "\t\tlayer_inputs = [img] + layer_predictions  # layer_input[i] is an input for network[i]\n",
    "\t\tlast_prediction = layer_predictions[-1]\n",
    "\n",
    "\t\t# Compute the loss and the initial gradient\n",
    "\t\tloss = crossentropy_loss(last_prediction, label)\n",
    "\t\tloss_grad = crossentropy_loss_grad_derivative(last_prediction, label)\n",
    "\n",
    "\t\treturn np.mean(loss), loss_grad, layer_inputs\n",
    "\n",
    "\tdef validation_epoch_end(self, img_train, label_train, img_val, label_val):\n",
    "\t\ttrain_log = np.mean(self.predict(img_train) == label_train)\n",
    "\t\tval_log = np.mean(self.predict(img_val) == label_val)\n",
    "\t\treturn val_log, train_log\n",
    "\n",
    "\tdef backward(self, layer_inputs, loss_grad):\n",
    "\t\t# Propagate gradients through the network\n",
    "\t\t# Reverse loop as this is backprop\n",
    "\t\tfor layer_index in range(len(self.network))[::-1]:\n",
    "\t\t\tlayer = self.network[layer_index]\n",
    "\t\t\tlayer_input = layer_inputs[layer_index]\n",
    "\t\t\tloss_grad = layer.backward(layer_input, loss_grad)  # grad w.r.t. input, also weight updates\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-06-03T18:49:20.383230-05:00",
     "start_time": "2018-06-03T18:47:53.604185Z"
    },
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 300
    },
    "id": "0zhhjgoR748q",
    "outputId": "e2fc1614-f29f-4f50-ef87-d641c9bb794b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Losses: 0.03483389020421904\n",
      "Val accuracy: 0.9952244508118434\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAD4CAYAAAATpHZ6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3hUVfrA8e+ZmRTSGxlaClV6gARpCkEs2LCBoi4Ciuiqu+v6W1fXtbLurqu7rmtbFuvaFrFjA0GJqBQpBimh1wRIQiBlgJSZOb8/ZhICpEwmEyZz836eZ57M3Ln3zHnnZt45c+659yitNUIIIYzB5O8KCCGE8B1J6kIIYSCS1IUQwkAkqQshhIFIUhdCCAOx+OuFExISdGpqqlfbHj16lPDwcN9WyM+MFpPR4gHjxWS0eMB4MdUVz5o1aw5prdvXt43fknpqaiqrV6/2atusrCwyMzN9WyE/M1pMRosHjBeT0eIB48VUVzxKqT0NbSPdL0IIYSCS1IUQwkAkqQshhIH4rU9dCNE6VVVVkZubS3l5ub+r0mTR0dHk5OT4uxo+ERoailKqydtJUhdCnCQ3N5fIyEhSU1O9Sir+VFZWRmRkpL+r0Wxaa4qKirwaySPdL0KIk5SXlxMfHx9wCd1IlFLEx8djNpubvK0kdSHEaSSh+5+3+yDgkvqWg2W8v7WS4mOV/q6KEEK0OgGX1HcXHeWznVXkHjnu76oIIVpIRESEv6sQsAIuqVujQgHILw28I/NCCNHSAjCphwCQX1rh55oIIVqa1pp7772X/v37M2DAAN59910ADhw4wOjRoxk0aBD9+/fnu+++w+FwcPvtt9es+89//hOAHTt2MH78eNLT0zn33HPZvHkzAO+99x79+/cnLS2N0aNH+y1GXwu4IY0JESEopKUuxJnw2Kcb2bS/1Kdl9u0UxSOX9/No3Q8//JDs7GzWrVvHoUOHGDp0KKNHj+add97hoosu4o9//CMOh4Njx46RnZ3NgQMH2LBhAwDFxcUAzJw5k9mzZ9OzZ09WrlzJHXfcwTfffMOsWbNYuHAhnTt3rlnXCAIuqQeZTUQGKwrKJKkLYXTff/89119/PWazGavVypgxY1i1ahVDhw7l5ptvpqqqiiuvvJJBgwbRrVs3du3axa9+9SsuvfRSLrzwQmw2G8uWLWPSpEk1ZVZUuH7ljxo1imnTpnHttddy9dVX+ytEnwu4pA4QG6qk+0WIM8DTFnVL0VrXuXz06NEsXbqUzz//nClTpnDvvfdy0003sWzZMpYtW8YLL7zAvHnzeOaZZ4iJiSE7O/u0MmbPns3KlSv5/PPPGTRoENnZ2cTHx7d0SC2u0T51pdSrSqkCpdSGRtYbqpRyKKUm+q56dYsOUdL9IkQbMHr0aN59910cDgeFhYUsXbqUs88+mz179pCYmMitt97KLbfcwtq1azl06BBOp5NrrrmGP/3pT6xdu5aoqCi6du3Ke++9B7i+JNatWwe4+tqHDRvGrFmzSEhIYN++ff4M1Wc8aam/DjwPvFHfCkopM/A3YKFvqtWwmBBFTom01IUwuquuuorly5eTlpaGUoonn3ySDh068N///pennnqKoKAgIiIieOONN8jLy2Pq1Kk12/71r38F4O233+aXv/wljz/+OFVVVUyePJm0tDTuvfdetm3bhtaacePGkZaW5q8wfarRpK61XqqUSm1ktV8BHwBDfVCnRsWGKA7ZKrA7nFjMATeARwjRCJvNBrjOqnzqqad46qmnTnp+6tSpJyXwat99991p137p2rUrCxYsOG3dDz/80Ic1bj2anRGVUp2Bq4DZza+OZ2JCFFrDIZucVSqEELX54kDpM8B9WmtHY9cqUErNBGYCWK1WsrKyvHrBUF0BKL7I+oFu0U2/4E1rZLPZvH4/WiOjxQPGi6m+eKKjoykrKzvzFfIBh8MRsHWvi9a6yf9zvkjqGcBcd0JPAC5RStm11h/XUcE5wByAjIwM7e1cgrs/+Roop0uPfmT26+BtvVuVtjC3YqAzWkz1xZOTkxOwl681yqV3qymlmvw/1+ykrrXuWqsCrwOf1ZXQfSkmxPWLIL9MDpYKIURtjSZ1pdT/gEwgQSmVCzwCBAForc9YP3ptUSEKk4ICGdYohBAn8WT0y/WeFqa1ntas2njIpBQJESEUyAlIQghxkoAdD2iNCiVfLhUghEAu1VtbACf1ELlUgBCiVbDb7f6uQo2ATeqJUaHSpy6EAd133328+OKLNY8fffRR/vGPf2Cz2Rg3bhxDhgxhwIABfPLJJ42WdeWVV5Kenk6/fv2YM2dOzfIFCxYwZMgQ0tLSGDduHOAa4jl9+nQGDBjAwIED+eCDD4CTfwW8//77TJs2DYBp06Zxzz33MHbsWO677z5+/PFHRo4cyeDBgxk5ciRbtmwBXMMsf/e739WU+9xzz/H1119z1VVX1ZS7aNEin11ULCAv6AVgjQyl6GgllXYnwZaA/W4SonX78n44uN63ZXYYABc/Ue/TkydP5u677+aOO+4AYN68eSxYsIDQ0FA++ugjoqKiOHToEMOHD2fChAkNzuX56quvEhcXx/Hjxxk6dCjXXHMNTqeTW2+9laVLl9K1a1cOHz4MwJ/+9Ceio6NZv94V75EjRxoNZevWrSxevBiz2UxpaSlLly7FYrGwePFiHnjgAT744APmzJnDrl27+Omnn7BYLBw+fJjY2FjuvPNOCgsLad++Pa+99hrTp09vyrtYr8BN6u7JMgptFXSOaefn2gghfGXw4MEUFBSwf/9+CgsLiY2NJTk5maqqKh544AGWLl2KyWQiLy+P/Px8OnSo/1yVZ599lo8++giAffv2sW3bNgoLCxk9ejRdu7pGY8fFxQGwePFi5s6dW7NtbGxso3WdNGkSZrPrBMiSkhKmTp3Ktm3bUEpRVVVVU+7tt9+OxWI56fWmTJnCW2+9xfTp01m+fDlvvFHv5bWaJGCTemLNDEjlktSFaCkNtKhb0sSJE3n//fc5ePAgkydPBlwX5iosLGTNmjUEBQWRmppKeXn9XbBZWVksXryY5cuXExYWRmZmJuXl5Wit62zd17e89rJTXy88PLzm/kMPPcTYsWP56KOP2L17d81JQ/WVO336dC6//HJCQ0OZNGlSTdJvroDtt0iMdM1VKsMahTCeyZMnM3fuXN5//30mTnRdzbukpITExESCgoJYsmQJe/bsabCMkpISYmNjCQsLY/PmzaxYsQKAESNG8O2337Jr1y6Amu6XCy+8kOeff75m++ruF6vVSk5ODk6ns6bVX9/rde7cGYDXX3+9ZvmFF17I7Nmzaw6mVr9ep06d6NSpE48//nhNP70vBGxSr56AWmZAEsJ4+vXrR1lZGZ07d6Zjx44A3HjjjaxevZqMjAzefvttevfu3WAZ48ePx263M3DgQB566CGGDx8OQPv27ZkzZw5XX301aWlpXHfddQA8+OCDHDlypGbe0iVLlgDwxBNPcNlll3HeeefV1KUuv//97/nDH/7AqFGjcDgcNctnzJhBcnIyAwcOJC0tjXfeeafmuRtvvJGkpCT69u3r3RtVF621X27p6enaW0uWLNEOh1N3+8Pn+skFOV6X05osWbLE31XwKaPFo7XxYqovnk2bNp3ZivhQaWmpv6vQJHfeead++eWX631+7dq1py0DVusGcmvA9qmbTIrESBmrLoQITOnp6YSHh/OPf/zDp+UGbFIH11h1mdZOCBGI1qxZ0yLlBmyfOoA1Uq7/IkRL0PVM+CzOHG/3QUAn9cSoELn+ixA+FhoaSlFRkSR2P9JaU1RUdNIBV08FdPeLNTKU4mNVVNgdhFiMMQOSEP7WpUsXcnNzKSws9HdVmqy8vJzQ0FB/V8MnQkNDOXr0aJO3C+ykHnVirHpSXJifayOEMQQFBdWcbRlosrKyGDx4sL+r4TONjcWvS8B3v4CMVRdCiGoBndSrW+oyrFEIIVwMktSlpS6EEBDgST02LIggs5KWuhBCuDWa1JVSryqlCpRSG+p5/kal1M/u2zKlVJrvq1lv3UiMlMkyhBCimict9deB8Q08vwsYo7UeCPwJmNPAuj6XGBVCQZm01IUQAjxI6lrrpcDhBp5fprWuniJkBdDFR3XziDVSLhUghBDVlCdnjSmlUoHPtNb9G1nvd0BvrfWMep6fCcwEsFqt6bVnGWkKm81WM2/gm5sqWL7fzovnhzeyVetWOyYjMFo8YLyYjBYPGC+muuIZO3bsGq11Rr0bNXQJx+obkApsaGSdsUAOEO9Jmc299G6157/ZplPu+0wfq7B7XV5r0FYu6xrIjBaT0eLR2ngx1RUPjVx61yejX5RSA4GXgSu01kW+KNNTMlmGEEKc0OykrpRKBj4Epmittza/Sk2TGFk9V6kcLBVCiEav/aKU+h+QCSQopXKBR4AgAK31bOBhIB540T25ql031N/jY3ICkhBCnNBoUtdaX9/I8zOAOg+MngnWmuu/SEtdCCEC+oxSgOh2QQRbTHICkhBCYICkrpTCGhUi3S9CCIEBkjpUn4Ak3S9CCGGMpB4VKtPaCSEEBknq7WUCaiGEAAyS1K1Rodgq7BytsPu7KkII4VcGSeoyrFEIIcAwSV1OQBJCCDBMUq++VIAkdSFE22aIpJ5YfVEvOVgqhGjjDJHUI0MstAsyS0tdCNHmGSKpK6VIjAohXw6UCiHaOEMkdXCdVSrXfxFCtHWGSeoyAbUQQhgoqVujXBNQaw/mXBVCCKMyUFIP4VilA5ucVSqEaMMMlNSrT0CSLhghRNtlmKSeGFk9Vl0Olgoh2q5Gk7pS6lWlVIFSakM9zyul1LNKqe1KqZ+VUkN8X83GJVafVSqX4BVCtGGetNRfB8Y38PzFQE/3bSbw7+ZXq+msclapEEJ4NPH0UqVUagOrXAG8oV3DTlYopWKUUh211gd8VEePRIRYCA82S5+6EA1xOsFRCY5KLFU2cDrAZG5mmQ4oL4Fjh6HqGIREQGgMhESBudEUUz+toeo42D389a01wRVHoHhfTYw4KsFe676jChwV4LSDMrtir/lrApPllGVmQLti1A7XdjX3ne6/7sfa6Xls8T2hQ3+v3pbGNOMdr9EZ2Ffrca572WlJXSk1E1drHqvVSlZWllcvaLPZ6tw20uJkw469ZGUVeFWuP9UXU6DydTxm+zGCKw/jMLfDbonAaQoGpXxWfkOCKouJPbKO9rYCtu/7GKWd7psDpZ2As9ay6uV2TM6qmr8mp/20ZUrb0cqM0xSM0xSCw+z66zQF4zCHuJe5HmtldpdTgdlRiclZ4b5fgclZWfPXtbz69Wq/rvs1OZF4zgH4Aezmdtgt4fXeHOYwTM4qLPZSgqrKCKqyYbGXue+XYbHbUNQ9lNhuDnWXcWq5YSjtxOw47r6V1/y12E8sq11fT4wEWO7tnj5z9iZdzc7uUxtdz5vPkS+Sel2frDr3sNZ6DjAHICMjQ2dmZnr1gllZWdS1berW5TicmszMkV6V60/1xdTiHFVQdhCC2kFwBFhCfJIsvY7n6CEo3AKHtkDhVvffLVCad/J6JguERp98C4k6cT82FToNBms/V2xN4bBD3hrYvgi2L4b92dTzL+2mTm7dmSxgDnbfglzvqaX6cdiJZaYgVwuv6rj7ZgN7Ya3Hx12tylNfK6id62Zx/w0KhbAwsMS6l4eAOeTk1zcHuR+fuL99xw56JFmxlJdgKS9xtbZrbgVQUgIVpSdaoMER0C4OwmIhphOExbkfu/+2i4XgMKiw1ZRjKS/BUnFq2XugrNT1XgVHuFr24REQ3AGCw12PgyNP3Le08/h/cuv2nfTq098Va8177n4far8nJsvJrezarW6n/eRlKDC5W/F1tu7d+1x5Pu4kOSye5IjERtfz5nPki6SeCyTVetwF2O+DcpvMGhXKT3uL/fHSgeF4MeRvgIMb4OB6yF8PBTmun6XVlPn0D1VwxIkPnznIo5c668BBKP2gjg+C6fQPRul+OLTVlbyPHz5RSFAYJPSElFHQvhdEJ0PV0VrJofTkZFF6wH2/+MRPdmWG9r2hYxp0GuT622GAK7bayg66Evj2xbBjiasMZYIuZ8N5f4Qe5/P9xjzOGT2mjg92C/5icDrBftz15WsJ9dmXLkBuZRY9GksYWkOlzZ0kQ3zyui1p//Eseg3J9Hc1/MoXSX0+cJdSai4wDCg50/3p1WqfVarO0E/zZtMaivcSWboV9oSc3O9Xc78S7BWu+057rVZX8CktktqtEQuU5LmT+HpXIi/Ze+J1wxJcyW3YbRDX3VV2ZZmrlVV51PVBrig7cf/oIdfzTodHYcWVl4Ntw+mtntp9k9UtwHZx0P4s6HO562/CWa4kHtXF9SXgzXtass/Vwj6wznXbvgjWveN6XplcfZqdBkF4e9j1res9AojoAL0vg57nQ7dMV+vTzb61BEIim16f5jCZTv8COpOUOvMxi2ZpNKkrpf4HZAIJSqlc4BEgCEBrPRv4ArgE2A4cA6a3VGUbkxgZQoXdSelxO9FhnrUozyinE47sggPZJyec8mLSAda2wGtWJ7CkoZAxHToMdB2gibC2aAtzuSc/G7V2JfbmHqg7lVIQk+y69Z1w4rXKDtR637Nh11I4WghJw2HcI9DzArD2P2N99UK0BE9Gv1zfyPMauNNnNWqGmskyysr9m9SrRwPYCuDgzycSycGfXX2U4GpNW/tBv6ugYxo/7yli4KD0OlrftVvgQa4E6LCffHS/9pF9e8WJx+GJkNjH1c/ZGinlHl1whl4rqpPr1vuSE8sd9uaN0BCilTHUf7M1snpauwp6Wn34k1FrV59r9cE7W74raVec0qdb3c9bWXby9pZQV1fHwGuho7tfN7HPSf3Th21Z0D3Td3UWnpGELgzGUP/RzZ6A2umAI7tPHLSr/be6hQ2uLo3aIy1CoyGu2+kjMcLiXck8oZckDyHEGWGoTNPkSwXYK2HvMti60NW/emjbyUPIIqyuhDzw2hMH7xLOgsgO0u8qhGiVDJXUw4ItRIZaGr5UgK0Ati2CrQtcQ9cqy1z91SkjoftYd/I+yzWUrtbIByGECASGSurgGgFzUveL1q4DlFsXuhJ53lpAQ2RHGHAN9LwIuo3x77AxIYTwEcMl9eqx6gAsew6Wv+AayoaCzukw9gHodZFraJ90oQghDMaQSX3V7sOuYYRfPeQ6G/G8h1xjkD04LVcIIQKZ4ZJ6YlQIBaXl6EUPocLi4Pp3XKNRhBCiDTDMzEfVrJGhjNRrUbuWwpj7JKELIdoUwyX1DhEWHrC8Q2VUKqT77YoFQgjhF4ZL6n0L5tPLlMe2tHtdp9oLIUQbYqykXmEjKfsZVjl7sTFqtL9rI4QQZ5yxkvry5zEfK+CvVTdQUCbT2gkh2h7jJPWyg/DDs9D3CnaE9pOkLoRok4yT1Jf8xXW52XGPYI0K8f6iXkIIEcCMkdQLcuCnN2HoDIjv7j6rVFrqQoi2xxhJfdEjrjk1x/wegMTIUAqkpS6EaIMCP6nv/Ba2LYRz73HNag5Yo0IoKKvA6WxoBnghhDAej5K6Umq8UmqLUmq7Uur+Op5PVkotUUr9pJT6WSl1SV3l+JzTCV89CNFJMOz2msWJkSHYnZrDxyrPSDWEEKK1aDSpK6XMwAvAxUBf4HqlVN9TVnsQmKe1HgxMBl70dUXrtP4912V1xz0MQaE1i5s9A5IQQgQoT1rqZwPbtdY7tdaVwFzgilPW0UCU+340sN93VaxH1XH4epZrvs/+E0966sQE1HKwVAjRtiitG+53VkpNBMZrrWe4H08Bhmmt76q1TkfgKyAWCAfO11qvqaOsmcBMAKvVmj537lyvKm2z2ehzeCHdd75BdtqfKI4deNLzh447+d23x5neP5gxXYLqKaV1sdlsRERE+LsaPmO0eMB4MRktHjBeTHXFM3bs2DVa64x6N9JaN3gDJgEv13o8BXjulHXuAf7PfX8EsAkwNVRuenq69tb3Cz/R+i9dtH772jqfL6+y65T7PtP/WrzV69c405YsWeLvKviU0eLR2ngxGS0erY0XU13xAKt1A7nVk+6XXCCp1uMunN69cgswz/0lsRwIBRI8KNsrKXvehUobnP9Ync+HWMzEhQdLn7oQos3xJKmvAnoqpboqpYJxHQidf8o6e4FxAEqpPriSeqEvK1qjaAed9n8JQ26CxN71ruaaq1T61IUQbUujSV1rbQfuAhYCObhGuWxUSs1SSk1wr/Z/wK1KqXXA/4Bp7p8Jvle0g8rgOMh8oMHVEqNCKSiTlroQom3xaDo7rfUXwBenLHu41v1NwCjfVq0evS5k5bD/MCbS2uBq1sgQthwsPSNVEkKI1iIgzyjVJnOj61ijQjlkq8QhZ5UKIdqQgEzqnrBGheBwaoqOSr+6EKLtMGxSrzkBSQ6WCiHaEMMmdblUgBCiLTJwUg8BkGGNQog2xbBJPSEiBKWkpS6EaFsMm9SDzCbiw4NlrLoQok0xbFIHSI4LY3uBzd/VEEKIM8bQSX1IcizrckuosDv8XRUhhDgjDJ3UM1JjqbQ72bhfziwVQrQNhk7qQ1JiAViz+4ifayKEEGeGoZN6YmQoyXFhrN5z2N9VEUKIM8LQSR0gIyWWNXuO0FIXjRRCiNbE8Ek9PTWWQ7ZK9hQd83dVhBCixRk+qWekxAGwZo/0qwshjM/wSb1nYgSRoRZWS1IXQrQBhk/qJpNiSHIsa+RgqRCiDTB8UgfXwdKt+TZKjlX5uypCCNGiPErqSqnxSqktSqntSqn761nnWqXUJqXURqXUO76tZvOkp7rGq6/dK10wQghjazSpK6XMwAvAxUBf4HqlVN9T1ukJ/AEYpbXuB9zdAnX12qCkGMwmJQdLhRCG50lL/Wxgu9Z6p9a6EpgLXHHKOrcCL2itjwBorQt8W83mCQu20K9TlJyEJIQwPNXYSTlKqYnAeK31DPfjKcAwrfVdtdb5GNgKjALMwKNa6wV1lDUTmAlgtVrT586d61WlbTYbERERTdrm7ZwKvt1n58Xzw7CYlFev25K8iak1M1o8YLyYjBYPGC+muuIZO3bsGq11Rn3bWDwot64MeOo3gQXoCWQCXYDvlFL9tdbFJ22k9RxgDkBGRobOzMz04OVPl5WVRVO3tcXtZ9E7P9G+52DSkmK8et2W5E1MrZnR4gHjxWS0eMB4MXkTjyfdL7lAUq3HXYD9dazzida6Smu9C9iCK8m3GtUnIcl4dSGEkXmS1FcBPZVSXZVSwcBkYP4p63wMjAVQSiUAvYCdvqxoc3WIDqVzTDvWSlIXQhhYo0lda20H7gIWAjnAPK31RqXULKXUBPdqC4EipdQmYAlwr9a6qKUq7a2M1FhW7zksF/cSQhiWJ33qaK2/AL44ZdnDte5r4B73rdXKSInlk+z95B45TlJcmL+rI4QQPtcmziitVjNphnTBCCEMqk0l9d4doogIsch4dSGEYbWppG42KQYnx7BmT3HjKwshRABqU0kdID0lli0HSykrl4t7CSGMp80l9YyUOJwaftorrXUhhPG0uaQ+KDkGk5KTkIQQxtTmknpEiIXeHaJk0gwhhCG1uaQOrpOQsvcWY3c4/V0VIYTwqTaZ1NNTYjla6WDzwTJ/V0UIIXyqTSb1jFTXxb3kJCQhhNG0yaTeKTqUDlGhcrBUCGE4bTKpK6VIT41lzW45WCqEMJY2mdTBdXGv/SXl7C8+7u+qCCGEz7ThpC796kII42mzSb1Px0jCgs2S1IUQhtJmk7rFbGJQUoxcsVEIYShtNqmDa7x6zoEyjlbY/V0VIYTwiTaf1B1Ozbp9cnEvIYQxeJTUlVLjlVJblFLblVL3N7DeRKWUVkpl+K6KLWdISixKLu4lhDCQRpO6UsoMvABcDPQFrldK9a1jvUjg18BKX1eypUSFBnGWNVKSuhDCMDxpqZ8NbNda79RaVwJzgSvqWO9PwJNAuQ/r1+LSU2L5ac8RHE7t76oIIUSzWTxYpzOwr9bjXGBY7RWUUoOBJK31Z0qp39VXkFJqJjATwGq1kpWV1eQKA9hsNq+3PVXYsSrKKuy88/kSkiL9d4jBlzG1BkaLB4wXk9HiAePF5E08niR1VceymmatUsoE/BOY1lhBWus5wByAjIwMnZmZ6VElT5WVlYW3256qW9ExXlq/BNW+O5nDU3xSpjd8GVNrYLR4wHgxGS0eMF5M3sTjSdM0F0iq9bgLsL/W40igP5CllNoNDAfmB8rB0qS4drSPDJGTkIQQhuBJUl8F9FRKdVVKBQOTgfnVT2qtS7TWCVrrVK11KrACmKC1Xt0iNfYxpRQZKbFyEpIQwhAaTepaaztwF7AQyAHmaa03KqVmKaUmtHQFz4T0lFj2HT7O5oOl/q6KEEI0i0dHBrXWX2ite2mtu2ut/+xe9rDWen4d62YGSiu92oS0TiREhDDzjTUcPlrp7+oIIYTX2vQZpdUSo0J56aZ0DpaWc/uba6iwO/xdJSGE8IokdbfBybH8fVIaP+4+zAMfbkBrGbcuhAg8ngxpbDMmpHViZ6GNZxZvo0diBL/M7O7vKgkhRJNIUj/Fb8b1ZGfhUf62YDNdE8IY37+jv6skhBAek+6XUyileHLiQAYnx3D3u9mszy3xd5WEEMJjktTrEBpkZs6UDOLDQ5jxxioOlgTU5WyEEG2YJPV6tI8M4ZVpGdjK7cx4YxXHKmUiDSFE6ydJvQG9O0Tx3A2D2bS/lN++m41TruQohGjlJKk34rzeVv54aV8Wbsznqa+2+Ls6QgjRIEnqHrh5VCo3DEvm31k7eH9Nrk/LrrQ7+cdXW1h90HfdO1prZn26iXmr9zW+shDCUGRIoweUUjw2oR97io7yhw9/plN0KCN7JDS73CJbBb98ay0/7j5MOwtMLaugfWRIs8v9OqeAV3/YRXS7IC4Z0JGIENnNQrQV0lL3UJDZxIs3pJMSH86UV3/k2a+3YXc4vS5vy8EyrnjhB7Jzi/n9+LOodMBTCzc3u54VdgePf76JxMgQSo5X8c7KPc0uUwgROCSpN0F0WBAf3jGSywZ25OlFW5k8ZwX7Dh9rcjlf5+Rz9Ys/UGF3Mu+2EdyR2YMLUiy8tyaXn3OLm1XH137Yze6iYyTCg5UAABd2SURBVPx9UhqjesTz0ne7KK+Sa9kI0VZIUm+iqNAg/jV5MM9cN4gtB8u4+F/f8dFPnvWza635z7c7mPHGarq1j2D+XaMYlBQDwITuwcSHB/Po/I1ej7IpKC3nua+3cX4fK6N7tefOsT0oLKvw+XEAIUTrJUndS1cO7swXvzmX3h0i+e276/jN3J8oOV5V7/oVdge/e+9n/vrlZi7p35F5t42gY3S7mufDghS/H9+btXuL+Tg7z6s6PblwC1UOzYOX9gFgRLd4BifHMPvbHVQ1o6tICBE4JKk3Q1JcGHNnDueeC3rx2c8HuORf3/HjrtNnUCosq+D6OSv4YG0ud5/fk+dvGEy7YPNp600c0oW0LtE88eVmbBVNGw2Tva+Y99fkcvM5XUlNCAdcB3jvzOxB7pHjfLpufyMlCCGMQJJ6M1nMJn49rifv3T4Ci1kxec5y/vHVlpqW8ab9pVz5wg9sOlDKCzcM4e7ze6FUXXN5g8mkeGRCPwrKKnhhyXaP6+B0ah6dv5H2kSHcdV6Pk54b1yeR3h0ieTFrh5w8JUQbIEndR4Ykx/L5r8/lmiFdeO6b7UycvZy3Vuxh4uxlOJya924byaUDG7/i45DkWK4e0plXvtvF7kNHPXrtj7PzyN5XzH3je582fFEpxR1je7C9wMZXmw56FZsQInB4lNSVUuOVUluUUtuVUvfX8fw9SqlNSqmflVJfK6VSfF/V1i8ixMJTk9J44YYh7Cq08eDHG+iZ6DogOqBLtMfl3D++N0FmxeOf5zS6rq3CzhNfbiYtKYarB3euc51LB3QkNT6MF5bskMk/hDC4RpO6UsoMvABcDPQFrldK9T1ltZ+ADK31QOB94ElfVzSQXDqwIwvuHs2sK/rx7m0jSIwKbdL2iVGh3HVeTxbn5PPt1sIG131xyXYKyip45PK+mEx1d+uYTYrbx3RnfV4J32071KS6CCECiyct9bOB7VrrnVrrSmAucEXtFbTWS7TW1QO2VwBdfFvNwNMpph03jUglNOj0A6KeuPmcVFLjw5j16cZ6R67sKTrKy9/t4urBnRmSHNtgeVcP6ULH6NAm9dULIQKPauznuFJqIjBeaz3D/XgKMExrfVc96z8PHNRaP17HczOBmQBWqzV97ty5XlXaZrMRERHh1batVV0xZRfYeWZtBdf3Duai1KDTtnl2bTkbixw8cW47YkMb/37+ancV72yu5I/DQukZ692Xjafayj4KZEaLB4wXU13xjB07do3WOqPejbTWDd6AScDLtR5PAZ6rZ91f4GqphzRWbnp6uvbWkiVLvN62taorJqfTqW96ZaXu//ACXVhWftJz320t1Cn3faaf/2abx69xtKJKD571lZ726srmVrdRbWUfBTKjxaO18WKqKx5gtW4gt3rS/ZILJNV63AU4bdCzUup84I/ABK11hQflikYopXjosr4cr3Lw1IITl/21O5w89ulGkuPCuOWcrh6XFxZs4eZRqSzZUsjG/TJNnxBG5ElSXwX0VEp1VUoFA5OB+bVXUEoNBv6DK6EX+L6abVePxAimj0pl3pp9NfOlvrViD9sKbPzx0j5N7rOfMiKVyBALL2btaInqCiH8rNGkrrW2A3cBC4EcYJ7WeqNSapZSaoJ7taeACOA9pVS2Ump+PcUJL/xqXE/XdWE+3UiRrYKnF23lnB4JXNjX2uSyotsFMWVECl+sP8COQlsL1FYI4U8ejVPXWn+hte6lte6utf6ze9nDWuv57vvna62tWutB7tuEhksUTREVGsTvL+rNmj1HuOGllRytdPDw5X3rPTO1MTef05UQi4nZ0loXwnDkjNIAMTG9CwO7RLMlv4wpw1PoZY30uqyEiBAmD03mo5/yyCs+7sNaCiH8TZJ6gDCZFH+7ZiBXDurE3ef3bHZ5M0d3A2DOt9JaF8JIJKkHkD4do3hm8mBiwoKbXVanmHZcPaQzc1fto7Cs4cFKTqcmr/g4y3YcYkNeCZV2uYyvEK2VTF7Zht0+pjvvr8nl1R928X8X9OJASTm7i46yu+gYuw8dZY/7/t7Dx05K5MFmE306RjKgSzQDOkczoHMMPa0RBJmljSCEv0lSb8O6tY/gkgEdeWnpTl7+bidVjhNnF4cGmUiND6d7+3DG9U4kNSGc5LgwjhyrZH1uCevzSvgkez9vrdgLQLDFRN+OUa4k3yWaqjInWmuvD+YKIbwjSb2Nu/eiswg2m7BGh5IaH0ZqfDipCeEkRobUm5AvG9gJcHXL7Dl8jPV5JazPLebn3BI++imPN1e4Jrt+ceMSzuudyHl9EhnRLd7r6+C0BFuFnVW7DnPkWCUT0jphkV8Zog55xceJDw9uVf+7jZGk3salxIfz9HWDvNrWZFJ0TQina0I4E9JOJPpdRUd5a+Fy8pxRfLA2lzdX7KFdkJlRPRIY1yeR83onYm3ilSub63ilg9V7DrNsRxHLdxSxPq8Eh3vSkAUbDvLs9YPP6Ae3yuHkq435hIe43hfpump91uw5wvUvraBH+wjevOVs4iNC/F0lj0hSFz5lMim6t49gTJcgMjMzKK9ysHLXYb7OyefrnAIW5+QDMKBzNOf1TmRcn0T6d4qu97LB3iqvcvDT3mKW7zjE8p1FZO8rpsqhsZgUg5JiuCOzOyO6xbP5YBmzPtvE1Fd/5KWpGUSFnn7hNF+yO5x8kr2ff329jb2HXRc2jQ0L4uIBHbl8YCfO7hqH2cfvhbecTs23Wwt558e9RIZauLCva0LzsGDjp4284uPc9uZqEsKD2VFo49r/LOftGcPpEH1mGyPeMP7eEX4VGmRmTK/2jOnVnscmaLbm21ick883mwt49ptt/OvrbaR1iebPVw2gf2fPJxKpz85CG098uZmsrYVU2p2YlOsL5JZzujGiezwZKbGE15odamSPBOIjgvm/eeuY/J8V/Pfms2kf6fsWmcOp+exnVzLfWXiUfp2ieOkm14X2Pl23n4/W5vHOyr0kRoZw6cCOXJ7WicFJMX45JlFe5eDDtXm88v1OdhQeJTEyhAq7kw/X5hFiMXFuzwQu6GtlXB8rCQHSem2KY5V2bv3vaiqqnMydOZwiWyW3/Hc1k/6zjLdvGU5yfJi/q9ggSerijFFKcVaHSM7qEMmdY3tQZKtg4cZ8nl60lQnPf8+0kV2558Jep03J54ljlXae+2Y7L3+3k1CLmV8MS2FUj3iGdo1rtPV9xaDOxIQFc/uba5g0exlv3jKMpDjffHCdTs2CjQd5ZvFWtubb6N0hktm/SOeiftaahH1BXyvHKu18s7mAT9ft5+2Ve3nth910jmnH5WmduDytI307RrV4gi8oK+fN5Xt4a8Uejhyron/nKP41eRCXDOiIAn7cfZhFm/L5amM+i3MKUGo96cmxXNjPygV9O9DVPeF5IHM6Nfe8u47NB0t5ddpQeiRG0iMR3p4xjKmv/cjE2ct4e8Ywejbj5L+WJkld+E18RAg3DEvm0gEdeXLhZl5btosvNxzgkcv7nZT0GqK15vP1B/jz5zkcKCnnmiFduO/is0iMbNrP5DG92vP2rcO4+fVVXPPvZbxxy9n07hDlbWhorVm0KZ9/Lt5GzoFSeiRG8PwNg7mkf8c6u5rCgi1cNrATlw3sRGl5FYs25vPpz/t5+budzP52B93bh/OL4Slck97F511EOQdKeeX7XczP3k+V08n5fazMOKcrZ3eNO2kfjOyewMjuCTx8WV82HSitSfB/+WIzf/liMz0TI7h4QEemjkgJmP7nU/1z8VYWbDzIQ5f1JfOsxJrlaUkxvDtzBL94ZSXXzVnBGzef7ZNfli2h0UkyWkpGRoZevXq1V9tmZWWRmZnp2wr5mdFi8iaetXuP8MCH69l8sIxxvRN57Ip+dImtv8W8vaCMR+Zv5IftRfTtGMWsK/qRkRrXrHpvyy9jyis/cqzSzivThjK0VnmexKS1JmtLIU8v2sr6vBK6JoTzm3E9uTytk1d95YePVrJgw0HeW7OPn/YWExZs5uohnblpRGqzLhXhcGpe+OBrfiyJ5Pvth2gXZGZSRhemj+ra5BZ37pFjLNqUz6JN+azYWUSIxcxNI1K4dXS3M9o9U3K8itXLv2fceWO92v6T7Dx+Mzeb6zKSeOKaAXU2KnYfOsqNL6+k9HgVr00f2uz/t8bU9T+nlGpwkgxJ6q2E0WLyNp4qh5PXftjFPxdtA+Du83ty8zldTxodYquw8+zX23j1+12EBZv53UVnceOwFJ8dYMw9coybXvmRvOLj/PsXQzivt7XBmCrtTlbuKqpJbAdKykmKa8evz+vJVYM7+2y45M+5xbyxfA/z1+2n0u5kRLd4po5M4fw+1kZfQ2vX8YxlOw7xw/YiVu4qoqzcjjUqhKkjU7nh7GSfnKm8o9DG899s55PsvDOW3LXWvLliD3/5Ioe4EM2/fjH8pC9jT2TvK+a6/ywnrUsMb80YRrCl/vdzf/FxfvHySg6UlDPnpnTO7dm+uSHUS5J6ADNaTM2NJ/fIMR6dv5HFOQX07hDJn68awJDkGOav28+fP8+hoKyC6zKS+P34s1rkp36RrYLpr69i4/5Snpo4kKuHdDkpprLyKr7dWshXG/NZsqWAsnI7oUEmRvdsz8UDOnDZwE4tNkzx8NFK3l21j7dW7CGv+DidokO5cXgKk4cm1bwXWmv2FB1j2Y4ilu04xPIdRRQdrQQgJT6Mkd3jia0s4O6J5zWYwLx1anKfMiKFmS2Q3PNLy7n3/Z9ZurWQkd3j2ZJ3mKJyzeShSdx/cW+PvqgOlpQz4fnvCbaY+OTOUR79PxWWVTDllZXsLDzKczcM5qJ+HXwRzmkkqQcwo8Xki3i01izcmM+j8zeSX1ZOj/YRbCuwMaBzNLOu6MfgRibbbi5bhZ3b3lzND9uLePDSPsQf3c3RmO58tSmf5TsOUeXQxIUHc36fRC7o24FzeiTQLvjMjXV3ODVf5+TzxvI9fL/9EMFmE5cO7IhJKZbvOMT+knIArFEhjOqewIju8YzoHl/TpXUm/udaMrl/uf4Af/hoPeVVDh64pA9Thqfw1ddZrKnswCvf7yKmXRAPXtaHKwd1rvf4zPFKB9f+Zzk7C218eMcozurgeZdW8bFKpr22ivV5Jfx90kCuGtyl2TGdypukLgdKRaullGJ8/w6c0zOBp7/aysKNB/nzVf2ZPDT5jIzljgix8Oq0ofz23Wwe/zzHvXQDqfFhTB/VlQv6WhmSHOu3ceVmk+LCfh24sF8HtheU8ebyPby/Jpdgi4mR3RO4o3s8I7vH0zUh3G+Xa+jePoJ/XjeIX53Xg+fdo5PeWL6bKcNTmDoytcFjJvUpLa/i0fkb+XBtHgO7RPP0tYPokeianDnEonjgfFcif+Cj9fz23XW8vyaXx68ccNqxAq01v3t/HRv2l/DSlIwmJXSAmLBg3poxjBn/XcU989aRd+Q4vayRmJTCbFIo5dpHJlV9cz82KaxRoXSOadfk2D0hSV20ehEhFh6+vC8PX973jL92iMXMc9cPYUjyLnK2buf2y0bQIzGi1V3TpkdiJI9d0Z+HLuvrSiCt5ASmat3aR/D0dYO4y53cX/l+Fy99t4u0pBguHdCBi/t39GgY6cqdRdwzbx0HSo7z6/N68KtxPevs5urbKYoPfjmSd37cy5NfbuaiZ5Zy19ge3DamGyEW16+pZ7/ezuc/H+D+i3tzvheziIHrf/P16Wdzx9tr+ftXWz3e7vYx3bn/4t5evWZjJKkL0QizSTHj3G5kOfa26vHJQKu/hk11cv/tBb349Of9fLH+QM2QyIFdorm4f0cuHdDxtBN8KuwOnl60lTlLd5IcF8Z7t48kPaXh7jezSTFleAoX9bXy2GebeHrRVj7OzuMvVw3g8NFK/rl4K1cP6cxt7rkFvBUaZOblmzLYXmij0u7EqTVO7eoe01rjcLoeO2vua5+dB1EXj5K6Umo88C/ADLystX7ilOdDgDeAdKAIuE5rvdu3VRVCGEVSXBh3ZPbgjswe7C06xhcbDvDl+gP8bcFm/rZgM/06RXHJAFeCr7A7ufvdbHIOlHL92Uk8eGnfk84KbkxiVCgv3DCEiekFPPTxBibPWUGQWTEkOYa/XFX30MWmMplUs4aY+lKj74xSygy8AFwA5AKrlFLztdabaq12C3BEa91DKTUZ+BtwXUtUWAhhLMnxYdw+pju3j+nOvsPHWLDhIJ+vP8BTC7fw1MItmBTEhgXz0k0ZXOBlNwnA2LMSWfTbMTz7zTZW7z7MizemB9TVFz3lydfd2cB2rfVOAKXUXOAKoHZSvwJ41H3/feB5pZTS/hpaI4QISElxYdw6uhu3ju5GXvFxvlx/gPzScm4b090nI2baBZu5b3zL9GW3Fo0OaVRKTQTGa61nuB9PAYZpre+qtc4G9zq57sc73OscOqWsmcBMAKvVmj537lyvKm2z2YiIiPBq29bKaDEZLR4wXkxGiweMF1Nd8YwdO7bZQxrr6nA69ZvAk3XQWs8B5oBrnLq3Y2SNNqYbjBeT0eIB48VktHjAeDF5E48nh8pzgaRaj7sA++tbRyllAaKBw02qiRBCiGbzJKmvAnoqpboqpYKBycD8U9aZD0x1358IfCP96UIIceY12v2itbYrpe4CFuIa0viq1nqjUmoWsFprPR94BXhTKbUdVwt9cktWWgghRN08Guyptf4C+OKUZQ/Xul8OTPJt1YQQQjRV6z79TAghRJNIUhdCCAORpC6EEAbit+upK6UKgT1ebp4AHGp0rcBitJiMFg8YLyajxQPGi6mueFK01vVOt+S3pN4cSqnVDZ1RFYiMFpPR4gHjxWS0eMB4MXkTj3S/CCGEgUhSF0IIAwnUpD7H3xVoAUaLyWjxgPFiMlo8YLyYmhxPQPapCyGEqFugttSFEELUQZK6EEIYSMAldaXUeKXUFqXUdqXU/f6ujy8opXYrpdYrpbKVUqv9XZ+mUkq9qpQqcE+WUr0sTim1SCm1zf234VmCW5l6YnpUKZXn3k/ZSqlL/FnHplBKJSmlliilcpRSG5VSv3EvD8j91EA8gbyPQpVSPyql1rljesy9vKtSaqV7H73rvlpu/eUEUp+6e77UrdSaLxW4/pT5UgOOUmo3kHHqTFGBQik1GrABb2it+7uXPQkc1lo/4f7yjdVa3+fPejZFPTE9Cti01n/3Z928oZTqCHTUWq9VSkUCa4ArgWkE4H5qIJ5rCdx9pIBwrbVNKRUEfA/8BrgH+FBrPVcpNRtYp7X+d33lBFpLvWa+VK11JVA9X6rwI631Uk6fFOUK4L/u+//F9YELGPXEFLC01ge01mvd98uAHKAzAbqfGognYGkXm/thkPumgfNwzf0MHuyjQEvqnYF9tR7nEuA70k0DXyml1rjncTUCq9b6ALg+gECin+vjK3cppX52d88ERFfFqZRSqcBgYCUG2E+nxAMBvI+UUmalVDZQACwCdgDFWmu7e5VGc16gJXWP5kINQKO01kOAi4E73T/9Revzb6A7MAg4APzDv9VpOqVUBPABcLfWutTf9WmuOuIJ6H2ktXZorQfhmjb0bKBPXas1VEagJXVP5ksNOFrr/e6/BcBHuHZmoMt393tW938W+Lk+zaa1znd/6JzASwTYfnL3034AvK21/tC9OGD3U13xBPo+qqa1LgaygOFAjHvuZ/Ag5wVaUvdkvtSAopQKdx/oQSkVDlwIbGh4q4BQe97aqcAnfqyLT1QnP7erCKD95D4I9wqQo7V+utZTAbmf6osnwPdRe6VUjPt+O+B8XMcKluCa+xk82EcBNfoFwD1E6RlOzJf6Zz9XqVmUUt1wtc7BNb3gO4EWk1Lqf0AmrsuE5gOPAB8D84BkYC8wSWsdMAce64kpE9fPeg3sBm6r7o9u7ZRS5wDfAesBp3vxA7j6oQNuPzUQz/UE7j4aiOtAqBlXg3ue1nqWO0fMBeKAn4BfaK0r6i0n0JK6EEKI+gVa94sQQogGSFIXQggDkaQuhBAGIkldCCEMRJK6EEIYiCR1IYQwEEnqQghhIP8PYRfvPb3UKn0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "epoch = 30\n",
    "model = NeuralNetwork(learning_rate=0.1)\n",
    "train_log, val_log = model.train(epoch, X_train, y_train, X_val, y_val)\n",
    "clear_output()\n",
    "print(\"Losses:\", train_log[-1])\n",
    "print(\"Val accuracy:\", val_log[-1])\n",
    "plt.plot(train_log, label='losses')\n",
    "plt.plot(val_log, label='val accuracy')\n",
    "plt.legend(loc='best')\n",
    "plt.grid()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YgxZze99748q"
   },
   "source": [
    "As we can see we have successfully trained a MLP which was purely written in numpy with high validation accuracy!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zum-oFsFQNCW"
   },
   "source": [
    "## Classification report.\n",
    "\n",
    "Look at the precision column and you will see the percentage of right prediction to each number. The closer to 1 the best"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "H5fFtXfKOWCm",
    "outputId": "8bfdf77d-1e40-458d-c658-10ec82d7db28"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00       100\n",
      "           1       0.99      0.98      0.99       113\n",
      "           2       1.00      1.00      1.00       114\n",
      "           3       1.00      1.00      1.00       106\n",
      "           4       1.00      1.00      1.00        90\n",
      "           5       1.00      0.99      0.99        93\n",
      "           6       1.00      1.00      1.00       108\n",
      "           7       1.00      1.00      1.00       114\n",
      "           8       0.97      0.99      0.98       105\n",
      "           9       0.99      0.99      0.99       104\n",
      "\n",
      "    accuracy                           1.00      1047\n",
      "   macro avg       1.00      1.00      1.00      1047\n",
      "weighted avg       1.00      1.00      1.00      1047\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def evaluate(testData, testLabel):\n",
    "\t# the predict expect a list of imgs that is why we need np.expand_dims\n",
    "\tpredictions = [model.predict(np.expand_dims(item, axis=0)) for item in testData]\n",
    "\tprint(classification_report(testLabel, predictions))\n",
    "\n",
    "\n",
    "evaluate(X_train, y_train)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FjhEtuSvQXvn"
   },
   "source": [
    "## Random test\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FyE43GXZQfpk",
    "outputId": "23e4b717-3a81-4045-b235-49c19fcfcb74"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] Test network with  5\n",
      "Result =  [5]\n"
     ]
    }
   ],
   "source": [
    "print(\"[INFO] Test network with \", y_train[0])\n",
    "arr = np.expand_dims(X_train[0], axis=0) # the predict expect a list of imgs that is why we need np.expand_dims\n",
    "predictions = model.predict(arr)\n",
    "print(\"Result = \", predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "anaconda-cloud": {},
  "colab": {
   "name": "Building_neural_network_from_scratch.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "navigate_num": "#000000",
    "navigate_text": "#333333",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700",
    "sidebar_border": "#EEEEEE",
    "wrapper_background": "#FFFFFF"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "264px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false,
   "widenNotebook": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}