05. Memory Augmented Networks/5.1 Neural Turing Machine.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Turing Machine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NTM is an interesting algorithm which has the ability to store and retrieve information from the memory. The idea of NTM is to augment the neural network with the external memory, that is, instead of using hidden states as a memory it uses an external memory for storing and retrieving the information. The architecture of Neural Turing machine is shown in the following figure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](Images/1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The important components of the NTM are as follows:\n",
    "\n",
    "__Controller__: It is basically a feed-forward neural network or the recurrent neural network. It reads from and writes to the memory. \n",
    "\n",
    "__Memory__: Memory matrix or memory bank or simply the memory is where we will store the information. Memory is basically the two-dimensional matrix composed of memory cells. Memory matrix contains N rows and M columns. Using the controller we access the content from the memory. So the controller receives input from the external environment and emits the response by interacting with the memory matrix. \n",
    "\n",
    "__Read and write heads__: Read head and write head are the pointers containing an address of the memory from which it has to read from and write to. \n",
    "\n",
    "Okay, but how can we access the information from the memory? Can we access the information from the memory by specifying the row and column index? yes, we can. But the problem is if we access information by index we can't train our NTM using gradient descent because how can we calculate the gradients for an index? So, the author of NTM  defines blurry operations for reading and writing using a controller. The blurry operation will interact with all the elements in the memory to some extent. It is basically the attention mechanism which strongly focuses on a region on the memory which is important to read/write while ignoring the focus on the other location. So we use a special read and write operations to determine which location on the memory to focus. We will explore more about read and write operations in the upcoming section. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}