Deep_Q_Network.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Open AI Lunar Lander V2 Write Up\n",
    "\n",
    "As part of the Deep Reinforcement Learing nanodegree from Udacity one of the lessons focusses on Deep Q Networks.\n",
    "\n",
    "Here you can see that by taking the default network from the lesson and enchancing it by implementing double Q learning and dueling networks, along with some hyper-parameter tweaks I have been able to comfortably beat the benchmark set by the default network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Q-Network (DQN)\n",
    "---\n",
    "In this notebook, you will implement a DQN agent with OpenAI Gym's LunarLander-v2 environment.\n",
    "\n",
    "### 1. Import the Necessary Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: box2d in /opt/conda/lib/python3.6/site-packages\n",
      "\u001b[33mYou are using pip version 9.0.1, however version 18.0 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n",
      "Requirement already satisfied: pyvirtualdisplay in /opt/conda/lib/python3.6/site-packages\n",
      "Requirement already satisfied: EasyProcess in /opt/conda/lib/python3.6/site-packages (from pyvirtualdisplay)\n",
      "\u001b[33mYou are using pip version 9.0.1, however version 18.0 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "import gym\n",
    "!pip3 install box2d\n",
    "import random\n",
    "import torch\n",
    "import numpy as np\n",
    "from collections import deque\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "!python -m pip install pyvirtualdisplay\n",
    "from pyvirtualdisplay import Display\n",
    "display = Display(visible=0, size=(1400, 900))\n",
    "display.start()\n",
    "\n",
    "is_ipython = 'inline' in plt.get_backend()\n",
    "if is_ipython:\n",
    "    from IPython import display\n",
    "\n",
    "plt.ion()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Instantiate the Environment and Agent\n",
    "\n",
    "Initialize the environment in the code cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n",
      "State shape:  (8,)\n",
      "Number of actions:  4\n"
     ]
    }
   ],
   "source": [
    "env = gym.make('LunarLander-v2')\n",
    "env.seed(0)\n",
    "print('State shape: ', env.observation_space.shape)\n",
    "print('Number of actions: ', env.action_space.n)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before running the next code cell, familiarize yourself with the code in **Step 2** and **Step 3** of this notebook, along with the code in `dqn_agent.py` and `model.py`.  Once you have an understanding of how the different files work together, \n",
    "- Define a neural network architecture in `model.py` that maps states to action values.  This file is mostly empty - it's up to you to define your own deep Q-network!\n",
    "- Finish the `learn` method in the `Agent` class in `dqn_agent.py`.  The sampled batch of experience tuples is already provided for you; you need only use the local and target Q-networks to compute the loss, before taking a step towards minimizing the loss.\n",
    "\n",
    "Once you have completed the code in `dqn_agent.py` and `model.py`, run the code cell below.  (_If you end up needing to make multiple changes and get unexpected behavior, please restart the kernel and run the cells from the beginning of the notebook!_)\n",
    "\n",
    "You can find the solution files, along with saved model weights for a trained agent, in the `solution/` folder.  (_Note that there are many ways to solve this exercise, and the \"solution\" is just one way of approaching the problem, to yield a trained agent._)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD8CAYAAAB9y7/cAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAABlNJREFUeJzt3d1xFEcYQNEeF1EQB2EoDsehOByHwiAOpzF+wMKLrF+0u9N995wqPSEVrRm486m3WbZ93wcAPX8cvQAALkPgAaIEHiBK4AGiBB4gSuABogQeIErgAaIEHiDqy9ELGGOMbdv8c1qAJ/Z93z7z9SZ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogWca+76P79+PXsXxXAPO5cvRC4Cnngvct2/XX8eRXor8rV0HPkfgWYLg/eDhx0fYogGIMsGzBFPqD64DHyHwTEfEXAPOY9v3/eg1jG3bjl8Eh9v3fWzbdvQyYBr7vn/qL4QJ/mDPPWBFDjgHgT/Iaz85Pf6a0AOf4RTNAd67LTbD9hmwrikm+FvZe/2dYJvmz+fu7v7Nz3l4ePtzYBVTBH6MdsjOMYmXr8+5vRTyb1//fMcXizwd0wT+0WkMV47ZpbZXhP593hXzF9zd3Ys8CdMF/tSKsb/WvrnQP+/u7v5TcYeSZV5k3ff958eMjlrbzNdkRR4OlCwT+FMzRW2WtcywBmAuU2/RvOXIbYoZg2rbBji15AT/1DW3b2aZ2F8z+3bW7L59/fNdRyphdonAn7pU3FYN5qrrfi8hhpctvUXzlnOcwqnEsbx98xj5SxxtPH2AODrJanIT/Es+OtlXJ9/S93WtI5GPv4efFljNzQT+1Ftv9FUJ4Gtu4XuEW3eTgR/j/xP9rYT9VOl7vtT2ycPD/fj+91/Ox7Okmw38qUrkfpdTN+/jdA2rEXh+cRr7px8zedx///73Xxd/8fNxiofVpE/RcF5vRX6WEzrniPFzD42fWzXecZJFCDxnM8MD4JJ78bZnWI3AczWvPQC2bfvwA+Aa2zOnHh7uTe9czTm2RbdJ9lanWATMYpbtLo5x0uVP/UHwIitMaMYXtrmOc953gYeJCf1tOfe9FnhYgND3XeL+epEVFrLif2PJ2y718DbBw6JM9Q2XvIcCD4sT+XVd+t4JPASY5tdzjftlDx5C7NGv4VoPYxM8RJnq53TNeyLwECf087j2fRB4uBFCf6wjrr3Aw40R+es76pp7kRVukBdjr+fIB6oJHm6crZvLOfq6muCBMcZ/MfrIRH/pgK3808XRcR/D+8EDC1gt9GfsqveDB9pW2kaaaZ0CDyxjpng+Z7b1CTywlFmn+RnXJPDAkmYK/SzreMopGmBpR5/pnzXuY5jggZBrT/Uzx30MgQeCrhH62eM+hsADYZeK8ApxH0PggbhzT/OrxH0MgQduxDlCv1Lcx3CKBrgxv3vqZrW4j2GCB27Ye6f6FeM+hsADvBr6VeM+hsAD/PQ05ivHfQx78AC/WD3qp0zwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8Q9eXoBfxrO3oBADUmeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gKh/ABsYVY2pq747AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f033d2a4710>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from dqn_agent import Agent\n",
    "from importlib import reload\n",
    "\n",
    "\n",
    "agent = Agent(state_size=8, action_size=4, seed=0)\n",
    "\n",
    "# watch an untrained agent\n",
    "state = env.reset()\n",
    "img = plt.imshow(env.render(mode='rgb_array'))\n",
    "for j in range(200):\n",
    "    action = agent.act(state)\n",
    "    img.set_data(env.render(mode='rgb_array')) \n",
    "    plt.axis('off')\n",
    "    display.display(plt.gcf())\n",
    "    display.clear_output(wait=True)\n",
    "    state, reward, done, _ = env.step(action)\n",
    "    if done:\n",
    "        break \n",
    "        \n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Train the Agent with DQN\n",
    "\n",
    "Run the code cell below to train the agent from scratch.  You are welcome to amend the supplied values of the parameters in the function, to try to see if you can get better performance!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 100\tAverage Score: -189.62\n",
      "Episode 200\tAverage Score: -99.489\n",
      "Episode 300\tAverage Score: -47.13\n",
      "Episode 400\tAverage Score: 84.739\n",
      "Episode 500\tAverage Score: 139.22\n",
      "Episode 600\tAverage Score: 170.17\n",
      "Episode 700\tAverage Score: 177.52\n",
      "Episode 800\tAverage Score: 181.09\n",
      "Episode 900\tAverage Score: 194.02\n",
      "Episode 1000\tAverage Score: 181.07\n",
      "Episode 1087\tAverage Score: 201.46\n",
      "Environment solved in 987 episodes!\tAverage Score: 201.46\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEKCAYAAAA8QgPpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJztnXe8FNX5uJ/3Fu6ldwQpAoKKFfUK2LGCmkiKBk2M2EIsSUyPLdaYGGNs0aj406gxwZiviRrFAthRFCwoVYooTZr0esv5/bEze2dnp+1svZf3+Xzg7p45M3NmZ/d95y3nPWKMQVEURVEypazYA1AURVGaJqpAFEVRlFioAlEURVFioQpEURRFiYUqEEVRFCUWqkAURVGUWBRNgYhItYi8JyIzRGSWiNxgtfcTkXdFZL6I/EtEWljtVdb7Bdb2vsUau6IoilJcC2QHcLwx5iBgMDBSRIYBfwTuMMYMBNYBF1r9LwTWGWMGAHdY/RRFUZQiUTQFYhJstt5WWv8McDzwf1b7o8A3rNejrPdY208QESnQcBVFURQXFcU8uYiUA+8DA4B7gYXAemNMndVlKdDTet0TWAJgjKkTkQ1AZ2CN3/G7dOli+vbtm5/BK4qiNFPef//9NcaYrmH9iqpAjDH1wGAR6QD8Fxjk1c3662VtpNVhEZGxwFiAPn36MH369ByNVlEUZddARD6P0q8ksrCMMeuB14BhQAcRsRVbL2C59Xop0BvA2t4e+MrjWOOMMTXGmJquXUMVqKIoihKTYmZhdbUsD0SkJXAiMAd4FTjD6jYGeMZ6/az1Hmv7K0YrQSqKohSNYrqwegCPWnGQMuBJY8xzIjIbeEJEfgd8CDxk9X8I+LuILCBheZxVjEEriqIoCYqmQIwxHwMHe7QvAoZ4tG8HzizA0BRFUZQIlEQMRFEURWl6qAJRFEVRYqEKRFEURYmFKhBFUZQ8MWn2Sr7csL3Yw8gbqkAURWlWGGN44r0v2F5bX+yhcNFj0/n2fW+zvbaeRas3h+/QxFAFoigFYkddPXO/3Fjw8y5es4UFqzYV/LzFYuLslVzxn0+47aV5RR1HfUNimtqy9dv4xZMzOP7Pr7N1Z13IXk0LVSCKUiB++/RMRt75Jqs2FtalMfy21zjx9jcKes5isnlHQkiv2bwjJ8czxtD/yud54PWFvDhzBVHmL6/atD3F4nh7YaJk37adubOKjDFpY1m7eQfL12/L2TnCUAWiNBnWb91Z7CFkxbTF6wDYtKO0nkL/MGEOA66akNNjrty4nQ3banN6zKjkukZ3fYOhwcAfXpjLxY9/wGufrg7dZ8jNkznpjkalXV6WGFRDFrUzttfWJ60aYwz9rpzA7yfMSelTc/MkjrjllfgnyRBVIEqTYOayDQy+cSJPf7gsp8d9de4q7nllfk6P2dR44I1F1DUYNmzNncAf+vvJHHPrq57bRt07hSv/80nOzpVLNmyrZclXW1Pa6lxSf6NDMd78/Gz6XvF86HHtlScaQqyXLzds5w8T5tDgoWn2+e2L/PDv0/nXtC+Y+2XCJfnQW58lt9c3GApd3EkViNIkmLMiETt4c37CFfD0h8s4/2/vcdek+Zzw59diH/f8R6Zx28ufsqOuPsU/vWLDNvpe8TxvRHjazBTnA7Ltzlq4enPS9RKXV+auZN6X0WIdi9dsYdP2hCBsW5UoSLFk3dagXTJmw7Za+l7xfJrbZsaS9Yx/7wvf/dZu3sGRt7zCrOUb2F5bz/MfrwAST+Azl23g0bcXA4nYwn8+WMqOuty5hU6+43WOdik+twIpc5g4D775GVEot/bZurOem5+fnaKsX571Jas2Jb4Hv/q/GTzwxiKmf76OhgbD1EVrU44zac4qfvPUJ5x695tp53hy+pJIY8klRS3nrihRsZ/gbJ/vT//1EQCvzsuNgP/a3W8xf9VmFt9yGgAffL4egPHvfcExe2Vf1fnkO17nszVbUtreWbiWsx+cSnmZUN9gOHSPjjx1yREpffa79kX27NaGZ390FAB19Q388O/vc+lxAzh0j44pfS94JLF0gX0NQQy/7TX27dGOCZcfnXT51GfhX9lRV88tL8xl1rKNPHnx4SnbNm6vpWWL8sjHenHWlyxbv41H315Mm6pKHp7yGR1bD+WuSfN597NEAe7j9+mWFPRTFqzlz985KO04T3+0nD6dWvHzk/dOto17YyHH77MbA7q18Tz3yo3pcZP6+tTPxXZHOTHGELS+nb3P5DkrefDNz1izeSd3jB7M9tp6xv79fQb1aMcLlx+dzBwzxvDI24u58bnZPDSmhhMG7eY6n/XX0fbVlsK7eNUCUZoE9k8zroi74JFp/Hj8h8n3xhiuebrRjTJ/VWqKZb31Cy3zEBZx+HRl4/FtOT1z2YbEuayG9z9fl7bflp31fLx0Q/L9ojVbmDx3Fb/69wwWrNpE3yue59OV8TKsZltWnS2M6hz+9cfeWczazTuSCntHXT37Xfsiz3yU7kLcurOOva95kb9NWcx7i9NWWMg4JrHkq0QQ+JW5q/loSeIzWblxe1J5OMcK8NQHS7nyPx+zYWstdfUNiMPGu/uVBcnX22vr+f2EuZxx/9vsrGuIPJ66htS+Xl+JnfXBx1tmBbZbVCRErn3PdtQm9rPdZva9uO/1hTwzY3nKvn4sXrOFx95ZnNKWzcNAJqgCUZoEthCKW8H/lbmr+N+M5cn39Q2Gx6f6u1FsH3S5j/T7ZOkG6hsMk2av5P7XF2Y0FlsgmRjq0BYmXdpW8Zzl2nnqg6X8bUo0V4oX9ihs//zHSzdw7TOzOPR3kzjw+peBhDtqy856bvjf7LT9V28KznaaNHtVRuOxXWtrNu/ggy8SluDmHaluqnqXUB//3hIOuvFl9rrmBd/j7rCUxvqttcl+zliD0xU2ZcEalny1laP++Arj3lyUcpzL/vkhA69OTTrY4VBIQd/RLdZ12JlSWyy3aZnAvC83sdDK3Hpt3mpmLFkfekwBznzgHa59ZhZ1DktpW4HmwKgCUZoEtt/ZAC/O/DLSPlt31nmmzH68dD1TFq712KMRW5g+O2M5r81LFYAzlqzn6/e8xV9emc9Fj03nlhfmeh5jxpL1nrOQ7R96pg+JL3yygkemLAbgvc++YtKclQCMe2ORp2B3M39lwmJ5Ze7KlHZbONnjWrulUSEkM8assW7xiNMc+6fXAs971X/9A+ZbdtQlBeVna7bwydINnoFm93mnLkq3dCDxmT4+1XsxvR0uofrktCX0v2oCK63vyMuzGj+X7/2/dznj/rdZum4bD7yeqkDqGwy1LreW06JxWkpubEth3dZa3l6wJpkxVV4mjLjzDdZ5JDL87rk5rNns756y4yn//XBpss1WwvlGFYhSEHbWNfD52i3hHX2wDYEGA7/694xI+4x+YCpDfj85rf30e6Yw5uH3Avd1ugDO+9u0lG0rLKUwe3njpECvWcaj7p3CMX9Kz0SqbzBc+Z9PfBWPH5f84wNedwT1Zy5LdUGF8aH1NP/8x6kK2L5U+5o3bU8V1n2veJ6JlrLa4XL9eGULRWXzjjrOeehdRt07hZ11DRx322t8/Z63PN0vbgVyzdMzfY873eUKtMe4vTZ17P+0Avm2Vec+b5hltW1nPRWWP+udhWv5x7sJxXXWuKm++6xwPFDcPvHT5Osgl9PO+gZu+N+swLEALF7bmASxZlNh4iGqQJSCcPV/P+HYP70WO1XUmQYZdR7FJ8s2hHdysXbzDh57Z3FouqWb4//8OsYYvvvgVEbc8UZyzoqXr72uwfhmIfW94vm8zaC24zm1Ln+97UrbWZ94Qt+4Pf3zvfq/jQLb6U75ymNuTlQ34zfvnZJUatsd7iMvWZpNhpodn5j+eaplYGfdtW6RyCVyZ1t1blMVeNwhN0+iojzxmf54/Icpn1EUnIrO6zN3Yrsro3Lx4++Hxk5ygSoQpSDY6bdbYpZysCMRy9Z5/yhsofXeZ19lVbzuZ0/O4NpnZjFnReaB6boGw9sL1zJv5aY0q8VJWIDzQcvnHlUAuIO6M5as93TdPTktkeb5rCMWBI0WzAWPTOflWV+yOUSYOce/yiNrye3ece9j40xccN5XL6vmESt1Nw7XPD2TL9Zu5edPplqu9nXe9NxsLnp0elo6cEVIAsWmHXVUlqeK0JkRH1pO2b97pH5h+H2Vlq3fxoWP+H8Hc0Ux10TvLSKvisgcEZklIpdb7Z1EZKKIzLf+drTaRUTuFpEFIvKxiBxSrLEr8QnLyHl30VpPAWLv95EjsOjE3uU7D7zDKXellu3IJCPFnvcRp+SE073jN05Iz+rx4t1FaznSMaM4aLJahUuIjbp3Cifd8QbzvtyUMhvcK0MKUl1gv/z3jNC5L799Zib1DYaZyzawYkO6kvvwi/RssrDChs55DYvWpLs6s5kg93/vL+WSf7yf1r7cetB4a8EaJs1ZyQJXJt6KCA8iZa4vdBRXE8CJrrTcbPDLAHOnjeeDYlogdcAvjDGDgGHAZSKyL3AFMNkYMxCYbL0HOAUYaP0bC9xX+CEr+eT1T1czetxU/t9bi9K2CcGa55NlG5KKxx2IdGZfRa2PVBtByLtl2tU+weJPlqY+lYYptB11DRml5rYoT/8Zb9hWy4g73+A7978Tur8zG2zj9jreWRScYDD+vSXcOelTvvaXt7hzUvos/tEeMYDttfVs2Frr695yNgcp37h4Bf/duJVBFNwPO3a5miAqyoTObVpkfK5M+dqBu+f9HEVTIMaYFcaYD6zXm4A5QE9gFPCo1e1R4BvW61HAYybBVKCDiPQo8LCVmERJWbVTG2ct38gN/5uV8qMP+21/494pKYrH6f6xJx0CjLgjWlHBoHkC9lgmzk7NZnrmo+UeveHr97yV8t7ta/cik3pZleX+H868EEW0fuvOWE/3r1kTOGcuj+ay+XLjdg668WXODUleiEKn1pkLX2eA2Y84SR5x6ppVVZTRqkV+5nC3rGycsNmiIsdFwTwoiZnoItIXOBh4F9jNGLMCEkpGRLpZ3XoCzrn6S622zKJLSlEJsySgURC3qargF9Ys4ijz+ca/1/j1ONKnoNzaiLN1vVxYKzdup33Lykj7B/H2gjWhfW59MXog3e2Hz4TBN06M9Nm6sZV7VOVz2t0JJWrHwkqRSXMym68Sl4rysuSEwlxTVVmWnAOyozb6ZMm4FD2ILiJtgKeAnxpjghZL8Pqap319RWSsiEwXkemrV+e+jpFSOFKf1MOlXC59vpPnpguTob+fzPkBwfGoRK2fFJVsFAjEqxCbbd2ubAgL8pc6leXi6XbMBVUOxeROuc4HRVUgIlJJQnn8wxjzH6t5pe2asv7av+SlQG/H7r2ANJ+BMWacMabGGFPTtWv2NYyU3BDHTXLfawv5eOl6lny1NeclujPFdom9s2htsrBjqVCIdE03Gws0Uc2L4/fpFt6phCkvk7xZIFedOij5uhArMhYzC0uAh4A5xpjbHZueBcZYr8cAzzjaz7WysYYBG2xXl9J0yFQRnH7PFI6+9dVYAc5cco+jppJX4HhXwz0pr1A89+OjuPOswVkfp0+nVjkYTTyG9uucYinkim8d3JPTDmgMCxfCAilmDORI4PvAJyJiRzmvAm4BnhSRC4EvgDOtbROAU4EFwFbg/MIOVykmRTZAAsuPK9kxavDurN9amzLL3o/eHVtRXRm9sq+b3dtXs3zD9rTJlIXi0QuGMLRfp7wstiUiKZWCc1nm3o+iKRBjzFv4y4UTPPob4LK8DkrJG9nWBi22CysO5x3RN6sJcLsKm7fX8egFQ9iwtZaDbnw5uHOW34PbvnMQvTu24hv3TsnuQDE51loaIO7StpcO35MvN27nPx+kV0UuLyOlpHzHVvlPFS56EF1RbIJkQ7FdWHHIZA2MpsCt3z4wL8e1n8bbtwrPcAvLGBtd0ztwe+sWFfTu1Iphe3aONLbWAfewe7vqSMfwIigG4rXeiM1RA7vw29P29dzm/o2MO7cm3uAyQBWIkheMMdz64ty0dQ7sv4fcNJFRrvkRfwgoLnh+Acoy5JogYRdUJuOIiMKt0AQJtmwIW0vDiXvRJvfnuNxjZnxKf2vOzJ/PPIj7vhdezKIyQNA/dekRvtsgsbCXn5IJUiCjD/NXglUV5b77Bi1olS9UgSh54dOVm/nrawvTSkjYEwq/2rKTGY4Z2g0NJid+4SDBvPdubdPafnhM/6zP6UeQ1dTK9WQ7cr/G2kjZpuXmgxtO34+yiMPq1ja4CKEb53yF5358VGBf9yfqVmpffLU1cGKlnT5bXVlO/67eqxL6na9nh5Yp27y+a9WVqR9Spc9kvqDvaVCKb1VFme/346Kj+/nuly9K75uqNAvsaraL16TOAPZL592eo4Bf22r/sJ7X6oJ7dG6dk/N6ns+lQJzuFXf9qpu+sb9jv7wNKRatWpQz5oi+vgrxVyP2Tnl/YK8Oyde3O5aaffSCIZ77O4O9+/dsHzgWewx/Oftg7hzdmI1199kHA3aJ9cRne+Oo/ZLb21jrvjsVThQ97bxmd/aXl0X2rUN6pfbx+cxEhPvP8baAgqyT6soyTwX574sPZ88ICjHXqAJR8krjhDPj+D+dbIrlOQkqEeEUGIf0SQi5fFr9bgHjPJd7mzNeEsdV9PtvHpDxPl5c7ZhHYGOPxs9F4s6KcmY4ORXlkL6dPPfPJN3UHsLXD9qdbxzcM/l9sisEbKutTwpYp1XX1bKKnF+z8ggmlX2+7w3tk6bY3VbErWccyHVfT41PBLmVRu7fmHJrK40fHTcg0Dqpqij3PGa+JiaGoQpEKSjuYnp2jCTT9Tf8CMqvdz4N2k+p+Xza9zr2RUf1s87vUiAOIewWEPvt3s7z+G/86rjka78SK7u3zyzQG6RQ3YUDD+zVnn17tGPU4NSifc46YpWO6/RzLfXvGmwF/tURq0gbnzUk+/q319Zz//cPZfjeXVPW87CH4bwGP+sglUSfYf0743aguS3adtUVEY+Zzlu/OY6Xf3YMvxyxd6BrrarS+/udr4mJYagCUfKCnz5wtx9966us2LAtVjkNL4J+SM6Yi/2UHxZ4tF0fQRyxZ2du/ub+ae3uY4s0CsCN22rp4RDuTqvDfvmbkfsA0Ldza847om/a8ft0TkyG69eltW+BwW8c3JNJPz8mpW3GtSez+JbTeOD7h6b1P3tIH04/KFUh2NfhriLcr0trJlx+NF1cCy85g+JOf72XZbVvj3b89bup4xj/g2HJ17NuGMGpjslx7lpq9oNHB0uB1NYbjtizC4+cPyTlfLd8+0AO7tMh+ZkBlAfESmzsQzjvnY37IUBEYgeyu7WtZi8rRvftQ3ry5A8P9+xXVeGdFaYKRNkl8FIs67b4l/nOlIoIQsHZLyw9+KEx4amQ3dtX061t+pN++rElWd9ry876FEHpZEC3xBPoIX068PB5NfzxjAO5/vT9+POZB6X1ffqyI3nqkiM4fM/OSesm5YwCA7qlJg/YnpsR+3Wnoyt1tnVVBbee4Z2u67YS/VxtTgvEeT+8hOtRA7ukpe/u3b1xvK1dCtx9CHtE1ZXlnLBPNx70SF0dvndXDuvbif9eemSKAI5iLdhdyiS9DKj7/gq5sWhFhCH9vN19fha2urCUXQKvsu6V5ZIzCyTqfJGkBRLSr2OE0uGCePqt3b9p99D8Jnr99MS9eOyCIQzt35nj99ktaQV5Xdrg3h2S1seIiKvcOQV5hwwmm7l1vFMA//HbB3DD6Ymg9RiHteTOGBqxX+pCSl7XlMl8IPvBo7xMeOi8wzhp39Tjf3z9yYz7vvdDQKuq9Kf5oS7BbasNITyFOBsLJAr77d7OV4HkozRKFFSBKAXFy9AQkZxZIFF/wHYMxN39AFcWUCQvuXhneKULu9TrF9evr6qijBMHdaOyvIxj9kovBBqmHL0sgpH7pS+ZE5TmCumfif02yAIZfVgfxhzRl8W3nMYZhzZmIrkVyOkH9XQdO30sQZfpEwLxDTy3q670de+0q06PG7knf9pjEQ8LxP155zqedtqBqffu+Z8cnfx+H7Fn5xRXo/savVLW80FJrAeiND/cloYtex55ezHXn75fyrb6BpMzCySiByspcNzCami/TnziWNc6ij7yc124lZlbSboVwrzfnRJ8npCxuF0yXdtWcUCvVIU449qTff3oyfO4RaX1tt41/ijK2u1STFNOnhaI/3HTXFjWkKLEM6Lg5Zayz5s+9uBry5Z7zj6Ym0btzyE3TUzb9k/L/Wmvb+9UIHNvGpm3SZ9u1AJR8oJT1jgFp1dtqLqGhtgWiPvJM+oPx0/gpM88Dj9eyxblnkLPPRT3E3ymP/H+XYLz/KNce5RyIX6C0J2FFcXtXulKlU2PCmWGn9IKSn3NhDQFIg4XVshooyyW5ub+cw7lmcuO9D6eSOSsLqelV11ZXrDJqGqBKHnHmOBiinX18S2Q8jJJWXgqqgvL74dZmebXDj7OiYO68csRe6ete26PzYn7GjOt7+W2JsLO53x31an7JIPzYfgJeff4owg3O85wkzWpL4oFEqdgYkXUafIhuPVQShA9ZFxe28MejEaGxa0ytKgLjSqQXZxVG7dTXiYpOfO5Jkw31DWYSGume+EWmlGf2Ozd3L9v9wzxsKP9ZuQ+tKuu9OznVmZpFkiOf/NBFsjYY/b0bI9i+dnXkT7+8AtoUV7G4ltOc+6Vst1Licb5XPJngaT+DSIfAfSol1WMOligLqxdniG/n8yhv5uU13OECakddfXU1sVUIK4fTlQXlv2DS8ss8sisiXIcLw3iHorzXGOP6Z9zBeIn/DLF75rdsZM4FZLTnvC9zp/B8S4dnlCMXkkMcUi34hpjZaEWiEdbj/YtPVqjUyzFEBVVIEreaTDBSuS7D77LMX96Ndax3YIj6u/Nr5tfENX3/A4XR9ixnE/w3dpW5bxEfVpaaczFM9JcWFbDmTW90vqGkX5/wn1YmQjNX4/cx2XhZIff8BJZWMHj8rqf90ao+Bs4nqz2zj+qQJS8E9c9FQX3g2eQBfJ1K+3xoN6Nxf7cI/PzgfufvzHIGja2bGMgYeQq88YvjdcdmI2aoZbJe7+2QpGemusIoseIgfhVCIhK2DlvO/Og0DVQ8klRFYiIPCwiq0RkpqOtk4hMFJH51t+OVruIyN0iskBEPhaR7FS7kjcemfIZy9Y3rsuQq0KJUQgSyqcd4AhY+nTLVKgnBUyGFojXvIJsCSremAlRLYAovcIMjlLz0PgpuEhB9LyMJ/ioZxzaiz/6VA4oBMW2QB4BRrrargAmG2MGApOt9wCnAAOtf2OB+wo0RiUD1mzewfX/m80P/+5eB6QwBCuA9G1pczMydAMFBVnTxuJw5ZVF8KlnSr5y/7Pxw7s/P/ehchVEzxXpLja7Pcp3IfcDLzUF66aoCsQY8wbwlat5FPCo9fpR4BuO9sdMgqlABxFJn2arxGbT9lpenbcqq2N4VdXNpwUS5oLy3GaMrzBwTw8JnbwXUBLFnVmaYoGQe4Hj53rK+rgRzxelT5pC8Txf4aRmzR4dU967Hygk+TdeGm+2qALJnN2MMSsArL/drPaewBJHv6VWm5IjLn/iI87/2zRWhCwLmin5jIG4CbJAvAR2mgLK8CnePp/XxK18rOP+yi+O9d2W/rRffOnjF5D3e+/Xli8evWAIr/9quO92+x6WSbhC9tseVq4++JjFv4dBlKIC8cPrk0yTTCIyVkSmi8j01atXF2BYzYeFqzcDsL02+gI/bry+8A0mf1ZIWBquE+eWpJAKCWyHB9ETfw/s1Z5fjdg7pbS5Vy2sbAlaKyJfgtfvuJEUVJrCKC0l17qqInBVyuTwfFyOznVY/B4+/GaaR6EEngECKUUFstJ2TVl/bZ/KUsCZbtALWO7e2RgzzhhTY4yp6do1vSCdUnhyVSgxCkEWhPPH6NfLr5SF/zEbg+iXHTeALm0as27SFEieLbH8yRrvI0c5X6Zp0VBaQtN+IEpcR/rAZlx3Mn06tbL6etPWo2hj9POXNqWoQJ4FxlivxwDPONrPtbKxhgEbbFeXkluy+dJ6CckCJmGFxEC8XFjBs8PDPougyYJB2/JBx1Yt+NbBPZPL25aCIA5zYXkG0UtIbDrniRYjBpIPN2guKWopExEZDwwHuojIUuA64BbgSRG5EPgCONPqPgE4FVgAbAXOL/iAmxnuFeZygZeQNCETCbM7n6vAX9APzmmB+HTLNJEpyMootHumrEy4ffRg5n25KafHzeYywi24+MfOJX07t2Lx2q1p7U4LMzQGsgtmYRVVgRhjzvbZdIJHXwNclt8R7VqMfuCdlPeZyPiv/eVN2lZVMn5s6qp6nkqpkPNAIsZAbNzXnGmJbvf5nMdzx2MKl8qc+Jt34ZN5CCTS9mIIzWcuO4o1W3bwl8nzU8di/U2kXYcowzyMq9gxojC0mOIuzPTP13m2R/nOzly20bPdM43X5SjKZ0wkigvL0OgmcY8kzeUSWr4i9b0J2FYoMhE62dyJKK6m0DReryysLMYUl/atKj1L3Tvn+RTDAil1SjEGohSJXAR5/VxYYX1in8/1PjALK5ILK9MsLLcFYgK2eR+jf5f4aZ7eY0r8zVUsIZujxFlDo5QEceNQ/OeB2L+b0hl14VALRAESgm/JV4n5H9kIHi8LxN3m1SdXBM4D8ZyJ7trfvY55hudzHs5rKO4rn/yLY1NSf3NBo98+Qt9Ix8usPZMTlIoF4of9/UjMAwmzRktp5IVBFYgCwJPTl4R3ioBvCMQE94lNBgUKo1ggfsu5RjmmezzpY0kPsO8ZMK+jEOQ7LhPmxvOyNkpRDktALSxnyfddDXVhNVOMMazetCNy/wWrNufkvH6lTDbtqAvsE5e4LqzG/bNb5CnIAkkLohcoim670XIlz/yevKNZL4UPPOeDYsWzSh1VIM2Uf7+/lMNunuS51KoXubIK3OtmQ2LN80IRnMUrKe8gQhZWiIhLVxIBMRCP8+WTKLGErOIbMVxk0UqZlJ60DqqFZT+E7IouLFUgzZSpC9cCMG9ltDkBzvRb20K4feKn9L3i+YyyprwU0bad9a4+ObRA3NV0s3RhZb4eiGs8QdtSyrkHHzcbMvl0o/TN5uk7dPLu4+anAAAgAElEQVRd/EMXFImSxttULiaHqAJprlhf5qjC39nPfvWXVxI58ZlMOPRSDlvTFEjkw/nSrto7fOe2CB45/7Dka895IK73UarFpmwPyLTK1TKrccmZC8tHMkZK4w0LPJeYbyjoq+k3Uo2BKM0O55yHKNQ7FYj12p7VXVufnQLZVpuqQHIxA/7usw/2bHf/iGv6dkq+toWVMf7CIN0CyUwq/PTEgY5j+cdH8kmu3WTuzLRMaC4WCPhfi7qwlGaH/VWOaoE4Zbr92n6a31kfPYbhdTq3BZKLcvH22DKxIDx/3u71HzJLwkrjW4c0rhtevIdr65pydH7fIHqUGEi4CZf5gAqI8+sRPqdl10MVSDMlaYEY+Hjpevpe8TzTFrvX7mokVdEkXttrXNRloEC8rIttO+tS3o+8883Ix/MjqlDzi3v4W2guBZSFVMhHOfcodGtXDcA5Q/fIyfH8FGG0aSDNR+gWo5hiqaMKpJlif5kbDLw5fw0Ar8z1X23QmSjlnDwFuXdh5QL/nHz3+8YWpzvq5yftxZmH9uLMQ3sTlWxTfKEwi2u1q65k8S2nccFR/XJyvGxcM2EWXakL3ShLADT2LfGLyQOqQJopknzCjiawnDEQ24iosCyQ2gwsEO8srPhpvAf1au/Zbv9W09Nw/d87N3Vs3YI/nXkQLVuUB++fxTOyO3aQMks99lELT1bVeEO2l3rcwGRw00r7SvKDKpBmitMCsQlyoTSkZGFZQfQyO4gergBWbtzOeX97j43batO25WMeiP/ktqA03hjzIjJ4Ag0bSyEX1solzs/t8hMGOjdktK/n9tijKjyh7rgSV4b5QEuZNFOSfmtj+MCn6q4T5wRAW95XJBVIuOD7yyvzeW3eatpUpX+lssm68tvTvr7QBaEyVACZlnMPPlb8fUsJZwzkZyftBcBdk+dHjIGEbC/xzyjK3CGbEstILghqgTQj5q/cxAdfJJSF/bTUYGCyFfv4MiD7KcVScVkgI+58g+8/9G7gue2Hay+Bn5UC8dnVd25CwExy8VE6QWQjE0pdOEYlzZLKZN8m/hmkZmEFU0orKRYKVSDNiJPueINv/fVtoPFpyCm8n/5oOcvWeyuRFBeWHQNxPFLZgXg/TNoL72PnishB9BQLJHMXllMhZeqiSHdhZbR7QbDHVFkuvmXls6nGm/aZud5G/Uz26d42Wsc8siu6qMJoci4sERkJ3AWUA//PGHNLkYdUkthf9vVbd6a0r9q4nZ4dWqb1Nx6xkkxmCdv+fa+n+zysnNvowgo5tvMKolgeuZQRTUneTPzZsfT1VSAxpX4Oef4nRxc9huR3O0vxwaBQNCkLRETKgXuBU4B9gbNFZN/ijqq0qXd9u/2+605LxRa0FTGcuhu31aW13fLC3IyP4x5LOvEnt2VCVi4s1/tCpPBmSpTPy6/eVy5cNlHvV3mZJLMCldKhqd2RIcACY8wiY8xO4AlgVJHHVJJI0oWV2u73tOTtwor+9bD3eWtBsKsrV0RdzyPFBRXFhZVlED7oWKVIlKdnd6ptp9YtrL/pS8Dm4/ylgu+jTBO4z/miqbmwegLOlY+WAkOLNJaSZXttPX+bshhIjz/4uQFenr0y+dreJ2htDTe5inMM7deJdz/znzFv4zeTPNtSJOmlUNKD8JkczYkxpSswA8vgu7ade3hfWldV8G1H2ZZdmVK9p4WgqVkgXl/zlNsnImNFZLqITF+9enWBhlVavOqYce7OgIoSj/jBY++zeUddRgok7o/o1AO688+LGp8Bxv9gWKT9/EaW65nOmQbhI5+7xB5bg+6fOwZSXiZ8p6Z3Rt+P5LGyyOgqBLuyMohDU1MgSwFn7YlewHJnB2PMOGNMjTGmpmvXrgUdnBcNDabgwT9n3GO7q4xIlLGs2byD52YszygGEvcKhVTftjtw75/GG63dK3Mm8CMIFPoB2yJ0b6rCaVec31BK9OnUiotyVJYm1zQ1F9Y0YKCI9AOWAWcB3y3ukILpf9UEDu/fmfFjoz1Z5wKnlfGPd79I2WZvmuRwWfkdI8oT5rad9ZSVZSEcJZFCmilJv3xaKZPspF2QBZOpIE1bK6TknrcbCfrY8llupCnppmJlgb3x6+OKct4oNCkLxBhTB/wIeAmYAzxpjJlV3FGF886itQU9n9eysjbGwKzlG7josemBx6g3hooIgn3QtS9ywp9fZ/OO9BImURAaq/7aXP/1RGLdnl2900qD8EpRTjtnoL/ff2PGLizXe2NKz2VjE+jCyuN5W3tULiglRu7fHYBu7apC+5aYV7IglPbd88AYMwGYUOxxlDJBM78Nhs/WbAk9hjGG8ohZWEvXbWPpunhrfIhImgI578h+DNuzMz3ateTsB6d67tcYRE+91s5tWvDp705hr2te8D1nJg+STqWRuQUSdNymQz4skN6dWjLm8L6cYgnoUuVHxw3g3MP3oEOrFmnuYDf29+q/lx5BdWVqkc6JPzsmX0MsKk3KAtmVGXXPWzz01mdAorihrSS+3LCdZ2ekhIHS5n44WbBqc6TzNTQYYniWMkbwdpXt070d7Vv5p4kGpfG2qIj/tQ6cyZ7tTPSYYyoEmWRh5eI8Pdq35KKj+5f83I6yMqFDq0TacnVlOZ/+7pTQfQ7u05FBPdqltA3crS0Ddyv+bPpcU9p3T0kyY+kGbnpuNgADr36BUfe+BcB3H5zKT8Z/yI66xqejoOq51z4zK1ImViIGkv712LazPtBFlikiwU/2mebe2+3/vGgo959zaKR9gvpIwLZMj1WKVFcm7nFwFeNCjab0CXo42RU/J1UgAQy5eRIPvrGo2MPwZOayjQAsWbcVgOXrtye37agNLp8eRQEsXruFj5akVvGtbzAMuvZFbvhf7sJOfhZIGM4VF93HAzhiQJek/zobUiciZkaaIViCJshDYw7jZyfuRe9O/rGjXLqwulurJQ7r1ymkZ9Phu0P7AIk1ZnY1mlwMpJCs2rSDmyfM4QfH9C/2UHyxdcFxt72WbAtbw3xnXfj6HI+983lam23ZPOqxLS4iEktA+e4R4VBBMZCWLt+183CZ1AaLc+5i0LtTKy4/cWBgn1wqkL5dWvPaL4fTu1OrnB0zl8S5PZccuyeXHLvnLllsURVIE8drBniYgti6M71eVRT2+e2LsfYLQohn+ufjt3rTN/bn0D06+m7PVpA6A/5NSdbEGWt5mfgmc/gVbSwlrjltEMfv081z25jD9+DwPTsn3++KisNGFUgJ09BgeOTtxYw+zH/dbq8n2lAFkoc1ymMjwS4sv9z7xiV73YfLvN6VzfeH7RHYN1sxUWrWR1TiCMhXfzGcRWuiJWyUIl3bVtG/axvPbTeM2r/AoyldVIGUMJPmrOTG52b7/hD9hOs9ry4IPO62naWjQITcurByXo03xnog5wzrw5PTlqa1N0X9ccI+3bj6tEEZ79encyv6dC5NN5WSOzSIXsJssyyF9Vu9J+n5tYextUgK5LLj9kxrK5O4Liw7iO5a0jbCvnEtgajj/N03DuDTm09pUm4qPx467zDfJ3FFUQVSwiSXifUReLUN4cFwL+LGQDLBaxb5gG7pgiiRxhsnC8u7PZ/+aPc5Lz8hOPjsvm/FXhBJCceu/7YrxzUyQRVIgeh35fNp8zNmLFnP41P9M5rsoKtfqfQnpy3xbA+jEBbIoxcMSWvzUhSCUB7wY/UtpligBaWCzvn9w9NjJkGo+ih9fvu1fTnviNKfIV8qqALJI7e9NC/52hjYuiNVcI+6dwrXPD0z9Dh+CuS2lz+NNa6wkgy5oHWL9PCaV7A8qgVy5Sn7cIIjK8anlmKgC6t/19Z0bFXJr0fuHXo+L9JW5lON0Ozo1LoF15++X1p5HcUb/ZTySFowO+bTcci0joyprfeWfAf17pCzc3gpBa/y8CIgEb6FRw7owkPnHZaynxdBuqhViwo+vPZkhu/tnZ4ZRlp13Qw1iCocpbmhCsSHbP3V0xanr6oX173itEB+PP7DuENK4lfqpEUOi195KQU/SyPIheV7fN998lh63HXooJpj/qgWUZoPkRWIiBwlIudbr7taa3I0W7LRH4tWb+bM+9/J+ph2f6cC+Z+rcGIcdvjME8mmCKEbL2XhPd8jOI3XjgO5u9iHSitlkscYiHucYRVh3JWCnRZYpqXhFaUUiSQxROQ64DfAlVZTJfB4vgZVCmTznLhhm3d6bVyXx2vzcrs0r58Fko3f9+6zD05576UrvEqBSMQ03qC1ylPb84d7nFGLSvbq2JIfHtOfu1yfUVx+PXJvrokxN0NRck1UifFN4HRgC4AxZjnQ/GoTO8hHymUmRWzrG4yvpZAtTgXinH2djQLZw1XbKHIMxKdvWj93ldyipPEmjn32kETxvE4hxfNatkjU1dq3RzuuPHVQpMWuonDp8AFcdHTp1mdTdh2izkTfaYwxImIARKT0i9lkSTbqw29fv2wqL3749/eZNCd42dm42JV8AVpVNRYPbBFRgfTp1IoHz63hlbmr+OOLc4GEe6p1i3K2WCnCni4srzTekFImzn5B7wuBfcoLj+rHH751QGj/bm2r+dfYYezfs31+B6YoRSLqI+eTIvIA0EFEfgBMAh6Me1IROVNEZolIg4jUuLZdKSILRGSeiIxwtI+02haIyBVxzx2VuAbIjCXr+dZf3/bc5lQgazfvCDxOvpSHG2e67baI6b2XHbcne3dvyyXDG2eWl7mq6kZ2YSHB64GYxn4AfzrjQP76vUNSXFhOS6AQLqxMrNOh/TunLNvalDKxOgQs6JVvDt2jo2flAqW0iGSBGGNuE5GTgI3A3sC1xpiJWZx3JvAt4AFno4jsC5wF7AfsDkwSkb2szfcCJwFLgWki8qwxZnYWYwjEHQCNytsL/dc/t4XH65+uZszD7/n2u+bpT2KdOw7Op/+1W3ZG2uf4fXZLaysrI0V6Rw2iJ2IgQUH0xn4AZ9YkCks6FfB7V53AgKtfSOmXD+zx52I9rVKf6PzRtScVdS7EU5ccUbRzK9EJVSAiUg68ZIw5EchGaSQxxsyxju3eNAp4whizA/hMRBYA9pTmBcaYRdZ+T1h986dAYgqJIMVjWyCzl2/07bN+604en/pFvJNnybaQEifjfzAspYy1kzJJDWt7CUjvmejRcPdzfncqyssok4Rgz2d2k33sTFyRTRV7GVdFCSL0EcMYUw9sFZFCOHJ7As76HEutNr/2vOElI95esIYfPDbdM/tm28565q/cFKh47N3atUzV206XyDkPvRtrvHF46zfHpbwPKnFy1mG9fZUH2EURgyvXek8kjCfwC1WN1+vY2eiPMUf0pU1Vhe9aE4rSlIgaRN8OfCIiE7EysQCMMT/x20FEJgFeBWWuNsY847ebR5vBW9F5/oxFZCwwFqBPnz5+wwvFy5K44NFpbK9tYHtdPa1cpTp+PP5DJs1ZGei3/Wz1Frq2qaJddapvucGAPYfPGeDOBYf17ci0xes8t/XqmJo5lRjXNs++tuvIDxEJFd5xlq71ize4rRkRyXuAwVZ22Vggg3q0Y+YNI8I7KkoTIKoCed76FxnL5ZUpSwGnpOoF2DPn/Nrd5x0HjAOoqamJ/Ut3yohX565iv57tkm11HhbIlAVrfLfZnPPQu/zw2P4c0id11bsGY3jh4xU88PqilEymXFBVUR7eCWjdopxbzziQr/3lLc/tYYHjKKm4ni6siDolrV+MUibZkoMVbRWlWRE1iP6oiLQA7ID2PGNMvMUognkW+KeI3E4iiD4QeI+EuBhozX5fRiLQ/t08nD+JU1ye/8g0+ndpnWyr9ZifkSxrEaKyFqzczD7dU6fQnH7PFOasSFgeXdpU5VSB+HHlKfsAjQL3+4f3Zbd21Z59K8qEgbsFT/spk/B4hmcQPWbMwnceSB5jIGOP6c/lT3xUsut5K0qhiaRARGQ48CiwmISc6C0iY4wxb8Q5qYh8E/gL0BV4XkQ+MsaMMMbMEpEnSQTH64DLrBgMIvIj4CWgHHjYGDMrzrmj4n7i/mztFirLEp602Ss2snDVZs47MlHN5cWZXyaXkQ1zb7RrWcnP/jUjpc1WHgBrQtJ7c4XXutSVHrWw2lZX8Mn14S6XMpHQeIZXUk9ci8HP4vE73omD0jPHwrjFNddj1OCejBqc19CbojQporqw/gycbIyZB2Cl1o4HDo1zUmPMf4H/+my7GbjZo30CMCHO+eLgpQbsuMj3H0qk4NoK5OLH30/2CaucW5+LHNAQXvrpMTzwxkIG9+7AxNne80ncAthgqPCQ8FFTOSWCBeKlYML28fu0MgmiL77ltMBz/POioZ7X3ra6ePMgFKUpEFWBVNrKA8AY86mINOtfl4lZRaQ+ZJXAeBVcM2Pv7m25/TuDAQIUSOKv0+XjlSUVNfBdFiGInk0MxK0y0mamJ/9mbtIcMaBLxvsoihJ9Jvp0EXlIRIZb/x4E3g/dq4myetMODrrx5ZQ2Y9LX0fBK5500Z1Xgset91uIoNF7C3Mva8FIqfscb1KNdSJ/0trhpvJm6sOJQ6pP9FKXYRFUglwCzgJ8Al5OIUVycr0EVm6hlzb3iHcvWe6fB2gRlaUXhsL4dwztFIE04Gm9rI+qyrWUCf/3eISF9YkwkjPhxqbBXlMIT1YVVAdxljLkdkrPTq/I2qiJTFVGB1BsT+QO0eWfhmswH5CBqWm4YUdJuh/XvxCXHRqtHJCLxYgYhw3CXMrHxtUAyH0EaRw/swpvzs7tPirIrENUCmQw4a1G3JFFQsVkStSptnHBGtim61ZW5qU+UJoA9JG9leVlkF1MUT5dfMcUopJcySX1vuxdzoWDbVKU+Fnzz4J7cMfqgrI+rKM2NqNKo2hiz2X5jvW62yfBegs6LQmRU2RxglQQPE5CDXeua2wrgnu+GL/jkJpOZ49EmEqa3xXU9+e22W/vcGcb2A8IdowfzzYN75ey4itJciKpAtohI0sFtlWAPdvbvAsxctqFg57KFb1WABfLaL4fz+EVDU9rs+Szup+o0y8JDF0YNoCfGl+g7pF+nZNv715zo2SdlHNbff198OK//anjG57OxS7rnwgLReIqiRCOqC/+nwL9FZDkJUbM7MDpvo2oijB43NTmjOx8c0qcDH3yxPvHGkmrVlf4C0mtyoB/JNN4AYZmJBSKWXvvnRUOTqcqd26RaA15HO8pKoT2sbyePrf4lVNzjfvHyo1nvs5RwXOKW9FeUXYVAC0REDhOR7saYacA+wL9IzBB/EfisAOMref7wwty8Hfs7jgKGtryszvAJ2y+GEcVNV1EWPd5iWwQV5WX+VoDrlA+eWxN5Dob7Otzvu7WrZq+QcitRyWc5FEVpToRJiAcAe5Whw4GrSCzstA6rYKGSP5yzo2152a9LvNCT+1narT/s7YtvOY27zkpMQswsBhLexy2YKzxKp5QSu8CyH4qSFWEKpNwY85X1ejQwzhjzlDHmt8CA/A5NcSaD2aJ2z65tePuK41NiDUH4i2gJ2Z5ZxldYEP27Q9NL6+dCfRyVj1nkpa3XFKVkCFUgImLHSU4AXnFsy3QKhJIhTqFsu2zKy4TdO7SMPMHumL26AtCnUysW/v5UurRpYR07dfuI/RqLDZ56QA9+cHQ/rjp1UOSxhgWef//NA4I7+JCcB+Kx7c1fH8eD59bEOm4m51YUxZswJTAeeF1E1pDIunoTQEQGAIVLQSoCe+3Whk9Xbg7vmEdSVvhztdmz4Ad2a8P8Vf7jvODIvpx+0O50bVuVsr+tnAb1aJdWbLCyvIyrT9s3o7FGSeONg+1G8jp8vsqqqwGiKNEIVCDGmJtFZDLQA3jZNKbElAE/zvfgiklN307FVyCO12UuxWH/veH0/fjzxE859YAe3scQSSoP5zFzLfDzpUCKSdgiWoqyqxPqhjLGTPVo+zQ/w9k1qa4sY3ttehXfFKFsvbRlml2TsbxMeOqSIyKfyz5kruV9pJno7snvEQYxoFsbvvhqa2D6cq6JW+BRUXY1NI7hQ65ESJkk1jwPomOrFqzYsN1z3+u+vi97d2/LXZPmA41zE+yn47iZTLm2GKII3c5tqvj1yL3p36U19766kJo9wgtD3nXWYD78Yr3vaomKohQPVSA+5Eq+VpSVsTNklam21RWs8IgoiQjnW4tW2QrExi6jUp7BXA1oTKXNcLeccenwRPLeyP29XW5u2lZXJgP9hULtD0WJRlHEiIj8SUTmisjHIvJfEeng2HaliCwQkXkiMsLRPtJqWyAiV+R9jDkSI1HmUnRs1cJ7DOLx2rJmbKsmk3IjTppjzCLXaAhEUYIp0nMoE4H9jTEHAp8CVwKIyL7AWcB+wEjgryJSbpWPvxc4BdgXONvqmzdyZoFEcDF96xDvdbadQv7kfbsDjZlH9mJWmSoCu3tMvbNLoLpVUaJRFBeWMca53N9U4Azr9SjgCWPMDuAzEVkADLG2LTDGLAIQkSesvrPzNcZcyZAoFsLow/pwzF5duf+1hTz6zufJdueu5x/Zl28f0ov2rRJrbthZWJnMFneSq0Dx4xcO5aVZX+bkWKWG1sJSlGBKIQZyAYkaWwA9SSgUm6VWG8ASV3tq2dkckysB614G148e7VtS3SI108g9kdBWHuBUIJmNJ9dpvEcN7MJRA/1ng3dp04I1m3f6bi9F1ABRlGjkTYGIyCSgu8emq40xz1h9riZRnPEf9m4e/Q3erjZPySwiY4GxAH36pJfPKDRbdtZF7lvpjmwHSLKG5AS7uDGQWLtlzJu/Pp66huAkglJFYyCKEkzeFIgx5sSg7SIyBvgacIJjguJSoLejWy9gufXar9193nFYhR5ramqKLgIyEULueEmQlZC0QDKOgaTORM83LVuUA4Wbw5ELdB6IokSjWFlYI4HfAKcbY7Y6Nj0LnCUiVSLSDxgIvAdMAwaKSD8RaUEi0P5sfseYz6N7446XBFkJtgLRbKr8oRaIogRTrBjIPUAVMNF62ptqjLnYGDNLRJ4kERyvAy4zxtQDiMiPgJdIPM4+bIyZlc8BFmNNiApXQCPQArG8Qqo/co9+pIoSjWJlYfmWgjfG3Azc7NE+AZiQz3E5yZdgfmLsMG7832xmr9gIwKSfH5vc5rZAgoaQbRaWEo4aIIoSTLHmgZQ8+RLLw/p3ZsLlRyffD+jWJvna7TIJ8sVfdHR/wH8Soh9qsURAPyNFiUQppPGWJMUQtNtq61PeBxkXFx7VjwuP6hf7XOrfD0er8SpKMGqB+JCLTJyfnDCQqzNYlGn91tqU91HWLc8UtUDC0TXRFSUaqkB8iCpCfnbiXr7bfn7SXgzu08F3u5uLh/fnFyftxb492gH5nauhs6zD0U9IUYJRBZIFZQKdWleG9ImuBbq1rebHJwx0zAfJvQa50Kru27lNVUjPXRe10hQlGhoD8SOCEDGEu7qyyZLKhwVy3pH9OO/I+LGTXQo1QRQlELVAfIjiBzcm/Gk105ni9nFBJwkWC/3UFSUaqkB8yJXsjrNwkx2fUP1RXDROpCjBqALJkjBLxc+FVV2pH32poopbUaKhMRAfosqQMGHjtx7I9GtOoj5iqXelOOg0EEUJRhWID1GfQsO6+cUx2lT5f/S24NL5CMVBP3dFiYb6UXyIKkRCg+gxUqmSCkTlWFFRA0RRglEF4kN0CyS4o9MCGbHfbtkMSSkQqrgVJRqqQPKMbYF0bt2CB75fU+TRKJmgMRBFCUYViA+RH0J9Or756+OARgWSiSxSuVVc1AJRlGioAvEjohTx69W7UyugUYE0ZPA4a1eBVUFWXHQeiKIEowrEh+hpvCHzQKztDQ2ZCyPNBioW+rkrShSKtSb6TSLysYh8JCIvi8juVruIyN0issDafohjnzEiMt/6Nyb/Y4zYL2S7HUTXZ9mmh8ZAFCWYYlkgfzLGHGiMGQw8B1xrtZ8CDLT+jQXuAxCRTsB1wFBgCHCdiHTM5wBzlcabPIwKoyaDug4VJRpFUSDGmI2Ot61pFK+jgMdMgqlABxHpAYwAJhpjvjLGrAMmAiPzOcaavrnRT7YwyiiIrvNASgLV+YoSTNFiICJys4gsAb5HowXSE1ji6LbUavNrzxtHDugSqZ/tohpkLQLlprqiHIDRh/WOfG4tplhc9GNXlGjkrZSJiEwCuntsutoY84wx5mrgahG5EvgRCReV12/XBLR7nXcsCfcXffr0iTP0JGUC7th3+5aVbNjWuPSsLeT37NoagDkrNqb0b1FRxtybRtKiXPMVmhwaBFGUQPIm1YwxJxpj9vf494yr6z+Bb1uvlwLOR/VewPKAdq/zjjPG1Bhjarp27ZrdNXi0tW/pvQKhiPDC5Ud7bquuLM/L+uZKflDLT1GiUawsrIGOt6cDc63XzwLnWtlYw4ANxpgVwEvAySLS0Qqen2y15RWvB9B6n3Rck8OnVS2mWBqo/aEowRTLr3KLiMwUkY9JKIPLrfYJwCJgAfAgcCmAMeYr4CZgmvXvRqut4LgnBOYjTffsIQnXW/d21Tk8qhIVVdyKEo2ilHM3xnzbp90Al/lsexh4OJ/jioJbgUge0nQvOKofFxyl65YXGw2BKEowGtnNkPoG73Yte9F80BiIokRDFUiG7N29Tcp7292hT6vNj1zGtRSlOaIKJAPat6zkvnMOTWnTp9Xmh95SRYmGKpAMGNqvE+2qU9N4kyEQfVhtdugtVZRgVIFkwE9OGJjWZlsgmZRrV0qbsArLiqIkUAWSAfv3bJ/y/ofH9kcdHs0XfSZQlGBUgWTBlacMSr5WWdN8uPCofgzo1oavH7R7sYeiKCVNUeaBNCeS1XZVgzQbendqxaSfH1vsYShKyaMKJEuO3asrpx7QPcUaURRF2RVQBZIl1ZXl/PV7jam9px7QnY6tWhRxRIqiKIVBFUiOcSoTRVGU5owG0RVFUZRYqAJRFEVRYqEKRFEURYmFKhBFURQlFhpEj8H95xxKmyr96BRF2bVRKSB3NbgAAAqWSURBVBiDkft3L/YQFEVRik5RXVgi8ksRMSLSxXovInK3iCwQkY9F5BBH3zEiMt/6N6Z4o1YURVGgiBaIiPQGTgK+cDSfAgy0/g0F7gOGikgn4DqghkTZqfdF5FljzLrCjlpRFEWxKaYFcgfwa1LrEI4CHjMJpgIdRKQHMAKYaIz5ylIaE4GRBR+xoiiKkqQoCkRETgeWGWNmuDb1BJY43i+12vzaFUVRlCKRNxeWiEwCvKLNVwNXASd77ebRZgLavc47FhgL0KdPn0hjVRRFUTInbwrEGHOiV7uIHAD0A2ZYK7/1Aj4QkSEkLIveju69gOVW+3BX+2s+5x0HjAOoqanRIuuKoih5ouAuLGPMJ8aYbsaYvsaYviSUwyHGmC+BZ4FzrWysYcAGY8wK4CXgZBHpKCIdSVgvLxV67IqiKEojpTYPZAJwKrAA2AqcD2CM+UpEbgKmWf1uNMZ8VZwhKoqiKFACCsSyQuzXBrjMp9/DwMMFGpaiKIoSgtbCUhRFUWKhCkRRFEWJRdFdWE2BHu2raVlZXuxhKIqilBSqQCLw1CVHsHuHlsUehqIoSkmhLqwIlInXPEZFUZRdG1UgEVD9oSiKko4qkAhUlKkGURRFcaMKJAIV5foxKYqiuFHJGIHKcrVAFEVR3KgCiUBFmX5MiqIoblQyRkAtEEVRlHRUgURANA1LURQlDVUgiqIoSixUgSiKoiixUAWiKIqixEIViKIoihILVSCKoihKLIqiQETkehFZJiIfWf9OdWy7UkQWiMg8ERnhaB9ptS0QkSuKMW5FURSlkWKWc7/DGHObs0FE9gXOAvYDdgcmiche1uZ7gZOApcA0EXnWGDO7kANWFEVRGim19UBGAU8YY3YAn4nIAmCItW2BMWYRgIg8YfVVBaIoilIkiqlAfiQi5wLTgV8YY9YBPYGpjj5LrTaAJa72ofke4OMXDmXtlh35Po2iKEqTJG8KREQmAd09Nl0N3AfcBBjr75+BCwCvKd8G71iN8TnvWGAsQJ8+fTIet5OjBnbJan9FUZTmTN4UiDHmxCj9RORB4Dnr7VKgt2NzL2C59dqv3X3eccA4gJqaGk8loyiKomRPsbKwejjefhOYab1+FjhLRKpEpB8wEHgPmAYMFJF+ItKCRKD92UKOWVEURUmlWDGQW0VkMAk31GLghwDGmFki8iSJ4HgdcJkxph5ARH4EvASUAw8bY2YVY+CKoihKAjGm+Xp5ampqzPTp04s9DEVRlCaFiLxvjKkJ66cz0RVFUZRYqAJRFEVRYqEKRFEURYmFKhBFURQlFs06iC4iq4HPszhEF2BNjoZTiuj1NW30+po2pXx9exhjuoZ1atYKJFtEZHqUTISmil5f00avr2nTHK5PXViKoihKLFSBKIqiKLFQBRLMuGIPIM/o9TVt9PqaNk3++jQGoiiKosRCLRBFURQlFqpAPGgO66+LSG8ReVVE5ojILBG53GrvJCITRWS+9bej1S4icrd1zR+LyCHFvYJoiEi5iHwoIs9Z7/uJyLvW9f3Lqt6MVeH5X9b1vSsifYs57iiISAcR+T8RmWvdx8Ob0/0TkZ9Z382ZIjJeRKqb8v0TkYdFZJWIzHS0ZXy/RGSM1X++iIwpxrVERRWICxEpJ7H++inAvsDZ1lrtTY06Eis9DgKGAZdZ13EFMNkYMxCYbL2HxPUOtP6NJbHoV1PgcmCO4/0fgTus61sHXGi1XwisM8YMAO6w+pU6dwEvGmP2AQ4icZ3N4v6JSE/gJ0CNMWZ/ElW2z6Jp379HgJGutozul4h0Aq4jseLqEOA6W+mUJMYY/ef4BxwOvOR4fyVwZbHHlYPregY4CZgH9LDaegDzrNcPAGc7+if7leo/EguLTQaOJ7EomZCYmFXhvpcklgI43HpdYfWTYl9DwLW1Az5zj7G53D8SS1UvATpZ9+M5YERTv39AX2Bm3PsFnA084GhP6Vdq/9QCScf+Yts412Vvkljm/sHAu8BuxpgVANbfbla3pnjddwK/Bhqs952B9caYOuu98xqS12dt32D1L1X6A6uBv1kuuv8nIq1pJvfPGLMMuA34AlhB4n68T/O5fzaZ3q8mdR9VgaTjty57k0RE2gBPAT81xmwM6urRVrLXLSJfA1YZY953Nnt0NRG2lSIVwCHAfcaYg4EtNLo/vGhS12e5ZUYB/YDdgdYk3Dpumur9C8PveprUdaoCSSdoXfYmhYhUklAe/zDG/MdqXmkvKWz9XWW1N7XrPhI4XUQWA0+QcGPdCXQQEXulTec1JK/P2t4e+KqQA86QpcBSY8y71vv/I6FQmsv9OxH4zBiz2hhTC/wHOILmc/9sMr1fTeo+qgJJp1msvy4iAjwEzDHG3O7Y9CxgZ3aMIREbsdvPtbJDhgEbbNO7FDHGXGmM6WWM6UviHr1ijPke8CpwhtXNfX32dZ9h9S/ZJztjzJfAEhHZ22o6gcRSz83i/pFwXQ0TkVbWd9W+vmZx/xxker9eAk4WkY6WlXay1VaaFDsIU4r/gFOBT4GFwNXFHk/MaziKhOn7MfCR9e9UEn7jycB8628nq7+QyD5bCHxCIjum6NcR8VqHA89Zr/sD7wELgH8DVVZ7tfV+gbW9f7HHHeG6BgPTrXv4NNCxOd0/4AZgLjAT+DtQ1ZTvHzCeRDynloQlcWGc+wVcYF3nAuD8Yl9X0D+dia4oiqLEQl1YiqIoSixUgSiKoiixUAWiKIqixEIViKIoihILVSCKoihKLFSBKIoPIlIvIh85/gVWZhaRi0Xk3Bycd7GIdImx3wgRud6aQzAh23EoShgV4V0UZZdlmzFmcNTOxpj78zmYCBxNYiLeMcCUIo9F2QVQBaIoGWKVT/kXcJzV9F1jzAIRuR7YbIy5TUR+AlxMoqz+bGPMWVap7odJTJbbCow1xnwsIp1JTELrSmKSnDjOdQ6JsuctSBTDvNQYU+8az2gSVaP7k6gvtRuwUUSGGmNOz8dnoCigLixFCaKly4U12rFtozFmCHAPiRpcbq4ADjbGHEhCkUBi5vWHVttVwGNW+3XAWyZRNPFZoA+AiAwCRgNHWpZQPfA994mMMf8iUSdrpjHmABIzuw9W5aHkG7VAFMWfIBfWeMffOzy2fwz8Q0SeJlGGBBLlZb4NYIx5RUQ6i0h7Ei6nb1ntz4vIOqv/CcChwLREuSha0liMz81AEmUxAFoZYzZFuD5FyQpVIIoSD+Pz2uY0EorhdOC3IrIfwaW6vY4hwKPGmCuDBiIi04EuQIWIzAZ6iMhHwI+NMW8GX4aixEddWIoSj9GOv+84N4hIGdDbGPMqiQWvOgBtgDewXFAiMhxYYxJrtDjbTyFRNBESxffOEJFu1rZOIrKHeyDGmBrgeRLxj1tJFAAdrMpDyTdqgSiKPy2tJ3mbF40xdipvlYi8S+Ih7GzXfuXA45Z7Skis8b3eCrL/TUQ+JhFEt8t83wCMF5EPgNdJlDrHGDNbRK4BXraUUi1wGfC5x1gPIRFsvxS43WO7ouQcrcarKBliZWHVGGPWFHssilJM1IWlKIqixEItEEVRFCUWaoEoiqIosVAFoiiKosRCFYiiKIoSC1UgiqIoSixUgSiKoiixUAWiKIqixOL/A//mj4bfVePeAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f0340a70c50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from double_dqn_agent import Agent\n",
    "agent = Agent(state_size=8, action_size=4, seed=42)\n",
    "\n",
    "def dqn(n_episodes=2000, max_t=1000, eps_start=1.0, eps_end=0.01, eps_decay=0.995):\n",
    "    \"\"\"Deep Q-Learning.\n",
    "    \n",
    "    Params\n",
    "    ======\n",
    "        n_episodes (int): maximum number of training episodes\n",
    "        max_t (int): maximum number of timesteps per episode\n",
    "        eps_start (float): starting value of epsilon, for epsilon-greedy action selection\n",
    "        eps_end (float): minimum value of epsilon\n",
    "        eps_decay (float): multiplicative factor (per episode) for decreasing epsilon\n",
    "    \"\"\"\n",
    "    scores = []                        # list containing scores from each episode\n",
    "    scores_window = deque(maxlen=100)  # last 100 scores\n",
    "    eps = eps_start                    # initialize epsilon\n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        state = env.reset()\n",
    "        score = 0\n",
    "        for t in range(max_t):\n",
    "            action = agent.act(state, eps)\n",
    "            next_state, reward, done, _ = env.step(action)\n",
    "            agent.step(state, action, reward, next_state, done)\n",
    "            state = next_state\n",
    "            score += reward\n",
    "            if done:\n",
    "                break \n",
    "        scores_window.append(score)       # save most recent score\n",
    "        scores.append(score)              # save most recent score\n",
    "        eps = max(eps_end, eps_decay*eps) # decrease epsilon\n",
    "        print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)), end=\"\")\n",
    "        if i_episode % 100 == 0:\n",
    "            print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)))\n",
    "        if np.mean(scores_window)>=200.0:\n",
    "            print('\\nEnvironment solved in {:d} episodes!\\tAverage Score: {:.2f}'.format(i_episode-100, np.mean(scores_window)))\n",
    "            torch.save(agent.qnetwork_local.state_dict(), 'checkpoint.pth')\n",
    "            break\n",
    "    return scores\n",
    "\n",
    "scores = dqn()\n",
    "\n",
    "# plot the scores\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(len(scores)), scores)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Watch a Smart Agent!\n",
    "\n",
    "In the next code cell, you will load the trained weights from file to watch a smart agent!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD8CAYAAAB9y7/cAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAABjVJREFUeJzt3cFx00oAx+EVkypow2ngXXJyHVAADaSBVwDUkRMXGggNvAK4UYPegZgxjj1YkaXd/ev7Zrgw0WiRzc+blSwN4zgWAPK8qz0AAJYh8AChBB4glMADhBJ4gFACDxBK4AFCCTxAKIEHCHVXewCllDIMg6/TApwYx3GYs70ZPEAogQcIJfAAoQQeIJTAA4QSeIBQAg8QSuABQgk8QCiBBwgl8AChBB4glMADhBJ4gFACDxBK4AFCCTxAKIEHCCXwAKEEHiCUwAOEEniAUAIPEErgAUIJPEAogQcIJfAAoQQeIJTAA4QSeIBQAg8QSuABQgk8QCiBBwgl8AChBB4glMADhBJ4gFACDxBK4AFCCTxAKIEHCCXwAKEEHiCUwAOEEniAUAIPEErgAUIJPEAogQcIJfAAoQQeIJTAA4QSeIBQAg8QSuABQgk8QCiBBwgl8AChBB4glMADhBJ4gFACDxBK4AFCCTxAKIEHCCXwAKEEHiCUwAOEEniAUAIPEErgAUIJPEAogQcIJfAAoQQeIJTAA4QSeIBQAg8QSuABQgk8QCiBBwgl8AChBB4glMADhBJ4gFACDxBK4AFCCTxAKIEHCCXwAKEEHiCUwAOEEniAUAIPEErgAUIJPEAogQcIJfAAoQQeIJTAA4QSeIBQd7UHAHPt949X/+zT0/U/C70TeLpxKeT37z9ctf3zjy83HA20T+DpzrVBh62zBk9XxB2uJ/Bsxv37D5PW66F3Ag8QSuABQgk8XdjvH62/w0QCDxBK4AFCCTxAKF90oknHlzMudXuBNfYBNZnB06wlTqqeXgvvxC3JBJ7m+DIS3IYlGpq0xsz66emxlP3LvvaWacgj8DTr+ceXP6LrbpAwjcDTlEtfaDK7hukEniadzt6XYpmGZE6yAoQSeJpxuHqmxlq79X0SDeM41h5DGYah/iBoxn7/aKkESinjOA5zthd4gEbNDbwlGoBQAg8QSuABQgk8QCiBBwgl8AChBB4glMADhBJ4gFACDxBK4AFCCTzNGMexPD/XHkV9jgG34oEfNOdc4O7v1x9HTZciv7XjwDwCTxcE7xcffkxhiQYglBk8XTBL/cVxYAqBpzki5hhwG57oRDPGcSzDMOsBNhDFE50AOEvgAUIJPEAogQcIJfAAoQQeIJTAA4QSeIBQAg8QSuABQgk8QCiBBwgl8GzWp92u9hBgUQJP9749PJRvDw+TtjnEXeRJJvBsjqizFe4Hz2+H90Kte7K/5X7wpzP3f75+vXrbT7td+ff790n7gzXNvR+8wG/clNd/6fC/9YEfh8hPiTv0YG7gPbJvo97ywX5pm5pPYfq825X/fv784+/2L0swT2bnbJzAb8wSv7G1GH7ASdZNGMfx959a+11q/x9fZukfzdbhFWvwwVp4ba9ltg+vWYPnrJ7iXkp/412aDzxuQeDDCGUG5zW4BYEPIezbcO51Fn0uEfiOiTqliD6XCXyHhJ2/EX1KEfiuCDtziP72CHwnxJ0lHL+vxD6PwDdO2FmLGX4egW+UsNMCM/y+CXxDRJ2WmeH3R+ArE3V6dvr+Ffy2CPyKxJx0lnTaIvArEHa2SOzrE/gbE3N4TezrEPiZBB2msW6/HoGfSNDhtlyds5wmAv/WaK7xEGhgfYf/e0I/TxNPdCqlVBvE4Q3UyHEAztha6I96NOsfvvnAA/3YQuhPmuyRfcA2pC7dLDXRFnigOymhX3oFReCBbvUa+rWWxgUe6F4Poa9xvlPggRgtfmO25oUs76rtGWBBLVwhWHsMZvBArBpLN7WjfkzggXhrhL6lsB8IPLAZS6zRtxj2A4EHNmnOrL7lqB8TeGDTpoS+l7AfuIoGoPw93r3FvRQzeIDfTtfoe4z6MTN4gDN6j3spAg8QS+ABQgk8QCiBBwgl8AChBB4glMADhBJ4gFACDxBK4AFCCTxAKIEHCCXwAKEEHiCUwAOEEniAUAIPEErgAUIJPEAogQcIJfAAoe5qD+DFUHsAAGnM4AFCCTxAKIEHCCXwAKEEHiCUwAOEEniAUAIPEErgAUIJPEAogQcIJfAAoQQeIJTAA4QSeIBQAg8QSuABQgk8QCiBBwgl8AChBB4glMADhBJ4gFD/A5qolaC9MgerAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f42e3c15a20>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from double_dqn_agent import Agent\n",
    "agent = Agent(state_size=8, action_size=4, seed=0)\n",
    "\n",
    "# load the weights from file\n",
    "agent.qnetwork_local.load_state_dict(torch.load('checkpoint.pth'))\n",
    "\n",
    "for i in range(3):\n",
    "    state = env.reset()\n",
    "    img = plt.imshow(env.render(mode='rgb_array'))\n",
    "    for j in range(200):\n",
    "        action = agent.act(state)\n",
    "        img.set_data(env.render(mode='rgb_array')) \n",
    "        plt.axis('off')\n",
    "        display.display(plt.gcf())\n",
    "        display.clear_output(wait=True)\n",
    "        state, reward, done, _ = env.step(action)\n",
    "        if done:\n",
    "            break \n",
    "            \n",
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Explore\n",
    "\n",
    "In this exercise, you have implemented a DQN agent and demonstrated how to use it to solve an OpenAI Gym environment.  To continue your learning, you are encouraged to complete any (or all!) of the following tasks:\n",
    "- Amend the various hyperparameters and network architecture to see if you can get your agent to solve the environment faster.  Once you build intuition for the hyperparameters that work well with this environment, try solving a different OpenAI Gym task with discrete actions!\n",
    "- You may like to implement some improvements such as prioritized experience replay, Double DQN, or Dueling DQN! \n",
    "- Write a blog post explaining the intuition behind the DQN algorithm and demonstrating how to use it to solve an RL environment of your choosing.  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}