Deep_Q_Network-Dueling-DDQN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dueling Double Deep Q-Network (DDDQN)\n",
    "---\n",
    "Implementation of the agent with OpenAI Gym's LunarLander-v2 environment. The code is based on materials from Udacity Deep Reinforcement Learning Nanodegree Program. \n",
    "\n",
    "### 1. Import the Necessary Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: box2d in e:\\anaconda3\\lib\\site-packages (2.3.2)\n",
      "Requirement already satisfied: pyvirtualdisplay in e:\\anaconda3\\lib\\site-packages (0.2.1)\n",
      "Requirement already satisfied: EasyProcess in e:\\anaconda3\\lib\\site-packages (from pyvirtualdisplay) (0.2.3)\n"
     ]
    }
   ],
   "source": [
    "import gym\n",
    "!pip install box2d\n",
    "import random\n",
    "import torch\n",
    "import numpy as np\n",
    "from collections import deque\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "!python -m pip install pyvirtualdisplay\n",
    "#from pyvirtualdisplay import Display\n",
    "#display = Display(visible=0, size=(1400, 900))\n",
    "#display.start()\n",
    "\n",
    "is_ipython = 'inline' in plt.get_backend()\n",
    "if is_ipython:\n",
    "    from IPython import display\n",
    "\n",
    "plt.ion()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Instantiate the Environment and Agent\n",
    "\n",
    "Initialize the environment in the code cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARN: gym.spaces.Box autodetected dtype as <class 'numpy.float32'>. Please provide explicit dtype.\u001b[0m\n",
      "State shape:  (8,)\n",
      "Number of actions:  4\n"
     ]
    }
   ],
   "source": [
    "env = gym.make('LunarLander-v2')\n",
    "env.seed(0)\n",
    "print('State shape: ', env.observation_space.shape)\n",
    "print('Number of actions: ', env.action_space.n)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Train the Agent \n",
    "\n",
    "Run the code cell below to train the agent from scratch. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 100\tAverage Score: -249.33\n",
      "Episode 200\tAverage Score: -224.96\n",
      "Episode 300\tAverage Score: -147.08\n",
      "Episode 400\tAverage Score: -86.99\n",
      "Episode 500\tAverage Score: -31.10\n",
      "Episode 600\tAverage Score: 56.62\n",
      "Episode 700\tAverage Score: 66.13\n",
      "Episode 800\tAverage Score: 121.63\n",
      "Episode 900\tAverage Score: 197.63\n",
      "Episode 914\tAverage Score: 200.22\n",
      "Environment solved in 814 episodes!\tAverage Score: 200.22\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEKCAYAAAA8QgPpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztnXecFOX9xz/f3b1+Rzk6HEhXAaV4IkZQjIIgRqJRsZvkZ9TEaIwm9sRuiD2WxBYTjb1Go4gCNixIUaR3kSLlKB4cx5XdfX5/TNmZ2ek7W27v+3697nW7zzwz88zszPN9vuX5PiSEAMMwDMN4JZTtBjAMwzAtExYgDMMwjC9YgDAMwzC+YAHCMAzD+IIFCMMwDOMLFiAMwzCML1iAMAzDML5gAcIwDMP4ggUIwzAM44tIthuQTjp27Ch69+6d7WYwDMO0KBYsWLBDCNHJqV5eC5DevXtj/vz52W4GwzBMi4KIvnNTj01YDMMwjC9YgDAMwzC+YAHCMAzD+IIFCMMwDOMLFiAMwzCML1iAMAzDML5gAcIwDMP4ggUIwzA5x7TFW7B7X1O2m8E4wAKEYZicYvveBvzmua9w8X8WZLspGaExGsNXG3b73n/T7nrc8r+liMVFgK1yBwsQhmEC4Yf6Jrw0b0PKx4nGpI5w4+76lI+VbfY1RhF36Nhvf3s5Tv3751hbU+frHFe+/A3+9dl6LNzoXwj5hQUIwzCoc9HROfH7lxbimtcWY+XWvSkdh0j6Hxf+2rNpdz027Myu8NnXGMW2PQ0YfNN7uPv9lbZ1l35fCwC+TXaK5pEFBYQFCMO0dvY0NGPITe/hvhmrXNW//o3F+P1LC5PKt9Q2AIArU8r2vQ149OO1ECZCIiRLEJ/yA6P/+iGOvvtDfzu74Nsd+/CHV75BcyyetK1mbyOisTiqb5+JI+6cBQB4dcEmAEA8LnDDG4ux7Ps9un1IkZgumb26BnsamtXvIXn3WFzg1v8tw5LNtZ6OlwosQBimlbOzThr5/m/R967qP//lBrzx9eak8qgsOCJh5w7x2tcWY+q7K7Bw4w9J25S9szGidsM1ry7Cqws24esN+rbX7m/G4XfMxB3TlmN/c0wtVzS7Tbv347kvN+DiZ6UEr82xOE56aDYWfCeZntxc7p6GZpz3z7kYf98naIxK55i3Xtr/7x+txVOffYuTHvo01Ut0DQsQhmlFfLdzH256c4lOS4jFpZF0UzSOusaoWv7k7HV45MM1ro8dlUfkkZCzAFFG77X7m5O2JUbkuSlBCiNSt/nxqu3YK2sCf3l3uaqVzVy+TVdfEawNcodfGJb231HXiCWbE9qIG42rsVm6b1v3NOC61xdj+54Gddsnq2r8XE5KsABhmFbE5S98jae/+E5n5miWndZbahsw5Kb31PLb31mOu9/T2+//8dFay2Mrx7lz2go0RZPNO1rKCqWVJOqbYknbEj4Q20NkjeICqdt85MO1uOJFSWg89vE6fLBiOwCgIKzvVhUNZL98rcUFYdPjKua8sx6fg/H3f2xaR9E6AElgRC1u0pOz15maB4OGBQjDtEK0XYuX8M+/Tl9huS0qazIzl2/DO4vtzWGlRVInuk+j8SS1Ue4AY3GBP77yjepstqv/l2nL1e+zV9fggxXbbPbwR5FGAKwwCRgoCOm7VaWT39sgXWtxQRjNsTjqGvTX3hwT+HT1DnyxbidWbUtEZNU3RfH+0q1YsXWPau5SjmsVaHD7O8uxcdd+j1fmnbxeUIphGD2KeSgWF2iOxXHOE1/iuIM7O+7XFI1j+ZY9tnWU8FsA2FrbqH4WQkAIIKQxbZUXSV3PnHW78Mr8TXjm/0aqI3OlT1Tk2sZd9XhlwSZ8+e0ufHL1seoxbvzvYsxYtg0f/eFYlBSGUbO3EY99sk7dft4/5wIAelWW6vZzixACf/9oLU4e2h09K0vV8uJIQoCY+XuMZTEh0BiNYec+6Z4UF4Qw7r6Psd4QKXblywuxfW/ivt305hK8s3grRvRqh/eXJQvCWEzo7rmRH/Y3oRdKLbcHAQsQhskS+5tiOPjP03HXaYfijOqervcTQqChOY6SQnNTiB1KHy6EwNbaBsxdvwtz1+9KqvehbI5RuOmtpXhhrv0cD21U0v7mGIQQuH/majw4azXaFEew6OYT1O1lsgB57SspQmnF1r0Y1rOd1DYoYanS/0bZHKb4HhSenSO1Z1d9E3oUliASNjeobNjlPaT3zYWb0b9zOe5+byVe+2oTPrhqrLpN6+P5bmc9nv9Sf1+MIcSxuMDx932sagSF4VCS8ACgEx4A8PQX0qKAZsIDAPY2RjH2no8sr2HPfmvtLiiyZsIiop5E9CERLSOipUT0O7m8kohmENFq+X97uZyI6EEiWkNEi4hoRLbazjBBsH2v5AB9+AP3jmoAePyTdTj4z9PV/bV8uHK7LsTTiBoi63COX/x7nu77fBMhM33JVt13rT2+vjGKjbv248FZqwEAewzmmhKDH0Bnrxf6//uapH2LIubdleJjCGomds3eRvzuxYW48GkpWmpdzT688fUm9RxGgaQ1mwFSx64lFhc6c5KVoAuay174Ku3nyKYPJArgKiHEIACjAFxKRIMAXAtglhBiAIBZ8ncAmAhggPx3EYB/ZL7JDJN9lHDbbbX6Eeu2PQ34xb/m4bLnv8aYuz7AiNtm6LZ/vWE35ss2dK+dbZPJnIep7+o7Tq0AWVNTZzuz2mj4UXYVQuC2d5bLZVLhHjlSy1KACL3GkirKvVHmtQDAve9Lc2QemLkKX6zbqatvFBhOuAhSC4Td9dYDiaDImglLCLEFwBb5814iWg6gB4DJAMbK1Z4G8BGAa+TyZ4Q0VJlDRO2IqJt8HIZpsfgxsQAJU49Cgzz34Nsd+0wdqNe/sUT9vOC73bjomfmuz6WEj2oxRhtFNULmo5U1+GileVjpmu11WLnN6HxOzJX43zffa0oS2kuRxvegjUaKBayBmHXwm3bvlxzca3cmb/R8fEJhJOQYqZYqt/10SFqPD+SID4SIegMYDuBLAF00QmErgC7y5x4ANmp22ySXsQBhWjxrttehf+fytBz7l/+eh9LCsM5HYQzPdULbYSsYTTFu++/j70sOUb32tcWYfsXRujYqGoUSraR1Tjc0JdcLSoBYHeWSZxdgUPc2qR9f2JwkIP5xzghMPKRbek+CHAjjJaJyAK8BuEIIoQvzkLUNT7eaiC4iovlENL+mJvMTaxjGDzvrGp0rGaAkQ1Ay8bjAByu24+1FW3Qagh1mKToaTDSQQhczzt2yensdVm3bqxMCikWqyUR4abUvpbmpmrDmrd+F4+79SA23NVLXGEXYY9oRM6Yv3WpqEgyS8uLM6AZZFSBEVABJeDwnhHhdLt5GRN3k7d0AKOEgmwFoQ1Wq5DIdQojHhRDVQojqTp06pa/xDBMgjQGYM8z6T21n2GwT8qll2C3vJ5WZaSBGE1aqlBdFdG0UAB79eC1u/t8yAHoBob3WoDSQ299ehrU1+2zDld2kackmg7q10f1PN9mMwiIA/wSwXAhxn2bTWwAukD9fAOBNTfn5cjTWKAC17P9g8gU/AsToA1HQDpK1na4y0Q8Awjae3H0ms8PN+uagO9NQiHRtFEJg6rsrNN+17dFqIN6c6P/9erNpSLKyt91xZq/eYVp+RnWVq3NrufiYvnji/GrP+507qldSFJvCxCFdsX7qJHQoL/J8XD9kUwM5CsB5AH5MRAvlvxMBTAUwjohWAzhe/g4A0wCsA7AGwBMAfpOFNjNMWmhoTu60rXAyXVl1tNrRvZ0AMTLwxndNy+es22U7k9wMu5TxQghdJJdRY9IKWe2WhAbirg1XvLQQ172+OLltKURzaTt0q87dyNUnHIRxg7rg3tOHejrX+EFdsfy2CabbOlZkRnAoZE2ACCE+FUKQEOJQIcQw+W+aEGKnEOI4IcQAIcTxQohdcn0hhLhUCNFPCHGIEMJ9CAnD5DhOGsjcb3dh3H0fo6E5Zql5mDHmrkRac62Jp8CDALGLFvrzm0vx+CfW+bG07GuMou/10yy3C2Fvhvp6w241+aK2jw8qCks5ph/3xE7NWh7PXjhSt61DWSFmm8yEV4T4zw6rwlB5EqUbFNPhTT8ZlLQtXYEYVmTdic4wrYXpS7bibYuU6UYfw4LvdqO+KTG6v/XtpVi9vQ6rNTmSrDQRrQlLm6xQO4M6qMlsr321CXdOs86PpWV3vf2CSUKYO/AV4iKRO0urQQQ1D0TZ/T9zvvO8r+7eGnJhLfjTOF0qFAD4zdh+3hsoU9W+BAAwun9HXfkDU4ah+oD2vo/rBxYgDJMhLnl2AX77/Nem27RRTjV7G/Gzf3yOq17+JqmeVvvwookA+lFyQRacwU79e3M8jl89bW9YeGW+NCNcmzI9rmoOzvfDaqXCa19bhGWy8/wbkzVKnNCe2so39PdzRmDikK4AgIuO7qvbNqpvJQAkaSIDNBpFh7JCzLnuOFUYGYMYfjq8h+fFqVIlJ+aBMExrRNuhan0giuaxRJN9VqttuAnfdcI4Sk6Vz9eYO5e1OGkIq7fVmTrwtdw/cxUmHtJVV6aasGyOf/+MVThleA/84ZWEUBZCqB3ui/M2Wu1qS3lRBHWNUd25raLTTjykG060mJvxx/EH4rQRVehZWYqD/jQdANCutECnKYZChK5ti9XvuRARxgKEYbKEWXe3tbYBf5PzR5nuE9AENC9OdDec/eSXjnWcwohrXM6FudCgpSiOeTsH/d9mrca7S7agfWmhWtYYjVuuzWHHleMGoke7EuxvjuFnI6ogIHDlSwnB5GZBLSORcAgDulQAkDSV3zyXnMfKeFTtIOBmE39IJmABwjAZZvueBgiYj8gvff4r3ZoPClrLhFfTlRkBKyCu0IbomlGzJzk5pBnG1C8xl/NAYnGhy+hbu7/ZlwAZPaAjRvTS+xq0Gkiqwlm7u/ZIVtapzhVF+PlRfVI6p1/YB8IwGWbknbNwxJ2zTFeMM1viVYtdF+lFrOx3MBWlA7u1KwApLbsfFMFxz/v26VkKwiGdecmYsj4VtNpPKGU/hMZcqTmUMeV/mxJp/H/WyF4pns8/rIEwTJYwM0dZRSHpRqIOPhA33deOOn+ddSo4pe/wm91DuY/z1idrblqKCsK64AGrlCVOHNw1eZa3ooE8dt5hvo6pRdFACHoBcuW4gbp6pYURrL5joi+TWVCwBsIwWcLM4qIdpW/cJWWA1WK1zvX0JVuSstjmGs5RUubbe1WW4si+HVI4rkSRQQNRTIhuc4QBUgit2UJeShuKC8LqyotlhWF89adxro+toI2kUgYLL/xqlGmEVUE4lPHIKy2sgTBMljDzZRgnFJ77zy+xfuokawO4zCXPpn/xoHRj5SI5a2QvVLUvSVqHQ+HCZ+arqxnaURgJoVAnQKSOf/itM2z2cociQMJE6NamGGeN7IlzRx2AyrJChz2T0flA5M9+Vp/MBCxAGCZLmHWYdhPpAP0YXVFGjJMQsx/caY5TBJlVmO/g7m1sV1kEgIUu5m5s2l2POk3qlbgQqGuIeloQyuoaFAESCknhtn859VDXxzRCFk70XIRNWAyTJcw0ECcfiFkHdvNbSwNsVfpwmgditvXALhUY3b9jAI5pYP3Oep2gEUK4imgrLnDuJpVrCyLdu2KS0pqmrEyX2YYFCMNkCV3SQ3kEaylATPolpWzRplpdeW52Nc4CxGz778cNRCgUxNTJZNbV7MOqbdbL7iq0KS5QP1t15GVFkjGnyEdYsBGdsHS5hn22YBMWw2QJbV9074xV2FXf5GLNjlztSlzg49Lalkiddzocxa9/vRmvf520pFASN5882HRin5a7TxuKVxdswtCqtim3S3ulZbLvIwgNLB2wBsIwWcI44v7XZ+st65p1H8ruxkFxbnY1zkvemmkgylyHbPafE4d0xZzrjgNgvdJfp4oi/Hpsv0AEnVZY3D9lGC4/bkAggikdsABhmCzhJ3usEInOdPIjn+GH+qYWo5PY5aqStieXKeajbI7AiQhd2hThuokH4Z8XHJ6B8yU+d2lTjCvHDcxqqK4dbMJimCzhpePXO1QT5Rt37Q+uQWnG0QdioqJ0byelLs9290lEuPgY/ynYvZ1L/p+Rs6UGayAMkyX8RNa0FG3DDKfrNU4I7N62WM0rlY3cXdkiV/0dZrSin4Vhcgsv8kPpUuZ+uwuLNxuirnI0xNOI04TvqEGA/Pkng9XP6YnDsuaRs0dk9HxaWo74YBMWw2SE9Tv2JZX5WYH17vfsEwbmMk4pR7SLRAGGCXVp7lWJ9AI9RMD9U4Zif5PPBF0pEMpibiuvsABhGA8oo30vTs1d+5ow9p6PLI8VNH4drmMP7ISPVtYE3JoEXq9Xn8o8vZ1qmAhRTfuICKcMr0rrOa1oOeKDTVgM44kxd32Iw++Y6WmfvRZpOPxoIEbMZlLnqknLKQrLiFZopHtQblzDI5tKQGImevba4BbWQBjGA5t2Bxf15GVhKLvOJCh5kW654zZrroKXFPapYhQg2QybbQmCQ4E1EIbJEkF02ARKEkR+O7906y1er1d7Ga1JA0lEYeW+JGEBwjBZwlsUVuqdSc/KEtvt6TZ9eZ04SXoVxDd9OpY51kkWIFlcpEk9dW6aIrWwAGGYLOFnJnoqlBVm12Lt3YSl9YGk34muO3c2fSAtQPNQYAHCMGnGqkPwJEAy4ANJN6kIzHR3qbmkgSROnfuChAUIw2SJoPr9FiI/vEedaX0gKTgl3Jjmkp3ovk+XMuxEZxhGxSrayovPwapP8RLJ5XS63I7CSi9GjSO7PpCWE8bLAoRh0oAQAlE5d4dVx+nJgmXRmZgd2k+/8+T51T728obniYSai053WG0knHsaSEswTbIAYZg08NRn69H/hnexs67R0nQTxETCuBBJHbPVYe20lSIXy7amSkoaSMbDeLOvgbQEWIAwTABs39OArbUN6vfXFmwCAGypbbAceXszYZl3KtK63qkTouT5JEHjVWDq54Gk4ANxUccYhZULYbwtQY7wTHSGCYCRd84CAKyfOilpW3o1kOQyS3+JzfmMyQTTged5IJoryXQUVnY77xYgOWRYA2GYNKDtKq06Tk8aiJUPJC4CCcPKxIg7lYmEaZ8HklMz0aX/LUGMsABhmDRjZfs3rn/hB7ND+DlqJgSI03ogRlL1gXjZxyhAstl95+rytWawAGGYNGM18PYyIreOwgrG7hTKQROWFl8CRP7v5rRJ4iMHNJCWAAsQhkkzVh1nc8xdh/rRyu22xzYexXrOiDWUCSe6Zy+69qP3XtWTVmVMZeL5bMHBUVgMw6hYCZBY3J1N5+f/mmeTDiXZl+LPhOVjJ494lx+aXFg+eip1PoWLO2K8/hYwBSMnYAHCMGlA6dTjQtgIkNTPY3ZsP2HDIaL0z0RPwYnuRwNpSUkJtShpW1qCItLiBAgRTSCilUS0hoiuzXZ7GEbL9CVbUVufWIEwGheWI2+3Gghg3ZmYCQU/vvmMRGGlMJEw3RpSLvXVudQWJ1rUPBAiCgN4BMA4AJsAzCOit4QQy7LbMoaRuOTZBRgzoKP6PR4Xlh1nIFFY8WRzix9nNVH6zTbew3i1qUz8n9ePZpXNTrwlaB4KLU0DGQlgjRBinRCiCcCLACZnuU0Mo0O77G3MVgMJIoxXJHWQdh3m59f+GBMGd00qz8w8EG/1dSasNLfPZTwDY6ClCZAeADZqvm+Sy1SI6CIimk9E82tqajLaOCs+WVWD0X/9AA3NsWw3JS/YuKsezUE4ENKEtq+LxZNzVWm3uWX26h2m5V46ZQGge7sS05UJQyGkXQXxPhPd/HM6MJoTc0GetAQfTksTII4IIR4XQlQLIao7deqU7eYAAG59exk27d6Pjbvqs92UFs/ufU0Yc9eHuOV/S7PaDieHtEJMCEvncRAmLHMfiMVx5WKz0XwmcmF5TqaY4kx0L9fjwR3FaGhpAmQzgJ6a71VyWU6j2MBb0gzTXGVfUxQA8MFy67kRmcCuL9Q6fKct3ooF3+02rRcNwG4iZTIRhjL745o9hZkJ4/W/opSfV0c5nZvTZnp54XyhRTnRAcwDMICI+kASHGcCODu7TXJGeThb0gzTXKUgLI15mlIwYdU1RvH5mh0Yb+ILcEssLkzSX0hoR8svzN1geYymWOomTS8+EKXYbCCTicGN54mEGrT3tKI4gr0NUcd9vFySURvkV9UdLUoDEUJEAfwWwHsAlgN4WQiRXVuGC5RHsyXNMM0m0VhcXYzp5fkb8aKmE1bMIE1R/wLkmlcX4aL/LMDamjrX+yzZXItvNv6gfrcbsbr9nRuaU7ebmM4DcdjHrHmZmAfy9BffeapvdRsriiLo16ksgBYlSEW4BU1LUoZamgYCIcQ0ANOy3Q4vJDQQFiAKQgg8OftbnDS0G7q11Tt1fzT1AzRG4/jmpvG4+tVFAIAzR/YCoBEgKWggm3ZLvig3o1iFkx76VPfdToCEQ4S4C/PUvkbp/F3aFGHbnkbXbdFi1gynztBMcQplIIzXK7p5IIZGlxWZd10hSpgX7Tri9qUF2K2Zr+N1kmMmaAndRYvSQFoqioMu1Qdi254GPPrxWs9Lg+Yia2vqcMe05bjixYVJ27bvbUTt/maTvRICxG0eKTOUzigWj2PXvia8MHeD6T1dsrkWlz73lWnEl51D2K2psk4WIDvqmtztYIKZCcuPcJU0kNx6rnTzQHzsY8eDZw1HYSTR/QXhjwqKdqUFAIBzRx2Q5ZY4wwLEBz/UN+GHevcvvfJipipAfvPcV5j67gqs2e7e9JKrKKM/u85u1ba9SWWKrTqVORQRVYAAN7yxGNe9vhhLv9+TVO/8p+bincVbdCsNKthF7TRG41ixNbntRhQBksq1mIUJN1qY95R6u/YlC+dcHO36SeeuFd4dy4sQDhGunnBgUr0xAzrhoz+MVb/nkhO9tDCC9VMn4dJj+2e7KY6wAPHBsFtnYNitM1zXd6NSu2GPPCrPnUfdPbNX12BvQ6LjUj5XFBdY7nPuk18mlQXxoiumxDe+3oR9TZIje/veZCGxa580SDBzlmtNHsYO3I3wABImrFQQIlkIO82RsfKBBM3NPxmEMw/v6VzRArsmWW3SaiCFkRDW3nkiJg/rYVpX+7sqQvznP+oNAOjXudxTW1srLEDSyI66RtTsbQxsdKNG0QRytPQSiwv1pfxu5z6c98+5uPG/S9Tte/ZLnWdFsbUbzmxgHoSpIRKW7uALczeiUjYXKKPyeFzg5reWove176j1zTSE+et3qZ//+em3vtpR1xhDYTi1V3DJ97VJJjCnKCwzH4k0DyRYjuzXEUf26+B7f38p3D0cX1NXeUfHD+6C9VMnoY3NwIZJwAIkjVTfPhOH3zFT7QhTFSRBmcIywRF3zkL17ZKWtllO7aE1Be2UR/dtbASI2XXa3cNnvliP3te+g937mmxn/Yc1ucHblkgdhaLdjbh9Bv79+XrHc740L5EQ4fZ3lluey466xmadHd4Pz3iMbALMBWKQa6JXFEdw/5ShOLBrBSJ+8rBr2qRg1bYbTjwYz114RGIfjdBxmkiom/Ap35OCFAV6a4PvVgZQOv5UX1Bl/1yZkDh7dQ2O/Mss7G9K7qx31DVid30z4nGBmjopwqhDeaG6fdc+qay4IAxAukdunLh2s7ef/1IK9739neU46E/TsaPOPLIpohmmvvaVNA9VMfv8UJ/sHzDrcGet2I4NO+stJwm6oa4haipAUtVKrBDqQCZ5W1GKgkxLm+ICnDK8CoC5+S9IDu9TiaP6J5JXDuiSMD05PU5hzXs0eoCUtaJb2+JgG5jnsADJAKrpIEUJouyfG+IDuP3t5dhS24D1O/dZ1jn/qbn4nRxp1aGsCLX7m1Fb36z6F5TO+Zi7P8IRd87S7au9zr/NXA0hhK3DWbm9/1v0PQDg+x/2m9bTjjwVR/Zf3l2Bmr3mAsd4zh7tpLDjpz77Fl9+u9OyPU7s1twHXfvS/FaaPYepakJWRAISIFa/unHQcecph+D2nw5JqnfDiQer/g0F7X244vgB+OK6H6OqfWmqTW1VsADJAEoHlKqFwG4msRuisbjtKH/jrnr8dfoK1+GcUTkUSekkfqhvwmsLNunqfLomkQSwQ3khht7yPobe+j52ynZ7ZeS/YVc9tu9txEqNA3q7pkO/f+YqrNy2V9eZ76xr1GkZSoegTDJUtJVHP16Lv0xbjlhc4OEPVmPrHnPB8uZC86w4+w3msLOPkOak/Pvz9Xho1hrTfVIh3fOFzIRwYTiUluCMcNj/tdjeBsPGElmTLSuK4JiBkjahfYx/dXRf3HzyYN0+2gCOSIiS5iMxzrS4iYQtkXjQJiwf+9bWN2Pore/jmgkH4ddj+5nWufT5r7BoUy0mD+uOg7q2cTym0g8p8youf3EhPllVg+G92pnWLytMPG7KJL6mqP6mnPDAJ5bnW7SpFre8lUg8cNjtMwEA66dOktujP5bSUU59dwUAoH/nctzz/irL4+9tiKJvpzKsq9FrVPsa9QJEO6o2CpcgCKdZgJhpIJFwyNMDaucz0TY/FQ1E58/QnIyIUF4kCQzFRFZcEML+5pgnJ3phJIRubYuxpbahRWS+zUVYA8kEahhvik50+HeiK36IV+ZvtKyjjNzdZiZVOmilw9ssz/C2MtVpnZpKHS9p2a9+dZEadmt+fD3GY/9RntVut7/Z6Ly+SR9um24NwTjr2i8FhtG/cv+DiAr8w/jkuRVmpCIMtbt2qijSbbt/yjD8YfxAHNKjLYCEBuJVECi+nyin4/UFC5AMoMwZSPW1VZ5xP++/0icFOWFKFSAhwv6mGNbKI/ewhRFfm/tJaUUQ63oIIVDfFE26L1YT6qz4fM0OfLczOeX+/z09X/c93TEMQfmdiyJh03KrvtLuyfjmpvG670r0mulxNAdKxYmuvc9FkTA+veZY9XvnimL89scDVHOuEozRFIuroeGjNc51K5R7FERestYIC5AMoHTan6yqwcerarCzrhE7LSKE7BApCCJl1BxEzrimaBwn/m02NstOaiGA0x793HG/+2YkzEfKtby7ZCvOePSLlNrz4ryNGPTn99RwYYUGG23FjPmMqw1zAAAgAElEQVQuI6qC1ECG9Uw29/ntdHu0K8HIPpXqd+NhlI7dKu+TnVBoW1KAERrTpF2kWEeNthBJIaLMizahCJCG5hjalRbikz8ei9tMnOlGlPtVbpFbi7GHBYgLxt33MY6/72Pf+yvv6+3vLMcFT83FYbfPVO33dnyxdic+X5twQqcSzZUQIKlLkJF3zsSyLYnUH2tq9upSgbhJzaFtxlzNpDw/zFq+DUDyjOyHP1yDLbXmDnM7+nUqw6RDu1luL/ARsWQ0Jym8/usfJZXZdeR29O5YqstS/NPh5jOwld/HKAQemDIMvzyqT1L9f//icKmtvzkKp8jHjITJchLoE+cfpn5OyQfiYddbJg/GgV0q0F+eQd6rQ6mryLI//2QQ3r5sNHp14OgrP7AAccHq7XUp5Z/y22ef9cQcnP1EIp2HlwVyjCgvo5t9nSZgGedK1BtG+m6EVLC5h8x7mqXf78GRf/nA89EqywpxRrV1Co6iSAjXTDjI8Tj/OGeE+tlKpoZChAtHJzrt/p3LXY+Gv7z+OHx+7Y+x7NYT8Oi5h+GRs0dgoZxy/vHzDsOfTxqk1j1rZC88eUG13BapMfeeMVR3vA7lRfjFUb2TzqMVaFGN2dJM0FW1L0HnisRcCi/alFHYGPdUJiW2MTnv4b0r8d7vj1Y1EbcUhEMYIvtRGO+w3pYBgktlokRz+dBAQs4aiGJP9pouxChA3OwfpPhQ0rMHRVNM4OCuFZbbiwvC+PXYfvjr9BW2x+mjWbPC7r7feNIgrNuxDx+s2I47fjoE3dqW4J+frsOpI6ow+ZHPLPfr0ibRUU8Yol8ca9ygLrpw77+ceoj6WdFA7NLIaNEeR1mnJRIKmZryHpgyTPc94iGMNxIm3URR4+G7ti3GLScPxvjBXVwfk0kvrIFkAKvOY29Dsy7BoPNxpP9+Ol9F6GypbbBN8wF4X6vbOBPdjQkryPV73CYvdEs0FkfnNsVJnaGC21nb2gikR889zKYmcOvkwbjkmH6o7l2JXh1KccvkIRjasx1OHWFuhnJC6fT/eMKBuPzH+qyuyvPox9eiPBtawfDQWcPVz9W9K3X1vURhuUl7csGPevN8jRyCNZAMYNVZHnLz+wAS8xicSMWE9cXaxIzptTV1GNw9WW1XXvWoTWRUnUkG2X2GMFdXi/PkQPrsSIhMhaWiQfVob95RGc0k5UURTLt8DL7euBvRmMBVr3wDINFB96osxQmDu2LC4K6YvnSrWvbu78aox6hqX4prJyabxW6bPAQ/Gdodv/jXPF35yxcfadq2T685VhdlZZYSXNkeJsK0y8eYCvyq9iWoLCvEok21OlNSQgMhVUOwCyrwIqTCIcKsq47Bcfcq/kaem5HrsABpUfiL499a26CbA2GlISj9gJ0GMfSW95PKvtuhNyHFXMTUZ2sF0RMGd8F7SyWn+zc3jYcAMOSm93R1lNBiq77PqIF0b1eMXh1KVUfsp2t2IByipI5V+7tFQmS5qp6WsqIIjj2wc1K5NtpKi5tUHIqAJyIM6m49YfSXR/XBFS8txAEaB7PWB6JcnV2glZm/woqCMKFfp0QuqxxJ+cbYwCYsDzRGYxiXQjSWHUu/r8WSzbW2dfyuK2KcLe1komq22W4mXF4yTE505wNJvwTRLhik8JuxiRF5WVEE5UURLLp5PC46uq9afvfpknPZamRt1ECMv8f9U4bhntP1DmpALzTNFjnKFPG4OxPWT4f3wPqpk9CuNJEEU5u1VjGT2WkgHcuL8MT51UnlvztuAKZdPkZXZjRhpUN+zLzyGEy/YoxzRcYVrIF4YGttA1anaTXASQ9Ka27bmbMUP4ZXDcQY3RKLC9TWN6OiOKKb9az0A1YmLLcLIH2wcrtjnUxM/O3dsQwDOpdj9fY69OtUhlOG90BJYXKUTpviAlxx/AAcWtUWYw/srEZBaTvYg7pWqL6W5BnebpFqPnF+NcYNyp4jOOED8b6vMvgIkVYDse/qh1Ylm0sHdqnQaT+lhWHcMOlgXZ10ZJ3uzwtFBQprIBYs37IHl73wta4zzfZaAX7H7MbUGJ+t2YGht76PB2aa54Xa09CMxZtq8fAHq/HhioQwMPN/mPHYx+sc66R7CVFl1Kv0QTeeNAi//fEAFFvMzi4tjOCkQ7vrQmi1I2v7mdfurkXNHZZl04yiIJp10F3aFKNfpzLLSXgxrRNd8YE4XZDJZuOpl906AT8Z2t1pNybHcK2BENFoAAOEEP8iok4AyoUQ36avadnlihcXYuW2vbj02ETiwVTWrg6CxHoO3tphXIHugZmrAQDvL9uGK8cfiFhc4M9vLlHnuvz+pW909R899zBc8uyCJJNDKtithR4EykJVyjhZyZVUXOh+EKAdWT901nCMlNPNG2+/m6SCQOJ3S2Um+7TLx6QufBUNxKQdhZEQZl011nJXRcCWFITVDt7perxebzhEiMUF+0BaAK7eJiK6CcA1AK6TiwoAPJuuRuUCSucx4YHZalm6R81O+M3qa1Vf0aiWb9mD577cYJkP6PFP1gIAFm/+wduJbXCbA2vFbRNw+mFVro+rdDrKLGTlu+K3aK+x5zuh7fg6tynGh38Yi3OO6IWDu0mmF8WZbvVzdCiXUnqcI6d/j6sjf9dNSGJQ9zYpT3x78KzhOG/UAb6Oc+8ZQ3HjpIMxWGN+cgrVVZ6/yrJCdCiT7r/dHorJlTPk5j5uh2OnADgZwD4AEEJ8D8B6plUeYGbXtXM+CyFc+wj8oobxetzPKqxW6WSV5VytUJyoyzTpSlKlyWWiw+KCMO4+fSjuPu1Q0+2XHtsPs69OJNk7WTaDGOcKKH6LgnAIZx7e0zKlvRajxbJPxzLcccoh6rOhhOFambDKiyJYP3USLj6mn65etleUPKCDZKLyMw+kY3kRLhzTF0SUcKKHgK//NA4LbjzedB/tvBM3l66kWGENJPdxa8JqEkIIIhIAQERlTju0dMxeLqMJKx4Xqv33X5+tx61vL8Pc649D5zbpWRbTrxPdyvSmdKo1Dokd25VK9v+nfay/bUVzTFjOwzDDKr1H7w5l6FmZCDO985RDcN3Eg9HVZmnSqT8zF0ZG3Jpm3P4ays+WbR9I0ISJ0L7MWrNT/SYhQnNMEaLWx/Mye53JLm4FyMtE9BiAdkT0KwC/BPBE+pqVfcyS3xnDU2NC4MxH5yAuhBrds3DjDxg/uGvSvkGgnN27CctKgEgjPbN1wLW8s2iLtxO6oDkaT0pdYUephQAxJswrK4ro5leMH9QFK7buRafyIuOujjiN0Lu2LUZhJIQ/nuAuJDcIH0guodwep/ukXG6H8kJsrdUPVt689KiktThSyeDLZBZXAkQIcQ8RjQOwB8CBAP4shJiR1pZlGbOXPEkDEULNJPuzEZKdftuehrS1KTET3aMGYiNANuysd4yu8rquhts2FYRCaIC7Yysr0Blxioy74viBOHfUAb60QqeOvrggjFW3T3R9PHVN+/yQH6qPwskk161tCW776RCMO7gLTnroU3VvABhqks6+IN9UtDzGUYAQURjATCHEsQDyWmhoMVOjjSOlWs3IXckvFYsLvLt4C3793FeBt0l1onvdz6KP3ranAUff/WFqjfJJLC5Q6CEiyirLqiJA/n7OCFPHfChEvk2KQa0MqJAI482vDtKNL+W8UQcAcCc8FQ0k21GPjDOOAkQIESOiOBG1FULYT5XOI8xW1TP6Hi574Wv18155FB8X+oWTgsTJhNUYjUEIacJjj/Ylaudq5TNZGqBT3CuxuHCVPE/BqGl0rijC9r2NqqnxxEOs1+/wS9Brk4s8M2Epl+HnPrnxgfAys7mPWx9IHYDFRDQDciQWAAghLk9Lq3IAs4VwjD6Qb3eot0KNwBIINlW5Dod5IMNvnaGmVj9v1AHqZLBcHMnFhNDd43//4nD8XJMw8I5TJJOHguLIB6Tfpqp9CbbvbXS1aJBfPMg3Vyg/W57ID5Wg71OBfMCmaO49t4wetz/96wD+BOATAAs0f3mLqygsTUeumLD8rNWxa1+TY50563aqk++sTqFdl0O7kqHbqK3TDqtKSv2dLoTQmwmN97tLRbHO9NS5olhdGa9daYFqDrJbWjVVAtdA5P/5Ij9SCUe227NbO+l3T2U9dSYzuHWiP01EhQAGykUrhRDuF7JogZhqIAYB0qzRSBThEhfCsxD57fP2/pLvf9iPMx+fo343Hr8pGtdpQ0bcCJDubYtxz+lD8cwX6x3rBoXWLGXsrM1GtaP6dgAA3DDpYPz7cymkOJ2dTLk8m71zhfcILjMSTvT86BiVq/DyuLu58gemDMP0JVtxoM2iXkxu4EqAENFYAE8DWA/pGehJRBcIIT5JX9OyixsNRNuRb9otrb0dF95NWJ9r1uowY71BOBiPf+vbS/HsnA2W+7uxYClO6kza57VC2tipmnWyxQVhNdnkvz5bb1kvKIoiYcz4/dGBmSRzJRdWLmD3u7UrLcSZI3tlsDWMX9z6QO4FMF4IsRIAiGgggBcA2C+z1oIxe8A/WKHPMqsdedWpTvTg7bZnP/ml7rvxFPPX707aR9t+Nz4QxZfgJEAKwqTTvFJBK6SNAtupHZmalDegS3Cj4Hx1ovvaN7hmMFnErQApUIQHAAghVhGR+5ViWiBmtvX/zNHPxDYTFm8t/B7raqzNSUEQFwJxOdmc3Uju3599i5iA7freCkWqBmJfLxIKoTlmvySuW3QmLMPtdmqHag5qQV3RQV0rsGhTrW1mXyuO6t8hae35XCEHFpdksoRbATKfiJ5EIoHiOQDmp6dJuUH7UueX3GxgH/T63GbsaWhG3+un4bqJB+HiY/qZjmjXbK/Dzf9bBgB45pcjHY9ZFHangRgXp0oFrdZhPK9TO4JITJhpbp08BGdU90Tvjt4zAT134ag0tCg1lHvvZ3GwlvS7Mda4DWH5NYBlAC6X/5bJZXlLXAAVDkuOBtmZemHVNintesIPYF///KfmOh5TcVpn8sXWanBGE5ZTO37+I2liWk8XS7jmCsUFYVT3Nl+KtiWiaH+enOgsOPIKtwIkAuBvQohThRCnAngQgPnUYBcQ0d1EtIKIFhHRG0TUTrPtOiJaQ0QriegETfkEuWwNEV3r99xuiYvcXY/gwVnSeh6K3yWIdioj/kza5+saEilUjKYop3ZMObwX1k+dhLYuNEUmPfzppEHo16kMA334iXL13WK84VaAzAKgzY9dAmBmCuedAWCIEOJQAKsgrzNCRIMAnAlgMIAJAP5ORGE5ncojACYCGATgLLlu2ogLEXgqi6BRBUgAfgBVgGQwj53dwDVfHM35zMg+lZh11VjTZYKZ1oHb7qJYCKEuBi5/9m07EEK8L4RQhp9zACgrBk0G8KIQolFe7XANgJHy3xohxDohRBOAF+W6aSMuROATydzw6oJNeGX+Rk/7BCHnlEvNpFNaa8Iy3uocl91MirSk4AfGGrcCZB8RjVC+EFE1gP0BteGXAN6VP/cAoO09N8llVuVpIy6kCKdMz4b9wyvf4I+vLvK2UwCCThnxByUz3RzHznaeL5PtGD0sOPILtwLkCgCvENFsIpoNSQP4rd0ORDSTiJaY/E3W1LkBQBTAc34vwOS8FxHRfCKaX1NT4/s48bhAiID7pwwLqmlpIwgZpwjKoExHbrQ3Y/TOyxcfqX5mDSTP4d83L7AVIER0OBF1FULMA3AQgJcANAOYDuBbu32FEMcLIYaY/L0pH/vnAE4CcI5ITOneDKCn5jBVcplVudl5HxdCVAshqjt16mTXRFviQkhLcPo+Qubw0ulbVVU67KAEyF9OPcTzPh3LE6vasQ+EYXIfJw3kMQBKpr8jAVwPyZm9G8Djfk9KRBMAXA3gZCFEvWbTWwDOJKIiIuoDYACAuQDmARhARH3knFxnynXTRlxInViu92PRWNyTkLOqq65vHdD1Ht67Ev07l9vW0aZzT/aB5PiNZ1KCf938wGkiYVgIsUv+PAXA40KI1wC8RkQLUzjvwwCKAMyQO645QohLhBBLiehlSPNMogAuFULEAICIfgvgPUjhw08JIZamcH5H1JneOf6o/7C/2ZOQIyJT54MiOILyPbjxHVW1L9ElgdS2iuVHfsK/a37hKECIKCJHTB0H4CIP+1oihLDMGS6EuAPAHSbl0wBM83tOrygmrFy3xTfH4p6EnFXNUMAaiJsQaCGAww5ojwXf7UY4ROiqSd/OHU1+w0ES+YGTCesFAB8T0ZuQoq5mAwAR9QeQ16sTZtuE5TYlfDQmPEWKWftAvDvRD+/d3nKbGyd6XAg8fPZwXDVuIA7sUoGyoggOkvN25brmx/iDf9X8wlaAyNrAVQD+DWC0xtkdAnBZepuWXWLqTPTsPPJuFxGMxoWnVfmsOmZFY3CaSHjv6UPVz8cMtA5SiISdRUBcCHRrW4LLjhvAI9JWBv/a+YGbNdHnmJSlZ9HvHELIEwmz1a+5TQsfi8e9pZB3iMJy6vYryzSRUjaaT1mhs4UzB1faZRjGAxlMXNGyiMclc062ooGcFplSaI6JpLXa7XDygThdrlZo2JmpigtcPFosQFotrHDmByxALFBMWG6f8yIPZiQ3XOAigy4gLRbllBX4+hMPUj9bvbhKuZPA1G61q+vGJOUnDTjTsmFTZX7BAsQCoUwkdPm8Z0tTOemhT7Fw4w+2dXq0K0WPdiW6ssnDuuu+u3Wia7eneslswmq9cJBEfsACxAKvUVi5PLAKUcI5rgiAAYZJfomZ6PbH0l5nqnnC3EaaMQyTm7AAsSAm58Jyq3Ln8sxp7YRIpZVGB3jCB+LfhHXqcG/5LVkDab3k8OvCeMD3ZMB8R1kPxO1znssvBGk0KUVAGB3gahiv03Vothvr3jdlGA7oUIYmmzXTfzWmD56YLaVRY/nBMC0bFiAWCNWE1fI1kBAlBKFiNjKan1QTloME0V6nWd3fHT/Adv9hPdtDycPJJqzWS+6+LYwX2IRlgWLCcmvmDzrliXbCXqpom6Z02UbB6DaVidsoLOkcyWVaweVp/grDMDkHCxALpDXRyXW0SNDhiQd1q8DVEw4M5FihUKJ9Sqcdtsx+6yQUNBqIj0uOaAVIPHn7tRMPQmVZIfp0LPN+cKblwCpIXsACxILmWByF4ZCHMN7gzj28Vzv071wemFmMdCYs6b/R/JSYB+J0LP1x3XDsgYmUJ2GN5DLTP8Ye2Blf/Wkcr7Odp+SwpZfxAQsQC5picRRGQh6c6MG9GW/85igURcKBDdIIiRdX0UCMwsn9PJDEZ7drxl878WD1s1YDYR9I64XngeQHLEAsaI4KFITdO9F31jXabh/Ws53nNgQlk7TBAEqfbelEdzyp1onuVFM+p0bX0Aodlh+tD9ZA8gsWIBZIGkjY9QOfy3MatFFYCR+IeRiv0/XqTFh+RpGaXdiJ3nphQZIfsACxoCkq+0ACOp6frjKo/pVIa8KS/ieF4CrlaU5lot2fxQfDtGxYgFggaSDuTVi5DBEw6RAp99VFR/cFAHRrW6yrU7u/GYAbs5T2uN7vjXYP1kBaLy3/rWIAnkhoiaKBZHNJ26C61xARLvtxf/xydG+UF0Vw1fiBurXIAel6lbp2eFt/XfqvlRNa38vPf9Tb/cGYvICd5/kFayAWNEXlKKxsChATCfLYeYd5Pg5BMllVFBeAiOQIL/2FNceV6CynY7mfB2LW/oKw9Mj17lCK84/s7dR0Jk/JB82eYQFiSXMsLnd2AT3oPsw1ZkvVnjC4K9qXFng6jlnKEeP72xxVZvW510D8jCYVAcIdSOuGf/78gAWICfG4UNcaz/SDrtUwurQpMq1jZma65eTBGN2/o0X95DJjUdStBkLmn53qKpQXSVZTq2tjGKblwD4QE5pi0mi8MBLKeJLEiuLET9K1TbFpHbMmdaoowrmjDsCna3aY7eF43mjcpQ/E4lhu1wbp1aEU908ZimMGdnZVn8lPWAHJD1gDMUEVIFkI49V24IdUtdUJlATJrQpRcid+UNcKdVvSEYwmrJg7AaKN0vIzKx0AThlehcqyQtf1mfyBTVf5BQsQE5SIpCBMWDdOOhg3TjrYuaKM9nRFkTBevGhUUh1zgUAIW/yabrSo5piQj5O8rbggcWC9BuJuVjpH6zJGWJDkByxATGhbUoD/XnoUJgzpmnLY4YVj+uLCMX1Nt7UzcYYbncsRk57ZTCCEiCwFhZuXVdVATKTT4b0rTY+l/WzWToZh8ht+600oCIcwrGc7dK4oTutIaeKQbkllxvP16ViGH/XrYFsHkNc9t2isGw0kGrN2oivCBYCF/pH6+uhMa4Ofl3yABYgDXgTI8786wrW56u3LRuPmkwcllRv74cJICM//Sm/GMmtSiMiyEze/BsM8EBsfSMwi0ZdWW2IBwriBn5L8ggVIgPyoX0eMHmAeSmtkSI+2KIqYrXnh/Iqp65prOm2y0UDMzHDGqhHjClMaFP+IcT/tHmbnVtKmVFWWWB6baZ2wDyQ/4DDegPESjWSGm92VOsYO3I8W0KdjGX4ytDtOP6wKgN5cpaDVQLQOcW1bzRz4p46owqkjqjy3iWGYlgELEAe8OtGtZli7jURyc7bE6oEEJUA4ZBOFZdakLvIck/8b3QfnjjpALY/GkhsatTRhJT6zE53xAisg+QELkIBxowUM72W9uJSbFB8hExXEzoluRnlRBOunTkoq796uBAd1rcCUw3vilv8tAwBETbQS6fTsA2G8wSls8gsWIAHj1I8+eNZwnDy0u+/9AY3c0JmTrMN4vVAYCWH6FUdjzfa9almfjmVYvb1OPk+ibofyxGTAXpWltsd9+eIjOX07o8KCJD9gu0PAOKcCsceNyUw5h3apWLOZ6KnQrlQSDn06luG+KcOSthdFQrpUKw+fPdz2eCP7VGJU3w62dZj8h8VGfsEaiANeB0pmE/GARGfvd72Nn42oglBG8HKdu047FL9/6Rv1vEHm7epYXoTZVx+Lrm2L1Qy6Wqral6jtaF9aoAochnEDC5L8gAVIQChmKacoLL99/L1nDE0cQ/4/pHtbtSxoDQQAepqapRTtJxEYELHy3jMMk9fwmx8QJx0qzSp36sOdt7ufByIMZcZ+PN0uh2ZN0kmG8QK7QPIDfvMDQun4rUbjqvXJ8Oa89usjdd/dvFiKENIKiBDp128//8gDkG6U8/VozxMFGZew4MgrsipAiOgqIhJE1FH+TkT0IBGtIaJFRDRCU/cCIlot/12QuTZ6q1dZVmi71rfxcO0NvgM353vk7BE47bAq9O9crpZpw3h7dyjFrZOHuGm2fwTQo10J7jtjKB491/syu0zrhtdGzw+yJkCIqCeA8QA2aIonAhgg/10E4B9y3UoANwE4AsBIADcRUfuMNtgBbcd/2mHWs6+NJiqjlcnNizWgSwXuOX2ozucRIko42W3aFjSnjuC1PRimtZJNDeR+AFdD34dOBvCMkJgDoB0RdQNwAoAZQohdQojdAGYAmJDxFtugNR/ZddjGbdo+v3vbYhzQwX4+hd1xhfrZIKTS4QvhASSTAuwDyQ+yIkCIaDKAzUKIbwybegDYqPm+SS6zKjc79kVENJ+I5tfU1ATW5oFdym2369OcW78dVk7yvh3L8Pl1x6G4wCzBojPa42bk3eQ5gYwPWG7kF2kL4yWimQC6mmy6AcD1kMxXgSOEeBzA4wBQXV2dcjfn1lbrVgMJ+g0KhwixuJBNWPpzKAtWFdhk2vUKjxwZhlFImwARQhxvVk5EhwDoA+AbudOtAvAVEY0EsBlAT031KrlsM4CxhvKPAm90CmjDc+06WcswXZ8dc0KAJJae7d2hDADw8Nkj8M6i79G/c4W/g2sY1K0NovE42hRLQumIvpUOezAMk+9kfCKhEGIxgM7KdyJaD6BaCLGDiN4C8FsiehGSw7xWCLGFiN4DcKfGcT4ewHWZbbf9dq2mYqa1KPsnzwNJTUmKhAhNkDSgqvaleOL8arVz71RRhJ8f1Sel4ytM+90Y9fPMK49xzH3FMHawJpsf5NpM9GkATgSwBkA9gF8AgBBiFxHdBmCeXO9WIcSuTDTIaxiv0z5Bhy8qkViKYBo3qEugxzdDGz7MMF7gJIr5RdYFiBCit+azAHCpRb2nADyVoWbZ8vZlo3HSQ5/qyqxW6jNi1EAUk9Dwnv6ikiOqAOEXk2k58DyQ/CDrAqQlUhRJBK+VFoZR3xTTm7A8ONE7tynGO5ePRr9O/kb1YXkhJxYgTEuAn9L8glOZONCrshT9OpXh5pMHm25XXgh9/23iA5H/m3X0g7u39R2+q2ggLD+YlgQ/r/kBayAOFBeEMeuqsY71dPNAbH0gwcIrATIMky1YAwkI7Togyiezvt1qvRC/KAIkZrFuOcPkIqyB5AcsQAJCr4EkO7aVPFVBKwyKCSvGy8UyDJNhWIAEhFkUlrljmzUQpvWivBIchZUfsADxgTD5bJbKJGRyd4PWQO467VAc1b+DOvucYRgmU7ATPSDMkimaaSBBT6Qa3qs9nrtwVKDHZBiGcQNrICli5npQZITZ+ugcNMUwgOB0znkBCxAfaPNAKS+CmWZhpmyw7ZdpzfDzn1+wAPFBcUE4KR+U2TwQszkaHL7IMGla5IzJOCxAUsRsRGUWxqvAKUcYhskXWICkiJktVw3jZQ2EYZg8hgVIiiiquFk6d3aYM4weHkDlFyxAfGL3HsTVxaP4bWEYM9gHkh+wAEmRxDLkCWERjytpS7SpTDLZKoZhmPTDAiRVTExYcSXvFd9dhmHyGO7i0oBiwjKbSMgwDJMvsABJEbMorJiZCUudcJiZdjFMLsMz0fMDFiBpoCAsSYkubYqz3BKGyS2CzgXHZBdOppgGDuhQhrtPOxTHHdwl201hGIZJGyxAUsRsHggAnF7dM/ONYRiGySBswkoRszBehmHs4bD2/IAFiE/+fs4InFFdhX6dvC3kxIKGac3w059fsADxyYAuFRKakEgAAAkHSURBVLjrtKE825xhmFYLC5AMwSo7wzD5BguQgHCriLDCwjBMvsBRWB5474qjfQuAtiUFADjBItO64cc/v2AB4oEDu1YklVmF8Rr5+zkj8NY333t2ujNMPsIm3fyABUiKqClKHOJLOrcpxoVj+maiSQzDMBmBfSAMwzCML1iApAir4gzjHvaB5BcsQFJkeK92AIA2JWwNZBi3cDbe/IB7vRS57adDcP6RvdGtbUm2m8IwOQ9nYsgvWANJkaJIGEN6tM12MxiGYTIOCxCGYRjGFyxAGIZhGF9kTYAQ0WVEtIKIlhLRXZry64hoDRGtJKITNOUT5LI1RHRtdlrNMEwQcPRifpAVJzoRHQtgMoChQohGIuoslw8CcCaAwQC6A5hJRAPl3R4BMA7AJgDziOgtIcSyzLeeYRi/lBSEAXBKn3whW1FYvwYwVQjRCABCiO1y+WQAL8rl3xLRGgAj5W1rhBDrAICIXpTrsgBhmBbEg2cNx/NzN2BIjzbZbgoTANkyYQ0EMIaIviSij4nocLm8B4CNmnqb5DKrcoZhWhBd2xbjynEDQayB5AVp00CIaCaAriabbpDPWwlgFIDDAbxMRIEkiiKiiwBcBAC9evUK4pAMwzCMCWkTIEKI4622EdGvAbwuhBAA5hJRHEBHAJsB9NRUrZLLYFNuPO/jAB4HgOrqanbVMQzDpIlsmbD+C+BYAJCd5IUAdgB4C8CZRFRERH0ADAAwF8A8AAOIqA8RFUJytL+VlZYzDMMwALLnRH8KwFNEtARAE4ALZG1kKRG9DMk5HgVwqRAiBgBE9FsA7wEIA3hKCLE0O01nGIZhAIBEHgdkV1dXi/nz52e7GQzDMC0KIloghKh2qscz0RmGYRhfsABhGIZhfMEChGEYhvFFXvtAiKgGwHcpHKIjpOiw1g7fhwR8LxLwvZDIx/twgBCik1OlvBYgqUJE8904kvIdvg8J+F4k4Hsh0ZrvA5uwGIZhGF+wAGEYhmF8wQLEnsez3YAcge9DAr4XCfheSLTa+8A+EIZhGMYXrIEwDMMwvmABYkJrWz6XiHoS0YdEtExeYvh3cnklEc0gotXy//ZyORHRg/L9WUREI7J7BcFCRGEi+pqI3pa/95HXrllDRC/JCT0hJ/18SS7/koh6Z7PdQUNE7YjoVXnp6eVEdGQrfiZ+L78bS4joBSIqbq3PhRYWIAaIKAxp+dyJAAYBOEteajefiQK4SggxCNIaLZfK13wtgFlCiAEAZsnfAeneDJD/LgLwj8w3Oa38DsByzfe/ArhfCNEfwG4A/yeX/x+A3XL5/XK9fOJvAKYLIQ4CMBTSPWl1zwQR9QBwOYBqIcQQSAldz0TrfS4SCCH4T/MH4EgA72m+Xwfgumy3K8P34E1I68+vBNBNLusGYKX8+TEAZ2nqq/Va+h+ktWZmAfgxgLcBEKRJYhHj8wEpO/SR8ueIXI+yfQ0B3Ye2AL41Xk8rfSaUFVEr5d/5bQAntMbnwvjHGkgyrXr5XFndHg7gSwBdhBBb5E1bAXSRP+fzPXoAwNUA4vL3DgB+EEJE5e/aa1Xvg7y9Vq6fD/QBUAPgX7I570kiKkMrfCaEEJsB3ANgA4AtkH7nBWidz4UOFiCMChGVA3gNwBVCiD3abUIaTuV1yB4RnQRguxBiQbbbkgNEAIwA8A8hxHAA+5AwVwFoHc8EAMh+nsmQhGp3AGUAJmS1UTkCC5Bk7JbVzVuIqACS8HhOCPG6XLyNiLrJ27sB2C6X5+s9OgrAyUS0HsCLkMxYfwPQjoiUxde016reB3l7WwA7M9ngNLIJwCYhxJfy91chCZTW9kwAwPEAvhVC1AghmgG8DulZaY3PhQ4WIMm0uuVziYgA/BPAciHEfZpNbwG4QP58ASTfiFJ+vhx5MwpArcas0WIRQlwnhKgSQvSG9Lt/IIQ4B8CHAE6Tqxnvg3J/TpPr58WIXAixFcBGIjpQLjoO0kqhreqZkNkAYBQRlcrvinIvWt1zkUS2nTC5+AfgRACrAKwFcEO225OB6x0NyRSxCMBC+e9ESHbbWQBWA5gJoFKuT5Ai1dYCWAwpOiXr1xHwPRkL4G35c18AcwGsAfAKgCK5vFj+vkbe3jfb7Q74HgwDMF9+Lv4LoH1rfSYA3AJgBYAlAP4DoKi1PhfaP56JzjAMw/iCTVgMwzCML1iAMAzDML5gAcIwDMP4ggUIwzAM4wsWIAzDMIwvWIAwjAVEFCOihZo/28zMRHQJEZ0fwHnXE1FHH/udQES3yBlz3021HQzjRMS5CsO0WvYLIYa5rSyEeDSdjXHBGEiT28YA+DTLbWFaAayBMIxHZA3hLiJaTERziai/XH4zEf1B/ny5vL7KIiJ6US6rJKL/ymVziOhQubwDEb0vrzfxJKRJecq5zpXPsZCIHpOXGzC2ZwoRLYSUcvwBAE8A+AUR5XUGBSb7sABhGGtKDCasKZpttUKIQwA8DKnTNnItgOFCiEMBXCKX3QLga7nsegDPyOU3AfhUCDEYwBsAegEAER0MYAqAo2RNKAbgHOOJhBAvQcqgvERu02L53CencvEM4wSbsBjGGjsT1gua//ebbF8E4Dki+i+kNCCAlDLmZwAghPhA1jzaADgawKly+TtEtFuufxyAwwDMk1IwoQSJ5IVGBgJYJ38uE0LsdXF9DJMSLEAYxh/C4rPCJEiC4ScAbiCiQ3ycgwA8LYS4zrYS0XwAHQFEiGgZgG6ySesyIcRsH+dlGFewCYth/DFF8/8L7QYiCgHoKYT4EMA1kNJ5lwOYDdkERURjAewQ0rornwA4Wy6fCClpISAlLTyNiDrL2yqJ6ABjQ4QQ1QDegbRmxV2QEoAOY+HBpBvWQBjGmhJ5JK8wXQihhPK2J6JFABoBnGXYLwzgWSJqC0mLeFAI8QMR3QzgKXm/eiRSft8C4AUiWgrgc0jpwyGEWEZENwJ4XxZKzQAuBfCdSVtHQHKi/wbAfSbbGSZwOBsvw3hEXnCqWgixI9ttYZhswiYshmEYxhesgTAMwzC+YA2EYRiG8QULEIZhGMYXLEAYhmEYX7AAYRiGYXzBAoRhGIbxBQsQhmEYxhf/D/kTFD2SS0AsAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from dqn_agent import Agent\n",
    "\n",
    "agent = Agent(state_size=8, action_size=4, seed=0)\n",
    "\n",
    "def train(n_episodes=2000, max_t=1000, eps_start=1.0, eps_end=0.01, eps_decay=0.995):\n",
    "    \"\"\"Deep Q-Learning.\n",
    "    \n",
    "    Params\n",
    "    ======\n",
    "        n_episodes (int): maximum number of training episodes\n",
    "        max_t (int): maximum number of timesteps per episode\n",
    "        eps_start (float): starting value of epsilon, for epsilon-greedy action selection\n",
    "        eps_end (float): minimum value of epsilon\n",
    "        eps_decay (float): multiplicative factor (per episode) for decreasing epsilon\n",
    "    \"\"\"\n",
    "    scores = []                        # list containing scores from each episode\n",
    "    scores_window = deque(maxlen=100)  # last 100 scores\n",
    "    eps = eps_start                    # initialize epsilon\n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        state = env.reset()\n",
    "        score = 0\n",
    "        for t in range(max_t):\n",
    "            action = agent.act(state, eps)\n",
    "            next_state, reward, done, _ = env.step(action)\n",
    "            agent.step(state, action, reward, next_state, done)\n",
    "            state = next_state\n",
    "            score += reward\n",
    "            if done:\n",
    "                break \n",
    "        scores_window.append(score)       # save most recent score\n",
    "        scores.append(score)              # save most recent score\n",
    "        eps = max(eps_end, eps_decay*eps) # decrease epsilon\n",
    "        print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)), end=\"\")\n",
    "        if i_episode % 100 == 0:\n",
    "            print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)))\n",
    "        if np.mean(scores_window)>=200.0:\n",
    "            print('\\nEnvironment solved in {:d} episodes!\\tAverage Score: {:.2f}'.format(i_episode-100, np.mean(scores_window)))\n",
    "            torch.save(agent.qnetwork_local.state_dict(), 'checkpoint_Dueling_DDQN.pth')\n",
    "            break\n",
    "    return scores\n",
    "\n",
    "scores = train()\n",
    "\n",
    "# plot the scores\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(len(scores)), scores)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Watch a Smart Agent!\n",
    "\n",
    "In the next code cell, you will load the trained weights from file to watch a smart agent!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD8CAYAAAB9y7/cAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAABgJJREFUeJzt3dtx2zgAQFFqJ1WkDpehNtKH+0gbLsN1pA3uh9cbr/PYyJRE4PKcmXxllMA0cAXBlHxa13UBoOevvQcAwG0IPECUwANECTxAlMADRAk8QJTAA0QJPECUwANEfdp7AMuyLKfTydtpAd5Z1/W05fF28ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQI/gHVdl3Vd9x4GEPNp7wEcyf9F/PXvT6fTPYYDxAn8DWzdjQs9cA0CfyW3OGJZ11XkgQ8T+Avd+6z87f8n9sAlBP4XRvyhp6Mb4BLuolm+38Xy9s/IZhjjR6zrujw/7z2K/bkGXMvhd/Azh7K6o/9Z4B4e7j+OPf0q8ke7Dmxz2MDPHPb3qqF/S/BeePLjEo5oQkpPWsB2h9vB1yNYvevGLvWF68AlDhX4etzfm/XoRsRcA67jNEL0TqfTzQcxwte5t9FD741d8F/rum5aEPkdvLB/N+uOHu6pdMyZDry4/5zQf3c+P256/NPTtsezrz/9AMBlmXO9pAPP7zkSefHw+cvHH3wW+dl8dOM343pJBt7O/c/NvkOBX7nVBwAuyzxrJRd4cf+42SYvvNrrQwBHXyuZNzpVP59lD67lZbae43OZkT47aoQx/E5iBz/qxZ3dLLsUmmZb1yOul+kDP9skmNGIE/cazufHbT9g5aoqa3mk9TL1EU1lQszC9f6RJ4jtRj7i2GKEr2naHfwIF++IRtqdMJ+jrdu918t0gT/aBBnV3hOXOVivL/a6HXmqIxqTZTwzvbx2t8vtjXJ3y8jueW2m2cGbLGObZUf/Gvlbvfv01v/+SKzJbe6xZqbYwZtI8xh15/b2jplbxv3h85fl4fOX5KsFu/PbuOU1HT7wJtKcjvh9e3p6XJ6/fd17GFcn6Pdz7es8bOBNqvkd/Xs44y5+pHeJHtU1r/uQZ/AmVcve5/OvRyfP377+cDxT3HH/CWtsfNf4Hg0XeBOva+/Qv3eLs/inp8dlOS//PqHszXo6tqGOaEzGYzjKS/97/rD1Z0crR7jG/N4wO3iT8XjusaP/3fHMjKwTLjHEL91elmWIQbCvW4X+fH68e9yvcT/8IGuTfW1aFAIPMK5NgR/qDB6A6xF4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeIEniAKIEHiBJ4gCiBB4gSeIAogQeI+rT3AP5x2nsAADV28ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFECDxAl8ABRAg8QJfAAUQIPECXwAFF/A5cfBSnoydRoAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from dqn_agent import Agent\n",
    "\n",
    "agent = Agent(state_size=8, action_size=4, seed=0)\n",
    "# load the weights from file\n",
    "agent.qnetwork_local.load_state_dict(torch.load('checkpoint_Dueling_DDQN.pth', map_location=lambda storage, loc: storage))\n",
    "\n",
    "for i in range(3):\n",
    "    state = env.reset()\n",
    "    img = plt.imshow(env.render(mode='rgb_array'))\n",
    "    for j in range(200):\n",
    "        action = agent.act(state)\n",
    "        img.set_data(env.render(mode='rgb_array')) \n",
    "        plt.axis('off')\n",
    "        display.display(plt.gcf())\n",
    "        display.clear_output(wait=True)\n",
    "        state, reward, done, _ = env.step(action)\n",
    "        if done:\n",
    "            break \n",
    "            \n",
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}