gopher_solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### [sample solution that works]\n",
    "\n",
    "# This tutorial is will bring you through your first deep reinforcement learning model\n",
    "\n",
    "\n",
    "* Seaquest game as an example\n",
    "* Training a simple lasagne neural network for Q_learning objective\n",
    "\n",
    "\n",
    "## About OpenAI Gym\n",
    "\n",
    "* Its a recently published platform that basicly allows you to train agents in a wide variety of environments with near-identical interface.\n",
    "* This is twice as awesome since now we don't need to write a new wrapper for every game\n",
    "* Go check it out!\n",
    "  * Blog post - https://openai.com/blog/openai-gym-beta/\n",
    "  * Github - https://github.com/openai/gym\n",
    "\n",
    "\n",
    "## New to Lasagne and AgentNet?\n",
    "* We only require surface level knowledge of theano and lasagne, so you can just learn them as you go.\n",
    "* Alternatively, you can find Lasagne tutorials here:\n",
    " * Official mnist example: http://lasagne.readthedocs.io/en/latest/user/tutorial.html\n",
    " * From scratch: https://github.com/ddtm/dl-course/tree/master/Seminar4\n",
    " * From theano: https://github.com/craffel/Lasagne-tutorial/blob/master/examples/tutorial.ipynb\n",
    "* This is pretty much the basic tutorial for AgentNet, so it's okay not to know it.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Experiment setup\n",
    "* Here we basically just load the game and check that it works"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: THEANO_FLAGS=\"device=gpu0\"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using gpu device 0: GeForce GTX TITAN X (CNMeM is disabled, cuDNN 5005)\n"
     ]
    }
   ],
   "source": [
    "%env THEANO_FLAGS=\"device=gpu0\"\n",
    "import theano\n",
    "theano.config.floatX = 'float32'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#global params.\n",
    "\n",
    "#game title. full list of games = http://yavar.naddaf.name/ale/list_of_current_games.html\n",
    "GAME=\"GopherDeterministic-v0\"\n",
    "\n",
    "#number of parallel agents and batch sequence length (frames)\n",
    "N_AGENTS = 50\n",
    "SEQ_LENGTH = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "#game image will be resized from (210,160) to your image_size. \n",
    "#You may want a bigger image for your homework assignment IF you want a larger NN\n",
    "IMAGE_W,IMAGE_H = IMAGE_SIZE =(105,80)\n",
    "from scipy.misc import imresize\n",
    "def preprocess(obs):\n",
    "    obs= imresize(obs,IMAGE_SIZE)\n",
    "    return obs.mean(-1)/255."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-12-07 21:05:20,127] Making new env: GopherDeterministic-v0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f11436d6150>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAM0AAAD/CAYAAABSDlLPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEfJJREFUeJzt3W+MnWWZx/Hvj8IApd2WWWr/0TBNpYBY0C5gLNKJWBUb\nQV4QRINpWOIL94+6m7iAhk1847IkG90XarK7arrEsqnoNhBgl1LZ0QZXATsw0pbClpGppdPS0gKF\nSluufXGe55zTYc609zlPz585v08ymec8z3Ouued0rl7Pv/u+FRGY2Yk7pdUNMOs0ThqzRE4as0RO\nGrNEThqzRE4as0SFJ42kayRtlfS8pNuKjm/WairyPo2kKcBzwArgD8ATwOciYkthP8SsxYquNFcA\nL0TEcEQcBv4D+EzBP8OspYpOmvnASNXrHdk6s0nj1ILjHfdYT5Kf27GOEREau67oSvMHYEHV6wWU\nqo1ZR1m+fHnNbUUnzZPA+ZL6JPUAnwXuL/hnmJ10/f39NbcVengWEUck/RXw38AU4Ae+cmaTTdHn\nNETEw8DDRcc1axd+IsAskZPGLJGTxiyRk8YskZPGLJGTxiyRk8YskZPGLJGTxiyRk8YskZPGLJGT\nxiyRk8YskZPGLJGTxiyRk8YskZPGLJGTxiyRk8YskZPGLJGTxiyRk8YskZPGLJGTxiyRk8YskZPG\nLFHhw9JaZ5oyZUp5+eKLLwagt7cXgKGhIQD27t3b/Ia1IVcas0ROGrNEPjwzAGbMmFFevvbaawF4\n6qmnALjkkksAeOyxx5rfsDbkSmOWyJXGgGMvBPT09AAwMlKac3jWrFktaVO7cqUxS+RKYwAcPXq0\nvHzo0CEALrzwQsCXmseqq9JIWiDpMUnPSvqdpC9n63slrZe0TdIjkmYW21yz1qu30hwG/iYiBiVN\nA56StB64BVgfEXdLug24PfuyNvfqq6+Wl9esWQPA9OnTARgeHm5Fk9pWXZUmInZFxGC2/AawBZgP\nXAesznZbDVxfRCPN2knD5zSS+oAPAr8GZkfEaLZpFJjdaHxrjogoL//+979vYUvaX0NXz7JDs58C\nX4mI16u3RelfIcZ9o1mbGxgYqLmt7qSRdBqlhLknItZlq0clzcm2zwV21xvfrJX6+/trbqv36pmA\nHwCbI+I7VZvuB1Zly6uAdWPfa9bp6j2nuRK4GXhG0qZs3R3AXcBaSbcCw8CNDbfQrM3UlTQRsZHa\nVWpF/c0xa39+jMYskZPGLJGTxiyRk8YskZPGLJGTxiyRk8YskZPGLJGTxiyRk8YskZPGLJGTxiyR\nk8YskZPGLJGTxiyRk8YskZPGLJGTxiyRk8YskZPGLJGTxiyRk8YskZPGLJGTxiyRk8YskZPGLJGT\nxiyRk8YskZPGLJGTxiyRk8YskZPGLJGTxixRo7M7T5G0SdID2eteSeslbZP0iKSZxTTTrH00Wmm+\nAmymMvX57cD6iFgMbMhem00qjUyJfi6wEvg3QNnq64DV2fJq4PqGWmfWhhqpNN8Gvga8U7VudkSM\nZsujwOwG4pu1pbqSRtKngd0RsYlKlTlGRASVwzazSaOuKdGBZcB1klYCZwB/IukeYFTSnIjYJWku\nsLuohpo108DAQM1tdVWaiPh6RCyIiIXATcDPI+ILwP3Aqmy3VcC6euKbtVp/f3/NbUXdp8kPw+4C\nPi5pG3B19tpsUqn38KwsIgaAgWx5H7Ci0Zhm7cxPBJglctKYJXLSmCVy0pglctKYJXLSmCVy0pgl\nctKYJXLSmCVy0pglctKYJXLSmCVy0pglctKYJXLSmCVy0pglctKYJXLSmCVquLuzNd+MGTMAmDVr\nFgBHjhwpb3vppZcAeOedd979RiuEK41ZIleaDrRy5UoAPvnJTwJw6NCh8rY777wTgD179jS/YV3C\nlcYskStNB5g6dSoA73//+4/5np/b9PT0lPf92Mc+BsDjjz8OwMjICAClUYKtCK40ZolcaTrAwoUL\nAbj55puBSoV55plnADjzzDPL+95www0AnH766QCsXbsWgLfeeqs5je0CrjRmiZw0Zol8eNYBNm/e\nDMDGjRuBysn+3r17Aejt7S3v+/LLLwPw8MMPAz4sOxlcacwSudJ0gPxy8dDQEACXX345AOeddx4A\np55a+Wd8+umnAdi/f38zm9hVXGnMErnSdJDR0dIcwPv27QMqj89Mnz69vM+OHTsAePvtt5vcuu7h\nSmOWyJWmg+TnKY8++igAX/rSl4DKuQ7Ar371q+Y3rMvUXWkkzZR0n6QtkjZL+pCkXknrJW2T9Iik\nmUU21qwdNHJ49s/AQxFxEXAJsBW4HVgfEYuBDdlrK0hEEBEcPHiQgwcPcuDAAQ4cOMD+/fvLX4cP\nH+bw4cOtbuqkVlfSSJoBXBURPwSIiCMRcQC4Dlid7bYauL6QVpq1kXrPaRYCeyT9CLgUeAr4KjA7\nIkazfUaB2Y030XKLFy8GKp3Q3njjDQD6+vrK+1xzzTUAPPTQQ8CxHdSsGPUenp0KLAW+FxFLgYOM\nORSL0h05d+KwjjQwMFBzW72VZgewIyKeyF7fB9wB7JI0JyJ2SZoL7K4zvlXJO6EtW7YMOPa+DICk\n8vJFF10EwPbt2wEYHBxsRhMnnf7+fn7xi1+Mu62uShMRu4ARSYuzVSuAZ4EHgFXZulXAunrim7Wz\nRu7T/DXwY0k9wP8BtwBTgLWSbgWGgRsbbqGVuzPnz5g9//zzAFxwwQVA5dwGYOfOnQCcffbZzWxi\nV6k7aSLiaeDycTatqL85Zu3Pj9GYJfJjNB0o72iWdxl4/fXXy9vysQHs5HGlMUvkStMB8sf88/GZ\nr7rqKqDSDXrRokXlfadNmwbAgw8+2MwmdhVXGrNErjQd4M033wQqnc+WL18OUH4wc968eeV988dm\n8tkDrHiuNGaJXGk6yJQpU4DKFbK88syZM6e8z2mnnQYc+2iNFcuVxiyRK80k4wpz8rnSmCVy0pgl\nctKYJXLSmCVy0pgl8tWzDrRt2zagMtj5a6+9Vt526aWXtqRN3cSVxiyRk8YskQ/POkD++EwuH685\nn/b8ueeeK297z3veA8AZZ5zRpNZ1H1cas0QtqTT5KJB2YvIpz/POZ/lj/3l35+op0fPp0/Op0bds\n2dK0dk4m733ve2tuc6UxS9SSSnP11VePuz4/Dr/iiisADxKRq/UQZl5pxtued1TL97E0s2fXHobc\nlcYsUUsqzZVXXjnu+lNOKeVwPlZx/noyyQfHyIddqr7KlXcgO97v3S4xJrPqGbPH6t5PxaxOLak0\nE12ZmKzyYZg2bdoEwPDwMFAZjxng4MGDAFx22WXAu8/p2iVGN6i+IjmWK41ZopZUmvyYuZvkd/XP\nPfdcAHbt2gXA3r17y/vkV7ry4+mxn1O7xOgGY5/CqOZKY5aoJZWmG+dOyc8lXnzxRaByV7/6ClX+\nWH9vby9Q+3yk1TG6nSuNWaKWVJq33nqrFT+2pfIrUvPnzwfgj3/8I3DssXM+X2Z+D2Xs59QuMbrB\nSblPI+kOSc9KGpK0RtLpknolrZe0TdIjkmbWG9+sXdWVNJL6gC8CSyNiCaW5Nm+iNC36+ohYDGxg\nzDTpZpNBvYdnrwGHgamSjgJTgZ2UpkXvz/ZZDfwP4yRO3nmqG+X9+asv8ea2bt0KVCabbfcYk1lf\nX1/NbfVOib4P+CfgJUrJsj8i1gOzI2I0220UqP2oqFmHqqvSSFoEfBXoAw4AP5F0c/U+ERGSxn0u\nvRtPLHP5A5L5pd/xth3vZmK7xJjMjhw5UnNbvYdnlwGPR8ReAEk/Az4M7JI0JyJ2SZoL7B7vzWvW\nrCkvL1myhCVLltTZDLPiDA0NMTQ0BFTuUY2n3qTZCtwp6UzgELAC+A1wEFgF/GP2fd14b/785z9f\n54/tfD09PcD4j93n2zolxmRT/R/4+eefz3e/+91x96sraSLiaUn/DjwJvAP8FvgXYDqwVtKtwDBw\nYz3xzdpZ3Tc3I+Ju4O4xq/dRqjpWQ97Za7wHAk902KV2idGt/BiNWSIPFthk+SAY4503nOgsZu0S\no1u50pglcqVpsvwcYrzutBN1fGrHGN3KlcYskStNixRx3tAuMbqNK41ZIleaFpkxY8akidFtXGnM\nEjlpzBK15PCs3jGC8z7rk0ER4yS3S4x2Vu/vN9EFksn9iZmdBC2pNBPN/TGRo0ePArBv377yuok6\nC1l3yUeQmTmzMp7Lm2++CcChQ4eAShfuCy+88Jj35tPML168GJh4rDdXGrNELak09Y7YmFca35Cz\n8eRVpXrglryCzJs3D4AXXngBqFxqz3tobtiwAYDzzjsPmPgIxpXGLJFvbtqkkVeajRs3ltflQ1WN\njIwAlYFEvv/97wOwYMECAGbNmgVUjoI8E5pZgVxpbNKYOnUqAMuWLSuvGxwcBOCVV14B3j1jxZ49\newC45ZZbgMqVt4m6fLvSmCXqqEqTz9BlNp6zzjoLgJUrV5bXnXPOOUClW3d+Hyb/W9q+ffsx3/MY\nvnpmVqCWVJrR0dHj7zSO/H+H/H6NWbX87yKfgwcqd/7zq2H5Pb78+6JFi4BKZdm/fz8w8Wx9rjRm\niTwTmk0a+VPw+exu1U70GcV8v4mOZlxpzBI5acwSteTwLL/hZNaupk2bVnObK41ZopZUmm9+85ut\n+LFmJ+wb3/hGzW2uNGaJWlJp3EXZ2t1Eg7i40pglctKYJZowaST9UNKopKGqdb2S1kvaJukRSTOr\ntt0h6XlJWyV94mQ23KxVjldpfgRcM2bd7cD6iFgMbMheI+l9wGeB92Xv+Z4kVzKbdCb8o46IXwKv\njll9HbA6W14NXJ8tfwa4NyIOR8Qw8AJwRXFNNWsP9VSC2RGRP9s/CuQj/80DdlTttwOY30DbzNpS\nQ4dPUergMlF3Sne1tI40MDBQc1s9STMqaQ6ApLnA7mz9H4AFVfudm60z6zj9/f01t9WTNPcDq7Ll\nVcC6qvU3SeqRtBA4H/hNHfHN2tqETwRIuhfoB86RNAL8PXAXsFbSrcAwcCNARGyWtBbYDBwB/iI8\nEoZNQhMmTUR8rsamFTX2/xbwrUYbZdbOfB/FLJGTxiyRk8YskZPGLFFL+tMsXbqUnTt3lifaKUo3\nxzxZcbs15ty5c2tuU7OvCkvyZWjrGBHxrmn3mp40Zp3O5zRmiZw0ZomanjSSrsl6dj4v6bY6YyT1\nKD3BmAskPSbpWUm/k/TlguKeIenXkgYlbZb0D0XEzWJMkbRJ0gMFtXVY0jNZzN8UFHOmpPskbcl+\n/w8VEPOCrI351wFJXy7iMz0hEdG0L2AKpc5pfcBpwCBwUR1xrgI+CAxVrbsb+Lts+TbgrsSYc4AP\nZMvTgOeAixqNm71vavb9VOB/gY8UFPdvgR8D9xf0GbwI9I5Z12jM1cCfV/3+M4r43avinwK8TOkJ\n+8LiTvgzT0bQCX7BDwP/VfX6duD2OmP1jUmarZQ6yOUJsLXBtq6j9IxdYXGBqcATwMWNxqXU9eJR\n4KPAA0V8BlnS/OmYdXXHzBJk+zjri/xMPwH88mT8DdT6avbh2XxgpOp1kb07a/UoTSapj1Il+3UR\ncSWdImkwe/9jEfFsAXG/DXwNqB6gq9GYATwq6UlJXywg5kJgj6QfSfqtpH+VdFYB7ax2E3BvAW09\nYc1OmqZc347SfzV1/SxJ04CfAl+JiNeLiBsR70TEByhVh+WSPtpIXEmfBnZHxCbgXfcRGmjrlRHx\nQeBTwF9KuqrBmKcCS4HvRcRS4CDZQCwNthMAST3AtcBPxm5rJO7xNDtpxvbuXMCx4wo0olaP0hMm\n6TRKCXNPROSd6xqOm4uIA8CDwJ81GHcZcJ2kFyn9L3u1pHsabWtEvJx93wP8J6WBURqJuQPYERFP\nZK/vo5REuwr6TD8FPJW1lwbbesKanTRPAudL6sv+l/gspR6fRajVo/SEqDQJ4w+AzRHxnQLjnpNf\nxZF0JvBxYFMjcSPi6xGxICIWUjo8+XlEfKGRmJKmSpqeLZ9F6VxhqMF27gJGJC3OVq0AngUeqDfm\nGJ+jcmhGI21NcjJOlI5z4vYpSlemXgDuqDPGvcBO4G1K50i3AL2UToy3AY8AMxNjfoTS+cEgpT/q\nTZTGb2s07hLgt1ncZ4CvZesbilsVv5/K1bO6Y1I6/xjMvn6X/9sU8PtfSunix9PAzyhdHGj4dwfO\nAl4BpletK+QzPd6XH6MxS+QnAswSOWnMEjlpzBI5acwSOWnMEjlpzBI5acwSOWnMEv0/OgjGb/32\nkdYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f114360ca90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import gym\n",
    "\n",
    "#creating a game\n",
    "atari = gym.make(GAME)\n",
    "\n",
    "action_names = np.array(atari.get_action_meanings())\n",
    "\n",
    "obs = atari.step(0)[0]\n",
    "\n",
    "plt.imshow(preprocess(obs),interpolation='none',cmap='gray')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic agent setup\n",
    "Here we define a simple agent that maps game images into Qvalues using simple convolutional neural network.\n",
    "\n",
    "![scheme](https://s18.postimg.org/gbmsq6gmx/dqn_scheme.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import lasagne"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from lasagne.layers import *\n",
    "#image observation at current tick goes here, shape = (sample_i,x,y)\n",
    "observation_layer = InputLayer((None,IMAGE_W,IMAGE_H))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from agentnet.memory import WindowAugmentation,LSTMCell,RNNCell\n",
    "\n",
    "#store 4-tick window in order to perceive motion of objects\n",
    "\n",
    "prev_window = InputLayer((None,4,IMAGE_W,IMAGE_H))\n",
    "\n",
    "current_window = WindowAugmentation(observation_layer,prev_window)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from lasagne.layers import Conv2DLayer,Pool2DLayer,DenseLayer,batch_norm,dropout\n",
    "\n",
    "#main neural network body\n",
    "conv0 = Conv2DLayer(current_window,16,filter_size=(8,8),stride=(4,4),name='conv0')\n",
    "\n",
    "conv1 = Conv2DLayer(conv0,32,filter_size=(4,4),stride=(2,2),name='conv1')\n",
    "\n",
    "dense0 = DenseLayer(conv1,256,name='dense',nonlinearity = lasagne.nonlinearities.tanh)\n",
    "\n",
    "#please set this to your last layer for convenience\n",
    "last_layer = dense0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#a layer that predicts Qvalues\n",
    "qvalues_layer = DenseLayer(last_layer,\n",
    "                   num_units = atari.action_space.n,\n",
    "                   nonlinearity=lasagne.nonlinearities.linear,\n",
    "                   name=\"q-evaluator layer\")\n",
    "\n",
    "#To pick actions, we use an epsilon-greedy resolver (epsilon is a property)\n",
    "from agentnet.resolver import EpsilonGreedyResolver\n",
    "action_layer = EpsilonGreedyResolver(qvalues_layer,name=\"e-greedy action picker\")\n",
    "\n",
    "action_layer.epsilon.set_value(np.float32(0.1))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from agentnet.target_network import TargetNetwork\n",
    "targetnet = TargetNetwork(qvalues_layer,dense0)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "qvalues_old = targetnet.output_layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Finally, agent\n",
    "We declare that this network is and MDP agent with such and such inputs, states and outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from agentnet.agent import Agent\n",
    "#all together\n",
    "agent = Agent(observation_layers=observation_layer,\n",
    "              policy_estimators=(qvalues_layer,qvalues_old),\n",
    "              agent_states={current_window:prev_window},\n",
    "              action_layers=action_layer)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[conv0.W,\n",
       " conv0.b,\n",
       " conv1.W,\n",
       " conv1.b,\n",
       " dense.W,\n",
       " dense.b,\n",
       " q-evaluator layer.W,\n",
       " q-evaluator layer.b]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Since it's a single lasagne network, one can get it's weights, output, etc\n",
    "weights = lasagne.layers.get_all_params(action_layer,trainable=True)\n",
    "weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create and manage a pool of atari sessions to play with\n",
    "\n",
    "* To make training more stable, we shall have an entire batch of game sessions each happening independent of others\n",
    "* Why several parallel agents help training: http://arxiv.org/pdf/1602.01783v1.pdf\n",
    "* Alternative approach: store more sessions: https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-12-07 21:06:14,035] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,080] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,114] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,146] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,179] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,212] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,244] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,279] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,312] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,345] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,378] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,410] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,442] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,475] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,507] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,539] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,572] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,605] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,638] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,671] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,704] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,737] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,770] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,803] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,836] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,869] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,902] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,934] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:14,966] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:15,000] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:15,033] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:15,066] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:15,098] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:15,132] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:15,165] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:15,198] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:15,232] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:15,264] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:15,296] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:15,329] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:15,361] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:15,393] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:15,426] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:15,460] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:15,492] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:15,525] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:15,559] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:15,593] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:15,626] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:15,659] Making new env: GopherDeterministic-v0\n"
     ]
    }
   ],
   "source": [
    "from agentnet.experiments.openai_gym.pool import EnvPool\n",
    "\n",
    "pool = EnvPool(agent,GAME, N_AGENTS,max_size=1000,preprocess_observation=preprocess) #may need to adjust for speed\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['NOOP' 'NOOP' 'UP' 'UP' 'FIRE' 'UP' 'LEFT']\n",
      " ['NOOP' 'NOOP' 'UP' 'LEFTFIRE' 'UP' 'UP' 'LEFTFIRE']]\n",
      "[[ 0.  0.  0.  0.  0.  0.  0.]\n",
      " [ 0.  0.  0.  0.  0.  0.  0.]]\n",
      "CPU times: user 1.29 s, sys: 47 ms, total: 1.33 s\n",
      "Wall time: 1.33 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#interact for 7 ticks\n",
    "_,action_log,reward_log,_,_,_  = pool.interact(7)\n",
    "\n",
    "\n",
    "print(action_names[action_log][:2])\n",
    "print(reward_log[:2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#load first sessions (this function calls interact and remembers sessions)\n",
    "pool.update(SEQ_LENGTH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Q-learning\n",
    "* An agent has a method that produces symbolic environment interaction sessions\n",
    "* Such sessions are in sequences of observations, agent memory, actions, q-values,etc\n",
    "  * one has to pre-define maximum session length.\n",
    "\n",
    "* SessionPool also stores rewards (Q-learning objective)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#get agent's Qvalues obtained via experience replay\n",
    "replay = pool.experience_replay.sample_session_batch(100,replace=True)\n",
    "\n",
    "_,_,_,_,(qvalues_seq,old_qvalues_seq) = agent.get_sessions(\n",
    "    replay,\n",
    "    session_length=SEQ_LENGTH,\n",
    "    optimize_experience_replay=True,\n",
    ")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#get reference Qvalues according to Qlearning algorithm\n",
    "from agentnet.learning import qlearning\n",
    "\n",
    "#crop rewards to [-1,+1] to avoid explosion.\n",
    "import theano.tensor as T\n",
    "rewards = T.maximum(-1,T.minimum(replay.rewards,1))\n",
    "\n",
    "#loss for Qlearning = (Q(s,a) - (r+gamma*Q(s',a_max)))^2\n",
    "\n",
    "elwise_mse_loss = qlearning.get_elementwise_objective(qvalues_seq,\n",
    "                                                      replay.actions[0],\n",
    "                                                      rewards,\n",
    "                                                      replay.is_alive,\n",
    "                                                      Qvalues_target=old_qvalues_seq,\n",
    "                                                      gamma_or_gammas=0.99,)\n",
    "\n",
    "#compute mean over \"alive\" fragments\n",
    "loss = elwise_mse_loss.sum() / replay.is_alive.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Compute weight updates\n",
    "updates = lasagne.updates.adadelta(loss,weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#compile train function\n",
    "import theano\n",
    "train_step = theano.function([],loss,updates=updates)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Demo run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "action_layer.epsilon.set_value(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-12-07 21:06:55,004] Making new env: GopherDeterministic-v0\n",
      "[2016-12-07 21:06:55,052] Clearing 2 monitor files from previous run (because force=True was provided)\n",
      "[2016-12-07 21:07:59,659] Finished writing results. You can upload them to the scoreboard via gym.upload('/home/apanin/jheuristic/drqn/records')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 10001 timesteps with reward=0.0\n"
     ]
    }
   ],
   "source": [
    "untrained_reward = pool.evaluate(save_path=\"./records\",record_video=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<video width=\"640\" height=\"480\" controls>\n",
       "  <source src=\"records/openaigym.video.32.9829.video000000.mp4\" type=\"video/mp4\">\n",
       "</video>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "\n",
    "video_path=\"records/openaigym.video.32.9829.video000000.mp4\"\n",
    "\n",
    "HTML(\"\"\"\n",
    "<video width=\"640\" height=\"480\" controls>\n",
    "  <source src=\"{}\" type=\"video/mp4\">\n",
    "</video>\n",
    "\"\"\".format(video_path))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#starting epoch\n",
    "epoch_counter = 1\n",
    "\n",
    "#full game rewards\n",
    "rewards = {epoch_counter:untrained_reward}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=30570\tepsilon=0.052\treward/step=0.61800\n",
      "iter=30580\tepsilon=0.052\treward/step=0.66400\n",
      "iter=30590\tepsilon=0.052\treward/step=0.58800\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-12-09 05:26:12,554] Making new env: GopherDeterministic-v0\n",
      "[2016-12-09 05:26:12,614] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=30600\tepsilon=0.052\treward/step=0.55600\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-12-09 05:26:28,778] Finished writing results. You can upload them to the scoreboard via gym.upload('/home/apanin/jheuristic/drqn/records')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 1938 timesteps with reward=1820.0\n",
      "iter=30610\tepsilon=0.052\treward/step=0.64200\n",
      "iter=30620\tepsilon=0.052\treward/step=0.64000\n",
      "iter=30630\tepsilon=0.052\treward/step=0.57800\n",
      "iter=30640\tepsilon=0.052\treward/step=0.68200\n",
      "iter=30650\tepsilon=0.052\treward/step=0.79400\n",
      "iter=30660\tepsilon=0.052\treward/step=0.71000\n",
      "iter=30670\tepsilon=0.052\treward/step=0.59800\n",
      "iter=30680\tepsilon=0.052\treward/step=0.62400\n",
      "iter=30690\tepsilon=0.052\treward/step=0.67000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-12-09 05:32:38,198] Making new env: GopherDeterministic-v0\n",
      "[2016-12-09 05:32:38,254] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=30700\tepsilon=0.052\treward/step=0.62000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-12-09 05:32:44,270] Finished writing results. You can upload them to the scoreboard via gym.upload('/home/apanin/jheuristic/drqn/records')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 647 timesteps with reward=120.0\n",
      "iter=30710\tepsilon=0.052\treward/step=0.61400\n",
      "iter=30720\tepsilon=0.052\treward/step=0.63600\n",
      "iter=30730\tepsilon=0.052\treward/step=0.78400\n",
      "iter=30740\tepsilon=0.052\treward/step=0.78800\n",
      "iter=30750\tepsilon=0.052\treward/step=0.70400\n",
      "iter=30760\tepsilon=0.052\treward/step=0.79800\n",
      "iter=30770\tepsilon=0.052\treward/step=0.82800\n",
      "iter=30780\tepsilon=0.052\treward/step=0.75200\n",
      "iter=30790\tepsilon=0.052\treward/step=0.63200\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-12-09 05:38:47,367] Making new env: GopherDeterministic-v0\n",
      "[2016-12-09 05:38:47,423] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=30800\tepsilon=0.052\treward/step=0.66000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-12-09 05:38:57,816] Finished writing results. You can upload them to the scoreboard via gym.upload('/home/apanin/jheuristic/drqn/records')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 1112 timesteps with reward=1020.0\n",
      "iter=30810\tepsilon=0.052\treward/step=0.61800\n",
      "iter=30820\tepsilon=0.052\treward/step=0.51200\n",
      "iter=30830\tepsilon=0.052\treward/step=0.59600\n",
      "iter=30840\tepsilon=0.052\treward/step=0.68200\n",
      "iter=30850\tepsilon=0.052\treward/step=0.69000\n",
      "iter=30860\tepsilon=0.052\treward/step=0.66600\n",
      "iter=30870\tepsilon=0.052\treward/step=0.70800\n",
      "iter=30880\tepsilon=0.052\treward/step=0.71200\n",
      "iter=30890\tepsilon=0.052\treward/step=0.74000\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-12-09 05:45:03,201] Making new env: GopherDeterministic-v0\n",
      "[2016-12-09 05:45:03,253] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter=30900\tepsilon=0.052\treward/step=0.70800\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-12-09 05:45:15,301] Finished writing results. You can upload them to the scoreboard via gym.upload('/home/apanin/jheuristic/drqn/records')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 1306 timesteps with reward=960.0\n",
      "iter=30910\tepsilon=0.052\treward/step=0.65600\n",
      "iter=30920\tepsilon=0.052\treward/step=0.74600\n",
      "iter=30930\tepsilon=0.052\treward/step=0.76400\n",
      "iter=30940\tepsilon=0.052\treward/step=0.78000\n",
      "iter=30950\tepsilon=0.052\treward/step=0.75000\n"
     ]
    }
   ],
   "source": [
    "\n",
    "#the loop may take eons to finish.\n",
    "#consider interrupting early.\n",
    "for i in xrange(10**7):    \n",
    "    \n",
    "    \n",
    "    #train\n",
    "    pool.update(SEQ_LENGTH,append=True)\n",
    "    \n",
    "    for i in range(5): #you may want to increase the number of training iterations per one update\n",
    "        loss = train_step()\n",
    "        \n",
    "    targetnet.load_weights(0.01)\n",
    "    \n",
    "    ##update resolver's epsilon (chance of random action instead of optimal one)\n",
    "    current_epsilon = 0.05 + 0.95*np.exp(-epoch_counter/5000.)\n",
    "    action_layer.epsilon.set_value(np.float32(current_epsilon))\n",
    "    \n",
    "    if epoch_counter%10==0:\n",
    "        #average reward per game tick in current experience replay pool\n",
    "        pool_mean_reward = pool.experience_replay.rewards.get_value().mean()\n",
    "        print(\"iter=%i\\tepsilon=%.3f\\treward/step=%.5f\"%(epoch_counter,\n",
    "                                                         current_epsilon,\n",
    "                                                         pool_mean_reward))\n",
    "        \n",
    "    ##record current learning progress and show learning curves\n",
    "    if epoch_counter%100 ==0:\n",
    "        action_layer.epsilon.set_value(0)\n",
    "        rewards[epoch_counter] = pool.evaluate(record_video=False)\n",
    "        action_layer.epsilon.set_value(np.float32(current_epsilon))\n",
    "#         plt.title(\"random frames\")\n",
    "#         for i in range(min((len(pool.games),6))):\n",
    "#             plt.subplot(2,3,i+1)\n",
    "#             plt.imshow(pool.games[i].get_observation())\n",
    "#         plt.show()\n",
    "\n",
    "    \n",
    "    epoch_counter  +=1\n",
    "\n",
    "    \n",
    "# Time to drink some coffee!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating results\n",
    " * Here we plot learning curves and sample testimonials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/apanin/anaconda/lib/python2.7/site-packages/pandas/computation/__init__.py:19: UserWarning: The installed version of numexpr 2.4.4 is not supported in pandas and will be not be used\n",
      "\n",
      "  UserWarning)\n",
      "/home/apanin/anaconda/lib/python2.7/site-packages/ipykernel/__main__.py:3: FutureWarning: pd.ewm_mean is deprecated for ndarrays and will be removed in a future version\n",
      "  app.launch_new_instance()\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f1116dee890>]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEACAYAAAC+gnFaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXmcVdWV77+LUeYqBqGKYlJBwBEHMIqmNIrYcUo+idIv\nMbx0bLs/msS8JB0l6ddip1uNGUwnaftp4kDSiYmaqNhOoLGMcUJRRChKCmQqoAqkoAahCqpqvz/W\nPd5zb915vnXX9/Phc8/dZ7jrnkvt31lr7b22OOcwDMMwSpd++TbAMAzDyC8mBIZhGCWOCYFhGEaJ\nY0JgGIZR4pgQGIZhlDgmBIZhGCVOXCEQkSUisl5E3hOR34nIYBEZLSIrRWSjiKwQkbKw4+tFpE5E\nFmTXfMMwDCNdJNY8AhGZCvwZmOWc6xSRPwBPAycAHzrn7hSRm4By59zNIjIb+B1wJjAReB6Y4Zzr\nye7XMAzDMFIlnkfQChwBhorIAGAosAu4HFgWOGYZcGVg+wrgIefcEefcVmATMDfTRhuGYRiZI6YQ\nOOeagR8D21EBOOCcWwmMd841BQ5rAsYHtiuBBt8lGlDPwDAMwyhQYgqBiBwLfAOYinbyw0Xki/5j\nnMaWYtWpsBoWhmEYBcyAOPvPAF51zu0DEJE/AZ8AGkVkgnOuUUQqgD2B43cCk3znVwXaQhAREwfD\nMIwUcM5Jpq8ZL0dQB5wlIkNERIALgVrgSWBx4JjFwOOB7eXAIhEZJCLTgOnAqkgXds4V7b9bbrkl\n7zaY/fm3o9RsN/vz/y9bxPQInHPvisivgbeAHuBt4F5gBPCwiHwF2ApcFTi+VkQeRsWiC7jeZdN6\nwzAMI23ihYZwzt0J3BnW3Ix6B5GOvw24LX3TDMMwjFxgM4tToLq6Ot8mpIXZnz+K2XYw+/sqMSeU\nZe1DRSxiZBiGkSQigstDstgwDMPo45gQGIZhlDgmBIZhGCWOCYFhGEaJY0JgGIZR4pgQGIZRUtxz\nD3zta9DeDp2d+bamMDAhMAyjpNi9G/7zP2H6dPjFL/JtTWEQd2axYRhGX+LIEfjUp2DsWKitzbc1\nhYF5BIZhlBRdXXDhhXDddVBfn29rCgMTAsMwSoojR2DgQJgxAzZuzLc1hYEJgWEYJYUnBJWV0NYG\nra35tij/mBAYhlFSHDkCAwaACBx3nIWHwITAMIwCZNMmWL06O9f2PALQ8JAJgQmBYRgFyOOPw//7\nf9m5tl8Ivv1tOPPM7HxOMWHDRw3DKDg6O+HDD7Nz7a6uoBDMm5edzyg2zCMwSp6ennxbUNq8/z58\n+cuhbR0dsHdvdj7P7xEYigmBUdIcOgQzZ1qcOJ+sXQurVoW2dXRkzyPwksVGkLhCICLHi8g7vn8t\nIvJ1ERktIitFZKOIrBCRMt85S0SkXkTqRGRBdr+CYaTOL3+pIlBXl29LenP33fCnP+XbiuyzbZsO\n4/STbSEwjyCUuELgnHvfOTfHOTcHOB04CDwG3AysdM7NAF4IvEdEZgNXA7OBhcDdImKeh1GQLFsG\nJ50EW7fm25LerFoFv/99vq3IPtu39x7L39EBzc3Q3Z35zzMh6E2yHfSFwCbn3A7gcmBZoH0ZcGVg\n+wrgIefcEefcVmATMDcDthpGxtm6FS6+uDCF4MABePHFvpvD6OmB114LegT+Zcw7OvR9c3PmP9ef\nLDaUZIVgEfBQYHu8c64psN0EjA9sVwINvnMagIkpW2gYKdDeHr9zP3gQPvoIzjhDO6NCo6VFwyPr\n1+fbkuywapUWf9u4UUXh4MHgPq88dDbCQ+YR9CZhIRCRQcBlwCPh+5xzDnC9TvIdkrxphpE6v/0t\nfOMbsY9paICqKpg2LTsewdq16T3Nt7TAiSfCG29kzqZCYvVqTdbX1cHgwaHhoY4Ofc3GyCFLFvcm\nmdtxCbDaOef9NE0iMsE51ygiFcCeQPtOYJLvvKpAWwhLly79eLu6uprq6uokTDGM2NTXwwcfxD5m\nxw4VgilTsuMRLFoEv/61ehyp0NICc+b07gz/8Ae49FIYNix9G/PJ6tUwaRLs26d1f1pboaJC93V0\nwNCh6XsEra3wyitwySXBtmLyCGpqaqipqcn65yQjBH9LMCwEsBxYDPwg8Pq4r/13IvITNCQ0HQgb\nHBYqBEZp0t6unfGsWZm/9qZNKgTOaU0ZjzvvhJNPhoUL9bMnTYKjj9YQ0UcfwdKlcPvtmXli7OjQ\nRVBS5cABOOYY7Sj9LFkCw4fDpz+dnn354PTTtWM+6igVgiVL4N57oV+/0JFDHR3626TrESxfDtdc\nA/fdB3/3d9pWTEIQ/pB86623ZuVzEgoNicgwNFHsH8x2B3CRiGwELgi8xzlXCzwM1ALPANcHQkeG\nEcKzz+oU/2ywaZN27OFPlGvWwMsv63ZDg3Y2IuoZbNwIP/qRCsj//t/p23D4MDQ2pnauc/o0G0kI\nDhyAN99M375c090Nb78N+/drR19fr53z66/DyJG9Q0NVVekLwZo1mod44IFgWzEJQa5ISAiccx85\n58Y659p8bc3OuQudczOccwuccwd8+25zzh3nnJvpnHsuG4YbxU9bm4Y/Mk1Pj3bmM2f2Dg91dARr\n0HseAWhI4p13dPuvf9WQTrqjdTo7U/cIDh7UzmrChFAh6OnRe1aMQtDerq+trSrCFRWaGxg8OLIQ\nVFaqaKTDmjXw+c/Dli3BNhs11Bsb32/kjfb27AjBrl1QVqbzA8KF4NChUCGoqtLtcCFwTj2KdEjH\nIzhwAEaNgjFjQoXAG2b51luhwy2LAS/009KiT/rjxgX3jRjRWwjGj0/v/4dz8O678Dd/o5/njUSy\nZHFvTAiMvNHenvqiIHPmBP+ww9m0SevMH3NM6JMgqBBs2qRP1tE8glde0dd0Rerw4dQ9gpYWFbNw\nIThwQMVLRO0vJvweQbgQjBwZmiPo7ExfCHbtCob9Jk4M3i8LDfXGhMDIGx99lNofeleXuvzRwgZ7\n9mgCeNq0yEJw8KB2El6OAFQI3n1Xtz2PId2Vq9LxCFpaensE3/2uDiUtL9fSycUWHvI6+tZW/Y3C\nhSDcIzj66PSE4IMPYPp0FQP//wUTgt6YEBh5w/MIkg1xeCGbAwci7z90SIceTpyoHb6fjg7tYN9+\nW586R4/W9ooKtef444PHptMJdXWp15GORzBqlNq3f79e69FHdaZxWVlhCcGjj8JDD8U/LpZHECk0\nlK4QHDigogkwdWpwrogJQW9MCIy80d6uIuB1EMmcB9E7iUOHYMgQTTaGC8GhQ3DaafDcc8EQCwTH\nr3tj/ocOTc8jOHxYh0Q2NqYWy/dyBAMH6nyB5mad67B+vQrBGWcUjhA8/ji89FL848JzBEcfHdwX\nHhrycgT+3+D223WkWaIcOKD3CkwI4mFCYOQN/xNiKuclIgQ7d/beN28ePPVUMCwEQSHwVqs64YT0\nhWDECB0RE81ziYWXIwAND61fr9dcvz4YGlq9ujDqEK1dq6GeePhDQ5FyBN797unRznrcuNDf+C9/\n0eHGiRaiiyYEXV2WLA7HhMDIG/E69FTP84Tg6KO1Mzh8OLivowPmztWn60hCcNppKgKzZqUXljh8\nGAYN0s9IZdayFxoCGDs2uH5vc7N2buPGqcikmoPIFJ2dsGEDNDXFPzbR0FBnp967srLQ32D/fu3M\nX3wxMdv8YlpZGQzTmUfQGxMCI294sf5MC8HBgyoE/fppeMHfWR46pEIAoUJQXq4d66RJsG6dxubT\n8Qi8zizVxdH9QjBmjArB+EBZR7+nkI3qnMmwYYN+z0Q9gmHD9LuFJ4tHjQp6Tp2dOvN42DDdPnJE\n25ubVaATzbt44TXQ+RiNjept9PRA//6Jf8dSwITAyBvt7dqpZSM0NHSobvvzBM7pvvHjNVTgzSEA\nzRU8+ihMnqzvR43KjEcwY0ZwFFIy+MMas2bB00/D2Wfrey8B6iWS88natXDeeYkJQXu7JvAjeQTj\nxwe9io4OFQKR0JDR/v06+ife77J7t4bO/PfQEwLPG/CXHTFMCIw80t6uHXWqHkGsUUNDhui2Xwi6\nutRLGDAAPvc5rXvj59JLdT/0Hs6YLIcPq4dx/PGpCYHfI/jqV9UWTwi8zm306Px7BDt3au2mjo5g\nxdBotLWpEESaUFZREfTcPCEAvQc33ggPPqhCMGVKYkLw7rt6vHevysv1/017u4WFImFCYOQN/xNi\nsudB/BwB6PW9hLG//Yc/jF0VNJMewfvvJ3++XwiOOUaLs110kcbS/Z1bvj2Ctja16eij43sFnhBs\n2aIdvee1gYrawYP6G4ULwTPPaD2iIUM0XxLvd2lp0Sf/Dz4I3isvTLhzpyWKI2FCYOSNdDyCESMS\nE4LKSp04Ft4ej0x4BH4hSHYIqV8IAP7t3+CUU/T7+END+fYIvN/CH9qJdezEiXo/vNFZHiLBa3R0\nqDcFeg+amzVvU16emEB7++vqQu/hhAk6u9g8gt6YEBg5p7ER7rorKASpeAReiCES/g7/zDO1dpDX\n7j1pxiNdj6CzUzuzsWP1fXgF0Xj449t+fvQjHdkEQY+gpye0umYuScUjADjnnN77Kyo0rOP3CEaO\n1FcvgZ+MEHR0hN5DE4LomBAYOae+Hu64Q7fDx4onwkcfJS4E554L770XLH2ca49AJLU8QbhH4HHp\npboWAQQ9guZmLeecD+/AE4JEPYLKSt2eP7/3fi+h640aAr0HXucf7hHs3BnZ0/L/vzAhSAwTAiPn\ndHfr0+Pw4ak9ebe364ifeCUmQDuUc8+F559PLjQ0alRmhABSGzkUTQj8eB6BN1HLK5qXbVavhu99\nT7f9QhBvToPnEZSX66S+cCoqYPt2DeX5heCCC3TbEwLvd7/oomB9KD8tLcFRQRYaSgwTAiPndHXp\nqycEmQoNOaejUcI7/PPP18Vokg0NpTIj2CNcCJJJGPf06Hf0wiLR8DyCXAtBXR2sWKHbnhBMnqyd\neCza2vQJfdcuPSecCRPgX/5FFwXyfqc5c+DKK/X/SnhoaOfOYP7HT0uLji466qhgrsG7/o4dliyO\nhAmBkXO8EgHDh2tnl4pHEEkI/vpXuOKK4IQyjxNP1IlPyYSGxo3TuL4nWsni5QggeY+grU09mniT\nnjyPwBtFlSshaGsLdsCeEPhLOESjvV1/82hiXFGhDwUDBgTv3bXXwpe+FPQkPCHo6NBjI00ua2nR\nuRfhOZYxYzR8ZR5Bb0wbjZzjF4J0PILWVn169sb+b94cLPLm7/Bnz4ba2uRCQwMHqhjs3h06AzlR\n0gkNJRIWglCPYPx4raiaC9ratEM9fDhUCMJLfodz8GDokNFwTjhB1xeO5LlNnKjf1ys74SWmI4Wj\nWlr0WuGlPcrK1GP0TyQ0FPMIjJzjDw2l6hGUl+sftn9N4m3b9H14hz9pkopGY2PioSHvvEihh0Tw\nC8H06SpSiRaIS1QI/DmC007TjjgXRei8VdJ27w4Vgu3bY3++fzRQJObP1yVC/+EfeieTp0zRkUnD\nh+t1vEmC0TyCCy6AJ58MbS8vVy/PPILeJLp4fZmIPCoiG0SkVkTmichoEVkpIhtFZIWIlPmOXyIi\n9SJSJyILsme+UYx0d2ui78c/Tt0jGD5cn+z8HfW2bcF1kP1CIKKhgtWrE/cIQK+/Y4fG96+/PvIx\n990XuRa/XwiGDdMn4USHkPqLpcWivFzzGK2t6r2MHKlPyv4ie9nAy0k0NASFYOhQ/fxoI4ecCw2X\nxeLCC+G660Lb7roLvvhF/S1HjNCRZyLRhaCsTCfi+SkvVztMCHqTqEfwH8DTzrlZwMlAHXAzsNI5\nNwN4IfAeEZkNXA3MBhYCd4uIeR7Gx3R36x/zaael7hFEEwLQJ8bwEMTs2ckLgecRPP989Hr7772n\n6weHE97pjR0b6r3Ewl8sLRYDBqjI7Nyp93PSJK1JdNZZiX1OqnhCsHmzvnrfM1Z46MgRzXmkWuxt\n1KjQSWYbN2rILZoQRLp/nrhasrg3cTtoERkFnOucux/AOdflnGsBLgeWBQ5bBlwZ2L4CeMg5d8Q5\ntxXYBMzNtOFG8dLdHewQhg7VJ1ivwmQieEIwcaJ21K2tuozjtm3BqqPhT30nnaTLWyYTGvI8glWr\nopdyOHhQ487h+D0CSE4IEg0NgXoCW7YEhWD5ck2MZ3Nh+7Y2HYGzYUPo6J9YCWP/3IB08YTg1FOT\nEwJvRrZ5BL1J5El9GrBXRB4QkbdF5JciMgwY75zzHMEmIFAkl0rAH1ltACZmzGKj6PEvDBJeYTIR\nvKRjVZU+DW/erKtXbdmiBdCGDOldXfKMM7RzTsUjWLUq9pyFfArB2LH6vYcP1yGcf/6zekSJVANN\nlbY2DbXV1oYKQaycir9sRLqUl6sndsopmvf5z/8MfZCIdv+81d5MCHqTiJM0ADgN+Kpz7k0R+SmB\nMJCHc86JSKxnkF77li5d+vF2dXU11dXVidhr9AH8HgEEhwSOGZPY+V4yuKpKFynxZtSOG6deghey\n8DNnjopDsjmCNWv0qfPIkcgx7kSFYNy45IQgkRwBqBCsW6cd8qBBwbDNo49q+3/9V2LXSYa2Nq3c\n+qc/hXa448ZFvhcQP1GcDAsXahG+yZP1geCrX4UFCzQp7/1Ow4ZFPresrLiEoKamhpqamqx/TiJC\n0AA0OOe8FVIfBZYAjSIywTnXKCIVgPcMshPwD7irCrSF4BcCo7SIJASJegTd3epRDBoUzBE0N2vt\nmk9/WkMTkTr7kSO11EMyndGsWdq5fOUr8JOfqFfgLQ7jcfBg5Kfvzs7QCWHJ5gi8MEY8xo7V8JU3\nuxd03sQ996gN2aCtTUfl/Md/hA7FHDtWJ5tFIpOhoa98BW65Rb/vnXfCT3+qnsH06XrvRo6Mvt5A\neXlxCUH4Q/Ktt96alc+JGxpyzjUCO0RkRqDpQmA98CSwONC2GHg8sL0cWCQig0RkGjAdWJVRq42i\npqsrVAiSSRj7Fy3xhGDfPh03vmSJdkbRnvrnzg3W6UmE8nJNvt58sz5JRgoPeR5BeEw+lzmCnh79\nXpMmacjtwgs1dLJtW3aGk7a3q0hWVISGhuJ5BJkKDY0bB//931pQ8O//HmbODM4n2LBBBT8a5eWW\nLI5Eorfka8BvRWQQsBn4MtAfeFhEvgJsBa4CcM7VisjDQC3QBVzvXDZTV0ax0d0d+seYjEfgn2zk\nJYv37dPJRhBbCO66K/XOKFrtf69+/kcfhYpMJCFYuzaxz0o2NATaIc+eDTfcoE/GHnv2aGI3k3hD\nRufNCxX0WEKQSY8A4POfD277F7V5663eCw75KbbQUK5IaFinc+5d59yZzrlTnHOfdc61OOeanXMX\nOudmOOcWOOcO+I6/zTl3nHNupnPuueyZbxQj4aGhZDwC/2SxESNUULZsCQrBmDHRZ6+OHh09dhyP\nsjIt4XDvvaHtBw/q6969GrJYuVLfRxKCaJ1kOMkmiyG4YM1Pf6rLOU6cqInzeGUfUsEvBH47Y3k9\nmfQIwpkwITh6aPXq2AsOFVtoKFfY+H4j56STIwivF1RVpU/anhBMmBC5oFm6lJfD736nT9ybNgXb\nDx3Sz3v8cbj/fnjiCW3PxTwC77oQ6o188pOwbJnOxg0vs5AuXV363YYO1Xvx/e8H9+UqWRyOV74a\n1CMwIUgeEwIj5/iHj0LqHgGoEHiLlgB86lNapiDTlJVpLZ+jj9bEsd+eKVO0auY//mNwEZxoOQKv\nQFwsks0RQKj4DR2q98Eb1+9c5nIF3hwOb4avt74AqM0dHZGT1JkODfnxQkMtLTqceObM6MdaaCgy\nJgRGzok2fDQRIgnBoUNBIejfv/fInkxQXq5hoMsuCx2eevCgDmOcPFlzEJs36xN9JCHYulXtjZcx\nSzVHEI7nEfz2t7oAfCZoa4teHltEQ3ORPJ9sh4YaGzV0d8opsZPBJ5wQmkMxFBMCI+ekExoKFwJv\n6UNPCLKF1zGffXZo+OPQIR25cv/9+sQ7d66Wozh8OLTjGzlS1x3u7o6/zkG6oSGPSZN0aOmuXaHh\nrHTw8gPRiBYeyqZH4OUI4iWKQZPM3/hGduwoZkwIjJyTbmjI36F449gTnYyWKuXl+sQ7b16wo+vp\n0Q7u8suDK25ddRX85jfa7vcIRLQMRlVV5LIIHl1d+vSc6DDXsjINSUXqZP1F6bxqnemSqhBkM0dw\n9NHqhbz2Wuz8gBEdEwIj52TSI/CEINEJWKlSVqafNWlScN6AF+7o5/srWrRIi9Tt2hUqBB4VFdE7\n5YceUu8i1oSocPr109nDkY73Qm5e7DwTtLamLgTZCg0NHKizjf/0JxOCVDEhMHJOpoaPgnbOQ4Yk\nVzoiFaZM0bCPt3JYe3vkhW5GjYIvf1kT2NGEIJpHsH69LgGZaFgoHp4QtLbqXItMzDTevz+26Hpz\nO8LJZmgI4IEHdInLWJPJjOjYHDsj53R3h3aS6QwfPfZYfYrONvPmaf0eCD71DhoUWYDuvFOTyOH1\n8CG2EDQ0qLdwyimZsdlbzcsT2d27dSRROsQTgunTdSx/ONn0CEBzJQ88kL3r93XMIzByTiaHjw4b\npjVvcoknBNGWXhw4UGv9VFT03hdLCHbs0NdMeQQjRmhM30tOZyJPsH9/7MT8ccdFTkxn2yMw0sM8\nAiPnZHL4aD4YNw7++EcVoWRtqayEN9+MvK+hQe9FokNH49G/f3DhmsmTM5MnaG6OnZiPJgQdHdGH\nnRr5xzwCI+dEyhG0tia2mEqkhc1zzbhx8LOf6cS1ZIUgmkfgnHoE55+fOY8A9Fo7dmiRuHQ8Am+d\n6XihoUmTdASPV3rDI9uhISM9TAiMnBMeGho4UOPt4Z1HJArFI+js1BpH0eoaRSPaqKHmZu0o587N\n7FDYsjK1ddasYOgpWbZt07pFED801L+/5iE++CC03UJDhY0JgZFzwj0CSDxhXChC4C2EngmPoLkZ\nHnxQR0DdeKPW2s8UnnexYAG8/HJq19iyRZeG7O6O7xGAhofq64Pvb79dv7MJQeFiQmDknEhCkGjC\nOHzUUD445xwttpbsimegCVzngiuJATz7LHz72xpWGTo0czkCUCEYMEBrD9XVwQ9/CM88k9w1du7U\n36yxUUUrnhDMnq3LWHrccw+88YaFhgoZEwIj54SvRwDF5RHMn6+VNysrkw8NifT2Cj78EC6+GL7z\nnczaCSoqo0Zp6O388/UzHnkkuWt4oawdO+KHhkBXSFu3Lvi+pcU8gkLHhMDIOeErlEHiHkEhCIHH\nlCmp2VJZGSoE+/bBWWdBNpbtHjUqOFrnm9+Ef/5nXb0sGbzRRp4QxPMITjop+BnOBQXePILCxYTA\nyDnRcgSJCkGhPFlOnpyaEIQnjD/8MHu1kkaNCuYJzjtPPYLaWv0NnnlGJ72tWBH7Gjt3amG3LVt0\nJbZ46z3MnKlVWA8f1hnYXgnsQvndjN6YEBg5J1qOoFhCQx5TpiQfGoLeoaF9+4JVRDNNWVno+P0R\nI7RI2+bN8POf6/5nn419jV271GNZt06P7xen1zjqKB05tHFjqLibEBQuJgRGzgkfPgrJeQSFIgTf\n+AZ87WvJnxcpR5ALj8DDC93s3g1f+lJwMZ1o7NypJTbWrk28uN/MmZqcbmkJir6FhgqXhIRARLaK\nyFoReUdEVgXaRovIShHZKCIrRKTMd/wSEakXkToRWZAt443ipNiHj3pMmBC6QleiRBKCbHkEp5yi\nS1f6mTVLO+ndu7WE9vr10VdO6+nR4zyPwKv2Go9Jk1RAWlqCC8GYR1C4JOoROKDaOTfHOTc30HYz\nsNI5NwN4IfAeEZkNXA3MBhYCd4uIeR7Gx0QTgv37459bCMNH0yWXoaFPfAK+9a3QtmnTdMLXvn2a\n5zjpJF2GMxLNzRr+mj8fnnwyWHgvHlVVWjLjwAENoY0YYUJQyCTTQYdXPL8cWBbYXgZcGdi+AnjI\nOXfEObcV2ATMxTACRBo+6q07G4/29uIXgspK2L49+D6boaFITJ0Kq1bpMNABA7TTjnbvveTwgAFw\nySWJ2+mVo/bWX77jDvUSjMIkGY/geRF5S0S8or/jnXNNge0mwFspthLwVyRvACambanRZ4g0fLSy\nMn5RtF27VAgmT86ebblg5kwdUfPOO8GyGqkknVNl6lQNB3nVUaMtJgOph+KqqoKhoVGj4PrrzSMo\nZBKtPnqOc263iIwDVopInX+nc86JSKySYb32LV269OPt6upqqrMxiNooSCKFhiZOjF8U7fnndVJU\n+LnFRv/+cO21cO+9sGSJPmUnuiJZJpgyRcf3e0IwdmzkBech9SUm/R5BJmdKlxo1NTXU1NRk/XMS\nEgLn3O7A614ReQwN9TSJyATnXKOIVAB7AofvBPxOYFWgLQS/EBilRSQh8MbWOxe9U1y5Ei66KPv2\n5YJrr9XY/AUXZC8/EI0hQ3QIqd8j8NcG8pOOEOzapTmCTFZTLTXCH5JvvfXWrHxO3NCQiAwVkRGB\n7WHAAuA9YDmwOHDYYuDxwPZyYJGIDBKRacB0YFWmDTeKl0jDR4cN0+GFsRLGb72ldX76AhUVOrkr\nX8srTp0a6hFECw2lmpwfMkR/082bTQiKgUQ8gvHAY6KPaQOA3zrnVojIW8DDIvIVYCtwFYBzrlZE\nHgZqgS7geucSqTRvlAqRPAIIPkVGq2XT0pLbpGq2+fa34dJLYcaM3H/2tGnBoa/jxkUPDaUzk7uq\nSmcxX3ZZaucbuSOuEDjntgCnRmhvBi6Mcs5twG1pW2f0SaIJgZcwPvHEyOe1tvatVa769dNKnfng\nzjuDT+pejmD7dh3Z4w/NpRoaAl1b4Ve/Mo+gGLDx/UbOiTR8FGInjLu79el02LDs2lYqTJ4cKgR7\n9+qcg9deCz0unXkb3/2u/s4mBIWPCYGRcyINHwX1CKIJQVubjmfP5eiaUmHsWL3vu3bB66+H7ksn\nNDRtmk5AO+209G00sosJgZFzYuUIos0l6GthoULCS+wOHqwTzSA4vyGd0BDAFVfEr1Zq5B8TAiPn\nRAsNxfIIWlutQ8km48bBF76gQtDTo7mC1ta+UdLDiI8JgZFzYoWGzCPIDzNmwD/+o9YWWrtWX3fs\nKKz1H4y+6UQ1AAAcVUlEQVTskejMYsPIGPGGj0bChCC7PPecvp5wQnB9goaG9ENDRnFgHoGRc6IJ\nwfjxOnqlu7v3PhOC3DBjRlAUTAhKBxMCI+dEyxEMHKgTxpqaeu8zIcgNxx8Pr7yicxy80JDlCPo+\nJgRGzomWI4DoCWMTgtxw/PFw5IguaGMeQelgQmDknGihIYg+hLStzYQgF3h1j845x4SglDAhMHJO\ntNAQmEeQb449VsNC8+erEFhoqDQwITByTqzQ0NSpWrEyHBOC3DB4MPz851BdrTkC8whKAxMCI+fE\nCg2dcIKunhWOCUHuuP56Xa/g8GEtRmdC0PcxITByTqzQkAlBYSACEybAtm0WGioFTAiMnBPLI5g6\nVWe1traGtluJidxTUaFDec0j6PuYEBg5J1aOoF8/mDVLFzTxYx5B7pkwQV9NCPo+JgRGzonlEYCG\nh9atC20zIcg93lKWFhrq+5gQGDmlp0fjz/1i/M877TRYvTq0zYQg95hHUDqYEBg5JVZYyGPu3GBd\nfADnLEeQD0wISoeEhEBE+ovIOyLyZOD9aBFZKSIbRWSFiJT5jl0iIvUiUiciC7JluFGcxAsLAZx6\nKtTV6WQm0EVSBg/WWkRG7rDQUOmQqEdwI1ALuMD7m4GVzrkZwAuB94jIbOBqYDawELhbRMzrMD4m\n1tBRj6OOgpkzYc0afW9hofzgeQSDB+fXDiP7xO2kRaQK+BvgV4C3YuzlwLLA9jLgysD2FcBDzrkj\nzrmtwCZgbiYNNoqbRDwCgLPPhpoa3TYhyA8VFTBoUOx8jtE3SOQnvgv4J6DH1zbeOecVC24Cxge2\nK4EG33ENwMR0jTSKk5df1vi+n0RyBKBr3T72mG5bwbn8MGEC/OQn+bbCyAUxnXQRuRTY45x7R0Sq\nIx3jnHMi4iLt8w6J1Lh06dKPt6urq6mujnh5o0jp6oJPfQrq62HKlGB7IqEhgE9+Ej74AC66CC6+\n2IQgH/TvDzfckG8rSpuamhpqPNc4i4gLf2Tz7xS5DbgG6AKOAkYCfwLOBKqdc40iUgG86JybKSI3\nAzjn7gic/yxwi3PujbDrulifaxQ/9fW62tVrr8FZZwXbd+/W4aG7d8e/xg9+APfdp9cZMAAefzx7\n9hpGMSAiOOck/pHJETM05Jz7rnNuknNuGrAI+LNz7hpgObA4cNhiwPsTXQ4sEpFBIjINmA6sCr+u\n0fepq9PX8JLSiYaGAG66CRYt0lnG5hEYRvZINg3kPcbfAVwkIhuBCwLvcc7VAg+jI4yeAa63R//S\n5P339TVcCBJNFntUVcHWrTaHwDCySQLRWsU59xLwUmC7GbgwynG3AbdlxDqjaKmr007cCwF9+CGM\nHaseQSI5Ao+qKk04m0dgGNnDBoYZWaGuDs4/Xz2C/fvhuONg+3Y4/fTkPIKJgTFnJgSGkT1MCIys\nsGWLLne4a5eWMm5pgbVrdU5AsqEhMCEwjGxiQmBkHOc0FHTKKSoEe/dq+9q1+ppMaGj0aJ3ZakJg\nGNkjiT9Jw0iMgwf1qf/YY3sLwWWXaZnpRBFRr8CEwDCyh3kERsbZt0+f5MeMgfZ2XQQdVAjOOANu\nvz256516ajBEZBhG5jGPwMg4+/apCIjA5Mnw1lva/v77MH587HMj8eijmbXPMIxQzCMwMo4nBADT\npsGbb2pop6cnNSEwDCO7mBAYGae5OVQI3n8fTjxR33uljQ3DKBxMCIyE6eqCw4fjH+flCECFAIJC\nYB6BYRQeJgRGwvzmNzBvHnR2xj4uPDQEJgSGUciYEBgJs3s3rF8P3/lO7OPCQ0OgQjB8OAwdml0b\nDcNIHhMCI2EOHIBvflPLQT/zTPTjInkEc+fCI49k30bDMJLHhCBHOJdYDf5C5sABOOYY+Kd/ir02\ngD9HMHYsLFsGw4bBwoW5sdMwjOQwIcgR69bpOrzFzIEDUFamC8t7ZabDefpp2Lgx6BGIwJe+lDsb\nDcNIHhOCHNHernX1GxvzbUlyvPeehoMgKAQzZmhnH4mbbtIFz485Jnc2GoaRHiYEOcIbafPGG7GP\nKzTWrYMnn9RtTwiqqnS7ra338S0t8NRTUFGRWzsNw0gdE4Ic4QnB66/n145k2bNHPZmurqAQ9OsH\n06dH9gq8YwzDKB5MCHJEZ6cOnSw2Idi7V0Vgx47QTj5SeKi7WyuP2rKShlFcmBDkiM5O+MQnYPVq\n7TCLBa+E9ObNutLYqFH6/vjjewtBa6vOFehn/6sMo6iI+ScrIkeJyBsiskZEakXk9kD7aBFZKSIb\nRWSFiJT5zlkiIvUiUiciC7L9BYqFjg6ts1NZqZOyioW9e6G8XEtIDxyoi8SAegThI4csLGQYxUlM\nIXDOdQDnO+dOBU4GzheR+cDNwErn3AzghcB7RGQ2cDUwG1gI3C0i9nyIegSDB2uJhmJKGO/ZA2ed\npZ5MeXmwPZJH0NJiQmAYxUjcTto5dzCwOQjoD+wHLgeWBdqXAVcGtq8AHnLOHXHObQU2AXMzaXCx\n4gnBWWfBa6/l25rE2btX5z+89lpoJ+8li50Lth04EAwdGYZRPMQVAhHpJyJrgCbgRefcemC8c64p\ncEgT4JUSqwQafKc3ABMzaG/R4gnBySdDbW3049at06fwQmHvXrj6ati+PVQIvLWE/fMizCMwjOIk\n7gplzrke4FQRGQU8JyLnh+13IuIin62HRGpcunTpx9vV1dVUV1cnYm/R0tkJRx0FkyYFl26MxD/8\nA3zmM/Dtb+fOtmgcOaJzBY49Fs4/XyeK+fHCQy0tOm/APALDyCw1NTXU1NRk/XMSXqrSOdciIk8B\npwNNIjLBOdcoIhWA9wy7E5jkO60q0NYLvxCUAp5HUFmpT9lHjmjy1U9Tk4ZgZs/Oj43hfPihloro\n1w++8AV49dXQ/V7C+Ic/hPnzYcgQ8wgMI5OEPyTfeuutWfmceKOGxnojgkRkCHAR8A6wHFgcOGwx\n4JUgWw4sEpFBIjINmA6syobhxYYnBAMGwNFHw65dofudgz/8QZ+so9XxyTV798K4cbq9eDHcc0/o\n/qlTYds2aGjQ2cQWGjKM4iRejqAC+HMgR/AG8KRz7gXgDuAiEdkIXBB4j3OuFngYqAWeAa53zsUK\nG5UMHR3BoZeTJmnn6eenP4U774Sf/ATq6nJvXyR27QqWihDRf34mT9YwV0ODLlD/wQcWGjKMYiRm\naMg59x5wWoT2ZuDCKOfcBtyWEev6EJ5HAJHzBOvWwS23wFVXwXXXhdb0zxdbt+pTfzQmT1bvpa0N\nLrkEHnsMzj03V9YZhpEpbIx/jvALQVVVbyH48EMNw4hoErYQwkPbtsGUKdH3T54M77wDEyfCtdfq\nzGLzCAyj+DAhyBHxPIIPP9RFXCB2medcEs8jqKrSpHdVlXoEkycHF6QxDKN4SHjUkJEefiGYMgX+\n/OfQ/X4h8JKw+SaeR3DUUboYfVUV9O+vo4pscXrDKD7MI8gR3jwCgNNPh1WrQmflFqIQxPMIQL2A\nqirdnjhRR0UZhlFcmBDkCP+oocmTtcPcskXfd3Xp0Euvls+UKdoJZ4OnnkpslbTOTk1Yx1tgZvJk\nFQDDMIoXE4Ic4Q8NiWhJam+C1v79KgL9++t7zyOINvC2qwv+9V9Ts+P22+HZZ+Mft327dvCeTdH4\nv/9XS1AYhlG8mBDkCL8QgArBSy/ptj8sBPqU3dAAF18ceSGbXbt0qGlLS/J27N6tawvEY926xGY4\nn3KKltc2DKN4MSHIEeFC8L/+l46737hRZ/D6hWDwYJ1DsHKlhnLC8UI7GzYkZ4NzKgSbNsU/dvVq\nzWUYhtH3MSHIEeFCUFEBN96oM4nDPQLQ8NC8eUGvwU9ToO5rrCqmkWhthUOHEhOCt9+G03pNJTQM\noy9iQpBFenqC2+FCAFqorbY2shD867/CQw9ph3zoUOg+TwiS9Qh271ZPo74+ev4BdN/q1SYEhlEq\nmBBkibY2OO447eQhdNSQh7e4S1NTbyG48EKYNg1mzdLZu34aG7U9FSE44QTdbm5WDyHacT09wWGh\nhmH0bUwIssTrr+vw0Acf1Pf+eQQelZUqGDU1mnSNxKxZvYvQNTVBdXVqQlBRoQJVXw9z5mixuHDe\nflvzA+FF5gzD6JuYEGSJV1+F886DX/5SQy2RQkP9+qlX8OKLuhxkJCLVHWpqgnPOgZ07e4eNYuEJ\nwamnaiL6gw9UhMKxsJBhlBYmBBnm2WehuxteeQW+9S3tqN9/P7IQgNYVqqzU+kORiCQEjY06xv+Y\nY5KrSeQJwdy5cO+9Okfgr3/tfZwlig2jtDAhyCA9PfDZz8LatfDGGzpX4JJL4OmntThb+FKPoEJw\n9tnRwzDRPILx45PPE3hCMG+ezlP47GdVCPxJbTAhMIxSw4Qgg2zbph7ACy/AsGFaVnrhQnjiCRWB\nSJ39dddBrNXnjjtOcw1HjgTbGhtTE4I9e/S8E06AoUPhsstgxIhQoWlshI8+0kS1YRilgQlBBvE6\n5See0E4a4FOf0gJzkcJCoLOIvWMjMWSIho682cAffaRhpvJynfm7YYOWqPjDH+Lb5y09OWAAfPOb\nuiD9uefCyy8Hj3nhBU1EW6LYMEoHE4IMUlur9fhfeSVYnmHkSDjrrOhCkAjz5mlnfeqpGraZOlU7\n6lmz9DNfeknzEfHYsye4BvH3v6/DQ889F5Yv1/DUwYOaRL7ootRtNQyj+DAhyCC1tfDpT+soIf9T\n/sKF6QnBeefBD38I776rJSeOOUbbjz9ePYW6Oh1B5E00i4RzwVXQ/Myfr9d87TUNEa1YAQsWpG6r\nYRjFR1whEJFJIvKiiKwXkXUi8vVA+2gRWSkiG0VkhYiU+c5ZIiL1IlInIiXTrdTWwpVX6ra/YNtl\nl6W3YMu55+q4/379NPHsCcHQoVrwbcUKfb96dfRrtLRomClckGbO1I5/7lwVhAED4NhjU7fVMIzi\nIxGP4Ajwf5xzJwBnATeIyCzgZmClc24G8ELgPSIyG7gamA0sBO4WkT7veTinQvDJT2pHeuKJwX2z\nZ8Obb6Z+7dmzVUiuvhreey80kTtrFvzlLxo2iiUE/rCQHxF47jkVg1//WsNQhmGUFnE7aOdco3Nu\nTWC7HdgATAQuB5YFDlsGBJ6FuQJ4yDl3xDm3FdgEzM2w3QXHzp36hD5mjBZ1Cy8Z0S8NKezXTz2C\nq67S955HACoE3d0qErGEYO9eOPro6PtnztTPmNvnfynDMMJJqnsSkanAHOANYLxzzotKNwFe8KMS\naPCd1oAKR5+mtjax+v2pMmKEdtbQ2yMYMAA+85n4QhDJI/Dwrm1CYBilR8IrzIrIcOCPwI3OuTbx\njS90zjkRiVHPkl77li5d+vF2dXU11dXViZpSkGzYkF0hAA05jRwZ6hGcdJLONZgxQ+sW7dkT+ck/\nnhDMmKFzH2wimWEUDjU1NdREqgOTYRISAhEZiIrAb5xzjweam0RkgnOuUUQqgD2B9p2Av2BCVaAt\nBL8QFDP79umY/tpaOPnk7H7WwIGwY4eKgcfcuTp8VEQ78dWrdTZzONEEwmPECL32iBGZt9swjNQI\nf0i+Ndbs0zRIZNSQAPcBtc65n/p2LQcWB7YXA4/72heJyCARmQZMB1ZlzuTC4gtfgJ/9TKt4Ztsj\ngFARABUAr4M//fTo4aF4HgGooBmGUXokkiM4B/gicL6IvBP4txC4A7hIRDYCFwTe45yrBR4GaoFn\ngOudi7UMSnGzfTssWaIlIObPz68tZ5wRuaw0BBejNwzDCEfy0UeLSJ/RhvJyuPxyuOGG/Cdat29X\nMWhq6l0i4rjj4H/+J5gUNgyj+BARnHMZLwCTcLLY6M2hQ1qW4cEHC6M2z+TJuvhNfb0mfwEeeUSH\nl+7erWJgGIYRjglBGnhlnQtBBDzOOUdrHXlC8OCDWhLbG2ZqGIYRTp+f8ZtNdu1SISgk5s8PLjbj\nnC6ZeehQ9kc0GYZRvJgQpMHu3VoiupCYN0/LXoOGiEaMgMWLtQKqYRhGJCxYkCIbN8K6dYUnBCef\nrGsRt7erN3DWWXD33fm2yjCMQsY8ghT56lfh3/6t8EJDgwbpbOO339ay1XPm5NsiwzAKHROCBNi9\nW5/+Pbq7NfwyenTheQQAZ56p1U7r62H69HxbYxhGoWNCkADLlsHNNwff19ZqWeiaGvj85/NmVlTm\nzlWh2rTJhMAwjPiYECTA5s3w6qvQ06PvX30VPvEJXQR+2LD82haJuXM1P7Bliy0yYxhGfEwIEmDz\nZl0gvq5O37/8so7XL1SmT4cDB3RthKFD822NYRiFjglBAmzapB2/5xWsWFHYC7z366d5AptJbBhG\nIpgQxKGzU2v3LFqkM3bXrNH6QlOn5tuy2Myda/kBwzASw+YRxGHLFpg0Cc47T8tNT5+u6/sWOt/6\nFnR05NsKwzCKAROCGHR3w5NPasL1hBN0cZdf/hIeeCDflsVnzJh8W2AYRrFgQhCB7dt17sArr2in\nf9dd0L+/ztLdsEG9A8MwjL6CCUEYmzbpSl8DBujSkP/zP1rjH+Bzn4OPPtJkrGEYRl/BhCCM99+H\ns8/WJSgfeSQoAgDXXps/uwzDMLKFPduGsXOnLun4xS/CE0/k2xrDMIzsk8ji9feLSJOIvOdrGy0i\nK0Vko4isEJEy374lIlIvInUiUgTja0LZtasw6wcZhmFki0Q8ggeAhWFtNwMrnXMzgBcC7xGR2cDV\nwOzAOXeLSFF5HZ5HYBiGUSrE7aSdcy8D+8OaLweWBbaXAVcGtq8AHnLOHXHObQU2AXle0j05TAgM\nwyg1Un1aH++cawpsNwHjA9uVQIPvuAagqLpVCw0ZhlFqpB22cc45wMU6JN3PyCXmERiGUWqkOny0\nSUQmOOcaRaQC2BNo3wlM8h1XFWjrxdKlSz/erq6uprq6OkVT0qe9HYYP17pCLS0wblzeTDEMw/iY\nmpoaampqsv45og/0cQ4SmQo86Zw7KfD+TmCfc+4HInIzUOacuzmQLP4dmheYCDwPHOfCPkREwpvy\nRmendvzvvQff/z68+KKWnTYMwyg0RATnnGT6unE9AhF5CPgkMFZEdgD/AtwBPCwiXwG2AlcBOOdq\nReRhoBboAq4vmB4/Cvv3Q1sbfPazMHIkPPdcvi0yDMPILQl5BBn/0ALyCGpr4cQTwTn4y1/g3HPz\nbZFhGEZksuURFNUY/2ywf7+WkVi2DObPz7c1hmEYuafkaw01N2uO4EtfyrclhmEY+aHPeASvvgr3\n3Zf8efv364pjhmEYpUqf8AgOHoRrrtGn+899DkaNSvzc/fth9Ojs2WYYhlHo9AmP4K67YM4cuOQS\n+NWvkju3udk8AsMwSpui9QgOHoSXXtJVw+66S0NDra06DPTrX9dFZRJh/35ditIwDKNUKVqPYPly\nuPxyuPFGuOIKmDFDR/9Mm6YLyiSKeQSGYZQ6ResRPPusFof7/e+hvj7Y/r3vwQ03aK5g0KD417Fk\nsWEYpU7ReQRHjsCvfw0rVsAf/wiPPQZTpgT3L1ig3sEvfpHY9ZqbLVlsGEZpU3QewT33wJ13alzf\nv56wnx//WGcIX3NN7AJyTz4Je/aYR2AYRmlTVCUmDh6EY45Rb+Dkk2Mf+73vwc9/Dj/6EVx3Xe/9\nhw/DkCHQ0wO7d8OECUmbYxiGkVOsxATw1FMqAPFEAODf/x3efBP++Z/h7bd779+6FcaOhepqCw0Z\nhlHaFJUQ/P73sGhR4scff7x6BYsWaYVRP/X1cOqpWnY6kaSyYRhGX6VohODwYS0R/ZnPJHfe1VfD\nhRfqKKLOzmD7pk0wfXpmbTQMwyhGikYIGhpgzJjUErs/+xkMG6bJ444Obauvh+OOy6yNhmEYxUjR\nCMG2baHDRJNhwAD43e+guxvGj1ev4pVXzCMwDMOAEhECgKOO0nkHH3yg6w6sWwczZ2bOPsMwjGKl\naIRg+3aYPDn964wZA9/6FjQ2Wo0hwzAMKCIhSNcjCGfMmMxdyzAMo5jJihCIyEIRqRORehG5KRPX\nzLQQGIZhGErGhUBE+gO/ABYCs4G/FZFZqV7vppvg3nth8+bCEYKampp8m5AWZn/+KGbbwezvq2TD\nI5gLbHLObXXOHQF+D1yRyoU2b4b779eSEu3tmckRZIJi/89k9uePYrYdzP6+SjaEYCKww/e+IdCW\nNI89pkM9H30U9u6FoUMzYp9hGIbhIxvVRxOqJnfZZfGPWb1aPQLDMAwje2S8+qiInAUsdc4tDLxf\nAvQ4537gOyb3JU8NwzD6ANmoPpoNIRgAvA98CtgFrAL+1jm3IaMfZBiGYWSEjIeGnHNdIvJV4Dmg\nP3CfiYBhGEbhkpeFaQzDMIzCIeczi7Mx2SwTiMhWEVkrIu+IyKpA22gRWSkiG0VkhYiU+Y5fEvgO\ndSKywNd+uoi8F9j3H1m0934RaRKR93xtGbNXRAaLyB8C7a+LSEZncUSxf6mINAR+g3dE5JJCtF9E\nJonIiyKyXkTWicjXA+1Fcf9j2F8s9/8oEXlDRNaISK2I3B5oL5b7H83+/N1/51zO/qGhok3AVGAg\nsAaYlUsbYti2BRgd1nYn8J3A9k3AHYHt2QHbBwa+yyaC3tUqYG5g+2lgYZbsPReYA7yXDXuB64G7\nA9tXA7/Pgf23AN+McGxB2Q9MAE4NbA9Hc2KziuX+x7C/KO5/4JpDA68DgNeB+cVy/2PYn7f7n2uP\nIGOTzbJEeDb+cmBZYHsZcGVg+wrgIefcEefcVvSHmSciFcAI59yqwHG/9p2TUZxzLwP7s2iv/1p/\nRJP/2bYfev8GUGD2O+canXNrAtvtwAZ0rkxR3P8Y9kMR3P+A3QcDm4PQB8z9FMn9j2E/5On+51oI\nMjbZLAs44HkReUtE/j7QNt451xTYbgLGB7YrUds9vO8R3r6T3H6/TNr78W/lnOsCWkQkF6s7f01E\n3hWR+3yufcHaLyJTUc/mDYrw/vvsfz3QVBT3X0T6icga9D6/6JxbTxHd/yj2Q57uf66FoJAz0+c4\n5+YAlwA3iMi5/p1OfaxCtj+EYrM3wH8B04BTgd3Aj/NrTmxEZDj6tHWjcy5kVexiuP8B+x9F7W+n\niO6/c67HOXcqUAWcJyLnh+0v6Psfwf5q8nj/cy0EO4FJvveTCFW0vOGc2x143Qs8hoaxmkRkAkDA\nDdsTODz8e1Sh32NnYNvfvjO7loeQCXsbfOdMDlxrADDKOdecPdPBObfHBQB+hf4Gni0FZb+IDERF\n4DfOuccDzUVz/332/7dnfzHdfw/nXAvwFHA6RXT/I9h/Rj7vf66F4C1guohMFZFBaBJjeY5t6IWI\nDBWREYHtYcAC4D3UtsWBwxYD3h/8cmCRiAwSkWnAdGCVc64RaBWReSIiwDW+c3JBJux9IsK1Pge8\nkG3jA3+8Hp9Bf4OCsz/wWfcBtc65n/p2FcX9j2Z/Ed3/sV7YRESGABcB71A89z+i/Z6IBcjt/Y+X\n3c70PzT08j6a8FiS68+PYtM0NCu/Bljn2QWMBp4HNgIrgDLfOd8NfIc64GJf++mBH3AT8LMs2vwQ\nOnP7MBoL/HIm7QUGAw8D9Wj8eGqW7f87NNm1FngX/SMeX4j2oyM8egL/X94J/FtYLPc/iv2XFNH9\nPwl4O2D/WuCfMv33mif783b/bUKZYRhGiVM0S1UahmEY2cGEwDAMo8QxITAMwyhxTAgMwzBKHBMC\nwzCMEseEwDAMo8QxITAMwyhxTAgMwzBKnP8P+YajttfqeQgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f1117066a50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "t,r = zip(*sorted(rewards.items(),key=lambda k:k[0]))\n",
    "plt.plot(t,pd.ewma(np.concatenate(r),alpha=0.1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from agentnet.utils.persistence import save\n",
    "save(action_layer,\"gopher.pcl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "###LOAD FROM HERE\n",
    "from agentnet.utils.persistence import load\n",
    "load(action_layer,\"gopher.pcl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-12-09 05:10:23,215] Making new env: GopherDeterministic-v0\n",
      "[2016-12-09 05:10:23,282] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 741 timesteps with reward=200.0\n",
      "Episode finished after 796 timesteps with reward=480.0\n",
      "Episode finished after 1956 timesteps with reward=1540.0\n",
      "Episode finished after 1094 timesteps with reward=360.0\n",
      "Episode finished after 1052 timesteps with reward=600.0\n",
      "Episode finished after 756 timesteps with reward=320.0\n",
      "Episode finished after 1456 timesteps with reward=800.0\n",
      "Episode finished after 1021 timesteps with reward=500.0\n",
      "Episode finished after 991 timesteps with reward=240.0\n",
      "Episode finished after 1013 timesteps with reward=580.0\n",
      "Episode finished after 942 timesteps with reward=320.0\n",
      "Episode finished after 1050 timesteps with reward=400.0\n",
      "Episode finished after 1424 timesteps with reward=520.0\n",
      "Episode finished after 1653 timesteps with reward=920.0\n",
      "Episode finished after 791 timesteps with reward=560.0\n",
      "Episode finished after 1035 timesteps with reward=840.0\n",
      "Episode finished after 1170 timesteps with reward=400.0\n",
      "Episode finished after 1037 timesteps with reward=760.0\n",
      "Episode finished after 1187 timesteps with reward=320.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2016-12-09 05:13:31,946] Finished writing results. You can upload them to the scoreboard via gym.upload('/home/apanin/jheuristic/drqn/records')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 1104 timesteps with reward=700.0\n",
      "mean session score=568.000000.5\n"
     ]
    }
   ],
   "source": [
    "action_layer.epsilon.set_value(0.05)\n",
    "rw = pool.evaluate(n_games=20,save_path=\"./records\",record_video=False)\n",
    "print(\"mean session score=%f.5\"%np.mean(rw))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from agentnet.utils.persistence import save\n",
    "save(action_layer,\"dqn_seaquest_mon2am_50k.pcl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from IPython.display import HTML\n",
    "\n",
    "#select the one you want\n",
    "video_path=\"./records/openaigym.video.0.13.video000000.mp4\"\n",
    "\n",
    "HTML(\"\"\"\n",
    "<video width=\"640\" height=\"480\" controls>\n",
    "  <source src=\"{}\" type=\"video/mp4\">\n",
    "</video>\n",
    "\"\"\".format(video_path))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Once you got it working,\n",
    "Try building a network that maximizes the final score\n",
    "\n",
    "* Moar lasagne stuff: convolutional layers, batch normalization, nonlinearities and so on\n",
    "* Recurrent agent memory layers, GRUMemoryLayer, etc\n",
    "* Different reinforcement learning algorithm (p.e. qlearning_n_step), other parameters\n",
    "* Experience replay pool\n",
    "\n",
    "\n",
    "Look for info?\n",
    "* [lasagne doc](http://lasagne.readthedocs.io/en/latest/)\n",
    "* [agentnet doc](http://agentnet.readthedocs.io/en/latest/)\n",
    "* [gym homepage](http://gym.openai.com/)\n",
    "\n",
    "\n",
    "You can also try to expand to a different game: \n",
    " * all OpenAI Atari games are already compatible, you only need to change GAME_TITLE\n",
    " * Other discrete action space environments are also accessible this way\n",
    " * For continuous action spaces, either discretize actions or use continuous RL algorithms (e.g. .learning.dpg_n_step)\n",
    " * Adapting to a custom non-OpenAI environment can be done with a simple wrapper\n",
    " \n",
    " \n",
    "__Good luck!__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}