examples/Deep Kung-Fu with GRUs and A2c algorithm (OpenAI Gym).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Agentnet intro\n",
    "\n",
    "In this tutorial we build above the [basic tutorial](http://bit.ly/29EvHvZ) withsome new algorithms\n",
    "* this time we're solving KungFuMaster\n",
    "* use single-layer LSTM on top of convolutional layers\n",
    "* actor-critic with n-step critic objective and n-step advantage for actor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: THEANO_FLAGS=device=gpu0,floatX=float32\n"
     ]
    }
   ],
   "source": [
    "#setup theano/lasagne. Prefer GPU\n",
    "%env THEANO_FLAGS=device=gpu0,floatX=float32"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#If you are running on a server, launch xvfb to record game videos\n",
    "#Please make sure you have xvfb installed\n",
    "import os\n",
    "if os.environ.get(\"DISPLAY\") is str and len(os.environ.get(\"DISPLAY\"))!=0:\n",
    "    !bash ../xvfb start\n",
    "    %env DISPLAY=:1\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Experiment setup\n",
    "* Here we basically just load the game and check that it works"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "%matplotlib inline\n",
    "\n",
    "#number of parallel agents and batch sequence length (frames)\n",
    "N_AGENTS = 10\n",
    "SEQ_LENGTH = 25"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using gpu device 0: GeForce GTX 1080 (CNMeM is enabled with initial size: 45.0% of memory, cuDNN 5105)\n",
      "[2017-02-21 20:34:31,740] Making new env: KungFuMaster-v0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f435c8e55d0>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWEAAAFiCAYAAAAna2l5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAG+5JREFUeJzt3WuMnNWd5/Hv3zd8w5ib7bBAMDG3EWCCO2bYhGQTchki\nTQY2o0xmR4MyoyxiMkhsFGlItETxYikbeUSWzQyMkEarhLDZEW9YElYbx4HJDJibbBwTJsQsF8cN\npjsGbGzcDb6dfVHVme56ju2q7uo6Vd3fj9RS16lTVf/TVf3r0895LpFSQpJUxozSBUjSdGYIS1JB\nhrAkFWQIS1JBhrAkFWQIS1JBhrAkFWQIS1JBhrAkFWQIS1JBkxbCEfGXEfFyRAxHxBMR8YHJei1J\n6lWTEsIR8UfA7cA3gPcDW4H1EXHaZLyeJPWqmIwT+ETEE8CTKaWb67cD6Ae+k1Ja19D3VOBTwHbg\nnbYXI0mdNxc4B1ifUnrjWB1ntfuVI2I2sAr45khbSilFxE+BKzMP+RTwP9tdhyR1gT8BfnCsDm0P\nYeA0YCYw2NA+CFyQ6b+92SeeNata7mc/+9lK25lnntnsU7bN+9//fgBuv/12vvKVrwCwa9euMX2u\nuuqqyuNeffXVSlt/f3+l7cILL6y07du3r9LW+DN68sknK3327NlTaRsaGvrt9xs2bOATn/hEpU+v\naEf9tX/exlq4cGGlLfeZnKj777+f6667ru3P20m9PoaJ1j84OMi9994LTeTbZIRwq5reBJH7xViy\nZEml7ayzzppYReNw0UUXAXDiiSf+9vuTTjppTJ/LL7+88riTTz650jZ37txK28qVKyttu3fvrrTN\nnj17zO2BgYFKnwULFlTa3n777TGv/573vKfSp1e0o/7cZ63x/YTqz7sd5s2bV+Qz3E69PoY21n/c\nfJuMEH4dOAwsbWhfClQTQZJ62ObNm3n66afHtA0PDzf9+LaHcErpYERsBq4Gfgi/XZi7GvhOu19P\nkkpatWoVq1atGtPW39/P7bff3tTjJ2tzxLeB79bD+Cngy8B84LuT9HrFvfFGbQH0wIEDv/1+zpw5\nY/ps3Lix8rjDhw9X2nLbGbds2VJp27t373Efu3///qae/4QTTvjt9zNmzBhzu9e0o/7c5ogZMzy2\nSe03KSGcUrqvvk/wbdQ2Q/wc+FRKadexH9n7PvrRj5YuYcIuueSS0iVMSK/Xn1s76DW9PoZO1j9p\nC3MppbuAuybr+bvVVAjhSy+9tHQJE9Lr9Tf+a9uLen0Mnazf/68kqSBDWJIK6ob9hJt25MiRStvW\nrVsrbTt37uxEOWM88sgjHX9NSd0ptw//0TgTlqSCDGFJKsgQlqSCDGFJKqinFuZycidVOfXUUwtU\nIkk1rZyn3ZmwJBVkCEtSQYawJBVkCEtSQT2/MPfuu+9W2kZfqkeSOq2Vk7o7E5akggxhSSrIEJak\nggxhSSqo5xfm9uzZU2nLXR9Mkjold/3Ho3EmLEkFGcKSVJAhLEkF9dQ24dyZiWbMqP4dcZuwpF7h\nTFiSCjKEJakgQ1iSCjKEJamgnlqYmzlzZqVt5cqVlbbly5d3ohxJynr11Vd5/PHHm+rrTFiSCjKE\nJakgQ1iSCjKEJamgnlqYyx0dd9lllzXVNmvW2KEeOnSo0id3RF6vaTxasHHcMHXH3vj5yC3kHjx4\nsFPldFRurI0OHz7cgUo6b/bs2WNu58Z55MiRTpUDwLZt25ru60xYkgoyhCWpIENYkgoyhCWpoJ5a\nmMstHj3//PNN9WvceD9VF6eaWZibqotTjQtzuYXc3Ps+FUznhbnGz3huEa7TC3M7duxoum/LM+GI\nuCoifhgRr0bEkYj4TKbPbRGxMyKGImJDRKxo9XUkaToYz+aIBcDPgS8BlaljRNwC3ATcAKwG9gPr\nI2LOBOqUpCmp5c0RKaUfAz8GiPwlLG4G1qaUHqz3uR4YBK4F7ht/qZI09bR1YS4ilgPLgIdG2lJK\ne4EngSvb+VqSNBW0e2FuGbVNFIMN7YP1+9out8E9twDRuEgzVRdoGuX+WZmqCzSNC6u5hdapOvbG\n93k6j70bFuZaeT13UZOkgto9Ex4AAljK2NnwUmBLm19LkorbtGkTmzdvHtM2PDzc9OPbGsIppZcj\nYgC4GngGICIWAVcAd7bztSSpG/T19dHX1zemrb+/n3Xr1jX1+JZDOCIWACuozXgBzo2IlcCbKaV+\n4A7g1oh4AdgOrAVeAR5o9bUkaaobz0y4D/hHagtwCbi93v494M9TSusiYj5wN7AYeAS4JqV0oA31\nStIx9dqRr+PZT/ifOM6CXkppDbBmfCVJ0vTh3hGSVJAhLEkFGcKSVFBPncpyIjp9xEy3mE7jnk5j\nbTRdjgDNaXzfe21hzpmwJBVkCEtSQYawJBVkCEtSQYawJBVkCEtSQYawJBVkCEtSQT1/sEbjZYsA\nZs6cWWmbNev4Q+21nbxzGn8euZ/FVBhnTuPYc5+NqXBQQ+6SVbmxNpoKlzfKjb3xM94NlzfKXwM5\nz5mwJBVkCEtSQYawJBVkCEtSQT21MJfb2L1ixYpK28UXX9yJctTlcguQrSyYqDc0vs/d8B7PmTOn\n6b7OhCWpIENYkgoyhCWpIENYkgrqqYW53BE/jz76aKVt+/btHahGkvIGBweb7utMWJIKMoQlqSBD\nWJIKMoQlqaCeWpjLnZZx9erVlbZLL720E+VIUtbzzz/PPffc01RfZ8KSVJAhLEkFGcKSVJAhLEkF\n9dTCXO7UhAMDA5W2RYsWdaIcScp67bXXmu7rTFiSCjKEJakgQ1iSCuqpbcI577zzTqVtaGioQCWS\nVJPLpaNxJixJBbUUwhHxtYh4KiL2RsRgRNwfEedn+t0WETsjYigiNkRE9WqckqSWZ8JXAX8DXAF8\nHJgN/CQi5o10iIhbgJuAG4DVwH5gfUQ0f/lRSZomWtomnFL69OjbEfEF4DfAKmDkEhc3A2tTSg/W\n+1wPDALXAvdNsF5JmlImuk14MZCANwEiYjmwDHhopENKaS/wJHDlBF9LkqaccYdwRARwB/BoSumX\n9eZl1EK58QJLg/X7JEmjTGQXtbuA3wE+2KZaJGnaGVcIR8TfAp8GrkopjT5IegAIYCljZ8NLgS3j\nLVKSutWmTZvYvHnzmLbh4eGmH99yCNcD+A+Aj6SUdoy+L6X0ckQMAFcDz9T7L6K2N8Wdrb6WJHW7\nvr4++vr6xrT19/ezbt26ph7fUghHxF3AHwOfAfZHxNL6XW+llEYOEbkDuDUiXgC2A2uBV4AHWnkt\nSZoOWp0J30ht4e1nDe1/BtwDkFJaFxHzgbup7T3xCHBNSunAxEqVpKmn1f2Em9qbIqW0Blgzjnok\naVrx3BGSVJAhLEkFGcKSVJAhLEkFGcKSVJAhLEkFGcKSVJAhLEkFGcKSVJAhLEkFGcKSVJAhLEkF\nGcKSVJAhLEkFGcKSVJAhLEkFGcKSVJAhLEkFGcKSVJAhLEkFGcKSVJAhLEkFGcKSVJAhLEkFGcKS\nVJAhLEkFGcKSVJAhLEkFGcKSVJAhLEkFGcKSVJAhLEkFGcKSVJAhLEkFGcKSVJAhLEkFGcKSVFBL\nIRwRN0bE1oh4q/71WET8XkOf2yJiZ0QMRcSGiFjR3pIlaepodSbcD9wCXA6sAh4GHoiIiwAi4hbg\nJuAGYDWwH1gfEXPaVrEkTSEthXBK6f+klH6cUnoxpfRCSulW4G3gd+tdbgbWppQeTCk9C1wPnAFc\n29aqJWmKGPc24YiYERGfB+YDj0XEcmAZ8NBIn5TSXuBJ4MqJFipJU9GsVh8QERcDjwNzgX3AdSml\nbRFxJZCAwYaHDFILZ0lSg5ZDGPgVsBI4CfhD4J6I+HBbq5KkaaLlEE4pHQJeqt/cEhGrqW0LXgcE\nsJSxs+GlwJYJ1ilJXWnTpk1s3rx5TNvw8HDTjx/PTLjRDOCElNLLETEAXA08AxARi4ArgDvb8DqS\n1HX6+vro6+sb09bf38+6deuaenxLIRwR3wT+L7ADOBH4E+AjwCfrXe4Abo2IF4DtwFrgFeCBVl5H\nkqaLVmfCS4DvAe8B3qI24/1kSulhgJTSuoiYD9wNLAYeAa5JKR1oX8mSNHW0FMIppS820WcNsGac\n9UjStOK5IySpIENYkgoyhCWpIENYkgoyhCWpIENYkgoyhCWpIENYkgoyhCWpIENYkgoyhCWpIENY\nkgoyhCWpIENYkgoyhCWpIENYkgoyhCWpIENYkgoyhCWpIENYkgoyhCWpIENYkgoyhCWpIENYkgoy\nhCWpIENYkgoyhCWpIENYkgoyhCWpIENYkgoyhCWpIENYkgoyhCWpIENYkgoyhCWpIENYkgoyhCWp\nIENYkgqaUAhHxFcj4khEfLuh/baI2BkRQxGxISJWTKxMSZqaxh3CEfEB4AZga0P7LcBN9ftWA/uB\n9RExZwJ1StKUNK4QjoiFwL3AF4E9DXffDKxNKT2YUnoWuB44A7h2IoVK0lQ03pnwncCPUkoPj26M\niOXAMuChkbaU0l7gSeDK8RYpSVPVrFYfEBGfBy4D+jJ3LwMSMNjQPli/T5I0SkshHBFnAncAH08p\nHZyckiRp+mh1JrwKOB14OiKi3jYT+HBE3ARcCASwlLGz4aXAlgnWKkldZ9OmTWzevHlM2/DwcNOP\nbzWEfwpc0tD2XeA54FsppZciYgC4GngGICIWAVdQ244sSVNKX18ffX1jt8729/ezbt26ph7fUgin\nlPYDvxzdFhH7gTdSSs/Vm+4Abo2IF4DtwFrgFeCBVl5LkqaDlhfmMtKYGymti4j5wN3AYuAR4JqU\n0oE2vJYkTSkTDuGU0scybWuANRN9bkma6jx3hCQVZAhLUkGGsCQVZAhLUkGGsCQVZAhLUkGGsCQV\n1I6DNTompVRpGxoaqrTt2dN4imNJ6px9+/Y13deZsCQVZAhLUkGGsCQVZAhLUkE9vzD35ptvVtrm\nzZvXiXIkKWv37t1N93UmLEkFGcKSVJAhLEkF9dQ24X+9tui/OvXUUyttS5Ys6UQ5kpR16NChpvs6\nE5akggxhSSrIEJakggxhSSqopxbmcgdr5HaKXrBgQSfKkaSsVs7k6ExYkgoyhCWpIENYkgoyhCWp\noJ5amJsxo/o3Y8WKFZW28847rxPlSFLWokWLmu7rTFiSCjKEJakgQ1iSCjKEJamgnlqYO3z4cKXt\nF7/4RaXt9ddf70Q5kpTVSgY5E5akggxhSSrIEJakggxhSSqopxbmcteYO/HEEyttixcv7kQ5kpT1\n7rvvNt23pZlwRHwjIo40fP2yoc9tEbEzIoYiYkNEVI8rliQB49sc8SywFFhW//rQyB0RcQtwE3AD\nsBrYD6yPiDkTL1WSpp7xbI44lFLadZT7bgbWppQeBIiI64FB4FrgvvGVKElT13hmwudFxKsR8WJE\n3BsRZwFExHJqM+OHRjqmlPYCTwJXtqVaSZpiWp0JPwF8AdgGvAdYA/xzRFxMLYATtZnvaIP1+yYs\nd4251157rdJ28ODBStvChQvH3M5dh25oaKjSNm/evErbrl3VfwRytXWDU045ZdyPbfaaft2iceF2\nyZIllT6593j+/PmVtv3791fa3n777QlUN7lmzpw55vbpp59e6bNv375KW25hO/cet7LQ1GknnHDC\nmNsnn3xypU+zY8/9bueO1D2eVn5PWgrhlNL6UTefjYingF8DnwN+1cpzSZImuItaSumtiHgeWAH8\nDAhqi3ajZ8NLgS0TeR1J6lY7duygv79/TFvuv/GjmVAIR8RCagH8vZTSyxExAFwNPFO/fxFwBXDn\nRF5HkrrV2Wefzdlnnz2mbffu3Tz00ENHecRYLYVwRPw18CNqmyD+DfBfgIPAP9S73AHcGhEvANuB\ntcArwAOtvE4r5s6dW2nLbe89//zzx9w+55xzKn127txZaTvjjDMqbQ8//HCl7cCBA8cqs2Mat4uu\nWrWqrc+/cePGtj5fOzVuG/zgBz9Y6dPse7x9+/ZK269+1R1b3HIHLTUeoHTlldW18B07dlTaGsMD\n4Jlnnqm0vfLKK62UOGlyYz/zzDPH3L7kkksqfZod+2OPPVZp27NnTyslAjA8PNx031ZnwmcCPwBO\nBXYBjwK/m1J6AyCltC4i5gN3A4uBR4BrUkrdkVCS1GVaXZj74yb6rKG214Qk6Tg8gY8kFWQIS1JB\nPXUWtXbK7bydW1x75513mnq+3IJBNzh06FClrdndZ2bPnl1p69Zx5uTeu9x7nPss5A5U6eaxN9aW\nO9gk91nIjb2Z5+9muTHlxp77GeXGOZ6xt/IYZ8KSVJAhLEkFGcKSVJAhLEkF9fzCXG7xJbfBvfHs\nSLlFp5zcWdpyG/7Hc6alThgcbDypXf6IwtxC1BtvvFFpy/1su0XjZ2FgYKDSJ7dg8tZbb1XacmfT\n6uaxN75/ufcuN/bc2b5ef/31Sls3j73xvcpd3iw39tw4cz+P3Jn3jqeVI+acCUtSQYawJBVkCEtS\nQYawJBUUpS/LExGXA5ub6TtjRvVvxrnnnltpW7Ro0XGfa86c6gWgG0+FCM0fUdRLcpdsymllcaFX\n5C5pk7t0T7ecmrSdTjrppEpb7vN95MiRTpQzaXI5kXvfcwuy7TI0NDRy6tNVKaWnj9XXmbAkFWQI\nS1JBhrAkFWQIS1JBPX/EXG6xIdfWaP78+ZW23JFkM2fOHF9hXayZhUuAvXv3TnIlkyt3lNRpp51W\nacsdDTaeo6S6Se5ze/rpp1facovRvb4omVt0P+WUUyptuZ9RuxYlW8kNZ8KSVJAhLEkFGcKSVJAh\nLEkF9dTCXG6hZcmSJZW2pUuXHve55s6dW2mbNav641i4cGGlrVtPW9ms3AJkTu6UgL0ktziSO1rw\n5JNPrrQ1e23BbpU7VWvuM9/sEYS9JLfYmFusy/0eNHv9xePJnUr0aJwJS1JBhrAkFWQIS1JBhrAk\nFdTzC3NnnHFGpe29731vpa1xw/yFF15Y6ZM7rWfuGm2/+c1vjllnN8ktLL7vfe9r6rEvvvhipe3t\nt9+ecE2d0uyibe5z9dxzz1Xa2rVo0wnnnHNOpS23EJU7Qmzbtm1N9esWjQuw559/fqVP7vSWuSMl\nt2/f3paacougR+NMWJIKMoQlqSBDWJIK6qltwrntOrltnrmDDBq3++V2xs/t5J3TSwcx5A5AyW3b\nzG0Pz/08cs/XrXLbenNyl3Fq9oCWbnXo0KFKW+69y13ip9mz7HWLZn63c2PK/R6063e7lcuiOROW\npIIMYUkqyBCWpIIMYUkqqHdWWcgvHuUuwbN79+7jPtebb77Zlpp60a5du0qXUMzAwEDpEjoi9zvQ\n399foJLO64bf7VYuDeZMWJIKajmEI+KMiPh+RLweEUMRsTUiLm/oc1tE7KzfvyEiVrSvZEmaOloK\n4YhYDGwE3gU+BVwEfAXYParPLcBNwA3AamA/sD4iqmdVlqRprtVtwl8FdqSUvjiq7dcNfW4G1qaU\nHgSIiOuBQeBa4L7xFipJU1HkFruO2jniX4AfA2cBHwFeBe5KKf19/f7lwIvAZSmlZ0Y97mfAlpTS\nlzPPeTmwebwDyF2epZeO6pI09Rw6dGjkqLlVKaWnj9W31W3C5wJ/AWwDPgn8HfCdiPjT+v3LgERt\n5jvaYP0+SdIorU4ZZwBPpZS+Xr+9NSIuBm4Evt/WyiSpBxw4cIADBw6MaWvl/MuthvBrQOPZrp8D\n/n39+wEggKWMnQ0vBba0+FqS1PXmzJlTuWjEqM0Rx9Xq5oiNwAUNbRdQX5xLKb1MLYivHrkzIhYB\nVwCPtfhakjTltToT/m/Axoj4GrU9Ha4Avgj8x1F97gBujYgXgO3AWuAV4IEJV5vRyinjJKnbtBTC\nKaVNEXEd8C3g68DLwM0ppX8Y1WddRMwH7gYWA48A16SUDuSeU5Kms5Z2UZuUAia4i5okdbG276Im\nSWojQ1iSCjKEJakgQ1iSCjKEJakgQ1iSCjKEJakgQ1iSCjKEJakgQ1iSCuqGEJ5bugBJmiTHzbdu\nCOFzShcgSZPknON16IYT+JxK7crN24F3ihYjSe0xl1oAr08pvXGsjsVDWJKms27YHCFJ05YhLEkF\nGcKSVJAhLEkFdU0IR8RfRsTLETEcEU9ExAdK13Q0EXFVRPwwIl6NiCMR8ZlMn9siYmdEDEXEhohY\nUaLWnIj4WkQ8FRF7I2IwIu6PiPMz/bpyDBFxY0RsjYi36l+PRcTvNfTpytpzIuKr9c/Rtxvau3YM\nEfGNes2jv37Z0Kdr6weIiDMi4vsR8Xq9xq31y62N7jPpY+iKEI6IPwJuB74BvB/YCqyPiNOKFnZ0\nC4CfA18CKruXRMQtwE3ADcBqYD+18czpZJHHcBXwN9Sulv1xYDbwk4iYN9Khy8fQD9wCXA6sAh4G\nHoiIi6Drax+jPtm4gdpnfnR7L4zhWWApsKz+9aGRO7q9/ohYDGwE3qW2i+xFwFeA3aP6dGYMKaXi\nX8ATwH8fdTuAV4C/Kl1bE7UfAT7T0LYT+PKo24uAYeBzpes9yhhOq4/jQz08hjeAP+ul2oGFwDbg\nY8A/At/ulZ8/tQnT08e4v9vr/xbwT8fp05ExFJ8JR8RsarOZh0baUm3EPwWuLFXXeEXEcmqzgtHj\n2Qs8SfeOZzG1Gf2b0FtjiIgZEfF5YD7wWC/VDtwJ/Cil9PDoxh4aw3n1TXIvRsS9EXEW9Ez9vw9s\nioj76pvkno6IL47c2ckxFA9harOwmcBgQ/sgtR9Cr1lGLdB6YjwREcAdwKMppZFtel0/hoi4OCL2\nUft38i7gupTSNnqgdoD6H47LgK9l7u6FMTwBfIHav/I3AsuBf46IBfRG/ecCf0HtP5FPAn8HfCci\n/rR+f8fGMKudT6aedBfwO8AHSxfSol8BK4GTgD8E7omID5ctqTkRcSa1P3wfTykdLF3PeKSU1o+6\n+WxEPAX8Gvgctfem280Ankopfb1+e2tEXEztD8r3O11Iaa8Dh6lt4B9tKTDQ+XImbIDaNu2uH09E\n/C3waeDfpZReG3VX148hpXQopfRSSmlLSuk/U1vYupkeqJ3a5rfTgacj4mBEHAQ+AtwcEQeozba6\nfQxjpJTeAp4HVtAb78FrwHMNbc8BZ9e/79gYiodwfSawGbh6pK3+L/LVwGOl6hqvlNLL1N6k0eNZ\nRG1PhK4ZTz2A/wD4aEppx+j7emUMDWYAJ/RI7T8FLqG2OWJl/WsTcC+wMqX0Et0/hjEiYiG1AN7Z\nI+/BRuCChrYLqM3mO/s7UHqVsr7q+DlgCLgeuBC4m9pq9+mlaztKvQuo/eJcRm2vgv9Uv31W/f6/\nqtf/+9R+2f438P+AOaVrr9d3F7Vdca6i9pd95GvuqD5dOwbgm/Xa3wtcDPxX4BDwsW6v/Rhjatw7\noqvHAPw18OH6e/BvgQ3UZvCn9kj9fdTWE74GvA/4D8A+4POdfg+K/zBGDfhL1E5nOQw8DvSVrukY\ntX6kHr6HG77+x6g+a6jt4jIErAdWlK57VG252g8D1zf068oxAH8PvFT/rAwAPxkJ4G6v/Rhjenh0\nCHf7GID/RW030mFgB/ADYHmv1F+v79PAM/X6/gX480yfSR+Dp7KUpIKKbxOWpOnMEJakggxhSSrI\nEJakggxhSSrIEJakggxhSSrIEJakggxhSSrIEJakggxhSSrIEJakgv4/tBhnI7wZKToAAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f435cec1bd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import gym\n",
    "from agentnet.experiments.openai_gym.wrappers import PreprocessImage\n",
    "#game maker consider https://gym.openai.com/envs\n",
    "def make_env():\n",
    "    env = gym.make(\"KungFuMaster-v0\")\n",
    "    return PreprocessImage(env,height=64,width=64,\n",
    "                           grayscale=True,\n",
    "                           crop=lambda img:img[50:])\n",
    "\n",
    "#spawn game instance\n",
    "env = make_env()\n",
    "observation_shape = env.observation_space.shape\n",
    "n_actions = env.action_space.n\n",
    "\n",
    "obs = env.step(0)[0]\n",
    "\n",
    "plt.imshow(obs[0],interpolation='none',cmap='gray')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic agent setup\n",
    "Here we define a simple agent that maps game images into action probas and state values via convolutional neural network.\n",
    "\n",
    "![a2c](https://s12.postimg.org/odg35favx/a2c_scheme.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import theano, lasagne\n",
    "import theano.tensor as T\n",
    "from lasagne.layers import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#observation\n",
    "observation_layer = InputLayer((None,)+observation_shape,)\n",
    "\n",
    "#4-tick window over images\n",
    "from agentnet.memory import WindowAugmentation, LSTMCell\n",
    "\n",
    "prev_wnd = InputLayer((None,4)+observation_shape)\n",
    "new_wnd = WindowAugmentation(observation_layer,prev_wnd)\n",
    "        \n",
    "#reshape to (frame, h,w). If you don't use grayscale, 4 should become 12.\n",
    "wnd_reshape = reshape(new_wnd, (-1,4*observation_shape[0])+observation_shape[1:])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from lasagne.nonlinearities import elu,tanh,softmax\n",
    "\n",
    "#network body\n",
    "conv0 = Conv2DLayer(wnd_reshape,32,5,stride=2,nonlinearity=elu)\n",
    "conv1 = Conv2DLayer(conv0,32,5,stride=2,nonlinearity=elu)\n",
    "conv2 = Conv2DLayer(conv1,64,5,stride=2,nonlinearity=elu)\n",
    "        \n",
    "dense = DenseLayer(dropout(conv2,0.1),512,nonlinearity=tanh,name='dense')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\n",
    "prev_cell = InputLayer((None,256))\n",
    "prev_out = InputLayer((None,256))\n",
    "\n",
    "new_cell,new_out = LSTMCell(prev_cell,prev_out,dense,name='lstm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#actor head\n",
    "logits_layer = DenseLayer(concat([dense,new_out]),n_actions,nonlinearity=None) \n",
    "#^^^ store policy logits to regularize them later\n",
    "policy_layer = NonlinearityLayer(logits_layer,softmax)\n",
    "\n",
    "#critic head\n",
    "V_layer = DenseLayer(concat([dense,new_out]),1,nonlinearity=None)\n",
    "\n",
    "#sample actions proportionally to policy_layer\n",
    "from agentnet.resolver import ProbabilisticResolver\n",
    "action_layer = ProbabilisticResolver(policy_layer)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Finally, agent\n",
    "We declare that this network is and MDP agent with such and such inputs, states and outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from agentnet.agent import Agent\n",
    "#all together\n",
    "agent = Agent(observation_layers=observation_layer,\n",
    "              policy_estimators=(logits_layer,V_layer),\n",
    "              agent_states={new_wnd:prev_wnd,new_cell:prev_cell,new_out:prev_out},\n",
    "              action_layers=action_layer)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[W,\n",
       " b,\n",
       " W,\n",
       " b,\n",
       " W,\n",
       " b,\n",
       " dense.W,\n",
       " dense.b,\n",
       " lstm.b_to_ingate,\n",
       " lstm.W_ctrl0_to_ingate,\n",
       " lstm.b_to_forgetgate,\n",
       " lstm.W_ctrl0_to_forgetgate,\n",
       " lstm.b_to_cell,\n",
       " lstm.W_ctrl0_to_cell,\n",
       " lstm.b_to_outgate,\n",
       " lstm.W_ctrl0_to_outgate,\n",
       " lstm.W_cell_to_ingate_peephole.scales,\n",
       " lstm.W_cell_to_forgetgate_peephole.scales,\n",
       " lstm.W_cell_to_outgate_peephole.scales,\n",
       " W,\n",
       " b,\n",
       " W,\n",
       " b]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Since it's a single lasagne network, one can get it's weights, output, etc\n",
    "weights = lasagne.layers.get_all_params([V_layer,policy_layer],trainable=True)\n",
    "weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create and manage a pool of atari sessions to play with\n",
    "\n",
    "* To make training more stable, we shall have an entire batch of game sessions each happening independent of others\n",
    "* Why several parallel agents help training: http://arxiv.org/pdf/1602.01783v1.pdf\n",
    "* Alternative approach: store more sessions: https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-02-21 20:34:33,360] Making new env: KungFuMaster-v0\n",
      "[2017-02-21 20:34:33,418] Making new env: KungFuMaster-v0\n",
      "[2017-02-21 20:34:33,462] Making new env: KungFuMaster-v0\n",
      "[2017-02-21 20:34:33,499] Making new env: KungFuMaster-v0\n",
      "[2017-02-21 20:34:33,533] Making new env: KungFuMaster-v0\n",
      "[2017-02-21 20:34:33,571] Making new env: KungFuMaster-v0\n",
      "[2017-02-21 20:34:33,608] Making new env: KungFuMaster-v0\n",
      "[2017-02-21 20:34:33,646] Making new env: KungFuMaster-v0\n",
      "[2017-02-21 20:34:33,684] Making new env: KungFuMaster-v0\n",
      "[2017-02-21 20:34:33,721] Making new env: KungFuMaster-v0\n"
     ]
    }
   ],
   "source": [
    "from agentnet.experiments.openai_gym.pool import EnvPool\n",
    "\n",
    "pool = EnvPool(agent,make_env, N_AGENTS) #may need to adjust\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "actions:\n",
      "[10  1  5 13  8  1  7  0  5 12]\n",
      "rewards\n",
      "[ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.]\n",
      "CPU times: user 236 ms, sys: 12 ms, total: 248 ms\n",
      "Wall time: 247 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#interact for 7 ticks\n",
    "_,action_log,reward_log,_,_,_  = pool.interact(10)\n",
    "\n",
    "print('actions:')\n",
    "print(action_log[0])\n",
    "print(\"rewards\")\n",
    "print(reward_log[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#load first sessions (this function calls interact and remembers sessions)\n",
    "pool.update(SEQ_LENGTH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advantage actor-critic\n",
    "\n",
    "* An agent has a method that produces symbolic environment interaction sessions\n",
    "* Such sessions are in sequences of observations, agent memory, actions, q-values,etc\n",
    "  * one has to pre-define maximum session length.\n",
    "\n",
    "* SessionPool also stores rewards, alive indicators, etc.\n",
    "* Code mostly copied from [here](https://github.com/yandexdataschool/tinyverse/blob/0b359aa6a5a9f666d2fa9eab97669c7930b7acb3/atari.py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/envs/py27/lib/python2.7/site-packages/agentnet/utils/logging.py:14: UserWarning: [Verbose>=2] Recurrent loop without unroll_scan got nonempty random state updates list. That happened because there is some source of randomness (e.g. dropout) inside recurrent step graph. To compile such graph, one must either call .get_automatic_updates() right after .get_output and pass these updates to a function when compiling theano.function.\n",
      "  default_warn(\"[Verbose>=%s] %s\"%(verbosity_level,message),**kwargs)\n"
     ]
    }
   ],
   "source": [
    "#get agent's Qvalues obtained via experience replay\n",
    "#we don't unroll scan here and propagate automatic updates\n",
    "#to speed up compilation at a cost of runtime speed\n",
    "replay = pool.experience_replay\n",
    "\n",
    "_,_,_,_,(logits_seq,V_seq) = agent.get_sessions(\n",
    "    replay,\n",
    "    session_length=SEQ_LENGTH,\n",
    "    experience_replay=True,\n",
    "    unroll_scan=False,\n",
    ")\n",
    "\n",
    "auto_updates = agent.get_automatic_updates()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\n",
    "# compute pi(a|s) and log(pi(a|s)) manually [use logsoftmax]\n",
    "# we can't guarantee that theano optimizes logsoftmax automatically since it's still in dev\n",
    "logits_flat = logits_seq.reshape([-1,logits_seq.shape[-1]])\n",
    "policy_seq = T.nnet.softmax(logits_flat).reshape(logits_seq.shape)\n",
    "logpolicy_seq = T.nnet.logsoftmax(logits_flat).reshape(logits_seq.shape)\n",
    "        \n",
    "# get policy gradient\n",
    "from agentnet.learning import a2c\n",
    "elwise_actor_loss,elwise_critic_loss = a2c.get_elementwise_objective(policy=logpolicy_seq,\n",
    "                                                                     treat_policy_as_logpolicy=True,\n",
    "                                                                     state_values=V_seq[:,:,0],\n",
    "                                                                     actions=replay.actions[0],\n",
    "                                                                     rewards=replay.rewards/100.,\n",
    "                                                                     is_alive=replay.is_alive,\n",
    "                                                                     gamma_or_gammas=0.99,\n",
    "                                                                     n_steps=None,\n",
    "                                                                     return_separate=True)\n",
    "        \n",
    "# add losses with magic numbers \n",
    "# (you can change them more or less harmlessly, this usually just makes learning faster/slower)\n",
    "# also regularize to prioritize exploration\n",
    "reg_logits = T.mean(logits_seq**2)\n",
    "reg_entropy = T.mean(T.sum(policy_seq*logpolicy_seq,axis=-1))\n",
    "loss = 0.1*elwise_actor_loss.mean() + 0.25*elwise_critic_loss.mean() + 1e-3*reg_entropy + 1e-3*reg_logits\n",
    "\n",
    "        \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Compute weight updates, clip by norm\n",
    "grads = T.grad(loss,weights)\n",
    "grads = lasagne.updates.total_norm_constraint(grads,10)\n",
    "\n",
    "updates = lasagne.updates.adam(grads, weights,1e-4)\n",
    "\n",
    "#compile train function\n",
    "train_step = theano.function([],loss,updates=auto_updates+updates)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Demo run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-02-21 20:37:53,468] Making new env: KungFuMaster-v0\n",
      "[2017-02-21 20:37:53,526] DEPRECATION WARNING: env.spec.timestep_limit has been deprecated. Replace your call to `env.spec.timestep_limit` with `env.spec.tags.get('wrapper_config.TimeLimit.max_episode_steps')`. This change was made 12/28/2016 and is included in version 0.7.0\n",
      "[2017-02-21 20:37:53,527] Clearing 2 monitor files from previous run (because force=True was provided)\n",
      "[2017-02-21 20:37:53,550] Starting new video recorder writing to /home/hedgedir/rl_projects/records/openaigym.video.1.27736.video000000.mp4\n",
      "[2017-02-21 20:37:57,702] Finished writing results. You can upload them to the scoreboard via gym.upload('/home/hedgedir/rl_projects/records')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 931 timesteps with reward=100.0\n"
     ]
    }
   ],
   "source": [
    "untrained_reward = np.mean(pool.evaluate(save_path=\"./records\",\n",
    "                                         record_video=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<video width=\"640\" height=\"480\" controls>\n",
       "  <source src=\"./records/openaigym.video.1.27736.video000000.mp4\" type=\"video/mp4\">\n",
       "</video>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#show video\n",
    "from IPython.display import HTML\n",
    "import os\n",
    "\n",
    "video_names = list(filter(lambda s:s.endswith(\".mp4\"),os.listdir(\"./records/\")))\n",
    "\n",
    "HTML(\"\"\"\n",
    "<video width=\"640\" height=\"480\" controls>\n",
    "  <source src=\"{}\" type=\"video/mp4\">\n",
    "</video>\n",
    "\"\"\".format(\"./records/\"+video_names[-1])) #this may or may not be _last_ video. Try other indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#starting epoch\n",
    "epoch_counter = 1\n",
    "\n",
    "#full game rewards\n",
    "rewards = {}\n",
    "loss,reward_per_tick,reward =0,0,0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAi0AAAFkCAYAAADsVgtLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3XeYU2Xax/HvDQIqygCiYG8oYlmlCqiIYm+o2Aa7qygr\nqKxt7bzq7lp2xYJg1xWVVewuCCKKojRlQEGKArIoCIjAgCh1nvePO9kJYWaYkuRMZn6f68qVyTlP\nzrnPIZo7T7UQAiIiIiKVXY2oAxAREREpDSUtIiIikhWUtIiIiEhWUNIiIiIiWUFJi4iIiGQFJS0i\nIiKSFZS0iIiISFZQ0iIiIiJZQUmLiIiIZAUlLSIiIpIVypS0mNktZjbBzFaY2SIze8vM9k0q87yZ\nFSQ9hiaVqWNmj5vZEjNbaWavm9kOSWUamNnLZpZvZsvM7Bkzq5tUZlczG2Jmq8xsoZk9YGZKxERE\nRKqgsn7BHwE8BhwKHAPUAj4ws62Syr0PNAaaxB65SfsfBk4GugIdgZ2AN5LKvAI0BzrHynYEnozv\njCUnQ4EtgHbAxcAlwN1lvCYRERHJAlaRBRPNrBGwGOgYQvgstu15ICeEcGYx76kH/AycF0J4K7at\nGTAdaBdCmGBmzYFvgFYhhEmxMscDQ4BdQggLzexE4F1gxxDCkliZK4H7gO1DCOvLfWEiIiJS6VS0\nKaU+EIClSds7xZqPZphZfzNrmLCvFV47MjK+IYQwE5gHtI9tagcsiycsMR/GznVoQpkp8YQlZjiQ\nAxxQscsSERGRymaL8r7RzAxv5vkshDAtYdf7eFPP98DewN+BoWbWPni1ThNgbQhhRdIhF8X2EXte\nnLgzhLDBzJYmlVlUxDHi+74qIubtgOOBucDq0l2piIiIAFsCewDDQwi/RBFAuZMWoD+wP3BY4sYQ\nwmsJL78xsynAbKAT8HEFzpcKxwMvRxyDiIhINjsf73eaceVKWsysH3AScEQI4aeSyoYQvjezJUBT\nPGlZCNQ2s3pJtS2NY/uIPSePJqoJNEwq0ybpdI0T9hVlLsBLL71E8+bNSwpbUqh379707ds36jCq\nFd3zzNM9zzzd88yaPn06F1xwAcS+S6NQ5qQllrB0AY4MIcwrRfldgO2AeHIzEViPjwpK7Ii7GzA2\nVmYsUN/MWiT0a+kMGDA+ocytZtYooV/LcUA+kNhclWg1QPPmzWnZsmUprlZSIScnR/c7w3TPM0/3\nPPN0zyMTWfeKMiUtZtYfH758GrDKzOI1G/khhNWxeVTuwvu0LMRrV+4HvsU7yRJCWGFmzwIPmdky\nYCXwKPB5CGFCrMwMMxsOPG1mPYDa+FDrQSGEeC3KB3hyMtDMbgZ2BO4B+oUQ1pXjXoiIiEglVtaa\nlqvwETyjkrZfCrwIbAD+AFyEjyxagCcrdyYlEr1jZV8H6gDDgKuTjtkN6IePGiqIlb02vjOEUGBm\npwADgDHAKuAFPGkSERGRKqZMSUsIocQh0iGE1cAJpTjOGqBX7FFcmeXABZs5zg/AKZs7n4iIiGQ/\nTXkvaZebmzwhsqSb7nnm6Z5nnu559VOhGXGzjZm1BCZOnDhRnbdERETKIC8vj1atWoHPVp8XRQyq\naREREZGsoKRFREREsoKSFhEREckKSlpEREQkKyhpERERkaygpEVERESygpIWERERyQpKWkRERCQr\nKGkRERGRrKCkRURERLKCkhYRERHJCkpaREREJCsoaREREZGsoKRFREREsoKSFhEREckKSlpEREQk\nKyhpERERkaygpEVERESygpIWERERyQpKWkRERCQrKGkRERGRrKCkRURERLKCkhYRERHJCkpaRERE\nJCsoaREREZGsoKRFREREsoKSFhEREckKSlpEREQiFgIUFEQdReWnpEVERCRil1wCbdoocdmcLaIO\nQEREpLr67DOYMAFefNFf9+8PPXtGG1NlppoWERGRiDz2GFx/vf/dpAm8+mq08VR2SlpEREQiMmGC\nP+fkQO/eMHEirFsXbUyVmZqHREREIvDzzzB3Lpx9NnToAK1bw++/w9Sp0KJF1NFVTkpaREREIvDF\nF/58332w117w229QsyaMH6+kpThqHhIREYlAXh40aAB77umvt94a2raFwYOjjasyU9IiIiISge++\ng2bNwKxw2w03wEcfweefp++8CxfCxx/DihXpO0e6KGkRERGJwHffwT77bLzt9NNh553hnXfSd97z\nz4ejj4bLL0/fOdJFSYuIiEiGrVnjSUvTphtvr1HDt/3wQ3rOu3YtjBnj53j7bVi8OD3nSRclLSIi\nIhk0ZAhsuSUsWbJpTQvAbrvBvHnpOfekSbB6tc8PU6MG3H13ds3Cq6RFREQkgxI72mY6afnsM9hq\nK+jcGe6/Hx5/HB54ID3nSgcNeRYREcmQggJ4//3C18nNQ+BJy4IFsH49bJHib+lx43yNo1q14Npr\nYdYsT1r23RdOPhnq1Ent+VJNNS0iIiIZ8tFH3o9kwAC49VaoX3/TMrvt5snNggWpP39eHrRqVfj6\nL3+BVauga1fv41LZqaZFREQkA1as8NWcDz8crrjCJ5Iryq67+vO8eZ7ApEp+PsyZs/HEdTvvDP/9\nrz/PmZO6c6WLkhYREZEMmDAB5s+H4cOLT1hg46QllSZP9ufk2XabNIGDDkpfP5pUUvOQiIhIBsyZ\n48nKvvuWXK5ePdh+e/j227Idf906OP5477dSlMmTfdTSfvttum/33b3GpbJT0iIiIpIBs2d7c0+t\nWpsve/DB8NVX/ve8edCpEyxbVvJ7xo6FDz6AgQM33bdyJfTvD+3bF925V0mLiIiI/M+cOb4wYmkc\nckhhc86oUfDJJz71fkmGDvXnESM23ffII9409eSTRb83nrSEULr4oqKkRUREJANmz4a99y5d2YMP\nhrlzYflymDbNt33+ua8EPX160e8ZOtT7p3z3HXz66cb7vvwSDjus6HlhwJOWVatg6dLSxRcVJS0i\nIiJpFoInLWWpaQEYP74wSfn8c7jnHp9nZdWqjcv/+CNMmQL/93/ekbdTJ09U4qZOhQMPLP58u+/u\nzzfc4OsSrVxZujgzrUxJi5ndYmYTzGyFmS0ys7fMbJMuRWZ2t5ktMLPfzGyEmTVN2l/HzB43syVm\nttLMXjezHZLKNDCzl80s38yWmdkzZlY3qcyuZjbEzFaZ2UIze8DMlIiJiEilsnSpD3kubdKy336e\nZJx9Nrz7rs/nkpfn/VVWrdq0Cej9931a/rPO8pqWrbbyZiXw8nPm+Aih4hxwABxzDLzwgjdDjR5d\nuO+55+Dvfy/L1aZPWb/gjwAeAw4FjgFqAR+Y2VbxAmZ2M9AT6A60BVYBw82sdsJxHgZOBroCHYGd\ngDeSzvUK0BzoHCvbEfhfa1wsORmKD9tuB1wMXALcXcZrEhERSatvvvHn5s1LV36LLXxhw+2399c3\n3QQNG3q/lG22gbfe2rj80KHeybZhQ5/VtkWLwpqWb77xmp6Salq23toToYICaNzYa3XA1ym6+WZ4\n8MHKsUZRmeZpCSGclPjazC4BFgOtgM9im68F7gkh/CdW5iJgEXA68JqZ1QMuA84LIXwSK3MpMN3M\n2oYQJphZc+B4oFUIYVKsTC9giJndEEJYGNu/H3BUCGEJMMXM7gDuM7M+IYT1Zb0ZIiIi6TBpkicT\nRQ03Ls622/r6QGef7bUgRx4JL7/sScuLL8Lvv8OwYd4cNGIE3HZb4XvbtPEaGvBRSGaw//6bP6eZ\nT373t795c9OZZ/rCjuA1OFGraFNKfSAASwHMbE+gCTAyXiCEsAIYD7SPbWqNJ0uJZWYC8xLKtAOW\nxROWmA9j5zo0ocyUWMISNxzIAQ6o4HWJiIikzKRJ3jxT1rWEzjrLp/Nv0wY6dPAFDg89FBYuhC5d\nPKk47DBvAjrnnML3tW7tTUKLF8N770G7dl6bUhpt2/rze+95H5n2sW/mG28sW+zpUO6kxcwMb+b5\nLIQQ69tMEzyxWJRUfFFsH0BjYG0smSmuTBO8Bud/Qggb8OQosUxR5yGhjIiISOQmTy7sXFtWO+64\n8euWLf15xAhfM2jdOk9qEkcmHXOM19Rce63PwHvuuaU/X48evjZS7do+gqlrV++cO39++eJPpYpM\n498f2B84LEWxZEzv3r3JycnZaFtubi65ubkRRSQiIlXVe+95v5IrrkjN8XbfHRo08MnmrrnGm42a\nNdu4TOPG3g/lqqt8Ft6zzir98bfdFnJyBpGTM4iff4YhQ3xCvLZt85kwITXXUF7lSlrMrB9wEnBE\nCOGnhF0LAcNrUxJrQRoDkxLK1Dazekm1LY1j++JlkkcT1QQaJpVpkxRa44R9xerbty8t46mqiIhI\nmixfDmec4YlFt26pOaaZd7QdM8abijp2LLrclVf6vrVrfUHEssjNzWX9+lyeew5GjvRz5uXl0Spx\niegIlLl5KJawdME7wG60vFII4Xs8YeicUL4e3g9lTGzTRGB9UplmwG7A2NimsUB9M0tc1qkznhCN\nTyhzkJk1SihzHJAPTENERCRio0fDhg3w9NNeO5IqF14IvXp5596SNG/uE9WV9xwff+wJS2VRppoW\nM+sP5AKnAavMLF6zkR9CWB37+2HgdjObBcwF7gF+BN4B75hrZs8CD5nZMmAl8CjweQhhQqzMDDMb\nDjxtZj2A2vhQ60GxkUMAH+DJycDYMOsdY+fqF0JYV8b7ICIiknKjRvnonj33TO1xL7kktcfLFmVt\nHroK72g7Kmn7pcCLACGEB8xsa3xOlfrAaODEEMLahPK9gQ3A60AdYBhwddIxuwH98FFDBbGy18Z3\nhhAKzOwUYABei7MKeAG4q4zXJCIikhajRsFRR1Wu2opsVtZ5WkrVnBRC6AP0KWH/GqBX7FFcmeXA\nBZs5zw/AKaWJSUREJJN+/dVHDV11VdSRVB2a8l5ERCQNvvzSZ5GNz3MiFaekRUREJA3Gj/fZa0s7\ndb9snpIWERGRNBg3zid9q1kz6kiqDiUtIiIiafD114Wz10pqKGkREREpwiOP+No75REC/PQT7LJL\namOq7ioyjb+IiEiVtHq1JyyrV8Mf/wjbbQdbbVX6969Y4aswJ68bJBWjmhYREZEkb77pa/v8/jvs\nsQd071629/8UW+BGSUtqKWkREZEq4b//9WaZiioogAce8JWNTzoJtt4aXn0VFi8u/TEWxuZub9Kk\n4vFIISUtIlKlpeJLTCq/b7+FvfaCYcMqfqy33oKvvvLmoTffhDlzoEYNOPVUmDKldMdQTUt6KGkR\nkSqroABatYJ+/aKORNLt3Xf933vIkIof65NPYL/94PDDfUHCRo3gtddg6VJvJtpcIvzKK9C3L9St\nC9tuW/F4pJCSFhGpsj7+GCZN8mep2uLJyogRFT/W/Pm+yGGi006DAQN87pX33y/5/Q8+CF98oaah\ndFDSIiJV1tNP+3Npq/QlO/3yC3z2GRxzjDcTzZpVseMtWAA777zp9mOO8dlt33ij+PeuXw/Tp/vf\nZekDI6WjpEVEqqQlS7xvwh/+4F9iq1ZFHZGkyyuv+PNTT3ntxl/+UrHjzZ8PO+1U9L6TToKhQwub\niELwUUZx330Ha9b43ytXViwO2ZSSFhGpkl580b9Q7rvPn6dNizoiSYUFCzwxAK/VCAGefx5OOQX2\n3BP++U+vCRk4sHzHLyjwTrRF1bSAJy0LF8LYsf767behYUN46CF//fXX/nz//anpFCwbU9IiIlXS\nwIHQpQsceSSYQV5e4b6Cgo1fS/bo3RvOOMMTltatoUUL77f0pz/5/txcuOQSuPxyePLJsh//55/9\n2MXVtBxxBBx0EJx7rtfgffaZb7/hBq+hmTLFRwzddBMcf3y5LlFKoKRFRKqc77+HyZPh7LN9jo1j\nj4XHHvNfwevX+0iTVq18Xg/JLuPHwzff+Oicr77yJKFLF/83Bk9Qn3gCLrgAevSAwYO9Y2xpLVjg\nz8XVtNSqBcOH+8igDh1g1Cjo2NG3Dx7sTUeHHlqhS5QSKGkRkSrn7bd9qOqJJ/rrPn38i+7gg33C\nsFGjfPvkyVFFKOWxZElhonnTTf7vO3kyvPTSxuXq1IF77/Wmo4sugptvhtmz4bLL4O67iz722rU+\nOuicc/x1cTUt4DUpo0Z5X5a8PK99Of54b4qcNAmuvLLClyrFUNIiIlXKhg3eIfPEEwvnyGjfHt55\nxztsjh7tX06gUUXZZuLEjV/36+dNNdtss2nZHXf0fatXe/Lyxz9635fnntt0npXvvoNu3aBXr8KR\nR40blxxLkyaeAIN39r79dk+W9t8fjjuufNcnm6cFE0WkSnnlFZgxY9OOmKed5s//+Edhf5Z4p0nJ\nDuPGQU6Oz5OyYYPPgFuS446DmTPhzjvhrru8qfC///XEZJ99vMx338G++3rzzhtv+PE//BC2KMW3\n45lnwgcfeNKy335+3HXrfPZcSQ8lLSIR+/13rwU491xvj5eKef116NTJO2kW5YQTPGlp3FhJSzZZ\nsMD7sXTt6jVnpXHrrZ5YdOgAF1/sQ5GbN/c+KfGkJZ7AzpoFu+3mf3fqVLrjX3KJz5bbrJm/rlXL\nH5I+ygdFIvbccz7ioW9fH64p5ReCd9Ts0KH4Ml26+BfLZZf5r+xff81cfFI+P/zgicTWW5etU23D\nhoWfhV12gb339v4n77xTWGbqVG9KiicsZVGnjidR+rGROUpaRCIWnzXz+ut92OTvv0cbTzYqKPBO\nmj/+CIsWQdu2xZdt29bLXnCBvy8+34Zs3ty56V2A8vPPfXbbZAMG+PbRoz0RqYjzzoOPPipchXnq\nVDjwwIodUzJHSYtIxGbO3Pj1999HE0e2WrUKTj/dO0YedphvKylpAahXz5sJtt/eF8eTzXv/fZ+8\nbejQ9Bz/t998gcJ99vFh6Yk+/ND7p+y9d8XPc+aZ3udk8GBPWr/5RklLNlHSIhKxGTN85dj48NuK\nrptS3Vx/vf9yvvpqb0bYdVev7t8cM59fQ0nL5hUUwDXX+N/xdXVS7dNP/XnZMu8j8tRT3gdl+XIf\nNdS5c2rOs912fqwXX4QGDbyJUElL9lDSIhKhggJf4K15cx+BsPXWSlrKYtIkn/X0wQfhkUcgP99X\n1y2tk07yGU2fey59MVYFP/1U+Ln89tvUHXfqVB8RBN45dpdd/HXbtj7Xya67+nwrBQWpS1rAR5J9\n+aU3xV51lS8BINlBSYtIhq1f7x1v//Y3H4r5++8+XNIMmjZV0lKS55+HMWP874ICH25aty5ccYVv\nq1dv8/NrJLr0UrjwQq+tSWdfjWw3b54/779/apOWXr08cVyyBP7zH5+g7dBDYdAgr9HZaSfvnJ6b\n601TqRIf/t61q/eX2WGH1B1b0ktDnkUyaOlS/2X/73/7IyfH2/BbtfL9SlqK98svPuIH4Omnfd6N\nLbbw0SGlmVOjKGY+qdjAgTBnTmr6TFRFP/zgz8ccA6+9VrFjrVrl93qvvbzj7bp1XtMxa5b/u8bt\nt5+v0t23r9e2pNIuu3iyEp/6X7KHalpEMmTWLE9KzjzT29BXrPAk5ttvvUMo+CRXEycWjmyortau\n9dqTpk19CvYffvBf4uD36IorfN6OefN8CGtFtGzpz9VtAcWCAhgxYvM1TDfdBP/6l88u3LatfzZX\nrPAawokTvQNtvCamNO64w5tCBw3yhOWmm3yYerNmvrhloj33hEcf9Rq0VLvqKiWp2UhJi0iGdO8O\n9ev7o2dP/xJInjnzT3/yuR9OO82/DEryyy/exPTee+mLOdVC8JlMN+f++/2Lsl07//V94on+C799\ne588rlYtHzEEhVOpl9cOO/gv7+Qp4qu6117zETnxDrBFWbzY+wsNHerzmMQnUZsxAx5+2Cfw23Zb\n2H33zZ8v/u++ZIk/9+7t9/2++7xfUv/+mu9ENk9Ji0gGrF3rVeG9e/sv1eIWVNt1VxgyBKZN89k2\nS/oVPGgQ3HabJzh33ZWWsFPujDM2P1IjBHjhBZ9H5aWXvIblm2/8i7N7d19PZv58ePNNHy4eH+Zc\nES1b+r9PdenXEoI3u4A3VxZn5MjCv3fd1fu0mHkH2vhswgUF/hyfb6go06d7E96NN/roIPDXzz/v\nx+veveLJp1QPSlpEMmDqVE9c2rTZfP+LFi28lmHwYH8uzqRJXvaOO7zNP97voDJ75x3/lf7RRz7K\nZ9GiTcuMGeN9Hi680F936OATgnXv7lOxgzenmXlTUSpccIF/eT/ySGqOV9lNnw4TJnitX7xjc1E+\n/LCw9mO33Xx0W9OmnrBMnOgrIj/2mO+fNKn448Rrsf7xDx9i3qOHj0g65pjUXI9UH0paRDLgiy+g\nZk04+ODSle/a1b9Ib7jB56koyuTJnrTceKN/mTz7bOriTZcGDfz5jju8f0STJr7S8qWX+j1avdqb\nyPbff+P+DYMGeRNCupoPzj4bzj8/O+5hKnz0kTexXXWVJy3x2pJEv/7qSeYFF/jr+DT3f/gDjBrl\n85ucdprPj5OTU3KfoFmzCtfkWbkS9tgDttwylVck1YWSFpEM+PJLbxbZaqvSv+f++/1LvKi1VvLy\n/Mu+RQvvU5Cb67UylaV5Y+FC+POfPf64337zpoHDDtv41/3pp3tzUNu23kfi2299peZMr5TboYPX\nAq1dm9nzpkNBQclrKn38sfcXOvVUT4oHDNi0TP/+3uH23nvhgQe8VgU8afnqK/+7bVtPJFu0KLlP\n0KxZPpQ5nviUpg+MSFGUtIhkwLRp3hejLHbayZtDBg70uV3iv4ZHjPAh0uvWwSGH+LYzzvB1YWbM\nSGnY5TZggPeZ+Ne/fJRPCL4uEHgys8UWvgBer17eFLTPPvCXv3in2I8/Ln2NVCodeKDf5+++23j7\nTz95f44GDfzfMRtcf71/foYN23Tfhg1eU3LUUZ6o9ezpfa3y8wvLzJvnycoVV3iiceONhasi/+EP\n/tylizcVgfcJ2lxNS9OmntxA+RYnFAElLSIZsXAh7Lxz2d937rneV6VWLZ9E7fnnPYkB76jburX/\n3amTV7ena12Yshg3zmM08+aHnXf2ScPifW4OPtj7Qdx7L5x1lm/LzYW//92bLdq1iybuAw7w56lT\nN97+r3/5iJdtt4V77infsefP96G76V5R+uWXvb/PY4/5KLQePTYt89ZbPtT+1FP99RVXeAI8ZYq/\nXrfO+xPl5PjotGSnnOLnee21wua6li19zax4J9tE337rfWgSkxbVtEi5hRCqzQNoCYSJEycGkUwp\nKAhhq61C6Nu37O/dsCGEhg1D2GmnEHJzQ6hVK4Qttwzh7rs3LXvSSSEcfXTF462ISZNC8HqVEPr3\nD+Hcc0O47TZ/fccd/vz774Xl16/37T/+GF3MiXbcMYTbby98XVAQwr77hnD++SE88UQIZiGMG1f2\n43bp4td+0EH+bxq3YEEIK1ZUPO74sXJyQjjggBCOOiqE55/3cy5ZUlhm7doQDjkkhM6dC7etWRPC\nFlv4v1cIIdx5p3/ORo8u/bmnTfNzjRy58fa5cws/D//+dwhTp4Zw4YUb3wPJHhMnTgxAAFqGqL7H\nozpxJBerpEUisHx54f+0y+OXX/yLZe1a/0I9/fQQ5s/ftFy/fv5lk59fsXgr4oEH/FrHjCnclp/v\nX4o77hjC9ttHF1tpHHtsCKedVvj688/9ej78MIR160Jo29aTmPXrS3e8+fNDOPtsP8aVV/rzp5/6\nvpUr/XXi+Sri6qs9wV261F/PnOnHP+64EO6/37dddZV/RsaO3fi9BxwQQo8e/nezZiH88Y9lO/f6\n9SFsvXUIDz7or9euDeHGG0P48589hn32CeGHH8p/bVI5KGlR0iJV0Ndfh3DWWf7LN4QQZszw/9JG\njUrveWfP9vO8+WZ6z1OSE07wL8lkHTt6bC1bZj6msvjLX0LYeecQzjknhAEDQrjiihB2262wZuDD\nD/06pk7d/LHWrg2hQ4cQmjQJ4b77/It91139s9GjRwgHHujH2mWXisc9enQItWuH8Ne/Fm7bsCGE\nbbf1c2y7rdeGmIXw8MObvv+880I4/PDCmpHBg8seQ4cOXrMWgic98RqWAw8s3zVJ5VMZkhb1aRFJ\nobVrvb3/9dd9yO6yZd6RE2DHHdN77r328hlLhwxJ73mK89FHPrtqUavx/vnPPkrozjszH1dZtG7t\n/U9ee81jHTTIO0PHRzK1aePPX365+WO9956Pkho82JciqFnT1056/XUfHbXjjt7xeOXKio36eu01\n79PUurV3bI6rUcP7kNSp4+c45xwfGv/HP256jIMO8nlW+vTx95VnReWjjvKVmp991h8nnujbDz+8\nPFclUoyosqUoHqimRVLkxx9DaN8+hNdfL9w2ebL3NznwQK8Ob9DAmwYGDfJfnKnqu1CS3r29pqCg\nIISXXvKYMmHpUm8Catmy6KarbPH994U1BGYh7L13YXNL3L77htCr1+aP1bVrCC1abLytoMA/O/Em\nvHff9XOVt+lk4UJv7jn/fG++Svbmm95XpVs3P8/llxd9nAULvKakZk1vZiqPqVML79tFF/nnfY89\nQhg6tHzHk8qnMtS0RJ5IZPRilbRIirRt6//1dOlSuO2ppwq/8G6+2Tve1qoVwt/+5u39BQXpj2vE\niPC/PhjxWGbOTP9533nHzzVnTvrPlU4FBSFst10IrVr5F35R19Otm3/Bl2TZshDq1AnhH/8oudyc\nOX7f3n+/5HLDhnmfpWQvv+zvX7iw5PcXFPjnYOXKksusXl3ycTbnkENCaNo0Mwm6ZF5lSFrUPCRS\nRjNm+BTo22/vc6Pk5fmKtzNnFpY5+GBvHlq3ztfO2XHHzCwGd8QRPjT6kkv8dZ06PnV6un36qc9l\nssce6T9XOpn5HCfXX+9z3+y556ZlWrf2ppT4xHk33eSPRC+95HO+dOtW8vl2393/vb76yodVX3ml\nz2uTrE8fuOUWP2aiESO8aadx481f1777wjbblFymTp2Sj7M5Q4f6is3bblux44gUZzOroIhIssGD\n/X/KN97oXyStWsGxx248g+shh/hkXHXrer+GVCzqVxp16sBFF/nkbi1awMkn+/wgjzxSttl4y+qT\nT6Bjx6qSxW94AAAgAElEQVSxSu8tt5S8/7jjvI/OqFF+zf37e3+Vn3/2fkV160K/fj752ub6MdWo\n4Z+NESN8/pTx432+mGuuKSzz008+9w14gty2rf+9YYOvDXT22eW+1JRLd78tESUtImXw++/w9NP+\nK7xNG//i2HJLXw23oMDXaTngAO8QW6OGz6K6alXh+i2Z8NhjPuPo4Yf72j733utr+7zwQnrWe1m3\nztdBitfuVHX77+81So884knEqlW+/YUX/Ll2ba91uv760h3vpJPguusKXycvPPjmm54U1anjnZ3j\nSctTT/ksw+edV5GrEckuah4SKcbXX0P79v6lATB2rH9B/PQT3H574bT8p59eONri8MN9Ovp4rcsj\nj3jZK6/MXNw1a3oMhx/us5C+8oqPWHnmmfSc7/vvvdmiefP0HL+yMfOFAocNg9tu8+R1r728du29\n93xm2VmzfIr80jjpJH/u2NFnEP7ii8J9K1bAX/8KZ57pI4T+/W9PEjds8NFNl15amMSIVAeqaREp\nxnvvebX8WWf5r9pevbz6++GHC9dhueYab46ZOtWr+JP7QJx5pj+ilJvrTVpPPeUr8qa6CSfel6dZ\ns9QetzK76y444QTYe2/v2zRvnjcLxdfiKYt99vHPUbdu/jl66imf7n+bbXzo8NKlvmjm4sWeCPXp\n401CS5ZUn9otkTglLSLFyMvzdXC++87XZ+nc2edASeys+Mgj/tyypa+xU575LTLhiiv8F/3XX6dm\nMcIQvLPpLrt40rLNNr5AX3XRsGHhPCTgzYAVEf8c1a3rzYw33gj//a/XqLRr5x12d98d7r7ba3em\nTfPPoWpZpLpR85BIMSZN8k6Sf/6zj84YOLD40RVmcMwxlbcj6lFHeV+LTz9NzfHuvttrmHJzPWnZ\nd9/Ke+3Z5MADvcbliSfg/ffhgw82XkDyppu8mejtt73fTDr6KIlUZkpaRGLWr/fZQvfeGz7/3Ptq\ntGjho0nmzs3ukRFbbumjnMaMKft7lyzxfhUFBYXbRo70BG7RInjjjerVNJRujz8O99/v/WRg46Sl\nZk149VVo1GjjEUYi1YWSFpGYN96A556DOXO8dgW82cesavyi7dChfEnL0097Z+LJkwu3zZ5d+KW5\nbBkcfXRqYhSoX99rVOIjzg49dOP922/v/Vt69sx8bCJRU9Ii1d6UKT7HxkMP+Zfvuef65HH77w/7\n7Rd1dKlz2GHeYXTkyM2X/fln76Py0Uc+fwj4kN7HH/dh3wsW+P0ZNMg7+V5+eVpDr5auu87vbVE1\nfGZqjpPqqcxJi5kdYWbvmtl8Mysws9OS9j8f2574GJpUpo6ZPW5mS8xspZm9bmY7JJVpYGYvm1m+\nmS0zs2fMrG5SmV3NbIiZrTKzhWb2gJkpEZNSW7fO+3ucdJInKj17FnawvPzyqvXFcPLJPgne6ad7\nk09J3n7bh3b37OlDcGvW9Plfevb0BfrAmy/OO89HV0nqNWigeyuSrDxf8HWBycCf8DUIivI+0Bho\nEnvkJu1/GDgZ6Ap0BHYC3kgq8wrQHOgcK9sReDK+M5acDMVHQLUDLgYuAe4uxzVJNfXxx/DLL75q\nb06OJy9nnAE9evgcGFVJ7dq+AvXq1V5DUpI33/QViKdP93lI4jUptWr5SsXgfX9ERDKpzElLCGFY\nCOHOEMI7QHG/Q9eEEH4OISyOPfLjO8ysHnAZ0DuE8EkIYRJwKXCYmbWNlWkOHA/8MYTwZQhhDNAL\nOM/MmsQOdTywH3B+CGFKCGE4cAdwtZlpKLds1siRcMcdXv1eo4bXQNSpA/Xq+dTs9etHHWHqbb89\nnHIKPP988WUmTvTp4e+805vOxo71+UCOPdbvS7xDbjZ3TBaR7JSuppROZrbIzGaYWX8za5iwrxVe\nO/K/lvUQwkxgHtA+tqkdsCyW0MR9iNfsHJpQZkoIIbGieziQAxyQ0quRKunee71JqE8fb/Lo0yfq\niDLj/PN9OPfcuZvuCwHOOcdHTf3xjz4Et2ZNH8HywQeFtSyw8VpLIiKZkI4aiffxpp7vgb2BvwND\nzax9CCHgzUVrQwgrkt63KLaP2PPixJ0hhA1mtjSpzKIijhHf91UKrkWqsB9/hBtugO7do44ks449\n1hOR4cM3XV7gu+989NTjjxc9YqpGDZ/0bOnSzMQqIpIo5UlLCOG1hJffmNkUYDbQCfg41ecrj969\ne5OTk7PRttzcXHJzk7veSFUVAsyf77PYVjc5OT78ediwTZOWMWO883H79kW/F3wxxt12S2+MIhKt\nQYMGMSip81t+fn4xpTMn7X0/Qgjfm9kSoCmetCwEaptZvaTalsaxfcSek0cT1QQaJpVpk3S6xgn7\nitW3b19atmxZ1kuRKmT5ch+6Wx2TFvB1c+67D9au9Q66BQX+GDPGm4SScnoRqWaK+iGfl5dHq1at\nIorIpb1V2sx2AbYDfoptmgisx0cFxcs0A3YDxsY2jQXqm1mLhEN1xjv+jk8oc5CZNUoocxyQD0xL\n8WVIFfPjj/5cXZOW44+HlSt9yvhrroEuXXzdm5dfLrmWRUQkSmWuaYnNldKUwpFDe5nZwcDS2OMu\nvE/Lwli5+4Fv8U6yhBBWmNmzwENmtgxYCTwKfB5CmBArM8PMhgNPm1kPoDbwGDAohBCvRfkAT04G\nmtnNwI7APUC/EMK6sl6XVC/z5/tzdU1aWrTwkURvvOHDmNet8wX5cnO9A66ISGVUnuah1ngzT4g9\n/hnb/i987pY/ABcB9YEFeLJyZ1Ii0RvYALwO1AGGAVcnnacb0A8fNVQQK3ttfGcIocDMTgEGAGOA\nVcALeNIkUqL5873vRnUdtlujhte2vPaaNxEBfPKJJy4iIpVVmZOWEMInlNysdEIpjrEGn3elVwll\nlgMXbOY4PwCnbO58Isnmz/fJ02rXjjqS6Nx7rw9hvvFG79+jhEVEKjtNwibV0o8/Vt+mobjdd/fH\nU0/5CtciIpWdkhaplr77TtPQx2kgnYhkC81pKdXS9Om+SrGIiGQPJS1S7fzyCyxapKRFRCTbKGmR\namf6dH9u3jzaOEREpGyUtEi1M22aD/ndd9+oIxERkbJQ0iLVyq+/wosvwn77QZ06UUcjIiJlodFD\nUm0sXQqnnAJTp8L770cdjYiIlJVqWqTKueceGD++8PXPP0P37t7xduZMGDkSDjssuvhERKR8lLRI\nlfLll3DnnXD22RBfRf2223y6+i5d4OuvoU3y2uAiIpIV1DwkVUK/frBsGcyY4esJLVoE//oXnHoq\nPPcc/OMfcN11UUcpIiIVoaRFsl4I0CthFavHHoN//xs+/9ynp69VC664Irr4REQkNZS0SNabM8ef\nr7/em4UOPRR++AFeftn7sxx9NNStG22MIiJScerTIllh3jxPRvr3hylTYNUq71RbUADDhoEZ3H67\nlwHvaDt/Pnz8sY8YEhGR7KeaFskKf/0rTJ4MEyb466ZNfaXms8+GgQOhRQuoX7+w/GGHwZZb+nws\n554bTcwiIpJaqmmRSm/xYnj+eR/K/OOPcOWVXvOyZo0nLOed5xPGJdpuO1i+HCZNgoYNo4lbRERS\nS0mLVHpDh3qH2osvhp13hgEDYOFC6NTJ9//1r3DggZu+TzPeiohULWoekkpvyBBo2xYaN/bXZtCg\nAdxxBxx5JOy1V7TxiYhIZqimRSqtoUPh4IPhrbeK7kx71FFw112Zj0tERKKhpEUqpbVrvQNtgwZw\n991w1VVRRyQiIlFT85BUSmPH+orMDz0ELVtGHY2IiFQGqmmRSumDD6BRIzjkkKgjERGRykI1LVKp\nLFniE8YNGQLHHgs1lFaLiEiMkhapNKZOhWOO8b8XLfJ5WUREROL0O1YqhbVr4ZxzICfHa1t22QVO\nPDHqqEREpDJRTYtEbvx4uPxy+PZbyMuDL76A7beHLfTpFBGRBPpakEhs2ACffuqz2t55p9e0vPUW\n/OEP/hAREUmmpEUi8c470LUr/OlPMGIEPPMMnHpq1FGJiEhlpj4tEonRo/25f39o0sRXaxYRESmJ\nkhaJxCefwIUXwjffwMyZsO22UUckIiKVnZIWybjly2HyZO/Psv/+SlhERKR0lLRIxr34ok8aF5+T\nRUREpDSUtEhGrV0LDz4I3brBbrtFHY2IiGQTJS2SUWPHwo8/wjXXRB2JiIhkGyUtklEffQQNG2rl\nZhERKTslLZJRH30ERx2lhRBFRKTs9NUhGbN6NYwb50mLiIhIWSlpkYz5/ntYvx4OOijqSEREJBsp\naZGMmTPHn/faK9o4REQkOylpkYz5/nuoXRt22inqSEREJBspaZGMmTMH9thDnXBFRKR89PUhKbN4\nMbz+evH758xR05CIiJSfkhZJmT59fLXmpUv99ZQpMG1a4f45c2DPPSMJTUREqgAlLZISv/0Gr7zi\nf48bB88+Cy1aQKtWMGyYT9+vmhYREamILaIOQKqGd9+F/HxfsfmTT2DgQOjaFebPh/vvh5UrYdUq\nOOGEqCMVEZFspaRFUuKdd3xq/t13h8ceg99/h1tvhVGj4OabYcMG6NABDjww6khFRCRbKWmRCvnp\nJ2jUCN5/H667DvbZxxOVk0+Ggw/2MmvWwOjRMGhQpKGKiEiWU9Ii5TZ7NjRtCqee6k1Dp57qfVjO\nP7+wzEEHQYMGXgNzzjnRxSoiItlPSYuU24QJ/vzee3DkkUWv3FyjBrz8sictmp9FREQqQkmLlFte\nnj/vsw88/DCYFV3uxBMzF5OIiFRdSlqk3PLy4Mwz4Y03oo5ERESqgzJX2JvZEWb2rpnNN7MCMzut\niDJ3m9kCM/vNzEaYWdOk/XXM7HEzW2JmK83sdTPbIalMAzN72czyzWyZmT1jZnWTyuxqZkPMbJWZ\nLTSzB8xMjRAZEIInLUU1CYmIiKRDeb7g6wKTgT8BIXmnmd0M9AS6A22BVcBwM6udUOxh4GSgK9AR\n2AlI/r3+CtAc6Bwr2xF4MuE8NYCheG1RO+Bi4BLg7nJck5TRtGmwfDm0aRN1JCIiUl2UuXkohDAM\nGAZgVmQvhmuBe0II/4mVuQhYBJwOvGZm9YDLgPNCCJ/EylwKTDeztiGECWbWHDgeaBVCmBQr0wsY\nYmY3hBAWxvbvBxwVQlgCTDGzO4D7zKxPCGF9Wa9NSm/YMNhySzjiiKgjERGR6iKlTSlmtifQBBgZ\n3xZCWAGMB9rHNrXGk6XEMjOBeQll2gHL4glLzId4zc6hCWWmxBKWuOFADnBAii5JijFsGBx1FGy1\nVdSRiIhIdZHq/h9N8MRiUdL2RbF9AI2BtbFkprgyTYDFiTtDCBuApUllijoPCWUkDVatgk8/1ZT8\nIiKSWdVy9FDv3r3JycnZaFtubi65ubkRRZQ9/vlP+PlnXwBRSYuISNU0aNAgBiVNY56fnx9RNIVS\nnbQsBAyvTUmsBWkMTEooU9vM6iXVtjSO7YuXSR5NVBNomFQmuRto44R9xerbty8tNeylzMaMgRtu\n8L/32svnZxERkaqnqB/yeXl5tGrVKqKIXEqbh0II3+MJQ+f4tljH20OBMbFNE4H1SWWaAbsBY2Ob\nxgL1zaxFwuE74wnR+IQyB5lZo4QyxwH5wLQUXZLEbNgA11/vM9uC17IUN5mciIhIOpS5piU2V0pT\nPIEA2MvMDgaWhhB+wIcz325ms4C5wD3Aj8A74B1zzexZ4CEzWwasBB4FPg8hTIiVmWFmw4GnzawH\nUBt4DBgUGzkE8AGenAyMDbPeMXaufiGEdWW9LinZgAEwbpz3ZZk/H9q1izoiERGpbsrTPNQa+Bjv\ncBuAf8a2/wu4LITwgJltjc+pUh8YDZwYQlibcIzewAbgdaAOPoT66qTzdAP64aOGCmJlr43vDCEU\nmNkpwAC8FmcV8AJwVzmuSTbj6achN1dDnEVEJDrlmaflEzbTrBRC6AP0KWH/GqBX7FFcmeXABZs5\nzw/AKSWVkYrLz4cpU+C666KOREREqjNNeS+bNX68T9vfoUPUkYiISHWmpEU267PPoFEj2HffqCMR\nEZHqTEmLFGvdOnjlFXjoITjtNI0WEhGRaClpkWL16QPnnw+tW8Ojj0YdjYiIVHdKWqRY774LF18M\no0ZB3bpRRyMiItWdkhYp0pw5MHWqpuoXEZHKQ0mLbGLIENh7b//72GOjjUVERCROSYts4ssv/fnV\nV2G77aKNRUREJE5Ji2xi1iyfk+Wcc6KOREREpJCSFtnE7NnQtGnUUYiIiGxMSYtsYtaswj4tIiIi\nlYWSFtnIihXw889KWkREpPJR0iIbmT3bn9U8JCIilY2SFtnI6NGwxRbQrFnUkYiIiGxMSYv8Twjw\n5JNw+ulQv37U0YiIiGxMSYv8z2efwbRpcNVVUUciIiKyKSUt8j9PPAH77ANHHRV1JCIiIptS0lLN\nhQCDBvlqzoMHQ/fuUEOfChERqYT09VTNXXYZdOsG333nzUJXXhl1RCIiIkXbIuoAJDpz58ILL8Bj\nj0HPnlFHIyIiUjLVtFRj770HtWvDxRdHHYmIiMjmKWmppgoK4LXXvNPttttGHY2IiMjmqXmoGsrP\n974rn38Ob70VdTQiIiKlo6SlGrrgAvjkE69p6dIl6mhERERKR0lLNZOfD8OGwcMPw1lnRR2NiIhI\n6alPSzXzwQewfj2cckrUkYiIiJSNkpZq5r334KCDYPfdo45ERESkbJS0VCMbNsDQoaplERGR7KSk\npRoZPx5++QVOPTXqSERERMpOSUs18tJL0KgRtG0bdSQiIiJlp9FD1cTLL8OAAfDgg1CzZtTRiIiI\nlJ1qWqqBESN8qv6LL4brr486GhERkfJR0lLFhQA33QQdOsCzz4JZ1BGJiIiUj5qHqrAvv/ROtwsX\nwscfq1lIRESym2paqrB33oE1a6BfPzjyyKijERERqRjVtFRho0dDp05w9dVRRyIiIlJxqmmpotas\n8XlZjjgi6khERERSQ0lLFTRvHrRsCatXQ8eOUUcjIiKSGkpaqpixY6FnT1i6FAYP9uRFRESkKlCf\nlirkhx/gsMN8mPMLL8BZZ0UdkYiISOooaalC/vMfH9Y8eTIccEDU0YiIiKSWmoeqkHff9Y63SlhE\nRKQqUtJSRSxbBh99BKedFnUkIiIi6aGkpYoYPBjWr4dzz406EhERkfRQ0lIFLFkCjz8Oxx0HO+4Y\ndTQiIiLpoY64WSwEyMuDM86AVatgwICoIxIREUkfJS1ZpqDA+67k50P//v73/vvDmDGwyy5RRyci\nIpI+SlqyzN//Drff7n9vvTW8+iqccor/LSIiUpUpackia9bAo49C9+5w7bXePKThzSIiUl0oacki\ngwbB4sXQuzfst1/U0YiIiGSWkpYsUFDgHW1vu82n5lfCIiIi1VHKhzyb2V1mVpD0mJZU5m4zW2Bm\nv5nZCDNrmrS/jpk9bmZLzGylmb1uZjsklWlgZi+bWb6ZLTOzZ8ysbqqvJ0rr18Of/gS1akGzZt75\n9sEHo45KREQkGumap2Uq0BhoEnscHt9hZjcDPYHuQFtgFTDczGonvP9h4GSgK9AR2Al4I+kcrwDN\ngc6xsh2BJ9NwLZF56SUfxnzdddCqla/gvMceUUclIiISjXQ1D60PIfxczL5rgXtCCP8BMLOLgEXA\n6cBrZlYPuAw4L4TwSazMpcB0M2sbQphgZs2B44FWIYRJsTK9gCFmdkMIYWGaritjNmyA//s/6NoV\n/vnPqKMRERGJXrpqWvYxs/lmNtvMXjKzXQHMbE+85mVkvGAIYQUwHmgf29QaT6YSy8wE5iWUaQcs\niycsMR8CATg0PZeUWV9/DXPnQq9eUUciIiJSOaQjaRkHXILXhFwF7Al8Gutv0gRPLBYlvWdRbB94\ns9LaWDJTXJkmwOLEnSGEDcDShDJZbexY2GILaNs26khEREQqh5Q3D4UQhie8nGpmE4D/AucAM1J9\nvvLo3bs3OTk5G23Lzc0lNzc3oog2NXYsHHIIbLVV1JGIiEh1M2jQIAYNGrTRtvz8/IiiKZT2Ic8h\nhHwz+xZoCowCDK9NSaxtaQzEm3oWArXNrF5SbUvj2L54meTRRDWBhgllitW3b19atmxZ9ovJoLFj\n4eSTo45CRESqo6J+yOfl5dGqVauIInJpX+XZzLbBE5YFIYTv8aSic8L+eng/lDGxTROB9UllmgG7\nAWNjm8YC9c2sRcKpOuMJ0fj0XEnmTJsGs2dDp05RRyIiIlJ5pLymxcweBN7Dm4R2Bv4PWAf8O1bk\nYeB2M5sFzAXuAX4E3gHvmGtmzwIPmdkyYCXwKPB5CGFCrMwMMxsOPG1mPYDawGPAoKowcuj552G7\n7VTTIiIikigdzUO74HOobAf8DHwGtAsh/AIQQnjAzLbG51SpD4wGTgwhrE04Rm9gA/A6UAcYBlyd\ndJ5uQD981FBBrOy1abiejHrmGV+9uXt3qF178+VFRESqCwshRB1DxphZS2DixIkTK2WfltmzfYr+\n3Fx4/HHYdtuoIxIREXEJfVpahRDyoohBaw9VIn/9KzRqBE88AVtvHXU0IiIilUvaO+JK6axY4as4\nX3utEhYREZGiKGmpJN54A9asgQsuiDoSERGRyknNQxEbMwY++cRHDB19NOyyS9QRiYiIVE5KWiIU\nAvTo4esMNWoEQ4dGHZGIiEjlpeahiMyeDVde6QnLiy9CXh40bRp1VCIiIpWXaloybOZMuPxyyMmB\nkSPh+OO9H4tZ1JGJiIhUbkpaMuzVV+Gzz/zvv/0Nbrkl2nhERESyhZqHMuzrr/25Th24+OJoYxER\nEckmSloybNw4uOkmmDsXdtop6mhERESyh5KWDJo3D+bPh/btoUmTqKMRERHJLkpaMui++6BePejY\nMepIREREso864mbIF1/Ak0/Cgw9Cw4ZRRyMiIpJ9VNOSZuvWwcMPwznnQMuW0KtX1BGJiIhkJ9W0\npFG3bjBqFPz8s0/R368f1KoVdVQiIiLZSTUtabJypc/J0rIljB4Nw4fDPvtEHZWIiEj2Uk1Lmnz5\nJRQUwP33wwEHRB2NiIhI9lNNS5qMH+8jhZo3jzoSERGRqkFJSxpMnw6DB0ObNlBDd1hERCQl1DyU\nYitXQufO3jR0111RRyMiIlJ1KGlJsVtvheXLvbZl992jjkZERKTqUNKSQs8958Oa+/VTwiIiIpJq\n6nGRIitXwl/+AhdeCFdfHXU0IiIiVY+SlhR54AHIz4d77406EhERkapJzUMVtHw59O/viyHecgvs\ntlvUEYmIiFRNqmmpoGeegTvvhJNOgttvjzoaERGRqks1LRX06adw5JHwzjtRRyIiIlK1qaalAgoK\n4LPP4Igjoo5ERESk6lPSUgHTpsGyZUpaREREMkFJSwU88QTUrw/t2kUdiYiISNWnpKUcfv0VbroJ\nnnoKbr4Z6taNOiIREZGqTx1xy2j9ejjtNPjiC7jsMrjmmqgjEhERqR6UtJTRv/8NH3/sj06doo5G\nRESk+lDzUBkUFPgkciedpIRFREQk01TTUgb/+Q988413wBUREZHMUk1LKYUAf/sbHH64P0RERCSz\nVNNSSk89BePHw8iRUUciIiJSPammpRSeeAJ69oTLL4ejj446GhERkepJSctm5OUVJiz9+0cdjYiI\nSPWlpGUzbr4Z9t8fHn0UatWKOhoREZHqS0lLCfLzYdQo6NFDCYuIiEjUlLSU4MMPfQbcE0+MOhIR\nERFR0lKM4cPh1lu9aWiPPaKORkRERJS0FOGbb+DMM6FRI3j66aijEREREdA8LUW6+WbYdVf44AOt\n4CwiIlJZKGlJ8tVXMGQIPP+8EhYREZHKRM1DCUaNglatvA9Lbm7U0YiIiEgiJS0J3n0Xdt4ZJk+G\nOnWijkZEREQSKWlJMHEiHHoo5OREHYmIiIgkU9ISU1AAkyZ585CIiIhUPkpaYmbNgpUrlbSkw6BB\ng6IOodrRPc883fPM0z2vfrI+aTGzq83sezP73czGmVmb8hxn4ECoWRNatkx1hKL/sWSe7nnm6Z5n\nnu559ZPVSYuZnQv8E7gLaAF8BQw3s0ZlOc748fD3v8Mdd0DDhmkIVERERCosq5MWoDfwZAjhxRDC\nDOAq4DfgstIe4Ouv4cILvVno1lvTFaaIiIhUVNYmLWZWC2gFjIxvCyEE4EOgfUnvDQE+/RTuugsO\nOQTWrPHmIa3kLCIiUnll84y4jYCawKKk7YuAZsW8Z0uAbt2m8+23vuGii+Dqq+HXXyEvL02RVnP5\n+fnk6eZmlO555umeZ57ueWZNnz49/ueWUcVgXjmRfcxsR2A+0D6EMD5h+/1AxxDCJrUtZtYNeDlz\nUYqIiFQ554cQXonixNlc07IE2AA0TtreGFhYzHuGA+cDc4HVaYtMRESk6tkS2AP/Lo1E1ta0AJjZ\nOGB8COHa2GsD5gGPhhAejDQ4ERERSalsrmkBeAh4wcwmAhPw0URbAy9EGZSIiIikXlYnLSGE12Jz\nstyNNwtNBo4PIfwcbWQiIiKSalndPCQiIiLVR9bO0yIiIiLVi5IWERERyQrVJmlJ1cKKVZ2ZHWFm\n75rZfDMrMLPTiihzt5ktMLPfzGyEmTVN2l/HzB43syVmttLMXjezHZLKNDCzl80s38yWmdkzZlY3\nqcyuZjbEzFaZ2UIze8DMqtxn1sxuMbMJZrbCzBaZ2Vtmtm8R5XTfU8TMrjKzr2L3Id/MxpjZCUll\ndL/TxMz+Evv/y0NJ23XPU8jM7ord58THtKQy2XXPQwhV/gGci8/LchGwH/AksBRoFHVsle0BnIB3\nbO6Cz4NzWtL+m2P37hTgQOBtYDZQO6HMAHwunCPxhSzHAKOTjvM+kAe0BjoA3wIvJeyvAUzB5wM4\nCDgeWAzcG/U9SsM9HwpcCDSPXet/YvdvK933tN3zk2Of9b2BpsC9wBqgue532u99G2AOMAl4SJ/x\ntN7ru4Cvge2BHWKPhtl8zyO/qRn6hxsHPJLw2oAfgZuijq0yP4ACNk1aFgC9E17XA34Hzkl4vQY4\nI4eJ+64AAAPqSURBVKFMs9ix2sZeN4+9bpFQ5nhgPdAk9vpEYB0JiSVwJbAM2CLqe5Pm+94odn8O\n133P6H3/BbhU9zut93gbYCZwNPAxGyctuuepv993AXkl7M+6e17lqsOSWQUWVpSNmdmeQBM2vpcr\ngPEU3svW+FD6xDIz8Un/4mXaActCCJMSDv8hEIBDE8pMCSEsSSgzHMgBDkjRJVVW9fF7sRR039PN\nzGqY2Xn4HE9jdL/T6nHgvRDCR4kbdc/Tah/z5v7ZZvaSme0K2XvPq3zSQskLKzbJfDhZrQn+QSzp\nXjYG1sY+/MWVaYJXDf5PCGED/iWdWKao80AV/nczMwMeBj4LIcTbnnXf08DMDjSzlfgvyf74r8mZ\n6H6nRSwxPAS4pYjduufpMQ64BK/5uArYE/g01t8kK+95Vk8uJ1IF9Qf2Bw6LOpBqYAZwMP5r7yzg\nRTPrGG1IVZOZ7YIn48eEENZFHU91EUJIXCNoqplNAP4LnIN//rNOdahpKc/CilK0hXh/oJLu5UKg\ntpnV20yZ5N7nNYGGSWWKOg9U0X83M+sHnAR0CiH8lLBL9z0NQgjrQwhzQgiTQgi3AV8B16L7nQ6t\n8M6geWa2zszW4R07rzWztfivbt3zNAsh5OOdZJuSpZ/zKp+0xLL6iUDn+LZYFXxnvBe0lFII4Xv8\nA5Z4L+vh7ZbxezkR74CVWKYZsBswNrZpLFDfzFokHL4z/h/Q+IQyB5kv0xB3HJAPbDRkryqIJSxd\ngKNCCPMS9+m+Z0wNoI7ud1p8iI8aOQSv3ToY+BJ4CTg4hDAH3fO0M7Nt8IRlQdZ+zqPu3ZyJB14V\n9hsbD3n+Bdg+6tgq2wOoi/8P5RC8R/h1sde7xvbfFLt3p+L/E3ob+I6Nh8j1B74HOuG/sD5n0yFy\nQ/H/abXBm0JmAgMT9tfAf/m+D/wBb5NdBNwT9T1Kwz3vj/eiPwL/9RF/bJlQRvc9tff8b7H7vTs+\n1PPv+P+cj9b9zti/QfLoId3z1N/jB4GOsc95B2BE7Fq3y9Z7HvlNzeA/3p/wsea/41lf66hjqowP\nvMq2AG9SS3w8l1CmDz5U7je8B3jTpGPUAR7Dm+ZWAoOBHZLK1Md/ZeXjX9hPA1snldkVn7Pk19gH\n/H6gRtT3KA33vKj7vQG4KKmc7nvq7vkz+Fwhv+O/Nj8glrDofmfs3+AjEpIW3fO03ONB+PQev+Mj\nfl4B9szme64FE0VERCQrVPk+LSIiIlI1KGkRERGRrKCkRURERLKCkhYRERHJCkpaREREJCsoaRER\nEZGsoKRFREREsoKSFhEREckKSlpEREQkKyhpERERkaygpEVERESywv8DuVXzrLbnACkAAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f4279148ad0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 50000/50000 [8:26:59<00:00,  2.81s/it]\n"
     ]
    }
   ],
   "source": [
    "from tqdm import trange\n",
    "from IPython.display import clear_output\n",
    "\n",
    "#the algorithm almost converges by 15k iterations, 50k is for full convergence\n",
    "for i in trange(50000):    \n",
    "    \n",
    "    #play\n",
    "    pool.update(SEQ_LENGTH)\n",
    "\n",
    "    #train\n",
    "    loss = 0.95*loss + 0.05*train_step()\n",
    "    targetnet.load_weights(0.01)\n",
    "    \n",
    "    \n",
    "    if epoch_counter%10==0:\n",
    "        #average reward per game tick in current experience replay pool\n",
    "        reward_per_tick = 0.95*reward_per_tick + 0.05*pool.experience_replay.rewards.get_value().mean()\n",
    "        print(\"iter=%i\\tloss=%.3f\\treward/tick=%.3f\"%(epoch_counter,\n",
    "                                                      loss,\n",
    "                                                      reward_per_tick))\n",
    "        \n",
    "    ##record current learning progress and show learning curves\n",
    "    if epoch_counter%100 ==0:\n",
    "        reward = 0.95*reward + 0.05*np.mean(pool.evaluate(record_video=False))\n",
    "        rewards[epoch_counter] = reward\n",
    "        \n",
    "        clear_output(True)\n",
    "        plt.plot(*zip(*sorted(rewards.items(),key=lambda (t,r):t)))\n",
    "        plt.show()\n",
    "        \n",
    "\n",
    "    \n",
    "    epoch_counter  +=1\n",
    "\n",
    "    \n",
    "# Time to drink some coffee!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating results\n",
    " * Here we plot learning curves and sample testimonials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f4275f43410>]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAi0AAAFkCAYAAADsVgtLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3XeYU2Xax/HvDQIqygCiYG8oYlmlCqiIYm+o2Aa7qygr\nqKxt7bzq7lp2xYJg1xWVVewuCCKKojRlQEGKArIoCIjAgCh1nvePO9kJYWaYkuRMZn6f68qVyTlP\nzrnPIZo7T7UQAiIiIiKVXY2oAxAREREpDSUtIiIikhWUtIiIiEhWUNIiIiIiWUFJi4iIiGQFJS0i\nIiKSFZS0iIiISFZQ0iIiIiJZQUmLiIiIZAUlLSIiIpIVypS0mNktZjbBzFaY2SIze8vM9k0q87yZ\nFSQ9hiaVqWNmj5vZEjNbaWavm9kOSWUamNnLZpZvZsvM7Bkzq5tUZlczG2Jmq8xsoZk9YGZKxERE\nRKqgsn7BHwE8BhwKHAPUAj4ws62Syr0PNAaaxB65SfsfBk4GugIdgZ2AN5LKvAI0BzrHynYEnozv\njCUnQ4EtgHbAxcAlwN1lvCYRERHJAlaRBRPNrBGwGOgYQvgstu15ICeEcGYx76kH/AycF0J4K7at\nGTAdaBdCmGBmzYFvgFYhhEmxMscDQ4BdQggLzexE4F1gxxDCkliZK4H7gO1DCOvLfWEiIiJS6VS0\nKaU+EIClSds7xZqPZphZfzNrmLCvFV47MjK+IYQwE5gHtI9tagcsiycsMR/GznVoQpkp8YQlZjiQ\nAxxQscsSERGRymaL8r7RzAxv5vkshDAtYdf7eFPP98DewN+BoWbWPni1ThNgbQhhRdIhF8X2EXte\nnLgzhLDBzJYmlVlUxDHi+74qIubtgOOBucDq0l2piIiIAFsCewDDQwi/RBFAuZMWoD+wP3BY4sYQ\nwmsJL78xsynAbKAT8HEFzpcKxwMvRxyDiIhINjsf73eaceVKWsysH3AScEQI4aeSyoYQvjezJUBT\nPGlZCNQ2s3pJtS2NY/uIPSePJqoJNEwq0ybpdI0T9hVlLsBLL71E8+bNSwpbUqh379707ds36jCq\nFd3zzNM9zzzd88yaPn06F1xwAcS+S6NQ5qQllrB0AY4MIcwrRfldgO2AeHIzEViPjwpK7Ii7GzA2\nVmYsUN/MWiT0a+kMGDA+ocytZtYooV/LcUA+kNhclWg1QPPmzWnZsmUprlZSIScnR/c7w3TPM0/3\nPPN0zyMTWfeKMiUtZtYfH758GrDKzOI1G/khhNWxeVTuwvu0LMRrV+4HvsU7yRJCWGFmzwIPmdky\nYCXwKPB5CGFCrMwMMxsOPG1mPYDa+FDrQSGEeC3KB3hyMtDMbgZ2BO4B+oUQ1pXjXoiIiEglVtaa\nlqvwETyjkrZfCrwIbAD+AFyEjyxagCcrdyYlEr1jZV8H6gDDgKuTjtkN6IePGiqIlb02vjOEUGBm\npwADgDHAKuAFPGkSERGRKqZMSUsIocQh0iGE1cAJpTjOGqBX7FFcmeXABZs5zg/AKZs7n4iIiGQ/\nTXkvaZebmzwhsqSb7nnm6Z5nnu559VOhGXGzjZm1BCZOnDhRnbdERETKIC8vj1atWoHPVp8XRQyq\naREREZGsoKRFREREsoKSFhEREckKSlpEREQkKyhpERERkaygpEVERESygpIWERERyQpKWkRERCQr\nKGkRERGRrKCkRURERLKCkhYRERHJCkpaREREJCsoaREREZGsoKRFREREsoKSFhEREckKSlpEREQk\nKyhpERERkaygpEVERESygpIWERERyQpKWkRERCQrKGkRERGRrKCkRURERLKCkhYRERHJCkpaRERE\nJCsoaREREZGsoKRFREREsoKSFhEREckKSlpEREQiFgIUFEQdReWnpEVERCRil1wCbdoocdmcLaIO\nQEREpLr67DOYMAFefNFf9+8PPXtGG1NlppoWERGRiDz2GFx/vf/dpAm8+mq08VR2SlpEREQiMmGC\nP+fkQO/eMHEirFsXbUyVmZqHREREIvDzzzB3Lpx9NnToAK1bw++/w9Sp0KJF1NFVTkpaREREIvDF\nF/58332w117w229QsyaMH6+kpThqHhIREYlAXh40aAB77umvt94a2raFwYOjjasyU9IiIiISge++\ng2bNwKxw2w03wEcfweefp++8CxfCxx/DihXpO0e6KGkRERGJwHffwT77bLzt9NNh553hnXfSd97z\nz4ejj4bLL0/fOdJFSYuIiEiGrVnjSUvTphtvr1HDt/3wQ3rOu3YtjBnj53j7bVi8OD3nSRclLSIi\nIhk0ZAhsuSUsWbJpTQvAbrvBvHnpOfekSbB6tc8PU6MG3H13ds3Cq6RFREQkgxI72mY6afnsM9hq\nK+jcGe6/Hx5/HB54ID3nSgcNeRYREcmQggJ4//3C18nNQ+BJy4IFsH49bJHib+lx43yNo1q14Npr\nYdYsT1r23RdOPhnq1Ent+VJNNS0iIiIZ8tFH3o9kwAC49VaoX3/TMrvt5snNggWpP39eHrRqVfj6\nL3+BVauga1fv41LZqaZFREQkA1as8NWcDz8crrjCJ5Iryq67+vO8eZ7ApEp+PsyZs/HEdTvvDP/9\nrz/PmZO6c6WLkhYREZEMmDAB5s+H4cOLT1hg46QllSZP9ufk2XabNIGDDkpfP5pUUvOQiIhIBsyZ\n48nKvvuWXK5ePdh+e/j227Idf906OP5477dSlMmTfdTSfvttum/33b3GpbJT0iIiIpIBs2d7c0+t\nWpsve/DB8NVX/ve8edCpEyxbVvJ7xo6FDz6AgQM33bdyJfTvD+3bF925V0mLiIiI/M+cOb4wYmkc\nckhhc86oUfDJJz71fkmGDvXnESM23ffII9409eSTRb83nrSEULr4oqKkRUREJANmz4a99y5d2YMP\nhrlzYflymDbNt33+ua8EPX160e8ZOtT7p3z3HXz66cb7vvwSDjus6HlhwJOWVatg6dLSxRcVJS0i\nIiJpFoInLWWpaQEYP74wSfn8c7jnHp9nZdWqjcv/+CNMmQL/93/ekbdTJ09U4qZOhQMPLP58u+/u\nzzfc4OsSrVxZujgzrUxJi5ndYmYTzGyFmS0ys7fMbJMuRWZ2t5ktMLPfzGyEmTVN2l/HzB43syVm\nttLMXjezHZLKNDCzl80s38yWmdkzZlY3qcyuZjbEzFaZ2UIze8DMlIiJiEilsnSpD3kubdKy336e\nZJx9Nrz7rs/nkpfn/VVWrdq0Cej9931a/rPO8pqWrbbyZiXw8nPm+Aih4hxwABxzDLzwgjdDjR5d\nuO+55+Dvfy/L1aZPWb/gjwAeAw4FjgFqAR+Y2VbxAmZ2M9AT6A60BVYBw82sdsJxHgZOBroCHYGd\ngDeSzvUK0BzoHCvbEfhfa1wsORmKD9tuB1wMXALcXcZrEhERSatvvvHn5s1LV36LLXxhw+2399c3\n3QQNG3q/lG22gbfe2rj80KHeybZhQ5/VtkWLwpqWb77xmp6Salq23toToYICaNzYa3XA1ym6+WZ4\n8MHKsUZRmeZpCSGclPjazC4BFgOtgM9im68F7gkh/CdW5iJgEXA68JqZ1QMuA84LIXwSK3MpMN3M\n2oYQJphZc+B4oFUIYVKsTC9giJndEEJYGNu/H3BUCGEJMMXM7gDuM7M+IYT1Zb0ZIiIi6TBpkicT\nRQ03Ls622/r6QGef7bUgRx4JL7/sScuLL8Lvv8OwYd4cNGIE3HZb4XvbtPEaGvBRSGaw//6bP6eZ\nT373t795c9OZZ/rCjuA1OFGraFNKfSAASwHMbE+gCTAyXiCEsAIYD7SPbWqNJ0uJZWYC8xLKtAOW\nxROWmA9j5zo0ocyUWMISNxzIAQ6o4HWJiIikzKRJ3jxT1rWEzjrLp/Nv0wY6dPAFDg89FBYuhC5d\nPKk47DBvAjrnnML3tW7tTUKLF8N770G7dl6bUhpt2/rze+95H5n2sW/mG28sW+zpUO6kxcwMb+b5\nLIQQ69tMEzyxWJRUfFFsH0BjYG0smSmuTBO8Bud/Qggb8OQosUxR5yGhjIiISOQmTy7sXFtWO+64\n8euWLf15xAhfM2jdOk9qEkcmHXOM19Rce63PwHvuuaU/X48evjZS7do+gqlrV++cO39++eJPpYpM\n498f2B84LEWxZEzv3r3JycnZaFtubi65ubkRRSQiIlXVe+95v5IrrkjN8XbfHRo08MnmrrnGm42a\nNdu4TOPG3g/lqqt8Ft6zzir98bfdFnJyBpGTM4iff4YhQ3xCvLZt85kwITXXUF7lSlrMrB9wEnBE\nCOGnhF0LAcNrUxJrQRoDkxLK1Dazekm1LY1j++JlkkcT1QQaJpVpkxRa44R9xerbty8t46mqiIhI\nmixfDmec4YlFt26pOaaZd7QdM8abijp2LLrclVf6vrVrfUHEssjNzWX9+lyeew5GjvRz5uXl0Spx\niegIlLl5KJawdME7wG60vFII4Xs8YeicUL4e3g9lTGzTRGB9UplmwG7A2NimsUB9M0tc1qkznhCN\nTyhzkJk1SihzHJAPTENERCRio0fDhg3w9NNeO5IqF14IvXp5596SNG/uE9WV9xwff+wJS2VRppoW\nM+sP5AKnAavMLF6zkR9CWB37+2HgdjObBcwF7gF+BN4B75hrZs8CD5nZMmAl8CjweQhhQqzMDDMb\nDjxtZj2A2vhQ60GxkUMAH+DJycDYMOsdY+fqF0JYV8b7ICIiknKjRvnonj33TO1xL7kktcfLFmVt\nHroK72g7Kmn7pcCLACGEB8xsa3xOlfrAaODEEMLahPK9gQ3A60AdYBhwddIxuwH98FFDBbGy18Z3\nhhAKzOwUYABei7MKeAG4q4zXJCIikhajRsFRR1Wu2opsVtZ5WkrVnBRC6AP0KWH/GqBX7FFcmeXA\nBZs5zw/AKaWJSUREJJN+/dVHDV11VdSRVB2a8l5ERCQNvvzSZ5GNz3MiFaekRUREJA3Gj/fZa0s7\ndb9snpIWERGRNBg3zid9q1kz6kiqDiUtIiIiafD114Wz10pqKGkREREpwiOP+No75REC/PQT7LJL\namOq7ioyjb+IiEiVtHq1JyyrV8Mf/wjbbQdbbVX6969Y4aswJ68bJBWjmhYREZEkb77pa/v8/jvs\nsQd071629/8UW+BGSUtqKWkREZEq4b//9WaZiioogAce8JWNTzoJtt4aXn0VFi8u/TEWxuZub9Kk\n4vFIISUtIlKlpeJLTCq/b7+FvfaCYcMqfqy33oKvvvLmoTffhDlzoEYNOPVUmDKldMdQTUt6KGkR\nkSqroABatYJ+/aKORNLt3Xf933vIkIof65NPYL/94PDDfUHCRo3gtddg6VJvJtpcIvzKK9C3L9St\nC9tuW/F4pJCSFhGpsj7+GCZN8mep2uLJyogRFT/W/Pm+yGGi006DAQN87pX33y/5/Q8+CF98oaah\ndFDSIiJV1tNP+3Npq/QlO/3yC3z2GRxzjDcTzZpVseMtWAA777zp9mOO8dlt33ij+PeuXw/Tp/vf\nZekDI6WjpEVEqqQlS7xvwh/+4F9iq1ZFHZGkyyuv+PNTT3ntxl/+UrHjzZ8PO+1U9L6TToKhQwub\niELwUUZx330Ha9b43ytXViwO2ZSSFhGpkl580b9Q7rvPn6dNizoiSYUFCzwxAK/VCAGefx5OOQX2\n3BP++U+vCRk4sHzHLyjwTrRF1bSAJy0LF8LYsf767behYUN46CF//fXX/nz//anpFCwbU9IiIlXS\nwIHQpQsceSSYQV5e4b6Cgo1fS/bo3RvOOMMTltatoUUL77f0pz/5/txcuOQSuPxyePLJsh//55/9\n2MXVtBxxBBx0EJx7rtfgffaZb7/hBq+hmTLFRwzddBMcf3y5LlFKoKRFRKqc77+HyZPh7LN9jo1j\nj4XHHvNfwevX+0iTVq18Xg/JLuPHwzff+Oicr77yJKFLF/83Bk9Qn3gCLrgAevSAwYO9Y2xpLVjg\nz8XVtNSqBcOH+8igDh1g1Cjo2NG3Dx7sTUeHHlqhS5QSKGkRkSrn7bd9qOqJJ/rrPn38i+7gg33C\nsFGjfPvkyVFFKOWxZElhonnTTf7vO3kyvPTSxuXq1IF77/Wmo4sugptvhtmz4bLL4O67iz722rU+\nOuicc/x1cTUt4DUpo0Z5X5a8PK99Of54b4qcNAmuvLLClyrFUNIiIlXKhg3eIfPEEwvnyGjfHt55\nxztsjh7tX06gUUXZZuLEjV/36+dNNdtss2nZHXf0fatXe/Lyxz9635fnntt0npXvvoNu3aBXr8KR\nR40blxxLkyaeAIN39r79dk+W9t8fjjuufNcnm6cFE0WkSnnlFZgxY9OOmKed5s//+Edhf5Z4p0nJ\nDuPGQU6Oz5OyYYPPgFuS446DmTPhzjvhrru8qfC///XEZJ99vMx338G++3rzzhtv+PE//BC2KMW3\n45lnwgcfeNKy335+3HXrfPZcSQ8lLSIR+/13rwU491xvj5eKef116NTJO2kW5YQTPGlp3FhJSzZZ\nsMD7sXTt6jVnpXHrrZ5YdOgAF1/sQ5GbN/c+KfGkJZ7AzpoFu+3mf3fqVLrjX3KJz5bbrJm/rlXL\nH5I+ygdFIvbccz7ioW9fH64p5ReCd9Ts0KH4Ml26+BfLZZf5r+xff81cfFI+P/zgicTWW5etU23D\nhoWfhV12gb339v4n77xTWGbqVG9KiicsZVGnjidR+rGROUpaRCIWnzXz+ut92OTvv0cbTzYqKPBO\nmj/+CIsWQdu2xZdt29bLXnCBvy8+34Zs3ty56V2A8vPPfXbbZAMG+PbRoz0RqYjzzoOPPipchXnq\nVDjwwIodUzJHSYtIxGbO3Pj1999HE0e2WrUKTj/dO0YedphvKylpAahXz5sJtt/eF8eTzXv/fZ+8\nbejQ9Bz/t998gcJ99vFh6Yk+/ND7p+y9d8XPc+aZ3udk8GBPWr/5RklLNlHSIhKxGTN85dj48NuK\nrptS3Vx/vf9yvvpqb0bYdVev7t8cM59fQ0nL5hUUwDXX+N/xdXVS7dNP/XnZMu8j8tRT3gdl+XIf\nNdS5c2rOs912fqwXX4QGDbyJUElL9lDSIhKhggJf4K15cx+BsPXWSlrKYtIkn/X0wQfhkUcgP99X\n1y2tk07yGU2fey59MVYFP/1U+Ln89tvUHXfqVB8RBN45dpdd/HXbtj7Xya67+nwrBQWpS1rAR5J9\n+aU3xV51lS8BINlBSYtIhq1f7x1v//Y3H4r5++8+XNIMmjZV0lKS55+HMWP874ICH25aty5ccYVv\nq1dv8/NrJLr0UrjwQq+tSWdfjWw3b54/779/apOWXr08cVyyBP7zH5+g7dBDYdAgr9HZaSfvnJ6b\n601TqRIf/t61q/eX2WGH1B1b0ktDnkUyaOlS/2X/73/7IyfH2/BbtfL9SlqK98svPuIH4Omnfd6N\nLbbw0SGlmVOjKGY+qdjAgTBnTmr6TFRFP/zgz8ccA6+9VrFjrVrl93qvvbzj7bp1XtMxa5b/u8bt\nt5+v0t23r9e2pNIuu3iyEp/6X7KHalpEMmTWLE9KzjzT29BXrPAk5ttvvUMo+CRXEycWjmyortau\n9dqTpk19CvYffvBf4uD36IorfN6OefN8CGtFtGzpz9VtAcWCAhgxYvM1TDfdBP/6l88u3LatfzZX\nrPAawokTvQNtvCamNO64w5tCBw3yhOWmm3yYerNmvrhloj33hEcf9Rq0VLvqKiWp2UhJi0iGdO8O\n9ev7o2dP/xJInjnzT3/yuR9OO82/DEryyy/exPTee+mLOdVC8JlMN+f++/2Lsl07//V94on+C799\ne588rlYtHzEEhVOpl9cOO/gv7+Qp4qu6117zETnxDrBFWbzY+wsNHerzmMQnUZsxAx5+2Cfw23Zb\n2H33zZ8v/u++ZIk/9+7t9/2++7xfUv/+mu9ENk9Ji0gGrF3rVeG9e/sv1eIWVNt1VxgyBKZN89k2\nS/oVPGgQ3HabJzh33ZWWsFPujDM2P1IjBHjhBZ9H5aWXvIblm2/8i7N7d19PZv58ePNNHy4eH+Zc\nES1b+r9PdenXEoI3u4A3VxZn5MjCv3fd1fu0mHkH2vhswgUF/hyfb6go06d7E96NN/roIPDXzz/v\nx+veveLJp1QPSlpEMmDqVE9c2rTZfP+LFi28lmHwYH8uzqRJXvaOO7zNP97voDJ75x3/lf7RRz7K\nZ9GiTcuMGeN9Hi680F936OATgnXv7lOxgzenmXlTUSpccIF/eT/ySGqOV9lNnw4TJnitX7xjc1E+\n/LCw9mO33Xx0W9OmnrBMnOgrIj/2mO+fNKn448Rrsf7xDx9i3qOHj0g65pjUXI9UH0paRDLgiy+g\nZk04+ODSle/a1b9Ib7jB56koyuTJnrTceKN/mTz7bOriTZcGDfz5jju8f0STJr7S8qWX+j1avdqb\nyPbff+P+DYMGeRNCupoPzj4bzj8/O+5hKnz0kTexXXWVJy3x2pJEv/7qSeYFF/jr+DT3f/gDjBrl\n85ucdprPj5OTU3KfoFmzCtfkWbkS9tgDttwylVck1YWSFpEM+PJLbxbZaqvSv+f++/1LvKi1VvLy\n/Mu+RQvvU5Cb67UylaV5Y+FC+POfPf64337zpoHDDtv41/3pp3tzUNu23kfi2299peZMr5TboYPX\nAq1dm9nzpkNBQclrKn38sfcXOvVUT4oHDNi0TP/+3uH23nvhgQe8VgU8afnqK/+7bVtPJFu0KLlP\n0KxZPpQ5nviUpg+MSFGUtIhkwLRp3hejLHbayZtDBg70uV3iv4ZHjPAh0uvWwSGH+LYzzvB1YWbM\nSGnY5TZggPeZ+Ne/fJRPCL4uEHgys8UWvgBer17eFLTPPvCXv3in2I8/Ln2NVCodeKDf5+++23j7\nTz95f44GDfzfMRtcf71/foYN23Tfhg1eU3LUUZ6o9ezpfa3y8wvLzJvnycoVV3iiceONhasi/+EP\n/tylizcVgfcJ2lxNS9OmntxA+RYnFAElLSIZsXAh7Lxz2d937rneV6VWLZ9E7fnnPYkB76jburX/\n3amTV7ena12Yshg3zmM08+aHnXf2ScPifW4OPtj7Qdx7L5x1lm/LzYW//92bLdq1iybuAw7w56lT\nN97+r3/5iJdtt4V77infsefP96G76V5R+uWXvb/PY4/5KLQePTYt89ZbPtT+1FP99RVXeAI8ZYq/\nXrfO+xPl5PjotGSnnOLnee21wua6li19zax4J9tE337rfWgSkxbVtEi5hRCqzQNoCYSJEycGkUwp\nKAhhq61C6Nu37O/dsCGEhg1D2GmnEHJzQ6hVK4Qttwzh7rs3LXvSSSEcfXTF462ISZNC8HqVEPr3\nD+Hcc0O47TZ/fccd/vz774Xl16/37T/+GF3MiXbcMYTbby98XVAQwr77hnD++SE88UQIZiGMG1f2\n43bp4td+0EH+bxq3YEEIK1ZUPO74sXJyQjjggBCOOiqE55/3cy5ZUlhm7doQDjkkhM6dC7etWRPC\nFlv4v1cIIdx5p3/ORo8u/bmnTfNzjRy58fa5cws/D//+dwhTp4Zw4YUb3wPJHhMnTgxAAFqGqL7H\nozpxJBerpEUisHx54f+0y+OXX/yLZe1a/0I9/fQQ5s/ftFy/fv5lk59fsXgr4oEH/FrHjCnclp/v\nX4o77hjC9ttHF1tpHHtsCKedVvj688/9ej78MIR160Jo29aTmPXrS3e8+fNDOPtsP8aVV/rzp5/6\nvpUr/XXi+Sri6qs9wV261F/PnOnHP+64EO6/37dddZV/RsaO3fi9BxwQQo8e/nezZiH88Y9lO/f6\n9SFsvXUIDz7or9euDeHGG0P48589hn32CeGHH8p/bVI5KGlR0iJV0Ndfh3DWWf7LN4QQZszw/9JG\njUrveWfP9vO8+WZ6z1OSE07wL8lkHTt6bC1bZj6msvjLX0LYeecQzjknhAEDQrjiihB2262wZuDD\nD/06pk7d/LHWrg2hQ4cQmjQJ4b77/It91139s9GjRwgHHujH2mWXisc9enQItWuH8Ne/Fm7bsCGE\nbbf1c2y7rdeGmIXw8MObvv+880I4/PDCmpHBg8seQ4cOXrMWgic98RqWAw8s3zVJ5VMZkhb1aRFJ\nobVrvb3/9dd9yO6yZd6RE2DHHdN77r328hlLhwxJ73mK89FHPrtqUavx/vnPPkrozjszH1dZtG7t\n/U9ee81jHTTIO0PHRzK1aePPX365+WO9956Pkho82JciqFnT1056/XUfHbXjjt7xeOXKio36eu01\n79PUurV3bI6rUcP7kNSp4+c45xwfGv/HP256jIMO8nlW+vTx95VnReWjjvKVmp991h8nnujbDz+8\nPFclUoyosqUoHqimRVLkxx9DaN8+hNdfL9w2ebL3NznwQK8Ob9DAmwYGDfJfnKnqu1CS3r29pqCg\nIISXXvKYMmHpUm8Catmy6KarbPH994U1BGYh7L13YXNL3L77htCr1+aP1bVrCC1abLytoMA/O/Em\nvHff9XOVt+lk4UJv7jn/fG++Svbmm95XpVs3P8/llxd9nAULvKakZk1vZiqPqVML79tFF/nnfY89\nQhg6tHzHk8qnMtS0RJ5IZPRilbRIirRt6//1dOlSuO2ppwq/8G6+2Tve1qoVwt/+5u39BQXpj2vE\niPC/PhjxWGbOTP9533nHzzVnTvrPlU4FBSFst10IrVr5F35R19Otm3/Bl2TZshDq1AnhH/8oudyc\nOX7f3n+/5HLDhnmfpWQvv+zvX7iw5PcXFPjnYOXKksusXl3ycTbnkENCaNo0Mwm6ZF5lSFrUPCRS\nRjNm+BTo22/vc6Pk5fmKtzNnFpY5+GBvHlq3ztfO2XHHzCwGd8QRPjT6kkv8dZ06PnV6un36qc9l\nssce6T9XOpn5HCfXX+9z3+y556ZlWrf2ppT4xHk33eSPRC+95HO+dOtW8vl2393/vb76yodVX3ml\nz2uTrE8fuOUWP2aiESO8aadx481f1777wjbblFymTp2Sj7M5Q4f6is3bblux44gUZzOroIhIssGD\n/X/KN97oXyStWsGxx248g+shh/hkXHXrer+GVCzqVxp16sBFF/nkbi1awMkn+/wgjzxSttl4y+qT\nT6Bjx6qSxW94AAAgAElEQVSxSu8tt5S8/7jjvI/OqFF+zf37e3+Vn3/2fkV160K/fj752ub6MdWo\n4Z+NESN8/pTx432+mGuuKSzz008+9w14gty2rf+9YYOvDXT22eW+1JRLd78tESUtImXw++/w9NP+\nK7xNG//i2HJLXw23oMDXaTngAO8QW6OGz6K6alXh+i2Z8NhjPuPo4Yf72j733utr+7zwQnrWe1m3\nztdBitfuVHX77+81So884knEqlW+/YUX/Ll2ba91uv760h3vpJPguusKXycvPPjmm54U1anjnZ3j\nSctTT/ksw+edV5GrEckuah4SKcbXX0P79v6lATB2rH9B/PQT3H574bT8p59eONri8MN9Ovp4rcsj\nj3jZK6/MXNw1a3oMhx/us5C+8oqPWHnmmfSc7/vvvdmiefP0HL+yMfOFAocNg9tu8+R1r728du29\n93xm2VmzfIr80jjpJH/u2NFnEP7ii8J9K1bAX/8KZ57pI4T+/W9PEjds8NFNl15amMSIVAeqaREp\nxnvvebX8WWf5r9pevbz6++GHC9dhueYab46ZOtWr+JP7QJx5pj+ilJvrTVpPPeUr8qa6CSfel6dZ\ns9QetzK76y444QTYe2/v2zRvnjcLxdfiKYt99vHPUbdu/jl66imf7n+bbXzo8NKlvmjm4sWeCPXp\n401CS5ZUn9otkTglLSLFyMvzdXC++87XZ+nc2edASeys+Mgj/tyypa+xU575LTLhiiv8F/3XX6dm\nMcIQvLPpLrt40rLNNr5AX3XRsGHhPCTgzYAVEf8c1a3rzYw33gj//a/XqLRr5x12d98d7r7ba3em\nTfPPoWpZpLpR85BIMSZN8k6Sf/6zj84YOLD40RVmcMwxlbcj6lFHeV+LTz9NzfHuvttrmHJzPWnZ\nd9/Ke+3Z5MADvcbliSfg/ffhgw82XkDyppu8mejtt73fTDr6KIlUZkpaRGLWr/fZQvfeGz7/3Ptq\ntGjho0nmzs3ukRFbbumjnMaMKft7lyzxfhUFBYXbRo70BG7RInjjjerVNJRujz8O99/v/WRg46Sl\nZk149VVo1GjjEUYi1YWSFpGYN96A556DOXO8dgW82cesavyi7dChfEnL0097Z+LJkwu3zZ5d+KW5\nbBkcfXRqYhSoX99rVOIjzg49dOP922/v/Vt69sx8bCJRU9Ii1d6UKT7HxkMP+Zfvuef65HH77w/7\n7Rd1dKlz2GHeYXTkyM2X/fln76Py0Uc+fwj4kN7HH/dh3wsW+P0ZNMg7+V5+eVpDr5auu87vbVE1\nfGZqjpPqqcxJi5kdYWbvmtl8Mysws9OS9j8f2574GJpUpo6ZPW5mS8xspZm9bmY7JJVpYGYvm1m+\nmS0zs2fMrG5SmV3NbIiZrTKzhWb2gJkpEZNSW7fO+3ucdJInKj17FnawvPzyqvXFcPLJPgne6ad7\nk09J3n7bh3b37OlDcGvW9Plfevb0BfrAmy/OO89HV0nqNWigeyuSrDxf8HWBycCf8DUIivI+0Bho\nEnvkJu1/GDgZ6Ap0BHYC3kgq8wrQHOgcK9sReDK+M5acDMVHQLUDLgYuAe4uxzVJNfXxx/DLL75q\nb06OJy9nnAE9evgcGFVJ7dq+AvXq1V5DUpI33/QViKdP93lI4jUptWr5SsXgfX9ERDKpzElLCGFY\nCOHOEMI7QHG/Q9eEEH4OISyOPfLjO8ysHnAZ0DuE8EkIYRJwKXCYmbWNlWkOHA/8MYTwZQhhDNAL\nOM/MmsQOdTywH3B+CGFKCGE4cAdwtZlpKLds1siRcMcdXv1eo4bXQNSpA/Xq+dTs9etHHWHqbb89\nnHIKPP988WUmTvTp4e+805vOxo71+UCOPdbvS7xDbjZ3TBaR7JSuppROZrbIzGaYWX8za5iwrxVe\nO/K/lvUQwkxgHtA+tqkdsCyW0MR9iNfsHJpQZkoIIbGieziQAxyQ0quRKunee71JqE8fb/Lo0yfq\niDLj/PN9OPfcuZvuCwHOOcdHTf3xjz4Et2ZNH8HywQeFtSyw8VpLIiKZkI4aiffxpp7vgb2BvwND\nzax9CCHgzUVrQwgrkt63KLaP2PPixJ0hhA1mtjSpzKIijhHf91UKrkWqsB9/hBtugO7do44ks449\n1hOR4cM3XV7gu+989NTjjxc9YqpGDZ/0bOnSzMQqIpIo5UlLCOG1hJffmNkUYDbQCfg41ecrj969\ne5OTk7PRttzcXHJzk7veSFUVAsyf77PYVjc5OT78ediwTZOWMWO883H79kW/F3wxxt12S2+MIhKt\nQYMGMSip81t+fn4xpTMn7X0/Qgjfm9kSoCmetCwEaptZvaTalsaxfcSek0cT1QQaJpVpk3S6xgn7\nitW3b19atmxZ1kuRKmT5ch+6Wx2TFvB1c+67D9au9Q66BQX+GDPGm4SScnoRqWaK+iGfl5dHq1at\nIorIpb1V2sx2AbYDfoptmgisx0cFxcs0A3YDxsY2jQXqm1mLhEN1xjv+jk8oc5CZNUoocxyQD0xL\n8WVIFfPjj/5cXZOW44+HlSt9yvhrroEuXXzdm5dfLrmWRUQkSmWuaYnNldKUwpFDe5nZwcDS2OMu\nvE/Lwli5+4Fv8U6yhBBWmNmzwENmtgxYCTwKfB5CmBArM8PMhgNPm1kPoDbwGDAohBCvRfkAT04G\nmtnNwI7APUC/EMK6sl6XVC/z5/tzdU1aWrTwkURvvOHDmNet8wX5cnO9A66ISGVUnuah1ngzT4g9\n/hnb/i987pY/ABcB9YEFeLJyZ1Ii0RvYALwO1AGGAVcnnacb0A8fNVQQK3ttfGcIocDMTgEGAGOA\nVcALeNIkUqL5873vRnUdtlujhte2vPaaNxEBfPKJJy4iIpVVmZOWEMInlNysdEIpjrEGn3elVwll\nlgMXbOY4PwCnbO58Isnmz/fJ02rXjjqS6Nx7rw9hvvFG79+jhEVEKjtNwibV0o8/Vt+mobjdd/fH\nU0/5CtciIpWdkhaplr77TtPQx2kgnYhkC81pKdXS9Om+SrGIiGQPJS1S7fzyCyxapKRFRCTbKGmR\namf6dH9u3jzaOEREpGyUtEi1M22aD/ndd9+oIxERkbJQ0iLVyq+/wosvwn77QZ06UUcjIiJlodFD\nUm0sXQqnnAJTp8L770cdjYiIlJVqWqTKueceGD++8PXPP0P37t7xduZMGDkSDjssuvhERKR8lLRI\nlfLll3DnnXD22RBfRf2223y6+i5d4OuvoU3y2uAiIpIV1DwkVUK/frBsGcyY4esJLVoE//oXnHoq\nPPcc/OMfcN11UUcpIiIVoaRFsl4I0CthFavHHoN//xs+/9ynp69VC664Irr4REQkNZS0SNabM8ef\nr7/em4UOPRR++AFeftn7sxx9NNStG22MIiJScerTIllh3jxPRvr3hylTYNUq71RbUADDhoEZ3H67\nlwHvaDt/Pnz8sY8YEhGR7KeaFskKf/0rTJ4MEyb466ZNfaXms8+GgQOhRQuoX7+w/GGHwZZb+nws\n554bTcwiIpJaqmmRSm/xYnj+eR/K/OOPcOWVXvOyZo0nLOed5xPGJdpuO1i+HCZNgoYNo4lbRERS\nS0mLVHpDh3qH2osvhp13hgEDYOFC6NTJ9//1r3DggZu+TzPeiohULWoekkpvyBBo2xYaN/bXZtCg\nAdxxBxx5JOy1V7TxiYhIZqimRSqtoUPh4IPhrbeK7kx71FFw112Zj0tERKKhpEUqpbVrvQNtgwZw\n991w1VVRRyQiIlFT85BUSmPH+orMDz0ELVtGHY2IiFQGqmmRSumDD6BRIzjkkKgjERGRykI1LVKp\nLFniE8YNGQLHHgs1lFaLiEiMkhapNKZOhWOO8b8XLfJ5WUREROL0O1YqhbVr4ZxzICfHa1t22QVO\nPDHqqEREpDJRTYtEbvx4uPxy+PZbyMuDL76A7beHLfTpFBGRBPpakEhs2ACffuqz2t55p9e0vPUW\n/OEP/hAREUmmpEUi8c470LUr/OlPMGIEPPMMnHpq1FGJiEhlpj4tEonRo/25f39o0sRXaxYRESmJ\nkhaJxCefwIUXwjffwMyZsO22UUckIiKVnZIWybjly2HyZO/Psv/+SlhERKR0lLRIxr34ok8aF5+T\nRUREpDSUtEhGrV0LDz4I3brBbrtFHY2IiGQTJS2SUWPHwo8/wjXXRB2JiIhkGyUtklEffQQNG2rl\nZhERKTslLZJRH30ERx2lhRBFRKTs9NUhGbN6NYwb50mLiIhIWSlpkYz5/ntYvx4OOijqSEREJBsp\naZGMmTPHn/faK9o4REQkOylpkYz5/nuoXRt22inqSEREJBspaZGMmTMH9thDnXBFRKR89PUhKbN4\nMbz+evH758xR05CIiJSfkhZJmT59fLXmpUv99ZQpMG1a4f45c2DPPSMJTUREqgAlLZISv/0Gr7zi\nf48bB88+Cy1aQKtWMGyYT9+vmhYREamILaIOQKqGd9+F/HxfsfmTT2DgQOjaFebPh/vvh5UrYdUq\nOOGEqCMVEZFspaRFUuKdd3xq/t13h8ceg99/h1tvhVGj4OabYcMG6NABDjww6khFRCRbKWmRCvnp\nJ2jUCN5/H667DvbZxxOVk0+Ggw/2MmvWwOjRMGhQpKGKiEiWU9Ii5TZ7NjRtCqee6k1Dp57qfVjO\nP7+wzEEHQYMGXgNzzjnRxSoiItlPSYuU24QJ/vzee3DkkUWv3FyjBrz8sictmp9FREQqQkmLlFte\nnj/vsw88/DCYFV3uxBMzF5OIiFRdSlqk3PLy4Mwz4Y03oo5ERESqgzJX2JvZEWb2rpnNN7MCMzut\niDJ3m9kCM/vNzEaYWdOk/XXM7HEzW2JmK83sdTPbIalMAzN72czyzWyZmT1jZnWTyuxqZkPMbJWZ\nLTSzB8xMjRAZEIInLUU1CYmIiKRDeb7g6wKTgT8BIXmnmd0M9AS6A22BVcBwM6udUOxh4GSgK9AR\n2AlI/r3+CtAc6Bwr2xF4MuE8NYCheG1RO+Bi4BLg7nJck5TRtGmwfDm0aRN1JCIiUl2UuXkohDAM\nGAZgVmQvhmuBe0II/4mVuQhYBJwOvGZm9YDLgPNCCJ/EylwKTDeztiGECWbWHDgeaBVCmBQr0wsY\nYmY3hBAWxvbvBxwVQlgCTDGzO4D7zKxPCGF9Wa9NSm/YMNhySzjiiKgjERGR6iKlTSlmtifQBBgZ\n3xZCWAGMB9rHNrXGk6XEMjOBeQll2gHL4glLzId4zc6hCWWmxBKWuOFADnBAii5JijFsGBx1FGy1\nVdSRiIhIdZHq/h9N8MRiUdL2RbF9AI2BtbFkprgyTYDFiTtDCBuApUllijoPCWUkDVatgk8/1ZT8\nIiKSWdVy9FDv3r3JycnZaFtubi65ubkRRZQ9/vlP+PlnXwBRSYuISNU0aNAgBiVNY56fnx9RNIVS\nnbQsBAyvTUmsBWkMTEooU9vM6iXVtjSO7YuXSR5NVBNomFQmuRto44R9xerbty8tNeylzMaMgRtu\n8L/32svnZxERkaqnqB/yeXl5tGrVKqKIXEqbh0II3+MJQ+f4tljH20OBMbFNE4H1SWWaAbsBY2Ob\nxgL1zaxFwuE74wnR+IQyB5lZo4QyxwH5wLQUXZLEbNgA11/vM9uC17IUN5mciIhIOpS5piU2V0pT\nPIEA2MvMDgaWhhB+wIcz325ms4C5wD3Aj8A74B1zzexZ4CEzWwasBB4FPg8hTIiVmWFmw4GnzawH\nUBt4DBgUGzkE8AGenAyMDbPeMXaufiGEdWW9LinZgAEwbpz3ZZk/H9q1izoiERGpbsrTPNQa+Bjv\ncBuAf8a2/wu4LITwgJltjc+pUh8YDZwYQlibcIzewAbgdaAOPoT66qTzdAP64aOGCmJlr43vDCEU\nmNkpwAC8FmcV8AJwVzmuSTbj6achN1dDnEVEJDrlmaflEzbTrBRC6AP0KWH/GqBX7FFcmeXABZs5\nzw/AKSWVkYrLz4cpU+C666KOREREqjNNeS+bNX68T9vfoUPUkYiISHWmpEU267PPoFEj2HffqCMR\nEZHqTEmLFGvdOnjlFXjoITjtNI0WEhGRaClpkWL16QPnnw+tW8Ojj0YdjYiIVHdKWqRY774LF18M\no0ZB3bpRRyMiItWdkhYp0pw5MHWqpuoXEZHKQ0mLbGLIENh7b//72GOjjUVERCROSYts4ssv/fnV\nV2G77aKNRUREJE5Ji2xi1iyfk+Wcc6KOREREpJCSFtnE7NnQtGnUUYiIiGxMSYtsYtaswj4tIiIi\nlYWSFtnIihXw889KWkREpPJR0iIbmT3bn9U8JCIilY2SFtnI6NGwxRbQrFnUkYiIiGxMSYv8Twjw\n5JNw+ulQv37U0YiIiGxMSYv8z2efwbRpcNVVUUciIiKyKSUt8j9PPAH77ANHHRV1JCIiIptS0lLN\nhQCDBvlqzoMHQ/fuUEOfChERqYT09VTNXXYZdOsG333nzUJXXhl1RCIiIkXbIuoAJDpz58ILL8Bj\nj0HPnlFHIyIiUjLVtFRj770HtWvDxRdHHYmIiMjmKWmppgoK4LXXvNPttttGHY2IiMjmqXmoGsrP\n974rn38Ob70VdTQiIiKlo6SlGrrgAvjkE69p6dIl6mhERERKR0lLNZOfD8OGwcMPw1lnRR2NiIhI\n6alPSzXzwQewfj2cckrUkYiIiJSNkpZq5r334KCDYPfdo45ERESkbJS0VCMbNsDQoaplERGR7KSk\npRoZPx5++QVOPTXqSERERMpOSUs18tJL0KgRtG0bdSQiIiJlp9FD1cTLL8OAAfDgg1CzZtTRiIiI\nlJ1qWqqBESN8qv6LL4brr486GhERkfJR0lLFhQA33QQdOsCzz4JZ1BGJiIiUj5qHqrAvv/ROtwsX\nwscfq1lIRESym2paqrB33oE1a6BfPzjyyKijERERqRjVtFRho0dDp05w9dVRRyIiIlJxqmmpotas\n8XlZjjgi6khERERSQ0lLFTRvHrRsCatXQ8eOUUcjIiKSGkpaqpixY6FnT1i6FAYP9uRFRESkKlCf\nlirkhx/gsMN8mPMLL8BZZ0UdkYiISOooaalC/vMfH9Y8eTIccEDU0YiIiKSWmoeqkHff9Y63SlhE\nRKQqUtJSRSxbBh99BKedFnUkIiIi6aGkpYoYPBjWr4dzz406EhERkfRQ0lIFLFkCjz8Oxx0HO+4Y\ndTQiIiLpoY64WSwEyMuDM86AVatgwICoIxIREUkfJS1ZpqDA+67k50P//v73/vvDmDGwyy5RRyci\nIpI+SlqyzN//Drff7n9vvTW8+iqccor/LSIiUpUpackia9bAo49C9+5w7bXePKThzSIiUl0oacki\ngwbB4sXQuzfst1/U0YiIiGSWkpYsUFDgHW1vu82n5lfCIiIi1VHKhzyb2V1mVpD0mJZU5m4zW2Bm\nv5nZCDNrmrS/jpk9bmZLzGylmb1uZjsklWlgZi+bWb6ZLTOzZ8ysbqqvJ0rr18Of/gS1akGzZt75\n9sEHo45KREQkGumap2Uq0BhoEnscHt9hZjcDPYHuQFtgFTDczGonvP9h4GSgK9AR2Al4I+kcrwDN\ngc6xsh2BJ9NwLZF56SUfxnzdddCqla/gvMceUUclIiISjXQ1D60PIfxczL5rgXtCCP8BMLOLgEXA\n6cBrZlYPuAw4L4TwSazMpcB0M2sbQphgZs2B44FWIYRJsTK9gCFmdkMIYWGaritjNmyA//s/6NoV\n/vnPqKMRERGJXrpqWvYxs/lmNtvMXjKzXQHMbE+85mVkvGAIYQUwHmgf29QaT6YSy8wE5iWUaQcs\niycsMR8CATg0PZeUWV9/DXPnQq9eUUciIiJSOaQjaRkHXILXhFwF7Al8Gutv0gRPLBYlvWdRbB94\ns9LaWDJTXJkmwOLEnSGEDcDShDJZbexY2GILaNs26khEREQqh5Q3D4UQhie8nGpmE4D/AucAM1J9\nvvLo3bs3OTk5G23Lzc0lNzc3oog2NXYsHHIIbLVV1JGIiEh1M2jQIAYNGrTRtvz8/IiiKZT2Ic8h\nhHwz+xZoCowCDK9NSaxtaQzEm3oWArXNrF5SbUvj2L54meTRRDWBhgllitW3b19atmxZ9ovJoLFj\n4eSTo45CRESqo6J+yOfl5dGqVauIInJpX+XZzLbBE5YFIYTv8aSic8L+eng/lDGxTROB9UllmgG7\nAWNjm8YC9c2sRcKpOuMJ0fj0XEnmTJsGs2dDp05RRyIiIlJ5pLymxcweBN7Dm4R2Bv4PWAf8O1bk\nYeB2M5sFzAXuAX4E3gHvmGtmzwIPmdkyYCXwKPB5CGFCrMwMMxsOPG1mPYDawGPAoKowcuj552G7\n7VTTIiIikigdzUO74HOobAf8DHwGtAsh/AIQQnjAzLbG51SpD4wGTgwhrE04Rm9gA/A6UAcYBlyd\ndJ5uQD981FBBrOy1abiejHrmGV+9uXt3qF178+VFRESqCwshRB1DxphZS2DixIkTK2WfltmzfYr+\n3Fx4/HHYdtuoIxIREXEJfVpahRDyoohBaw9VIn/9KzRqBE88AVtvHXU0IiIilUvaO+JK6axY4as4\nX3utEhYREZGiKGmpJN54A9asgQsuiDoSERGRyknNQxEbMwY++cRHDB19NOyyS9QRiYiIVE5KWiIU\nAvTo4esMNWoEQ4dGHZGIiEjlpeahiMyeDVde6QnLiy9CXh40bRp1VCIiIpWXaloybOZMuPxyyMmB\nkSPh+OO9H4tZ1JGJiIhUbkpaMuzVV+Gzz/zvv/0Nbrkl2nhERESyhZqHMuzrr/25Th24+OJoYxER\nEckmSloybNw4uOkmmDsXdtop6mhERESyh5KWDJo3D+bPh/btoUmTqKMRERHJLkpaMui++6BePejY\nMepIREREso864mbIF1/Ak0/Cgw9Cw4ZRRyMiIpJ9VNOSZuvWwcMPwznnQMuW0KtX1BGJiIhkJ9W0\npFG3bjBqFPz8s0/R368f1KoVdVQiIiLZSTUtabJypc/J0rIljB4Nw4fDPvtEHZWIiEj2Uk1Lmnz5\nJRQUwP33wwEHRB2NiIhI9lNNS5qMH+8jhZo3jzoSERGRqkFJSxpMnw6DB0ObNlBDd1hERCQl1DyU\nYitXQufO3jR0111RRyMiIlJ1KGlJsVtvheXLvbZl992jjkZERKTqUNKSQs8958Oa+/VTwiIiIpJq\n6nGRIitXwl/+AhdeCFdfHXU0IiIiVY+SlhR54AHIz4d77406EhERkapJzUMVtHw59O/viyHecgvs\ntlvUEYmIiFRNqmmpoGeegTvvhJNOgttvjzoaERGRqks1LRX06adw5JHwzjtRRyIiIlK1qaalAgoK\n4LPP4Igjoo5ERESk6lPSUgHTpsGyZUpaREREMkFJSwU88QTUrw/t2kUdiYiISNWnpKUcfv0VbroJ\nnnoKbr4Z6taNOiIREZGqTx1xy2j9ejjtNPjiC7jsMrjmmqgjEhERqR6UtJTRv/8NH3/sj06doo5G\nRESk+lDzUBkUFPgkciedpIRFREQk01TTUgb/+Q988413wBUREZHMUk1LKYUAf/sbHH64P0RERCSz\nVNNSSk89BePHw8iRUUciIiJSPammpRSeeAJ69oTLL4ejj446GhERkepJSctm5OUVJiz9+0cdjYiI\nSPWlpGUzbr4Z9t8fHn0UatWKOhoREZHqS0lLCfLzYdQo6NFDCYuIiEjUlLSU4MMPfQbcE0+MOhIR\nERFR0lKM4cPh1lu9aWiPPaKORkRERJS0FOGbb+DMM6FRI3j66aijEREREdA8LUW6+WbYdVf44AOt\n4CwiIlJZKGlJ8tVXMGQIPP+8EhYREZHKRM1DCUaNglatvA9Lbm7U0YiIiEgiJS0J3n0Xdt4ZJk+G\nOnWijkZEREQSKWlJMHEiHHoo5OREHYmIiIgkU9ISU1AAkyZ585CIiIhUPkpaYmbNgpUrlbSkw6BB\ng6IOodrRPc883fPM0z2vfrI+aTGzq83sezP73czGmVmb8hxn4ECoWRNatkx1hKL/sWSe7nnm6Z5n\nnu559ZPVSYuZnQv8E7gLaAF8BQw3s0ZlOc748fD3v8Mdd0DDhmkIVERERCosq5MWoDfwZAjhxRDC\nDOAq4DfgstIe4Ouv4cILvVno1lvTFaaIiIhUVNYmLWZWC2gFjIxvCyEE4EOgfUnvDQE+/RTuugsO\nOQTWrPHmIa3kLCIiUnll84y4jYCawKKk7YuAZsW8Z0uAbt2m8+23vuGii+Dqq+HXXyEvL02RVnP5\n+fnk6eZmlO555umeZ57ueWZNnz49/ueWUcVgXjmRfcxsR2A+0D6EMD5h+/1AxxDCJrUtZtYNeDlz\nUYqIiFQ554cQXonixNlc07IE2AA0TtreGFhYzHuGA+cDc4HVaYtMRESk6tkS2AP/Lo1E1ta0AJjZ\nOGB8COHa2GsD5gGPhhAejDQ4ERERSalsrmkBeAh4wcwmAhPw0URbAy9EGZSIiIikXlYnLSGE12Jz\nstyNNwtNBo4PIfwcbWQiIiKSalndPCQiIiLVR9bO0yIiIiLVi5IWERERyQrVJmlJ1cKKVZ2ZHWFm\n75rZfDMrMLPTiihzt5ktMLPfzGyEmTVN2l/HzB43syVmttLMXjezHZLKNDCzl80s38yWmdkzZlY3\nqcyuZjbEzFaZ2UIze8DMqtxn1sxuMbMJZrbCzBaZ2Vtmtm8R5XTfU8TMrjKzr2L3Id/MxpjZCUll\ndL/TxMz+Evv/y0NJ23XPU8jM7ord58THtKQy2XXPQwhV/gGci8/LchGwH/AksBRoFHVsle0BnIB3\nbO6Cz4NzWtL+m2P37hTgQOBtYDZQO6HMAHwunCPxhSzHAKOTjvM+kAe0BjoA3wIvJeyvAUzB5wM4\nCDgeWAzcG/U9SsM9HwpcCDSPXet/YvdvK933tN3zk2Of9b2BpsC9wBqgue532u99G2AOMAl4SJ/x\ntN7ru4Cvge2BHWKPhtl8zyO/qRn6hxsHPJLw2oAfgZuijq0yP4ACNk1aFgC9E17XA34Hzkl4vQY4\nI4eJ+64AAAPqSURBVKFMs9ix2sZeN4+9bpFQ5nhgPdAk9vpEYB0JiSVwJbAM2CLqe5Pm+94odn8O\n133P6H3/BbhU9zut93gbYCZwNPAxGyctuuepv993AXkl7M+6e17lqsOSWQUWVpSNmdmeQBM2vpcr\ngPEU3svW+FD6xDIz8Un/4mXaActCCJMSDv8hEIBDE8pMCSEsSSgzHMgBDkjRJVVW9fF7sRR039PN\nzGqY2Xn4HE9jdL/T6nHgvRDCR4kbdc/Tah/z5v7ZZvaSme0K2XvPq3zSQskLKzbJfDhZrQn+QSzp\nXjYG1sY+/MWVaYJXDf5PCGED/iWdWKao80AV/nczMwMeBj4LIcTbnnXf08DMDjSzlfgvyf74r8mZ\n6H6nRSwxPAS4pYjduufpMQ64BK/5uArYE/g01t8kK+95Vk8uJ1IF9Qf2Bw6LOpBqYAZwMP5r7yzg\nRTPrGG1IVZOZ7YIn48eEENZFHU91EUJIXCNoqplNAP4LnIN//rNOdahpKc/CilK0hXh/oJLu5UKg\ntpnV20yZ5N7nNYGGSWWKOg9U0X83M+sHnAR0CiH8lLBL9z0NQgjrQwhzQgiTQgi3AV8B16L7nQ6t\n8M6geWa2zszW4R07rzWztfivbt3zNAsh5OOdZJuSpZ/zKp+0xLL6iUDn+LZYFXxnvBe0lFII4Xv8\nA5Z4L+vh7ZbxezkR74CVWKYZsBswNrZpLFDfzFokHL4z/h/Q+IQyB5kv0xB3HJAPbDRkryqIJSxd\ngKNCCPMS9+m+Z0wNoI7ud1p8iI8aOQSv3ToY+BJ4CTg4hDAH3fO0M7Nt8IRlQdZ+zqPu3ZyJB14V\n9hsbD3n+Bdg+6tgq2wOoi/8P5RC8R/h1sde7xvbfFLt3p+L/E3ob+I6Nh8j1B74HOuG/sD5n0yFy\nQ/H/abXBm0JmAgMT9tfAf/m+D/wBb5NdBNwT9T1Kwz3vj/eiPwL/9RF/bJlQRvc9tff8b7H7vTs+\n1PPv+P+cj9b9zti/QfLoId3z1N/jB4GOsc95B2BE7Fq3y9Z7HvlNzeA/3p/wsea/41lf66hjqowP\nvMq2AG9SS3w8l1CmDz5U7je8B3jTpGPUAR7Dm+ZWAoOBHZLK1Md/ZeXjX9hPA1snldkVn7Pk19gH\n/H6gRtT3KA33vKj7vQG4KKmc7nvq7vkz+Fwhv+O/Nj8glrDofmfs3+AjEpIW3fO03ONB+PQev+Mj\nfl4B9szme64FE0VERCQrVPk+LSIiIlI1KGkRERGRrKCkRURERLKCkhYRERHJCkpaREREJCsoaRER\nEZGsoKRFREREsoKSFhEREckKSlpEREQkKyhpERERkaygpEVERESywv8DuVXzrLbnACkAAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f43050f06d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "plt.plot(*zip(*sorted(rewards.items(),key=lambda k:k[0])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from agentnet.utils.persistence import save\n",
    "save(action_layer,\"kung_fu.pcl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<agentnet.resolver.probabilistic.ProbabilisticResolver at 0x7f435bfea590>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "###LOAD FROM HERE\n",
    "from agentnet.utils.persistence import load\n",
    "load(action_layer,\"kung_fu.pcl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-02-22 05:14:20,638] Making new env: KungFuMaster-v0\n",
      "[2017-02-22 05:14:20,701] DEPRECATION WARNING: env.spec.timestep_limit has been deprecated. Replace your call to `env.spec.timestep_limit` with `env.spec.tags.get('wrapper_config.TimeLimit.max_episode_steps')`. This change was made 12/28/2016 and is included in version 0.7.0\n",
      "[2017-02-22 05:14:20,703] Clearing 2 monitor files from previous run (because force=True was provided)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode finished after 3300 timesteps with reward=20700.0\n",
      "Episode finished after 4671 timesteps with reward=35300.0\n",
      "Episode finished after 3272 timesteps with reward=21200.0\n",
      "Episode finished after 3275 timesteps with reward=21000.0\n",
      "Episode finished after 2637 timesteps with reward=14500.0\n",
      "Episode finished after 3671 timesteps with reward=22200.0\n",
      "Episode finished after 3433 timesteps with reward=21300.0\n",
      "Episode finished after 3582 timesteps with reward=24600.0\n",
      "Episode finished after 4278 timesteps with reward=31700.0\n"
     ]
    }
   ],
   "source": [
    "rw = pool.evaluate(n_games=20,save_path=\"./records\",record_video=True)\n",
    "print(\"mean session score=%f.5\"%np.mean(rw))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#show video\n",
    "from IPython.display import HTML\n",
    "import os\n",
    "\n",
    "video_names = list(filter(lambda s:s.endswith(\".mp4\"),os.listdir(\"./videos/\")))\n",
    "\n",
    "HTML(\"\"\"\n",
    "<video width=\"640\" height=\"480\" controls>\n",
    "  <source src=\"{}\" type=\"video/mp4\">\n",
    "</video>\n",
    "\"\"\".format(\"./videos/\"+video_names[-1])) #this may or may not be _last_ video. Try other indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## How to enhance\n",
    "* More parallel agents\n",
    "* Different constructs for recurrent memory\n",
    "* Try PGQ-like algorithms\n",
    "* Maybe tune parameters for regularization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}