Example_6.3_random_walk_batch_update.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "範例說明大部份參考[ShangtongZhang](https://github.com/ShangtongZhang/reinforcement-learning-an-introduction/blob/master/chapter06/random_walk.py)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example 6.3: Random walk under batch updating\n",
    "\n",
    "Example 6.2的那個[隨機漫步](http://terms.naer.edu.tw/detail/576952/)預測問題如果用batch-updating version的TD(0)、constant-$\\alpha$ MC的話，會是這樣的：在每一個新的episode之後，所有已經過看的episodes都會被視為一個batch。它們會反覆的被用到演算法裡面，TD(0)跟constant-$\\alpha$ MC都一樣，然後$\\alpha$是小到能夠讓value function收斂的那種。最終得到的value function再來跟$v_\\pi$比較，然後每5個states計算其平均平方根誤差(整個實驗是100次獨立的反覆計算)，得到的學習曲線結果如Figure 6.2所示。\n",
    "\n",
    "![](https://i.imgur.com/VYPbyFh.png)\n",
    "Figure 6.2：隨機漫步任務上使用批次訓練(batch training)的TD(0)與constant-$\\alpha$ MC的比較\n",
    "\n",
    "上圖也可以看的出來，TD(0)的結果還是比MC來的好。\n",
    "\n",
    "使用batch training，constant-$\\alpha$ MC收斂至$V(s)$，這是在看過(visiting)每一個state $s$之後的實際回報(actual returns)的樣本平均。某種程度上這算是最佳估測(optimal estimates)，它們最小化來自訓練集內的實際回報(actual returns)的均方誤差(mean square error)。但是你從Figure 6.2來看卻發現，batch TD method的表現竟然比MC還要來的好。它老兄怎麼有辦法比這個optimal method還要來的好？答案是這樣的，MC只在某些有限方面上是最佳的，而TD則是與預測回報(prediction returns)相關方面是最佳的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先載入需求套件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from tqdm import tqdm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "根據說明，我們會有A、B、C、D、E這五個state再加上A的左邊跟E的右邊兩個terminal state，總共7個state，每個state都有它的true value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.        , 0.16666667, 0.33333333, 0.5       , 0.66666667,\n",
       "       0.83333333, 1.        ])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TRUE_VALUE = np.ones(7)\n",
    "TRUE_VALUE[0] = 0\n",
    "TRUE_VALUE[1:6] = np.arange(1, 6) / 6.0\n",
    "TRUE_VALUE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "每個state的value都初始化為0.5，然後我們會把右邊那個terminal state的預設值設置為1，這計算過程中是有幫助的"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0. , 0.5, 0.5, 0.5, 0.5, 0.5, 1. ])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "VALUES = np.zeros(7)\n",
    "VALUES[1:6] = 0.5\n",
    "VALUES[6] = 1\n",
    "VALUES"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "這個範例的狀態轉移很簡單，不是向左就是向右"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "TRANSITION_LEFT = 0\n",
    "TRANSITION_RIGHT = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 左右的機率各半\n",
    "probs = [0.5, 0.5]\n",
    "np.random.choice([TRANSITION_LEFT, TRANSITION_RIGHT], p=probs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "過程中的reward皆為0，然後只有進到最右邊的terminal state才會得到reward+1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面給出公式6.1做為參考  \n",
    "$$V(S_t) \\leftarrow V(S_t) + \\alpha [G_t - V(S_t)] \\tag{6.1}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "先來看看Monte Carlo，因為是批次處理，因此我們並不會在過程中就直接計算value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def monte_carlo(values, alpha=0.1):\n",
    "    # 所有的state都是從3開始\n",
    "    state = 3\n",
    "    trajectory = [3]\n",
    "    \n",
    "    while True:\n",
    "        # 狀態的轉機轉移\n",
    "        if np.random.choice([TRANSITION_LEFT, TRANSITION_RIGHT], p=probs) == TRANSITION_LEFT:\n",
    "            state -= 1\n",
    "        else:\n",
    "            state += 1\n",
    "        # 寫入軌跡記錄\n",
    "        trajectory.append(state)\n",
    "        # 判斷是否結束\n",
    "        if state == 6:\n",
    "            returns = 1.0\n",
    "            break\n",
    "        elif state == 0:\n",
    "            returns = 0.0\n",
    "            break\n",
    "            \n",
    "    \n",
    "    # 批次處理，因此不在這邊計算\n",
    "#     for _state in trajectory[: -1]:\n",
    "#         values[_state] += alpha * (returns - values[_state])\n",
    "        \n",
    "    return trajectory, [returns] * (len(trajectory) - 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([3, 4, 5, 4, 3, 4, 5, 6], [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "_values = np.copy(VALUES)\n",
    "trajectory, reward = monte_carlo(_values)\n",
    "trajectory, reward"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下來是TD，下面給出公式6.2參考  \n",
    "$$V(S_t) \\leftarrow V(S_t) + \\alpha[R_{t+1} + \\gamma V(S_{t+1}) - V(S_t)] \\tag{6.2}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一樣的，我們並不會立即的計算value，因此這邊也會同步註解掉計算的部份"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def temporal_difference(values, alpha=0.1):\n",
    "    # 所有的state都是從3開始\n",
    "    state = 3\n",
    "    trajectory = [3]\n",
    "    rewards = [0]\n",
    "    \n",
    "    while True:\n",
    "        old_state = state\n",
    "        if np.random.choice([TRANSITION_LEFT, TRANSITION_RIGHT], p=probs) == TRANSITION_LEFT:\n",
    "            state -= 1\n",
    "        else:\n",
    "            state += 1\n",
    "        \n",
    "        # 這邊每一個state的變化得到的reward皆為0\n",
    "        # 跟上面設置的VALUES有關，算是一種小技巧吧，我猜        \n",
    "        reward = 0\n",
    "        \n",
    "        trajectory.append(state)\n",
    "        \n",
    "        # 批次更新，因此不在這邊計算\n",
    "#         values[old_state] += alpha * (reward + values[state] - values[old_state])\n",
    "        \n",
    "        if state == 6 or state == 0:\n",
    "            break\n",
    "        \n",
    "        rewards.append(reward)\n",
    "    \n",
    "    return trajectory, rewards"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 1, 2, 3, 2, 3, 2, 3, 4, 5, 6],\n",
       " [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "_values = np.copy(VALUES)\n",
    "trajectory, reward = temporal_difference(_values)\n",
    "trajectory, reward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3 0\n",
      "2 0\n",
      "3 0\n",
      "2 0\n",
      "3 0\n",
      "2 0\n",
      "3 0\n",
      "2 0\n",
      "3 0\n",
      "2 0\n",
      "1 0\n",
      "2 0\n",
      "3 0\n",
      "2 0\n",
      "3 0\n",
      "2 0\n",
      "3 0\n",
      "4 0\n",
      "5 0\n"
     ]
    }
   ],
   "source": [
    "for i, j in zip(trajectory, reward):\n",
    "    print(i, j)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面開始處理批次更新(batch updating)的部份"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def batch_updating(method, episodes, alpha=0.001):\n",
    "    \"\"\"批次更新\n",
    "    method: MC or TD\n",
    "    episodes: episodes\n",
    "    alpha: step size，要夠小\n",
    "    \"\"\"\n",
    "    # 執行次數\n",
    "    runs = 100\n",
    "    # 記錄每個episode的total error\n",
    "    total_errors = np.zeros(episodes)\n",
    "    for run in tqdm(range(0, runs)):\n",
    "        # 初始值，每一run都重來\n",
    "        current_values = np.copy(VALUES)\n",
    "        # 記錄error\n",
    "        errors = []\n",
    "        # 記錄trajectory\n",
    "        trajectories = []\n",
    "        # 記錄reward\n",
    "        rewards = []\n",
    "        for episode in range(episodes):            \n",
    "            if method == 'TD':\n",
    "                _trajectory, _rewards = temporal_difference(current_values)\n",
    "            else:\n",
    "                _trajectory, _rewards = monte_carlo(current_values)\n",
    "            \n",
    "            # 每經過一個episode，你的trajectories就會多一個episode的記錄\n",
    "            # 每次都會利用目前為止所有的記錄來做計算\n",
    "            trajectories.append(_trajectory)\n",
    "            rewards.append(_rewards)\n",
    "            while True:\n",
    "                # 這邊我們會利用目前為止所看過的所有的軌跡來更新\n",
    "                # 作法上跟DP一樣，一直到更新的幅度小於某一個閥值之後才會停止\n",
    "                # 5個state加上左右各一個terminal state\n",
    "                updates = np.zeros(7)\n",
    "                # 這邊為了有區隔，用的符號就有所不同，雖然應該是無所謂\n",
    "                for trajectory_, rewards_ in zip(trajectories, rewards):\n",
    "                    for i in range(0, len(trajectory_) -1):\n",
    "                        # 計算的公式都還是根據6.1、6.2\n",
    "                        if method == 'TD':\n",
    "                            updates[trajectory_[i]] += rewards_[i] + current_values[trajectory_[i + 1]] - current_values[trajectory_[i]] \n",
    "                        else:\n",
    "                            updates[trajectory_[i]] += rewards_[i] -current_values[trajectory_[i]]\n",
    "                \n",
    "                updates *= alpha\n",
    "                # 當更新的幅度已經低於閥的時候就停止\n",
    "                if np.sum(np.abs(updates)) < 1e-3:\n",
    "                    break\n",
    "                \n",
    "                current_values += updates\n",
    "            # 計算每個episode的error\n",
    "            errors.append(np.sqrt(np.sum(np.power(current_values - TRUE_VALUE, 2)) / 5.0))\n",
    "        # 加總每一run的error\n",
    "        total_errors += np.asarray(errors)\n",
    "    # 計算平均\n",
    "    total_errors /= runs\n",
    "    return total_errors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 100/100 [00:41<00:00,  2.39it/s]\n",
      "100%|██████████| 100/100 [00:35<00:00,  2.85it/s]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXzcdbXw8c+ZmUwm+94t6ZIutLTQBcomq4LIZkHlYkEQBUW88rggKl69sjzq9brciwpX4UEUAdmvWAVkLfvWlNJ9oXvTLWnS7Nss5/njOynTkCZpMpNJMuf9es0r89vPMCUn311UFWOMMaYrT7IDMMYYMzRZgjDGGNMtSxDGGGO6ZQnCGGNMtyxBGGOM6ZYv2QHES3FxsU6aNCnZYRhjzLCydOnSfapa0t2xEZMgJk2aREVFRbLDMMaYYUVEth3qmFUxGWOM6ZYlCGOMMd2yBGGMMaZbI6YNwhhjEiEYDFJZWUlbW1uyQxmQQCBAWVkZaWlpfb7GEoQxxvSgsrKSnJwcJk2ahIgkO5x+UVVqamqorKykvLy8z9dZFZMxxvSgra2NoqKiYZscAESEoqKiwy4FJTRBiMg5IrJeRDaKyI3dHL9WRFaKyHsi8pqIzIw59v3odetF5BOJjNMYY3oynJNDp/58hoQlCBHxAncA5wIzgUtjE0DUX1T1aFWdC/wc+K/otTOBhcAs4Bzgf6L3i7/WOnjpZ7BzaUJub4wxw1Ui2yCOBzaq6mYAEXkIuBBY03mCqjbEnJ8FdC5OcSHwkKq2A1tEZGP0fm/GPUrxwEv/Ab4AlB4b99sbY8xA1NTUcOaZZwKwZ88evF4vJSVu4PPy5cuZM2cOwWAQn8/H5z//eb71rW/h8cTnb/9EJohSYEfMdiVwQteTRORrwPWAH/hYzLVvdbm2tJtrrwGuAZgwYUL/ogzkQmYR7N/Sv+uNMSaBioqKeO+99wC4+eabyc7O5oYbbgAgOzv7wLGqqiouu+wyGhoauOWWW+Ly7KQ3UqvqHao6Bfge8MPDvPYuVZ2vqvM7M2q/FE6G2s39v94YY5Js1KhR3HXXXdx+++3Ea6XQRJYgdgLjY7bLovsO5SHgd/28dmAKymH7W72fZ4xJabf8fTVrdjX0fuJhmDkul5s+OSsu95o8eTLhcJiqqipGjx494PslsgSxBJgmIuUi4sc1Oi+KPUFEpsVsng+8H32/CFgoIukiUg5MA95JWKSF5dBQCaH2hD3CGGOGm4SVIFQ1JCLXAc8AXuAeVV0tIrcCFaq6CLhORM4CgsB+4MrotatF5BFcg3YI+JqqhhMVK4WTQSNQtx2Kp/V+vjEmJcXrL/1E2bx5M16vl1GjRsXlfgkdSa2qTwFPddn3o5j33+jh2p8AP0lcdDEKoiMLa7dYgjDGDEvV1dVce+21XHfddXEbt2FTbYArQYA1VBtjhpXW1lbmzp17oJvrFVdcwfXXXx+3+1uCAMgqBn+2dXU1xgxpN99880Hb4XDiat5hCHRzHRJEXDVTrSUIY4zpZAmiU2G5VTEZY0wMSxCdCsuhbhtEEltkM8aY4cISRKeCcgh3QEPixuMZY8xwYgmi04GeTNYOYYwxYAniA4WdYyGsHcIYY8ASxAdyS8Hrt66uxpghR0S4/PLLD2yHQiFKSkq44IILDux7+umnmT9/PjNnzmTevHl8+9vfHvBzLUF08nghf6JVMRljhpysrCxWrVpFa2srAM899xylpR+sgLBq1Squu+467r//ftasWUNFRQVTp04d8HMtQcQqtLEQxpih6bzzzuPJJ58E4MEHH+TSSy89cOznP/85P/jBD5gxYwYAXq+Xr371qwN+po2kjlU4Gba9Aapu8JwxxsR6+kbYszK+9xxzNJz7s15PW7hwIbfeeisXXHABK1as4KqrruLVV18FXAkiHlVKXVkJIlZBOXQ0QXN1siMxxpiDzJ49m61bt/Lggw9y3nnnDcozrQQRqzBmVtfs+EyXa4wZQfrwl34iLViwgBtuuIGXXnqJmpqaA/tnzZrF0qVLmTNnTlyfZyWIWJ1jIarWJDcOY4zpxlVXXcVNN93E0UcffdD+73znO/z0pz9lw4YNAEQiEX7/+98P+HlWgoiVPxHyxsOT34a9q+Gj/waZhcmOyhhjACgrK+PrX//6h/bPnj2b2267jUsvvZSWlhZE5KAusP0l8VrcOtnmz5+vFRUVA79RSy0s/ilU/AHSc+Gyh2HCiQO/rzFmWFq7di1HHnlkssOIi+4+i4gsVdX53Z1vVUxdZRbC+b+Ea18DFJbdl+yIjDEmKSxBHMroWVB6LOxekexIjDEmKSxB9GTsHKhaC6H2ZEdijEmikVAV35/PYAmiJ2PnQCTokoQxJiUFAgFqamqGdZJQVWpqaggEAod1nfVi6smY2e7nnhUwbm5yYzHGJEVZWRmVlZVUVw/vAbSBQICysrLDusYSRE8Kyl1Ppt3Lkx2JMSZJ0tLSKC8vT3YYSWFVTD3xeFwpwhKEMSYFWYLozdjZsGeVrVVtjEk5liB6M3YOhFph3/vJjsQYYwaVJYjejI1OfmXVTMaYFGMJojdF08AXcD2ZjDEmhSQ0QYjIOSKyXkQ2isiN3Ry/XkTWiMgKEXlBRCbGHAuLyHvR16JExtkjrw9GH2UlCGNMyklYghARL3AHcC4wE7hURGZ2OW0ZMF9VZwOPAT+POdaqqnOjrwWJirNPxs5xU24M44EyxhhzuBJZgjge2Kiqm1W1A3gIuDD2BFVdrKot0c23gMMbxTFYxs6G9nrYvzXZkRhjzKBJZIIoBXbEbFdG9x3K1cDTMdsBEakQkbdE5KLuLhCRa6LnVCR0lKM1VBtjUtCQaKQWkcuB+cAvYnZPjM5Rfhlwm4hM6Xqdqt6lqvNVdX5JSUniAhw1Ezw+SxDGmJSSyASxExgfs10W3XcQETkL+AGwQFUPTJuqqjujPzcDLwHzEhhrz3zpUDzdrTJnjDEpIpEJYgkwTUTKRcQPLAQO6o0kIvOAO3HJoSpmf4GIpEffFwMnA8ldKHr0TFur2hiTUhKWIFQ1BFwHPAOsBR5R1dUicquIdPZK+gWQDTzapTvrkUCFiCwHFgM/U9UkJ4hZUL8DWuuSGoYxxgyWhM7mqqpPAU912fejmPdnHeK6N4CjExnbYRs1y/2sWgsTT0puLMYYMwiGRCP1sDA6miD2rkpuHMYYM0gsQfRV7jgI5Fk7hDEmZViC6CsRN+WG9WQyxqQISxCHY9RM2LvGptwwxqSElF9yNBiOcN1f3mXaqByOGJPDkWNymDoqGxH58MmjZ0FHI9Rth4KJHz5ujDEjSMoniJqmDjZXN/P82irCEVcy+PlnZnPJceM/fHJnQ3XVGksQxpgRL+UTxJi8AM9dfzrtoTAbq5o4/zevsau+tfuTRx3pfu5dBdPPHbwgjTEmCawNIird52XWuDzSfR5aOw6x/nR6DuRPdO0QxhgzwlmC6CLT76U1eIgEAdaTyRiTMixBdJGR5qXlUCUIcHMy1WyEYNvgBWWMMUlgCaKLjF5LELNAw7Bv/eAFZYwxSWAJoosMv/fQbRDwwZxM1g5hjBnhLEF0kZnm6zlBFE4GXwD2rBi8oIwxJgksQXQR8Htp6amKyeuDiSfD+qdtRLUxZkSzBNFFZpqXtp5KEACzLoL9W2wJUmPMiGYJoosMv5eWYKjnk2Zc4NaoXvPE4ARljDFJYAmii14bqQEyC6H8dFj9V6tmMsaMWJYgushI60OCAJj1Kdi/1aqZjDEjliWILjKjjdTaW8lgxvmummn1XwcnMGOMGWSWILrI8HtRhfZQpOcTMwth8hmuHcKqmYwxI5AliC4y0rwAfatmmnlRtJrpvcQGZYwxSWAJootMfzRB9DQWotOBaibrzWSMGXksQXQRiJYgepywr1NmIUw4CTa9kOCojDFm8FmC6CLT79ZQautLCQJcd9c9K6GlNoFRGWPM4LME0UXG4ZQgAMpPcz+3vpqgiIwxJjksQXSRcThtEAClx0BaFmx5JYFRGWPM4LME0cUHvZh6mW6jkzcNJn7EEoQxZsSxBNHFYfVi6lR+GuzbAA27ExSVMcYMPksQXXRWMfW5DQKsHcIYMyIlNEGIyDkisl5ENorIjd0cv15E1ojIChF5QUQmxhy7UkTej76uTGScsQ60QRxOghhzNATyYfPLCYrKGGMGX8IShIh4gTuAc4GZwKUiMrPLacuA+ao6G3gM+Hn02kLgJuAE4HjgJhEpSFSssQ5rJHUnjxfKT4UtL9u0G8aYESORJYjjgY2qullVO4CHgAtjT1DVxaraEt18CyiLvv8E8Jyq1qrqfuA54JwExnpAmtdDmlcOrw0C3HiI+h1u6g1jjBkBekwQIuIRkUv6ee9SYEfMdmV036FcDTx9ONeKyDUiUiEiFdXV1f0M88MCad7Da4OAD9ohrDeTMWaE6DFBqGoE+G6igxCRy4H5wC8O5zpVvUtV56vq/JKSkrjFk9mXRYO6Kj4CskfDphfjFocxxiRTX6qYnheRG0RkvIgUdr76cN1OYHzMdll030FE5CzgB8ACVW0/nGsTJSPNe/hVTCJuKdIN/4S2hsQEZowxg6gvCeKzwNeAV4Cl0VdFH65bAkwTkXIR8QMLgUWxJ4jIPOBOXHKoijn0DHC2iBREG6fPju4bFBl+3+FXMQHMuRRCbbDmb/EPyhhjBpmvtxNUtbw/N1bVkIhch/vF7gXuUdXVInIrUKGqi3BVStnAoyICsF1VF6hqrYj8X1ySAbhVVQdtNryMNE/fJ+uLVTYfCqfAiofhmCviH5gxxgyiXhOEiKQBXwWirbC8BNypqsHerlXVp4Cnuuz7Ucz7s3q49h7gnt6ekQiZfh8tfZ1qI5aIK0Us/jHUbYf8CfEPzhhjBklfqph+BxwL/E/0dWx034iV4ffSGuxlydFDmR3t9LXi4fgFZIwxSdBrCQI4TlXnxGy/KCLLExXQUJCR5u37ZH1dFUyEiSfD8ofg1BtcqcIYY4ahvpQgwiIypXNDRCYD/aigHz4y/f3oxRRrzkKo2Qg7341fUMYYM8j6kiC+AywWkZdE5GXgReDbiQ0rufo1UC7WzAvBF4DlD8YvKGOMGWQ9VjGJiAdoBaYB06O718eMVxiRMv3e/vVi6hTIg2lnw9q/w7k/B49NmmuMGX76MpL6DlVtV9UV0deITg7g2iCCYSUY7mdDNbhBc017YNey+AVmjDGDqC9/2r4gIp8RSZ3W1sNedrQ70z4O4oX1T8YpKmOMGVx9SRBfAR4F2kWkQUQaRWREzyXRrzUhusosdEuRrnuq93ONMWYI6m02VwFmqapHVf2qmquqOaqaO0jxJUVmPBIEwIzzoXot1G6OQ1TGGDO4emuDUCDl6kg6Fw0aUE8mgOnnuZ9WijDGDEN9qWJ6V0SOS3gkQ0iG33XuGlAbBLhBc6OPgvWWIIwxw09fEsQJwJsisim6dvRKEVmR6MCSqV/Ljh7K9PNg+5vQXDPwexljzCDqy1Qbn0h4FENMZxtEvybs62rGefDKz+H9Z2DuZQO/nzHGDJJeSxCqug23eM/Hou9b+nLdcBZIi0M3105j50JuqRtVHRnAuApjjBlkvf6iF5GbgO8B34/uSgPuT2RQyRa3XkzgJus7+RturerFPx74/YwxZpD0pYrpU8A84F0AVd0lIjkJjSrJMuJZggA4/hrYuxpe/RUUTbWqJmPMsNCXqqKOaHdXBRCRrMSGlHwZ/jh1c+0kAuf/CspPh0Vfh62vxee+xhiTQH1JEI+IyJ1Avoh8GXge+H+JDSu50n0ePMLAJuzrypsGl/wZCsvh8S9BR0v87m2MMQnQl0bqXwKPAY/jZnT9kar+NtGBJZOIkDHQKb+7k5EPC34Ljbvhzdvje29jjImzvrRBoKrPAc8lOJYhJcPvi18bRKwJJ8KRC+C12+CYKyFndPyfYYwxcTCiu6sORIbfE59eTN0562YIt8PinyTm/sYYEweWIA4hM82XuARRNAWO+zIsuw/2rknMM4wxZoD6nCBEJE1E5onIqEQGNFQE/F5aElHF1On070J6DvzzexAZ0Ut8G2OGqUMmCBH5vYjMir7PA5YDfwaWicilgxRf0mSmeWlLVAkC3HoRH7/VDaB7/ubEPccYY/qppxLEqaq6Ovr+i8AGVT0aOBb4bsIjS7IMv5eWYBzmYurJsV+A+VfDG7+BZQ8k9lnGGHOYekoQHTHvPw48AaCqexIa0RCR4fcmrg0i1rn/6QbQ/eObsP2txD/PGGP6qKcEUSciF4jIPOBk4J8AIuIDMgYjuGTKSBukBOFNg0vuhbzx8OgXINiW+GcaY0wf9JQgvgJcB/wR+GZMyeFMUmCVuUy/NzHjILqTUQAX/JcbQLf8wcF5pjHG9OKQCUJVN6jqOao6V1X/FLP/GVX9dl9uLiLniMh6EdkoIjd2c/w0EXlXREIicnGXY2EReS/6WnQYnykuEjKSuiflp8O4Y+D1X0M4wW0fxhjTB4ccSS0iv+npQlX9ek/HRcQL3IFrv6gElojIIlWN7fi/HfgCcEM3t2hV1bk9PSORMvxe2kMRIhHF45HEP1AETvkWPHIFrP0bHPWZxD/TGGN60NNUG9cCq4BHgF3A4f6WPB7YqKqbAUTkIeBC4ECCUNWt0WNDbiWd2Cm/s9L7NCPJwM24AIqmwav/DbM+7ZKGMcYkSU9tEGOBu3BLjl6BWyjob6p6r6re24d7lwI7YrYro/v6KiAiFSLylohcdBjXxUVmvKf87guPB075JuxdCRufH7znGmNMN3pqg6hR1d+r6kdx4yDygTUicsUgxTZRVecDlwG3iciUrieIyDXRJFJRXV0d14d3Ljsa1ym/++LoS9wSpa/8ElQH99nGGBOjL0uOHgN8A7gceBpY2sd778StZd2pLLqvT1R1Z/TnZuAl3Kp2Xc+5S1Xnq+r8kpKSvt66TzL9rlppUEsQAD6/m4Zjx1vw9u8H99nGGBOjp6k2bhWRpcD1wMvAfFW9uksjc0+WANNEpFxE/MBCoE+9kUSkQETSo++LceMwBnVWuwy/+08zaF1dYx1zJUw/D577EexePvjPN8YYei5B/BBXrTQH+A/gXRFZISIrRWRFbzdW1RBuHMUzwFrgEVVdHU08CwBE5DgRqQT+BbhTRDqn9jgSqBCR5cBi4GeHkZjiIiOtswSRhC6nInDhHZBZDI9dBe1Ngx+DMSbl9dQ9p3ygN1fVp4Cnuuz7Ucz7Jbiqp67XvQEcPdDnD0RnI/Wgt0EcCKAQPn0n3LsAnv4eXHRHcuIwxqSsnhqpt3X3wvVMOmXwQkyOjGT0Yuqq/DQ49Xp4735Y+4/kxWGMSUk9tUHkisj3ReR2ETlbnP8DbAYuGbwQk+PAOIhkJgiA02+EMbPh79+Apvj21DLGmJ701AZxHzAdWAl8CdcWcDFwkapeOAixJVVnCSIpjdSxfH741J3Q3uBmfLWur8aYQdJTG8Tk6PoPiMjdwG5ggqqmxHSjnW0QSS9BAIyeCR/7oevVtOJhmLMw2REZY1JATyWIYOcbVQ0DlamSHAACPi8egerG9mSH4px0HUw4Cf7xLXjjdggHe7/GGGMGoKcEMUdEGqKvRmB253sRaRisAJPF4xHOmD6KJ97bSXtoCJQiPF64+B6YeDI8+wP43cmw+aVkR2WMGcF66sXkVdXc6CtHVX0x73MHM8hk+cJHJrGvqYN/LN+d7FCc3HHwuUfh0ocg3A5/vhAeuRLqdvR+rTHGHKZep9pIZadOK2bqqGz+9MZWdKg0DovA9HPhX9+Gj/4QNjwDdxzv5m4KDZHqMGPMiGAJogciwhc+MomVO+tZum1/ssM5WFoATv8OXPcOTD0LXvy/8LuPwKYXkx2ZMWaEsATRi08fU0puwMcfX9+a7FC6lz8BPnsfXP6/rgvsfZ9y1U5t9cmOzBgzzFmC6EWm38fC4yfwz9V72FXXmuxwDm3qmfCvb7pqp3VPwj3nQsOuZEdljBnGLEH0wRUnTkRVue35DckOpWe+dFft9LlHoG4b/OFsqF6f7KiMMcOUJYg+GF+YyTWnTeGRikoeXrI92eH0bsrH4AtPukbrP5wNS+62cRPGmMNmCaKPbjj7CE6ZWsy/P7Ga93bUJTuc3o2bC196DkbPgie/7Xo6rfpfm6rDGNNnliD6yOf18NtL51GSk85X7186dEZY96RgkitJXPYo+DLgsS/C8oeSHZUxZpiwBHEYCrL83HnFsdQ2d3Deb17lkSU7CEeG+F/kInDE2XDtq1B2HDz7Q2gdBiUgY0zSWYI4TEeV5vHIV05ifEEG3318Bef/5lX+uWo3HaFIskPrmccL5/8KWmvhxR8nOxpjzDAgQ2aE8ADNnz9fKyoqBu15qsqTK3fzs6fXUbm/lYLMNBbMGce/zB/PUaV5gxbHYXvqO67R+suLXTuFMSalichSVZ3f7TFLEAMTCkd49f19PPZuJc+t2UtHKMJRpbksPG4CF80rJTu9pxnVk6C1Dm4/DvLHw9XPg8cKkcakMksQg6S+JcgT7+3kwXe2s25PI9NGZfPYtR8hLzMtqXF9yPKH4a/XwNg5cOaPYMqZrq3CGJNyLEEMMlVl8foqrr3vXeaOz+fPVx9PILqE6ZCgCisegcU/hrrtbgrxoz7tfhZPt1KFMSnEEkSSLFq+i68/uIzzjh7D7Zceg8czxP5KD7XD0nvh9V9DQ6Xbl1EAR5wDsz4Fkz/qljw1xoxYliCS6O5XN/PjJ9dyyfwybr3wqKFVkuikCvu3wLY3YeursP4pN9lfIA/mXAYnXAOFk9254SDsXQ2RMKTnQCAXskpcLyljzLBjCSLJfvXsen774kZmjMnh1wvnMX1MTrJD6lmow61Wt/IRWP0EREIw7WwItUFlBQSbDz7flwGjZsCoWZBdAp408PqhaDJMOs3tM8YMSZYghoDF66v4zqPLaWgL8a9nTGHBnHFMLslOdli9a9gNFX+AZQ9AZhFMPAkmnAj+bGhvhLY6qNkMe1dB1Vq3He44+B6jZsKsT8MJX3ElDmPMkGEJYojY19TOjY+v5Pm1ewE4YnQ25x09louPLaOsIDPJ0cWRarQqaiVseQU2vuCqrjIK4ZRvwjGfd20dxpikswQxxOysa+XZ1Xv456o9vLO1FoBTp5Xw+RMncuaRo5CR2OV051JY/FPY+LzbDuS7uaLGzoFZF7mqKO8QGzNiTAqwBDGEVe5v4dGKSh6t2MGu+jZOnVbMLQtmDY/qp/7YsQS2vwH7t8H+rbDjbehoctVX86+CM75vDd7GDCJLEMNAKBzh/re28atnN9AWCnP1KZP56hlTyMsYYoPs4i3Y6qqglj8I6/4BMy6Az9wNaRnJjsyYlNBTgkjoiCgROUdE1ovIRhG5sZvjp4nIuyISEpGLuxy7UkTej76uTGScQ4HP6+ELJ5fz4g1n8Mk54/j9y5s4/ReLueuVTbQFw8kOL3HSMuDIC2DhA3DOz9xyqfcugOaaZEdmTMpLWAlCRLzABuDjQCWwBLhUVdfEnDMJyAVuABap6mPR/YVABTAfUGApcKyq7j/U84Z7CaKrVTvr+cUz63l5QzVlBRn8v8/P58ixKdADaM3f4PEvQ+44lzRGz0p2RMaMaMkqQRwPbFTVzaraATwEXBh7gqpuVdUVQNe5sj8BPKeqtdGk8BxwTgJjHXKOKs3j3quO5y9fOoFgOMLFv3uDxeuqkh1W4s28EK78OwRb4O6zYOVjyY7ImJSVyARRCuyI2a6M7ovbtSJyjYhUiEhFdXV1vwMdyj4ytZi/fe0UJhVncfW9S7j71c1EhvoiRQM14QT4yiswZjY8fjU8cAk8fDnc/xlXuti5NNkRGpMShnW/QlW9C7gLXBVTksNJmDF5AR699iS++dB7/PjJtfx9+S5+9MlZHDtxBI8lyBnjShIv3OLaJdIy3KtyiRvhPfkMmL3QTQnSuBtQNxhv7BybmdaYOElkgtgJjI/ZLovu6+u1Z3S59qW4RDVMZfp9/P7yY/nb8p387Ol1fOZ3b7Bgzji+9tGpQ3/qjv7y+eETP3GvTm0NsPSP8Mbt8MS1bp8n2tPr9V/D6KNg9iUw/kQYczT4MyEcgvrtbuba1v0uqXQ0u9HggVzIHu3Ot1lsjTlIIhupfbhG6jNxv/CXAJep6upuzv0T8I8ujdRLgWOip7yLa6SuPdTzRlojdU+a20PcsXgj97y+hbZghFOnFfOV06ZwyrTiZIc2eIJtULsZske5Edrt9bDqcVh2P+xa5s4RD+SMg6Y9bj6pnkw9Cy76nbufMSkkaeMgROQ84DbAC9yjqj8RkVuBClVdJCLHAX8FCoA2YI+qzopeexXwb9Fb/URV/9jTs1IpQXTa39zBA29v4943t1Hd2M7HZ47m5gWzKM1P8TEEDbtdkti1zA3Gyx8PhVOgYKJLJoE8V7LoaHYlki2vwHM/cvs/fSdM+ViyP4Exg8YGyo1w7aEwf3x9K7c9vwGPCNd//Ai+eHI53qG2/sRQtnc1PHYVVK+DcfNgxvkw/XwYdaS1aZgRzRJEithR28LNi1bzwroqTp1WzK8XzqMwyxb86bOOFnjnLlj7d9gZ/bcUyIfSY6D0WFeyKDve5owyI4oliBSiqjy0ZAc3LVpNcZafOz53DPMmjODeTonSsBs2PufWv9j5LlStAQ27KqppZ0PZfCiZ4V5ZxVbKMMOWJYgUtLKynq8+sJS9DW381yVz+eSccckOaXhra4BNL8D6f8L7z0JrTH8Jfw7kT3CvwslQPBWKprmShz8reTEb0weWIFJUfUuQL99XwZKttfzkoqO57IQJyQ5pZFB1Yy+q10HVOqjb5rrQ7t/melaFWt15vgyYeqYbHT75DOshZYYkSxAprC0Y5l8feJcX11XxvXNm8NUzpiQ7pJEtEoGGnVC93pU01i6KDuQDsse4sRmBXLcaX3uTq5pKz3FjMvLHuwbycfMgb7xVW5lBYQkixQXDEb79yHIWLd/FFz4yiX+/YKb1cBoskYibGqTyHdi9AvasdGt7p+e4lyq0N7iEUV8JkaC7LqPADfobM9sljIknQV5Zcj+LGZF6ShDWHSMFpHk9/Pdn5zIqJ527X9tC5f4Wfr1wHlnp9vusCLIAABRESURBVPUnnMcD449zr94E26BqtRu/sXuFW+e74p4PqqzyJ8DEk6H8dCg/DfL6OrWZMf1jJYgUc9+bW7lp0WpmjsvlV/8yd+RO0zFSRMJujMa2N2Dba7D19Q8ayAunQPmpMOlU1w03I981mFs3XHMYrIrJHGTxuiq+/tAymtpDfGpeKd866wjGF2YmOyzTF5GIK2VsftmNAN/2BnQ0HnxOVgmMP8GVNkqPgcxilzwC+ZY8zIdYgjAfsr+5g9+/vIk/vbGViCo/PH8mnz9pImINo8NLOAR7lsPeNdGG7wbXo2rb626akYOIG7ORMxYKy2HeFTDlTJukMMVZgjCHtKe+jR/8dSUvrKvi0uMncMuCWfh99gtjRKjf6aqn2uqgtQ5aatzEhQ27Yfd70LQXio+A+VdD8TTXDTd7tCtxWNJIGZYgTI8iEeWXz67nf17axAnlhfzi4jlMKLIqpxEt1AFrnoA3b4fdyw8+5vW7UkbeeDfor/gIKJ4OJUdAbpkljxHGEoTpk7+9t5PvPraCYDjC+bPH8ZXTJnNUaV6ywzKJpAr7t0DjXleiaKpy4zgadkLdDti34eBR42lZLlFMOxuO+gyUTE9e7CYuLEGYPtvb0MY9r23hgbe309QeYnZZHhfMHsv5s8fZNOKpqnmfG/i3bz1Ub4A9K1zjOAqjj4a5l7rV/bKKkh2p6QdLEOaw1bcGebRiB4uW72JFZT0Ap04r5vITJ3LmjFH4vFbNkNIa98DqJ9zyrzuXulX9ZpzvVvObciakBZIdoekjSxBmQLbua2bR8l08+M52dte3MS4vwL/MH8/Fx5ZZ91jjGsLfvQ9WPOyqo/w5MP0cKJ3veksVToaCSeBNS3akphuWIExchMIRXlhXxf1vbeO1jftQhZMmF3H+7LGcfkSJJYtUFw7ClpddyWLdkwe3XXj9rrF71Ey3ml+wzU05klcGE050a4Jnl3xwfkeLq8ratQxC7W6eqvyJLtlkFg7+ZxvBLEGYuNtZ18rjSyt5/N1KttW0ADClJIsFc0r53IkTKM5OT3KEJqlUobkaardA7SY38+3e1VC1FsIdbqZbn9+N2Qh3uGvSslwpw5vmuuRqpPt754yD0bOgaCrkjHZdc/PGw9jZbtnYWJGwe2kEPD4bKNgNSxAmYVSVTdXNvLS+ihfXVfHGphr8Pg+fmlvKp48pZe6EfNJ93mSHaYaqUDvseg+2v+kSSjgI4XY3GnzcMW4keFom1O9wyWTf+y7R7F3lplfvOoq8cIrrotu017WTxB73psP442HSKa4k07rf9dqKhNy07KXzU7ILryUIM2g2VjXyx9e38vi7lbQFI6T7PMybkM+Jk4s4cXIRc8fnE0izhGHipKPZ/ZKv2QS7l7lk01LjShU5Y9wKgB4PiNf1xtr2mpsIkdjfe+K2s8fA9HPd2h2TTk2ZXlmWIMygq28N8vbmGt7eUstbm2tYs7sBVUj3eThybC5TSrKZMiqLGWNyOLo0n5Icq5Iyg6S1zo39yCx2JZVQW3Ttjr/Dpheho8mdV3KkG/NROMVVZ5Ue4wYMjrBShiUIk3T1LUGWbP0gWWyqbmJvQ/uB42PzAswdn88J5YWcMLmI6aNz8NiaFWawhYOuYXzLK7DjHajZ6FYMjITc8fQ8lyiKpnywzGxe9OcwXZvcEoQZkhragqzb3ciKyjpWVNazdNt+dta5tQ/8Xg8lOekU56RTlp/BjDE5zByXy+SSbPIz0sgJ+Gwshhkc4ZArcVQucUlj17uu/aOt7uDzfBkfdOstnOySSGfpI2fMkE0eliDMsFG5v4W3N9eyYW8j1U3tVDe2s62mhe21LR86Nyfdx6jcdEbnBijJSScvI43cQBqFWX6mjc5m+pgcSrLTbYZakxht9W46ks4G9Lrtbk3ymk0uoXT2zgKXMI44x71Kj3GrCQ4RliDMsNfYFmT9nka21bTQ2BakvjXE/pYOqhrb2FPfxr6mDhragjS0BonE/JMuyvIzc1wuM8fmMnOca/uYVJxFtq2mZxIpEnZLyNZuctOUbHzBVVuFo9Wq2aNd6aJgkhvjkTfejRMZPQvSswc1VEsQJmWoKjXNHWzY08i6PY2s29PA6l0NbNjbSDD8wb/10bnplBdnMbkkm8nFWYzLz2BMXoAxuQEy0rz4fR78Pg9pVo1l4qW9Cba+6saC1GxyyaNuOzTuPnjMR+Fk1xheWA4F5a60EQm69pHOZWf98RuUagnCpLyOUITN+5rYuq+Zzfua2VzdzJZ9zWyubmJ/S/CQ1/l9HnLSfeQEfORl+inIdFVYs8blcUJ5IUeOzcVrjelmIMJBV9qoXgd7VrkR5J3VVMEPV63iC7gkMf4EKJjoSiEF5W7QYD9YgjCmB/UtQXbWtbKnoZU99e20h8J0hCK0hyI0d4RobAvR0BqkvjXI/pYO9jV2sKehDXDtIEeOy2X66ByOGJPDnLI8Zo7NtQZ0M3CqboxHsMWNLhcvVK2Bjc/D+89BzfsfnDt2DnzllX49xhKEMXG2u76Vd7bUsmRrLWt3N7JhTyON7a4rZKbfy7wJ+Rw5JpfykizKi7IYkxegKCudnIDPuu+a+Ai2uQby/dvcWL+pZ/XrNklLECJyDvBrwAvcrao/63I8HfgzcCxQA3xWVbeKyCRgLbA+eupbqnptT8+yBGGSSVXZWdfKsu11VGytpWLbfjZWNdEeOng+IZ9HKM5OZ2x+gLF5AcbkZrifeQEml2RxxOgca/cwg6qnBJGwrhwi4gXuAD4OVAJLRGSRqq6JOe1qYL+qThWRhcB/Ap+NHtukqnMTFZ8x8SQilBVkUlaQySfnjAPcUq57GtrYsq+ZqsY2apo6qGnuoKqhnT0Nrazb3chL66tp6QgfuE+6z8PMcbnMGJNDaX4GpQUZjMnNoCTHT1GW68prJRAzWBLZ1+94YKOqbgYQkYeAC4HYBHEhcHP0/WPA7WKd1s0I4fEI4/IzGNfDSnyqSmN7iN11bazf28iKHXUsr6zj2dV7qWnu+ND5HoHcjDTyMtLIz0ijKDudwiw/Rdl+RuUEGJWTzti8ABMKMynJsTEgZmASmSBKgR0x25XACYc6R1VDIlIPdM6QVS4iy4AG4Ieq+mrXB4jINcA1ABMmTIhv9MYMAhEhN5BG7pg0po/JYUG09AHQ2hFmZ10rVQ1t7GvuoKapndrmDuqjDea1zR3sbWhj7e4G9jW1H9SNFyCQ5mFcfgY56T4y/T6y0n1kp3vJSveRm5HGuLwApQUZjM3LoCjbT2Gm3xrXzUGG6mih3cAEVa0RkWOBJ0Rklqo2xJ6kqncBd4Frg0hCnMYkTIbfy9RR2Uwd1fvAKVWlriVIVWM7u+tb2V7bwraaFvbUt9HUHqKlI8TOulaa20M0t4doaAt+KKEA5EWnMclOd680rwefV/B6hHSfh3Sfl3Sfh9yMNAoy08jP9JOV7iXT7yPT78XbWWIRyE73UZDpJz8zjex0n5VmhqFEJoidwPiY7bLovu7OqRQRH5AH1KhrOW8HUNWlIrIJOAKwVmhjuiEiFGT5KcjyM31M79M4RCLKvqZ2Kuta2V3XRm1zO/uaOtjf0kFTW4jGaCIJhZX2UJhQRA90/W3tCNPQFjyo7aQ3Oek+JhRlMqEwk4lFWUwuzqK8JItJRVkUZ/steQxRiUwQS4BpIlKOSwQLgcu6nLMIuBJ4E7gYeFFVVURKgFpVDYvIZGAasDmBsRqTUjweYVRugFG5Aehn7Wx7KEx9S5DmjjDN7SFag2Ei0XlOIgpN7W46lP3NHeyqc6Wa9XsbeX7t3oNKL9npPiYVZ1Kan0FJTjol2QHG5rt2lAmFmYzODdhgxCRJWIKItilcBzyD6+Z6j6quFpFbgQpVXQT8AbhPRDYCtbgkAnAacKuIBIEIcK2q1n74KcaYZEn3eRmVe/iLP4XCEXbVtR0Y2b5lXzNbalrYsq+Zt7fUUtdlZHua1zX2jy/IPNAleHRugKIsPwG/l4DPS3qaB5/HVYX5vR4CaW5fdrqPjDSvlVD6yQbKGWOGlPZQmD31bWyvdbP4Vu5vpXJ/KztqW9hd30p1Y/tBEzL2Jt3noTDLT24gDZ9X8HmENK+H7ICPnEAauQEfRdnpFGf7Ke7sFRatrsvLSBvx41KSMg7CGGP6I93nZWJRFhOLsro9HgpHDrSXtAXDtAbDtIcihMNKKKIEwxHagmHagmGa2sPsb+mgtrmDhtYg4YgeaE+pbe5gW03LgSlUDvW3cqbfS07Ad6CBPpDmpSjbz+icAKNy0xmVG2B0TjolOekE0rykeQWvx4NXBI8HvB450Og/3EoyliCMMcOKz+txM+/mBeJ2z1A4wv6WIPuiXYlrml3bSUNrMDqNfMjN0RV2jfT7mjpYs8t1L+5racbrEfKivb+KslxJpSDL9fLKz0ijIFpyKYyWXDL9PjLTXRVamleSklwsQRhjUp4vuoLh4a6NHo4oNU3t7G1op7qpjfZghFBECUUihCMQUSUcUZraQtS1dlDXEp3wsamDjdVN1G0LUt/a0W2X467SvILP47odp3k9pEV/+r0eZpXm8dtL5/X34x+SJQhjjOknb2xvMPL6dQ9VpbkjzP5oyaW2uZ3GthAt0d5h7aEIHaEIHeEIoXCEYNhVo4XCSjDitscXHHq0/kBYgjDGmCQS+aCNYnxh/BYCioeR3TxvjDGm3yxBGGOM6ZYlCGOMMd2yBGGMMaZbliCMMcZ0yxKEMcaYblmCMMYY0y1LEMYYY7o1YmZzFZFqYNsAblEM7ItTOMNFqn3mVPu8YJ85VQzkM09U1ZLuDoyYBDFQIlJxqClvR6pU+8yp9nnBPnOqSNRntiomY4wx3bIEYYwxpluWID5wV7IDSIJU+8yp9nnBPnOqSMhntjYIY4wx3bIShDHGmG5ZgjDGGNOtlE8QInKOiKwXkY0icmOy40kEERkvIotFZI2IrBaRb0T3F4rIcyLyfvRnQbJjjTcR8YrIMhH5R3S7XETejn7fD4uIP9kxxpOI5IvIYyKyTkTWishJI/17FpFvRf9drxKRB0UkMNK+ZxG5R0SqRGRVzL5uv1dxfhP97CtE5Jj+PjelE4SIeIE7gHOBmcClIjIzuVElRAj4tqrOBE4Evhb9nDcCL6jqNOCF6PZI8w1gbcz2fwL/rapTgf3A1UmJKnF+DfxTVWcAc3CffcR+zyJSCnwdmK+qRwFeYCEj73v+E3BOl32H+l7PBaZFX9cAv+vvQ1M6QQDHAxtVdbOqdgAPARcmOaa4U9Xdqvpu9H0j7pdGKe6z3hs97V7gouREmBgiUgacD9wd3RbgY8Bj0VNG1GcWkTzgNOAPAKraoap1jPDvGbd0coaI+IBMYDcj7HtW1VeA2i67D/W9Xgj8WZ23gHwRGduf56Z6gigFdsRsV0b3jVgiMgmYB7wNjFbV3dFDe4DRSQorUW4DvgtEottFQJ2qhqLbI+37LgeqgT9Gq9XuFpEsRvD3rKo7gV8C23GJoR5Yysj+njsd6nuN2++1VE8QKUVEsoHHgW+qakPsMXX9nUdMn2cRuQCoUtWlyY5lEPmAY4Dfqeo8oJku1Ukj8HsuwP3FXA6MA7L4cFXMiJeo7zXVE8ROYHzMdll034gjImm45PCAqv5vdPfezqJn9GdVsuJLgJOBBSKyFVd1+DFc/Xx+tCoCRt73XQlUqurb0e3HcAljJH/PZwFbVLVaVYPA/+K++5H8PXc61Pcat99rqZ4glgDToj0e/LjGrUVJjinuonXvfwDWqup/xRxaBFwZfX8l8LfBji1RVPX7qlqmqpNw3+uLqvo5YDFwcfS0kfaZ9wA7RGR6dNeZwBpG8PeMq1o6UUQyo//OOz/ziP2eYxzqe10EfD7am+lEoD6mKuqwpPxIahE5D1dX7QXuUdWfJDmkuBORU4BXgZV8UB//b7h2iEeACbip0i9R1a4NYcOeiJwB3KCqF4jIZFyJohBYBlyuqu3JjC+eRGQurlHeD2wGvoj7Q3DEfs8icgvwWVxvvWXAl3B17iPmexaRB4EzcNN67wVuAp6gm+81mihvx1W1tQBfVNWKfj031ROEMcaY7qV6FZMxxphDsARhjDGmW5YgjDHGdMsShDHGmG5ZgjDGGNMtSxDGDICI3CoiZ8XhPk3xiMeYeLJursYMASLSpKrZyY7DmFhWgjCmCxG5XETeEZH3ROTO6JoSTSLy39F1B14QkZLouX8SkYuj738WXXNjhYj8Mrpvkoi8GN33gohMiO4vF5E3RWSliPy4y/O/IyJLotfcEt2XJSJPisjy6LoHnx3c/yomFVmCMCaGiByJG5V7sqrOBcLA53CTwFWo6izgZdxI1tjrioBPAbNUdTbQ+Uv/t8C90X0PAL+J7v81blK9o3GzkHbe52zcPP7HA3OBY0XkNNyo2F2qOie67sE/4/7hjenCEoQxBzsTOBZYIiLvRbcn46YoeTh6zv3AKV2uqwfagD+IyKdxUxwAnAT8Jfr+vpjrTgYejNnf6ezoaxnwLjADlzBWAh8Xkf8UkVNVtX6An9OYXvl6P8WYlCK4v/i/f9BOkX/vct5BjXeqGhKR43EJ5WLgOtwMsj3prgFQgP9Q1Ts/dMAtHXke8GMReUFVb+3l/sYMiJUgjDnYC8DFIjIKDqz7OxH3/0rn7KCXAa/FXhRdayNPVZ8CvoVb7hPgDdxssuCqql6Nvn+9y/5OzwBXRe+HiJSKyCgRGQe0qOr9wC9w03gbk1BWgjAmhqquEZEfAs+KiAcIAl/DLb5zfPRYFa6dIlYO8DcRCeBKAddH9/8f3Apv38Gt9vbF6P5vAH8Rke8RMxW1qj4bbQd5003KSRNwOTAV+IWIRKIxfTW+n9yYD7Nursb0gXVDNanIqpiMMcZ0y0oQxhhjumUlCGOMMd2yBGGMMaZbliCMMcZ0yxKEMcaYblmCMMYY063/D9g7ljJqKrBdAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "episodes = 100 + 1\n",
    "td_errors = batch_updating('TD', episodes)\n",
    "mc_errors = batch_updating('MC', episodes)\n",
    "\n",
    "plt.plot(td_errors, label='TD')\n",
    "plt.plot(mc_errors, label='MC')\n",
    "plt.xlabel('episodes')\n",
    "plt.ylabel('RMS error')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}