Example_6.5_Windy Gridworld.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "這邊參考的主要來源是[ShangtongZhang](https://github.com/ShangtongZhang/reinforcement-learning-an-introduction/blob/master/chapter06/windy_grid_world.py)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example 6.5: Windy Gridworld\n",
    "![](https://i.imgur.com/MTwwarC.png)\n",
    "\n",
    "如上圖的小插圖所示，那是一個標準的gridworld，有開始(S)與目標(G)狀態，有一點不同：有一個側風在中間由下往上吹。那actions一樣是標準的那四個，上、下、右、左，但是到中間之後你的next state會被風往上吹而導致偏移，造成偏移的強度會依不同的column而有所不同。風吹的強度也在每個column上給出，一看你就知道會被吹幾格。舉例來說，如果你在目標(G)右邊的格子，然後你執行action是向左，那你就會被帶到目標(G)的上面一格。這是undiscounted episodic task，在你到達目標之前，你每次都會得到reward=-1。\n",
    "\n",
    "那個趨勢線則是你用$\\epsilon-$greedy Sarsa來處理這個任務的結果，其$\\epsilon=0.1, \\alpha=0.5$，然後所有的$s, a$的初始$Q(s,a)=0$。上圖可以看的到，斜率增加說明的是，隨著時間的推移，我們可以更快的到達目標(G)。在經過8000個time steps之後，greedy policy早就是optimal(插圖說明的就是它的trajectory)；一直是$\\epsilon-$greedy exploration的話，大概每個episodes都是平均17個steps可以完成，比最小15個steps還要多2個steps。注意到，這個任務用MC可能不是那麼輕鬆，因為不是所有的policies都能夠保證終止。如果曾經有policy被發現會導致agent停留在同一個state，那下一個episode它就永遠不會終止。像是Sarsa這種online learning methods不會有這樣的問題，因為它們會很快的在episode的期間學習，如果policies不夠好，那就換一個。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$(S_t, A_t, R_{t+1}, S_{t+1}, A_{t+1})$，每個元素的第一個字拿出來，就是Sarsa，下面給出公式6.7做為參考：\n",
    "$$Q(S_t,A_t)\\leftarrow Q(S_t,A_t) + \\alpha[R_{t+1} + \\gamma Q(S_{t+1}, A_{t+1}) - Q(S_t, A_t)] \\tag{6.7}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://i.imgur.com/cpcvCNi.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先載入需求套件"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先定義環境參數"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "WORLD_HEIGHT = 7\n",
    "WORLD_WIDTH = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定義風速在每個column的影響"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "WIND = [0, 0, 0, 1, 1, 1, 2, 2, 1, 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定義ACTION"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 2, 3]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ACTION_UP = 0\n",
    "ACTION_DOWN = 1\n",
    "ACTION_LEFT = 2\n",
    "ACTION_RIGHT = 3\n",
    "ACTIONS = [ACTION_UP, ACTION_DOWN, ACTION_LEFT, ACTION_RIGHT]\n",
    "ACTIONS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定義$\\epsilon$，這代表著你的agent去探索其它非最佳的action的機率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "EPSILON = 0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定義step size，$\\alpha$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "ALPHA = 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定義REWARD，根據範例說明，每個time step都會-1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "REWARD = -1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定義起始位置、結束位置"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "START_STATE = [3, 0]\n",
    "GOAL_STATE = [3, 7]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "設置一個函數來定義每一個time step選擇action的結果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def step(state, action):\n",
    "    \"\"\"每個time step的結果    \n",
    "    state: 當下的state\n",
    "    action: 選擇的action\n",
    "    \n",
    "    每次的移動都要考慮進去風對於移動的影響，\n",
    "    因此會做一個判斷，避免出界\n",
    "    \n",
    "    return:\n",
    "        next state\n",
    "    \"\"\"\n",
    "    i, j = state\n",
    "    if action == ACTION_UP:\n",
    "        # 往上，column不動，row要注意出界        \n",
    "        return [max(i - 1 - WIND[j], 0), j]\n",
    "    elif action == ACTION_DOWN:\n",
    "        # 往下，column不動，row要注意出界\n",
    "        # 取min是因為避免出下界\n",
    "        # 取max是怕風大太強吹出上界\n",
    "        return [max(min(i + 1 - WIND[j], WORLD_HEIGHT - 1), 0), j]\n",
    "    elif action == ACTION_LEFT:\n",
    "        # 往左，要考慮到風吹的影響，也要注意出界\n",
    "        return [max(i - WIND[j], 0), max(j - 1, 0)]\n",
    "    elif action == ACTION_RIGHT:\n",
    "        return [max(i - WIND[j], 0), min(j + 1, WORLD_WIDTH - 1)]\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([3, 0], [2, 0])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "state = START_STATE\n",
    "action = ACTION_UP\n",
    "next_state = step(state, action)\n",
    "state, next_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "設置一個函數來執行episode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def episode(q_values):\n",
    "    time = 0\n",
    "    # 設置初始狀態\n",
    "    state = START_STATE\n",
    "    # 決定action\n",
    "    # 如果根據EPSILON的機率來抽真的抽到1，那就是探索，隨機從中選擇一個action\n",
    "    if np.random.binomial(1, EPSILON) == 1:\n",
    "        action = np.random.choice(ACTIONS)\n",
    "    else:\n",
    "        # 如果是greedy，那就從q_value裡面找一個最大的\n",
    "        # 如果有很多個最大值，那就隨便來一個吧\n",
    "        # 先取得該state的value，會分別記錄上、下、左、右的value\n",
    "        _values = q_values[state[0], state[1], :]\n",
    "        # 從中取最大值，np.random.choice的目的在於，如果有多個最大值就隨機取\n",
    "        # 下面的List comprehension說的是，對_values做for loop\n",
    "        # 然後如果_value是_values中最大的那個值，那就記錄_action\n",
    "        action = np.random.choice([_action for _action, _value in enumerate(_values) if _value == np.max(_values)])\n",
    "    \n",
    "    # 開始跑至GOAL STATE\n",
    "    while state != GOAL_STATE:\n",
    "        next_state = step(state, action)\n",
    "        # 利用next state來決定next action\n",
    "        if np.random.binomial(1, EPSILON) == 1:\n",
    "            next_action = np.random.choice(ACTIONS)\n",
    "        else:\n",
    "            _values = q_values[next_state[0], next_state[1], :]        \n",
    "            next_action = np.random.choice([_action for _action, _value in enumerate(_values) if _value == np.max(_values)])\n",
    "\n",
    "        # 每過一個time step就做一次的SARSA update\n",
    "        # 因為是undiscount，因此就沒有特別計算discount\n",
    "        q_values[state[0], state[1], action] += \\\n",
    "            ALPHA * (REWARD + \n",
    "                         q_values[next_state[0], next_state[1], next_action] - \n",
    "                             q_values[state[0], state[1], action])\n",
    "        state = next_state\n",
    "        action = next_action\n",
    "        time += 1\n",
    "    \n",
    "    return time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "q_value = np.random.randint(1, 10, (WORLD_HEIGHT, WORLD_WIDTH, 4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([6, 5, 1, 6])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "_values = q_value[state[0], state[1], :]\n",
    "_values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 3]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[_action for _action, _value in enumerate(_values) if _value == np.max(_values)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面等價於上面的list comprehension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 6\n",
      "3 6\n"
     ]
    }
   ],
   "source": [
    "for a, v in enumerate(_values):\n",
    "    if v == np.max(_values):\n",
    "        print(a, v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面我們設置一個函數來執行"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run(episodes):\n",
    "    # 那個4就是4個action\n",
    "    q_values = np.zeros((WORLD_HEIGHT, WORLD_WIDTH, 4))\n",
    "    # 記錄每個episode執行幾個time step\n",
    "    steps = []    \n",
    "    \n",
    "    # 跑500次\n",
    "    for i in range(episodes):\n",
    "        steps.append(episode(q_values))\n",
    "        \n",
    "    # 計算累加\n",
    "    steps = np.add.accumulate(steps)\n",
    "    \n",
    "    plt.plot(steps, np.arange(1, len(steps) + 1))\n",
    "    plt.xlabel('Time Steps')\n",
    "    plt.ylabel('Episodes')\n",
    "    \n",
    "    plt.show()\n",
    "    # 這樣寫應該不是很好，又是show圖又是回傳的\n",
    "    # 不過就只是為了方便\n",
    "    return q_values\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZMAAAEGCAYAAACgt3iRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXhV1dn38e8tM8g8yRRBBWQQEA/g9FhHVKyiLVXUVhz6Yh/bap0qqK1T20elrUO1KlatVosCgqBVEUXq0CoGZQiThDCGQJB5CmS43z/2ih6RISQ5OSfJ73Nd58rea093Fklu9tprr2XujoiISFkckuwARESk8lMyERGRMlMyERGRMlMyERGRMlMyERGRMquZ7AASoUWLFt6xY8dkhyEiUqnMnDnzK3dvWZpjq2Qy6dixI+np6ckOQ0SkUjGz5aU9Vs1cIiJSZkomIiJSZkomIiJSZkomIiJSZkomIiJSZglNJmZ2o5nNM7MMMxtjZnXNrJOZfWpmmWb2ipnVDvvWCeuZYXvHuPOMDOWLzOzsRMYsIiIHL2HJxMzaAdcDMXfvCdQAhgIPAA+5+1HARuCacMg1wMZQ/lDYDzPrHo7rAZwD/NXMaiQqbhEROXiJbuaqCdQzs5pAfSAHOB0YH7Y/D1wYlgeHdcL2M8zMQvnL7r7L3ZcCmUD/BMctIlKpuDtjZqxg2sK1Sbl+wpKJu2cDfwRWECWRzcBMYJO7F4TdVgHtwnI7YGU4tiDs3zy+fC/HfM3MhptZupmlr1u3rvy/IRGRFLVh+26ue+lzRk6Yy4TPs5MSQyKbuZoS3VV0AtoCDYiaqRLC3Ue7e8zdYy1blmo0ABGRSmfawrWc/fAHvLtgLbedczSPDj02KXEkcjiVM4Gl7r4OwMwmACcBTcysZrj7aA8Up9FsoAOwKjSLNQbWx5UXiz9GRKRa2rargN//az5jZqzk6MMa8vxV/enetlHS4knkM5MVwPFmVj88+zgDmA+8DwwJ+wwDJoXlyWGdsH2aR3MKTwaGht5enYDOwIwExi0iktJmLN3AuY98wMufreRn3zuSSb84KamJBBJ4Z+Lun5rZeOBzoAD4AhgN/At42cx+F8qeCYc8A/zDzDKBDUQ9uHD3eWY2ligRFQA/d/fCRMUtIpKqdhUU8ud3vmT0h1l0aFqfsdeeQL+OzZIdFgAW/ee/aonFYq5Rg0WkKsnI3szNY2ezaO1WLhuQxh2DutGgTvneD5jZTHePlebYKjkEvYhIVVFU5PztoywefHsRzRrU5rmr+nFa11bJDus7lExERFLU+m27uGnsbP795TrO7tGaB3/Ym8b1ayU7rL1SMhERSUFvZ+Rwx8QMtu4q4L4Le/LjAWlEfZlSk5KJiEgK2bwjn7smZ/DarNX0bNeIf/6oD10Pa5jssA5IyUREJEW8vzCXERPmsH7bbm48swvXnXYktWpUjsHdlUxERJJs264C7p48j/EzV9G1dUOeGdaPnu0aJzusg6JkIiKSRDOXb+TGV2axauMOfnHaUfzyjKOoU7PyDYyuZCIikgS7Cgp5aOpiRn+whLZN6vFKCr2AWBpKJiIiFWzx2q38cswXLFyzlaH9OnDHed1oWDc1u/yWlJKJiEgFKSpynv14KQ9OWUTDOjV5ZliMM7q1TnZY5ULJRESkAmRv2snNY2fxSdYGzuzWij/84BhaNayb7LDKjZKJiEiC/WtODiMmzKGoyHnwh734Uax9Sr+AWBpKJiIiCbJjdwH3vbGAMTNW0LtDE/4y9FjSmtdPdlgJoWQiIpIAxV1+V27cwf+eeiQ3ndWl0ryAWBpKJiIi5aigsIjH3s/kL9MyadO4Lq8MP4H+nSpvl9+SUjIRESkn81Zv5jevZfD5ik384Nh23D24B40qeZffkkrYPZeZdTWzWXGfLWb2KzNrZmZTzWxx+No07G9m9qiZZZrZHDPrG3euYWH/xWY2bN9XFRGpeLsKChk1ZSEXPPYxWV9t55GhffjzJX2qTSKBxE7buwjoA2BmNYBsYCIwAnjP3e83sxFh/TbgXKL53TsDA4AngAFm1gy4C4gBDsw0s8nuvjFRsYuIlNTM5RsZOWEOX67dxo+Oa8+d53VP2TlHEqmimrnOAJa4+3IzGwycGsqfB6YTJZPBwAsezSP8iZk1MbM2Yd+p7r4BwMymAucAYyoodhGR79ial8+oKYv4xyfLadOobsrOgFhRKiqZDOWbP/6t3T0nLK8Bil//bAesjDtmVSjbV/m3mNlwYDhAWlpauQUuIrKnd+ev5c7XMli7NY8rT+zILQO7lvt87JVNwr97M6sNXACM3HObu7uZeXlcx91HA6MBYrFYuZxTRCTeuq27uPv1efxrTg5dWzfkiR/35di0pskOKyVURCo9F/jc3deG9bVm1sbdc0IzVm4ozwY6xB3XPpRl802zWHH59IRGLCKyh8mzV/PbSRns2FXILQO7MPyUI6lds+q+N3KwKqImLuXbzzcmA8U9soYBk+LKrwi9uo4HNofmsCnAQDNrGnp+DQxlIiIJt37bLn455guuH/MFHZs34M0bTuYXp3dWItlDQu9MzKwBcBZwbVzx/cBYM7sGWA5cHMrfBAYBmcAO4CoAd99gZvcBn4X97i1+GC8ikijuzutzcrh78jy25uVz01lduO7UI6lZhd9iLwuLOk9VLbFYzNPT05MdhohUUrlb8rjjtQymzl9L7/aNeXBIb7oe1jDZYSWcmc1091hpjq3e3Q9EROK4O6/NyubuyfPJyy9k5LlHc83JnXQ3UgJKJiIiwNotedwxcS7vLsjluMOb8uCQXhzZ8tBkh1VpKJmISLW2q6CQv3+8jMemZbK7sIg7z+vGVSd1osYhVWu+kURTMhGRasndmTJvLX94cwErNuzg9KNb8Zvvd6dTiwbJDq1SUjIRkWonI3sz970xn0+XbqBL60N54er+nNKlZbLDqtSUTESk2sjdmscfpyxi3MxVNK1fm99d2JOh/TroAXs5UDIRkSpvd0ERf/soi8fDc5GfntyJX5zemcb1qt/ovomiZCIiVVr6sg3c+VoGC9ds5azurbljUDc66rlIuVMyEZEqafOOfO59Yz6vfr6KwxrV5ekrYpzVvfWBD5RSUTIRkSpn6vy13DFxLuu37+bnpx3JdaceVe2HiE801a6IVBkbt+/mntfn8dqs1Rx9WEOevbIfPds1TnZY1YKSiYhUCW9n5HDna/PYtGM3vzqzM9edepRG9q1ASiYiUqmt37aL306OJqzq0bYRL1zdn+5tGyU7rGpHyUREKiV3519zc/jtpGiI+FsGduHa7x1JLb0zkhRKJiJS6WzYvps7Js7lrYw19GrfmFFDjq8WQ8SnMiUTEalU3l+Yy63j57BlZz6/Pqcrw//nCL3BngIS+i9gZk3MbLyZLTSzBWZ2gpk1M7OpZrY4fG0a9jUze9TMMs1sjpn1jTvPsLD/YjMbtu8rikhVlbs1j5+/9DlX/f0zmjeozaRfnMR1px6lRJIiEn1n8gjwtrsPMbPaQH3gduA9d7/fzEYAI4DbgHOBzuEzAHgCGGBmzYC7gBjgwEwzm+zuGxMcu4ikiDfmrObO1zLYubuQm8/qwv875Qjq1qqR7LAkTsKSiZk1Bk4BrgRw993AbjMbDJwadnsemE6UTAYDL3g0j/An4a6mTdh3avG872Y2FTgHGJOo2EUkNWzcvpvfTMrgjTk59O7QhD/9qDdHtdKEVakokXcmnYB1wHNm1huYCdwAtHb3nLDPGqB4fIN2wMq441eFsn2Vf4uZDQeGA6SlpZXfdyEiSTFt4Vpue3Uum3bs5tazu3LtKXo2ksoS+S9TE+gLPOHuxwLbiZq0vhbuQrw8Lubuo9095u6xli01L4FIZbV9VwEjJ8zh6r+nR89Gfn4yPz9Nz0ZSXSLvTFYBq9z907A+niiZrDWzNu6eE5qxcsP2bKBD3PHtQ1k23zSLFZdPT2DcIpIk0xau5a7J81i1cSfXfu8IbjqrC3Vq6tlIZZCwVO/ua4CVZtY1FJ0BzAcmA8U9soYBk8LyZOCK0KvreGBzaA6bAgw0s6ah59fAUCYiVUR0NzKXq/+eTp2aNRh77QmMPLebEkklkujeXL8EXgo9ubKAq4gS2FgzuwZYDlwc9n0TGARkAjvCvrj7BjO7D/gs7Hdv8cN4Ean80pdt4OZxs1mxYQfXnnIENw/sqjG1KiGLHltULbFYzNPT05MdhojsR15+IQ9N/ZLRH2bRvmk9/jikNwOOaJ7ssKo1M5vp7rHSHKs34EWkwmVkb+amsbP4cu02LhuQxu2DunGo5hup1PSvJyIVpqjIefbjpTzw9kKa1q/Nc1f147SurZIdlpQDJRMRqRC5W/O4eexsPlz8FQO7t+aBH/aiaYPayQ5LyomSiYgk3LSFa7l13By27y7g9xf15LL+aZhZssOScqRkIiIJU1BYxKh3FvHUv7M4+rCGvHzp8XRuraHiqyIlExFJiHmrN3PruDnMz9nCZQPS+O33u2twxipMyUREylV+YRFPTF/Co+8tpkn92jz5476c07NNssOSBFMyEZFy88WKjfx20jzmZm9mcJ+23HNBD5rU10P26kDJRETKrPgFxKc/zKLFoXX46+V9GXSM7kaqEyUTESmTOas2cfPY2SzO3cal/dO4fdDRNKxbK9lhSQVTMhGRUsnLL+Tx9zP56/QltDy0Dn+/qh+n6gXEakvJREQOyvptu3ht1mqe+TCL1Zvz+EHfdtx1fg8a19PdSHWmZCIiJTZl3hpGTpjLhu276d2+MQ9d0keDMwqgZCIiJZCXX8h9b8znpU9X0LNdI168ZgDd2zZKdliSQpRMRGS/Pslaz+0T5pL11XaGn3IEt2i+EdmLhP5EmNkyM5trZrPMLD2UNTOzqWa2OHxtGsrNzB41s0wzm2NmfePOMyzsv9jMhu3reiJSfvLyC7nn9XkMHf0JBUXOi9cM4PZB3ZRIZK8q4s7kNHf/Km59BPCeu99vZiPC+m3AuUDn8BkAPAEMMLNmwF1ADHBgpplNdveNFRC7SLX0xYqN3DxuNlnrtjPshMO57dyjqV9bDRmyb8n46RgMnBqWnwemEyWTwcALHk39+ImZNTGzNmHfqcVT9ZrZVOAcYEzFhi1S9e0uKOLR9xbz1+mZHNaoLi/9dAAnHdUi2WFJJZDoZOLAO2bmwFPuPhpo7e45YfsaoHVYbgesjDt2VSjbV/m3mNlwYDhAWlpaeX4PItXCojVbufGVWczP2cKQ49rz2/O700gvH0oJJTqZnOzu2WbWCphqZgvjN7q7h0RTZiFRjYZoDvjyOKdIdZBfWMRT/17Co+9l0qheTZ6+IsZZ3Vsf+ECROAlNJu6eHb7mmtlEoD+w1szauHtOaMbKDbtnAx3iDm8fyrL5plmsuHx6IuMWqS4WrtnCTa/MZn7OFs7r1YZ7L+hB80PrJDssqYQS1i3DzBqYWcPiZWAgkAFMBop7ZA0DJoXlycAVoVfX8cDm0Bw2BRhoZk1Dz6+BoUxESqmoyBn9wRIu+MvH5G7N46mfHMfjl/VVIpFSS+SdSWtgYpiasybwT3d/28w+A8aa2TXAcuDisP+bwCAgE9gBXAXg7hvM7D7gs7DfvcUP40Xk4K1Yv4Nbx8/m06UbOLtHa/5w0TFKIlJmFnWeqlpisZinp6cnOwyRlOLujJmxkvvemE+NQ4zffr87P4q111zs8jUzm+nusdIcW6I7EzO7AXgO2Ar8DTgWGOHu75TmoiJSsdZt3cXICXN4d0EuJx3VnFFDetO2Sb1khyVVSEmbua5290fM7GygKfAT4B+AkolIins7Yw23T5zLtl0F3HleN64+qROHHKK7ESlfJU0mxT95g4B/uPs8072xSErbvDOfe16fx4TPs+nZrhEPXdyHzq0bJjssqaJKmkxmmtk7QCdgZOilVZS4sESkLD7O/Ipbx81m7dZdXH9GZ355+lHUqqExtSRxSppMrgH6AFnuvsPMmhN6W4lI6ti5u5AH3l7I3/+zjCNaNuDV/z2RPh2aJDssqQZKmkwc6A58H7gXaADUTVRQInLwZi7fyK3jo8EZrzyxI7edczT1atdIdlhSTZQ0mfyVqFnrdKJkshV4FeiXoLhEpIR2FxTx0Ltf8tS/l9CmcT0NzihJUdJkMsDd+5rZFwDuvtHMaicwLhEpgSXrtnH9mC+Yt3oLl8Q68Jvzu3NoHQ0VLxWvpD91+WZWg6i5CzNriR7AiySNu/PyZyu59/X51K11CKN/chwDexyW7LCkGitpMnkUmAi0MrPfA0OAOxMWlYjs0/L127nztQw+XPwVJxzRnIcu6cNhjfUIU5KrRMnE3V8ys5nAGUTvnFzo7gsSGpmIfMu2XQWM/iCLJ/+9hFqHGPcN7sHlAw7XC4iSEvabTMKUucVyiZvd0MyaacBFkYrx2bIN3PjKLFZt3Mn5vdty53ndaN1IdyOSOg50ZzKT6DmJAWnAxrDcBFhB9BKjiCTI7oIiHn73S5789xLaN63PuJ+dQL+OzQ58oEgF228ycfdOAGb2NDDR3d8M6+cCFyY+PJHqa/HarfzqlVnqqSWVQkl/Mo939/9XvOLub5nZgwmKSaRaKypyXvjvMv7vrYU0qFOTp35yHGerp5akuJImk9VmdifwYli/HFidmJBEqq+1W/K4ZdxsPlz8Fad1bckDQ3rRqqGejUjqK+nIb5cCLYm6B08EWoWyAzKzGmb2hZm9EdY7mdmnZpZpZq8Uv/xoZnXCembY3jHuHCND+aIwDL5IlfPm3BzOfvgD0pdt5HcX9uTZK/spkUilUdKuwRuAG8Jowe7u2w7iGjcAC4BGYf0B4CF3f9nMniQaRPKJ8HWjux9lZkPDfpeYWXdgKNADaAu8a2Zd3L3wIGIQSVlb8vK5e3I0VHzv9o156JI+HNHy0GSHJXJQSnRnYmbHhKFUMoB5ZjbTzHqW4Lj2wHlEszMS5kA5HRgfdnmebx7kDw7rhO1nhP0HAy+7+y53X0o0R3z/ksQtkuo+zVrPuQ9/yKRZq7n+jM6M/98TlUikUirpM5OngJvc/X0AMzsVGA2ceIDjHgZ+DRTPyNMc2OTuBWF9FdAuLLcDVgK4e4GZbQ77twM+iTtn/DFfM7PhwHCAtLS0En5bIsmxq6CQP0/9ktEfZJHWLOry2zetabLDEim1kiaTBsWJBMDdp5tZg/0dYGbfB3LdfWZIPgnl7qOJEhyxWMwTfT2R0vpy7VZueHkWC3K2cGn/NO48rxsN1OVXKrmS/gRnmdlviOZ9B/gxkHWAY04CLjCzQURznzQCHgGamFnNcHfSHsgO+2cDHYBVZlYTaAysjysvFn+MSKVRVOQ8959lPPD2QhrWqcnfrohxZvfWyQ5LpFyUtDfX1US9uSaET4tQtk/uPtLd27t7R6IH6NPc/XLgfaKBIgGGAZPC8uSwTtg+zd09lA8Nvb06AZ2BGSWMWyQlLP1qOz988j/c98Z8Tuncgik3nqJEIlVKSXtzbQSuh6irL1Gz15ZSXvM24GUz+x3wBfBMKH8G+IeZZQIbiBIQ7j7PzMYC84EC4OfqySWVhbvz0qcr+P2/FlC75iH8+eLeXHRsO6K+JSJVh0X/+T/ATmb/BH4GFAKfEZqs3H1UYsMrnVgs5unp6ckOQ6q53C15/PrVOUxftI7/6dyCUUN6a6h4SWlmNtPdY6U5tqTPTLq7+xYzuxx4CxhBNAhkSiYTkWR7a24Ot0+cy878Qu4d3IOfHH+47kakSitpMqllZrWI3gl5zN3zzUw9pkT2sCUvn7snzWPCF9n0at+YP1/ch6Na6b0RqfoO5j2TZcBs4AMzOxwo7TMTkSrpv0vWc8u42azZksf1Z3Tml6cfRa0aJe3jIlK5lfQB/KNEU/cWW25mpyUmJJHKJS+/kD9OWcQzHy+lY/MGjP/ZCRyrFxClmjnQTIs/dvcXzeymfezy5wTEJFJpfJK1nhGvzmHZ+h38+Pg0bh/Ujfq19QKiVD8H+qkvfsu94X73Eqlmtublc/9bC3np0xWkNavPi9cM4OTOLZIdlkjSHGimxafC13sqJhyR1PefJV9x67g55GzeyTUnd+LmgV10NyLVXol+A8zsCKKhUI4nmhP+v8CN7n6gIVVEqoy8/EIemvoloz/MolPzBoz/3xM1OKNIUNL/Tv0TeBy4KKwPBcYAAxIRlEiqmbVyE7eMm01m7jYuGxANzqi7EZFvlPS3ob67/yNu/UUzuzURAYmkkoLCIh6dlslj0xbTulFdXri6P6d0aZnssERSTkmTyVtmNgJ4maiZ6xLgTTNrBl/PxChSpSxZt41bx83m8xWb+MGx7bh7cA8a1a2V7LBEUlJJk8nF4eu1e5QPJUouR5RbRCJJVlBYxOgPs3j43cXUq1WDR4b2YXCf78zHJiJxSvrSYqdEByKSCuav3sKvX51NRvYWzulxGPde2INWDTU4o8iB7HesBzP7ddzyj/bY9odEBSVS0XYVFPKndxZxwWMfsWZzHn+9vC9P/uQ4JRKREjrQwEFD45ZH7rHtnHKORSQpPl+xke8/+hF/mZbJBb3bMvXG7zHomDbJDkukUjlQM5ftY3lv6yKVSl5+IaOmLOLZj5dyWKO6PHdlP047ulWywxKplA50Z+L7WN7b+reYWV0zm2Fms81snpndE8o7mdmnZpZpZq+YWe1QXiesZ4btHePONTKULzKzs0v83Ynsw8I1W7jw8Y955qOlXD4gjXduPEWJRKQMDnRn0tvMthDdhdQLy4T1AzUm7wJOd/dtYS6Uj8zsLeAm4CF3f9nMngSuAZ4IXze6+1FmNhR4ALjEzLoTNbf1ANoC75pZF03dK6VRVOQ8+/FSHpyyiEZ1a/HcVf04rauSiEhZHWhsrhqlPbFH8wFvC6u1wseB04HLQvnzwN1EyWRwWAYYDzxm0dR0g4GX3X0XsDTMEd+faEgXkRLLyN7MXZPnMXP5Rs7s1or7f9iLFofWSXZYIlVCQseDMLMaRNP7HkU0HMsSYJO7F4RdVgHFHfjbASsB3L3AzDYDzUP5J3GnjT8m/lrDgeEAaWlp5f69SOVVUFjEE9OX8Mh7i2lSvzajhvRiyHHtNY2uSDlKaDIJTVF9zKwJMBE4OoHXGg2MBojFYppSWABY9tV2bhw7iy9WbOLCPm25Z3BPGtfTW+wi5a1CRqpz901m9j5wAtDEzGqGu5P2QHbYLRvoAKwys5pAY2B9XHmx+GNE9srdGTNjJfe9MZ9aNYy/XHos5/dum+ywRKqshE1QbWYtwx0JZlYPOAtYALwPDAm7DQMmheXJYZ2wfVp47jIZGBp6e3UCOgMzEhW3VH65W/O45vl0bp84l+MOb8qUG09RIhFJsETembQBng/PTQ4Bxrr7G2Y2H3jZzH4HfAE8E/Z/BvhHeMC+gfDCpLvPM7OxwHygAPi5enLJvkyZt4aRE+ayfVcBd5/fnStO6Mghh+jZiEiiWfSf/6olFot5enp6ssOQCrQ1L597X5/PuJmr6NmuEQ9d3IfOrTXbtMjBMLOZ7h4rzbGa3UcqvRlLN3DT2Fms3rSTX5x2FNef0ZnaNRPWgisie6FkIpVW/DS6HZrWZ9zPTuC4w5slOyyRaknJRCqluas2c8u42Sxau5VL+3fgzvO606COfpxFkkW/fVKp5OUX8sh7ixn9QRYtDq3N36/qx6kaDkUk6ZRMpNKYtXITt4ybTWbuNi6OteeO87rrBUSRFKFkIikv/m6kdcM6PH91f77XpWWywxKROEomktI+zVrPba/OYdn6HVwca8+d3+9Oo7q6GxFJNUomkpLy8qNpdP/20VI6NK3PP//fAE48skWywxKRfVAykZQzb/Vmbnol6ql1+YA0bh/UTT21RFKcfkMlZRQVOX/7KItRUxbRpH5tTVwlUokomUhKyNm8k5vHzuY/S9Zzdo/W/N8PetGsQe1khyUiJaRkIknl7rz6eTb3vTGf/MIiHvjhMVwc66CJq0QqGSUTSZqVG3YwYsIcPs5cT9+0Jvzp4j50atEg2WGJSCkomUiFKyxynv/PMkZNWUSNQ4zfX9STS/ulaah4kUpMyUQqVGbuVn49fg6fr9jEaV1b8vuLjqFtk3rJDktEyiiRMy12MLP3zWy+mc0zsxtCeTMzm2pmi8PXpqHczOxRM8s0szlm1jfuXMPC/ovNbNi+rimpK7+wiMemLWbQIx+R9dV2HrqkN89e2U+JRKSKSOSdSQFws7t/bmYNgZlmNhW4EnjP3e83sxHACOA24FyiKXk7AwOAJ4ABZtYMuAuIAR7OM9ndNyYwdilH/12ynjtem0vWuu2c16sN91zQgxaH1kl2WCJSjhKWTNw9B8gJy1vNbAHQDhgMnBp2ex6YTpRMBgMvhHnfPzGzJmbWJuw71d03AISEdA4wJlGxS/nYvDOf/3tzAS9/tpLDm9fnmWExzujWOtlhiUgCVMgzEzPrCBwLfAq0DokGYA1Q/NelHbAy7rBVoWxf5ZLC3l+Uy8hX55K7NY/hpxzBjWd2oV7tGskOS0QSJOHJxMwOBV4FfuXuW+LfH3B3N7NymYTezIYDwwHS0tLK45RSCpt35vO7N6K52Du3OpSnfnISvTs0SXZYIpJgCU0mZlaLKJG85O4TQvFaM2vj7jmhGSs3lGcDHeIObx/KsvmmWay4fPqe13L30cBogFgsVi4JSg7OB1+u49fj57Bu2y6uO/VIbjizM3Vq6m5EpDpIZG8uA54BFrj7n+M2TQaKe2QNAybFlV8RenUdD2wOzWFTgIFm1jT0/BoYyiRFbNtVwB0T53LFszM4tG5NJl53Ir8+52glEpFqJJF3JicBPwHmmtmsUHY7cD8w1syuAZYDF4dtbwKDgExgB3AVgLtvMLP7gM/CfvcWP4yX5Ju+KJc7JmawevNOfnpyJ245uyt1aymJiFQ3FnWeqlpisZinp6cnO4wqLXdrHve9sYDXZ6/miJYNGDWkF8cd3izZYYlIGZjZTHePleZYvQEvB6WoyPnnjBU88PZCduUXcdNZXbj2e0eoSUukmlMykRJbuGYLt0+Yy+crNnHikc353YU9OaLlockOS0RSgJKJHFBefiGPvLeYpz/IolG9Wvz54t5cdGw7DfdJZe4AAA+0SURBVBMvIl9TMpH9ysjezI2vzGJx7jYujrVn5LndaKpJq0RkD0omslfbdhXwwFsLefHT5bQ4tA4vXN2fU7q0THZYIpKilEzkOz7JWs8t42aTvWknw07oyI1ndaFxvVrJDktEUpiSiXwtL7+QUVMW8ezHS0lrVp9x155ArKO6+4rIgSmZCACzV27iprGzWLJuOz85/nBGDjqa+rX14yEiJaO/FtVcXn4hD7+7mKc/zKJVwzr845r+/E9nPRsRkYOjZFKNzV65iZvHzSYz9NS647zuejYiIqWiZFIN7S4o4i/TFvPX6Uto1VA9tUSk7JRMqpkFOVu4aexsFuRs4Yd92/Pb83U3IiJlp2RSTRQUFvHUB1k8/O6XNK5Xm6eviHFWd02hKyLlQ8mkGsjM3cbN42Yze+UmzuvVhvsG96SZ3mIXkXKkZFKFFRU5z368lFFTFlGvdg3+cumxnN+7bbLDEpEqSMmkilq+fju3jpvDjGUbOLNbK/7wg2No1bBussMSkSoqkdP2PmtmuWaWEVfWzMymmtni8LVpKDcze9TMMs1sjpn1jTtmWNh/sZkN29u15BvuzoufLOfcRz5kQc4W/vij3jx9RUyJREQSKmHJBPg7cM4eZSOA99y9M/BeWAc4F+gcPsOBJyBKPsBdwACgP3BXcQKS71q9aSdXPDuDO1/L4LjDmzLlxlMYclx7DRUvIgmXsGYud//AzDruUTwYODUsPw9MB24L5S94NIfwJ2bWxMzahH2nFs/5bmZTiRLUmETFXRm5O+NnruLe1+dT6M7vLuzJ5QPSlEREpMJU9DOT1u6eE5bXAMV9U9sBK+P2WxXK9lX+HWY2nOiuhrS0tHIMObXlbs3j9glzeXdBLv07NuOPP+pNWvP6yQ5LRKqZpD2Ad3c3My/H840GRgPEYrFyO2+qcnfemJPD3ZPnsW1XAXee142rT+rEIYfobkREKl5FJ5O1ZtbG3XNCM1ZuKM8GOsTt1z6UZfNNs1hx+fQKiDOlfbVtF795LYO3MtbQq31j/vSj3nRu3TDZYYlINZbIB/B7Mxko7pE1DJgUV35F6NV1PLA5NIdNAQaaWdPw4H1gKKu23pqbw9kPfcB7C3IZce7RTLzuJCUSEUm6hN2ZmNkYoruKFma2iqhX1v3AWDO7BlgOXBx2fxMYBGQCO4CrANx9g5ndB3wW9ru3+GF8dbN5Zz53TcrgtVmrOaZdY/50cW+6KImISIqwqANV1RKLxTw9PT3ZYZSbjxZ/xa3jZ5O7dRfXn96Z6047klo1KvqmUkSqOjOb6e6x0hyrN+BT2M7dhTzw9kL+/p9lHNmyAROvO5Fe7ZskOywRke9QMklRs8I0ulnrtnPVSR257ZyjqVurRrLDEhHZKyWTFJNfWMRfpmXy+PuZtG5Yh3/+dAAnHtUi2WGJiOyXkkkKWb5+O9e/PIvZKzfxg77tuOv8Hpq4SkQqBSWTFODujE1fyb2vz6fGIcbjl/XlvF5tkh2WiEiJKZkkWda6bdwxMYP/Zq1nQKdmPHRJH9o2qZfssEREDoqSSZLsKijk6Q+yeHRaJnVqHsIfLjqGof06aDgUEamUlEwq2K6CQsalr+Lx9zPJ2ZzHoGMO4+7ze9CqkeYbEZHKS8mkAn2StZ4Rr85h2fod9E1rwqghvTm5s3pqiUjlp2RSAXbuLuTBKQt57uNlpDWrz3NX9ePULi0134iIVBlKJglUVOS8NiubP0/9klUbd3LFCYcz4tyjqV9b1S4iVYv+qiWAuzNtYS5/eudL5udsoUfbRjw4pBcnHqkmLRGpmpRMypG783Hmev74ziJmrdzE4c3r88jQPpzfq616aYlIlaZkUk6y1m3jztcy+M+S9bRtXJf7f3AMPzyuvUb3FZFqQcmkjNZuyeMv0xbzymcrqVurBned353LBqRRp6YGZRSR6kPJpJQ2bN/NE9MzeeG/yyksci7u14FfndFZ74uISLVUaZKJmZ0DPALUAP7m7vdXdAw7dhfwwZfrmDJvLW9l5LC7oIgLj23Hr87oQlrz+hUdjohIyqgUycTMagCPA2cBq4DPzGyyu89P1DXdndWb85i/egufr9hIRvZmZizdwK6CIprUr8VFx7bnmpM7clQrTZ0rIlIpkgnQH8h09ywAM3sZGAyUazKZv3oLN7z8BVvy8tm8M5+8/CIAatUwOrdqyKX90zi7x2H069iUmnqwLiLytcqSTNoBK+PWVwED4ncws+HAcIC0tLRSXaRh3Zp0bn0ojerWomHdmnRoVp8ebRvTvU0j6tXWA3URkX2pLMnkgNx9NDAaIBaLeWnO0aFZff56+XHlGpeISHVQWdpqsoEOcevtQ5mIiKSAypJMPgM6m1knM6sNDAUmJzkmEREJKkUzl7sXmNkvgClEXYOfdfd5SQ5LRESCSpFMANz9TeDNZMchIiLfVVmauUREJIUpmYiISJkpmYiISJkpmYiISJmZe6ne70tpZrYOWF6KQ1sAX5VzOOUlVWNL1bhAsZWWYiudVI3tYOI63N1bluYiVTKZlJaZpbt7LNlx7E2qxpaqcYFiKy3FVjqpGltFxaVmLhERKTMlExERKTMlk28bnewA9iNVY0vVuECxlZZiK51Uja1C4tIzExERKTPdmYiISJkpmYiISJkpmQBmdo6ZLTKzTDMbkcDrdDCz981svpnNM7MbQnkzM5tqZovD16ah3Mzs0RDXHDPrG3euYWH/xWY2LK78ODObG4551MzsIOKrYWZfmNkbYb2TmX0azvVKGP4fM6sT1jPD9o5x5xgZyheZ2dlx5aWuYzNrYmbjzWyhmS0wsxNSqM5uDP+WGWY2xszqJqvezOxZM8s1s4y4soTX076uUYLYRoV/0zlmNtHMmpS2PkpT5/uLLW7bzWbmZtYiVeotlP8y1N08M3swGfX2He5erT9EQ9ovAY4AagOzge4JulYboG9Ybgh8CXQHHgRGhPIRwANheRDwFmDA8cCnobwZkBW+Ng3LTcO2GWFfC8eeexDx3QT8E3gjrI8FhoblJ4H/DcvXAU+G5aHAK2G5e6i/OkCnUK81ylrHwPPAT8NybaBJKtQZ0XTSS4F6cfV1ZbLqDTgF6AtkxJUlvJ72dY0SxDYQqBmWH4iL7aDr42Dr/ECxhfIORNNeLAdapFC9nQa8C9QJ662SUW/fiTURfzQr0wc4AZgStz4SGFlB154EnAUsAtqEsjbAorD8FHBp3P6LwvZLgafiyp8KZW2AhXHl39rvALG0B94DTgfeCD/4X/HNL/vX9RR+wU4IyzXDfrZn3RXvV5Y6BhoT/cG2PcpToc7aASuJ/oDUDPV2djLrDejIt//wJLye9nWNA8W2x7aLgJf29n0eqD5K87NaktiA8UBvYBnfJJOk1xtRAjhzL/tVeL3Ff9TM9c0fhGKrQllChdvGY4FPgdbunhM2rQFaHyC2/ZWv2kt5STwM/BooCuvNgU3uXrCXc319/bB9c9j/YOMtiU7AOuA5i5rg/mZmDUiBOnP3bOCPwAogh6geZpIa9VasIuppX9c4GFcT/a+9NLGV5md1v8xsMJDt7rP32JQK9dYF+J/Q/PRvM+tXytjKtd6UTJLAzA4FXgV+5e5b4rd59F+BCu2vbWbfB3LdfWZFXreEahLd5j/h7scC24maBL6WjDoDCG3cg4kSXlugAXBORcdRUhVRT6W5hpndARQALyUkqINkZvWB24HfVtQ1D7LeahLdDR8P3AqMLX4Ok0xKJpBN1DZarH0oSwgzq0WUSF5y9wmheK2ZtQnb2wC5B4htf+Xt91J+ICcBF5jZMuBloqauR4AmZlY8G2f8ub6+ftjeGFhfinhLYhWwyt0/DevjiZJLsusM4Exgqbuvc/d8YAJRXaZCvRWriHra1zUOyMyuBL4PXB7+oJYmtvUcfJ3vz5FE/0GYHX4n2gOfm9lhpYgtEfW2CpjgkRlErQktShFb+dbb/trAqsOHKMtnEf3wFD+c6pGgaxnwAvDwHuWj+PaDuAfD8nl8+2HfjFDejOg5QtPwWQo0C9v2fNg36CBjPJVvHsCP49sP564Lyz/n2w/nxoblHnz7AWAW0cO/MtUx8CHQNSzfHeor6XUGDADmAfXDsc8Dv0xmvfHd9vWE19O+rlGC2M4B5gMt99jvoOvjYOv8QLHtsW0Z3zwzSYV6+xlwb1juQtQcZcmot2/FWZo/ilXtQ9RD40uiHg93JPA6JxPdys4BZoXPIKK2yPeAxUS9NIp/CA14PMQ1F4jFnetqIDN8roorjwEZ4ZjHOMBDs73EeCrfJJMjwi9CZvihK+49UjesZ4btR8Qdf0e49iLiekWVpY6BPkB6qLfXwi9rStQZcA+wMBz/j/CLnJR6A8YQPbvJJ/rf6zUVUU/7ukYJYssk+kNY/LvwZGnrozR1vr/Y9ti+jG+SSSrUW23gxXDOz4HTk1Fve340nIqIiJSZnpmIiEiZKZmIiEiZKZmIiEiZKZmIiEiZKZmIiEiZ1TzwLiJVm5kVd9EEOAwoJBrCBWCHu59YzterDzwN9CLqarqJ6J2LmsBl7v7X8ryeSEVQ12CROGZ2N7DN3f+YwGuMJHpR76aw3pXoXYY2RO/49EzUtUUSRc1cIvthZtvC11PDoHqTzCzLzO43s8vNbEaYq+LIsF9LM3vVzD4Ln5P2cto2xA2L4u6L3H0XcD9wpJnNMrNR4Xy3hvPMMbN7QlnHMJfFSxbN7zI+3O0Q4pof9k9YQhTZk5q5REquN9AN2EA0PMXf3L2/RZOc/RL4FdGYZg+5+0dmlkY0lHe3Pc7zLPCOmQ0hal573t0XEw2p0dPd+wCY2UCgM9CfqDlsspmdQjRKcVeiN7U/NrNngevM7DmiodyPdne3uMmmRBJNdyYiJfeZu+eEu4glwDuhfC7R+EkQDf74mJnNAiYDjcIo0V9z91lEw1iMIhrT6TMz2zPhQDR51EDgC6JhM44mSi4AK93947D8ItFQPZuBPOAZM/sBsKNs365IyenORKTkdsUtF8WtF/HN79IhwPHunre/E7n7NqJRhieYWRHR2Emv7rGbAf/n7k99qzCaC2fPh53u7gVm1h84AxgC/IJoBGiRhNOdiUj5eoeoyQsAM+uz5w5mdpJ9Mxd7baLpVpcDW4mmcy42Bbi6+M7GzNqZWauwLc3MTgjLlwEfhf0au/ubwI1EzXIiFUJ3JiLl63rgcTObQ/T79QHRkOHxjgSeCBMaHQL8C3g1POf42MwygLfc/dbQ/PXfMPfRNuDHRF2XFwE/D89L5gNPEM05McnM6hLd1dyU4O9V5GvqGixSyYRmLnUhlpSiZi4RESkz3ZmIiEiZ6c5ERETKTMlERETKTMlERETKTMlERETKTMlERETK7P8DUiVHmFp2xeQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "q_values = run(8000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "有了q values就可以決定control，做法很簡單，greedy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimal policy is:\n",
      "['D', 'U', 'R', 'R', 'R', 'R', 'R', 'R', 'R', 'D']\n",
      "['R', 'R', 'R', 'R', 'R', 'R', 'R', 'U', 'U', 'D']\n",
      "['R', 'U', 'R', 'R', 'R', 'R', 'R', 'R', 'R', 'D']\n",
      "['R', 'R', 'R', 'R', 'R', 'U', 'U', 'G', 'R', 'D']\n",
      "['R', 'R', 'R', 'R', 'R', 'R', 'U', 'D', 'L', 'L']\n",
      "['R', 'R', 'R', 'R', 'R', 'U', 'U', 'D', 'R', 'U']\n",
      "['R', 'R', 'R', 'R', 'U', 'U', 'U', 'U', 'U', 'L']\n",
      "Wind strength for each column:\n",
      "['0', '0', '0', '1', '1', '1', '2', '2', '1', '0']\n"
     ]
    }
   ],
   "source": [
    "optimal_policy = []\n",
    "for i in range(0, WORLD_HEIGHT):\n",
    "    optimal_policy.append([])\n",
    "    for j in range(0, WORLD_WIDTH):\n",
    "        if [i, j] == GOAL_STATE:\n",
    "            optimal_policy[-1].append('G')\n",
    "            continue\n",
    "        bestAction = np.argmax(q_values[i, j, :])\n",
    "        if bestAction == ACTION_UP:\n",
    "            optimal_policy[-1].append('U')\n",
    "        elif bestAction == ACTION_DOWN:\n",
    "            optimal_policy[-1].append('D')\n",
    "        elif bestAction == ACTION_LEFT:\n",
    "            optimal_policy[-1].append('L')\n",
    "        elif bestAction == ACTION_RIGHT:\n",
    "            optimal_policy[-1].append('R')\n",
    "print('Optimal policy:')\n",
    "for row in optimal_policy:\n",
    "    print(row)\n",
    "print('Wind strength for each column:\\n{}'.format([str(w) for w in WIND]))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}