Chap14-Recurrent_Neural_Networks/Chap14_2-Recurrent_Neural_Networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chap14-2. 순환 신경망 (RNN, Recurrent Neural Network) - (2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "저번 포스팅인 [07-1. 순환 신경망 - (1)](http://excelsior-cjh.tistory.com/183)에서는 시계열 데이터에 적합한 모델인 RNN의 구조와 텐서플로(TensorFlow)의 `BasicRNNCell`과 `static_rnn()`, `dynamic_rnn()`을 이용해 RNN을 구현하는 방법에 대해 알아보았다. 이번 포스팅에서는 RNN을 학습시키는 방법과 심층 RNN에 대해 알아보도록 하자."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. RNN 학습시키기"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 BPTT (BackPropagation Through Time)\n",
    "\n",
    "RNN은 기존 신경망의 역전파(backprop)와는 달리 타임 스텝별로 네트워크를 펼친 다음, 역전파 알고리즘을 사용하는데 이를 **BPTT**(BackPropagation Through Time)라고 한다. \n",
    "\n",
    "![BPTT](./images/bptt01.png)\n",
    "\n",
    "BPTT 또한 일반적인 역전파와 같이 먼저 순전파(forward prop)로 각 타임 스텝별 시퀀스를 출력한다. 그런다음 이 출력 시퀀스와 손실(비용)함수를 사용하여 각 타임 스텝별 Loss를 구한다. 그리고 손실 함수의 그래디언트는 위의 그림과 같이 펼쳐진 네트워크를 따라 역방향으로 전파된다. BPTT는 그래디언트가 마지막 타임 스텝인 출력뿐만 아니라 손실함수를 사용한 모든 출력에서 역방향으로 전파된다.  \n",
    "\n",
    "RNN은 각 타임 스텝마다 같은 매개변수 $\\mathbf{W}$ 와 $\\mathbf{b}$ 이 사용되기 때문에 역전파가 진행되면서 모든 타임 스텝에 걸쳐 매개변수 값이 합산된다. 이렇게 업데이트된 가중치는 순전파 동안에는 모든 타임 스텝에 동일한 가중치가 적용된다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Truncated BPTT\n",
    "\n",
    "3.1에서 살펴본 BPTT는 전체의 타임 스텝마다 처음부터 끝까지 역전파를 하기 때문에 타임 스텝이 클 수록 계산량이 많아지는 문제가 있다. 이러한 계산량 문제를 해결하기 위해 전체 타임 스텝을 일정 구간(예를들어 3 또는 5 구간)으로 나눠 역전파를 하는 **Truncated BPTT**를 사용한다. \n",
    "\n",
    "\n",
    "\n",
    "![truncated-bptt](./images/bptt02.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 RNN을 이용한 분류기 구현\n",
    "\n",
    " 이번에는 RNN을 이용해 MNIST 숫자 이미지 데이터셋을 분류하는 분류기를 구현해보자. MNIST와 같은 이미지 데이터는 이전 포스팅 [06. 합성곱 신경망](http://excelsior-cjh.tistory.com/180)에서 살펴본 이미지의 공간(spatial) 구조를 활용하는 CNN 모델이 더 적합하지만, 인접한 영역의 픽셀은 서로 연관되어 있으므로 이를 시퀀스 데이터로 볼 수도 있다.   아래의 그림처럼 MNIST 데이터에서 `28 x 28` 픽셀을 시퀀스의 각원소는 `28`개의 픽셀을 가진 길이가 `28` 시퀀스 데이터로 볼 수 있다.\n",
    "\n",
    "![](./images/mnist_seq.png)\n",
    "\n",
    "\n",
    "\n",
    "아래의 코드는 텐서플로(TensorFlow)의 `BasicRNNCell`과 `dynamic_rnn()`을 이용해 MNIST 분류기를 구현한 코드이다."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set Up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "# 일관된 출력을 위해 유사난수 초기화\n",
    "def reset_graph(seed=42):\n",
    "    tf.reset_default_graph()\n",
    "    tf.set_random_seed(seed)\n",
    "    np.random.seed(seed)\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['axes.labelsize'] = 14\n",
    "plt.rcParams['xtick.labelsize'] = 12\n",
    "plt.rcParams['ytick.labelsize'] = 12\n",
    "\n",
    "# 한글출력\n",
    "# matplotlib.rc('font', family='AppleGothic')  # MacOS\n",
    "matplotlib.rc('font', family='Malgun Gothic')  # Windows\n",
    "plt.rcParams['axes.unicode_minus'] = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3.1 MNIST Data Load"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train_x.shape : (55000, 784)\n",
      "valid_x.shape : (5000, 28, 28)\n",
      "test_x.shape : (10000, 28, 28)\n"
     ]
    }
   ],
   "source": [
    "################\n",
    "# Layer Params #\n",
    "################\n",
    "n_steps = 28\n",
    "n_inputs = 28\n",
    "n_neurons = 150\n",
    "n_outputs = 10\n",
    "\n",
    "(train_x, train_y), (test_x, test_y) = tf.keras.datasets.mnist.load_data()\n",
    "train_x = train_x.astype(np.float32).reshape(-1, 28*28) / 255.0  # (784,)\n",
    "test_x = test_x.astype(np.float32).reshape(-1, 28*28) / 255.0\n",
    "train_y = train_y.astype(np.int32)\n",
    "test_y = test_y.astype(np.int32)\n",
    "valid_x, train_x = train_x[:5000], train_x[5000:]\n",
    "valid_y, train_y = train_y[:5000], train_y[5000:]\n",
    "test_x = test_x.reshape([-1, n_steps, n_inputs])\n",
    "valid_x = valid_x.reshape([-1, n_steps, n_inputs])\n",
    "\n",
    "print('train_x.shape :', train_x.shape)\n",
    "print('valid_x.shape :', valid_x.shape)\n",
    "print('test_x.shape :', test_x.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3.2 RNN for MNIST\n",
    "\n",
    "그런 다음, 아래의 코드와 같이 MNIST 데이터셋을 분류하는 RNN 모델을 만들어 준다. MNIST를 분류하는 RNN 모델은 아래의 그림과 같이 RNN모델 중 28개의 타임 스텝(28 pixels)의 시퀀스를 입력받아 하나의 벡터를 출력하는 Sequence-to-Vector 모델이다.\n",
    "\n",
    "![](./images/rnn-mnist.PNG)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Mini-batch\n",
    "def shuffle_batch(features, labels, batch_size):\n",
    "    rnd_idx = np.random.permutation(len(features))\n",
    "    n_batches = len(features) // batch_size\n",
    "    for batch_idx in np.array_split(rnd_idx, n_batches):\n",
    "        batch_x, batch_y = features[batch_idx], labels[batch_idx]\n",
    "        yield batch_x, batch_y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "inputs = tf.placeholder(tf.float32, [None, n_steps, n_inputs])\n",
    "labels = tf.placeholder(tf.int32, [None])\n",
    "\n",
    "# RNN Model\n",
    "basic_cell = tf.nn.rnn_cell.BasicRNNCell(num_units=n_neurons)\n",
    "outputs, states = tf.nn.dynamic_rnn(basic_cell, inputs, dtype=tf.float32)\n",
    "# dense layer\n",
    "logits = tf.layers.dense(states, n_outputs)  # states = outputs[-1]\n",
    "\n",
    "# loss\n",
    "xentropy = tf.reduce_mean(\n",
    "    tf.nn.sparse_softmax_cross_entropy_with_logits(labels=labels, logits=logits))\n",
    "\n",
    "################\n",
    "# Train Params #\n",
    "################\n",
    "learning_rate = 0.001\n",
    "n_epochs = 5\n",
    "batch_size = 150\n",
    "\n",
    "# optimizer\n",
    "train_op = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(xentropy)\n",
    "\n",
    "# metric\n",
    "correct = tf.nn.in_top_k(logits, labels, 1)\n",
    "accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch : 000 acc_batch : 0.9533, acc_valid : 0.9322 loss_batch : 0.1928\n",
      "epoch : 001 acc_batch : 0.9600, acc_valid : 0.9530 loss_batch : 0.1873\n",
      "epoch : 002 acc_batch : 0.9600, acc_valid : 0.9520 loss_batch : 0.2149\n",
      "epoch : 003 acc_batch : 0.9600, acc_valid : 0.9606 loss_batch : 0.1332\n",
      "epoch : 004 acc_batch : 0.9800, acc_valid : 0.9730 loss_batch : 0.0784\n"
     ]
    }
   ],
   "source": [
    "# Train\n",
    "with tf.Session() as sess:\n",
    "    tf.global_variables_initializer().run()\n",
    "    for epoch in range(n_epochs):\n",
    "        for batch_x, batch_y in shuffle_batch(train_x, train_y, batch_size):\n",
    "            batch_x = batch_x.reshape([-1, n_steps, n_inputs])\n",
    "            sess.run(train_op, \n",
    "                     feed_dict={inputs: batch_x, labels: batch_y})\n",
    "        acc_batch = accuracy.eval(feed_dict={inputs: batch_x, labels: batch_y})\n",
    "        acc_valid = accuracy.eval(feed_dict={inputs: valid_x, labels: valid_y})\n",
    "        loss_batch = xentropy.eval(feed_dict={inputs: batch_x, labels: batch_y})\n",
    "        print('epoch : {:03d}'.format(epoch),\n",
    "              'acc_batch : {:.4f}, acc_valid : {:.4f}'.format(acc_batch, acc_valid),\n",
    "              'loss_batch : {:.4f}'.format(loss_batch))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 RNN을 이용한 시계열 데이터 예측\n",
    "\n",
    "이번에는 주식가격과 같은 시계열(time series) 데이터를 예측하는 RNN 모델을 구현해 보도록 하자."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.4.1 가상의 시계열 데이터 만들기\n",
    "\n",
    "RNN 모델에 사용하기 위한 가상의 시계열 데이터를 다음과 같이 만들어 준다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "t_min, t_max = 0, 30\n",
    "resolution = 0.1\n",
    "\n",
    "def time_series(t):\n",
    "    return t * np.sin(t) / 3 + 2 * np.sin(t*5)\n",
    "\n",
    "def next_batch(batch_size, n_steps):\n",
    "    t0 = np.random.rand(batch_size, 1) * (t_max - t_min - n_steps * resolution)\n",
    "    Ts = t0 + np.arange(0., n_steps + 1) * resolution\n",
    "    ys = time_series(Ts)\n",
    "    return ys[:, :-1].reshape(-1, n_steps, 1), ys[:, 1:].reshape(-1, n_steps, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqYAAAEbCAYAAAAIx0apAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzsvXd8XNWZ//9+RnVGvUu2ZMmdYsBgx0AgdHASaiCUZUPbb8iG7JIC/BIgBAjJJiSwWbLspkAWTAlxIAQIAUJx6AEMphkXXGVZxeq9S3N+f5x7p0gzatbM1YzO+/XSa245995zR9KdZ57yeUQphcFgMBgMBoPB4DQupydgMBgMBoPBYDCAMUwNBoPBYDAYDDMEY5gaDAaDwWAwGGYExjA1GAwGg8FgMMwIjGFqMBgMBoPBYJgRGMPUYDAYDAaDwTAjMIbpLEREckXkZRHJcXouk0VEjheRhyJ4/kdE5KJInX+mIyJrROQmEUkRkXdE5FCn52QwGEIjIgki8oKILJ7g+GtF5FuRntdMQEQqReRYp+dhmDzGMJ0FiMgVInKXva6UalFKnaiUanVyXlNBKfWqUuqSSJxbRM4H3EqptZE4f6QRkd9Ol1GtlOoH/hV4UERkOs5pMBiCGflsnixKqWGl1GlKqe0THP+fSqlfTvV6BkM0MIbp7KAcSHd6EvuLiET67/Um4McRvsaUmOC9LwUSp+uaSqkPgb3A2dN1ToPBEMSYz+YoPPMMhhmH+aOPc0RkLfBt4EIrtHGpiFSIyFDAmFdE5HoReV5E6kXkTREpE5HbRGSHiNSJyM0jznuWiHxgnfNNETlsnHlcKyKbRaRaRLaKSJK1PVVEfiEi20Rkt4j8RkQ81r4TrOt/W0QqgW+KyOUi8lLAeXNE5H4R2Wn9/IeIJFj7CkTkMRHZLiINIvIfY8xvEZChlNoQsK1ARP4gIlUisktEbhwZHhrrfbC2XWGlTdSKyAYROSRg/2TvPc9KNagUkb0i8qyIFNrXAo4C/svav8TafoyIvGVt+0BETgq4fqKI3Gq9P3tF5FFGf0j+CThvrN+twWCYPGM9m0XkQhHZAdwpIknWs2G39X/6RmDoXkSUiJRay2tE5Kci8qCI7LF+zg8Yu0ZEbrKW7WfMFSLyiYg0Ws/SpIDxh4vIq9Z1t4jINyTgsyPMfV0iIh8FPDftuZ1iPYvsed0YcMytIvKAiNxj7dspIieJyHki8rHoz6VHRzwft4rI+SKy0Xq+vioiB48xr0l9ZhkcRCllfuL8B7gV+F3AegUwFLD+CrARKEJ/WXkI7Sm7zdo/H+gGDrPWT7T2L7XWvwTsAVLDXP8kYAfa8ANYBLis5QeAh4EUIAl4DPiZte8EoBO43lp3AZcDLwWc+2XgdmtfBvAGcJW178GAc7mAxWO8R5cAD43Y9gZwB9oL6QL+AxgGjp3I+wBUAm8CBdb6L4BXA84/2Xsvt66RYK2vBf5rxO/xKwHrS4Em4Chr/UhrvdBa/xGwDsix1r8I9AM3BZxjEbDD6b9h82N+4vGH0M9mL3AnINb/uQf4f0CyNeYO4ImAYxRQai2vAfYBR1jrZ1vPkcyA/TdZyycAvcAPrWvloZ/TV1j784Fm4EvWehbwLAGfHSHuZyHQBhRb63PwP/fPwnoGA3OBVvyfKbcCHcCR1vqVQD3wV8BtvQfvAd8MmHsX8Bvr+SnAdcAuIMUaU8kEn9XmZ2b9GI+pwWaNUqpeKeVFG0y5wG0ASqnd6IeCXQhzDfAjpdSn1v4n0A+Jz4Q5dx/6oXagNX6HUsorIvnAhWhDsl8pNQj8HPhywLEJaIMOa24+RGQFsBi4USnlVUp1Ar8MOL4PWCIiedb+sfKwitEPdPvch1vnvkEpNWRd+yb0Q95mIu/DL5VSjdbyvcAK6/yTvnel1B7rGrnA0egPgLAeAuDfgXuUUm9b53gHeAdYbe3/JvBtZeUaK6WeBf424hz70F9YDAZDdBD0c0NZ//c9Sqn/AzwisgroYez/+z8rpd4HUEo9ZY1fGmbsENoBoZRSzcDjwEpr3yXAK9YzB6VUO/D9ceY+iP6Svdw6ptZ6LqOU+guw0/L2HoI2PAPv41XrGQX6M6gA+IlSqlcp1QM8AwR6OV3At6znp1JK3WltWxViXpP9zDI4yLTloxlinvqA5S6gUSkVGLLpBNKs5QXADwNDMegQcGGoEyul/iEi3wB+JyK9wA8tI2g+2vjaKP76Ghf6G7DNPqXUQJg5L0B/y98VcHwiUGctX4P+Jr5JRJ5DewpqwpzLhfZU2NieQt97oJRSItI24vrjvQ+1Acut+N/DSd+7ZSz/Du1Z2IY2UJPD3I89v1UicnHANg/wkogUAJnA1hHHjCyI81rzNBgM0cELVNsrIlKBjv4kAFuszWP939eOWA987oykXik1PGKs/UV0EeM/H4JQSlWJyFnAbSLyC7Rh+TDodC7ga8AmYDfaAA+8j/qA8wyIyMCIewn8DAKoVbpIM5AG9GfCSCb1mWVwFmOYGqZCLXCrUuqxiR5gjX1MRE4B/iQiJ1vn6QMWjPSGBhBuuz2PXUqpkN4DpVQXcJ3o/NgfAs/h9/qOpAHLo2vRDMwVEVFKx35ExA2UjLj+pN6HEcdO9t5/Cfy3UuoBaz7fAC4Y5xo/V0rdMXKHlUc2BJShw182C9DhPJsi9HtjMBiig7KfORa3AeuUUj8EEJEvAqdEYR7N6NSCQBaOd5BSah2wzopoPSEiPcBHwM3olINOABH53H7OL3fE8zkFy6EQYuz+PKsNUcaE8mcHLWiDAxGZji8ja4Dvi8g865wpIhK2cltEDhSRMmv1LfQDL93yXr4F/FT8xVBlInLUBOfxjj5ErhbL7SgiB4hVYGQlyCdbYaC/o3NQxzrX0QHrb6G/0X9PNClowzDQWFzDJN6HQKZ47yloLylWQcG/jtjfAiy05ptgze9quyBAtObhudb1B4E/A/8ZUFDwL1ghuACORr83BoNh+pnIsznw/z4HHQmKBn8CzhORE61rl6DTmcIiIvNE5ABr9WN0HmcG2jOagE7pQkQuRIfz94cs4BYRcVnv3c+AD5RSn4QYu4YpPqsN0ccYprODtUCGiOwBzh9v8HgopX4P/B/wguhq8A8J9jaOpAR4VUT2AuuBXyulXrX2/TNQis492gE8wgT/Lq0w++nAacBeEdkJ3IX2BIJOtq+2znsDOqcz3Lk2A14RWWat9wJnAF9Af9t+D3gB/UHSPcX3YSSTvfdrga+KSBX+wqlAfgFcAXwKVCil3kQXBKy1fvdbgOMDxl+Fzj/bZV1/EboAK5Dz0AaswWCYfibybL4VOFpEqtFRnz9GY2JKqY/Reab/a137UeBurOdfGDLQXtJadEHtP4CHlVJbgP8E1lvP6eXAq+FPMyF2o3Nad6G9pLmEeQ+n4VltiCISHDEwGGYvVi7mWUqpkCL1lreiCci2w1HxjIgchP4wOnSMdAODwTBLsLyMP1BKrRx3cGTncQJazWCRk/MwRAbjMTUYLJRSj6BTA85DL5whIqnWcg5wD/rb/2wwSpPRKgKXGKPUYJh9iNaIPjEgTWop8BPgf52dmSHeMR5TgyEMIvJbtLanF12o9ARacmSsUJbBYDDEPCKSh04bOBAdMm9Dh/LvUw4bDsZjGt8Yw9RgMBgMBoPBMCMwoXyDwWAwGAwGw4wgbnVM8/PzVUVFhdPTMBgMccaGDRualFIFTs8jUphnp8FgiAQTfXbGrWFaUVHBe++95/Q0DAZDnGFJ+8Qt5tlpMBgiwUSfnSaUbzAYDAaDwWCYERjD1GAwGAwGgyFO+NWvIjs+0hjD1GAwGAwGgyFGUQreeQfOPx88HlizRr9ecAGsX6/378/4aBO3OaYGg8FgmB4GBweprq6mr6/P6ak4TmpqKqWlpSQlJTk9FYOBwUG49FL4y1+grw8WLoTnnoOjj4bHH4dnn4Uzz4QHH4SkpMmPdwJjmBoMBoNhTKqrq8nIyKCiogKrEdCsRClFc3Mz1dXVzJ8/3+npGGY5SvmNzJ4evW3FCsjL06/bt0N3Nzz1lB73+99Pbvwjj4AT/+4mlG8wGAyGMenr6yMvL29cozQwRJiWBi6Xfp0pIcL9RUTIy8sznmPDjGD9enj6ab+RmZMD11+vl7/3Pb0O0Nurx61ZM7nx774btVsJwnhMDQaDwTAu4xmlI0OEXq/e3tMzc0KE08Fs9hgbZg733gtr18LKlf5txcVw2GF6eflyOO002LdPr7e3w333wZFHwvDw+OMBbrgBLroIrrwy8vcTiDFMDQaDwbBfhAopBuL1zowQocEQL1x2Gfz853DjjXDFFaHHrF3rX77nHrj7brjmmomNv+8+uP12fZ1oY0L5BoPBYNgvRoYUw+F0iDAcl19+OVu3bnV6GgbDhElOhp074Tvf0V/2BgZCj+vvh0suge9+Fz75ZOLjr7lGnz85OXL3EA5jmBoMBoNhv/jP/9RG50To7dXjDQbD/uF26xD9Qw/BhReGHnPRRfDww3ocTG682x2ZeY+HMUwNBoPBsF8884w/p3Q8vF49fjLs2LGDE088kc997nPcdNNN3H///Zx88smsWLGCe+65B4Bbb72V7373u5xzzjl87nOfY926dZx22mksW7aM1157DdCe0VtuuYVTTz2VI444gr/97W+jrnXrrbdy/PHHc9xxx7Fhw4bJTdRgiCKnn64LDAFSUkKPsT2eLheUlk5u/OmnT99cJ4MxTA0Gg8GwX0zUWzrV8c888wxf+cpXeP3117nttts444wzWLduHa+99hq//vWvfePS0tJ48sknufDCC7njjjt4/vnnuf/++/lVQGubvLw8XnzxRV588UW+//3vB13npZdeoq2tjVdffZUnn3ySW265ZXITNRiiyLXX+r2axx6rX3fsgDPO0K+B21NT4Yc/nNz4a6+N/D2EwhQ/GQyGmMLrVQx5FcmJ5nv1TMHtHj+/dOT4yXDllVfyi1/8gmuuuYYrr7ySZ555hoaGBhITE4Okm1atWgXAokWLOPLIIxER5s+fT1tbm2/MqaeeCmgDNSUlBRWgYfX++++zbt06TjjhBACG7fJlg2EGsmqVVrpYtw5OOUUXLP3oR1BZCZs2wc036+0FBXDyyXD55fDiixMf/5nPhL/2r34F3/hGZO7LGKYGgyGmuOHPG3l9eyPPf+c4MlJjWHcojjj9dC0JNZFw/lRChCLCTTfdREdHB0ceeSTz5s3j+eefp7a2lkceeSRoXKjlQNavX8+BBx7Inj17SExMDBq3ZMkSLrjgAn7wgx8A0DMZa9tgiABjGYAiWn5t9Wr4yU90YaH9HayyUns8zzgDli3T41yuyY0Pp5zxySfw5JNw/PFw8MHTfccmlG8wGGKI/qFh/vjeXmrb+/jJs6aKeqYQGFIcj6mECB955BGOPvpozjrrLL75zW/i8Xg45phjuOuuuygsLJzUuT744ANWr17NV77yFe66666gfWeffTY1NTUce+yxfOELX+Cvf/3r5CZqMOwnk+1jn5Skx557Lpx6anBji1NP1dsvuMCvHTzZ8aF47DH4wx/g0Ucj8x6IivVWHGFYuXKleu+995yehsFgmEZe3trAFWveJSMlkfTURN664eSoz0FENiilVo4/MjYJ9ezcsmULBx54YNhjlIKLL9Y6pWPlj7rdcPbZzumYXn755Vx//fUccMAB+3We8d4Pg2EqhOpj/9Zbuo/9zp36/8fpJhU9PVo/9dZb4ZZbdOeoiX4pneiz03hMDQZDzPDypw14khP42nELqGvvo60njBifIarYIcWzz/Z7YAJxubTX5+yzxw4RGgyzlZFNKrze4D72I5tUOOVTfOwx7XEF/frYY9N/jRmXYyoixwB3AN9QSn0oOgHoP4DVwDDwqFLqTifnaDAYnGFnYxdLijI4tCwbgK37OjlqQZ7DszKA9uA88ogWz7/zTt2CtLdXe1NOPx2uu27sYoposGbNGmcnYDCEYby+988/D62twU0qrFq/iHHvvXpec+f6t3m9/m5Qy5bpcP6tt/r319Toee1PG9MZZZiKyINABpAZsPlcYDGwEkgC3hCRN5RSbzswRYPB4CCVTT18piKHA4szANhS12EM03EQkWTgZ8CJQBfwJ6XUXWMfNdVr6Q8lO/cskpW7BkO8MNm+9xCdPvaXXaYNzeXL4ZxzQo+57Tb/8hNP6GfA/rYxnWmh/K8rpb4ENAVs+wpwt9IMAA8AYd4ig8EQr/QPDVPb3kt5XhoFGSnkpiWzpa7D6WnFAj8FepRSy5VSxwK/jMZF7crdTZuicTWDIXa57DKoqtKvr7yifwL71oNet/ddeins3Rv5PvbJydobqpR+HauN6S23aKP0llv2v43pjDJMlVKhtDkqgF0B63uAuSHGISJfE5H3ROS9xsbGCMzQYDA4xd6WHpSC+flpiAiLCtKpbDJyPmMhIrnAF4Cb7W0qShWvka7cNRjihcn2vY92H/svfUkbwZaK2ihuvllrpIbzqk6WGWWYhiEFGApYHwZCquUppe5RSq1USq0sKCiIyuQMBkN0sI3Q8jwPAHNz3NS0TbKF0OzjUOBT4H9E5FUReVJElowcNN1f6nt6tPckz8qymGynJ4NhtjHZvvfR7mM/b174NqYpKVBePn3XigXDtBqYF7BeDlQ6MxWDweAUlc3dAFTkpQEwJzuVfR19DHvjU/JumihG5+f/p1LqeODXwKg62un+Uh+Jyt1XXnll/08Sgv7+ft5+25QsGJxlsn3vo93Hfts2WLpUL+/eDd/+tn4FWLJE758uZlTxUxj+BHwDWC8iKcBl1o/BYJhFVLf2kp6SSE6afjrPyXYz7FU0dPZRkhVl90HssA/YoJTaAaCUel5EHhSRJKXU4HRcIFqVu9dff31EDMi6ujruuusu1o5M6jMYosi112oli+7u4D723/423HUXLFqktz/6qDN97Dds0MVZTzwBH38MP/6xVt9YvlzLWW3YAPspD+wjFgzT+4BlIvI2kAD8Vik1jba5wWCIBRo6+yjK9LsS5mRrY7S2rdcYpuF5G1gsIqVKqWoROQrYM11GKUSncvfqq69m8+bNnHDCCVxwwQU8/vjjdHV1sXr1am677TbWrFnDhg0b+OSTT/jxj3/MwMAAN954I1lZWZSXl5OTk8Ptt9/OW2+9xQ033IDX6+W0007jhhtu4KKLLmLXrl2cdtppvPDCC/v3ZhgMU2Syfe+jLb22ebP2ih5xhC5wAv1l88kndS750NCYh0+KGWmYKqVOCFgeAr7p3GwMBsNMoL6jn6LMVN/6XMswrWnrY8U05jfFE0qpPhH5KvAHERkG+oB/ns5r2JW7TzyhX2+8MXRRRn+/7s99+OH+D7aJcvfdd/Puu+/yyiuv0NjYyDe+8Q2Gh4c59NBDudVyxTY3N/Pyyy8DsGLFCl544QXy8vK44447aG5uRinFddddx3PPPUdmZiYXXXQR1dXVrF27luuvv954TA2OMtm+99FuUjFvnp7b/PnB2885R8taPf/89F1rRhqmBoPBMJL6jj4+U5HrWy/J0kZqrSmAGhOl1FvA5yJ9nS99SXtNf/AD+NnPRu+/+Wb4+tdHf7BNlmeeeYaNGzeSnJxMT08PA1YJ82c/+1kAGhoaKC0tJc+qvLKN1MbGRrZt28ZZZ50FQFtbG9XV1cydG1LkxWCIOnYf+5IS3ZI0sEmF3cd+3z5n2pF+/evh982fP/b+yWIMU4PBMONRStHQ0U9hQCg/IzWJzNREalqNYTpTiGTl7pAVK7z77rvZsGEDXV1dPPDAA779iYn64yw3N5cdO3bQ1dVFeno6f//73wHIz8/ngAMO4IUXXvAZtR6Ph71799Lf3z/1iRkM08hVV+nX6ZJeikVioSrfYDCMw8CQlzVv7ua3r+50eioRoa1nkIFhL0UZqUHb52S7qWvvc2hWhpFEsnL3uOOOY9WqVRQWFrJy5UquvfZa5s2bN2pcYmIiN910E8cccwyrV6+mq6uL1NRUXC4X3/3udznuuOM49dRT+fa3vw3AnDlzaGpqYvXq1VOfnMFgmDaMx9RgiAPuf3M3P31uKwBnLZ8Td8VA9Z3a+AzMMbXX6zuMYTpTiGTl7i9+8Yuw+y6//PKg9S9/+cv80z/9EwDf+ta3WL58OQBnnnkmZ555ZtDYhIQEXn/99alNymAwTDvGY2owxAHrd7f4lt/a2ezgTCJDfYcOtQZW5QMUZ2otU8PMYPNmeOQRf2vC9HS/TNQf/hC99qTf+973OPHEEzn22GNJTEzk7LPPjs6FDQbDfmM8pgZDjKOU4oO9bZx3RCl/31rPP3Y2c+4RpU5Pa1qxvaKjPKZZqTR19TM47CUpwXzPdppoVu6OxVjeVYPBMLMxhqnBEONUtfTQ0j3AEeXZ9AwM8fau+POYNliGaUFGsMe0JCsVpaChs98nH2WIDEopZByNmmhW7jqFUqbTmMEQSYyLwWCIcT6qbgdgeVk2B5ZkUt3aS9/gsMOzml7qO/rJ9iSRmpQQtL3Y8qDuMwVQESU1NdWnBTqbUUrR3NxMamrq+IMNhhH86ldOzyA2MB5TgyHGqbJ6yC/IT2dHXpfe1tLDkqIMJ6c1rdR39I2qyAd/aN8UQEWW0tJSqquraWxsdHoqjpOamkppaXylyhgig1K6Xe+dd8Izz2hx/Ouu00L5112nuzdFWyg/FjCGqcEQ49S295HjScKdnMC8XA8Ae5rjzDDtDNYwtSm2RPaNZFRkSUpKYv7+KuMbDLOIwUG49FL4y1+0WP7ChfDcc3D00fD441o8/8wzdRcnJwTzZzImlG8wxDh1Ab3iK/LSANhjeVHjhYaOvlGFTwA5niSSE13GY2owGGYMSvmN0p4e8Hq1XFpenn71eqG7G556So+b5RkyozCGqcEQ49S19zEnWxtt2Z4kMlITqWrpcXhW04fXq2jo7B8lFQUgIloyynhMDQbDDGH9et3rvsd6DOfkwPXX6+XvfU+vg243+vTT8O67zsxzpmJC+QZDjFPX7u8hLyKU53mobI4fw7S5e4BhrwrpMQWjZWowGGYO994La9fqRhM2xcVaLg10s4nTTtM9721uuAEuugiuvDK6c52pGMPUYIhhegaGaO8d9OVaApTnprG5rsPBWU0vdpi+METxE2gt04+r26I5JYPBYAjJZZfBz38ON94IV1wReszatf7l++6D22/Xxxk0JpRvMMQwtW3aaLND+fZybVtv3Ej7NPjakY4O5YPWMq1r74ub+zUYDLFLcjLs3Anf+Y7OHx0YCD2uvx8uuQSuuUaPT06O7jxnMsYwNRhimLr2XgBf8RNAcZab/iEvbT2DTk1rWtnXbrcjDeMxzUxlII7u12AwxDZuN7S3w0MPwYUXhh5z0UXw8MN6nNv0BgnCGKaGuEYpnYDe0qIlO+KNOttjGmCYzrHC+rWW0Rrr1LX34hIozAjtMfWJ7Js8U4PBMAM4/XRwWdZVSujHls9D6nLp8QY/xjA1xC2bNsG8ci9paVqm409/cnpG049tfBZl+Z9+JVZrTttojXXq2vsozEglMSH046rYundjmBoMhpnAtdf6vaDHHqtfd+zQwvo7dgRvT03V4w1+jGFqiFvuuAOq9/r/xHvjw4EYxL72PvLTU0hJ9LfqLLFF5+PEUKtr76UkO3wLyGLLW2wkowwGw0xg1Sotnl9QAKecogucTj1Vd3869VS4/369vaAAzjpLd4Ay+DFV+Ya45eVXvQR+9xryBq/HA7UBGqY2+ekpJLqEurb4sMTr2vo4sCQz7P7CjBREiJv7NRgMsY2I7ui0ejX85Cdaq7TNEg6prNQe0jPO0C1KH3zQtCUdiTFMDXHJvn1QVamN0KRkLyVXP8+qzx8F5Dg7sWmmrq2XBQVpQdsSXEJRnIjOK6Wobe/lpAMKw45JSnAxN9sdV9qtBoMhtklKgvPPh5ISXd/w7LM6aud2a6/puefqzynTjnQ0xjA1xCVvvulfPnyFoj7Ry/b6TlaUx5lh2t7HMYvyR20vyUqNi+Kn9t5B+ga9vrzZcMzPT6MyztqwGgyG2Oaqq/TrOec4O49YI77imgaDxdtv+5dPOt6FOymBT+s7nZtQBOjoG6Srf8iXUxpISbabujjwmNo6raHuMZAF+Wnsbuw2WqYGg8EQ4xjD1BCX2JWPAIcdJiwpSmd7fZdzE4oAdtV9KG9ivIjO+3VaxzZM5+en0dk/RFNXGDVrg8FgMMQExjA1xCXbd3h9ywsWwOKiDLbFmcfUNtrmhPKYZmnR+Zbu2DbUdjfp8HxFXtqY4+YXpAeNNxgMBkNsYgxTQ9yhFFRW+ssc58+HJUXpNHT20x5H3YHsUH04j2ngmFhle30XeWnJ5KSN3a9vvmW47mrUXvFhr+Lnf9vKKb94laau/ojP02AwGAzTgzFMDXFHczN0d2nD1JOmyM+HebnacNnbGj+V23VtuiNSUYiOSHaL0lg3THc0drGwMH3ccaU5bnI8SayvbAHgobcq+dUrO9nR0MWD/6iM7CQNBoPBMG0Yw9QQd+za5V+ev0BrxJXmaEOtujX2K9VtasfoiGQL0tfFUGX+JzXtfFLT7suLVUqxo6GLRRMwTF0u4ZhF+byxvYn+oWF++9ouVlXkcsqBRTz49h6Ghr3jnsNgMBgMzhMThqmI/EpEPhaRV6yfLzs9J8PMJdAwXbxQv5bleACojieP6RgdkfLTUkhKkJjxmHb1D3Her//BGXe/wf/8XVeuNXUN0N47yOIJGKYAxy0uoKGzn+sf30hdex//ftIivrCsmLaeQfa0xM/v3WAwGOKZmDBMgVzg35RSJ1g/cdj13DBdBBqmCxbokH6mO5GMlMS48pjWtfWFrVZ3WSL7sdINad2WevqHvGS5k/j1qztp7upnU207wIQ8pgAnLC3Ak5zAEx/U8IVlxRy3pIDFRfrYeFNkMBgMhngllgzTpvEGicjXROQ9EXmvsbExCtMyzET27PEvV1ToVxFhbo47bgxTuyOSnUsaijnZsXO/T39US0lWKo9fdTR9g8P85tWdPL9pH2nJCawsz53QOQozU3n2m5/j309cxI/PWQbAQqtaf0dDfCkyGAwGQ7wSK4ZpAvA7EXlTRH4qIiE/jZWlFFOjAAAgAElEQVRS9yilViqlVhYUFER5ioaZQnW1f7mszL9cmuOJm1C+ryPSGPqeFXmemGnT+d6eVk5YWsiiwgy+dHgp971ZyR/W7+XUg4pwJydM+DwV+Wlct3opeem6ICwtJZG52W62NxiPqcFgmF5+9SunZxCfxIRhqpQ6WSl1DHAykAn8yOEpxTyvbWukoTM28g8ny969flH50lL/9tIcNzWtvTEvOg9QY4Xo54zRqrMiP42mrn66+od825RS/O/LO/j8Xa+xZ4a08GztHqCtZ5CFBVo54VsnLyYzVXdL/vKKsrEOnRCLCuOvucJkEZHjRUSJSLbTczEYYhWl4J134PzzweOBNWv06wUXwPr1er9h/4kJw9RGKdUH/AZY5fRcYplt9Z1cet96Vv3HOrbHmeg8wN4Aj+lIw7Szf4j23tjXMq2yPKHzcj1hx9ii9JUBovMvf9rAHc9/ytZ9nTy+oTrcoVFlt2Ugz8/X852X5+GdG0/ho1tO49jF+ft9/kWF6exq6oqLLyRTwYow3Qw0Oz0XgyFWGRyEiy+Gk06CP/9Zf7Y895x+ffxxvf3ii/U4w/4RE4apiBRYrwL8M/CWszOKbV7cXO9bfnbjPgdnMv309EBbqy54SkxSFBb695X6KvNjI+8StCH5UAi5I7vKvDxvfMN0T0A4/6kPa8n2JLGiPIdnP5kZv/vdjVZ3p3x/d6fkRBdZ7qRpOX9JVip9g146eofGHxyf3A3cCYR1G5v8fIMhPErBpZfCX/6iP2O8XlixAvLy9KvXC93d8NRTetws/Q48bcSEYQo8IyJvAm8CacAtDs8npvn71gYOmZvF0qIMNlS1Oj2daSUwv7S4ROEK+Asvy7W1TGMj79LrVVz18AZ+8OQn3PjExqB9e5q7yUtLJiM1vPFmG62VlkeyZ2CIFzbV88VDSjjrsDnsaOgK8qY6RWVzNy7xS3pNNwVWA4J4TV0ZCxH5PlCrlHpurHEmP99gCM/69fD009ooBcjJgeuv18vf+55eB+jt1ePefdeZecYLiU5PYCIopUzofproGRji/apWrj5xEY1dA/z141q8XoXLJeMfHAMEGqbzyoLvKdY8pu9WttA36MWTnMBTH9Zy85kHk56i/2X3NPcwbwxvKejCn8KMFHZabTrf2N5E7+Awpx9SQpp1nu0NXUGeSifY3dRNWa6H5MTIfE8uytQFYg2d/SwuyojINWYiInIZsAy42Om5GAyxyr33wtq1sHKlf1txMRx2mF5evhxOOw32BQSgbrgBLroIrrwyunONF2LFY2qYJnY1dqMUHFCSyYryHDr7htjRGD+FIXv3+pfL5wUbplnuJDJSY0fL9JmNdbiTEvjNV1bQP+TlpYAUjD3NPZSPkV9qc2hpNhv2aK/437c2kJGSyGcqcqmwjNqZUABV1dIzZq7s/lJoeUzrO2adx/R64BDgAxH5EJgDvC4iBzk7LYMhdrjsMqiq0q+vvKJ/1q4NHrN2rX/fpZfqz6HLLov+XOMFY5jOMnZbodsFBWkcMjcLgK374qcAqjpM4ZNv2xiSUf/Y0cTft9YzMDQz2lduqevgkNIsjl2UT0lWKs9urAOgf2iY2vZeyvPG93QetSCXPc091LT18vetDRy3pIDkRBfZnmSy3Em+ML+TNHcN+MLtkaAwwGM6m1BKHaiUWqaUWq6UWg7UAp9TSm12em6GyWFkiZwjORl27oTvfEcbnQMDocf198Mll8A11+jxycnRnWc8YQzTWcauxm5EdGGM7aXaMwPyDCfD8DB0hrGlq6r8y6EN09Ci83tberj4d+/wL2ve44/vVo0+0AF2N3WzID8Nl0s46YBC3tyh+8Bvr+9CKVg4gY5IRy/MA+CHf9lEQ2c/Xzik2LevIs8TVBjlFK09A+R6IvcUT09JxJOcQEPH7DJMDbGLkSWaWbjd0N4ODz0EF14YesxFF8HDD+tx7vAqfoYJYAzTWcaupi7mZLlJTUrAnZxAcWZqzIiwAzQ3w6LFitxcxQ03KIaHg/cHGqbl5aOPL81xs7elZ5R0kO2NBHhzh/OqOu29gzR1DfgklE46oJDugWHe3d3K27v0/FZVjN8R6cDiTAozUnhhcz1luW6+sKzEt688L81xj2nf4DA9A8PkpEXWvVCYkTIri58CUUpVKKXanJ6HYWymS5bIeFmnj9NPx1dImxImuGN7SF0uPd4wdYxhOsvY3dTNggJ/CLg8zzMj8gwnygMPQOVuYWhIuP124a67gvfvrvQbnKENUw/dA8O09QQ/1Z/9ZB/L5mZy3hGlrK9swet11iVhV8vbhulnF+aTkujiuU/qeGd3C+V5HorH6Ppk43IJv//qkXz+4GJuO3sZCQFFbhV5Hmpaex1NXWjt0XGx3EgbppmpxmNqmPHsjyyR8bJGjmuv9XtBjz1Wv+7YAWecoV8Dt6em6vGGqWMM01mEUordjTo8bFORl+bTxIwF/vxE8NP1/gf860oFe0znzRt9fGmOLRnlD+e39wzy0d42TjuomKMW5NLSPeB4C8vAXGAAd3ICZx02h9+/U8WLm+s5cv7E+scDLC7K4DeXrODEpYVB28tyPXgV1LU7VwzW0q0N0xzP9GiWhsN4TA2xwFRliYz4e2RZtQrOPBMKCuCUU+C+++DUU+GZZ/Tr/ffr7QUFcNZZ8JnPOD3j2CYm5KIM00NH7xCd/UOUBVRAl+d7aHyvn+7+IZ+E0EyloQHe+kfwtk0bhbo6KCmBlhbo7dEeQU+aIidntARWmU8yqodDSnXx1/t7ddX6yoocn7TQxpp2lhY7Jy20q8nS9gz4XX39hIU8ZnVrOu+IEAm0k8QuCmrq6p9QIVUkaO3Wn5Q5EcwxBchPT6G5K0zVgsEwA5iqLNGFF8LLL/u9rBDsZd2+PdjL+sgjIPGhDhg1RODBB2H1avjJT/SXgjYrKaayUntIzzgDli3T48z7u38Yj+ksorpNP7XmBvRX97WtHCOcf80fP+TO5z91vKXjm2+C1zv6P/6ll/RroLe0rEyFfDjMDeExfX9PKy6Bw0qzKc1xI6KLoZykurWH4sxUUhITfNsWFqTzhyuP4vXvnsiRC/L2+xoF6TpZqtHBavWWKIXyczzJdPYPMTg8MxQXDIaRTFWW6KCDjPh7NEhK0mkS556rvaRpaTqfNC1Nr597rk6bSIps8GdWMLNdZIZppbZNhzLnBBimdmX+3pZeDp6TNeqYzbUd/PmDGgDy0pO54pj5UZhpaLZvD7193Tot0xFomM6vCP2VNcudRGZqYpBk1PtVrRxYkunzGBdlpDquddrY2U9B5ugcUrvKfjqwJZqcNExb7VB+pA3TNP1p0dYzGFFpKoNhqgTKEr38Mvzud6Elh/r74atf1UYmwC23GPH3aHHVVfr1nHOcnUe8YzymccLupm4uvvdtNtW2hx1TYxljttcQgkPboXj8/WqSEoSCjBRe2+ZsD207yRyCJTveelt7cvfs8W8rLw8fS9FaptrwHPYqPqxq44h5Ob79Zblu9jrctrSho98nDB8pctOScYnDHlPLMM12R9bNkG2lCrT1mHC+wVnGqpafrCzR4KARfzfEH8YwHcH2+k66+oecnsakWfPmbv6xs5l/uudtOvtCZ7jXtPWSmuQiL8A7leXR3ZCqwoSuX9pSz3GLCzhhSQEba9odDedv3+6/9vnnQ0KCXt++DTo6YNs2/9iKivDnKc1x++73032ddA8Ms6I8J2C/rlZ3kobOvogbpgkuIS89hcYuBz2mPQNkuZNITIjso8gurmrtMdUfhugymWr5ycoSnXGGEX83xB/GMA3g9+/s4dT/eo2bn/rE6alMioEhL3/5qJYsdxIdfUO8s6sl5Liatl7mZLuREcmX83I9IXMq23sH2dPcwxHlORxamkVT1wB17c5VNn8aYJgeeigcdLBeVkrYsAE2bfLvP/jg8Oc5oCSTnY1ddPUPsaFKFz4FeUxz3NS19zqWjzgw5KW1Z9BXiBVJCtJTHPeYRjq/FPzFVbaH1mCIBpOtlp+KLJERfzfEG8YwtRj2Km5/disAT39US//Q8DhHwP++vINfvhQm8TGKfFDVSmvPID86ZxnJCS7erQxjmLb2BhU+2ZTleNgbwkO4ubYDgIPnZHJIaTYAH1c7o8/d2wt1NfrPNSFBUV4ORx3pN7DfeQc+2eQfP5ZhurI8B6/S79sHe1rJT0+hLNf/vpTaMkptzhjhtgcz0h5T0HmmThqmbT2DEZeKAn8OqwnlG6LFVDRJpyJLNF3i70aQ3zBTMIapxfaGTjr7hzjrsDkMDite29Y05viq5h5+8eI27lq3jU8d7jW/zdLcXFmewyGlWeEN07a+0IZpbuhuSHa+6sFzsjigOAMR2FLnzL3u2uVfLi3TD9pVq/zbnnpK0dSoDdVUtxozlH/4vGxcAu/sauHNnU2sLM8J8iKXZtuV+87kmTZ0aIO4MDP+DdPoeUxNKN8QXaaiSWrLEi1bpmWJrr1WyxGBX5bopz8NliWaqvi7EeQ3zFTi2jDtGxzm6w9t4O9b68cd+0GV9gRefdIiUpNcvraP4bj39V0kuIT05ER+8+rOaZnvVNnZ0EVacgIlWal8piKXjTXt9A0Ge3z7Bodp6uoPaZjOy/XQP+QdZaBsru2gMCOFgowUUpMSHK1W373bv7x4kX494QT/trff9huWBx7o9yCEIiM1iaXFmfzPyzuo7+jnyyuCNUFtg9Cp3MuGTttjGvlQfr6VY+pU7nBrz0DENUwB3EkJJCe6jMfUEBXuvRduvFFXyx9/vP457bTR1fL2vpUrdbX8vfdOXpZoKl5WI8hvmMnEtVzUMx/X8bdN+/jbpn28fN0JvvaOoXh/Tys5niQWFaazqDCdbfVjewbX727hmIV5pCQm8OFeZ9tP72joYmFhOiLCoaVZDA4rdjR0sWyuX/6ptk0blIEV+TaltmRUa49PdB1gy75ODpqT6R+X46amzRkvYm2tf7msTBuhCxfCnLleamuCrdBDlo2vbvzvJy7i3x55n2xPEicdENwRKd/S92xySJDdb5hG3mOan57M4LCivXfQV7keLZRSUfOYigg5niRfC1SDIZJcdhn8/OfaOL3iitBjAqvn77sPbr/dXy0/GVmiyYq/Q3CKARhBfsPMIq49pmvfrSI1Sd/isxvrxhy7saadw8qyERGWFGawvT58S8q+wWF2NHZx8JwsDizJpLK5m54B5yr5dzR0saggHYAlRbpb0UjDusY2TMN4TIGgyvxhr2JnY5fvfKCNWqc8pnUBv745c/SrCJx6yug/4SOPHP98px9awrPf/ByPX/VZXK7gp26WO4mkBKHJIY9pY0cfLoG89MgbpnnpzhUF9Q4O0z/kjbiGqU2OJ5mWbuMCMkSeQE3SaFTLT8bLOtW2p/GOybGdOcStx3TYq3i3spXvnLKEpz6sGdOrqZSiqqXHJ16+qCidP39QQ3vvIFkh9BU/3dfJsFdx8JxMElyCUrB1X2dQZXe06OwbZF9HHwsLtWFakechOcHFpyMMU9tjOieEYWobq3tb/EZnVUsPA0NeFlvnBe0xfebjOoaGvRGX9xlJoMfUNkwBvvhFeOAB/7rLpbj44ol9vQ/0BgciIuSlpdDkUO5lc7cObye4Iu+msMPoTngSbWM4N0qe2hxPsgnlG6JGYLV8Zyc88cToMRddBE8+qZfT9rMr8ES8rFNtexqPgvxKaSP9zjt12sOyZXDdddq7fN11OuXBeIqdIW49pn1WVf0hpZkcVpbNh3vbwubRNXcP0DMw7PMcLinUXsIdDaG9ppvrdLX6QXMyObBEGzdbrG3RZk+z/tq7wEpTSExwsaAgjW0jCrJqWntxCRRnjc5bTE1KoCgzJUgyartl2C4O9JhmexjyKuodMNgCDdOSEv/yeefBaav9v9evflXIzt7/6+VnJDvmMW3pHoiaF9EOozvhSWy1rhk1j2maCeUbosd0VctPJ1Ntexpvgvwmx3ZmE7+G6aDWoFxanMnysmwaO/upDaPBaYew7S5Idvh6e5g80821HWSkJFKW46E0x01GaiJbHapW32fdU0mAJ3RpcQbbRqQiVLf1UpyZSlIYT+e8XE9QKH+7ZZQvGuExBRwRn6+p9RufgR7ThAR49I/CV69U/PM/K372s+m5Xl5aimM5ps1RyruEAI+pA6H8FstIzE2LTnPpLHcS7b2x1zzDEJtMtVo+kkQ7xWAmMhUZL0N0iWPDdJiM1ETmZKVyaKkuAtpYHbpdp+0pnJenDdO5OW6SEoTK5tCFPrubullQmI7LJYgI8/PT2BOmc1KkqWu3QvQBntAlRRnUtPUGdYCqae0NWfhkUxbQphN0jurcbDfpKf5sD/t4J2SUwoXyAbKy4N57hIcfnh5vKegCqGaHPKat3QNB3bkiiS/H1AFPom0MR6voKsudTEfvoKPdywyzh6lUy0eD2S7Ib3JsZz5xm2PaPzjM0qIMRMSXf7m7qTvk2L0jPKYJLqEsx0NVS+jxlc3dQS0sy3I8joXy69r7SLRaS9r4PL4NXb6816qWHo5akBf2PKW5Hp78sIaBIS/JiS421Xb40hRs5mTpJ9S+jugKzw8NQWODf72oKPLX1KH8AZRSozplRZpoVaqDllFKSXQ54zGNco5pljuJgWEvvYPDeJLj9tFnmCFMtlo+Wo+Z00/X4Wqvd+akGEQLk2MbG8Sxx9TLkmJtoGWmJpGfnsLuptA5o1UtPRRkpOBOTvBtK8v1+PI3A+kfGqa2rZfyPH+memmOm+q2Xrze8J6Ytp4BnttYx67G8NX+U6GuvY+izNSgQpmlI1IRuvqHqGvvCwrLj2RRYTpepRsNdPUPsbOxi0MC5KYA3MkJZKQm0tARXU9iQwN4vfr+cvK8UQkrFaSnMDDspaMvuqFfr1fR2hM9w1REyE1LptkBw7S1ZwCXQGaIAsNIYBcytveaxDFDdJisJmk0mIkpBtHC5NjGBnHrNhhWylcQBLo4qLIpdAh6b0svZSPC3OV5Ht7f0zrKY1bd2otX6ep3m9JcDwNDXhq7+sP2N//Z3z7lD+uryE9PYf2NJ4+SKJoqde29lIwoaCrNceNOSuDTfdoI3t2oPb8LC8KXfS63Wo5+tLedrr4hlMKXAhFIUWYq9VH2mIYrfIokfi3T/pDKDJGirXcQryJqhinoPFOnPKbZUVIfAMj2+A3Tkqw4i08aZiyT0SSNBnaKwbp1/hSDH/1Ie3E3bYKbb/anGJx8cvRSDKJBYI7tyy/D734XOn+2vx+++lXt5e7sjK8c21ggbj2moL2eNvPz09gVJpRf1dLjq8i3mZfrobN/aFQLwz3N+hwjPaYQPvdSKcXLW3Usuqmrny37pi/sX9feF1T4BOByCYuL0tneoD2mOxr161ge07JcNzmeJD7a28bGGp2Lu2zuaMO0MCMl6oZpYFhl7pzoGDE+wzTKCgS+8HYUDdPctGRnckx7BnytQqOB/QWjzbQlNcxiptL2NJ6Y7Tm2sUBcG6aBxub8gjSauvrp6Av+UBoY8lLX3jvKMLUNz6oRRU221zXQY2rnpgbqgAaydV8n+zr6uObUJQC8uaNp3Lm/uLmeFT96kTue3xq2WEMppQ3TEBJQS4oy2FLXiVKKnQ3dJLiEebnhPaYi4pPVen17E3Oz3RSE6DykPabRNdYCDdM5JdF5StpFQdGuzLcN07y0yIvr2+SkOecxjaYBbkL5BoNmulIMYlGUfibKeBmCiWvDdKTHFKByhNe0tk2H5ktHGaZ63faQ2uxp7iYjJTHoA3U8j+kb27UheuFnylhUmM4bO5rHnftj7+2luXuA/315Z8hcV9Af7AND3pCG6fKybJq6+qls7mFHQxfluR6SE8f+dX+mIpdtDZ28uq2RC1aWhRxTmJlCY2d0e6vX1/uXi4ujc03bY9rcHW2Pqb5eTpQklADy0pId6fzU2j3ok6uKBsYwNRj8XHWVTi949FHo6oLhYf36xz/q7V//+uhjlIJ33tFGrccDa9bo1wsu0NXusSB4MZtzbGOFmDFMReREEVkvIv8QkSdFJHes8YkuCZI6svNNR1bm77WMyZEeU9sLWjXCKKxs7qE83xOUd5qalEBBRkpYj+mn9Z0UZaZQlJnKEfOyx63g7xsc5vXtTRy7KB+AN3eG9rDW2RqmIQzTY6xjX9/eyPrKFg4JkS86kiuOqWBpUQbJiS4uPnJeyDFFGakMDHujGg4NNEyjUZEPOrztkuiH8pud8Jh6kunoG2Jw2Bu1a4KWqIqqx9TOMTWhfINh0sSLKP1MlfEy+IkJw1REUoHfAF9WSn0WeBX4yVjHjPQOzsvzIAK7GoMNUztUP9IwdScnUJiRMkqfdE9zd1B+qU1Zjttn5I5kZ2MXC61e9vPz02ns7A/SGB3JW7ua6R0c5srjFlCSlRo29O83TEcnwVTkeSjJSuW/XtxGS/cAnz94fFejJzmRP/7r0fz16mNDhvEBX3FXfWf08kwDQ/nRMkwTXLpavTHKoXw7pB5Nj6ktcB/NrkhKKVqj2OEKID05EZcYj6lheonFcPZkiSdR+tmeYxsLxIRhCqwG3lRKVVnrvwPOGuuA5BEdjlISEyjNcY/ymFa19JCUICGr6cvzPEEe08FhL9WtvUH5pTalIwTqbZRS7GgINEztlILwIvV2O9HlZdl8dmE+b+1sDhk632eJ65dkj567iHDygYW+4q3jlxaEvV4gWe4knw5qKIoytcEazTzT+nr/vUcrlA86nB/ttqTN3QNkpCSSkpgw/uBpwjYOW6PYlrSrf4ghr4qahinookDd/ckYpoapEw/h7MkSb6L0M1HGy+AnVuSiKoBd9opSqlNEEkUkSSnl+5QRka8BXwMoKSsfdZL5+emjQ/ktPZTmeEJK1szLTQvyVta29TLkVaE9prlunt1Yx7BXBZ2rsaufzr4hn1TTAut1V1NX2PB6ZXMPuWnJZLmTOGRuJo+/X01jZz+FI4zn2vY+khKE/DBh3+9/8SCKMlLJSE2cNkHxwgw9h4YoVubX1fmXo+UxBWcM05YoexHBL3AfzTxT2wiO9r1muZNoM4apYYoMDvo9h319sHChDmcffbQOZz/7rA4TP/hg/Bg18SpKP9NkvAx+YsVjmgKMVDofAoK+myql7lFKrVRKrZxTmD/qJAvy09jd1B3kfdzT3OMrdBpJeZ6HfR199A0OA/halFaEMExLczwMedWorkg7GywNUUuqaV6uTikI14UKoKql2zenxZb3clv9aGH+fZa4fjhNVHdyAlefvJjLj5kf9lqTpdDymDZEMfdynwM5pqAr85sdqMqPZt4lQK6lQBDNUL4tT5UbxZQFwHhMDVMmnsLZk8GI0huizZiGqYj8RUTuDLPvfhH5r3GOf0NELtqfCVpUA75qHBFJA/qUUpNqyzM/P42u/iEaLS+YUoo9zT0hDU3wV+bbLUvtCv1QoXy/ZFRwiH6n1enJ1hBNTUpgbrZ7lDpAIJVNPZTn2oapPs7WJA2ktm20uH6kSU1KIMudFDUt074+6OzQhndCoiJ3zJK36cUpj2nerPCYWrm0UQzlA2R5ko1hapgS8RbOniiBovSXXgoDYR4T/f1wySVwzTV6vJOi9LMh7zeeGc9j+iDwTyISNE5E3MB5wJoIzWskzwGfFxHbX/Y14JHJnsTO77Q7ITV3D9DVPzSGx1SP32mN37qvk4zUxJCFQX7JqOA80x0NXXiSEygOCMNX5KWxO4wEVP/QMLXt/panBekpZHuSQntMO/oc6WBTlBk9kf2GBv9yfr7y6c9Fg/z0FHoGhukZiF5bUic8ptkeO8c0ih5TBxoJgOUxdaCZgCG2ufdeuPFGHc4+/nj9c9ppo8PZ9r6VK3U4+957nZ33dDHTRelnY95vPDPex/zTgBs4fsT2c4CdSqmPIjKrESilWoGrgadF5G3gM8BPJ3ue+SMko2yvZTiP6dKiDFwCm2t1J6TNtR0cPCczSCrKZk62G5HQHtOFBelBx5TluqkJUSgFWqRfKajI18ayiLC4MN3X995mLHH9SDOWyL5Sit++upNH3qkKuX+yBOYtFRdHtzwy3xbZ74yOIaOUotkBwzQ50UVGSmJUuz/ZaQPRzzFNnHUeUxH5oYiss6T2HheR8bXjDEHM9nD2TBaljxcZK4OfMQ1TpVQ/8CjwzyN2XYLlLRWR74jIVhGpEpHNIvL5UOcSkR+LyG9GbHtJRL4SsP45EXlbRCpF5H0R8RnESqlnlFKrlFJHKaUuVkqFj4WHYU62m+REl98wtXNG80Mbpu7kBBYVprOxpp1hr2Lrvg4OnhP6mZ6c6KI4M3WUx3RXY/eoHvWlOR6auvp9uauBhGp5urgog+0NXUG5sU1dWlx/Tnb0PaaFGak0hskxfWR9FT99bis3PrFxVJetqRBY+BStdqQ2+ZZnvDFK4fzugWEGhrxRN0xBG4jRDOW3dA+Q6BIyUqJbf5nt1qF8r3dWuVC2KqVOtqT2NgM3Oj2hWCMWw9nTyXSK0k9nmH225v3GOxMJjD4InCciKQBWOP0E/KH0DuAopdQ84AZr/KQRkYOAPwPfVEpVAP8O/ElERlcxTZEEl1CR52GXZZjuadatOueOYdwtm5vFxpoOdjV20Tfo5aCSzLBjy3I8QVqmPQND1LT1+qSibOzrhZKX2hOiwGpJYTrtvYNBxqDdZcpOIYgmhZkpNHT2hfxwX7t+r0+V4M8bqid0vqFhL0NhxN1ra/3Lc+ZMfq77Q4HV/SmcET7dtDoU3oboG6atPQNke5JDRh8iSZY7Ca+CriimZziNUuoPAavvAiVOzSWWmenh7EiyP6L0kQyzz9a833hnXMNUKfUPoAn4orXpYuB5pVSjtf//gD4RORRIBgpEpHAKc7ka+LVSan3Add8HTp3CucIy36rMB50zOl6rzkPmZtHU1c/zm3RM+eC54Q3T0pzgEL0t5m9X5AeOg9AtTO2Wpzkef7WyXY77twIAACAASURBVJm/vcGfZ1rT1mudK3R+bCQpykhhcFiNquLuGxxmS10H/3rcApYWZbBua0OYMwTzw6c3s/y2F3ltW+OofY4appbHNFoFUL6uT+nRN0zz0pKjW5XfPRD1inwIaEs6C7s/iUgS8E10FGzkvq+JyHsi8l5j4+j/Q8PMDmdHmqmK0kcyzD7b837jmYmWkjyENkghOIyfIiKPAO8A3wUOA7xoA3WyLACussL4lSJSCRwOTEwZfoLMz09nT3M3Q8NePqhq5fB5OWOOP2pBHgB3vbSdijwPiwvDi8+X5nqoa+/1tXbcYRmSIz2mtjEZymMaquWpXZm/LSDP1D52rgMeU1/3pxF5ppvrOhjyKg4ry2bZ3Cw+3TdaSWAkvQPDPPT2Hrr6h7j+8Y9HNRJw0jC1PZfRMkxbuvV1ol2pbl8zmgL7rd2Djtynry3p7MszXQS8CDymlPrryP2BUnsFBdP6yI0bZnuP9cmK0k9nmD1U+H+25/3GMxM1TB8ETheRo4FS4Flr+2VAvlLqMKXUV4A7xjhnJ5A+Ylug+E8t8BOlVEXAT6FS6r8nOMcJsbQ4ncFhxTMb62jqGmBF+diG6YElmZx7xFyGvIpvnLgopBC/TWmOG6+CujZdsb51XydJCeIrurIpzEghKUHChPJHtzwtSE8hyx1cmV/d2kO2J4n0KOfoAT6h/30dwfP/eG8bAIeVZrOkKJ2Gzn7axvHCvbhFi5SecWgJte19PgUEGycN06QEl25LGqVQvq2ZmhemYUIkyU1Lim6OaU/0i7wgwGM6iwxTETkLeAD4N6XUb52eT6xieqxrUfpzzoFHH4WuLhge1q9//KPe/vWv+8fuT5h9IuH/2Z73G89MyKpRSlWKyHvAfwG/D+i2lAKkW/mnw8CtY5zmPeBbIpKplOoQkeOAQwP2rwEeEJG/KaW2iEgCcLZS6s+Tu6WxOWlpEUkJwo+f2QLAEeXZ4x5zy5kHc/SCPL50+Nwxx/m0TFt7mJfn4dN9HSwsSB+VKuCy8lpHhvLtlqenHxqcAiYiLClKZ0eAlmlNa68j+aUAZdZ1RyoLfFTdTlFmCsVZqSwp9jcGWDU/vPjoG9sbyfEk8f+tXspfP67jje2NPs1XgJpaBegvA9E2TEFX5kfLY9pkGab5Gc7kmPYODtM7MIw7OfLtUFsd6HAFs88wtWoC7gBWKaXanZ5PLGOHs1ev1uHsp5+GNv1d3BfOPuMM02Md9q9b1OWXT7y7VmDeb2cnPPHE6LlcdBE8+aReTgtd52yYYUxGFfIB4EiCtUvvBxqASuBj4E20gToKpdQ669g3RORvwLnA6wH7X0UXTz0mInuALcAxk5jfhMjyJHHUgjzd4jMjZczQvO8YdxLnrywjMWHst6ssVxtsVZZk1Kf7OllaHPr8pTmeUR7TsVqeLi7KYFu9vzK/urV3zKKtSJKfnkJyomvU/D/a28ahpdrQX2rlxX5aP3Y4f1t9FwcUZ1Kel0ZFnoc3AlrAgrMeU7BF9qPjSWzs7CctOWHa2sdOBp/IfhTyTL1enZ+c60AoP3v2hfIPB/KBp0TkFetnWr/szyZMj/WJMdUw+6WXTi78P5vzfuOZCX8CWkVO/zdiWxda0zTkOZVSx44YfyNjSJVY1aN/CLd/uvjGCYtITnDx/dMPHDM0P1nmZLnJTE3k4+p2vnjIILXtfWMYpm5e2hJcHGRX5NtdnwJZbFfmd/WTl5bC3tYejlviTC6YyyWUZruDDNP23kF2NXVz3opSAEqyUslISRylvxqIUoodDV2cd4T2RB9RnsMb2/2G6cAANDeJdU1FYWH0XRAFGSl8UNUWlWs1dvWHbN4QDeywemv3QMS/8HT0DeJV0dcwBb/HtG2WFD8ppf4G5Dk9j3jC9Fgfn8Aw+8svw+9+FzqE3t8PX/2q9j53dsKHH44d/n/+eWht9Yf///u/tQe1uzs47/fb34a77oJFi/T2Rx+Nz7zfeCX6rpkZwNEL8zh64fQ/q10u4YjyHN7f08qWug4ADhjDMLW1TFOTdOjU1/I0hK7qErsyv76Ljswh+ga9Yc8dDUpzg6WxNlbrKOGhpVrnVUSYl+cZ1XAgkLr2Prr6h1hk3duyOVn8+f0aGjr6KMxMDQrzFBQqEhKib5hGsy1pU6fzhmk08kz9XZ+i71ZyJyWQlCCzyWNqMDjCZMPseXn+Knub8cL/v/+99qRu2eLP+/3Rj3RqxaZNcPPN/rzfk0+Oz7zfeCSKDR5nByvm5bCtoZMn3q8hKUFYMS90fmWoyvzK5h5Sk1wUhjBOAivzN1mdqMKJ/UeD0pxgj+lH1dqreOhcf87u3BFe1ZHYKgNLrJzSZXP1/Wyq1Ub93r0B1yt1JmGrIEO3Je3uj7zupZMeU9t7GQ3JKF/XJwdC+SJCliWybzAYIsdkw+wnnDC18P+zz05exsowszGG6TSzojwHpeCP7+3luMUFPnmakdgyT7YeKWiPaUVeWkjRcbsyf3tDF5trO0hOcPmMVScozXHT0j3gM9g+2tvGgvy0oPudm+Ompq13lASUjS2nZeu0HjRHa8R+UqMN7+3b/WMXLnDmiWK3fK1rH21gD09z96DGzn7y0x3ymHqi6THVRqETVflgtyWNngKBwTAbmay81ne/O7Uq+7Q0k/cbb8zKUH4kWVmRy/KybD7c28ZZy8NX64QS2a9s7hnVvtRGRDigOIMNla0UZKSwpDidpHGKsSJJaYACwQHFmXxU3cbRC/JGjekZGKatZzBkPuGe5h4yUxN9Bkp6SiIL8tP4xPII2w8vgMWLI3Qj42C3fK1p62NRQKFc3+AwZ979BgeWZHLXhctx7Weucv/QMO29g75uU9Em052ES/zdpyKJkx5T+7rR1Gw1GGYjtrzWunUTD7NPtcre5P3GF8ZjOs0kJ7p4/KrP8vhVR3PWYeEN08KM1CAtU69XUdXSE7Ii3+bs5XP5tL6Tt3Y1c3jZ2Pqrkcauut9U08G+9j7qO/o5rCxYemus1qugvcUjO1cdPDeLT2p0KD/QY+q0YVrbFnwPa9dXsb2hi798VMtDb+/Z7+vYlf9OhfITXEK2J9nXfSqSONl61b5uNLtcGQyzkal0izJV9gYwhmlESHAJK8pzx+wDnuASSnM87LYE5es6+hgY8jIvREW+zTmHzyEjNZEsdxJXn7xo2uc9GRYVppOWnMCHe9v8+aWlwYZpqS9dIXQBVHVrz6jOVcvmZFLT1ktr9wDbtvtD5U4ZpkUZKSS4ZJRm631vVrKqIpeFBWkhW6mG4/2q1qAOXja2iL9ThilEz2Br6R4gOdGFJwp6qaHITYuOAW4wzHYmK68127trGTQmlO8gS4syfDqfn+7TXsJw8lIAnuREHvyXVaSnJFKYkRqVOYYjwSUcWqpTFhJcQnKCi4OtHFEbf7rCaI+pUoqa1l6OWZQftN0ugPqkpoPt2/ypAU4ZpokJLoozU4M8pntbeqhq6eFfjqlgY00Hr3zagFJqzC8iAN39Q5z7q38A8PD/O5JjF/vvvaFDdwtzKscUdJ5pNHJMGzr7KUhPGff9ihS5acm0dg9M6HdmMBj2j8mE2acS/jfEH8Zj6iBLizOobO6md2CYTTVjy0vZHD4vx1cs5DSHlWWz8f9v787joyzPxf9/ruz7SlZCCIuyyCoBBG1BUSgFZFEoVQr4tVqXX0sFjgiv31FaFVtrPfX06OlRv6JiFWtRluNaBVoU2VQEFVGWsIVAWBJCNkLm/v7xzGSdyUaSZ2ZyvV+veU3mmfvJXPPMZHLN/dz3dR8r5JUth7ihb0pV2SuX2PBgIkMC3SamBSUVFF+orFcz05Xcbv7qPMXFVtIQFW3o1Kner2g36XFhtSapfbr/NAAje3ZiUGYcp4svNFh9wGX1zmNVPz+9YV+t+1z727WaF0B8Oy1Lml9UTnKMvT3DFx2Gc2VtX2lBKdV0LTn9r/yPJqY26pMWjTFW2aRvjp8jKzGC6DDfmTp4fZ9kAC46DDc7C+vXJGINVzhWUD9pc22rm4jFRYTQvVMk696rThqu6GvvB1DnuPDaiemB03SKCuGy5CgGO8fVfnGk8SL8q784Ru/UaO69tgfbcs5QUOO0+eEzJUSGBNo27hIgJSaMvMKyNn+cfGePqV3as2arUqp5dHUtpafybdQ71eod3JtXxNe55+jXOaaRPbxLdlYCr91xFf/8Lp8fXOa+S7NzfHi98ZlQs4ew/pja8f1TWfpsdSmsSZPs/VrcOT6c/911nAsXHQQHCpv3n+Kq7omICL1SowkMEL7LK4KBnn+Hw2H4JvccNw/J4Ia+qTy9YT8b9p5k6mAroT9ypoQuCRG2nlpOiw3nXNlFissvEhnadh8NJ4vKyM6yb/JezcS0m5vFLJRS9tJZ9h2b9pjaKDMhgqjQINbtyuXwmRJbC+a31IgeiTwwvjdBHkpXWUX2609+cm1zt/zl6G7plByoXm51xoxWCraF+qbFctFh2JtXxIFTxZw4V87IHlYiHhwYQFpsWK1VsNw5VlBK8YVKeqXGMKBzLImRIWz6rnr51cNnShqc+NYe0uM812xtLRcuOjhbUmHrGGntMVVKKe+liamNAgKEGdld2PT9KYIDhYkD0uwOqdVlxFu9cEVltetGHisoJTIkkDg3CxB882k0VFrjVQcNMrZNfHJxLbO682hB9fjSGkvaZiZEcLiBpVcBvs2zJrn1So0mIEAY3j2BrQfPANZEMO9ITF2lsdrudL5reVe7qw8AnClun6Vmle945hm7I1BKaWJqszt+2I3QoABmXdW1wRqmvsrdCldgncrvHB/u9tT19dcLTzxhzdCcOdP+0e0Z8eEkRoaw60gB//oun7TYMLomVieRXeIjOHKm4V7GquVXnat1De+WyLGCUo6cKSG/qJzyiw4yE+1NTF2rXNWt2dqaXGWx3C27214SI63HPqNF9js8Y2DrVmtMY0QEvPiidT1jBmzbZt2vlGpfOsbUZmmx4Wz8t9G2lglqS1VF9s+UVo2pBTh2ttTtaXyAzp2t2ZcLFoDD0S5hNkhEGJARy+qdx6ioNPzih91rJdSZiRGcOl9OyYWLRIS4/5Pam1dE57jwqsltw7olALDlwGm6OHtKu9jcY5oSE4YI5LbhBKiTXlCvNTwkkLDgAO0x7eAqKqylL9euhbIy6NED3n0XRoyAVausNdgnTbJmf+tEG6Xaj/aYeoG02HBblxdtS67JTfV7TEvcTnyqK8BLDsu0KzOoqLS6T+aMzKp1nyuhbKhk1KEzJXSvsdxsr5RoUmJC+cc3J9i87xQBAlfavJpXcGAAydGhHK/xWhljeGPHEd7dfRzTCt1HVT2mNpaLAqvX9PR5HWPaURlTnZSWlFhfgIcMgcRE69rhgOJiWLPGaqc9p0q1H+0xVW2qU1QIoUEBtRLTc2UVnCu7WG/VJ282aWA64cGBnC25UDUW06WL83kcPl3C5R5qzOYWlNKnd3LV7YAAYXy/NF7ddph9+ecZnBlPrJvxtu0tPS6c3MLapbH+7e+7AHhsWn9+Oizzkn5/bkEpgQFi+xmClJhQ8s61fWks5Z22bYN166ykFCA+Hh54wPp50SJ4/304exZKS61227dbQ4uUUm1PE1PVpkSk3sz8Y15QTL4lru+b4na7q+fXXfUBgPKLleQXlddLaCcMSOPFzTkcyC9mwQ2dWzfYFuqRFFVrJaunN+wjITKEsKAAPvg6r1mJqbuVlQ6cOk9mQoTtZwjS4sL5JvecrTEoezz3HKxcCdnZ1dtSU2Ggs9zboEEwdizk5VXfv3gxzJwJd9zRvrEq1RF5yYlS5c/q1jJ1/expjKmvSYwMISQwgLxz7scsuorW101Ms7vG88iUfkwYkMY0NwsU2GFgRiynzl8gt7CMg6eK+WTfaX7+g26M65fK5v2nKauobNLvOXW+nOHLPmLhG1/W2udAfjHdvaB2aHqstcxsawxPUL5lzhw4fNi63rjRuqxcWbvNypXV982eDUeOWO2VUm1PE1PV5jLiw2uNvzzkLK1kd3mk1hIQIKTEhnqs/+kaxuCqE+oiIsy6qitP33Kl1yTpAzKslay+PFLA2p25iMDUwZ0Z3SuZ8osOtjlLXDXmxU9yOFlUzt8/O8pLm3MAa5GBnNPFXlHUPi02nHJnTVXVsYSEwP79cN99VtJ5wcNQ4/Jy+NnPYP58q32IfYuyKdWhaGKq2lznuHBOF1+g9ILVc3Yg/zyx4cG2Lr/Z2tJiwjnuYTa7qy6otySfDemdFk1woPD5obO8+cVRhndLIC02nP6drVqu35883+jvqHQYVmw5xLgrUhiWlcDK7UcwxnD8XBllFQ66J0U1+jvamutLQluWxlLeKzwcCgthxQr4yU/ct5k5E155xWoX7v1/ukr5DZ9ITEXkHRHZJiIbnZdr7I5JNV3dmfkH8ovpnhRp6/KbrS0tzvM6867kJzXWvtWOmio0KJCBGXG89GkOh06X8H+u7gZAfEQwseHBHDzVeGKac7qYwtIKru+TwsxhXTh4qpgtB85wIN/at2Z1ArukxlqZhqcvE8q/TZhQXfEj1MM8PFcPaUCA1V4p1T58IjEFEoCbjDGjnZeP7Q5INZ1r9r1rctDBU95xOrc1pcZaiam7MYu5BaUkRYcSGhRoQ2TN97ubBhATFsy1vZK4wTnhS0To1imSg6eKG93fNamoT1oMP+6fRkxYEK9tO8xnh84C0DPZC3pMY9t++VXlvRYsqO4FvcbZzbFvH0ycaF3X3B4WZrVXSrUPX5mVHw+carSV8kpdnD2mh8+UUFx+kbxzZfTwgtO5rSktJowLlQ7OFF8gsU4ppGMFpfUmPnmznslR/Ov+awkJCqjVq929UyRbDpxudP89x88RFCBclhJFaFAgUwd35tVth6moNFzdM9H2UlEAnaJCCQ6UevV1VccwbJhVPP+jj+D66+GFF+DhhyEnB77+Gh580NqelARjxsDQoXZHrFTH4Ss9pgDvicgmEVkkIm7jFpE7RWSHiOzIz89v7/iUBykxoXSKCuHLI4VVPW7eMDO7NTV0aji3oJTOcd5/Gr+myNCgeiWdunWKJLewrGqssCd7jp+jR1JUVQ/x7JFZVYsTzMju0jYBN1NAgNC9UxR784rsDkXZQMRa0alfP1i2zOoRzcmx7svJsW4/9ph1/8svW+2VUu3D6xJTEUmtMZZ0o4hkGmN6GWNGAROBq4B73O1rjHnWGJNtjMlOSkpq17iVZyLCoC7xfHHkLDuPFADQK9V9IXpf5Vpnvu44U2MMuQVlpMf6To+pJ92cY0MbO52/53gRfdKqX98eSVFsuv9aHprUlx/3T2vTGJtjQEYsu44WasmoDio4GKZPh2nT4IYbIDLSGk8aGWndnjYNZszQ5UiVam9edyrfGJMHjPZwX6GIvABMb9eg1CUbnBnHh3tO8NYXx+iSEO53Y0zTPIxZLCipoLSi0qdO5XvSpcYktr7pMW7blF+sJO9cGV0Ta7++XRIiuM05kcpbDOgSxxufHeXoWWuoxRs7jnDgVDG/GnMZUaFe99Go2sDdd1vXU6bYG4dSqprXf/qKSCgQYowpEpEg4CfAJzaHpZppcKZVH/OzQ2eZM6KrX83IB0iMCiUoQOqdyq+uYer7iWl1r7DncZknCq1FBurWbPVGAzOsEli7jxWyblcuj7+3F4Dk6FB+/oPudoamlFIdltedyncjBviniGwCPgb2AX+xNyTVXMOyEhjfLxXAq07ntpbAACElpn7JKFepKF+oYdoYV/Ld0Brzuc6kNc0Hhi70So0mKjSI//zoe/70j+8Z3y+VKzPj+OvWwzgcenpfKaXs4PU9psaYfOBKu+NQlyYoMID/njWEwpIKYiP8c9BWWmxYvR7TXA+rPvmiwAAhOTq0wdqfrqEMvtBDHBoUyMNTruC+17+kb1oMy6b258M9J/i3v+9iT945rkiPtTtEpZTqcLw+MVX+xV+TUrBqmX7trOHpkltYRmhQgN+scuWq1+qJa5UrX0nEpw7OoHNcBH3SookOC2ZwZjwA3x4v0sRUKaVs4Aun8pXyCVaPaWmtWd5HzpTQOT7cb8bUpsaGNXgq/3hhKbHhwUSE+M533mHdEogOs74wZSVGEBIUwN4T/lNGSkSuda6ct1lEVotIgt0xKaWUJ5qYKtVKUmPDKatwUFhaUbXt6NnSqtns/iA1JtzjClcAxwvKqiZJ+aKgwAAuS45iz/FzjTf2ASIShjUm/2ZjzEjgn8Aye6NSSinPNDFVqpW4EjLX6WyAI2dLyIj3/vGWTZUWG0bJhUrOlV10e39uYZlPjC9tSK/UaH8qvD8O+MQYc9h5+3ngRhvjUUqpBmliqlQrcSVkR8+WAFBUVkFBSQVdEvynxzTFmXyf8HA6/3hhqU/3mAL0To3mZFE5BSUX7A6lNWQBB1w3jDFFQJCI1BrsravmKaW8hSamSrWSrEQrAc05ba2MdOSMNUPdn07lVy8kUD8xLb1QSUFJhc/3mGYmWIsDHD3ruV6rDwkF6nZvXwRqjcXQVfOUUt5CE1OlWklcRAjxEcEcPGX1mLp6Trsk+HaiVlNqjOci+9U1TH27x9RVc9a1OIKPOwpkum6ISCRQZoxxPxZDKaVspompUq0oq1MkOc615I+c9b8e05SqxLS83n3HnWNrfaG4fkNcpa5y/SMxfRf4kYikOG/fCbxqYzxKKdUg36npopQP6NYpks37TgOw72QRseHBxPlR7daQoAA6RYWQd85zj6mv1DD1JCEyhNCgAL9ITI0xZ0Xkl8A6EXFgjTe9w+awlFLKI01MlWpF3RIjefPzY5ReqGTX0UIGZMT6TQ1Tl1Q3K1xBdY9pqo+fyhcROseF16qu4MuMMW8Db9sdh1JKNYWeyleqFXVLsibOfJ1byHcniujX2f9WD0qNcb/60/HCUjpFhRAaFGhDVK0rPS7cX8aYKqWUT9HEVKlWNLxbIiLwl3/up6LS0N8fE1MPqz/lFpb5/PhSl/S4ML84ld+RPfOM3REopVpCE1OlWlFSdCiDusTx4Z6TAH6ZmKbFhlNQUkFZRWWt7ccLSn1+fKlLWmw4+efLuXDRYXcoqomMga1bYfp0iIiAF1+0rmfMgG3brPuVUt5PE1OlWtmY3skA3Dwkw6+K67tUz8yv3Wt63I96TFNjwzAGTp2vX31AeZ+KCrjlFrjuOnjzTcjIgHffta5XrbK233KL1U4p5d00MVWqlf38B915fnY2j980wO5Q2oS7Ivvnyio4X37Rb3pMk6NDAc8rXCnvYQzMng1r10JJCTgcMGQIJCZa1w4HFBfDmjVWO+05Vcq7aWKqVCsLCw7k+r4pBAT412x8F9es+5olo/ylhqlLcrT1HE8WaY+pt9u2Ddats5JSgPh4eOAB6+dFi6zbAKWlVrvt2+2JUynVNFouSinVLKluiuz7Sw1Tl+QYq8dUE1Pv9txzsHIlZGdXb0tNhYEDrZ8HDYKxYyEvr/r+xYth5ky4Q6u5KuWVNDFVSjVLZGgQ0WFBtZYl9bce08TIEEQgX0/le7U5c+Dxx2HJErjtNvdtVq6s/vmFF+B3v7P2U0p5Jz2Vr5RqtrQ6RfaPF5YSINVjM31dUGAAiZGh2mPq5UJCYP9+uO8+a/zohQvu25WXw89+BvPnW+1DQto3TqVU02mPqVKq2VJiwmpNDMotKCMlJoygQP/5rpscrYmpLwgPh8JCWLECiorgrbfqt5k5E1avtn6OjGzf+JRSzeM//0WUUu2mbo/psYIS0uP84zS+S3JMKCeL9FS+t5swAQKc/8lCPXTYu3pIAwKs9kop76WJqVKq2VKdBegrKq0C9Afyi+nWyb+6opKjQzl5TntMvd2CBVavKcA111jX+/bBxInWdc3tYWFWe6WU99LEVCnVbF0TIjAGDp0upqisgpNF5fRIirI7rFaVHB3GqfPlVDq08KU3GzYMJk2CpCS4/nprgtMNN8Dbb1vXy5db25OS4MYbYehQuyNWSjVEx5gqpZqtV2o0AN+dOE9xubU0afckP+sxjQnFYeB0cXlVXVPlfUTg5Zdh3DhYtsyqVVpQYN2Xk2P1kE6cCP36We3EP8sLK+U3NDFVSjVbz+QoRGBvXhFlFVZi2sPfElNnhYGT5zQx9XbBwTB9OqSlQVkZvPOOVVA/PNzqNZ02zaplGhxsd6RKqcZoYqqUaraw4ECyEiP57kQRlQ5DYICQmeBfiWmSMxnN15n5PuHuu63rKVPsjUMpdWk0MVVKtchlyVHsPVHE+fKLZCVGEBLkX0PWq3pMdWa+Ukq1G69KTEVEgPHAE8BIY0yBc3sI8AwwEDDAfxpjXmnp4zgcDk6dOkVBQQGVlZWtELlSviEsLIyMjAyCW+Gc5sAucfxjzwlyThVzz+ierRCdd3EtS3pCZ+YrpVS78arEFPgIOAQk1Nn+S6DAGDNUROKAz0RkozHmaEse5OjRo4gIWVlZBAcHIzoaXnUAxhhOnz7N0aNH6dat2yX/vlnDu/L8pgOcLangpiEZrRChdwkNCiQuIlh7TJVSqh15W2I60RhTIiI5dbbPAiYDGGMKRORNYCLwl5Y8SHFxMb169SIgwL9OPSrVEBEhMTGR/Pz8Vvl9sRHB/P6mAew6Wuh3NUxdtJapUkq1L69KTI0xJR7u6myMOVzj9iGgc91GInIncCdAZmZmg4+lSanqiFr77MDYK1IZe0Vqq/5Ob5IcHabLkiqlVDuyLTsTkVQR2Vjj0lAmWTeBrgQcdRsZY541xmQbY7KTkpJaNV6lVMdj9ZjqqXyllGovtvWYGmPygNFNbH5KRJKNMSedt7sCe9skMKWUckqLC+NEUTkXKx0EBepZFqWUamu+8kn7d+AeABGJxxpf+patESml/F5GfASVDsMJPZ2vlFLtwlcS00eBy0RkM/A+sNhVSkoppdpK57hwAI6e8TT8XSmlVGvyysTUGJNVM/E0xhQbY241xow0UhOsswAAHZBJREFUxgwzxqyzMz5/MXfuXCZOnNjs/c6ePUtKSgr79+9vsN3NN9/Mk08+2dLwfE5bH8/GdLTj3R46x1uJ6bGCUpsjUUqpjsErE1PVsIULF/KjH/3okn/PU089xSuvNH+dgmXLlvHjH/+YHj16NBjXQw89xCOPPEJhYaHb33PXXXdx33338fTTTzNgwABiYmKIiYlhxIgRvP32282OqzGPPfYYQ4cOJSYmhqSkJCZNmsRXX33Var+/tY7n0qVLEZFal9TU+jPfm3u8VfO5ekyPndXEVCml2oMmpj5o+/btDBs27JJ/T2xsLHFxcc3ap6SkhOeff57bb7+90bj69+9P9+7d3SZrxhjWrVvH5MmTycjI4Pe//z2ff/45O3bs4LrrrmPKlCns2rWrSTHNnTuXpUuXNtpu48aN3HPPPWzevJn169cTFBTE9ddfz5kzZ5r0OI1pzePZq1cvjh8/XnXZvXt3vX2bc7xVy4QFB9IpKlR7TJVSqp1oYupDKioqCAkJ4V//+hcPP/wwIsIVV1zR4D7/+te/uOqqq4iKiiI2Npbhw4dX9RLWPfU8evRo7rnnHpYsWUKnTp1ITk5m4cKFOBzVlbneeecdAgICuPrqq5sU14033shrr71WL67t27dTVlbGNddcw+TJkxk/fjw9e/bk8ssv59FHHyU6OppPP/30ko5XXe+//z633XYb/fr1o3///qxYsYL8/Hw++eSTJv+O9jieAEFBQaSmplZdapY/a8nxVi3XOT6co84e06+OaW+0XZ55xu4IlFLtwasK7NvlN+u+5pvcc+36mH3TY3hoUsNJZV2BgYF8+umnZGdns3XrVjIzMwkNDfXY/uLFi0yePJnbb7+dv/71r1RUVPD5558TGBjocZ+//vWvzJs3j82bN7Nz505uueUWhgwZwk9/+lMANm3axJAhQ2oVam8ormHDhvHII49QWlpKeHh41T6rV69mwoQJBAXVfgtWVlbyxhtvcP78eUaOHNms49NcRUVFOBwO4uPjm9S+vY4nwIEDB+jcuTMhISEMHz6cZcuW0b17d6Blx1u1XNeECD47dBaAX6z4zOZoOg5jYNs2eOIJePtt6NcPFi6EiROt66FDQVeTVsr/aGLqQwICAjh+/DjR0dEMHTq00VV8zp07R0FBAZMmTaoav9i7d+8G9+nbty+//e1vAbj88st57rnn+Oijj6oSqUOHDpGWltbkuNLT06moqCA3N7fWmNQ1a9bw8MMPV93evXs3I0aMoKysjKioKN566y369+/fhKPScvPmzWPQoEGMGDGiSe3b63gOHz6cF198kd69e3Py5EkeeeQRRo4cyddff01iYmKLjrdquQEZsaz9MpddRwv0lH47qaiA2bNh7VooK4MePeDdd2HECFi1Ct55ByZNgpdfhuBgu6NVSrUmTUyh2T2Xdvriiy8YOHBgk5aWTEhIYO7cuYwbN44xY8YwZswYpk+fTpcuXTzuM2DAgFq309PTOXnyZNXt0tJSUlJSmhyXq9eutLT6H/q+ffs4cOAA48aNq9rWq1cvdu7cSUFBAatWrWLOnDls3LiRfv361XusZcuWsWzZsqrb5eXliAhPPPFE1bZ3332XH/zgBx6f5/z58/n444/5+OOPG+zxrKm9juf48eNr3b7qqqvo3r07L730EvPnzwead7zVpRmcafWo/9+PD9ocSfOISCjwBNALiAe+BO42xlTYGlgjjKlOSkucVbqGDIHEROv6+++huBjWrLHavfqq9pwq5U90jKmP2blzJ4MHD25y++XLl7N161Z++MMfsnbtWi6//HLef/99j+2D63Q/iEitMZGdOnXi7NmzTY7LNbGo5hjJ1atXM2bMGCIjI6u2hYSE0LNnT7Kzs3nssccYNGgQ//Ef/+E2xrvuuoudO3dWXW688cZ627Kzsz0+x/vuu4/XXnuN9evXV50eb6r2Op41RUVFccUVV/D9999XbWvO8VaX5or0GIIDhTU7cwkN8qmPzEhgnTFmrDFmKBAC/NzmmBq1bRusW1edlMbHwwMPWD8vWmTdBigttdpt325PnEqptuFTn7IKvvzyy3q9cI0ZOHAgixYtYuPGjYwePZqXXnqpxY8/ePBgvvnmmybH9dVXX5Genl6rV3DNmjVMmTKlwcdxOByUl7tfbSchIYGePXtWXaKjo+tt8zS+ct68ebz66qusX7++0dPwnrTH8ayprKyMb7/9ttYp/+Ycb3VpwoID6d85FoBBXZpXdcFOxpgzxpgPamzaAaR5au8NnnsOliyB7GwYNcq6jB0LAwda9w8aZN123ZedDYsXW/sppfyDJqY+5uLFi3z77bfk5uZSUGCtQfBf//VfbpOsgwcP8sADD7B582YOHTrEhg0b2LVrF3379m3x448bN449e/Zw+vTpRuMCa3JPzVqb+fn5bNmyhUmTJlVte+CBB9i0aRM5OTns3r2bxYsXs3HjRm699dYWx+nOvffey/Lly3nttdeIj48nLy+PvLw8zp8/X9XG07GE9jueCxcu5J///CcHDx5k69at3HzzzRQXFzNnzpyqNk093qp1PDF9IMum9ufRqfWHlvgCEYkBbsPDUs4icqeI7BCRHfn5+e0bXA1z5sDhw9b1xo3WZeXK2m1Wrqy+b/ZsOHLEaq+U8g+amPqYRx99lJUrV5KRkcHixYsBOHXqFHv37q3XNiIigu+++47p06dz+eWXM2fOHG699VYWLVrU4sfv378/w4YNY2Wd/xbu4iorK+Ott97ijjvuqGq3bt06hg4dWqtHLy8vj1mzZtGrVy/GjBnD9u3beffdd+uNtbxUzzzzDEVFRYwZM4a0tLSqS82xqZ6OJbTf8Tx69Cg//elP6dWrF9OmTSM0NJQtW7bQtWvXqjZNPd6qdXRPiuKW4Zn0TI62OxSPRCRVRDbWuGQ6t2cDHwBLjTFfuNvXGPOsMSbbGJNt5zCQkBDYvx/uu89KOi9ccN+uvBx+9jOYP99qHxLSvnEqpdqOGGPsjqFNZGdnmx07dri9b8+ePfTp06edI/If7733HvPmzeObb75pcOLQ008/zZo1a/jgg+qziZMnT+bqq6/m/vvvb49QfUJTj2dj3B1vd/T9f2lE5DNjjOdBzF5ERH4BTAN+bow50pR9GvrsbA+RkdXjS6dMgbfc9PFOnQqrV1e3r3HSQynlpZr62ak9pqrZfvSjH3Hvvfdy9OjRBtsFBwfz5z//uda2q6++uqpUkrI09Xg2xt3xVh2XiAzAmuw0salJqTeYMAECnP+ZPJVpdvWQBgRY7ZVS/kMTU9Uiv/rVr2qdWnbnzjvvpFevXrW23X///Q2WV+qomnI8G+PueKsObSSQDvyjxul9r18/acECcM1dvOYa63rfPquw/r59tbeHhVntlVL+Q+uYKqWUHzLG/AX4i91xNNewYVbx/I8+guuvhxdegIcfhpwc+PprePBBa3tSEowZY60ApZTyH5qYKqWU8hoi1opO48bBsmVWrVJX4YmcHKuHdOJEa4nSl1/W4vpK+RtNTJVSSnmV4GCYPh3S0qwlSd95xyqoHx4ON9wA06ZBXp4uR6qUP9LEVCmllNe5+27rupG1OJRSfkYnPymllFJKKa+gialSSimllPIKmpgqpZRSSimvoIlpB1JZWcnYsWP5/vvvm9T+j3/8I0899VQbR+UdsrKy+Pjjj+0OQ6kO4Rmvr6aqlLKLJqY+ZPny5fz6179u8f6BgYF88MEHXHbZZU1qv2DBAubNm9fix1NKKQBjYOtWa6Z9RAS8+KJ1PWMGbNtm3a+UUqCJqU85dOgQ5xtYFNrhcLRjNEop1biKCrjlFrjuOnjzTcjIgHffta5XrbK233KL1U4ppTQxxSrQbNelqWbOnMmf/vQnXn/9dbKysnj55ZfJyckhKCiI119/nZ49e7Jw4UIqKiq466676NatG126dOGaa66pdepeRKrWZJ87dy6LFy9m9uzZdO3ala5du/LGG29UtZ07dy6PPPIIABs3bqRnz54sX76cfv36kZSUxG233UZFjf8mX3zxBaNGjaJLly706dOHZ555hqCghiuSrVixgoEDB5KZmUn37t2rYvvwww8ZMWJEVVzLli2r2mfp0qXMmTOHO++8k65du9KjRw/Wr1/PqlWrGDBgACkpKcyYMYOSkpKq2Hv37s0bb7xB//79SU9PZ9SoUXz99dce41q7di2DBw8mKyuLq6++mi+//LKpL5VSyskYmD0b1q6FkhJwOGDIEEhMtK4dDiguhjVrrHbac6qUwhjjl5chQ4YYT7755ptat62PQ3suzfHQQw+Z22+/ver2wYMHjYiYBQsWGIfDYSorK01xcbF5/vnnTXl5uTHGmIULF5opU6bUeK6YI0eOGGOMmTNnjklJSTGfffaZMcaY1atXm6ioKFNYWFh1/8MPP2yMMWbDhg0mLCzMPPjgg8bhcJhTp06ZHj16mBdeeMEYY0x+fr5JSEgwb775pjHGmIKCAjN+/HgTGBjo8fns27fPxMbGmuPHjxtjjDl27Jg5d+6cMcaYNWvWmO+++84YY8zRo0dNXFyc2blzZ9VxiI6ONlu2bDHGGPPss8+a5ORkM2HCBFNSUmKKi4vNkCFDzFNPPVUVe2RkpPnFL35hysrKjMPhMH/4wx9Mt27dTFlZmTHGmK5du5pNmzYZY4xZv369ycjIMN9++60xxpg333zTZGZmmtLS0ma9Xt6q7vtfNQ+ww3jBZ1xbXRr67GyuLVuMiYys/ryLjzfG+WdsvvjCuu26LzLSmK1bW+2hlVJepqmfndpj6uOMMcybNw8RISAggIiICG6//XZKSkrYtm0bERERDfYMTps2jSuvvBKAyZMnExERwd69e922DQoK4sEHH0RESExM5KabbmLHjh2A1fM5evRopk6dCkBsbCyPPvpog7EHBwdTUVHBzp07AUhPTyc6OhqAG2+8kR49evD999+ze/duUlJSaj2PUaNGMXz4cADmzJlDfn4+S5YsITw8nIiICCZMmFCrl9PhcPDUU08RGhqKiLBw4UIcDgfbtm2rF9eTTz7Jv//7v9OrVy8Apk6dSlRUFNu3b2/w+Silqj33HCxZAtnZMGqUdRk7FgYOtO4fNMi67bovOxsWL7b2U0p1XLryE759+iggIICMjIyq2zk5OcyePZvKykr69OkDwIULFzzun56eXut2fHw8xcXFbtumpKQQGBhYq+2JEycA2LdvH7179673uxqSmZnJ2rVrefDBB5k/fz5Llixh1qxZgFUR4Nlnn+WKK66gW7duGGNqPY+UlJSqn0NCQggJCan1XKKjo2s9j/T0dEJDQ2s9fnJyMqdPn64X14EDB3jooYdqDR84f/48J0+ebPD5KKWqzZkDjz9uJae33ea+zcqV1T+/8AL87nfWfkqpjksTUx8nIkiNwaoPPvggY8aM4aGHHgLgnXfe4cMPP2zzOBITE8nJyam1bf/+/Y3uN2bMGMaMGcNnn33G1KlTiYiIYODAgfz2t7/l6NGjVT2omzZtuqT4zpw5gzGm6liVl5ezb98+evbsWa9teno6S5cuZfr06Zf0mEp1ZCEhsH8/3HcfbNgAzz9vbaurvBx+/nNYtw6Kity3UUp1HF51Kl8sPxaRb0Qkrsb2ZBE5JyIbnZf/tTNOuyQkJHDgwAEALl686LZNeXk5Z86cAeDs2bM8+eST7RLbzTffzKpVq9iwYQMAx48fr5o45cnhw4f59ttvARgwYABdu3alqKiICxcuUFlZSWFhIQCvv/46u3fvvqT4CgsL+c1vfoPD4eDixYssWrSIwYMH069fv3pt586dy6OPPsrhw4cB65iuWbPmkh5fqY4oPBwKC2HFCvjJT9y3mTkTXnnFahce3r7xKaW8j1clpsBHwHQgoc72BGC7MWa08zKx/UOz38yZMykqKqo3e76mpUuX8umnn5KRkcH48eP5iaf/Bq1swIABrFixgnvvvZeMjAxmzJjBL3/5SyIjIz3uU1RUxNSpU0lPT6d///6MHDmSWbNm0adPHxYsWMCwYcPo0aMHO3fuZNSoUZcUX7du3QgODqZ79+707NmTM2fOeDyGt956K7fffjtjx44lKyuLQYMGsWfPnkt6fKX8halRkzQyEgICrGt3NUknTLDuB6gzkqaKq4c0IMBqr5Tq4JoyQ6q9LkCE8zoHiKuxfSTwehP2vxPYAezIzMz0ODNMZyW3j9WrV5vWnOHbUhs2bDA9evSwOwyvoe//S0MHnpV/4YIxM2caExFhTECAqVVhJCDAmlk/c6bVzpjas/L//Gdr2/ffGzNhgnVtjDH/+Z/W/REROitfKX/W1M9Or+oxNcaUeLgrGhgsIp+IyNsico2H/Z81xmQbY7KTkpLaLlBVz9mzZ9mwYYPrCwJ79+5lyZIl3HvvvTZHppRqDcZNTdKa3NUkHTYMJk2CpCS4/nprgtMNN8Dbb1vXy5db25OS4MYbYehQe56bUsp72Db5SURSgRpzMpltjDnsrq0x5n3gcud+g4HVIjLcGJPX9pGqpnA4HDz66KPMmjWL4OBg4uLimD9/PnPnzrU7NKVUK9i2zZqgVOKp+8CptNRqt327lZi+/DKMGwfLllnbCwqsdjk5sGABTJwI/fpZ7Zqz6IhSyj/Zlpg6k8rRLdjvCxHZAfQFNDH1EomJie0y+78lRo8ezb59++wOQymf9sc/WklnU5SWWu1ffx2Cg63xqGlpUFYG77xj3R8ebvWaTpsGeXlWO6WU8olyUSLSyRhzyvlzV2AQ8JW9USmlVMfx9tv1T9974nBY7V3uvtu6njKl9eNSSvkXn0hMgQkich9QAAhwuzHmkqqdmxo1LZXqKFxjgJVqrqb2lra0vVJKgZcmpsaYrDq3XwJeaq3fHxwcTGlpKREREa31K5XyCRUVFQQFeeWfvfJy4eGNjy+t214ppZrLq2blt5fk5GSOHTtGSUmJ9iCpDsPhcHDixAliY2PtDkX5oJo1SRujNUmVUi3VIbtOYmJiAMjNzaWiosLmaJRqP5GRkXTq1MnuMJQPWrDAmrhUXNx427Awq71SSjVXh0xMwUpOXQmqUkqphrlqkq5Z0/D40fBwrUmqlGq5DnkqXymlVPOIWLVGJ0+uXoq0poAAiIiw7teapEqpltLEVCmlVJMEB8Orr8L69XDTTdUJamQk3HwzbNwIr72mNUmVUi3XYU/lK6WUaj4R67T+3/5mdyRKKX+kPaZKKaWUUsoraGKqlFJKKaW8gvhrHU8RKQL22h1HC3QCTtkdRAv5auy+Gjf4buy+GjdAL2NMtN1BtBURyQcO2R3HJfLl91dL6PP1f/7wnLsaY5Iaa+TPY0z3GmOy7Q6iuURkhy/GDb4bu6/GDb4bu6/GDVbsdsfQlpryj8Pb+fL7qyX0+fq/jvSc9VS+UkoppZTyCpqYKqWUUkopr+DPiemzdgfQQr4aN/hu7L4aN/hu7L4aN/h27B1FR3uN9Pn6vw7znP128pNSSimllPIt/txjqpRSSimlfIgmpkoppZRSyitoYqqUUkoppbyC3yWmInKtiGwTkc0islpEEuyOqSlE5BkR2SUiG52Xm+2OqTEicrXzOA9y3hYRWSYinzlfg4V2x+hO3bid2wpqHPuNIhJlZ4zuiMhvROQjZ+yrRCTWF465u7id233hmP9ZRD50HtsVIhLuC8fcn7n53Ml0vq/+ISJfiMj/52G/d5yvl+v9dk37Rt4ybp5vdxF5T0TWO9+Dcz3sN0NEPnfu+6KIhLVr4C10Cc/XJ19fcP8/qcZ9y0VktYf9fPI1boxfFdh3vih/AW4wxhwWkfuAZcBd9kbWJAnAvcaYTXYH0hQi8jIQDcTU2DwNuAzIBoKBj0XkY2PMFhtCdMtd3CISCpwyxoy2K64m+tYY8xCAiDwMLAG24eXHHDdxi8iD+MYx//+NMYVQ9d6ZAZzH+4+5X/LwuZMCLDLG7BORCOAzEVlvjPmmzu4JwE3GmCPtFO4l8/B8TwCTjDEVzi95B0XkFWPMxRr7pQFLgZHGmAIReQr4NfC79ou++Vr6fJ187vUFj8/Zdd+1wBDggJv7fPI1bgp/6zEdB3xijDnsvP08cKON8TRHAr613Nhdxpip1I55FvBnY7kAvARMsSU6z9zFnQictimeJjPGvFbj5nYgDR845h7i9pVj7kpKY7Di/hIfOOZ+rN7frzFmuzFmn/PnEmAPkOpm33h86zMW3D/fYmNMhfPmZcAeN0naDGClMabAeftZfOM92tLnC775+oL7/0mISCpW4rnEw36++ho3yt8S0yxqfLMwxhQBQSISbFtETRcIPC8in4jIYyISbndADXH+A6gri9rf7A4BndsloCbyEHcskCIim5ynbSe3d1zN4Xw//wr4Gz5wzF3qxO0Tx1xErhORrVjH9X1jzE586Jj7Gw9/v1VEpB/QF/jUQ5P3nO+5RSLi9f//PD1fEXlcRL4FVgL3ummShQ++Ry/h+br41OsL7p+zc1jT68Avsc7QuJOFD77GTeETL1wzhAJ1v0ldBLy+WKsxZowx5mpgDFaX/sM2h9QSdY9/JeCwKZYmM8bsMcZkGWN+ANwJPC4iA+2Oyx0R6Qn8A3jDGPO/+Mgxrxu3rxxzY8x6Y8xwoCtwpYj8Gh855h2NiNyKdZZsijGmtO79xphexphRwETgKuCedg6x1Rhj7jfG9AamAn8TkS51mvjVe7QJz9dvXl/nF/i/AU8YY3Y10NSvXuOa/C0xPQpkum6ISCRQ5qHb3ysZY8qwxskOszuWFqh1/LH+mefYE0rLGGMOAP+LNa7Hq4jIjVinje81xvyPc7PXH3MPcVfx5mPuYow5B/w31hdHrz/mHYmIBIjIi8Bw4DpjzLcNtXcOz3gB3/yMrcUYsxvYRP3n4pfv0Qaeb802vv76Xgv0Bx4WkZ1YX7auFZG/12nnl68x+F9i+i7wIxFJcd6+E3jVxniaTESSnNcC3IrnU1He7O84v6U6JxTNAf5qa0RNICLxIhLo/DkBGIs1FtJrON/TfwB+bIz5usZdXn3MPcXtI8c80TnOy/V3OQXrn6JXH/MO6B7gvDHmVw2cCg4VkWjnz0HAT/DNz1hEpE+dv51hwI46zVYBP3N2zgDcDaxovyhbT1Oerz+9vsaYD4wxXYwxg4wxg4CfAxuMMXUr9fjNa1yXX83KN8acFZFfAutExIE1/uIOm8NqqrdFpAIQ4DPg32yOpyVeAPqJyBasMbP/Y4z5zuaYmiIbeEJECrD+Jn7j/GbuTQYDnYA1Vo4EwBmsAfDefMw9xf3feP8xj8I6bViJdZrsn8Afsf5GvfmYdzTXAP1FZGONba9i9TQ9C8zHOu35vogUY1VSeA/rzJQvuhp4zfm34wDmG2MOOcdUPuu8vVdE/gRsdP4v3A4sty/kS9Lo88W/Xl+P/Pg1rkWM8frhl0oppZRSqgPwt1P5SimllFLKR2liqpRSSimlvIImpkoppZRSyitoYqqUUkoppbyCJqZKKaWUUsoraGKqlFJKKaW8giamyi+IyBsikuO8XBSR4zVuzxKRx+2OUSmlvI1+dipvo3VMld8RkRxgljHmY7tjUUopX6GfncobaI+pUkoppZTyCpqYKr8nIktF5Hnnz1nO01XTReRLETkrIo+LSIaIvC8iR0Vkt4hcVWP/MBF5UkS+E5GDIvIXEYmw7xkppVTb089OZQdNTFVHFAgMBQYB/YC5wD+A+40xGcArwDM12v8PkAz0By4HEoGH2jFepZTyBvrZqdqcJqaqo/q9sRwDPgA2GWO+dN73N6C/iASISCfgJ8DdxphyY0wF8Dhwsz1hK6WUrfSzU7WpILsDUMoGlcaY0zVunwfyatwuwvrbCAG6YfUS7BYR1/0BQGg7xKmUUt5EPztVm9PEVKmG5QJlQHdjjMPuYJRSykfoZ6dqET2Vr1QDnKerPgUeE5FgABHpUnOAv1JKqdr0s1O1lCamSjXuViAD2C8i+4BX0b8dpZRqjH52qmbTAvtKKaWUUsor6DcXpZRSSinlFTQxVUoppZRSXkETU6WUUkop5RU0MVVKKaWUUl5BE1OllFJKKeUVNDFVSimllFJeQRNTpZRSSinlFTQxVUoppZRSXuH/AbCboEogam7yAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 792x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "t = np.linspace(t_min, t_max, int((t_max - t_min) / resolution))\n",
    "\n",
    "n_steps = 20\n",
    "t_instance = np.linspace(12.2, 12.2 + resolution * (n_steps + 1), n_steps + 1)\n",
    "\n",
    "plt.figure(figsize=(11,4))\n",
    "plt.subplot(121)\n",
    "plt.title(\"time series (generated)\", fontsize=14)\n",
    "plt.plot(t, time_series(t), label=r\"$t . \\sin(t) / 3 + 2 . \\sin(5t)$\")\n",
    "plt.plot(t_instance[:-1], time_series(t_instance[:-1]), \"b-\", linewidth=3, label=\"training sample\")\n",
    "plt.legend(loc=\"lower left\", fontsize=14)\n",
    "plt.axis([0, 30, -17, 13])\n",
    "plt.xlabel(\"Time\")\n",
    "plt.ylabel(\"Value\", rotation=0)\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.title(\"training sample\", fontsize=14)\n",
    "plt.plot(t_instance[:-1], time_series(t_instance[:-1]), \"bo\", markersize=12, label=\"sample\")\n",
    "plt.plot(t_instance[1:], time_series(t_instance[1:]), \n",
    "         \"w*\", markeredgewidth=0.5, markeredgecolor=\"b\", markersize=14, label=\"target\")\n",
    "plt.legend(loc=\"upper left\")\n",
    "plt.xlabel(\"Time\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.4.2 RNN for Time Series\n",
    "\n",
    "이제 시계열 데이터를 예측하는 RNN을 모델링 해보자. 100개의 순환뉴런을 가지고, 입력 시퀀스의 길이가 20이므로 20개의 타임 스텝을 가지며, 출력 시퀀스 또한 길이가 20인 시퀀스를 출력한다. RNN의 출력결과는 아래의 그림과 같이 [미니배치, 뉴런 개수] (`[batch_size, n_neurons]`)이기 때문에, 미니 배치를 50으로 할 경우, `(50, 100)`의 형태를 가지는 출력 벡터가 나오게 된다. 하지만, 실제로 우리가 원하는 출력은 각 타임 스텝 마다 하나의 출력, 즉 `(batch_size, 1)`의 형태를 원하기 때문에 아래의 그림과 같이 다음의 과정이 필요하다.\n",
    "\n",
    "1.  각 타임 스텝마다 `(batch_size, n_neurons)` 출력 벡터를 `tf.reshape()`을 이용해 쌓는다(stack).\n",
    "2. 그런 다음, 1개의 유닛을 가지는 Fully-Connected Layer를 적용한다. 이때의 FC Layer는 단지 차원 축소를 위한 레이어이기 때문에 활성화 함수를 사용하지 않는다.\n",
    "3. 마지막으로, `tf.reshape()`을 이용해 차원 축소시킨 출력을 다시 타임 스텝별로 풀어준다(unstack).\n",
    "\n",
    "![](./images/time-series02.PNG)\n",
    "\n",
    "\n",
    "\n",
    "아래의 코드는 위의 과정을 적용하여 시계열 데이터를 예측하는 RNN 모델을 구현한 코드이다. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "################\n",
    "# Layer Params #\n",
    "################\n",
    "n_steps = 20\n",
    "n_neurons = 100\n",
    "n_inputs = 1\n",
    "n_outputs = 1\n",
    "\n",
    "X = tf.placeholder(tf.float32, [None, n_steps, n_inputs])\n",
    "y = tf.placeholder(tf.float32, [None, n_steps, n_outputs])\n",
    "\n",
    "# RNN Model\n",
    "cell = tf.nn.rnn_cell.BasicRNNCell(num_units=n_neurons, activation=tf.nn.relu)\n",
    "rnn_outputs, states = tf.nn.dynamic_rnn(cell, X, dtype=tf.float32)\n",
    "\n",
    "# 하나의 출력을 위한 작업\n",
    "stacked_rnn_outputs = tf.reshape(tensor=rnn_outputs, shape=[-1, n_neurons])\n",
    "stacked_outputs = tf.layers.dense(stacked_rnn_outputs, n_outputs)\n",
    "predictions = tf.reshape(stacked_outputs, [-1, n_steps, n_outputs])\n",
    "\n",
    "################\n",
    "# Train Params #\n",
    "################\n",
    "learning_rate = 0.001\n",
    "n_iterations = 600\n",
    "batch_size = 50\n",
    "\n",
    "# loss\n",
    "mse = tf.losses.mean_squared_error(labels=y, predictions=predictions)\n",
    "# optimizer\n",
    "train_op = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(mse)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step: 000, MSE: 13.9070\n",
      "step: 100, MSE: 0.5057\n",
      "step: 200, MSE: 0.1974\n",
      "step: 300, MSE: 0.1012\n",
      "step: 400, MSE: 0.0685\n",
      "step: 500, MSE: 0.0629\n",
      "y_pred:(1, 20, 1)\n",
      "[[[-3.5476217]\n",
      "  [-2.5151467]\n",
      "  [-1.1526244]\n",
      "  [ 0.5834651]\n",
      "  [ 2.015859 ]\n",
      "  [ 3.030329 ]\n",
      "  [ 3.5977817]\n",
      "  [ 3.4671748]\n",
      "  [ 2.7913451]\n",
      "  [ 2.1190128]\n",
      "  [ 1.6537794]\n",
      "  [ 1.5421822]\n",
      "  [ 1.940395 ]\n",
      "  [ 2.7001028]\n",
      "  [ 3.8418205]\n",
      "  [ 5.09896  ]\n",
      "  [ 6.124487 ]\n",
      "  [ 6.6779537]\n",
      "  [ 6.6244407]\n",
      "  [ 6.0005555]]]\n"
     ]
    }
   ],
   "source": [
    "# Train\n",
    "with tf.Session() as sess:\n",
    "    tf.global_variables_initializer().run()\n",
    "    for iteration in range(n_iterations):\n",
    "        batch_x, batch_y = next_batch(batch_size, n_steps)\n",
    "        sess.run(train_op, feed_dict={X: batch_x, y: batch_y})\n",
    "        if iteration % 100 == 0:\n",
    "            loss = mse.eval(feed_dict={X: batch_x, y: batch_y})\n",
    "            print('step: {:03d}, MSE: {:.4f}'.format(iteration, loss))\n",
    "            \n",
    "    # 새로운 데이터 예측하기\n",
    "    X_new = time_series(np.array(t_instance[:-1].reshape(-1, n_steps, n_inputs)))\n",
    "    y_pred = sess.run(predictions, feed_dict={X: X_new})\n",
    "    \n",
    "print('y_pred:{}\\n{}'.format(y_pred.shape, y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXMAAAEbCAYAAAAoHWSlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xt8VNW5//HPM0kMDAFR1KpERVCP8KsYNIhCxURRjOKFUypa8acHPbZS/VWNLbR6aqttbbVVWzm2R6la9bSitKAgAVQEby0a5OZBVCK0om3FCxwySSAZnt8fe5IOgVyYmWQmk+/79dqvzb4/s1/DMytrr72WuTsiItK1hdIdgIiIJE/JXEQkCyiZi4hkASVzEZEsoGQuIpIFlMxFRLKAkrm0yMyeMrONsanBzP4Wt1yYwuvkmtlzZjYwVeds53V/aGa/TvE5XzGzj82sRwvbTzEzN7NJSV7ncTOblup9petSMpcWuftX3H2Auw8ANgFNy+6+KdHzmtk5Zvb7uOs0uPuZ7v5+CsJu7bqLzOykjrxGTD7QUrK+Cfi8E2KQbkbJXNLhUGC/NFx3CJ3znZ8DfLP5SjMbBIwA3uyEGKSbUTKXpJjZeDNbGat6ecXMjovbdoWZrTazv5pZlZkdamb3AD8BTosdMzVWzeJmdnDsuMdjVSCPm9lfYvt9Oe68ZmY3mNk6M/vAzOaZ2a9aqjIxsyPMbCNwMDA7dr782OYcM7s3Ft9HZlbe3s/XigWx845ptv4G4H5gl9euzexwM5tpZutj92q+mf1L3PZcM7sttv0DM3sCCDc7xxfN7IVYnGvNbEI74pRs4u6aNLU5ARuBLzVbNwb4C3B0bPkrwPvAPsAxwGfAQbFt/YFesX9fBSyIO08uQYI7OLb8OPARUBRb/jKwFSiILV8DrAD6x5ZPAbYAv27jM2wCTo5b/mHsvGfGlouBHcBRbX2+Vq7xCnBx7DPOi1u/P/BxbP48MCm2viewHignKFwZ8HXgg7jPewewCOgbWz4vFue02PJBwD+AcbHlY2LXGhx3P6el+zukqWMnlcwlGeXAD9z9PQB3fwqoB06MzfcBimLbPnT3yF6ce5a7r4wd+wegATg6tu1a4Lvu/mFs+5+A/07wM7zk7s/FzlMJrAYaS9+tfb62PA6caGaNMV8T+0yfNdvvPOBTd/+5u+/0wK+BD4GzzcyA64BvuvuWWBxzgefizvFvwEJ3nxfb/i4wG7iw3XdBujwlc0nGQOBHcS1cNgL9gC+4+wZgPPADM3vLzL66l+f+qNny50Cv2L8HAev2sD0RHzZb3hJ3nRY/X1sndfc64FfAN2NVOlOAe/aw654+C0AVcBhB1VAYeKfZ9vjPOxA4r1mcF8RilW4iN90BSJf2EcGf77P3tDFW4n3OzIYT1FVXu/szKbjuZwSJbkPcukEEVSap1Orna4f7gbUEybqysYTfzAcEpfPmjgRmEnxWBwqBv8ZtHwi8FRfno+6+20NX6T5UMpdkPALcYmaHAZhZvpmdH/v3gLiHeCsJklbv2PJnwBFmlmNmiRQoZhKUmPePXet8YGw7jvsMGLQX13yEFj5fe7j7J8AfgZ8Bd7ew29PAYWZ2vZmFYg93JwN9CJ4rbCdoHXO3mfWMxXE18MW4c/wOuMTMRsa2m5mdaWZ92hurdH1K5pKMR2PT87E/7VcCjQm8D/C0mX0IrAFeBBrbllcQlCY3EtR/762bCeq23zKz94Ezgd8CbdXJ/4ggsa4xs33acZ3WPl973QOscfele9ro7tuA02PTRuBdggevZ7n7jthuVxPU1W8ws/cI/iqZHXeO94CvAr80s78C7xE8hG3Yy1ilCzN3DU4hXZ+ZzQWejT08FOl2VDKXLsfMRlisO4FY1cQVwEkE1S8i3ZIegEpXVAg8bGa9CaoS3gJOd3e9Ji/dlqpZRESygKpZRESyQKdVsxxwwAE+YMCAzrqciEhWWL58+SfufmBb+3VaMh8wYACVlZWddTkRkaxgZn9pz36qZhERyQJK5iIiWUDJXEQkC6S1nXl9fT2bNm2irq4unWF0aT169KCwsJC8vLx0hyIiaZTWZL5p0yZ69+7NgAEDCLptlr3h7nz66ads2rSJI488Mt3hiGScqir4+c/hmWe2ct99V3DddY9w/vn7Ul4OgwZ13LHpkNZqlrq6Ovr166dEniAzo1+/fvrLRmQPKipg6FB46MEoNxz9A8a/NYfrj7qNhx6MMnRosL2tY2fMgNLSZxg/fg4lJXOZMYM2j02XtNeZtzeRV1XBlCnQpw+EQsF8ypRgfXemH0KR3VVVwYQJUFcTZV7DWL657BdwK1z/+r3MaxhLXU2UCRP2nD8aj62pgWh9lG8NvhNuh28NvpNofZSaGlo8Np3SnszbI/5Xcts2cA/mmfwrKSLps2PHGCIRIzo3lzEFL5BbuxMccmt3MqbgBaJzc4lEjPr65mNu//NYbzCiZ+Ry3I/fglth6B1riJ6RizdYi8emU8Yn8/hfyfr6XbfV15PUr+SHH37Ivffe2+79t2/fzp///Oe9v5CIdKqbbrqZSCQcDPvdvJf7CLASIpEw5eW3tHxsBbAMLAJ4bL4MqGj52HTK+GT+85/vnsSbq6+He/Y0umIb+vfvz/XXX9/u/f/2t7/tVfIXkfSoqChl3Lh51A3O/+eIro16Qd2x+Zx77rMsWFDS4rE7luXt8Ydgx+t5LR6bThmfzB9/vH3J/LHH9v7cGzdu5OKLL+aKK67g1ltv5cwzz6SoqIh33gnGzr3mmmsYNWoUp5xyCtu3b+fiiy9m8eLFnHXWWQDccccdnH766ZxwwgnMnTsXoMVzrVixgjFjxlBSUsJNN90EwAMPPMCpp57KqFGjqFBdkUjKFBTAkiWlXPTwTKLFBgWAAQUQLTYuengmS5eWUFDQ8rF3LJiGh3fd5mG4o2Jai8emU8Yn8+rq1O7XkkMPPZTnnnuO733vezzwwAN8/vnnrF27lldffZXXXnuN/Px8nnjiCU4//XQWLVoEwFVXXcXixYuZPXs2v/zlL1s8F8DXvvY1HnnkEZYsWcKdd97JO++8w6JFi3jppZd48cUX+elPf5rcBxCRJpMmQV4e9NlvGzWzw0QfN3b8Ry7Rx42a2WH67LeNvDy47LKWj11/9FFEi3PwXuAG3guixTmsP/qoFo9Np4xP5u399Uv2V3L06NEADB48mM8++4z99tuP8vJyrr32Wn73u9/ttv/OnTu59957mTp1Kg8++CDbtm1r8VyffPIJBx98MIWFhQCEQiFWrVrFqlWrKC0t5ayzzuIf//gHDQ0aslEkFcrLg4R85ZW/Idy7llWHFXHua/NZdVgR4d61TJ78EHl5cMMNLR87+d8fxhbtpOqHA3nkyMup+uFAbNFO/u2qR1o8Np0yPpk3/kq2JhW/ko1N/MwMd6e+vp5zzjmH6dOnM2/ePNasWUNOTg7bt28HgmqTTz75hJ/+9KeMHz++1XPtv//+bNiwgU8//RQI3nw95phjOO2001iyZAlLlixh+fLl5OZq4CeRVBg0CGbNgurqfZk69S6Kiyt5/vkzGT78DaZNu5NIpA+zZu355Z9djv3uzzjmxveY/P4j/Ev5u0y7+a5Wj02njE/mjb+SremIX8lPP/2UUaNGcfrppxONRjn66KM59NBD+eSTTxg7dizHHnss69ato7S0lOeff77Vc4VCIe655x7GjRtHaWkpd9xxB0VFRRx++OGccsopjB07lt/85jep/QAi3VxZGQwZMoe6uhvp3TtEKAQFBTnU1pYzZMgcyso65ti0cfdOmU488URvbu3atbut25P5893DYfe8PPeglXkw5eUF6+fPb9dpslZ776NIV7R+vfs117j377/F//jHC71//y1+zTXB+sy3xd0vjM0TA1R6O3JsxpfMIfiVXL0arr561zdAr746WJ+Rv5IikrRkXsnPDM8Ac4C5HX6lThvQubi42JuPNPT2228zePDgTrl+NtN9lGxUVRUk8rqaKAsZS0nPF8mt20lDjxBLaksZy0J6hHNYvTrz6q//qRRYEpsvTugMZrbc3Yvb2q9LlMxFpPtJ5pX89BlD0KDdIGow7xW4HZj3crDcuI3Ux5xwMjezfczsHjNbaWavmFn7X6UUEWlDMq/kp8/NQBiiwFjgkga4NTYfS7CeMJD6mJMpmd8B1Lh7kbt/CfhFimJqw1ZgfGwuItkqmVfy06cUmAcV+UE/LtWAx+bLCNbzLFCS8isnlMzNbH+gDPhe4zrfQ+W7mV1tZpVmVrl58+bEo9xF5z1QEJH0SeaV/PQqhRUTWvhr4it0RCKHxEvmQ4F3gOlmttTM5pjZMc13cvcH3L3Y3YsPPPDApAL9p4eazZOzZMmSlJynOfWwKJKcZF7JT7thX9jjXxMUHdRhl0w0mR8MFAM/d/fTgF8BT6Usql3EPVDAgNdi619ttj6xBwrTpk1LPsQ9UA+LIslJ5pX8tCtbDiOAghCYBfMRQNmbHXbJRN8f/zuw3N3XA7j7QjN71Mzy3L2NPg731s3An4Ca2PKOZnNI9IHCddddx9q1aykpKeGiiy7iD3/4A9XV1YwdO5bbbruNRx55hOXLl/PWW2/xwx/+kB07dvDd736XfffdlyOOOIL99tuPn/zkJ/zpT3/iO9/5Djt37uSss87iO9/5DhdffDHvv/8+Z511VlPHXCLSfs1fyb/77utxDzF8+OnceOO9jB79cka+Vg9ATl9YeBdUHAMrV0PRcVD2LuS82nHXbM+bRc0noAfwP0BhbPlk4PXWjknmDVD3xe4ebuHUYXd/sZ3n2d2IESPc3f3jjz92d/eGhgYfMmSIR6NRf/jhh/2SSy5p2veEE07wTz75xN3d77zzTp86darv3LnTR44c6Vu3bnV394kTJ/rGjRt9w4YNPnHixITj2ht6A1Sy2fr17t/4hnufPu6hUDD/xje6yhugyaOdb4AmVDJ39zozuwr4vZlFgTrg0pT8uuxRKTAT+ErsUo16xNaXJH2FZ599ljVr1rDPPvtQU1PDjh1ByX/kyJEAfPzxxxQWFtKvXz8ATjzxRBYtWsTmzZt59913Of/88wHYsmULmzZton///knHJCJByXv69GCSliXcTZ+7/wk4NYWxtGELQbghIB/YHlvektRZG7udve+++1i+fDnV1dX89re/bdre2JPh/vvvz/r166murqagoIDFi4O3uQ444ACOPfZYFi1a1PRDEA6H+eCDD5p6WBQR6Whd6A3Q3xDUmx8PPB2b15Bsq5bRo0dz0kkncdBBB1FcXEx5eTmHH374bvvl5uZyyy23MGrUKMaOHUt1dTU9evQgFArx7W9/m9GjR3PmmWc2DUMX38OiiEhH60J9s1wIjAauJ/gNigL3Ai8TtDvvePX19eTF+uP95je/SWlpKRdeeGGnXLs16ptFJHu1t2+WLjQaQvOEnQOUx6bOMXXqVFasWEF9fT0jRozgggsu6LRri4i0pgsl8/S7++670x2CiMgedaE6cxERaYmSuYhIFlAyFxHJAkrmItKhqqpgyhQoLNzK7NnjKSzcypQpwXpJna6VzKNRmDcPbr89mEej6Y6oyRVXXMG6dev48MMPW+1ga9WqVWzZErzotHDhQp577rnOClGk03X9MTy7jq7TmiUahbFjYdkyiESgVy8YMQIWLoScnA67rLtjZu3ev3///k0vDu3JPffcw7Rp0+jbt69eKJKsVlUFEybEjeG57EVYCtf3uJeihlWMbVjIhAmZPoZn19F1knlFRZDIq6uD5erqYLmiAsaNS/i0GzduZMqUKey777588MEHHHLIIfzoRz/iW9/6Frm5uYwePZqJEyfyta99jW3btnHggQfy2GOPsc8++/Cd73yHl156iX79+lFTU9N0vmnTpvHEE0+wYcMGrrvuOqqrqyksLKS0tJQFCxawdu1arr/+enbs2EFdXR1f//rXeeaZZ7jrrrsIhUL07duXhx56iH79+nHyySczZswYli5dSn5+Ps8++yz5+fmpuKMiHSoYw/MFmAdcQjDaDnFjeP4+F8bBunVnAM+nMdIs0Z7euFIxJddrorvfdpu7mTv8czJzv/329p9jDzZs2OADBgxo6vXw1ltv9e9///s+cOBAr62tdXf3SZMm+Ztvvunu7v/5n//pjz76qD/33HN+1VVXubv79u3bvaioyN9+++1deks8++yzfcWKFe7uHo1G3d398ssv97ffftvd3R9++GH/1a9+5Z9//rmffPLJHolE3N39ySef9BtvvNHd3QcOHNh0n6677jp/5plndvsM6jVRMtE55yz26uqw+224G83+7+J+O15dHfZzznkx3aFmNNrZa2LXqTMfNiyoWonXqxcUFSV96pNOOok+ffoAMGLECEKhEMOGDaNHjx4ArF69mhtuuIGSkhIeffRRPv74Y958803OOeccAPbZZx+GDh2623m3bNlCUSy+UKjlW/3ee+8xfPhwwuEwAGPGjGHdunVA0JFX46v6gwcP5rPPPkv684p0hq45hmfX1XWSeVlZUEdeUBAbuaMgWC4rS/rUa9asoa4u6Fr32WefpW/fvk29JQIcffTRPPLIIyxZsoTXXnuNa6+9liOOOIJXXnkFgNraWpYtW7bbeUOhEOvXrweCfl0AcnJydutNceDAgbz++uvU1tYCsHjxYoYNGwawS329mTX2Jy+S8bruGJ5dU9epM8/JCR52VlTAypVBibysLCUPPw855BAmTZrE3//+d4qLiznvvPN49dV/jgjy4x//mMmTJwOw7777cv/99/PlL3+Zp59+mlNOOYX+/fvvsaOr6dOnM3nyZEKhEEOGDOH++++nrKyMiy++mNtvv71pv379+lFeXk5paSm9evWif//+3H///Ul/LpF0mjQJZsyIjeH532HCS2uIvplDzglRak4L0+faDB7DsytqT11MKqak68w7SGeOCNRRMuE+ijS3fr17OOy+eHGJNzSEfPnyYT5mzCJfvnyYNzSE/IUXSj0c7j4jBiWKrKszF5EupfkYnsXFlTz//JkMH/4G06bdSSTSJ3PH8OyCuk41SwcZMGAATzzxRLrDEMlKZWVQVTWHhQuhd++gRXFBQQ61teUMGVKuRJ5CaU/mvpcv5ciuXA9EJcNpDM/OkdZqlh49evDpp58qISXI3fn000+bmlCKSPeV1pJ5YWEhmzZtYvPmzekMo0vr0aMHhYWF6Q5DRNIsrck8Ly+PI488Mp0hiIhkBbVmERHJAkrmIiJZQMlcRCQLKJmLiGQBJXMRkSygZC4ikgWUzEVEskBSydzMTjMzN7O+qQpIRET2XsLJ3Mx6At8DPk1dOCIikohkSub3AT+jaZhWEclWVVUwZQoUFm5l9uzxFBZuZcqUYL1khoSSuZndDHzk7hVt7He1mVWaWaX6XxHpmioqYOhQeOjBKDcc/QPGvzWH64+6jYcejDJ0aLBd0s/2tsdCM7scOBv4qru7mW0Eitx9S2vHFRcXe2VlZcKBikjnq6oKEnldTZSFjKWk54vk1u2koUeIJbWljGUhPcI5rF6tQSY6ipktd/fitvZLpGQ+DTgOWGFmK4FDgZfNbEgC5xKRDLZjxxgiESM6N5cxBS+QW7sTHHJrdzKm4AWic3OJRIz6+jHpDrXb2+tk7u6D3f2L7l7k7kXAR8Cp7r429eGJSDrddNPNRCJhWAFEmm2MACshEglTXn5LGqKTeGpnLiItqqgoZdy4edQNzodezTb2grpj8zn33GdZsKAkHeFJnKSTubsPaKu+XLJXVRV84+tRLu83k7WXDObyfjP5xtejauWQJQoKYMmSUi56eCbRYoMCwIACiBYbFz08k6VLSygoSHekopK5JKyiAoqOi/LlB8byX/97OUNmruPX/3s5X35gLEXHRdXKIQtMmgR5edBnv23UzA4TfdzY8R+5RB83amaH6bPfNvLy4LLL0h2pKJlLQqqqYMIEOK22guG+jB4N28GhZ8N2hvsyTqutYMKEttshq2Sf2crLg2R+5ZW/Idy7llWHFXHua/NZdVgR4d61TJ78EHl5cMMN6Y5UlMwlIY2tHObddh69bdf3xnpbNfNuP6/NVg4q2We+QYNg1iyort6XqVPvori4kuefP5Phw99g2rQ7iUT6MGuWmiVmAiVzSUhTK4dh7PHBGEWtt3JIVcleOl5ZGQwZMoe6uhvp3TtEKAQFBTnU1pYzZMgcysrSHaGAkrkkqLGVQ2R0TxjBLg/GGAGR0T1bbeWQipK9dJ5Bg2D6dNi6FaLRYD59ukrkmUTJXBLS2Mph4lefpHZOPvweuA34PdTOyWfiV59stZVDsiV7EdmVkrkkpLGVQ9++W2jwPBrODlFzY08azg7R4Hn07bul1VYOyZbsRWRXSuaSUI94u7RyCNewevXxXHDB06xefTzhcE2brRySLdmLyK6UzLu5xh7xZsyA0tJnGD9+DiUlc5kxg1Z7xEu2lUOyJXsR2ZWSeTfW2KKkpgbq62Hy5IeAYF5fH6xvrUVJMq0cki3Zi8iu9roL3ESpC9zM8/bbYxg8+IVgIQr1z+SS91YD9V/MJe/8BsgJNq1bdwbHHvt8yq9fUQENDReydOlo7r77etxDhEJRbrzxXkaPfpncXDV7E+nILnAlSzS1KIkCYyHv/zbArbH5WCDasS1Kkm2/rNFvROK4e6dMJ554oktmMXMvKVnstU/luxfgTtxUgNc+le+nnfaih0LpjnR38+e7h8PueXnukyY96u74pZc+5nl5wfr589MdoUhqAJXejhyrknk31tiiZNbNE/BmfVV7BJ66+SsZ2aIk2bp+kWykZN6NNbYo+fshX4Bws41h+MchB2Vki5LGt0fdDW8wvrTlFbgdTt36Mt4QrNfbo7tTtVR2UzLvxhpblJx4y3IYAdGeIXZiRHuGYASccPObGdmiJN11/V1Rok1QpetQa5ZurrFFyUsvfol19xzD8axmtR3HsTe8y6klr2Zki5JQCE477UUqvlFGj3/bDvFduxRA3cP5nD19AS+/XEI0mrYwM0bjoMw1NRAiyoofFTE0+harQsdxwi0r2EkO4TAalDlDtbc1S25nBCOZq6wMqqrmsHAhvNQH5lefT0EBHL4dhgz5Vkb+526q6/9oApdG/huL29ZU1/9uCX36pC3EjBJUS73Q9JeM/xiogaHhNUTPyIWFQE7QBBVS3wRVOoeqWaTL9YjXVev606WpWqoCWAYWATw2XwZUqFoqGyiZS5fTVev606WxU7Mdy/KgWaslIrDj9Tx1apYFVM0iXU5TvzC1ffn28XexbnGsrr/uOI4tepdT617V6DdxGqul7qiexvfCtwcl8hgPwx0V01haqWqprk7JXLqkrljXny6TJgWtWNYffRTRXjnkVEahBghDtDiH9YceRd4qVUt1dWrNIpLlGluzzJtXyuhRS9lw/5G8fN+pnHrdyxw5ZQNLXynhvPMWqzVLhlLfLCKt6E4v0OzSXfF3f8YxN77H5Pcf4V/K32XazXdpUOYsoWQu3U53fIFGgzJnP1WzSLcS/wINwOLFpZSWLmHx4lLOOGMxgF6gkYyiapZupjtVGyRjl35d3Bg58jUARo16tWmd+nWRrkjJPAt0x2qDRDW9QAMQhfzndsDtsXns1X+9QCNdkZJ5F6fuYPdO4ws0kf/tGXTKdQlwa2w+FiL/21Mv0EiXpGTexak72L3T+ALNz864Cf8zQSddHsz9z/CzM27KyD7cRdqScDI3sx+Y2Qtm9pqZ/cHM9k1lYNI+6g527zT26/J/dqwNXpyJVwNDdqzN6H5d9GxEWpJMyXydu5/h7iOBtcB3UxST7IXGaoO62flBp0lxJU2WQd3sfFUbxGns1+WYie9Ar2Ybw3DMRe9mbL8ujc9GHnowyg1H/4Dxb83h+qNu46EHo3o2Ioknc3f/fdziG8AhyYcje6urDv2WLo0v0PxlyEDe2/9otlFAFGMbvVjf7yj++n+OzMgXaBqfjdTVRJnXMJZvLvsF3ArXv34v8xrGUlcT1bORbi7pOnMzywP+H/DkHrZdbWaVZla5efPmZC8le6DuYPdeWRkMOe5p7jv3ba7s+Xu+b7dxZc8n+OW56xhy3NNtvkCTjqqOxmcj0bm5jCl4gdzaneCQW7uTMQUvEJ2bq2cj3V17Rn1uaQKOApYAX2tr3xNPPLHDRq/uztavD0ajX/zcab7zdLyhZ8ijmDf0DPnO0/EXFpV4OBzsJ8mbPz+433l57pMmPeru+KWXPuZ5ecH6+fM75rrnnLPYq6vD7rfFmsMTNxnut+PV1WE/55wXOyYASRug0tuRj5N5AHo+8FvgG+7+X6n4YZG917w72AtrZ/M9bmN83R+ZWnQnkbp9M7LaoCtKZzPQpmcjg/N3r+vvBXXH6tlId5dQMjezLwB3Aee4+/+kNiTZW439btTuuImX+pzPHaFbWNr7Amq2f0v9bqRQqpqBJlJN0/hs5KKHZxItNigADCiAaLFx0cMz9Wykm0u0ZD4MOAB42syWxKY/pjAu2Utdbei3rigVzUATfVu38dlIn/22UTM7TPRxY8d/5BJ93KiZHabPftv0bKS7a09dTCom1ZlLV2fmXlKy2GufyncvaFZvXYDXPpXvp532oodCez6+8flG4yGLF5e4O/7CC6VN61p6vtH0bGRxiTc0hHz58mE+ZswiX758mDc0hPyFF0r1bCRL0dF15iLdTbLNQJOpptmlT/Kpd1FcXMnzz5/J8OFvMG3aneqTXJTMRdor2WagyVbTqE9yaY2SuUg7Nb49euIty2EERHuG2IkR7RmCEXDCzW+2+vZoKt7W1bMRaYkGdBZpp+bNQNctPobjWc3quuM4tuhdTq17tdWqjqZqmo8mcGnkv7G4bU3VNO+W0KdPp3wcyTJK5iJ7oawMqqrmsHAhvNQH5lefT0EBHL4dhgz5Vqsl5EmTglYsfz/kC/AhEF/v3lhNs0EtUiQxqmYR2UuJVnUkW00j0hqVzEU6SbLVNCKtUTIX6UTJVNOItMaCNukdr7i42CsrKzvlWiIi2cLMlrt7cVv7qc5cRCQLKJmLiGQBJXMRkSygZC4ikgWUzDOIRl4XkUQpmWeIRPu5FhEBJfOMkM7hyEQkOyhJFIGzAAAN4klEQVSZZ4Bd+rl2Y+TI1wAYNerVpnUaeV1EWqNkngGa+rmOyc/fscsc2h6OTES6NyXzDNDYz3XTwAXzgNtj89iABRp5XURao75ZMkBjP9eXXPQ7ZteMJ6fSg+5RewUjr18S/h1Ll6qfaxFpmUrmGaBxOLLRkZcIveG7jEATesMZHXlJI6+LSKuUzDNAYz/X/zrwj1DTbGMN/OvA2ernWkRapWSeARr7ud5c+AV25O6zy7YdufuwufAg9XMtIq1SMs8QZWVwwKRX2XDQqVRTQBSjmgI2HHQqB0x6VSOvi0ir9AA0gww6Jgf+sjB43XPlSgqKiji2rAxyctIdmohkOCXzTJOTA+PGBZOISDupmkVEJAsomYuIZAElcxGRLKBkLiKSBRJO5mZWamavm9lrZjbHzPZPZWAiItJ+CSVzM+sB/BqY4O4jgaXAj1MZmIiItF+iJfOxwKvu/tfY8gzg/OY7mdnVZlZpZpWbN29ONEYREWlDosl8APB+44K7bwNyzSwvfid3f8Ddi929+MADD0w8ShERaVWiyTwfaGi2roGgrz8REelkiSbzTcDhjQtm1guoc/fmCV5ERDpBosm8AjjbzL4QW74a+F1qQhIRkb2VUN8s7v65mV0HzDWznQT15/+e0shERKTdEu5oy92fBZ5NYSwiIpIgvQEqIpIFlMxFRLKAknmKVVXBlClQWLiV2bPHU1i4lSlTgvUiIh1FyTyFKipg6FCYMQNKS59h/Pg5lJTMZcaMYH1FRbojFJFspWSeIlVVMGEC1NRAfT1MnvwQEMzr64P1EyaohC4iHUPJPEV27BhDJGK4B9PIka8BMGrUq03rIhGjvn5MmiMVkWykZJ4iN910M5FIuGk5P3/HLnOASCRMefktnR6biGQ/JfMUqagoZdy4ebsk9HiRSJhzz32WBQtKOjcwEekWlMxTpKAAliwpZeLEmdTW9thlW21tDyZOnMnSpSUUFKQpQBHJakrmKTJpEuTlQd++W2hoyKWhIURNTU8aGkI0NOTSt+8W8vLgssvSHamIZCMl8xQpLw+S+ZVX/oZwuIbVq4/nggueZvXq4wmHa5g8+SHy8uCGG9IdqYhkIyXzFBk0CGbNgurqfZk69S6Kiyt5/vkzGT78DaZNu5NIpA+zZgX7iYikWsIdbcnuysqgqmoOCxdC795QXQ0FBTnU1pYzZEi5ErmIdBgl8xQbNAim/yLK9LMrYMUKGDYsyPI5OekOTUSymJJ5qkWjMHYsLFsGkQj06gUjRsDChUroItJhVGeeahUVQSKvrgb3YL5smTpmEZEOpWSeaitWBCXyeJEIrFyZnnhEpFtQMk+1YcOCqpV4vXpBUVF64hGRbkHJPNXKyoI68oICMAvmI0YE60VEOogegKZaTk7wsLOiIqhaKSpSaxYR6XBK5h0hJwfGjQsmEZFOoGoWEZEsoGQuIpIFlMxFRLKAkrmISBZQMhcRyQJK5iIiWUDJXEQkCyiZN1NVBVOmQGHhVmbPHk9h4VamTAnWi4hkqr1O5maWb2b3mdkiM3vDzGaYWV5HBNfZKipg6FCYMQNKS59h/Pg5lJTMZcaMYL06PhSRTJVIybwXMNfdz3L34cA+wFWpDavzVVXBhAlQUwP19TB58kNAMK+vD9ZPmKASuohkpr1O5u7+mbsviltVCRySupDSY8eOMUQihnswjRz5GgCjRr3atC4SMerrx6Q5UhGR3SVVZ25mfYB/A2a3sP1qM6s0s8rNmzcnc6kOd9NNNxOJhJuW8/N37DIHiETClJff0umxiYi0pc1kbmYHm9mSuOnw2PpiYBHwfXdfsadj3f0Bdy929+IDDzwwtZGnWEVFKePGzdsloceLRMKce+6zLFhQ0rmBiYi0Q5u9Jrr734GS+HVm9jXgX4GvuPsHHRNa5yoogCVLSpk4cSZPPfUVevasa9pWW9uDiRNnsnRpCX36pDFIEZEWJNKaZSjBA89x2ZLIASZNgrw86Nt3Cw0NuTQ0hKip6UlDQ4iGhlz69t1CXh5cdlm6IxUR2V0ideYjgUOB5+KqXu5PcVydrrw8SOZXXvkbwuEaVq8+ngsueJrVq48nHK5h8uSHyMuDG25Id6QiIrszd++UCxUXF3tlZWWnXCtRFRXQ0HAhS5eO5u67r8c9RCgU5cYb72X06JfJzZ2j0d9EpFOZ2XJ3L25rP400FKesDKqq5rBwIfTuDdXVUFCQQ21tOUOGlDNoULojFBHZMyXzZgYNgum/iDL97ApYsQKGDdMYniKS8ZTMm4tGYexYWLYMIhHo1QtGjAgGaVZCF5EMpY62mquoCBJ5dTW4B/Nly9Qxi4hkNCXz5lasCErk8SIRWLkyPfGIiLSDknlzw4YFVSvxevWCoqL0xCMi0g5K5s2VlQV15AUFYBbMR4xAbRJFJJPpAWhzOTnBw86KiqBqpahIrVlEJONlZck86dGCcnJg3Di45ZZgrkQuIhku65K5RgsSke4oq5K5RgsSke4qq5K5RgsSke4qq5K5RgsSke4qq5K5RgsSke4qq5J5/GhBtbU9dtkWP1pQQUGaAhQR6SBZlcw1WpCIdFcZm8wb24r36QOhUDBvq624RgsSke4qI0caqqgImhDW1wdTo7y8YJo1q+W36zVakIhkk/aONJRxJfPmbcXjtaeteFkZDBkyh7q6G+ndO0QoFD9akBK5iGSnjEvmP//57km8ufp6uOeelrcPGgTTp8PWrcFYE1u3Bssa9k1EslXGJfPHH29fMn/ssc6JR0SkK8i4XhOrq1O0XzQaVKBrHE8R6QYyLpkXFMC2be3br0Uax1NEupmMq2ZpbCvemjbbimscTxHpZjIumTe2FW9Nm23FNY6niHQzGZfMBw0K2pGHw7sn9by8YP2sWW20TNE4niLSzWRcMofgWeXq1XD11bu+AXr11cH6NtuKaxxPEelmMvIN0JRobM2icTxFpAtr7xugGdeaJWUax/EcNy7dkYiIdLiMrGYREZG9k1QyN7PLzWxLqoIREZHEJJzMzewg4P+mMBYREUlQQsnczHKAGcCNqQ1HREQSkWjJ/FfAHHdf1dpOZna1mVWaWeXmzZsTvJSIiLSlzaaJZnYw8ETcqheBsLtPjW3f4u5927yQ2WbgL0nE2lUcAHyS7iAynO5R63R/2tad7tER7n5gWzvtVTtzMysA3gM+BhoPPA5YA5S4e7d/GGpmle1pE9qd6R61TvenbbpHu9urdubuXg0cEr8uVjLXe/IiImmkduYiIlkg6WTenvrybuaBdAfQBegetU73p226R810Wt8sIiLScVTNIiKSBZTMRUSygJK5iEgWUDLfC2Y2ysxeM7Oi2PLhZvYHM3vOzFaY2bUtHDffzF43syWx6UudG3nn2cM9GmhmC8xssZktN7MrWjjuIjN7M3bsI2bWo1MD7yRJ3J9u+x1qtu1hM5vTwnHd4jvUkuztzzzFzOxRoDfQJ271F4Cp7r7ezMLAcjNb7O5rmx2+P/Bld/+gk8JNixbu0T+A89y93sz2BTaY2ePu3hB33CHA94GR7r7FzH4BXA/8pPOi73iJ3p+Y7vwdatxWCpwIvL+Hbd3iO9Qalczb7+vuPp64V4jd/Q13Xx/7dw3wNnDwHo7dj+7x6vGe7lHE3etji0cDb+8hUV0EPBH3BvEDwIUdHm3nS/T+QDf+DkFTtyLfB77bwnHd5TvUIpXM2ymWrFtkZl8EhgB/amGXBWYWAuYBd7n7zhSHmHYt3SMzuxM4n+D7NmEPuwwAlsct/wXon+r40i2J+9OoW36HYt2IzASuI/gLZU8G0A2+Q61RyTwFzOxSgi6BL3T32ubb3f1f3P00YBxwMjClk0NMK3f/trsfC4wHnjSzw5rtkg/El0ajQNYlqpa04/502++QmeUBTwI/c/fVrezarb9DoGSeFDMLmdkjwAjgdHdf19r+7r4VeAg4qRPCyzjuvgZ4md0//ybg8LjlI4CNnRRWxmjl/sTv092+Q6UEnfndbmYrCQpNpWY2q9l+3f47pGSenClAtbv/v1b+hM43s96xf+cCE2m5KibrmNng2GAmmNn+BEmostlufwAuM7NeseVrgMc6L8r0ac/96c7fIXdf5O6HuXtRrEO/q4AX3b15dVS3/Q41Up15cr4EHGdmS+LW/Y6g9PAAwUhM+cBCM4sAecAC4NedHGc6jQJ+Hxsrdidwo7v/JVb3+0Bs+R0zuxdYYmY7gTeAh9MXcqdq8/6g79Ae6Tu0K/XNIiKSBVTNIiKSBZTMRUSygJK5iEgWUDIXEckCSuYiIllAyVxEJAsomUtWMLOnzGxjbGows7/FLU+K9X8ikrXUzlyyjpltBCa5+yvpjkWks6hkLiKSBZTMJeuZ2ffNbEbs3wNi1TBfMbNVZva5md1pZoVmttDMNpnZGjM7Oe74HmZ2t5m9a2YbzOzXscFIRDKGkrl0RznAcKAI+CJwBfAc8G13LwQeB+6P2/+/gIMIeu87BugH3NqJ8Yq0SclcuqufeuBDYBHwsruvim17kqADtZCZHUDQS+E17r49NirQnbQ+iIRIp1OvidIdRd3907jlauDvccvbCP5v7AMcSVCSX2NmjdtDBD0ZimQMJXOR1n0E1AEDs3GYNskeqmYRaUWsGuZPwB2xIcwws8PiH5CKZAIlc5G2XQoUAlVmtp5gABL935GMopeGRESygEoXIiJZQMlcRCQLKJmLiGQBJXMRkSygZC4ikgWUzEVEsoCSuYhIFlAyFxHJAv8fyk7Gd2TMZlIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.title(\"Testing the Model\", fontsize=14)\n",
    "plt.plot(t_instance[:-1], time_series(t_instance[:-1]), \"bo\", markersize=10, label=\"instance\")\n",
    "plt.plot(t_instance[1:], time_series(t_instance[1:]), \"w*\", markersize=10, label=\"target\", color='yellow')\n",
    "plt.plot(t_instance[1:], y_pred[0,:,0], \"r.\", markersize=10, label=\"prediction\")\n",
    "plt.legend(loc=\"upper left\")\n",
    "plt.xlabel(\"Time\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.4.3 OutputProjectionWrapper를 사용한 RNN 모델링\n",
    "\n",
    "1.4.2에서는 RNN의 출력결과를 우리가 원하는 형태로 맞춰주기 위해 *stack → FC layer → unstack*과정이 필요했다.  텐서플로에서는 [`tf.contrib.rnn.OutputProjectionWrapper`](https://www.tensorflow.org/api_docs/python/tf/contrib/rnn/OutputProjectionWrapper) 를 이용해 Cell을 감싸서 아래의 그림과 같은 역할을 해준다.  `OutputProjectionWrapper`의 FC Layer는 출력 시퀀스(outputs)에만 적용되고 state($h_t$)에는 적용되지 않는다.\n",
    "\n",
    "![](./images/time-series04.PNG)\n",
    "\n",
    "\n",
    "\n",
    "아래의 코드는 `OutputProjectionWrapper`를 이용해 1.4.2의 동일한 예제를 작성한 코드이다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "################\n",
    "# Layer Params #\n",
    "################\n",
    "n_steps = 20\n",
    "n_neurons = 100\n",
    "n_inputs = 1\n",
    "n_outputs = 1\n",
    "\n",
    "X = tf.placeholder(tf.float32, [None, n_steps, n_inputs])\n",
    "y = tf.placeholder(tf.float32, [None, n_steps, n_outputs])\n",
    "\n",
    "# RNN Model using OutputProjectionWrapper\n",
    "cell = tf.contrib.rnn.OutputProjectionWrapper(\n",
    "    tf.nn.rnn_cell.BasicRNNCell(num_units=n_neurons, activation=tf.nn.relu),\n",
    "    output_size=n_outputs)\n",
    "predictions, states = tf.nn.dynamic_rnn(cell, X, dtype=tf.float32)\n",
    "\n",
    "################\n",
    "# Train Params #\n",
    "################\n",
    "learning_rate = 0.001\n",
    "n_iterations = 600\n",
    "batch_size = 50\n",
    "\n",
    "# loss\n",
    "mse = tf.losses.mean_squared_error(labels=y, predictions=predictions)\n",
    "# optimizer\n",
    "train_op = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(mse)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step: 000, MSE: 11.9673\n",
      "step: 100, MSE: 0.5258\n",
      "step: 200, MSE: 0.1496\n",
      "step: 300, MSE: 0.0728\n",
      "step: 400, MSE: 0.0616\n",
      "step: 500, MSE: 0.0594\n",
      "y_pred:(1, 20, 1)\n",
      "[[[-3.5285556 ]\n",
      "  [-2.4768994 ]\n",
      "  [-1.1221677 ]\n",
      "  [ 0.59313875]\n",
      "  [ 2.119323  ]\n",
      "  [ 3.1693287 ]\n",
      "  [ 3.457877  ]\n",
      "  [ 3.2451653 ]\n",
      "  [ 2.8801768 ]\n",
      "  [ 2.2912362 ]\n",
      "  [ 1.7846028 ]\n",
      "  [ 1.4566743 ]\n",
      "  [ 1.8437719 ]\n",
      "  [ 2.712548  ]\n",
      "  [ 3.904094  ]\n",
      "  [ 5.13178   ]\n",
      "  [ 6.1285706 ]\n",
      "  [ 6.707691  ]\n",
      "  [ 6.6761217 ]\n",
      "  [ 6.0609703 ]]]\n"
     ]
    }
   ],
   "source": [
    "# Train\n",
    "with tf.Session() as sess:\n",
    "    tf.global_variables_initializer().run()\n",
    "    for iteration in range(n_iterations):\n",
    "        batch_x, batch_y = next_batch(batch_size, n_steps)\n",
    "        sess.run(train_op, feed_dict={X: batch_x, y: batch_y})\n",
    "        if iteration % 100 == 0:\n",
    "            loss = mse.eval(feed_dict={X: batch_x, y: batch_y})\n",
    "            print('step: {:03d}, MSE: {:.4f}'.format(iteration, loss))\n",
    "            \n",
    "    # 새로운 데이터 예측하기\n",
    "    X_new = time_series(np.array(t_instance[:-1].reshape(-1, n_steps, n_inputs)))\n",
    "    y_pred = sess.run(predictions, feed_dict={X: X_new})\n",
    "    \n",
    "print('y_pred:{}\\n{}'.format(y_pred.shape, y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXMAAAEbCAYAAAAoHWSlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xt8VOW97/HPb5IQGAJSUaslWoTqFloxaBCFFhLBYgRFKi1acePGHi+op2poYW/d1UpbT9VW9ynbuq1Vt3qqKC0ISEC8YOul1CAX3UgVClbsRcSCZBJCZvidP9YkJoHcJpeZzHzfr9d6raz7bxbDL0+e9aznMXdHRES6t1CyAxARkfZTMhcRSQNK5iIiaUDJXEQkDSiZi4ikASVzEZE0oGQuTTKzp8xse3yKmtlf6y3nd+B1ss1slZkN6qhztvK6PzCz+zr4nC+b2Ydm1rOJ7WeamZvZ9HZe5zEzm9vR+0r3pWQuTXL3r7v7QHcfCOwA6pbdfUei5zWzc83s8XrXibr72e7+pw4Iu7nrPmtmp3fmNeJygaaS9WzgH10Qg2QYJXNJhs8Bn0nCdYfSNd/5xcC3G680s8HASOCNLohBMoySubSLmU0xs/XxqpeXzezketsuM7ONZvZnM9tqZp8zs7uB/wOMjR8zJ17N4mZ2dPy4x+JVII+Z2Xvx/S6sd14zsxvMbLOZvW9my8zs501VmZjZ581sO3A0sCh+vtz45iwzuyce31/MrLS1n68ZK+LnHd9o/Q3AvUCD167N7DgzW2BmW+L3armZ/VO97dlmdlt8+/tm9gQQbnSOL5nZ8/E4N5nZ1FbEKenE3TVpanECtgNfbrRuPPAecEJ8+evAn4AewInAx8BR8W0DgN7xn78FrKh3nmyCBHd0fPkx4C9AQXz5QmAPkBdfvhpYBwyIL58J7Abua+Ez7ADOqLf8g/h5z44vFwL7gS+09PmaucbLwEXxz7is3vrDgQ/j8+eA6fH1vYAtQClB4cqAq4D3633e24FngX7x5fPicc6NLx8F/B2YFF8+MX6tIfXu59xkf4c0de6kkrm0RynwfXd/F8DdnwJqgNPi8x5AQXzbB+4eacO5F7r7+vixvwaiwAnxbdcC/+buH8S3vwb8vwQ/w2/dfVX8POXARqC29N3c52vJY8BpZlYb89Xxz/Rxo/3OA3a5+0/c/YAH7gM+AM4xMwOuA77t7rvjcSwFVtU7x78AK919WXz7O8Ai4IJW3wXp9pTMpT0GAT+s18JlO9Af+Ky7bwOmAN83s7fM7JttPPdfGi3/A+gd/3kwsPkQ2xPxQaPl3fWu0+Tna+mk7r4P+Dnw7XiVzizg7kPseqjPArAVOJagaigM/LHR9vqfdxBwXqM4J8djlQyRnewApFv7C8Gf74sOtTFe4l1lZiMI6qor3H1JB1z3Y4JEt63eusEEVSYdqdnP1wr3ApsIknV5bQm/kfcJSueNHQ8sIPisDuQDf663fRDwVr04H3H3gx66SuZQyVza42HgZjM7FsDMcs3s/PjPA+s9xFtPkLT6xJc/Bj5vZllmlkiBYgFBifnw+LXOBya04riPgcFtuObDNPH5WsPdPwJ+A9wF/LSJ3Z4GjjWz680sFH+4OxPoS/BcoZqgdcxPzaxXPI4rgC/VO8evgIvNbFR8u5nZ2WbWt7WxSvenZC7t8Uh8ei7+p/16oDaB9wWeNrMPgDeBF4HatuVlBKXJ7QT13211E0Hd9ltm9ifgbOC/gZbq5H9IkFjfNLMerbhOc5+vte4G3nT3lw610d33AmfFp+3AOwQPXr/q7vvju11BUFe/zczeJfirZFG9c7wLfBP4v2b2Z+Bdgoew0TbGKt2YuWtwCun+zGwp8Ez84aFIxlHJXLodMxtp8e4E4lUTlwGnE1S/iGQkPQCV7igfeMjM+hBUJbwFnOXuek1eMpaqWURE0oCqWURE0kCXVbMcccQRPnDgwK66nIhIWli7du1H7n5kS/t1WTIfOHAg5eXlXXU5EZG0YGbvtWY/VbOIiKQBJXMRkTSgZC4ikgaS2s68pqaGHTt2sG/fvmSG0a317NmT/Px8cnJykh2KiCRRUpP5jh076NOnDwMHDiTotlnawt3ZtWsXO3bs4Pjjj092OCKSREmtZtm3bx/9+/dXIk+QmdG/f3/9ZSPShK1b4ZqrYszov4BNFw9hRv8FXHNVjK1bW3fsrFmQn7+HRYumkJ+/h1mzaNWxyZD0OvPWJvLaG9u3L4RCwTyVb2xX0S9CkUMrK4OCk2NceP8E/uuTGQxdsJn7PpnBhfdPoODkGGVlzR87bBg8+IsYN5zwfaa8tZjrv3AbD/4ixrBhNHts0nTV+HSnnXaaN7Zp06aD1h3K8uXu4bB7To47fDrl5ATrly9v1WnSVmvvo0im2LIlyA0TWeqfkNcgcXxCnk9kqYfDwX5NHRsi6qsY5zW9Qu6G1/QK+SrGeYhok8d2BoKBTbr/GKBbt8LUqVBZCTU1DbfV1ATrp05NrIT+wQcfcM8997R6/+rqan7/+9+3/UIi0qX27x9PJGIsu+08+lhFg219rIJl884jEjFqasY3eWxsaTbj854nu+oAOGRXHWB83vPElmY3eWwypXwy/8lPDk7ijdXUwN2HGl2xBQMGDOD6669v9f5//etf25T8RSQ5Zs++iUgkDMP5dETXWr2BAohEwpSW3tz0ses4eLiTCLC+6WOTKeWT+WOPtS6ZP/po28+9fft2LrroIi677DJuueUWzj77bAoKCvjjH4Oxc6+++mpGjx7NmWeeSXV1NRdddBEvvPACX/3qVwG4/fbbOeusszj11FNZunQpQJPnWrduHePHj6eoqIjZs2cDcP/99/OVr3yF0aNHU5aSlXAi3VNZWTGTJi0jMqYXjATyAIvPR0JkTC8mTnyGFSuKmjx235DcQ/4i2HdSbpPHJlVr6mI6Ykq0ztysYT15U1Mo1OaqKN+2bZtPmzbNZ8yY4ffdd5+7u//617/2G2+80T/++GMfM2aMu7sfOHCgwf61PvzwQ3d33759u48fP97d/ZDncncfMWKEv//+++7uHovFfPPmzX7hhRf6gQMHvLq62seOHdv2DxCnOnORhvr0CfLCxIlLvXJvrvtS3OfhvhSv3JvrEycudXDv27fpY887d7FHi8w9D3fDPQ+PFpmfd+7iJo/tDLSyzjzlB6fIy4O9e1u3X3uMGTMGgCFDhrB06VI+85nPUFpayrXXXsuZZ57JJZdc0mD/AwcOcM899xCNRsnJyWFvvSAbn+ujjz7i6KOPJj8/H4BQKMSGDRvYsGEDxcXFAPz9738nGo2SnZ3y/yQiKW/6dHjgAejXbzdRzyF6Tg37z8qlR49qolU59Ou3m5wcuPTSpo/t+5m9VP6/MOGXKom9kUXWqTEqx4bpe+3eJo9NppSvZpk+HVp6ubEjbmxtEz8zw92pqanh3HPPZf78+Sxbtow333yTrKwsqqurgaDa5KOPPuLHP/4xU6ZMafZchx9+ONu2bWPXrl1A8ObriSeeyNixY1m9ejWrV69m7dq1SuQiHaS0NMgLl1/+S8LhSjZuPIXJk59m48ZTCIcrmTnzQXJy4IYbWji2TxUbji1g4qvL2XBsAeE+Vc0em0wpn8xrb2xzOuPG7tq1i9GjR3PWWWcRi8U44YQT+NznPsdHH33EhAkTOOmkk9i8eTPFxcU899xzzZ4rFApx9913M2nSJIqLi7n99tspKCjguOOO48wzz2TChAn88pe/7NgPIJLBBg+GhQuhouIw5sy5k8LCcp577mxGjHiduXPvIBLpy8KFwX4deWxStaYupiMmtTPvPKozFzm0LVvcr7kmqN8OhYL5Nde0ro14e4791G53vyA+Twzp0s4coKQENm6EK65o+AboFVcE60tKkh2hiHSW9rySP3gwzJ8Pe/ZALBbM589vXam6Pcd+agmwGFjaloMS0mUDOhcWFnrjkYbefvtthgwZ0iXXT2e6j5KuysrgGxfGeHrfBEZlvUzPWDVVWbm8Fvsyk3uu5MlfZ6V4Ya4YWB2fv5DQGcxsrbsXtrRftyiZi0jmqX37e2xVGSN8DT2j1eDQK1rNCF/D2KqyhN/+7jzjCRq0G8QMlr0M84BlvwuWa7fR8W+PJpzMzayHmd1tZuvN7GUza/2rlCIiLWjPK/nJcxMQhhgwAbg4CrfE5xMI1hMGOv7t0faUzG8HKt29wN2/DPxHB8XUgj3AlPhcRNJVe17JT55iYBmU5cIaoALw+HwNwXqeAYo6/MoJJXMzOxwoAb5Xu867qvK9Cx8oiEjytOeV/OQqhnVTm+jX5et0RiKHxEvmw4A/AvPN7CUzW2xmJzbeycyuMLNyMyvfuXNnuwL91ION5u2zevXqDjlPY+phUaR98vJg9epipn3zSaoW58LjwG3A41C1OJdp33ySl14qavfb351i+Geb+GviqE67ZKLJ/GigEPiJu48Ffg481Xgnd7/f3QvdvfDII49M8FL1HihgwKvx9a80Wp9YvdncuXMTjKt56mFRpH1q3/7+9JX8EJU39iJ6ToioN/9KftKVrI3/NRECs2A+Eih5o9MumWgy/xuw1t23ALj7SuBoM+uEUYXjDxTq7G80h0QfKFx33XVs2rSJoqIi7r33XsaNG8fIkSP53veC2qOHH36Y6667juLiYl555RVefPFFzjzzTM455xyuvPLKul8Er732GkVFRYwZM4Yf/OAHxGKxg3pYFJG2ac8r+UmX1Q9W3gmPL4LbboPHfwMr74Cswzrvmq15s6jxBPQE/gfIjy+fAfyhuWPa8wao+wvuHm7i1GF3f7GV5znYyJEj3f3THhCj0agPHTrUY7GYP/TQQ37xxRfX7Xvqqaf6Rx995O7ud9xxh8+ZM8cPHDjgo0aN8j179ri7+7Rp03z79u0H9bDYmfQGqKSr5cvdlyyZ7KWlP3GzWLyH1KjPnn2XL1kyOSPe/qYze010931m9i3gcTOLAfuAS1o4rB2KgQXA1+OXqtUzvr6o3Vd45plnePPNN+nRoweVlZXs3x+U/EeNGgXAhx9+SH5+Pv379wfgtNNO49lnn2Xnzp288847nH/++QDs3r2bHTt2MGDAgHbHJJLpSkpg69bFrFwJffpARQXk5WVRVVXK0KGlqdc/ShIl3E2fu78GfKUDY2nBboJwQ0AuUB1f3t2us0ajUQB+9rOfsXbtWioqKvjv//7vuu21PRkefvjhbNmyhYqKCvLy8njhheBtriOOOIKTTjqJZ599tu4XQTgc5v3336/rYVFEElf7Wv38+cmOJLV1ozdAfwlUAqcAT8fnlbS3VcuYMWM4/fTTOeqooygsLKS0tJTjjjvuoP2ys7O5+eabGT16NBMmTKCiooKePXsSCoX47ne/y5gxYzj77LPrhqGr38OiiEhn60Z9s1wAjAGuJ/gdFAPuAX5H0O6889XU1JAT74/329/+NsXFxVxwwQVdcu3mqG8WkfTV2r5ZutFoCI0TdhZQGp+6xpw5c1i3bh01NTWMHDmSyZMnd9m1RUSa042SefL99Kc/TXYIIiKH1I3qzEVEpClK5iIiaUDJXEQkDSiZi4ikge6VzGMxWLYM5s0L5rFYsiOqc9lll7F582Y++OCDZjvY2rBhA7t3By86rVy5klWrVnVViCJJ0Z4xPKX1uk9rllgMJkyANWsgEoHevWHkSFi5ErKyOu2y7o6ZtXr/AQMG1L04dCh33303c+fOpV+/fnqhSNLeQWN4LqjmvqwZvHb/Lyh4pDuM4dl9dJ9kXlYWJPKK+PBRFRXBclkZTJqU8Gm3b9/OrFmzOOyww3j//fc55phj+OEPf8h3vvMdsrOzGTNmDNOmTePKK69k7969HHnkkTz66KP06NGDf/3Xf+W3v/0t/fv3p7Kysu58c+fO5YknnmDbtm1cd911VFRUkJ+fT3FxMStWrGDTpk1cf/317N+/n3379nHVVVexZMkS7rzzTkKhEP369ePBBx+kf//+nHHGGYwfP56XXnqJ3NxcnnnmGXJzczvijop0qtoxPIuryhhBfAxP4mN4UjuG5yQ2bmzriPdySK3pjasjpvb1mujut93mbuYOn05m7vPmtf4ch7Bt2zYfOHBgXa+Ht9xyi996660+aNAgr6qqcnf36dOn+xtvvOHu7v/5n//pjzzyiK9atcq/9a1vubt7dXW1FxQU+Ntvv92gt8RzzjnH161b5+7usVjM3d1nzJjhb7/9tru7P/TQQ/7zn//c//GPf/gZZ5zhkUjE3d2ffPJJv/HGG93dfdCgQXX36brrrvMlS5Yc9BnUa6Kkok2bxrk77rfhbjT6v4v7vCA9vP32uGSHmtJoZa+J3afOfPjwoGqlvt69oaCg3ac+/fTT6du3LwAjR44kFAoxfPhwevbsCcDGjRu54YYbKCoq4pFHHuHDDz/kjTfe4NxzzwWgR48eDBs27KDz7t69m4J4fKFQ07f63XffZcSIEYTDQb/t48ePZ/PmzUDQkVftq/pDhgzh448/bvfnFekK3XMMz+6r+yTzkpKgjjwvLz5yR16w3AEVbm+++Sb79gVd6z7zzDP069evrrdEgBNOOIGHH36Y1atX8+qrr3Lttdfy+c9/npdffhmAqqoq1qxZc9B5Q6EQW7ZsAYJ+XQCysrIO6k1x0KBB/OEPf6CqqgqAF154geHDhwM0qK83s9r+5EVSXvcdw7N76j515llZwcPOsjJYvz4okZeUdMjDz2OOOYbp06fzt7/9jcLCQs477zxeeeWVuu0/+tGPmDlzJgCHHXYY9957LxdeeCFPP/00Z555JgMGDDhkR1fz589n5syZhEIhhg4dyr333ktJSQkXXXQR8+bNq9uvf//+lJaWUlxcTO/evRkwYAD33ntvuz+XSDLVH8PzqcVT6bW6GtYDBVBVlMu0i4IxPON/FEt7taYupiOmdteZd5KuHBGos6TCfRRp7Oqr3XNy3C+55FH/5JM8r6kJeSTSy2tqQv7JJ3l+ySWPek6O+zXXJDvS1Eba1ZmLSLfSrcfw7IYyPpkPHDiQJ554ItlhiKSdwYNh4UKoqDiMOXPupLCwnOeeO5sRI15n7tw7iET6snChmiV2lKTXmXsbX8qRhlwPRCWFaQzPrpPUZN6zZ0927dpF//79ldAT4O7s2rWrrgmlSCrSGJ5dI6nJPD8/nx07drBz585khtGt9ezZk/z8/GSHISJJltRknpOTw/HHH5/MEERE0kLGPwAVEUkHSuYiImlAyVxEJA0omYuIpAElcxGRNKBkLiKSBpTMRUTSgJK5iEgaaFcyN7OxZuZm1q+jAhIRkbZLOJmbWS/ge8CujgtHRFLR1q1wzVUxZvRfwKaLhzCj/wKuuSrG1q3Jjkxqtadk/jPgLqCig2IRkRRUVgYFJ8e48P4J/NcnMxi6YDP3fTKDC++fQMHJMcrKkh2hQILJ3MxuAv7i7s3+M5rZFWZWbmbl6kxLpPvZuhWmToWxVWWM8DX0jFaDQ69oNSN8DWOrypg6FZXQU0Cbk7mZzQC+BNzS0r7ufr+7F7p74ZFHHplIfCKSRPv3jycSMZbddh59rOEf4X2sgmXzziMSMWpqxicpQqmVSMl8LnAysM7M1gOfA35nZkM7NDIRSbrZs28iEgnDcKB3o429gQKIRMKUlt6chOikvjYnc3cf4u5fcvcCdy8A/gJ8xd03dXx4IpJMZWXFTJq0jMiYXjASyAMsPh8JkTG9mDjxGVasKEpqnKJ25iLSjLw8WL26mGnffJKqxbnwOHAb8DhULc5l2jef5KWXisjLS3ak0u5k7u4D3X13RwQj3c/WrTBrFuTn72HRoink5+9h1iw9EEsX06dDTg7067ebqOcQPSdE5Y29iJ4TIuo59Ou3m5wcuPTSZEcqKplLwsrKYNgweOABKC5ewpQpiykqWsoDDwTr1WSt+ystDZL55Zf/knC4ko0bT2Hy5KfZuPEUwuFKZs58kJwcuOGGZEcqSuaSkNoma5WVEKuJ8Z0hd8A8+M6QO4jVxKispFVN1lSyT22DB8PChVBRcRhz5txJYWE5zz13NiNGvM7cuXcQifRl4cJgP0kuJXNJSG2TNY8asXHZnPyjt+AWGHb7m8TGZeNRa7HJmkr23UNJCQwduph9+26kT58QoRDk5WVRVVXK0KGLKSlJdoQCSuaSoLoma2XAGrAI4PH5GqCs+SZr9Uv2NTUwc+aDQDCvqaHVJXvpGoMHw/z5sGcPxGLBfP58lchTiZK5JKS2ydr+NTkQabQxAvv/kNNsk7W6kr0H06hRrwIwevQrdev0MopI6ymZS0Jqm6zdvmIuHm64zcNwe9ncZpus1ZXsAWKQu2o/zIvPY8FqvYwi0npK5pKQ2iZrW074ArHCLLw3uIH3hlhhFltO+EKzTdbqXkb5pBdMAC4m6CDiYmACRD7RyygibaFkLgl1b1rbZG3m/3oIe/YAW38wiIePn8HWHwzCnj3Av3zr4WabrNWW7O8aNxv/PUHfmx7M/fdw17jZehlFpA2UzDNcot2bNmiy9m93ceKN7zLzTw/zT6XvMPemO1tsslZbsv/i/k1Q2WhjJQzdv0kvo4i0gZJ5Bmtv96btabJWW7I/cdofD+7AKQwnfuMdvYwi0gZK5hmsI7o3TbTJWm3J/r2hg3j38BPYSx4xjL30Zkv/L/DnLx7f4ssoeuFI5FNK5hks2d2blpTA0JOf5mcT3+byXo9zq93G5b2e4P9O3MzQk59utmSvF45EGjJ375ILFRYWenl5eZdcS1onFIKxY19k2dMT6f21quBlnwhBIh8Jkd/0YuL5y/nd74qIxZIcbD1btwYJu7ISQsRY98MChsXeYkPoZE69eR0HyCIcho0b9VKLdH9mttbdC1vaL7srgpHUVL9706cWT6XX6mpYDxRAVVEu0y4Kujft2zfZkTYUVA89H7RHnwD+I6AShoWDrgRYCWTB5s3jgOeSG6xIF1E1Swbrrt2btrcrgUylZwzpTck8g3XX7k3b25VAJqp9xvDgL2LccML3mfLWYq7/wm08+IuYnjGkC3fvkum0005zST3Ll7svWTLZS0t/4mYxB/dQKOqzZ9/lS5ZM9uXLkx3hwfr0cQf3Wwv/3Q/0JliITwd647cW/ruDe9++yY40NWzZ4h4Ou4eI+irGeU2vkLvhNb1CvopxHiLq4XCwn6QeoNxbkWNVMs9w3bF70/Z2JZBpapugxpZmMz7vebKrDoBDdtUBxuc9T2xptjo1SwNK5tLtujdtb1cCmabuGcM6DlktxXo9Y0gHSubS7bS3K4FMU/uMYd+Q3EO+T7DvpFw9Y0gDSubSLXXH6qFkqW2C+o2HFhArNMgDDMiDWKHxjYcWqFOzNKBkLt1Wd6seSpbaZwx9P7OXykVhYo8Z+/89m9hjRuWiMH0/s1fPGNKAkrlkpExqc92gCWqfKjYcW8DEV5ez4dgCwn2qUrYJqrSNkrlknEzr16XBM4Y5d1JYWM5zz53NiBGvM3fuHXrGkCaUzCWj1B9IOlYT4ztD7oB58J0hdxCriaXtQNJ6xpD+lMzTRCZVG7RH3UDSUSM2LpuTf/QW3ALDbg/6dfFo+g4krWcM6U3JPA1kWrVBe6hfF0lXSubdXKZWGyRK/bpIulIy7+YyudogEbVtrm9fMRcPN9zmYbi9bK7aXEu3lHAyN7Pvm9nzZvaqmf3azA7ryMCkdVRt0DbdvV8XPRuRJrWmN65DTcDF9X6eB/y4uf3Va2LnMHMvKnrBq2/OcbeGPQi64dX/nuNjx77ooVCyI00NtT0IvvBCkUerzd+9e5A/OGiGv3v3II9Wmz//fHHK9iC4fHkQe06O+/Tpj7g7fsklj3pOTrA+FXu4lPajs3tNdPfH6y2+DhzTnl8qkhhVG7RNd+3Xpf6zkZoamDnzQSCY19SgZyPS/jpzM8sB/jfw5CG2XWFm5WZWvnPnzvZeSg6hu1cbJEN3bHNd92zEg+cjX979MsyDr+z5HR4N1uvZSIZrTfG9qQn4ArAauLKlfVXN0jm6c7VBd7Vli/usK6P+z4c/4f9z0Un+z4c/4bOujHbqPT733Be8oiLsHsV9HO55QV73vPhyFK+oCPu5577YeUFIUtDKapb2JPLzgVeAL7ZmfyXzztMdRwvqrpYvd8/rFfXnbZxXZee6G16ZnevP2zjP6xXttHtd+2yk6qncIIHXfzaSh1c9latnI2mqtck8oWoWM/sscCdwrrv/Twf8gSDt0B2rDbqj2nrrsVVljPA19IxWg0OvaDUjfA1jq8o6rd669tnIwpum4o3ax3sEnrrp63o2kuESrTMfDhwBPG1mq+PTbzowLmkjvard+WrrrZfddh59rKLBtj5WwbJ557Wq3jqR5oW1z0b+dsxnodGDbsLw92OO0rORDJdQMnf3Fe7e392L6k1f6+jgRFJJXZv+4RxyxB4KWm7Tn2jXC7Xd2J5281oYCbFeIQ5gxHqFYCScetMb6sY2w+kNUJFWqu0KIDKmF4ykwYg9jITImF7NdgXQnuaFdU0qq/rx3VPu5IKqRXyP25iy7zfMKbiDyL7DUrJJpXSd7GQHINJd1NZbT/vmkzy1eCq9VlfDeqAAqopymXbRk7z0UhF9+x76+KCa5vm65erqHgCMHv0K7la3fvPmccBzBx1fUgJbty5m5Ur4bV9YXnE+eXlwXDUMHfodJfIMp5K5SCvV1lv367ebqOcQPSdE5Y29iJ4TIuo59Ou3u9l667pqGoAY5K7aD/Pi81iwuqVqGj0bkaYomYu0UoPh18KVbNx4CpMnP83GjacQDle2OPxaXTXNJ71gAnAxcEt8PgEinzRfTSPSHCVzkVZq7/BrtdU0d42bjf8eqAA8mPvv4a5xs9W8UBKmZC7SBu1p019bTfPF/ZugstHGShi6f5OaF0rClMxF2ijReuvaapoTp/3x4KaNYTjxG++oeaEkTK1ZRLpIbTXNe9WD6Hl4NcdU/JUwESoJ87f+x/DnLx6v5oWSMCVzkS4UNC98mnsmxvj7w2X80771/LFnAZ+dWML1J2cpkUvCLOjHpfMVFhZ6eXl5l1xLRCRdmNlady9saT/VmYuIpAElcxGRNKBkLiKSBpTMU4hGXheRRCmZp4hEu0YVEQGU0N86AAANxElEQVQl85SgkddFpL2UzFNAg5HX3Rg16lXg065RNfK6iLREyTwFNOgaFcjN3d9gDi13jSoimU3JPAXUdY0aCQf9Wi8D5sXnsSCRq2tUEWmOXudPAbVdo178jV+xqHIKWeUOEaA3xAqNi8O/anYEGxERlcxTQG3XqGMivyX0ujfo5zr0ujMm8lt1jSoizVIyTwG1XaN+bdBvDtnP9dcGLVLXqCLSLCXzFFDbNerO/M+yP7tHg237s3uwM/8odY0qIs1SMk8RJSVwxPRX2HbUV6ggjxhGBXlsO+orHDH9lWZHsBER0QPQFDL4xCx4b2Xwuuf69eQVFHBSSQlkZSU7NBFJcUrmqSYrCyZNCiYRkVZSNYuISBpQMhcRSQNK5iIiaUDJXEQkDSSczM2s2Mz+YGavmtliMzu8IwMTEZHWSyiZm1lP4D5gqruPAl4CftSRgYmISOslWjKfALzi7n+OLz8AnN8xIYmISFslmswHAn+qXXD3vUC2meXU38nMrjCzcjMr37lzZ+JRiohIsxJN5rlAtNG6KEFff3Xc/X53L3T3wiOPPDLBS4mISEsSTeY7gONqF8ysN7DP3RsneBER6QKJJvMy4Bwz+2x8+QrgVx0TkoiItFVCfbO4+z/M7DpgqZkdIKg//18dGpmIiLRawh1tufszwDMdGIuIiCRIb4CKiKQBJXMRkTSgZC4ikgaUzDvY1q0waxbk5+9h0aIp5OfvYdasYL2ISGdRMu9AZWUwbBg88AAUFy9hypTFFBUt5YEHgvVlZcmOUETSlZJ5B9m6FaZOhcpKqKmBmTMfBIJ5TU2wfupUldBFpHMomXeQ/fvHE4kY7sE0atSrAIwe/UrdukjEqKkZn+RIRSQdKZl3kNmzbyISCdct5+bubzAHiETClJbe3OWxiUj6UzLvIGVlxUyatKxBQq8vEgkzceIzrFhR1LWBiUhGUDLvIHl5sHp1MdOmLaCqqmeDbVVVPZk2bQEvvVREXl6SAhSRtKZk3kGmT4ecHOjXbzfRaDbRaIjKyl5EoyGi0Wz69dtNTg5cemmyIxWRdKRk3kFKS4NkfvnlvyQcrmTjxlOYPPlpNm48hXC4kpkzHyQnB264IdmRikg6Mndvea8OUFhY6OXl5V1yrWQpK4No9AJ+++KX2Xz3iRSwgQ02jJNueIevFL1CdvZiSkqSHaWIdCdmttbdC1vcT8m8Y219J0bNWRPI/2ANvYhQRW92DBhJzgsrGXxiVrLDE5FuprXJXNUsHWzwO2WctGcNeVSQhZNHBSftWcPgd/T6p4h0HiXzjrZuHUQiDddFIrB+fXLiEZGMoGTe0YYPh969G67r3RsKCpITj4hkBCXzjlZSAiNHBg3PzYL5yJHoyaeIdKaEh42TJmRlwcqVQdOW9euDEnlJSbBeRKSTKJl3hqwsmDQpmEREuoCqWURE0oCSuYhIGlAyFxFJA0rmIiJpQMlcRCQNKJmLiKQBJXMRkTSgZC4ikgaUzBvZuhVmzYL8/D0sWjSF/Pw9zJoVrBcRSVVK5vWUlcGwYfDAA1BcvIQpUxZTVLSUBx4I1pepF1sRSVFtTuZmlmtmPzOzZ83sdTN7wMxyOiO4rrR1K0ydCpWVUFMDM2c+CATzmppg/dSpKqGLSGpKpGTeG1jq7l919xFAD+BbHRtW19u/fzyRiOEeTKNGvQrA6NGv1K2LRIyamvFJjlRE5GBtTubu/rG7P1tvVTlwzKH2NbMrzKzczMp37tyZaIxdYvbsm4hEwnXLubn7G8wBIpEwpaU3d3lsIiItaVeduZn1Bf4FWHSo7e5+v7sXunvhkUce2Z5LdbqysmImTVrWIKHXF4mEmTjxGVasKOrawEREWqHFZG5mR5vZ6nrTcfH1hcCzwK3uvq6zA+1seXmwenUx06YtoKqqZ4NtVVU9mTZtAS+9VEReXpICFBFpRov9mbv734Ci+uvM7Erga8DX3f39zgmta02fHrRi6ddvN9FoNtFoiP37c+nRo5poNJt+/XaTkwOXXprsSEVEDpZIa5ZhBA88J6VLIgcoLYWcHLj88l8SDleyceMpTJ78NBs3nkI4XMnMmQ+SkwM33JDsSEVEDpZInfko4HPAqnpVL/d2cFxdbvBgWLgQKioOY86cOyksLOe5585mxIjXmTv3DiKRvixcGOwnIpJq2jxsnLvfB9zXCbEkXUkJbN26mJUroU8fqKiAvLwsqqpKGTq0VIlcRFKWxgBtZPBgmP8fMeafUwbr1sHw4RqQWURSnpJ5Y7EYTJgAa9ZAJAK9e8PIkbBypRK6iKQs9c3SWFlZkMgrKsA9mK9Zo45ZRCSlKZk3tm5dUCKvLxKB9euTE4+ISCsomTc2fHhQtVJf795QUJCceEREWkHJvLGSkqCOPC8PzIL5yJHBehGRFKUHoI1lZQUPO8vKgqqVggK1ZhGRlKdkfihZWTBpUjCJiHQDaVnNoqHfRCTTpF0y19BvIpKJ0iqZa+g3EclUaZXMNfSbiGSqtErmGvpNRDJVWiVzDf0mIpkqrZK5hn4TkUyVssm8tnlh374QCgXzlpoXTp8ejBZUf+i3yspeRKMhDf0mImktJZN5/eaFe/cGnRfu3UuLzQs19JuIZKqUS+aNmxfW11LzQg39JiKZKuWS+U9+cnASb6ymBu6++9DbSkpg6NDF7Nt3I336hAiF6g/9tlj9ZYlIWjJ375ILFRYWenl5eYv79e0bVKm0Zr89ezogMBGRFGZma929sKX9Uq6jrYqKDtovFgsq1zWOp4hkgJRL5nl5rSuZN9u8UON4ikiGSbk689rmhc1psXmhxvEUkQyTcsm8tnlhc1psXqhxPEUkw6RcMq9tXhgOH5zUc3KC9S02L9Q4niKSYVIumUPwrHLjRrjiioZvgF5xRbC+xeaFGsdTRDJMyjVN7DC1rVk0jqeIdGPdtmlih9E4niKSQVKymkVERNqmXcnczGaY2e6OCkZERBKTcDI3s6OAf+7AWEREJEEJJXMzywIeAG7s2HBERCQRiZbMfw4sdvcNze1kZleYWbmZle/cuTPBS4mISEtabJpoZkcDT9Rb9SIQdvc58e273b1fixcy2wm8145Yu4sjgI+SHUSK0z1qnu5PyzLpHn3e3Y9saac2tTM3szzgXeBDoPbAk4E3gSJ3z/iHoWZW3po2oZlM96h5uj8t0z06WJvambt7BXBM/XXxkrnekxcRSSK1MxcRSQPtTuatqS/PMPcnO4BuQPeoebo/LdM9aqTL+mYREZHOo2oWEZE0oGQuIpIGlMxFRNKAknkbmNloM3vVzAriy8eZ2a/NbJWZrTOza5s4brmZ/cHMVsenL3dt5F3nEPdokJmtMLMXzGytmV3WxHHfMLM34sc+bGY9uzTwLtKO+5Ox36FG2x4ys8VNHJcR36GmpG9/5h3MzB4B+gB9663+LDDH3beYWRhYa2YvuPumRocfDlzo7u93UbhJ0cQ9+jtwnrvXmNlhwDYze8zdo/WOOwa4FRjl7rvN7D+A64H/03XRd75E709cJn+HarcVA6cBfzrEtoz4DjVHJfPWu8rdp1DvFWJ3f93dt8R/rgTeBo4+xLGfITNePT7UPYq4e0188QTg7UMkqm8AT9R7g/h+4IJOj7brJXp/IIO/Q1DXrcitwL81cVymfIeapJJ5K8WTdZPM7EvAUOC1JnZZYWYhYBlwp7sf6OAQk66pe2RmdwDnE3zfph5il4HA2nrL7wEDOjq+ZGvH/amVkd+heDciC4DrCP5COZSBZMB3qDkqmXcAM7uEoEvgC9y9qvF2d/8ndx8LTALOAGZ1cYhJ5e7fdfeTgCnAk2Z2bKNdcoH6pdEYkHaJqimtuD8Z+x0ysxzgSeAud9/YzK4Z/R0CJfN2MbOQmT0MjATOcvfNze3v7nuAB4HTuyC8lOPubwK/4+DPvwM4rt7y54HtXRRWymjm/tTfJ9O+Q8UEnfnNM7P1BIWmYjNb2Gi/jP8OKZm3zyygwt3/dzN/QueaWZ/4z9nANJquikk7ZjYkPpgJZnY4QRIqb7Tbr4FLzax3fPlq4NGuizJ5WnN/Mvk75O7Puvux7l4Q79DvW8CL7t64Oipjv0O1VGfePl8GTjaz1fXW/Yqg9HA/wUhMucBKM4sAOcAK4L4ujjOZRgOPx8eKPQDc6O7vxet+748v/9HM7gFWm9kB4HXgoeSF3KVavD/oO3RI+g41pL5ZRETSgKpZRETSgJK5iEgaUDIXEUkDSuYiImlAyVxEJA0omYuIpAElc0kLZvaUmW2PT1Ez+2u95enx/k9E0pbamUvaMbPtwHR3fznZsYh0FZXMRUTSgJK5pD0zu9XMHoj/PDBeDfN1M9tgZv8wszvMLN/MVprZDjN708zOqHd8TzP7qZm9Y2bbzOy++GAkIilDyVwyURYwAigAvgRcBqwCvuvu+cBjwL319v8v4CiC3vtOBPoDt3RhvCItUjKXTPVjD3wAPAv8zt03xLc9SdCBWsjMjiDopfBqd6+Ojwp0B80PIiHS5dRromSimLvvqrdcAfyt3vJegv8bPYDjCUryb5pZ7fYQQU+GIilDyVykeX8B9gGD0nGYNkkfqmYRaUa8GuY14Pb4EGaY2bH1H5CKpAIlc5GWXQLkA1vNbAvBACT6vyMpRS8NiYikAZUuRETSgJK5iEgaUDIXEUkDSuYiImlAyVxEJA0omYuIpAElcxGRNKBkLiKSBv4/Sy3shUeamLMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.title(\"Testing the Model\", fontsize=14)\n",
    "plt.plot(t_instance[:-1], time_series(t_instance[:-1]), \"bo\", markersize=10, label=\"instance\")\n",
    "plt.plot(t_instance[1:], time_series(t_instance[1:]), \"w*\", markersize=10, label=\"target\", color='yellow')\n",
    "plt.plot(t_instance[1:], y_pred[0,:,0], \"r.\", markersize=10, label=\"prediction\")\n",
    "plt.legend(loc=\"upper left\")\n",
    "plt.xlabel(\"Time\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 심층 RNN\n",
    "\n",
    "심층 RNN(deep RNN)은 아래의 그림처럼 RNN Cell을 여러 층으로 쌓은 것을 말한다. 일반적으로 딥러닝에서는 이러한 Deep RNN 모델이 많이 사용된다. \n",
    "\n",
    "![deep-rnn](./images/deep-rnn.PNG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 텐서플로에서의 Deep RNN\n",
    "\n",
    "텐서플로에서는 [`tf.nn.rnn_cell.MultiRNNCell`](https://www.tensorflow.org/api_docs/python/tf/nn/rnn_cell/MultiRNNCell)을 이용해 RNN Cell을 여러층으로 쌓을 수 있는데 먼저 `BasicRNNCell`을 이용해 여러개의 Cell을 만들고, 이것을 `MultiRNNCell`로 쌓아서 deep RNN을 구현할 수 있다.  아래의 예제는 deep RNN을 구현한 간단한 예제 코드이다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "outputs_val:(2, 5, 100)\n"
     ]
    }
   ],
   "source": [
    "reset_graph()\n",
    "\n",
    "################\n",
    "# layer params #\n",
    "################\n",
    "n_inputs = 2\n",
    "n_steps = 5\n",
    "n_neurons = 100\n",
    "n_layers = 3\n",
    "\n",
    "X = tf.placeholder(tf.float32, [None, n_steps, n_inputs])\n",
    "\n",
    "# Deep RNN\n",
    "layers = [tf.nn.rnn_cell.BasicRNNCell(num_units=n_neurons)\n",
    "          for layer in range(n_layers)]\n",
    "multi_layer_cell = tf.nn.rnn_cell.MultiRNNCell(layers)\n",
    "outputs, states = tf.nn.dynamic_rnn(multi_layer_cell, X, dtype=tf.float32)\n",
    "\n",
    "# Sample data\n",
    "batch_x = np.random.rand(2, n_steps, n_inputs)  # shape: (2, 5, 2)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    tf.global_variables_initializer().run()\n",
    "    outputs_val, states_val = sess.run([outputs, states], feed_dict={X: batch_x})\n",
    "    \n",
    "print('outputs_val:{}'.format(outputs_val.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Dropout 적용하기\n",
    "\n",
    "2.1에서처럼 여러 층의 RNN Cell을 쌓게되면 오버피팅(overfitting)되기 쉽기 때문에 RNN layer의 전과 후에 드롭아웃(dropout) layer를 추가할 수 있다. 또한, RNN 층(layer) 사이에도 드롭아웃을 적용할 수 있는데, 텐서플로에서는 [`tf.nn.rnn_cell.DropoutWrapper`](https://www.tensorflow.org/api_docs/python/tf/nn/rnn_cell/DropoutWrapper)를 이용해 RNN 층 사이에도 드롭아웃을 적용할 수 있다.  아래의 예제는 `DropoutWrapper`를 이용해 RNN 층에 드롭아웃을 적용한 예제이다. 아래의 코드에서 사용한 데이터는 1.4.1에서 생성한 가상의 시계열 데이터를 사용했다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "################\n",
    "# layer params #\n",
    "################\n",
    "n_inputs = 1\n",
    "n_outputs = 1\n",
    "n_neurons = 100\n",
    "n_layers = 3\n",
    "n_steps = 20\n",
    "\n",
    "X = tf.placeholder(tf.float32, [None, n_steps, n_inputs])\n",
    "y = tf.placeholder(tf.float32, [None, n_steps, n_outputs])\n",
    "\n",
    "# without dropout rate\n",
    "keep_prob = tf.placeholder_with_default(1.0, shape=())\n",
    "cells = [tf.nn.rnn_cell.BasicRNNCell(num_units=n_neurons)\n",
    "         for layer in range(n_layers)]\n",
    "cells_drop = [tf.nn.rnn_cell.DropoutWrapper(\n",
    "                    cell, input_keep_prob=keep_prob,\n",
    "                    state_keep_prob=keep_prob) for cell in cells]\n",
    "multi_layer_cell = tf.nn.rnn_cell.MultiRNNCell(cells_drop)\n",
    "rnn_outputs, states = tf.nn.dynamic_rnn(multi_layer_cell, X, dtype=tf.float32)\n",
    "\n",
    "# 하나의 출력을 위한 작업\n",
    "stacked_rnn_outputs = tf.reshape(rnn_outputs, [-1, n_neurons])\n",
    "stacked_outputs = tf.layers.dense(stacked_rnn_outputs, n_outputs)\n",
    "predictions = tf.reshape(stacked_outputs, [-1, n_steps, n_outputs])\n",
    "\n",
    "# loss\n",
    "mse = tf.losses.mean_squared_error(labels=y, predictions=predictions)\n",
    "\n",
    "################\n",
    "# Train Params #\n",
    "################\n",
    "learning_rate = 0.01\n",
    "train_keep_prob = 0.3\n",
    "n_iterations = 1500\n",
    "batch_size = 50\n",
    "\n",
    "# optimizer\n",
    "train_op = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(mse)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step: 000, MSE: 4.1924\n",
      "step: 300, MSE: 10.9267\n",
      "step: 600, MSE: 12.2163\n",
      "step: 900, MSE: 14.6663\n",
      "step: 1200, MSE: 14.5340\n"
     ]
    }
   ],
   "source": [
    "# Train\n",
    "with tf.Session() as sess:\n",
    "    tf.global_variables_initializer().run()\n",
    "    for iteration in range(n_iterations):\n",
    "        batch_x, batch_y = next_batch(batch_size, n_steps)\n",
    "        sess.run(train_op, feed_dict={X: batch_x, y: batch_y,\n",
    "                                      keep_prob: train_keep_prob})\n",
    "        if iteration % 300 == 0:\n",
    "            loss = mse.eval(feed_dict={X: batch_x, y: batch_y})\n",
    "            print('step: {:03d}, MSE: {:.4f}'.format(iteration, loss))\n",
    "            \n",
    "    # 새로운 데이터 예측하기\n",
    "    X_new = time_series(np.array(t_instance[:-1].reshape(-1, n_steps, n_inputs)))\n",
    "    y_pred = sess.run(predictions, feed_dict={X: X_new})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXMAAAEbCAYAAAAoHWSlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3X101PWZ9/H3NSEQkwBRtNWSKpW1q6zVUINUaCFRlI2illtadMW7LvamlWqrxhZsPaurtrZaH3brcb09SKl6r6XiigIGUBBrtbUNRdEqrURpwT4IWjhmEkgmXPcfvwlOQh4mM5N5/LzOmTP8nr/zc7zynev3fTB3R0REclso0wUQEZHkKZiLiOQBBXMRkTygYC4ikgcUzEVE8oCCuYhIHlAwl16Z2aNmti36ipjZX2KWK1N4nSFm9rSZHZuqc8Z53VvM7L4Un/MXZvaumZX0sv00M3Mzm5PkdR42s4Wp3ldyl4K59Mrdv+DuY9x9DLADOLDs7jsSPa+ZnW1mj8RcJ+LuZ7r7Wykodl/XXWtmpw7mNaKGAb0F62uBv6ehDFJgFMwlEz4GHJqB644jPd/55cA3uq80s7HAROC3aSiDFBgFc0mKmc00s5ejqZdfmNmnYrZdamabzexPZtZkZh8zs7uA7wNTo8csiKZZ3MyOjB73cDQF8rCZ/TG63wUx5zUzu9rMtpjZdjNbaWb/1VvKxMyOMbNtwJHA49HzDYtuLjKzu6Pl+7OZ1cf7+fqwOnread3WXw3cC3Tpdm1mR5vZUjPbGr1XT5nZP8ZsH2JmN0W3bzeznwKl3c5xopmti5bzdTObFUc5JZ+4u1569fsCtgGf7bZuGvBH4Ljo8heAt4ChwCeB94GPRLeNBsqi//4ysDrmPEMIAtyR0eWHgT8DVdHlC4A9QHl0+XJgEzA6unwasBu4r5/PsAP4TMzyLdHznhldrgbagH/o7/P1cY1fABdGP+PKmPWHAe9G358B5kTXHwJsBeoJKlcGfBXYHvN5bwXWAhXR5XOj5VwYXf4I8DdgRnT5k9FrnRBzPxdm+juk1+C+VDOXZNQD/+7ubwK4+6NAO3BK9H0oUBXd9o67hwdw7mXu/nL02MeACHBcdNsVwLfd/Z3o9l8C/y/Bz/Bzd386ep5GYDPQWfvu6/P152HgFDPrLPPl0c/0frf9zgXec/c73H2/B+4D3gH+2cwMuBL4hrvvjpZjBfB0zDn+FVjj7iuj2/8APA58Pu67IDlPwVyScSzw3ZgWLtuAUcBH3f1tYCbw72b2mpn9ywDP/eduy38HyqL/Hgts6WF7It7ptrw75jq9fr7+Turue4H/Ar4RTenMB+7qYdeePgtAE/BxgtRQKfD7bttjP++xwLndynl+tKxSIIZkugCS0/5M8PP98Z42Rmu8T5vZBIJcdbO7P5mC675PEOjejlk3liBlkkp9fr443Au8ThCsGztr+N1sJ6idd/cJYCnBZ3WgEvhTzPZjgddiyvmgux/00FUKh2rmkowlwPVm9nEAMxtmZudF/z0m5iHeywRBa3h0+X3gGDMrMrNEKhRLCWrMh0WvdR4wPY7j3gfGDuCaS+jl88XD3XcB/wP8ELizl92eAD5uZleZWSj6cHcuMILgucI+gtYxd5rZIdFyzANOjDnHfwMXmdmk6HYzszPNbES8ZZXcp2AuyXgw+nom+tP+ZaAzgI8AnjCzd4BXgWeBzrblDQS1yW0E+e+B+g5Bbvs1M3sLOBP4CdBfTv67BIH1VTMbGsd1+vp88boLeNXdn+tpo7t/AJwefW0D/kDw4PUsd2+L7jaPIFf/tpm9SfCr5PGYc7wJ/Avwn2b2J+BNgoewkQGWVXKYuWtyCsl9ZrYCWBV9eChScFQzl5xjZhMtOpxANDVxKXAqQfpFpCDpAajkokrgx2Y2nCCV8Bpwururm7wULKVZRETygNIsIiJ5IG1plsMPP9zHjBmTrsuJiOSFjRs37nL3I/rbL23BfMyYMTQ2NqbrciIiecHM/hjPfkqziIjkAQVzEZE8oGAuIpIHMtrOvL29nR07drB3795MFiOnlZSUUFlZSXFxcaaLIpJ1mprgjjvgySf38KMfXcqVVy7hvPNGUl8PY8cO3rEZka6B00855RTv7q233vKdO3f6/v37D9om/du/f7/v3LnT33rrrUwXRSTrPPWUe2mpe3Gx+5w5D7o7fvHFD3lxcbD+qacG59hUIxhxM7snp9i7dy+jRo0iGH9fBsrMGDVqlH7ZiHTT1ASzZkFLC7S3w9y5i4Hgvb09WD9rVrBfKo/NpIznzOMN5E1NMH8+jBgBoVDwPn9+9t3QdNMfQpGDtbVNIxw23IPXpEkvAjB58gsH1oXDRnt792lakzs2kzIezOPR0AAnnQSLFsEHH4B78L5oUbC+oSHTJRSRbHLttd8hHP5wzuthw9q6vAOEw6XU11+f0mMzKeuDefefPLGS/cnzzjvvcPfdd8e9/759+/jVr3418AuJSFo1NNQyY8bKLkE5VjhcyjnnrGL16pqUHptJWR/M77jj4CDeXXs73NXT7Ir9GD16NFdddVXc+//lL38ZUPAXkcwoL4cNG2qZPXspra0lXba1tpYwe/ZSnnuuhvLy1B6bSVkfzB9+OL5g/tBDAz/3tm3buPDCC7n00ku54YYbOPPMM6mqquL3vw/mzr388suZPHkyp512Gvv27ePCCy9k/fr1nHXWWQDceuutnH766Xz6059mxYoVAL2ea9OmTUybNo2amhquvfZaAO6//34+97nPMXnyZBqUKxJJmTlzoLgYKip2E4kMIRIJ0dJyCJFIiEhkCBUVuykuhksuSe2xGRVPk5dUvHpqmvj666/32yzHzD3Ikvf9CoUG3OLH3377bZ89e7Z/6Utf8vvuu8/d3R977DG/5ppr/P333/cpU6a4ux9oOtm5f6d3333X3d23bdvm06ZNc3fv8Vzu7hMmTPDt27e7u3tHR4dv2bLFL7jgAt+/f7/v27fPp06dOvAPEBXPfRQpJFu3Bk0I16+v8Ugk5Bs3jvdp09b6xo3jPRIJ+bp1tV5aGuyXymMHA3E2Tcz6ySnKy4OHnfHsl4wpU6YAcMIJJ7BixQoOPfRQ6uvrueKKKzjttNO4+OKLu+y/f/9+7r77biKRCMXFxXwQU8ju59q1axdHHnkklZWVAIRCIV555RVeeeUVamtrAfjb3/5GJBJhyJCs/08ikvXGjoVly6C5eSQLFtzOnXdehXuICRNO55pr7mbKlOdZtqznzj/JHJtJWZ9m6fzJ05dU/OTpbOJnZrg77e3tnH322dxzzz2sXLmSV199laKiIvbt2wcEaZNdu3bxgx/8gJkzZ/Z5rsMOO4y3336b9957Dwh6vn7yk59k6tSpbNiwgQ0bNrBx40YFcpEUqquDceOWs3fvNQwfHiIUgvLyIlpb6xk3bjl1dYNzbFd7gJnR98GV9cG8vj6+YH711am97nvvvcfkyZM5/fTT6ejo4LjjjuNjH/sYu3btYvr06Rx//PFs2bKF2tpannnmmT7PFQqFuOuuu5gxYwa1tbXceuutVFVVcfTRR3Paaacxffp0HnjggdR+AJGedHTAypVw883Be0dHeo5NQmcfk8rKPTz++EwqK/fE3cdk7Fi45x7Ysyco7p49wXI8tepkjv3Qk8ByYMVADkpMPLmYVLwSzZm7d+1aG5snz0TX2myknLnEJRJxP+MM9/Ly4GFUeXmwHIkM7rFJyKZu9Ymp8SAE1iZ8BnKhO3+86upg82aYN69rD9B584L18f/kEckDidaQGxrgpZeguTmoDzU3B8vxtKRK5tgEpaxbfVp/UUwDLOb1YnT9C93Wp773aMJJWjMbCvwAqAWagWXuPmiNsDt/8txzz2BdQSQHdHTA9OlBIA2HoawMJk6ENWugqKjvYzdtCo6JFQ7Dyy/DjBmDd2yCgm716w4s79s3FPiwW32nLVvOAHpJdSZzvxLyHeCXQEvnp+j2DlAKpL73aDI181uBFnevcvfPAv+RojKJSG+SqSGPHx8Es1hlZVBVNbjHJigl3erT/ouiFlhJELB7UgqsAmpSfuWEgrmZHQbUAf/WuS6a2xGRwdRXDbk/dXVBrbS8HMyC94kT48tTJnNsglLSrT6Z+5WwWmApdAwL4vrNBO8dw4L1gxDIIfE0y0nA74F7zGwc8HfgW+7+h9idzGweMA/g6KOPTqacMfYAlwJLgJEpOqdIjuisITc3f7gu3hpyUVGQXmhoCIJZVVUQjONJNyRzbIJiu9U/+tNZHLJhH2wCxkNrzTBmXxh0qx8xoo+TJHO/ktHxPkyPwEtAGCgDJkZgzfswWLcsnqek3V/AhcB24B+iy9OBV/o6JpnWLF09GD3lQwkcm5/UmqWAZKhVSSZcfnm0FctFP/H2qUW+vxzfb/j+crx9apHPuegnXlzs/rWv9XGSZO9XJOK+YoX7TTcF7/Eet+JE9/Ju3dTLcV/xqfiOj8Egt2b5K7DR3bdG/yCsAY40szTMXba423tyNmzYkJLzdKcRFmVQdNaQH3kEbropeB+0h3mZ1dnH5Jsn3k7Rxg6sGczBmqGosYNr/+mH/fcxSeZ+dT48vegiuOGG4H369Phaw2yKBDXyWGHg5X4GmkpCosH8V8BxZlYJYGafAf7o7oNQ0sFt6rNw4cLki9gDjbAofUqmuVxRUdCC5Prrg/c8DOTwYbf6ob+L4M1dt3kYhr3RHl+3+kTvV1IPm2+Hsm5jjJSVQ9Xt8V07AQnlzN19r5l9GXjEzDqAvcDF/RyWoMFr6nPllVfy+uuvU1NTwxe/+EUee+wxmpubmT59OjfddBNLlixh48aNvPbaa9xyyy20tbXx7W9/m5EjR3LMMcdw6KGH8v3vf59f/vKXXHfddezfv5+zzjqL6667jgsvvJC33nqLs846i7Vr1yZ1ByTPpL25XO6qq4O/7LidtkcvoqT9w4jeVlzOyKm3D24fk2SaY3Y+MO7+33gwCxxPLiYVr+Ry5uvdvbSXU5e6+7NxnudgEydOdPcPR0CMRCI+btw47+jo8B//+Md+0UUXHdj305/+tO/atcvd3W+77TZfsGCB79+/3ydNmuR79uxxd/fZs2f7tm3bDhphcTApZ55jVqwIcrdd8qnlwfp8lWjuufPYTDwnSPa/U+dnvvnmgX/mGOTLqImBaFMfvkDwI6BTCalq6rNq1SpeffVVhg4dSktLC21tQc1/0qRJALz77rtUVlYyatQoAE455RTWrl3Lzp07+cMf/sB5550HwO7du9mxYwejR49OukySpzLQASejkv0lkoGWNEDytevO9E6a/pvmSDAH2E1Q3BAwDNgXXd6d1FkjkQgAP/rRj9i4cSPNzc385Cc/ObC9cyTDww47jK1bt9Lc3Ex5eTnr168H4PDDD+f4449n7dq1B/4QlJaWsn379gMjLIp0kanmcpkSm3uGrrnneANdmgPjgWtm4o9IgnJibJbAAwR585OBJ6LvLSTbqmXKlCmceuqpfOQjH6G6upr6+voe28QPGTKE66+/nsmTJzN9+nSam5spKSkhFArxrW99iylTpnDmmWcemIYudoRFkS4y0AEnozLScSdFcuhhs3maOm5WV1d7Y2Njl3VvvPEGJ5xwQpxn+DwwBbiK4G9QB3A38DzBEJODr729neLoeLzf+MY3qK2t5fOf/3xart2Xgd1HyQodHTlT40vaypVBs77YXyLl5UEzwXxMK6WYmW109+r+9suhNEv3gF0E1Edf6bFgwQI2bdpEe3s7EydO5Pzzz0/btSXPZCJtkCmZaNlRgHIomGfenXfemekiiOSeHMs956ocypmLSC5qaoL5VxZR+dXP8fg/baTyq59j/pVFcc0UJPFTMBeRQdPQACedBIsWQW3tk8ycuZyamhUsWhSsH8S5LQqOgrmIDIqUzRQkcVEwF5FBEcwUZLgHr0mTgnGVOmcKcjfCYaO9PfVTqBWi3ArmGZodPB6XXnopW7Zs4Z133ulzgK1XXnmF3buDjk5r1qzh6aefTlcRRdIqJTMFSdxypzVLhgYncnfMrP8do0aPHn2g41BP7rrrLhYuXEhFRYU6FEle65wpaOXKGZSVtRy0vXOmoOefr0l/4fJQ7gTzVHQJ7sG2bduYP38+I0eOZPv27Rx11FF897vf5Zvf/CZDhgxhypQpzJ49m6985St88MEHHHHEETz00EMMHTqU6667jp///OeMGjWKlpaWA+dbuHAhP/3pT3n77be58soraW5uprKyktraWlavXs3rr7/OVVddRVtbG3v37uWrX/0qTz75JLfffjuhUIiKigoWL17MqFGj+MxnPsO0adN47rnnGDZsGKtWrWLYsGGpuKMig6rLTEGPfoFDDvlwXKXW1hJmz45jpiCJW+4E80EcnOiNN97glVdeYcSIEdx444088sgjvPbaa/zud7+jpKSESy65hBtvvJHx48dz7733snTpUo466ih27drFCy+8QFtbGxMnTjzovPPnz+fWW2+lqqqK/fv3EwqFeP7551m4cCHHH388S5YsAYLBuW699VbWrVtHaWkpjz76KN/73ve444472LlzJxdffDG33HILX//611m7di3nnntuUp9XJB3mzAlasVRU7CYSGUIkEqKtbRhDh+4jEhlCRcVuiovhkksyXdL8kDs580GcHfzUU09lRLR6MHHiREKhEOPHj6ekpASAzZs3c/XVV1NTU8ODDz7Iu+++y29/+1vOPvtsAIYOHcpJJ5100Hl3795NVbR8oVDvt/rNN99kwoQJlJYG+cVp06axZcsWIBjIq7Or/gknnMD777+f9OcVSYfOmYIuu+wBSktb2Lz5ZM4//wk2bz6Z0tIW5s5d3P9MQRK33Anmgzg40auvvsrevcFPwFWrVlFRUXFgtESA4447jiVLlrBhwwZefPFFrrjiCo455hh+8YtfANDa2spLL7100HlDoRBbt24FgnFdAIqKig4aTfHYY4/l17/+Na2trQCsX7+e8ePHA3TJ15sZ6RpLRyRZnTMFNTePZMGC26mubuSZZ85kwoTfsHDhbYTDI+KbKUjikjtplkHsEnzUUUcxZ84c/vrXv1JdXc25557LCy+8cGD79773PebOnQvAyJEjuffee7ngggt44oknOO200xg9enSPA13dc889zJ07l1AoxLhx47j33nupq6vjwgsv5Oabbz6w36hRo6ivr6e2tpaysjJGjx7Nvffem/TnEsm0ujpoalrOmjUwfHjwqKu8vIjW1nrGjatXIE+hHBo1cXDEPrDMVdlwH0VkcMQ7amLupFlERKRXBR/Mx4wZk9O1chERyIJgrgd6ydH9ExHIcDAvKSnhvffeU0BKkLvz3nvvHWhCKSKFK6OtWSorK9mxYwc7d+7MZDFyWklJCZWVlZkuhohkWEaDeXFxMZ/4xCcyWQQRkbyQO+3MRbJJ54TMmzYFvZM1DZpkmIK5yEBlaARPkb5kvDWLSM6JHcHTvesIniIZklQwN7OpZuZmVpGqAolkvb5G8BTJkISDuZkdAvwb8F7qiiOSAwZxBM9s1dQE8+dDZeUeHn98JpWVe5g/X/N3ZpNkauY/An4INPe2g5nNM7NGM2tU80PJG4M4gmc2amiAk04KxiavrX2SmTOXU1OzgkWLgvXKLmWHhAbaMrPvAMPc/d/MbBtQ5e67+zqmp4G2RHJWZ2uWFI/gmW2amoKAHZ1Ii/Xra6mt3cD69bWcccZ6AEpLYfNmDWU7WOIdaGvArVnM7EvAicC/JFIwkbxQVBTMcJXkLFfZrq1tGuHwugPL+/YNBWDy5Bdw/3Cs/S1bzgCeSXfxJEYiaZaFwKeATWb2MvAx4HkzG5fSkolIxl177XcIh0sPLA8b1tblHYKJmevrr0972aSrAQdzdz/B3U909yp3rwL+DHzO3V9PffFEJJMaGmqZMWNll4AeKxwu5ZxzVrF6dU16CyYHUTtzSYpaOeS38nLYsKGW2bOX0tradUC31tYSZs9eynPP1VBenqECygFJB3N3H9Pfw0/JT6lo5aA/BtltzpxgUuaKit1EIkOIREK0tBxCJBIiEhlCRcVuiovhkksyXVLB3dPyOuWUU1zyx9at7qWl7kEXSPf162vcHV+3rvbAutLSYL/ePPVUsE9xsfucOQ+6O37xxQ95cXGw/qmn0vVppDed/53Xr6/xSCTkGzeO92nT1vrGjeM9Egn5unW1/f53luQAjR5HjFWaRRIStHIw3IPXpEkvAh+2cnA3wmGjvX1aj8c3NcGsWUGTt/Z2mDt3MRC8t7cH62fNUg0908aOhWXLoLl5JAsW3E51dSPPPHMmEyb8hoULbyMcHsGyZWqWmA0UzCUhybZySPaPgaRPXR2MG7ecvXuvYfjwEKEQlJcX0dpaz7hxy/O1r1TOSajTUCLUaSi/hEIwdeqzrFw5g7KyloO2d7ZyeP75Gjo6Dj7+nHOe5Wc/6/nY2HN88YurWLWqJoUlF8kt8XYaUs1cEpJsKwc1eRNJLQVzSahFSbKtHNTkTSS1FMwLXKLNC+vrg2B+2WUPUFrawubNJ3P++U+wefPJlJa2MHfuYoqL4eqrez5eTd5EUkvBvIAl06Ik2VYOyf4xEJGuFMwLWLItSpJp5ZCKJm/qcCQSI57G6Kl4qdNQ9jn77PXe3Fzqff2na24u9bPPfnbQyrB1q/vXvuY+YoR7KBS8f+1r/XdCUYcjKRTE2WlITRMLWLLNCzNFY2xLIVHTROlXrrYoUYcjkYMpmBewXG1RojG2E6NnDHkunlxMKl7KmWefXB1Eycy9pqb3fH9zc6lPnfqsh0IZLWZW0TOG3IUG2pL+5OogSrmaHsoUDWpWGBTMC1wuDqKUq+mhTNEzhsKgYC6MHQv33AN79gSTzu/ZEyxnW428kzocDYyeMRQGBXPJObmaHsoUDWpWGBTMJSflYnooU/SMoTAomEvOyrX0UKboGUNhUDCXglRIba71jKEwKJhLwUl02N9cpWcMhUHBPE8UUk0zGYXa5lrPGPKfgnkeKLSaZjIKuc21njHkNwXzHFeoNc1Eqc215CsF8xxXyDXNRKjNteSrhIO5mf27ma0zsxfN7DEzG5nKgkl8VNMcmFxvc61nI9KbZGrmW9z9DHefBLwOfDtFZZIBUE1zYHK5zbWejUhfEg7m7v5IzOJvgKOSL44MVK7XNNMtV9tc69mI9CfpnLmZFQNfB37Ww7Z5ZtZoZo07d+5M9lLSg1yuaWZCrk4krWcj0q94Bj3v7QX8A7AB+Ep/+2pyisGRqxNMZFquTSSdDZNvS2YQ5+QUyQTy84AXgH+KZ38F88Hz1FPuTz55vtfX3+FmHQ7uoVDEr732h/7kk+drFpkU6fzDCcFr/foad8fXras9sG6w/nBqdqXCFW8wTyjNYmYfBW4Hznb336XgB4IkQb370iNVqY5E0jR6NiL9STRnPh44HHjCzDZEX/+TwnLJAKl33+BLRTPQRFuk6NmI9Cue6nsqXkqzSK5LNtWRTJpGz0YKF5rQWSS1kk11JJOm0ciH0h8Fc5E4JZvqSDZNo2cj0qd4qu+peCnNIrku2VSHWqRIIlCaRSS1kk11qEWKDCYFc5EBSCbVoRYpMpgUzEUGKNFmoLk6LozkBgVzkTRRixQZTEMyXQCRQlJXB01Ny1mzBoYPh+bm2DRNvQK5JMyCh6WDr7q62hsbG9NyLRGRfGFmG929ur/9VDOXwtXREfSf37QJxo8Pqs1FRZkulUhCFMylMHV0wPTp8NJLEA5DWRlMnAhr1iigS07SA9Asovkd06ihIQjkzc3BsCjNzcGy5l6THKVgniU0v2OabdoU1MhjhcPw8suZKY9IkhTMs4Dmd8yA8eOD1EqssjKoqspMeUSSpGCeBTS/YwbU1QU58vJyMAveJ05Eo1VJrlIwzwKpmPRABqioKHjY+cgjcNNNwbsefkoOUzvzLBAKwdSpz7Jy5QzKyloO2h4Ol3LOOat4/vkaOjoyUEARyZh425mrZp4FNJqeiCRLwTwLaDQ9EUmWgnkW0Gh6IpIsBfMsoNH0RCRZ6s6fJTSanogkQ61ZRESymFqziIgUEAVzEZE8oGAuIpIHFMxFRPJAwsHczGrN7Ndm9qKZLTezw1JZMBERiV9CwdzMSoD7gFnuPgl4DvheKgsmIiLxS7RmPh14wd3/FF1eBJyXmiKJiMhAJRrMxwBvdS64+wfAEDMrjt3JzOaZWaOZNe7cuTPxUoqISJ8SDebDgEi3dRGgSw8kd7/f3avdvfqII45I8FIiItKfRIP5DuDozgUzKwP2unv3AC8iImmQaDBvAP7ZzD4aXZ4H/HdqiiQiIgOV0EBb7v53M7sSWGFm+wny5/8npSUTEZG4JTxqoruvAlalsCwiIpIg9QAVEckDCuYiInlAwTzFmppg/nyorNzD44/PpLJyD/PnB+tFRAaLgnkKNTTASSfBokVQW/skM2cup6ZmBYsWBesbGjJdQhHJVwrmKdLUBLNmQUsLtLfD3LmLgeC9vT1YP2uWaugiMjgUzFOkrW0a4bDhHrwmTXoRgMmTXziwLhw22tunZbikIpKPFMxT5Nprv0M4XHpgediwti7vAOFwKfX116e9bCKS/xTMU6ShoZYZM1Z2CeixwuFSzjlnFatX16S3YCJSEBTMU6S8HDZsqGX27KW0tpZ02dbaWsLs2Ut57rkaysszVEARyWsK5ikyZw4UF0NFxW4ikSFEIiFaWg4hEgkRiQyhomI3xcVwySWZLqmI5CMF8xSprw+C+WWXPUBpaQubN5/M+ec/webNJ1Na2sLcuYspLoarr850SUUkHymYp8jYsbBsGTQ3j2TBgtuprm7kmWfOZMKE37Bw4W2EwyNYtizYT1KoowNWroSbbw7eOzoyXSKRjEh4oC05WF0dNDUtZ80aGD4cmpuhvLyI1tZ6xo2rVyBPtY4OmD4dXnoJwmEoK4OJE2HNGigqynTpRNLK3L3/vVKgurraGxsb03ItKRArV8JFFwV/NTuVl8Mjj8CMGZkrl0gKmdlGd6/ubz+lWSR3bdoU1MhjhcPw8suZKY9IBimYS+4aPz5IrcQqK4OqqsyURySDFMwld9XVBTny8nIwC94nTgzWixQYPQBJ6PBTAAALAUlEQVSV3FVUFDzsbGgIUitVVUEg18NPKUAK5pLbioqCh5164CkFTmkWEZE8oGAuIpIHFMxFRPKAgrmISB5QMBcRyQMK5iIieUDBvJumJpg/Hyor9/D44zOprNzD/PmaiFlEspuCeYyGBjjpJFi0CGprn2TmzOXU1Kxg0aJgfUNDpksoItKzAQdzMxtmZj8ys7Vm9hszW2RmxYNRuHRqaoJZs6ClBdrbYe7cxUDw3t4erJ81SzV0EclOidTMy4AV7n6Wu08AhgJfTm2x0q+tbRrhsOEevCZNehGAyZNfOLAuHDba26dluKQiIgcbcDB39/fdfW3MqkbgqJ72NbN5ZtZoZo07d+5MtIxpce213yEcLj2wPGxYW5d3gHC4lPr669NeNhGR/iSVMzezEcC/Ao/3tN3d73f3anevPuKII5K51KBraKhlxoyVXQJ6rHC4lHPOWcXq1TXpLZiISBz6DeZmdqSZbYh5HR1dXw2sBW50902DXdDBVl4OGzbUMnv2UlpbS7psa20tYfbspTz3XA3l5RkqoIhIH/odNdHd/wrUxK4zs68A/wv4grtvH5yipdecOUErloqK3UQiQ4hEQrS1DWPo0H1EIkOoqNhNcTFcckmmSyoicrBEWrOcRPDAc0a+BHKA+nooLobLLnuA0tIWNm8+mfPPf4LNm0+mtLSFuXMXU1wMV1+d6ZKKiBwskfHMJwEfA542s851r7v7/JSVKgPGjoVly6C5eSQLFtzOnXdehXuICRNO55pr7mbKlOdZtizYT1KsoyNoxL9pUzAVnCaYEBmwAQdzd78PuG8QypJxdXXQ1LScNWtg+PBg0vfy8iJaW+sZN65egXwwdHTA9Onw0kvBZMxlZcHUb2vWKKCLDIC5e1ouVF1d7Y2NjWm5luSQlSvhoouCv5ydysvhkUc0e5AIYGYb3b26v/3UnV8ya9OmoEYeKxwO5vQUkbgpmEtmjR8fpFZilZUFkzOLSNwUzCWz6uqCHHl5OZgF7xMnButFJG6JtGYRSZ2iouBhZ0NDkFqpqlJrFpEEKJhL5hUVBQ879cBTJGF5mWbRBBMiUmjyLphrggkRKUR5Fcw1wYSIFKq8ypkHE0ysO7C8b99Q4MMJJjpt2XIG8Ey6i5e/1B1fJOPyqmauCSYyoLM7/kUXwQ03BO/TpwfrRSRt8iqYa4KJJHR0BF3rb745eI83GDc0BOOqNDeDe/D+0kt6OCGSZnkVzDXBRIKSqV2rO75IVsjaYN7ZvHDECAiFgvf+mhfOmROMSR47wURLyyFEIiFNMNGXZGrX6o4vkhWyMpjHNi/84IMgvnzwAf02L9QEEwlKpnat7vgiWSHrWrPENi/srr09eM2aBZs3HzxRhCaYSFBn7Tp2GNp4a9fqji+SFbKuZn7HHUHA7kt7O9x1V8/b6upg3Ljl7N17DcOHhwiFYieYWK4KY0+SrV13dse//vrgXYFcJO2ybnKKESOClEo8++3Zk4KCSaCzrbhq1yJZJd7JKbIuzRL7Sz8V+0mcNNiVSE7LumBeXh5fzVzNC3ugnpgiBSvrgvmcOUGrlb7y5mpe2ANNjCxS0LLuAWhn88K+qHlhD9QTU6SgZV0w72xeWFp6cFAvLg7Wq3lhD9QTU6SgZV0whyDVu3kzzJvXtQfovHnB+rxuXpjoGCnqiSlS0LKuaWJBSybvrZy5SF7K2aaJBS027w1d8979NRlUT0yRgqZgnk36ynvH0/5bbcVFClZSOXMz+5KZ7U5VYQqe8t4ikqCEg7mZfQT43yksi2gEQhFJUEJpFjMrAhYB1wDP9bHfPGAewNFHH53IpQqL8t4ikqCEWrOY2f3Ar9x9sZntdveK/o5RaxYRkYFLWWsWMzsS+GnMqmeBv7v74iTKl980RoqIpFm/wdzd/wrUAJhZOfAm8K6ZTY/uMtzMXgZq3D0/HoYmE4zV3ltEMmBAOXN3bwaOil0XTbPkT3OLZINxMm3FRUQSlJXd+VMi0W7xyQ5YpTFSRCQDku40FM/Dz7RLpnadbMedZObTFBFJUH7WzJOpXSfbcUdtxUUkA/KzO38ytevOYNy9Vj+QyY3VVlxE0iw/g3kyqY5UBGONkSIiaZafwTwVtWsFYxHJIfkZzJXqEJECk5/BHFS7FpGCkp+tWURECoyCuYhIHlAwFxHJAwrmIiJ5QMFcRCQPJDQ5RUIXMtsJ/DEtF8usw4FdmS5EltM96pvuT/8K6R4d4+5H9LdT2oJ5oTCzxnhmBSlkukd90/3pn+7RwZRmERHJAwrmIiJ5QME89e7PdAFygO5R33R/+qd71I1y5iIieUA1cxGRPKBgLiKSBxTMRUTygIL5AJjZZDN70cyqostHm9ljZva0mW0ysyt6Oe4pM/u1mW2Ivj6b3pKnTw/36FgzW21m681so5ld2stxXzSz30aPXWJmJWkteJokcX8K9jvUbduPzWx5L8cVxHeoN/k7nnmKmdmDwHBgRMzqjwIL3H2rmZUCG81svbu/3u3ww4AL3H17moqbEb3co78B57p7u5mNBN42s4fdPRJz3FHAjcAkd99tZv8BXAV8P32lH3yJ3p+oQv4OdW6rBU4B3uphW0F8h/qimnn8vuruM4npQuzuv3H3rdF/twBvAEf2cOyhFEbX457uUdjd26OLxwFv9BCovgj81N13R5fvBz4/6KVNv0TvDxTwdwjAzI4kCNbf7uW4QvkO9Uo18zhFg3WvzOxEYBzwy152WW1mIWAlcLu7709xETOut3tkZrcB5xF832b1sMsYYGPM8h+B0akuX6YlcX86FeR3yMzKgaXAlQS/UHoyhgL4DvVFNfMUMLOLgUXA5929tft2d/9Hd58KzAA+A8xPcxEzyt2/5e7HAzOBn5nZx7vtMgyIrY12AHkXqHoTx/0p2O+QmRUDPwN+6O6b+9i1oL9DoGCeFDMLmdkSYCJwurtv6Wt/d98DLAZOTUPxso67vwo8z8GffwdwdMzyMcC2NBUra/Rxf2L3KbTvUC3wKeBmM3uZoNJUa2bLuu1X8N8hBfPkzAea3f3rffyEHmZmw6P/HgLMpvdUTN4xsxPMrCj678MIglBjt90eAy4xs7Lo8uXAQ+krZebEc38K+Tvk7mvd/ePuXuXuVcCXgWfdvXs6qmC/Q52UM0/OZ4FPmdmGmHX/TVB7uB+4huDn3xozCwPFwGrgvjSXM5MmA4+Y2W6Cn73XuPsfo7nf+6PLvzezu4ENZrYf+A3w48wVOa36vT/oO9QjfYe60tgsIiJ5QGkWEZE8oGAuIpIHFMxFRPKAgrmISB5QMBcRyQMK5iIieUDBXPKCmT1qZtuir4iZ/SVmeU50/BORvKV25pJ3zGwbMMfdf5Hpsoiki2rmIiJ5QMFc8p6Z3Whmi6L/HhNNw3zBzF4xs7+b2W1mVmlma8xsh5m9amafiTm+xMzuNLM/mNnbZnZfdDISkayhYC6FqAiYAFQBJwKXAk8D33L3SuBh4N6Y/f8v8BGC0fs+CYwCbkhjeUX6pWAuheoHHngHWAs87+6vRLf9jGAAtZCZHU4wSuHl7r4vOivQbfQ9iYRI2mnURClEHe7+XsxyM/DXmOUPCP7fGAp8gqAm/6qZdW4PEYxkKJI1FMxF+vZnYC9wbD5O0yb5Q2kWkT5E0zC/BG6NTmGGmX089gGpSDZQMBfp38VAJdBkZlsJJiDR/zuSVdRpSEQkD6h2ISKSBxTMRUTygIK5iEgeUDAXEckDCuYiInlAwVxEJA8omIuI5AEFcxGRPPD/Ac7qu9/EPLBIAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.title(\"Testing the Model\", fontsize=14)\n",
    "plt.plot(t_instance[:-1], time_series(t_instance[:-1]), \"bo\", markersize=10, label=\"instance\")\n",
    "plt.plot(t_instance[1:], time_series(t_instance[1:]), \"w*\", markersize=10, label=\"target\", color='yellow')\n",
    "plt.plot(t_instance[1:], y_pred[0,:,0], \"r.\", markersize=10, label=\"prediction\")\n",
    "plt.legend(loc=\"upper left\")\n",
    "plt.xlabel(\"Time\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 마무리\n",
    "\n",
    "이번 포스팅에서는 RNN을 학습시키는 방법인 BPTT와 텐서플로를 이용해 MNIST 분류기와 시계열 데이터를 예측하는 RNN 모델을 구현해 보았다. 그리고 심층 RNN을 구현하는 방법과 RNN에 드롭아웃을 적용하는 방법에 대해 알아보았다.\n",
    "\n",
    "다음 포스팅에서는 타임스텝이 매우 긴 RNN의 학습단계에서의 문제점과 이를 보완한 RNN의 변형인 LSTM과 GRU에 대해 알아보도록 하자. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tensorflow",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}