Ch10_Recurrent_Neural_Networks/Recurrent_Neural_Networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recurrent Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In  n-gram models the conditional probability of word w<sub>t</sub> at position t only\n",
    "depends on the n - 1 previous words. If we want to check the possible effect of words earlier than t - (n - 1) on wt,\n",
    "we need to increase n. However, the number of model parameters would also increase exponentially with it, as we need\n",
    "to store |V|<sup>n</sup> numbers for a vocabulary V . Hence, rather than modeling p(w<sub>t</sub>|w<sub>t-1</sub>; . . . ;w<sub>t-n+1</sub>) it is preferable to use a\n",
    "latent variable model in which we have"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$p(x_t|x_{t-1}, \\ldots x_1) \\approx p(x_t|x_{t-1}, h_{t}).$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a sufficiently powerful function h<sub>t</sub> this is not an approximation. After all, ht could simply store all the data it observed\n",
    "so far. Let’s see why building such models is a bit more tricky than simple autoregressive\n",
    "models where"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$h_t = f(x_{t-1}, h_{t-1}).$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a warmup we will review the latter for discrete outputs and n = 2, i.e. for Markov model of first order. To simplify\n",
    "things further we use a single layer in the design of the RNN. Later on we will see how to add more expressivity efficiently\n",
    "across items."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recurrent Networks Without Hidden States"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us take a look at a multilayer perceptron with a single hidden layer. Given a mini-batch of instances X ~R<sup>n * d</sup> with\n",
    "sample size n and d inputs (features or feature vector dimensions). Let the hidden layer’s activation function be ϕ. Hence\n",
    "the hidden layer’s output H~R<sup>n * h</sup> is calculated as"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\mathbf{H} = \\phi(\\mathbf{X} \\mathbf{W}_{xh} + \\mathbf{b}_h).$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we have the weight parameter W<sub>xh</sub> ~ R<sup>d * h</sup>, bias parameter b<sub>h</sub> ~ R<sup>1 * h</sup>, and the number of hidden units h, for\n",
    "the hidden layer. Recall that bh is just a vector - its values are replicated using the broadcasting mechanism \n",
    "to match those of the matrix-matrix product."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also note that hidden state and hidden layer refer to two very different concepts. Hidden layers are, as explained, layers\n",
    "that are hidden from view on the path from input to output. Hidden states are technically speaking inputs to whatever we\n",
    "do at a given step. Instead, they can only be computed by looking at data at previous iterations. In this sense they have\n",
    "much in common with latent variable models in statistics, such as clustering or topic models where e.g. the cluster ID\n",
    "affects the output but cannot be directly observed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The hidden variable <b>H</b> is used as the input of the output layer. For classification purposes, such as predicting the next\n",
    "character, the output dimensionality q might e.g. match the number of categories in the classification problem. Lastly the\n",
    "output layer is given by"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\mathbf{O} = \\mathbf{H} \\mathbf{W}_{hq} + \\mathbf{b}_q.$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, O ~ R<sup>n * q</sup> is the output variable, W<sub>hq</sub> ~ R<sup>h * q</sup> is the weight parameter, and b<sub>q</sub> ~ R<sup>1 * q</sup> is the bias parameter of\n",
    "the output layer. If it is a classification problem, we can use softmax(O) to compute the probability distribution of the\n",
    "output category.  we can pick (w<sub>t</sub>;w<sub>t-1</sub>) pairs at random and estimate the parameters W and b of our network\n",
    "via autograd and stochastic gradient descent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recurrent Networks with Hidden States"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Matters are entirely different when we have hidden states. Let’s look at the structure in some more detail. Assume that\n",
    "X<sub>t</sub> ~ R<sup>n * d</sup> is the mini-batch input and  H<sub>t</sub> ~ R<sup>n * h</sup> is the hidden variable of time step t from the sequence. Unlike the\n",
    "multilayer perceptron, here we save the hidden variable H<sub>t-1</sub> from the previous time step and introduce a new weight\n",
    "parameter W<sub>hh</sub> ~ R<sup>h * h</sup>, to describe how to use the hidden variable of the previous time step in the current time step.\n",
    "Specifically, the calculation of the hidden variable of the current time step is determined by the input of the current time\n",
    "step together with the hidden variable of the previous time step:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\mathbf{H}t = \\phi(\\mathbf{X}t \\mathbf{W}{xh} + \\mathbf{H}{t-1} \\mathbf{W}_{hh} + \\mathbf{b}_h).$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compared with the multilayer perceptron, we added one more H<sub>t-1</sub>W<sub>hh</sub> here. From the relationship between hidden\n",
    "variables H<sub>t</sub> and H<sub>t-1</sub> of adjacent time steps, we know that those variables captured and retained the sequence’s historical\n",
    "information up to the current time step, just like the state or memory of the neural network’s current time step. Therefore,\n",
    "such a hidden variable is also called a hidden state. Since the hidden state uses the same definition of the previous time\n",
    "step in the current time step, the computation of the equation above is recurrent, hence the name recurrent neural network\n",
    "(RNN)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many different RNN construction methods. RNNs with a hidden state defined by the equation above are very\n",
    "common. For time step t, the output of the output layer is similar to the computation in the multilayer perceptron:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\mathbf{O}_t = \\mathbf{H}t \\mathbf{W}{hq} + \\mathbf{b}_q$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "RNN parameters include the weight W<sub>xh</sub> ~ R<sup>d * h</sup>; W<sub>hh</sub> ~ R<sup>h * h </sup>of the hidden layer with the bias b<sub>h</sub> ~ R<sup>1 * h</sup>, and the\n",
    "weight W<sub>hq</sub> ~ R<sup>h * q</sup> of the output layer with the bias b<sub>q</sub> ~ R<sup>1 * q</sup> . It is worth mentioning that RNNs always use these\n",
    "model parameters, even for different time steps. Therefore, the number of RNN model parameters does not grow as the\n",
    "number of time steps increases."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The figure below shows the computational logic of an RNN at three adjacent time steps. In time step t, the computation\n",
    "of the hidden state can be treated as an entry of a fully connected layer with the activation function ϕ after concatenating\n",
    "the input X<sub>t</sub> with the hidden state H<sub>t-1</sub> of the previous time step. The output of the fully connected layer is the hidden\n",
    "state of the current time step H<sub>t</sub>. Its model parameter is the concatenation of W<sub>xh</sub> and W<sub>hh</sub>, with a bias of b<sub>h</sub>. The\n",
    "hidden state of the current time step t H<sub>t</sub> will participate in computing the hidden state H<sub>t+1</sub> of the next time step t+1,\n",
    "the result of which will become the input for the fully connected output layer of the current time step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"175pt\" version=\"1.1\" viewBox=\"0 0 405 175\" width=\"405pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<defs>\n",
       "<g>\n",
       "<symbol id=\"glyph0-0\" overflow=\"visible\">\n",
       "<path d=\"M 1.125 0 L 1.125 -5.625 L 5.625 -5.625 L 5.625 0 Z M 1.265625 -0.140625 L 5.484375 -0.140625 L 5.484375 -5.484375 L 1.265625 -5.484375 Z M 1.265625 -0.140625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-1\" overflow=\"visible\">\n",
       "<path d=\"M 4.859375 -2.828125 C 4.859375 -2.304688 4.757812 -1.820312 4.5625 -1.375 C 4.34375 -0.882812 4.023438 -0.515625 3.609375 -0.265625 C 3.273438 -0.0546875 2.914062 0.0625 2.53125 0.09375 L 2.1875 1.78125 L 1.390625 1.78125 L 1.765625 0.03125 C 1.628906 -0.0078125 1.492188 -0.0625 1.359375 -0.125 C 0.742188 -0.445312 0.4375 -0.992188 0.4375 -1.765625 C 0.4375 -2.679688 0.707031 -3.4375 1.25 -4.03125 C 1.644531 -4.46875 2.144531 -4.710938 2.75 -4.765625 L 3.109375 -6.4375 L 3.90625 -6.4375 L 3.546875 -4.703125 C 3.859375 -4.628906 4.125 -4.476562 4.34375 -4.25 C 4.6875 -3.894531 4.859375 -3.421875 4.859375 -2.828125 Z M 4.078125 -2.828125 C 4.078125 -3.253906 3.96875 -3.582031 3.75 -3.8125 C 3.65625 -3.914062 3.539062 -4 3.40625 -4.0625 L 2.671875 -0.546875 C 2.910156 -0.609375 3.128906 -0.734375 3.328125 -0.921875 C 3.597656 -1.191406 3.800781 -1.546875 3.9375 -1.984375 C 4.03125 -2.285156 4.078125 -2.566406 4.078125 -2.828125 Z M 2.625 -4.125 C 2.28125 -4.039062 1.976562 -3.816406 1.71875 -3.453125 C 1.394531 -3.015625 1.234375 -2.476562 1.234375 -1.84375 C 1.234375 -1.414062 1.335938 -1.085938 1.546875 -0.859375 C 1.648438 -0.742188 1.765625 -0.660156 1.890625 -0.609375 Z M 2.625 -4.125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-0\" overflow=\"visible\">\n",
       "<path d=\"M 1.125 0 L 1.125 -5.625 L 5.625 -5.625 L 5.625 0 Z M 1.265625 -0.140625 L 5.484375 -0.140625 L 5.484375 -5.484375 L 1.265625 -5.484375 Z M 1.265625 -0.140625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-1\" overflow=\"visible\">\n",
       "<path d=\"M 1.4375 0 L -0.265625 0 L 2.578125 -3.265625 L 0.96875 -6.4375 L 2.359375 -6.4375 L 2.984375 -5.234375 C 3.003906 -5.191406 3.148438 -4.894531 3.421875 -4.34375 C 3.429688 -4.3125 3.445312 -4.269531 3.46875 -4.21875 C 3.832031 -4.65625 4.132812 -5.015625 4.375 -5.296875 L 5.390625 -6.4375 L 7.046875 -6.4375 L 4.109375 -3.046875 L 5.6875 0 L 4.203125 0 L 3.71875 -0.96875 C 3.457031 -1.476562 3.289062 -1.847656 3.21875 -2.078125 C 3.101562 -1.898438 2.804688 -1.539062 2.328125 -1 Z M 1.4375 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-2\" overflow=\"visible\">\n",
       "<path d=\"M 4.8125 -2.875 L 2.328125 -2.875 L 1.71875 0 L 0.390625 0 L 1.734375 -6.4375 L 3.0625 -6.4375 L 2.546875 -3.953125 L 5.03125 -3.953125 L 5.5625 -6.4375 L 6.875 -6.4375 L 5.53125 0 L 4.203125 0 Z M 4.8125 -2.875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-0\" overflow=\"visible\">\n",
       "<path d=\"M 0.875 0 L 0.875 -4.375 L 4.375 -4.375 L 4.375 0 Z M 0.984375 -0.109375 L 4.265625 -0.109375 L 4.265625 -4.265625 L 0.984375 -4.265625 Z M 0.984375 -0.109375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-1\" overflow=\"visible\">\n",
       "<path d=\"M 1.671875 -0.5 L 1.578125 0 C 1.429688 0.0390625 1.285156 0.0625 1.140625 0.0625 C 0.898438 0.0625 0.707031 0 0.5625 -0.125 C 0.445312 -0.207031 0.390625 -0.332031 0.390625 -0.5 C 0.390625 -0.570312 0.421875 -0.753906 0.484375 -1.046875 L 0.921875 -3.15625 L 0.4375 -3.15625 L 0.53125 -3.625 L 1.015625 -3.625 L 1.203125 -4.515625 L 1.921875 -4.953125 L 1.640625 -3.625 L 2.25 -3.625 L 2.140625 -3.15625 L 1.546875 -3.15625 L 1.125 -1.140625 C 1.070312 -0.890625 1.046875 -0.738281 1.046875 -0.6875 C 1.046875 -0.613281 1.066406 -0.554688 1.109375 -0.515625 C 1.148438 -0.484375 1.21875 -0.46875 1.3125 -0.46875 C 1.445312 -0.46875 1.566406 -0.476562 1.671875 -0.5 Z M 1.671875 -0.5 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-2\" overflow=\"visible\">\n",
       "<path d=\"M 0.328125 -1.5 L 0.453125 -2.125 L 2.34375 -2.125 L 2.21875 -1.5 Z M 0.328125 -1.5 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-3\" overflow=\"visible\">\n",
       "<path d=\"M 1.6875 0 L 2.484375 -3.78125 C 2.140625 -3.507812 1.65625 -3.296875 1.03125 -3.140625 L 1.15625 -3.703125 C 1.457031 -3.828125 1.757812 -3.988281 2.0625 -4.1875 C 2.363281 -4.382812 2.585938 -4.554688 2.734375 -4.703125 C 2.828125 -4.796875 2.914062 -4.90625 3 -5.03125 L 3.359375 -5.03125 L 2.3125 0 Z M 1.6875 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-4\" overflow=\"visible\">\n",
       "<path d=\"M 2 -0.8125 L 2 -2.1875 L 0.625 -2.1875 L 0.625 -2.765625 L 2 -2.765625 L 2 -4.125 L 2.578125 -4.125 L 2.578125 -2.765625 L 3.9375 -2.765625 L 3.9375 -2.1875 L 2.578125 -2.1875 L 2.578125 -0.8125 Z M 2 -0.8125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-0\" overflow=\"visible\">\n",
       "<path d=\"M 1.125 0 L 1.125 -5.625 L 5.625 -5.625 L 5.625 0 Z M 1.265625 -0.140625 L 5.484375 -0.140625 L 5.484375 -5.484375 L 1.265625 -5.484375 Z M 1.265625 -0.140625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-1\" overflow=\"visible\">\n",
       "<path d=\"M 0.71875 0 L 0.71875 -6.4375 L 1.578125 -6.4375 L 1.578125 -3.796875 L 4.921875 -3.796875 L 4.921875 -6.4375 L 5.78125 -6.4375 L 5.78125 0 L 4.921875 0 L 4.921875 -3.03125 L 1.578125 -3.03125 L 1.578125 0 Z M 0.71875 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-2\" overflow=\"visible\">\n",
       "<path d=\"M 0.59375 -5.53125 L 0.59375 -6.4375 L 1.390625 -6.4375 L 1.390625 -5.53125 Z M 0.59375 0 L 0.59375 -4.671875 L 1.390625 -4.671875 L 1.390625 0 Z M 0.59375 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-3\" overflow=\"visible\">\n",
       "<path d=\"M 3.625 0 L 3.625 -0.59375 C 3.320312 -0.125 2.882812 0.109375 2.3125 0.109375 C 1.945312 0.109375 1.609375 0.00390625 1.296875 -0.203125 C 0.984375 -0.410156 0.738281 -0.695312 0.5625 -1.0625 C 0.394531 -1.425781 0.3125 -1.847656 0.3125 -2.328125 C 0.3125 -2.796875 0.390625 -3.21875 0.546875 -3.59375 C 0.703125 -3.976562 0.929688 -4.269531 1.234375 -4.46875 C 1.546875 -4.664062 1.894531 -4.765625 2.28125 -4.765625 C 2.5625 -4.765625 2.8125 -4.707031 3.03125 -4.59375 C 3.25 -4.476562 3.425781 -4.320312 3.5625 -4.125 L 3.5625 -6.4375 L 4.359375 -6.4375 L 4.359375 0 Z M 1.125 -2.328125 C 1.125 -1.734375 1.25 -1.285156 1.5 -0.984375 C 1.75 -0.691406 2.046875 -0.546875 2.390625 -0.546875 C 2.734375 -0.546875 3.023438 -0.6875 3.265625 -0.96875 C 3.515625 -1.25 3.640625 -1.679688 3.640625 -2.265625 C 3.640625 -2.898438 3.515625 -3.367188 3.265625 -3.671875 C 3.015625 -3.972656 2.710938 -4.125 2.359375 -4.125 C 2.003906 -4.125 1.707031 -3.976562 1.46875 -3.6875 C 1.238281 -3.394531 1.125 -2.941406 1.125 -2.328125 Z M 1.125 -2.328125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-4\" overflow=\"visible\">\n",
       "<path d=\"M 3.78125 -1.5 L 4.609375 -1.40625 C 4.472656 -0.925781 4.226562 -0.550781 3.875 -0.28125 C 3.53125 -0.0195312 3.085938 0.109375 2.546875 0.109375 C 1.867188 0.109375 1.328125 -0.0976562 0.921875 -0.515625 C 0.523438 -0.941406 0.328125 -1.535156 0.328125 -2.296875 C 0.328125 -3.078125 0.53125 -3.679688 0.9375 -4.109375 C 1.34375 -4.546875 1.867188 -4.765625 2.515625 -4.765625 C 3.140625 -4.765625 3.644531 -4.550781 4.03125 -4.125 C 4.425781 -3.707031 4.625 -3.113281 4.625 -2.34375 C 4.625 -2.289062 4.625 -2.21875 4.625 -2.125 L 1.140625 -2.125 C 1.171875 -1.613281 1.316406 -1.222656 1.578125 -0.953125 C 1.835938 -0.679688 2.164062 -0.546875 2.5625 -0.546875 C 2.851562 -0.546875 3.097656 -0.617188 3.296875 -0.765625 C 3.503906 -0.921875 3.664062 -1.164062 3.78125 -1.5 Z M 1.1875 -2.78125 L 3.796875 -2.78125 C 3.765625 -3.175781 3.664062 -3.472656 3.5 -3.671875 C 3.25 -3.972656 2.921875 -4.125 2.515625 -4.125 C 2.148438 -4.125 1.84375 -4 1.59375 -3.75 C 1.351562 -3.507812 1.21875 -3.1875 1.1875 -2.78125 Z M 1.1875 -2.78125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-5\" overflow=\"visible\">\n",
       "<path d=\"M 0.59375 0 L 0.59375 -4.671875 L 1.3125 -4.671875 L 1.3125 -4 C 1.644531 -4.507812 2.140625 -4.765625 2.796875 -4.765625 C 3.078125 -4.765625 3.332031 -4.710938 3.5625 -4.609375 C 3.800781 -4.515625 3.976562 -4.382812 4.09375 -4.21875 C 4.207031 -4.0625 4.289062 -3.867188 4.34375 -3.640625 C 4.375 -3.492188 4.390625 -3.238281 4.390625 -2.875 L 4.390625 0 L 3.59375 0 L 3.59375 -2.84375 C 3.59375 -3.164062 3.5625 -3.40625 3.5 -3.5625 C 3.4375 -3.71875 3.328125 -3.84375 3.171875 -3.9375 C 3.015625 -4.039062 2.832031 -4.09375 2.625 -4.09375 C 2.289062 -4.09375 2 -3.984375 1.75 -3.765625 C 1.507812 -3.554688 1.390625 -3.148438 1.390625 -2.546875 L 1.390625 0 Z M 0.59375 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-6\" overflow=\"visible\">\n",
       "<path d=\"M 0.28125 -1.390625 L 1.0625 -1.515625 C 1.101562 -1.203125 1.222656 -0.960938 1.421875 -0.796875 C 1.628906 -0.628906 1.910156 -0.546875 2.265625 -0.546875 C 2.628906 -0.546875 2.898438 -0.617188 3.078125 -0.765625 C 3.253906 -0.910156 3.34375 -1.082031 3.34375 -1.28125 C 3.34375 -1.46875 3.265625 -1.609375 3.109375 -1.703125 C 2.992188 -1.773438 2.722656 -1.867188 2.296875 -1.984375 C 1.710938 -2.128906 1.304688 -2.253906 1.078125 -2.359375 C 0.859375 -2.460938 0.691406 -2.609375 0.578125 -2.796875 C 0.460938 -2.984375 0.40625 -3.191406 0.40625 -3.421875 C 0.40625 -3.628906 0.453125 -3.820312 0.546875 -4 C 0.640625 -4.175781 0.769531 -4.328125 0.9375 -4.453125 C 1.0625 -4.535156 1.226562 -4.609375 1.4375 -4.671875 C 1.65625 -4.734375 1.882812 -4.765625 2.125 -4.765625 C 2.488281 -4.765625 2.804688 -4.710938 3.078125 -4.609375 C 3.359375 -4.503906 3.566406 -4.363281 3.703125 -4.1875 C 3.835938 -4.007812 3.929688 -3.769531 3.984375 -3.46875 L 3.203125 -3.359375 C 3.171875 -3.597656 3.066406 -3.785156 2.890625 -3.921875 C 2.722656 -4.054688 2.488281 -4.125 2.1875 -4.125 C 1.820312 -4.125 1.5625 -4.0625 1.40625 -3.9375 C 1.25 -3.820312 1.171875 -3.679688 1.171875 -3.515625 C 1.171875 -3.410156 1.203125 -3.320312 1.265625 -3.25 C 1.328125 -3.15625 1.429688 -3.082031 1.578125 -3.03125 C 1.648438 -3 1.878906 -2.929688 2.265625 -2.828125 C 2.828125 -2.679688 3.21875 -2.5625 3.4375 -2.46875 C 3.664062 -2.375 3.84375 -2.234375 3.96875 -2.046875 C 4.09375 -1.867188 4.15625 -1.644531 4.15625 -1.375 C 4.15625 -1.101562 4.078125 -0.851562 3.921875 -0.625 C 3.765625 -0.394531 3.539062 -0.210938 3.25 -0.078125 C 2.96875 0.046875 2.640625 0.109375 2.265625 0.109375 C 1.660156 0.109375 1.195312 -0.015625 0.875 -0.265625 C 0.5625 -0.523438 0.363281 -0.898438 0.28125 -1.390625 Z M 0.28125 -1.390625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-7\" overflow=\"visible\">\n",
       "<path d=\"M 2.328125 -0.703125 L 2.4375 -0.015625 C 2.207031 0.0351562 2.007812 0.0625 1.84375 0.0625 C 1.550781 0.0625 1.328125 0.015625 1.171875 -0.078125 C 1.015625 -0.171875 0.898438 -0.289062 0.828125 -0.4375 C 0.765625 -0.582031 0.734375 -0.890625 0.734375 -1.359375 L 0.734375 -4.046875 L 0.15625 -4.046875 L 0.15625 -4.671875 L 0.734375 -4.671875 L 0.734375 -5.828125 L 1.53125 -6.296875 L 1.53125 -4.671875 L 2.328125 -4.671875 L 2.328125 -4.046875 L 1.53125 -4.046875 L 1.53125 -1.328125 C 1.53125 -1.097656 1.539062 -0.953125 1.5625 -0.890625 C 1.59375 -0.828125 1.640625 -0.773438 1.703125 -0.734375 C 1.765625 -0.691406 1.851562 -0.671875 1.96875 -0.671875 C 2.0625 -0.671875 2.179688 -0.679688 2.328125 -0.703125 Z M 2.328125 -0.703125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-8\" overflow=\"visible\">\n",
       "<path d=\"M 3.640625 -0.578125 C 3.347656 -0.328125 3.066406 -0.148438 2.796875 -0.046875 C 2.523438 0.0546875 2.234375 0.109375 1.921875 0.109375 C 1.410156 0.109375 1.015625 -0.015625 0.734375 -0.265625 C 0.460938 -0.515625 0.328125 -0.835938 0.328125 -1.234375 C 0.328125 -1.460938 0.378906 -1.671875 0.484375 -1.859375 C 0.585938 -2.046875 0.722656 -2.195312 0.890625 -2.3125 C 1.054688 -2.425781 1.242188 -2.515625 1.453125 -2.578125 C 1.609375 -2.609375 1.84375 -2.644531 2.15625 -2.6875 C 2.800781 -2.757812 3.273438 -2.851562 3.578125 -2.96875 C 3.578125 -3.070312 3.578125 -3.140625 3.578125 -3.171875 C 3.578125 -3.492188 3.503906 -3.71875 3.359375 -3.84375 C 3.148438 -4.03125 2.847656 -4.125 2.453125 -4.125 C 2.078125 -4.125 1.800781 -4.054688 1.625 -3.921875 C 1.445312 -3.796875 1.316406 -3.566406 1.234375 -3.234375 L 0.46875 -3.328125 C 0.53125 -3.660156 0.640625 -3.925781 0.796875 -4.125 C 0.960938 -4.332031 1.195312 -4.488281 1.5 -4.59375 C 1.8125 -4.707031 2.164062 -4.765625 2.5625 -4.765625 C 2.96875 -4.765625 3.289062 -4.71875 3.53125 -4.625 C 3.78125 -4.53125 3.960938 -4.410156 4.078125 -4.265625 C 4.203125 -4.128906 4.285156 -3.953125 4.328125 -3.734375 C 4.359375 -3.597656 4.375 -3.359375 4.375 -3.015625 L 4.375 -1.953125 C 4.375 -1.222656 4.390625 -0.757812 4.421875 -0.5625 C 4.453125 -0.363281 4.519531 -0.175781 4.625 0 L 3.796875 0 C 3.710938 -0.164062 3.660156 -0.359375 3.640625 -0.578125 Z M 3.578125 -2.34375 C 3.285156 -2.226562 2.851562 -2.128906 2.28125 -2.046875 C 1.957031 -1.992188 1.726562 -1.9375 1.59375 -1.875 C 1.457031 -1.820312 1.351562 -1.738281 1.28125 -1.625 C 1.207031 -1.507812 1.171875 -1.382812 1.171875 -1.25 C 1.171875 -1.039062 1.25 -0.863281 1.40625 -0.71875 C 1.5625 -0.582031 1.796875 -0.515625 2.109375 -0.515625 C 2.410156 -0.515625 2.679688 -0.582031 2.921875 -0.71875 C 3.160156 -0.851562 3.335938 -1.035156 3.453125 -1.265625 C 3.535156 -1.441406 3.578125 -1.703125 3.578125 -2.046875 Z M 3.578125 -2.34375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-9\" overflow=\"visible\">\n",
       "<path d=\"M 0.734375 0 L 0.734375 -6.4375 L 5.078125 -6.4375 L 5.078125 -5.6875 L 1.59375 -5.6875 L 1.59375 -3.6875 L 4.609375 -3.6875 L 4.609375 -2.921875 L 1.59375 -2.921875 L 1.59375 0 Z M 0.734375 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-10\" overflow=\"visible\">\n",
       "<path d=\"M 5.296875 -2.265625 L 6.140625 -2.046875 C 5.960938 -1.347656 5.640625 -0.8125 5.171875 -0.4375 C 4.710938 -0.0703125 4.144531 0.109375 3.46875 0.109375 C 2.78125 0.109375 2.21875 -0.03125 1.78125 -0.3125 C 1.34375 -0.59375 1.007812 -1 0.78125 -1.53125 C 0.5625 -2.070312 0.453125 -2.648438 0.453125 -3.265625 C 0.453125 -3.941406 0.578125 -4.53125 0.828125 -5.03125 C 1.085938 -5.53125 1.453125 -5.90625 1.921875 -6.15625 C 2.398438 -6.414062 2.921875 -6.546875 3.484375 -6.546875 C 4.128906 -6.546875 4.671875 -6.378906 5.109375 -6.046875 C 5.554688 -5.722656 5.863281 -5.265625 6.03125 -4.671875 L 5.1875 -4.484375 C 5.039062 -4.953125 4.828125 -5.289062 4.546875 -5.5 C 4.265625 -5.71875 3.90625 -5.828125 3.46875 -5.828125 C 2.976562 -5.828125 2.566406 -5.707031 2.234375 -5.46875 C 1.898438 -5.226562 1.664062 -4.90625 1.53125 -4.5 C 1.394531 -4.101562 1.328125 -3.695312 1.328125 -3.28125 C 1.328125 -2.726562 1.40625 -2.242188 1.5625 -1.828125 C 1.726562 -1.421875 1.976562 -1.117188 2.3125 -0.921875 C 2.644531 -0.722656 3.007812 -0.625 3.40625 -0.625 C 3.882812 -0.625 4.285156 -0.757812 4.609375 -1.03125 C 4.941406 -1.3125 5.171875 -1.722656 5.296875 -2.265625 Z M 5.296875 -2.265625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-11\" overflow=\"visible\">\n",
       "<path d=\"\" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-12\" overflow=\"visible\">\n",
       "<path d=\"M 0.578125 0 L 0.578125 -6.4375 L 1.359375 -6.4375 L 1.359375 0 Z M 0.578125 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-13\" overflow=\"visible\">\n",
       "<path d=\"M 0.5625 1.796875 L 0.46875 1.0625 C 0.644531 1.101562 0.796875 1.125 0.921875 1.125 C 1.097656 1.125 1.238281 1.09375 1.34375 1.03125 C 1.445312 0.976562 1.535156 0.898438 1.609375 0.796875 C 1.648438 0.710938 1.726562 0.515625 1.84375 0.203125 C 1.863281 0.160156 1.890625 0.0976562 1.921875 0.015625 L 0.140625 -4.671875 L 1 -4.671875 L 1.96875 -1.96875 C 2.09375 -1.625 2.207031 -1.265625 2.3125 -0.890625 C 2.394531 -1.242188 2.5 -1.597656 2.625 -1.953125 L 3.625 -4.671875 L 4.421875 -4.671875 L 2.640625 0.078125 C 2.453125 0.585938 2.304688 0.941406 2.203125 1.140625 C 2.054688 1.398438 1.894531 1.585938 1.71875 1.703125 C 1.539062 1.828125 1.320312 1.890625 1.0625 1.890625 C 0.914062 1.890625 0.75 1.859375 0.5625 1.796875 Z M 0.5625 1.796875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-14\" overflow=\"visible\">\n",
       "<path d=\"M 0.578125 0 L 0.578125 -4.671875 L 1.296875 -4.671875 L 1.296875 -3.953125 C 1.472656 -4.285156 1.640625 -4.503906 1.796875 -4.609375 C 1.953125 -4.710938 2.125 -4.765625 2.3125 -4.765625 C 2.570312 -4.765625 2.84375 -4.679688 3.125 -4.515625 L 2.84375 -3.78125 C 2.65625 -3.894531 2.460938 -3.953125 2.265625 -3.953125 C 2.097656 -3.953125 1.941406 -3.898438 1.796875 -3.796875 C 1.660156 -3.691406 1.5625 -3.546875 1.5 -3.359375 C 1.414062 -3.078125 1.375 -2.769531 1.375 -2.4375 L 1.375 0 Z M 0.578125 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-15\" overflow=\"visible\">\n",
       "<path d=\"M 1.453125 0 L 0.03125 -4.671875 L 0.84375 -4.671875 L 1.59375 -1.96875 L 1.859375 -0.96875 C 1.867188 -1.019531 1.953125 -1.34375 2.109375 -1.9375 L 2.84375 -4.671875 L 3.65625 -4.671875 L 4.359375 -1.953125 L 4.59375 -1.0625 L 4.859375 -1.96875 L 5.65625 -4.671875 L 6.421875 -4.671875 L 4.96875 0 L 4.15625 0 L 3.40625 -2.796875 L 3.21875 -3.59375 L 2.28125 0 Z M 1.453125 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-16\" overflow=\"visible\">\n",
       "<path d=\"M 0.59375 0 L 0.59375 -6.4375 L 1.390625 -6.4375 L 1.390625 -4.125 C 1.753906 -4.550781 2.21875 -4.765625 2.78125 -4.765625 C 3.125 -4.765625 3.421875 -4.695312 3.671875 -4.5625 C 3.929688 -4.425781 4.113281 -4.238281 4.21875 -4 C 4.332031 -3.757812 4.390625 -3.410156 4.390625 -2.953125 L 4.390625 0 L 3.609375 0 L 3.609375 -2.953125 C 3.609375 -3.347656 3.519531 -3.632812 3.34375 -3.8125 C 3.175781 -4 2.9375 -4.09375 2.625 -4.09375 C 2.382812 -4.09375 2.160156 -4.03125 1.953125 -3.90625 C 1.742188 -3.789062 1.597656 -3.628906 1.515625 -3.421875 C 1.429688 -3.210938 1.390625 -2.921875 1.390625 -2.546875 L 1.390625 0 Z M 0.59375 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-17\" overflow=\"visible\">\n",
       "<path d=\"M 3.640625 -1.703125 L 4.421875 -1.609375 C 4.335938 -1.066406 4.117188 -0.644531 3.765625 -0.34375 C 3.410156 -0.0390625 2.976562 0.109375 2.46875 0.109375 C 1.832031 0.109375 1.320312 -0.0976562 0.9375 -0.515625 C 0.550781 -0.929688 0.359375 -1.53125 0.359375 -2.3125 C 0.359375 -2.820312 0.441406 -3.265625 0.609375 -3.640625 C 0.773438 -4.015625 1.023438 -4.296875 1.359375 -4.484375 C 1.703125 -4.671875 2.078125 -4.765625 2.484375 -4.765625 C 2.984375 -4.765625 3.394531 -4.632812 3.71875 -4.375 C 4.039062 -4.125 4.25 -3.765625 4.34375 -3.296875 L 3.578125 -3.171875 C 3.503906 -3.484375 3.375 -3.71875 3.1875 -3.875 C 3 -4.039062 2.773438 -4.125 2.515625 -4.125 C 2.109375 -4.125 1.78125 -3.976562 1.53125 -3.6875 C 1.289062 -3.40625 1.171875 -2.957031 1.171875 -2.34375 C 1.171875 -1.707031 1.289062 -1.25 1.53125 -0.96875 C 1.769531 -0.6875 2.082031 -0.546875 2.46875 -0.546875 C 2.78125 -0.546875 3.039062 -0.640625 3.25 -0.828125 C 3.457031 -1.015625 3.585938 -1.304688 3.640625 -1.703125 Z M 3.640625 -1.703125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-18\" overflow=\"visible\">\n",
       "<path d=\"M 1.890625 0 L 0.109375 -4.671875 L 0.953125 -4.671875 L 1.953125 -1.875 C 2.054688 -1.570312 2.15625 -1.257812 2.25 -0.9375 C 2.320312 -1.175781 2.421875 -1.46875 2.546875 -1.8125 L 3.578125 -4.671875 L 4.390625 -4.671875 L 2.625 0 Z M 1.890625 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-19\" overflow=\"visible\">\n",
       "<path d=\"M 0.296875 -2.328125 C 0.296875 -3.191406 0.535156 -3.832031 1.015625 -4.25 C 1.421875 -4.59375 1.910156 -4.765625 2.484375 -4.765625 C 3.128906 -4.765625 3.65625 -4.554688 4.0625 -4.140625 C 4.46875 -3.722656 4.671875 -3.144531 4.671875 -2.40625 C 4.671875 -1.800781 4.578125 -1.328125 4.390625 -0.984375 C 4.210938 -0.640625 3.953125 -0.367188 3.609375 -0.171875 C 3.265625 0.015625 2.890625 0.109375 2.484375 0.109375 C 1.828125 0.109375 1.296875 -0.0976562 0.890625 -0.515625 C 0.492188 -0.941406 0.296875 -1.546875 0.296875 -2.328125 Z M 1.109375 -2.328125 C 1.109375 -1.734375 1.238281 -1.285156 1.5 -0.984375 C 1.757812 -0.691406 2.085938 -0.546875 2.484375 -0.546875 C 2.878906 -0.546875 3.207031 -0.691406 3.46875 -0.984375 C 3.726562 -1.285156 3.859375 -1.742188 3.859375 -2.359375 C 3.859375 -2.929688 3.726562 -3.367188 3.46875 -3.671875 C 3.207031 -3.972656 2.878906 -4.125 2.484375 -4.125 C 2.085938 -4.125 1.757812 -3.972656 1.5 -3.671875 C 1.238281 -3.378906 1.109375 -2.929688 1.109375 -2.328125 Z M 1.109375 -2.328125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-20\" overflow=\"visible\">\n",
       "<path d=\"M 0.78125 0 L 0.78125 -4.046875 L 0.078125 -4.046875 L 0.078125 -4.671875 L 0.78125 -4.671875 L 0.78125 -5.15625 C 0.78125 -5.476562 0.804688 -5.710938 0.859375 -5.859375 C 0.941406 -6.066406 1.078125 -6.234375 1.265625 -6.359375 C 1.460938 -6.484375 1.734375 -6.546875 2.078125 -6.546875 C 2.296875 -6.546875 2.539062 -6.519531 2.8125 -6.46875 L 2.6875 -5.78125 C 2.53125 -5.8125 2.378906 -5.828125 2.234375 -5.828125 C 1.992188 -5.828125 1.820312 -5.773438 1.71875 -5.671875 C 1.613281 -5.566406 1.5625 -5.375 1.5625 -5.09375 L 1.5625 -4.671875 L 2.484375 -4.671875 L 2.484375 -4.046875 L 1.5625 -4.046875 L 1.5625 0 Z M 0.78125 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-21\" overflow=\"visible\">\n",
       "<path d=\"M 3.65625 0 L 3.65625 -0.6875 C 3.289062 -0.15625 2.796875 0.109375 2.171875 0.109375 C 1.898438 0.109375 1.644531 0.0546875 1.40625 -0.046875 C 1.164062 -0.160156 0.984375 -0.296875 0.859375 -0.453125 C 0.742188 -0.609375 0.664062 -0.800781 0.625 -1.03125 C 0.59375 -1.1875 0.578125 -1.4375 0.578125 -1.78125 L 0.578125 -4.671875 L 1.359375 -4.671875 L 1.359375 -2.078125 C 1.359375 -1.660156 1.378906 -1.382812 1.421875 -1.25 C 1.460938 -1.039062 1.5625 -0.875 1.71875 -0.75 C 1.882812 -0.632812 2.085938 -0.578125 2.328125 -0.578125 C 2.566406 -0.578125 2.789062 -0.632812 3 -0.75 C 3.207031 -0.875 3.351562 -1.039062 3.4375 -1.25 C 3.519531 -1.457031 3.5625 -1.765625 3.5625 -2.171875 L 3.5625 -4.671875 L 4.359375 -4.671875 L 4.359375 0 Z M 3.65625 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-22\" overflow=\"visible\">\n",
       "<path d=\"M 0.59375 1.78125 L 0.59375 -4.671875 L 1.3125 -4.671875 L 1.3125 -4.0625 C 1.476562 -4.300781 1.664062 -4.476562 1.875 -4.59375 C 2.09375 -4.707031 2.359375 -4.765625 2.671875 -4.765625 C 3.066406 -4.765625 3.414062 -4.660156 3.71875 -4.453125 C 4.019531 -4.253906 4.25 -3.96875 4.40625 -3.59375 C 4.5625 -3.21875 4.640625 -2.8125 4.640625 -2.375 C 4.640625 -1.894531 4.550781 -1.460938 4.375 -1.078125 C 4.207031 -0.691406 3.960938 -0.394531 3.640625 -0.1875 C 3.316406 0.0078125 2.972656 0.109375 2.609375 0.109375 C 2.347656 0.109375 2.113281 0.0507812 1.90625 -0.0625 C 1.695312 -0.175781 1.523438 -0.316406 1.390625 -0.484375 L 1.390625 1.78125 Z M 1.3125 -2.3125 C 1.3125 -1.707031 1.429688 -1.257812 1.671875 -0.96875 C 1.921875 -0.6875 2.21875 -0.546875 2.5625 -0.546875 C 2.90625 -0.546875 3.203125 -0.691406 3.453125 -0.984375 C 3.710938 -1.285156 3.84375 -1.75 3.84375 -2.375 C 3.84375 -2.96875 3.71875 -3.410156 3.46875 -3.703125 C 3.226562 -4.003906 2.9375 -4.15625 2.59375 -4.15625 C 2.257812 -4.15625 1.960938 -3.992188 1.703125 -3.671875 C 1.441406 -3.359375 1.3125 -2.90625 1.3125 -2.3125 Z M 1.3125 -2.3125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-23\" overflow=\"visible\">\n",
       "<path d=\"M 0.84375 0 L 0.84375 -6.4375 L 1.6875 -6.4375 L 1.6875 0 Z M 0.84375 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph3-24\" overflow=\"visible\">\n",
       "<path d=\"M 0.4375 -3.140625 C 0.4375 -4.203125 0.722656 -5.035156 1.296875 -5.640625 C 1.867188 -6.253906 2.609375 -6.5625 3.515625 -6.5625 C 4.109375 -6.5625 4.644531 -6.414062 5.125 -6.125 C 5.601562 -5.84375 5.96875 -5.445312 6.21875 -4.9375 C 6.46875 -4.425781 6.59375 -3.851562 6.59375 -3.21875 C 6.59375 -2.5625 6.460938 -1.972656 6.203125 -1.453125 C 5.941406 -0.941406 5.566406 -0.550781 5.078125 -0.28125 C 4.597656 -0.0195312 4.078125 0.109375 3.515625 0.109375 C 2.910156 0.109375 2.367188 -0.0351562 1.890625 -0.328125 C 1.410156 -0.617188 1.046875 -1.019531 0.796875 -1.53125 C 0.554688 -2.039062 0.4375 -2.578125 0.4375 -3.140625 Z M 1.3125 -3.125 C 1.3125 -2.34375 1.519531 -1.726562 1.9375 -1.28125 C 2.351562 -0.84375 2.878906 -0.625 3.515625 -0.625 C 4.148438 -0.625 4.675781 -0.847656 5.09375 -1.296875 C 5.507812 -1.742188 5.71875 -2.382812 5.71875 -3.21875 C 5.71875 -3.738281 5.628906 -4.191406 5.453125 -4.578125 C 5.273438 -4.972656 5.015625 -5.28125 4.671875 -5.5 C 4.328125 -5.71875 3.945312 -5.828125 3.53125 -5.828125 C 2.925781 -5.828125 2.40625 -5.617188 1.96875 -5.203125 C 1.53125 -4.785156 1.3125 -4.09375 1.3125 -3.125 Z M 1.3125 -3.125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "</g>\n",
       "</defs>\n",
       "<g id=\"surface1\">\n",
       "<path d=\"M 302 300 L 370 300 C 373.3125 300 376 302.6875 376 306 L 376 350 C 376 353.3125 373.3125 356 370 356 L 302 356 C 298.6875 356 296 353.3125 296 350 L 296 306 C 296 302.6875 298.6875 300 302 300 Z M 302 300 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 326.203125 320 L 354.203125 320 L 354.203125 336 L 326.203125 336 Z M 326.203125 320 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.902344%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"225.6986\" xlink:href=\"#glyph0-1\" y=\"70.5\"/>\n",
       "</g>\n",
       "<path d=\"M 272 328 L 320.300781 328 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 324.300781 328 L 320.300781 328 M 320.300781 326.5 L 324.300781 328 L 320.300781 329.5 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 308 368 L 308 334 C 308 330.6875 310.6875 328 314 328 L 321.257812 328 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 354.203125 328 L 425.839844 328 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 429.839844 328 L 425.839844 328 M 425.839844 326.5 L 429.839844 328 L 425.839844 329.5 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 356.914062 328 L 358 328 C 361.3125 328 364 325.3125 364 322 L 364 285.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 364 281.898438 L 364 285.898438 M 362.5 285.898438 L 364 281.898438 L 365.5 285.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"193.0261\" xlink:href=\"#glyph1-1\" y=\"121\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"199.0291\" xlink:href=\"#glyph2-1\" y=\"123\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"6.7336\" xlink:href=\"#glyph3-1\" y=\"69\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"13.2334\" xlink:href=\"#glyph3-2\" y=\"69\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"15.2332\" xlink:href=\"#glyph3-3\" y=\"69\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"20.239\" xlink:href=\"#glyph3-3\" y=\"69\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"25.2448\" xlink:href=\"#glyph3-4\" y=\"69\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"30.2506\" xlink:href=\"#glyph3-5\" y=\"69\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"37.7566\" xlink:href=\"#glyph3-6\" y=\"69\"/>\n",
       "  <use x=\"42.2566\" xlink:href=\"#glyph3-7\" y=\"69\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"44.7568\" xlink:href=\"#glyph3-8\" y=\"69\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"49.7626\" xlink:href=\"#glyph3-7\" y=\"69\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"52.2628\" xlink:href=\"#glyph3-4\" y=\"69\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"269.2778\" xlink:href=\"#glyph1-2\" y=\"61\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"275.7773\" xlink:href=\"#glyph2-1\" y=\"63\"/>\n",
       "</g>\n",
       "<path d=\"M 160 410 L 188 410 L 188 426 L 160 426 Z M 160 410 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.902344%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"59.4973\" xlink:href=\"#glyph0-1\" y=\"160.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"96.7449\" xlink:href=\"#glyph3-9\" y=\"156.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"102.2421\" xlink:href=\"#glyph3-10\" y=\"156.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"108.7419\" xlink:href=\"#glyph3-11\" y=\"156.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"111.2421\" xlink:href=\"#glyph3-12\" y=\"156.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"113.2419\" xlink:href=\"#glyph3-8\" y=\"156.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"118.2477\" xlink:href=\"#glyph3-13\" y=\"156.5\"/>\n",
       "  <use x=\"122.7477\" xlink:href=\"#glyph3-4\" y=\"156.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"127.7535\" xlink:href=\"#glyph3-14\" y=\"156.5\"/>\n",
       "  <use x=\"130.7505\" xlink:href=\"#glyph3-11\" y=\"156.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"133.2507\" xlink:href=\"#glyph3-15\" y=\"156.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"139.7505\" xlink:href=\"#glyph3-2\" y=\"156.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"141.7503\" xlink:href=\"#glyph3-7\" y=\"156.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"144.2505\" xlink:href=\"#glyph3-16\" y=\"156.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"149.2563\" xlink:href=\"#glyph3-11\" y=\"156.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"89.4807\" xlink:href=\"#glyph3-8\" y=\"166.7944\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"94.4865\" xlink:href=\"#glyph3-17\" y=\"166.7944\"/>\n",
       "  <use x=\"98.9865\" xlink:href=\"#glyph3-7\" y=\"166.7944\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"101.4867\" xlink:href=\"#glyph3-2\" y=\"166.7944\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"103.4865\" xlink:href=\"#glyph3-18\" y=\"166.7944\"/>\n",
       "  <use x=\"107.9865\" xlink:href=\"#glyph3-8\" y=\"166.7944\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"112.9923\" xlink:href=\"#glyph3-7\" y=\"166.7944\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"115.4925\" xlink:href=\"#glyph3-2\" y=\"166.7944\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"117.4923\" xlink:href=\"#glyph3-19\" y=\"166.7944\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"122.4981\" xlink:href=\"#glyph3-5\" y=\"166.7944\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"127.5039\" xlink:href=\"#glyph3-11\" y=\"166.7944\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"130.0041\" xlink:href=\"#glyph3-20\" y=\"166.7944\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"132.5043\" xlink:href=\"#glyph3-21\" y=\"166.7944\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"137.5101\" xlink:href=\"#glyph3-17\" y=\"166.7944\"/>\n",
       "  <use x=\"142.0101\" xlink:href=\"#glyph3-7\" y=\"166.7944\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"144.5103\" xlink:href=\"#glyph3-2\" y=\"166.7944\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"146.5101\" xlink:href=\"#glyph3-19\" y=\"166.7944\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"151.5159\" xlink:href=\"#glyph3-5\" y=\"166.7944\"/>\n",
       "</g>\n",
       "<path d=\"M 404 412.273438 L 426.101562 412.273438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 430.101562 412.273438 L 426.101562 412.273438 M 426.101562 410.773438 L 430.101562 412.273438 L 426.101562 413.773438 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 413 424.273438 L 413 418.273438 C 413 414.960938 415.6875 412.273438 419 412.273438 L 429.789062 412.273438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"325.481\" xlink:href=\"#glyph3-10\" y=\"161\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"331.9808\" xlink:href=\"#glyph3-19\" y=\"161\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"336.9866\" xlink:href=\"#glyph3-5\" y=\"161\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"341.9924\" xlink:href=\"#glyph3-17\" y=\"161\"/>\n",
       "  <use x=\"346.4924\" xlink:href=\"#glyph3-8\" y=\"161\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"351.4982\" xlink:href=\"#glyph3-7\" y=\"161\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"353.9984\" xlink:href=\"#glyph3-4\" y=\"161\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"359.0042\" xlink:href=\"#glyph3-5\" y=\"161\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"364.01\" xlink:href=\"#glyph3-8\" y=\"161\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"369.0158\" xlink:href=\"#glyph3-7\" y=\"161\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"371.516\" xlink:href=\"#glyph3-4\" y=\"161\"/>\n",
       "</g>\n",
       "<path d=\"M 308 424 L 330.101562 424 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 334.101562 424 L 330.101562 424 M 330.101562 422.5 L 334.101562 424 L 330.101562 425.5 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 308 424 L 314 424 C 317.3125 424 320 421.3125 320 418 L 320 413.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 320 409.898438 L 320 413.898438 M 318.5 413.898438 L 320 409.898438 L 321.5 413.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"230.1312\" xlink:href=\"#glyph3-10\" y=\"161.0455\"/>\n",
       "  <use x=\"236.631\" xlink:href=\"#glyph3-19\" y=\"161.0455\"/>\n",
       "  <use x=\"241.6368\" xlink:href=\"#glyph3-22\" y=\"161.0455\"/>\n",
       "  <use x=\"246.6426\" xlink:href=\"#glyph3-13\" y=\"161.0455\"/>\n",
       "</g>\n",
       "<path d=\"M 191.949219 300 L 259.949219 300 C 263.261719 300 265.949219 302.6875 265.949219 306 L 265.949219 350 C 265.949219 353.3125 263.261719 356 259.949219 356 L 191.949219 356 C 188.636719 356 185.949219 353.3125 185.949219 350 L 185.949219 306 C 185.949219 302.6875 188.636719 300 191.949219 300 Z M 191.949219 300 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 216.148438 320 L 244.148438 320 L 244.148438 336 L 216.148438 336 Z M 216.148438 320 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.902344%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"115.6473\" xlink:href=\"#glyph0-1\" y=\"70.5\"/>\n",
       "</g>\n",
       "<path d=\"M 176 327.058594 L 210.25 327.652344 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 214.25 327.722656 L 210.25 327.652344 M 210.277344 326.152344 L 214.25 327.722656 L 210.226562 329.152344 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 197.949219 369.085938 L 197.949219 333.660156 C 197.949219 330.480469 200.425781 327.855469 203.597656 327.671875 L 205.347656 327.566406 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 244.148438 328 L 320.300781 328 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 324.300781 328 L 320.300781 328 M 320.300781 326.5 L 324.300781 328 L 320.300781 329.5 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 247.019531 328 L 247.949219 328 C 251.261719 328 253.949219 325.3125 253.949219 322 L 253.949219 285.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 253.949219 281.898438 L 253.949219 285.898438 M 252.449219 285.898438 L 253.949219 281.898438 L 255.449219 285.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 407.539062 300 L 475.539062 300 C 478.851562 300 481.539062 302.6875 481.539062 306 L 481.539062 350 C 481.539062 353.3125 478.851562 356 475.539062 356 L 407.539062 356 C 404.226562 356 401.539062 353.3125 401.539062 350 L 401.539062 306 C 401.539062 302.6875 404.226562 300 407.539062 300 Z M 407.539062 300 \" style=\"fill-rule:nonzero;fill:rgb(100%,100%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 431.738281 320 L 459.738281 320 L 459.738281 336 L 431.738281 336 Z M 431.738281 320 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.902344%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"331.2371\" xlink:href=\"#glyph0-1\" y=\"70.5\"/>\n",
       "</g>\n",
       "<path d=\"M 377.539062 328 L 425.839844 328 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 429.839844 328 L 425.839844 328 M 425.839844 326.5 L 429.839844 328 L 425.839844 329.5 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 413.117188 368 L 413.5 331.617188 C 413.519531 329.613281 415.152344 328 417.15625 328 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 459.738281 328 L 510.101562 328 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 514.101562 328 L 510.101562 328 M 510.101562 326.5 L 514.101562 328 L 510.101562 329.5 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 461.707031 328 L 463.539062 328 C 466.851562 328 469.539062 325.3125 469.539062 322 L 469.539062 285.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 469.539062 281.898438 L 469.539062 285.898438 M 468.039062 285.898438 L 469.539062 281.898438 L 471.039062 285.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"162.5319\" xlink:href=\"#glyph1-2\" y=\"61\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"169.0314\" xlink:href=\"#glyph2-1\" y=\"63\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"170.976\" xlink:href=\"#glyph2-2\" y=\"63\"/>\n",
       "  <use x=\"173.307\" xlink:href=\"#glyph2-3\" y=\"63\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"21.4915\" xlink:href=\"#glyph3-23\" y=\"121\"/>\n",
       "  <use x=\"23.9917\" xlink:href=\"#glyph3-5\" y=\"121\"/>\n",
       "  <use x=\"28.9939\" xlink:href=\"#glyph3-22\" y=\"121\"/>\n",
       "  <use x=\"33.9961\" xlink:href=\"#glyph3-21\" y=\"121\"/>\n",
       "  <use x=\"38.9983\" xlink:href=\"#glyph3-7\" y=\"121\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"81.4141\" xlink:href=\"#glyph1-1\" y=\"121\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"87.417\" xlink:href=\"#glyph2-1\" y=\"123\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"89.3616\" xlink:href=\"#glyph2-2\" y=\"123\"/>\n",
       "  <use x=\"91.6926\" xlink:href=\"#glyph2-3\" y=\"123\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"293.0356\" xlink:href=\"#glyph1-1\" y=\"121\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"299.0386\" xlink:href=\"#glyph2-1\" y=\"123\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"300.9832\" xlink:href=\"#glyph2-4\" y=\"123\"/>\n",
       "  <use x=\"305.0712\" xlink:href=\"#glyph2-3\" y=\"123\"/>\n",
       "</g>\n",
       "<path d=\"M 349.949219 264 L 377.949219 264 L 377.949219 280 L 349.949219 280 Z M 349.949219 264 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.902344%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 240 264 L 268 264 L 268 280 L 240 280 Z M 240 264 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.902344%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<path d=\"M 456 264 L 484 264 L 484 280 L 456 280 Z M 456 264 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.902344%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-112,-260)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"7.4873\" xlink:href=\"#glyph3-24\" y=\"13.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"14.4875\" xlink:href=\"#glyph3-21\" y=\"13.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"19.4933\" xlink:href=\"#glyph3-7\" y=\"13.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"21.9935\" xlink:href=\"#glyph3-22\" y=\"13.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"26.9993\" xlink:href=\"#glyph3-21\" y=\"13.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"32.0051\" xlink:href=\"#glyph3-7\" y=\"13.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"34.5053\" xlink:href=\"#glyph3-11\" y=\"13.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"37.0055\" xlink:href=\"#glyph3-12\" y=\"13.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"39.0053\" xlink:href=\"#glyph3-8\" y=\"13.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"44.0111\" xlink:href=\"#glyph3-13\" y=\"13.5\"/>\n",
       "  <use x=\"48.5111\" xlink:href=\"#glyph3-4\" y=\"13.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"53.5169\" xlink:href=\"#glyph3-14\" y=\"13.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"373.2874\" xlink:href=\"#glyph1-2\" y=\"61\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"379.7869\" xlink:href=\"#glyph2-1\" y=\"63\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"381.7315\" xlink:href=\"#glyph2-4\" y=\"63\"/>\n",
       "  <use x=\"385.8195\" xlink:href=\"#glyph2-3\" y=\"63\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import SVG\n",
    "SVG(filename=\"../img/rnn.svg\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As discussed, the computation in the hidden state uses H<sub>t</sub> = X<sub>t</sub>W<sub>xh</sub> + H<sub>t-1</sub>W<sub>hh</sub> to generate an object matching H<sub>t-1</sub>\n",
    "in dimensionality. Moreover, we use H<sub>t</sub> to generate the output O<sub>t</sub> = H<sub>t</sub>W<sub>hq</sub>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "X=torch.randn(3,1)\n",
    "H=torch.randn(3,2)\n",
    "\n",
    "W_xh=torch.randn(1,2)\n",
    "W_hh=torch.randn(2,2)\n",
    "W_hq=torch.randn(2,3)\n",
    "\n",
    "def net(X, H):\n",
    "    H = (torch.mm(X, W_xh) + torch.mm(H, W_hh))\n",
    "    O = (torch.mm(H, W_hq))\n",
    "    return H, O"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The recurrent network defined above takes observations X and a hidden state H as arguments and uses them to update the\n",
    "hidden state and emit an output O. Since this chain could go on for a very long time, training the model with backprop is\n",
    "out of the question (at least without some approximation). After all, this leads to a very long chain of dependencies that\n",
    "would be prohibitive to solve exactly: books typically have more than 100,000 characters and it is unreasonable to assume\n",
    "that the later text relies indiscriminately on all occurrences that happened, say, 10,000 characters in the past. Truncation\n",
    "methods such as BPTT  and long short term memory are useful to address this in a more\n",
    "principled manner. For now, let’s see how a state update works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.1791, -1.1110],\n",
      "        [ 1.8311, -4.8133],\n",
      "        [ 0.1405,  2.0094]])\n",
      "tensor([[ 1.4982, -0.4124,  0.5771],\n",
      "        [ 2.0720, -1.2001,  2.6391],\n",
      "        [-2.3988,  0.7046, -1.0535]])\n"
     ]
    }
   ],
   "source": [
    "(H, O) = net(X,H)\n",
    "\n",
    "print(H)\n",
    "print(O)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Steps in a Language Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We conclude this section by illustrating how RNNs can be used to build a language model. For simplicity of illustration\n",
    "we use words rather than characters, since the former are easier to comprehend. Let the number of mini-batch examples\n",
    "be 1, and the sequence of the text be the beginning of our dataset, i.e. “the time machine by h. g. wells”. The figure below\n",
    "illustrates how to estimate the next character based on the present and previous characters. During the training process,\n",
    "we run a softmax operation on the output from the output layer for each time step, and then use the cross-entropy loss\n",
    "function to compute the error between the result and the label. Due to the recurrent computation of the hidden state in\n",
    "the hidden layer, the output of time step 3 O<sub>3</sub> is determined by the text sequence “the”, “time”, “machine”. Since the\n",
    "next word of the sequence in the training data is “by”, the loss of time step 3 will depend on the probability distribution\n",
    "of the next word generated based on the sequence “the”, “time”, “machine” and the label “by” of this time step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"151pt\" version=\"1.1\" viewBox=\"0 0 288 151\" width=\"288pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<defs>\n",
       "<g>\n",
       "<symbol id=\"glyph0-0\" overflow=\"visible\">\n",
       "<path d=\"\" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-1\" overflow=\"visible\">\n",
       "<path d=\"M 2.0625 -6.421875 C 1.859375 -6.1875 1.609375 -5.96875 1.296875 -5.765625 C 0.984375 -5.59375 0.6875 -5.46875 0.390625 -5.375 L 0.390625 -4.640625 C 1 -4.828125 1.5 -5.109375 1.90625 -5.53125 L 1.90625 0 L 2.640625 0 L 2.640625 -6.421875 Z M 2.0625 -6.421875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-2\" overflow=\"visible\">\n",
       "<path d=\"M 2.796875 -6.546875 C 2.140625 -6.546875 1.609375 -6.34375 1.21875 -5.90625 C 0.8125 -5.484375 0.609375 -4.90625 0.609375 -4.1875 L 1.34375 -4.1875 C 1.359375 -4.75 1.5 -5.1875 1.75 -5.46875 C 1.984375 -5.78125 2.328125 -5.921875 2.765625 -5.921875 C 3.171875 -5.921875 3.515625 -5.8125 3.75 -5.59375 C 3.984375 -5.375 4.09375 -5.078125 4.09375 -4.6875 C 4.09375 -4.28125 3.921875 -3.890625 3.59375 -3.53125 C 3.421875 -3.359375 3.09375 -3.09375 2.59375 -2.734375 C 1.90625 -2.265625 1.453125 -1.875 1.203125 -1.59375 C 0.765625 -1.109375 0.5625 -0.578125 0.5625 0 L 4.84375 0 L 4.84375 -0.65625 L 1.46875 -0.65625 C 1.578125 -1.09375 2.046875 -1.578125 2.875 -2.140625 C 3.53125 -2.578125 3.984375 -2.9375 4.203125 -3.171875 C 4.609375 -3.625 4.828125 -4.125 4.828125 -4.6875 C 4.828125 -5.234375 4.640625 -5.703125 4.25 -6.03125 C 3.875 -6.375 3.390625 -6.546875 2.796875 -6.546875 Z M 2.796875 -6.546875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-3\" overflow=\"visible\">\n",
       "<path d=\"M 2.765625 -6.546875 C 2.15625 -6.546875 1.65625 -6.375 1.28125 -6.03125 C 0.875 -5.671875 0.640625 -5.1875 0.59375 -4.578125 L 1.328125 -4.578125 C 1.375 -5 1.515625 -5.34375 1.765625 -5.578125 C 2 -5.8125 2.34375 -5.90625 2.765625 -5.90625 C 3.1875 -5.90625 3.515625 -5.8125 3.75 -5.59375 C 3.96875 -5.40625 4.09375 -5.125 4.09375 -4.765625 C 4.09375 -4.40625 3.96875 -4.125 3.75 -3.921875 C 3.515625 -3.734375 3.1875 -3.640625 2.765625 -3.640625 L 2.265625 -3.640625 L 2.265625 -3.0625 L 2.796875 -3.0625 C 3.234375 -3.0625 3.578125 -2.96875 3.828125 -2.75 C 4.09375 -2.53125 4.21875 -2.234375 4.21875 -1.84375 C 4.21875 -1.46875 4.078125 -1.15625 3.828125 -0.90625 C 3.546875 -0.640625 3.171875 -0.5 2.71875 -0.5 C 2.3125 -0.5 1.984375 -0.625 1.703125 -0.84375 C 1.390625 -1.09375 1.21875 -1.484375 1.203125 -1.984375 L 0.453125 -1.984375 C 0.515625 -1.265625 0.765625 -0.71875 1.21875 -0.34375 C 1.609375 -0.03125 2.109375 0.125 2.71875 0.125 C 3.390625 0.125 3.921875 -0.0625 4.34375 -0.4375 C 4.75 -0.8125 4.953125 -1.28125 4.953125 -1.875 C 4.953125 -2.25 4.84375 -2.5625 4.625 -2.8125 C 4.421875 -3.0625 4.140625 -3.25 3.765625 -3.359375 C 4.46875 -3.59375 4.8125 -4.09375 4.8125 -4.8125 C 4.8125 -5.34375 4.625 -5.765625 4.25 -6.078125 C 3.875 -6.40625 3.375 -6.546875 2.765625 -6.546875 Z M 2.765625 -6.546875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-4\" overflow=\"visible\">\n",
       "<path d=\"M 2.203125 -4.78125 C 1.6875 -4.78125 1.25 -4.65625 0.921875 -4.421875 C 0.578125 -4.1875 0.40625 -3.890625 0.40625 -3.484375 C 0.40625 -3.09375 0.578125 -2.796875 0.9375 -2.578125 C 1.15625 -2.4375 1.53125 -2.296875 2.09375 -2.171875 C 2.59375 -2.046875 2.921875 -1.9375 3.078125 -1.875 C 3.359375 -1.734375 3.515625 -1.53125 3.515625 -1.265625 C 3.515625 -0.75 3.09375 -0.484375 2.265625 -0.484375 C 1.875 -0.484375 1.578125 -0.546875 1.375 -0.6875 C 1.171875 -0.859375 1.046875 -1.109375 0.984375 -1.484375 L 0.28125 -1.484375 C 0.390625 -0.40625 1.0625 0.125 2.28125 0.125 C 3.59375 0.125 4.25 -0.359375 4.25 -1.3125 C 4.25 -1.71875 4.0625 -2.03125 3.703125 -2.265625 C 3.453125 -2.421875 3.0625 -2.5625 2.5 -2.703125 C 2.03125 -2.828125 1.703125 -2.921875 1.53125 -3 C 1.265625 -3.140625 1.140625 -3.3125 1.140625 -3.515625 C 1.140625 -3.71875 1.234375 -3.875 1.4375 -4 C 1.625 -4.125 1.875 -4.171875 2.203125 -4.171875 C 2.5625 -4.171875 2.859375 -4.109375 3.046875 -3.953125 C 3.234375 -3.828125 3.34375 -3.625 3.40625 -3.328125 L 4.109375 -3.328125 C 3.984375 -4.296875 3.359375 -4.78125 2.203125 -4.78125 Z M 2.203125 -4.78125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-5\" overflow=\"visible\">\n",
       "<path d=\"M 1.796875 -6.15625 L 1.09375 -5.859375 L 1.09375 -4.65625 L 0.15625 -4.65625 L 0.15625 -4.046875 L 1.09375 -4.046875 L 1.09375 -1.09375 C 1.09375 -0.734375 1.15625 -0.484375 1.3125 -0.296875 C 1.46875 -0.09375 1.75 0 2.140625 0 L 2.828125 0 L 2.828125 -0.609375 L 2.234375 -0.609375 C 2.078125 -0.609375 1.96875 -0.640625 1.90625 -0.71875 C 1.828125 -0.796875 1.796875 -0.921875 1.796875 -1.09375 L 1.796875 -4.046875 L 2.953125 -4.046875 L 2.953125 -4.65625 L 1.796875 -4.65625 Z M 1.796875 -6.15625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-6\" overflow=\"visible\">\n",
       "<path d=\"M 2.46875 -4.78125 C 1.796875 -4.78125 1.265625 -4.53125 0.875 -4.046875 C 0.484375 -3.59375 0.28125 -3.015625 0.28125 -2.328125 C 0.28125 -1.578125 0.484375 -0.984375 0.90625 -0.53125 C 1.296875 -0.09375 1.84375 0.125 2.5 0.125 C 3.109375 0.125 3.59375 -0.046875 3.984375 -0.390625 C 4.296875 -0.6875 4.515625 -1.046875 4.609375 -1.5 L 3.890625 -1.5 C 3.78125 -1.171875 3.625 -0.921875 3.421875 -0.765625 C 3.1875 -0.578125 2.875 -0.484375 2.5 -0.484375 C 2.0625 -0.484375 1.703125 -0.625 1.453125 -0.90625 C 1.203125 -1.203125 1.078125 -1.609375 1.03125 -2.125 L 4.703125 -2.125 C 4.6875 -2.9375 4.515625 -3.578125 4.171875 -4.015625 C 3.78125 -4.53125 3.21875 -4.78125 2.46875 -4.78125 Z M 2.5 -4.171875 C 3.375 -4.171875 3.875 -3.6875 3.953125 -2.6875 L 1.046875 -2.6875 C 1.09375 -3.140625 1.234375 -3.5 1.5 -3.765625 C 1.75 -4.046875 2.078125 -4.171875 2.5 -4.171875 Z M 2.5 -4.171875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-7\" overflow=\"visible\">\n",
       "<path d=\"M 2.78125 -4.78125 C 2.078125 -4.78125 1.5625 -4.484375 1.234375 -3.890625 L 1.234375 -4.65625 L 0.5625 -4.65625 L 0.5625 1.78125 L 1.28125 1.78125 L 1.28125 -0.84375 C 1.640625 -0.203125 2.15625 0.125 2.8125 0.125 C 3.484375 0.125 4 -0.125 4.375 -0.609375 C 4.71875 -1.0625 4.890625 -1.625 4.890625 -2.3125 C 4.890625 -3.015625 4.71875 -3.578125 4.359375 -4.03125 C 3.984375 -4.53125 3.453125 -4.78125 2.78125 -4.78125 Z M 2.6875 -4.1875 C 3.1875 -4.1875 3.5625 -4 3.828125 -3.625 C 4.046875 -3.296875 4.15625 -2.859375 4.15625 -2.3125 C 4.15625 -1.75 4.046875 -1.3125 3.8125 -0.984375 C 3.546875 -0.640625 3.171875 -0.46875 2.65625 -0.46875 C 2.234375 -0.46875 1.90625 -0.640625 1.640625 -0.96875 C 1.375 -1.296875 1.25 -1.734375 1.25 -2.265625 L 1.25 -2.359375 C 1.25 -2.875 1.375 -3.3125 1.609375 -3.640625 C 1.859375 -4 2.21875 -4.1875 2.6875 -4.1875 Z M 2.6875 -4.1875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-8\" overflow=\"visible\">\n",
       "<path d=\"M 2.640625 -4.78125 C 1.9375 -4.78125 1.390625 -4.546875 0.96875 -4.0625 C 0.578125 -3.59375 0.375 -3.03125 0.375 -2.328125 C 0.375 -1.625 0.578125 -1.046875 0.96875 -0.609375 C 1.390625 -0.109375 1.9375 0.125 2.640625 0.125 C 3.328125 0.125 3.890625 -0.109375 4.3125 -0.609375 C 4.703125 -1.046875 4.890625 -1.625 4.890625 -2.328125 C 4.890625 -3.03125 4.703125 -3.59375 4.296875 -4.0625 C 3.890625 -4.546875 3.328125 -4.78125 2.640625 -4.78125 Z M 2.640625 -4.171875 C 3.125 -4.171875 3.515625 -3.984375 3.796875 -3.609375 C 4.046875 -3.265625 4.171875 -2.84375 4.171875 -2.328125 C 4.171875 -1.8125 4.046875 -1.390625 3.796875 -1.046875 C 3.515625 -0.671875 3.125 -0.484375 2.640625 -0.484375 C 2.140625 -0.484375 1.75 -0.671875 1.484375 -1.046875 C 1.234375 -1.390625 1.109375 -1.8125 1.109375 -2.328125 C 1.109375 -2.84375 1.234375 -3.265625 1.484375 -3.609375 C 1.75 -3.984375 2.140625 -4.171875 2.640625 -4.171875 Z M 2.640625 -4.171875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-9\" overflow=\"visible\">\n",
       "<path d=\"M 0.5625 -4.65625 L 0.5625 -1.765625 C 0.5625 -0.515625 1.109375 0.125 2.25 0.125 C 2.875 0.125 3.390625 -0.15625 3.765625 -0.71875 L 3.765625 0 L 4.484375 0 L 4.484375 -4.65625 L 3.765625 -4.65625 L 3.765625 -1.8125 C 3.703125 -1.421875 3.5625 -1.09375 3.296875 -0.84375 C 3.0625 -0.609375 2.78125 -0.484375 2.46875 -0.484375 C 2.046875 -0.484375 1.75 -0.609375 1.5625 -0.8125 C 1.375 -1.03125 1.28125 -1.375 1.28125 -1.8125 L 1.28125 -4.65625 Z M 0.5625 -4.65625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-10\" overflow=\"visible\">\n",
       "<path d=\"M 1.15625 -6.40625 C 1 -6.40625 0.875 -6.359375 0.765625 -6.25 C 0.65625 -6.171875 0.609375 -6.03125 0.609375 -5.875 C 0.609375 -5.71875 0.65625 -5.59375 0.765625 -5.484375 C 0.875 -5.390625 1 -5.34375 1.15625 -5.34375 C 1.3125 -5.34375 1.4375 -5.390625 1.53125 -5.484375 C 1.640625 -5.59375 1.703125 -5.71875 1.703125 -5.875 C 1.703125 -6.03125 1.640625 -6.171875 1.53125 -6.25 C 1.4375 -6.359375 1.3125 -6.40625 1.15625 -6.40625 Z M 0.796875 -4.65625 L 0.796875 0 L 1.515625 0 L 1.515625 -4.65625 Z M 0.796875 -4.65625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-11\" overflow=\"visible\">\n",
       "<path d=\"M 2.734375 -4.78125 C 2.4375 -4.78125 2.15625 -4.703125 1.90625 -4.5625 C 1.640625 -4.421875 1.4375 -4.234375 1.28125 -3.96875 L 1.28125 -4.65625 L 0.5625 -4.65625 L 0.5625 0 L 1.28125 0 L 1.28125 -2.8125 C 1.3125 -3.234375 1.4375 -3.5625 1.6875 -3.8125 C 1.921875 -4.046875 2.203125 -4.171875 2.53125 -4.171875 C 3.34375 -4.171875 3.75 -3.71875 3.75 -2.8125 L 3.75 0 L 4.46875 0 L 4.46875 -2.859375 C 4.46875 -4.140625 3.890625 -4.78125 2.734375 -4.78125 Z M 2.734375 -4.78125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-12\" overflow=\"visible\">\n",
       "<path d=\"M 0.15625 -6.421875 L 0.15625 -5.78125 L 2.421875 -5.78125 L 2.421875 0 L 3.15625 0 L 3.15625 -5.78125 L 5.421875 -5.78125 L 5.421875 -6.421875 Z M 0.15625 -6.421875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-13\" overflow=\"visible\">\n",
       "<path d=\"M 0.5625 -6.546875 L 0.5625 0 L 1.28125 0 L 1.28125 -2.65625 C 1.28125 -3.09375 1.390625 -3.453125 1.640625 -3.75 C 1.875 -4.03125 2.171875 -4.171875 2.53125 -4.171875 C 2.921875 -4.171875 3.21875 -4.0625 3.421875 -3.828125 C 3.625 -3.59375 3.734375 -3.265625 3.734375 -2.828125 L 3.734375 0 L 4.453125 0 L 4.453125 -2.9375 C 4.453125 -3.53125 4.3125 -3.984375 4.015625 -4.296875 C 3.734375 -4.625 3.3125 -4.78125 2.765625 -4.78125 C 2.453125 -4.78125 2.171875 -4.703125 1.921875 -4.5625 C 1.65625 -4.40625 1.4375 -4.1875 1.28125 -3.890625 L 1.28125 -6.546875 Z M 0.5625 -6.546875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-14\" overflow=\"visible\">\n",
       "<path d=\"M 2.59375 -4.78125 C 2.0625 -4.78125 1.625 -4.53125 1.28125 -4.03125 L 1.28125 -4.65625 L 0.5625 -4.65625 L 0.5625 0 L 1.28125 0 L 1.28125 -2.828125 C 1.28125 -3.203125 1.390625 -3.515625 1.609375 -3.765625 C 1.8125 -4.046875 2.09375 -4.171875 2.421875 -4.171875 C 3.125 -4.171875 3.484375 -3.765625 3.484375 -2.9375 L 3.484375 0 L 4.21875 0 L 4.21875 -2.921875 C 4.21875 -3.296875 4.3125 -3.59375 4.53125 -3.828125 C 4.71875 -4.0625 4.984375 -4.171875 5.296875 -4.171875 C 5.6875 -4.171875 5.96875 -4.0625 6.15625 -3.859375 C 6.328125 -3.65625 6.421875 -3.34375 6.421875 -2.9375 L 6.421875 0 L 7.140625 0 L 7.140625 -3.046875 C 7.140625 -3.578125 6.984375 -4 6.703125 -4.3125 C 6.390625 -4.625 6 -4.78125 5.515625 -4.78125 C 5.1875 -4.78125 4.921875 -4.703125 4.703125 -4.5625 C 4.453125 -4.421875 4.234375 -4.203125 4.046875 -3.890625 C 3.78125 -4.484375 3.296875 -4.78125 2.59375 -4.78125 Z M 2.59375 -4.78125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-15\" overflow=\"visible\">\n",
       "<path d=\"M 0.640625 -6.421875 L 0.640625 0 L 1.390625 0 L 1.390625 -5.109375 L 1.40625 -5.109375 L 3.640625 0 L 4.296875 0 L 6.53125 -5.109375 L 6.546875 -5.109375 L 6.546875 0 L 7.296875 0 L 7.296875 -6.421875 L 6.375 -6.421875 L 3.984375 -0.9375 L 3.953125 -0.9375 L 1.5625 -6.421875 Z M 0.640625 -6.421875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-16\" overflow=\"visible\">\n",
       "<path d=\"M 2.578125 -4.78125 C 2.03125 -4.78125 1.578125 -4.671875 1.234375 -4.453125 C 0.859375 -4.21875 0.625 -3.859375 0.515625 -3.359375 L 1.234375 -3.3125 C 1.3125 -3.609375 1.453125 -3.828125 1.703125 -3.984375 C 1.921875 -4.125 2.203125 -4.1875 2.53125 -4.1875 C 3.34375 -4.1875 3.75 -3.796875 3.75 -3.015625 L 3.75 -2.765625 L 2.640625 -2.75 C 1.9375 -2.75 1.390625 -2.609375 1.015625 -2.359375 C 0.578125 -2.09375 0.375 -1.6875 0.375 -1.15625 C 0.375 -0.78125 0.515625 -0.46875 0.8125 -0.21875 C 1.09375 0.015625 1.453125 0.125 1.921875 0.125 C 2.359375 0.125 2.75 0.03125 3.09375 -0.15625 C 3.375 -0.328125 3.625 -0.53125 3.796875 -0.78125 L 3.796875 0 L 4.46875 0 L 4.46875 -2.984375 C 4.46875 -3.53125 4.328125 -3.953125 4.0625 -4.25 C 3.734375 -4.609375 3.25 -4.78125 2.578125 -4.78125 Z M 3.75 -2.234375 L 3.75 -1.84375 C 3.75 -1.46875 3.578125 -1.15625 3.25 -0.875 C 2.90625 -0.578125 2.515625 -0.4375 2.078125 -0.4375 C 1.796875 -0.4375 1.5625 -0.515625 1.375 -0.640625 C 1.203125 -0.796875 1.109375 -0.96875 1.109375 -1.171875 C 1.109375 -1.859375 1.640625 -2.203125 2.6875 -2.21875 Z M 3.75 -2.234375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-17\" overflow=\"visible\">\n",
       "<path d=\"M 2.5625 -4.78125 C 1.875 -4.78125 1.328125 -4.53125 0.921875 -4.046875 C 0.5625 -3.59375 0.375 -3.015625 0.375 -2.296875 C 0.375 -1.59375 0.5625 -1.015625 0.921875 -0.578125 C 1.3125 -0.109375 1.859375 0.125 2.5625 0.125 C 3.125 0.125 3.59375 -0.03125 3.9375 -0.328125 C 4.296875 -0.625 4.53125 -1.09375 4.640625 -1.6875 L 3.921875 -1.6875 C 3.8125 -0.875 3.34375 -0.484375 2.5625 -0.484375 C 2.09375 -0.484375 1.734375 -0.640625 1.484375 -0.96875 C 1.234375 -1.296875 1.109375 -1.75 1.109375 -2.3125 C 1.109375 -2.875 1.234375 -3.328125 1.5 -3.65625 C 1.75 -4 2.109375 -4.171875 2.5625 -4.171875 C 2.9375 -4.171875 3.234375 -4.09375 3.46875 -3.921875 C 3.6875 -3.734375 3.828125 -3.46875 3.890625 -3.109375 L 4.609375 -3.109375 C 4.53125 -3.671875 4.3125 -4.09375 3.953125 -4.375 C 3.59375 -4.640625 3.140625 -4.78125 2.5625 -4.78125 Z M 2.5625 -4.78125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-18\" overflow=\"visible\">\n",
       "<path d=\"M 0.5625 -6.546875 L 0.5625 0 L 1.234375 0 L 1.234375 -0.765625 C 1.546875 -0.171875 2.0625 0.125 2.78125 0.125 C 3.453125 0.125 3.984375 -0.125 4.375 -0.609375 C 4.71875 -1.0625 4.890625 -1.640625 4.890625 -2.34375 C 4.890625 -3.03125 4.71875 -3.578125 4.375 -4.03125 C 4 -4.53125 3.46875 -4.78125 2.8125 -4.78125 C 2.15625 -4.78125 1.640625 -4.453125 1.28125 -3.796875 L 1.28125 -6.546875 Z M 2.65625 -4.1875 C 3.171875 -4.1875 3.546875 -4 3.8125 -3.640625 C 4.046875 -3.328125 4.15625 -2.890625 4.15625 -2.34375 C 4.15625 -1.78125 4.046875 -1.34375 3.828125 -1.015625 C 3.578125 -0.65625 3.1875 -0.46875 2.6875 -0.46875 C 2.21875 -0.46875 1.859375 -0.640625 1.609375 -1.015625 C 1.375 -1.328125 1.25 -1.75 1.25 -2.28125 L 1.25 -2.359375 C 1.25 -2.90625 1.375 -3.34375 1.640625 -3.6875 C 1.90625 -4.015625 2.234375 -4.1875 2.65625 -4.1875 Z M 2.65625 -4.1875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-19\" overflow=\"visible\">\n",
       "<path d=\"M 0.046875 -4.65625 L 1.890625 -0.109375 L 1.09375 1.78125 L 1.875 1.78125 L 4.421875 -4.65625 L 3.625 -4.65625 L 2.25 -0.984375 L 0.828125 -4.65625 Z M 0.046875 -4.65625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-20\" overflow=\"visible\">\n",
       "<path d=\"M 4 -6.546875 L 4 -3.796875 C 3.640625 -4.453125 3.125 -4.78125 2.453125 -4.78125 C 1.796875 -4.78125 1.28125 -4.53125 0.90625 -4.03125 C 0.546875 -3.578125 0.375 -3.03125 0.375 -2.34375 C 0.375 -1.640625 0.546875 -1.0625 0.90625 -0.609375 C 1.28125 -0.125 1.8125 0.125 2.5 0.125 C 3.1875 0.125 3.71875 -0.171875 4.046875 -0.765625 L 4.046875 0 L 4.71875 0 L 4.71875 -6.546875 Z M 2.609375 -4.1875 C 3.03125 -4.1875 3.359375 -4.015625 3.625 -3.6875 C 3.875 -3.34375 4.015625 -2.90625 4.015625 -2.359375 L 4.015625 -2.28125 C 4.015625 -1.75 3.890625 -1.328125 3.671875 -1.015625 C 3.40625 -0.640625 3.046875 -0.46875 2.578125 -0.46875 C 2.078125 -0.46875 1.703125 -0.65625 1.453125 -1.015625 C 1.21875 -1.34375 1.109375 -1.78125 1.109375 -2.34375 C 1.109375 -2.890625 1.21875 -3.328125 1.453125 -3.640625 C 1.703125 -4 2.09375 -4.1875 2.609375 -4.1875 Z M 2.609375 -4.1875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-21\" overflow=\"visible\">\n",
       "<path d=\"\" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-22\" overflow=\"visible\">\n",
       "<path d=\"M 3.453125 -6.421875 L 0.21875 -2.15625 L 0.21875 -1.453125 L 3.453125 -1.453125 L 3.453125 0 L 4.15625 0 L 4.15625 -1.453125 L 5.1875 -1.453125 L 5.1875 -2.0625 L 4.15625 -2.0625 L 4.15625 -6.421875 Z M 3.421875 -5.46875 L 3.453125 -5.46875 L 3.453125 -2.0625 L 0.84375 -2.0625 Z M 3.421875 -5.46875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-23\" overflow=\"visible\">\n",
       "<path d=\"M 0.640625 -6.421875 L 0.640625 0 L 1.390625 0 L 1.390625 -3 L 5.109375 -3 L 5.109375 0 L 5.84375 0 L 5.84375 -6.421875 L 5.109375 -6.421875 L 5.109375 -3.640625 L 1.390625 -3.640625 L 1.390625 -6.421875 Z M 0.640625 -6.421875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-24\" overflow=\"visible\">\n",
       "<path d=\"M 1.1875 -1.078125 C 1.03125 -1.078125 0.90625 -1.015625 0.796875 -0.90625 C 0.6875 -0.796875 0.640625 -0.671875 0.640625 -0.515625 C 0.640625 -0.359375 0.6875 -0.234375 0.796875 -0.125 C 0.90625 -0.015625 1.03125 0.03125 1.1875 0.03125 C 1.328125 0.03125 1.46875 -0.015625 1.578125 -0.125 C 1.6875 -0.234375 1.734375 -0.359375 1.734375 -0.515625 C 1.734375 -0.671875 1.6875 -0.796875 1.578125 -0.90625 C 1.46875 -1.015625 1.328125 -1.078125 1.1875 -1.078125 Z M 1.1875 -1.078125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-25\" overflow=\"visible\">\n",
       "<path d=\"M 0.9375 -6.421875 L 0.609375 -2.875 L 1.3125 -2.875 C 1.4375 -3.125 1.625 -3.328125 1.875 -3.453125 C 2.125 -3.59375 2.390625 -3.65625 2.703125 -3.65625 C 3.171875 -3.65625 3.53125 -3.515625 3.796875 -3.234375 C 4.0625 -2.953125 4.21875 -2.5625 4.21875 -2.046875 C 4.21875 -1.59375 4.0625 -1.21875 3.765625 -0.9375 C 3.46875 -0.640625 3.109375 -0.5 2.65625 -0.5 C 2.265625 -0.5 1.921875 -0.609375 1.671875 -0.796875 C 1.390625 -1.015625 1.21875 -1.328125 1.1875 -1.734375 L 0.453125 -1.734375 C 0.5 -1.140625 0.75 -0.671875 1.203125 -0.328125 C 1.59375 -0.03125 2.078125 0.125 2.640625 0.125 C 3.28125 0.125 3.8125 -0.078125 4.25 -0.453125 C 4.703125 -0.875 4.9375 -1.40625 4.9375 -2.046875 C 4.9375 -2.75 4.75 -3.28125 4.359375 -3.6875 C 3.984375 -4.0625 3.484375 -4.25 2.875 -4.25 C 2.578125 -4.25 2.3125 -4.21875 2.0625 -4.109375 C 1.78125 -4 1.546875 -3.828125 1.375 -3.625 L 1.328125 -3.625 L 1.546875 -5.765625 L 4.6875 -5.765625 L 4.6875 -6.421875 Z M 0.9375 -6.421875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph0-26\" overflow=\"visible\">\n",
       "<path d=\"M 3.390625 -6.546875 C 2.453125 -6.546875 1.703125 -6.21875 1.15625 -5.546875 C 0.625 -4.921875 0.375 -4.140625 0.375 -3.1875 C 0.375 -2.234375 0.640625 -1.453125 1.15625 -0.84375 C 1.71875 -0.203125 2.515625 0.125 3.53125 0.125 C 4.0625 0.125 4.546875 0.046875 5.015625 -0.09375 C 5.46875 -0.234375 5.84375 -0.421875 6.140625 -0.671875 L 6.140625 -3.296875 L 3.40625 -3.296875 L 3.40625 -2.65625 L 5.421875 -2.65625 L 5.421875 -1.03125 C 5.1875 -0.875 4.921875 -0.75 4.609375 -0.671875 C 4.28125 -0.578125 3.953125 -0.515625 3.59375 -0.515625 C 2.765625 -0.515625 2.140625 -0.78125 1.703125 -1.265625 C 1.3125 -1.734375 1.109375 -2.375 1.109375 -3.1875 C 1.109375 -4.015625 1.3125 -4.671875 1.734375 -5.15625 C 2.140625 -5.65625 2.6875 -5.890625 3.390625 -5.890625 C 3.953125 -5.890625 4.421875 -5.765625 4.765625 -5.484375 C 5.109375 -5.21875 5.3125 -4.84375 5.421875 -4.359375 L 6.15625 -4.359375 C 6.03125 -5.0625 5.765625 -5.59375 5.3125 -5.96875 C 4.828125 -6.359375 4.1875 -6.546875 3.390625 -6.546875 Z M 3.390625 -6.546875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-0\" overflow=\"visible\">\n",
       "<path d=\"M 1.125 0 L 1.125 -5.625 L 5.625 -5.625 L 5.625 0 Z M 1.265625 -0.140625 L 5.484375 -0.140625 L 5.484375 -5.484375 L 1.265625 -5.484375 Z M 1.265625 -0.140625 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-1\" overflow=\"visible\">\n",
       "<path d=\"M 4.8125 -2.875 L 2.328125 -2.875 L 1.71875 0 L 0.390625 0 L 1.734375 -6.4375 L 3.0625 -6.4375 L 2.546875 -3.953125 L 5.03125 -3.953125 L 5.5625 -6.4375 L 6.875 -6.4375 L 5.53125 0 L 4.203125 0 Z M 4.8125 -2.875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph1-2\" overflow=\"visible\">\n",
       "<path d=\"M 0.78125 -2.53125 C 0.78125 -2.90625 0.835938 -3.304688 0.953125 -3.734375 C 1.109375 -4.296875 1.335938 -4.785156 1.640625 -5.203125 C 1.941406 -5.617188 2.328125 -5.945312 2.796875 -6.1875 C 3.265625 -6.425781 3.796875 -6.546875 4.390625 -6.546875 C 5.179688 -6.546875 5.820312 -6.296875 6.3125 -5.796875 C 6.8125 -5.304688 7.0625 -4.65625 7.0625 -3.84375 C 7.0625 -3.164062 6.898438 -2.507812 6.578125 -1.875 C 6.253906 -1.238281 5.816406 -0.75 5.265625 -0.40625 C 4.722656 -0.0625 4.109375 0.109375 3.421875 0.109375 C 2.816406 0.109375 2.3125 -0.0234375 1.90625 -0.296875 C 1.5 -0.566406 1.207031 -0.898438 1.03125 -1.296875 C 0.863281 -1.703125 0.78125 -2.113281 0.78125 -2.53125 Z M 2.09375 -2.546875 C 2.09375 -2.109375 2.226562 -1.738281 2.5 -1.4375 C 2.769531 -1.144531 3.125 -1 3.5625 -1 C 3.914062 -1 4.257812 -1.113281 4.59375 -1.34375 C 4.925781 -1.582031 5.195312 -1.941406 5.40625 -2.421875 C 5.625 -2.898438 5.734375 -3.363281 5.734375 -3.8125 C 5.734375 -4.320312 5.597656 -4.71875 5.328125 -5 C 5.054688 -5.289062 4.707031 -5.4375 4.28125 -5.4375 C 3.632812 -5.4375 3.109375 -5.132812 2.703125 -4.53125 C 2.296875 -3.9375 2.09375 -3.273438 2.09375 -2.546875 Z M 2.09375 -2.546875 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-0\" overflow=\"visible\">\n",
       "<path d=\"M 0.875 0 L 0.875 -4.375 L 4.375 -4.375 L 4.375 0 Z M 0.984375 -0.109375 L 4.265625 -0.109375 L 4.265625 -4.265625 L 0.984375 -4.265625 Z M 0.984375 -0.109375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-1\" overflow=\"visible\">\n",
       "<path d=\"M 1.6875 0 L 2.484375 -3.78125 C 2.140625 -3.507812 1.65625 -3.296875 1.03125 -3.140625 L 1.15625 -3.703125 C 1.457031 -3.828125 1.757812 -3.988281 2.0625 -4.1875 C 2.363281 -4.382812 2.585938 -4.554688 2.734375 -4.703125 C 2.828125 -4.796875 2.914062 -4.90625 3 -5.03125 L 3.359375 -5.03125 L 2.3125 0 Z M 1.6875 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-2\" overflow=\"visible\">\n",
       "<path d=\"M 0.40625 0 C 0.46875 -0.300781 0.554688 -0.550781 0.671875 -0.75 C 0.785156 -0.945312 0.9375 -1.132812 1.125 -1.3125 C 1.3125 -1.5 1.671875 -1.804688 2.203125 -2.234375 C 2.523438 -2.492188 2.75 -2.6875 2.875 -2.8125 C 3.039062 -2.988281 3.160156 -3.160156 3.234375 -3.328125 C 3.285156 -3.441406 3.3125 -3.566406 3.3125 -3.703125 C 3.3125 -3.929688 3.226562 -4.125 3.0625 -4.28125 C 2.90625 -4.445312 2.707031 -4.53125 2.46875 -4.53125 C 2.238281 -4.53125 2.035156 -4.445312 1.859375 -4.28125 C 1.679688 -4.125 1.554688 -3.863281 1.484375 -3.5 L 0.875 -3.59375 C 0.9375 -4.039062 1.109375 -4.390625 1.390625 -4.640625 C 1.679688 -4.898438 2.039062 -5.03125 2.46875 -5.03125 C 2.75 -5.03125 3.003906 -4.96875 3.234375 -4.84375 C 3.460938 -4.726562 3.632812 -4.5625 3.75 -4.34375 C 3.875 -4.132812 3.9375 -3.914062 3.9375 -3.6875 C 3.9375 -3.351562 3.816406 -3.035156 3.578125 -2.734375 C 3.429688 -2.535156 3.003906 -2.15625 2.296875 -1.59375 C 1.984375 -1.351562 1.753906 -1.15625 1.609375 -1 C 1.460938 -0.84375 1.351562 -0.695312 1.28125 -0.5625 L 3.515625 -0.5625 L 3.390625 0 Z M 0.40625 0 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-3\" overflow=\"visible\">\n",
       "<path d=\"M 0.390625 -1.3125 L 0.984375 -1.390625 C 1.023438 -1.035156 1.125 -0.78125 1.28125 -0.625 C 1.4375 -0.476562 1.644531 -0.40625 1.90625 -0.40625 C 2.207031 -0.40625 2.46875 -0.515625 2.6875 -0.734375 C 2.914062 -0.953125 3.03125 -1.203125 3.03125 -1.484375 C 3.03125 -1.734375 2.945312 -1.9375 2.78125 -2.09375 C 2.613281 -2.257812 2.390625 -2.34375 2.109375 -2.34375 C 2.078125 -2.34375 2.007812 -2.335938 1.90625 -2.328125 L 2.015625 -2.84375 C 2.078125 -2.832031 2.132812 -2.828125 2.1875 -2.828125 C 2.539062 -2.828125 2.8125 -2.910156 3 -3.078125 C 3.1875 -3.253906 3.28125 -3.46875 3.28125 -3.71875 C 3.28125 -3.945312 3.203125 -4.140625 3.046875 -4.296875 C 2.890625 -4.453125 2.703125 -4.53125 2.484375 -4.53125 C 2.253906 -4.53125 2.050781 -4.445312 1.875 -4.28125 C 1.695312 -4.125 1.585938 -3.898438 1.546875 -3.609375 L 0.9375 -3.734375 C 1.03125 -4.148438 1.21875 -4.46875 1.5 -4.6875 C 1.789062 -4.914062 2.128906 -5.03125 2.515625 -5.03125 C 2.929688 -5.03125 3.265625 -4.90625 3.515625 -4.65625 C 3.773438 -4.40625 3.90625 -4.101562 3.90625 -3.75 C 3.90625 -3.476562 3.832031 -3.242188 3.6875 -3.046875 C 3.550781 -2.847656 3.347656 -2.6875 3.078125 -2.5625 C 3.265625 -2.445312 3.40625 -2.304688 3.5 -2.140625 C 3.601562 -1.972656 3.65625 -1.785156 3.65625 -1.578125 C 3.65625 -1.128906 3.488281 -0.738281 3.15625 -0.40625 C 2.820312 -0.0820312 2.421875 0.078125 1.953125 0.078125 C 1.492188 0.078125 1.125 -0.046875 0.84375 -0.296875 C 0.570312 -0.546875 0.421875 -0.882812 0.390625 -1.3125 Z M 0.390625 -1.3125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-4\" overflow=\"visible\">\n",
       "<path d=\"M 2.09375 0 L 2.359375 -1.28125 L 0.3125 -1.28125 L 0.453125 -1.890625 L 3.25 -5.015625 L 3.765625 -5.015625 L 3.09375 -1.828125 L 3.796875 -1.828125 L 3.6875 -1.28125 L 2.984375 -1.28125 L 2.703125 0 Z M 2.46875 -1.828125 L 2.90625 -3.921875 L 1.046875 -1.828125 Z M 2.46875 -1.828125 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "<symbol id=\"glyph2-5\" overflow=\"visible\">\n",
       "<path d=\"M 0.484375 -1.4375 L 1.125 -1.5 C 1.113281 -1.40625 1.109375 -1.347656 1.109375 -1.328125 C 1.109375 -1.179688 1.144531 -1.03125 1.21875 -0.875 C 1.300781 -0.726562 1.40625 -0.613281 1.53125 -0.53125 C 1.664062 -0.445312 1.804688 -0.40625 1.953125 -0.40625 C 2.140625 -0.40625 2.332031 -0.46875 2.53125 -0.59375 C 2.726562 -0.726562 2.890625 -0.921875 3.015625 -1.171875 C 3.140625 -1.429688 3.203125 -1.6875 3.203125 -1.9375 C 3.203125 -2.21875 3.117188 -2.441406 2.953125 -2.609375 C 2.785156 -2.773438 2.566406 -2.859375 2.296875 -2.859375 C 2.117188 -2.859375 1.945312 -2.8125 1.78125 -2.71875 C 1.625 -2.632812 1.476562 -2.507812 1.34375 -2.34375 L 0.796875 -2.375 L 1.5625 -4.9375 L 4 -4.9375 L 3.890625 -4.375 L 1.984375 -4.375 L 1.609375 -3.09375 C 1.742188 -3.195312 1.882812 -3.273438 2.03125 -3.328125 C 2.1875 -3.378906 2.34375 -3.40625 2.5 -3.40625 C 2.882812 -3.40625 3.195312 -3.28125 3.4375 -3.03125 C 3.6875 -2.78125 3.8125 -2.429688 3.8125 -1.984375 C 3.8125 -1.597656 3.726562 -1.242188 3.5625 -0.921875 C 3.394531 -0.597656 3.160156 -0.347656 2.859375 -0.171875 C 2.566406 -0.00390625 2.25 0.078125 1.90625 0.078125 C 1.625 0.078125 1.367188 0.015625 1.140625 -0.109375 C 0.921875 -0.234375 0.753906 -0.410156 0.640625 -0.640625 C 0.535156 -0.867188 0.484375 -1.097656 0.484375 -1.328125 C 0.484375 -1.347656 0.484375 -1.382812 0.484375 -1.4375 Z M 0.484375 -1.4375 \" style=\"stroke:none;\"/>\n",
       "</symbol>\n",
       "</g>\n",
       "</defs>\n",
       "<g id=\"surface1\">\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"68.53783\" xlink:href=\"#glyph0-1\" y=\"16\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"117.5063\" xlink:href=\"#glyph0-2\" y=\"16\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"169.1687\" xlink:href=\"#glyph0-3\" y=\"16\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"11.4955\" xlink:href=\"#glyph0-4\" y=\"16\"/>\n",
       "  <use x=\"16.0405\" xlink:href=\"#glyph0-5\" y=\"16\"/>\n",
       "  <use x=\"19.2355\" xlink:href=\"#glyph0-6\" y=\"16\"/>\n",
       "  <use x=\"24.2305\" xlink:href=\"#glyph0-7\" y=\"16\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"6.991\" xlink:href=\"#glyph0-8\" y=\"38.5\"/>\n",
       "  <use x=\"12.265\" xlink:href=\"#glyph0-9\" y=\"38.5\"/>\n",
       "  <use x=\"17.305\" xlink:href=\"#glyph0-5\" y=\"38.5\"/>\n",
       "  <use x=\"20.5\" xlink:href=\"#glyph0-7\" y=\"38.5\"/>\n",
       "  <use x=\"25.774\" xlink:href=\"#glyph0-9\" y=\"38.5\"/>\n",
       "  <use x=\"30.814\" xlink:href=\"#glyph0-5\" y=\"38.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"10.078\" xlink:href=\"#glyph0-10\" y=\"141.68067\"/>\n",
       "  <use x=\"12.382\" xlink:href=\"#glyph0-11\" y=\"141.68067\"/>\n",
       "  <use x=\"17.413\" xlink:href=\"#glyph0-7\" y=\"141.68067\"/>\n",
       "  <use x=\"22.687\" xlink:href=\"#glyph0-9\" y=\"141.68067\"/>\n",
       "  <use x=\"27.727\" xlink:href=\"#glyph0-5\" y=\"141.68067\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"63.49483\" xlink:href=\"#glyph0-12\" y=\"141.68067\"/>\n",
       "  <use x=\"69.06583\" xlink:href=\"#glyph0-13\" y=\"141.68067\"/>\n",
       "  <use x=\"74.06983\" xlink:href=\"#glyph0-6\" y=\"141.68067\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"111.6113\" xlink:href=\"#glyph0-12\" y=\"141.68067\"/>\n",
       "  <use x=\"117.1823\" xlink:href=\"#glyph0-10\" y=\"141.68067\"/>\n",
       "  <use x=\"119.4863\" xlink:href=\"#glyph0-14\" y=\"141.68067\"/>\n",
       "  <use x=\"127.1813\" xlink:href=\"#glyph0-6\" y=\"141.68067\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"155.1932\" xlink:href=\"#glyph0-15\" y=\"141.68067\"/>\n",
       "  <use x=\"163.1312\" xlink:href=\"#glyph0-16\" y=\"141.68067\"/>\n",
       "  <use x=\"168.1622\" xlink:href=\"#glyph0-17\" y=\"141.68067\"/>\n",
       "  <use x=\"173.0852\" xlink:href=\"#glyph0-13\" y=\"141.68067\"/>\n",
       "  <use x=\"178.0892\" xlink:href=\"#glyph0-10\" y=\"141.68067\"/>\n",
       "  <use x=\"180.3932\" xlink:href=\"#glyph0-11\" y=\"141.68067\"/>\n",
       "  <use x=\"185.4242\" xlink:href=\"#glyph0-6\" y=\"141.68067\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"60.99733\" xlink:href=\"#glyph0-12\" y=\"38.5\"/>\n",
       "  <use x=\"66.56833\" xlink:href=\"#glyph0-10\" y=\"38.5\"/>\n",
       "  <use x=\"68.87233\" xlink:href=\"#glyph0-14\" y=\"38.5\"/>\n",
       "  <use x=\"76.56733\" xlink:href=\"#glyph0-6\" y=\"38.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"104.2808\" xlink:href=\"#glyph0-15\" y=\"38.5\"/>\n",
       "  <use x=\"112.2188\" xlink:href=\"#glyph0-16\" y=\"38.5\"/>\n",
       "  <use x=\"117.2498\" xlink:href=\"#glyph0-17\" y=\"38.5\"/>\n",
       "  <use x=\"122.1728\" xlink:href=\"#glyph0-13\" y=\"38.5\"/>\n",
       "  <use x=\"127.1768\" xlink:href=\"#glyph0-10\" y=\"38.5\"/>\n",
       "  <use x=\"129.4808\" xlink:href=\"#glyph0-11\" y=\"38.5\"/>\n",
       "  <use x=\"134.5118\" xlink:href=\"#glyph0-6\" y=\"38.5\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"167.9372\" xlink:href=\"#glyph0-18\" y=\"38.5\"/>\n",
       "  <use x=\"173.2112\" xlink:href=\"#glyph0-19\" y=\"38.5\"/>\n",
       "</g>\n",
       "<path d=\"M 72.28125 111.101562 L 98.28125 111.101562 L 98.28125 131.101562 L 72.28125 131.101562 Z M 72.28125 111.101562 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"66.083541\" xlink:href=\"#glyph1-1\" y=\"102.486338\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"72.583053\" xlink:href=\"#glyph2-1\" y=\"104.486338\"/>\n",
       "</g>\n",
       "<path d=\"M 122.894531 111.101562 L 148.894531 111.101562 L 148.894531 131.101562 L 122.894531 131.101562 Z M 122.894531 111.101562 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"116.697511\" xlink:href=\"#glyph1-1\" y=\"102.486338\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"123.197023\" xlink:href=\"#glyph2-2\" y=\"104.486338\"/>\n",
       "</g>\n",
       "<path d=\"M 173.804688 111.101562 L 199.804688 111.101562 L 199.804688 131.101562 L 173.804688 131.101562 Z M 173.804688 111.101562 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"167.609911\" xlink:href=\"#glyph1-1\" y=\"102.486338\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"174.109423\" xlink:href=\"#glyph2-3\" y=\"104.486338\"/>\n",
       "</g>\n",
       "<path d=\"M 72.28125 74 L 98.28125 74 L 98.28125 94 L 72.28125 94 Z M 72.28125 74 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.902344%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"65.833053\" xlink:href=\"#glyph1-2\" y=\"65.385498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"72.833541\" xlink:href=\"#glyph2-1\" y=\"67.385498\"/>\n",
       "</g>\n",
       "<path d=\"M 122.894531 74 L 148.894531 74 L 148.894531 94 L 122.894531 94 Z M 122.894531 74 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.902344%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"116.447023\" xlink:href=\"#glyph1-2\" y=\"65.385498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"123.447511\" xlink:href=\"#glyph2-2\" y=\"67.385498\"/>\n",
       "</g>\n",
       "<path d=\"M 173.804688 74 L 199.804688 74 L 199.804688 94 L 173.804688 94 Z M 173.804688 74 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.902344%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"167.359423\" xlink:href=\"#glyph1-2\" y=\"65.385498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"174.359911\" xlink:href=\"#glyph2-3\" y=\"67.385498\"/>\n",
       "</g>\n",
       "<path d=\"M 98.28125 121.101562 L 116.992188 121.101562 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 120.992188 121.101562 L 116.992188 121.101562 M 116.992188 119.601562 L 120.992188 121.101562 L 116.992188 122.601562 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 148.894531 121.101562 L 167.90625 121.101562 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 171.90625 121.101562 L 167.90625 121.101562 M 167.90625 119.601562 L 171.90625 121.101562 L 167.90625 122.601562 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 85.28125 111.101562 L 85.28125 99.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 85.28125 95.898438 L 85.28125 99.898438 M 83.78125 99.898438 L 85.28125 95.898438 L 86.78125 99.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 135.894531 111.101562 L 135.894531 99.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 135.894531 95.898438 L 135.894531 99.898438 M 134.394531 99.898438 L 135.894531 95.898438 L 137.394531 99.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 186.804688 111.101562 L 186.804688 99.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 186.804688 95.898438 L 186.804688 99.898438 M 185.304688 99.898438 L 186.804688 95.898438 L 188.304688 99.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 85.28125 147.179688 L 85.28125 137 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 85.28125 133 L 85.28125 137 M 83.78125 137 L 85.28125 133 L 86.78125 137 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 135.894531 147.179688 L 135.894531 137 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 135.894531 133 L 135.894531 137 M 134.394531 137 L 135.894531 133 L 137.394531 137 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 186.804688 147.179688 L 186.804688 137 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 186.804688 133 L 186.804688 137 M 185.304688 137 L 186.804688 133 L 188.304688 137 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"6.559\" xlink:href=\"#glyph0-13\" y=\"97.10084\"/>\n",
       "  <use x=\"11.563\" xlink:href=\"#glyph0-10\" y=\"97.10084\"/>\n",
       "  <use x=\"13.867\" xlink:href=\"#glyph0-20\" y=\"97.10084\"/>\n",
       "  <use x=\"19.141\" xlink:href=\"#glyph0-20\" y=\"97.10084\"/>\n",
       "  <use x=\"24.415\" xlink:href=\"#glyph0-6\" y=\"97.10084\"/>\n",
       "  <use x=\"29.41\" xlink:href=\"#glyph0-11\" y=\"97.10084\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"10.0195\" xlink:href=\"#glyph0-4\" y=\"110.10084\"/>\n",
       "  <use x=\"14.5645\" xlink:href=\"#glyph0-5\" y=\"110.10084\"/>\n",
       "  <use x=\"17.7595\" xlink:href=\"#glyph0-16\" y=\"110.10084\"/>\n",
       "  <use x=\"22.7905\" xlink:href=\"#glyph0-5\" y=\"110.10084\"/>\n",
       "  <use x=\"25.9855\" xlink:href=\"#glyph0-6\" y=\"110.10084\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"6.991\" xlink:href=\"#glyph0-8\" y=\"60\"/>\n",
       "  <use x=\"12.265\" xlink:href=\"#glyph0-9\" y=\"60\"/>\n",
       "  <use x=\"17.305\" xlink:href=\"#glyph0-5\" y=\"60\"/>\n",
       "  <use x=\"20.5\" xlink:href=\"#glyph0-7\" y=\"60\"/>\n",
       "  <use x=\"25.774\" xlink:href=\"#glyph0-9\" y=\"60\"/>\n",
       "  <use x=\"30.814\" xlink:href=\"#glyph0-5\" y=\"60\"/>\n",
       "  <use x=\"34.009\" xlink:href=\"#glyph0-21\" y=\"60\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"10.0195\" xlink:href=\"#glyph0-4\" y=\"73\"/>\n",
       "  <use x=\"14.5645\" xlink:href=\"#glyph0-5\" y=\"73\"/>\n",
       "  <use x=\"17.7595\" xlink:href=\"#glyph0-16\" y=\"73\"/>\n",
       "  <use x=\"22.7905\" xlink:href=\"#glyph0-5\" y=\"73\"/>\n",
       "  <use x=\"25.9855\" xlink:href=\"#glyph0-6\" y=\"73\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"219.4927\" xlink:href=\"#glyph0-22\" y=\"16\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"218.2612\" xlink:href=\"#glyph0-18\" y=\"141.68067\"/>\n",
       "  <use x=\"223.5352\" xlink:href=\"#glyph0-19\" y=\"141.68067\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"218.7022\" xlink:href=\"#glyph0-23\" y=\"38.5\"/>\n",
       "  <use x=\"225.1822\" xlink:href=\"#glyph0-24\" y=\"38.5\"/>\n",
       "</g>\n",
       "<path d=\"M 224.128906 111.101562 L 250.128906 111.101562 L 250.128906 131.101562 L 224.128906 131.101562 Z M 224.128906 111.101562 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"217.933911\" xlink:href=\"#glyph1-1\" y=\"102.486338\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"224.433423\" xlink:href=\"#glyph2-4\" y=\"104.486338\"/>\n",
       "</g>\n",
       "<path d=\"M 224.128906 74 L 250.128906 74 L 250.128906 94 L 224.128906 94 Z M 224.128906 74 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.902344%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"217.683423\" xlink:href=\"#glyph1-2\" y=\"65.385498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"224.683911\" xlink:href=\"#glyph2-4\" y=\"67.385498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"269.9417\" xlink:href=\"#glyph0-25\" y=\"16\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"269.1512\" xlink:href=\"#glyph0-23\" y=\"141.68067\"/>\n",
       "  <use x=\"275.6312\" xlink:href=\"#glyph0-24\" y=\"141.68067\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"269.0252\" xlink:href=\"#glyph0-26\" y=\"38.5\"/>\n",
       "  <use x=\"275.7572\" xlink:href=\"#glyph0-24\" y=\"38.5\"/>\n",
       "</g>\n",
       "<path d=\"M 274.578125 111.101562 L 300.578125 111.101562 L 300.578125 131.101562 L 274.578125 131.101562 Z M 274.578125 111.101562 \" style=\"fill-rule:nonzero;fill:rgb(69.804382%,85.098267%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"268.382911\" xlink:href=\"#glyph1-1\" y=\"102.486338\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"274.882423\" xlink:href=\"#glyph2-5\" y=\"104.486338\"/>\n",
       "</g>\n",
       "<path d=\"M 274.578125 74 L 300.578125 74 L 300.578125 94 L 274.578125 94 Z M 274.578125 74 \" style=\"fill-rule:nonzero;fill:rgb(39.99939%,74.902344%,100%);fill-opacity:1;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"268.132423\" xlink:href=\"#glyph1-2\" y=\"65.385498\"/>\n",
       "</g>\n",
       "<g style=\"fill:rgb(0%,0%,0%);fill-opacity:1;\">\n",
       "  <use x=\"275.132911\" xlink:href=\"#glyph2-5\" y=\"67.385498\"/>\n",
       "</g>\n",
       "<path d=\"M 199.804688 121.101562 L 218.230469 121.101562 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 222.230469 121.101562 L 218.230469 121.101562 M 218.230469 119.601562 L 222.230469 121.101562 L 218.230469 122.601562 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 250.128906 121.101562 L 268.679688 121.101562 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 272.679688 121.101562 L 268.679688 121.101562 M 268.679688 119.601562 L 272.679688 121.101562 L 268.679688 122.601562 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 237.128906 147.179688 L 237.128906 137 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 237.128906 133 L 237.128906 137 M 235.628906 137 L 237.128906 133 L 238.628906 137 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 237.128906 111.101562 L 237.128906 99.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 237.128906 95.898438 L 237.128906 99.898438 M 235.628906 99.898438 L 237.128906 95.898438 L 238.628906 99.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 287.578125 147.179688 L 287.578125 137 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 287.578125 133 L 287.578125 137 M 286.078125 137 L 287.578125 133 L 289.078125 137 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 287.578125 111.101562 L 287.578125 99.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "<path d=\"M 287.578125 95.898438 L 287.578125 99.898438 M 286.078125 99.898438 L 287.578125 95.898438 L 289.078125 99.898438 \" style=\"fill:none;stroke-width:1;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;\" transform=\"matrix(1,0,0,1,-14,-21)\"/>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SVG(filename=\"../img/rnn-train.svg\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Word-level RNN language model. The input and label sequences are The Time Machine by H. and\n",
    "Time Machine by H. G. respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The number of words is huge compared to the number of characters. This is why quite often  we will use a character-level RNN instead. In the next few sections, we will introduce its implementation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* A network that uses recurrent computation is called a recurrent neural network (RNN).\n",
    "* The hidden state of the RNN can capture historical information of the sequence up to the current time step.\n",
    "* The number of RNN model parameters does not grow as the number of time steps increases.\n",
    "* We can create language models using a character-level RNN."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. If we use an RNN to predict the next character in a text sequence, how many output dimensions do we need?\n",
    "2. Can you design a mapping for which an RNN with hidden states is exact? Hint - what about a finite number of\n",
    "words?\n",
    "3. What happens to the gradient if you backpropagate through a long sequence?\n",
    "4. What are some of the problems associated with the simple sequence model described above?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}