lora_on_mnist.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a85fefe6-e3e9-44b2-9bdc-fc03d30080b6",
   "metadata": {},
   "source": [
    "# LoRA on MNIST  \n",
    "This notebook will attempt to implement Low-Rank Adaptation(LoRA) Finetuning from scratch on the MNIST dataset using pytorch lightning. I'm purposely using a simple model and data that way I can focus on LoRA implementation. \n",
    "\n",
    "Plan\n",
    "1. Build an MLP. Train this model to perform well on MNIST 0-4 (half of the data).\n",
    "3. Use standard finetuning approach to train this model to work on MNIST 5-9 (this will act as our baseline)\n",
    "4. Use LoRA finetuning to train the original model to work on MNIST 5-9\n",
    "5. Compare finetuning techniques (performance, memory etc)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37e02a5d-8dd2-4361-b7cf-9d33b025a410",
   "metadata": {},
   "source": [
    "Sources:\n",
    "1. https://arxiv.org/abs/2106.09685 - LoRA paper\n",
    "2. https://lightning.ai/pages/community/tutorial/lora-llm/ - (The first half offers useful starter pseudocode)\n",
    "3. https://colab.research.google.com/drive/1iERDk94Jp0UErsPf7vXyPKeiM4ZJUQ-a?usp=sharing#scrollTo=WuK0lPwcB7Ia - had some good ideas on metrics to compute about LoRA-ized model\n",
    "4. https://lightning.ai/docs/pytorch/stable/notebooks/lightning_examples/mnist-hello-world.html - starter code for building an MLP and training on MNIST\n",
    "5. https://discuss.pytorch.org/t/how-to-use-one-class-of-number-in-mnist/26276/21 - forum post on how to limit MNIST to only first or second half"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "423b1158-1d05-4a05-b5b2-f5a0edc9ba88",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Global seed set to 42\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "42"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import lightning as L\n",
    "from lightning.pytorch.loggers import CSVLogger\n",
    "from lightning.pytorch.callbacks import LearningRateFinder\n",
    "from lightning.pytorch.callbacks.early_stopping import EarlyStopping\n",
    "\n",
    "from pytorch_lightning import Callback\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch.nn import functional as F\n",
    "from torch.utils.data import DataLoader, random_split\n",
    "from torch.utils.data.sampler import SubsetRandomSampler\n",
    "\n",
    "import torchvision\n",
    "from torchvision.datasets import MNIST\n",
    "from torchvision import transforms\n",
    "\n",
    "from torchmetrics import Accuracy\n",
    "\n",
    "import pandas as pd\n",
    "import seaborn as sn\n",
    "import math\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# set precision to what lightning suggests for this gpu\n",
    "torch.set_float32_matmul_precision('high')\n",
    "# make results reproducible\n",
    "L.seed_everything(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c9fd83ca-88db-4682-85e1-904d4f5df759",
   "metadata": {},
   "outputs": [],
   "source": [
    "# reserved for constants\n",
    "PATH_DATASETS = '/home/sunil/lora_from_scratch/data'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d712261-bb19-459d-8272-b33e529bc613",
   "metadata": {},
   "source": [
    "Train an MLP on the first half of MNIST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "27414d79-396d-460d-8700-aee7169c20bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "class LitMNIST(L.LightningModule):\n",
    "    def __init__(self, data_dir=PATH_DATASETS, hidden_size=64, lr=2e-4):\n",
    "        super().__init__()\n",
    "\n",
    "        # Set our init args as class attributes\n",
    "        self.data_dir = data_dir\n",
    "        self.hidden_size = hidden_size\n",
    "        self.lr = lr\n",
    "\n",
    "        # Hardcode some dataset specific attributes\n",
    "        self.class_names = [0,1,2,3,4]\n",
    "        self.min_class = min(self.class_names)\n",
    "        self.num_classes = len(self.class_names) \n",
    "        self.dims = (1, 28, 28) # the shape of an example (C x W x H)\n",
    "        channels, width, height = self.dims\n",
    "\n",
    "        self.transform = transforms.Compose(\n",
    "            [\n",
    "                transforms.ToTensor(),\n",
    "                transforms.Normalize((0.1307,), (0.3081,)),\n",
    "            ]\n",
    "        )\n",
    "\n",
    "        self.batch_size = 1024\n",
    "\n",
    "        # Define layers for model\n",
    "        self.l1 = nn.Linear(channels * width * height, hidden_size)\n",
    "        self.l2 = nn.Linear(hidden_size, hidden_size)\n",
    "        self.l3 = nn.Linear(hidden_size, self.num_classes)\n",
    "\n",
    "        self.dropout = nn.Dropout(0.1)\n",
    "        self.relu = nn.ReLU()\n",
    "\n",
    "        \n",
    "        # Define metrics\n",
    "        self.val_accuracy = Accuracy(task=\"multiclass\", num_classes=self.num_classes)\n",
    "        self.test_accuracy = Accuracy(task=\"multiclass\", num_classes=self.num_classes)\n",
    "\n",
    "\n",
    "    def forward(self, x):\n",
    "        # preprocessing\n",
    "        x = torch.flatten(x,1)\n",
    "\n",
    "        # layer 1 (input size, hidden size)\n",
    "        x = self.l1(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.dropout(x)\n",
    "\n",
    "        # layer 2 (hidden size, hidden size)\n",
    "        x = self.l2(x)\n",
    "        x = self.relu(x)\n",
    "        x = self.dropout(x)\n",
    "\n",
    "        #layer 3 (hidden size, self.num_classes)\n",
    "        x = self.l3(x)\n",
    "                    \n",
    "        # notice that we return the log probabilities here as that is what nll loss expects in the training step\n",
    "        return F.log_softmax(x, dim=1)\n",
    "\n",
    "    def common_step(self, batch, batch_idx):\n",
    "        # define steps all of {train, val, test} will take in one place\n",
    "        x, y = batch\n",
    "        # rescale y to be 0 indexed if necessary (like when we start using mnist 5-9)\n",
    "        if self.min_class != 0:\n",
    "            y = y - self.min_class\n",
    "        \n",
    "        logits = self(x)\n",
    "        loss = F.nll_loss(logits, y)\n",
    "        return x,y, logits, loss\n",
    "        \n",
    "        \n",
    "\n",
    "    def training_step(self, batch, batch_idx):\n",
    "        _, _, _, loss = self.common_step(batch, batch_idx)\n",
    "        self.log(\"train_loss\", loss, batch_size=self.batch_size)\n",
    "        return loss\n",
    "\n",
    "    def validation_step(self, batch, batch_idx):\n",
    "        \n",
    "        x, y, logits, loss = self.common_step(batch, batch_idx)\n",
    "        preds = torch.argmax(logits, dim=1)\n",
    "        self.val_accuracy.update(preds, y)\n",
    "\n",
    "        self.log(\"val_loss\", loss, prog_bar=True)\n",
    "        self.log(\"val_acc\", self.val_accuracy, prog_bar=True)\n",
    "\n",
    "    def test_step(self, batch, batch_idx):\n",
    "        x, y, logits, loss = self.common_step(batch, batch_idx)\n",
    "        preds = torch.argmax(logits, dim=1)\n",
    "        self.test_accuracy.update(preds, y)\n",
    "\n",
    "        self.log(\"test_loss\", loss, prog_bar=True)\n",
    "        self.log(\"test_acc\", self.test_accuracy, prog_bar=True)\n",
    "    \n",
    "    def configure_optimizers(self):\n",
    "        # we'll use adamw to match the paper\n",
    "        optimizer = torch.optim.AdamW(self.parameters(), lr=self.lr)\n",
    "        return optimizer\n",
    "\n",
    "    # the rest of the class is helper functions/hooks for configuring data/dataloader building\n",
    "    @staticmethod\n",
    "    def get_indices(dataset,class_names):\n",
    "        if isinstance(dataset, torch.utils.data.dataset.Subset):\n",
    "            targets = torch.tensor([dataset.dataset.targets[i] for i in dataset.indices])\n",
    "        else:\n",
    "            targets = dataset.targets\n",
    "\n",
    "            \n",
    "        indices =  []\n",
    "        for i in range(len(targets)):\n",
    "            if targets[i] in class_names:\n",
    "                indices.append(i)\n",
    "        return indices\n",
    "        \n",
    "    def create_dataloader(self,dataset):\n",
    "        idx = self.get_indices(dataset, self.class_names)\n",
    "        loader = DataLoader(dataset,batch_size=self.batch_size, sampler = SubsetRandomSampler(idx), num_workers=16) # Note - this necessarily shuffles the data due to the sampler we are using\n",
    "        return loader\n",
    "\n",
    "    def prepare_data(self):\n",
    "        # download data\n",
    "        MNIST(self.data_dir, train=True, download=True)\n",
    "        MNIST(self.data_dir, train=False, download=True)\n",
    "\n",
    "    def setup(self, stage=None):\n",
    "        # Assign train/val datasets for use in dataloaders\n",
    "        if stage == \"fit\" or stage is None:\n",
    "            mnist_full = MNIST(self.data_dir, train=True, transform=self.transform)\n",
    "            self.mnist_train, self.mnist_val = random_split(mnist_full, [55000, 5000])\n",
    "\n",
    "            \n",
    "\n",
    "        # Assign test dataset for use in dataloader(s)\n",
    "        if stage == \"test\" or stage is None:\n",
    "            self.mnist_test = MNIST(self.data_dir, train=False, transform=self.transform)\n",
    "\n",
    "\n",
    "    def train_dataloader(self):\n",
    "        return self.create_dataloader(self.mnist_train)\n",
    "\n",
    "    def val_dataloader(self):\n",
    "        return self.create_dataloader(self.mnist_val)\n",
    "\n",
    "    def test_dataloader(self):\n",
    "        return self.create_dataloader(self.mnist_test)\n",
    "\n",
    "        \n",
    "        \n",
    "\n",
    "        \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "dcea2f7d-cb12-44df-96b0-e832632e905f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: True (cuda), used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "HPU available: False, using: 0 HPUs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "\n",
      "  | Name          | Type               | Params\n",
      "-----------------------------------------------------\n",
      "0 | l1            | Linear             | 50.2 K\n",
      "1 | l2            | Linear             | 4.2 K \n",
      "2 | l3            | Linear             | 325   \n",
      "3 | dropout       | Dropout            | 0     \n",
      "4 | relu          | ReLU               | 0     \n",
      "5 | val_accuracy  | MulticlassAccuracy | 0     \n",
      "6 | test_accuracy | MulticlassAccuracy | 0     \n",
      "-----------------------------------------------------\n",
      "54.7 K    Trainable params\n",
      "0         Non-trainable params\n",
      "54.7 K    Total params\n",
      "0.219     Total estimated model params size (MB)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/sunil/miniconda3/envs/lora_from_scratch/lib/python3.11/site-packages/lightning/pytorch/loops/fit_loop.py:280: PossibleUserWarning: The number of training batches (28) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.\n",
      "  rank_zero_warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0: 100%|█████████████████████████| 28/28 [00:03<00:00,  8.83it/s, v_num=1]\n",
      "Validation: 0it [00:00, ?it/s]\u001b[A\n",
      "Validation:   0%|                                         | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:   0%|                            | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:  33%|██████▎            | 1/3 [00:00<00:00, 271.41it/s]\u001b[A\n",
      "Validation DataLoader 0:  67%|████████████▋      | 2/3 [00:00<00:00, 315.63it/s]\u001b[A\n",
      "Epoch 0: 100%|█| 28/28 [00:04<00:00,  6.79it/s, v_num=1, val_loss=1.050, val_acc\u001b[A\n",
      "Epoch 1: 100%|█| 28/28 [00:03<00:00,  8.50it/s, v_num=1, val_loss=1.050, val_acc\u001b[A\n",
      "Validation: 0it [00:00, ?it/s]\u001b[A\n",
      "Validation:   0%|                                         | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:   0%|                            | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:  33%|██████▎            | 1/3 [00:00<00:00, 248.89it/s]\u001b[A\n",
      "Validation DataLoader 0:  67%|████████████▋      | 2/3 [00:00<00:00, 282.01it/s]\u001b[A\n",
      "Epoch 1: 100%|█| 28/28 [00:04<00:00,  6.47it/s, v_num=1, val_loss=0.473, val_acc\u001b[A\n",
      "Epoch 2: 100%|█| 28/28 [00:03<00:00,  8.53it/s, v_num=1, val_loss=0.473, val_acc\u001b[A\n",
      "Validation: 0it [00:00, ?it/s]\u001b[A\n",
      "Validation:   0%|                                         | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:   0%|                            | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:  33%|██████▎            | 1/3 [00:00<00:00, 273.62it/s]\u001b[A\n",
      "Validation DataLoader 0:  67%|████████████▋      | 2/3 [00:00<00:00, 317.64it/s]\u001b[A\n",
      "Epoch 2: 100%|█| 28/28 [00:04<00:00,  6.56it/s, v_num=1, val_loss=0.271, val_acc\u001b[A\n",
      "Epoch 3: 100%|█| 28/28 [00:03<00:00,  8.59it/s, v_num=1, val_loss=0.271, val_acc\u001b[A\n",
      "Validation: 0it [00:00, ?it/s]\u001b[A\n",
      "Validation:   0%|                                         | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:   0%|                            | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:  33%|██████▎            | 1/3 [00:00<00:00, 313.41it/s]\u001b[A\n",
      "Validation DataLoader 0:  67%|████████████▋      | 2/3 [00:00<00:00, 357.63it/s]\u001b[A\n",
      "Epoch 3: 100%|█| 28/28 [00:04<00:00,  6.49it/s, v_num=1, val_loss=0.208, val_acc\u001b[A\n",
      "Epoch 4: 100%|█| 28/28 [00:03<00:00,  8.41it/s, v_num=1, val_loss=0.208, val_acc\u001b[A\n",
      "Validation: 0it [00:00, ?it/s]\u001b[A\n",
      "Validation:   0%|                                         | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:   0%|                            | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:  33%|██████▎            | 1/3 [00:00<00:00, 302.71it/s]\u001b[A\n",
      "Validation DataLoader 0:  67%|████████████▋      | 2/3 [00:00<00:00, 355.34it/s]\u001b[A\n",
      "Epoch 4: 100%|█| 28/28 [00:04<00:00,  6.48it/s, v_num=1, val_loss=0.179, val_acc\u001b[A\n",
      "Epoch 5: 100%|█| 28/28 [00:03<00:00,  8.46it/s, v_num=1, val_loss=0.179, val_acc\u001b[A\n",
      "Validation: 0it [00:00, ?it/s]\u001b[A\n",
      "Validation:   0%|                                         | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:   0%|                            | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:  33%|██████▎            | 1/3 [00:00<00:00, 303.08it/s]\u001b[A\n",
      "Validation DataLoader 0:  67%|████████████▋      | 2/3 [00:00<00:00, 347.33it/s]\u001b[A\n",
      "Epoch 5: 100%|█| 28/28 [00:04<00:00,  6.33it/s, v_num=1, val_loss=0.162, val_acc\u001b[A\n",
      "Epoch 6: 100%|█| 28/28 [00:03<00:00,  8.42it/s, v_num=1, val_loss=0.162, val_acc\u001b[A\n",
      "Validation: 0it [00:00, ?it/s]\u001b[A\n",
      "Validation:   0%|                                         | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:   0%|                            | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:  33%|██████▎            | 1/3 [00:00<00:00, 304.84it/s]\u001b[A\n",
      "Validation DataLoader 0:  67%|████████████▋      | 2/3 [00:00<00:00, 355.07it/s]\u001b[A\n",
      "Epoch 6: 100%|█| 28/28 [00:04<00:00,  6.58it/s, v_num=1, val_loss=0.149, val_acc\u001b[A\n",
      "Epoch 7: 100%|█| 28/28 [00:03<00:00,  8.59it/s, v_num=1, val_loss=0.149, val_acc\u001b[A\n",
      "Validation: 0it [00:00, ?it/s]\u001b[A\n",
      "Validation:   0%|                                         | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:   0%|                            | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:  33%|██████▎            | 1/3 [00:00<00:00, 279.88it/s]\u001b[A\n",
      "Validation DataLoader 0:  67%|████████████▋      | 2/3 [00:00<00:00, 176.67it/s]\u001b[A\n",
      "Epoch 7: 100%|█| 28/28 [00:04<00:00,  6.53it/s, v_num=1, val_loss=0.140, val_acc\u001b[A\n",
      "Epoch 8: 100%|█| 28/28 [00:03<00:00,  8.36it/s, v_num=1, val_loss=0.140, val_acc\u001b[A\n",
      "Validation: 0it [00:00, ?it/s]\u001b[A\n",
      "Validation:   0%|                                         | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:   0%|                            | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:  33%|██████▎            | 1/3 [00:00<00:00, 316.67it/s]\u001b[A\n",
      "Validation DataLoader 0:  67%|████████████▋      | 2/3 [00:00<00:00, 357.34it/s]\u001b[A\n",
      "Epoch 8: 100%|█| 28/28 [00:04<00:00,  6.40it/s, v_num=1, val_loss=0.133, val_acc\u001b[A\n",
      "Epoch 9: 100%|█| 28/28 [00:03<00:00,  8.59it/s, v_num=1, val_loss=0.133, val_acc\u001b[A\n",
      "Validation: 0it [00:00, ?it/s]\u001b[A\n",
      "Validation:   0%|                                         | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:   0%|                            | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:  33%|██████▎            | 1/3 [00:00<00:00, 299.10it/s]\u001b[A\n",
      "Validation DataLoader 0:  67%|████████████▋      | 2/3 [00:00<00:00, 347.64it/s]\u001b[A\n",
      "Epoch 9: 100%|█| 28/28 [00:04<00:00,  6.47it/s, v_num=1, val_loss=0.129, val_acc\u001b[A\n",
      "Epoch 9: 100%|█| 28/28 [00:04<00:00,  6.47it/s, v_num=1, val_loss=0.129, val_acc\u001b[A"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`Trainer.fit` stopped: `max_epochs=10` reached.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 9: 100%|█| 28/28 [00:04<00:00,  6.46it/s, v_num=1, val_loss=0.129, val_acc\n"
     ]
    }
   ],
   "source": [
    "model = LitMNIST()\n",
    "trainer = L.Trainer(\n",
    "    accelerator=\"auto\",\n",
    "    devices=1,\n",
    "    max_epochs=10,\n",
    "    logger=CSVLogger(save_dir=\"logs/\"),\n",
    ")\n",
    "trainer.fit(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "fa45d80a-ea43-41a5-a8f4-a60ac95717bb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>val_loss</th>\n",
       "      <th>val_acc</th>\n",
       "      <th>train_loss</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>epoch</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.045979</td>\n",
       "      <td>0.861719</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.609744</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.472707</td>\n",
       "      <td>0.913672</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.271100</td>\n",
       "      <td>0.930078</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.233079</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       val_loss   val_acc  train_loss\n",
       "epoch                                \n",
       "0      1.045979  0.861719         NaN\n",
       "1           NaN       NaN    0.609744\n",
       "1      0.472707  0.913672         NaN\n",
       "2      0.271100  0.930078         NaN\n",
       "3           NaN       NaN    0.233079"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<seaborn.axisgrid.FacetGrid at 0x7f7a39132110>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkMAAAHpCAYAAACBVjiOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABlK0lEQVR4nO3dd3iUVd7G8e/MZDLpCemUQGgJHaWIFJUmCIJYEFQUUVFZ2yqWFV1ZxcKru3Y22GV1UbGhKLqKIkhTikSREnonJCSQ3qa8f0wKAQJJmMwkmftzXXNl5plnnvObrJp7zznPOQaHw+FARERExEsZPV2AiIiIiCcpDImIiIhXUxgSERERr6YwJCIiIl5NYUhERES8msKQiIiIeDWFIREREfFqDSIMORwOsrOz0ZJIIiIi4moNIgzl5OQQGhpKTk6Op0sRERGRRqZBhCERERGRuqIwJCIiIl5NYUhERES8msKQiIiIeDWFIREREfFqCkMiIiLi1RSGRERExKspDImIiIhXUxgSERERr6YwJCIiIl5NYUhERES8msKQiIiIeDWFIREREfFqCkMiIiLi1RSGRERExKspDImIiIhXUxgSERERr+ZVYWjRpsPc/eF65q3Z6+lSREREpJ7wqjC0PS2Xr34/yPLtGZ4uRUREROoJrwpDibFBAGxNzfFwJSIiIlJfeFUYSogJBmBHei7FVruHqxEREZH6wKvCUPMwf4IsPljtDnZn5Hm6HBEREakHvCoMGQwGEmKcQ2UpGioTERERvCwMASTGOofKFIZEREQEvDAMlc0bSjmsMCQiIiJeGIYSS8PQVoUhERERwQvDUELpMNnezHzyi60erkZEREQ8zevCUGSQhcggXxwO2HY419PliIiIiId5XRgCzRsSERGRCl4dhrQStYiIiHhlGCq/vV49QyIiIl7Pu8OQeoZERES8nleGofbRzlWo03KKOJpX7OFqRERExJO8MgwF+5lpHuYPaL0hERERb1fjMPTzzz8zevRomjVrhsFg4IsvvjjjZ5YuXUrPnj3x8/OjTZs2vPbaa7Wp1aXKhsoUhkRERLxbjcNQXl4e3bt3Z9asWdU6f9euXYwcOZILLriA9evX88gjj3DPPffw2Wef1bhYVyoLQ1s0b0hERMSr+dT0AyNGjGDEiBHVPv+1116jZcuWvPTSSwB07NiRtWvX8q9//Yurrrqqps27jLblEBEREXDDnKFVq1YxbNiwSseGDx/O2rVrKSkpOeVnioqKyM7OrvRwtfKFF1NzcDgcLr++iIiINAx1HoZSU1OJiYmpdCwmJgar1cqRI0dO+ZmZM2cSGhpa/oiLi3N5XW2jAzEZDWQXWjmcXeTy64uIiEjD4Ja7yQwGQ6XXZT0xJx4vM23aNLKyssof+/btc3lNFh8TrSMDAdiS6vqeJxEREWkY6jwMxcbGkpqaWulYWloaPj4+REREnPIzFouFkJCQSo+6oHlDIiIiUudhqG/fvixatKjSse+//55evXphNpvruvnTqpg3pN3rRUREvFWNw1Bubi7JyckkJycDzlvnk5OT2bt3L+Ac4po4cWL5+VOmTGHPnj1MnTqVzZs388477/D222/zwAMPuOYbnIXEWOdK1OoZEhER8V41vrV+7dq1DBo0qPz11KlTAbjxxhuZM2cOhw4dKg9GAK1bt+abb77hvvvu49///jfNmjXjlVde8eht9WUSY53Db9vScrDZHZiMp57DJCIiIo2XwdEA7ivPzs4mNDSUrKwsl84fstkddJr+P4qsdn56YGD5hGoRERHxHl65N1kZk9FA+xjnUJl2sBcREfFOXh2GoGISteYNiYiIeCevD0MdSvcoS1EYEhER8UpeH4aO35ZDREREvI/Xh6Gy3et3HcmjyGrzcDUiIiLibl4fhmJD/Aj288Fmd7AzPc/T5YiIiIibeX0YMhgM5fOGNIlaRETE+9R40cXGKCEmmDW7j7IlNYcxni5GRKSxs9vBYQN76dQEs5/zp80KBZnO42Xvl/0sex7eBsz+zvPTt0J+hvN4814V1xGpIYUhKuYNbdUkahGpLocDrEWAo+KPc85hSNsExblQnHfcz9KH3Qojnq24xkcTID+z6jbOuxW6XOl8vnE+/PpG1ecGhMM1cytezxlVETZO5aIHoe1g5/N1c+D3eVV9UWcAuTzJ+bIwC94ZcXJYcdgrXo99F+L7O89feD+sn1v5/OO1GQQTv3A+P7oLZvWqumaAKcshtqvz+aLpsPVb5/O//g5N4k//WZEqKAxx3B1lGiYTafyK8yBz1wlh5YSfFzxQ0cuw8AFI33JCsMmtCDcXPABDHnOeu3MJzL+t6rYNRrjk/8BQuvXPgXWQc6jq8zuOqniekwp7V1Z9blBs5dd7fwF7SdXn5x2peH5s7+mvXXTCfxvTNlZ9LoC1oOK5raTy6xM57BXPDaUzNwwmMJpO+Gl0/jQcN7sjpKkzqBlKzxGpJYUhILE0DO0/WkBukZUgi34tIm5ltzn/aJYFkJJC5x/ckgLnoziv9Hl+xbGeN0JQtPP8FS/DvtUV75XkVe6R6TQGxsxynntgHfxn9Onr6X1rRS0H18OBtVWfW3zcjRdB0RDdGXwDKx6W4ONeBzn/+Jf94R71ElgLq752TJeK5+2HQXDTqs/1OWGIaOzbzt6rqjTvWfG8y1iI7Vb1uf5hFc/NgXDDF1WHFaOpcg/NkOlwwVQw+pz6fJO54tzwNvB4VtV1nGjUi9U/V+Q09FcfaBLoS3SwhbScIrYezqFHyyaeLkmk/igbDirKAVsxhDZ3HreVwPYfKgJKcf5xYaX0Z+9bILqj8/yVr8KmBSeEmtKftiJoMxAmfuk8N/sAvDn49HW1H1oRhvavgS1fV33u8T0bfqEQGF0RTo4PLr5BYAmq/Ad64MNQlA2+J4Sa4z9Xpu0guOM0PSwnSryk+udGtHU+qqtTDWZAxnRyPqrD5OP8ntUVGOl8VIdBm2WLZygMlUqMDXaGoVSFIWlkctOcYaAwy/mz0iPL2ePQtLvz3N8/gt/ec/7xLzunMLtiuCUyEe5a7XxuK4EPrzl9220HV4ShY/tg/+qqzy05bijFNxBC45xzccwBpQ//4177O0NNmR43OueelJ8X4Aw1ZcHF/7h/p5t2hwe3Ve93B9D+4uqfKyINksJQqYSYYJZtO6J5Q+J5Doezx6TwuEASFAVhLZ3vp/4JKd9WDizHPwf4y4qK673a0/l+VfzDK8JQTirsWVH1ucfz8XMOtVQKKyeElvA2Feefez20vvCE84776RtQcW5wLNz3Z/XqAAUWETkrCkOlErVhq7iStRhyU6HgmLNHpvxx3OuLZ4CPxXn+h9dB6h8Voeb4SaUAFz4Ig//ufH54I/z0VNVtG4zOQFU25OAXBhicc1dO9YhoV/HZxBHO0GUJAb+Qyuf5Bjnne5QxGuHWxdX/nTTt5nyIiNQzCkOlym6v1x5lgsPhvFuoMMs5t8TH13l8x2LnuiYnhpqy122HwMVPOM/duwreu+z07QyYCsExzud5aZC1r/L7BmNpEAmpuHUbILK9c1io7L2ysFIeXkIqh6F7/6j+XIyoROdDRMSLKAyVah8TBMCR3GKO5BYRGWTxcEVy1koKnLct56Y5A0vBMWdY6DDS+X7OYfjmgVP02mRXrIVy+88VQ0hr34HNX1XdXpPWFc/9Qp3DSH6hzp4Zv9CKh3/p6+Mn6Y78l/M27bJw4xfiHDo6VYhp3sP5qC5NShUROS2FoVIBvj60DA9gb2Y+Ww/nKAzVZ7YSZ8DJSYWcg9CsR8UdTqv+Db+97wxBhcdO/myL8yrCEMDmBVW3YzRXvm067nzn7cFVBZzQuIpzm3aHvx+u/ndqdk71zxUREZdSGDpOQkywMwyl5tCvbTVvBRXXsdudS+sXZFYM1Tgc8PV9znCTfdAZgPLSgePWT7nyTeg2zvm8MAvSN1e85+PvHIryb+IMLdHH3T7s38TZI1NVz42PX+VelX53Vf+7qDdGRKTBUBg6TofYYH7YfFh3lLmaw+FcVM9U+o9b2hZIWVjas3Oo9Gfpw17iHCqaVjp/xmBw9t7kZ1S+ptHHueJuSNPK82m6joOW50NwM+cdSX6hVQcTH1/ndgciIuLVFIaOk6BJ1LV34DfnvkLZh44LOMc97zMFhv7DeW7aJvhxRhUXMjh7ZEoKKkLOwGnO+TXBTZ0BJ7gZBEQ472Y6UWQ750NERKSaFIaOU3F7fS4OhwODhjqcHA7nHJ30zZCeAmmlP6+Z69wcEuD7x2DP8qqvcfz+S1EdoPt1zmATUtqDUxZ0gmIqTywG9d6IiEidUhg6TuvIQHyMBnKLrBzMKqR5mP+ZP9RY5aY717IpCz+nmox8ZKtzSApKJwA7juu9OcXPMjGd4IrZbvgSIiIiZ6YwdBxfHyNto4JIOZzD1tScxhuGHA5nT01ZD0/6FufDNwhu+Nx5jtkP1s2p+IzB6Nx8Mapj6Vo0HSD8uH2Shj/tzm8gIiLiMgpDJ0iIDSblcA5bUnMY1CHa0+WcHYfDubFm2SrHW76B5S84A9CptmfwDa5YrM8SDEP+4bxdPLqDc5VicyMNhyIi4tUUhk6QGBPEVzSwbTkcDufqxcfP50nf4vzZ904YNM15nq3Iubs3gMHk3AE7KrFyb8/xLpjq3u8hIiLiAQpDJ0iIqcd3lNntztATHFvR2/PlnbDxC+f2EaeSvqXiect+MPZdZ+iJaFtxDRERES+mMHSCDrEhAGxPz8Vqs+NjOsXt23XN4YCju0t7eI67g+vIVudu5rf8AHG9K84vznWulhzRztnDE13FvJ7gGOhypdu/joiISH2mMHSCFk388TebKCixsTsjn3bRQe4tIGs//Hds5VWUj2fyhewDQGkYGjAV+t0D4W1OviVdREREzkhh6ARGo4GEmCB+35/F1sM57g9DwU3BYXeGnsjEih6e6A7On01aV6zkDM7hLhEREak1haFTSIgJ5vf9WaSk5jCya9Mzf+Bs7FvjXM/nkv9zDm8ZTXD1HOdihP5hddu2iIiI4IEJMfVfYmzZStR1OIn68Eb48Fp4eyjsXAJL/q/ivZhOCkIiIiJuop6hU0isyz3KMnbAkpmw4VPA4VzM8Jzr4KK/ub4tEREROSOFoVMo26Nsd0YehSU2/Myms79oTqqz92f9+2C3Oo91uhwGPQpRCWd/fREREakVhaFTiAq2EBZg5lh+CdvTcunSPPTsL5qfWbq9hQPaXQyD/166n5eIiIh4kuYMnYLBYDhuB/taDpUVZsMvr4Hd5nwd0wmGTIebvoXrP1UQEhERqSfUM1SFxNhgft2VWfN5QyUFsOZtWPY8FGSCfxPoPt75nra3EBERqXcUhqpQvi1HdXuGbCWw/r+w9DnIOeg8FtEeAsLrqEIRERFxBYWhKpTfXn+mniG7HTZ+Dj89DZk7ncdC42Dgw9DtmsoLJIqIiEi9o7/UVUiIdoahg1mFZBeWEOJXxVYX696Bhfc7nwdEwoUPQq+btAmqiIhIA6EJ1FUIDTDTNNQPOEXv0LG9Fc+7XePcIHXw3+Gvv8P5UxSEREREGhCFodM4ad7Q/nXw3hj49/mQm+Y8ZgmCO1c7e4Qsbt7HTERERM6awtBplM0bOrrrd/hoArw12Ll1hq0Y9qysONHogkUZRURExCM0Z+g0zgnO4nlzEldsWUH51hndroGBf4Mm8Z4uT0RERFxAYagqq/7NJT/9A6OpBABHx9EYBv0dojt4uDARERFxJQ2TVSW6I0Z7CT/bu3JZ0ZOkj3xLQUhERKQRUhgCKMqFn/8JH99YcazNILhtCf8IeYo/HG3ZmprrufpERESkznh3GCophF9mw8vdYfFTsOkL2Pur8z2DAZqdS0KM8w6xaq9ELSIiIg2Kd84Zslnh9w9gybOQvd95LLwNDHoUWvSudGpibAjfbTx85pWoRUREpEHyvjC08QtY/CRkbHe+Dm7mvDvsnAlgOnmV6bLd67eoZ0hERKRR8r4wtOtnZxAKiIAL7odet4DZr8rTE2Odw2TbDudgtzswGg3uqlRERETcwPvC0IUPQlAMnP8X8As54+mtIgLxNRnJL7Zx4FgBceEBbihSRERE3MX7JlCHNHUOi1UjCAGYTUbaRpdOota8IRERkUbH+8JQLSTqjjIREZFGS2GoGhJK9yhTz5CIiEjjozBUDWV3lG1Vz5CIiEijozBUDWW71+9Iz6XEZvdwNSIiIuJKCkPV0DzMn0BfEyU2B7uO5Hm6HBEREXEhhaFqMBgMmjckIiLSSCkMVZPmDYmIiDROCkPVlKieIRERkUZJYaiaynqGtNaQiIhI46IwVE1lc4b2ZuaTX2z1cDUiIiLiKgpD1RQZZCEi0BeHA7an5Xq6HBEREXERhaEa0LwhERGRxkdhqAYSYhSGREREGhuFoRoo7xnSJGoREZFGQ2GoBhK01pCIiEijozBUAwkxQQAczi7iWH6xh6sRERERV1AYqoFgPzPNw/wBzRsSERFpLBSGaqhs3pCGykRERBoHhaEaStBK1CIiIo2KwlANJcY65w1tTdXCiyIiIo1BrcJQUlISrVu3xs/Pj549e7Js2bLTnj937ly6d+9OQEAATZs25aabbiIjI6NWBXtaYkwI4OwZcjgcHq5GREREzlaNw9C8efO49957efTRR1m/fj0XXHABI0aMYO/evac8f/ny5UycOJFbbrmFjRs38sknn7BmzRomT5581sV7QpuoQExGA1kFJRzOLvJ0OSIiInKWahyGXnjhBW655RYmT55Mx44deemll4iLi2P27NmnPP+XX34hPj6ee+65h9atWzNgwABuv/121q5de9bFe4Kf2UR8RACgeUMiIiKNQY3CUHFxMevWrWPYsGGVjg8bNoyVK1ee8jP9+vVj//79fPPNNzgcDg4fPsynn37KpZdeWmU7RUVFZGdnV3rUJ+V3lOn2ehERkQavRmHoyJEj2Gw2YmJiKh2PiYkhNTX1lJ/p168fc+fOZfz48fj6+hIbG0tYWBivvvpqle3MnDmT0NDQ8kdcXFxNyqxzx88bEhERkYatVhOoDQZDpdcOh+OkY2U2bdrEPffcw/Tp01m3bh3/+9//2LVrF1OmTKny+tOmTSMrK6v8sW/fvtqUWWfK7ijTwosiIiINn09NTo6MjMRkMp3UC5SWlnZSb1GZmTNn0r9/fx588EEAunXrRmBgIBdccAFPPfUUTZs2PekzFosFi8VSk9LcqmytoW1pOdjsDkzGUwdBERERqf9q1DPk6+tLz549WbRoUaXjixYtol+/fqf8TH5+PkZj5WZMJhNAg701vVVEIBYfI4UldvZl5nu6HBERETkLNR4mmzp1Km+99RbvvPMOmzdv5r777mPv3r3lw17Tpk1j4sSJ5eePHj2azz//nNmzZ7Nz505WrFjBPffcw3nnnUezZs1c903cyGQ00L5001bNGxIREWnYajRMBjB+/HgyMjKYMWMGhw4dokuXLnzzzTe0atUKgEOHDlVac2jSpEnk5OQwa9Ys7r//fsLCwhg8eDDPPvus676FByTEBPPngWxSUnMY3jnW0+WIiIhILRkcDWCsKjs7m9DQULKysggJCfF0OQC8vnQHM7/dwqXdmvLv63p4uhwRERGpJe1NVksJWmtIRESkUVAYqqUOpWFo15E8iqw2D1cjIiIitaUwVEuxIX4E+/lgtTvYmZ7n6XJERESklhSGaslgMJBYut7QVt1RJiIi0mApDJ2FsnlDWolaRESk4VIYOgtl84bUMyQiItJwKQydhbJtObaoZ0hERKTBUhg6C2VhaP/RAnKLrB6uRkRERGpDYegshAf6EhXs3FB2m4bKREREGiSFobOkeUMiIiINm8LQWSobKktJzfVwJSIiIlIbCkNnqWytoZTD2R6uRERERGpDYegsVaw1pJ4hERGRhkhh6Cy1jw4C4EhuERm5RR6uRkRERGpKYegsBVp8aBkeAMDWw+odEhERaWgUhlygYhK15g2JiIg0NApDLpAY6xwqS1HPkIiISIOjMOQCCdq9XkREpMFSGHKBDrEhAGxNzcHhcHi4GhEREakJhSEXaB0ZiI/RQE6RlYNZhZ4uR0RERGpAYcgFfH2MtIkKBJy9QyIiItJwKAy5SPkdZZo3JCIi0qAoDLlI+Yat6hkSERFpUBSGXKSsZ2iLwpCIiEiDojDkIomlPUPb03Ox2uwerkZERESqS2HIReKaBOBvNlFstbMnM9/T5YiIiEg1KQy5iNFoICHGuRK15g2JiIg0HApDLqR5QyIiIg2PwpALlc0b0rYcIiIiDYfCkAtprSEREZGGR2HIhcrWGtp9JI/CEpuHqxEREZHqUBhyoahgC2EBZuwO2J6W6+lyREREpBoUhlzIYDCUD5Vp3pCIiEjDoDDkYomaNyQiItKgKAy5WKL2KBMREWlQFIZcrOL2es0ZEhERaQgUhlwsIdoZhg4cKyC7sMTD1YiIiMiZKAy5WGiAmdgQPwC2ad6QiIhIvacwVAfKhspSUjVUJiIiUt8pDNUBbcshIiLScCgM1YGKDVuzPVyJiIiInInCUB0oX2soNQeHw+HhakREROR0FIbqQLvoIAwGOJpfwpHcYk+XIyIiIqehMFQH/H1NxEcEApo3JCIiUt8pDNWRhJggALZoJWoREZF6TWGojpTNG9K2HCIiIvWbwlAdSYjVhq0iIiINgcJQHelQGoa2Hc7BbtcdZSIiIvWVwlAdaRURiK/JSF6xjQPHCjxdjoiIiFRBYaiOmE1G2kQ57yhL0bwhERGRekthqA4lat6QiIhIvacwVIe0R5mIiEj9pzBUh47flkNERETqJ4WhOlS2YeuO9FxKbHYPVyMiIiKnojBUh5qH+RPoa6LE5mD3kTxPlyMiIiKnoDBUh4xGgxZfFBERqecUhuqY5g2JiIjUbwpDdSxBYUhERKReUxiqY7q9XkREpH5TGKpjZWFoT2Y+BcU2D1cjIiIiJ1IYqmORQRYiAn1xOGB7Wq6nyxEREZETKAy5Qdm8oS2p2R6uRERERE6kMOQGmjckIiJSfykMuUHFhq0aJhMREalvFIbcoGyYbKturxcREal3FIbcICEmCIDU7EKy8ks8XI2IiIgcT2HIDYL9zDQP8we0LYeIiEh9ozDkJmW9QwpDIiIi9YvCkJskxoYAmjckIiJS3ygMuUlibGnPkMKQiIhIvaIw5CblG7YezsHhcHi4GhERESmjMOQmbaOCMBkNZBWUkJZT5OlyREREpJTCkJv4mU3ERwQAZx4qyynOYdb6Waw8sNIdpYmIiHi1WoWhpKQkWrdujZ+fHz179mTZsmWnPb+oqIhHH32UVq1aYbFYaNu2Le+8806tCm7IyleiPkMYevfPd3n9j9d58bcXsTvs7ihNRETEa/nU9APz5s3j3nvvJSkpif79+/P6668zYsQINm3aRMuWLU/5mXHjxnH48GHefvtt2rVrR1paGlar9ayLb2gSYoL5ZkPqGW+vn9hpIr8e+pVJXSZhwOCm6kRERLyTwVHD2bx9+vShR48ezJ49u/xYx44dufzyy5k5c+ZJ5//vf//jmmuuYefOnYSHh9eqyOzsbEJDQ8nKyiIkJKRW16gPvt1wiL/M/Y1uLUJZcNeAan3GaneGRh9jjXOriIiIVEONhsmKi4tZt24dw4YNq3R82LBhrFx56vktCxYsoFevXjz33HM0b96chIQEHnjgAQoKCqpsp6ioiOzs7EqPxuD43evt9jNn0J/3/8yVC67ky+1f1nVpIiIiXqtGYejIkSPYbDZiYmIqHY+JiSE1NfWUn9m5cyfLly/nzz//ZP78+bz00kt8+umn3HnnnVW2M3PmTEJDQ8sfcXFxNSmz3moVEYivj5HCEjt7M/PPeP6e7D3sytrFRykf6XZ8ERGROlKrCdQGQ+V5LA6H46RjZex2OwaDgblz53LeeecxcuRIXnjhBebMmVNl79C0adPIysoqf+zbt682ZdY7JqOB9tHV35ZjXOI47u1xL+8Of7fK36+IiIgrxcfH89JLL1XrXIPBwBdffFGn9bhDjcJQZGQkJpPppF6gtLS0k3qLyjRt2pTmzZsTGhpafqxjx444HA72799/ys9YLBZCQkIqPRqLxNLFF6uzLYfFZOGWrrcQ5BvExoyN5Bbn1nV5IiIiXqdGYcjX15eePXuyaNGiSscXLVpEv379TvmZ/v37c/DgQXJzK/6Qb926FaPRSIsWLWpRcsNWfnt9DTZsfX7t81zz9TW8v+n9uipLRETEa9V4mGzq1Km89dZbvPPOO2zevJn77ruPvXv3MmXKFMA5xDVx4sTy86+77joiIiK46aab2LRpEz///DMPPvggN998M/7+/q77Jg1EwnGTqKurc2RnDBjIKMyoq7JEROQMHA4H+cVWtz9qMmf09ddfp3nz5tjtldeou+yyy7jxxhvZsWMHY8aMISYmhqCgIHr37s0PP/zgst/Rhg0bGDx4MP7+/kRERHDbbbdV6gxZsmQJ5513HoGBgYSFhdG/f3/27NkDwO+//86gQYMIDg4mJCSEnj17snbtWpfVdjo1vl97/PjxZGRkMGPGDA4dOkSXLl345ptvaNWqFQCHDh1i79695ecHBQWxaNEi7r77bnr16kVERATjxo3jqaeect23aEDKhsl2pudRbLXj63PmPDqs1TASLk+gTWibui5PRESqUFBio9P079ze7qYZwwnwrd6f66uvvpp77rmHn376iSFDhgBw9OhRvvvuO7766ityc3MZOXIkTz31FH5+fvznP/9h9OjRpKSkVLlWYHXl5+dzySWXcP7557NmzRrS0tKYPHkyd911F3PmzMFqtXL55Zdz66238uGHH1JcXMzq1avL58ROmDCBc889l9mzZ2MymUhOTsZsNp9VTdVVq8Vr7rjjDu64445TvjdnzpyTjnXo0OGkoTVv1TTUj2A/H3IKrew8kkuH2DPPhzIajLQJbcOxwmO8s/EdrutwHbGBsW6oVkREGpLw8HAuueQSPvjgg/Iw9MknnxAeHs6QIUMwmUx07969/PynnnqK+fPns2DBAu66666zanvu3LkUFBTw3nvvERgYCMCsWbMYPXo0zz77LGazmaysLEaNGkXbtm0B5xziMnv37uXBBx+kQ4cOALRv3/6s6qkJreTnZgaDgcSYYNbuOUpKak61wlCZR5Y/wrIDy8gpzuEfff9Rh1WKiMiJ/M0mNs0Y7pF2a2LChAncdtttJCUlYbFYmDt3Ltdccw0mk4m8vDyeeOIJvv76aw4ePIjVaqWgoKDSiE5tbd68me7du5cHIXDOG7bb7aSkpHDhhRcyadIkhg8fzsUXX8zQoUMZN24cTZs2BZzTcCZPnsz777/P0KFDufrqq8tDU13TRq0eUJt5QwCTu06mQ3gHBscNrouyRETkNAwGAwG+Pm5/1HRpldGjR2O321m4cCH79u1j2bJlXH/99QA8+OCDfPbZZzz99NMsW7aM5ORkunbtSnFx8Vn/fk63zE7Z8XfffZdVq1bRr18/5s2bR0JCAr/88gsAjz/+OBs3buTSSy9l8eLFdOrUifnz5591XdWhMOQBZfOGzrRh64l6xPTg41Efc0GLC7SBq4iInJK/vz9XXnklc+fO5cMPPyQhIYGePXsCsGzZMiZNmsQVV1xB165diY2NZffu3S5pt1OnTiQnJ5OXl1d+bMWKFRiNRhISEsqPnXvuuUybNo2VK1fSpUsXPvjgg/L3EhISuO+++/j++++58soreffdd11S25koDHlAQkzNb68vYzAYWLhzIaPmj2JL5hZXlyYiIo3AhAkTWLhwIe+88055rxBAu3bt+Pzzz0lOTub333/nuuuuO+nOs7Np08/PjxtvvJE///yTn376ibvvvpsbbriBmJgYdu3axbRp01i1ahV79uzh+++/Z+vWrXTs2JGCggLuuusulixZwp49e1ixYgVr1qypNKeoLikMeUDZWkP7MgvIK7LW+PNL9y1lX84+5myc4+LKRESkMRg8eDDh4eGkpKRw3XXXlR9/8cUXadKkCf369WP06NEMHz6cHj16uKTNgIAAvvvuOzIzM+nduzdjx45lyJAhzJo1q/z9LVu2cNVVV5GQkMBtt93GXXfdxe23347JZCIjI4OJEyeSkJDAuHHjGDFiBE888YRLajuTGu9a7wmNZdf64/V++gfSc4r44s7+nBMXVqPP7snew6I9i7iuw3UEmAPqpkAREREvoZ4hD6mYN5Rd48+2CmnF5K6TCTAHsCVzizZxFREROQsKQx5SPm8otXb7jTkcDqYumcrVX13NioMrXFmaiIgIc+fOJSgo6JSPzp07e7o8l9I6Qx7SoZa315cxGAw0C2yGyWAiJTOFAc0HuLI8ERHxcpdddhl9+vQ55XvuWhnaXRSGPCShFhu2nmhy18mMTRhLfGi8i6oSERFxCg4OJjg42NNluIWGyTykfXQQAOk5RWTm1W6xqzC/MOJD4zlScISk5CSs9prfmSYiIuLtFIY8JNDiQ1y4P1DzxRePZ7PbmLBwArN/n82CHQtcVZ6IiIjXUBjyoLI7ymo7bwjAZDRxXcfr6BbZjfiQeBdVJiIi4j00Z8iDEmOD+WFz2lnNGwKY0HECEztNxGAwnHZvGBERETmZeoY8KKGWe5SdyMfog81h49OtnzL+6/HkleSd+UMiIiICKAx5VNm2HFtTc1yycOKcjXPYnLmZj1M+PutriYiId4qPj+ell17ydBlupWEyD2oTGYSP0UBOkZVDWYU0C/Ov9bV8jD7c3/N+9ufuZ1ziOBdWKSIi0rgpDHmQr4+RNlGBbD2cS8rhnLMKQwCDWg4qf74vZx9xwXFnW6KIiEijp2EyD3PVvKEyucW5TP5+Mld8eQWH8w675JoiInKc4rzTP2zHrflmLT79uSUFFec6HCe/X0Ovv/46zZs3x263Vzp+2WWXceONN7Jjxw7GjBlDTEwMQUFB9O7dmx9++KG2vwleeOEFunbtSmBgIHFxcdxxxx3k5lbeZmrFihVcdNFFBAQE0KRJE4YPH87Ro0cBsNvtPPvss7Rr1w6LxULLli15+umna11PbalnyMMSY4L5mkNsdVEYCjQHUmIrwe6wsz59PZcEXuKS64qISKlnmp3+/avnQOcrnM8Xz4CVr1Z9brNz4bYlzuf5GfDPtpXffzyrRqVdffXV3HPPPfz0008MGTIEgKNHj/Ldd9/x1VdfkZuby8iRI3nqqafw8/PjP//5D6NHjyYlJYWWLVvWqC0Ao9HIK6+8Qnx8PLt27eKOO+7goYceIikpCYDk5GSGDBnCzTffzCuvvIKPjw8//fQTNpsNgGnTpvHmm2/y4osvMmDAAA4dOsSWLVtqXMfZUhjyMFdsy3E8g8HAY+c/RoA5gGZBZ/gXVkREGpXw8HAuueQSPvjgg/Iw9MknnxAeHs6QIUMwmUx07969/PynnnqK+fPns2DBAu66664at3fvvfeWP2/dujVPPvkkf/nLX8rD0HPPPUevXr3KXwPlm7zm5OTw8ssvM2vWLG688UYA2rZty4AB7t9rU2HIw8o2bN2WlovN7sBkPPs1gto1aQdAal4qKw6s4KqEq876miIiUuqRg6d/32SpeD54OgycVvW5huNmqwREnPna1TBhwgRuu+02kpKSsFgszJ07l2uuuQaTyUReXh5PPPEEX3/9NQcPHsRqtVJQUMDevXtr1dZPP/3EM888w6ZNm8jOzsZqtVJYWEheXh6BgYEkJydz9dVXn/KzmzdvpqioqDy0eZLCkIfFNQnAz2yksMTOnow82kQFueS6RwqOMGr+KIpsRXSJ7EJieKJLrisi4vV8A6t/ro8v4Fu9cw2Gml27CqNHj8Zut7Nw4UJ69+7NsmXLeOGFFwB48MEH+e677/jXv/5Fu3bt8Pf3Z+zYsRQX13yPzD179jBy5EimTJnCk08+SXh4OMuXL+eWW26hpKQEAH//qm8MOt177qYJ1B5mNBpcPokaINI/kkFxg+gV00srUouIeBF/f3+uvPJK5s6dy4cffkhCQgI9e/YEYNmyZUyaNIkrrriCrl27Ehsby+7du2vVztq1a7FarTz//POcf/75JCQkcPBg5Z6tbt268eOPP57y8+3bt8ff37/K991JPUP1QEJMMH/szyLlcA4jujZ12XVn9J+Bn8lPYUhExMtMmDCB0aNHs3HjRq6//vry4+3atePzzz9n9OjRzjmmjz120p1n1dW2bVusViuvvvoqo0ePZsWKFbz22muVzpk2bRpdu3bljjvuYMqUKfj6+vLTTz9x9dVXExkZyd/+9jceeughfH196d+/P+np6WzcuJFbbrnlrL5/TalnqB4omzd0Nhu2noq/jz8l9hLmbp7LA0sfcMkq1yIiUv8NHjyY8PBwUlJSuO6668qPv/jiizRp0oR+/foxevRohg8fTo8ePWrVxjnnnMMLL7zAs88+S5cuXZg7dy4zZ86sdE5CQgLff/89v//+O+eddx59+/blyy+/xMfH2Rfz2GOPcf/99zN9+nQ6duzI+PHjSUtLq/0XryWDowH8hczOziY0NJSsrCxCQkI8XY7L/bw1nYnvrKZtVCA/3j/QpddOzUvl0s8vpdhezJvD3uT8pue79PoiIiINnYbJ6oGyPcp2Z+RTWGLDz2xy2bVjA2P5yzl/IcQ3hJ4xPV12XRERkcZCw2T1QHSwhVB/Mza7gx3puWf+QA1N7jqZcYnj8DH4kFGQ4fLri4hI4zN37lyCgoJO+ShbK6ixUM9QPWAwGEiMDWb1rky2Hs6hc7NQl7exL3sfj654lKOFR5k/Zj4+Rv1PLyIiVbvsssvo06fPKd8zm81urqZu6S9iPZEY4wxDKamu7xkCaOLXhF1ZuyiwFpCSmULnyMaV6kVExLWCg4MJDg72dBluoTBUT5Rvy5GaXSfXD/IN4rkLn6NtWFuiA6LrpA0REZGGSHOG6onEmLLb6+umZwigb7O+RAdEsy97HysOrKizdkRERBoShaF6oiwMHThWQE5hSZ2189vh37jsi8t4ZPkj5JXk1Vk7IiIiDYXCUD0RGmAmNsQPqNveoa5RXWkW1IyO4R3JKXbtIo8iIiINkeYM1SMJscGkZheSkppDz1ZN6qQNs9HMB5d+QKjF9XesiYiINETqGapHEmOcO9a7eluOE4VaQim0FvLun+/y5h9v1mlbIiLScMXHx/PSSy+55FpLlizBYDBw7Ngxl1zPldQzVI8kxjq3GnHl7vVVWZO6hhfWvYDFZGFMuzG6w0xEpJEYOHAg55xzjktCzJo1awgMDDz7ouo5haF6pOKOsroPQwOaD2Bk65H0a9aPCL+IOm9PRETqB4fDgc1mK98s9XSioqLcUJHnaZisHmkXHYTBABl5xaTnFNVpWwaDgWcvfJYx7cZgNBjJL8mv0/ZERBqL/JJ88kvyKdvnvMBaQH5JPja7DYAiWxH5JfmU2J13BpfYSsgvyafYVgyA1W4lvySfQmshAHaHvfyaJ7ZRU5MmTWLp0qW8/PLLGAwGDAYDc+bMwWAw8N1339GrVy8sFgvLli1jx44djBkzhpiYGIKCgujduzc//PBDpeudOExmMBh46623uOKKKwgICKB9+/YsWLCgxnWW+eyzz+jcuTMWi4X4+Hief/75Su8nJSXRvn17/Pz8iImJYezYseXvffrpp3Tt2hV/f38iIiIYOnQoeXm1u0taYage8fc10So8AHBP7xDAhvQNXLPwGmb8MsMt7YmINHR9PuhDnw/6cLToKADXfn0tfT7ow29pvwEwbdk0+nzQh0+3fgrAmxvepM8HfXhuzXMA/Lj3R/p80Ie//PAXAHYe20mfD/pwyWeXnNRGTb388sv07duXW2+9lUOHDnHo0CHi4uIAeOihh5g5cyabN2+mW7du5ObmMnLkSH744QfWr1/P8OHDGT16NHv37j1tG0888QTjxo3jjz/+YOTIkUyYMIHMzMwa17pu3TrGjRvHNddcw4YNG3j88cd57LHHmDNnDgBr167lnnvuYcaMGaSkpPC///2PCy+8EIBDhw5x7bXXcvPNN7N582aWLFnClVdeWR5Qa0rDZPVMQkwwuzPySUnNoX+7yDpvz2Q0sSljE3uz93Ks8BhhfmF13qaIiNSN0NBQfH19CQgIIDY2FoAtW7YAMGPGDC6++OLycyMiIujevXv566eeeor58+ezYMEC7rrrrirbmDRpEtdeey0AzzzzDK+++iqrV6/mkksuqfIzp/LCCy8wZMgQHnvsMQASEhLYtGkT//znP5k0aRJ79+4lMDCQUaNGERwcTKtWrTj33HMBZxiyWq1ceeWVtGrVCoCuXbvWqP3jKQzVMx1ig/l+02G39Qx1iujEU/2f4oIWFygIiYhUw6/X/QqAv48/AB+O+hCHw4HFZAFg5gUzear/U5hNzs1Mb+16K5M6TyrfIHtIyyH8et2vGA3OwZk2YW3Kr3liG67Uq1evSq/z8vJ44okn+Prrrzl48CBWq5WCgoIz9gx169at/HlgYCDBwcGkpaXVuJ7NmzczZsyYSsf69+/PSy+9hM1m4+KLL6ZVq1a0adOGSy65hEsuuaR8eK579+4MGTKErl27Mnz4cIYNG8bYsWNp0qR2y9JomKyeKdujbIsb7igrM6bdGML9wtl5bCfbj253W7siIg1RgDmAAHMABoMBcIaiAHMAJqMJAIvJQoA5ALPRGYbMJjMB5gB8Tb4A+Bh9CDAH4OfjXGjXaDCWX/PENlzpxLvCHnzwQT777DOefvppli1bRnJyMl27dqW4uPi01zlxx3qDwYDdbq9xPQ6Ho/x3ePyxMsHBwfz22298+OGHNG3alOnTp9O9e3eOHTuGyWRi0aJFfPvtt3Tq1IlXX32VxMREdu3aVeM6QGGo3im7o2zb4Rzs9tqNfdbGF9u/4IoFV/DM6mdqPeYqIiKe5+vri81mO+N5y5YtY9KkSVxxxRV07dqV2NhYdu/eXfcFlurUqRPLly+vdGzlypUkJCRgMjmDpY+PD0OHDuW5557jjz/+YPfu3SxevBhwhrD+/fvzxBNPsH79enx9fZk/f36tatEwWT0THxmI2WQgr9jGgWMFxIW79v8ZVKVPbB98DD4Em4MptBWWd/+KiEjDEh8fz6+//sru3bsJCgqqstemXbt2fP7554wePRqDwcBjjz1Wqx6e2rr//vvp3bs3Tz75JOPHj2fVqlXMmjWLpKQkAL7++mt27tzJhRdeSJMmTfjmm2+w2+0kJiby66+/8uOPPzJs2DCio6P59ddfSU9Pp2PHjrWqRT1D9YzZZKRzM+dWGfPXH3Bbu02DmrLwyoW8PPhlBSERkQbsgQcewGQy0alTJ6KioqqcA/Tiiy/SpEkT+vXrx+jRoxk+fDg9evRwW509evTg448/5qOPPqJLly5Mnz6dGTNmMGnSJADCwsL4/PPPGTx4MB07duS1117jww8/pHPnzoSEhPDzzz8zcuRIEhIS+Pvf/87zzz/PiBEjalWLwdEAxkSys7MJDQ0lKyuLkJAQT5dT5xb8fpB7PlxPqL+Z5X8bRLCf+cwfcpG8kjze3/Q+HcI7MDBuoNvaFRER8RT1DNVDl3ZtSpuoQLIKSnhv1R63tj1381z+nfxvXlj3QvkCYiIiIo2ZwlA9ZDIauHtwOwDeWraTvCKr29q+tsO1dI/qzh3d7zhplr+IiEhVpkyZQlBQ0CkfU6ZM8XR5p6VhsnrKarMz5IWl7MnI55GRHbjtwrZur8Fmt+HAUb42hoiISFXS0tLIzs4+5XshISFER9ffDcEVhuqxj9fu46FP/yAyyJdlDw3G39fktrZXHljJP9f+k7EJY5nQcYLb2hUREXE3DZPVY1ec25wWTfw5klvMB6tPvyKoq+3P3c/2Y9v5aMtH2B3uu9VSRETE3RSG6jGzycidg5xzh15buoPCEvdNaL6i/RX8tcdf+e/I/5YvGS8iItIY6a9cPXdVjxY0C/UjPaeIeWv2ua1ds9HM5K6TCbWEkpKZQlZRltvaFhERcSeFoXrO18fIXwY6J0+/tnQHRVb33u4+a/0sxn41lnf+fMet7YqIiLiLwlADcHWvOGJCLBzKKuTTdfvd2nbXyK4AHC086tZ2RURE3EV3kzUQ767YxRNfbaJ5mD8/PTAQXx/35FiHw8Gu7F20CW3jlvZERETcTT1DDcS157UkMsjCgWMFzF/vvt4hg8FAm9A2ZBVl8fJvL7Mvx33zlkRERNxBYaiB8DObmHKRs3fm3z/twGpz7+3uT6x6grc2vEVScpJb2xUREalrCkMNyHV9WhIe6MvezHy+TD7o1rZv6XoL7Zu0Z1irYW5tV0REpK4pDDUgAb4+3HqBs3do1k/bsdndN92rc0RnPhv9GYNaDtIijCIi0qgoDDUwN/RtRViAmV1H8vj6D/f2DhkMBr7b/R1jvhjD7+m/u7VtERGRuqIw1MAEWXyYPKA1AK8udm/vEMDyA8vZnb2bOX/OcWu7IiIidUVhqAGa2C+eED8ftqfl8u2fh9za9h3d7+DOc+7kqQFPubVdERGRuqIw1ACF+Jm5qb+zd2jW4u3Y3dg71DSoKVO6TyHQHEhKZgoNYJkqERGR01IYaqBu7t+aIIsPW1Jz+H7TYbe3/8iyRxj71Vh+3Puj29sWERFxJYWhBio0wMykfvEAvPLjNrf30DQNaorRYGTb0W1ubVdERMTVtB1HA3Y0r5j+zy4mv9jGWxN7MbRTjNvazinOIT0/nTZh2qZDREQaNvUMNWBNAn2Z2DcegFcWu7d3KNg3mDZhbcgszCQpOYliW7Hb2hYREXElhaEGbvIFrfEzG/ljfxZLt6a7tW2Hw8GN397I7N9n8+nWT93atoiIiKsoDDVwkUEWru/TCoCX3Tx3yGAwcH3H6+kU0Yl2Ye3c1q6IiIgrac5QI5CWXcgFz/1EkdXOf2/pw4D2kW5r22a3YTQYMRgMOBwODAaD29oWERFxhVr1DCUlJdG6dWv8/Pzo2bMny5Ytq9bnVqxYgY+PD+ecc05tmpUqRIf4ce15LQHnnWXuZDKasDvsfLn9S8Z9PY6soiy3ti8iInK2ahyG5s2bx7333sujjz7K+vXrueCCCxgxYgR79+497eeysrKYOHEiQ4YMqXWxUrUpF7XF12Rk9e5MftmZ4da2DQYD/9n0H7ZkbuGDLR+4tW0REZGzVeNhsj59+tCjRw9mz55dfqxjx45cfvnlzJw5s8rPXXPNNbRv3x6TycQXX3xBcnJytdvUMFn1/P2LDfz3l730axvBB7ee79a2lx9Yzraj27imwzX4+/i7tW0REZGzUaOeoeLiYtatW8ewYcMqHR82bBgrV66s8nPvvvsuO3bs4B//+Ee12ikqKiI7O7vSQ87sLwPbYTYZWLkjgzW7M93a9oDmA7ipy034+/izL2efW9sWERE5GzUKQ0eOHMFmsxETU3lxv5iYGFJTU0/5mW3btvHwww8zd+5cfHx8qtXOzJkzCQ0NLX/ExcXVpEyv1TzMn7E9WwDunzsEUGAtYMoPU7jsi8vYn7Pf7e2LiIjURq0mUJ94x1BVdxHZbDauu+46nnjiCRISEqp9/WnTppGVlVX+2LdPPQ3VdcfAdpiMBpZtO8L6vUfd2ra/jz92ux0csO7wOre2LSIiUlvV66opFRkZiclkOqkXKC0t7aTeIoCcnBzWrl3L+vXrueuuuwCw2+04HA58fHz4/vvvGTx48Emfs1gsWCyWmpQmpeLCA7jy3OZ8sm4/ry7ezjuTeru1/Wl9puFj8CEuRL15IiLSMNSoZ8jX15eePXuyaNGiSscXLVpEv379Tjo/JCSEDRs2kJycXP6YMmUKiYmJJCcn06dPn7OrXk7pzkHtMBpg8ZY0Nux3763urUNbExcSR3p+Oh+nfOzWtkVERGqjRj1DAFOnTuWGG26gV69e9O3blzfeeIO9e/cyZcoUwDnEdeDAAd577z2MRiNdunSp9Pno6Gj8/PxOOi6uEx8ZyJhzmjN//QFeWbyNNyf2cmv7WUVZjJo/inxrPh3DO9I1qqtb2xcREamJGoeh8ePHk5GRwYwZMzh06BBdunThm2++oVUr55YQhw4dOuOaQ1L37hzUji+SD7Bo02E2Hsyic7NQt7UdagllaKuh7Mneg8loclu7IiIitaHtOBqxuz9cz1e/H2REl1hmX9/TrW0XWAvwM/lpew4REan3tFFrI3b3YOfmqd/+mUpKao5b2/b38cfqsPJxysfc+9O9bt1AVkREpCYUhhqxhJhgRnSJBWDWT9vd3n5WURb/Wvsvftz7I0v3L3V7+yIiItWhMNTI3VXaO/T1HwfZnpbr1rYj/SO5o/sdPHzew/RrdvLdhiIiIvWBwlAj17lZKBd3isHhgH97oHdoUpdJTOg4AbPRTEaBezeQFRERqQ6FIS9wz+D2AHyZfIBdR/Lc3v6h3EPc/N3NTPhmAiW2Ere3LyIicjoKQ16ga4tQBneIxu6AJA/0DoVaQtmdvZv0/HQ2HNng9vZFREROR7fWe4n1e49yRdJKTEYDSx4YSFx4gFvbX31oNS1DWhIbGOvWdkVERM5EPUNe4tyWTbigfSQ2u4OkJe7vHTqv6XnEBsZyMPcgP+//2e3ti4iIVEVhyIv8dYhz7tCn6/az/2i+29vfeGQjo+aP4uFlD5NV5N4900RERKqiMORFesWH069tBCU2B68t3eH29juEd6BVSCs6hXcip9i9i0CKiIhURXOGvMwvOzO45o1f8DUZ+fmhQcSG+rm1/ayiLEIt7tsnTURE5EzUM+Rlzm8TwXnx4RTb7B7pHQq1hFJkK+L9Te+TlJzk9vZFREROpDDkhe4pnTv04eq9pGUXur39P9L/4Lk1z/Hmhjc5mHvQ7e2LiIgcT2HIC/VvF0GPlmEUWe288fNOt7ffO7Y3l7W9jEf7PEpUQJTb2xcRETme5gx5qSUpaUx6dw1+ZiPL/zaYyCCLR+pwOBzkW/MJNAd6pH0RERH1DHmpixKi6N4ilMISO28t2+WRGrZkbuH6b6/n78v/7pH2RUREQGHIaxkMBu4u3bPsvVW7ycwrdnsNZqOZDekbWHFwBUcKjri9fREREVAY8mpDOkbTqWkI+cU23lnu/t6htmFteXrA03xz5TdE+ke6vX0RERFQGPJqBoOh/M6yOSt3k5Xv/h3lR7cdTaR/JHuy95CSmeL29kVERBSGvNywTjF0iA0mt8jKOys8M3fo213fMuaLMcxYNYMGMJ9fREQaGYUhL2c0VswdenfFLrIL3d871Du2N74mX0ItoeSW5Lq9fRER8W4KQ8KILrG0iw4iu9DKeyt3u739SP9IFly+gKShSQT7Bru9fRER8W4KQ1LaO9QOgLeW7yK3yOr2GmIDYymwFvDWhrf4fvf3bm9fRES8l8KQADCqWzPaRAZyLL+E91ft8UgNH6d8zMu/vcyL616kxOb+4ToREfFOCkMCgMlo4M5Bpb1Dy3aSX+z+3qGrE66me1R37jjnDowG/aMpIiLuob84Um7MOc1oGR5ARl4xH/y61+3tB5gD+O/I/zK67WgMBgMldvUOiYhI3VMYknI+JiN3DmoLwGtLd1JYYvNIHWtS1zD+6/G8t/E9j7QvIiLeRWFIKrni3BY0D/PnSG4RH652f+8QwKG8Q2zJ3MLHKR9js3smkImIiPdQGJJKfH2M3FHeO7TDI71Dl7a+lL/2+CvzRs3DZDS5vX0REfEuCkNykrE9W9A01I/D2UV8sm6/29s3GU1M7jqZML8wth/dTkZBhttrEBER76EwJCex+Jj4y0Bn79Dsn7ZTbLV7pI43/3iTKxdcyZsb3vRI+yIi4h0UhuSUxvWKIzrYwsGsQj77zf29QwBdIrvgwMHRwqPas0xEROqMwdEA/spkZ2cTGhpKVlYWISEhni7Ha7y9fBdPfr2JFk38+emBgZhN7s/OO4/tpE1YG7e3KyIi3kM9Q1Kl685rSWSQL/uPFjB//QGP1NAmrA25xbkkJSex89hOj9QgIiKNm8KQVMnf18RtFzp7ZZJ+2o7V5pm5Q8/8+gyzf5/NrORZHmlfREQaN4UhOa0JfVoRHujL7ox8vvrjoEdquKnLTbQJbcMl8Zd4pH0REWncFIbktAItPtwyoDUAry7ejs3u/ilm7Zu054sxXzAsfhh2h2d6p0REpPFSGJIzmti3FaH+Znam57FwwyGP1GAwGFi8dzFXfHkFa1LXeKQGERFpnBSG5IyC/czlvUOzFm/D7oHeIYCVB1eyM2sn7/z5jkfaFxGRxklhSKrlxn7xBPv5sPVwLt9tTPVIDbd3u53bu93Osxc+65H2RUSkcVIYkmoJ9TdzU39n79DLP3qmdygqIIq7zr2LEN8Qth7dqvlDIiLiEgpDUm03948n0NfEltQcfth82GN1PLHqCa5acBXf7vrWYzWIiEjjoTAk1RYW4MuN/eIBeGXxNo9tkdEssBkGDGw7us0j7YuISOOi7TikRjLzihnw7GLyi228M6kXgzvEuL2G/JJ89uXsIzE80e1ti4hI46OeIamR8EBfbji/FQCv/LjdI71DAeYAEsMTySrKYnbybAqthW6vQUREGg+FIamxyRe0wc9sJHnfMZZtO+KRGhwOB7d8dwtJvyfx4ZYPPVKDiIg0DgpDUmNRwRauO8/ZO/TI/A2s2pHh9hoMBgPXd7qe9k3ak9Akwe3ti4hI46E5Q1IraTmFXD5rBQeznENU43q14JGRHQkL8HVbDTa7DYPBgNFgxOFwYDAY3Na2iIg0HuoZklqJDvbjf/ddyPXntwTg47X7GfrCUr5MPuC2eUQmowmAhTsXMu7rcWQUuL+HSkREGj6FIam1ED8zT13elU+n9KV9dBBHcov560fJ3DRnDfsy891SgwED7296ny2ZW3h/0/tuaVNERBoXDZOJSxRb7by+dAevLt5Osc2Ov9nE/cMSmNQvHh9T3Wbu1YdWk5yezPUdryfAHFCnbYmISOOjMCQutSM9l2mfb2D1rkwAujQP4f+u7EaX5qFuaX9/zn5aBLdwS1siItI4aJhMXKptVBAf3Xo+z17VlRA/H/48kM1ls5bz9MJN5Bdb66zdElsJdy++m1HzR7Eza2edtSMiIo2PwpC4nNFoYHzvlvx4/0BGd2+G3QFvLtvFxS/8zJKUtDpp02wyA+DAwdrUtXXShoiINE4aJpM699OWNP7+xZ8cOFYAwJhzmvHYqE5EBllc2s7e7L3YHDZah7Z26XVFRKRxUxgSt8grsvLCoq28u2IXdgeE+pt59NKOXN2zhcvXB8ooyOCHPT8wvsN4l15XREQaJ4Uhcas/9h/j4c82sOlQNgB920TwzJVdaR0Z6JLr55XkcfGnF5NTnMO7w9+lV2wvl1xXREQaL80ZErfq1iKMBXf155GRHfAzG1m1M4PhL/3MrMXbKLbaz/r6geZARsSPoEtEF3xN7lsNW0REGi71DInH7MvM59Ev/uTnrekAJMYE88yVXenZqslZXbfQWojFZNH2HCIiUi3qGRKPiQsP4D839ebla84hItCXlMM5jH1tJdO//JOcwpJaX9fPxw+bw8b8bfO568e7sDvOvsdJREQaL4Uh8SiDwcCYc5rzw9SLGNuzBQ4HvLdqDxe/8DPfbUyt9XXzSvJ4bs1zLN2/lO/3fO/CikVEpLHRMJnUKyu3H+GR+RvYneHc22x45xieuKwLsaF+Nb7W3M1zsdqtjE8cj59PzT8vIiLeQWFI6p3CEhuvLt7G60t3YrU7CLb48NCIDkw4ryVGY83nAeUU53DP4nsY024Mo9uMLt/tXkREBDRMJvWQn9nEg8M78PU9AzgnLoycIiuPffEnV7++iq2Hc2p8vQU7FrD28Fre/fNdjAYjVruVA7kH6qByERFpiNQzJPWaze5g7q97eO5/KeQWWTGbDEy5qC13DmqHn7l6PTyZhZl8vu1zmgY25dI2l7JozyLuX3I/V7S/gif6PVHH30BEROo79QxJvWYyGpjYN55FUy9kaMcYSmwOXl28nREvL2PVjoxqXSPcL5zJXSdzaZtLAdh4ZCMOHET4RThfZ2zk/U3vk12cXWffQ0RE6i/1DEmD4XA4+G5jKtO/3EhaThEA43vFMW1kB8ICarbA4q6sXQT7BhPpH8nffv4b3+z6hqvaX8Xj/R7HZrdpXpGIiBdRz5A0GAaDgUu6NOWH+y/i+vNbAjBv7T6GvrCUBb8fpCa5vnVoayL9IwHo07QP7cLaMS5xHAAvrnuRG7+9kTWpa1z/JUREpN5Rz5A0WGt3Z/Lw5xvYnpYLwMDEKJ4c04W48IAaX8vhcGAwGLDarQz5ZAiZhZnMGjyLi+IuIjktmWZBzYgOiHb1VxARkXpAYUgatCKrjdeX7mTW4u0U2+z4m03cPyyBSf3i8THVruMzLT+NhTsXMrHTRAwGA6Pmj+Jg7kGShibRr1k/F38DERHxNA2TSYNm8TFxz5D2fHvvBZzXOpyCEhtPLdzM5Ukr+PNAVq2uGR0QzU1dbsJkNJFZmEmUfxQB5gDOiToHgOkrpvNxysfkl+S78JuIiIinqGdIGg273cEn6/bx9MLNZBdaMRkN3DKgNfcObU+Ar89ZXTujIIMI/wg2Z2xm3Nfj8DH48P3Y74kKiOJIwZHy+UciItLwKAxJo5OWU8iMrzbx9R+HAGge5s+4XnEM6RhN52YhZ7WbfU5xDl9s/4L0gnSm9pxKal4qIz4bQe/Y3rwy+BVt+yEi0gDVapgsKSmJ1q1b4+fnR8+ePVm2bFmV537++edcfPHFREVFERISQt++ffnuu+9qXbDImUQH+zHruh68M6kXzcP8OXCsgBd/2MqoV5fT55kfefizP/h+Yyp5RdYaXzvYN5gbOt3A1J5TAViTugabw0axvRg/Hz/ySvJ45893yCzMdPXXEhGROlLjnqF58+Zxww03kJSURP/+/Xn99dd566232LRpEy1btjzp/HvvvZdmzZoxaNAgwsLCePfdd/nXv/7Fr7/+yrnnnlutNtUzJLWVV2Tl6z8OsnhLGsu2HSG/2Fb+nq/JyPltIxjSIZrBHaJrdRcawIHcA+QW55IYnsi8LfN46tenSGiSwGeXfVZ+l5qIiNRfNQ5Dffr0oUePHsyePbv8WMeOHbn88suZOXNmta7RuXNnxo8fz/Tp06t1vsKQuEKR1cavOzNZvCWNH7ccZl9mQaX320cHMbhjNEM6xNCjZVit7kZbsm8Jr/3+GqPajOL6TtezeO9i3vjjDW7ucjPD4oe56JuIiIgr1WhWaXFxMevWrePhhx+udHzYsGGsXLmyWtew2+3k5OQQHh5e5TlFRUUUFRWVv87O1jYJcvYsPiYuTIjiwoQo/jG6EzvSc/lxcxo/bklj3Z6jbEvLZVtaLq8v3Umov5mBiVEM7hDNRQlR1V7hemDcQAbGDcRmd/ZAfbL1EzZmbGRjxkaGxQ9jf85+DAYDzYOa1+VXFRGRGqhRGDpy5Ag2m42YmJhKx2NiYkhNTa3WNZ5//nny8vIYN25clefMnDmTJ57QBppSdwwGA+2ig2kXHcztF7UlK7+EpdvSWbz5MEu2pnMsv4Qvkw/yZfJBjAbo1SqcwR2dw2nto4POOPRVtp3H0wOe5vNtnzOi9QgAkpKT+Hrn1zzY+0Fu6HRDnX9PERE5s1rdb3ziH4Lqzov48MMPefzxx/nyyy+Jjq56Nd9p06YxderU8tfZ2dnExcXVplSRagkNMHNZ92Zc1r0ZVpud5H3H+HFLGos3p5FyOIfVuzNZvTuT//t2Cy2a+DvnGXWMoU/rcPzMVe9jVrZJLDj/PckuzsaBo3zNov9u+i8OHIxpN4YQXw0Bi4h4Qo3CUGRkJCaT6aReoLS0tJN6i040b948brnlFj755BOGDh162nMtFgsWi6UmpYm4jI/JSK/4cHrFh/O3Szqw/2g+P21xDqet3JHB/qMF/GfVHv6zag/+ZhMD2kcypEM0gzpEExNS9a31BoOBWUNmsS9nH3HBcRTZinj9j9c5VnSMZkHNGNJyCDnFOQT7Brvx24qISK0mUPfs2ZOkpKTyY506dWLMmDFVTqD+8MMPufnmm/nwww+5/PLLa1ykJlBLfZFfbGXF9gwWbznM4i1pHM4uqvR+l+YhDO4Qw5AO0XRtHorRWHWPaZGtiC+3f8nifYuZNXgWRoORMV+MIdwvnCf7P0nLkJPvzhQREder9a31r732Gn379uWNN97gzTffZOPGjbRq1Ypp06Zx4MAB3nvvPcAZhCZOnMjLL7/MlVdeWX4df39/QkNDq9WmwpDURw6Hg40Hs0vvTkvjj/3HOP7fpsggC4M7OCdhD2gfRZDl9B2xKZkpjP96PP4+/vx49Y8EmAP4fNvnDGg+QJvEiojUoVqtQJ2UlMRzzz3HoUOH6NKlCy+++CIXXnghAJMmTWL37t0sWbIEgIEDB7J06dKTrnHjjTcyZ86carWnMCQNQXpOEUtS0li8JY2ft6aTd9yaRmaTgfPbRDC4dE2jVhGBp7zG4bzDpBxN4cIWF7L16FauWnAVFpOFxeMWa06RiEgd0XYcInWg2Gpnze7M0lv3D7Mno/Kmrm2jAhnSMYbBHaLp2aoJ5lOsafTnkT/555p/EuEfwQsDXyA9P507f7yTsQljuTrhai3mKCLiIgpDInXM4XCw80iecxL25jTW7M7Eaq/41y7Yz4eLEqIY0C6SDk1DSIgJqrSxbJGtCIvJwmu/v8a/k//NudHn8t6I9yiwFpCal0rr0Nae+FoiIo2GwpCIm2UVlLBsWzqLN6exZGs6mXnFld43GKBleACJMcF0iA0mMTaExNhgIoKtfL3rK1qFtOLCFhfy6dZPeWLVE1zV/ioe7/e4Z76M1EhmYSY/7f2JYnsx13a4Fqj+0iQiUndqtc6QiNReqL+ZUd2aMapbM2x2B8n7jrF4y2GS9x0jJTWHI7nF7MnIZ09GPt9vOlz+OV8fI+2j4+kQG0JK7E42Fm7CgKG8Z2j1odVsOLKBK9tfSRO/Jp76el4tPT+dzZmbCbWE0j2qOysPrOTZNc8SHxLPy4NfJj0/ncdXPU6YJYxrO1xLfkk+N3x7A2MTxjI2YSxmo9nTX0HEKykMiXiQyWigZ6sm9GxVEV6O5BaRkprDltQcUlKzSUnNYevhXApKbGw8mM3Gg2Xb0/TEYG7Ni3tD+HbFKo4E/pvD1t9JSU/lH/3/ho+PDYtJ63W5Un5JPvty9hEdEE0TvyZ8uvVT/rf7f1za+lKuaH8FX+/8mhfWvcCI+BF0v6g7ZpOZnVk7sdqtAMQFx9G/WX9aBLfAarfy2bbP2Hp0K+9tfI+x7cdS1lGvniIR91IYEqlnIoMsRLaz0L9dZPkxm93Bvsz80oCUw5bSkLQ7A46VwC87M/EJTcC3SSaf/BTHx99/R0T8fMz+aQwIv4mLWvahY9Ng4iMCa7UBrbdwOBwcLTpKE0sTDAYD/9n4H1IyU/hL978QFxLHX3/6K78c+oUn+z/J5e0uZ3/Ofn499CvxIfFc0f4K2oS2oX2T9sQGxQLQMbwjr1/8Oi2DnWtGBZgDeO3i18rbu6bDNfgafYkJjMFsMvPz/p95e8Pb3N/rfrpFdfPI70DEGykMiTQAJqOB+MhA4iMDuaRLbPnxwhIb2w7nloaj1qQcHs4WvxzSrbkU+f5OMYV8/tshPl2xHlPgVky2aNo1aVk6F8n56BAbQkyIxWt6I8rm6OSV5PHtrm/JKMjg9u63Y7PbuHDehWQXZ/Pj1T8SHRDNwp0L2Zy5maGthhIXEkdccBxbMrdQbHPO87o4/mJah7amU0QnAC6Ku4iL4i4qbyvIN4h+zfpVWYvZaGZ8h/Hlr2cnz+bPjD9ZtGcR3aK6aT6RiJtoArVII5SZV8zqvXv4btcP+Ob3Y8vho+zwfxiM+RTsuR1bQXyl80P9zaXBKLj8Z0JMMMF+DXcOS2ZhJn8e+RMfow/9mvVjY8ZGHv75YQLMAcwbNY9jhce4YN4FAKyZsAY/Hz8u+ewSDuQe4L8j/0v3qO58nPIx2cXZDGk5hNahrbHarfgY6+7/Q6bmpfLWhre4+9y7CbWE8vjKxwk0B3Jbt9sItVRvkVoRqTmFIREvkJqXymPLH2PHsV080Pk/bEvNY/7BpyjKac3hA+dgt/ue8nPNw/zp2LSsFymEDrHBtAwPOO3mtO6SV5LH3uy9NPFrQmxgLP/b/T8+SfmEvs36MrnrZL7Z+Q1/W/Y3ekT34D8j/sPe7L1cOv9S/Ex+rJ6wGoD7l95PlH8Ud5xzB6GWUA7lHiLcP7xezLXanbWby764DAcOPrr0IzpHdvZ0SSKNlobJRLxAbGAsbw5/k9ziXIJ8g/g14lfeOrCegLAUVt80ldSjsGpPCunHgssnbh/OLuLAsQIOHCvgh81pla4XFWyheZg/LZr406JJAM2blD4P86d5E/9K6yTVlsPhIKMwg1DfUMwmM/O2zGN9+nqu7XAt3aO6M/PXmXy540vuPvdubut2G8cKj7E6dTWBZufq3vGh8SQ2SaRNWBsAmgY15c1hbxIXHAc4Jym/MPCFSm02DWp61nW7SquQViQNTSI5LZnOkZ3JKc7h9kW3M7HTRIbHD9fwmYgLKQyJeJEg3yAAukZ25bHzHyOnOIfIgFCyinfwyrZb6B3bm7cnvY7ZaOZoXjEph3PK72zbkprNtsO55BZZSc8pIj2niOR9x07ZTkSgb0VAahJQHpyal74u26fN7rBjNBgpsZcwf9t89ufs5+4ed2M2mrnsi8vYnb2bD0Z+QNeoriw/uJwl+5bQPao73aO6ExccR7hfOHaHHYC+zfryzIBnaN+kPQCdIjrx6WWfltdkNpo5v+n5dffLdTGDwcCA5gMY0HwAAP/d/F82HNlA0u9JDG01FB+D/vMt4ir6t0nECwWYAxiXOK789e/pv2MymAg2B2M2mskoyGDetnmMTRjL+W3iy89zOBwcyy9h/9ECDhzLZ//RguMe+Rw4WkBOkZWMvGIy8or5Y3+W84PGfEwBuzEY7FhzuhAanIOx6ZtgKuCK8DdpHubPv3c/R4m9iOEtx9A5ui3hfuHsyd5DWr6zV+rSNpdyTtQ59IjuAcCt3W7l9u63l9fWKqQVrUJa1f0vz0Nu7HQjBgx0juiMj9GHH/b8wIIdC7iv531ahVzkLGnOkIgAzk1iC22FtAppxZt/vMkr618p3/rDZrdhNBirHJrJLc5lT84eAnwCCPdtwcLtS/h46xyCjXG095nI5qPJ/OmYCSXh5Gx/CAzFBHeYDkBOynSwB2CJ/QIcRoozLiTIJ5LYJiXEhUXQsklIaQ+TP83DAmjRxJ+wALNXDxPZHXYu//JydmXt4vZut3PXuXd5uiSRBk09QyICQExgTPnzhCYJ9IjuwdiEsQAs2LGA9ze/z23dbuOS+Ev4asdXLDuwjJGtRzIwbiDv/PkOb254k6sTrmZ63+k0DfVhR+4fdIqwMn1UJ9Lzo7jjx/nEh8Tz2ISLOZhVyM97gykqCCE7MpADxwrZf/RGDhwtIMNaTI7VSs4hA9sOZQKZJ9Ua6GsqH3JzhqTKc5ciAn0bdVgyGoy8NPAl3trwFjd1uQmA6SumExccx/Wdrsffx9/DFYo0LApDInKSsvVyyjqOv9j+BduObuNAzgHAOaz27a5vaR7UnIFxA2kZ0pIIv4jyu7C6RnXlmQHP0CbUOXk5KiCKT0Z/Un79EH9fOsQOPWXb+cVWDhwtYP8x5/DbgdIhOOfQXAHpOUXkFdvYejiXrYdzT3kNi4+R6BAL0cF+RAdbiAnxIyrYQnSwheiQimNNGnAPU5uwNjxzwTMAbMrYxPzt8zEajFwUdxEJTRI8XJ1Iw6JhMhE5o6yiLL7c/iWj246miV8TVh1cxebMzfSK6eX2xQELS2wcPFYxV+n4uUsHjhZwOKeQ6v5XzWwyEBVkIao8IFUEqPIwFWIhItCCyVh/Q5PdYeebXd+wK2sXd597N8cKj3Hfkvu4rdtt9G3W19PlidR7CkMi0qgUW+2kZhWSllNIWk4RadmlP3OKOJxdSHrp88y84mpf02hwbpNSFpBiQixElYWm0t6mmBALkUEWzPVgu5Pn1z7PnI1zSGySyMejP8Zo8HxNIvWZhslEpFHx9THSMiKAlhEBpz2v2GonPbdyWErPLuRwdlFFkMopIiO3CLuD8teQXeU1DQYID/B1DsmF+BFzfA/Tcc+jgi11unDlzV1uxmq3cmGLCzEajHyz8xt+Tf2VO8+5k+iA6DprV6ShUs+QiMhpWG12MvKKSTs+JGUXcTinkLTsItJLj6XnFGG1V/8/p6H+ZiICfQn2NxPi50Oov5kQfzMhfubS5z7HPa98Tk16n0rsJYz6fBQH8w7y1x5/ZXLXybX5NYg0agpDIiIuYLc7yMw/MTRVhKfjg1SxzX5WbfmbTWcMTCF+pT/9fUgt3ML3Bz5mRt+niQwM4vFV/6BrVFeuaHdFne61JtJQKAyJiLiRw+Egq6CEtJwijuYVk11oJbughKyCErILS8gusB733Hk8p/ScnCLrWbdvCthJQKs3wGEkNmc6kZbmpwxVIf7O18F+ZgItJgJ9fQiwmAjw9SHAbMJYjyeUi9SU/i+BiIgbGQwGwgJ8CQs49ea4p2O12cktspJdYCW7sDRAlQYn5/MTj1cOWoUldmz5LSlMHY3BJ49t6X5sN+3Gr+lnFO24GHth82rX4m82EeBrIqA0KPn7lgYm37LjPgT6mvD3df4MsDhDVGBZoPJ1/gy0mMo/66+QJR6iMCQi0kD4mIy1DlIARVZbaWAaUh6S3t/2IqszttA8wsHg0BnkFNrKe6XKAlVekZX8Yhv5xVbKpkUVlNgoKLGRkefCL4gzZFUOTCYCLT6lx31OeF05WPVtG0Gov9m1BYlXUBgSEfESFh8TUcEmooIt5ccSmt/NK+sdXJ1wNT1jOvHl9i/ZmbWTyV0nE+wbXOnzDoeDIqv9uHBkI6/YSkGxrdKx/GIreUU28kus5Bcdd06xjfzjglVesa30uLV8baiykAXVX/qgzDf3XKAwJLWiMCQi4sWaBTXj/y74PwAKrAW89NtLHCk4QnRANBM6Tqh0rsFgwM9sws9sIsKFNTgcDgpL7OQXW8sDVn6x7YQg5QxWFUHq5HPCAhSEpHYUhkREBAA/kx//6PsPPk75mKsTrsbhcPDq+le5+9y763SFcYPBgL+vc+6QK0OWSHUpDImICOAMJQPjBjIwbiAAyw8sJyogqsHu3yZSXbq1XkRETimvJA9fky9mo4afpHFTz5CIiJxSoDnQ0yWIuIV27xMRERGvpjAkIiIiXk1hSERERLyawpCIiIh4NYUhERER8WoKQyIiIuLVFIZERETEqykMiYiIiFdTGBIRERGvpjAkIiIiXk1hSERERLyawpCIiIh4NYUhERER8WoKQyIiIuLVFIZERETEqykMiYiIiFfz8XQB1eFwOADIzs72cCUiIuINgoODMRgMni5D3KRBhKGcnBwA4uLiPFyJiIh4g6ysLEJCQjxdhriJwVHW7VKP2e12Dh486JKknp2dTVxcHPv27dM/6GdJv0vX0e/SdfS7dC1v/X2qZ8i7NIieIaPRSIsWLVx6zZCQEK/6F7su6XfpOvpduo5+l66l36c0ZppALSIiIl5NYUhERES8mteFIYvFwj/+8Q8sFounS2nw9Lt0Hf0uXUe/S9fS71O8QYOYQC0iIiJSV7yuZ0hERETkeApDIiIi4tUUhkRERMSrKQyJiIiIV/OqMJSUlETr1q3x8/OjZ8+eLFu2zNMlNTgzZ86kd+/eBAcHEx0dzeWXX05KSoqny2oUZs6cicFg4N577/V0KQ3WgQMHuP7664mIiCAgIIBzzjmHdevWebqsBsdqtfL3v/+d1q1b4+/vT5s2bZgxYwZ2u93TpYnUCa8JQ/PmzePee+/l0UcfZf369VxwwQWMGDGCvXv3erq0BmXp0qXceeed/PLLLyxatAir1cqwYcPIy8vzdGkN2po1a3jjjTfo1q2bp0tpsI4ePUr//v0xm818++23bNq0ieeff56wsDBPl9bgPPvss7z22mvMmjWLzZs389xzz/HPf/6TV1991dOlidQJr7m1vk+fPvTo0YPZs2eXH+vYsSOXX345M2fO9GBlDVt6ejrR0dEsXbqUCy+80NPlNEi5ubn06NGDpKQknnrqKc455xxeeuklT5fV4Dz88MOsWLFCPb4uMGrUKGJiYnj77bfLj1111VUEBATw/vvve7AykbrhFT1DxcXFrFu3jmHDhlU6PmzYMFauXOmhqhqHrKwsAMLDwz1cScN15513cumllzJ06FBPl9KgLViwgF69enH11VcTHR3Nueeey5tvvunpshqkAQMG8OOPP7J161YAfv/9d5YvX87IkSM9XJlI3WgQG7WerSNHjmCz2YiJial0PCYmhtTUVA9V1fA5HA6mTp3KgAED6NKli6fLaZA++ugjfvvtN9asWePpUhq8nTt3Mnv2bKZOncojjzzC6tWrueeee7BYLEycONHT5TUof/vb38jKyqJDhw6YTCZsNhtPP/001157radLE6kTXhGGyhgMhkqvHQ7HScek+u666y7++OMPli9f7ulSGqR9+/bx17/+le+//x4/Pz9Pl9Pg2e12evXqxTPPPAPAueeey8aNG5k9e7bCUA3NmzeP//73v3zwwQd07tyZ5ORk7r33Xpo1a8aNN97o6fJEXM4rwlBkZCQmk+mkXqC0tLSTeoukeu6++24WLFjAzz//TIsWLTxdToO0bt060tLS6NmzZ/kxm83Gzz//zKxZsygqKsJkMnmwwoaladOmdOrUqdKxjh078tlnn3mooobrwQcf5OGHH+aaa64BoGvXruzZs4eZM2cqDEmj5BVzhnx9fenZsyeLFi2qdHzRokX069fPQ1U1TA6Hg7vuuovPP/+cxYsX07p1a0+X1GANGTKEDRs2kJycXP7o1asXEyZMIDk5WUGohvr373/SMg9bt26lVatWHqqo4crPz8dorPznwWQy6dZ6abS8omcIYOrUqdxwww306tWLvn378sYbb7B3716mTJni6dIalDvvvJMPPviAL7/8kuDg4PLettDQUPz9/T1cXcMSHBx80lyrwMBAIiIiNAerFu677z769evHM888w7hx41i9ejVvvPEGb7zxhqdLa3BGjx7N008/TcuWLencuTPr16/nhRde4Oabb/Z0aSJ1wmturQfnoovPPfcchw4dokuXLrz44ou6HbyGqppj9e677zJp0iT3FtMIDRw4ULfWn4Wvv/6aadOmsW3bNlq3bs3UqVO59dZbPV1Wg5OTk8Njjz3G/PnzSUtLo1mzZlx77bVMnz4dX19fT5cn4nJeFYZERERETuQVc4ZEREREqqIwJCIiIl5NYUhERES8msKQiIiIeDWFIREREfFqCkMiIiLi1RSGRERExKspDImIiIhXUxgSkXJLlizBYDBw7NgxT5ciIuI2CkMiIiLi1RSGRERExKspDInUIw6Hg+eee442bdrg7+9P9+7d+fTTT4GKIayFCxfSvXt3/Pz86NOnDxs2bKh0jc8++4zOnTtjsViIj4/n+eefr/R+UVERDz30EHFxcVgsFtq3b8/bb79d6Zx169bRq1cvAgIC6NevHykpKXX7xUVEPEhhSKQe+fvf/867777L7Nmz2bhxI/fddx/XX389S5cuLT/nwQcf5F//+hdr1qwhOjqayy67jJKSEsAZYsaNG8c111zDhg0bePzxx3nssceYM2dO+ecnTpzIRx99xCuvvMLmzZt57bXXCAoKqlTHo48+yvPPP8/atWvx8fHh5ptvdsv3FxHxBO1aL1JP5OXlERkZyeLFi+nbt2/58cmTJ5Ofn89tt93GoEGD+Oijjxg/fjwAmZmZtGjRgjlz5jBu3DgmTJhAeno633//ffnnH3roIRYuXMjGjRvZunUriYmJLFq0iKFDh55Uw5IlSxg0aBA//PADQ4YMAeCbb77h0ksvpaCgAD8/vzr+LYiIuJ96hkTqiU2bNlFYWMjFF19MUFBQ+eO9995jx44d5ecdH5TCw8NJTExk8+bNAGzevJn+/ftXum7//v3Ztm0bNpuN5ORkTCYTF1100Wlr6datW/nzpk2bApCWlnbW31FEpD7y8XQBIuJkt9sBWLhwIc2bN6/0nsViqRSITmQwGADnnKOy52WO7/z19/evVi1ms/mka5fVJyLS2KhnSKSe6NSpExaLhb1799KuXbtKj7i4uPLzfvnll/LnR48eZevWrXTo0KH8GsuXL6903ZUrV5KQkIDJZKJr167Y7fZKc5BERLydeoZE6ong4GAeeOAB7rvvPux2OwMGDCA7O5uVK1cSFBREq1atAJgxYwYRERHExMTw6KOPEhkZyeWXXw7A/fffT+/evXnyyScZP348q1atYtasWSQlJQEQHx/PjTfeyM0338wrr7xC9+7d2bNnD2lpaYwbN85TX11ExKMUhkTqkSeffJLo6GhmzpzJzp07CQsLo0ePHjzyyCPlw1T/93//x1//+le2bdtG9+7dWbBgAb6+vgD06NGDjz/+mOnTp/Pkk0/StGlTZsyYwaRJk8rbmD17No888gh33HEHGRkZtGzZkkceecQTX1dEpF7Q3WQiDUTZnV5Hjx4lLCzM0+WIiDQamjMkIiIiXk1hSERERLyahslERETEq6lnSERERLyawpCIiIh4NYUhERER8WoKQyIiIuLVFIZERETEqykMiYiIiFdTGBIRERGvpjAkIiIiXu3/AbEpfFA4eK4HAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 616.125x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "metrics = pd.read_csv(f\"{trainer.logger.log_dir}/metrics.csv\")\n",
    "del metrics[\"step\"]\n",
    "metrics.set_index(\"epoch\", inplace=True)\n",
    "display(metrics.dropna(axis=1, how=\"all\").head())\n",
    "sn.relplot(data=metrics, kind=\"line\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f6dc1512-5f6d-46a3-8129-1cea54a92aa3",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/sunil/miniconda3/envs/lora_from_scratch/lib/python3.11/site-packages/lightning/pytorch/trainer/connectors/checkpoint_connector.py:148: UserWarning: `.test(ckpt_path=None)` was called without a model. The best model of the previous `fit` call will be used. You can pass `.test(ckpt_path='best')` to use the best model or `.test(ckpt_path='last')` to use the last model. If you pass a value, this warning will be silenced.\n",
      "  rank_zero_warn(\n",
      "Restoring states from the checkpoint path at logs/lightning_logs/version_1/checkpoints/epoch=9-step=280.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "Loaded model weights from the checkpoint at logs/lightning_logs/version_1/checkpoints/epoch=9-step=280.ckpt\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing DataLoader 0: 100%|██████████████████████| 6/6 [00:00<00:00, 193.85it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\">   Runningstage.testing    </span>┃<span style=\"font-weight: bold\">                           </span>┃\n",
       "┃<span style=\"font-weight: bold\">          metric           </span>┃<span style=\"font-weight: bold\">       DataLoader 0        </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_acc          </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.9754816293716431     </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_loss         </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.08690668642520905    </span>│\n",
       "└───────────────────────────┴───────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1m  Runningstage.testing   \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m                           \u001b[0m┃\n",
       "┃\u001b[1m \u001b[0m\u001b[1m         metric          \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      DataLoader 0       \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_acc         \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.9754816293716431    \u001b[0m\u001b[35m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_loss        \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.08690668642520905   \u001b[0m\u001b[35m \u001b[0m│\n",
       "└───────────────────────────┴───────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[{'test_loss': 0.08690668642520905, 'test_acc': 0.9754816293716431}]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# as expected our model performs well\n",
    "trainer.test()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "46379ebe-33f9-468a-a833-4dfefc489f8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# save model weights so we can finetune from them later\n",
    "trainer.save_checkpoint(\"model.ckpt\")\n",
    "torch.save(model.state_dict(), 'model.pt')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad0a1f3b-6fe5-4839-b8d3-92f177c86ecb",
   "metadata": {},
   "source": [
    "# Finetune on second half of MNIST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2c2a53b7-4004-43fb-87a3-c5129bb38746",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the model - we refer to it as \"no training\" as we are going to use this initialization to show that the model performs poorly on the second half of mnist (as it wasn't trained on it)\n",
    "model_loaded_no_training = LitMNIST.load_from_checkpoint(checkpoint_path=\"model.ckpt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "05702f82-da58-46c3-a3d7-1b985b167e9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# set it up to run on the second half of mnist (digits 5,6,7,8,9)\n",
    "model_loaded_no_training.class_names= [5,6,7,8,9]\n",
    "model_loaded_no_training.min_class = min(model_loaded_no_training.class_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "a0b47dc5-e317-4fda-863a-e33013804f59",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: True (cuda), used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "HPU available: False, using: 0 HPUs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "\n",
      "  | Name          | Type               | Params\n",
      "-----------------------------------------------------\n",
      "0 | l1            | Linear             | 50.2 K\n",
      "1 | l2            | Linear             | 4.2 K \n",
      "2 | l3            | Linear             | 325   \n",
      "3 | dropout       | Dropout            | 0     \n",
      "4 | relu          | ReLU               | 0     \n",
      "5 | val_accuracy  | MulticlassAccuracy | 0     \n",
      "6 | test_accuracy | MulticlassAccuracy | 0     \n",
      "-----------------------------------------------------\n",
      "54.7 K    Trainable params\n",
      "0         Non-trainable params\n",
      "54.7 K    Total params\n",
      "0.219     Total estimated model params size (MB)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0:   0%|                     | 1/26895 [00:00<5:09:29,  1.45it/s, v_num=2]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`Trainer.fit` stopped: `max_steps=1` reached.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0:   0%|                     | 1/26895 [00:00<5:11:30,  1.44it/s, v_num=2]\n"
     ]
    }
   ],
   "source": [
    "# we will train on the new data on exactly one element once (this is so we can access the trainer2.test() function, which allows us to easily evaluate how well our model works on the new data prior to training)\n",
    "model_loaded_no_training.batch_size = 1\n",
    "trainer_no_train = L.Trainer(\n",
    "    accelerator=\"auto\",\n",
    "    devices=1,\n",
    "    max_steps=1,\n",
    "    logger=CSVLogger(save_dir=\"logs/\"),\n",
    ")\n",
    "trainer_no_train.fit(model_loaded_no_training)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a7e6c74d-ca4a-4ca9-94c0-22e7ecaa8d25",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Restoring states from the checkpoint path at logs/lightning_logs/version_2/checkpoints/epoch=0-step=1.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "Loaded model weights from the checkpoint at logs/lightning_logs/version_2/checkpoints/epoch=0-step=1.ckpt\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing DataLoader 0: 100%|████████████████| 4861/4861 [00:08<00:00, 599.65it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\">   Runningstage.testing    </span>┃<span style=\"font-weight: bold\">                           </span>┃\n",
       "┃<span style=\"font-weight: bold\">          metric           </span>┃<span style=\"font-weight: bold\">       DataLoader 0        </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_acc          </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.32956182956695557    </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_loss         </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    3.0265371799468994     </span>│\n",
       "└───────────────────────────┴───────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1m  Runningstage.testing   \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m                           \u001b[0m┃\n",
       "┃\u001b[1m \u001b[0m\u001b[1m         metric          \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      DataLoader 0       \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_acc         \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.32956182956695557   \u001b[0m\u001b[35m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_loss        \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   3.0265371799468994    \u001b[0m\u001b[35m \u001b[0m│\n",
       "└───────────────────────────┴───────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[{'test_loss': 3.0265371799468994, 'test_acc': 0.32956182956695557}]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we can see our model performs extremely poorly on the new data that it hasn't seen, which is unsuprising. \n",
    "trainer_no_train.test()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "336a997e-8ec7-42be-ae7c-d2646899222e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: True (cuda), used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "HPU available: False, using: 0 HPUs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "\n",
      "  | Name          | Type               | Params\n",
      "-----------------------------------------------------\n",
      "0 | l1            | Linear             | 50.2 K\n",
      "1 | l2            | Linear             | 4.2 K \n",
      "2 | l3            | Linear             | 325   \n",
      "3 | dropout       | Dropout            | 0     \n",
      "4 | relu          | ReLU               | 0     \n",
      "5 | val_accuracy  | MulticlassAccuracy | 0     \n",
      "6 | test_accuracy | MulticlassAccuracy | 0     \n",
      "-----------------------------------------------------\n",
      "54.7 K    Trainable params\n",
      "0         Non-trainable params\n",
      "54.7 K    Total params\n",
      "0.219     Total estimated model params size (MB)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/sunil/miniconda3/envs/lora_from_scratch/lib/python3.11/site-packages/lightning/pytorch/loops/fit_loop.py:280: PossibleUserWarning: The number of training batches (28) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.\n",
      "  rank_zero_warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0: 100%|█████████████████████████| 28/28 [00:03<00:00,  8.67it/s, v_num=3]\n",
      "Validation: 0it [00:00, ?it/s]\u001b[A\n",
      "Validation:   0%|                                         | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:   0%|                            | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:  33%|██████▎            | 1/3 [00:00<00:00, 305.84it/s]\u001b[A\n",
      "Validation DataLoader 0:  67%|████████████▋      | 2/3 [00:00<00:00, 315.61it/s]\u001b[A\n",
      "Epoch 0: 100%|█| 28/28 [00:04<00:00,  6.66it/s, v_num=3, val_loss=0.102, val_acc\u001b[A\n",
      "Epoch 1: 100%|█| 28/28 [00:03<00:00,  8.49it/s, v_num=3, val_loss=0.102, val_acc\u001b[A\n",
      "Validation: 0it [00:00, ?it/s]\u001b[A\n",
      "Validation:   0%|                                         | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:   0%|                            | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:  33%|██████▎            | 1/3 [00:00<00:00, 333.52it/s]\u001b[A\n",
      "Validation DataLoader 0:  67%|████████████▋      | 2/3 [00:00<00:00, 374.41it/s]\u001b[A\n",
      "Epoch 1: 100%|█| 28/28 [00:04<00:00,  6.50it/s, v_num=3, val_loss=0.0966, val_ac\u001b[A\n",
      "Epoch 2: 100%|█| 28/28 [00:03<00:00,  8.36it/s, v_num=3, val_loss=0.0966, val_ac\u001b[A\n",
      "Validation: 0it [00:00, ?it/s]\u001b[A\n",
      "Validation:   0%|                                         | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:   0%|                            | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:  33%|██████▎            | 1/3 [00:00<00:00, 299.68it/s]\u001b[A\n",
      "Validation DataLoader 0:  67%|████████████▋      | 2/3 [00:00<00:00, 345.98it/s]\u001b[A\n",
      "Epoch 2: 100%|█| 28/28 [00:04<00:00,  6.01it/s, v_num=3, val_loss=0.093, val_acc\u001b[A\n",
      "Epoch 3: 100%|█| 28/28 [00:03<00:00,  8.49it/s, v_num=3, val_loss=0.093, val_acc\u001b[A\n",
      "Validation: 0it [00:00, ?it/s]\u001b[A\n",
      "Validation:   0%|                                         | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:   0%|                            | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:  33%|██████▎            | 1/3 [00:00<00:00, 311.52it/s]\u001b[A\n",
      "Validation DataLoader 0:  67%|████████████▋      | 2/3 [00:00<00:00, 359.30it/s]\u001b[A\n",
      "Epoch 3: 100%|█| 28/28 [00:04<00:00,  6.44it/s, v_num=3, val_loss=0.0892, val_ac\u001b[A\n",
      "Epoch 4: 100%|█| 28/28 [00:03<00:00,  8.36it/s, v_num=3, val_loss=0.0892, val_ac\u001b[A\n",
      "Validation: 0it [00:00, ?it/s]\u001b[A\n",
      "Validation:   0%|                                         | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:   0%|                            | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:  33%|██████▎            | 1/3 [00:00<00:00, 252.85it/s]\u001b[A\n",
      "Validation DataLoader 0:  67%|████████████▋      | 2/3 [00:00<00:00, 150.24it/s]\u001b[A\n",
      "Epoch 4: 100%|█| 28/28 [00:04<00:00,  6.38it/s, v_num=3, val_loss=0.0849, val_ac\u001b[A\n",
      "Epoch 5: 100%|█| 28/28 [00:03<00:00,  8.44it/s, v_num=3, val_loss=0.0849, val_ac\u001b[A\n",
      "Validation: 0it [00:00, ?it/s]\u001b[A\n",
      "Validation:   0%|                                         | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:   0%|                            | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:  33%|██████▎            | 1/3 [00:00<00:00, 310.90it/s]\u001b[A\n",
      "Validation DataLoader 0:  67%|█████████████▎      | 2/3 [00:00<00:00, 77.72it/s]\u001b[A\n",
      "Epoch 5: 100%|█| 28/28 [00:04<00:00,  6.44it/s, v_num=3, val_loss=0.0825, val_ac\u001b[A\n",
      "Epoch 6: 100%|█| 28/28 [00:03<00:00,  8.36it/s, v_num=3, val_loss=0.0825, val_ac\u001b[A\n",
      "Validation: 0it [00:00, ?it/s]\u001b[A\n",
      "Validation:   0%|                                         | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:   0%|                            | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:  33%|██████▎            | 1/3 [00:00<00:00, 319.76it/s]\u001b[A\n",
      "Validation DataLoader 0:  67%|████████████▋      | 2/3 [00:00<00:00, 121.76it/s]\u001b[A\n",
      "Epoch 6: 100%|█| 28/28 [00:04<00:00,  6.49it/s, v_num=3, val_loss=0.0799, val_ac\u001b[A\n",
      "Epoch 7: 100%|█| 28/28 [00:03<00:00,  8.62it/s, v_num=3, val_loss=0.0799, val_ac\u001b[A\n",
      "Validation: 0it [00:00, ?it/s]\u001b[A\n",
      "Validation:   0%|                                         | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:   0%|                            | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:  33%|██████▎            | 1/3 [00:00<00:00, 312.70it/s]\u001b[A\n",
      "Validation DataLoader 0:  67%|████████████▋      | 2/3 [00:00<00:00, 365.56it/s]\u001b[A\n",
      "Epoch 7: 100%|█| 28/28 [00:04<00:00,  6.41it/s, v_num=3, val_loss=0.0754, val_ac\u001b[A\n",
      "Epoch 8: 100%|█| 28/28 [00:03<00:00,  8.63it/s, v_num=3, val_loss=0.0754, val_ac\u001b[A\n",
      "Validation: 0it [00:00, ?it/s]\u001b[A\n",
      "Validation:   0%|                                         | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:   0%|                            | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:  33%|██████▎            | 1/3 [00:00<00:00, 305.02it/s]\u001b[A\n",
      "Validation DataLoader 0:  67%|█████████████▎      | 2/3 [00:00<00:00, 50.97it/s]\u001b[A\n",
      "Epoch 8: 100%|█| 28/28 [00:04<00:00,  6.56it/s, v_num=3, val_loss=0.0732, val_ac\u001b[A\n",
      "Epoch 9: 100%|█| 28/28 [00:03<00:00,  8.41it/s, v_num=3, val_loss=0.0732, val_ac\u001b[A\n",
      "Validation: 0it [00:00, ?it/s]\u001b[A\n",
      "Validation:   0%|                                         | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:   0%|                            | 0/3 [00:00<?, ?it/s]\u001b[A\n",
      "Validation DataLoader 0:  33%|██████▎            | 1/3 [00:00<00:00, 254.59it/s]\u001b[A\n",
      "Validation DataLoader 0:  67%|████████████▋      | 2/3 [00:00<00:00, 283.16it/s]\u001b[A\n",
      "Epoch 9: 100%|█| 28/28 [00:04<00:00,  6.43it/s, v_num=3, val_loss=0.0707, val_ac\u001b[A\n",
      "Epoch 9: 100%|█| 28/28 [00:04<00:00,  6.42it/s, v_num=3, val_loss=0.0707, val_ac\u001b[A"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`Trainer.fit` stopped: `max_epochs=10` reached.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 9: 100%|█| 28/28 [00:04<00:00,  6.42it/s, v_num=3, val_loss=0.0707, val_ac\n"
     ]
    }
   ],
   "source": [
    "# now finetune e2e on the new data\n",
    "model_5_9 = LitMNIST.load_from_checkpoint(checkpoint_path=\"model.ckpt\")\n",
    "# set it up to run on the second half of mnist (digits 5,6,7,8,9)\n",
    "model_loaded_no_training.class_names= [5,6,7,8,9]\n",
    "model_loaded_no_training.min_class = min(model_loaded_no_training.class_names)\n",
    "\n",
    "trainer = L.Trainer(\n",
    "    accelerator=\"auto\",\n",
    "    devices=1,\n",
    "    max_epochs=10,\n",
    "    logger=CSVLogger(save_dir=\"logs/\"),\n",
    ")\n",
    "trainer.fit(model_5_9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "4fd0c776-ced6-48d2-bfc2-ebc5fb873ae6",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Restoring states from the checkpoint path at logs/lightning_logs/version_3/checkpoints/epoch=9-step=280.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "Loaded model weights from the checkpoint at logs/lightning_logs/version_3/checkpoints/epoch=9-step=280.ckpt\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing DataLoader 0: 100%|███████████████████████| 6/6 [00:00<00:00, 37.82it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\">   Runningstage.testing    </span>┃<span style=\"font-weight: bold\">                           </span>┃\n",
       "┃<span style=\"font-weight: bold\">          metric           </span>┃<span style=\"font-weight: bold\">       DataLoader 0        </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_acc          </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.9844327569007874     </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_loss         </span>│<span style=\"color: #800080; text-decoration-color: #800080\">   0.053532421588897705    </span>│\n",
       "└───────────────────────────┴───────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1m  Runningstage.testing   \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m                           \u001b[0m┃\n",
       "┃\u001b[1m \u001b[0m\u001b[1m         metric          \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      DataLoader 0       \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_acc         \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.9844327569007874    \u001b[0m\u001b[35m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_loss        \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m  0.053532421588897705   \u001b[0m\u001b[35m \u001b[0m│\n",
       "└───────────────────────────┴───────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[{'test_loss': 0.053532421588897705, 'test_acc': 0.9844327569007874}]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# unsuprisingly finetuning allows us to perform equally well on the second half of MNIST. However this approach would require us to save the ENTIRE model binary for each dataset we wanted to finetune our initial model on, which doesn't scale well.\n",
    "trainer.test()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b37c4c1e-8141-4416-a0a1-1f8e596abd17",
   "metadata": {},
   "source": [
    "# Finetune on second half of MNIST using LoRA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "27563e84-9d50-480c-9202-91d7b5ec518c",
   "metadata": {},
   "outputs": [],
   "source": [
    "class LitMNISTLoRA(L.LightningModule):\n",
    "    def __init__(self, data_dir=PATH_DATASETS, hidden_size=64, lr=2e-4, lora_rank = 8):\n",
    "        super().__init__()\n",
    "\n",
    "        # Set our init args as class attributes\n",
    "        self.data_dir = data_dir\n",
    "        self.hidden_size = hidden_size\n",
    "        self.lr = lr\n",
    "\n",
    "        # Hardcode some dataset specific attributes\n",
    "        self.class_names = [0,1,2,3,4]\n",
    "        self.min_class = min(self.class_names)\n",
    "        self.num_classes = len(self.class_names) \n",
    "        self.dims = (1, 28, 28) # the shape of an example (C x W x H)\n",
    "        channels, width, height = self.dims\n",
    "\n",
    "        self.transform = transforms.Compose(\n",
    "            [\n",
    "                transforms.ToTensor(),\n",
    "                transforms.Normalize((0.1307,), (0.3081,)),\n",
    "            ]\n",
    "        )\n",
    "\n",
    "        self.batch_size = 64\n",
    "        \n",
    "        # Define layers for model\n",
    "        self.l1 = nn.Linear(channels * width * height, hidden_size)\n",
    "        self.l2 = nn.Linear(hidden_size, hidden_size)\n",
    "        self.l3 = nn.Linear(hidden_size, self.num_classes)\n",
    "\n",
    "        self.dropout = nn.Dropout(0.1)\n",
    "        self.relu = nn.ReLU()\n",
    "\n",
    "        # Define lora hyperparameters\n",
    "        self.lora_rank = lora_rank # The rank 'r' for the low-rank adaptation\n",
    "        self.lora_alpha = 1 # lora scaling factor\n",
    "        \n",
    "        # layer 1 lora layers\n",
    "        self.l1_lora_A = nn.Parameter(torch.empty(channels * width * height, self.lora_rank))\n",
    "        self.l1_lora_B = nn.Parameter(torch.empty(self.lora_rank, hidden_size))\n",
    "\n",
    "        # layer 2 lora layers\n",
    "        self.l2_lora_A =  nn.Parameter(torch.empty(hidden_size, self.lora_rank))\n",
    "        self.l2_lora_B = nn.Parameter(torch.empty(self.lora_rank, hidden_size))\n",
    "\n",
    "        # layer 3 lora layers\n",
    "        self.l3_lora_A = nn.Parameter(torch.empty(hidden_size, self.lora_rank))\n",
    "        self.l3_lora_B = nn.Parameter(torch.empty(self.lora_rank, self.num_classes))\n",
    "        \n",
    "        # Define initialization for lora layers (this ensures that the model behavior is identital to to the original model prior to finetuning)\n",
    "        for n,p in self.named_parameters():\n",
    "            if 'lora' in n:\n",
    "                if n[-1]=='A':\n",
    "                    nn.init.kaiming_uniform_(p, a=math.sqrt(5))\n",
    "                elif n[-1]=='B':\n",
    "                    nn.init.zeros_(p)\n",
    "\n",
    "        # freeze non lora weights\n",
    "        for n,p in self.named_parameters():\n",
    "            if 'lora' not in n:\n",
    "                p.requires_grad = False\n",
    "        \n",
    "        # Define metrics\n",
    "        self.val_accuracy = Accuracy(task=\"multiclass\", num_classes=self.num_classes)\n",
    "        self.test_accuracy = Accuracy(task=\"multiclass\", num_classes=self.num_classes)\n",
    "\n",
    "\n",
    "    def lora_linear(self, x, layer, lora_A, lora_B):\n",
    "        # does the work of combining outputs from normal layer and lora layer for x\n",
    "        # notice that h is the sum of two separate operations on x\n",
    "        h = layer(x)\n",
    "        h += x@(lora_A @ lora_B)*self.lora_alpha\n",
    "        return h\n",
    "        \n",
    "    def forward(self, x):\n",
    "        # preprocessing\n",
    "        x = torch.flatten(x,1)\n",
    "        \n",
    "        # layer 1 (input size, hidden size)\n",
    "        x = self.lora_linear(x, self.l1, self.l1_lora_A, self.l1_lora_B)\n",
    "        x = self.relu(x)\n",
    "        x = self.dropout(x)\n",
    "\n",
    "        # layer 2 (hidden size, hidden size)\n",
    "        x = self.lora_linear(x, self.l2, self.l2_lora_A, self.l2_lora_B)\n",
    "        x = self.relu(x)\n",
    "        x = self.dropout(x)\n",
    "\n",
    "        #layer 3 (hidden size, self.num_classes)\n",
    "        x = self.lora_linear(x, self.l3, self.l3_lora_A, self.l3_lora_B)\n",
    "                    \n",
    "        # notice that we return the log probabilities here as that is what nll loss expects in the training step\n",
    "        return F.log_softmax(x, dim=1)\n",
    "\n",
    "    def common_step(self, batch, batch_idx):\n",
    "        # define steps all of {train, val, test} will take in one place\n",
    "        x, y = batch\n",
    "        # rescale y to be 0 indexed if necessary (like when we start using mnist 5-9)\n",
    "        if self.min_class != 0:\n",
    "            y = y - self.min_class\n",
    "        \n",
    "        logits = self(x)\n",
    "        loss = F.nll_loss(logits, y)\n",
    "        return x,y, logits, loss\n",
    "        \n",
    "        \n",
    "\n",
    "    def training_step(self, batch, batch_idx):\n",
    "        _, _, _, loss = self.common_step(batch, batch_idx)\n",
    "        self.log(\"train_loss\", loss, batch_size=self.batch_size)\n",
    "        return loss\n",
    "\n",
    "    def validation_step(self, batch, batch_idx):\n",
    "        x, y, logits, loss = self.common_step(batch, batch_idx)\n",
    "        preds = torch.argmax(logits, dim=1)\n",
    "        self.val_accuracy.update(preds, y)\n",
    "\n",
    "        self.log(\"val_loss\", loss, prog_bar=False)\n",
    "        self.log(\"val_acc\", self.val_accuracy, prog_bar=True)\n",
    "\n",
    "    def test_step(self, batch, batch_idx):\n",
    "        x, y, logits, loss = self.common_step(batch, batch_idx)\n",
    "        preds = torch.argmax(logits, dim=1)\n",
    "        self.test_accuracy.update(preds, y)\n",
    "\n",
    "        self.log(\"test_loss\", loss, prog_bar=True)\n",
    "        self.log(\"test_acc\", self.test_accuracy, prog_bar=True)\n",
    "    \n",
    "    def configure_optimizers(self):\n",
    "        # we'll use adamw to match the paper\n",
    "        optimizer = torch.optim.AdamW(self.parameters(), lr=self.lr)\n",
    "        scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min', patience=10)\n",
    "\n",
    "        return {\n",
    "        \"optimizer\": optimizer,\n",
    "        \"lr_scheduler\": {\n",
    "            \"scheduler\": scheduler,\n",
    "            \"monitor\": \"val_loss\",\n",
    "            \"frequency\": 1\n",
    "        },\n",
    "        }\n",
    "\n",
    "      \n",
    "    # the rest of the class is helper functions/hooks for configuring data/dataloader building\n",
    "    @staticmethod\n",
    "    def get_indices(dataset,class_names):\n",
    "        if isinstance(dataset, torch.utils.data.dataset.Subset):\n",
    "            targets = torch.tensor([dataset.dataset.targets[i] for i in dataset.indices])\n",
    "        else:\n",
    "            targets = dataset.targets\n",
    "\n",
    "            \n",
    "        indices =  []\n",
    "        for i in range(len(targets)):\n",
    "            if targets[i] in class_names:\n",
    "                indices.append(i)\n",
    "        return indices\n",
    "        \n",
    "    def create_dataloader(self,dataset):\n",
    "        idx = self.get_indices(dataset, self.class_names)\n",
    "        loader = DataLoader(dataset,batch_size=self.batch_size, sampler = SubsetRandomSampler(idx), num_workers=16) # Note - this necessarily shuffles the data due to the sampler we are using\n",
    "        return loader\n",
    "\n",
    "    def prepare_data(self):\n",
    "        # download data\n",
    "        MNIST(self.data_dir, train=True, download=True)\n",
    "        MNIST(self.data_dir, train=False, download=True)\n",
    "\n",
    "    def setup(self, stage=None):\n",
    "        # Assign train/val datasets for use in dataloaders\n",
    "        if stage == \"fit\" or stage is None:\n",
    "            mnist_full = MNIST(self.data_dir, train=True, transform=self.transform)\n",
    "            self.mnist_train, self.mnist_val = random_split(mnist_full, [55000, 5000])\n",
    "\n",
    "        # Assign test dataset for use in dataloader(s)\n",
    "        if stage == \"test\" or stage is None:\n",
    "            self.mnist_test = MNIST(self.data_dir, train=False, transform=self.transform)\n",
    "\n",
    "    def train_dataloader(self):\n",
    "        return self.create_dataloader(self.mnist_train)\n",
    "\n",
    "    def val_dataloader(self):\n",
    "        return self.create_dataloader(self.mnist_val)\n",
    "\n",
    "    def test_dataloader(self):\n",
    "        return self.create_dataloader(self.mnist_test)\n",
    "\n",
    "        \n",
    "        \n",
    "\n",
    "        \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "44986317-f1ec-4b53-b68d-47b0db656ac1",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# try lora finetuning with different lora ranks\n",
    "def lora_experiment(rank):\n",
    "    state_dict = torch.load(\"model.pt\")\n",
    "    model = LitMNISTLoRA(lora_rank=rank,)\n",
    "    model.load_state_dict(state_dict, strict=False)\n",
    "    model.class_names= [5,6,7,8,9]\n",
    "    model.min_class = min(model.class_names)\n",
    "    \n",
    "    from lightning.pytorch.callbacks import LearningRateMonitor\n",
    "    lr_monitor = LearningRateMonitor(logging_interval='step')\n",
    "    \n",
    "    trainer = L.Trainer(\n",
    "        accelerator=\"auto\",\n",
    "        devices=1,\n",
    "        max_epochs=100,\n",
    "        callbacks=[lr_monitor, EarlyStopping(monitor=\"val_loss\", mode=\"min\", patience=10)],\n",
    "\n",
    "        enable_progress_bar=False\n",
    "    )\n",
    "    \n",
    "    trainer.fit(model)\n",
    "    return trainer.test()[0]['test_acc']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "b6dcee69-a9c7-4bb9-80a9-2bb5d8747cc9",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: True (cuda), used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "HPU available: False, using: 0 HPUs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "\n",
      "  | Name          | Type               | Params\n",
      "-----------------------------------------------------\n",
      "0 | l1            | Linear             | 50.2 K\n",
      "1 | l2            | Linear             | 4.2 K \n",
      "2 | l3            | Linear             | 325   \n",
      "3 | dropout       | Dropout            | 0     \n",
      "4 | relu          | ReLU               | 0     \n",
      "5 | val_accuracy  | MulticlassAccuracy | 0     \n",
      "6 | test_accuracy | MulticlassAccuracy | 0     \n",
      "-----------------------------------------------------\n",
      "1.0 K     Trainable params\n",
      "54.7 K    Non-trainable params\n",
      "55.8 K    Total params\n",
      "0.223     Total estimated model params size (MB)\n",
      "`Trainer.fit` stopped: `max_epochs=100` reached.\n",
      "Restoring states from the checkpoint path at /home/sunil/lora_from_scratch/lightning_logs/version_0/checkpoints/epoch=99-step=42100.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "Loaded model weights from the checkpoint at /home/sunil/lora_from_scratch/lightning_logs/version_0/checkpoints/epoch=99-step=42100.ckpt\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\">   Runningstage.testing    </span>┃<span style=\"font-weight: bold\">                           </span>┃\n",
       "┃<span style=\"font-weight: bold\">          metric           </span>┃<span style=\"font-weight: bold\">       DataLoader 0        </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_acc          </span>│<span style=\"color: #800080; text-decoration-color: #800080\">     0.874717116355896     </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_loss         </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.37462300062179565    </span>│\n",
       "└───────────────────────────┴───────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1m  Runningstage.testing   \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m                           \u001b[0m┃\n",
       "┃\u001b[1m \u001b[0m\u001b[1m         metric          \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      DataLoader 0       \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_acc         \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m    0.874717116355896    \u001b[0m\u001b[35m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_loss        \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.37462300062179565   \u001b[0m\u001b[35m \u001b[0m│\n",
       "└───────────────────────────┴───────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: True (cuda), used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "HPU available: False, using: 0 HPUs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "\n",
      "  | Name          | Type               | Params\n",
      "-----------------------------------------------------\n",
      "0 | l1            | Linear             | 50.2 K\n",
      "1 | l2            | Linear             | 4.2 K \n",
      "2 | l3            | Linear             | 325   \n",
      "3 | dropout       | Dropout            | 0     \n",
      "4 | relu          | ReLU               | 0     \n",
      "5 | val_accuracy  | MulticlassAccuracy | 0     \n",
      "6 | test_accuracy | MulticlassAccuracy | 0     \n",
      "-----------------------------------------------------\n",
      "2.1 K     Trainable params\n",
      "54.7 K    Non-trainable params\n",
      "56.8 K    Total params\n",
      "0.227     Total estimated model params size (MB)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1: 0.874717116355896}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`Trainer.fit` stopped: `max_epochs=100` reached.\n",
      "Restoring states from the checkpoint path at /home/sunil/lora_from_scratch/lightning_logs/version_1/checkpoints/epoch=99-step=42100.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "Loaded model weights from the checkpoint at /home/sunil/lora_from_scratch/lightning_logs/version_1/checkpoints/epoch=99-step=42100.ckpt\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\">   Runningstage.testing    </span>┃<span style=\"font-weight: bold\">                           </span>┃\n",
       "┃<span style=\"font-weight: bold\">          metric           </span>┃<span style=\"font-weight: bold\">       DataLoader 0        </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_acc          </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.9314955472946167     </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_loss         </span>│<span style=\"color: #800080; text-decoration-color: #800080\">     0.212742418050766     </span>│\n",
       "└───────────────────────────┴───────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1m  Runningstage.testing   \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m                           \u001b[0m┃\n",
       "┃\u001b[1m \u001b[0m\u001b[1m         metric          \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      DataLoader 0       \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_acc         \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.9314955472946167    \u001b[0m\u001b[35m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_loss        \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m    0.212742418050766    \u001b[0m\u001b[35m \u001b[0m│\n",
       "└───────────────────────────┴───────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: True (cuda), used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "HPU available: False, using: 0 HPUs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "\n",
      "  | Name          | Type               | Params\n",
      "-----------------------------------------------------\n",
      "0 | l1            | Linear             | 50.2 K\n",
      "1 | l2            | Linear             | 4.2 K \n",
      "2 | l3            | Linear             | 325   \n",
      "3 | dropout       | Dropout            | 0     \n",
      "4 | relu          | ReLU               | 0     \n",
      "5 | val_accuracy  | MulticlassAccuracy | 0     \n",
      "6 | test_accuracy | MulticlassAccuracy | 0     \n",
      "-----------------------------------------------------\n",
      "4.2 K     Trainable params\n",
      "54.7 K    Non-trainable params\n",
      "58.9 K    Total params\n",
      "0.236     Total estimated model params size (MB)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1: 0.874717116355896, 2: 0.9314955472946167}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`Trainer.fit` stopped: `max_epochs=100` reached.\n",
      "Restoring states from the checkpoint path at /home/sunil/lora_from_scratch/lightning_logs/version_2/checkpoints/epoch=99-step=42100.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "Loaded model weights from the checkpoint at /home/sunil/lora_from_scratch/lightning_logs/version_2/checkpoints/epoch=99-step=42100.ckpt\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\">   Runningstage.testing    </span>┃<span style=\"font-weight: bold\">                           </span>┃\n",
       "┃<span style=\"font-weight: bold\">          metric           </span>┃<span style=\"font-weight: bold\">       DataLoader 0        </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_acc          </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.9639991521835327     </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_loss         </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.1125415563583374     </span>│\n",
       "└───────────────────────────┴───────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1m  Runningstage.testing   \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m                           \u001b[0m┃\n",
       "┃\u001b[1m \u001b[0m\u001b[1m         metric          \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      DataLoader 0       \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_acc         \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.9639991521835327    \u001b[0m\u001b[35m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_loss        \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.1125415563583374    \u001b[0m\u001b[35m \u001b[0m│\n",
       "└───────────────────────────┴───────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: True (cuda), used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "HPU available: False, using: 0 HPUs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "\n",
      "  | Name          | Type               | Params\n",
      "-----------------------------------------------------\n",
      "0 | l1            | Linear             | 50.2 K\n",
      "1 | l2            | Linear             | 4.2 K \n",
      "2 | l3            | Linear             | 325   \n",
      "3 | dropout       | Dropout            | 0     \n",
      "4 | relu          | ReLU               | 0     \n",
      "5 | val_accuracy  | MulticlassAccuracy | 0     \n",
      "6 | test_accuracy | MulticlassAccuracy | 0     \n",
      "-----------------------------------------------------\n",
      "8.4 K     Trainable params\n",
      "54.7 K    Non-trainable params\n",
      "63.1 K    Total params\n",
      "0.252     Total estimated model params size (MB)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1: 0.874717116355896, 2: 0.9314955472946167, 4: 0.9639991521835327}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "`Trainer.fit` stopped: `max_epochs=100` reached.\n",
      "Restoring states from the checkpoint path at /home/sunil/lora_from_scratch/lightning_logs/version_3/checkpoints/epoch=99-step=42200.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "Loaded model weights from the checkpoint at /home/sunil/lora_from_scratch/lightning_logs/version_3/checkpoints/epoch=99-step=42200.ckpt\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\">   Runningstage.testing    </span>┃<span style=\"font-weight: bold\">                           </span>┃\n",
       "┃<span style=\"font-weight: bold\">          metric           </span>┃<span style=\"font-weight: bold\">       DataLoader 0        </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_acc          </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.9709936380386353     </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_loss         </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.09589049965143204    </span>│\n",
       "└───────────────────────────┴───────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1m  Runningstage.testing   \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m                           \u001b[0m┃\n",
       "┃\u001b[1m \u001b[0m\u001b[1m         metric          \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      DataLoader 0       \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_acc         \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.9709936380386353    \u001b[0m\u001b[35m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_loss        \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.09589049965143204   \u001b[0m\u001b[35m \u001b[0m│\n",
       "└───────────────────────────┴───────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: True (cuda), used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "HPU available: False, using: 0 HPUs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "\n",
      "  | Name          | Type               | Params\n",
      "-----------------------------------------------------\n",
      "0 | l1            | Linear             | 50.2 K\n",
      "1 | l2            | Linear             | 4.2 K \n",
      "2 | l3            | Linear             | 325   \n",
      "3 | dropout       | Dropout            | 0     \n",
      "4 | relu          | ReLU               | 0     \n",
      "5 | val_accuracy  | MulticlassAccuracy | 0     \n",
      "6 | test_accuracy | MulticlassAccuracy | 0     \n",
      "-----------------------------------------------------\n",
      "16.7 K    Trainable params\n",
      "54.7 K    Non-trainable params\n",
      "71.4 K    Total params\n",
      "0.286     Total estimated model params size (MB)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1: 0.874717116355896, 2: 0.9314955472946167, 4: 0.9639991521835327, 8: 0.9709936380386353}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Restoring states from the checkpoint path at /home/sunil/lora_from_scratch/lightning_logs/version_4/checkpoints/epoch=64-step=27430.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "Loaded model weights from the checkpoint at /home/sunil/lora_from_scratch/lightning_logs/version_4/checkpoints/epoch=64-step=27430.ckpt\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\">   Runningstage.testing    </span>┃<span style=\"font-weight: bold\">                           </span>┃\n",
       "┃<span style=\"font-weight: bold\">          metric           </span>┃<span style=\"font-weight: bold\">       DataLoader 0        </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_acc          </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.9765480160713196     </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_loss         </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.08745694160461426    </span>│\n",
       "└───────────────────────────┴───────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1m  Runningstage.testing   \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m                           \u001b[0m┃\n",
       "┃\u001b[1m \u001b[0m\u001b[1m         metric          \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      DataLoader 0       \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_acc         \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.9765480160713196    \u001b[0m\u001b[35m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_loss        \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.08745694160461426   \u001b[0m\u001b[35m \u001b[0m│\n",
       "└───────────────────────────┴───────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: True (cuda), used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "HPU available: False, using: 0 HPUs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "\n",
      "  | Name          | Type               | Params\n",
      "-----------------------------------------------------\n",
      "0 | l1            | Linear             | 50.2 K\n",
      "1 | l2            | Linear             | 4.2 K \n",
      "2 | l3            | Linear             | 325   \n",
      "3 | dropout       | Dropout            | 0     \n",
      "4 | relu          | ReLU               | 0     \n",
      "5 | val_accuracy  | MulticlassAccuracy | 0     \n",
      "6 | test_accuracy | MulticlassAccuracy | 0     \n",
      "-----------------------------------------------------\n",
      "33.4 K    Trainable params\n",
      "54.7 K    Non-trainable params\n",
      "88.2 K    Total params\n",
      "0.353     Total estimated model params size (MB)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1: 0.874717116355896, 2: 0.9314955472946167, 4: 0.9639991521835327, 8: 0.9709936380386353, 16: 0.9765480160713196}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Restoring states from the checkpoint path at /home/sunil/lora_from_scratch/lightning_logs/version_5/checkpoints/epoch=48-step=20678.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "Loaded model weights from the checkpoint at /home/sunil/lora_from_scratch/lightning_logs/version_5/checkpoints/epoch=48-step=20678.ckpt\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\">   Runningstage.testing    </span>┃<span style=\"font-weight: bold\">                           </span>┃\n",
       "┃<span style=\"font-weight: bold\">          metric           </span>┃<span style=\"font-weight: bold\">       DataLoader 0        </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_acc          </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.9796338081359863     </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_loss         </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.08891116082668304    </span>│\n",
       "└───────────────────────────┴───────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1m  Runningstage.testing   \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m                           \u001b[0m┃\n",
       "┃\u001b[1m \u001b[0m\u001b[1m         metric          \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      DataLoader 0       \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_acc         \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.9796338081359863    \u001b[0m\u001b[35m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_loss        \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.08891116082668304   \u001b[0m\u001b[35m \u001b[0m│\n",
       "└───────────────────────────┴───────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: True (cuda), used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "HPU available: False, using: 0 HPUs\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "\n",
      "  | Name          | Type               | Params\n",
      "-----------------------------------------------------\n",
      "0 | l1            | Linear             | 50.2 K\n",
      "1 | l2            | Linear             | 4.2 K \n",
      "2 | l3            | Linear             | 325   \n",
      "3 | dropout       | Dropout            | 0     \n",
      "4 | relu          | ReLU               | 0     \n",
      "5 | val_accuracy  | MulticlassAccuracy | 0     \n",
      "6 | test_accuracy | MulticlassAccuracy | 0     \n",
      "-----------------------------------------------------\n",
      "66.9 K    Trainable params\n",
      "54.7 K    Non-trainable params\n",
      "121 K     Total params\n",
      "0.486     Total estimated model params size (MB)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1: 0.874717116355896, 2: 0.9314955472946167, 4: 0.9639991521835327, 8: 0.9709936380386353, 16: 0.9765480160713196, 32: 0.9796338081359863}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Restoring states from the checkpoint path at /home/sunil/lora_from_scratch/lightning_logs/version_6/checkpoints/epoch=37-step=16036.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "Loaded model weights from the checkpoint at /home/sunil/lora_from_scratch/lightning_logs/version_6/checkpoints/epoch=37-step=16036.ckpt\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\">   Runningstage.testing    </span>┃<span style=\"font-weight: bold\">                           </span>┃\n",
       "┃<span style=\"font-weight: bold\">          metric           </span>┃<span style=\"font-weight: bold\">       DataLoader 0        </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_acc          </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.9804567098617554     </span>│\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_loss         </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.08482691645622253    </span>│\n",
       "└───────────────────────────┴───────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1m  Runningstage.testing   \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m                           \u001b[0m┃\n",
       "┃\u001b[1m \u001b[0m\u001b[1m         metric          \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      DataLoader 0       \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_acc         \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.9804567098617554    \u001b[0m\u001b[35m \u001b[0m│\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_loss        \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.08482691645622253   \u001b[0m\u001b[35m \u001b[0m│\n",
       "└───────────────────────────┴───────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1: 0.874717116355896, 2: 0.9314955472946167, 4: 0.9639991521835327, 8: 0.9709936380386353, 16: 0.9765480160713196, 32: 0.9796338081359863, 64: 0.9804567098617554}\n"
     ]
    }
   ],
   "source": [
    "results = {}\n",
    "for rank in [1, 2, 4, 8, 16, 32, 64]:\n",
    "    result = lora_experiment(rank)\n",
    "    results[rank] = result\n",
    "    print(results)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d712d24e-596b-49dc-a270-3ace8eae3b6d",
   "metadata": {},
   "source": [
    "Test results: {1: 0.874717116355896, 2: 0.9314955472946167, 4: 0.9639991521835327, 8: 0.9709936380386353, 16: 0.9765480160713196, 32: 0.9796338081359863, 64: 0.9804567098617554}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a9458be-b334-4da8-be94-c42b36c72f01",
   "metadata": {},
   "source": [
    "| model                         |approx. number of trainable parameters  | test accuracy      | percent trainable parameters relative to baseline | percent test accuracy relative to baseline |\n",
    "|-------------------------------|--------------------------------|--------------------|-------------------------------------------|------------------------------------|\n",
    "| baseline - whole model finetune | 54700                          | 0.9844327569007874 |                                         1 |                                  1 |\n",
    "| LoRA rank = 1                      | 1000                           | 0.874717116355896  |                             0.01828153565 |                       0.8885493806 |\n",
    "| LoRA rank = 2                      | 2100                           | 0.9314955472946167 |                             0.03839122486 |                       0.9462256724 |\n",
    "| LoRA rank = 4                      | 4200                           | 0.9639991521835327 |                             0.07678244973 |                       0.9792432702 |\n",
    "| LoRA rank= 8                       | 8400                           | 0.9709936380386353 |                              0.1535648995 |                       0.9863483628 |\n",
    "| LoRA rank = 16                     | 16700                          | 0.9765480160713196 |                              0.3053016453 |                       0.9919905745 |\n",
    "| LoRA rank = 32                     | 33400                          | 0.9796338081359863 |                              0.6106032907 |                       0.9951251635 |\n",
    "| LoRA rank = 64                     | 66900                          | 0.9804567098617554 |                               1.223034735 |                       0.9959610781 |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "8d647881-2fb5-4359-96fe-a5aed4f962c8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAm4AAAHFCAYAAABLm3WjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAACKJElEQVR4nO3deXhM1xsH8O9k3yN7gkiskTTWWIMSSqwNqo2qJbaWohS1FLW09qW0RdHauqCL+qGK2HchdiF2URIhSMiemfP7I50rk5kkM0wyYr6f55mHOXPuve+9c2fmzTn3nCsTQggQERER0SvPxNABEBEREZF2mLgRERERlRJM3IiIiIhKCSZuRERERKUEEzciIiKiUoKJGxEREVEpwcSNiIiIqJRg4kZERERUSjBxIyIiIiolSiRxW716NWQyGaysrHD79m2111u0aIHAwMCSCEXNvn37IJPJ8Mcffxhk+7q6desWOnToAGdnZ8hkMowYMaLYttWiRQu0aNGiyHrKY7hv3z6pLCIiAr6+vir1ZsyYgU2bNuk1xpelPDdv3bpVaL0jR45gypQpePLkSbHEoel46Zsu5/qUKVMgk8mKNR5js2TJEqxevdrQYZSoW7duQSaTaXzUq1cPAODr64uIiIhijaO4P7/5afvdWRxK4jy7d+8epkyZgjNnzhTrdkgzs5LcWGZmJiZOnIiffvqpJDf7Wvn0009x/PhxrFy5Ep6envDy8jJ0SBpNmjQJw4cPVymbMWMGunXrhs6dOxsmqJdw5MgRTJ06FREREShTpoze16/peNHrZcmSJXB1dS32JOVVNGzYMPTo0UOlzM7ODgDw119/wcHBoVi3X9yf3/yWLFlS7NsobNvFfZ7du3cPU6dOha+vL2rXrl1s2yHNSjRxa9u2LX799VeMHj0atWrVKslNG1x6ejqsrKxeuhXjwoULaNCgwSuf/FSuXNnQIRhUeno6rK2tta5v7MdLX9LS0mBjY2PoMEqMEAIZGRk6nWuGUKFCBTRq1Ejja3Xq1CnhaIpfQECAoUOgl6Cv3+viUqLXuI0ZMwYuLi4YO3ZsofWUzeuamntlMhmmTJkiPVd26Zw7dw7vvvsuHB0d4ezsjJEjRyInJwexsbFo27Yt7O3t4evrizlz5mjcZkZGBkaOHAlPT09YW1ujefPmOH36tFq9kydP4u2334azszOsrKxQp04d/Pbbbyp1lN1vO3fuRL9+/eDm5gYbGxtkZmYWuM9xcXHo2bMn3N3dYWlpCX9/f8yfPx8KhQLA826ua9eu4Z9//pG6Gwrr4lu8eDHefPNNuLu7w9bWFjVq1MCcOXOQnZ2tUk8IgTlz5sDHxwdWVlaoW7cu/vnnH43rvHz5Mtq2bQsbGxu4urpi0KBBePr0qVq9/F1/MpkMqampWLNmjRR73q6ECxcuICwsDE5OTrCyskLt2rWxZs0alXUqj8G6deswYcIElC1bFg4ODnjrrbcQGxurUjcyMhJhYWEoX748rKysUKVKFXz00Ud4+PBhgcerIFOmTMFnn30GAKhYsaIUv7Jr2NfXFx07dsTGjRtRp04dWFlZYerUqQC0fw80dZXKZDIMHToUP/30E/z9/WFjY4NatWph69atKvWuXbuGvn37omrVqrCxsUG5cuXQqVMnnD9/XuP+aHuua7JhwwY0btwYtra2sLOzQ2hoqFbLKj8TkZGR6Nu3L5ydnWFra4tOnTrhxo0bKnW1fe+Un/1Tp06hW7ducHJykhLgkydPonv37vD19YW1tTV8fX3x/vvvq12qoYxrz549GDhwIFxcXODg4IDevXsjNTUVCQkJeO+991CmTBl4eXlh9OjRau9dVlYWvvrqK1SvXh2WlpZwc3ND37598eDBA6mOr68vLl68iP3790vnT973OyUlBaNHj0bFihVhYWGBcuXKYcSIEUhNTVXZlvKc+P777+Hv7w9LS0vpc7J06VLUqlULdnZ2sLe3R/Xq1fH5558X+J5kZ2fD3d0dvXr1UnvtyZMnsLa2xsiRIwEACoUCX331Ffz8/GBtbY0yZcqgZs2aWLRoUYHr11b+rlJdPucAsGvXLrRq1QoODg6wsbFBkyZNsHv3bun1oj6/+X9TCopLea7s3bsXgwcPhqurK1xcXNC1a1fcu3dPZdn8XaXK37R58+ZhwYIFqFixIuzs7NC4cWMcO3ZMbdsrVqxAtWrVYGlpiYCAAPz6669aXU6hr/Ps999/R8OGDeHo6AgbGxtUqlQJ/fr1A5D7/tSvXx8A0LdvX2k7mo6h0oMHD/Dxxx8jICAAdnZ2cHd3R8uWLXHw4EG1upmZmZg2bRr8/f1hZWUFFxcXhISE4MiRI1IdhUKBb7/9FrVr15bOx0aNGmHz5s1SHV3fV02/17p8tz558gSjRo1CpUqVYGlpCXd3d7Rv3x6XL1+GEAJVq1ZFaGio2nLPnj2Do6MjhgwZUuDxy69EW9zs7e0xceJEDB8+HHv27EHLli31tu733nsPPXv2xEcffYTIyEjpx3HXrl34+OOPMXr0aPz6668YO3YsqlSpgq5du6os//nnn6Nu3br44YcfkJycjClTpqBFixY4ffo0KlWqBADYu3cv2rZti4YNG+L777+Ho6Mj1q9fj/DwcKSlpak1Tffr1w8dOnTATz/9hNTUVJibm2uM/cGDBwgODkZWVha+/PJL+Pr6YuvWrRg9ejSuX7+OJUuWoG7dujh69Ci6dOmCypUrY968eQBQaFfp9evX0aNHD+lDevbsWUyfPh2XL1/GypUrpXpTp07F1KlT0b9/f3Tr1g137tzBwIEDIZfL4efnJ9W7f/8+mjdvDnNzcyxZsgQeHh745ZdfMHTo0CLfn6NHj6Jly5YICQnBpEmTAEDqHomNjUVwcDDc3d3xzTffwMXFBT///DMiIiJw//59jBkzRu29atKkCX744QekpKRg7Nix6NSpEy5dugRTU1Np3xs3bowBAwbA0dERt27dwoIFC9C0aVOcP3++wPdCkwEDBuDRo0f49ttvsXHjRumY5/2r+tSpU7h06RImTpyIihUrwtbWVqf3oCB///03Tpw4gWnTpsHOzg5z5sxBly5dEBsbK52X9+7dg4uLC2bNmgU3Nzc8evQIa9asQcOGDXH69GmV91B5/Io61zWZMWMGJk6ciL59+2LixInIysrC3Llz0axZM0RFRWnVytC/f3+0bt0av/76K+7cuYOJEyeiRYsWOHfunNSFpet717VrV3Tv3h2DBg2SfoBu3boFPz8/dO/eHc7OzoiPj8fSpUtRv359xMTEwNXVVWUdAwYMQNeuXbF+/XqcPn0an3/+ufSHX9euXfHhhx9i165dmD17NsqWLauS0ISFheHgwYMYM2YMgoODcfv2bUyePBktWrTAyZMnYW1tjb/++gvdunWDo6Oj1I1maWkJILeVsHnz5vj333/x+eefo2bNmrh48SK++OILnD9/Hrt27VL5y3/Tpk04ePAgvvjiC3h6esLd3R3r16/Hxx9/jGHDhmHevHkwMTHBtWvXEBMTU+B7YW5ujp49e+L777/H4sWLVbor161bh4yMDPTt2xcAMGfOHEyZMgUTJ07Em2++iezsbFy+fFnra8YUCgVycnJUykxNTQtt0dDmc/7zzz+jd+/eCAsLw5o1a2Bubo5ly5YhNDQUO3bsQKtWrbT6/OpiwIAB6NChg3QOf/bZZ+jZsyf27NlT5LKLFy9G9erVsXDhQgC5l0i0b98eN2/ehKOjIwBg+fLl+Oijj/DOO+/g66+/RnJyMqZOnVroH/5K+jjPjh49ivDwcISHh2PKlCnStenK/atbty5WrVolfQ906NABAFC+fPkC43r06BEAYPLkyfD09MSzZ8/w119/oUWLFti9e7eU5Obk5KBdu3Y4ePAgRowYgZYtWyInJwfHjh1DXFwcgoODAeT+ofvzzz+jf//+mDZtGiwsLHDq1Kkir1UujKbfa22/W58+fYqmTZvi1q1bGDt2LBo2bIhnz57hwIEDiI+PR/Xq1TFs2DCMGDECV69eRdWqVaXtrl27FikpKTolbhAlYNWqVQKAOHHihMjMzBSVKlUS9erVEwqFQgghRPPmzcUbb7wh1b9586YAIFatWqW2LgBi8uTJ0vPJkycLAGL+/Pkq9WrXri0AiI0bN0pl2dnZws3NTXTt2lUq27t3rwAg6tatK8UjhBC3bt0S5ubmYsCAAVJZ9erVRZ06dUR2drbKtjp27Ci8vLyEXC5X2d/evXtrdXzGjRsnAIjjx4+rlA8ePFjIZDIRGxsrlfn4+IgOHTpotd685HK5yM7OFmvXrhWmpqbi0aNHQgghHj9+LKysrESXLl1U6h8+fFgAEM2bN5fKxo4dK2QymThz5oxK3datWwsAYu/evVJZnz59hI+Pj0o9W1tb0adPH7XYunfvLiwtLUVcXJxKebt27YSNjY148uSJEOL5e9W+fXuVer/99psAII4ePapx3xUKhcjOzha3b98WAMT//vc/6TXle3Xz5k2NyyrNnTu3wHo+Pj7C1NRU5X3SpKD3QAjNxwuA8PDwECkpKVJZQkKCMDExETNnzixwOzk5OSIrK0tUrVpVfPrpp1K5Lue68nOlFBcXJ8zMzMSwYcNUtvX06VPh6ekp3nvvvUL3XXmcCzrPvvrqK43LFfbeKWP84osvCt22ELnH5NmzZ8LW1lYsWrRILa78+9W5c2cBQCxYsEClvHbt2qJu3brS83Xr1gkA4s8//1Spd+LECQFALFmyRCp74403VD5PSjNnzhQmJibixIkTKuV//PGHACC2bdsmlQEQjo6OKueOEEIMHTpUlClTpoijoO7cuXMCgFi+fLlKeYMGDURQUJD0vGPHjqJ27do6r1/5Xa7pERkZKYTI/fzk/V7Q9nOempoqnJ2dRadOnVTqyeVyUatWLdGgQQOprLDPb/7fFKX8cSnPlY8//lil3pw5cwQAER8fL5U1b95c5b1WHocaNWqInJwcqTwqKkoAEOvWrZNi9/T0FA0bNlTZxu3bt4W5ubnad4QmL3uezZs3TwCQvnc1UZ7fmn6jtZGTkyOys7NFq1atVL4T1q5dKwCIFStWFLjsgQMHBAAxYcKEQreh6/uqze91Qd+t06ZNUzmnNUlJSRH29vZi+PDhKuUBAQEiJCSkyG3nVeLTgVhYWOCrr77CyZMn1boYX0bHjh1Vnvv7+0Mmk6Fdu3ZSmZmZGapUqaJxZGuPHj1U/vrz8fFBcHAw9u7dCyC3O+ry5cv44IMPAOT+ZaB8tG/fHvHx8WrN+O+8845Wse/ZswcBAQFo0KCBSnlERASEEFr9JafJ6dOn8fbbb8PFxQWmpqYwNzdH7969IZfLceXKFQC5LWEZGRnSfikFBwfDx8dHpWzv3r1444031K5PzH/Rsa727NmDVq1awdvbW6U8IiICaWlpOHr0qEr522+/rfK8Zs2aAKDyviYmJmLQoEHw9vaGmZkZzM3Npf25dOnSS8WrSc2aNVGtWjW1cm3eg8KEhITA3t5eeu7h4QF3d3eVfc3JycGMGTMQEBAACwsLmJmZwcLCAlevXtW4r0Wd65rs2LEDOTk56N27t8q5b2VlhebNm6uMKC5MQedZ3m3r+t5p+pw9e/ZMal03MzODmZkZ7OzskJqaqnEdmr4/AEitCXnL8x77rVu3okyZMujUqZPKcalduzY8PT21Oi5bt25FYGAgateurbKO0NBQtdHaANCyZUs4OTmplDVo0ABPnjzB+++/j//9739aXxJQo0YNBAUFYdWqVVLZpUuXEBUVJXWNKdd/9uxZfPzxx9ixYwdSUlK0Wr/S8OHDceLECZVHw4YNC12mqM/5kSNH8OjRI/Tp00fluCkUCrRt2xYnTpxQ6wLUB22+fwrSoUMHqbVQ07KxsbFS93xeFSpUQJMmTV4qbm3PM2U36HvvvYfffvsNd+/efantKn3//feoW7curKyspM/17t27VT6P//zzD6ysrFTOvfyUl/Ho1EKlBU3fI9p+t/7zzz+oVq0a3nrrrQLXb29vj759+2L16tXSeblnzx7ExMRo1WuVl0HmcevevTvq1q2LCRMmqF0v8qKcnZ1VnltYWMDGxgZWVlZq5RkZGWrLe3p6aixLSkoCkNtNCACjR4+Gubm5yuPjjz8GALUvS21HfCYlJWmsW7ZsWel1XcXFxaFZs2a4e/cuFi1ahIMHD+LEiRNYvHgxgNyLL/Ouu6D9zx+nNvV0pev+u7i4qDxXdgUo90mhUKBNmzbYuHEjxowZg927dyMqKkq6lkRZT580xa/te1CY/PsK5O5v3mVHjhyJSZMmoXPnztiyZQuOHz+OEydOoFatWhq3UdS5rony/K9fv77a+b9hwwatE4Witv0i752mY9+jRw989913GDBgAHbs2IGoqCicOHECbm5uGteh6fujoPK83x/379/HkydPYGFhoXZcEhIStDou9+/fx7lz59SWt7e3hxBCq++VXr16YeXKlbh9+zbeeecduLu7o2HDhoiMjCxy+/369cPRo0dx+fJlAMCqVatgaWmJ999/X6ozfvx4zJs3D8eOHUO7du3g4uKCVq1a4eTJk0WuH8jtRqtXr57KI+8fJJoU9TlXnpPdunVTO3azZ8+GEELqotOnouJ6mWWVnwMPDw+1ZTWV6ULb8+zNN9/Epk2bpD/Uypcvj8DAQKxbt+6Ft71gwQIMHjwYDRs2xJ9//oljx47hxIkTaNu2rcpxe/DgAcqWLQsTk4JTkwcPHsDU1PSlf3fy0/S50va79cGDB4V2FSsNGzYMT58+xS+//AIA+O6771C+fHmEhYXpFGuJXuOmJJPJMHv2bLRu3RrLly9Xe12ZbOXv03+RBEZbCQkJGsuUHzTlNTHjx49Xuz5OKf+1RNqOSHFxcUF8fLxaufKC1/zX42hj06ZNSE1NxcaNG1VazvLPu6Pcv4L2P++FrS4uLgXWexn63v8LFy7g7NmzWL16Nfr06SOVX7t27aXiLIym91rb9+BlKa/zmTFjhkr5w4cPNU59UNS5ronyPfjjjz/UWmJ1UdC2q1SpAuDF3rv8xz45ORlbt27F5MmTMW7cOKk8MzNT7z/kygvUt2/frvH1opIT5Tqsra0LvOYx//lf0PdK37590bdvX6SmpuLAgQOYPHkyOnbsiCtXrhT6nr3//vsYOXIkVq9ejenTp+Onn35C586dVVr1zMzMMHLkSIwcORJPnjzBrl278PnnnyM0NBR37twxyEhe5XH59ttvCxyxqk2yY2lpqfH6seL8vSmI8jOoTErzetnvWV3Os7CwMISFhSEzMxPHjh3DzJkz0aNHD/j6+qJx48Y6b/vnn39GixYtsHTpUpXy/APb3NzccOjQISgUigKTNzc3N8jlciQkJBTaOKLr+6rpc6Xtd6ubmxv+/fffAmNRqlKlCtq1a4fFixejXbt22Lx5M6ZOnarSCqsNg9054a233kLr1q0xbdo0PHv2TOU1Dw8PWFlZ4dy5cyrl//vf/4otnnXr1kEIIT2/ffs2jhw5Il006efnh6pVq+Ls2bNqfzlq+xdkQVq1aoWYmBicOnVKpXzt2rWQyWQICQnReZ3Kk1D5Fx2QO3p0xYoVKvUaNWoEKysr6S8ApSNHjqg1/YeEhODixYs4e/asSvmvv/6qVUz5W4qUWrVqhT179qiNzFq7di1sbGwK/FIuiKZ9B4Bly5bptJ68dPmrurA4NL0HL0smk6nt699//11gF0dR57omoaGhMDMzw/Xr1ws8/7VR0Hmm3LY+3juZTAYhhNo6fvjhB8jlcq3Xo42OHTsiKSkJcrlc4zHJ+8dcQed/x44dcf36dbi4uGhch64TM9va2qJdu3aYMGECsrKycPHixULrOzk5oXPnzli7di22bt2KhISEQruqypQpg27dumHIkCF49OjRS10Q/jKaNGmCMmXKICYmpsBzUtlyWtjn19fXV+23Zs+ePWq/SyXBz88Pnp6eapcRxcXFqYyqLIw+zzNLS0s0b94cs2fPBgBpBLmu34eavqPOnTundhlMu3btkJGRUegEwsrLn/Ingfnp433V9ru1Xbt2uHLlilaXNQ0fPhznzp1Dnz59YGpqioEDB2odj5JBWtyUZs+ejaCgICQmJuKNN96QymUyGXr27ImVK1eicuXKqFWrFqKiorROEF5EYmIiunTpgoEDByI5ORmTJ0+GlZUVxo8fL9VZtmwZ2rVrh9DQUERERKBcuXJ49OgRLl26hFOnTuH3339/oW1/+umnWLt2LTp06IBp06bBx8cHf//9N5YsWYLBgwdrvHaqKK1bt4aFhQXef/99jBkzBhkZGVi6dCkeP36sUs/JyQmjR4/GV199hQEDBuDdd9/FnTt3MGXKFLWm6BEjRmDlypXo0KEDvvrqK2lUqbKLpSg1atTAvn37sGXLFnh5ecHe3h5+fn6YPHkytm7dipCQEHzxxRdwdnbGL7/8gr///htz5syRRltpq3r16qhcuTLGjRsHIQScnZ2xZcsWrbqNCosdABYtWoQ+ffrA3Nwcfn5+hSbr2r4HL6tjx45YvXo1qlevjpo1ayI6Ohpz584tsOlem3M9P19fX0ybNg0TJkzAjRs30LZtWzg5OeH+/fuIioqCra2tNAVKYU6ePKlynk2YMAHlypWTLjfQx3vn4OCAN998E3PnzoWrqyt8fX2xf/9+/Pjjj3qffLV79+745Zdf0L59ewwfPhwNGjSAubk5/v33X+zduxdhYWHo0qULgNxzaP369diwYQMqVaoEKysr1KhRAyNGjMCff/6JN998E59++ilq1qwJhUKBuLg47Ny5E6NGjSryerCBAwfC2toaTZo0gZeXFxISEjBz5kw4OjpK1ywVpl+/ftiwYQOGDh2K8uXLq12r06lTJwQGBqJevXpwc3PD7du3sXDhQvj4+KiMkCtJdnZ2+Pbbb9GnTx88evQI3bp1g7u7Ox48eICzZ8/iwYMH0o97YZ/fXr16YdKkSfjiiy/QvHlzxMTE4LvvvtP5e0cfTExMMHXqVHz00Ufo1q0b+vXrhydPnmDq1Knw8vIqtAtR6WXPsy+++AL//vsvWrVqhfLly+PJkydYtGgRzM3N0bx5cwC5805aW1vjl19+gb+/P+zs7FC2bFnp8pb8OnbsiC+//BKTJ09G8+bNERsbi2nTpqFixYoqo43ff/99rFq1CoMGDUJsbCxCQkKgUChw/Phx+Pv7o3v37mjWrBl69eqFr776Cvfv30fHjh1haWmJ06dPw8bGBsOGDQMAvbyv2n63jhgxAhs2bEBYWBjGjRuHBg0aID09Hfv370fHjh1VGl9at26NgIAA7N27V5oCTGc6DWV4QXlHlebXo0cPAUBlVKkQQiQnJ4sBAwYIDw8PYWtrKzp16iRu3bpV4KjSBw8eqCzfp08fYWtrq7a9/CNYlSOYfvrpJ/HJJ58INzc3YWlpKZo1ayZOnjyptvzZs2fFe++9J9zd3YW5ubnw9PQULVu2FN9//71W+1uQ27dvix49eggXFxdhbm4u/Pz8xNy5c6WRqkq6jCrdsmWLqFWrlrCyshLlypUTn332mfjnn3/URoAqFAoxc+ZM4e3tLSwsLETNmjXFli1b1EZGCSFETEyMaN26tbCyshLOzs6if//+4n//+59Wo0rPnDkjmjRpImxsbNRGrJ4/f1506tRJODo6CgsLC1GrVi21EUvK9+r3339XKdc0ClkZp729vXBychLvvvuuiIuLUzt/tB1VKoQQ48ePF2XLlhUmJiYq+1vYe6Lte1DQqNIhQ4aorTP/qKjHjx+L/v37C3d3d2FjYyOaNm0qDh48qPb+6XKu5x9VqrRp0yYREhIiHBwchKWlpfDx8RHdunUTu3btKvTYKY/zzp07Ra9evUSZMmWEtbW1aN++vbh69apKXW3fu4I++0II8e+//4p33nlHODk5CXt7e9G2bVtx4cKFAkeU5f+s6vK9kp2dLebNmye9z3Z2dqJ69erio48+Utm3W7duiTZt2gh7e3sBQOX9fvbsmZg4caLw8/MTFhYWwtHRUdSoUUN8+umnIiEhQapX0DmxZs0aERISIjw8PISFhYUoW7aseO+998S5c+fU3wwN5HK58Pb2LnC03vz580VwcLBwdXUVFhYWokKFCqJ///7i1q1bha5X+dmcO3dugXUKGlWqzedcCCH2798vOnToIJydnYW5ubkoV66c6NChg9ryBX1+MzMzxZgxY4S3t7ewtrYWzZs3F2fOnNH6XFHGm/fzXNCoUk3HIf95LYQQy5cvF1WqVBEWFhaiWrVqYuXKlSIsLEzUqVNH80HM42XPs61bt4p27dqJcuXKCQsLC+Hu7i7at28vDh48qLKddevWierVqwtzc/MCR3AqZWZmitGjR4ty5coJKysrUbduXbFp0yaN33vp6eniiy++EFWrVhUWFhbCxcVFtGzZUhw5ckSqI5fLxddffy0CAwOl/WjcuLHYsmWLyjZf5n0VQvvvVmXd4cOHiwoVKghzc3Ph7u4uOnToIC5fvqy23ilTpggA4tixYwUes8LIhMjTZ0JEVAxWr16Nvn374sSJE1p3qxJRridPnqBatWro3LmzxuvCqXSpV68eZDIZTpw48ULLG7SrlIiIiJ5LSEjA9OnTERISAhcXF9y+fRtff/01nj59yvsZl2IpKSm4cOECtm7diujoaPz1118vvC4mbkRERK8IS0tL3Lp1Cx9//DEePXokDdD6/vvvVa4Fp9Ll1KlTUjI+efLkl7rfOLtKiYiIiEoJg00HQkRERES6YeJGREREVEowcSMiIiIqJTg4QQOFQoF79+7B3t5e69tWERERkWEJIfD06dMi73lamjFx0+DevXvw9vY2dBhERET0Au7cuaPVjd9LIyZuGihvY3Tnzh04ODgYOBoiIiLSRkpKCry9vV/43uGlARM3DZTdow4ODkzciIiISpnX+TKn17MDmIiIiOg1xMSNiIiIqJRg4kZERERUSjBxIyIiIiolmLgRERERlRJM3IiIiIhKCSZuRERERKUEEzciIiKiUoKJGxEREVEpwTsnEBERFUYhBx4cBNLjAWsvwK0ZYGJq6KjISDFxIyIiKsidjUD0cCDt3+dlNuWBoEWAd1fDxUVGi12lREREmtzZCBzsppq0AUDa3dzyOxsNExcZNSZuRERE+SnkuS1tEBpe/K8sekRuPaISxMSNiIgovwcH1VvaVAgg7U5uPaISxMSNiIgov/R4/dYj0hMmbkRERPlZe+m3HpGeMHEjIiLKz61Z7uhRyAqoIANsvHPrEZUgJm5ERET5mZjmTvkBQD15++950ELO50YljokbERGRJt5dgWZ/ADblVMttyueWcx43MgBOwEtERFQQ765AuTDeOYFeGUzciIiICmNiCni0MHQURACYuBEREVEJkisEom4+QuLTDLjbW6FBRWeYmhQ0CITyY+JGREREJWL7hXhM3RKD+OQMqczL0QqTOwWgbSCnVtGGwQcnLFmyBBUrVoSVlRWCgoJw8GDhs1AvXrwY/v7+sLa2hp+fH9auXatWZ+HChfDz84O1tTW8vb3x6aefIiMjQ8PaiIiIqCRsvxCPwT+fUknaACAhOQODfz6F7Rc4mbE2DNritmHDBowYMQJLlixBkyZNsGzZMrRr1w4xMTGoUKGCWv2lS5di/PjxWLFiBerXr4+oqCgMHDgQTk5O6NSpEwDgl19+wbhx47By5UoEBwfjypUriIiIAAB8/fXXJbl7RPQKYLcMkeHJFQJTt8QUeOdXGYCpW2LQOsCTn88iyIQQmo5jiWjYsCHq1q2LpUuXSmX+/v7o3LkzZs6cqVY/ODgYTZo0wdy5c6WyESNG4OTJkzh06BAAYOjQobh06RJ2794t1Rk1ahSioqKKbM1TSklJgaOjI5KTk+Hg4PCiu0dEBsZuGdIHhUJALgTkCgGFEFCI3EREWS79K/6rm6dcqiuelwshIFdoV567vdzy59sRebajqRx5tp8nRgU0rPd5ed7XlOVCqO577n5Ddb0iT938x+S/uunZcjxJyy7yWK8b2AiNK7u88HtlDL/fBmtxy8rKQnR0NMaNG6dS3qZNGxw5ckTjMpmZmbCyslIps7a2RlRUFLKzs2Fubo6mTZvi559/RlRUFBo0aIAbN25g27Zt6NOnT4GxZGZmIjMzU3qekpLyEntGRK8CZbdM/r9Mld0yS3vWLdXJmzbJhEIgz4+27slE3qShuJMJTUmDTslE3m1qmUwUtHz+cio5iU95WVNRDJa4PXz4EHK5HB4eHirlHh4eSEhI0LhMaGgofvjhB3Tu3Bl169ZFdHQ0Vq5ciezsbDx8+BBeXl7o3r07Hjx4gKZNm0IIgZycHAwePFgtQcxr5syZmDp1ql73j4gMI1uuwMOnmZi46UKB3TIAMPr3c7gc/xQKQKdkQlPSoEsykTdpYDLxejCRAaYmMshkMpjKZDA1kUllJjIZTEzylJsApjLVchMTGUxNkFv2X73cckjPVf+F+nplebZZRLmJMj5p27nlsv/q5d1eQeUmsjyx5dme7L/1qmxPJsOFe8kYv/F8kcfS3d6qyDrGzuCjSmUy1b5sIYRamdKkSZOQkJCARo0aQQgBDw8PREREYM6cOTA1zZ0Mcd++fZg+fTqWLFmChg0b4tq1axg+fDi8vLwwadIkjesdP348Ro4cKT1PSUmBt7e3nvaQiF5GVo4Cj1KzkJSaiUepWXiUmoWHz7Lw6L/nSc+y8pRnIiUjR6v1PsvMwcLdV4s5esNR/rCq/ogXnEyYyPIkEXl/3E20Sya0TRoK/nFHvkRGu6Th+XaeJzT5kwZlsqQsf76vyLNfmstl+fZddXu5sRT0m0XPBZR1wDe7ryIhOUPjH1QyAJ6OudegUuEMlri5urrC1NRUrXUtMTFRrRVOydraGitXrsSyZctw//59eHl5Yfny5bC3t4erqyuA3OSuV69eGDBgAACgRo0aSE1NxYcffogJEybAxER9IK2lpSUsLS31vIdEpElGtlxKtJJScxOwpGf//f/Z8zJlUvY0U7tELC+ZDNDm6t3gyi6o7GankjQUlUxokzQUlkyoJAdFJBNFJQ1MJqi0MDWRYXKnAAz++RRkgErypjxTJ3cK4MAELRgscbOwsEBQUBAiIyPRpUsXqTwyMhJhYWGFLmtubo7y5csDANavX4+OHTtKCVlaWppacmZqagrx3zUcRKRf6VlyqTXsefKVKf3/eYKWhaRnmUjNkuu8DVMTGZxsLOBiawEXOws42+b+39nWEs52/5VLr1ni0r0UfPDj8SLXO6xl1Ze6EJqItNc20AtLe9ZVGzDkyQFDOjFoV+nIkSPRq1cv1KtXD40bN8by5csRFxeHQYMGAcjtwrx79640V9uVK1cQFRWFhg0b4vHjx1iwYAEuXLiANWvWSOvs1KkTFixYgDp16khdpZMmTcLbb78tdacSkWZCCKRlydVaw5TPk/J2Uf73PD1b90TMzEQGZ9vcBMzVzlL6v4utxfNE7L9yF1sLOFiZw0SHv8QbVXaBl6MVu2WIXjFtA73QOsCTU/S8BIMmbuHh4UhKSsK0adMQHx+PwMBAbNu2DT4+PgCA+Ph4xMXFSfXlcjnmz5+P2NhYmJubIyQkBEeOHIGvr69UZ+LEiZDJZJg4cSLu3r0LNzc3dOrUCdOnTy/p3SMyOCEEnmXmqLSGPU+6MlVaw5TXkWVkK3TejoWpyfPky+55a5iydSw3QfuvhczWAg5WZsXalcduGaJXl6mJjC3dL8Gg87i9qoxhHhgq3Ks6aasQAk8zc6SWL02tYaqJWBaycnRPxCzNTKTWL2dbS7j+l3w550nKnidjFrCzLN5E7EVxHjci42IMv98GH1VK9KopyR97IQRS0nPwUG2EZGaeZEz1gv1sue5/a1mbm0qtYbndj6qtYS75ui1tLExfyURMV+yWIaLXDVvcNDCGjJ00K2jSVuXPfFGTtioUAsnp2bkX5+e7MF85XUXe1rDHqVnIeYFJuWwtTKXWMJd814Ypy553W1rC2oLXdxLR688Yfr/Z4kb0n6LupQcA4zaex/2UTDxOy5JayPLOL/Y4LRvyF0jE7C3N/kvELPIkXc8TMGUrmTI5szJnIkZEZIyYuBH9J+rmI5XuUU2epGVj8uaLRa7LwcpMGhWpTMZc7FRbw/J2XVqaMREjIqKiMXEjo5eeJcfhaw/x46GbWtWvWd4Rb5R1lC7Ml1rD/kvEnGwsYGGmPtEzERHRy2LiRkYpITkDuy/fx+5LiTh87SEydRh5Ob6dP4eyExGRQTBxI6OgUAhcuJeMXZcSsfvSfVy8l6Lyerky1mhZ3R1/n7uHR2nZGtfBSVuJiMjQmLjRaystKweHryVh96X72HM5EYlPM6XXZDKgjncZtPL3QCt/d/h52EMmk6FJFRcM/vkUAE7aSkRErx4mbvRaiU9Ox+7/WtWOXE9S6QK1tTDFm9Xc0LK6O0Kqu8PVzlJted5Lj4iIXmVM3KhUUygEzt1Nxp5L97HrUiJi4lW7QMs7WeMtfw+0rO6OhpWctRq9yUlbiYjoVcXEjUqdtKwcHLz6EHsuJWJPbCIe5OsCrVvBCa383dGqugeqedi90B0AeC89IiJ6FTFxo1Lh3pN07L78vAs07/037SzN8GY1V7Sq7oEWfm5w0dAFSkRE9Dpg4kavJIVC4Oy/T7DnciJ2XUrEpXxdoN7O1mhV3QNv+XugQUVnzptGRERGgYkbvTJSM//rAr18H3suP8DDZ8+7QE2kLlAPvOXvjiruL9YFSkREVJoxcSODuvskHbsv5U6Ee/R6ErLkql2gzau5oZW/O1r4ucPZ1sKAkRIRERkeEzcqUQqFwJl/n0jJ2uWEpyqvV3C2QSt/d7zl74H6vuwCJSIiyouJG+lMrhA6TZWR2wX6ALsuJWJfbCIePsuSXjORAfV8nNHS3x1v+bujshu7QImIiArCxI10sv1CvNrktF4aJqf993Fa7kS4lxNxLF8XqL1Vni7Qau5wYhcoERGRVpi4kda2X4jH4J9PqdwKCsi9Yfugn09hTGg1PMuUY89l9S5QHxeb/0aBuqN+RWeYm7ILlIiISFdM3EgrcoXA1C0xakkb8PyennN2XJHKTGRAPV9ntKrujlb+HqjsZssuUCIiopfExI20EnXzkUr3aEEaV3ZBeD1vtPBzQxkbdoESERHpExM30kri06KTNgDoXt8bYbXLFXM0RERExokXGpFW3O2t9FqPiIiIdMcWNypStlyBfbGJhdaRAfB0zJ0ahIiIiIoHEzcq1N0n6fhk3WlE335cYB3lkIPJnQIKnc+NiIiIXg4TNyrQrpj7GP3HWTxJy4a9pRlmd6sJExnU5nHz1DCPGxEREekfEzdSk5WjwJztl/HDoZsAgJrlHfHd+3VRwcUGANA6wFOnOycQERGRfjBxIxV3HqVh6LrTOHvnCQCgX5OKGNeuuso9Q01NZGhc2cVAERIRERkvJm4k2X4hAWP+OIuUjBw4WJlh3ru10OYNT0OHRURERP9h4kbIzJFj5rbLWH3kFgCgToUy+Pb9OijvZGPYwIiIiEgFEzcjdzspFUN/PY3zd5MBAB+9WQmjQ/14L1EiIqJXEBM3I/b3uXiM+/McnmbmwMnGHPPfq4WW1T0MHRYREREVgImbEcrIluOrv2Pw87E4AEA9Hyd826MOvBytDRwZERERFYaJm5G58eAZhvx6GpfiUwAAH7eojJGtq8GMXaNERESvPCZuRuR/Z+7i843nkZolh4utBRaE10bzam6GDouIiIi0xMTNCKRnyTF1y0WsP3EHANCokjMWda8DDwfeEJ6IiKg0YeL2mruW+BRDfjmN2PtPIZMBw1pWxfBWVXmnAyIiolKIidtrQK4QGm9B9Wf0v5i46QLSs+VwtbPEou610aSKq6HDJSIiohfExK2U234hXu2m7x4OlqjkaoujNx4BAJpUccHX4bXhbs+uUSIiotKMiVsptv1CPAb/fAoiX/n9lEzcT8mETAaMfKsaPg6pwq5RIiKi1wDngCil5AqBqVti1JK2vJxsLJi0ERERvUaYuJVSUTcfqXSPavIoNQtRNx+VUERERERU3Ji4lVKJTwtP2nStR0RERK8+Jm6llLYDDTgggYiI6PXBxK2UalDRGV6OVijo6jUZAC/H3KlBiIiI6PXAxK2UMjWRYXKnAI2vKZO5yZ0CODCBiIjoNcLErRRrG+iFpT3rIn9u5ulohaU966JtoJdhAiMiIqJiwXncSrngKq5Q/DcnyOx3aqCCs6105wQiIiJ6vTBxK+VuPUwFALjZWyK8fgUDR0NERETFiV2lpdzN/xI3XxcbA0dCRERExc3giduSJUtQsWJFWFlZISgoCAcPHiy0/uLFi+Hv7w9ra2v4+flh7dq1anWePHmCIUOGwMvLC1ZWVvD398e2bduKaxcM6tbDNACAr4utgSMhIiKi4mbQrtINGzZgxIgRWLJkCZo0aYJly5ahXbt2iImJQYUK6t1+S5cuxfjx47FixQrUr18fUVFRGDhwIJycnNCpUycAQFZWFlq3bg13d3f88ccfKF++PO7cuQN7e/uS3r0ScSvpvxY3VyZuRERErzuZEKKw210Wq4YNG6Ju3bpYunSpVObv74/OnTtj5syZavWDg4PRpEkTzJ07VyobMWIETp48iUOHDgEAvv/+e8ydOxeXL1+Gubn5C8WVkpICR0dHJCcnw8HB4YXWUVI6Lz6MM3eeYMkHddG+BkeREhGR8SpNv98vymBdpVlZWYiOjkabNm1Uytu0aYMjR45oXCYzMxNWVqp3ArC2tkZUVBSys7MBAJs3b0bjxo0xZMgQeHh4IDAwEDNmzIBcLi+eHTEwqcWNXaVERESvPYMlbg8fPoRcLoeHh4dKuYeHBxISEjQuExoaih9++AHR0dEQQuDkyZNYuXIlsrOz8fDhQwDAjRs38Mcff0Aul2Pbtm2YOHEi5s+fj+nTpxcYS2ZmJlJSUlQepcGTtCw8SctNWH1dOTiBiIjodWfwwQkymep8Y0IItTKlSZMmoV27dmjUqBHMzc0RFhaGiIgIAICpqSkAQKFQwN3dHcuXL0dQUBC6d++OCRMmqHTH5jdz5kw4OjpKD29vb/3sXDFTjij1cLCEjQVndiEiInrdGSxxc3V1hampqVrrWmJiolornJK1tTVWrlyJtLQ03Lp1C3FxcfD19YW9vT1cXV0BAF5eXqhWrZqUyAG5180lJCQgKytL43rHjx+P5ORk6XHnzh097WXxYjcpERGRcTFY4mZhYYGgoCBERkaqlEdGRiI4OLjQZc3NzVG+fHmYmppi/fr16NixI0xMcnelSZMmuHbtGhQKhVT/ypUr8PLygoWFhcb1WVpawsHBQeVRGtz8byqQihxRSkREZBQM2lU6cuRI/PDDD1i5ciUuXbqETz/9FHFxcRg0aBCA3Jaw3r17S/WvXLmCn3/+GVevXkVUVBS6d++OCxcuYMaMGVKdwYMHIykpCcOHD8eVK1fw999/Y8aMGRgyZEiJ719xU941gVOBEBERGQeDXhgVHh6OpKQkTJs2DfHx8QgMDMS2bdvg4+MDAIiPj0dcXJxUXy6XY/78+YiNjYW5uTlCQkJw5MgR+Pr6SnW8vb2xc+dOfPrpp6hZsybKlSuH4cOHY+zYsSW9e8WOXaVERETGxaDzuL2qSsM8MEII1Jy6E08zcrBjxJvw83w9JxgmIiLSVmn4/X5ZBh9VSi/mUWoWnmbkAAB8eJ9SIiIio8DErZS6lZQ7MKGsoxWszE2LqE1ERESvAyZupRQHJhARERkfnRO3ffv2FUMYpCveXJ6IiMj46Jy4tW3bFpUrV8ZXX31VaiaqfR0p75pQkSNKiYiIjIbOidu9e/cwfPhwbNy4ERUrVkRoaCh+++23Au9KQMWDLW5ERETGR+fEzdnZGZ988glOnTqFkydPws/PD0OGDIGXlxc++eQTnD17tjjipDyEELgl3TWBI0qJiIiMxUsNTqhduzbGjRuHIUOGIDU1FStXrkRQUBCaNWuGixcv6itGyufhsyw8y8yBiQzwdmbiRkREZCxeKHHLzs7GH3/8gfbt28PHxwc7duzAd999h/v37+PmzZvw9vbGu+++q+9Y6T/KbtKyZaxhacapQIiIiIyFzre8GjZsGNatWwcA6NmzJ+bMmYPAwEDpdVtbW8yaNUvlNlSkX9LABF7fRkREZFR0TtxiYmLw7bff4p133oGFhYXGOmXLlsXevXtfOjjSTJrDjSNKiYiIjIrOidvu3buLXqmZGZo3b/5CAVHRlF2lvNUVERGRcdH5GreZM2di5cqVauUrV67E7Nmz9RIUFe6mNKKULW5ERETGROfEbdmyZahevbpa+RtvvIHvv/9eL0FRwYQQuM053IiIiIySzolbQkICvLy81Mrd3NwQHx+vl6CoYIlPM5GWJc+dCsSJXaVERETGROfEzdvbG4cPH1YrP3z4MMqWLauXoKhgyhGl5Z1sYGH2UtPwERERUSmj8+CEAQMGYMSIEcjOzkbLli0B5A5YGDNmDEaNGqX3AEmVNKKU3aRERERGR+fEbcyYMXj06BE+/vhj6f6kVlZWGDt2LMaPH6/3AEnVzSTlzeXZTUpERGRsdE7cZDIZZs+ejUmTJuHSpUuwtrZG1apVYWlpWRzxUT5scSMiIjJeOiduSnZ2dqhfv74+YyEtKG8uz8SNiIjI+LxQ4nbixAn8/vvviIuLk7pLlTZu3KiXwEidQiGkyXcr8q4JRERERkfnYYnr169HkyZNEBMTg7/++gvZ2dmIiYnBnj174OjoWBwx0n/uP81AZo4CZiYylHeyNnQ4REREVMJ0TtxmzJiBr7/+Glu3boWFhQUWLVqES5cu4b333kOFChWKI0b6j3IqEG9nG5iZcioQIiIiY6Pzr//169fRoUMHAIClpSVSU1Mhk8nw6aefYvny5XoPkJ6Trm/jiFIiIiKjpHPi5uzsjKdPnwIAypUrhwsXLgAAnjx5grS0NP1GRypu8VZXRERERk3nwQnNmjVDZGQkatSogffeew/Dhw/Hnj17EBkZiVatWhVHjPQfZVcpby5PRERknHRO3L777jtkZGQAAMaPHw9zc3McOnQIXbt2xaRJk/QeID0nzeHGEaVERERGSafELScnB1u2bEFoaCgAwMTEBGPGjMGYMWOKJTh6TqEQuP0otyuaLW5ERETGSadr3MzMzDB48GBkZmYWVzxUgHvJ6cjKUcDC1ARly3AqECIiImOk8+CEhg0b4vTp08URCxVCOaLU29kapiYyA0dDREREhqDzNW4ff/wxRo0ahX///RdBQUGwtVXttqtZs6begqPnpJvLs5uUiIjIaOmcuIWHhwMAPvnkE6lMJpNBCAGZTAa5XK6/6EjCgQlERESkc+J28+bN4oiDiiAlbmxxIyIiMlo6J24+Pj7FEQcVgV2lREREpHPitnbt2kJf79279wsHQ5rlyBW4899UID683RUREZHR0jlxGz58uMrz7OxspKWlwcLCAjY2NkzcisG9JxnIlgtYmJmgrCOnAiEiIjJWOk8H8vjxY5XHs2fPEBsbi6ZNm2LdunXFEaPRU3aT+jjbwIRTgRARERktnRM3TapWrYpZs2aptcaRfnBgAhEREQF6StwAwNTUFPfu3dPX6igP3lyeiIiIgBe4xm3z5s0qz4UQiI+Px3fffYcmTZroLTB67lYS53AjIiKiF0jcOnfurPJcJpPBzc0NLVu2xPz58/UVF+XxvKuUI0qJiIiMmc6Jm0KhKI44qADZcgX+fZwOgF2lRERExk5v17hR8bj7OB05CgErcxN42FsZOhwiIiIyIJ0Tt27dumHWrFlq5XPnzsW7776rl6DouZt5rm/jVCBERETGTefEbf/+/ejQoYNaedu2bXHgwAG9BEXP8ebyREREpKRz4vbs2TNYWFiolZubmyMlJUUvQdFznMONiIiIlHRO3AIDA7Fhwwa18vXr1yMgIEAvQdFzN5Ny71FakSNKiYiIjJ7Oo0onTZqEd955B9evX0fLli0BALt378a6devw+++/6z1AY8euUiIiIlLSOXF7++23sWnTJsyYMQN//PEHrK2tUbNmTezatQvNmzcvjhiNVlaOAv8+Vra4MXEjIiIydjonbgDQoUMHjQMUSL/uPE6DQgC2FqZws7c0dDhERERkYDpf43bixAkcP35crfz48eM4efKkXoKiXMpuUh8XW8hknAqEiIjI2OmcuA0ZMgR37txRK7979y6GDBmicwBLlixBxYoVYWVlhaCgIBw8eLDQ+osXL4a/vz+sra3h5+eHtWvXFlh3/fr1kMlkarfpKi14c3kiIiLKS+eu0piYGNStW1etvE6dOoiJidFpXRs2bMCIESOwZMkSNGnSBMuWLUO7du0QExODChUqqNVfunQpxo8fjxUrVqB+/fqIiorCwIED4eTkhE6dOqnUvX37NkaPHo1mzZrptoOvEOnm8hxRSkRERHiBFjdLS0vcv39frTw+Ph5mZrrlgQsWLED//v0xYMAA+Pv7Y+HChfD29sbSpUs11v/pp5/w0UcfITw8HJUqVUL37t3Rv39/zJ49W6WeXC7HBx98gKlTp6JSpUo6xfQqufUwd2ACR5QSERER8AKJW+vWrTF+/HgkJydLZU+ePMHnn3+O1q1ba72erKwsREdHo02bNirlbdq0wZEjRzQuk5mZCSsr1ft1WltbIyoqCtnZ2VLZtGnT4Obmhv79+2sVS2ZmJlJSUlQerwJ2lRIREVFeOidu8+fPx507d+Dj44OQkBCEhISgYsWKSEhIwPz587Vez8OHDyGXy+Hh4aFS7uHhgYSEBI3LhIaG4ocffkB0dDSEEDh58iRWrlyJ7OxsPHz4EABw+PBh/Pjjj1ixYoXWscycOROOjo7Sw9vbW+tli0tGthz3ktMB8K4JRERElEvnxK1cuXI4d+4c5syZg4CAAAQFBWHRokU4f/78CyU8+UdLCiEKHEE5adIktGvXDo0aNYK5uTnCwsIQEREBADA1NcXTp0/Rs2dPrFixAq6urlrHoGxBVD40Db4oaXcepUEIwN7SDC626rcYIyIiIuPzQvO42dra4sMPP3ypDbu6usLU1FStdS0xMVGtFU7J2toaK1euxLJly3D//n14eXlh+fLlsLe3h6urK86dO4dbt26pDFRQKBQAADMzM8TGxqJy5cpq67W0tISl5as1T5qym9TH1YZTgRARERGAF0zcgNzRpXFxccjKylIpf/vtt7Va3sLCAkFBQYiMjESXLl2k8sjISISFhRW6rLm5OcqXLw8gd8qPjh07wsTEBNWrV8f58+dV6k6cOBFPnz7FokWLXokuUG1JI0o5MIGIiIj+o3PiduPGDXTp0gXnz5+HTCaDEALA8y5PuVyu9bpGjhyJXr16oV69emjcuDGWL1+OuLg4DBo0CEBuF+bdu3eludquXLmCqKgoNGzYEI8fP8aCBQtw4cIFrFmzBgBgZWWFwMBAlW2UKVMGANTKX3U3H/JWV0RERKRK52vchg8fjooVK+L+/fuwsbHBxYsXceDAAdSrVw/79u3TaV3h4eFYuHAhpk2bhtq1a+PAgQPYtm0bfHx8AOROMRIXFyfVl8vlmD9/PmrVqoXWrVsjIyMDR44cga+vr6678crjzeWJiIgoP5lQNplpydXVFXv27EHNmjXh6OiIqKgo+Pn5Yc+ePRg1ahROnz5dXLGWmJSUFDg6OiI5ORkODg4GiSF45m7cS87An4ODEeTjZJAYiIiISpNX4fe7uOnc4iaXy2FnZwcgN4m7d+8eAMDHxwexsbH6jc5I5U4FkgGAXaVERET0nM7XuAUGBuLcuXOoVKkSGjZsiDlz5sDCwgLLly8v1XcpeJXcTsq9vs3BygxONuYGjoaIiIheFTonbhMnTkRqau71V1999RU6duyIZs2awcXFBRs2bNB7gMYo7x0TOBUIERERKemcuIWGhkr/r1SpEmJiYvDo0SM4OTkxydCT5zeXZzcpERERPffC87jl5ezsrI/V0H84opSIiIg00XlwAhU/3lyeiIiINGHi9gpiVykRERFpwsTtFZOWlYP7KZkAgIrsKiUiIqI8dE7cDhw4gJycHLXynJwcHDhwQC9BGbNb/93qysnGHI6cCoSIiIjy0DlxCwkJwaNHj9TKk5OTERISopegjBm7SYmIiKggOiduQgiN034kJSXB1pbJxsuSBiawm5SIiIjy0Xo6kK5duwIAZDIZIiIiYGlpKb0ml8tx7tw5BAcH6z9CIyNNBcIWNyIiIspH68TN0dERQG6Lm729PaytraXXLCws0KhRIwwcOFD/ERoZdpUSERFRQbRO3FatWgUA8PX1xejRo9ktWkxu/jc4gV2lRERElJ/O17iNGTNG5Rq327dvY+HChdi5c6deAzM2coXA7kv38fBZ7lQg3s7WRSxBRERExkbnxC0sLAxr164FADx58gQNGjTA/PnzERYWhqVLl+o9QGOw/UI8ms7eg/5rTkpl7RYdxPYL8QaMioiIiF41Oidup06dQrNmzQAAf/zxBzw9PXH79m2sXbsW33zzjd4DfN1tvxCPwT+fQnxyhkp5QnIGBv98iskbERERSXRO3NLS0mBvbw8A2LlzJ7p27QoTExM0atQIt2/f1nuArzO5QmDqlhgIDa8py6ZuiYFcoakGERERGRudE7cqVapg06ZNuHPnDnbs2IE2bdoAABITE+Hg4KD3AF9nUTcfqbW05SUAxCdnIOqm+oTHREREZHx0Tty++OILjB49Gr6+vmjQoAEaN24MILf1rU6dOnoP8HWW+LTgpO1F6hEREdHrTevpQJS6deuGpk2bIj4+HrVq1ZLKW7VqhS5duug1uNedu72VXusRERHR603nFjcA8PT0hL29PSIjI5Geng4AqF+/PqpXr67X4F53DSo6w8vRCuo3EMslA+DlaIUGFZ1LMiwiIiJ6RemcuCUlJaFVq1aoVq0a2rdvj/j43FGPAwYMwKhRo/Qe4OvM1ESGyZ0CAEAteVM+n9wpAKYmBaV2REREZEx0Ttw+/fRTmJubIy4uDjY2NlJ5eHg4tm/frtfgjEHbQC8s7VkXno6q3aGejlZY2rMu2gZ6GSgyIiIietXofI3bzp07sWPHDpQvX16lvGrVqpwO5AW1DfRC6wBP1JyyA6lZcsx7txa61CnHljYiIiJSoXOLW2pqqkpLm9LDhw9haWmpl6CMkamJDFlyBQCgSRUXJm1ERESkRufE7c0335RueQUAMpkMCoUCc+fORUhIiF6DMybZcgWy5bkT7dqY69wQSkREREZA5wxh7ty5aNGiBU6ePImsrCyMGTMGFy9exKNHj3D48OHiiNEopGXJpf9bW5gaMBIiIiJ6Venc4hYQEIBz586hQYMGaN26NVJTU9G1a1ecPn0alStXLo4YjUL6f4mbmYkMFmYvNEsLERERveZ0bnGLi4uDt7c3pk6dqvG1ChUq6CUwY5OWlQOArW1ERERUMJ2bdipWrIgHDx6olSclJaFixYp6CcoYKbtKbZi4ERERUQF0TtyEEJDJ1Ec8Pnv2DFZWvDXTi0rPViZuHJhAREREmmmdJYwcORJA7ijSSZMmqUwJIpfLcfz4cdSuXVvvARqL1Mz/ukrN2eJGREREmmmduJ0+fRpAbovb+fPnYWFhIb1mYWGBWrVqYfTo0fqP0Eiks6uUiIiIiqB14rZ3714AQN++fbFo0SI4ODgUW1DGSHmNGwcnEBERUUF0vqBq1apVxRGH0UvLZosbERERFY4Thr0i0v+bDoSDE4iIiKggTNxeEewqJSIioqIwcXtFKAcn2DJxIyIiogIwcXtFPG9xY1cpERERaaZzlrB582aN5TKZDFZWVqhSpQrvoPACeOcEIiIiKorOiVvnzp0hk8kghFApV5bJZDI0bdoUmzZtgpOTk94Cfd2lZysHJzBxIyIiIs107iqNjIxE/fr1ERkZieTkZCQnJyMyMhINGjTA1q1bceDAASQlJXEyXh1JXaW8cwIREREVQOcWt+HDh2P58uUIDg6Wylq1agUrKyt8+OGHuHjxIhYuXIh+/frpNdDX3fOuUl7jRkRERJrp3OJ2/fp1jXdNcHBwwI0bNwAAVatWxcOHD18+OiOSlsWuUiIiIiqczolbUFAQPvvsMzx48EAqe/DgAcaMGYP69esDAK5evYry5cvrL0ojwHnciIiIqCg698v9+OOPCAsLQ/ny5eHt7Q2ZTIa4uDhUqlQJ//vf/wAAz549w6RJk/Qe7OuMN5knIiKiouicuPn5+eHSpUvYsWMHrly5AiEEqlevjtatW8PEJLcBr3PnzvqO87XH6UCIiIioKC90JbxMJkPbtm3Rtm1bfcdjtNI5AS8REREV4YWyhN27d2P37t1ITEyEQqFQeW3lypV6CcyY5MgVyJLnHkcbTgdCREREBdB5cMLUqVPRpk0b7N69Gw8fPsTjx49VHrpasmQJKlasCCsrKwQFBeHgwYOF1l+8eDH8/f1hbW0NPz8/rF27VuX1FStWoFmzZnBycoKTkxPeeustREVF6RxXSUrLlkv/5+AEIiIiKojOLW7ff/89Vq9ejV69er30xjds2IARI0ZgyZIlaNKkCZYtW4Z27dohJiYGFSpUUKu/dOlSjB8/HitWrED9+vURFRWFgQMHwsnJCZ06dQIA7Nu3D++//z6Cg4NhZWWFOXPmoE2bNrh48SLKlSv30jEXB2U3qamJDJZmvH0sERERaSYT+e9dVQQXFxdERUWhcuXKL73xhg0bom7duli6dKlU5u/vj86dO2PmzJlq9YODg9GkSRPMnTtXKhsxYgROnjyJQ4cOadyGXC6Hk5MTvvvuO/Tu3VuruFJSUuDo6Ijk5GSNc9bp282HqQiZtw/2lmY4PzW02LdHRET0Oirp329D0Ll5Z8CAAfj1119fesNZWVmIjo5GmzZtVMrbtGmDI0eOaFwmMzMTVlZWKmXW1taIiopCdna2xmXS0tKQnZ0NZ2fnAmPJzMxESkqKyqMkKSffZTcpERERFUbnrtKMjAwsX74cu3btQs2aNWFubq7y+oIFC7Raz8OHDyGXy+Hh4aFS7uHhgYSEBI3LhIaG4ocffkDnzp1Rt25dREdHY+XKlcjOzsbDhw/h5eWltsy4ceNQrlw5vPXWWwXGMnPmTEydOlWruIsD53AjIiIibeicuJ07dw61a9cGAFy4cEHlNZlMpnMA+ZcRQhS4nkmTJiEhIQGNGjWCEAIeHh6IiIjAnDlzYGqqnvTMmTMH69atw759+9Ra6vIaP348Ro4cKT1PSUmBt7e3zvvyotI4FQgRERFpQedMYe/evXrZsKurK0xNTdVa1xITE9Va4ZSsra2xcuVKLFu2DPfv34eXlxeWL18Oe3t7uLq6qtSdN28eZsyYIbUMFsbS0hKWlpYvt0MvgZPvEhERkTYMNoTRwsICQUFBiIyMVCmPjIxEcHBwocuam5ujfPnyMDU1xfr169GxY0fprg0AMHfuXHz55ZfYvn076tWrVyzx6xNvME9ERETa0KrFrWvXrli9ejUcHBzQtWvXQutu3LhR642PHDkSvXr1Qr169dC4cWMsX74ccXFxGDRoEIDcLsy7d+9Kc7VduXIFUVFRaNiwIR4/fowFCxbgwoULWLNmjbTOOXPmYNKkSfj111/h6+srtejZ2dnBzs5O69hKktRVysl3iYiIqBBaJW6Ojo7SdWeOjo5623h4eDiSkpIwbdo0xMfHIzAwENu2bYOPjw8AID4+HnFxcVJ9uVyO+fPnIzY2Fubm5ggJCcGRI0fg6+sr1VmyZAmysrLQrVs3lW1NnjwZU6ZM0Vvs+sTBCURERKQNnedxMwYlPQ/Mol1X8fWuK3i/QQXM7Fqj2LdHRET0OuI8blQi0rJ5jRsREREVTefE7f79++jVqxfKli0LMzMzmJqaqjxId+wqJSIiIm3oPB1IREQE4uLiMGnSJHh5eb3Q3G2k6vk8bkzciIiIqGA6J26HDh3CwYMHpUl46eUpW9xsOQEvERERFULnrlJvb29wPIN+8V6lREREpA2dE7eFCxdi3LhxuHXrVjGEY5x45wQiIiLShs59c+Hh4UhLS0PlypVhY2OjdpP5R48e6S04Y5GezcSNiIiIiqZz4rZw4cJiCMO4Pb9zAq9xIyIiooLpnCn06dOnOOIwammZnMeNiIiIiqZz4pb3FlSaVKhQ4YWDMVZp7ColIiIiLeicuPn6+hY6d5tcLn+pgIwR53EjIiIibeicuJ0+fVrleXZ2Nk6fPo0FCxZg+vTpegvMWMgVAlk5CgCADedxIyIiokLonCnUqlVLraxevXooW7Ys5s6di65du+olMGOhnMMNYFcpERERFU5vN5mvVq0aTpw4oa/VGQ3lXRNkMsDSTG9vBxEREb2GdG5xS0lJUXkuhEB8fDymTJmCqlWr6i0wYyFNvmtuyvu+EhERUaF0TtzKlCmjlmAIIeDt7Y3169frLTBj8XxgAq9vIyIiosLpnC3s3btX5bmJiQnc3NxQpUoVmJkx+dBVenbuNW62lry+jYiIiAqnc6Ylk8kQHByslqTl5OTgwIEDePPNN/UWnDF4ftcEJm5ERERUOJ2vhg8JCdF4P9Lk5GSEhIToJShjwhvMExERkbZ0TtyEEBovok9KSoKtra1egjIm6VLixm5mIiIiKpzW2YJyfjaZTIaIiAhYWlpKr8nlcpw7dw7BwcH6j/A1x7smEBERkba0TtwcHR0B5La42dvbw9raWnrNwsICjRo1wsCBA/Uf4WtOOQEvu0qJiIioKFonbqtWrQKQe6/S0aNHs1tUT3iNGxEREWlL52vcJk+eDEtLS+zatQvLli3D06dPAQD37t3Ds2fP9B7g6+75qFJe40ZERESF0zlbuH37Ntq2bYu4uDhkZmaidevWsLe3x5w5c5CRkYHvv/++OOJ8baWzq5SIiIi0pHOL2/Dhw1GvXj08fvxY5Tq3Ll26YPfu3XoNzhhwcAIRERFpS+cWt0OHDuHw4cOwsLBQKffx8cHdu3f1FpixSMvmNW5ERESkHZ1b3BQKBeRyuVr5v//+C3t7e70EZUzSOTiBiIiItKRz4ta6dWssXLhQei6TyfDs2TNMnjwZ7du312dsRkE5HQhvMk9ERERF0Tlb+PrrrxESEoKAgABkZGSgR48euHr1KlxdXbFu3briiPG1pmxxs2WLGxERERVB58StbNmyOHPmDNatW4dTp05BoVCgf//++OCDD1QGK5B2ODiBiIiItPVC/XPW1tbo168f+vXrp+94jE4a71VKREREWnqhbOHKlSvYt28fEhMToVAoVF774osv9BKYsUjnqFIiIiLSks6J24oVKzB48GC4urrC09MTMplMek0mkzFx05E0OMGciRsREREVTufE7auvvsL06dMxduzY4ojHqMgVAhnZuS2WbHEjIiKioug8Hcjjx4/x7rvvFkcsRkfZTQrwGjciIiIqms6J27vvvoudO3cWRyxGR9lNKpMBVuY6vxVERERkZHRu5qlSpQomTZqEY8eOoUaNGjA3N1d5/ZNPPtFbcK875Rxu1uamKtcKEhEREWmic+K2fPly2NnZYf/+/di/f7/KazKZjImbDtJ4uysiIiLSgc6J282bN4sjDqPEyXeJiIhIF7ywyoCkG8ybc2ACERERFU2rjGHkyJH48ssvYWtri5EjRxZad8GCBXoJzBg8v8E8W9yIiIioaFolbqdPn0Z2drb0/4LwAnvdKKcDsbVk4kZERERF0ypx27t3L27cuAFHR0fs3bu3uGMyGtI1buwqJSIiIi1ofY1b1apV8eDBA+l5eHg47t+/XyxBGQuOKiUiIiJdaJ24CSFUnm/btg2pqal6D8iYpP93jRsTNyIiItIGR5UaUCqnAyEiIiIdaJ24yWQytcEHHIzwctLZVUpEREQ60PqqeCEEIiIiYGlpCQDIyMjAoEGDYGtrq1Jv48aN+o3wNZYmdZVycAIREREVTeuMoU+fPirPe/bsqfdgjE1annuVEhERERVF68Rt1apVxRLAkiVLMHfuXMTHx+ONN97AwoUL0axZswLrL168GN999x1u3bqFChUqYMKECejdu7dKnT///BOTJk3C9evXUblyZUyfPh1dunQplvhfBrtKiYiISBcGHZywYcMGjBgxAhMmTMDp06fRrFkztGvXDnFxcRrrL126FOPHj8eUKVNw8eJFTJ06FUOGDMGWLVukOkePHkV4eDh69eqFs2fPolevXnjvvfdw/PjxktotrfFepURERKQLmcg/z0cJatiwIerWrYulS5dKZf7+/ujcuTNmzpypVj84OBhNmjTB3LlzpbIRI0bg5MmTOHToEIDc+eVSUlLwzz//SHXatm0LJycnrFu3Tqu4UlJS4OjoiOTkZDg4OLzo7hUpbPFhnL3zBCt610PrAI9i2w4REZExKKnfb0MyWItbVlYWoqOj0aZNG5XyNm3a4MiRIxqXyczMhJWVlUqZtbU1oqKipFtyHT16VG2doaGhBa5Tud6UlBSVR0ngPG5ERESkC4Mlbg8fPoRcLoeHh2pLk4eHBxISEjQuExoaih9++AHR0dEQQuDkyZNYuXIlsrOz8fDhQwBAQkKCTusEgJkzZ8LR0VF6eHt7v+TeaYddpURERKQLg0/Am38uOCFEgfPDTZo0Ce3atUOjRo1gbm6OsLAwREREAABMTZ8nP7qsEwDGjx+P5ORk6XHnzp0X3BvdKAcn2HI6ECIiItKCwRI3V1dXmJqaqrWEJSYmqrWYKVlbW2PlypVIS0vDrVu3EBcXB19fX9jb28PV1RUA4OnpqdM6AcDS0hIODg4qj5LAe5USERGRLgyWuFlYWCAoKAiRkZEq5ZGRkQgODi50WXNzc5QvXx6mpqZYv349OnbsCBOT3F1p3Lix2jp37txZ5DpLmkIhkJ7NrlIiIiLSnkH76EaOHIlevXqhXr16aNy4MZYvX464uDgMGjQIQG4X5t27d7F27VoAwJUrVxAVFYWGDRvi8ePHWLBgAS5cuIA1a9ZI6xw+fDjefPNNzJ49G2FhYfjf//6HXbt2SaNOXxUZOXLp/2xxIyIiIm0YNHELDw9HUlISpk2bhvj4eAQGBmLbtm3w8fEBAMTHx6vM6SaXyzF//nzExsbC3NwcISEhOHLkCHx9faU6wcHBWL9+PSZOnIhJkyahcuXK2LBhAxo2bFjSu1eo1MzniZuVGRM3IiIiKppB53F7VZXEPDBxSWl4c+5eWJub4tKXbYtlG0RERMaE87hRsUnL5hxuREREpBsmbgbCOdyIiIhIV0zcDIQ3mCciIiJdMXEzkOctbpx8l4iIiLTDxM1A0pT3KTVnixsRERFph4mbgbCrlIiIiHTFxM1AODiBiIiIdMXEzUCUt7viDeaJiIhIW0zcDER5jRtb3IiIiEhbTNwMJI3XuBEREZGOmLgZCAcnEBERka6YuBlIKudxIyIiIh0xcTOQ9Czeq5SIiIh0w+YeA+E1bkT0OpPL5cjOzjZ0GPQasrCwgImJ8bY7MXEzEGkeN945gYheI0IIJCQk4MmTJ4YOhV5TJiYmqFixIiwsLAwdikEwcTOQ54MT+BYQ0etDmbS5u7vDxsYGMpnM0CHRa0ShUODevXuIj49HhQoVjPL8YtZgIGnZnMeNiF4vcrlcStpcXFwMHQ69ptzc3HDv3j3k5OTA3Nzc0OGUOOPtJDYwTgdCRK8b5TVtNjY2Bo6EXmfKLlK5XG7gSAyDiZuBcHACEb2ujLH7ikqOsZ9fTNwMQAgh3auUXaVERESkLSZuBpCRrYAQuf/nTeaJiIAWLVpgxIgRhg5DaxEREejcubOhw3gt7Nu3DzKZjCORtcTEzQCUN5gHOB0IERmPiIgIyGQytce1a9ewceNGfPnll3rfXnElV4sWLcLq1auLZd2aHDlyBKampmjbtm2JbZNeTUzcDEB5fZuVuQlMTIy7r56IjEvbtm0RHx+v8qhYsSKcnZ1hb29v6PC05ujoiDJlypTY9lauXIlhw4bh0KFDiIuLK5FtcgLlVxMTNwNI4xxuRGSkLC0t4enpqfIwNTVV6yr19fXFjBkz0K9fP9jb26NChQpYvny5yrru3r2L8PBwODk5wcXFBWFhYbh16xYAYMqUKVizZg3+97//SS17+/bt09gtd+bMGchkMmnZ1atXo0yZMtixYwf8/f1hZ2cnJZxK+VvzWrRogU8++QRjxoyBs7MzPD09MWXKFJV4L1++jKZNm8LKygoBAQHYtWsXZDIZNm3aVOgxS01NxW+//YbBgwejY8eOai19yn36+++/UatWLVhZWaFhw4Y4f/68VEe5T5s2bUK1atVgZWWF1q1b486dO1KdKVOmoHbt2li5ciUqVaoES0tLCCEQFxeHsLAw2NnZwcHBAe+99x7u378vLXf9+nWEhYXBw8MDdnZ2qF+/Pnbt2qUSY2ZmJsaMGQNvb29YWlqiatWq+PHHH1XqREdHo169erCxsUFwcDBiY2MLPS7GiombASi7StlNSkRUsPnz56NevXo4ffo0Pv74YwwePBiXL18GAKSlpSEkJAR2dnY4cOAADh06JCVYWVlZGD16NN577z2VFr7g4GCtt52WloZ58+bhp59+woEDBxAXF4fRo0cXusyaNWtga2uL48ePY86cOZg2bRoiIyMB5E4c27lzZ9jY2OD48eNYvnw5JkyYoFUsGzZsgJ+fH/z8/NCzZ0+sWrUKQnmhdB6fffYZ5s2bhxMnTsDd3R1vv/22SqtZWloapk+fjjVr1uDw4cNISUlB9+7dVdZx7do1/Pbbb/jzzz9x5swZAEDnzp3x6NEj7N+/H5GRkbh+/TrCw8OlZZ49e4b27dtj165dOH36NEJDQ9GpUyeVlsHevXtj/fr1+Oabb3Dp0iV8//33sLOzU9n2hAkTMH/+fJw8eRJmZmbo16+fVsfH6AhSk5ycLACI5OTkYln/4asPhM/YreKt+fuKZf1ERIaQnp4uYmJiRHp6usbX+/TpI0xNTYWtra306NatmxBCiObNm4vhw4dLdX18fETPnj2l5wqFQri7u4ulS5cKIYT48ccfhZ+fn1AoFFKdzMxMYW1tLXbs2CFtLywsTCWGvXv3CgDi8ePHUtnp06cFAHHz5k0hhBCrVq0SAMS1a9ekOosXLxYeHh4q+5J33c2bNxdNmzZV2Vb9+vXF2LFjhRBC/PPPP8LMzEzEx8dLr0dGRgoA4q+//tJ4vJSCg4PFwoULhRBCZGdnC1dXVxEZGam2T+vXr5fKkpKShLW1tdiwYYPKPh07dkyqc+nSJQFAHD9+XAghxOTJk4W5ublITEyU6uzcuVOYmpqKuLg4qezixYsCgIiKiiow5oCAAPHtt98KIYSIjY0VAFRizksZ/65du6Syv//+WwDQeC4Vdp4V9+/3q4AtbgbAOdyIyFiFhITgzJkz0uObb74psG7NmjWl/8tkMnh6eiIxMRFAbrfatWvXYG9vDzs7O9jZ2cHZ2RkZGRm4fv36S8dpY2ODypUrS8+9vLykbWsTb/5lYmNj4e3tDU9PT+n1Bg0aFBlHbGwsoqKipJYxMzMzhIeHY+XKlWp1GzduLP3f2dkZfn5+uHTpklRmZmaGevXqSc+rV6+OMmXKqNTx8fGBm5ub9PzSpUvw9vaGt7e3VBYQEKCyXGpqKsaMGSOV29nZ4fLly1KL25kzZ2BqaormzZsXuq95j5+XlxcAFHnMjREvsjKANM7hRkRGytbWFlWqVNGqbv7bGclkMigUCgC5XY9BQUH45Zdf1JbLm3jkZ2KS214h8nQ1aroIX9O2hYbuSW3jFUK80MSxP/74I3JyclCuXDmpTAgBc3NzPH78GE5OToUun3+bmmLIW2Zra6vyWkFx5y3/7LPPsGPHDsybNw9VqlSBtbU1unXrhqysLACAtbV1EXuZK+/xU65befzoOba4GUD6f9e4cXACEdGLqVu3Lq5evQp3d3dUqVJF5eHo6Agg99ZI+W+LpEzq8g40UF7LVZyqV6+OuLg4lYv6T5w4UegyOTk5WLt2LebPn6/SSnn27Fn4+PioJa3Hjh2T/v/48WNcuXIF1atXV1nfyZMnpeexsbF48uSJSp38AgICEBcXpzKIISYmBsnJyfD39wcAHDx4EBEREejSpQtq1KgBT09PaaAHANSoUQMKhQL79+8vdH9JO0zcDEDZVcoWNyKiF/PBBx/A1dUVYWFhOHjwIG7evIn9+/dj+PDh+PfffwHkjkw9d+4cYmNj8fDhQ2RnZ6NKlSrw9vbGlClTcOXKFfz999+YP39+scfbunVrVK5cGX369MG5c+dw+PBhaXBCQS1xW7duxePHj9G/f38EBgaqPLp166Y2KnPatGnYvXs3Lly4gIiICLi6uqqMfDU3N8ewYcNw/PhxnDp1Cn379kWjRo0K7bJ96623ULNmTXzwwQc4deoUoqKi0Lt3bzRv3lzqdq1SpQo2btwoJZU9evRQaSnz9fVFnz590K9fP2zatAk3b97Evn378Ntvv73o4TRqTNwMQLrGjaNKiYheiI2NDQ4cOIAKFSqga9eu8Pf3R79+/ZCeng4HBwcAwMCBA+Hn54d69erBzc0Nhw8fhrm5OdatW4fLly+jVq1amD17Nr766qtij9fU1BSbNm3Cs2fPUL9+fQwYMAATJ04EAFhZWWlc5scff8Rbb70ltSDm9c477+DMmTM4deqUVDZr1iwMHz4cQUFBiI+Px+bNm6UbsgO5x2zs2LHo0aMHGjduDGtra6xfv77QuJXTlTg5OeHNN9/EW2+9hUqVKmHDhg1Sna+//hpOTk4IDg5Gp06dEBoairp166qsZ+nSpejWrRs+/vhjVK9eHQMHDkRqamrRB47UyERRnfZGKCUlBY6OjkhOTpa+APRp3o5YfLf3Gvo09sHUsEC9r5+IyBAyMjJw8+ZNVKxYscBkhJ47fPgwmjZtimvXrqkMhNDVvn37EBISgsePHxc4KfDq1asxYsSI1+K2UoWdZ8X9+/0q4EVWBvC8q5SHn4jIWPz111+ws7ND1apVce3aNQwfPhxNmjR5qaSNjA8zBwNIz1YOTmBXKRGRsXj69CnGjBmDO3fuwNXVFW+99VaJXF9HrxcmbgbAedyIiIxP79690bt3b72vt0WLFkVOVRIREYGIiAi9b5tKHgcnGADvVUpEREQvgombAaRlsauUiIiIdMfEzQA4jxsRERG9CCZuBpDOa9yIiIjoBTBxMwAOTiAiIqIXwcTNAKSuUnMOTiAiouLn6+uLhQsXSs+Vd0Sg0oeJmwGkc3ACEZHRiIiIgEwmkx4uLi5o27Ytzp07Z7CY4uPj0a5dO4Ntn14cE7cSJoRAWja7SomIjEnbtm0RHx+P+Ph47N69G2ZmZujYsaPB4vH09ISlpaXBtk8vjolbCcvMUUA5TyJHlRIRlTCFHLi/D7i1LvdfhbxENmtpaQlPT094enqidu3aGDt2LO7cuYMHDx4AAMaOHYtq1arBxsYGlSpVwqRJk5CdnS0tf/bsWYSEhMDe3h4ODg4ICgrCyZMnpdePHDmCN998E9bW1vD29sYnn3xS6E3c83aV3rp1CzKZDBs3bkRISAhsbGxQq1YtHD16VGUZXbdBxYOJWwlTXt8GcAJeIqISdWcjsNkX2B0CHOmR++9m39zyEvTs2TP88ssvqFKlClxcXAAA9vb2WL16NWJiYrBo0SKsWLECX3/9tbTMBx98gPLly+PEiROIjo7GuHHjYG5uDgA4f/48QkND0bVrV5w7dw4bNmzAoUOHMHToUJ3imjBhAkaPHo0zZ86gWrVqeP/995GTk6PXbdDLY+ZQwpST71qYmcDURGbgaIiIjMSdjcDBbgDy3Roq7W5uebM/AO+uxbb5rVu3ws7ODgCQmpoKLy8vbN26FSYmue0nEydOlOr6+vpi1KhR2LBhA8aMGQMAiIuLw2effYbq1asDAKpWrSrVnzt3Lnr06IERI0ZIr33zzTdo3rw5li5dCisrK61iHD16NDp06AAAmDp1Kt544w1cu3YN1atX19s26OWxxa2EcQ43IqISppAD0cOhlrQBz8uiRxRrt2lISAjOnDmDM2fO4Pjx42jTpg3atWuH27dvAwD++OMPNG3aFJ6enrCzs8OkSZMQFxcnLT9y5EgMGDAAb731FmbNmoXr169Lr0VHR2P16tWws7OTHqGhoVAoFLh586bWMdasWVP6v5eXFwAgMTFRr9ugl8cWtxKm7Cq1ZTcpEVHJeHAQSPu3kAoCSLuTW8+jRbGEYGtriypVqkjPg4KC4OjoiBUrVqBjx47o3r07pk6ditDQUDg6OmL9+vWYP3++VH/KlCno0aMH/v77b/zzzz+YPHky1q9fjy5dukChUOCjjz7CJ598orbdChUqaB2jsusVyL0GDgAUCoX0rz62QS+P2UMJS/2vq5QDE4iISkh6vH7r6YFMJoOJiQnS09Nx+PBh+Pj4YMKECdLrypa4vKpVq4Zq1arh008/xfvvv49Vq1ahS5cuqFu3Li5evKiSGOpbSWyDtMOu0hLGrlIiohJm7aXfei8gMzMTCQkJSEhIwKVLlzBs2DA8e/YMnTp1QpUqVRAXF4f169fj+vXr+Oabb/DXX39Jy6anp2Po0KHYt28fbt++jcOHD+PEiRPw9/cHkDsi9ejRoxgyZAjOnDmDq1evYvPmzRg2bJje4i+JbZB2DJ64LVmyBBUrVoSVlRWCgoJw8ODBQuv/8ssvqFWrFmxsbODl5YW+ffsiKSlJpc7ChQvh5+cnDVn+9NNPkZGRUZy7obXnd01g4kZEVCLcmgE25QEUNCBMBth459YrJtu3b4eXlxe8vLzQsGFDnDhxAr///jtatGiBsLAwfPrppxg6dChq166NI0eOYNKkSdKypqamSEpKQu/evVGtWjW89957aNeuHaZOnQog99q0/fv34+rVq2jWrBnq1KmDSZMmSdep6UNJbIO0IxNCaLpas0Rs2LABvXr1wpIlS9CkSRMsW7YMP/zwA2JiYjT2mR86dAjNmzfH119/jU6dOuHu3bsYNGgQqlatKv118ssvv6B///5YuXIlgoODceXKFURERCA8PFxlaHVhUlJS4OjoiOTkZDg4OOh1n387cQdj/jyHED83rOrbQK/rJiIypIyMDNy8eVP6Y/yVIo0qBVQHKfyXzBXzqFLSn8LOs+L8/X5VGLTFbcGCBejfvz8GDBgAf39/LFy4EN7e3li6dKnG+seOHYOvry8++eQTVKxYEU2bNsVHH32kMgnh0aNH0aRJE/To0QO+vr5o06YN3n//fZU6hpQm3e6KlxcSEZUY7665yZlNOdVym/JM2qhUMVjilpWVhejoaLRp00alvE2bNjhy5IjGZYKDg/Hvv/9i27ZtEELg/v37+OOPP6R5ZwCgadOmiI6ORlRUFADgxo0b2LZtm0odQ1Le7oqDE4iISph3V+DtW0CrvUDwr7n/vn2TSRuVKgZr9nn48CHkcjk8PDxUyj08PJCQkKBxmeDgYPzyyy8IDw9HRkYGcnJy8Pbbb+Pbb7+V6nTv3h0PHjxA06ZNIYRATk4OBg8ejHHjxhUYS2ZmJjIzM6XnKSkpL7l3BePgBCIiAzIxLbYpP4hKgsEHJyjnilESQqiVKcXExOCTTz7BF198gejoaGzfvh03b97EoEGDpDr79u3D9OnTsWTJEpw6dQobN27E1q1b8eWXXxYYw8yZM+Ho6Cg9vL299bNzGkiDE5i4ERERkY4M1uLm6uoKU1NTtda1xMREtVY4pZkzZ6JJkyb47LPPAOSOcrG1tUWzZs3w1VdfwcvLC5MmTUKvXr0wYMAAAECNGjWQmpqKDz/8EBMmTJBuL5LX+PHjMXLkSOl5SkpKsSVvysTNxpzXuBEREZFuDNbiZmFhgaCgIERGRqqUR0ZGIjg4WOMyaWlpaomXqWluy5VycGxBdYQQKGgAraWlJRwcHFQexSVdGpzAFjciIiLSjUGbfUaOHIlevXqhXr16aNy4MZYvX464uDip63P8+PG4e/cu1q5dCwDo1KkTBg4ciKVLlyI0NBTx8fEYMWIEGjRogLJly0p1FixYgDp16qBhw4a4du0aJk2ahLfffltK8gyJXaVERET0ogyauIWHhyMpKQnTpk1DfHw8AgMDsW3bNvj4+AAA4uPjVW6yGxERgadPn+K7777DqFGjUKZMGbRs2RKzZ8+W6kycOBEymQwTJ07E3bt34ebmhk6dOmH69Oklvn+apGdzcAIRERG9GINOwPuqKs4J/N5ZegTRtx/j+55BaBvoqdd1ExEZ0is9AS+9NjgBL5Wo1Exe40ZEREQvholbCWNXKRGRcYmIiEDnzp1feFmZTAaZTAYzMzNUqFABgwcPxuPHj9Xqpqenw8nJCc7OzkhPT3/JqOlVxcSthHFwAhER6aJt27aIj4/HrVu38MMPP2DLli34+OOP1er9+eefCAwMREBAADZu3GiASKkkMHErYc/vnMB53IiIjN3+/fvRoEEDWFpawsvLC+PGjUNOTo5KHUtLS3h6eqJ8+fJo06YNwsPDsXPnTrV1/fjjj+jZsyd69uyJH3/8saR2gUoYs4cSJITIc5N5trgREb0oIYR06UlJszY3LfAOP7q4e/cu2rdvj4iICKxduxaXL1/GwIEDYWVlhSlTpmhc5saNG9i+fTvMzc1Vyq9fv46jR49i48aNEEJgxIgRuHHjBipVqvTScdKrhYlbCUrPkkPx3xjeC3eT4WpnCVOTl//wExEZm/RsOQK+2GGQbcdMC9VLr8mSJUvg7e2N7777DjKZDNWrV8e9e/cwduxYfPHFF9Jk8lu3boWdnR3kcjkyMjIAAAsWLFBZ18qVK9GuXTs4OTkByO1eXblyJb766quXjpNeLewqLSHbL8QjZN4+6Xn/NSfRdPYebL8Qb7igiIjIYC5duoTGjRurtN41adIEz549w7///iuVhYSE4MyZMzh+/DiGDRuG0NBQDBs2THpdLpdjzZo16Nmzp1TWs2dPrFmzBnK5YVolqfiwxa0EbL8Qj8E/n0L+CfMSkjMw+OdTWNqzLtoGehkkNiKi0sja3BQx00INtm19EEKodbkqp1bNW25ra4sqVaoAAL755huEhIRg6tSp+PLLLwEAO3bswN27dxEeHq6yLrlcjp07d6Jdu3Z6iZdeDUzciplcITB1S4xa0gYAAoAMwNQtMWgd4MluUyIiLclkslI/yCsgIAB//vmnSgJ35MgR2Nvbo1y5cgUuN3nyZLRr1w6DBw9G2bJl8eOPP6J79+6YMGGCSr1Zs2bhxx9/ZOL2mindZ30pEHXzEeKTMwp8XQCIT85A1M1HaFzZpeQCIyKiEpOcnIwzZ86olH344YdYuHAhhg0bhqFDhyI2NhaTJ0/GyJEjpevbNGnRogXeeOMNzJgxA5MnT8aWLVuwefNmBAYGqtTr06cPOnTogAcPHsDNza04dosMgIlbMUt8WnDS9iL1iIio9Nm3bx/q1KmjUtanTx9s27YNn332GWrVqgVnZ2f0798fEydOLHJ9I0eORN++feHm5gZbW1u0atVKrU5ISAjs7e3x008/YeTIkXrbFzIs3qtUA33e6+zo9SS8v+JYkfXWDWzEFjciKtV4r1IqCbxXKRWrBhWd4eVohYKuXpMB8HK0QoOKziUZFhEREZVCTNyKmamJDJM7BQCAWvKmfD65UwAHJhAREVGRmLiVgLaBXljasy48HVWbdD0drTgVCBEREWmNgxNKSNtAL7QO8ETUzUdIfJoBd/vc7lG2tBEREZG2mLiVIFMTGQcgEBER0QtjVykREekVJyug4mTs5xcTNyIi0gtzc3MAQFpamoEjoddZVlYWAMDUVD+3Hitt2FVKRER6YWpqijJlyiAxMREAYGNjo3YvTqKXoVAo8ODBA9jY2MDMzDhTGOPcayIiKhaenp4AICVvRPpmYmKCChUqGO0fBUzciIhIb2QyGby8vODu7o7s7GxDh0OvIQsLi0Lv5fq6Y+JGRER6Z2pqarTXIBEVJ+NNWYmIiIhKGSZuRERERKUEEzciIiKiUoLXuGmgnNwvJSXFwJEQERGRtpS/26/zJL1M3DR4+vQpAMDb29vAkRAREZGunj59CkdHR0OHUSxk4nVOS1+QQqHAvXv3YG9v/0LzxKSkpMDb2xt37tyBg4NDMURYuvB4qOMxUcXjoY7HRBWPhzoeE1XK4xETEwM/P7/XdsoQtrhpYGJigvLly7/0ehwcHPhhyoPHQx2PiSoeD3U8Jqp4PNTxmKgqV67ca5u0ARycQERERFRqMHEjIiIiKiWYuBUDS0tLTJ48GZaWloYO5ZXA46GOx0QVj4c6HhNVPB7qeExUGcvx4OAEIiIiolKCLW5EREREpQQTNyIiIqJSgokbERERUSnBxI2IiIiolGDipmdLlixBxYoVYWVlhaCgIBw8eNDQIb2QAwcOoFOnTihbtixkMhk2bdqk8roQAlOmTEHZsmVhbW2NFi1a4OLFiyp1MjMzMWzYMLi6usLW1hZvv/02/v33X5U6jx8/Rq9eveDo6AhHR0f06tULT548UakTFxeHTp06wdbWFq6urvjkk0+QlZVVHLtdoJkzZ6J+/fqwt7eHu7s7OnfujNjYWJU6xnRMli5dipo1a0oTfzZu3Bj//POP9LoxHQtNZs6cCZlMhhEjRkhlxnZMpkyZAplMpvLw9PSUXje24wEAd+/eRc+ePeHi4gIbGxvUrl0b0dHR0uvGdkx8fX3VzhGZTIYhQ4YAML7joTVBerN+/Xphbm4uVqxYIWJiYsTw4cOFra2tuH37tqFD09m2bdvEhAkTxJ9//ikAiL/++kvl9VmzZgl7e3vx559/ivPnz4vw8HDh5eUlUlJSpDqDBg0S5cqVE5GRkeLUqVMiJCRE1KpVS+Tk5Eh12rZtKwIDA8WRI0fEkSNHRGBgoOjYsaP0ek5OjggMDBQhISHi1KlTIjIyUpQtW1YMHTq02I9BXqGhoWLVqlXiwoUL4syZM6JDhw6iQoUK4tmzZ1IdYzommzdvFn///beIjY0VsbGx4vPPPxfm5ubiwoULRncs8ouKihK+vr6iZs2aYvjw4VK5sR2TyZMnizfeeEPEx8dLj8TEROl1Yzsejx49Ej4+PiIiIkIcP35c3Lx5U+zatUtcu3ZNqmNsxyQxMVHl/IiMjBQAxN69e4UQxnc8tMXETY8aNGggBg0apFJWvXp1MW7cOANFpB/5EzeFQiE8PT3FrFmzpLKMjAzh6Ogovv/+eyGEEE+ePBHm5uZi/fr1Up27d+8KExMTsX37diGEEDExMQKAOHbsmFTn6NGjAoC4fPmyECI3gTQxMRF3796V6qxbt05YWlqK5OTkYtlfbSQmJgoAYv/+/UIIHhMhhHBychI//PCDUR+Lp0+fiqpVq4rIyEjRvHlzKXEzxmMyefJkUatWLY2vGePxGDt2rGjatGmBrxvjMclv+PDhonLlykKhUPB4FIJdpXqSlZWF6OhotGnTRqW8TZs2OHLkiIGiKh43b95EQkKCyr5aWlqiefPm0r5GR0cjOztbpU7ZsmURGBgo1Tl69CgcHR3RsGFDqU6jRo3g6OioUicwMBBly5aV6oSGhiIzM1Oli6GkJScnAwCcnZ0BGPcxkcvlWL9+PVJTU9G4cWOjPhZDhgxBhw4d8NZbb6mUG+sxuXr1KsqWLYuKFSuie/fuuHHjBgDjPB6bN29GvXr18O6778Ld3R116tTBihUrpNeN8ZjklZWVhZ9//hn9+vWDTCYz+uNRGCZuevLw4UPI5XJ4eHiolHt4eCAhIcFAURUP5f4Utq8JCQmwsLCAk5NToXXc3d3V1u/u7q5SJ/92nJycYGFhYbDjKoTAyJEj0bRpUwQGBgIwzmNy/vx52NnZwdLSEoMGDcJff/2FgIAAozwWALB+/XqcOnUKM2fOVHvNGI9Jw4YNsXbtWuzYsQMrVqxAQkICgoODkZSUZJTH48aNG1i6dCmqVq2KHTt2YNCgQfjkk0+wdu1aKU7AuI5JXps2bcKTJ08QEREBgMejMGaGDuB1I5PJVJ4LIdTKXhcvsq/562iq/yJ1StLQoUNx7tw5HDp0SO01Yzomfn5+OHPmDJ48eYI///wTffr0wf79+wuM8XU+Fnfu3MHw4cOxc+dOWFlZFVjPmI5Ju3btpP/XqFEDjRs3RuXKlbFmzRo0atRIY5yv8/FQKBSoV68eZsyYAQCoU6cOLl68iKVLl6J3794Fxvo6H5O8fvzxR7Rr106l1Qsw3uNRGLa46YmrqytMTU3VsvPExES1TL60U44MK2xfPT09kZWVhcePHxda5/79+2rrf/DggUqd/Nt5/PgxsrOzDXJchw0bhs2bN2Pv3r0oX768VG6Mx8TCwgJVqlRBvXr1MHPmTNSqVQuLFi0yymMRHR2NxMREBAUFwczMDGZmZti/fz+++eYbmJmZSbEY0zHJz9bWFjVq1MDVq1eN8hzx8vJCQECASpm/vz/i4uKkOAHjOiZKt2/fxq5duzBgwACpzJiPR1GYuOmJhYUFgoKCEBkZqVIeGRmJ4OBgA0VVPCpWrAhPT0+Vfc3KysL+/fulfQ0KCoK5ublKnfj4eFy4cEGq07hxYyQnJyMqKkqqc/z4cSQnJ6vUuXDhAuLj46U6O3fuhKWlJYKCgop1P/MSQmDo0KHYuHEj9uzZg4oVK6q8bozHJD8hBDIzM43yWLRq1Qrnz5/HmTNnpEe9evXwwQcf4MyZM6hUqZLRHZP8MjMzcenSJXh5eRnlOdKkSRO1KYSuXLkCHx8fAMb9HbJq1Sq4u7ujQ4cOUpkxH48iFf/4B+OhnA7kxx9/FDExMWLEiBHC1tZW3Lp1y9Ch6ezp06fi9OnT4vTp0wKAWLBggTh9+rQ0tcmsWbOEo6Oj2Lhxozh//rx4//33NQ7TLl++vNi1a5c4deqUaNmypcZh2jVr1hRHjx4VR48eFTVq1NA4TLtVq1bi1KlTYteuXaJ8+fIlPkx78ODBwtHRUezbt09l+HpaWppUx5iOyfjx48WBAwfEzZs3xblz58Tnn38uTExMxM6dO43uWBQk76hSIYzvmIwaNUrs27dP3LhxQxw7dkx07NhR2NvbS9+HxnY8oqKihJmZmZg+fbq4evWq+OWXX4SNjY34+eefpTrGdkyEEEIul4sKFSqIsWPHqr1mjMdDG0zc9Gzx4sXCx8dHWFhYiLp160rTRZQ2e/fuFQDUHn369BFC5A5dnzx5svD09BSWlpbizTffFOfPn1dZR3p6uhg6dKhwdnYW1tbWomPHjiIuLk6lTlJSkvjggw+Evb29sLe3Fx988IF4/PixSp3bt2+LDh06CGtra+Hs7CyGDh0qMjIyinP31Wg6FgDEqlWrpDrGdEz69esnnedubm6iVatWUtImhHEdi4LkT9yM7Zgo59wyNzcXZcuWFV27dhUXL16UXje24yGEEFu2bBGBgYHC0tJSVK9eXSxfvlzldWM8Jjt27BAARGxsrNprxng8tCETQgiDNPURERERkU54jRsRERFRKcHEjYiIiKiUYOJGREREVEowcSMiIiIqJZi4EREREZUSTNyIiIiISgkmbkRERESlBBM3Ii3cunULMpkMZ86cMXQoksuXL6NRo0awsrJC7dq19bZebfZ13759kMlkePLkCQBg9erVKFOmTJHrlslk2LRpk17i1Fb+WItbcZ0rvr6+WLhwYaF1DHF8iahkMXGjUiEiIgIymQyzZs1SKd+0aRNkMpmBojKsyZMnw9bWFrGxsdi9e7dBYwkPD8eVK1ek51OmTNGYTMbHx6Ndu3YlGJl29JnweHt7Iz4+HoGBgXpZH708bf+wICoNmLhRqWFlZYXZs2fj8ePHhg5Fb7Kysl542evXr6Np06bw8fGBi4uLHqPSnbW1Ndzd3Yus5+npCUtLyxKISP+ys7O1qmdqagpPT0+YmZkVc0Slx8uc568SuVwOhUJh6DDIyDFxo1LjrbfegqenJ2bOnFlgHU0tPQsXLoSvr6/0PCIiAp07d8aMGTPg4eGBMmXKYOrUqcjJycFnn30GZ2dnlC9fHitXrlRb/+XLlxEcHAwrKyu88cYb2Ldvn8rrMTExaN++Pezs7ODh4YFevXrh4cOH0ustWrTA0KFDMXLkSLi6uqJ169Ya90OhUGDatGkoX748LC0tUbt2bWzfvl16XSaTITo6GtOmTYNMJsOUKVM0rmf79u1o2rQpypQpAxcXF3Ts2BHXr19XqRMVFYU6derAysoK9erVw+nTp9XWs23bNlSrVg3W1tYICQnBrVu3VF7P26KxevVqTJ06FWfPnoVMJoNMJsPq1auluPO2bJ0/fx4tW7aEtbU1XFxc8OGHH+LZs2fS68r3at68efDy8oKLiwuGDBmikkT9/PPPqFevHuzt7eHp6YkePXogMTFR4/HQRHludOnSBTKZTHquPJdWrlyJSpUqwdLSEkKIIo9p/q5SZVft7t27Ua9ePdjY2CA4OBixsbHSMtevX0dYWBg8PDxgZ2eH+vXrY9euXWqxPn36FD169ICdnR3Kli2Lb7/9ttB9u3v3LsLDw+Hk5AQXFxeEhYWpvXd5KWP9+++/UatWLVhZWaFhw4Y4f/68VCcpKQnvv/8+ypcvDxsbG9SoUQPr1q1TWU9B5/mCBQtQo0YN2NrawtvbGx9//LHK+608j7Zu3Qo/Pz/Y2NigW7duSE1NxZo1a+Dr6wsnJycMGzYMcrlcWi4rKwtjxoxBuXLlYGtri4YNG0qfzX379qFv375ITk6Wzkfl56Ww5fLHExAQAEtLS9y+fRv79u1DgwYNYGtrizJlyqBJkya4fft2oe8Fkb4wcaNSw9TUFDNmzMC3336Lf//996XWtWfPHty7dw8HDhzAggULMGXKFHTs2BFOTk44fvw4Bg0ahEGDBuHOnTsqy3322WcYNWoUTp8+jeDgYLz99ttISkoCkNsN2Lx5c9SuXRsnT57E9u3bcf/+fbz33nsq61izZg3MzMxw+PBhLFu2TGN8ixYtwvz58zFv3jycO3cOoaGhePvtt3H16lVpW2+88QZGjRqF+Ph4jB49WuN6UlNTMXLkSJw4cQK7d++GiYkJunTpIrUapKamomPHjvDz80N0dDSmTJmitq47d+6ga9euaN++Pc6cOYMBAwZg3LhxBR7b8PBwjBo1Cm+88Qbi4+MRHx+P8PBwtXppaWlo27YtnJyccOLECfz+++/YtWsXhg4dqlJv7969uH79Ovbu3Ys1a9Zg9erVUiII5P74fvnllzh79iw2bdqEmzdvIiIiosD48jtx4gQAYNWqVYiPj5eeA8C1a9fw22+/4c8//5QSsaKOaUEmTJiA+fPn4+TJkzAzM0O/fv2k1549e4b27dtj165dOH36NEJDQ9GpUyfExcWprGPu3LmoWbMmTp06hfHjx+PTTz9FZGSkxu2lpaUhJCQEdnZ2OHDgAA4dOgQ7Ozu0bdu2yBawzz77DPPmzcOJEyfg7u6Ot99+W0qWMzIyEBQUhK1bt+LChQv48MMP0atXLxw/flxlHZrOcxMTE3zzzTe4cOEC1qxZgz179mDMmDFqcX/zzTdYv349tm/fjn379qFr167Ytm0btm3bhp9++gnLly/HH3/8IS3Tt29fHD58GOvXr8e5c+fw7rvvom3btrh69SqCg4OxcOFCODg4SOej8hwvbLm88cycORM//PADLl68CGdnZ3Tu3BnNmzfHuXPncPToUXz44YdGe8kGGYCBb3JPpJU+ffqIsLAwIYQQjRo1Ev369RNCCPHXX3+JvKfx5MmTRa1atVSW/frrr4WPj4/Kunx8fIRcLpfK/Pz8RLNmzaTnOTk5wtbWVqxbt04IIcTNmzcFADFr1iypTnZ2tihfvryYPXu2EEKISZMmiTZt2qhs+86dOwKAiI2NFUII0bx5c1G7du0i97ds2bJi+vTpKmX169cXH3/8sfS8Vq1aYvLkyUWuK6/ExEQBQJw/f14IIcSyZcuEs7OzSE1NleosXbpUABCnT58WQggxfvx44e/vLxQKhVRn7NixAoB4/PixEEKIVatWCUdHR+l1Te+DEEIAEH/99ZcQQojly5cLJycn8ezZM+n1v//+W5iYmIiEhAQhxPP3KicnR6rz7rvvivDw8AL3MSoqSgAQT58+FUIIsXfvXpVYNckbV959MDc3F4mJiQUuJ4T6MVWeK8rjp9z+rl27VPYTgEhPTy9wvQEBAeLbb7+Vnvv4+Ii2bduq1AkPDxft2rXTuB8//vij8PPzU3nfMjMzhbW1tdixY4fGbSpjXb9+vVSWlJQkrK2txYYNGwqMtX379mLUqFHSc23P899++024uLhIz1etWiUAiGvXrkllH330kbCxsZHeTyGECA0NFR999JEQQohr164JmUwm7t69q7LuVq1aifHjx0vrzXt+6rIcAHHmzBnp9aSkJAFA7Nu3r8j9IyoObHGjUmf27NlYs2YNYmJiXngdb7zxBkxMnp/+Hh4eqFGjhvTc1NQULi4ual1ujRs3lv5vZmaGevXq4dKlSwCA6Oho7N27F3Z2dtKjevXqAKDSlVavXr1CY0tJScG9e/fQpEkTlfImTZpI29LW9evX0aNHD1SqVAkODg6oWLEiAEgtOZcuXUKtWrVgY2OjcR+VdRo1aqTSopC/zotQbtvW1lYqa9KkCRQKhUo34htvvAFTU1PpuZeXl8r7cvr0aYSFhcHHxwf29vZo0aKFyj6+DB8fH7i5uamUFXVMC1KzZk2VfQAg7UdqairGjBmDgIAAlClTBnZ2drh8+bLaOvMf98aNGxd4TkRHR+PatWuwt7eXzkdnZ2dkZGSodZfnl3c7zs7O8PPzk7Yjl8sxffp01KxZEy4uLrCzs8POnTvVYtV0nu/duxetW7dGuXLlYG9vj969eyMpKQmpqalSHRsbG1SuXFl67uHhAV9fX9jZ2amUKY/dqVOnIIRAtWrVVD57+/fvL3Q/tV3OwsJC5b1zdnZGRESE1Cq6aNEixMfHF3o8ifSJV89SqfPmm28iNDQUn3/+uVqXmImJCYQQKmWaLio3NzdXeS6TyTSWaXMhsjKhUSgU6NSpE2bPnq1WR/lDDUAlUdFmvUpCCJ27Yzp16gRvb2+sWLECZcuWhUKhQGBgoNRVlv9YaaJNnRdR2P7kLS/sfUlNTUWbNm3Qpk0b/Pzzz3Bzc0NcXBxCQ0P1ckG8pveqqGNakLz7kfecAXK7Jnfs2IF58+ahSpUqsLa2Rrdu3bTah4KOoUKhQFBQEH755Re11/Ino9pQbmf+/Pn4+uuvsXDhQul6tREjRqjFmv/Y3b59G+3bt8egQYPw5ZdfwtnZGYcOHUL//v1VPqO6fjYVCgVMTU0RHR2tkuADUEn28tN2OWtra7VjvGrVKnzyySfYvn07NmzYgIkTJyIyMhKNGjUqcHtE+sLEjUqlWbNmoXbt2qhWrZpKuZubGxISElSSAn3Op3Xs2DG8+eabAICcnBxER0dL12TVrVsXf/75J3x9fV9qRKGDgwPKli2LQ4cOSdsCgCNHjqBBgwZarycpKQmXLl3CsmXL0KxZMwDAoUOHVOoEBATgp59+Qnp6OqytraV9zF8n/1QZ+evkZ2FhoXLxuCYBAQFYs2YNUlNTpR/5w4cPw8TERO19Lcjly5fx8OFDzJo1C97e3gCAkydParVsXubm5kXGC2h3TF/EwYMHERERgS5dugDIveZN0yCC/Mf92LFjUqtufnXr1sWGDRvg7u4OBwcHneI5duwYKlSoAAB4/Pgxrly5Im3n4MGDCAsLQ8+ePQHkJkBXr16Fv79/oes8efIkcnJyMH/+fKm1+7ffftMpLk3q1KkDuVyOxMRE6T3JT9P5qM1yRW23Tp06GD9+PBo3boxff/2ViRuVCHaVUqlUo0YNfPDBB2qj6lq0aIEHDx5gzpw5uH79OhYvXox//vlHb9tdvHgx/vrrL1y+fBlDhgzB48ePpYvMhwwZgkePHuH9999HVFQUbty4gZ07d6Jfv35aJQV5ffbZZ5g9ezY2bNiA2NhYjBs3DmfOnMHw4cO1XodyJOHy5ctx7do17NmzByNHjlSp06NHD5iYmKB///6IiYnBtm3bMG/ePJU6gwYNwvXr1zFy5EjExsbi119/VRkcoImvry9u3ryJM2fO4OHDh8jMzFSr88EHH8DKygp9+vTBhQsXsHfvXgwbNgy9evWCh4eHVvtYoUIFWFhY4Ntvv8WNGzewefNmfPnll1otmz/e3bt3IyEhodDpZrQ5pi+iSpUq2LhxI86cOYOzZ8+iR48eGlt7Dx8+jDlz5uDKlStYvHgxfv/99wLPiQ8++ACurq4ICwvDwYMHcfPmTezfvx/Dhw8vcnDPtGnTsHv3bly4cAERERFwdXVF586dpVgjIyNx5MgRXLp0CR999BESEhKK3MfKlSsjJydHeq9++uknfP/990UfnCJUq1YNH3zwAXr37o2NGzfi5s2bOHHiBGbPno1t27YByH1/nz17ht27d+Phw4dIS0vTajlNbt68ifHjx+Po0aO4ffs2du7ciStXrhSZuBLpCxM3KrW+/PJLtW48f39/LFmyBIsXL0atWrUQFRVV4IjLFzFr1izMnj0btWrVwsGDB/G///0Prq6uAICyZcvi8OHDkMvlCA0NRWBgIIYPHw5HR0eV6+m08cknn2DUqFEYNWoUatSoge3bt2Pz5s2oWrWq1uswMTHB+vXrER0djcDAQHz66aeYO3euSh07Ozts2bIFMTExqFOnDiZMmKDW1VuhQgX8+eef2LJlC2rVqoXvv/8eM2bMKHTb77zzDtq2bYuQkBC4ubmpTRcB5F7LtGPHDjx69Aj169dHt27d0KpVK3z33Xda76ObmxtWr16N33//HQEBAZg1a5Za4qmN+fPnIzIyEt7e3qhTp06B9bQ5pi/i66+/hpOTE4KDg9GpUyeEhoaibt26avVGjRqF6Oho1KlTB19++SXmz5+P0NBQjeu0sbHBgQMHUKFCBXTt2hX+/v7o168f0tPTi2yBmzVrFoYPH46goCDEx8dj8+bNsLCwAABMmjQJdevWRWhoKFq0aAFPT08pqStM7dq1sWDBAsyePRuBgYH45ZdfCp3aRxerVq1C7969MWrUKPj5+eHtt9/G8ePHpVbY4OBgDBo0COHh4XBzc8OcOXO0Wk4TGxsbXL58Ge+88w6qVauGDz/8EEOHDsVHH32kl30hKopMFNcFLEREVKrs27cPISEhePz4Me80QPSKYosbERERUSnBxI2IiIiolGBXKREREVEpwRY3IiIiolKCiRsRERFRKcHEjYiIiKiUYOJGREREVEowcSMiIiIqJZi4EREREZUSTNyIiIiISgkmbkRERESlBBM3IiIiolLi/13XTyhn29prAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots()\n",
    "\n",
    "baseline_num_parameters = 54700\n",
    "baseline_accuracy = 0.9844327569007874\n",
    "ax.scatter(baseline_num_parameters, baseline_accuracy, c = 'orange')\n",
    "\n",
    "lora_num_parameters = [1000,2100, 4200, 8400, 16700, 33400, 66900]\n",
    "lora_accuracies = [ 0.874717116355896, 0.9314955472946167, 0.9639991521835327, 0.9709936380386353, 0.9765480160713196, 0.9796338081359863, 0.9804567098617554]\n",
    "ax.plot(num_parameters, accuracies)\n",
    "plt.scatter(num_parameters, accuracies)\n",
    "\n",
    "legend1 = ax.legend(['Baseline', 'LoRA'],\n",
    "                    loc=\"lower right\", title=\"Finetuning Approach\")\n",
    "\n",
    "ax.set_xlabel('Number of additional trainable parameters')\n",
    "ax.set_ylabel('Finetuning test accuracy')\n",
    "ax.set_title('Number of additonal trainable parameters vs Finetuning test accuracy')\n",
    "plt.savefig('parameters_vs_accuracy.png')\n",
    "plt.show()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45e3f7af-5ff9-4edc-8f16-edfb17d5be17",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots()\n",
    "\n",
    "ax.scatter(baseline_num_parameters/baseline_num_parameters, baseline_accuracy/baseline_accuracy, c = 'orange')\n",
    "lora_parameters_relative_to_baseline = [i/baseline_num_parameters for i in lora_num_parameters]\n",
    "lora_accuracy_relative_to_baseline = [i/baseline_accuracy for i in lora_accuracies]\n",
    "ax.scatter(lora_parameters_relative_to_baseline, lora_accuracy_relative_to_baseline)\n",
    "ax.plot(lora_parameters_relative_to_baseline, lora_accuracy_relative_to_baseline)\n",
    "\n",
    "\n",
    "\n",
    "legend1 = ax.legend(['Baseline', 'LoRA'],\n",
    "                    loc=\"lower right\", title=\"Finetuning Approach\")\n",
    "\n",
    "\n",
    "ax.set_xlabel('Percent additional training parameters relative to baseline')\n",
    "ax.set_ylabel('Percent test accuracy relative to baseline')\n",
    "ax.set_title('Percent additional training parameters relative to baseline vs Finetuning test accuracy')\n",
    "plt.savefig('rel_parameters_vs_rel_accuracy.png', bbox_inches='tight')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c52a58f6-d245-484d-97ab-c6066f46c1af",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}