From 8b51f95c40bf689cceaf0b16c6f8ea7d79f20d0b Mon Sep 17 00:00:00 2001 From: Jyun-Yu Jiang Date: Sat, 13 Aug 2022 07:21:37 +0000 Subject: [PATCH] Update KDD22 Tutorial Instructions, Session 2, and Session 5 --- tutorials/kdd22/README.md | 27 +- ...ulti-label Classification with PECOS.ipynb | 1458 ++++++++++++++++ ...treme Multi-label Ranking with PECOS.ipynb | 1505 ----------------- .../kdd22/Session 4 Utilities in PECOS.ipynb | 1022 ++++++----- tutorials/kdd22/imgs/pecos_label_matrix.png | Bin 0 -> 73040 bytes tutorials/kdd22/imgs/pecos_pipeline.png | Bin 0 -> 18041 bytes tutorials/kdd22/imgs/pecos_xmc_examples.png | Bin 0 -> 48386 bytes 7 files changed, 1987 insertions(+), 2025 deletions(-) create mode 100644 tutorials/kdd22/Session 2 Extreme Multi-label Classification with PECOS.ipynb delete mode 100644 tutorials/kdd22/Session 2 Extreme Multi-label Ranking with PECOS.ipynb create mode 100644 tutorials/kdd22/imgs/pecos_label_matrix.png create mode 100644 tutorials/kdd22/imgs/pecos_pipeline.png create mode 100644 tutorials/kdd22/imgs/pecos_xmc_examples.png diff --git a/tutorials/kdd22/README.md b/tutorials/kdd22/README.md index c6760bf5..6e6eddfc 100644 --- a/tutorials/kdd22/README.md +++ b/tutorials/kdd22/README.md @@ -15,10 +15,33 @@ By the end of the tutorial, we believe that attendees will be easily capable of |---|---|---| | 8:00 AM - 8:30 AM | Check-in and Environment Setup | | | 8:30 AM - 8:50 AM | Session 1: Introduction to PECOS | | -| 8:50 AM - 9:30 AM | Session 2: Extreme Multi-label Ranking with PECOS | [Notebook](https://github.com/amzn/pecos/blob/mainline/tutorials/kdd22/Session%202%20Extreme%20Multi-label%20Ranking%20with%20PECOS.ipynb) | +| 8:50 AM - 9:30 AM | Session 2: Extreme Multi-label Classification with PECOS | [Notebook](https://github.com/amzn/pecos/blob/mainline/tutorials/kdd22/Session%202%20Extreme%20Multi-label%20Ranking%20with%20PECOS.ipynb) | | 9:30 AM - 10:00 AM | Coffee Break | | | 10:00 AM - 10:30 AM | Session 3: Approximate Nearest Neighbor (ANN) Search in PECOS | [Notebook](https://github.com/amzn/pecos/blob/mainline/tutorials/kdd22/Session%203%20Approximate%20Nearest%20Neighbor%20Search%20in%20PECOS.ipynb) | | 10:30 AM - 11:10 AM | Session 4: Utilities in PECOS | [Notebook](https://github.com/amzn/pecos/blob/mainline/tutorials/kdd22/Session%204%20Utilities%20in%20PECOS.ipynb) | -| 11:10 AM - 11:40 AM | Session 5: XR-Transformer cookbook and Distributed PECOS | [Notebook](https://github.com/amzn/pecos/blob/mainline/tutorials/kdd22/Session%205%20XR-Transformer%20cookbook%20and%20Distributed%20PECOS.ipynb) | +| 11:10 AM - 11:40 AM | Session 5: eXtreme Multi-label Ranking (XMR) with XR-Transformer | [Notebook](https://github.com/amzn/pecos/blob/mainline/tutorials/kdd22/Session%205%20eXtreme%20Multi-label%20Classification%20with%20Transformers.ipynb) | | 11:40 AM - 11:50 AM | Session 6: Research with PECOS | | | 11:50 AM - 12:00 PM | Closing Remarks | | + +## Tutorial Instructions + +### Miniconda Installation +```bash +mkdir -p ~/miniconda3 +wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh +bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3 +rm -rf ~/miniconda3/miniconda.sh +~/miniconda3/bin/conda init bash +~/miniconda3/bin/conda init zsh +``` + +### Tutorial Mateiral Execution +```bash +conda create -n tutorial_env python=3.9 -y +conda activate tutorial_env +python -m pip install libpecos==0.4.0 matplotlib panda requests jupyterlab +mkdir -p ~/pecos_tutorial_playground +cd ~/pecos_tutorial_playground +git clone https://github.com/amzn/pecos +python -m jupyterlab.labapp --ip=0.0.0.0 --port 8888 --no-browser --allow-root --notebook-dir=pecos/tutorials/kdd22 +``` \ No newline at end of file diff --git a/tutorials/kdd22/Session 2 Extreme Multi-label Classification with PECOS.ipynb b/tutorials/kdd22/Session 2 Extreme Multi-label Classification with PECOS.ipynb new file mode 100644 index 00000000..3d098965 --- /dev/null +++ b/tutorials/kdd22/Session 2 Extreme Multi-label Classification with PECOS.ipynb @@ -0,0 +1,1458 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "67e70878", + "metadata": {}, + "source": [ + "# eXtreme Multi-label Classification (XMC) Problem and PECOS\n", + "\n", + "Prediction for Enormous and Correlated Output Spaces (PECOS) is a versatile and modular machine learning framework for solving prediction problems with very large outputs spaces. For a given input instance, we apply PECOS to the eXtreme Multilabel Classification (XMC) problem to find and rank the most relevant items from an enormous but fixed and finite output space. Generally, PECOS trains an XMC model that takes numerical features to rank labels from the enormous output space. PECOS also provides feature extraction functions for text data, such as TF-IDF (this session) and Transformers (Session 5).\n", + "\n", + "

\n", + "\n", + "
\n", + "\n", + "Using PECOS, we can tackle lots of real-world large-scale applications with only few commands or limited programming codes.\n", + "\n", + "

\n", + "\n", + "
\n", + "\n", + "\n", + "In this part of the tutorial, we will use XR-Linear as an example to demonstrate how to use PECOS to tackle real-world problems and understrand the model architecture in PECOS." + ] + }, + { + "cell_type": "markdown", + "id": "11c281dc", + "metadata": {}, + "source": [ + "## Outline in this Session\n", + "\n", + "1. Experimental dataset preparation\n", + "2. Hands-on PECOS in only few commands \n", + "3. Code with the PECOS library\n", + "4. Build your customized PECOS XR-Linear model" + ] + }, + { + "cell_type": "markdown", + "id": "41d87d24", + "metadata": {}, + "source": [ + "## 1. Experimental Dataset Preparation\n", + "\n", + "`eurlex-4k`, `wiki10-31k`, `amazoncat-13k`, `amazon-670k`, `wiki-500k`, and `amazon-3m` are available." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "1073ac9c", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "xmc-base/wiki10-31k/output-items.txt\r\n", + "xmc-base/wiki10-31k/tfidf-attnxml\r\n", + "xmc-base/wiki10-31k/tfidf-attnxml/X.trn.npz\r\n", + "xmc-base/wiki10-31k/tfidf-attnxml/X.tst.npz\r\n", + "xmc-base/wiki10-31k/X.trn.txt\r\n", + "xmc-base/wiki10-31k/X.tst.txt\r\n", + "xmc-base/wiki10-31k/Y.trn.npz\r\n", + "xmc-base/wiki10-31k/Y.trn.txt\r\n", + "xmc-base/wiki10-31k/Y.tst.npz\r\n", + "xmc-base/wiki10-31k/Y.tst.txt\r\n" + ] + } + ], + "source": [ + "DATASET = \"wiki10-31k\"\n", + "! wget -nv -nc https://archive.org/download/pecos-dataset/xmc-base/{DATASET}.tar.gz\n", + "! tar --skip-old-files -zxf {DATASET}.tar.gz \n", + "! find xmc-base/{DATASET}/*" + ] + }, + { + "cell_type": "markdown", + "id": "057fb642", + "metadata": {}, + "source": [ + "### Numerical Feature and Label Format in PECOS\n", + "\n", + "In PECOS, numerical features of instances can be in either a [dense NumPy matrix](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html) or a [Compressed Sparse Row (CSR) matrix](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.html) of shape `(nr_inst, nr_feat)`, where `nr_inst` and `nr_feat` are numbers of instances and features. \n", + "\n", + "Similary, labels of instances can be also presented as a dense or a sparse matrix of shape `(nr_inst, nr_labels)`, where `nr_labels` is the number of labels in the XMC problem. The following figure shows an example of the sparse matrix for label representations of six instances.\n", + "\n", + "

\n", + "\n", + "
\n" + ] + }, + { + "cell_type": "markdown", + "id": "fcf8d41d", + "metadata": {}, + "source": [ + "## 2. Hands-on PECOS in Only Few Commands\n", + "\n", + "PECOS provides convenient command-line interfaces to establish a pipeline from feature extraction to training and inference. Specifically, a PECOS XR-Linear model for text data can be established and evaluated using the following command-line modules without writing any code.\n", + "\n", + "* Text Vectorizer: `pecos.utils.featurization.text.preprocess`\n", + "* XR-Linear Train/Predict/Evaluate: `pecos.xmc.xlinear.train`, `pecos.xmc.xlinear.predict`, `pecos.xmc.xlinear.evaluate`\n", + "\n", + "All of these commands can be supplied with JSON-format configuration files (See Section 4.2.2 and Appendix 2). In this section, we first use the default setting to have a quick hands-on demo.\n", + "\n", + "\n", + "### 2.1. Text Vectorizer\n", + "\n", + "PECOS text vectorizer `pecos.utils.featurization.text.preprocess` extracts TF-IDF features. The options `build` and `run` learn the vectorizer and extract features, respectively. The `--help` argument will list all available command-line options. In the default setting, the vectorizer learns a unigram TF-IDF without any filtering. If you have prepared a JSON-format vectorizer configuration file, the argument `--vectorizer-config-path` can help customize the vectorizer (See Section 4.2.2)." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "6b47e6d2", + "metadata": {}, + "outputs": [], + "source": [ + "! python3 -m pecos.utils.featurization.text.preprocess build \\\n", + " --text-pos 0 \\\n", + " --input-text-path xmc-base/{DATASET}/X.trn.txt \\\n", + " --output-model-folder simplest.{DATASET}.vectorizer \\\n", + " --from-file true" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "870f6b0d", + "metadata": {}, + "outputs": [], + "source": [ + "! python3 -m pecos.utils.featurization.text.preprocess run \\\n", + " --text-pos 0 \\\n", + " --input-preprocessor-folder simplest.{DATASET}.vectorizer \\\n", + " --input-text-path xmc-base/{DATASET}/X.trn.txt \\\n", + " --output-inst-path simplest.{DATASET}.X.trn.npz \\\n", + " --from-file true\n", + "! python3 -m pecos.utils.featurization.text.preprocess run \\\n", + " --text-pos 0 \\\n", + " --input-preprocessor-folder simplest.{DATASET}.vectorizer \\\n", + " --input-text-path xmc-base/{DATASET}/X.tst.txt \\\n", + " --output-inst-path simplest.{DATASET}.X.tst.npz \\\n", + " --from-file true" + ] + }, + { + "cell_type": "markdown", + "id": "a34bb4e2", + "metadata": {}, + "source": [ + "### 2.2. Train, Predict, Evaluate a PECOS Model\n", + "\n", + "With feature and label matrices, the pipeline of training, prediction, and evalution can be easily established with the modules `pecos.xmc.xlinear.train`, `pecos.xmc.xlinear.predict`, and `pecos.xmc.xlinear.evaluate`. The `--help` argument will list all available command-line options. If you have prepared a JSON-format configuration file, the argument `--params-path` can help customize training and prediction procedures (See Appendix 2)." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "b2316bff", + "metadata": {}, + "outputs": [], + "source": [ + "! python3 -m pecos.xmc.xlinear.train \\\n", + " -x simplest.{DATASET}.X.trn.npz \\\n", + " -y xmc-base/{DATASET}/Y.trn.npz \\\n", + " -m simplest.{DATASET}.model \\\n", + " --nr-splits 16 \\\n", + " -t 0.1" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "580ecfc7", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "==== evaluation results ====\r\n", + "prec = 83.89 78.41 72.53 67.68 63.45 59.72 56.48 53.53 50.86 48.39\r\n", + "recall = 4.95 9.21 12.64 15.60 18.11 20.34 22.30 24.07 25.63 27.00\r\n" + ] + } + ], + "source": [ + "! python3 -m pecos.xmc.xlinear.predict \\\n", + " -x simplest.{DATASET}.X.tst.npz \\\n", + " -y xmc-base/{DATASET}/Y.tst.npz \\\n", + " -m simplest.{DATASET}.model \\\n", + " -o simplest.{DATASET}.Y.tst.pred.npz \\\n", + " -b 10 \\\n", + " -k 10" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "5e75b408", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "==== evaluation results ====\r\n", + "prec = 83.89 78.41 72.53 67.68 63.45 59.72 56.48 53.53 50.86 48.39\r\n", + "recall = 4.95 9.21 12.64 15.60 18.11 20.34 22.30 24.07 25.63 27.00\r\n" + ] + } + ], + "source": [ + "! python3 -m pecos.xmc.xlinear.evaluate \\\n", + " -y xmc-base/{DATASET}/Y.tst.npz \\\n", + " -p simplest.{DATASET}.Y.tst.pred.npz \\\n", + " -k 10" + ] + }, + { + "cell_type": "markdown", + "id": "b0c731f5", + "metadata": {}, + "source": [ + "## 3. Code with the PECOS library\n", + "\n", + "PECOS includes the comprehensieve Python library and interfaces so that we can easily utilize PECOS in the code-level with more flexibility.\n", + "\n", + "### 3.1. Loading Features and Labels\n", + "For convenience, PECOS also provides APIs `load_feature_matrix` and `load_label_matrix` for loading features and labels from binary files in arbitary formats.\n", + "Note that for the sparse format, training labels should be loaded as a [Compressed Sparse Column (CSC) matrix](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csc_matrix.html) while testing labels should be loaded as a CSR matrix for the purpose of computational efficiency. " + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "c518d892", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Training features X_trn is a csr matrix of shape (14146, 101938).\n", + "Training labels Y_trn is a csc matrix of shape (14146, 30938).\n", + "Testing features X_tst is a csr matrix of shape (6616, 101938).\n", + "Testing labels Y_tst is a csr matrix of shape (6616, 30938).\n" + ] + } + ], + "source": [ + "import numpy as np\n", + "from pecos.xmc.xlinear.model import XLinearModel\n", + "\n", + "DATASET = \"wiki10-31k\"\n", + "\n", + "X_trn = XLinearModel.load_feature_matrix(f\"xmc-base/{DATASET}/tfidf-attnxml/X.trn.npz\".format())\n", + "Y_trn = XLinearModel.load_label_matrix(f\"xmc-base/{DATASET}/Y.trn.npz\", for_training=True)\n", + "\n", + "X_tst = XLinearModel.load_feature_matrix(f\"xmc-base/{DATASET}/tfidf-attnxml/X.tst.npz\")\n", + "Y_tst = XLinearModel.load_label_matrix(f\"xmc-base/{DATASET}/Y.tst.npz\", for_training=False)\n", + "\n", + "print(f\"Training features X_trn is a {X_trn.getformat()} matrix of shape {X_trn.shape}.\")\n", + "print(f\"Training labels Y_trn is a {Y_trn.getformat()} matrix of shape {Y_trn.shape}.\")\n", + "print(f\"Testing features X_tst is a {X_tst.getformat()} matrix of shape {X_tst.shape}.\")\n", + "print(f\"Testing labels Y_tst is a {Y_tst.getformat()} matrix of shape {Y_tst.shape}.\")" + ] + }, + { + "cell_type": "markdown", + "id": "150fea14", + "metadata": {}, + "source": [ + "### 3.2. Semantic Label Indexing and Cluster Chain in XR-Linear\n", + "\n", + "The first step of training an XR-Linear model is to conduct semantic label indexing and establish the *hierarchial label tree* for resursive training the XR-Linear model and its inference. \n", + "\n", + "

\n", + "\n", + "
\n", + "\n", + "PECOS supports any method for semantic label indexing. In the PECOS library, as a build-in method, we provide Label Representation via Positive Instance Feature Aggregation (PIFA) for semantic label indexing with only the need of positive instances and their features in training data. PECOS can also consider additional label features `Z` of shape `(nr_labels, nr_label_feat)` in either dense or sparse matrix format, where `nr_label_feat` is the number of label features. These representations and features for each label are concatenated or combined as label embedding in `LabelEmbeddingFactory` in PECOS.\n", + "\n", + "To conduct semantic label indexing, PECOS learns an indexer based on label embedding. PECOS currently supports to use the **Hierarchical K-Means** for semantic label indexing with a hyper-parameter `nr_splits` (the number of clusters in each layer, or `B` in [our report](https://arxiv.org/pdf/2010.05878.pdf)), which decides the depth `D` of the hierarchical label tree. " + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "26794215", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "4 layers in the trained hierarchical label tree.\n" + ] + } + ], + "source": [ + "from pecos.xmc import Indexer, LabelEmbeddingFactory\n", + "\n", + "label_feat = LabelEmbeddingFactory.create(Y_trn, X_trn, method=\"pifa\")\n", + "# label_feat = LabelEmbeddingFactory.create(Y_trn, X_trn, Z, method=\"pifa_lf_concat\") # for using label features Z\n", + "\n", + "cluster_chain = Indexer.gen(label_feat, nr_splits=16, indexer_type=\"hierarchicalkmeans\")\n", + "\n", + "print(f\"{len(cluster_chain)} layers in the trained hierarchical label tree.\")" + ] + }, + { + "cell_type": "markdown", + "id": "02ffda21", + "metadata": {}, + "source": [ + "### 3.3. Training XR-Linear Negative Sampling and Sparsification\n", + "\n", + "Negative sampling plays an important role in solving the XMC problem. PECOS currently provides two negative sampling schemes, including Teacher Forcing Negatives (TFN) and Matcher Aware Negatives (MAN). Please refer to [our report](https://arxiv.org/pdf/2010.05878.pdf) for more details about negative sampling schemes.\n", + "\n", + "To reduce model sizes and improve efficiency, PECOS conduct model sparsification with a hyper-parameter `threshold`. The model weights with absolute values smaller than the threshold will be discarded." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "bd3d6527", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Training time: 39.8403 seconds.\n" + ] + } + ], + "source": [ + "import time\n", + "start_time = time.time()\n", + "\n", + "# For negative_sampling_scheme in model training, \"man\" and tfn+man\" are also available.\n", + "xlm = XLinearModel.train(X_trn, Y_trn, C=cluster_chain, threshold=0.1, negative_sampling_scheme=\"tfn\")\n", + "\n", + "training_time = time.time() - start_time\n", + "print(f\"Training time: {training_time:.4f} seconds.\")" + ] + }, + { + "cell_type": "markdown", + "id": "20f5cfa7", + "metadata": {}, + "source": [ + "PECOS supports serializing and loading the trained model into binary on disk with convenient interfaces. Note that model loading with `is_predict_only=True` could lead to faster prediction speed by disabling the flexibility of model modification." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "3d5d468d", + "metadata": {}, + "outputs": [], + "source": [ + "xlm.save(f\"{DATASET}.xlm.model\")\n", + "xlm = XLinearModel.load(f\"{DATASET}.xlm.model\", is_predict_only=False)" + ] + }, + { + "cell_type": "markdown", + "id": "4b6038ec", + "metadata": {}, + "source": [ + "### 3.4. Prediction and Evaluation\n", + "\n", + "As a tree model, the inference method significantly affects the prediction efficiency of XR-Linear in PECOS. As illustrated in the following figure, the prediction process in PECOS employs a beam search with a hyper-parameter `beam_size`. The other hyper-parameter `only_topk` also needs to be decided to limit the predicted most relevant labels for each instance. The `predict` function of the trained model will result in a CSR matrix of shape `(nr_inst, nr_labels)` and exactly `only_topk` non-zero columns for each row (or instance).\n", + "\n", + "
\n", + "
\n", + "
" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "7f851bc1", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Y_pred is a csr matrix of shape (6616, 30938) and 66160 non-zero elements.\n" + ] + } + ], + "source": [ + "Y_pred = xlm.predict(X_tst, beam_size=10, only_topk=10)\n", + "\n", + "print(f\"Y_pred is a {Y_pred.getformat()} matrix of shape {Y_pred.shape} and {Y_pred.nnz} non-zero elements.\")" + ] + }, + { + "cell_type": "markdown", + "id": "fb1ed22c", + "metadata": {}, + "source": [ + "For evaluation, we evaluate the trained model with conventional ranking metrics, including Precision@K and Recall@K. PECOS also provides the evaluation interface for predicted sparse matrices." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "4c57da1a", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "prec = 84.36 78.20 72.67 68.03 63.85 60.23 56.86 53.85 51.07 48.63\n", + "recall = 4.99 9.17 12.65 15.67 18.28 20.56 22.50 24.21 25.74 27.15\n" + ] + } + ], + "source": [ + "from pecos.utils import smat_util\n", + "metrics = smat_util.Metrics.generate(Y_tst, Y_pred, topk=10)\n", + "print(metrics)" + ] + }, + { + "cell_type": "markdown", + "id": "ae9baf18", + "metadata": {}, + "source": [ + "### 3.5. PECOS and One-versus-All (OVA) Model\n", + "\n", + "PECOS also supports to train an OVA model without leveraing clustering hierarchy if needed. \n", + "\n", + "**Training OVA models is time-consuming, we suggest to try the following code offline after the tutorial. Note that training the above XR-Linear model is 26 times faster than training an OVA model using an AWS *i3.4xlarge* instance.**\n", + "\n", + "```python\n", + "import time\n", + "start_time = time.time()\n", + "\n", + "xlm_ova = XLinearModel.train(X_trn, Y_trn, C=None, negative_sampling_scheme=\"tfn\") \n", + "\n", + "training_time_ova = time.time() - start_time\n", + "print(f\"Training time for the OVA model: {training_time_ova:.4f} seconds.\")\n", + "pecos_faster_ratio = training_time_ova / training_time\n", + "print(f\"XR-Linear is {pecos_faster_ratio:.2f} times faster than the OVA model\")\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "1663277d", + "metadata": {}, + "source": [ + "# 4. Customized PECOS Model\n", + "\n", + "Besides pre-defined models in PECOS, such as XR-Linear, it is also convenient for users to customize PECOS for specific purposes and usage. Specifically, we suggest to establishing a model class to wrap fundamental PECOS functions and tailored operations. As a result, the customized model can be easily constructed and consumed for arbitrary data types and feature extractors. \n", + "\n", + "## 4.1. Structure of a Customized PECOS Model\n", + "\n", + "Even though a customized machine learning pipeline can be seperated into several independent scripts, we recommend declaring a customized PECOS model as a **model class** for better re-usability and code maintenance.\n", + "\n", + "A customized PECOS model should at least consist of the following components:\n", + "\n", + "* `preprocessor` or `encoder`: The procedure, which can be a method or a functionable object, pre-processes or encodes an arbitrary input with the designated data format into features. For example, text data and image data can be encoded by BERT and ResNet.\n", + "* `train()`: The training method takes a set of training data with a preprocessor, learns a primitive PECOS model, and returns a PECOS-based customized machine learning model. The training function could be a class method to construct the model object with the learned model and essential components after training.\n", + "* `model`: A primitive PECOS model taking pre-processed features is capable of deriving the predictions for arbitrary testing data. The model weights should be learned by `train()`. \n", + "* `predict()`: The prediction method takes arbitrary testing data and infers the prediction based on the pre-processor and the learned model.\n", + "* `save()`: The saving function serializes the trained model, including model weights and configuration, for further usage.\n", + "* `load()`: The loading function reads the serialized model so that the trained model can be loaded and re-used.\n", + "\n", + "
\n", + "
\n", + "
\n", + "\n", + "\n", + "In this part of the tutorial, we will use the task of *extreme multi-label text classification* as an example to demonstrate how to **customize a PECOS model that can handle text data with either a conventional bag-of-words (BoW) model or a deep learning model as the text encoder for feature extraction**.\n" + ] + }, + { + "cell_type": "markdown", + "id": "c3acc325", + "metadata": {}, + "source": [ + "## 4.2. Example: eXtreme Multi-label Text Classification (XMTC)\n", + "\n", + "The task of extreme multi-label text classification (XMTC) seeks to find relevant labels from an extreme large label collection for a given text input. Many real-world applications can be formulated as XMTC tasks, such as recommendation systems, document tagging, and semantic search. \n", + "\n", + "In this section, we guide through how to establish a customized PECOS model for XMTC tasks. We will walk through (1) PECOS' built-in BOW model for text preprocessing and vectorizing; (2) how to customize a PECOS model; and (3) \n", + "advanced usage of XR-Transformer based on deep learning." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "237164ca", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "text2text_demo/output-labels.txt\r\n", + "text2text_demo/pecos-CustomPECOS-model\r\n", + "text2text_demo/pecos-CustomPECOS-model/preprocessor\r\n", + "text2text_demo/pecos-CustomPECOS-model/preprocessor/0.base\r\n", + "text2text_demo/pecos-CustomPECOS-model/preprocessor/0.base/tokenizer\r\n", + "text2text_demo/pecos-CustomPECOS-model/preprocessor/0.base/tokenizer/vocab.txt\r\n", + "text2text_demo/pecos-CustomPECOS-model/preprocessor/0.base/tokenizer/config.json\r\n", + "text2text_demo/pecos-CustomPECOS-model/preprocessor/0.base/vectorizer\r\n", + "text2text_demo/pecos-CustomPECOS-model/preprocessor/0.base/vectorizer/tfidf-model.txt\r\n", + "text2text_demo/pecos-CustomPECOS-model/preprocessor/0.base/vectorizer/config.json\r\n", + "text2text_demo/pecos-CustomPECOS-model/preprocessor/meta.json\r\n", + "text2text_demo/pecos-CustomPECOS-model/preprocessor/config.json\r\n", + "text2text_demo/pecos-CustomPECOS-model/xlinear_model\r\n", + "text2text_demo/pecos-CustomPECOS-model/xlinear_model/ranker\r\n", + "text2text_demo/pecos-CustomPECOS-model/xlinear_model/ranker/0.model\r\n", + "text2text_demo/pecos-CustomPECOS-model/xlinear_model/ranker/0.model/W.npz\r\n", + "text2text_demo/pecos-CustomPECOS-model/xlinear_model/ranker/0.model/C.npz\r\n", + "text2text_demo/pecos-CustomPECOS-model/xlinear_model/ranker/0.model/param.json\r\n", + "text2text_demo/pecos-CustomPECOS-model/xlinear_model/ranker/param.json\r\n", + "text2text_demo/pecos-CustomPECOS-model/xlinear_model/param.json\r\n", + "text2text_demo/pecos-CustomPECOS-model/output_items.json\r\n", + "text2text_demo/pecos-text2text-model\r\n", + "text2text_demo/pecos-text2text-model/0.base\r\n", + "text2text_demo/pecos-text2text-model/0.base/tokenizer\r\n", + "text2text_demo/pecos-text2text-model/0.base/tokenizer/vocab.txt\r\n", + "text2text_demo/pecos-text2text-model/0.base/tokenizer/config.json\r\n", + "text2text_demo/pecos-text2text-model/0.base/vectorizer\r\n", + "text2text_demo/pecos-text2text-model/0.base/vectorizer/tfidf-model.txt\r\n", + "text2text_demo/pecos-text2text-model/0.base/vectorizer/config.json\r\n", + "text2text_demo/pecos-text2text-model/meta.json\r\n", + "text2text_demo/pecos-text2text-model/config.json\r\n", + "text2text_demo/pecos-text2text-model/2.base\r\n", + "text2text_demo/pecos-text2text-model/2.base/tokenizer\r\n", + "text2text_demo/pecos-text2text-model/2.base/tokenizer/vocab.txt\r\n", + "text2text_demo/pecos-text2text-model/2.base/tokenizer/config.json\r\n", + "text2text_demo/pecos-text2text-model/2.base/vectorizer\r\n", + "text2text_demo/pecos-text2text-model/2.base/vectorizer/tfidf-model.txt\r\n", + "text2text_demo/pecos-text2text-model/2.base/vectorizer/config.json\r\n", + "text2text_demo/pecos-text2text-model/1.base\r\n", + "text2text_demo/pecos-text2text-model/1.base/tokenizer\r\n", + "text2text_demo/pecos-text2text-model/1.base/tokenizer/vocab.txt\r\n", + "text2text_demo/pecos-text2text-model/1.base/tokenizer/config.json\r\n", + "text2text_demo/pecos-text2text-model/1.base/vectorizer\r\n", + "text2text_demo/pecos-text2text-model/1.base/vectorizer/tfidf-model.txt\r\n", + "text2text_demo/pecos-text2text-model/1.base/vectorizer/config.json\r\n", + "text2text_demo/testing-data.txt\r\n", + "text2text_demo/training-data.txt\r\n" + ] + } + ], + "source": [ + "! wget -nv -nc https://archive.org/download/text2text_demo.tar.gz/text2text_demo.tar.gz\n", + "! tar --skip-old-files -zxf text2text_demo.tar.gz\n", + "! find text2text_demo/*" + ] + }, + { + "cell_type": "markdown", + "id": "3b2ec0e6", + "metadata": {}, + "source": [ + "### 4.2.1. Preprocessor: Text Preprocessing and Vectorizing\n", + "\n", + "The preprocessor plays a role of encoding input data into machine readable vector representations. Any encoder that can transform text data into a vector representation can be considered as the preprocessor or encoder of a customized PECOS model for XMTC tasks.\n", + "\n", + "In the PECOS library, we provide [various text vectorizers](https://github.com/amzn/pecos/blob/mainline/pecos/utils/featurization/text/vectorizers.py), such as TF-IDF, hashing, and pretrained transformer, as **built-in preprocessors** to deal with text data. In this tutorial, we will utilize the [n-gram](https://en.wikipedia.org/wiki/N-gram) [TF-IDF](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) model as our preprocessor.\n", + "\n", + "#### Label Space File Format for Built-in Text Preprocessors\n", + "\n", + "Label space is also essential for text preprocessors, especially for understanding the label space size to create the appropriate label matrix. The label IDs start from zero and can be referred to the line numbers and corresponding text descriptions in the label space file." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "3f48f4f7", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Artificial intelligence researchers\r\n", + "Computability theorists\r\n", + "British computer scientists\r\n", + "Machine learning researchers\r\n", + "Turing Award laureates\r\n", + "Deep Learning\r\n" + ] + } + ], + "source": [ + "! cat \"./text2text_demo/output-labels.txt\"" + ] + }, + { + "cell_type": "markdown", + "id": "e0862645", + "metadata": {}, + "source": [ + "#### Data File Format for Built-in Text Preprocessors\n", + "\n", + "PECOS built-in text preprocessors majorly take the files of text data with labels in a tab-separated values (TSV) format. Each line in the TSV file consists of two elements that represent the comma-separated label IDs and the input text of a data instance. " + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "bd5ebfc6", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "0,1,2\tAlan Turing is widely considered to be the father of theoretical computer science and artificial intelligence.\r\n", + "0,2,3\tHinton was co-author of a highly cited paper published in 1986 that popularized the backpropagation algorithm for training multi-layer neural networks.\r\n", + "3,4,5\tHinton received the 2018 Turing Award, together with Yoshua Bengio and Yann LeCun, for their work on artificial intelligence and deep learning.\r\n", + "0,3,5\tYoshua Bengio is a Canadian computer scientist, most noted for his work on artificial neural networks and deep learning.\r\n" + ] + } + ], + "source": [ + "! cat ./text2text_demo/training-data.txt" + ] + }, + { + "cell_type": "markdown", + "id": "566d1eb5", + "metadata": {}, + "source": [ + "The data file format also supports to represent the label relevance for cost-sensitive learning by using double colons to separate a label and its relevance.\n", + "\n", + "

\n", + "0::0.1,1::0.2,2::0.8 <TAB> Alan Turing is widely considered to be the father of theoretical computer science and artificial intelligence.

\n" + ] + }, + { + "cell_type": "markdown", + "id": "4d4e4419", + "metadata": {}, + "source": [ + "#### Training a Text Preprocessor\n", + "\n", + "The preprocessor model `Preprocessor` is defined in `pecos.utils.featurization.text.preprocess`. Given a training text corpus and the configuration dictionary, the class method `Preprocessor.train` will train a corresponding text preprocesssor. Besides, the built-in preprocessors also support serialization with the function `save()` for the re-usability.\n", + "\n", + "With the previously mentioned data and label space file formats, the utility function `Preprocessor.load_data_from_file(input_text_path, output_text_path)` returns a dictionary with three keys:\n", + "\n", + "* `label_matrix`: a `(num_inst, num_labels)` CSR matrix for the labels of each instance.\n", + "* `label_relevance`: `None` or a `(num_inst, num_labels)` CSR matrix for the relevance of each label in cost-sensitive learning if available.\n", + "* `corpus`: a list of string as the text corpus in the input_text_path.\n", + "\n", + "The configuration settings of text preprocessor including the preprocessor type and hyper-parameters should be defined in a dictionary. Specifially, the key `type` defines the preprocessor choice while the key `kwargs` represents the hyper-parameters. In this tutorial, we adopt n-gram TFIDF features containing *word unigrams*, *word bigrams*, and *character trigrams*. Note that each of the n-gram feature can have different hyper-parameters, such as `max_feature` and `max_df`. Users need to properly set max_feature (e.g., hundred of thousands or millions) based on the corpus size and downstream tasks." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "b7f70a8f", + "metadata": {}, + "outputs": [], + "source": [ + "from pecos.utils.featurization.text.preprocess import Preprocessor\n", + "\n", + "input_text_path = \"./text2text_demo/training-data.txt\"\n", + "output_text_path = \"./text2text_demo/output-labels.txt\"\n", + "model_folder = \"./text2text_demo/pecos-text2text-model\"\n", + "\n", + "parsed_result = Preprocessor.load_data_from_file(input_text_path, output_text_path) # Read files\n", + "corpus = parsed_result[\"corpus\"] # Corpus input text: List of strings\n", + "Y = parsed_result[\"label_matrix\"] # Label Matrix: Sparse Matrix\n", + "\n", + "vectorizer_config = {\n", + " \"type\": \"tfidf\",\n", + " \"kwargs\": {\n", + " \"base_vect_configs\": [\n", + " \n", + " {\n", + " \"ngram_range\": [1, 1],\n", + " \"max_df_ratio\": 0.98,\n", + " \"analyzer\": \"word\",\n", + " },\n", + " {\n", + " \"ngram_range\": [2, 2],\n", + " \"max_df_ratio\": 0.98,\n", + " \"analyzer\": \"word\",\n", + " },\n", + " {\n", + " \"ngram_range\": [3, 3],\n", + " \"max_df_ratio\": 0.98,\n", + " \"analyzer\": \"char_wb\",\n", + " },\n", + " ],\n", + " },\n", + " }\n", + "\n", + "preprocessor = Preprocessor.train(corpus, vectorizer_config)\n", + "preprocessor.save(model_folder) " + ] + }, + { + "cell_type": "markdown", + "id": "a0300f8c", + "metadata": {}, + "source": [ + "#### Preprocessing with a Trained Text Preprocessor\n", + "\n", + "The function `predict` of a trained text preprocessor encodes texts in a **text data file** into a CSR matrix of shape `(num_inst, dim)` as numerical vector representations, where `num_inst` is the number of instances in the file; `dim` is the number of feature dimensions." + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "3b182171", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The file consists of 4 instances with {X.shape[1]}-dimensional features in a {X.getformat()} matrix.\n", + "\n", + "Text 0: Alan Turing is widely considered to be the father of theoretical computer science and artificial intelligence.\n", + "Text 1: Hinton was co-author of a highly cited paper published in 1986 that popularized the backpropagation algorithm for training multi-layer neural networks.\n", + "Text 2: Hinton received the 2018 Turing Award, together with Yoshua Bengio and Yann LeCun, for their work on artificial intelligence and deep learning.\n", + "Text 3: Yoshua Bengio is a Canadian computer scientist, most noted for his work on artificial neural networks and deep learning.\n", + "\n", + "The cosine similarity is 0.0076 between text 0 and text 1.\n", + "The cosine similarity is 0.0325 between text 0 and text 2.\n", + "The cosine similarity is 0.0082 between text 1 and text 2.\n", + "The cosine similarity is 0.0366 between text 0 and text 3.\n", + "The cosine similarity is 0.0267 between text 1 and text 3.\n", + "The cosine similarity is 0.0943 between text 2 and text 3.\n" + ] + } + ], + "source": [ + "# Obtaining numerical vectors from text\n", + "X = preprocessor.predict(corpus)\n", + "\n", + "print(f\"The file consists of {X.shape[0]} instances \"\n", + " \"with {X.shape[1]}-dimensional features \"\n", + " \"in a {X.getformat()} matrix.\\n\")\n", + "\n", + "from sklearn.metrics.pairwise import cosine_similarity\n", + "\n", + "sim = cosine_similarity(X)\n", + "\n", + "for i, ti in enumerate(corpus):\n", + " print(f\"Text {i}: {ti}\")\n", + "\n", + "print(\"\")\n", + "for i in range(X.shape[0]):\n", + " for j in range(i):\n", + " print(f\"The cosine similarity is {sim[i][j]:.4f} between text {j} and text {i}.\")" + ] + }, + { + "cell_type": "markdown", + "id": "18fcd09b", + "metadata": {}, + "source": [ + "#### Command-line Interface\n", + "\n", + "The above vectorizer operations can also be achieved by the following commands with a JSON-format configuration file:\n", + "\n", + "```bash\n", + "python3 -m pecos.utils.featurization.text.preprocess build \\\n", + " --text-pos 1 \\\n", + " --input-text-path ./text2text_demo/training-data.txt \\\n", + " --vectorizer-config-path /path/to/vectorizer-config.json \\\n", + " --output-model-folder ./text2text_demo/pecos-text2text-model\n", + "\n", + "python3 -m pecos.utils.featurization.text.preprocess run \\\n", + " --input-preprocessor-folder ./text2text_demo/pecos-text2text-model \\\n", + " --text-pos 1 \\\n", + " --input-text-path ./text2text_demo/training-data.txt \\\n", + " --output-inst-path /path/to/X.npz\n", + " --label-pos 0 \\\n", + " --output-label-path /path/to/Y.npz \\\n", + " --label-text-path ./text2text_demo/output-labels.txt\n", + "```\n", + "\n", + "#### Efficiency of PECOS Built-in TF-IDF Vectorizer\n", + "\n", + "Moreover, the TF-IDF vectorizer in PECOS is implemented in C++ and efficient." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "a3d6f675", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "PECOS TFIDF time: 27.30493s, result shape=(14146, 10858825), nnz=37194670\n" + ] + } + ], + "source": [ + "vectorizer_config = {\n", + " \"type\": \"tfidf\",\n", + " \"kwargs\": {\n", + " \"base_vect_configs\": [ \n", + " {\n", + " \"ngram_range\": [1, 2],\n", + " \"max_df_ratio\": 0.98,\n", + " \"analyzer\": \"word\",\n", + " },\n", + " ],\n", + " },\n", + " }\n", + "\n", + "input_text_path = \"xmc-base/wiki10-31k/X.trn.txt\"\n", + "corpus = Preprocessor.load_data_from_file(input_text_path, text_pos=0)[\"corpus\"]\n", + "\n", + "import time\n", + "start_time = time.time()\n", + "preprocessor = Preprocessor.train(corpus, vectorizer_config)\n", + "X = preprocessor.predict(input_text_path)\n", + "print(f\"PECOS TFIDF time: {time.time() - start_time:.5f}s, result shape={X.shape}, nnz={X.nnz}\")" + ] + }, + { + "cell_type": "markdown", + "id": "e63c62ce", + "metadata": {}, + "source": [ + "As a baseline method, we compare with the [Sklearn TFIDF vectorizer](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html):\n", + "```python\n", + "start_time = time.time()\n", + "preprocessor = Preprocessor.train(\n", + " corpus,\n", + " {\"type\": \"sklearntfidf\", \"kwargs\":{\"ngram_range\": [1, 2], \"max_df\": 0.98}},\n", + ")\n", + "X = preprocessor.predict(corpus)\n", + "print(f\"Sklearn TFIDF time: {time.time() - start_time:.5f}s, result shaepe={X.shape}, nnz={X.nnz}\")\n", + "```\n", + "\n", + "**Training Sklearn TFIDF models is time-consuming, we suggest to try the following code offline after the tutorial. The execution results using an AWS *i3.4xlarge* instance are as follows:**\n", + "```\n", + "Sklearn TFIDF time: 221.40709s, result shaepe=(14146, 7269690), nnz=33505461\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "75f5aaf8", + "metadata": {}, + "source": [ + "### 4.2.2 Customized PECOS Model with TF-IDF Preprocessor\n", + "\n", + "\n", + "After being powered with text preprocessors, following the [aforementioned illustration](#Structure-of-a-Customized-PECOS-Model), we demonstrate an example of declaring a **customized PECOS model class** based on a TF-IDF preprocessor and a XR-Linear model." + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "3893c23b", + "metadata": {}, + "outputs": [], + "source": [ + "import json\n", + "from os import path\n", + "import pathlib\n", + "from pecos.utils.featurization.text.preprocess import Preprocessor\n", + "from pecos.xmc.xlinear.model import XLinearModel\n", + "from pecos.xmc import Indexer, LabelEmbeddingFactory\n", + "from pecos.utils import smat_util\n", + "\n", + "class CustomPECOS:\n", + " def __init__(self, preprocessor=None, xlinear_model=None, output_items=None):\n", + " self.preprocessor = preprocessor\n", + " self.xlinear_model = xlinear_model\n", + " self.output_items = output_items\n", + " \n", + " @classmethod\n", + " def train(cls, input_text_path, output_text_path):\n", + " \"\"\"Train a CustomPECOS model\n", + " \n", + " Args: \n", + " input_text_path (str): Text input file name. \n", + " output_text_path (str): The file path for output text items.\n", + " vectorizer_config (str): Json_format string for vectorizer config (default None). e.g. {\"type\": \"tfidf\", \"kwargs\": {}}\n", + " \n", + " Returns:\n", + " A CustomPECOS object\n", + " \"\"\"\n", + " # Obtain X_text, Y\n", + " parsed_result = Preprocessor.load_data_from_file(input_text_path, output_text_path)\n", + " Y = parsed_result[\"label_matrix\"]\n", + " corpus = parsed_result[\"corpus\"]\n", + "\n", + " # Train TF-IDF vectorizer\n", + " preprocessor = Preprocessor.train(corpus, {\"type\": \"tfidf\", \"kwargs\":{}}) \n", + " X = preprocessor.predict(corpus) \n", + " \n", + " # Train a XR-Linear model with TF-IDF features\n", + " label_feat = LabelEmbeddingFactory.create(Y, X, method=\"pifa\")\n", + " cluster_chain = Indexer.gen(label_feat)\n", + " xlinear_model = XLinearModel.train(X, Y, C=cluster_chain)\n", + " \n", + " # Load output items\n", + " with open(output_text_path, \"r\", encoding=\"utf-8\") as f:\n", + " output_items = [q.strip() for q in f]\n", + " \n", + " return cls(preprocessor, xlinear_model, output_items)\n", + " \n", + " def predict(self, corpus):\n", + " \"\"\"Predict labels for given inputs\n", + " \n", + " Args:\n", + " corpus (list of strings): input strings.\n", + " Returns:\n", + " csr_matrix: predicted label matrix (num_samples x num_labels)\n", + " \"\"\"\n", + " X = self.preprocessor.predict(corpus)\n", + " Y_pred = self.xlinear_model.predict(X)\n", + " return smat_util.sorted_csr(Y_pred)\n", + "\n", + " def save(self, model_folder):\n", + " \"\"\"Save the CustomPECOS model\n", + "\n", + " Args:\n", + " model_folder (str): folder name to save\n", + " \"\"\"\n", + " self.preprocessor.save(f\"{model_folder}/preprocessor\")\n", + " self.xlinear_model.save(f\"{model_folder}/xlinear_model\")\n", + " with open(f\"{model_folder}/output_items.json\", \"w\", encoding=\"utf-8\") as fp:\n", + " json.dump(self.output_items, fp)\n", + "\n", + " @classmethod\n", + " def load(cls, model_folder):\n", + " \"\"\"Load the CustomPECOS model\n", + "\n", + " Args:\n", + " model_folder (str): folder name to load\n", + " Returns:\n", + " CustomPECOS\n", + " \"\"\"\n", + " preprocessor = Preprocessor.load(f\"{model_folder}/preprocessor\")\n", + " xlinear_model = XLinearModel.load(f\"{model_folder}/xlinear_model\")\n", + " with open(f\"{model_folder}/output_items.json\", \"r\", encoding=\"utf-8\") as fin:\n", + " output_items = json.load(fin)\n", + " return cls(preprocessor, xlinear_model, output_items)" + ] + }, + { + "cell_type": "markdown", + "id": "fcdbb2c6", + "metadata": {}, + "source": [ + "### 4.2.3. Operating the Customized PECOS Model\n", + "\n", + "With a well-declared model class, the customized PECOS model can be modularized and very convenient to use." + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "24134357", + "metadata": {}, + "outputs": [], + "source": [ + "# Declare the path for model serialization and preprocessor configuration.\n", + "model_folder = \"./text2text_demo/pecos-CustomPECOS-model\"\n", + "\n", + "# Train and save the trained model\n", + "input_text_path = \"./text2text_demo/training-data.txt\"\n", + "output_text_path = \"./text2text_demo/output-labels.txt\"\n", + "model = CustomPECOS.train(input_text_path, output_text_path)\n", + "model.save(model_folder)\n", + "\n", + "# Load the trained model and predict\n", + "model = model.load(model_folder)\n", + "testing_text_path = \"./text2text_demo/testing-data.txt\"\n", + "Y_pred = model.predict(testing_text_path)" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "31efd9ac", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Text Input: {text}\n", + "Score {pred_score:.4f}: {pred_label}\n", + "Score {pred_score:.4f}: {pred_label}\n", + "Score {pred_score:.4f}: {pred_label}\n", + "Score {pred_score:.4f}: {pred_label}\n", + "Score {pred_score:.4f}: {pred_label}\n", + "Score {pred_score:.4f}: {pred_label}\n" + ] + } + ], + "source": [ + "test_texts = Preprocessor.load_data_from_file(testing_text_path, output_text_path)[\"corpus\"]\n", + "\n", + "for i, text in enumerate(test_texts):\n", + " print(\"Text Input: {text}\")\n", + " for j in range(Y_pred.indptr[i], Y_pred.indptr[i + 1]):\n", + " pred_label = model.output_items[Y_pred.indices[j]]\n", + " pred_score = Y_pred.data[j]\n", + " print(\"Score {pred_score:.4f}: {pred_label}\")" + ] + }, + { + "cell_type": "markdown", + "id": "32908310", + "metadata": {}, + "source": [ + "## Appedix 1: Model Parameters in PECOS Implementation\n", + "\n", + "### A1.1. Cluster Chain in PECOS Implementation\n", + "\n", + "Specifically, PECOS trains a *cluster_chain* of `D` matching matrices `C[d]`, where `C[d]` is a CSC matrix of shape `(L[d], K[d])`; `L[d]` and `K[d]` are the numbers of labels and clusters in the layer `d`. Note that the clusters of a layer would be the labels of the next layer. The labels of the last layer `L[D - 1]` would be the labels of the overall XMC problem `nr_labels`." + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "6b0cb55e", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "4 layers in the trained hierarchical label tree with C[d] as:\n", + "cluster_chain[0] is a csc matrix of shape (2, 1)\n", + "cluster_chain[1] is a csc matrix of shape (32, 2)\n", + "cluster_chain[2] is a csc matrix of shape (512, 32)\n", + "cluster_chain[3] is a csc matrix of shape (30938, 512)\n" + ] + } + ], + "source": [ + "print(f\"{len(cluster_chain)} layers in the trained hierarchical label tree with C[d] as:\")\n", + "for d, C in enumerate(cluster_chain):\n", + " print(f\"cluster_chain[{d}] is a {C.getformat()} matrix of shape {C.shape}\")" + ] + }, + { + "cell_type": "markdown", + "id": "790b21dc", + "metadata": {}, + "source": [ + "### A2.2. Model Weights in PECOS Implementation\n", + "\n", + "Model weights in an XR-Linear model are also accessible as `model_chain` for analysis and computations. For the i-th layer in the hierarchy, the model weights of matchers/rankers are available as a CSC matrix of shape `(nr_feat + 1, L[i])`, which concatenates weights for features and the bias term. " + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "9e101f6b", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "model_chain[0].W is a csc matrix of shape (101939, 2)\n", + "model_chain[1].W is a csc matrix of shape (101939, 32)\n", + "model_chain[2].W is a csc matrix of shape (101939, 512)\n", + "model_chain[3].W is a csc matrix of shape (101939, 30938)\n" + ] + } + ], + "source": [ + "for d, m in enumerate(xlm.model.model_chain):\n", + " print(f\"model_chain[{d}].W is a {m.W.getformat()} matrix of shape {m.W.shape}\")" + ] + }, + { + "cell_type": "markdown", + "id": "73a599a0", + "metadata": {}, + "source": [ + "## Appendix 2: Customized Parameters and Advanced Training Options\n", + "\n", + "PECOS also supports using customized parameters and several advanced training options, such as different solvers and cost-sensitive learning.\n", + "\n", + "### A2.1. Customized Parameters\n", + "\n", + "The parameters for either of indexing, training, and inference can be easily customized by feeding a dictionary into the corresponding parameter class and its constructor:\n", + "\n", + "* Semantic Indexing (Hierarchical K-Means): `HierarchicalKMeans.TrainParams.from_dict(dict)`\n", + "* Training: `XLinearModel.TrainParams.from_dict(dict)`\n", + "* Inference: `XLinearModel.PredParams.from_dict(dict)`\n", + "\n", + "Although most of the parameters can be also passed by `kwargs` of Python methods, **we encourage to use the dictionary to designate the parameters because it is easier to manage, modularize, and store parameters in certain formats like JSON.**\n", + "\n", + "For XR-Linear models, the default values and skeleton of the parameters can be revealed and generated by the following command:" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "1ddc9bfa", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\r\n", + " \"train_params\": {\r\n", + " \"__meta__\": {\r\n", + " \"class_fullname\": \"pecos.xmc.xlinear.model###XLinearModel.TrainParams\"\r\n", + " },\r\n", + " \"mode\": \"full-model\",\r\n", + " \"ranker_level\": 1,\r\n", + " \"nr_splits\": 16,\r\n", + " \"min_codes\": null,\r\n", + " \"shallow\": false,\r\n", + " \"rel_mode\": \"disable\",\r\n", + " \"rel_norm\": \"no-norm\",\r\n", + " \"hlm_args\": {\r\n", + " \"__meta__\": {\r\n", + " \"class_fullname\": \"pecos.xmc.base###HierarchicalMLModel.TrainParams\"\r\n", + " },\r\n", + " \"neg_mining_chain\": \"tfn\",\r\n", + " \"model_chain\": {\r\n", + " \"__meta__\": {\r\n", + " \"class_fullname\": \"pecos.xmc.base###MLModel.TrainParams\"\r\n", + " },\r\n", + " \"threshold\": 0.1,\r\n", + " \"max_nonzeros_per_label\": null,\r\n", + " \"solver_type\": \"L2R_L2LOSS_SVC_DUAL\",\r\n", + " \"Cp\": 1.0,\r\n", + " \"Cn\": 1.0,\r\n", + " \"max_iter\": 100,\r\n", + " \"eps\": 0.1,\r\n", + " \"bias\": 1.0,\r\n", + " \"threads\": -1,\r\n", + " \"verbose\": 0,\r\n", + " \"newton_eps\": 0.01\r\n", + " }\r\n", + " }\r\n", + " },\r\n", + " \"pred_params\": {\r\n", + " \"__meta__\": {\r\n", + " \"class_fullname\": \"pecos.xmc.xlinear.model###XLinearModel.PredParams\"\r\n", + " },\r\n", + " \"hlm_args\": {\r\n", + " \"__meta__\": {\r\n", + " \"class_fullname\": \"pecos.xmc.base###HierarchicalMLModel.PredParams\"\r\n", + " },\r\n", + " \"model_chain\": {\r\n", + " \"__meta__\": {\r\n", + " \"class_fullname\": \"pecos.xmc.base###MLModel.PredParams\"\r\n", + " },\r\n", + " \"only_topk\": 20,\r\n", + " \"post_processor\": \"l3-hinge\"\r\n", + " }\r\n", + " }\r\n", + " },\r\n", + " \"indexer_params\": {\r\n", + " \"__meta__\": {\r\n", + " \"class_fullname\": \"pecos.xmc.base###HierarchicalKMeans.TrainParams\"\r\n", + " },\r\n", + " \"nr_splits\": 16,\r\n", + " \"min_codes\": null,\r\n", + " \"max_leaf_size\": 100,\r\n", + " \"imbalanced_ratio\": 0.0,\r\n", + " \"imbalanced_depth\": 100,\r\n", + " \"spherical\": true,\r\n", + " \"seed\": 0,\r\n", + " \"kmeans_max_iter\": 20,\r\n", + " \"threads\": -1\r\n", + " }\r\n", + "}\r\n" + ] + } + ], + "source": [ + "! python3 -m pecos.xmc.xlinear.train --generate-params-skeleton" + ] + }, + { + "cell_type": "markdown", + "id": "35472517", + "metadata": {}, + "source": [ + "### A2.2. Training Parameters for Hierarchial Models in XR-Linear\n", + "\n", + "Hierarchical models could have different parameters over layers. To have customized parameters for the hierarchical model, `hlm_args` needs to be designated in the parameter dictionary. The values of `model_chain` and `neg_mining_chain` in `hlm_args` can be **a single dictionary** of general parameters for all layers or **a list of dictinoaries** for specific parameters of individual layers.\n", + "\n", + "#### General Parameters for All Layers\n", + "\n", + "```\n", + "train_params_l1 = XLinearModel.TrainParams.from_dict(\n", + " {\n", + " ...\n", + " \"hlm_args\": {\n", + " ...\n", + " \"neg_mining_chain\": \"tfn\", # Negative sampling scheme for all layers\n", + " \"model_chain\":{...}, # Parameters for all layers\n", + " }\n", + " ...\n", + " })\n", + "```\n", + "\n", + "#### Specific Parameters of Individual Layers\n", + "\n", + "```\n", + "train_params_l1 = XLinearModel.TrainParams.from_dict(\n", + " {\n", + " ...\n", + " \"hlm_args\": {\n", + " ...\n", + " \"neg_mining_chain\": [\n", + " \"tfn\", # Negative sampling scheme for layer-0\n", + " \"tfn\", # Negative sampling scheme for layer-1\n", + " \"tfn+man\", # Negative sampling scheme for layer-2\n", + " ...\n", + " ],\n", + " \"model_chain\": [\n", + " {...}, # Parameters for layer-0\n", + " {...}, # Parameters for layer-1\n", + " {...}, # Parameters for layer-2\n", + " ...\n", + " ],\n", + " }\n", + " ...\n", + " })\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "81f86b8a", + "metadata": {}, + "source": [ + "### A2.3. Variety of Solvers\n", + "\n", + "The solver for optimization can be adjusted by the argument `solver_type` in the `train` function. PECOS currently provides the following solvers for training each matcher/ranker:\n", + "\n", + "* \"L2R_L2LOSS_SVC_DUAL\" (default): L2-regularized L2-loss Dual SVM\n", + "* \"L2R_L1LOSS_SVC_DUAL\": : L2-regularized L1-loss Dual SVM\n", + "* \"L2R_LR_DUAL\": L2-reguarlized Logistic Regression" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "8f26ee42", + "metadata": {}, + "outputs": [], + "source": [ + "xlm_l1_kwargs = XLinearModel.train(\n", + " X_trn, Y_trn,\n", + " C=cluster_chain,\n", + " threshold=0.1,\n", + " negative_sampling_scheme=\"tfn\",\n", + " solver_type=\"L2R_L1LOSS_SVC_DUAL\")" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "id": "197926a7", + "metadata": {}, + "outputs": [], + "source": [ + "train_params_l1 = XLinearModel.TrainParams.from_dict(\n", + " {\n", + " \"hlm_args\": {\n", + " \"threshold\": 0.1,\n", + " \"neg_mining_chain\": \"tfn\",\n", + " \"model_chain\":{\n", + " \"solver_type\": \"L2R_L1LOSS_SVC_DUAL\",\n", + " },\n", + " }\n", + " }\n", + ")\n", + "\n", + "xlm_l1_dict = XLinearModel.train(\n", + " X_trn, Y_trn,\n", + " C=cluster_chain,\n", + " train_params=train_params_l1)" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "id": "eddf91a6", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Evaluation Metrics with L2R_L1LOSS_SVC_DUAL (by method kwargs)\n", + "prec = 83.65 77.27 72.01 67.67 63.91 60.53 57.33 54.53 52.03 49.66\n", + "recall = 4.94 9.06 12.53 15.59 18.26 20.64 22.67 24.52 26.23 27.72\n", + "\n", + "Evaluation Metrics with L2R_L1LOSS_SVC_DUAL (by dictionary)\n", + "prec = 83.65 77.27 72.01 67.67 63.91 60.53 57.33 54.53 52.03 49.66\n", + "recall = 4.94 9.06 12.53 15.59 18.26 20.64 22.67 24.52 26.23 27.72\n" + ] + } + ], + "source": [ + "Y_pred_l1_kwargs = xlm_l1_kwargs.predict(X_tst, beam_size=10, only_topk=10)\n", + "Y_pred_l1_dict = xlm_l1_dict.predict(X_tst, beam_size=10, only_topk=10)\n", + "metrics_l1_kwargs = smat_util.Metrics.generate(Y_tst, Y_pred_l1_kwargs, topk=10)\n", + "metrics_l1_dict = smat_util.Metrics.generate(Y_tst, Y_pred_l1_dict, topk=10)\n", + "\n", + "print(\"Evaluation Metrics with L2R_L1LOSS_SVC_DUAL (by method kwargs)\")\n", + "print(metrics_l1_kwargs)\n", + "\n", + "print(\"\\nEvaluation Metrics with L2R_L1LOSS_SVC_DUAL (by dictionary)\")\n", + "print(metrics_l1_dict)" + ] + }, + { + "cell_type": "markdown", + "id": "9f481ed3", + "metadata": {}, + "source": [ + "## Appendix 3: Cost-sensitive Learning\n", + "\n", + "PECOS supports to adjust the cost of each training instance. To enable cost-sensitive learning, we need to provide a **relevance matrix** `R_trn` with the same shape to the label matrix `Y_trn` for the argument `R`. When `R` is `None` (default), cost-sensitive learning is disable. \n", + "\n", + "Since PECOS models are usually hierarhical, costs for upper layers also need to be decided as the cost-sensitive learning mode by the argument `rel_mode`. Currently, PECOS supports the following cost-sensitive learning modes:\n", + "\n", + "* `\"disable\"` (default): The cost-sensitive learning is disable.\n", + "* `\"induce\"`: Induce the costs into upper layers by the clustering chain.\n", + "* `\"ranker-only\"`: Only apply cost-sensitive learning to the model in the last ranker layer without induction.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "id": "382277f3", + "metadata": {}, + "outputs": [], + "source": [ + "# An exmaple of using training label frequency scores as costs. \n", + "import copy\n", + "from sklearn.preprocessing import normalize\n", + "\n", + "R_trn = copy.deepcopy(Y_trn)\n", + "\n", + "# Training parameters for cost-sensitive learning.\n", + "train_params_cost = XLinearModel.TrainParams.from_dict(\n", + " {\n", + " \"rel_mode\": \"induce\",\n", + " \"rel_norm\": \"l1\",\n", + " \"hlm_args\": {\n", + " \"neg_mining_chain\": \"tfn\",\n", + " \"model_chain\":\n", + " [\n", + " {\n", + " \"threshold\": 0.1,\n", + " \"Cp\": 1.0,\n", + " \"Cn\": 1.0,\n", + " },\n", + " {\n", + " \"threshold\": 0.1,\n", + " \"Cp\": 8.0,\n", + " \"Cn\": 1.0,\n", + " },\n", + " {\n", + " \"threshold\": 0.1,\n", + " \"Cp\": 4.0,\n", + " \"Cn\": 1.0,\n", + " },\n", + " {\n", + " \"threshold\": 0.1,\n", + " \"Cp\": 4.0,\n", + " \"Cn\": 1.0,\n", + " },\n", + " ],\n", + " }\n", + " })\n", + " \n", + "# Cost-sensitive learning.\n", + "xlm_cost = XLinearModel.train(\n", + " X_trn, Y_trn,\n", + " C=cluster_chain,\n", + " R=R_trn,\n", + " train_params=train_params_cost)" + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "id": "559c15cb", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Evaluation Metrics with Cost-sensitive Learning\n", + "prec = 84.93 80.15 74.50 69.32 64.73 61.05 57.68 54.52 51.70 49.10\n", + "recall = 5.01 9.42 13.00 15.99 18.52 20.83 22.85 24.54 26.09 27.43\n", + "\n", + "Original Evaluation Metrics\n", + "prec = 84.36 78.20 72.67 68.03 63.85 60.23 56.86 53.85 51.07 48.63\n", + "recall = 4.99 9.17 12.65 15.67 18.28 20.56 22.50 24.21 25.74 27.15\n" + ] + } + ], + "source": [ + "Y_pred_cost = xlm_cost.predict(X_tst, beam_size=10, only_topk=10)\n", + "metrics_cost = smat_util.Metrics.generate(Y_tst, Y_pred_cost, topk=10)\n", + "print(\"Evaluation Metrics with Cost-sensitive Learning\")\n", + "print(metrics_cost)\n", + "print(\"\\nOriginal Evaluation Metrics\")\n", + "print(metrics)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.10" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/tutorials/kdd22/Session 2 Extreme Multi-label Ranking with PECOS.ipynb b/tutorials/kdd22/Session 2 Extreme Multi-label Ranking with PECOS.ipynb deleted file mode 100644 index fdb579f2..00000000 --- a/tutorials/kdd22/Session 2 Extreme Multi-label Ranking with PECOS.ipynb +++ /dev/null @@ -1,1505 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "67e70878", - "metadata": {}, - "source": [ - "# eXtreme Multi-label Ranking (XMR) Problem and PECOS\n", - "\n", - "Prediction for Enormous and Correlated Output Spaces (PECOS) is a versatile and modular machine learning framework for solving prediction problems with very large outputs spaces. For a given input instance, we apply PECOS to the eXtreme Multilabel Ranking (XMR) problem to find and rank the most relevant items from an enormous but fixed and finite output space.\n", - "\n", - "

\n", - "\n", - "As shown in the above figure, to address the XMR problem, PECOS conceptually consists of three stages, including semantic label indexing, machine-learned matching, and ranking. In this part of the tutorial, we will use XR-Linear as an example to demonstrate how to use PECOS to tackle real-world problems and understrand the model architecture in PECOS.\n", - "\n", - "### Install PECOS through Python PIP" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "6d9fa78b", - "metadata": {}, - "outputs": [], - "source": [ - "! pip install libpecos" - ] - }, - { - "cell_type": "markdown", - "id": "41d87d24", - "metadata": {}, - "source": [ - "## Experimental Dataset\n", - "\n", - "`eurlex-4k`, `wiki10-31k`, `amazoncat-13k`, `amazon-670k`, `wiki-500k`, and `amazon-3m` are available." - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "1073ac9c", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "2022-07-14 08:54:02 URL:https://ia802308.us.archive.org/21/items/pecos-dataset/xmc-base/wiki10-31k.tar.gz [162277861/162277861] -> \"wiki10-31k.tar.gz\" [1]\n", - "xmc-base/wiki10-31k/output-items.txt\n", - "xmc-base/wiki10-31k/tfidf-attnxml\n", - "xmc-base/wiki10-31k/tfidf-attnxml/X.trn.npz\n", - "xmc-base/wiki10-31k/tfidf-attnxml/X.tst.npz\n", - "xmc-base/wiki10-31k/X.trn.txt\n", - "xmc-base/wiki10-31k/X.tst.txt\n", - "xmc-base/wiki10-31k/Y.trn.npz\n", - "xmc-base/wiki10-31k/Y.trn.txt\n", - "xmc-base/wiki10-31k/Y.tst.npz\n", - "xmc-base/wiki10-31k/Y.tst.txt\n" - ] - } - ], - "source": [ - "DATASET = \"wiki10-31k\"\n", - "! wget -nv -nc https://archive.org/download/pecos-dataset/xmc-base/{DATASET}.tar.gz\n", - "! tar --skip-old-files -zxf {DATASET}.tar.gz \n", - "! find xmc-base/{DATASET}/*" - ] - }, - { - "cell_type": "markdown", - "id": "73f0fa78", - "metadata": {}, - "source": [ - "### Analyze Sparse Features and Label Space" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "d680e1e0", - "metadata": {}, - "outputs": [], - "source": [ - "import numpy as np\n", - "import scipy.sparse as smat\n", - "import matplotlib.pyplot as plt\n", - "X_trn = smat.load_npz(f\"xmc-base/{DATASET}/tfidf-attnxml/X.trn.npz\")\n", - "Y_trn = smat.load_npz(f\"xmc-base/{DATASET}/Y.trn.npz\")" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "0b34281d", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "'14146 instances with 101938 features.'" - ] - }, - "execution_count": 3, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "\"{} instances with {} features.\".format(*X_trn.shape)" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "id": "7f16a000", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "'Overall Sparsity: 99.34%'" - ] - }, - "execution_count": 4, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "\"Overall Sparsity: {:.2f}%\".format(100 * (1 - X_trn.nnz / (X_trn.shape[0] * X_trn.shape[1])))" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "id": "dcf0f0cf", - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAfnUlEQVR4nO3dfbxVZZ338c8XlTQfAoW4EVTIqG6tJDujpuXjpKhTUpnZ3dyS+Yqm0UYnZyZqMit7QMucnBqLklvswYeyEhNTJMVyRgUUBVTySPgSQqE0FR8o5Hf/sa4ti+Pee63zsB8O+/t+vfbrrHWtda31O5vN/p1rXWtdlyICMzOzeoa0OgAzM2t/ThZmZlbIycLMzAo5WZiZWSEnCzMzK+RkYWZmhZwszGqQtF7Sa9LyZZK+VGO/70g6p7nRFZP0eUk/bHUctnVwsjCrISJ2iogVJfb7h4g4D0DSUEk/lbRSUkg6PL+vMudL+lN6nS9JtY4t6QJJj0p6WtIjkj7TY/sMScslbZL04T79omYlOFmYDbzfAn8PPFZl21RgMrAf8GbgXcDH6hzrUuANEbELcDDwIUnvzW2/F/hH4O7+h21Wm5OFdRxJp0q6Lrf+kKSf5NYflTQxtQxeW6X+zpJukXRxaim8dIkqIv4SEf8REb8FXqxy+inAhRGxKiJWAxcCH64Va0Qsj4hnc0WbgNfmtn87IuYBLxT8zttJukLSNZKG1tvXrBonC+tE84F3SBoiaXdgKPA2gNRHsRNwX7WKknYD5gG3R8Q/Re/Hy9mXrDVQcW8qq0nSNEnrgVXAjsCPe3NCSTsAvwA2ACdFxF96U98MnCysA6V+iGeAicChwI3AHyS9ATgM+E1EbKpSdXeyRPOTiPhsH0+/E/BUbv0pYKd6/RYRMR3YGdgf+EGP+kV2AX4FPAycGhHVWjtmhZwsrFPNBw4nSxbzgVvJEsVhab2a44EdgO/047zryb7AK3YB1kdEpLuq1qfXFh3ZkbkHeB74Qi/OdxBZ38j0PrSCzF7iZGGdqpIs3pGW51OcLL5H9lf6HEk79vG8y8g6tyv2S2WVu6p2Sq+v1Ki/LbB3L853E/BVYJ6kUX0J2AycLKxzzQeOAHaIiFXAb4BJwG7APXXqnQEsB65LfQEvI+kVkrZPq0MlbZ+7zHQ58ElJY1J/ydnAZTWOM0TSxyQNTx3pBwCnk/WZVPYZms4lYLt0ri3+X0fEBWT9HPMkjajzu5nV5GRhHSkifkd2Seg3af1pYAVZx3XN6/rpUs5Uss7ma3NJIW852eWiMWT9Ic8De6Vt3wWuA5YAS4HrU1kt7yHrb3gG+CHwn+lVcVM6/sHAjLR8aJW4zyPr5L5Z0q51zmdWlXwZ08zMirhlYWZmhZwszMyskJOFmZkVcrIwM7NC27Y6gEYYMWJEjBs3rtVhmJkNKosWLfpjRIystm2rTBbjxo1j4cKFrQ7DzGxQkfRIrW2+DGVmZoUalizSk6R3SbpX0jJJX0jl4yXdKalb0lWV4ZLTU69XpfI7JY3LHevTqXy5pGMaFbOZmVXXyJbFBuDIiNiPbHTPSZIOAs4HLoqI1wJPAqel/U8DnkzlF6X9kLQPcDLZMM6TgP+StE0D4zYzsx4alizSKJnr0+p26RXAkcBPU/ksslnDAE5I66TtR6XxdE4AroyIDRHxe6AbOKBRcZuZ2cs1tM9C0jaSFgNrgblkY9z8OSI2pl1WkY2fQ/r5KEDa/hTZoG4vlVepkz/XVEkLJS1ct25dA34bM7PO1dBkEREvRsREYCxZa+ANDTzXjIjoioiukSOr3vllZmZ91JS7oSLiz8AtZFNXDpNUuWV3LLA6La8G9gBI218F/ClfXqWOmZk1QSPvhhopaVha3gF4J/AAWdI4Me02Bbg2Lc9O66Ttv07DQc8GTk53S40HJgB3NSpuMzN7uUY+lDcamJXuXBoCXB0Rv5R0P3ClpC+RTTJzadr/UuAHkrqBJ8jugCIilkm6Grgf2Aic7nmEzcyaa6ucz6Krqyv8BLeZtdK4adf3ue7K6ccPYCTlSVoUEV3VtvkJbjMzK+RkYWZmhZwszMyskJOFmZkVcrIwM7NCThZmZlbIycLMzAo5WZiZWSEnCzMzK+RkYWZmhZwszMyskJOFmZkVcrIwM7NCThZmZlbIycLMzAo5WZiZWSEnCzMzK+RkYWZmhZwszMyskJOFmZkVcrIwM7NCThZmZlbIycLMzAo5WZiZWSEnCzMzK+RkYWZmhRqWLCTtIekWSfdLWibpzFT+eUmrJS1Or+NydT4tqVvScknH5MonpbJuSdMaFbOZmVW3bQOPvRE4OyLulrQzsEjS3LTtooj4en5nSfsAJwP7ArsDN0t6Xdr8beCdwCpggaTZEXF/A2M3M7OchiWLiFgDrEnLz0h6ABhTp8oJwJURsQH4vaRu4IC0rTsiVgBIujLt62RhZtYkTemzkDQOeAtwZyo6Q9J9kmZKGp7KxgCP5qqtSmW1ynueY6qkhZIWrlu3bqB/BTOzjtbwZCFpJ+Aa4KyIeBq4BNgbmEjW8rhwIM4TETMioisiukaOHDkQhzQzs6SRfRZI2o4sUfwoIn4GEBGP57Z/D/hlWl0N7JGrPjaVUafczMyaoJF3Qwm4FHggIr6RKx+d2+09wNK0PBs4WdIrJI0HJgB3AQuACZLGSxpK1gk+u1Fxm5nZyzWyZXEI8H+BJZIWp7LPAB+UNBEIYCXwMYCIWCbparKO643A6RHxIoCkM4AbgW2AmRGxrIFxm5lZD428G+q3gKpsmlOnzpeBL1cpn1OvnpmZNZaf4DYzs0JOFmZmVsjJwszMCjX01lkzs8Fq3LTrWx1CW3HLwszMCjlZmJlZIScLMzMr5GRhZmaFnCzMzKyQk4WZmRVysjAzs0JOFmZmVqhXyULSEEm7NCoYMzNrT4XJQtKPJe0iaUeyuSful/SvjQ/NzMzaRZmWxT5pOtTJwA3AeLJ5KszMrEOUSRbbpelRJwOzI+KvZBMXmZlZhyiTLL5LNqPdjsBtkvYCnm5kUGZm1l4KR52NiIuBi3NFj0g6onEhmZlZuynTwT1K0qWSbkjr+wBTGh6ZmZm1jTKXoS4DbgR2T+u/A85qUDxmZtaGyiSLERFxNbAJICI2Ai82NCozM2srZZLFs5J2I90BJekg4KmGRmVmZm2lzLSqnwRmA3tLuh0YCZzY0KjMzKytlLkb6m5JhwGvBwQsT89amJlZhyhzN9TpwE4RsSwilgI7SfrHxodmZmbtokyfxUcj4s+VlYh4EvhowyIyM7O2UyZZbCNJlRVJ2wBDiypJ2kPSLZLul7RM0pmpfFdJcyU9lH4OT+WSdLGkbkn3Sdo/d6wpaf+HJPkZDzOzJiuTLH4FXCXpKElHAVeksiIbgbMjYh/gIOD09EDfNGBeREwA5qV1gGOBCek1FbgEsuQCnAscCBwAnFtJMGZm1hxlksWngFuAj6fXPODfiipFxJqIuDstPwM8AIwBTgBmpd1mkQ1QSCq/PDJ3AMMkjQaOAeZGxBPpEthcYFK5X8/MzAZCmbuhNpH9lX9JX08iaRzwFuBOYFRErEmbHgNGpeUxwKO5aqtSWa3ynueYStYiYc899+xrqGZmVkWZu6EOSX0Lv5O0QtLvJa0oewJJOwHXAGeleTFeEhHBAA13HhEzIqIrIrpGjhw5EIc0M7OkzEN5lwL/DCyil8N8pHkwrgF+FBE/S8WPSxodEWvSZaa1qXw1sEeu+thUtho4vEf5rb2Jw8zM+qdMn8VTEXFDRKyNiD9VXkWV0h1UlwIPRMQ3cptms3nU2inAtbnyU9JdUQel864hG8TwaEnDU8f20anMzMyapEzL4hZJXwN+BmyoFFY6r+s4hGz61SWSFqeyzwDTgaslnQY8ApyUts0BjgO6geeAU9N5npB0HrAg7ffFiHiiRNxmZjZAyiSLA9PPrlxZAEfWqxQRvyUbHqSao6rsH8DpNY41E5hZGKmZmTVEmbuhPCuemVmHK9OyQNLxwL7A9pWyiPhio4IyM7P2UubW2e8AHwA+QXZZ6f3AXg2Oy8zM2kiZu6EOjohTgCcj4gvA24DXNTYsMzNrJ2WSxfPp53OSdgf+CoxuXEhmZtZuyvRZ/FLSMOBrwN1kd0J9v5FBmZlZeymTLC6IiA3ANZJ+SdbJ/UJjwzIzs3ZS5jLU/1QWImJDRDyVLzMzs61fzZaFpP9FNrrrDpLewuYH7HYBXtmE2MzMrE3Uuwx1DPBhsoH7LmRzsniGbNgOM7O2Nm7a9a0OYatRM1lExCxglqT3RcQ1TYzJzMzaTJk+i7GSdkmjwX5f0t2Sjm54ZGZm1jbKJIuPpEmLjgZ2IxtJdnpDozIzs7ZSJllU+iqOI5sjexm1R5M1M7OtUJlksUjSTWTJ4kZJOwObGhuWmZm1kzIP5Z0GTARWRMRzknYjTUxkZmadocx8FpskPQ7sI6nUkOZmZrZ1Kfzyl3Q+2RDl9wMvpuIAbmtgXGZm1kbKtBQmA69P40OZmVkHKtPBvQLYrtGBmJlZ+yrTsngOWCxpHvBS6yIi/qlhUZmZWVspkyxmp5eZmXWoMndDzWpGIGZm1r7qDVG+hOyup6oi4s0NicjMzNpOvZbF3zUtCjMza2v1hih/pJmBmJlZ+ypz66yZmXW4hiULSTMlrZW0NFf2eUmrJS1Or+Ny2z4tqVvScknH5MonpbJuSdMaFa+ZmdVWr4N7XkQcJen8iPhUH459GfAt4PIe5RdFxNd7nGsf4GRgX2B34GZJr0ubvw28E1gFLJA0OyLu70M8ZmaDQn+mg105/fgBjGSzeh3coyUdDLxb0pX0mMMiIu6ud+CIuE3SuJJxnABcmYYU+b2kbuCAtK07IlYApDhOIBunyszMmqResvgccA4wFvhGj20BHNnHc54h6RRgIXB2RDwJjAHuyO2zKpUBPNqj/MBqB5U0FZgKsOeee/YxNDMzq6Zmn0VE/DQijgUuiIgjerz6miguAfYmmx9jDXBhH4/zMhExIyK6IqJr5MiRA3VYMzOj3BPc50l6N3BoKro1In7Zl5NFxOOVZUnfAyrHWQ3skdt1bCqjTrmZmTVJ4d1Qkr4KnEnWT3A/cKakr/TlZJJG51bfA1TulJoNnCzpFZLGAxOAu4AFwARJ4yUNJesE9zhVZmZNVmYgweOBiRGxCUDSLOAe4DP1Kkm6AjgcGCFpFXAucLikiWR9HiuBjwFExDJJV5Mlo43A6RHxYjrOGcCNwDbAzIhY1rtf0czM+qvsNKnDgCfS8qvKVIiID1YpvrTO/l8GvlylfA4wp8w5zcysMcoki68C90i6hez22UMBPxxnZtZBynRwXyHpVuBvUtGnIuKxhkZlZmZtpdRlqIhYgzuWzcw6lgcSNDOzQmU7uM3MWqI/4yTZwKnbspC0jaQHmxWMmZm1p7rJIj3rsFySB1syM+tgZS5DDQeWSboLeLZSGBHvblhUZmbWVsoki3MaHoWZmbW1Ms9ZzJe0FzAhIm6W9EqyoTfMzKxDlBlI8KPAT4HvpqIxwC8aGJOZmbWZMs9ZnA4cAjwNEBEPAa9uZFBmZtZeyiSLDRHxl8qKpG3JRo01M7MOUSZZzJf0GWAHSe8EfgJc19iwzMysnZRJFtOAdcASsvkn5gCfbWRQZmbWXsrcDbUpTXh0J9nlp+UR4ctQZmYdpDBZSDoe+A7wMNl8FuMlfSwibmh0cGZm1h7KPJR3IXBERHQDSNobuB5wsjAz6xBl+iyeqSSKZAXwTIPiMTOzNlSzZSHpvWlxoaQ5wNVkfRbvBxY0ITYzM2sT9S5DvSu3/DhwWFpeB+zQsIjMzKzt1EwWEXFqMwMxM7P2VeZuqPHAJ4Bx+f09RLmZWecoczfUL4BLyZ7a3tTQaMzMrC2VSRYvRMTFDY/EzMzaVplk8U1J5wI3ARsqhRFxd8OiMjOztlLmOYs3AR8FppM9oHch8PWiSpJmSloraWmubFdJcyU9lH4OT+WSdLGkbkn3Sdo/V2dK2v8hSVN6+wuamVn/lWlZvB94TX6Y8pIuA74FXJ4rmwbMi4jpkqal9U8BxwIT0utA4BLgQEm7AucCXWTPeCySNDsinuxlLGbWQuOmXd/qEKyfyrQslgLDenvgiLgNeKJH8QnArLQ8C5icK788MncAwySNBo4B5kbEEylBzAUm9TYWMzPrnzIti2HAg5IWsGWfRV9unR0VEWvS8mPAqLQ8Bng0t9+qVFar/GUkTQWmAuy55559CM3MzGopkyzObcSJIyIkDdhQ5xExA5gB0NXV5SHUzcwGUJn5LOYP4PkelzQ6Itaky0xrU/lqYI/cfmNT2Wrg8B7ltw5gPGZmVkJhn4WkZyQ9nV4vSHpR0tN9PN9soHJH0xTg2lz5KemuqIOAp9LlqhuBoyUNT3dOHZ3KzMysicq0LHauLEsSWWf0QUX1JF1B1ioYIWkV2eWs6cDVkk4DHgFOSrvPAY4DuoHngFPTuZ+QdB6bR7n9YkT07DQ3M7MGK9Nn8ZI0neov0kN60wr2/WCNTUfVOO7pNY4zE5jZmzjNzGxglRlI8L251SFkzzy80LCIzMys7ZRpWeTntdgIrCS7FGVmZh2iTJ+F57UwM+tw9aZV/VydehER5zUgHjMza0P1WhbPVinbETgN2A1wsjAz6xD1plW9sLIsaWfgTLJbWq8kG3nWzMw6RN0+izTq6yeBD5EN/Le/R3w1M+s89fosvga8l2y8pTdFxPqmRWVmZm2l3nAfZwO7A58F/pAb8uOZfgz3YWZmg1C9Posyc12YmVkH6NVwH2bWuTzbXWdz68HMzAo5WZiZWSEnCzMzK+RkYWZmhZwszMyskJOFmZkVcrIwM7NCThZmZlbIycLMzAo5WZiZWSEnCzMzK+Sxocw6hMd2sv5wy8LMzAo5WZiZWSEnCzMzK9SSZCFppaQlkhZLWpjKdpU0V9JD6efwVC5JF0vqlnSfpP1bEbOZWSdrZcviiIiYGBFdaX0aMC8iJgDz0jrAscCE9JoKXNL0SM3MOlw7XYY6AZiVlmcBk3Pll0fmDmCYpNEtiM/MrGO1KlkEcJOkRZKmprJREbEmLT8GjErLY4BHc3VXpbItSJoqaaGkhevWrWtU3GZmHalVz1m8PSJWS3o1MFfSg/mNERGSojcHjIgZwAyArq6uXtU1M7P6WtKyiIjV6eda4OfAAcDjlctL6efatPtqYI9c9bGpzMzMmqTpyULSjpJ2riwDRwNLgdnAlLTbFODatDwbOCXdFXUQ8FTucpWZmTVBKy5DjQJ+Lqly/h9HxK8kLQCulnQa8AhwUtp/DnAc0A08B5za/JDN2oOH7LBWaXqyiIgVwH5Vyv8EHFWlPIDTmxCamZnV0E63zpqZWZtysjAzs0IeotysydzvYIORWxZmZlbIycLMzAo5WZiZWSEnCzMzK+RkYWZmhXw3lFkf+I4m6zRuWZiZWSEnCzMzK+RkYWZmhZwszMyskJOFmZkVcrIwM7NCThZmZlbIz1lYR/JzEma945aFmZkVcrIwM7NCThZmZlbIfRY2aLnfwax53LIwM7NCbllYS7l1YDY4uGVhZmaFnCzMzKyQL0NZv/lSktnWzy0LMzMrNGhaFpImAd8EtgG+HxHTWxzSVsWtAzOrZ1AkC0nbAN8G3gmsAhZImh0R97c2svbhL3sza6RBkSyAA4DuiFgBIOlK4ASgIcnCX7xmZlsaLMliDPBobn0VcGB+B0lTgalpdb2k5U2KrawRwB9bHUQfDebYYXDH79hbY9DGrvOBvse/V60NgyVZFIqIGcCMVsdRi6SFEdHV6jj6YjDHDoM7fsfeGoM5dmhM/IPlbqjVwB659bGpzMzMmmCwJIsFwARJ4yUNBU4GZrc4JjOzjjEoLkNFxEZJZwA3kt06OzMilrU4rN5q20tkJQzm2GFwx+/YW2Mwxw4NiF8RMdDHNDOzrcxguQxlZmYt5GRhZmaFnCz6SdKZkpZKWibprFR2laTF6bVS0uIadSdJWi6pW9K0Zsadi6E/8a+UtCTtt7CZcafzV4t9oqQ7KjFJOqBG3SmSHkqvKU0NnH7H/mLu36clN3rUiH8/Sf+TPhPXSdqlRt2Wfu77GXvTP/OSZkpaK2lprmxXSXPT53eupOGpXJIuTu/tfZL2r3HMt6bfozvtr8JAIsKvPr6ANwJLgVeS3SxwM/DaHvtcCHyuSt1tgIeB1wBDgXuBfQZL/GnbSmBEO733wE3AsWmf44Bbq9TdFViRfg5Py8MHQ+xp2/pWvOcl4l8AHJb2+QhwXpW6Lf3c9yf2tK3pn3ngUGB/YGmu7AJgWlqeBpyf+9zcAAg4CLizxjHvStuV9j+2KA63LPrnf5P9YzwXERuB+cB7KxtTtj4JuKJK3ZeGMImIvwCVIUyaqT/xt1qt2AOo/FX4KuAPVeoeA8yNiCci4klgLjCpCTFX9Cf2dlAr/tcBt6V95gLvq1K31Z/7/sTeEhFxG/BEj+ITgFlpeRYwOVd+eWTuAIZJGp2vmNZ3iYg7Isscl+fq1+Rk0T9LgXdI2k3SK8myev7hwXcAj0fEQ1XqVhvCZEzDIq2uP/FD9uV2k6RFabiVZqoV+1nA1yQ9Cnwd+HSVuq1+7/sTO8D26TLVHZImNyPgHmrFv4zNX/zvZ8vPUkW7vvdlYofWfubzRkXEmrT8GDAqLZd5f8ek8nr7vMygeM6iXUXEA5LOJ7t88CywGHgxt8sHac+/yoEBif/tEbFa0quBuZIeTH8FNVyd2D8O/HNEXCPpJOBS4G+bEVNZAxD7Xul9fw3wa0lLIuLhJoVfL/6PABdLOofsodm/NCumsgYg9pZ95muJiJDU8Gcg3LLop4i4NCLeGhGHAk8CvwOQtC1Z8/aqGlXbYgiTfsRPRKxOP9cCPye7xNA0NWKfAvws7fKTGjG1/L3vR+z5930FcCvwloYH/PIYXhZ/RDwYEUdHxFvJ/siolsDa8r0vGXvLP/M5j1cuL6Wfa1N5mfd3dSqvt8/LNbOjZmt8Aa9OP/cEHgSGpfVJwPw69bYl61gdz+aOvn0HUfw7Ajvnlv8bmNTq2IEHgMNT+VHAoir1dgV+T9a5PTwt7zpIYh8OvCItjwAeosk3RtSJv1I2hOw6+Eeq1Gv5574fsbfsMw+MY8sO7q+xZQf3BWn5eLbs4L6rxvF6dnAfVxhDsz9kW9sL+A3ZvBr3Akflyi8D/qHHvrsDc3Lrx5H9Rfkw8O+DKX6yu1nuTa9lrYi/WuzA24FFqexO4K2pvItshsVK3Y8A3el16mCJHTgYWJL2WQKc1i6fG+DM9Hn+HTCdzSNEtNXnvq+xt+ozT9bSWQP8lax/4TRgN2Ae2R8LN5P+2Elf/t9O7+0SoCt3nMW55S6y/puHgW9Vft96Lw/3YWZmhdxnYWZmhZwszMyskJOFmZkVcrIwM7NCThZmZlbIycIGjR6jrS6WNK4Px5gsaZ8GhIekIWkEz6VpRM8FksY34lzpfLtL+mlanijpuD4cY7Kkz6XlT6TY5yibvhhJb5d0UW7/kZJ+NVC/gw0eThY2mDwfERNzr5V9OMZkoFfJIj3NXsYHyO7Lf3NEvAl4D/Dn3pyrN+ePiD9ExIlpdSLZ8wu99W/Af6XlDwFvJnvY7Jg0kOQ5wHm5c64D1kg6pA/nskHMycIGtTQu//w0sNuNuSEQPpr+sr9X0jWSXinpYODdZIP1LZa0t6RbJXWlOiMkrUzLH5Y0W9KvgXmSdkzzCtwl6R5J1UZKHQ2siYhNABGxKrJRbZG0XtJFyuZQmCdpZK04U/llkr4j6U7gAkmH5VpU90jaWdK41BIYCnwR+EDa/gFl8xxUzjFE2bwFI3u8d68DNkTEHytFwHZkw3f/Ffh74IaI6Dni6S/IEot1ECcLG0x2yH1h/lzSdsB/AidGNqbPTODLad+fRcTfRMR+ZMNonBYR/002SNy/ppZJ0eB7+6djHwb8O/DriDgAOIIs4ezYY/+rgXel+C6UlB+zaUdgYUTsSzYs9rm14szVGQscHBGfBP4FOD0iJpKNBvx8ZafIhvr+HHBV+r2uAn7I5i/0vwXuTa2CvEOAu3Pr3wLuIBsG43bgVLKngXtamGKwDuJRZ20weT59WQIg6Y1kk9nMza6YsA3ZsAgAb5T0JbJxf3YCbuzD+ebm/qo+Gni3pH9J69uTfak+UNk5IlZJej1wZHrNk/T+iJgHbGLzoIw/ZPOAgfXi/ElEVEYBvh34hqQfkSWYVao/udlM4FrgP8iGNvl/VfYZDbyUQCLiB8APAFI/xsXAsZJOIRv2+uzUalpLdrnNOoiThQ1mApZFxNuqbLsMmBwR90r6MHB4jWNsZHMLe/se257tca73RcTyegFFxAaygdlukPQ4WR/JvGq7lojzpfNHxHRJ15P1S9wu6RjghTpxPCrpcUlHko2MWu2y0fNkkyxtQdLuwAER8UVJ88kS32fJBjecS/Y+Pd+znm3dfBnKBrPlwEhJbwOQtJ2kfdO2nck6Yrdjyy/KZ9K2ipXAW9PyidR2I/CJ1OlLj0tMpLL90xctkoaQdRY/kjYPyR3//wC/LYiz57H3joglEXE+2RSgb+ixS8/fC+D7ZK2YfAsl7wGyKUV7Oo/sshbADmSJbRNZXwZks8otrVLPtmJOFjZopWv1JwLnS7qXbCKbg9Pmc8hGbr2dbBjqiiuBf02dxHuTzUj3cUn3kA35Xct5ZJ2/90laRu4OoZxXA9dJWgrcR9Zq+Vba9ixwQNp2JFmHdL04ezordWbfR9b5fEOP7bcA+1Q6uFPZbLJLW9UuQUE2jehbKgkQNifBiKj0ZfyYbPTSQ4DKLbNHANfXidW2Qh511qwJJK2PiJ2afM4u4KKIqNkZLembwHURcXMvjnsbcELlTi/rDG5ZmG2FJE0DrqH2PN4VX2Hz5aUyxx0JfMOJovO4ZWFmZoXcsjAzs0JOFmZmVsjJwszMCjlZmJlZIScLMzMr9P8B7lbCTesJFoAAAAAASUVORK5CYII=\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], - "source": [ - "counts, bins = np.histogram(100 - 100 * X_trn.getnnz(1) / X_trn.shape[1], bins=20)\n", - "plt.hist(bins[:-1], bins, weights=counts)\n", - "plt.title(DATASET);\n", - "plt.xlabel(\"Feature Sparsity (%)\");\n", - "plt.ylabel(\"Number of Instances\");" - ] - }, - { - "cell_type": "markdown", - "id": "a1e0157b", - "metadata": {}, - "source": [ - "### Extremely large label space" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "id": "49f8fe28", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "'14146 instances with 30938 labels.'" - ] - }, - "execution_count": 6, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "\"{} instances with {} labels.\".format(*Y_trn.shape)" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "64f5fc9b", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "'Overall Sparsity: 99.94%'" - ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "\"Overall Sparsity: {:.2f}%\".format(100 * (1 - Y_trn.nnz / (Y_trn.shape[0] * Y_trn.shape[1])))" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "id": "b5cb2084", - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEWCAYAAACaBstRAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAhmUlEQVR4nO3deZgdVbnv8e+PIczIkJYTMhDAoAaVAJFBD7MigxrEK8JVCMMlKKCoXI8BB1AOGkDkHFCBIDmAQwBBIUoQAlcmLwESCBmAQMBwSQiTcJMwGAi8549aGyrN7t2rd/fuvZP+fZ6nnq5atVbVm87ufrtqVa2liMDMzCzHas0OwMzMVh5OGmZmls1Jw8zMsjlpmJlZNicNMzPL5qRhZmbZnDTMOiDpZUlbpfXLJP17B/UukvT93o2uc5JOl/SbZsdhqxYnDbMORMT6EfFERr2vRMQZAJL6SbpG0nxJIWnPcl0VzpL0j7ScJUkdHVvS2ZKekrRE0pOSTm23f7ykuZLeknRkXf9Qsy5w0jDreXcBXwaeqbJvDHAQsB3wEeAzwHE1jnUp8IGI2BD4GPAlSQeX9j8IHA/c3/2wzTrnpGF9jqSjJP2ptP2YpN+Xtp+SNCJdKbyvSvsNJP1V0vnpyuHtW1cR8XpE/EdE3AW8WeX0o4FzI2JBRCwEzgWO7CjWiJgbEa+Uit4C3lfa/4uIuBX4Zyf/5jUlTZR0raR+teqa1eKkYX3R7cBuklaTtDnQD9gVIPVhrA/MrNZQ0qbArcDfIuLr0fVxeLaluDqoeDCVdUjSWEkvAwuA9YDfdeWEktYBrgOWAYdExOtdaW9W5qRhfU7qp1gKjAB2B24Cnpb0AWAP4M6IeKtK080pEs7vI+J7dZ5+fWBxaXsxsH6tfo2IGAdsAOwA/Lpd+85sCPwFeBw4KiKqXf2YZXPSsL7qdmBPiqRxO3AbRcLYI21XcyCwDnBRN877MsUv8ooNgZcjItJTWC+nZYUO7yg8ALwG/LAL59uFou9kXB1XRWbv4qRhfVUlaeyW1m+n86RxCcVf7ZMlrVfneedQdIJXbJfKKk9hrZ+WH3fQfg1g6y6c72bgJ8CtkjarJ2CzMicN66tuB/YC1omIBcCdwH7ApsADNdqdCMwF/pT6Ct5F0lqS1k6b/SStXbr9dAXwLUkDU3/KycBlHRxnNUnHSdo4dbjvBJxA0adSqdMvnUvAmulcK/xcR8TZFP0gt0rqX+PfZtYpJw3rkyLiUYpbRXem7SXAExQd3B3e90+3eMZQdEpfX0oOZXMpbiMNpOgveQ3YIu27GPgTMAuYDdyQyjryOYr+iKXAb4AL0lJxczr+x4DxaX33KnGfQdEZfoukTWqcz6wm+TanmZnl8pWGmZllc9IwM7NsThpmZpbNScPMzLKt0ewAGqV///4xdOjQZodhZrbSmD59+gsR0VarziqbNIYOHcq0adOaHYaZ2UpD0pOd1fHtKTMzy+akYWZm2Zw0zMwsm5OGmZllc9IwM7NsThpmZpatYUlD0uA0j/JDkuZIOimVbyJpSpqXeYqkjVO50pzL8yTNlLRD6VijU/3HJI1uVMxmZlZbI680lgMnR8RwitnDTpA0HBgL3BoRwyjmBRib6u8PDEvLGOBCKJIMcBqwM7ATcFol0ZiZWe9qWNKIiEURcX9aXwo8TDG/wCjg8lTtcuCgtD4KuCJNazkV2EjSAOBTwJSIeDEiXgKmUEyWY2ZmvaxX3giXNBTYHrgH2CwiFqVdzwCVKSgHAk+Vmi1IZR2VVzvPGIqrFIYMGdJD0ZsZwNCxN9Tddv64A3swEmumhneES1ofuBb4Rpod7W1pFrQemwUqIsZHxMiIGNnWVnP4FDMzq0NDk4akNSkSxm8j4g+p+Nl024n09blUvhAYXGo+KJV1VG5mZr2skU9PCbgUeDgiflbaNQmoPAE1Gri+VH5EeopqF2Bxuo11E7CvpI1TB/i+qczMzHpZI/s0Pg4cDsySNCOVnQqMA66WdAzwJHBI2jcZOACYB7wKHAUQES9KOgO4L9X7UUS82MC4zcysAw1LGhFxF6AOdu9TpX4AJ3RwrAnAhJ6LzszM6uE3ws3MLJuThpmZZXPSMDOzbE4aZmaWzUnDzMyyOWmYmVk2Jw0zM8vmpGFmZtmcNMzMLJuThpmZZXPSMDOzbE4aZmaWzUnDzMyyOWmYmVk2Jw0zM8vmpGFmZtkaOd3rBEnPSZpdKrtK0oy0zK/M6CdpqKTXSvsuKrXZUdIsSfMknZ+mkTUzsyZo5HSvlwE/B66oFETEFyvrks4FFpfqPx4RI6oc50LgWOAeiilh9wNu7PlwzcysMw270oiIO4Cqc3mnq4VDgIm1jiFpALBhRExN08FeARzUw6GamVmmZvVp7AY8GxGPlcq2lPSApNsl7ZbKBgILSnUWpDIzM2uCRt6equUwVrzKWAQMiYh/SNoRuE7Stl09qKQxwBiAIUOG9EigZmb2jl6/0pC0BnAwcFWlLCKWRcQ/0vp04HFgG2AhMKjUfFAqqyoixkfEyIgY2dbW1ojwzcz6tGbcnvoE8EhEvH3bSVKbpNXT+lbAMOCJiFgELJG0S+oHOQK4vgkxm5kZjX3kdiJwN/B+SQskHZN2Hcq7O8B3B2amR3CvAb4SEZVO9OOBXwHzKK5A/OSUmVmTNKxPIyIO66D8yCpl1wLXdlB/GvChHg3OzMzq4jfCzcwsm5OGmZlla9Yjt2Z91tCxN9Tddv64A3swErOu85WGmZllc9IwM7NsThpmZpbNScPMzLI5aZiZWTYnDTMzy+akYWZm2Zw0zMwsm5OGmZllc9IwM7NsThpmZpbNScPMzLJ5wELrszxwoFnX+UrDzMyydSlpSFpN0oaZdSdIek7S7FLZ6ZIWSpqRlgNK+06RNE/SXEmfKpXvl8rmSRrblXjNzKxndZo0JP1O0oaS1gNmAw9J+nbGsS8D9qtSfl5EjEjL5HSO4RRzh2+b2vxS0uqSVgd+AewPDAcOS3XNzKwJcq40hkfEEuAg4EZgS+DwzhpFxB3Ai5lxjAKujIhlEfF3YB6wU1rmRcQTEfE6cGWqa2ZmTZCTNNaUtCZF0pgUEW8A0Y1znihpZrp9tXEqGwg8VaqzIJV1VF6VpDGSpkma9vzzz3cjRDMzqyYnaVwMzAfWA+6QtAWwpM7zXQhsDYwAFgHn1nmcqiJifESMjIiRbW1tPXloMzMj45HbiDgfOL9U9KSkveo5WUQ8W1mXdAnw57S5EBhcqjoolVGj3MzMellOR/hmki6VdGPaHg6MrudkkgaUNj9H0bEOMAk4VNJakrYEhgH3AvcBwyRtKakfRWf5pHrObWZm3Zfzct9lwH8B303bjwJXAZfWaiRpIrAn0F/SAuA0YE9JIyj6ROYDxwFExBxJVwMPAcuBEyLizXScE4GbgNWBCRExJ/tfZ2ZmPSonafSPiKslnQIQEcslvdlZo4g4rEpxh4kmIs4EzqxSPhmYnBGnmZk1WE5H+CuSNiU9MSVpF2BxQ6MyM7OWlHOl8S2KfoStJf0NaAP+R0OjMjOzlpTz9NT9kvYA3g8ImJve1TAzsz4m5+mpE4D1I2JORMwG1pd0fONDMzOzVpNze+rYiPhFZSMiXpJ0LPDLxoVl1rnuDG1uZvXJ6QhfXZIqG2kQwX6NC8nMzFpVzpXGX4CrJF2cto9LZWZm1sfkJI3vUCSKr6btKcCvGhaRmZm1rJynp96iGGjwwsaHY2ZmrazTpCHp48DpwBapvoCIiK0aG5pZ63InvPVVObenLgW+CUwHOh0+xMzMVl05SWNxRNzY8EjMbJXVnSuz+eMO7MFIrLtyksZfJZ0D/AFYVimMiPsbFpWZmbWknKSxc/o6slQWwN49H46ZmbWynKen6pqlz8zMVj05VxpIOhDYFli7UhYRP2pUUGZm1ppyBiy8CPgi8DWKx22/QPH4rZmZ9TE5Y099LCKOAF6KiB8CuwLbdNZI0gRJz0maXSo7R9IjkmZK+qOkjVL5UEmvSZqRlotKbXaUNEvSPEnnl8fBMjOz3pWTNF5LX1+VtDnwBjAgo91lwH7tyqYAH4qIj1DMNX5Kad/jETEiLV8plV8IHAsMS0v7Y5qZWS/JSRp/TlcE5wD3A/OBiZ01iog7gBfbld0cEcvT5lRgUK1jSBoAbBgRUyMigCuAgzJiNjOzBsjpCD87IpYB10r6M0Vn+D974NxHA1eVtreU9ACwBPheRNwJDAQWlOosSGVVSRoDjAEYMmRID4RoZmZlOVcad1dWImJZRCwul9VD0neB5cBvU9EiYEhEbE8xJ/nvJG3Y1eNGxPiIGBkRI9va2roTopmZVdHhlYakf6H4q34dSdtTPDkFsCGwbr0nlHQk8Glgn3TLiXQlsyytT5f0OEVn+0JWvIU1KJWZmVkT1Lo99SngSIpf1OfyTtJYCpxaz8kk7Qf8G7BHRLxaKm8DXoyINyVtRdHh/UREvChpiaRdgHuAI4AL6jm3mZl1X4dJIyIuBy6X9PmIuLarB5Y0EdgT6C9pAXAaxdNSawFT0pOzU9OTUrsDP5L0BvAW8JWIqHSiH0/xJNY6wI1pMbMu8nDu1hNyOsIHpf6FpcAlwA7A2Ii4uVajiDisSvGlHdS9FqiamCJiGvChjDjNzKzBcjrCj46IJcC+wKbA4cC4hkZlZmYtKSdpVPoyDgCuiIg5pTIzM+tDcpLGdEk3UySNmyRtQNHvYGZmfUxOn8YxwAiKp5lelbQpcFRDozIzs5aUM5/GW5KeBYZLyhpK3czMVk2dJgFJZ1EMjf4Q8GYqDuCOBsZlZmYtKOfK4SDg/emtbTOzXtWd90vmjzuwByMxyOsIfwJYs9GBmJlZ68u50ngVmCHpVtL4UAAR8fWGRWVmZi0pJ2lMSouZmfVxOU9PXd4bgZiZWeurNTT6LIqnpKpKU7aamVkfUutK49O9FoWZZfFItdZstYZGf7I3AzEzs9bnN7zNbJXV3Sszv+fxbjnvaZiZmQE1kkZ6L6MyjIiZmVnNK40Bkj4GfFbS9pJ2KC85B5c0QdJzkmaXyjaRNEXSY+nrxqlcks6XNE/SzPI5JI1O9R+TNLref6yZmXVPrT6NHwDfBwYBP2u3L4C9M45/GfBz4IpS2Vjg1ogYJ2ls2v4OsD8wLC07AxcCO0vahGJ+8ZHpvNMlTYqIlzLOb2ZmPajW01PXANdI+n5EnFHPwSPiDklD2xWPAvZM65cDt1EkjVEUMwMGMFXSRpIGpLpTIuJFAElTgP2AifXEZGZm9ct5I/wMSZ8Fdk9Ft0XEn7txzs0iYlFafwbYLK0PBJ4q1VuQyjoqfxdJY4AxAEOGDOlGiGZmVk2nT09J+glwEsV8Gg8BJ0n6cU+cPF1VdPjWeR3HGx8RIyNiZFtbW08d1szMkpxHbg8EPhkREyJiAsWtoe68Lf5suu1E+vpcKl8IDC7VG5TKOio3M7Nelvuexkal9fd085yTgMoTUKOB60vlR6SnqHYBFqfbWDcB+0raOD1ptW8qMzOzXpbzRvhPgAck/RUQRd/G2JyDS5pI0ZHdX9ICiqegxgFXSzoGeBI4JFWfDBwAzKOYw+MogIh4UdIZwH2p3o8qneJmZta7cjrCJ0q6DfhoKvpORDyTc/CIOKyDXftUqRvACR0cZwIwIeecZmbWOFljT6XbRJ6Iycysj/PYU2Zmls1Jw8zMstVMGpJWl/RIbwVjZmatrWbSiIg3gbmS/Hq1mZlldYRvDMyRdC/wSqUwIj7bsKjMzKwl5SSN7zc8CjMzWynkvKdxu6QtgGERcYukdYHVGx+amZm1mpwBC48FrgEuTkUDgesaGJOZmbWonEduTwA+DiwBiIjHgPc2MigzM2tNOUljWUS8XtmQtAY9OJy5mZmtPHKSxu2STgXWkfRJ4PfAnxoblpmZtaKcpDEWeB6YBRxHMRrt9xoZlJmZtaacp6feknQ5cA/Fbam5aURaMzPrYzpNGpIOBC4CHqeYT2NLScdFxI2NDs7MzFpLzst95wJ7RcQ8AElbAzcAThpmZn1MTp/G0krCSJ4AljYoHjMza2EdXmlIOjitTpM0Gbiaok/jC7wz9WqXSXo/cFWpaCvgBxTzkB9L0ekOcGpETE5tTgGOAd4Evh4RniPczKwJat2e+kxp/Vlgj7T+PLBOvSeMiLnACCiGXgcWAn+kmBP8vIj4abm+pOHAocC2wObALZK2SSPwmplZL+owaUTEUb1w/n2AxyPiSUkd1RkFXBkRy4C/S5oH7ATc3QvxmZlZSc7TU1sCXwOGluv30NDohwITS9snSjoCmAacHBEvUYx1NbVUZ0EqqxbrGGAMwJAhngLEzKyn5XSEXwfMBy6geJKqsnSLpH7AZyneMAe4ENia4tbVonrOERHjI2JkRIxsa2vrbohmZtZOziO3/4yI8xtw7v2B+yPiWYDKVwBJlwB/TpsLgcGldoNSmZmZ9bKcK43/lHSapF0l7VBZeuDch1G6NSVpQGnf54DZaX0ScKiktdKtsmHAvT1wfjMz66KcK40PA4cDewNvpbJI23WRtB7wSYqxrCrOljQiHXt+ZV9EzJF0NfAQsBw4wU9OmZk1R07S+AKwVXl49O6KiFeATduVHV6j/pnAmT11fjMzq0/O7anZFC/emZlZH5dzpbER8Iik+4BllcIeeuTWzMxWIjlJ47SGR2FmZiuFnPk0bu+NQMzMrPXlvBG+lHfmBO8HrAm8EhEbNjIwMzNrPTlXGhtU1lUMEDUK2KWRQZmZWWvKeXrqbVG4DvhUY8IxM7NWlnN76uDS5mrASOCfDYvIzMxaVs7TU+V5NZZTvK09qiHRmJlZS8vp0+iNeTXMzGwlUGu61x/UaBcRcUYD4jEzsxZW60rjlSpl61HM1b0p4KRhZtbH1Jru9e1JkCRtAJxEMY/3lfTAJExmZrbyqdmnIWkT4FvAl4DLgR3SFKxmZtYH1erTOAc4GBgPfDgiXu61qMzMrCXVernvZGBz4HvA05KWpGWppCW9E56ZmbWSWn0aXXpb3MzMVn1NSwyS5kuaJWmGpGmpbBNJUyQ9lr5unMol6XxJ8yTN7KE5ys3MrIuafTWxV0SMiIiRaXsscGtEDANuTdsA+wPD0jIGuLDXIzUzs6YnjfZGUTylRfp6UKn8ijRg4lRgI0kDmhCfmVmf1sykEcDNkqZLGpPKNouIRWn9GWCztD4QeKrUdkEqW4GkMZKmSZr2/PPPNypuM7M+K2fAwkb514hYKOm9wBRJj5R3RkRIig7aVhUR4ykeEWbkyJFdamtmZp1r2pVGRCxMX58D/gjsBDxbue2Uvj6Xqi8EBpeaD0plZmbWi5qSNCStl4YmQdJ6wL7AbGASMDpVGw1cn9YnAUekp6h2ARaXbmOZmVkvadbtqc2APxazx7IG8LuI+Iuk+4CrJR0DPAkckupPBg4A5gGvUoyBZWZmvawpSSMingC2q1L+D2CfKuUBnNALoZmZWQ2t9sitmZm1MCcNMzPL5qRhZmbZnDTMzCybk4aZmWVz0jAzs2xOGmZmls1Jw8zMsjlpmJlZNicNMzPL5qRhZmbZnDTMzCybk4aZmWVz0jAzs2xOGmZmls1Jw8zMsjlpmJlZtl5PGpIGS/qrpIckzZF0Uio/XdJCSTPSckCpzSmS5kmaK+lTvR2zmZkVmjHd63Lg5Ii4X9IGwHRJU9K+8yLip+XKkoYDhwLbApsDt0jaJiLe7NWozcys9680ImJRRNyf1pcCDwMDazQZBVwZEcsi4u/APGCnxkdqZmbtNeNK422ShgLbA/cAHwdOlHQEMI3iauQlioQytdRsAR0kGUljgDEAQ4YMaVzgZtYnDB17Q91t5487sAcjaR1N6wiXtD5wLfCNiFgCXAhsDYwAFgHndvWYETE+IkZGxMi2traeDNfMzGhS0pC0JkXC+G1E/AEgIp6NiDcj4i3gEt65BbUQGFxqPiiVmZlZL2vG01MCLgUejoiflcoHlKp9Dpid1icBh0paS9KWwDDg3t6K18zM3tGMPo2PA4cDsyTNSGWnAodJGgEEMB84DiAi5ki6GniI4smrE/zklJlZc/R60oiIuwBV2TW5RpszgTMbFpSZmWXxG+FmZpbNScPMzLI5aZiZWTYnDTMzy+akYWZm2Zw0zMwsm5OGmZllc9IwM7NsThpmZpbNScPMzLI5aZiZWTYnDTMzy+akYWZm2Zw0zMwsm5OGmZlla8YkTGZmq7yhY2+ou+38cQf2YCQ9y1caZmaWbaVJGpL2kzRX0jxJY5sdj5lZX7RSJA1JqwO/APYHhlPMJz68uVGZmfU9K0ufxk7AvIh4AkDSlcAo4KGmRmVm1gCt3B+ysiSNgcBTpe0FwM7tK0kaA4xJmy9LmtsLseXoD7zQ7CA60eoxtnp80Poxtnp84Bi7TWd1K74tOquwsiSNLBExHhjf7DjakzQtIkY2O45aWj3GVo8PWj/GVo8PHGNPaHR8K0WfBrAQGFzaHpTKzMysF60sSeM+YJikLSX1Aw4FJjU5JjOzPmeluD0VEcslnQjcBKwOTIiIOU0Oqyta7pZZFa0eY6vHB60fY6vHB46xJzQ0PkVEI49vZmarkJXl9pSZmbUAJw0zM8vmpNFFkk6SNFvSHEnfSGXbSbpb0ixJf5K0YQdtv5nazZY0UdLaqVySzpT0qKSHJX29BWPcR9L9kmZIukvS+5oY47vapvJNJE2R9Fj6unGLxXeOpEckzZT0R0kb1Rtfo2Is7T9ZUkjq32rxSfpa+j7OkXR2vfE1KkZJIyRNTT8r0yTt1MWYJkh6TtLsUlnVz3b63XG+iuGVZkraoYNj7pj+PfNSfdU6bk0R4SVzAT4EzAbWpXiI4BbgfRRPd+2R6hwNnFGl7UDg78A6aftq4Mi0fhRwBbBa2n5vC8b4KPDBtH48cFmTYqzaNu07Gxib1scCZ7VYfPsCa6T1s+qNr5Expv2DKR46eRLo30rxAXul7bWa/LNSK8abgf3T+gHAbV2Ma3dgB2B2qazqZzsd/0ZAwC7APR0c8960X6n+/rWOW2vxlUbXfJDiP+XViFgO3A4cDGwD3JHqTAE+30H7NYB1JK1B8WF7OpV/FfhRRLwFEBHPtWCMAVT+4npPqby3Y+yoLRRDy1ye1i8HDmql+CLi5lQGMJXifaN6Nep7CHAe8G8U/+etFt9XgXERsQya+rNSK8Zu/axExB3Ai+2KO/psjwKuiMJUYCNJA8oN0/aGETE1iuxwRbv2XfqZcdLomtnAbpI2lbQuRZYfDMyh+OYDfIEVX0QEICIWAj8F/h+wCFgcETen3VsDX0yXsjdKGtaCMf4vYLKkBcDhwLhmxFijLcBmEbEorT8DbNZi8ZUdTfEXX70aEqOkUcDCiHiwG7E1LD6KX+i7SbpH0u2SPtqCMX4DOEfSUxQ/T6d0I8aKjj7b1YZYGtiu7cBUXq1Ol39mnDS6ICIepritcDPwF2AG8CbFL4DjJU0HNgBeb9823SscBWwJbA6sJ+nLafdawD+jePX/EmBCC8b4TeCAiBgE/Bfws2bEWKNt+3pBnX8pNzo+Sd8FlgO/rSe+RsWYfvmdCvyg3rgaGV/avQawCcWtlm8DV1fuz7dQjF8FvhkRgyl+bi6tJ74acdf92e6R49Z7P9BLAPwYOL5d2TbAvVXqfgG4tLR9BPDLtP4IsGVaF8Vf+C0TI9AGPF4qHwI81IwYa7UF5gID0voAYG4rxZe2jwTuBtZt1mexo7bAh4HngPlpWU5x1fkvrRBfWv8LsFdp3+NAW6t8D9P6Yt55B07AkjpiGcqKfRpVP9vAxcBh1eqVygYAj5S2DwMurnXcmrH15Ae3Lyykjrf0i/MRYKNS2WoU9wuPrtJuZ4rL3nXTB+ly4Gtp37hKG2BP4L5WipHir7sXgG1SvWOAa5sRY0dt0/Y5rNipd3aLxbcfxXD+PfVLrsdjbFdnPnV2hDfwe/gViv4/KH6hP0X6Bd1CMT4M7JnW9wGm1xHXUFZMGlU/28CBrNgRXjXB8e6O8APq/Znp9ge3ry3AnekH/0Fgn1R2EsXTRY9SJIDKXxmbA5NLbX+YPlyzgV/zzhMgGwE3ALMo/grdrgVj/FyK70HgNmCrJsb4rrapfFPgVuAxiqdZNmmx+OZR/JKbkZaLWu172O748+le0mjE97Af8Jv0+bwf2LvVvofAvwLTU/k9wI5djGkiRZ/iGxT9D8d09NmmSAK/oLjimgWMLB1nRml9ZPqePQ78vPRv6vLPjIcRMTOzbO4INzOzbE4aZmaWzUnDzMyyOWmYmVk2Jw0zM8vmpGErPUkvd6Hu6ZL+d08cX9J30winM9OIpjt35bhdJen/pq9DJf3POtpvL+nStP75FPudkjZNZVtLuqpUv5+kO9I4ZGaAk4ZZXSTtCnwa2CEiPgJ8ghXHAKr3uB3+go6Ij6XVoUCXkwbFECHnp/WvAR+leKO4cqx/B75XOt/rFM/wf7GOc9kqyknDVkmSPpMGtXtA0i2SygOxVeZLeEzSsaU235Z0X7py+GEnpxgAvBDvjLb6QkQ8nY4zX9LZaf6Ce5XmHukopnT182tJfwN+LWnb1G5GimVYqle54hlHMVjeDBXzn9whaUTp33GXpO3afT82AD4S7wxE+BbFmGfrAm9I2g14JiIea/fvvA74UiffC+tDnDRsVXUXsEtEbA9cSTHUd8VHgL2BXYEfSNpc0r7AMGAnYASwo6Tdaxz/ZmCwiomzfilpj3b7F0fEhynevv2PjJiGA5+IiMMohsr4z4gYQfEmb3mEUiiGe7gzIkZExHkUA+IdCSBpG2DtePcotZU3git+QvEG8Gco3kD+PnBGlX/nbIorEjPAScNWXYOAmyTNohgNddvSvusj4rWIeAH4K0Wi2DctD1AMT/EBiiRSVUS8DOwIjAGeB66SdGSpysTS110zYpoUEa+l9buBUyV9B9iiVN6R3wOflrQmxQitl1WpMyDFWYl/SkTsGBGfoRjZeDKwjaRrJF2SRrwlIt4EXk9XKmZOGrbKugD4efpr/zhg7dK+9mPnBMUYPj9Jf72PiIj3RUTNIa0j4s2IuC0iTgNOZMXJeqLKeq2YXikd93fAZ4HXKOYw2buTOF6lmCxoFHAI1Ydcf63d+QBIyeFIivGLfgiMprgiKt+SWgv4Z60YrO9w0rBV1XuAhWl9dLt9oyStnZ4a2pNies+bgKMlrQ8gaaCk93Z0cEnv14qTZY2gmB614oulr3dnxFQ+9lbAExFxPnA9xe20sqUU8zyU/Yqik/u+iHipymEfppjKtL1vA+dHxBvAOhQJ7i2Kvg7S9+iFtN8MP0pnq4J1VcwoWPEz4HTg95JeAv4PxcRSFTMpbkv1p5j/+WngaUkfBO5Oc/q8DHyZYn6JatYHLpC0EcW8E/MoblVVbCxpJrCMYv4COomp7BDgcElvUMym9uN2+2dSTJr0IMVc7edFxHRJSygmyHqXiHhE0nskbRARSwEkbQ7sFBGVTv8LKBLo/+edaT/3ohiB2QzAo9ya9TRJ8ymGqH6hF8+5OcWQ9R+INNd8lTrfBJZGxK+6cNw/UMy38GiPBGorPd+eMlvJSTqCYt6G73aUMJILKa58co/bD7jOCcPKfKVhZmbZfKVhZmbZnDTMzCybk4aZmWVz0jAzs2xOGmZmlu2/AYOspcnMX/19AAAAAElFTkSuQmCC\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], - "source": [ - "counts, bins = np.histogram(100 - 100 * Y_trn.getnnz(1) / Y_trn.shape[1], bins=20, range=(99.85, 100))\n", - "plt.hist(bins[:-1], bins, weights=counts)\n", - "plt.title(DATASET);\n", - "plt.xlabel(\"Label Sparsity (%)\");\n", - "plt.ylabel(\"Number of Instances\");" - ] - }, - { - "cell_type": "markdown", - "id": "057fb642", - "metadata": {}, - "source": [ - "## Numerical Feature and Label Format in PECOS\n", - "\n", - "In PECOS, numerical features of instances can be in either a [dense NumPy matrix](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html) or a [Compressed Sparse Row (CSR) matrix](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.html) of shape `(nr_inst, nr_feat)`, where `nr_inst` and `nr_feat` are numbers of instances and features. Similary, labels of instances can be also presented as a dense or a sparse matrix of shape `(nr_inst, nr_labels)`, where `nr_labels` is the number of labels in the XMR problem. Note that for the sparse format, training labels should be a [Compressed Sparse Column (CSC) matrix](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csc_matrix.html) while testing labels should be a CSR matrix for the purpose of computational efficiency. For convenience, PECOS also provides APIs for loading features and labels from binary files in arbitary formats.\n", - "\n", - "In addition to numerical features, PECOS also supports handling text data with transformer." - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "c518d892", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Training features X_trn is a csr matrix of shape (14146, 101938).\n", - "Training labels Y_trn is a csc matrix of shape (14146, 30938).\n", - "Testing features X_tst is a csr matrix of shape (6616, 101938).\n", - "Testing labels Y_tst is a csr matrix of shape (6616, 30938).\n" - ] - } - ], - "source": [ - "import numpy as np\n", - "from pecos.xmc.xlinear.model import XLinearModel\n", - "\n", - "DATASET = \"wiki10-31k\"\n", - "\n", - "X_trn = XLinearModel.load_feature_matrix(\"xmc-base/{}/tfidf-attnxml/X.trn.npz\".format(DATASET))\n", - "Y_trn = XLinearModel.load_label_matrix(\"xmc-base/{}/Y.trn.npz\".format(DATASET), for_training=True)\n", - "\n", - "X_tst = XLinearModel.load_feature_matrix(\"xmc-base/{}/tfidf-attnxml/X.tst.npz\".format(DATASET))\n", - "Y_tst = XLinearModel.load_label_matrix(\"xmc-base/{}/Y.tst.npz\".format(DATASET), for_training=False)\n", - "\n", - "print(f\"Training features X_trn is a {X_trn.getformat()} matrix of shape {X_trn.shape}.\")\n", - "print(f\"Training labels Y_trn is a {Y_trn.getformat()} matrix of shape {Y_trn.shape}.\")\n", - "print(f\"Testing features X_tst is a {X_tst.getformat()} matrix of shape {X_tst.shape}.\")\n", - "print(f\"Testing labels Y_tst is a {Y_tst.getformat()} matrix of shape {Y_tst.shape}.\")" - ] - }, - { - "cell_type": "markdown", - "id": "b0c731f5", - "metadata": {}, - "source": [ - "## Hands-on Example: XMR with XR-Linear\n", - "\n", - "XR-LINEAR is a recursive linear machine learned realization of our PECOS framework. As shown in the below figure, XR-Linear treats machine-learned matching as a smaller XMR problem, thereby recursively apply the three-stage framework of PECOS to address the problem.\n", - "\n", - "

\n", - "\n", - "
" - ] - }, - { - "cell_type": "markdown", - "id": "150fea14", - "metadata": {}, - "source": [ - "### Semantic Label Indexing and Cluster Chain in XR-Linear\n", - "\n", - "The first step of training an XR-Linear model is to conduct semantic label indexing and establish the *hierarchial label tree* for resursive training the XR-Linear model and its inference. \n", - "\n", - "PECOS supports any method for semantic label indexing. In the PECOS library, as a build-in method, we provide Label Representation via Positive Instance Feature Aggregation (PIFA) for semantic label indexing with only the need of positive instances and their features in training data. PECOS can also consider additional label features `Z` of shape `(nr_labels, nr_label_feat)` in either dense or sparse matrix format, where `nr_label_feat` is the number of label features. These representations and features for each label are concatenated or combined as label embedding in `LabelEmbeddingFactory` in PECOS.\n", - "\n", - "To conduct semantic label indexing, PECOS learns an indexer based on label embedding. PECOS currently supports to use the Hierarchical K-Means for semantic label indexing with a hyper-parameter `nr_splits` (the number of clusters in each layer, or `B` in [our report](https://arxiv.org/pdf/2010.05878.pdf)), which decides the depth `D` of the hierarchical label tree. " - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "id": "26794215", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "4 layers in the trained hierarchical label tree.\n" - ] - } - ], - "source": [ - "from pecos.xmc import Indexer, LabelEmbeddingFactory\n", - "\n", - "label_feat = LabelEmbeddingFactory.create(Y_trn, X_trn, method=\"pifa\")\n", - "# label_feat = LabelEmbeddingFactory.create(Y_trn, X_trn, Z, method=\"pifa_lf_concat\") # for using label features Z\n", - "\n", - "cluster_chain = Indexer.gen(label_feat, nr_splits=8, indexer_type=\"hierarchicalkmeans\")\n", - "\n", - "print(f\"{len(cluster_chain)} layers in the trained hierarchical label tree.\")" - ] - }, - { - "cell_type": "markdown", - "id": "02ffda21", - "metadata": {}, - "source": [ - "### Training XR-Linear Negative Sampling and Sparsification\n", - "\n", - "Negative sampling plays an important role in solving the XMR problem. PECOS currently provides two negative sampling schemes, including Teacher Forcing Negatives (TFN) and Matcher Aware Negatives (MAN). Please refer to [our report](https://arxiv.org/pdf/2010.05878.pdf) for more details about negative sampling schemes.\n", - "\n", - "To reduce model sizes and improve efficiency, PECOS conduct model sparsification with a hyper-parameter `threshold`. The model weights with absolute values smaller than the threshold will be discarded." - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "id": "bd3d6527", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Training time: 40.9793 seconds.\n" - ] - } - ], - "source": [ - "import time\n", - "start_time = time.time()\n", - "\n", - "# For negative_sampling_scheme in model training, \"man\" and tfn+man\" are also available.\n", - "xlm = XLinearModel.train(X_trn, Y_trn, C=cluster_chain, threshold=0.1, negative_sampling_scheme=\"tfn\")\n", - "\n", - "training_time = time.time() - start_time\n", - "print(f\"Training time: {training_time:.4f} seconds.\")" - ] - }, - { - "cell_type": "markdown", - "id": "20f5cfa7", - "metadata": {}, - "source": [ - "PECOS supports serializing and loading the trained model into binary on disk with convenient interfaces. Note that model loading with `is_predict_only=True` could lead to faster prediction speed by disabling the flexibility of model modification." - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "id": "3d5d468d", - "metadata": {}, - "outputs": [], - "source": [ - "xlm.save(\"{}.xlm.model\".format(DATASET))\n", - "xlm = XLinearModel.load(\"{}.xlm.model\".format(DATASET), is_predict_only=False)" - ] - }, - { - "cell_type": "markdown", - "id": "4b6038ec", - "metadata": {}, - "source": [ - "### Prediction and Evaluation\n", - "\n", - "As a tree model, the inference method significantly affects the prediction efficiency of XR-Linear in PECOS. As illustrated in the following figure, the prediction process in PECOS employs a beam search with a hyper-parameter `beam_size`. The other hyper-parameter `only_topk` also needs to be decided to limit the predicted most relevant labels for each instance. The `predict` function of the trained model will result in a CSR matrix of shape `(nr_inst, nr_labels)` and exactly `only_topk` non-zero columns for each row (or instance).\n", - "\n", - "
\n", - "
\n", - "
" - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "id": "7f851bc1", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Y_pred is a csr matrix of shape (6616, 30938) and 66160 non-zero elements.\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEGCAYAAACQO2mwAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAZU0lEQVR4nO3de5gldX3n8feHm1dguIwsK+IAi/fIxV4VVFSMd0XX1XhhFVmUoCai++iKMQYTYqLramLWILKoYKJ4QRRERRAQ1BikUS4DiCDeQHTGC4qwosB3/6hqadquPjXdfbprZt6v5znPqfpVnapP62G+p+pX9atUFZIkzWaT5Q4gSRoui4QkqZNFQpLUySIhSepkkZAkddpsuQMspu23375WrVq13DEkab1y4YUX/rSqVs62bIMqEqtWrWJycnK5Y0jSeiXJ97uWebpJktTJIiFJ6mSRkCR1skhIkjpZJCRJnSwSkqROFglJUieLhCSpk0VCktTJIiFJ6mSRkCR1skhIkjpZJCRJnSwSkqROFglJUieLhCSpk0VCktTJIiFJ6mSRkCR1skhIkjpZJCRJnSwSkqROFglJUieLhCSpk0VCktRprEUiyQeSrEmyelrbtknOTHJV+77NHJ/fKsm1Sd4zzpySpNmN+0jieOApM9qOAM6qqt2Bs9r5LkcB540nmiRplLEWiao6D/j5jOZnASe00ycAz57ts0keBuwAnDGufJKkuS1Hn8QOVXV9O/1jmkJwJ0k2Ad4JvG7UxpIcmmQyyeTatWsXN6kkbeQ2m2thkm3nWl5VM48S1klVVZKaZdErgc9V1bVJRm3jWOBYgImJidm2JUmapzmLBHAhUECAnYFftNMrgB8Au8xjnz9JsmNVXZ9kR2DNLOvsAzwmySuBewJbJPl1Vc3VfyFJWmRznm6qql2qalfgi8Azq2r7qtoOeAbz7ys4FTionT4IOGWW/R5YVTtX1SqaU04fskBI0tLr2yfxyKr63NRMVX0e2HfUh5KcCHwNuH97KeshwNuAJya5Cvjjdp4kE0mOW9c/QJI0PqNON035UZK/BP61nT8Q+NGoD1XVCzsWPWGWdSeBl83SfjzNpbSSpCXW90jihcBK4FPAye10VwGQJG0geh1JtFcxHZ7kHlV105gzSZIGoteRRJJ9k1wOXNHO75Hk6LEmkyQtu76nm/4BeDLwM4CquhjYb1yhJEnD0PuO66r64Yym2xY5iyRpYPpe3fTDJPsClWRz4HDaU0+SpA1X3yOJw4BXAfcGrgP2bOclSRuwkUcSSTYF3l1VBy5BHknSgIw8kqiq24D7JtliCfJIkgakb5/ENcBXk5wK/P4+iap611hSSZIGoW+R+E772gTYcnxxJElD0veO67+G5pnTzWzdONZUkqRB6HvH9USSS4FLgEuTXNw+XlSStAHre7rpA8Arq+rLAEkeDXwQeOi4gkmSll/f+yRumyoQAFX1FeDW8USSJA1F3yOJc5O8DziR5nGmzwe+lGRvgKr6xpjySZKWUd8isUf7fuSM9r1oisb+i5ZIkjQYfa9uevxcy5McVFUnLE4kSdJQ9B4FdoTDF2k7kqQBWawikUXajiRpQBarSNQibUeSNCAeSUiSOs1ZJJKs6rmdry48iiRpaEYdSXwxyRFJ5rwKqqr+bBEzSZIGYlSR2AvYAbgwyWOWII8kaUBGHSHcCLy2HczvrCTXArfT9EFUVTl2kyRtwPo8vnR/4N3AccA/0xQJSdJGYM4ikeSjwE7Ai6rq0qWJJEkailFHEl+squOWJIkkaXDm7Li2QEjSxm2xbqabVZIPJFmTZPW0tm2TnJnkqvZ9m1k+t2eSryW5LMklSZ4/zpySpNmNtUgAxwNPmdF2BHBWVe0OnNXOz3Qz8JKqenD7+X9MsmKMOSVJs+j7PAmS7Ausmv6ZqvrQXJ+pqvNmuWv7WcDj2ukTgC8Bb5jxuW9Pm/5RkjXASuCGvnklSQvXq0gk+RdgN+Ai4La2uYA5i0SHHarq+nb6xzQ3682174cDWwDfmce+JEkL0PdIYgJ4UFUt6mivVVVJOreZZEfgX4CDqmrW+zOSHAocCrDzzjsvZjxJ2uj17ZNYDfyHRdrnT9p//KeKwJrZVkqyFfBZ4E1V9e9dG6uqY6tqoqomVq5cuUgRJUnQ/0hie+DyJF8HbplqrKoD5rHPU4GDgLe176fMXCHJFsCngA9V1Unz2IckaRH0LRJvmc/Gk5xI00m9fTvu05E0xeHjSQ4Bvg/8SbvuBHBYVb2sbdsP2C7JS9vNvbSqLppPDknS/GSRuxmW1cTERE1OTi53DElaryS5sKomZls2auymr1TVo5PcyJ0fUTo1CuxWi5hTkjQwo4YKf3T7vuXSxJEkDcm477iWJK3HLBKSpE4WCUlSp15FIsk9kmzSTt8vyQFJNh9vNEnScut7JHEecNck9wbOAF5MM8KrJGkD1rdIpKpuBp4DHF1VzwMePL5YkqQh6F0kkuwDHEgznhLApuOJJEkair5F4nDgjcCnquqyJLsC54wvliRpCHqN3VRV59H0S0zNXwO8elyhJEnD0PehQ/cDXscfPplu//HEkiQNQd9RYD8BHAMcxx1PppMkbeD6Folbq+q9Y00iSRqcvh3Xn0nyyiQ7Jtl26jXWZJKkZdf3SOKg9v3109oK2HVx40iShqTv1U27jDuIJGl4+l7dtDnwCppHigJ8CXhfVf1uTLkkSQPQ93TTe4HNgaPb+Re3bS8bRyhJ0jD0LRL/uar2mDZ/dpKLxxFIkjQcfa9uui3JblMz7bAc3i8hSRu4vkcSrwfOSXINEOC+wMFjSyVJGoS+VzedlWR34P5t05VVdcv4YkmShmDOIpFk/6o6O8lzZiz6T0moqpPHmE2StMxGHUk8FjgbeOYsywqwSEjSBmzOIlFVR7aTf1NV352+LIk32EnSBq7v1U2fnKXtpMUMIkkanlF9Eg+geZb11jP6JbYC7jrOYJKk5TeqT+L+wDOAFdy5X+JG4OVjyiRJGohRfRKnAKck2aeqvrZEmSRJA9G3T+KwJCumZpJsk+QD44kkSRqKvkXioVV1w9RMVf0C2GvUh5J8IMmaJKuntW2b5MwkV7Xv23R89qB2nauSHDTbOpKk8epbJDaZ/o95+1S6PndrHw88ZUbbEcBZVbU7cFY7fyft9o8EHgE8HDiyq5hIksan79hN7wS+luQTNGM3PRd466gPVdV5SVbNaH4W8Lh2+gSaZ1O8YcY6TwbOrKqfAyQ5k6bYnNgz7zq59hc389bPXjGOTUvSknjoTit4xeN2G73iOuo7dtOHkkwC+7dNz6mqy+e5zx2q6vp2+sfADrOsc2/gh9Pmr23b/kCSQ4FDAXbeeed5BfrtrbfznbW/ntdnJWkIdthqPHcljLpPYquq+lV7+ufHwEemLdt26pf+fFVVJakFbuNY4FiAiYmJeW1r15X35IzXPnYhMSRpgzTqSOIjNPdJXEgzVtOUtPO7zmOfP0myY1Vdn2RHYM0s61zHHaekAHaiOS0lSVpCc3ZcV9Uz2vddqmrXaa9dqmo+BQLgVGDqaqWDgFNmWecLwJPaS223AZ7UtkmSltCo0017z7W8qr4x4vMn0hwRbJ/kWporlt4GfDzJIcD3gT9p150ADquql1XVz5McBVzQbupvFnpqS5K07lLVfRo/yTnt5F2BCeBimlNNDwUmq2qfsSdcBxMTEzU5ObncMSRpvZLkwqqamG3ZqNNNj6+qxwPXA3tX1URVPYzmRrrrFj+qJGlI+t5Md/+qunRqpqpWAw8cTyRJ0lD0vZnukiTHAf/azh8IXDKeSJKkoehbJA4GXgEc3s6fB7x3LIkkSYPR947r3yQ5BvhcVV055kySpIHo1SeR5ADgIuD0dn7PJKeOMZckaQD6dlwfSTMa6w0AVXURsMt4IkmShqJvkfhdVf1yRtuCxlySJA1f347ry5K8CNg0ye7Aq4F/G18sSdIQ9D2S+HPgwcAtNIP+/RJ4zZgySZIGYuSRRJJNgc+2d16/afyRJElDMfJIoqpuA25PsvUS5JEkDUjfPolfA5e2jxG9aaqxql49llSSpEHoWyRObl+SpI1Inz6JZwMrgUurygf/SNJGZM4+iSRHA68FtgOOSvLmJUklSRqEUUcS+wF7VNVtSe4OfBk4avyxJElDMOrqpt+2VzdRVTfTPJVOkrSRGHUk8YAkU8+NCLBbOx+gquqhY00nSVpWo4qET5+TpI3YnEWiqr6/VEEkScPTd+wmSdJGyCIhSeo06j6Js9r3ty9NHEnSkIzquN4xyb7AAUk+yoxLYKvqG2NLJkladqOKxF8BbwZ2At41Y1kB+48jlCRpGEZd3XQScFKSN1eVd1pL0kam1yiwVXVUkgNohukA+FJVnTa+WJKkIeh1dVOSvwcOBy5vX4cn+btxBpMkLb++z5N4OrBnVd0OkOQE4JvAX4wrmCRp+a3LfRIrpk0v+FGmSQ5PsjrJZUleM8vyrZN8JsnF7ToHL3SfkqR10/dI4u+BbyY5h+Yy2P2AI+a70yQPAV4OPBz4LXB6ktOq6uppq70KuLyqnplkJXBlkg9X1W/nu19J0rrpdSRRVScCj6R5hOkngX2q6mML2O8DgfOr6uaquhU4F3jOzN0CWyYJcE/g58CtC9inJGkd9T2SoKquB05dpP2uBt6aZDvg/wFPAyZnrPOedn8/ArYEnj/VJyJJWhrLMnZTVV0BvB04AzgduAi4bcZqT27b/yOwJ/CeJFvN3FaSQ5NMJplcu3btGFNL0sZn2Qb4q6r3V9XDqmo/4BfAt2escjBwcjWuBr4LPGCW7RxbVRNVNbFy5crxB5ekjcjIIpFk0yTfWuwdJ7lX+74zTX/ER2as8gPgCe06OwD3B65Z7BySpG4j+ySq6rYkVybZuap+sIj7/mTbJ/E74FVVdUOSw9p9HgMcBRyf5FKaK6reUFU/XcT9S5JG6NtxvQ1wWZKvAzdNNVbVAfPdcVU9Zpa2Y6ZN/wh40ny3L0lauL5F4s1jTSFJGqS+A/ydm+S+wO5V9cUkdwc2HW80SdJy6zvA38uBk4D3tU33Bj49pkySpIHoewnsq4BHAb8CqKqrgHuNK5QkaRj6Folbpo+ZlGQzmmEzJEkbsL5F4twkfwHcLckTgU8AnxlfLEnSEPQtEkcAa4FLgT8FPgf85bhCSZKGoe/VTbe3Dxo6n+Y005VV5ekmSdrA9SoSSZ4OHAN8h+bu512S/GlVfX6c4SRJy6vvzXTvBB4/9VCgJLsBnwUsEpK0AevbJ3HjjKfGXQPcOIY8kqQBmfNIIsnU0+Imk3wO+DhNn8TzgAvGnE2StMxGnW565rTpnwCPbafXAncbSyJJ0mDMWSSq6uClCiJJGp6+VzftAvw5sGr6ZxYyVLgkafj6Xt30aeD9NHdZ3z62NJKkQelbJH5TVf801iSSpMHpWyTeneRI4AzglqnGqvrGWFJJkgahb5H4I+DFwP7ccbqp2nlJ0gaqb5F4HrDr9OHCJUkbvr53XK8GVowxhyRpgPoeSawAvpXkAu7cJ+ElsJK0AetbJI4cawpJ0iD1fZ7EueMOIkkanr53XN/IHc+03gLYHLipqrYaVzBJ0vLreySx5dR0kgDPAh45rlCSpGHoe3XT71Xj08CTFz+OJGlI+p5ues602U2ACeA3Y0kkSRqMvlc3TX+uxK3A92hOOUmSNmB9+yR8roQkbYRGPb70r+ZYXFV11CLnkSQNyKiO65tmeQEcArxhITtOcniS1UkuS/KajnUel+Sidh3v1ZCkJTbq8aXvnJpOsiVwOHAw8FHgnV2fGyXJQ4CXAw8HfgucnuS0qrp62jorgKOBp1TVD5Lca777kyTNz8hLYJNsm+RvgUtoisreVfWGqlqzgP0+EDi/qm6uqluBc4HnzFjnRcDJVfUDgAXuT5I0D3MWiSTvAC4AbgT+qKreUlW/WIT9rgYek2S7JHcHngbcZ8Y69wO2SfKlJBcmeUlHxkOTTCaZXLt27SJEkyRNSVV1L0xupxn19VbuGJYDIDQd1/MeliPJIcArafo5LgNuqarXTFv+Hpr7MZ4A3A34GvD0qvp21zYnJiZqcnJyvpEkaaOU5MKqmpht2ag+iXW+I7uvqno/8H6AJH8HXDtjlWuBn1XVTcBNSc4D9gA6i4QkaXGNrQiMMtURnWRnmv6Ij8xY5RTg0Uk2a09JPQK4YmlTStLGre8d1+PwySTbAb8DXlVVNyQ5DKCqjqmqK5KcTtNhfjtwXFWtXsa8krTRWbYiUVWPmaXtmBnz7wDesWShJEl3smynmyRJw2eRkCR1skhIkjpZJCRJnSwSkqROFglJUieLhCSpk0VCktTJIiFJ6mSRkCR1skhIkjpZJCRJnSwSkqROFglJUieLhCSpk0VCktTJIiFJ6mSRkCR1skhIkjpZJCRJnSwSkqROFglJUieLhCSpk0VCktQpVbXcGRZNkrXA9xewie2Bny5SnKVk7qVl7qVl7vG7b1WtnG3BBlUkFirJZFVNLHeOdWXupWXupWXu5eXpJklSJ4uEJKmTReLOjl3uAPNk7qVl7qVl7mVkn4QkqZNHEpKkThYJSVIniwSQ5ClJrkxydZIjBpDnA0nWJFk9rW3bJGcmuap936ZtT5J/arNfkmTvaZ85qF3/qiQHLUHu+yQ5J8nlSS5Lcvj6kD3JXZN8PcnFbe6/btt3SXJ+m+9jSbZo2+/Szl/dLl81bVtvbNuvTPLkceaets9Nk3wzyWnrS+4k30tyaZKLkky2bYP+nrT7W5HkpCTfSnJFkn3Wh9wLUlUb9QvYFPgOsCuwBXAx8KBlzrQfsDewelrb/wKOaKePAN7eTj8N+DwQ4JHA+W37tsA17fs27fQ2Y869I7B3O70l8G3gQUPP3u7/nu305sD5bZ6PAy9o248BXtFOvxI4pp1+AfCxdvpB7ffnLsAu7fdq0yX4vvwP4CPAae384HMD3wO2n9E26O9Ju88TgJe101sAK9aH3Av6m5c7wHK/gH2AL0ybfyPwxgHkWsWdi8SVwI7t9I7Ale30+4AXzlwPeCHwvmntd1pvif6GU4Anrk/ZgbsD3wAeQXO37GYzvyfAF4B92unN2vUy87szfb0x5t0JOAvYHzitzbE+5P4ef1gkBv09AbYGvkt7wc/6knuhL083wb2BH06bv7ZtG5odqur6dvrHwA7tdFf+Zf272lMZe9H8Kh989vaUzUXAGuBMml/TN1TVrbNk+H2+dvkvge2WIzfwj8D/BG5v57dj/chdwBlJLkxyaNs29O/JLsBa4IPt6b3jktxjPci9IBaJ9VA1Pz8Ge+1yknsCnwReU1W/mr5sqNmr6raq2pPml/nDgQcsb6LRkjwDWFNVFy53lnl4dFXtDTwVeFWS/aYvHOj3ZDOa08Dvraq9gJtoTi/93kBzL4hFAq4D7jNtfqe2bWh+kmRHgPZ9TdvelX9Z/q4km9MUiA9X1clt83qRHaCqbgDOoTlNsyLJZrNk+H2+dvnWwM9Y+tyPAg5I8j3gozSnnN69HuSmqq5r39cAn6IpzEP/nlwLXFtV57fzJ9EUjaHnXhCLBFwA7N5eEbIFTYfeqcucaTanAlNXQRxEc75/qv0l7ZUUjwR+2R76fgF4UpJt2qstntS2jU2SAO8Hrqiqd60v2ZOsTLKinb4bTT/KFTTF4rkduaf+nucCZ7e/IE8FXtBeRbQLsDvw9XHlrqo3VtVOVbWK5nt7dlUdOPTcSe6RZMupaZr/f1cz8O9JVf0Y+GGS+7dNTwAuH3ruBVvuTpEhvGiuQvg2zXnoNw0gz4nA9cDvaH69HEJz7vgs4Crgi8C27boB/rnNfikwMW07/x24un0dvAS5H01zqH0JcFH7etrQswMPBb7Z5l4N/FXbvivNP5ZXA58A7tK237Wdv7pdvuu0bb2p/XuuBJ66hN+Zx3HH1U2Dzt3mu7h9XTb139zQvyft/vYEJtvvyqdprk4afO6FvByWQ5LUydNNkqROFglJUieLhCSpk0VCktTJIiFJ6mSR0EYtya/n+blnJ3nQYufpue+XJnlPO/2WJNe1o6leleTk5cqlDZNFQpqfZ9OMnjoE/1BVe1bV7sDHgLOTrFzuUNowWCQkIMnjknxp2rMCPtzeQU6St6V5RsYlSf53kn2BA4B3tL/gd0vy8iQXpHkmxSeT3L397PHtMwX+Lck1SZ47bZ9vSPNMhYuTvK1t2y3J6e3Ad19Osk5jSFXVx4AzgBct1v822rhtNnoVaaOxF/Bg4EfAV4FHJbkC+C/AA6qqkqyoqhuSnEpzh/NJAEluqKr/207/Lc1d8v+n3e6ONHejP4BmqIaTkjwVeBbwiKq6Ocm27brHAodV1VVJHgEcTTMm07r4BuvBAIVaP1gkpDt8vaquBWiHDV8F/DvwG+D9aZ78dlrHZx/SFocVwD2581g8n66q24HLk0wNI/3HwAer6maAqvp5O3ruvsAn2oMYaB4EtK4yehWpH4uEdIdbpk3fRvPgnluTPJxmMLfnAn/G7L/sjweeXVUXJ3kpzVhKs213rn/AN6F5FsSe65z8zvaiGV9IWjD7JKQ5tL/ut66qzwGvBfZoF91I84jWKVsC17dDpR/YY9NnAgdP67vYtppnb3w3yfPatiTZY66NzJL3v9KMKnriunxO6mKRkOa2JXBakkuAr9A8Txqa5ze8Ps0TynYD3kzzFL6vAt8atdGqOp2mf2KyPbX1unbRgcAhSaZGSH1Wj4yvnboEFvhvwP5VtbbvHyjNxVFgJUmdPJKQJHWySEiSOlkkJEmdLBKSpE4WCUlSJ4uEJKmTRUKS1On/A4kVwKqexgYuAAAAAElFTkSuQmCC\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], - "source": [ - "Y_pred = xlm.predict(X_tst, beam_size=10, only_topk=10)\n", - "\n", - "print(f\"Y_pred is a {Y_pred.getformat()} matrix of shape {Y_pred.shape} and {Y_pred.nnz} non-zero elements.\")\n", - "\n", - "import matplotlib.pyplot as plt\n", - "plt.plot(range(Y_pred.shape[0]), Y_pred.getnnz(1))\n", - "plt.xlabel(\"Instance ID\");\n", - "plt.ylabel(\"Number of Predictions in Y_pred\");" - ] - }, - { - "cell_type": "markdown", - "id": "fb1ed22c", - "metadata": {}, - "source": [ - "For evaluation, we evaluate the trained model with conventional ranking metrics, including Precision@K and Recall@K. PECOS also provides the evaluation interface for predicted sparse matrices." - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "id": "4c57da1a", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "prec = 84.07 78.17 72.68 67.79 63.79 60.06 56.63 53.51 50.83 48.33\n", - "recall = 4.97 9.16 12.68 15.60 18.25 20.49 22.40 24.05 25.60 26.95\n" - ] - } - ], - "source": [ - "from pecos.utils import smat_util\n", - "metrics = smat_util.Metrics.generate(Y_tst, Y_pred, topk=10)\n", - "print(metrics)" - ] - }, - { - "cell_type": "markdown", - "id": "32908310", - "metadata": {}, - "source": [ - "### Dive Deep in Cluster Chain\n", - "\n", - "Specifically, PECOS trains a *cluster_chain* of `D` matching matrices `C[d]`, where `C[d]` is a CSC matrix of shape `(L[d], K[d])`; `L[d]` and `K[d]` are the numbers of labels and clusters in the layer `d`. Note that the clusters of a layer would be the labels of the next layer. The labels of the last layer `L[D - 1]` would be the labels of the overall XMR problem `nr_labels`." - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "id": "6b0cb55e", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "4 layers in the trained hierarchical label tree with C[d] as:\n", - "cluster_chain[0] is a csc matrix of shape (8, 1)\n", - "cluster_chain[1] is a csc matrix of shape (64, 8)\n", - "cluster_chain[2] is a csc matrix of shape (512, 64)\n", - "cluster_chain[3] is a csc matrix of shape (30938, 512)\n" - ] - } - ], - "source": [ - "print(f\"{len(cluster_chain)} layers in the trained hierarchical label tree with C[d] as:\")\n", - "for d, C in enumerate(cluster_chain):\n", - " print(f\"cluster_chain[{d}] is a {C.getformat()} matrix of shape {C.shape}\")" - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "id": "55eeb4e5", - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbgAAAEyCAYAAACI4cUNAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAA3aElEQVR4nO3debhcVZnv8e+PMIU5MQjpYDgBmb2CMQwKQpQZh9BeQUAREMQBWnAkIBfQtvuJqKjdKDQgQ2gFQUSC0BBAZGgFkmAIswmYSGJIQKaEIUB47x9rFSlO6pxTp3aN5/w+z1NPdq091Lt37dR79tprr6WIwMzMbKBZpdUBmJmZNYITnJmZDUhOcGZmNiA5wZmZ2YDkBGdmZgOSE5yZmQ1ITnBmZjYgOcGZtYCkuZL2anUc3Uk6TNI8SS9K+q2k4a2OyaxWTnBmg5CkVSuUbQf8F3A4sBHwEvCzJodmVjdOcGZtRNIwSb+T9JSkZ/P0JnneQZJmdFv+q5KuydNrSPqBpL9JWiTpXElD87zxkuZLOknSk8BFFT7+U8C1EXF7RCwF/h/wcUnrNnSnzRrECc6svaxCSj6bAqOBl4Gz87wpwBhJ25QtfzgwOU9PArYEdgDeCYwCTitbdmNgeN72sRU+ezvgvtKbiHgMeDVv06zjOMGZtZGI+EdEXBURL0XEEuDfgD3yvGXAr4BPw5tVil3A7ySJlLS+EhHP5HX/HTikbPNvAKdHxLKIeLnCx68DPN+t7HnAV3DWkVaqhzez1pG0FvAjYD9gWC5eV9KQiFgOXAJcJulU0tXbFRGxTNLbgbWAGSnXpc0BQ8o2/1REvNLLxy8F1utWth6wpMg+mbWKr+DM2svXgK2AnSNiPWD3XC6AiLiLVG34AeAw4NI8/2lSdeZ2EbFBfq0fEeuUbbuvoUMeBLYvvZG0GbAG8Jdiu2TWGk5wZq2zmqQ1y16rkqoDXwaey030T6+w3mTSfbnXIuJOgIh4Azgf+FG+mkPSKEn79iOeXwAflfQBSWsD3wF+k6s7zTpOVQlO0q75hEfSpyWdJWnTxoZmNuBdT0pmpdcZwI+BoaQrsruAGyqsdynwLuC/u5WfBMwB7pL0AnAz6WqwKhHxIPAFUqJbTEq2X6p2fbN2o2oGPJU0i1R18W7gYuAC4OCI2KOh0ZnZSnLT/8XA2IiY3ep4zNpVtVWUr0fKhBOAsyPip7hllVmrfBGY5uRm1rtqW1EukXQyqdXWByStAqzWuLDMrBJJc0kNTg5sbSRm7a/aKsqNSS22pkXEHZJGA+MjYnIfq5qZmbVEVQkOIDcq2SIibs7P6gxx6yozM2tXVVVRSvocqZeE4cDmpC6AzgX2bFxotRsxYkR0dXW1OgwzM2uwGTNmPB0RG1aaV+09uOOAnYC7ASJidulZm3bU1dXF9OnTWx2GmZk1mKR5Pc2rNsEti4hXS10A5QdSq6vbNLOm6Jp4XatDqNrcSR9udQg2CFT7mMBtkk4BhkraG7gSuLZxYZmZmRVT7RXcROBo4H7g86QeGC5oVFBmNrD5atOaodoENxS4MCLOB5A0JJe91KjAzMzMiqi2ivIWUkIrGUrq587MzKwtVZvg1sxD2AOQp9dqTEhmZmbFVZvgXpQ0tvRG0ntJvZ+bmZm1pWrvwZ0IXCnp76R+8DYGPtmooMzMzIqqKsFFxDRJW7NibKlHI+K1xoVlZmZWTLVXcAA7Al15nbGScGfLZmbWrqrti/JSUh+UM4HluTgAJzgzM2tL1V7BjQO2jWqHHjAbIDrpgWQze6tqW1E+QGpYUjVJF0paLOmBsrLhkm6SNDv/OyyXS9J/SJojaVa3FptH5OVnSzqiPzGYmdngVW2CGwE8JOlGSVNKrz7WuRjYr1vZROCWiNiC9PD4xFy+P7BFfh0LnAMpIQKnAzuTRjM4vZQUzczMelNtFeUZ/d1wRNwuqatb8QRgfJ6+BPgDcFIun5yrQO+StIGkkXnZmyLiGQBJN5GS5mX9jcfMzAaXah8TuK1On7dRRCzM008CG+XpUcATZcvNz2U9la9E0rGkqz9Gjx5dp3DNbLDrtPuw7hx6haqqKCXtImmapKWSXpW0XNILRT44X63VrdFKRJwXEeMiYtyGG1Yc3NXMzAaRau/BnQ0cCswmdbR8DPDTGj5vUa56JP+7OJcvAN5RttwmuayncjMzs15Vm+CIiDnAkIhYHhEXsXIDkmpMAUotIY8Arikr/0xuTbkL8HyuyrwR2EfSsNy4ZJ9cZmZm1qtqG5m8JGl1YKakM4GF9JEcJV1GaiQyQtJ8UmvIScAVko4G5gEH58WvBw4A5pDGmDsKICKekfSvwLS83HdKDU7MzMx6U22CO5yU0I4HvkKqNvx4bytExKE9zNqzwrIBHNfDdi4ELqwyTjMzM6D6KsoDI+KViHghIr4dEV8FPtLIwMzMzIqoNsFV6kHkyDrGYWZmVle9VlFKOhQ4DBjTreeS9QDfCzMzs7bV1z24P5IalIwAflhWvgSY1aigzMzMiuo1wUXEPGCepL2AlyPiDUlbAlsD9zcjQBt4Oq1nCDPrTNXeg7sdWFPSKGAqqVXlxY0KyszMrKhqE5wi4iXSowE/i4iDgO0aF5aZmVkxVSc4Se8DPgWU6peGNCYkMzOz4qpNcCcAJwNXR8SDkjYDbm1cWGZmZsVUO1zO7aT7cKX3jwNfblRQZmZWm05qxNXooX2qSnC55eTXga7ydSLiQ40Jy8zMrJhq+6K8EjgXuABY3rhwrFad9FebmVkzVJvgXo+IcxoaSR8k7Qf8hNS45YKImNTKeMzMrL1V28jkWklfkjRS0vDSq6GRlZE0hDTA6v7AtsChkrZt1uebmVnnqfYKrtTZ8jfKygLYrL7h9GgnYE5u3IKky4EJwENN+nwzM+sw1baiHNPoQPowCnii7P18YOfyBSQdCxyb3y6V9GgdPncE8HQdttNK3of2MBD2AQbGfngf2oS+V5f92LSnGX2NJtDXoKa/qTWieouI84Dz6rlNSdMjYlw9t9ls3of2MBD2AQbGfngf2kej96OvK7iP9jIvgGYluAWkUcRLNsllZmZmFfU1msBRAJJOjYjv5uk1ImJZM4IrMw3YQtIYUmI7hDROnZmZWUW9tqKUdFLug/ITZcV/amxIK4uI14HjgRuBh4ErIuLBJnx0Xas8W8T70B4Gwj7AwNgP70P7aOh+KCJ6nilNAPYAjgHuAx4B9gH2iYh6NOIwMzNriL4S3B7A3aSRvXcEtiGNJvB7YKuIeH8zgjQzM+uvvhqZ7AucBmwOnAXMAl4s3ZszMzNrV73eg4uIUyJiT2AucCmpm6wNJd0p6domxNcSki6UtFjSA62OpRaS3iHpVkkPSXpQ0gmtjqkWktaUdI+k+/J+fLvVMdVK0hBJf5b0u1bHUgtJcyXdL2mmpOmtjqdWkjaQ9GtJj0h6OLcx6BiStsrfQen1gqQTWx1Xf0n6Sv4//YCkyySt2ZDP6a2KsiyYMyPim3n6zxHxHkkjIqLjHzSsRNLuwFJgckS8q9Xx9JekkcDIiLhX0rrADODAiOionl8kCVg7IpZKWg24EzghIu5qcWj9JumrwDhgvYj4SKvj6S9Jc4Fxnf5/XtIlwB0RcYGk1YG1IuK5FodVk9yF4QJg54iY1+p4qiVpFOn/8rYR8bKkK4DrI+Lien9WVX1RlpJbdmQu6+gTvTd5/LtnWh1HrSJiYUTcm6eXkFqejmptVP0XydL8drX86vsvsjYjaRPgw6TROKxFJK0P7A78HCAiXu3U5JbtCTzWScmtzKrAUEmrAmsBf2/Eh1Tb2fKbIuK+RgRijSGpC3gPqbFQx8lVezOBxcBNEdGJ+/Fj4JvAGy2Oo4gApkqakbvF60RjgKeAi3J18QWS1m51UAUcAlzW6iD6KyIWAD8A/gYsBJ6PiKmN+Kx+JzjrHJLWAa4CToyIF1odTy0iYnlE7EDqvWYnSR1VZSzpI8DiiJjR6lgK2i0ixpJG9DguV+N3mlWBscA5EfEe4EVgYmtDqk2uXv0YaazOjiJpGKmz/DHAPwFrS/p0Iz7LCW6AyvesrgJ+0U59htYqVyXdCuzX4lD6a1fgY/ke1uXAhyT9d2tD6r/8VzcRsRi4mjTCR6eZD8wvqwX4NSnhdaL9gXsjYlGrA6nBXsBfI+KpiHiN1OVjQx45c4IbgHLjjJ8DD0fEWa2Op1aSNpS0QZ4eCuxN6mygY0TEyRGxSUR0kaqUfh8RDflrtVEkrZ0bK5Gr9PYBOq6FcUQ8CTwhaatctCedO+TWoXRg9WT2N2AXSWvl36o9Se0E6s4JrgJJl5G6JNtK0nxJR7c6pn7aFTicdLVQak58QKuDqsFI4FZJs0j9kd4UER3ZzL7DbQTcKek+4B7guoi4ocUx1epfgF/kc2oH4N9bG07/5T8y9qZ5nd3XVb6C/jVwL3A/KQ81pMuuqh4TMDMz6zS+gjMzswHJCc6sBXLPIHu1Oo5ykkZKmiLp75IiP2Ji1rGc4MwGofyAbXdvADcA/7fJ4Zg1hBOcWRuRNEzS7yQ9JenZPL1JnneQpBndlv+qpGvy9BqSfiDpb5IWSTo3tz5F0vjcYOokSU8CF3X/7IhYFBE/IzXoMet4TnBm7WUVUvLZFBgNvAycnedNAcZI2qZs+cOByXl6ErAlqXXgO0nds51WtuzGwPC87U7tjcSsam5FadYC+cHvYyLi5j6W2wG4NSKG5ffnAM9ExLckbUfqtHZj4FVSB+HvjojH8rLvA34ZEWMkjQemkjp7fqWPz1wVeA0YExFza91Hs1brazw4M2siSWsBPyL12DIsF68raUhELAcuAS6TdCrp6u2KiFgm6e2kTmtnpGdn0+ZIQ1yVPNVXcjMbSFxFadZevgZsRRoCZT1S7/eQkhV5qKBXgQ8Ah5HGaQR4mlSduV1EbJBf60fEOmXbdnWNDSpVJThJu5Z63Zb0aUlnSdq0saGZDXirKQ3qWnqtCqxLSlTPSRoOnF5hvcmk+3KvRcSdABHxBnA+8KN8NYekUZL27U9AeeDJNfLbNRo1EKVZM1R7BXcO8JKk7Ul/YT7GihvbZlab60nJrPQ6gzS0zlDSFdldpGb73V0KvAvo3mnzScAc4C5JLwA3k64G++Nl0r08SP1+vtzP9c3aRrUjet8bEWMlnQYsiIifl8oaH6KZlctN/xcDYyNidqvjMWtX1TYyWSLpZNJN7Q9IWoU0urKZNd8XgWlObma9qzbBfZJ0Q/uzEfGkpNHA9xsXlplVkh8vEHBgayMxa39VPweXG5VsERE356bMQyJiSUOjMzMzq1G1rSg/Rxq/579y0Sjgtw2KyczMrLBqqyiPIw1RfzdARMwuNUVuRyNGjIiurq5Wh2FmZg02Y8aMpyNiw0rzqk1wyyLi1VIPCfl5nbZ9aLSrq4vp06e3OgyzpuqaeF1Dtjt30ocbsl2zepA0r6d51T4Hd5ukU4ChkvYGrgSurUdwZmZmjVBtgpsIPAXcD3ye9IDqqY0KyszMrKhqqyiHAhdGxPkAkobkspcaFZiZmVkR1V7B3UJKaCVDSd0AmZmZtaVqE9yaEVHqn448vVZvK0i6UNJiSQ+UlQ2XdJOk2fnf0hhXkvQfkuZImiVpbNk6R+TlZ0s6on+7Z2Zmg1W1Ce7FbknnvfTdCevFpDGtyk0EbomILUhXhRNz+f7AFvl1LKlzZ8p6U9+Z9JjC6aWkaGZm1ptq78GdCFwp6e+kboI2JnXf1aOIuF1SV7fiCcD4PH0J8AdSD+gTgMmRulW5S9IGkkbmZW+KiGcAJN1ESpqXVRm3mZkNUlUluIiYJmlrVgy98WhEvFbD520UEQvz9JPARnl6FPBE2XLzc1lP5WZmZr2q9goOYEegK68zVhIRUfOYcBERkur2sLikY0nVm4wePbpemzUzsw5VVYKTdCmwOTATWJ6Lg/4PerpI0siIWJirIBfn8gXAO8qW2ySXLWBFlWap/A+VNhwR5wHnAYwbN65te1mxxvS44d42zKy7aq/gxgHbRrVDD/RsCnAEMCn/e01Z+fGSLic1KHk+J8EbgX8va1iyD3BywRjMzGwQqDbBPUBqWLKwrwVLJF1GuvoaIWk+qTXkJOAKSUcD84CD8+LXAwcAc0gPjx8FEBHPSPpXYFpe7julBidmZma9qTbBjQAeknQPsKxUGBEf62mFiDi0h1l7Vlg2SCMWVNrOhcCFVcZpZmYGVJ/gzmhkEGZmZvVW7WMCtzU6EDMzs3qqdkTvXSRNk7RU0quSlkt6odHBmZmZ1ararrrOBg4FZpM6Wj4G+GmjgjIzMyuq6ge9I2KOpCERsRy4SNKfcZN9G+AaNUq2mTVetQnuJUmrAzMlnUl6XKDaqz8zM7OmqzbBHU5KaMcDXyH1OvLxRgVlZu3DPc9Yp6o2wR0YET8BXgG+DSDpBOAnjQrMrD9clWhm3VWb4I5g5WR2ZIUyM7M+NeoPEl8ZWrleE5ykQ4HDgDGSppTNWg9wl1lmZta2+rqC+yOpQckI4Idl5UuAWY0KysysXfhqs3P1muAiYh4wT9JewMsR8YakLYGtgfubEaCZmVktqm3qfzuwpqRRwFRSq8qLGxWUmZlZUdU2MlFEvJSHuflZRJwpaWYD4zIz6ze3prVy1V7BSdL7gE8BpTNoSGNCMjMzK67aK7gTSN1yXR0RD0raDLi1cWFZO/Bfw2bWyaodLud20n240vvHgS83KigzM7OiqkpwueXk14Gu8nUi4kONCcvMzKyYaqsorwTOBS4AljcuHKuVqxPNzN6q2gT3ekSc09BI+iBpP1LXYEOACyJiUivjMTMrwg+QN161Ce5aSV8CrgaWlQojoinddUkaQhpgdW9gPjBN0pSIeKgZn19PvtIyM2uO/nS2DPCNsrIANqtvOD3aCZiTG7cg6XJgAtBxCc7MrJE8vNEK1baiHNPoQPowCnii7P18YOfyBSQdCxyb3y6T9ECTYitqBPB0q4Poh06K17E2hmNtjLaNVd9bqaidYt20pxl9jSbQ66CmEfGbWiOqt4g4DzgPQNL0iBjX4pCq0kmxQmfF61gbw7E2hmOtv76u4D7ay7wAmpXgFpBGES/ZJJeZmZlV1NdoAkcBSDo1Ir6bp9eIiGW9rdcA04AtJI0hJbZDSOPUmZmZVdRrX5SSTsp9UH6irPhPjQ1pZRHxOnA8cCPwMHBFRDzYyyrnNSWw+uikWKGz4nWsjeFYG8Ox1pkioueZ0gRgD+AY4D7gEWAfYJ+IeLQpEZqZmdWgrwS3B3A3aWTvHYFtSKMJ/B7YKiLe34wgzczM+quvRib7AqcBmwNnAbOAF0v35szMzNpVr/fgIuKUiNgTmAtcSuoma0NJd0q6tgnx9UrSfpIelTRH0sQK89eQ9Ks8/25JXS0IE0nvkHSrpIckPSjphArLjJf0vKSZ+XVaK2LNscyVdH+OY3qF+ZL0H/m4zpI0thVx5li2KjtmMyW9IOnEbsu07NhKulDS4vLnMiUNl3STpNn532E9rHtEXma2pCMqLdOEWL8v6ZH8PV8taYMe1u31nGlSrGdIWlD2PR/Qw7q9/m40KdZflcU5t6cBpFtwXCv+VrXrOduniOjzBZxZNv3n/O+IatZt1IuUbB8j9aayOuke4bbdlvkScG6ePgT4VYtiHQmMzdPrAn+pEOt44HetPKZlsczt7fsFDgD+BxCwC3B3q2MuOyeeBDZtl2ML7A6MBR4oKzsTmJinJwLfq7DecODx/O+wPD2sBbHuA6yap79XKdZqzpkmxXoG8PUqzpFefzeaEWu3+T8ETmuT41rxt6pdz9m+XlWN6B0R3yx7e2Qua/VT7G923xURrwKl7rvKTQAuydO/BvaUpCbGCEBELIyIe/P0ElJL0FHNjqOOJgCTI7kL2EDSyFYHBewJPBYR81odSEmksRS799lafl5eAhxYYdV9gZsi4pmIeBa4CdivUXFC5VgjYmqkVswAd5GeQW25Ho5rNar53air3mLNv0cHA5c1MoZq9fJb1ZbnbF+qSnDlIuK+RgRSg0rdd3VPGm8uk/+TPg+8rSnR9SBXk76H1Hinu/dJuk/S/0jarrmRvUUAUyXNUOoCrbtqjn0rHELPPxTtcmwBNoqIhXn6SWCjCsu04zH+LOnKvZK+zplmOT5Xp17YQzVaux3XDwCLImJ2D/Nbdly7/VZ15Dnb7wRntZO0DnAVcGJEvNBt9r2kqrXtgf8Eftvk8MrtFhFjgf2B4yTt3sJYqiJpdeBjpLELu2unY/sWkep2em7K3CYkfQt4HfhFD4u0wzlzDqlB3A7AQlLVX7s7lN6v3lpyXHv7reqUcxY6O8FV033Xm8tIWhVYH/hHU6LrRtJqpBPmF1GhD8+IeCEilubp64HVJI1ocpilWBbkfxeThkjaqdsi7dh12v7AvRGxqPuMdjq22aJSlW7+d3GFZdrmGEs6EvgI8Kn847aSKs6ZhouIRRGxPCLeAM7vIYZ2Oq6rAh8HftXTMq04rj38VnXUOVvSyQnuze678l/vhwBTui0zhRVD/XwC+H1P/0EbKdez/xx4OCLO6mGZjUv3ByXtRPpump6MJa0tad3SNKmRQfeRGaYAn1GyC/B8WfVFq/T4l3C7HNsy5eflEcA1FZa5EdhH0rBc1bZPLmsqpYGGvwl8LCJe6mGZas6Zhut2H/ife4ihmt+NZtkLeCQi5lea2Yrj2stvVcecs2/RyhYuRV+k1nx/IbWK+lYu+w7pPyPAmqQqqznAPcBmLYpzN9Il/SxgZn4dAHwB+EJe5njgQVKrrruA97co1s1yDPfleErHtTxWkQagfQy4HxjX4vNgbVLCWr+srC2OLSnpLgReI92TOJp0H/gWYDZwMzA8LzuONFp9ad3P5nN3DnBUi2KdQ7qvUjpvS62S/wm4vrdzpgWxXprPx1mkH+SR3WPN71f63Wh2rLn84tI5WrZsq49rT79VbXnO9vXqtScTMzOzTtXJVZRmZmY9coIza4HcQ8VerY6jnKQPK/VS9JykJyVdULoHZNaJnODMBqHcgq+79YHvku4DbUN6hun7zYzLrJ6c4MzaSG6B9jtJT0l6Nk9vkucdJGlGt+W/KumaPL2GpB9I+pukRZLOlTQ0zxsvab7SGI9PAhd1/+yI+GVE3BARL0XqieJ8YNeG77RZgzjBmbWXVUjJZ1NgNPAycHaeNwUYI2mbsuUPBybn6UnAlqQHnd9JugIr71h6Y1I/gZsC1fSKsTup9Z5ZR3IrSrMWkDQXOCYibu5juR2AWyNiWH5/DvBMRHwrdzl2JylxvQosBd4dEY/lZd8H/DIixkgaD0wF1ouIV6qIb2/gCmDniPhLTTtp1mJ9jQdnZk0kaS3gR6ROakt9Ka4raUhELCd1dHuZpFNJV29XRMQySW8H1gJmlPUnLlLv+SVPVZncdgF+CXzCyc06masozdrL14CtSFdO65GqCSElKyKN3vAqqZPew0gPNwM8TarO3C4iNsiv9SNinbJt91ldI+k9pKrQz0bELfXYIbNWcYIza53VJK1Z9lqVNAbXy8BzkoYDp1dYbzLpvtxrEXEnQKzof/FH+WoOSaMk7VttMJLeBdwA/EtEtHxAY7OiCiU4SbvmPtKQ9GlJZ0natD6hmQ1415OSWel1BvBjYCjpiuwuUsLp7lLgXcB/dys/idRF0l2SXiB1qbRVP+L5GrAh8HNJS/PLjUysYxVqZCJpFrA98G5Sv2oXAAdHxB51ic7MVpKb/i8mjbzc0zhiZoNe0SrK1yNlyAnA2RHxU1IVi5k1zheBaU5uZr0r2opyiaSTSa25PiBpFWC14mGZWSX58QIBB7Y2ErP2V7SKcmNSS65pEXGHpNHA+IiY3MeqZmZmDVX4Qe/cqGSLiLg5P8MzJCKW1CU6MzOzGhWqopT0OVKXP8OBzUldA50L7Fk8tNqNGDEiurq6WhmCmZk1wYwZM56OiA0rzSt6D+44YCfgboCImF16BqeVurq6mD59eqvDMDOzBpM0r6d5RRPcsoh4tdQ1UH5Q1Z1bmrVA18TrGrLduZM+3JDtmjVa0ccEbpN0CjA0d856JeAeEMzMrOWKXsFNBI4G7gc+T+qZ4YKiQZkNdI262uoUvtq0Ziia4IYCF0bE+QCShuSyl4oGZmZmVkTRKspbSAmtZCip/zszM7OWKprg1oyIpaU3eXqtgts0MzMrrGgV5YuSxkbEvQCS3kvqFd2sqXxPx8y6K5rgTgSulPR3Uv94GwOfLBqUmZlZUYUSXERMk7Q1K8acejQiXiselpmZWTFFr+AAdgS68rbGSsKdLZuZWasV7YvyUlIflDOB5bk4ACc4MzNrqaJXcOOAbaPokARmZmZ1VvQxgQdIDUvMzMzaStEruBHAQ5LuAZaVCiPiY72tlEclXkKq1nw9IsZJGg78inQ/by5wcEQ8q9ST80+AA0g9pBxZeizBzKzR/AhK5yqa4M4osO4HI+LpsvcTgVsiYpKkifn9ScD+wBb5tTNwTv7XzMysR0UfE7itXoEAE4DxefoS4A+kBDcBmJzv890laQNJIyNiYR0/28zMBphC9+Ak7SJpmqSlkl6VtFzSC1WsGsBUSTMkHZvLNipLWk8CG+XpUcATZevOz2VmZmY9KlpFeTZwCGkcuHHAZ4Atq1hvt4hYkEf/vknSI+UzIyIk9atlZk6UxwKMHj26P6uamdkAVLQVJRExBxgSEcsj4iJgvyrWWZD/XQxcDewELJI0EiD/uzgvvgB4R9nqm+Sy7ts8LyLGRcS4DTfcsMgumZnZAFD0Cu4lSasDMyWdCSykj6QpaW1glYhYkqf3Ab4DTAGOACblf6/Jq0wBjpd0OalxyfO+/9Ycbj3mgUnBx8A6V9EEdzgpoR0PfIV0pfXxPtbZCLg6tf5nVeCXEXGDpGnAFZKOBuYBB+flryc9IjCH9JjAUQVjNjOzQaBogjswIn4CvAJ8G0DSCaTn1iqKiMeB7SuU/wPYs0J5AMcVjNPMBgFfbVq5ovfgjqhQdmTBbZqZmRVW0xWcpEOBw4AxkqaUzVoPeKYegdnA5b+yzawZaq2i/COpQckI4Idl5UuAWUWDMjMzK6qmBBcR84B5kvYCXo6INyRtCWwN3F/PAM3MzGpR9B7c7cCakkYBU0mtKi8uGpSZmVlRRROcIuIl0qMBP4uIg4DtiodlZmZWTNHHBCTpfcCngKNz2ZCC2zQzG/DckULjFb2COwE4Gbg6Ih6UtBlwa/GwzMzMiik6XM7tpPtwpfePA18uGpSZmVlRhRJcbjn5ddIo3G9uKyI+VCws6y8/W2Zm0Jjfgk6t9ix6D+5K4FzgAmB58XB6Jmk/UhdgQ4ALImJSIz/PzMw6W9EE93pEnFOXSHohaQjwU2Bv0oCn0yRNiYiHGv3Z9eYrLTOz5ijayORaSV+SNFLS8NKrLpG91U7AnIh4PCJeBS4HJjTgc8zMbIAoegVX6mz5G2VlAWxWcLvdjQKeKHs/nzQ23JvKR/QGlkp6tGz2CODpOsfU6XxMVuZjsjIfk5UNumOi7/W5SCuPyaY9zSjainJMkfXrKSLOA86rNE/S9IgY1+SQ2pqPycp8TFbmY7IyH5OVtesxqXU0gV4HNY2I39QWTo8WkAZTLdkkl5mZmVVU6xXcR3uZF0C9E9w0YAtJY0iJ7RDScD1mZmYV1TqawFEAkk6NiO/m6TUiYlk9gyv7vNclHQ/cSHpM4MKIeLAfm6hYdTnI+ZiszMdkZT4mK/MxWVlbHhNFRP9Xkk4i9WByTkTskMvujYix9Q3PzMysNrVWUT4CHARsJumO/P5tkraKiEd7X9XMzKzxar2C2wO4mzSy947ANsB1wO+BrSLi/fUM0szMrL9qfdB7X1JC2xw4i/RM2osRcVQ7JTdJ+0l6VNIcSRNbHU87kDRX0v2SZkqa3up4WkXShZIWS3qgrGy4pJskzc7/DmtljM3WwzE5Q9KCfL7MlHRAK2NsJknvkHSrpIckPSjphFw+aM+TXo5JW54nNV3BvbmydB9pHLixwL8BjwLPRkRvrSybInfv9RfKuvcCDu3E7r3qSdJcYFxEDKoHVbuTtDuwFJgcEe/KZWcCz0TEpPwH0bCIOKmVcTZTD8fkDGBpRPyglbG1gqSRwMiIuFfSusAM4EDgSAbpedLLMTmYNjxPinbVdWNETM8PWc+PiN2Ao+oQVz24ey/rUR7q6ZluxROAS/L0JaT/uINGD8dk0IqIhRFxb55eAjxM6lVp0J4nvRyTtlQowUXEN8veHpnL2uXKoFL3Xm37RTRRAFMlzcjdm9kKG0XEwjz9JLBRK4NpI8dLmpWrMAdNdVw5SV3Ae0htD3yesNIxgTY8T4pewb0pIu6r17asoXbLj3PsDxyXq6Wsm0h197XX3w8c55Dute8ALAR+2NJoWkDSOsBVwIkR8UL5vMF6nlQ4Jm15ntQtwbUhd+9VQUQsyP8uBq4mVeVasijfYyjda1jc4nhaLiIWRcTyiHgDOJ9Bdr5IWo30Q/6Lsi4IB/V5UumYtOt5MpAT3Jvde0landS915QWx9RSktbON4aRtDawD/BA72sNKlNYMULGEcA1LYylLZR+yLN/ZhCdL5IE/Bx4OCLOKps1aM+Tno5Ju54nhVpRtrvcVPXHrOje699aG1FrSdqMdNUG6SH/Xw7WYyLpMmA8aZiPRcDpwG+BK4DRwDzg4IgYNI0uejgm40nVTgHMBT5fdv9pQJO0G3AHcD/wRi4+hXTPaVCeJ70ck0Npw/NkQCc4MzMbvAZyFaWZmQ1iTnBmLZB7lNmr1XGUk/TB3MvNc5L+IelqSX60xjqWE5zZICSpUkfrDwH7RsQGwD8Bs0nNv806khOcWRuRNEzS7yQ9JenZPL1JnneQpBndlv+qpGvy9BqSfiDpb5IWSTpX0tA8b7yk+ZJOkvQkcFH3z85Nvf9eVrQceGfDdtaswZzgzNrLKqTksympld7LwNl53hRgjKRtypY/HJicpycBW5Jas72T1HPPaWXLbgwMz9uu2IuNpNGSnsuf+3XgzKI7ZNYqbkVp1gK50+tjIuLmPpbbAbg1Iobl9+eQOvr9lqTtgDtJietVUkfJ746Ix/Ky7yM9CjJG0nhgKrBeRLxSRXzDgc8Bt0XEXTXtpFmL1TrgqZk1gKS1gB8B+wGl/vzWlTQkIpaTOve9TNKppKu3KyJimaS3A2sBM9KzuGlzpGdAS56qJrkBRMQzki4B7pM0KiJeL7xzZk3mKkqz9vI1YCtg54hYDyj1FSqAfDX1KvAB4DDg0jz/aVK14nYRsUF+rR8R65Rtu7/VNasCbwfWq2lPzFrMCc6sdVaTtGbZa1VgXVKiei5XE55eYb3JpPtyr0XEnQBlfQD+KF/NIWmUpH2rDUbSxyVtJWkVSRuSBjP+82DppcMGnkIJTtKuuU9DJH1a0lmSNq1PaGYD3vWkZFZ6nUHqWm4o6YrsLuCGCutdCrwL+O9u5ScBc4C7JL0A3Ey6GqzWqPx5S1jRFdM/92N9s7ZSdETvWcD2wLuBi4ELSP2y7VGX6MxsJbnp/2JgbETMbnU8Zu2qaBXl63k8pAnA2RHxU1IVi5k1zheBaU5uZr0r2opyiaSTSa25PiBpFWC14mGZWSX58QIBB7Y2ErP2V7SKcmNSS65pEXGHpNHA+IiY3MeqZmZmDVX4Qe/cqGSLiLg5P8MzJCKW1CU6MzOzGhWqopT0OVKXP8OBzUmtsM4F9iweWu1GjBgRXV1drQzBzMyaYMaMGU9HxIaV5hW9B3ccsBNphFsiYnbpGZxW6urqYvr06a0Ow8zMGkzSvJ7mFU1wyyLi1VLXQPlBVXduaWbWh66J1zVku3Mnfbgh2+1ERR8TuE3SKcBQSXsDVwLXFg/LzMysmKIJbiLwFKnXg8+TemY4tWhQZmZmRRWtohwKXBgR5wNIGpLLXioamJmZ9V8jqj47tdqz6BXcLaSEVjKU1P+dmZlZSxVNcGtGxNLSmzy9VsFtmpmZFVa0ivJFSWMj4l4ASe8l9YpuNiB0Uku3TorVrBmKJrgTgSsl/Z3UP97GwCeLBmVm1i4a9YeDNV6hBBcR0yRtzYoxpx6NiNf6Wi93GLsEWE4akWBcHtzxV0AXMJc07M6zSg/Z/QQ4gNR45cjSFaNZiX+EzKy7eozovSNpPLixwKGSPlPleh+MiB0iYlx+PxG4JSK2IDVemZjL9we2yK9jgXPqELOZmQ1wRfuivJTUB+VM0tUYpJ5MahlNYAIwPk9fAvyBNELxBGByHnfuLkkbSBoZEQtrj9zMquVm59apit6DGwdsG/0fkiCAqZIC+K+IOA/YqCxpPQlslKdHAU+UrTs/lznBmXUoN4ixZiia4B4gNSzpb7LZLSIW5I6Zb5L0SPnMiIic/Kom6VhSFSajR4/uZzhmZjbQFE1wI4CHJN0DLCsVRsTHelspIhbkfxdLupo0IsGiUtWjpJHA4rz4AuAdZatvksu6b/M84DyAcePGucNnM7NBrmiCO6O/K0haG1glIpbk6X2A7wBTgCOASfnfa/IqU4DjJV0O7Aw87/tvZmbWl6KPCdxWw2obAVfnIXZWBX4ZETdImgZcIeloYB5wcF7+etIjAnNIjwkcVSRmaz036TezZijainIX4D+BbYDVgSHAixGxXk/rRMTjwPYVyv9BhZHAcwOW44rEaWZmtevURkFFn4M7GzgUmE3qaPkY4KdFgzIzMyuq8IPeETEHGBIRyyPiImC/4mGZmZkVU7SRyUuSVgdmSjqT9LhAPXpHMTMzK6RoMjo8b+N44EVSc/6PFw3KzMysqKJXcAdGxE+AV4BvA0g6gdQ5spn1wC1JzRqv6BXcERXKjiy4TTMzs8JquoKTdChwGDBG0pSyWesBz9QjMDMzsyJqraL8I6lByQjgh2XlS4BZRYMyM6uFq36tXE0JLiLmAfMk7QW8HBFvSNoS2Bq4v54BmpmZ1aLoPbjbgTUljQKmklpVXlw0KDMzs6KKJjhFxEukRwN+FhEHAdsVD8vMzKyYoo8JSNL7gE8BR+eyIQW3aTXwqMtmZm9VNMGdAJwMXB0RD0raDLi1eFjWDnzD3sw6WdHhcm4n3YcrvX8c+HLRoMzMzIoqOlzOlsDXga7ybUXEh4qFZWZmVkzRKsorgXOBC4DlxcMxMzOrj6IJ7vWIOKcukfRB0n6kPi6HABdExKRmfK6ZmXWmognuWklfAq4GlpUKI6Ku3XVJGkIaSHVvYD4wTdKUiHionp/TDG64YWbWHEUTXKmz5W+UlQWwWcHtdrcTMCc3YkHS5cAEoOMSnJmZNUfRVpRj6hVIH0YBT5S9nw/sXL6ApGOBY/PbpZIebVJszTYCeLrVQTTZYNxnGJz7PRj3GQbpfut7ddnvTXuaUetoAr0OahoRv6llu0VExHnAec3+3GaTND0ixrU6jmYajPsMg3O/B+M+g/e7Uduv9Qruo73MC6DeCW4BabTwkk1ymZmZWUW1jiZwFICkUyPiu3l6jYhY1vuaNZsGbCFpDCmxHUIaj87MzKyimjpblnRS7oPyE2XFf6pPSCuLiNeB44EbgYeBKyLiwUZ9Xpsb8NWwFQzGfYbBud+DcZ/B+90Qioj+ryRNAPYAjgHuAx4B9gH2iYiB2rjDzMw6SK0Jbg/gbtLI3jsC2wDXAb8HtoqI99czSDMzs/6qtZHJvsBpwObAWcAs4MXSvTkzM7NWq+keXEScEhF7AnOBS0ndZ20o6U5J19YxPsskzZV0v6SZkqa3Op5GkXShpMWSHigrGy7pJkmz87/DWhljI/Sw32dIWpC/85mSDmhljPUm6R2SbpX0kKQHJZ2Qywfs993LPg/073pNSfdIui/v97dz+RhJd0uaI+lXklav6+fWUkX55srSmRHxzTz954h4j6QRETHoHlhsNElzgXED/dhK2h1YCkyOiHflsjOBZyJikqSJwLCIOKmVcdZbD/t9BrA0In7QytgaRdJIYGRE3CtpXWAGcCBwJAP0++5lnw9mYH/XAtaOiKWSVgPuJI0n+lXgNxFxuaRzgfvq2b9xTVdwJaXklh2Zywb0D7A1Vh5jsHtfphOAS/L0JaQfhAGlh/0e0CJiYUTcm6eXkFpIj2IAf9+97POAFsnS/Ha1/ArgQ8Cvc3ndv+tCCa5cRNxXr21ZRQFMlTQjd0s2mGwUEQvz9JPARq0MpsmOlzQrV2EOmKq67iR1Ae8hNV4bFN93t32GAf5dSxoiaSawGLgJeAx4Lj8GBqkLxrom+7olOGu43SJiLLA/cFyu0hp0ItWp116v3lnOITXk2gFYCPywpdE0iKR1gKuAEyPihfJ5A/X7rrDPA/67jojlEbEDqSeqnYCtG/2ZTnAdIiIW5H8Xk4Yn2qm1ETXVonzvonQPY3GL42mKiFiUfxTeAM5nAH7n+X7MVcAvyvqwHdDfd6V9HgzfdUlEPAfcCrwP2EBSqTV/3btgdILrAJLWzjekkbQ26aH6B3pfa0CZwoqhmY4ArmlhLE1T+pHP/pkB9p3nhgc/Bx6OiLPKZg3Y77unfR4E3/WGkjbI00NJY3s+TEp0pR6x6v5dF2pFac0haTPSVRukZxd/GRH/1sKQGkbSZcB40vAhi4DTgd8CVwCjgXnAwfUeVLfVetjv8aQqqyA9kvP5sntTHU/SbsAdwP3AG7n4FNI9qQH5ffeyz4cysL/rd5MakQwhXVhdERHfyb9tlwPDgT8Dn65nn8ZOcGZmNiC5itLMzAYkJzgzMxuQnODMzGxAcoIzM7MByQnOzMwGJCc4azhJIemHZe+/njsSrse2L5b0ib6XLPw5B0l6WNKt3cq7JL2ce4C/T9IfJW3Vx7a6ykcNaDalkSlGVChfR9J/SXosdwn3B0k753lLV95SVZ91oKRti8bcbZv/I2mTHN+4KtcZL+l3/fycqrdv7ckJzpphGfDxSj+qrVTWg0I1jgY+FxEfrDDvsYjYISK2Jz3rc0pdAmy+C0gdPm8REe8FjiI9l1fEgUC/Elxv30t+SPhtETG/YFw2CDjBWTO8DpwHfKX7jO5XYKUrhfwX922SrpH0uKRJkj6Vx5S6X9LmZZvZS9J0SX+R9JG8/hBJ35c0LXdg+/my7d4haQrwUIV4Ds3bf0DS93LZacBuwM8lfb+PfV0PeLa3GLp93pqSLsqf+WdJH8zlR0r6jaQblMZFO7NsnaPzvt4j6XxJZ+fyDSVdlT9vmqRdc/nbJE1VGofrAkAV4tgc2Bk4NXcXRUT8NSKu67bcW66EJJ0t6cg8PUlpnLNZkn4g6f3Ax4Dv5yvczfPrhnyFeIekrfO6F0s6V9LdwJmS9tCKsdH+rNyTD+nh9z/0dPDz1fEdku7Nr/eXfzeSrpP0aP6sVfI6+0j6U17+SqV+Isu3OSTH90D+nlY6j6091Tqit1l//RSYVf5DXYXtgW1IVxWPAxdExE5Kg0T+C3BiXq6L1Hff5sCtkt4JfAZ4PiJ2lLQG8L+SpublxwLvioi/ln+YpH8Cvge8l5Skpko6MPe48CHg6xFRabDZzZV6SV8XWIuUKCBd9VWKobx3heNIfQr/n/xjP1XSlnneDqTe5pcBj0r6T2A58P/yPiwBfg+URvL4CfCjiLhT0mjgxnz8TgfuzPvx4RxXd9sBMyNieYV5fZL0NlIXU1tHREjaICKey39I/C4ifp2XuwX4QkTMVqr+/BlpyBRIfRG+PyKWKw2cfFxE/G9OOK/kZfYn9WzTk8XA3hHxiqQtgMuAUjXjTqSryXnADaRahT8ApwJ7RcSLkk4ijVH2nbJt7gCMKhunb4MaDpG1gBOcNUVEvCBpMvBl4OUqV5tW6q5I0mNAKUHdD5RXFV6RrzpmS3qc1Ev5PsC7teLqcH1gC+BV4J7uyS3bEfhDRDyVP/MXwO70/oMKuYoyr/NJ0tXqfr3E8JeydXcD/hMgIh6RNA8oJbhbIuL5vN2HgE1JVYa3lbquknRl2fJ7AdtKb16grZeTw+7Ax/NnXCfp2T72pxbPk5LQz/MV3kr3u3Is7weuLItxjbJFrixLsP8LnJW/g9+UVUnuCny9lzhWA86WtAPpj4Ety+bdExGP51guIx37V0hJ739zTKsDf+q2zceBzfIfGNex4jy0NucEZ830Y+Be4KKystfJVeW5yqh8yPryPuneKHv/Bm89d7v3Nxekarh/iYgby2dIGg+8WEvwVZrCiv3rKYauKrdVvv/L6fv/6yrALhHxSnlhWTLpzYPA9pKG9HEV9+b3la0JEBGvS9oJ2JPUee7xrLgyK4/vudIfAxW8+b3k0byvAw4gJZ99SX+cPBERr/YS31dIfXlunz+v/Fj0dJ7cFBGH9rTBiHhW0vbAvsAXSKNvf7aXGKxN+B6cNU2+6riCt1aRzSVVCUK6X7NaDZs+SNIq+T7SZsCjpOq5LyoNTYKkLZVGYujNPcAekkZIGkLqAPe2fsayG2kgR6qM4Q7gU6X5pA6GH+1l+9NyjMOUGmP837J5U0lVt+Tt7ZAnbwcOy2X7AysNphkRjwHTgW8rZ8R8P+vD3RadR7pKXCNX1e2Zl10HWD8iriclme3z8ktIVbfkcc/+KumgvI5y4liJpM0j4v6I+F7e561J1ZM39HJsIF0lL8xX9IeTOvct2UnSmPyH1CeBO4G7gF1ztXZp5I7yqz6UGketEhFXkaozx/YRg7UJX8FZs/2Q9Nd9yfnANZLuI/141XJ19TdSclqPdH/nFaXGFF3AvfkH+ylSi74eRcRCSRNJQ3gIuC4iqhm+o3QPTqSrjGNyeTUx/Aw4R9L9pKujIyNiWU9XXRGxQNK/5/19BniEVD0Iqfr3p5Jmkf5v30664vg2cJmkB4E/ko5XJceQvp85kl4Gnga+0e3zn5B0BWk4l7+SeoCHlMSukbRmPg5fzeWXA+dL+jLpyu5TeX9PJf0xczkr7iGWO1Gpwc0bpKvL/wF+TVkCz66T9Fqe/hOpBetVkj7DyufTNOBs4J2k7/jqiHhDqZHMZfk+KaQkVl6NPAq4KCdGgJMrxGttyKMJmHUYSetExNJ8BXc1cGFEXN3Xep2s1EgnIvxcmlXNCc6sw0j6AalByZqkaskTwv+RzVbiBGdmZgOSG5mYmdmA5ARnZmYDkhOcmZkNSE5wZmY2IDnBmZnZgPT/AaO8B7imYR5xAAAAAElFTkSuQmCC\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], - "source": [ - "import matplotlib.pyplot as plt\n", - "from pecos.core import clib\n", - "from pecos.utils import smat_util\n", - "\n", - "fig, axes = plt.subplots(nrows=len(cluster_chain), ncols=1)\n", - "fig.tight_layout()\n", - "\n", - "cur_Y = Y_tst\n", - "\n", - "counts, bins = np.histogram(cur_Y.getnnz(1), bins=16) \n", - "ax = plt.subplot(len(cluster_chain), 1, len(cluster_chain))\n", - "ax.hist(bins[:-1], bins, weights=counts)\n", - "ax.set_title(\"Layer {}\".format(len(cluster_chain) - 1))\n", - "plt.ylabel(\"#Instances\")\n", - "\n", - "for d in range(len(cluster_chain) - 1, 0, -1):\n", - " cur_Y = smat_util.binarized(clib.sparse_matmul(cur_Y, cluster_chain[d]))\n", - " counts, bins = np.histogram(cur_Y.getnnz(1), bins=min(16, cluster_chain[d].shape[1])) \n", - " ax = plt.subplot(len(cluster_chain), 1, d)\n", - " ax.hist(bins[:-1], bins, weights=counts)\n", - " ax.set_title(\"Layer {}\".format(d - 1))\n", - " plt.ylabel(\"#Instances\")\n", - " \n", - " \n", - "plt.subplot(len(cluster_chain), 1, len(cluster_chain))\n", - "plt.xlabel(\"Number of Belonged Clusters/Labels\");" - ] - }, - { - "cell_type": "markdown", - "id": "790b21dc", - "metadata": {}, - "source": [ - "### Dive Deep in Model Weights\n", - "\n", - "Model weights in an XR-Linear model are also accessible as `model_chain` for analysis and computations. For the i-th layer in the hierarchy, the model weights of matchers/rankers are available as a CSC matrix of shape `(nr_feat + 1, L[i])`, which concatenates weights for features and the bias term. " - ] - }, - { - "cell_type": "code", - "execution_count": 17, - "id": "9e101f6b", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "model_chain[0].W is a csc matrix of shape (101939, 8)\n", - "model_chain[1].W is a csc matrix of shape (101939, 64)\n", - "model_chain[2].W is a csc matrix of shape (101939, 512)\n", - "model_chain[3].W is a csc matrix of shape (101939, 30938)\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAD8CAYAAACYebj1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAA2RklEQVR4nO3dd5hU5fXA8e+ZvgVYFpbeEUFQQV2x907s0Yg9MQlJLNEk5hdLmhrTTGKKSQyx944SRUXsWFlApEmRIr0sLLvs7tR7fn/MuO6yswXY2TuznM/z7MPMe9+59ywwc+Z971tEVTHGGGMAPG4HYIwxJntYUjDGGFPHkoIxxpg6lhSMMcbUsaRgjDGmjiUFY4wxddokKYjIfSKyUUTm1SsrFpHXRGRJ6s+uTbz28lSdJSJyeVvEY4wxZte0VUvhAeDUHcpuAF5X1WHA66nnDYhIMfAr4BBgLPCrppKHMcaYzGuTpKCq7wBbdig+C3gw9fhB4Ow0Lz0FeE1Vt6jqVuA1GicXY4wx7cSXwXP3VNV1qcfrgZ5p6vQFVtV7vjpV1oiITAAmABQUFBw0YsSINgzVGGM6vpkzZ25W1ZLm6mQyKdRRVRWR3VpPQ1UnAhMBSktLtaysrE1iM8aYPYWIrGypTiZHH20Qkd6pQHoDG9PUWQP0r/e8X6rMGGOMCzKZFCYDX44muhx4IU2dV4GTRaRr6gbzyakyY4wxLmirIamPAx8Aw0VktYh8G/g9cJKILAFOTD1HREpF5B4AVd0C3AbMSP3cmiozxhjjAsnFpbPtnoIxxuw8EZmpqqXN1bEZzcYYY+pYUjDGGFPHkoIxxpg6lhSMMcbUaZfJa8bsSVQdNtTOoTq+nm7BERQFB7sdkjGtZknBmDZUE9/Mq6uvoiZeDiiKQ9/8Qzmm9214xN5uJvtZ95Exbejd9bdQFVtLXGuIay0JjbCm5kMWbH3S7dCMaRVLCsa0kUiiko21n6IkGpQnNMKibZNcisqYnWNJwZg24mgMRNIeS2iknaMxZtdYUjCmjYS8xRT6Gq8Q78HHgMJj2z8gY3aBJQVj2oiIcGTPX+CTPDz4AfBJiDxfd8Z0+7bL0RnTOjYcIkNUlW3RMAX+AH6P1+1wTDspyduXcwY9weJtk6mMrqJn3hiGdD4FvyfP7dCMaRVLChnw4oqF3DpjGlsjYXzi4aK9x3DjQcfh81jDbE+Q7yuxloHJWZYU2tj0tSv46XsvUZuIAxAjwWOLZxNzEtx6yMkuR2eMMc2zr65t7O+fTq9LCF+qTcR5cumnVMeiLkVljDGtY0mhja2sqkhb7hFhS7imfYMxxpidZEmhje3frRfpRqp7EHrkF7Z7PDtDtRaNzkBjn5GLmy8ZY3afJYU29qMxRxHy+RuU5Xn9XDv6CILe7L2F49Q8jW48FN36PXTLBejm09D4KrfDMsa0s4wmBREZLiKf1PupFJHrdqhzrIhsq1fnl5mMKdNGFvfkqVMu5vBeA+nkDzKkczG/PfQUJow6xO3QmqTROVB5G2gt6Pbkn4kV6NYrrMVgzB4mo19dVXURMAZARLzAGiDdIjDvqurpmYylPe3XrRePnXyh22G0mtY8Auy4DIMDziaIfQqB0W6EZYxxQXt2H50AfK6qK9vxmqY1nM1AuhaBB7SinYMxxripPZPCeODxJo4dJiJzRORlERnVjjEZgOBxQKhxucbAb60EY/Yk7ZIURCQAnAk8nebwLGCgqo4G/gE838Q5JohImYiUbdq0KWOx7okk7zzw9qFhYsiDwmsQT5FLURlj3NBeLYXTgFmqumHHA6paqarbU4+nAH4R6Z6m3kRVLVXV0pKSksxHvAcRTz7S7VkovBb8B0DweKTrv/AUTnA7NGNMO2uvMZIX0kTXkYj0AjaoqorIWJKJqryd4jIp4ilACr8NhbZmjzF7sownBREpAE4Cvlev7PsAqno3cB7wAxGJA7XAeLVxkMYY44qMJwVVrQa67VB2d73HdwF3ZToOY4wxLbMZzcYYY+pYUjAmR6mG0dgi1NnidiimA8nexXiMMU1ytv8Xqu8iOcEwhgaPR4r+gIjt8GZ2j7UUjMkxGn4Ztt+VWquqGohC5E102y/cDs10ANZSMCbH6Pb/kByoV18Ewq+gzq8RT/Ys0f7Fuq08MqWMJV9sZMSgnlw8rpR+PYvcDss0w5KCMbnGaWpGvwe0EsiOpDD/83Vc9btniMbiJBxl0YqNvPzeQv7z8wsYPqiH2+GZJlj3kTG5xn8wad+6kgeenu0eTlPuePANaiMxEk5y2lHCUWojMf788BsuR2aaY0nBmBwjna4DyQe89UpD0OlmkivUu09V+Wx5o1VtAJj3+fp2jsbsDEsKxuQY8Q1Cuj0PeWeDdyAEjkC6/hdP/pluh1ZHRAiF/GmP5TdRbrKD3VMwJgeJbwDS5Xduh9Gsc4/bn2emzSESi9eVBQM+zjtxjHtBmRZZS8EYkxE/+MaRHH3QEAJ+L4V5AQJ+L8cfPIzvnH2o26GZZlhLwRiTEX6fl99cdTobt1SxekMFA3p3pXtRdoyMMk2zpGCMyagexZ3oUdzJ7TBMK1n3kTHGmDqWFIwxxtSxpGCMMaaOJQVjjDF1LCkYY4ypY0nBGGNMnYwnBRFZISJzReQTESlLc1xE5O8islREPhWRAzMdkzHGmPTaa57Ccaq6uYljpwHDUj+HAP9O/WmMMaadZUP30VnAQ5r0IVAkIr3dDsoYY/ZE7ZEUFJgqIjNFZEKa432BVfWer06VNSAiE0SkTETKNm1qapMRY4wxu6M9ksKRqnogyW6iq0Tk6F05iapOVNVSVS0tKSlp2wiNMcYA7ZAUVHVN6s+NwCRg7A5V1gD96z3vlyozxhjTzjKaFESkQEQ6ffkYOBmYt0O1ycBlqVFIhwLbVHVdJuMy7tDEJjQ2D3Wq2+d6GsWpfhBn85k4m8/Aqb4f1Wi7XNuYXJXp0Uc9gUki8uW1HlPVV0Tk+wCqejcwBRgHLAVqgG9lOCbTztSpQbddD5F3QAKgcbRwAlJwFan/G21/TVV063chOhsIJwur7kTDr0Pxwxm7rjG5LqNJQVWXAaPTlN9d77ECV2UyDuMurfw5RN4FovDlN/Xq/4J3EOSdnpmLxsogNoe6hADJx/F5EP0Agodn5rrG5LhsGJJqOjB1tkN4KhDZ4UAtWv2fzF04OhM03LhcayE2K3PXNSbHWVIwmaVVNPnfzCnP3HU9JUAozYFQ6pgxJh1LCiazPD3BU5DuAAQyOHE9dAqIt3G5eCE0LnPXNSbHWVIwGSXigU6/pOG3dh9IAVJ4Xeau6ylEih8Gb38gL/nj6YcUP4h4bGtIY5piezSbjPPknYZ6e6Db/wOJ1RAoRQomIL5+Gb2u+EdC92mQWAEoeAfbqCNjWmBJwbQLCRyEFE9s/+uKgG9wu1/XmFxl3UfGGGPqWFIwxhhTx5KCMcaYOpYUjDHG1LGkYIwxpo4lBWPaWLg2ypyy5SxZuJbk0l7G5A4bkmpMG5o6eTZ3/eElvF4PjqMUdyvkN/+4hL4DurkdmjGtYi0FY9rIkoVruesPLxEJx6ipjhCujbJuzVZu+MGDOI7jdnjGtIolBWPayP+e+phYNN6gTFWpqqxlwZxVTbzKmOxiScGYNrJ1SzWO0/gegohQua3WhYiM2XkdOik4jsP2qloSCWu6m8w77JjhBEP+RuXxWIJRo/uneYUx2afD3mie/ORHPHT3m9TWRAmGfIy/4mjOv+wIWxDNZMwJ40Yz+amPWbtqC5FwDIBQyM8FVxxFl67plg83JvtkLCmISH/gIZL7NCswUVX/tkOdY4EXgOWpoudU9dbdvfarL8zinr+/VvfGjG9P8MjEt/D7vZxz0WG7e3rjgrgTpiq2lnxfd4Lezm6Hk1Yw5Oev93+Hl5+fyfRpC+jUJY8zzh/LQYft5XZoxrSaZGoctYj0Bnqr6iwR6QTMBM5W1QX16hwLXK+qO7VRb2lpqZaVlTV5/NKv/YWN67c1Ku9SlM9Tr/9sZy5lXKaqfLrlfuZufRjBi0OcwYUncljPn+GVxl01xpimichMVS1trk7G7imo6jpVnZV6XAUsBPpm6nr1lW+uSlu+raLG7i/kmKWVLzF36yMkNEJca3A0yortr1O26e9uh2ZMh9QuN5pFZBBwAPBRmsOHicgcEXlZREY1c44JIlImImWbNm1q9nr9BnZPW96zdxFeb4e+t97hzN36EAkNNyhLaIQllS+S0JhLURnTcWX8E1JECoFngetUtXKHw7OAgao6GvgH8HxT51HViapaqqqlJSXNb7z+3etOJhhseLskGPLz3etO3oXfwLgpnKhIW66aIO7YMM/amgiLF6xh88Yd31rG7JqMjj4SET/JhPCoqj634/H6SUJVp4jIv0Sku6pu3p3rHnz4MH5950Xc949prF65md79ivnmlSdwyFF7785pjQtKQiNZW/Nxo/KQt5jAHr7X8hP3vcNj97yN1+clFkswpnQwN/7uPAoKQy2/2JgmZHL0kQD3AgtV9S9N1OkFbFBVFZGxJFsu5W1x/QMPGcqBhwxti1MZFx3U/Wo2rvoeCY2gJO8HeSXI2JIf79HDi9+dNp/H7n2HSCQOkeQs6k9mLONPv5rEr/58ocvRmVyWyZbCEcClwFwR+SRVdhMwAEBV7wbOA34gInGgFhivtqykqac4uBdfG3Avn5bfz+bwAjoF+rF/8TfpmTfa7dBc9dSD79UNuf5SLJZgxvtLqNxWQ+cu+S5FZnJdxpKCqk4Hmv0qp6p3AXdlKgbTMRQFBnF071vcDiOrbN2yPW251+uhqrLWkoLZZTYUx5gcdMDYIXi8jb9z+f0+evUuav+ATIdhScGYHHTJhGPJzw/i9X31Fg6G/Fz5f6fh9XldjMzkug679pExHVnP3kXc/eSVPP3Qe8yZsZyefYr4xuVHsu8BA90OzeQ4SwrG5KiSnl248qfj3A7DdDDWfWSMMaaOJQXTYTiqODai2ZjdYt1HJuetr6nipg9e4e21ywA4ru9Qbj/0FHrm79kznuuLJRIs3VhOp1CQfl27uB2OyWKWFFxWvqmKtavK6dO/G91K7ENsZ0UScc6e8hCbareTSLUS3lzzOee+/DBvnfM9/B4bifPyvEX88sXXcRwl7jgM69GNf44/k56dC90OzWQhSwouiccS/PmW53n39QUEAl6i0QRHnTCSn/zqbHx++yBrrVe/WExVNFyXEAASqlREwkxbtYTTBo5wMTr3LVy3kRtfmEo4Fm9QdsXDz/LilZft0UuFmPQsKbjkwbvf4L03FhKLxolFk2/Y995YSEmvzlxx9UkuR5c7llVuoTreeAnt2niMZZVb2uw6qsqsj5Yx470ldO6Sx/Hj9qdXn65tdv5MefjjT4jGEw3KEqqsq6hi/rqN7Nunp0uRmWxlScElLz0zg0ik4YdZJBLjxafLLCnshOFF3SnwBaiORxuU5/l87F3U9BLrqgqJlUACvEOa/cacSDjccv0TzJmxnHBtFJ/fyxP3vcP//ebrHHn8yLb6VTJi/baqtDffPR5h8/ZqFyIy2c5GH7mkpiaatry2JtLOkeS2E/sPo1soH5989V/Z7/HQI78Tx/VNv0quxhahm09GN5+Jbj4X3XQsGv2kyWu8+/oC5sxYRrg2+W8WjyWIROLc8cvnGi1Klw1UlY21c5m75SGOGrWeTqHGSSGaSLBfn14uRLfzVBWnZhLOppNxNhyAU34xGp3jdlgdliUFlwwfmX5n0r1HtcuOpR2G3+Nl0rjLOGPwPuT5/OT7/Jw1eBTPnXopPk/j/96qteiWS1KthDBQC846dOu3UGdr2mu8MWUO4drGH/4ej4e5s1a28W+0exyN88ban/HammuZXf5f/F2eY8KZL9C/pKKuTp7fz7cOPZBuhbmxaJ7W3AeVv4bECtBqiM1At1yKxua7HFnHZN1HLrnqZ+P46YT7iUbjOAnF6/XgD/i4+mdfczu0nNMtlM+dR57Bna2pHH4diDcu1wTUvggFlzY61NSNf1XNukEBS7e9xPraMuKpLUwTWkvADxcd/zHPvTmeLqE8Ljv0AE7aZy+XI20d1Shsv4vkyvr1RdCqO5Hie9wIq0OzpOCSvUf25V+P/YCnH3qPzxetY+jw3px/2RH0HdDN7dA6NmcjaLquuzDqbEi71vtpZx/EzA+WNmot1NZEKXt/CaNLB2XNKJ4lVS/WJYT6Av4oD377CIqDuZEM6jibILW5UkMK8YXtHc0ewZKCi/oO6MZ1Pz/T7TD2LP6DSP6336E7SPKRQGnal5QevhennHUgk5/6GHUa9s9PfupjhgzrxfHj9m/TMNXZhlb9CcJTkgWh05BO1yOeohZe2NyM7hyc7e0pBk2XFABv//aNZQ9h9xRMTqmKRvjLJ+9y/PMT+dqL9/Pkkjk7t7SFf38IHgKSV68wBL7hEDg67UtEhEsmHIvX2/Dtsn2Qh8VnCj9Y/TIT3nyWz7ZubPHyqsrrq5fy3Tef5fJpTzFp2TzijrNDnThaPh5qnwOtSv7UTkLLL0A1TddXPUM7j8Mrjfdo9nsK6BrIve1pRfIgfzyw4+8UQgqvcSOkDs9aCiZnhFOzl1dtryDqJMfe/3rGND7esIo/H3l6q84hIlD0L7TmCah9GkhA6Byk4BJEmv6OVLM9gtfrIR5LXnfbPl42HOdnQLdyxvZcQZ7vPW4pe52fH3Qjo4qb/gZ7y4xpPLn0U2pTcytmbFzFpGXzeeCEb+D5sgsq8jY462jYmoklu74ib0HoxCbPv3eXM1lV/Q4ba+cS11q8EkLwcGzv25v9/bKZdLoBJQC1j4DGkq2HTjchwSPcDq1DynhSEJFTgb8BXuAeVf39DseDwEPAQUA5cIGqrsh0XCb3/G/5AtbVVNYlBEhOUntx5Wdctf/hDOlc3KrziPiQgkug4JJWX7tH7y6E8gJEwjFUYONRfg4bsJTj+i/G50ngEejfaStvrruaEUVP4fUEG51jWeUWHl8yh0jiq2/7NfEYZRtX887a5Rzbd0iyML4I0twXQGsg/hnvzxnKvc9/yPrNlYwc0ovvn3cEQ/t3B8AjPk7scyfra2exoXY2IW8xgzudSNDbudW/a7YR8SKdf4p2+lHy70UKsuYeTkeU0a8OIuIF/gmcBowELhSRHWf7fBvYqqp7AXcCf8hkTCZ3vb9+JTVpZi97RZi9aU1Gr+3xePjhTacTDPlJFAihUJTjBywi4E0mBICAN0HQu4VlVVPTnuP9dSvT3siuicd4c/XnXxV4B+7QvZUiecxa4uPGv/+PeUvXsbmimndnf863b3mcJV9s+qqaCL3zD2JMt+8woujcnE4I9Yn4EE+hJYQMy3R7ciywVFWXqWoUeAI4a4c6ZwEPph4/A5wg9q9u0uhb0DntAneC0CMv84u7HXn8SP58zxUce8Q+DCjaSsJp/PYJeBOs3P5W2td3DgTxpunC8Xu8FIfqJYHQSSCFJBvXX/KgUsAv740Sjn7V0lCFcCTGv5+evou/lTENZTop9AVW1Xu+OlWWto4m76JtAxqNyxSRCSJSJiJlmzZt2vGw2QOMHzam0YQ0jwhdgiEO79W6bSjX1ZTxxtqfMWXV95i35VFizs4t9TBsnz784rcXcGTfkaT/6iKEvOnXRDqp/7Cv7hvU4xXh3KH7fnUGCSDdnoLAYSQTgxcCh1Hhe4CqmsbnVWD+5+t36vfYU2nkbZwtE3DKL8SpfhhN1023h8uZO0+qOlFVS1W1tKSk6TVtTMfVr7AL9x73dXrkFZDv8xP0+tinaw+ePPkivGlmL+9o/tYneH3t/7Gq+l02hefyyZZ7+N8XV+x0YgD46Zhv4SGfHUao4pUAw4vOTfuaPJ+fh068gOJgHoX+AIX+AAU+P3898gz6FxY1qCve3niK70N6zkF6zsFTfD8FhYOajKekqy2D3RKn6q9oxQ8h+hbEZkLVHWj5+OQEOVMn0zea1wD1h2L0S5Wlq7NaRHxAF5I3nI0BYM7b85n0t5coX1fBoacfyLQrv8VGT4Q8r4++ha3bMCaa2M7s8rtJ1PsASGiEmvhGFm+bzKiuF+5UTEGfn/OH/Iepq68j4lTiwYNDnIO7/5CSUNOL5B1Q0oePz7+GWZvWEHMS9AsV8cHnX/D05rkcP3xoo6UnRAJ1j0MBP6cfPYoX351PpF4XUijg44qzD9mp+Pc0mtgI1fcA9RNAGBLLIfwi5KVP5HuiTCeFGcAwERlM8sN/PHDRDnUmA5cDHwDnAW+o2p6KJmnyv19l4k8fJpJaKHDZnBVMued17p51B526tP7b8ebIQjz4SdDwW2FCIyxe/xoPnL+YRR8vpahHF8bfcDanffuEFm9odgkM5LzBz7EpPJ8NtXOIJipIaJTa+BbyfE2PhPJ5PIzt2Z/HZszhylcn13Up/eblt7jtjBM5c/Q+Tb72x5cci+M4TJm+APF48HqEH5x/BMcfvHer/y72SLFZIP7Gs9m1Fg2/gVhSqJPRpKCqcRG5GniVZOfofao6X0RuBcpUdTJwL/CwiCwFtpBMHMZQWx1ukBAAouEYFRu28cJdL3PJL85v9bmCns4oicYHFOa/vopP3qhKXnN7mH9f9wDla7dy6S9bPr/iMHfLg6yvnZWaFxBkdvl/OL7PH+mdn36GNMDK8gr+8OrbRHbY6+AX/3uNw4YMYJsTZn75evp1KuLA7n3qEpTP5+WGK07ihxcdQ0VlLT2KC/H5smv9pawkTbUoPeCx7uj6Mj5PQVWnAFN2KPtlvcdhoPXvbrPHWDZnJV5f43sF0XCMD/43c6eSQnFwbwp8PamMrULrraXjxLwsfLBhl024JsKTf3yB868/k1B+4/kG9S2vmsr62pn1FqBLJrC31v2cC4a8iEfSv8Vemb+IxI43JEjOeL586pMsC2/BKx4UZUBhEY+efCHdQl/FmR8KkB8KNHp9NlB1iCS24fcW4hW/2+EkBcYmR3RpDQ2X+wgg+fY9tL6cudFs9jyduxWSiKf5dg8U9yraqXOJCCf2/QudAwPwSQi/pwCvhFh0d282zmz84erxChu/2NzieZdWvpR2ATrVOJvDTS/YFnOctMtzVBdGWVC1kXAiTnU8Sk08xtJt5Vz/3ostxpINlm57iSeXnc7TK87m8c9PZsamf+C0sDRHexDxIsUPgLcfSH4yQUg+dL4N8Q93O7ysYstcmKzVf3hfBozoy+dzVuIkvvp2H8wPcu516ZcYdzTBwoqnWLTtOWJOLQMKjmJMt++S5yum0N+bswY8SkV0GZFEJd1Cw5m35C9A4w1b4rEE3XoXtRij0HTXjaSdqpZ04oi9uGd6GeF4ww/MROdEo69qcXWYvm4F1bEoBf7sbB1Eo3Gen/Yg2wc+gDfwVSJftG0SoBxc8kP3gksR31DoPg3iC5ItBv9+SJp1ovZ01lIwWe3WyTcweL8BBPOD5HfOI5gX4IrfXsgBx++Xtv709bcyu/y/VMXWEE5sYUnlS/zvi28STWwHki2GrsGh9Mo/AL8nn0t+/nWC+Q0/aIP5AU669BgKuhS0GN+wLqfjS/PB4vUE6BZq+obxiF4lXHLIGEL+Hb6XNfmOFKKJ9K0mtzmOw01XPcRa39MNEgJAQsMs2jaJhJMdOwqKCOIfhQQOtoTQBGspmKzWvU8xd8+6g5ULV1O5uYqhYwaR3ynNEhBAZXQ1X1S/3WDYqRIn6lSxtPIlRna9oNFr9j1yH2569Dr+ee19lK/bis/v4/Tvnch3ft+6dZEGFZ7Aqu3T+aL6XVTjeMQPCMf1/gMeaf4G8PUnHcVpo/bmkvufojaWbDF4qj04nRx2bGQM7FRE11D639ttsz5cxpKF6xjWbceNcL4ScarIT7MelMk+lhRMThi4T78W62yJLELwQZphp+trZ6dNCgCHn3Uwh51ZSk1VLaH8IN6dGM0j4uHo3rdQHv6MdbWzCHo6M7DwOALellsZAKP69OSQwf15e/FyFPBv9RHJjyZbDB4IeLz4PB7uODx7d+T7dNYKwrVRti7vRK/9trDjSh5eCRDyFrkSm9l5lhSyyPZwhE/XrKdzXohRvXvYwl87qcDfi3QbyXjw0dnf/IYsIkJB513fs7hbaATdQiN26bXXHnc4Hy1fRW0sjiSE4KoAniJlSP9ijhs8lIv3HkPvguxd1K5bSScCQR+fPDGMk0bMwOt36hKDOAEOLPl+k6OwTPaxf6ks8chHs/nTa9PxeT04jlLSqYB7LjmH/sVFboeWM7oHR9LJ34eK6Eq03j7MHvExoomlJzJl4xebeOeZD3ESDoefPZZ+w3o3WXef3j149IoL+PO06cxbu4GenQq58phDOHVUbkxIO+6U/bj/rmlsXd6ZabeUMubCpXQdXEmkIo9T9/8Jw4pOdjtEsxMkFycPl5aWallZmdthtJmZK9fwnUeeq+tXhuRCb/27duGVa75pLYadEI5v5d0Nt7K+ZhaIUOAr4YieP6dn3uh2i+Gl/77Gv669H1VFHcXj83LRTedy8c1fb7cYdsaHc1fw4OSPWV9exZjhffnOOYfSt0fRTp1j/pwvuP2Gp6iuCoNCUXEBv/zTeIYObzoZmvYnIjNVtelZlVhSyArXPf0Sr85f3KjjIz/g56Fvns++fXq6Elcuiya2k9AIIW9xuybVzWvKuXzYNUTDDfd9COYFuOvj3zNoVHbtK/zCW3P5y8Nv1i3H7fEIeUE/D912CUWd8pg+exmxRILD9h9E96LmlxVxHIcvlm1CPMKAwSX2ZSYLtSYpWPdRFthaXZN2S3WPCNtqbWnfXbE1ojy6eAGLKzZxYPe+XDBsNF2CmR+C+MHksrQfhrFonLeffp9Bo9Lf7HZDPJ7g74+/02B/BsdRasMxbr9nKguWrUc8AgoJx+Hq8UdxwckHNnk+j8fDoL3sC0yus6SQBU4YsRdzVq9vNJEplnAY3beXS1Hlrnnl67lg6mPEEgmiToK31izjPws+YvK4y1u9ququypWGt6ry1sJlbA/EScTAW69h46gy67PVjV7zzyemc/DIAQzp170dIzXtzSavZYHzDtyXvkWdCfmSOVqAkN/HT048gsKQje3eWT/74GWqY9G6vZzDiThbw7X8ftZbGb/24WeVkq5L1h/wcfR5h2X8+q2xtaaWr098jJ9OfoVtXR2294Pqng3HbaXr+YknErzy/mftFqdxh7UUskB+wM/TEy7imVnzmPbZUorz87jkkDGUDmx5bH4uiToxvOLB28Kkrt1RE4vy2daNjcodlDfXfJ7mFW2re99u/OCv3+Lf192POoqjitfrYfwNZzN43wEZv35r3PzCayzZsJmY46S+FgrxfCVSBKEK8Hk9iAixHdadclSJRBvvkW06FksKWSI/4OeyQw/gskMPcDuUNreociV3LXmCFdVr8YqXY3ocxPf3Oo88b9u3gnweLx4REmm+rYe87bNi5+kTTqL05NG8++xHJOIJjjj7YPoP33EXWnfURmO8u2R5MiHU5xFiXaA47OfSr5Vy/+SPGr02GPBz7MHD2ilS4xZLCiaj1teWc9On/yDsJGcZOxrn7Y0z2RTZym/3v7rNrxfwejm5/95MXbW4wQdf0Ovjwr3bb1hqr0E9OP8nZ7Tb9VormkikHdQAUFAQ4NV/fY+A30cw4OM/z75PLJ5AHSUY9HHyocMZs3d2JDeTOZYUTEZNXvM2cW3YDRHTOAsrl7OqZgP989t+tMpvDz2VVdsrWLqtHBEh4Tgc3msg1+x3RKvPsXlNOYtnLqN732KGHTikwwyv7JIXYlC3IpZu2tKg3CvCCSOGEkgt0HfxuFLG7juQl99bQDSW4ISxezNmeN8O8/dgmmZJwWTUypq1jZICgE+8rKvdlJGk0CUY4oVxlzO3fD0rqyoY0bWEYUWtGzGjqvzzh/cx5d7X8Qd8OAmH3kN68oepv6Brz6I2j3VXqSoLlq1nyReb6NejiAP36Y/H07oP7NvPOplvPfQs8YRDNJEg5PNRGAzwoxOObFBv2IAShg04JhPhmyxmSSFLbItt5+W177GwcjkDCnpxep+j6Bnq5nZYu21gsC9z+ZwEjVsLAwsyN9tVRNi/e2/27978NWJOnHAiSqEvDxFh6oNv8eoDbxILx4ilJqB9sXANt11wJ39565aMxbszwtEY190xiYXLN4AqHo9Q0rUTd9/8DYq7tLx+0+h+vXnpqst5fMYclm3ewoED+nDeAfvSOc+WkjYZSgoicgdwBsnlKj8HvqWqFWnqrQCqgAQQb2mmXUe1MbyFa2fdQdiJEnVifFKxiClrp3P7/lcxovNgt8PbZY9OKeORl5ZReI6Cj7pF0gIeP4cU7+tq0os5MSZ+PolpGz7CUYeiQCeu3Ot8Jv19CuHqhmv/J+IJPvtoMVs3VGRFa+Ge5z5g/ufriMa+SrRrNlbw23un8qcfn92qc/Tu0okfn3hkyxXNHidT8xReA/ZV1f2BxcCNzdQ9TlXH7KkJAeC+5S9QFa8h6iS/mcY1QdiJ8rfFj7sc2a6bvWg1E599n0ilsHVSb6Kr8tCYQNjLef1O5PoRl7ka398WP860DR8RdWLENcHmSAV/WPgAFd1r0tb3+rxUVza9X0B7+t878xskBIB4wuGDT1cQjbm/9aXJbRlJCqo6VbVuY9YPgY414L6NzdryGZpmTMia2o3UxLPjg2hnPTttDpHU8glOpZ+qqT0pf2AgNU8PYb/oaHyezM1VaEllrJrpmz6pS8Jfijgx8r7TFV+gcQM6r1MefYZmxxIO8YSTtlxVcZwcmVJtslZ7zGi+Ani5iWMKTBWRmSIyobmTiMgEESkTkbJNmza1eZBuCnnT77srCD5Pbt72qawOpx36KCJUh93dmrE8UtHk36tngJ+iHp0J5iX/TTxeD8H8INffeyUeT3YsAHDUAUPw7nBTWYARg3oSCrbPXAzTce3yJ46ITAPSLcxzs6q+kKpzMxAHHm3iNEeq6hoR6QG8JiKfqeo76Sqq6kRgIiRXSd3VuLPR1/ocyRNfTG3wzdUnXg7ttj8BT26+yY8rHcacRWsaLLYGEI877Desj0tRJfXK604izYgoD8I+RYP5wdzrmXLP68x+fS69h/TkzKtObdXOb+3lmvFHM3PhKqqqI9RGYgQDPvw+Lz//ru1bYHZfxpbOFpFvAt8DTlDV9B21Dev/Gtiuqn9qqW5HWzo77iT402cP8dGWefjES0IdBhf04Zb9vk+hb9d3A3NTJBpnwm1PsmJdOeFIHBEh6Pfyw4uO4esntN8ksqY8suIlnlv9JhHnq607Q54gfzvwevplYJhsW6sNx3jl/YUsXL6egb2LOf2oUXRpYu9qY77k2n4KInIq8BfgGFVN29cjIgWAR1WrUo9fA25V1VdaOn9HSwpfWle7iRXV6+gZ6saQwtyfORqNxXn1g894q2wpRYV5nHvC/owamh2brqgqU9d/wDOrprEttp0RnQdzxZCzGFTgbivGmExyMyksBYJAearoQ1X9voj0Ae5R1XEiMgSYlDruAx5T1dtbc/6OmhSyXWV5FQs/XEyXks4MP3gvm91qTI5xbZMdVd2rifK1wLjU42WA+/0IplUeue1pHv/dJHwBP47j0K13V37/6s/pNaiH26EZY9pQdgynMFnt45dn8+QfXyAajlFTWUN4e5h1n6/nF2f83u3QjDFtLDfHO5pWiToxJq95mzc2zMArHk7tdTin9jl8p/czSDfL13GUdcs3snLh6qwamWOM2T2WFDqohDrcMOfvLK9eWzfU9d7lL1C2dSG/2rfZKSGNVJZXpS33+jxUV1TvdqzGmOxh3UcdVNmW+aysWd9g7kPEiTKnYjGLKlfu1LmOPGcsgbzGE+zUUfY6IHfXZjLGNGZJoYOav+1zwonGM4cTmmBh5bKdOtdZV59Gj/7dCOYnE4N4hGB+gGv++R0CofSzsY0xucm6jzqoboEiAh5/o/V9/B4fxYEuO3Wu/E55/GvmH3nlvjf46MWZFPfpytlXn8beBw1ty5CNMVnAkkIHdWzPUh5e+VKDMgF84uOQbvvu9PnyCkKcc804zrlmXBtFaIzJRtZ91EF18Rdy235XUhLsStATIOjx0yevB38Y/UOCTSzAZ4wx1lLowPbpPJj7x/6a1bUb8YqH3qHuNgvZGNMsSwodnIi0eh/kD5ev4umZc6mNxfjaviM4ZeQwfF5rTBqzJ7GkYAD46+vv8eCHs6hN7dz1wbJVTPpkPv+5+Gy8WbKPgDEm8+zdblhbUcn9H8ysSwgAtbEYs1at5Z0lK9wLzBjT7iwpGD5YvgpPmnsNNdEYbyz63IWIjDFusaRg6BQMpE0KPo9QlBdyISJjjFssKRiOHjYYjzT+r+D1eDlnzCgXIjLGuMWSgiHk93HPpedQlBeiIBigMBgg5Pdxy+knMKSk2O3wjDHtyEYfGQBG9+vNu9dPoGzlGiLxOKUD+1EYtEluxuxpLCmYOn6vl8OGDHA7DGOMizLWfSQivxaRNSLySeon7aI5InKqiCwSkaUickOm4jHGGNOyTN9TuFNVx6R+pux4UES8wD+B04CRwIUiMjLDMRmTVtXW7axcsIpIbeMlx43ZU7jdfTQWWKqqywBE5AngLGCBq1GZPUo0EuPOCXfz9lMf4Av4UMfhopu/zvifnW1rRZk9TqaTwtUichlQBvxEVbfucLwvsKre89XAIelOJCITgAkAAwZYv/eu2B6v4ZV17zOnYgl9Qt05o+/R9Gvlukgd2b+uvY93n/mQWCRGLJLcf+Kx25+lR//unHDxUS5HZ0z72q3uIxGZJiLz0vycBfwbGAqMAdYBf96da6nqRFUtVdXSkpKS3TnVHmlrtJIflP2OR1e+zKytC3l53Xv8cNYfmbVloduhuSoajvLaQ28TqY02KA9XR3j8d8+5FJUx7tmtloKqntiaeiLyX+DFNIfWAP3rPe+XKjNt7ImVr7ItVkVCHQASOCQch78ufowHD7l1j+0mqd5WgzZxbOuGivYMJWNeXvkZf53zHutrKtm3uBc/O/BY9u/e2+2wTJbK5Oij+v/rzgHmpak2AxgmIoNFJACMByZnKqY92Udb5tYlhPqq4jVsjGxxIaLs0KWkM4VFBY3KRWDk4SNciKhtPbpoNj+e/iKLKjaxLRrhvfUrueDVR/l08zq3QzNZKpOjj/4oInNF5FPgOOBHACLSR0SmAKhqHLgaeBVYCDylqvMzGFPWiycc/vnWBxz+x7sZ/Zu/c8VDz7J0Y/lunzfPm34NI1Ul5A3u9vlzlcfj4aq/XUEwP1CvTAgVhLji9gtdjGz3xR2HP85+m9pEvEF5bSLOn2a/41JUJttl7Eazql7aRPlaYFy951OARsNV91Q3vfAqUxcuJVy3r8EXXHDPE/zvykvpU9R5l897Zp9j+O+ySUScr/rOvXgY2WUIXfyFux13Ljvm/MMoKunMo7c/y/rlG9jn0L255Bfn0X94X7dD2y3l4WoiOySEL83bsr6dozG5wu0hqaaeDZXbeWX+EqKJRF2ZApF4nAc/nMWNpx67y+c+pfdhLN2+itc3fIzf48NRhz55JfzfiMt3P/AOYPSxoxh9bMda/K9LMK/JY30Lu7RjJCaXWFLIIss2byHo8zZICpDsBvh0ze59s/OIh2v2Hs+FA09hSdUqSoJdGVrYb4+9wbwnCHl9XLz3ATy2eHaDLqQ8r49r9z/CxchMNrOkkEX6d+3SKCEAeEXYu0f3NrlG92BXuge7tsm5TPa78aDj8IjwyKLZJNSh0B/gxoOO48T+w9wOzWQpSwpZpF/XLhw+ZCDvL1tJJP5Vcgj4vHzr8INcjMzkKp/Hw82lx/PTA46hMhqmOJSfdkMlY75k+ylkmTvP/xrnjhlF0OfDI8KIXiXcf9l5DOpm3+7Nrgt4vXTPK7CEYFokqk1N3clepaWlWlZW5nYYGeU4StxxCPi8bodijOkgRGSmqpY2V8e6j7KUxyMEPJYQjDHty7qPjDHG1LGkYIwxpo4lBWOMMXUsKRhjjKljScEYY0wdSwrGGGPqWFIwxhhTx5KCMcaYOpYUjDHG1LGkYIwxpo4lBWOMMXUysvaRiDwJDE89LQIqVHVMmnorgCogAcRbWqjJGGNMZmUkKajqBV8+FpE/A9uaqX6cqm7ORBzGGGN2TkZXSZXkXo/fAI7P5HWMMca0jUzfUzgK2KCqS5o4rsBUEZkpIhMyHIsxxpgW7HJLQUSmAb3SHLpZVV9IPb4QeLyZ0xypqmtEpAfwmoh8pqrvNHG9CcAEgAEDBuxq2MYYY5qRsZ3XRMQHrAEOUtXVraj/a2C7qv6ppbp7ws5rxhjT1tzeee1E4LOmEoKIFAAeVa1KPT4ZuDWD8XRYqxatYfpzH6PqcMQ5hzBwn35uh2SMyVGZTArj2aHrSET6APeo6jigJzApeS8aH/CYqr6SwXg6pKf/PJkHfvkkiVgCgEdvf46Lbz6Xi276usuRGWNyUca6jzLJuo+S1i3bwHf2/RHRcKxBeSAU4O7Zf6T/8L4uRWaMyUat6T6yGc057L3nP0adxkk9kUjw3qSPXYjIGJPrLCnkMI/HA8nutwYEEI/90xpjdp59cuSwI88dmy4n4PF5Oerrh7R/QMaYnGdJIYf1GFDC9/9yOYGQv8HPt393EX2GpptCYowxzcvoMhcm8874/ikcMu5A3nt+BqrK4WcdTK9BPdwOyxiToywpdAA9BpRwzg/HuR2GMaYDsO4jY4wxdSwpGGOMqWNJwRhjTB27p2BMG1NVJn0yn4nTyyivrmFM315cf9JRDO9V4nZoxrTIkoIxbezf73zEf6fPoDYWB2D65yuZuWotz3z3IoaUFLscnTHNs+4jY9pQbTTWICFAciepcCzOP9/+0L3AjGklSwrGtKHVFdvwpJlm7qjy6Zr1LkRkzM6xpGBMG+rRqZBYwkl7bEBxUfsGY8wusKRgTBvqkhfia/sNJ+hreLsu5Pdx5dG2HpXJfnaj2Zg2dsvpJ5Ln9/Ps7Hk4qhTn5/Pzccdy0EDb38JkP9tkx5gMicYTVEejFOWFkHTL2RrTztzeo9mYPVrA5yXgy3M7DGN2it1TMMYYU2e3koKInC8i80XEEZHSHY7dKCJLRWSRiJzSxOsHi8hHqXpPikhgd+Ixxhize3a3pTAPOBd4p36hiIwExgOjgFOBf4mIN83r/wDcqap7AVuBb+9mPMYYY3bDbiUFVV2oqovSHDoLeEJVI6q6HFgKjK1fQZJ33o4HnkkVPQicvTvxGGOM2T2ZutHcF6g/p391qqy+bkCFqsabqVNHRCYAE1JPIyIyr41izaTuwGa3g2hBLsQIFmdbszjbVq7EObylCi0mBRGZBqTb8PdmVX1hV6LaFao6EZiYiqmspWFV2SAX4syFGMHibGsWZ9vKpThbqtNiUlDVE3fh2muA/vWe90uV1VcOFImIL9VaSFfHGGNMO8rUkNTJwHgRCYrIYGAY8HH9CpqcNfcmcF6q6HKg3VoexhhjGtvdIanniMhq4DDgJRF5FUBV5wNPAQuAV4CrVDWRes0UEemTOsXPgB+LyFKS9xjubeWlJ+5O3O0oF+LMhRjB4mxrFmfb6jBx5uQyF8YYYzLDZjQbY4ypY0nBGGNMnZxOCiLyExFREenudizpiMhtIvKpiHwiIlPr3UvJKiJyh4h8lop1kogUuR1TOs0tq5INROTU1LIuS0XkBrfjSUdE7hORjdk8z0dE+ovImyKyIPXvfa3bMaUjIiER+VhE5qTivMXtmJojIl4RmS0iLzZXL2eTgoj0B04GvnA7lmbcoar7q+oY4EXgly7H05TXgH1VdX9gMXCjy/E0Je2yKtkgtYzLP4HTgJHAhanlXrLNAySXnslmceAnqjoSOBS4Kkv/LiPA8ao6GhgDnCoih7obUrOuBRa2VClnkwJwJ/B/JPdFz0qqWlnvaQFZGquqTq03s/xDknNGsk4zy6pkg7HAUlVdpqpR4AmSy71kFVV9B9jidhzNUdV1qjor9biK5AdZ1u1QpEnbU0/9qZ+sfI+LSD/ga8A9LdXNyaQgImcBa1R1jtuxtEREbheRVcDFZG9Lob4rgJfdDiIH9QVW1Xve7LItpnVEZBBwAPCRy6GkleqS+QTYCLymqlkZJ/BXkl+i028gXk/WbrLT3PIawE0ku45c19IyIKp6M3CziNwIXA38ql0DTGnNciUicjPJpvuj7RlbfdmyrIpxn4gUAs8C1+3Q6s4aqflXY1L34SaJyL6qmlX3a0TkdGCjqs4UkWNbqp+1SaGp5TVEZD9gMDAntcVhP2CWiIxV1fXtGCKwU8uAPApMwaWk0FKcIvJN4HTgBHVx8souLquSDVqztItpJRHxk0wIj6rqc27H0xJVrRCRN0ner8mqpAAcAZwpIuOAENBZRB5R1UvSVc657iNVnauqPVR1kKoOItlMP9CNhNASERlW7+lZwGduxdIcETmVZNPyTFWtcTueHDUDGJbaOCpAcj+RyS7HlJNSy+rfCyxU1b+4HU9TRKTky5F6IpIHnEQWvsdV9UZV7Zf6vBwPvNFUQoAcTAo55vciMk9EPiXZ3ZWVQ+uAu4BOwGup4bN3ux1QOk0tq5INUjfqrwZeJXlj9KnUci9ZRUQeBz4AhovIahHJxo2tjgAuBY5P/X/8JPUtN9v0Bt5Mvb9nkLyn0Oxwz1xgy1wYY4ypYy0FY4wxdSwpGGOMqWNJwRhjTB1LCsYYY+pYUjDGGFPHkoIxxpg6lhSMMcbU+X/m+9RunV/tRQAAAABJRU5ErkJggg==\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], - "source": [ - "for d, m in enumerate(xlm.model.model_chain):\n", - " print(\"model_chain[{}].W is a {} matrix of shape {}\".format(d, m.W.getformat(), m.W.shape))\n", - "\n", - "layer_d = 1\n", - "from sklearn.decomposition import TruncatedSVD\n", - "svd = TruncatedSVD(n_components=2, random_state=0)\n", - "Wt = svd.fit_transform(xlm.model.model_chain[layer_d].W.transpose())\n", - "\n", - "import numpy as np\n", - "color = cluster_chain[layer_d].tocsr() * np.arange(cluster_chain[layer_d].shape[1])\n", - "\n", - "import matplotlib.pyplot as plt\n", - "plt.scatter(Wt[:, 0], Wt[:, 1], c=color)\n", - "plt.xlim(-4, 4);\n", - "plt.ylim(-10, 10);" - ] - }, - { - "cell_type": "markdown", - "id": "ae9baf18", - "metadata": {}, - "source": [ - "### PECOS and One-versus-All (OVA) Model\n", - "\n", - "PECOS also supports to train an OVA model without leveraing clustering hierarchy if needed.\n", - "\n", - "**Training OVA models is time-consuming, we suggest to try it offline after the tutorial.**" - ] - }, - { - "cell_type": "code", - "execution_count": 18, - "id": "c95f0acf", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Training time for the OVA model: 1047.3194 seconds.\n", - "XR-Linear is 25.56 times faster than the OVA model\n" - ] - } - ], - "source": [ - "import time\n", - "start_time = time.time()\n", - "\n", - "xlm_ova = XLinearModel.train(X_trn, Y_trn, C=None, negative_sampling_scheme=\"tfn\") \n", - "\n", - "training_time_ova = time.time() - start_time\n", - "print(\"Training time for the OVA model: {:.4f} seconds.\".format(training_time_ova))\n", - "\n", - "print(\"XR-Linear is {:.2f} times faster than the OVA model\".format(training_time_ova / training_time))" - ] - }, - { - "cell_type": "markdown", - "id": "73a599a0", - "metadata": {}, - "source": [ - "## Customized Parameters and Advanced Training Options\n", - "\n", - "PECOS also supports using customized parameters and several advanced training options, such as different solvers and cost-sensitive learning.\n", - "\n", - "### Customized Parameters\n", - "\n", - "The parameters for either of indexing, training, and inference can be easily customized by feeding a dictionary into the corresponding parameter class and its constructor:\n", - "\n", - "* Semantic Indexing (Hierarchical K-Means): `HierarchicalKMeans.TrainParams.from_dict(dict)`\n", - "* Training: `XLinearModel.TrainParams.from_dict(dict)`\n", - "* Inference: `XLinearModel.PredParams.from_dict(dict)`\n", - "\n", - "Although most of the parameters can be also passed by `kwargs` of Python methods, **we encourage to use the dictionary to designate the parameters because it is easier to manage, modularize, and store parameters in certain formats like JSON.**\n", - "\n", - "For XR-Linear models, the default values and skeleton of the parameters can be revealed and generated by the following command:" - ] - }, - { - "cell_type": "code", - "execution_count": 19, - "id": "1ddc9bfa", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{\r\n", - " \"train_params\": {\r\n", - " \"__meta__\": {\r\n", - " \"class_fullname\": \"pecos.xmc.xlinear.model###XLinearModel.TrainParams\"\r\n", - " },\r\n", - " \"mode\": \"full-model\",\r\n", - " \"ranker_level\": 1,\r\n", - " \"nr_splits\": 16,\r\n", - " \"min_codes\": null,\r\n", - " \"shallow\": false,\r\n", - " \"rel_mode\": \"disable\",\r\n", - " \"rel_norm\": \"no-norm\",\r\n", - " \"hlm_args\": {\r\n", - " \"__meta__\": {\r\n", - " \"class_fullname\": \"pecos.xmc.base###HierarchicalMLModel.TrainParams\"\r\n", - " },\r\n", - " \"neg_mining_chain\": \"tfn\",\r\n", - " \"model_chain\": {\r\n", - " \"__meta__\": {\r\n", - " \"class_fullname\": \"pecos.xmc.base###MLModel.TrainParams\"\r\n", - " },\r\n", - " \"threshold\": 0.1,\r\n", - " \"max_nonzeros_per_label\": null,\r\n", - " \"solver_type\": \"L2R_L2LOSS_SVC_DUAL\",\r\n", - " \"Cp\": 1.0,\r\n", - " \"Cn\": 1.0,\r\n", - " \"max_iter\": 100,\r\n", - " \"eps\": 0.1,\r\n", - " \"bias\": 1.0,\r\n", - " \"threads\": -1,\r\n", - " \"verbose\": 0,\r\n", - " \"newton_eps\": 0.01\r\n", - " }\r\n", - " }\r\n", - " },\r\n", - " \"pred_params\": {\r\n", - " \"__meta__\": {\r\n", - " \"class_fullname\": \"pecos.xmc.xlinear.model###XLinearModel.PredParams\"\r\n", - " },\r\n", - " \"hlm_args\": {\r\n", - " \"__meta__\": {\r\n", - " \"class_fullname\": \"pecos.xmc.base###HierarchicalMLModel.PredParams\"\r\n", - " },\r\n", - " \"model_chain\": {\r\n", - " \"__meta__\": {\r\n", - " \"class_fullname\": \"pecos.xmc.base###MLModel.PredParams\"\r\n", - " },\r\n", - " \"only_topk\": 20,\r\n", - " \"post_processor\": \"l3-hinge\"\r\n", - " }\r\n", - " }\r\n", - " },\r\n", - " \"indexer_params\": {\r\n", - " \"__meta__\": {\r\n", - " \"class_fullname\": \"pecos.xmc.base###HierarchicalKMeans.TrainParams\"\r\n", - " },\r\n", - " \"nr_splits\": 16,\r\n", - " \"min_codes\": null,\r\n", - " \"max_leaf_size\": 100,\r\n", - " \"imbalanced_ratio\": 0.0,\r\n", - " \"imbalanced_depth\": 100,\r\n", - " \"spherical\": true,\r\n", - " \"seed\": 0,\r\n", - " \"kmeans_max_iter\": 20,\r\n", - " \"threads\": -1\r\n", - " }\r\n", - "}\r\n" - ] - } - ], - "source": [ - "! python3 -m pecos.xmc.xlinear.train --generate-params-skeleton" - ] - }, - { - "cell_type": "markdown", - "id": "35472517", - "metadata": {}, - "source": [ - "### Training Parameters for Hierarchial Models in XR-Linear\n", - "\n", - "Hierarchical models could have different parameters over layers. To have customized parameters for the hierarchical model, `hlm_args` needs to be designated in the parameter dictionary. The values of `model_chain` and `neg_mining_chain` in `hlm_args` can be **a single dictionary** of general parameters for all layers or **a list of dictinoaries** for specific parameters of individual layers.\n", - "\n", - "#### General Parameters for All Layers\n", - "\n", - "```\n", - "train_params_l1 = XLinearModel.TrainParams.from_dict(\n", - " {\n", - " ...\n", - " \"hlm_args\": {\n", - " ...\n", - " \"neg_mining_chain\": \"tfn\", # Negative sampling scheme for all layers\n", - " \"model_chain\":{...}, # Parameters for all layers\n", - " }\n", - " ...\n", - " })\n", - "```\n", - "\n", - "#### Specific Parameters of Individual Layers\n", - "\n", - "```\n", - "train_params_l1 = XLinearModel.TrainParams.from_dict(\n", - " {\n", - " ...\n", - " \"hlm_args\": {\n", - " ...\n", - " \"neg_mining_chain\": [\n", - " \"tfn\", # Negative sampling scheme for layer-0\n", - " \"tfn\", # Negative sampling scheme for layer-1\n", - " \"tfn+man\", # Negative sampling scheme for layer-2\n", - " ...\n", - " ],\n", - " \"model_chain\": [\n", - " {...}, # Parameters for layer-0\n", - " {...}, # Parameters for layer-1\n", - " {...}, # Parameters for layer-2\n", - " ...\n", - " ],\n", - " }\n", - " ...\n", - " })\n", - "```" - ] - }, - { - "cell_type": "markdown", - "id": "81f86b8a", - "metadata": {}, - "source": [ - "### Variety of Solvers\n", - "\n", - "The solver for optimization can be adjusted by the argument `solver_type` in the `train` function. PECOS currently provides the following solvers for training each matcher/ranker:\n", - "\n", - "* \"L2R_L2LOSS_SVC_DUAL\" (default): L2-regularized L2-loss Dual SVM\n", - "* \"L2R_L1LOSS_SVC_DUAL\": : L2-regularized L1-loss Dual SVM\n", - "* \"L2R_LR_DUAL\": L2-reguarlized Logistic Regression" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "id": "8f26ee42", - "metadata": {}, - "outputs": [], - "source": [ - "xlm_l1_kwargs = XLinearModel.train(\n", - " X_trn, Y_trn,\n", - " C=cluster_chain,\n", - " threshold=0.1,\n", - " negative_sampling_scheme=\"tfn\",\n", - " solver_type=\"L2R_L1LOSS_SVC_DUAL\")" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "id": "197926a7", - "metadata": {}, - "outputs": [], - "source": [ - "train_params_l1 = XLinearModel.TrainParams.from_dict(\n", - " {\n", - " \"hlm_args\": {\n", - " \"threshold\": 0.1,\n", - " \"neg_mining_chain\": \"tfn\",\n", - " \"model_chain\":{\n", - " \"solver_type\": \"L2R_L1LOSS_SVC_DUAL\",\n", - " },\n", - " }\n", - " }\n", - ")\n", - "\n", - "xlm_l1_dict = XLinearModel.train(\n", - " X_trn, Y_trn,\n", - " C=cluster_chain,\n", - " train_params=train_params_l1)" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "id": "eddf91a6", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Evaluation Metrics with L2R_L1LOSS_SVC_DUAL (by method kwargs)\n", - "prec = 83.43 77.66 72.47 67.67 63.73 60.18 56.90 54.04 51.45 49.04\n", - "recall = 4.93 9.11 12.62 15.59 18.19 20.51 22.49 24.28 25.92 27.36\n", - "\n", - "Evaluation Metrics with L2R_L1LOSS_SVC_DUAL (by dictionary)\n", - "prec = 83.43 77.66 72.47 67.67 63.73 60.18 56.90 54.04 51.45 49.04\n", - "recall = 4.93 9.11 12.62 15.59 18.19 20.51 22.49 24.28 25.92 27.36\n" - ] - } - ], - "source": [ - "Y_pred_l1_kwargs = xlm_l1_kwargs.predict(X_tst, beam_size=10, only_topk=10)\n", - "Y_pred_l1_dict = xlm_l1_dict.predict(X_tst, beam_size=10, only_topk=10)\n", - "metrics_l1_kwargs = smat_util.Metrics.generate(Y_tst, Y_pred_l1_kwargs, topk=10)\n", - "metrics_l1_dict = smat_util.Metrics.generate(Y_tst, Y_pred_l1_dict, topk=10)\n", - "\n", - "print(\"Evaluation Metrics with L2R_L1LOSS_SVC_DUAL (by method kwargs)\")\n", - "print(metrics_l1_kwargs)\n", - "\n", - "print(\"\\nEvaluation Metrics with L2R_L1LOSS_SVC_DUAL (by dictionary)\")\n", - "print(metrics_l1_dict)" - ] - }, - { - "cell_type": "markdown", - "id": "9f481ed3", - "metadata": {}, - "source": [ - "### Cost-sensitive Learning\n", - "\n", - "PECOS supports to adjust the cost of each training instance. To enable cost-sensitive learning, we need to provide a **relevance matrix** `R_trn` with the same shape to the label matrix `Y_trn` for the argument `R`. When `R` is `None` (default), cost-sensitive learning is disable. \n", - "\n", - "Since PECOS models are usually hierarhical, costs for upper layers also need to be decided as the cost-sensitive learning mode by the argument `rel_mode`. Currently, PECOS supports the following cost-sensitive learning modes:\n", - "\n", - "* `\"disable\"` (default): The cost-sensitive learning is disable.\n", - "* `\"induce\"`: Induce the costs into upper layers by the clustering chain.\n", - "* `\"ranker-only\"`: Only apply cost-sensitive learning to the model in the last ranker layer without induction.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "id": "382277f3", - "metadata": {}, - "outputs": [], - "source": [ - "# An exmaple of using training label frequency scores as costs. \n", - "import copy\n", - "from sklearn.preprocessing import normalize\n", - "\n", - "R_trn = copy.deepcopy(Y_trn)\n", - "\n", - "# Training parameters for cost-sensitive learning.\n", - "train_params_cost = XLinearModel.TrainParams.from_dict(\n", - " {\n", - " \"rel_mode\": \"induce\",\n", - " \"rel_norm\": \"l1\",\n", - " \"hlm_args\": {\n", - " \"neg_mining_chain\": \"tfn\",\n", - " \"model_chain\":\n", - " [\n", - " {\n", - " \"threshold\": 0.1,\n", - " \"Cp\": 1.0,\n", - " \"Cn\": 1.0,\n", - " },\n", - " {\n", - " \"threshold\": 0.1,\n", - " \"Cp\": 8.0,\n", - " \"Cn\": 1.0,\n", - " },\n", - " {\n", - " \"threshold\": 0.1,\n", - " \"Cp\": 4.0,\n", - " \"Cn\": 1.0,\n", - " },\n", - " {\n", - " \"threshold\": 0.1,\n", - " \"Cp\": 4.0,\n", - " \"Cn\": 1.0,\n", - " },\n", - " ],\n", - " }\n", - " })\n", - " \n", - "# Cost-sensitive learning.\n", - "xlm_cost = XLinearModel.train(\n", - " X_trn, Y_trn,\n", - " C=cluster_chain,\n", - " R=R_trn,\n", - " train_params=train_params_cost)" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "id": "559c15cb", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Evaluation Metrics with Cost-sensitive Learning\n", - "prec = 85.02 80.58 74.57 69.37 64.79 60.82 57.34 54.29 51.37 48.81\n", - "recall = 5.02 9.46 13.02 15.99 18.54 20.74 22.69 24.42 25.92 27.27\n", - "\n", - "Original Evaluation Metrics\n", - "prec = 84.07 78.17 72.68 67.79 63.79 60.06 56.63 53.51 50.83 48.33\n", - "recall = 4.97 9.16 12.68 15.60 18.25 20.49 22.40 24.05 25.60 26.95\n" - ] - } - ], - "source": [ - "Y_pred_cost = xlm_cost.predict(X_tst, beam_size=10, only_topk=10)\n", - "metrics_cost = smat_util.Metrics.generate(Y_tst, Y_pred_cost, topk=10)\n", - "print(\"Evaluation Metrics with Cost-sensitive Learning\")\n", - "print(metrics_cost)\n", - "print(\"\\nOriginal Evaluation Metrics\")\n", - "print(metrics)" - ] - }, - { - "cell_type": "markdown", - "id": "1663277d", - "metadata": {}, - "source": [ - "# Customized PECOS Model\n", - "\n", - "Besides pre-defined models in PECOS, such as XR-Linear, it is also convenient for users to customize PECOS for specific purposes and usage. Specifically, we suggest to establishing a model class to wrap fundamental PECOS functions and tailored operations. As a result, the customized model can be easily constructed and consumed for arbitrary data types and feature extractors. \n", - "\n", - "## Structure of a Customized PECOS Model\n", - "\n", - "Even though a customized machine learning pipeline can be seperated into several independent scripts, we recommend declaring a customized PECOS model as a **model class** for better re-usability and code maintenance.\n", - "\n", - "A customized PECOS model should at least consist of the following components:\n", - "\n", - "* `preprocessor` or `encoder`: The procedure, which can be a method or a functionable object, pre-processes or encodes an arbitrary input with the designated data format into features. For example, text data and image data can be encoded by BERT and ResNet.\n", - "* `train()`: The training method takes a set of training data with a preprocessor, learns a primitive PECOS model, and returns a PECOS-based customized machine learning model. The training function could be a class method to construct the model object with the learned model and essential components after training.\n", - "* `model`: A primitive PECOS model taking pre-processed features is capable of deriving the predictions for arbitrary testing data. The model weights should be learned by `train()`. \n", - "* `predict()`: The prediction method takes arbitrary testing data and infers the prediction based on the pre-processor and the learned model.\n", - "* `save()`: The saving function serializes the trained model, including model weights and configuration, for further usage.\n", - "* `load()`: The loading function reads the serialized model so that the trained model can be loaded and re-used.\n", - "\n", - "
\n", - "
\n", - "
\n", - "\n", - "\n", - "In this part of the tutorial, we will use the task of *extreme multi-label text classification* as an example to demonstrate how to **customize a PECOS model that can handle text data with either a conventional bag-of-words (BoW) model or a deep learning model as the text encoder for feature extraction**.\n" - ] - }, - { - "cell_type": "markdown", - "id": "c3acc325", - "metadata": {}, - "source": [ - "## Example: eXtreme Multi-label Text Classification (XMTC)\n", - "\n", - "The task of extreme multi-label text classification (XMTC) seeks to find relevant labels from an extreme large label collection for a given text input. Many real-world applications can be formulated as XMTC tasks, such as recommendation systems, document tagging, and semantic search. \n", - "\n", - "In this section, we guide through how to establish a customized PECOS model for XMTC tasks. We will walk through (1) PECOS' built-in BOW model for text preprocessing and vectorizing; (2) how to customize a PECOS model; and (3) \n", - "advanced usage of XR-Transformer based on deep learning.\n" - ] - }, - { - "cell_type": "markdown", - "id": "3b2ec0e6", - "metadata": {}, - "source": [ - "### Preprocessor: Text Preprocessing and Vectorizing\n", - "\n", - "The preprocessor plays a role of encoding input data into machine readable vector representations. Any encoder that can transform text data into a vector representation can be considered as the preprocessor or encoder of a customized PECOS model for XMTC tasks.\n", - "\n", - "In the PECOS library, we provide [various text vectorizers](https://github.com/amzn/pecos/blob/mainline/pecos/utils/featurization/text/vectorizers.py), such as TF-IDF, hashing, and pretrained transformer, as **built-in preprocessors** to deal with text data. In this tutorial, we will utilize the [n-gram](https://en.wikipedia.org/wiki/N-gram) [TF-IDF](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) model as our preprocessor.\n", - "\n", - "#### Label Space File Format for Built-in Text Preprocessors\n", - "\n", - "Label space is also essential for text preprocessors, especially for understanding the label space size to create the appropriate label matrix. The label IDs start from zero and can be referred to the line numbers and corresponding text descriptions in the label space file." - ] - }, - { - "cell_type": "code", - "execution_count": 25, - "id": "3f48f4f7", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Artificial intelligence researchers\r\n", - "Computability theorists\r\n", - "British computer scientists\r\n", - "Machine learning researchers\r\n", - "Turing Award laureates\r\n", - "Deep Learning\r\n" - ] - } - ], - "source": [ - "! cat \"./text2text_demo/output-labels.txt\"" - ] - }, - { - "cell_type": "markdown", - "id": "e0862645", - "metadata": {}, - "source": [ - "#### Data File Format for Built-in Text Preprocessors\n", - "\n", - "PECOS built-in text preprocessors majorly take the files of text data with labels in a tab-separated values (TSV) format. Each line in the TSV file consists of two elements that represent the comma-separated label IDs and the input text of a data instance. " - ] - }, - { - "cell_type": "code", - "execution_count": 26, - "id": "bd5ebfc6", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "0,1,2\tAlan Turing is widely considered to be the father of theoretical computer science and artificial intelligence.\r\n", - "0,2,3\tHinton was co-author of a highly cited paper published in 1986 that popularized the backpropagation algorithm for training multi-layer neural networks.\r\n", - "3,4,5\tHinton received the 2018 Turing Award, together with Yoshua Bengio and Yann LeCun, for their work on artificial intelligence and deep learning.\r\n", - "0,3,5\tYoshua Bengio is a Canadian computer scientist, most noted for his work on artificial neural networks and deep learning.\r\n" - ] - } - ], - "source": [ - "! cat ./text2text_demo/training-data.txt" - ] - }, - { - "cell_type": "markdown", - "id": "566d1eb5", - "metadata": {}, - "source": [ - "The data file format also supports to represent the label relevance for cost-sensitive learning by using double colons to separate a label and its relevance.\n", - "\n", - "

\n", - "0::0.1,1::0.2,2::0.8 <TAB> Alan Turing is widely considered to be the father of theoretical computer science and artificial intelligence.

\n" - ] - }, - { - "cell_type": "markdown", - "id": "4d4e4419", - "metadata": {}, - "source": [ - "#### Training a Text Preprocessor\n", - "\n", - "The preprocessor model `Preprocessor` is defined in `pecos.utils.featurization.text.preprocess`. Given a training text corpus and the configuration dictionary, the class method `Preprocessor.train` will train a corresponding text preprocesssor. Besides, the built-in preprocessors also support serialization with the function `save()` for the re-usability.\n", - "\n", - "With the previously mentioned data and label space file formats, the utility function `Preprocessor.load_data_from_file(input_text_path, output_text_path)` returns a dictionary with three keys:\n", - "\n", - "* `label_matrix`: a `(num_inst, num_labels)` CSR matrix for the labels of each instance.\n", - "* `label_relevance`: `None` or a `(num_inst, num_labels)` CSR matrix for the relevance of each label in cost-sensitive learning if available.\n", - "* `corpus`: a list of string as the text corpus in the input_text_path.\n", - "\n", - "The configuration settings of text preprocessor including the preprocessor type and hyper-parameters should be defined in a dictionary. Specifially, the key `type` defines the preprocessor choice while the key `kwargs` represents the hyper-parameters. In this tutorial, we adopt n-gram TFIDF features containing *word unigrams*, *word bigrams*, and *character trigrams*. Note that each of the n-gram feature can have different hyper-parameters, such as `max_feature` and `max_df`. Users need to properly set max_feature (e.g., hundred of thousands or millions) based on the corpus size and downstream tasks." - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "id": "b7f70a8f", - "metadata": {}, - "outputs": [], - "source": [ - "from pecos.utils.featurization.text.preprocess import Preprocessor\n", - "\n", - "input_text_path = \"./text2text_demo/training-data.txt\"\n", - "output_text_path = \"./text2text_demo/output-labels.txt\"\n", - "model_folder = \"./text2text_demo/pecos-text2text-model\"\n", - "\n", - "parsed_result = Preprocessor.load_data_from_file(input_text_path, output_text_path) # Read files\n", - "corpus = parsed_result[\"corpus\"] # Corpus input text: List of strings\n", - "\n", - "vectorizer_config = {\n", - " \"type\": \"tfidf\",\n", - " \"kwargs\": {\n", - " \"base_vect_configs\": [\n", - " \n", - " {\n", - " \"ngram_range\": [1, 1],\n", - " \"max_df_ratio\": 0.98,\n", - " \"analyzer\": \"word\",\n", - " },\n", - " {\n", - " \"ngram_range\": [2, 2],\n", - " \"max_df_ratio\": 0.98,\n", - " \"analyzer\": \"word\",\n", - " },\n", - " {\n", - " \"ngram_range\": [3, 3],\n", - " \"max_df_ratio\": 0.98,\n", - " \"analyzer\": \"char_wb\",\n", - " },\n", - " ],\n", - " },\n", - " }\n", - "\n", - "preprocessor = Preprocessor.train(corpus, vectorizer_config)\n", - "preprocessor.save(model_folder) " - ] - }, - { - "cell_type": "markdown", - "id": "a0300f8c", - "metadata": {}, - "source": [ - "#### Preprocessing with a Trained Text Preprocessor\n", - "\n", - "The function `predict` of a trained text preprocessor encodes texts in a **text data file** into a CSR matrix of shape `(num_inst, dim)` as numerical vector representations, where `num_inst` is the number of instances in the file; `dim` is the number of feature dimensions." - ] - }, - { - "cell_type": "code", - "execution_count": 28, - "id": "3b182171", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "The file consists of 4 instances with 405-dimensional features in a csr matrix.\n", - "\n", - "Text 0: Alan Turing is widely considered to be the father of theoretical computer science and artificial intelligence.\n", - "Text 1: Hinton was co-author of a highly cited paper published in 1986 that popularized the backpropagation algorithm for training multi-layer neural networks.\n", - "Text 2: Hinton received the 2018 Turing Award, together with Yoshua Bengio and Yann LeCun, for their work on artificial intelligence and deep learning.\n", - "Text 3: Yoshua Bengio is a Canadian computer scientist, most noted for his work on artificial neural networks and deep learning.\n", - "\n", - "The cosine similarity is 0.0076 between text 0 and text 1.\n", - "The cosine similarity is 0.0325 between text 0 and text 2.\n", - "The cosine similarity is 0.0082 between text 1 and text 2.\n", - "The cosine similarity is 0.0366 between text 0 and text 3.\n", - "The cosine similarity is 0.0267 between text 1 and text 3.\n", - "The cosine similarity is 0.0943 between text 2 and text 3.\n" - ] - } - ], - "source": [ - "# Obtaining numerical vectors from text\n", - "X = preprocessor.predict(corpus)\n", - "\n", - "print(\"The file consists of {} instances \"\n", - " \"with {}-dimensional features \"\n", - " \"in a {} matrix.\\n\".format(*X.shape, X.getformat()))\n", - "\n", - "from sklearn.metrics.pairwise import cosine_similarity\n", - "\n", - "sim = cosine_similarity(X)\n", - "\n", - "for i, ti in enumerate(corpus):\n", - " print(\"Text {}: {}\".format(i, ti))\n", - "\n", - "print(\"\")\n", - "for i in range(X.shape[0]):\n", - " for j in range(i):\n", - " print(\"The cosine similarity is {:.4f} between text {} and text {}.\".format(sim[i][j], j, i))" - ] - }, - { - "cell_type": "markdown", - "id": "18fcd09b", - "metadata": {}, - "source": [ - "#### Efficiency of PECOS Built-in TF-IDF Vectorizer\n", - "\n", - "Moreover, the TF-IDF vectorizer in PECOS is implemented in C++ and efficient." - ] - }, - { - "cell_type": "code", - "execution_count": 29, - "id": "a3d6f675", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "PECOS TFIDF time: 27.30768s, result shaepe=(14146, 10858825), nnz=37194670\n" - ] - } - ], - "source": [ - "vectorizer_config = {\n", - " \"type\": \"tfidf\",\n", - " \"kwargs\": {\n", - " \"base_vect_configs\": [ \n", - " {\n", - " \"ngram_range\": [1, 2],\n", - " \"max_df_ratio\": 0.98,\n", - " \"analyzer\": \"word\",\n", - " },\n", - " ],\n", - " },\n", - " }\n", - "\n", - "input_text_path = \"xmc-base/wiki10-31k/X.trn.txt\"\n", - "corpus = Preprocessor.load_data_from_file(input_text_path, text_pos=0)[\"corpus\"]\n", - "\n", - "import time\n", - "start_time = time.time()\n", - "preprocessor = Preprocessor.train(corpus, vectorizer_config)\n", - "X = preprocessor.predict(input_text_path)\n", - "print(f\"PECOS TFIDF time: {time.time() - start_time:.5f}s, result shaepe={X.shape}, nnz={X.nnz}\")" - ] - }, - { - "cell_type": "markdown", - "id": "e63c62ce", - "metadata": {}, - "source": [ - "As a baseline method, we compare with the [Sklearn TFIDF vectorizer](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html):" - ] - }, - { - "cell_type": "code", - "execution_count": 30, - "id": "677f77de", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Sklearn TFIDF time: 221.65870s, result shaepe=(14146, 7269690), nnz=33505461\n" - ] - } - ], - "source": [ - "start_time = time.time()\n", - "preprocessor = Preprocessor.train(\n", - " corpus,\n", - " {\"type\": \"sklearntfidf\", \"kwargs\":{\"ngram_range\": [1, 2], \"max_df\": 0.98}},\n", - ")\n", - "X = preprocessor.predict(corpus)\n", - "print(f\"Sklearn TFIDF time: {time.time() - start_time:.5f}s, result shaepe={X.shape}, nnz={X.nnz}\")" - ] - }, - { - "cell_type": "markdown", - "id": "75f5aaf8", - "metadata": {}, - "source": [ - "### Customized PECOS Model with TF-IDF Preprocessor\n", - "\n", - "\n", - "After being powered with text preprocessors, following the [aforementioned illustration](#Structure-of-a-Customized-PECOS-Model), we demonstrate an example of declaring a **customized PECOS model class** based on a TF-IDF preprocessor and a XR-Linear model." - ] - }, - { - "cell_type": "code", - "execution_count": 31, - "id": "3893c23b", - "metadata": {}, - "outputs": [], - "source": [ - "import json\n", - "from os import path\n", - "import pathlib\n", - "from pecos.utils.featurization.text.preprocess import Preprocessor\n", - "from pecos.xmc.xlinear.model import XLinearModel\n", - "from pecos.xmc import Indexer, LabelEmbeddingFactory\n", - "from pecos.utils import smat_util\n", - "\n", - "class CustomPECOS:\n", - " def __init__(self, preprocessor=None, xlinear_model=None, output_items=None):\n", - " self.preprocessor = preprocessor\n", - " self.xlinear_model = xlinear_model\n", - " self.output_items = output_items\n", - " \n", - " @classmethod\n", - " def train(cls, input_text_path, output_text_path):\n", - " \"\"\"Train a CustomPECOS model\n", - " \n", - " Args: \n", - " input_text_path (str): Text input file name. \n", - " output_text_path (str): The file path for output text items.\n", - " vectorizer_config (str): Json_format string for vectorizer config (default None). e.g. {\"type\": \"tfidf\", \"kwargs\": {}}\n", - " \n", - " Returns:\n", - " A CustomPECOS object\n", - " \"\"\"\n", - " # Obtain X_text, Y\n", - " parsed_result = Preprocessor.load_data_from_file(input_text_path, output_text_path)\n", - " Y = parsed_result[\"label_matrix\"]\n", - " corpus = parsed_result[\"corpus\"]\n", - "\n", - " # Train TF-IDF vectorizer\n", - " preprocessor = Preprocessor.train(corpus, {\"type\": \"tfidf\", \"kwargs\":{}}) \n", - " X = preprocessor.predict(corpus) \n", - " \n", - " # Train a XR-Linear model with TF-IDF features\n", - " label_feat = LabelEmbeddingFactory.create(Y, X, method=\"pifa\")\n", - " cluster_chain = Indexer.gen(label_feat)\n", - " xlinear_model = XLinearModel.train(X, Y, C=cluster_chain)\n", - " \n", - " # Load output items\n", - " with open(output_text_path, \"r\", encoding=\"utf-8\") as f:\n", - " output_items = [q.strip() for q in f]\n", - " \n", - " return cls(preprocessor, xlinear_model, output_items)\n", - " \n", - " def predict(self, corpus):\n", - " \"\"\"Predict labels for given inputs\n", - " \n", - " Args:\n", - " corpus (list of strings): input strings.\n", - " Returns:\n", - " csr_matrix: predicted label matrix (num_samples x num_labels)\n", - " \"\"\"\n", - " X = self.preprocessor.predict(corpus)\n", - " Y_pred = self.xlinear_model.predict(X)\n", - " return smat_util.sorted_csr(Y_pred)\n", - "\n", - " def save(self, model_folder):\n", - " \"\"\"Save the CustomPECOS model\n", - "\n", - " Args:\n", - " model_folder (str): folder name to save\n", - " \"\"\"\n", - " self.preprocessor.save(f\"{model_folder}/preprocessor\")\n", - " self.xlinear_model.save(f\"{model_folder}/xlinear_model\")\n", - " with open(f\"{model_folder}/output_items.json\", \"w\", encoding=\"utf-8\") as fp:\n", - " json.dump(self.output_items, fp)\n", - "\n", - " @classmethod\n", - " def load(cls, model_folder):\n", - " \"\"\"Load the CustomPECOS model\n", - "\n", - " Args:\n", - " model_folder (str): folder name to load\n", - " Returns:\n", - " CustomPECOS\n", - " \"\"\"\n", - " preprocessor = Preprocessor.load(f\"{model_folder}/preprocessor\")\n", - " xlinear_model = XLinearModel.load(f\"{model_folder}/xlinear_model\")\n", - " with open(f\"{model_folder}/output_items.json\", \"r\", encoding=\"utf-8\") as fin:\n", - " output_items = json.load(fin)\n", - " return cls(preprocessor, xlinear_model, output_items)" - ] - }, - { - "cell_type": "markdown", - "id": "fcdbb2c6", - "metadata": {}, - "source": [ - "### Operating the Customized PECOS Model\n", - "\n", - "With a well-declared model class, the customized PECOS model can be modularized and very convenient to use." - ] - }, - { - "cell_type": "code", - "execution_count": 32, - "id": "24134357", - "metadata": {}, - "outputs": [], - "source": [ - "# Declare the path for model serialization and preprocessor configuration.\n", - "model_folder = \"./text2text_demo/pecos-CustomPECOS-model\"\n", - "\n", - "# Train and save the trained model\n", - "input_text_path = \"./text2text_demo/training-data.txt\"\n", - "output_text_path = \"./text2text_demo/output-labels.txt\"\n", - "model = CustomPECOS.train(input_text_path, output_text_path)\n", - "model.save(model_folder)\n", - "\n", - "# Load the trained model and predict\n", - "model = model.load(model_folder)\n", - "testing_text_path = \"./text2text_demo/testing-data.txt\"\n", - "Y_pred = model.predict(testing_text_path)" - ] - }, - { - "cell_type": "code", - "execution_count": 33, - "id": "31efd9ac", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Text Input: In 1989, Yann LeCun et al. applied the standard backpropagation algorithm on neural networks for hand digit recognition.\n", - "Score 0.9515: Machine learning researchers\n", - "Score 0.8233: Artificial intelligence researchers\n", - "Score 0.4659: Deep Learning\n", - "Score 0.2779: British computer scientists\n", - "Score 0.0569: Turing Award laureates\n", - "Score 0.0129: Computability theorists\n" - ] - } - ], - "source": [ - "test_texts = Preprocessor.load_data_from_file(testing_text_path, output_text_path)[\"corpus\"]\n", - "\n", - "for i, text in enumerate(test_texts):\n", - " print(\"Text Input: {}\".format(text))\n", - " for j in range(Y_pred.indptr[i], Y_pred.indptr[i + 1]):\n", - " pred_label = model.output_items[Y_pred.indices[j]]\n", - " pred_score = Y_pred.data[j]\n", - " print(\"Score {:.4f}: {}\".format(pred_score, pred_label))" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.7.10" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/tutorials/kdd22/Session 4 Utilities in PECOS.ipynb b/tutorials/kdd22/Session 4 Utilities in PECOS.ipynb index 81757997..20f4f90b 100644 --- a/tutorials/kdd22/Session 4 Utilities in PECOS.ipynb +++ b/tutorials/kdd22/Session 4 Utilities in PECOS.ipynb @@ -5,794 +5,780 @@ "id": "b1ebc316", "metadata": {}, "source": [ - "# Utilities in PECOS\n", + "# PECOS Utilities\n", "\n", - "PECOS provides various useful interfaces and utility functions for XMR problems and related tasks. In this session, we will introduce how to tackle arbitrary data formats for XMR, and then present some utilities in PECOS for efficient matrix operations and hierarchical clustering.\n", + "PECOS provides various useful interfaces and utility functions for XMC problems and related tasks. In this session, we will present some utilities in PECOS for efficient matrix operations and hierarchical clustering.\n", "\n", - "### Install PECOS through Python PIP" + "## Outline\n", + "\n", + "1. Sparse Matrix Operations\n", + "2. Hierarchical Clustering" ] }, { - "cell_type": "code", - "execution_count": null, - "id": "4eba0f0b", + "cell_type": "markdown", + "id": "c2b3f61e", "metadata": {}, - "outputs": [], "source": [ - "! pip install libpecos" + "## 1. Sparse Matrix Operations\n", + "\n", + "Most of the computations in PECOS are based on sparse matrices, so PECOS also provides various useful and efficient operation utilities for sparse matrices." ] }, { "cell_type": "markdown", - "id": "3e9cc45c", + "id": "5fba58d4", "metadata": {}, "source": [ - "## Working with Arbitrary Data Formats\n", + "### 1.1 Genric Matriox IO and Conversion\n", "\n", - "PECOS is a general machine learning framework and able to fit arbitary data format and interact with different data manipulation and analysis libraries like [Pandas](https://pandas.pydata.org/). In the following example, we will show how to learn a PECOS model with Pandas-loaded data of text, categorical, and numerical features." + "`smat_util.load_matrix` and `smat_util.save_matrix` provide generic interfaces for loading and storing matrices in arbitrary common formats, including [dense matrix](https://numpy.org/doc/stable/reference/generated/numpy.array.html) in NumPy or different sparse matrix formats (i.e., [sparse Compressed Sparse Row (CSR) matrix](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.html), [sparse Compressed Sparse Column (CSC) matrix](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csc_matrix.html), and [sparse COOrdinate (COO) matrix](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csc_matrix.html))." ] }, { "cell_type": "code", "execution_count": 1, - "id": "9f0f17a7", - "metadata": {}, - "outputs": [], - "source": [ - "import pecos\n", - "import pandas as pd\n", - "import numpy as np" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "526820f8", + "id": "385ed0ba", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Archive: drugLib_raw.zip\r\n", - " inflating: drugLibTest_raw.tsv \r\n", - " inflating: drugLibTrain_raw.tsv \r\n" + "Dense Matrtix IO\n", + "mat is a matrix with a shape (2, 3).\n", + "[[0.32706124 0.94765886 0.16764024]\n", + " [0.29065096 0.23160388 0.3871939 ]]\n", + "mat_loaded is a matrix with a shape (2, 3).\n", + "[[0.32706124 0.94765886 0.16764024]\n", + " [0.29065096 0.23160388 0.3871939 ]]\n", + "\n", + "csr Sparse Matrix IO\n", + "mat is a matrix with a shape (5, 4) and 4 non-zero values.\n", + " (1, 1)\t0.3859982113301277\n", + " (2, 1)\t0.5399444869534915\n", + " (3, 1)\t0.008896715300809821\n", + " (4, 2)\t0.9634283904734527\n", + "mat_loaded is a matrix with a shape (5, 4) and 4 non-zero values.\n", + " (1, 1)\t0.3859982113301277\n", + " (2, 1)\t0.5399444869534915\n", + " (3, 1)\t0.008896715300809821\n", + " (4, 2)\t0.9634283904734527\n", + "\n", + "csc Sparse Matrix IO\n", + "mat is a matrix with a shape (5, 4) and 4 non-zero values.\n", + " (2, 0)\t0.8430107100693552\n", + " (0, 3)\t0.5602000410939516\n", + " (3, 3)\t0.4358575080842668\n", + " (4, 3)\t0.454532975053182\n", + "mat_loaded is a matrix with a shape (5, 4) and 4 non-zero values.\n", + " (2, 0)\t0.8430107100693552\n", + " (0, 3)\t0.5602000410939516\n", + " (3, 3)\t0.4358575080842668\n", + " (4, 3)\t0.454532975053182\n", + "\n", + "coo Sparse Matrix IO\n", + "mat is a matrix with a shape (5, 4) and 4 non-zero values.\n", + " (2, 0)\t0.6008844736009242\n", + " (3, 3)\t0.5296164005351621\n", + " (3, 2)\t0.6884529935778093\n", + " (0, 3)\t0.847894528567365\n", + "mat_loaded is a matrix with a shape (5, 4) and 4 non-zero values.\n", + " (2, 0)\t0.6008844736009242\n", + " (3, 3)\t0.5296164005351621\n", + " (3, 2)\t0.6884529935778093\n", + " (0, 3)\t0.847894528567365\n", + "\n" ] } ], "source": [ - "! wget -nv -nc https://archive.ics.uci.edu/ml/machine-learning-databases/00461/drugLib_raw.zip\n", - "! unzip -o drugLib_raw.zip" + "from pecos.utils import smat_util\n", + "import numpy as np\n", + "import scipy.sparse as smat\n", + "\n", + "print(\"Dense Matrtix IO\")\n", + "mat = np.random.rand(2, 3)\n", + "print(f\"mat is a {type(mat)} matrix with a shape {mat.shape}.\")\n", + "print(mat)\n", + "smat_util.save_matrix(\"mat.npz\", mat)\n", + "mat_loaded = smat_util.load_matrix(\"mat.npz\")\n", + "print(f\"mat_loaded is a {type(mat_loaded)} matrix with a shape {mat_loaded.shape}.\")\n", + "print(mat)\n", + "print(\"\") \n", + "\n", + "for matrix_format in [\"csr\", \"csc\", \"coo\"]:\n", + " print(f\"{matrix_format} Sparse Matrix IO\")\n", + " mat = smat.random(5, 4, density=0.2, format=matrix_format)\n", + " print(f\"mat is a {type(mat)} matrix\"\n", + " f\" with a shape {mat.shape} and {mat.nnz} non-zero values.\")\n", + " print(mat)\n", + " \n", + " smat_util.save_matrix(\"mat.npz\", mat)\n", + " mat_loaded = smat_util.load_matrix(\"mat.npz\")\n", + " print(f\"mat_loaded is a {type(mat_loaded)} matrix\"\n", + " f\" with a shape {mat_loaded.shape} and {mat_loaded.nnz} non-zero values.\")\n", + " print(mat_loaded)\n", + " print(\"\") " ] }, { "cell_type": "code", - "execution_count": 3, - "id": "ac8ab46f", + "execution_count": 2, + "id": "2579b855", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Training DataFrame consists of 3107 instances.\n", - "Testing DataFrame consists of 1036 instances.\n", - "Index(['Unnamed: 0', 'urlDrugName', 'rating', 'effectiveness', 'sideEffects',\n", - " 'condition', 'benefitsReview', 'sideEffectsReview', 'commentsReview'],\n", - " dtype='object')\n" + "Original Matrix mat\n", + " [[0.16936286 0.78425304 0.8562633 0.61722574 0.4486684 0.23233178]\n", + " [0.79099373 0.3961628 0.91564054 0.58414229 0.43155964 0.55876417]\n", + " [0.44718835 0.05151288 0.42833526 0.12533758 0.2968885 0.82826553]\n", + " [0.57886779 0.45415528 0.24104546 0.04155873 0.7281743 0.08374103]] \n", + "\n", + "csr_mat = dense_to_csr(mat)\n", + "csr_mat is a matrix with a shape (4, 6) and 24 non-zero values.\n", + "[[0.16936286 0.78425304 0.8562633 0.61722574 0.4486684 0.23233178]\n", + " [0.79099373 0.3961628 0.91564054 0.58414229 0.43155964 0.55876417]\n", + " [0.44718835 0.05151288 0.42833526 0.12533758 0.2968885 0.82826553]\n", + " [0.57886779 0.45415528 0.24104546 0.04155873 0.7281743 0.08374103]] \n", + "\n", + "csr_mat_topk = dense_to_csr(mat, topk=2)\n", + "csr_mat is a matrix with a shape (4, 6) and 8 non-zero values.\n", + "[[0. 0.78425304 0.8562633 0. 0. 0. ]\n", + " [0.79099373 0. 0.91564054 0. 0. 0. ]\n", + " [0. 0. 0.42833526 0. 0. 0.82826553]\n", + " [0. 0.45415528 0. 0. 0.7281743 0. ]] \n", + "\n" ] } ], "source": [ - "train_df = pd.read_csv(\"drugLibTrain_raw.tsv\", sep=\"\\t\")\n", - "test_df = pd.read_csv(\"drugLibTest_raw.tsv\", sep=\"\\t\")\n", - "print(f\"Training DataFrame consists of {len(train_df)} instances.\")\n", - "print(f\"Testing DataFrame consists of {len(test_df)} instances.\")\n", - "print(train_df.columns)" + "mat = np.random.rand(4, 6)\n", + "\n", + "print(f\"Original Matrix mat\\n\", mat, \"\\n\")\n", + "\n", + "print(\"csr_mat = dense_to_csr(mat)\")\n", + "csr_mat = smat_util.dense_to_csr(mat)\n", + "print(f\"csr_mat is a {type(csr_mat)} matrix\"\n", + " f\" with a shape {csr_mat.shape} and {csr_mat.nnz} non-zero values.\")\n", + "print(csr_mat.toarray(), \"\\n\")\n", + "\n", + "print(\"csr_mat_topk = dense_to_csr(mat, topk=2)\")\n", + "csr_mat_topk = smat_util.dense_to_csr(mat, topk=2)\n", + "print(f\"csr_mat is a {type(csr_mat_topk)} matrix\"\n", + " f\" with a shape {csr_mat_topk.shape} and {csr_mat_topk.nnz} non-zero values.\")\n", + "print(csr_mat_topk.toarray(), \"\\n\")" ] }, { - "cell_type": "code", - "execution_count": 4, - "id": "6d6ee989", + "cell_type": "markdown", + "id": "b2746e3c", "metadata": {}, - "outputs": [], "source": [ - "label_name = \"effectiveness\"\n", - "text_features = [\"condition\", \"benefitsReview\", \"sideEffectsReview\", \"commentsReview\"]\n", - "categorical_features = [\"sideEffects\"]\n", - "numerical_features = [\"rating\"]" + "### 1.2. Memory-efficient Sparse Matrix Operations\n", + "\n", + "To manipulate with sparse matrix, PECOS provides many useful memory-efficient functions. For example, for CSR matrices, we have following functions to combine multiple matrices.\n", + "\n", + "* `hstack_csr([mat, mat, mat]`\n", + "* `vstack_csr([mat, mat, mat]`\n", + "* `block_diag_csr([mat, mat, mat]`\n", + "\n", + "These funcations are also available for CSC matrices as `hstack_csc`, `vstack_csr`, and `block_diag_csr`.\n" ] }, { "cell_type": "code", - "execution_count": 5, - "id": "c25b1cb2", + "execution_count": 3, + "id": "b9dac617", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Original Matrix mat\n", + " [[0.77445782 0.45465131]\n", + " [0. 0.43456434]\n", + " [0. 0. ]] \n", + "\n", + "hstack_csr([mat, mat, mat])\n", + "[[0.77445782 0.45465131 0.77445782 0.45465131 0.77445782 0.45465131]\n", + " [0. 0.43456434 0. 0.43456434 0. 0.43456434]\n", + " [0. 0. 0. 0. 0. 0. ]] \n", + "\n", + "vstack_csr([mat, mat, mat])\n", + "[[0.77445782 0.45465131]\n", + " [0. 0.43456434]\n", + " [0. 0. ]\n", + " [0.77445782 0.45465131]\n", + " [0. 0.43456434]\n", + " [0. 0. ]\n", + " [0.77445782 0.45465131]\n", + " [0. 0.43456434]\n", + " [0. 0. ]] \n", + "\n", + "block_diag_csr([mat, mat, mat])\n", + "[[0.77445782 0.45465131 0. 0. 0. 0. ]\n", + " [0. 0.43456434 0. 0. 0. 0. ]\n", + " [0. 0. 0. 0. 0. 0. ]\n", + " [0. 0. 0.77445782 0.45465131 0. 0. ]\n", + " [0. 0. 0. 0.43456434 0. 0. ]\n", + " [0. 0. 0. 0. 0. 0. ]\n", + " [0. 0. 0. 0. 0.77445782 0.45465131]\n", + " [0. 0. 0. 0. 0. 0.43456434]\n", + " [0. 0. 0. 0. 0. 0. ]] \n", + "\n" + ] + } + ], "source": [ - "X_trn_list = []\n", - "X_tst_list = []" + "from pecos.utils import smat_util\n", + "import scipy.sparse as smat\n", + "\n", + "mat = smat.random(3, 2, density=0.5, format=\"csr\")\n", + "print(f\"Original Matrix {type(mat)} mat\\n\", mat.toarray(), \"\\n\")\n", + "\n", + "print(f\"hstack_csr([mat, mat, mat])\")\n", + "print(smat_util.hstack_csr([mat, mat, mat]).toarray(), \"\\n\")\n", + "\n", + "print(f\"vstack_csr([mat, mat, mat])\")\n", + "print(smat_util.vstack_csr([mat, mat, mat]).toarray(), \"\\n\")\n", + "\n", + "print(f\"block_diag_csr([mat, mat, mat])\")\n", + "print(smat_util.block_diag_csr([mat, mat, mat]).toarray(), \"\\n\")" ] }, { "cell_type": "markdown", - "id": "4f72d047", + "id": "5f9cf7f1", "metadata": {}, "source": [ - "### Label Encoding\n", + "### 1.3. Sparse-to-sparse Matrix Multiplication (SpMM)\n", + "\n", + "Many operations in PECOS or XMC problems rely on Sparse-to-sparse Matrix Multiplication (SpMM), such as the computation of PIFA features. It is also one of the key primitives in large-scale linear algebra operations, with a broad range of applications in machine learning and natural language processing.\n", "\n", - "To encode labels into the sparse matrix format compatible to PECOS, [OneHotEncoder](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MultiLabelBinarizer.html#sklearn.preprocessing.MultiLabelBinarizer) and [MultiLabelBinarizer](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MultiLabelBinarizer.html#sklearn.preprocessing.MultiLabelBinarizer) are helpful for the scenarios of multi-class and multi-label classification." + "For SpMM, PECOS provides a highly optimized multi-core CPU implementation with state-of-the-art performance, where the underlying operations are implemented and optimized in C/C++.\n", + "Specifically, the Python interface and parameters are as follows:\n", + "\n", + "```python\n", + "from pecos.core import clib as pecos_clib\n", + "Z = pecos_clib.sparse_matmul(X, Y, eliminate_zeros=False, sorted_indices=True, threads=-1)\n", + "```\n", + "* Parameters\n", + " * `X` (scipy.sparse.csr_matrix or scipy.sparse.csc_matrix): the first sparse matrix to be multiplied.\n", + " * `Y` (scipy.sparse.csr_matrix or scipy.sparse.csc_matrix): the second sparse matrix to be multiplied.\n", + " * `eliminate_zeros` (bool, optional): if true, then eliminate (potential) zeros created by maxnnz in output matrix Z. Default is false.\n", + " * `sorted_indices` (bool, optional): if true, then sort the Z.indices for the output matrix Z. Default is true.\n", + " * `threads` (int, optional): The number of threads. Default -1 to use all CPU cores." ] }, { "cell_type": "code", - "execution_count": 6, - "id": "4f0d9ec7", + "execution_count": 4, + "id": "b2e6c54a", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Y_trn is a csr matrix with a shape (3107, 5) and 3107 non-zero values.\n", - "Y_tst is a csr matrix with a shape (1036, 5) and 1036 non-zero values.\n" + "||Z_true - Z_pred|| = 0.0\n" ] } ], "source": [ - "from sklearn.preprocessing import OneHotEncoder\n", - "\n", - "label_encoder = OneHotEncoder(dtype=np.float32)\n", - "Y_trn = label_encoder.fit_transform(train_df[[label_name]])\n", - "Y_tst = label_encoder.transform(test_df[[label_name]])\n", - "\n", - "print(f\"Y_trn is a {Y_trn.getformat()} matrix with a shape {Y_trn.shape} and {Y_trn.nnz} non-zero values.\")\n", - "print(f\"Y_tst is a {Y_tst.getformat()} matrix with a shape {Y_tst.shape} and {Y_tst.nnz} non-zero values.\")" + "import numpy as np\n", + "import scipy.sparse as smat\n", + "from scipy.sparse import linalg\n", + "from pecos.core import clib as pecos_clib\n", + "X = smat.random(1000, 1000, density=0.01, format='csr', dtype=np.float32)\n", + "Y = smat.random(1000, 1000, density=0.01, format='csr', dtype=np.float32)\n", + "Z_true = X.dot(Y)\n", + "Z_pred = pecos_clib.sparse_matmul(X, Y)\n", + "print(\"||Z_true - Z_pred|| = \", linalg.norm(Z_true - Z_pred))" ] }, { "cell_type": "code", - "execution_count": 7, - "id": "62ec7371", + "execution_count": 5, + "id": "14e4bded", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Y_trn_mlb is a csr matrix with a shape (3107, 5) and 3107 non-zero values.\n", - "Y_tst_mlb is a csr matrix with a shape (1036, 5) and 1036 non-zero values.\n" - ] - } - ], + "outputs": [], "source": [ - "from sklearn.preprocessing import MultiLabelBinarizer\n", - "\n", - "label_encoder_multilabel = MultiLabelBinarizer(sparse_output=True)\n", - "Y_trn_mlb = label_encoder.fit_transform([[lbl] for lbl in train_df[label_name].tolist()])\n", - "Y_tst_mlb = label_encoder.fit_transform([[lbl] for lbl in test_df[label_name].tolist()])\n", - "print(f\"Y_trn_mlb is a {Y_trn_mlb.getformat()} matrix with a shape {Y_trn_mlb.shape} and {Y_trn_mlb.nnz} non-zero values.\")\n", - "print(f\"Y_tst_mlb is a {Y_tst_mlb.getformat()} matrix with a shape {Y_tst_mlb.shape} and {Y_tst_mlb.nnz} non-zero values.\")" + "import time\n", + "DATASET = \"wiki10-31k\"\n", + "X = smat_util.load_matrix(f\"xmc-base/{DATASET}/tfidf-attnxml/X.trn.npz\").astype(np.float32)\n", + "Y = smat_util.load_matrix(f\"xmc-base/{DATASET}/Y.trn.npz\").astype(np.float32)\n", + "YT_csr = Y.T.tocsr()\n", + "X_csr = X.tocsr()" ] }, { "cell_type": "markdown", - "id": "260a9a8a", + "id": "bb5af1e9", "metadata": {}, "source": [ - "### Text Feature Encoding\n", + "#### Benchmarking Sparse Matrix Muplication\n", "\n", - "As introduced in Session 1, we can use PECOS vectorizer for featurize text data. In addition, the encoder of [XR-Transformer](https://github.com/amzn/pecos/tree/mainline/pecos/xmc/xtransformer) can be also utilized for deriving text features with proper fine-tuning." + "The SpMM utility has state-of-the-art performance in efficiency as shown in the following figure.\n", + "\n", + "
\n", + "
\n", + "
\n", + "\n", + "In this part, we provide some hands-on instructions for benchmarking different methods for SpMM." ] }, { "cell_type": "code", - "execution_count": 8, - "id": "96fef619", + "execution_count": 6, + "id": "c71cffdb", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "condition: (3107, 3759) and (1036, 3759) in training and testing.\n", - "benefitsReview: (3107, 72861) and (1036, 72861) in training and testing.\n", - "sideEffectsReview: (3107, 64321) and (1036, 64321) in training and testing.\n", - "commentsReview: (3107, 91731) and (1036, 91731) in training and testing.\n" - ] - } - ], + "outputs": [], "source": [ - "from pecos.utils.featurization.text.vectorizers import Vectorizer\n", + "# Benchmarking SciPy\n", "\n", - "for feature_name in text_features:\n", - " vectorizer_config = {\n", - " \"type\": \"tfidf\",\n", - " \"kwargs\": {\n", - " \"base_vect_configs\": [\n", + "start = time.time()\n", + "Z = YT_csr.dot(X_csr)\n", + "Z.sort_indices()\n", + "run_time_scipy = time.time() - start" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "2f2b9795", + "metadata": {}, + "outputs": [], + "source": [ + "# Benchmarking PyTorch\n", "\n", - " {\n", - " \"ngram_range\": [1, 2],\n", - " \"max_df_ratio\": 0.98,\n", - " \"analyzer\": \"word\",\n", - " },\n", - " ],\n", - " },\n", - " } \n", - " train_texts = [str(x) for x in train_df[feature_name].tolist()]\n", - " test_texts = test_df[feature_name].tolist()\n", - " vectorizer = Vectorizer.train(train_texts, config=vectorizer_config)\n", - " X_trn_local = vectorizer.predict(train_texts)\n", - " X_tst_local = vectorizer.predict(test_texts)\n", - " print(f\"{feature_name}: {X_trn_local.shape} and {X_tst_local.shape} in training and testing.\")\n", + "import torch\n", + "\n", + "def csr_to_coo(A):\n", + " A_coo = smat.coo_matrix(A)\n", + " indices = np.vstack([A_coo.row, A_coo.col]).T\n", + " values = A_coo.data\n", + " return indices, values\n", + "\n", + "def get_pt_data(A_csr):\n", + " A_indices, A_values = csr_to_coo(A_csr)\n", + " A_pt = torch.sparse_coo_tensor(\n", + " A_indices.T.astype(np.int64),\n", + " A_values.astype(np.float32),\n", + " A_csr.shape,\n", + " )\n", + " return A_pt\n", " \n", - " X_trn_list.append(X_trn_local)\n", - " X_tst_list.append(X_tst_local)" + "YT_pt = get_pt_data(YT_csr)\n", + "X_pt = get_pt_data(X_csr)\n", + "start = time.time()\n", + "Z_pt = torch.sparse.mm(YT_pt, X_pt)\n", + "run_time_pytorch = time.time() - start" ] }, { - "cell_type": "markdown", - "id": "38e75fa2", + "cell_type": "code", + "execution_count": 8, + "id": "54b24694", "metadata": {}, + "outputs": [], "source": [ - "### Categorical Feature Encoding\n", + "# Benchmarking PECOS\n", "\n", - "Similar to labels, categorical features can also be considered as one-hot or multi-hot embeddings." + "start = time.time()\n", + "Z = pecos_clib.sparse_matmul(\n", + " YT_csr, X_csr,\n", + " eliminate_zeros=False,\n", + " sorted_indices=True\n", + ")\n", + "run_time_pecos = time.time() - start" ] }, { "cell_type": "code", "execution_count": 9, - "id": "386b96ad", + "id": "e12f29e9", "metadata": {}, "outputs": [ { - "name": "stdout", - "output_type": "stream", - "text": [ - "sideEffects: (3107, 5) and (1036, 5) in training and testing.\n" - ] + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD4CAYAAADmWv3KAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAW/0lEQVR4nO3deZSldX3n8fcHEFkEDNIqitDigqMMspQoB6Oho44oOh4Vlbjr0CbHBUYzc3BiEjGTBCfBNY5nWkARFAVUNhcUFKMJA3RDsx+HBkVRAo0aWWQR/M4f9yktiqpbT917n9vVl/frnHvqPvv3culv/er7/J7fL1WFJGnybLKhA5AkdcMEL0kTygQvSRPKBC9JE8oEL0kTarMNHcBMO+ywQy1fvnxDhyFJG401a9bcUlXL5tq2pBL88uXLWb169YYOQ5I2Gkmun2+bJRpJmlAmeEmaUCZ4SZpQJnhJmlAmeEmaUCZ4SZpQJnhJmlAmeEmaUCZ4SZpQnT3JmmQ34IszVu0K/FVVfaSL6y0/4qtdnFbAj4568YYOQdIAOkvwVfUDYE+AJJsCPwW+0tX1JEn3N64SzR8D11bVvGMmSJJGa1wJ/jXASXNtSLIyyeokq9evXz+mcCRp8nWe4JNsDrwUOGWu7VW1qqqmqmpq2bI5R7yUJA1gHC34A4GLq+qmMVxLktQYR4I/hHnKM5Kk7nSa4JNsDTwf+HKX15EkPVCnMzpV1R3AI7q8hiRpbj7JKkkTasEWfJJHAvsDjwHuBK4AVlfVbzuOTZI0hHkTfJIDgCOA7YFLgJuBLYCXAU9IcipwdFXdOoY4JUmL1K8F/yLg0Kr68ewNSTYDDqJ3A/VLHcUmSRrCvAm+qv5bn233Aqd1EZAkaTQWvMma5LAk26bn2CQXJ3nBOIKTJA2uTS+atzR19hcAfwC8Hjiq06gkSUNrk+DT/HwRcEJVXTljnSRpiWqT4Nck+Sa9BH92km0Au0hK0hLX5knWt9KbuOO6qvp1kkcAb+40KknS0Pr1g9971qpdEyszkrSx6NeCP7r5uQWwD3AZvdr7HsBqYL9uQ5MkDWPeGnxVHVBVBwA3Avs0k3LsA+xFb35VSdIS1uYm625Vdfn0QlVdAfyH7kKSJI1Cm5uslyU5BjixWX4tvXKNJGkJa5Pg3wz8GXBYs/zPwCc7i0iSNBILJviqugv4cPOSJG0k2owHvz/wfmCXmftX1a7dhSVJGlabEs2xwH8F1gD3dRuOJGlU2iT4X1XV1zuPRJI0Um0S/HeS/APwZeDu6ZVVdfFCByZ5OHAMsDtQ9EamPH+wUCVJi9EmwT+z+Tk1Y10BK1oc+1HgG1X1yiSbA1stMj5J0oDa9KI5YJATJ9kOeA7wpuY89wD3DHIuSdLitZnRabskH0qyunkd3STvhTweWA98OsklSY5JsvUc5185fe7169cP8BEkSXNpM1TBccBtwKua163Ap1sctxmwN/DJqtoLuAM4YvZOVbWqGedmatmyZa0DlyT116YG/4SqesWM5SOTrG1x3A3ADVV1QbN8KnMkeElSN9q04O9M8uzphebBpzsXOqiq/g34SZLdmlV/DFw1UJSSpEVr04L/M+D4GXX3X9LcOG3hncDnmh401+FMUJI0Nm160awFnp5k22b51rYnb46dWmg/SdLotelF83dJHl5Vt1bVrUn+IMn/HEdwkqTBtanBH1hV/z69UFW/BF7UWUSSpJFok+A3TfLQ6YUkWwIP7bO/JGkJaHOT9XPAuUmm+76/GTi+u5AkSaPQ5ibrB5NcCjyvWfU3VXV2t2FJkobVpgUPcDVwb1Wdk2SrJNtU1W1dBiZJGk6bXjSH0nsK9f80qx4LnNZhTJKkEWhzk/XtwP70xqChqq4BHtllUJKk4bVJ8Hc3Q/0CkGQzeuPBS5KWsDYJ/rtJ/gewZZLnA6cAZ3YbliRpWG0S/BH0xnW/HHgb8DXgfV0GJUkaXptukr8FPgV8Ksn2wE5VZYlGkpa4Nr1ozkuybZPc19BL9B/uPjRJ0jDalGi2a0aQfDnw2ap6Jr2x3SVJS1ibBL9Zkh3pTdd3VsfxSJJGpE2C/wBwNrCuqi5KsitwTbdhSZKG1eYm6yn0ukZOL18HvGL+IyRJS8G8Lfgk72turM63fUWSg7oJS5I0rH4t+MuBM5PcBVxMry/8FsCTgD2Bc4C/6zpASdJg5k3wVXU6cHqSJ9Ebi2ZHeuPRnAisrKo7xxOiJGkQbWrw1zDgTdUkPwJuA+6jN9ywE3BL0pi0HQ9+GAdU1S1juI4kaYY23SQlSRuhrhN8Ad9MsibJyrl2SLIyyeokq9evX99xOJL04NFmLJonJzk3yRXN8h5J2o4m+eyq2hs4EHh7kufM3qGqVlXVVFVNLVu2bFHBS5Lm16YF/yngvcBvAKrqMuA1bU5eVT9tft4MfAXYd7AwJUmL1SbBb1VVF85ad+9CByXZOsk20++BFwBXLD5ESdIg2vSiuSXJE2im6UvySuDGFsc9CvhKkunrfL6qvjFooJKkxWmT4N8OrAKekuSnwA+B1y10UDNmzdOHC0+SNKg2DzpdBzyvKbNsUlW3dR+WJGlYCyb4JA8H3gAspzc2PABV9a4uA5MkDadNieZrwP+lN/jYb7sNR5I0Km0S/BZV9e7OI5EkjVSbbpInJDk0yY5Jtp9+dR6ZJGkobVrw9wD/APwFTVfJ5ueuXQUlSRpemwT/HuCJjggpSRuXNiWadcCvuw5EkjRabVrwdwBrk3wHuHt6pd0kJWlpa5PgT2tekqSNSJsnWY8fRyCSpNGaN8EnObmqXpXkcn7fe+Z3qmqPTiOTJA2lXwv+w83Pg8YRiB5clh/x1Q0dwsT60VEv3tAhaInol+A/AexdVdePKxhJ0uj06yaZsUUhSRq5fi34xyb52Hwb7SYpSUtbvwR/J7BmXIFIkkarX4L/uV0kJWnj1a8Gf8/YopAkjdy8Cb6qnjXOQCRJo9VmsLGhJNk0ySVJzur6WpKk3+s8wQOHAVeP4TqSpBlaJfimFf6YJDtPv1oetxPwYuCYYYKUJC3egoONJXkn8NfATfx+0u0C2oxF8xHgvwPb9Dn/SmAlwM47t/q9IUlqoc1wwYcBu1XVzxdz4iQHATdX1ZokfzTfflW1ClgFMDU19YBBzSRJg2lTovkJ8KsBzr0/8NIkPwK+AKxIcuIA55EkDaBNC/464LwkX+X+Mzp9qN9BVfVe4L0ATQv+z6vqdQNHKklalDYJ/sfNa/PmJUnaCLSZ0elIgCQPa5ZvX+xFquo84LzFHidJGtyCNfgkuye5BLgSuDLJmiRP6z40SdIw2txkXQW8u6p2qapdgPcAn+o2LEnSsNok+K2r6jvTC025ZevOIpIkjUSrXjRJ/hI4oVl+Hb2eNZKkJaxNC/4twDLgy81rWbNOkrSEtelF80vA6fkkaSMzb4JP8pGqOjzJmfTGnrmfqnppp5FJkobSrwU/XXP/x3EEIkkarXkTfFVNT7i9Z1V9dOa2JIcB3+0yMEnScNrcZH3jHOveNOI4JEkj1q8GfwjwJ8Djk5wxY9M2wC+6DkySNJx+Nfh/BW4EdgCOnrH+NuCyLoOSJA2vXw3+euB6YL/xhSNJGpU2g409K8lFSW5Pck+S+5LcOo7gJEmDa3OT9Z+AQ4BrgC2B/wJ8osugJEnDa5Pgqap1wKZVdV9VfRp4YbdhSZKG1WawsV8n2RxYm+R/0bvx2uoXgyRpw2mTqF/f7PcO4A7gccArugxKkjS8Ni34W4B7quou4MgkmwIP7TYsSdKw2rTgzwW2mrG8JXBON+FIkkalTYLfYuZE2837rfrsD0CSLZJcmOTSJFcmOXKYQCVJi9Mmwd+RZO/phST7AHe2OO5uYEVVPR3YE3hhkmcNFKUkadHa1OAPB05J8jMgwKOBVy90UFUVMN3yf0jzesC48pKkbrSZ0emiJE8BdmtW/aCqftPm5M0N2TXAE4FPVNUFc+yzElgJsPPOO7eNW5K0gHlLNElWND9fDrwEeHLzekmzbkHNg1F7AjsB+ybZfY59VlXVVFVNLVu2bICPIEmaS78W/HOBb9NL7rMVvQm4W6mqf0/yHXpPwF6xqAglSQPpN5rkXzc/3zzIiZMsA37TJPctgecDHxwoSknSovWb8OPd/Q6sqg8tcO4dgeObOvwmwMlVddbiQ5QkDaJfiWabYU5cVZcBew1zDknS4PqVaHwwSZI2Ym0m/Ng1yZlJ1ie5OcnpSXYdR3CSpMG1eZL188DJ9GrqjwFOAU7qMihJ0vDaJPitquqEqrq3eZ0IbNF1YJKk4bQZquDrSY4AvkCv//urga8l2R6gqn7RYXySpAG1SfCvan6+bdb619BL+NbjJWkJajMWzePHEYgkabT6Pei0oqq+Pd+4M1XVeqgCSdL4jWUsGknS+C04Fg3wgar64cxtSSzbSNIS16ab5JfmWHfqqAORJI1Wvxr8U4CnAdvNqsNvi/3gJWnJ61eD3w04CHg496/D3wYc2mFMkqQR6FeDPx04Pcl+VXX+GGOSJI1AvxLNx2kmyU5yyOztVfWuDuOSJA2pX4lm9diikCSNXL8SzfHjDESSNFoLDlXQTJZds9dX1YpOIpIkjUSbwcb+fMb7LYBXAPd2E44kaVTaDDa2Ztaqf0lyYUfxSJJGpE2JZvsZi5sA+wDbtTjuccBngUfRK/GsqqqPDhinJGmR2pRo1tBL0KFXmvkh8NYWx90LvKeqLk6yDbAmybeq6qqBo5UktdbZePBVdSNwY/P+tiRXA48FTPCSNAb9HnSacxz4aYsZDz7JcmAv4II5tq0EVgLsvPPObU8pSVpAvxb8qcDa5gW9Es201uPBJ3kYvREpD6+qW2dvr6pVwCqAqampB3THlCQNpl+Cfzm9eVf3AE4HTqqqdYs5eZKH0Evun3MGKEkar3nHg6+q06rqNfRmdroWODrJ95M8t82JkwQ4Fri6qj40kmglSa21mfDjLuBXwK3Aw2g/Fvz+wOuBFUnWNq8XDRamJGmx+k66Ta9Esy9wDvDRqmo9AFlVfZ/71+0lSWPUrwZ/DnAZ8H3gocAbkrxheqPDBUvS0tYvwb95bFFIkkbO4YIlaUK1uckqSdoImeAlaUINlOCTbD7qQCRJo7Vggk9yXjOWzPTyvsBFXQYlSRpem+GC/x74RpKP0RsN8kDsYSNJS16b4YLPTvKnwLeAW4C9qurfOo9MkjSUNiWavwQ+DjwHeD9wXpIXdxyXJGlIbUo0jwD2rao7gfOTfAM4Bvhqp5FJkobSpkRz+Kzl64HndxWQJGk0+g029pGqOjzJmfQm+Lifqnppp5FJkobSrwV/QvPzH8cRiCRptPqNRbMmyabAyqp67RhjkiSNQN9eNFV1H7CLT65K0sanTS+a64B/SXIGcMf0Sqfhk6SlrU2Cv7Z5bQJs06x7wE1XSdLS0ibBX1VVp8xckeTgjuKRJI1Im9Ek39tynSRpCenXD/5A4EXAY5uBxqZtC9y70ImTHAccBNxcVbsPG6gkaXH6teB/BqwG7gLWzHidAfynFuf+DPDCIeOTJA2oXz/4S4FLk3y+qn6z2BNX1T/PHEdekjRebW6yLk/y98BTgS2mV1bVrp1FJUkaWpubrJ8GPkmv7n4A8FngxFEFkGRlktVJVq9fv35Up5WkB702CX7LqjoXSFVdX1XvB0Y2HnxVraqqqaqaWrZs2ahOK0kPem1KNHcn2QS4Jsk7gJ8CD+s2LEnSsNq04A8DtgLeBewDvB5440IHJTkJOB/YLckNSd46TKCSpMVpM+HHRc3b21nEZNtVdcigQUmShtfvQacz+h3ohB+StLT1a8HvB/wEOAm4AMhYIpIkjUS/BP9oenOvHgL8Cb1Jtk+qqivHEZikpWX5EV/d0CFMrB8dNbKOifcz703Wqrqvqr5RVW8EngWsA85retJIkpa4vjdZkzyUXp/3Q4DlwMeAr3QfliRpWP1usn4W2B34GnBkVV0xtqgkSUPr14J/Hb0p+g4D3pX87h5rgKqqbTuOTZI0hH6jSbZ5CEqStESZxCVpQpngJWlCmeAlaUKZ4CVpQpngJWlCmeAlaUKZ4CVpQpngJWlCmeAlaUKZ4CVpQpngJWlCmeAlaUKZ4CVpQnWa4JO8MMkPkqxLckSX15Ik3V9nCT7JpsAngAOBpwKHJHlqV9eTJN1fly34fYF1VXVdVd0DfAH4zx1eT5I0Q985WYf0WOAnM5ZvAJ45e6ckK4GVzeLtSX7QYUxLxQ7ALRs6iLbywQ0dwZKw0Xxnfl+/82D5znaZb0OXCb6VqloFrNrQcYxTktVVNbWh41B7fmcbH7+zbks0PwUeN2N5p2adJGkMukzwFwFPSvL4JJsDrwHO6PB6kqQZOivRVNW9Sd4BnA1sChxXVVd2db2NzIOqJDUh/M42Pg/67yxVtaFjkCR1wCdZJWlCmeAlaUKZ4EckyV8kuTLJZUnWJnlAn/9mv6kkH2vevynJ+mb/q5IcOt6oJ1+S+5r/vlckOSXJVvPs9x+b/dYm+UWSHzbvzxni2p9J8srBo39wm++7m7F++nVEs/4hSY5Kck2Si5Ocn+TAZtt2ST7bDJtybfN+u2bbJkk+1lzn8iQXJXn8hvvko7PB+8FPgiT7AQcBe1fV3Ul2ADafa9+qWg2snrHqi1X1jiSPBK5MckZV3dR91A8ad1bVngBJPgf8KfCh2TtV1eXA9H6fAc6qqlPbXCDJplV134ji1e/N9939bv0sfwPsCOze/Dt8FPDcZtuxwBVV9YbmfEcCxwAHA68GHgPsUVW/TbITcEdnn2qMbMGPxo7ALVV1N0BV3VJVP0vyjCT/muTSJBcm2SbJHyU5a/YJqupm4Fpgl6YFsgx+17pYN72soXwPeGKSDyQ5fHplkr9NcthcByQ5pGnVXZH8/nnDJLcnOTrJpcB+Sd7Q/PV2aZITZpziOc3/A9fZmh/K94Anzrexad0fCrxzxr/Dm6rq5CRPBPah9wtg2geAqSRPoPfv98aq+m1z3A1V9cuOPsdYmeBH45vA45L8vyT/O8lzm77/XwQOq6qnA88D7pzvBEl2BXYF1gEnAq9tNj0PuLSq1nf6CSZcks3oDXx3OXAcMN2S24TeMxonznHMY4APAivote6fkeRlzeatgQua7/aXwPuAFc3yzF8WOwLPpvcX3lGj/lwPBrO+O4AtZ5VoXk0v+f+4qm6d4xRPBdbO/Cureb8WeBpwMvCS5lxHJ9mry88zTpZoRqCqbk+yD/CHwAH0Evvf0msVXNTscytAktmHvzrJs4G7gbdV1S+SHAecDnwEeAvw6XF8jgm1ZZK1zfvvAcdW1T1Jft78Q34UcElV/XyOY58BnDf9y7UpEzwHOA24D/hSs98K4JSqugWgqn4x4xynNS3Dq5qSgdp7wHfXvH9AiSbJHoNepKpuSLIbve9xBXBukoOr6txBz7lUmOBHpGkRnAecl+Ry4O0tD/1iVb1j1rl+kuSmJCvojcr52rkPVQvz1WuPAd4EPJpei36x7mpZd797xvsH/HZXX/N9d3NZB+ycZNs5WvFXAXsm2WS6DNP85bZns42mrPN14OtJbgJeBmz0Cd4SzQgk2S3Jk2as2hO4GtgxyTOafbZp/tRs6xh6ZYNTvIHXia8AL6TXSj97nn0uBJ6bZIf05jc4BPjuHPt9Gzg4ySMAkmzfQbzqo6p+Ta+F/9GmPEqSZU1LfB1wCb0y2rT3ARdX1bokezfluOnEvwdw/Xg/QTdM8KPxMOD49Lo6Xkav5vdX9O7Of7y5EfctYItFnPOM5ryWZzrQzFHwHeDk+X6BVtWNwBHNfpcCa6rq9Dn2u5JeSe67zXf9gF46GqnZNfjpexvvA9bTK4ddAZwFTLfm3wo8uekieS3w5GYdwCOBM5tjLgPuBf5pXB+mSw5VsEQlmQI+XFV/uKFjmURNS+1i4OCqumZDxyN1wRb8EpTegxtfAt67oWOZROlNHbkOONfkrklmC16SJpQteEmaUCZ4SZpQJnhJmlAmeEmaUCZ4SZpQ/x/ET4IYm4/UTQAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" } ], "source": [ - "from sklearn.preprocessing import OneHotEncoder\n", + "from matplotlib import pyplot as plt\n", + "plt.bar(\n", + " [1,2,3],\n", + " [run_time_scipy, run_time_pytorch, run_time_pecos],\n", + " tick_label = [\"SciPy\", \"PyTorch\", \"PECOS\"])\n", "\n", - "for feature_name in categorical_features:\n", - " local_encoder = OneHotEncoder(dtype=np.float32)\n", - " X_trn_local = local_encoder.fit_transform(train_df[[feature_name]])\n", - " X_tst_local = local_encoder.transform(test_df[[feature_name]])\n", - " print(f\"{feature_name}: {X_trn_local.shape} and {X_tst_local.shape} in training and testing.\")\n", - " \n", - " X_trn_list.append(X_trn_local)\n", - " X_tst_list.append(X_tst_local)" + "plt.ylabel(\"Matrix Multiplication Time (seconds)\");" ] }, { "cell_type": "markdown", - "id": "6deaf4e8", + "id": "3e9cc45c", "metadata": {}, "source": [ - "### Numerical Features Encoding\n", + "### 1.4. Sparse Matrix Operations for Working with Arbitrary Data Formats\n", "\n", - "Numberical features can be directly incorporated as model inputs after some simple normalization." + "PECOS is a general machine learning framework and able to fit arbitary data format and interact with different data manipulation and analysis libraries like [Pandas](https://pandas.pydata.org/). In the following example, we will show how to learn a PECOS model with Pandas-loaded data of text, categorical, and numerical features based on sparse matrix operations." ] }, { "cell_type": "code", "execution_count": 10, - "id": "90668ea4", + "id": "9f0f17a7", + "metadata": {}, + "outputs": [], + "source": [ + "import pecos\n", + "import pandas as pd\n", + "import numpy as np" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "526820f8", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "rating: (3107, 1) and (1036, 1) in training and testing.\n" + "2022-08-13 06:42:02 URL:https://archive.ics.uci.edu/ml/machine-learning-databases/00461/drugLib_raw.zip [1133354/1133354] -> \"drugLib_raw.zip\" [1]\n", + "Archive: drugLib_raw.zip\n", + " inflating: drugLibTest_raw.tsv \n", + " inflating: drugLibTrain_raw.tsv \n" ] } ], "source": [ - "from scipy.sparse import csr_matrix\n", - "from sklearn.preprocessing import StandardScaler\n", - "\n", - "for feature_name in numerical_features:\n", - " X_trn_values = train_df[[\"rating\"]].values\n", - " X_tst_values = test_df[[\"rating\"]].values\n", - " scaler = StandardScaler()\n", - " X_trn_local = csr_matrix(scaler.fit_transform(X_trn_values), dtype=np.float32)\n", - " X_tst_local = csr_matrix(scaler.transform(X_tst_values), dtype=np.float32)\n", - " print(f\"{feature_name}: {X_trn_local.shape} and {X_tst_local.shape} in training and testing.\")\n", - " \n", - " X_trn_list.append(X_trn_local)\n", - " X_tst_list.append(X_tst_local)" - ] - }, - { - "cell_type": "markdown", - "id": "d2441580", - "metadata": {}, - "source": [ - "### Feature Concatenation\n", - "\n", - "PECOS provides easy-going utility functions for efficient matrix operations. The `hstack_csr` function can concatenate different features for each individual instance. More detils about other utilities will be introduced later in this session." + "! wget -nv -nc https://archive.ics.uci.edu/ml/machine-learning-databases/00461/drugLib_raw.zip\n", + "! unzip -o drugLib_raw.zip" ] }, { "cell_type": "code", - "execution_count": 11, - "id": "d0d3e69c", + "execution_count": 12, + "id": "ac8ab46f", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "X_trn is a csr matrix with a shape (3107, 232678) and 653987 non-zero values.\n", - "X_tst is a csr matrix with a shape (1036, 232678) and 164272 non-zero values.\n" + "Training DataFrame consists of 3107 instances.\n", + "Testing DataFrame consists of 1036 instances.\n", + "Index(['Unnamed: 0', 'urlDrugName', 'rating', 'effectiveness', 'sideEffects',\n", + " 'condition', 'benefitsReview', 'sideEffectsReview', 'commentsReview'],\n", + " dtype='object')\n" ] } ], "source": [ - "from pecos.utils import smat_util\n", - "\n", - "X_trn = smat_util.hstack_csr(X_trn_list)\n", - "X_tst = smat_util.hstack_csr(X_tst_list)\n", - "\n", - "print(f\"X_trn is a {X_trn.getformat()} matrix with a shape {X_trn.shape} and {X_trn.nnz} non-zero values.\")\n", - "print(f\"X_tst is a {X_tst.getformat()} matrix with a shape {X_tst.shape} and {X_tst.nnz} non-zero values.\")" + "train_df = pd.read_csv(\"drugLibTrain_raw.tsv\", sep=\"\\t\")\n", + "test_df = pd.read_csv(\"drugLibTest_raw.tsv\", sep=\"\\t\")\n", + "print(f\"Training DataFrame consists of {len(train_df)} instances.\")\n", + "print(f\"Testing DataFrame consists of {len(test_df)} instances.\")\n", + "print(train_df.columns)" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "6d6ee989", + "metadata": {}, + "outputs": [], + "source": [ + "label_name = \"effectiveness\"\n", + "text_features = [\"condition\", \"benefitsReview\", \"sideEffectsReview\", \"commentsReview\"]\n", + "categorical_features = [\"sideEffects\"]\n", + "numerical_features = [\"rating\"]" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "c25b1cb2", + "metadata": {}, + "outputs": [], + "source": [ + "X_trn_list = []\n", + "X_tst_list = []" ] }, { "cell_type": "markdown", - "id": "5e61775b", + "id": "4f72d047", "metadata": {}, "source": [ - "### Model Training and Testing" + "#### Label Encoding\n", + "\n", + "To encode labels into the sparse matrix format compatible to PECOS, [OneHotEncoder](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html) and [MultiLabelBinarizer](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MultiLabelBinarizer.html#sklearn.preprocessing.MultiLabelBinarizer) are helpful for the scenarios of multi-class and multi-label classification." ] }, { "cell_type": "code", - "execution_count": 12, - "id": "38189597", + "execution_count": 15, + "id": "4f0d9ec7", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "prec = 52.80 40.69 30.92 24.52 20.00\n", - "recall = 52.80 81.37 92.76 98.07 100.00\n" + "Y_trn is a csr matrix with a shape (3107, 5) and 3107 non-zero values.\n", + "Y_tst is a csr matrix with a shape (1036, 5) and 1036 non-zero values.\n" ] } ], "source": [ - "from pecos.xmc.xlinear.model import XLinearModel\n", - "xlm = XLinearModel.train(X_trn, Y_trn)\n", + "from sklearn.preprocessing import OneHotEncoder\n", "\n", - "Y_pred = xlm.predict(X_tst, beam_size=10, only_topk=5)\n", - "metrics = smat_util.Metrics.generate(Y_tst, Y_pred, topk=5)\n", - "print(metrics)" + "label_encoder = OneHotEncoder(dtype=np.float32)\n", + "Y_trn = label_encoder.fit_transform(train_df[[label_name]])\n", + "Y_tst = label_encoder.transform(test_df[[label_name]])\n", + "\n", + "print(f\"Y_trn is a {Y_trn.getformat()} matrix with a shape {Y_trn.shape} and {Y_trn.nnz} non-zero values.\")\n", + "print(f\"Y_tst is a {Y_tst.getformat()} matrix with a shape {Y_tst.shape} and {Y_tst.nnz} non-zero values.\")" ] }, { - "cell_type": "markdown", - "id": "c2b3f61e", + "cell_type": "code", + "execution_count": 16, + "id": "62ec7371", "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Y_trn_mlb is a csr matrix with a shape (3107, 5) and 3107 non-zero values.\n", + "Y_tst_mlb is a csr matrix with a shape (1036, 5) and 1036 non-zero values.\n" + ] + } + ], "source": [ - "## Sparse Matrix Operations\n", + "from sklearn.preprocessing import MultiLabelBinarizer\n", "\n", - "Most of the computations in PECOS are based on sparse matrices, so PECOS also provides various useful and efficient operation utilities for sparse matrices." + "label_encoder_multilabel = MultiLabelBinarizer(sparse_output=True)\n", + "Y_trn_mlb = label_encoder.fit_transform([[lbl] for lbl in train_df[label_name].tolist()])\n", + "Y_tst_mlb = label_encoder.fit_transform([[lbl] for lbl in test_df[label_name].tolist()])\n", + "print(f\"Y_trn_mlb is a {Y_trn_mlb.getformat()} matrix with a shape {Y_trn_mlb.shape} and {Y_trn_mlb.nnz} non-zero values.\")\n", + "print(f\"Y_tst_mlb is a {Y_tst_mlb.getformat()} matrix with a shape {Y_tst_mlb.shape} and {Y_tst_mlb.nnz} non-zero values.\")" ] }, { "cell_type": "markdown", - "id": "5fba58d4", + "id": "260a9a8a", "metadata": {}, "source": [ - "### Genric Matriox IO and Conversion\n", + "#### 1.4.2. Text Feature Encoding\n", "\n", - "`smat_util.load_matrix` and `smat_util.save_matrix` provide generic interfaces for loading and storing matrices in arbitrary common formats, including [dense matrix](https://numpy.org/doc/stable/reference/generated/numpy.array.html) in NumPy or different sparse matrix formats (i.e., [sparse Compressed Sparse Row (CSR) matrix](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_matrix.html), [sparse Compressed Sparse Column (CSC) matrix](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csc_matrix.html), and [sparse COOrdinate (COO) matrix](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csc_matrix.html))." + "As introduced in Session 1, we can use PECOS vectorizer for featurize text data. In addition, the encoder of [XR-Transformer](https://github.com/amzn/pecos/tree/mainline/pecos/xmc/xtransformer) can be also utilized for deriving text features with proper fine-tuning." ] }, { "cell_type": "code", - "execution_count": 13, - "id": "385ed0ba", + "execution_count": 17, + "id": "96fef619", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Dense Matrtix IO\n", - "mat is a matrix with a shape (2, 3).\n", - "[[0.6757516 0.42168422 0.40557039]\n", - " [0.86806547 0.9198075 0.7494449 ]]\n", - "mat_loaded is a matrix with a shape (2, 3).\n", - "[[0.6757516 0.42168422 0.40557039]\n", - " [0.86806547 0.9198075 0.7494449 ]]\n", - "\n", - "csr Sparse Matrix IO\n", - "mat is a matrix with a shape (5, 4) and 4 non-zero values.\n", - " (1, 2)\t0.17821196669035588\n", - " (2, 1)\t0.8259001065480657\n", - " (4, 2)\t0.5111159408743305\n", - " (4, 3)\t0.6337428297507509\n", - "mat_loaded is a matrix with a shape (5, 4) and 4 non-zero values.\n", - " (1, 2)\t0.17821196669035588\n", - " (2, 1)\t0.8259001065480657\n", - " (4, 2)\t0.5111159408743305\n", - " (4, 3)\t0.6337428297507509\n", - "\n", - "csc Sparse Matrix IO\n", - "mat is a matrix with a shape (5, 4) and 4 non-zero values.\n", - " (0, 1)\t0.868116915403953\n", - " (1, 2)\t0.7454473997071077\n", - " (0, 3)\t0.21167432752493887\n", - " (1, 3)\t0.4685535255015949\n", - "mat_loaded is a matrix with a shape (5, 4) and 4 non-zero values.\n", - " (0, 1)\t0.868116915403953\n", - " (1, 2)\t0.7454473997071077\n", - " (0, 3)\t0.21167432752493887\n", - " (1, 3)\t0.4685535255015949\n", - "\n", - "coo Sparse Matrix IO\n", - "mat is a matrix with a shape (5, 4) and 4 non-zero values.\n", - " (1, 0)\t0.3217900085041965\n", - " (3, 3)\t0.15316424313380772\n", - " (2, 3)\t0.7835729602784944\n", - " (2, 2)\t0.396664789900256\n", - "mat_loaded is a matrix with a shape (5, 4) and 4 non-zero values.\n", - " (1, 0)\t0.3217900085041965\n", - " (3, 3)\t0.15316424313380772\n", - " (2, 3)\t0.7835729602784944\n", - " (2, 2)\t0.396664789900256\n", - "\n" + "condition: (3107, 3759) and (1036, 3759) in training and testing.\n", + "benefitsReview: (3107, 72861) and (1036, 72861) in training and testing.\n", + "sideEffectsReview: (3107, 64321) and (1036, 64321) in training and testing.\n", + "commentsReview: (3107, 91731) and (1036, 91731) in training and testing.\n" ] } ], "source": [ - "from pecos.utils import smat_util\n", - "import numpy as np\n", - "import scipy.sparse as smat\n", + "from pecos.utils.featurization.text.vectorizers import Vectorizer\n", "\n", - "print(\"Dense Matrtix IO\")\n", - "mat = np.random.rand(2, 3)\n", - "print(f\"mat is a {type(mat)} matrix with a shape {mat.shape}.\")\n", - "print(mat)\n", - "smat_util.save_matrix(\"mat.npz\", mat)\n", - "mat_loaded = smat_util.load_matrix(\"mat.npz\")\n", - "print(f\"mat_loaded is a {type(mat_loaded)} matrix with a shape {mat_loaded.shape}.\")\n", - "print(mat)\n", - "print(\"\") \n", + "for feature_name in text_features:\n", + " vectorizer_config = {\n", + " \"type\": \"tfidf\",\n", + " \"kwargs\": {\n", + " \"base_vect_configs\": [\n", "\n", - "for matrix_format in [\"csr\", \"csc\", \"coo\"]:\n", - " print(f\"{matrix_format} Sparse Matrix IO\")\n", - " mat = smat.random(5, 4, density=0.2, format=matrix_format)\n", - " print(f\"mat is a {type(mat)} matrix\"\n", - " f\" with a shape {mat.shape} and {mat.nnz} non-zero values.\")\n", - " print(mat)\n", + " {\n", + " \"ngram_range\": [1, 2],\n", + " \"max_df_ratio\": 0.98,\n", + " \"analyzer\": \"word\",\n", + " },\n", + " ],\n", + " },\n", + " } \n", + " train_texts = [str(x) for x in train_df[feature_name].tolist()]\n", + " test_texts = test_df[feature_name].tolist()\n", + " vectorizer = Vectorizer.train(train_texts, config=vectorizer_config)\n", + " X_trn_local = vectorizer.predict(train_texts)\n", + " X_tst_local = vectorizer.predict(test_texts)\n", + " print(f\"{feature_name}: {X_trn_local.shape} and {X_tst_local.shape} in training and testing.\")\n", " \n", - " smat_util.save_matrix(\"mat.npz\", mat)\n", - " mat_loaded = smat_util.load_matrix(\"mat.npz\")\n", - " print(f\"mat_loaded is a {type(mat_loaded)} matrix\"\n", - " f\" with a shape {mat_loaded.shape} and {mat_loaded.nnz} non-zero values.\")\n", - " print(mat_loaded)\n", - " print(\"\") " + " X_trn_list.append(X_trn_local)\n", + " X_tst_list.append(X_tst_local)" + ] + }, + { + "cell_type": "markdown", + "id": "38e75fa2", + "metadata": {}, + "source": [ + "#### 1.4.3. Categorical Feature Encoding\n", + "\n", + "Similar to labels, categorical features can also be considered as one-hot or multi-hot embeddings." ] }, { "cell_type": "code", - "execution_count": 14, - "id": "2579b855", + "execution_count": 18, + "id": "386b96ad", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Original Matrix mat\n", - " [[7.75480297e-01 3.06141999e-01 1.48439313e-01 4.82371302e-01\n", - " 7.94080899e-01 4.22684703e-04]\n", - " [1.22368102e-01 9.06639305e-02 5.88889479e-01 3.37880581e-01\n", - " 7.45595442e-01 9.58851999e-01]\n", - " [5.65962202e-01 7.18344997e-01 1.99721347e-01 2.02474399e-01\n", - " 5.86650110e-01 1.93471414e-01]\n", - " [5.72012816e-02 1.20470044e-01 1.27986695e-01 6.43206432e-01\n", - " 5.42874998e-01 7.05113274e-01]] \n", - "\n", - "csr_mat = dense_to_csr(mat)\n", - "csr_mat is a matrix with a shape (4, 6) and 24 non-zero values.\n", - "[[7.75480297e-01 3.06141999e-01 1.48439313e-01 4.82371302e-01\n", - " 7.94080899e-01 4.22684703e-04]\n", - " [1.22368102e-01 9.06639305e-02 5.88889479e-01 3.37880581e-01\n", - " 7.45595442e-01 9.58851999e-01]\n", - " [5.65962202e-01 7.18344997e-01 1.99721347e-01 2.02474399e-01\n", - " 5.86650110e-01 1.93471414e-01]\n", - " [5.72012816e-02 1.20470044e-01 1.27986695e-01 6.43206432e-01\n", - " 5.42874998e-01 7.05113274e-01]] \n", - "\n", - "csr_mat_topk = dense_to_csr(mat, topk=2)\n", - "csr_mat is a matrix with a shape (4, 6) and 8 non-zero values.\n", - "[[0.7754803 0. 0. 0. 0.7940809 0. ]\n", - " [0. 0. 0. 0. 0.74559544 0.958852 ]\n", - " [0.5659622 0. 0. 0. 0.58665011 0. ]\n", - " [0. 0. 0. 0. 0.542875 0.70511327]] \n", - "\n" + "sideEffects: (3107, 5) and (1036, 5) in training and testing.\n" ] } ], "source": [ - "mat = np.random.rand(4, 6)\n", - "\n", - "print(f\"Original Matrix mat\\n\", mat, \"\\n\")\n", - "\n", - "print(\"csr_mat = dense_to_csr(mat)\")\n", - "csr_mat = smat_util.dense_to_csr(mat)\n", - "print(f\"csr_mat is a {type(csr_mat)} matrix\"\n", - " f\" with a shape {csr_mat.shape} and {csr_mat.nnz} non-zero values.\")\n", - "print(csr_mat.toarray(), \"\\n\")\n", + "from sklearn.preprocessing import OneHotEncoder\n", "\n", - "print(\"csr_mat_topk = dense_to_csr(mat, topk=2)\")\n", - "csr_mat_topk = smat_util.dense_to_csr(mat, topk=2)\n", - "print(f\"csr_mat is a {type(csr_mat_topk)} matrix\"\n", - " f\" with a shape {csr_mat_topk.shape} and {csr_mat_topk.nnz} non-zero values.\")\n", - "print(csr_mat_topk.toarray(), \"\\n\")" + "for feature_name in categorical_features:\n", + " local_encoder = OneHotEncoder(dtype=np.float32)\n", + " X_trn_local = local_encoder.fit_transform(train_df[[feature_name]])\n", + " X_tst_local = local_encoder.transform(test_df[[feature_name]])\n", + " print(f\"{feature_name}: {X_trn_local.shape} and {X_tst_local.shape} in training and testing.\")\n", + " \n", + " X_trn_list.append(X_trn_local)\n", + " X_tst_list.append(X_tst_local)" ] }, { "cell_type": "markdown", - "id": "b2746e3c", + "id": "6deaf4e8", "metadata": {}, "source": [ - "### Memory-efficient Sparse Matrix Operations\n", + "#### 1.4.4. Numerical Features Encoding\n", "\n", - "To manipulate with sparse matrix, PECOS provides many useful memory-efficient functions. For example, for CSR matrices, we have following functions to combine multiple matrices.\n", - "\n", - "* `hstack_csr([mat, mat, mat]`\n", - "* `vstack_csr([mat, mat, mat]`\n", - "* `block_diag_csr([mat, mat, mat]`\n", - "\n", - "These funcations are also available for CSC matrices as `hstack_csc`, `vstack_csr`, and `block_diag_csr`.\n" + "Numberical features can be directly incorporated as model inputs after some simple normalization." ] }, { "cell_type": "code", - "execution_count": 15, - "id": "b9dac617", + "execution_count": 19, + "id": "90668ea4", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Original Matrix mat\n", - " [[0.45989041 0. ]\n", - " [0.77400378 0. ]\n", - " [0. 0.44557291]] \n", - "\n", - "hstack_csr([mat, mat, mat])\n", - "[[0.45989041 0. 0.45989041 0. 0.45989041 0. ]\n", - " [0.77400378 0. 0.77400378 0. 0.77400378 0. ]\n", - " [0. 0.44557291 0. 0.44557291 0. 0.44557291]] \n", - "\n", - "vstack_csr([mat, mat, mat])\n", - "[[0.45989041 0. ]\n", - " [0.77400378 0. ]\n", - " [0. 0.44557291]\n", - " [0.45989041 0. ]\n", - " [0.77400378 0. ]\n", - " [0. 0.44557291]\n", - " [0.45989041 0. ]\n", - " [0.77400378 0. ]\n", - " [0. 0.44557291]] \n", - "\n", - "block_diag_csr([mat, mat, mat])\n", - "[[0.45989041 0. 0. 0. 0. 0. ]\n", - " [0.77400378 0. 0. 0. 0. 0. ]\n", - " [0. 0.44557291 0. 0. 0. 0. ]\n", - " [0. 0. 0.45989041 0. 0. 0. ]\n", - " [0. 0. 0.77400378 0. 0. 0. ]\n", - " [0. 0. 0. 0.44557291 0. 0. ]\n", - " [0. 0. 0. 0. 0.45989041 0. ]\n", - " [0. 0. 0. 0. 0.77400378 0. ]\n", - " [0. 0. 0. 0. 0. 0.44557291]] \n", - "\n" + "rating: (3107, 1) and (1036, 1) in training and testing.\n" ] } ], "source": [ - "from pecos.utils import smat_util\n", - "import scipy.sparse as smat\n", - "\n", - "mat = smat.random(3, 2, density=0.5, format=\"csr\")\n", - "print(f\"Original Matrix {type(mat)} mat\\n\", mat.toarray(), \"\\n\")\n", - "\n", - "print(f\"hstack_csr([mat, mat, mat])\")\n", - "print(smat_util.hstack_csr([mat, mat, mat]).toarray(), \"\\n\")\n", - "\n", - "print(f\"vstack_csr([mat, mat, mat])\")\n", - "print(smat_util.vstack_csr([mat, mat, mat]).toarray(), \"\\n\")\n", + "from scipy.sparse import csr_matrix\n", + "from sklearn.preprocessing import StandardScaler\n", "\n", - "print(f\"block_diag_csr([mat, mat, mat])\")\n", - "print(smat_util.block_diag_csr([mat, mat, mat]).toarray(), \"\\n\")" + "for feature_name in numerical_features:\n", + " X_trn_values = train_df[[\"rating\"]].values\n", + " X_tst_values = test_df[[\"rating\"]].values\n", + " scaler = StandardScaler()\n", + " X_trn_local = csr_matrix(scaler.fit_transform(X_trn_values), dtype=np.float32)\n", + " X_tst_local = csr_matrix(scaler.transform(X_tst_values), dtype=np.float32)\n", + " print(f\"{feature_name}: {X_trn_local.shape} and {X_tst_local.shape} in training and testing.\")\n", + " \n", + " X_trn_list.append(X_trn_local)\n", + " X_tst_list.append(X_tst_local)" ] }, { "cell_type": "markdown", - "id": "5f9cf7f1", + "id": "d2441580", "metadata": {}, "source": [ - "### Sparse-to-sparse Matrix Multiplication (SpMM)\n", - "\n", - "Many operations in PECOS or XMR problems rely on Sparse-to-sparse Matrix Multiplication (SpMM), such as the computation of PIFA features. It is also one of the key primitives in large-scale linear algebra operations, with a broad range of applications in machine learning and natural language processing.\n", - "\n", - "For SpMM, PECOS provides a highly optimized multi-core CPU implementation with state-of-the-art performance, where the underlying operations are implemented and optimized in C/C++.\n", - "Specifically, the Python interface and parameters are as follows:\n", + "#### 1.4.5. Feature Concatenation\n", "\n", - "```python\n", - "from pecos.core import clib as pecos_clib\n", - "Z = pecos_clib.sparse_matmul(X, Y, eliminate_zeros=False, sorted_indices=True, threads=-1)\n", - "```\n", - "* Parameters\n", - " * `X` (scipy.sparse.csr_matrix or scipy.sparse.csc_matrix): the first sparse matrix to be multiplied.\n", - " * `Y` (scipy.sparse.csr_matrix or scipy.sparse.csc_matrix): the second sparse matrix to be multiplied.\n", - " * `eliminate_zeros` (bool, optional): if true, then eliminate (potential) zeros created by maxnnz in output matrix Z. Default is false.\n", - " * `sorted_indices` (bool, optional): if true, then sort the Z.indices for the output matrix Z. Default is true.\n", - " * `threads` (int, optional): The number of threads. Default -1 to use all CPU cores." + "PECOS provides easy-going utility functions for efficient matrix operations. The `hstack_csr` function can concatenate different features for each individual instance. More detils about other utilities will be introduced later in this session." ] }, { "cell_type": "code", - "execution_count": 16, - "id": "b2e6c54a", + "execution_count": 20, + "id": "d0d3e69c", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "||Z_true - Z_pred|| = 0.0\n" + "X_trn is a csr matrix with a shape (3107, 232678) and 653987 non-zero values.\n", + "X_tst is a csr matrix with a shape (1036, 232678) and 164272 non-zero values.\n" ] } ], "source": [ - "import numpy as np\n", - "import scipy.sparse as smat\n", - "from scipy.sparse import linalg\n", - "from pecos.core import clib as pecos_clib\n", - "X = smat.random(1000, 1000, density=0.01, format='csr', dtype=np.float32)\n", - "Y = smat.random(1000, 1000, density=0.01, format='csr', dtype=np.float32)\n", - "Z_true = X.dot(Y)\n", - "Z_pred = pecos_clib.sparse_matmul(X, Y)\n", - "print(\"||Z_true - Z_pred|| = \", linalg.norm(Z_true - Z_pred))" - ] - }, - { - "cell_type": "code", - "execution_count": 17, - "id": "14e4bded", - "metadata": {}, - "outputs": [], - "source": [ - "import time\n", - "DATASET = \"wiki10-31k\"\n", - "X = smat_util.load_matrix(f\"xmc-base/{DATASET}/tfidf-attnxml/X.trn.npz\").astype(np.float32)\n", - "Y = smat_util.load_matrix(f\"xmc-base/{DATASET}/Y.trn.npz\").astype(np.float32)\n", - "YT_csr = Y.T.tocsr()\n", - "X_csr = X.tocsr()" - ] - }, - { - "cell_type": "markdown", - "id": "bb5af1e9", - "metadata": {}, - "source": [ - "#### Benchmarking Sparse Matrix Muplication\n", - "\n", - "The SpMM utility has state-of-the-art performance in efficiency as shown in the following figure.\n", - "\n", - "
\n", - "
\n", - "
\n", - "\n", - "In this part, we provide some hands-on instructions for benchmarking different methods for SpMM." - ] - }, - { - "cell_type": "code", - "execution_count": 18, - "id": "c71cffdb", - "metadata": {}, - "outputs": [], - "source": [ - "# Benchmarking SciPy\n", - "\n", - "start = time.time()\n", - "Z = YT_csr.dot(X_csr)\n", - "Z.sort_indices()\n", - "run_time_scipy = time.time() - start" - ] - }, - { - "cell_type": "code", - "execution_count": 19, - "id": "2f2b9795", - "metadata": {}, - "outputs": [], - "source": [ - "# Benchmarking PyTorch\n", - "\n", - "import torch\n", + "from pecos.utils import smat_util\n", "\n", - "def csr_to_coo(A):\n", - " A_coo = smat.coo_matrix(A)\n", - " indices = np.vstack([A_coo.row, A_coo.col]).T\n", - " values = A_coo.data\n", - " return indices, values\n", + "X_trn = smat_util.hstack_csr(X_trn_list)\n", + "X_tst = smat_util.hstack_csr(X_tst_list)\n", "\n", - "def get_pt_data(A_csr):\n", - " A_indices, A_values = csr_to_coo(A_csr)\n", - " A_pt = torch.sparse_coo_tensor(\n", - " A_indices.T.astype(np.int64),\n", - " A_values.astype(np.float32),\n", - " A_csr.shape,\n", - " )\n", - " return A_pt\n", - " \n", - "YT_pt = get_pt_data(YT_csr)\n", - "X_pt = get_pt_data(X_csr)\n", - "start = time.time()\n", - "Z_pt = torch.sparse.mm(YT_pt, X_pt)\n", - "run_time_pytorch = time.time() - start" + "print(f\"X_trn is a {X_trn.getformat()} matrix with a shape {X_trn.shape} and {X_trn.nnz} non-zero values.\")\n", + "print(f\"X_tst is a {X_tst.getformat()} matrix with a shape {X_tst.shape} and {X_tst.nnz} non-zero values.\")" ] }, { - "cell_type": "code", - "execution_count": 20, - "id": "54b24694", + "cell_type": "markdown", + "id": "5e61775b", "metadata": {}, - "outputs": [], "source": [ - "# Benchmarking PECOS\n", - "\n", - "start = time.time()\n", - "Z = pecos_clib.sparse_matmul(\n", - " YT_csr, X_csr,\n", - " eliminate_zeros=False,\n", - " sorted_indices=True\n", - ")\n", - "run_time_pecos = time.time() - start" + "#### 1.4.6. Model Training and Testing" ] }, { "cell_type": "code", "execution_count": 21, - "id": "e12f29e9", + "id": "38189597", "metadata": {}, "outputs": [ { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD4CAYAAADmWv3KAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAW/ElEQVR4nO3deZRkZZnn8e8PEFlkaaRUXKBEFEcdZEkXDrZKtTqitONxp911KLuPC4z2zIFpuhV7uhunGwTU8UwJKoKioLK6oKDY2u0oVVBAgcehQFGUhkJpWWQRfOaPuClJkhl1MyJuVFbw/ZwTJ+PuTxDUk28+973vm6pCkjR5NtnQAUiSumGCl6QJZYKXpAllgpekCWWCl6QJtdmGDmCmHXfcsZYuXbqhw5CkjcaqVatuqqolc21bVAl+6dKlrFy5ckOHIUkbjSTXzrfNEo0kTSgTvCRNKBO8JE0oE7wkTSgTvCRNKBO8JE0oE7wkTSgTvCRNKBO8JE2ozp5kTbI78IUZq3YF/qaqju3ieksP+0oXpxXw06NeuqFDkDSAzhJ8Vf0Y2BMgyabAL4AzurqeJOn+xlWi+RPg6qqad8wESdJojSvBvw44da4NSZYnWZlk5bp168YUjiRNvs4TfJLNgZcBp8+1vapWVNVUVU0tWTLniJeSpAGMowV/AHBxVd0whmtJkhrjSPAHMU95RpLUnU4TfJKtgRcCX+7yOpKkB+p0Rqequh14eJfXkCTNzSdZJWlCrbcFn+QRwH7Ao4E7gDXAyqr6fcexSZKGMG+CT7I/cBiwA3AJcCOwBfBy4AlJvggcXVW3jCFOSdIC9WvBvwQ4uKp+NntDks2AA+ndQP1SR7FJkoYwb4Kvqv/WZ9s9wJldBCRJGo313mRNckiSbdNzYpKLk7xoHMFJkgbXphfN25o6+4uAPwLeCBzVaVSSpKG1SfBpfr4EOLmqrpixTpK0SLVJ8KuSfINegj8vyTaAXSQlaZFr8yTr2+lN3HFNVf02ycOBt3YalSRpaP36we89a9WuiZUZSdpY9GvBH9383ALYB7iMXu19D2AlsG+3oUmShjFvDb6q9q+q/YHrgX2aSTn2AfaiN7+qJGkRa3OTdfequnx6oarWAP+hu5AkSaPQ5ibrZUlOAE5pll9Pr1wjSVrE2iT4twJ/ARzSLP8z8PHOIpIkjcR6E3xV3Ql8uHlJkjYSbcaD3w/4ALDLzP2ratfuwpIkDatNieZE4L8Cq4B7uw1HkjQqbRL8b6rqa51HIkkaqTYJ/ttJ/hH4MnDX9Mqqunh9BybZHjgBeBpQ9Eam/P5goUqSFqJNgn9W83NqxroClrU49jjg61X1qiSbA1stMD5J0oDa9KLZf5ATJ9kOeC7wluY8dwN3D3IuSdLCtZnRabskxyRZ2byObpL3+jweWAd8KsklSU5IsvUc518+fe5169YN8BEkSXNpM1TBJ4Fbgdc0r1uAT7U4bjNgb+DjVbUXcDtw2OydqmpFM87N1JIlS1oHLknqr00N/glV9coZy0cmWd3iuOuA66rqB83yF5kjwUuSutGmBX9HkudMLzQPPt2xvoOq6t+AnyfZvVn1J8CVA0UpSVqwNi34vwBOmlF3v5nmxmkL7wY+2/SguQZngpKksWnTi2Y18PQk2zbLt7Q9eXPs1Pr2kySNXpteNH+fZPuquqWqbknyR0n+5ziCkyQNrk0N/oCq+vfphaq6GXhJZxFJkkaiTYLfNMlDpxeSbAk8tM/+kqRFoM1N1s8CFySZ7vv+VuCk7kKSJI1Cm5usH0pyKfCCZtXfVtV53YYlSRpWmxY8wI+Ae6rq/CRbJdmmqm7tMjBJ0nDa9KI5mN5TqP+nWfUY4MwOY5IkjUCbm6zvBPajNwYNVXUV8Igug5IkDa9Ngr+rGeoXgCSb0RsPXpK0iLVJ8N9J8j+ALZO8EDgdOKfbsCRJw2qT4A+jN6775cA7gK8CR3QZlCRpeG26Sf4e+ATwiSQ7AI+tKks0krTItelFc2GSbZvkvopeov9w96FJkobRpkSzXTOC5CuAz1TVs+iN7S5JWsTaJPjNkuxEb7q+czuOR5I0Im0S/AeB84C1VXVRkl2Bq7oNS5I0rDY3WU+n1zVyevka4JXzHyFJWgzmbcEnOaK5sTrf9mVJDuwmLEnSsPq14C8HzklyJ3Axvb7wWwBPBPYEzgf+vusAJUmDmTfBV9VZwFlJnkhvLJqd6I1HcwqwvKruGE+IkqRBtKnBX8WAN1WT/BS4FbiX3nDDTsAtSWPSdjz4YexfVTeN4TqSpBnadJOUJG2Euk7wBXwjyaoky+faIcnyJCuTrFy3bl3H4UjSg0ebsWielOSCJGua5T2StB1N8jlVtTdwAPDOJM+dvUNVraiqqaqaWrJkyYKClyTNr00L/hPA4cDvAKrqMuB1bU5eVb9oft4InAE8c7AwJUkL1SbBb1VVP5y17p71HZRk6yTbTL8HXgSsWXiIkqRBtOlFc1OSJ9BM05fkVcD1LY57JHBGkunrfK6qvj5ooJKkhWmT4N8JrACenOQXwE+AN6zvoGbMmqcPF54kaVBtHnS6BnhBU2bZpKpu7T4sSdKw1pvgk2wPvAlYSm9seACq6j1dBiZJGk6bEs1Xgf9Lb/Cx33cbjiRpVNok+C2q6r2dRyJJGqk23SRPTnJwkp2S7DD96jwySdJQ2rTg7wb+Efgrmq6Szc9duwpKkjS8Ngn+fcBujggpSRuXNiWatcBvuw5EkjRabVrwtwOrk3wbuGt6pd0kJWlxa5Pgz2xekqSNSJsnWU8aRyCSpNGaN8EnOa2qXpPkcu7rPfMHVbVHp5FJkobSrwX/4ebngeMIRA8uSw/7yoYOYWL99KiXbugQtEj0S/AfA/auqmvHFYwkaXT6dZPM2KKQJI1cvxb8Y5IcP99Gu0lK0uLWL8HfAawaVyCSpNHql+B/ZRdJSdp49avB3z22KCRJIzdvgq+qZ48zEEnSaLUZbGwoSTZNckmSc7u+liTpPp0neOAQ4EdjuI4kaYZWCb5phT86yc7Tr5bHPRZ4KXDCMEFKkhZuvYONJXk38H7gBu6bdLuANmPRHAv8d2CbPudfDiwH2HnnVr83JEkttBku+BBg96r61UJOnORA4MaqWpXk+fPtV1UrgBUAU1NTDxjUTJI0mDYlmp8Dvxng3PsBL0vyU+DzwLIkpwxwHknSANq04K8BLkzyFe4/o9Mx/Q6qqsOBwwGaFvxfVtUbBo5UkrQgbRL8z5rX5s1LkrQRaDOj05EASR7WLN+20ItU1YXAhQs9TpI0uPXW4JM8LcklwBXAFUlWJXlq96FJkobR5ibrCuC9VbVLVe0CvA/4RLdhSZKG1SbBb11V355eaMotW3cWkSRpJFr1okny18DJzfIb6PWskSQtYm1a8G8DlgBfbl5LmnWSpEWsTS+amwGn55Okjcy8CT7JsVV1aJJz6I09cz9V9bJOI5MkDaVfC3665v5P4whEkjRa8yb4qpqecHvPqjpu5rYkhwDf6TIwSdJw2txkffMc694y4jgkSSPWrwZ/EPBnwOOTnD1j0zbAr7sOTJI0nH41+H8Frgd2BI6esf5W4LIug5IkDa9fDf5a4Fpg3/GFI0kalTaDjT07yUVJbktyd5J7k9wyjuAkSYNrc5P1o8BBwFXAlsB/AT7WZVCSpOG1SfBU1Vpg06q6t6o+Bby427AkScNqM9jYb5NsDqxO8r/o3Xht9YtBkrThtEnUb2z2exdwO/A44JVdBiVJGl6bFvxNwN1VdSdwZJJNgYd2G5YkaVhtWvAXAFvNWN4SOL+bcCRJo9ImwW8xc6Lt5v1WffYHIMkWSX6Y5NIkVyQ5cphAJUkL0ybB355k7+mFJPsAd7Q47i5gWVU9HdgTeHGSZw8UpSRpwdrU4A8FTk/ySyDAo4DXru+gqipguuX/kOb1gHHlJUndaDOj00VJngzs3qz6cVX9rs3Jmxuyq4DdgI9V1Q8GjlSStCD9RpNcVlXfSvKKWZuelISq+vL6Tl5V9wJ7JtkeOCPJ06pqzazrLAeWA+y8884L/gCSpLn1a8E/D/gW8KdzbCt6E3C3UlX/nuTb9J6AXTNr2wpgBcDU1JQlHEkakX6jSb6/+fnWQU6cZAnwuya5bwm8EPjQQFFKkhasX4nmvf0OrKpj1nPunYCTmjr8JsBpVXXuwkOUJA2iX4lmm2FOXFWXAXsNcw5J0uD6lWh8MEmSNmJtJvzYNck5SdYluTHJWUl2HUdwkqTBtXmS9XPAafRq6o8GTgdO7TIoSdLw2iT4rarq5Kq6p3mdAmzRdWCSpOG0Garga0kOAz5Pr//7a4GvJtkBoKp+3WF8kqQBtUnwr2l+vmPW+tfRS/jW4yVpEWozFs3jxxGIJGm0BhmLBqDVWDSSpA1nLGPRSJLGb71j0QAfrKqfzNyWxLKNJC1ybbpJfmmOdV8cdSCSpNHqV4N/MvBUYLtZdfhtsR+8JC16/WrwuwMHAttz/zr8rcDBHcYkSRqBfjX4s4CzkuxbVd8fY0ySpBHoV6L5CM0k2UkOmr29qt7TYVySpCH1K9GsHFsUkqSR61eiOWmcgUiSRmu9QxU0k2U/YDLsqlrWSUSSpJFoM9jYX854vwXwSuCebsKRJI1Km8HGVs1a9S9JfthRPJKkEWlTotlhxuImwD7Adi2OexzwGeCR9Eo8K6rquAHjlCQtUJsSzSp6CTr0SjM/Ad7e4rh7gPdV1cVJtgFWJflmVV05cLSSpNY6Gw++qq4Hrm/e35rkR8BjABO8JI1Bvwed5hwHftpCxoNPshTYC/hB68gkSUPp14L/IrC6eUGvRDOt9XjwSR5Gb0TKQ6vqljm2LweWA+y8885tTilJaqFfgn8FvXlX9wDOAk6tqrULOXmSh9BL7p+dr8VfVSuAFQBTU1MP6G8vSRrMvOPBV9WZVfU6ejM7XQ0cneR7SZ7X5sRJApwI/KiqjhlJtJKk1tpM+HEn8BvgFuBhtB8Lfj/gjcCyJKub10sGC1OStFB9J92mV6J5JnA+cFxVtR6ArKq+x/3r9pKkMepXgz8fuAz4HvBQ4E1J3jS90eGCJWlx65fg3zq2KCRJI+dwwZI0odrcZJUkbYRM8JI0oQZK8Ek2H3UgkqTRWm+CT3JhM5bM9PIzgYu6DEqSNLw2wwX/A/D1JMfTGw3yAOxhI0mLXpvhgs9L8ufAN4GbgL2q6t86j0ySNJQ2JZq/Bj4CPBf4AHBhkpd2HJckaUhtSjQPB55ZVXcA30/ydeAE4CudRiZJGkqbEs2hs5avBV7YVUCSpNHoN9jYsVV1aJJz6E3wcT9V9bJOI5MkDaVfC/7k5uc/jSMQSdJo9RuLZlWSTYHlVfX6McYkSRqBvr1oqupeYBefXJWkjU+bXjTXAP+S5Gzg9umVTsMnSYtbmwR/dfPaBNimWefk2JK0yLVJ8FdW1ekzVyR5dUfxSJJGpM1okoe3XCdJWkT69YM/AHgJ8JhmoLFp2wL3dB2YJGk4/VrwvwRWAncCq2a8zgb+0/pOnOSTSW5MsmYUgUqSFqZfP/hLgUuTfK6qfjfAuT8NfBT4zICxSZKG0OYm69Ik/wA8BdhiemVV7drvoKr655kThUiSxqvNTdZPAR+nV3ffn16L/JRRBZBkeZKVSVauW7duVKeVpAe9Ngl+y6q6AEhVXVtVHwBGNh58Va2oqqmqmlqyZMmoTitJD3ptSjR3JdkEuCrJu4BfAA/rNixJ0rDatOAPAbYC3gPsA7wReHOXQUmShtdmwo+Lmre3sYDJtpOcCjwf2DHJdcD7q+rEQYKUJC1cvwedzu534Pom/KiqgwYNSpI0vH4t+H2BnwOnAj8AMpaIJEkj0S/BP4re3KsHAX9Gb5LtU6vqinEEJmlxWXrYVzZ0CBPrp0eNrGPi/cx7k7Wq7q2qr1fVm4FnA2uBC5ueNJKkRa7vTdYkD6XX5/0gYClwPHBG92FJkobV7ybrZ4CnAV8FjqwqBw2TpI1Ivxb8G+hN0XcI8J7kD/dYA1RVbdtxbJKkIfQbTbLNQ1CSpEXKJC5JE8oEL0kTygQvSRPKBC9JE8oEL0kTygQvSRPKBC9JE8oEL0kTygQvSRPKBC9JE8oEL0kTygQvSRPKBC9JE6rTBJ/kxUl+nGRtksO6vJYk6f46S/BJNgU+BhwAPAU4KMlTurqeJOn+umzBPxNYW1XXVNXdwOeB/9zh9SRJM/Sdk3VIjwF+PmP5OuBZs3dKshxY3izeluTHHca0WOwI3LShg2grH9rQESwKG8135vf1Bw+W72yX+TZ0meBbqaoVwIoNHcc4JVlZVVMbOg6153e28fE767ZE8wvgcTOWH9uskySNQZcJ/iLgiUken2Rz4HXA2R1eT5I0Q2clmqq6J8m7gPOATYFPVtUVXV1vI/OgKklNCL+zjc+D/jtLVW3oGCRJHfBJVkmaUCZ4SZpQJvgRSfJXSa5IclmS1Uke0Oe/2W8qyfHN+7ckWdfsf2WSg8cb9eRLcm/z33dNktOTbDXPfv+x2W91kl8n+Unz/vwhrv3pJK8aPPoHt/m+uxnrp1+HNesfkuSoJFcluTjJ95Mc0GzbLslnmmFTrm7eb9ds2yTJ8c11Lk9yUZLHb7hPPjobvB/8JEiyL3AgsHdV3ZVkR2DzufatqpXAyhmrvlBV70ryCOCKJGdX1Q3dR/2gcUdV7QmQ5LPAnwPHzN6pqi4Hpvf7NHBuVX2xzQWSbFpV944oXt1nvu/uD+tn+VtgJ+Bpzb/DRwLPa7adCKypqjc15zsSOAF4NfBa4NHAHlX1+ySPBW7v7FONkS340dgJuKmq7gKoqpuq6pdJnpHkX5NcmuSHSbZJ8vwk584+QVXdCFwN7NK0QJbAH1oXa6eXNZTvArsl+WCSQ6dXJvm7JIfMdUCSg5pW3ZrkvucNk9yW5OgklwL7JnlT89fbpUlOnnGK5zb/D1xja34o3wV2m29j07o/GHj3jH+HN1TVaUl2A/ah9wtg2geBqSRPoPfv9/qq+n1z3HVVdXNHn2OsTPCj8Q3gcUn+X5L/neR5Td//LwCHVNXTgRcAd8x3giS7ArsCa4FTgNc3m14AXFpV6zr9BBMuyWb0Br67HPgkMN2S24TeMxqnzHHMo4EPAcvote6fkeTlzeatgR803+3NwBHAsmZ55i+LnYDn0PsL76hRf64Hg1nfHcCWs0o0r6WX/H9WVbfMcYqnAKtn/pXVvF8NPBU4DfjT5lxHJ9mry88zTpZoRqCqbkuyD/DHwP70Evvf0WsVXNTscwtAktmHvzbJc4C7gHdU1a+TfBI4CzgWeBvwqXF8jgm1ZZLVzfvvAidW1d1JftX8Q34kcElV/WqOY58BXDj9y7UpEzwXOBO4F/hSs98y4PSqugmgqn494xxnNi3DK5uSgdp7wHfXvH9AiSbJHoNepKquS7I7ve9xGXBBkldX1QWDnnOxMMGPSNMiuBC4MMnlwDtbHvqFqnrXrHP9PMkNSZbRG5Xz9XMfqhbmq9eeALwFeBS9Fv1C3dmy7n7XjPcP+O2uvub77uayFtg5ybZztOKvBPZMssl0Gab5y23PZhtNWedrwNeS3AC8HNjoE7wlmhFIsnuSJ85YtSfwI2CnJM9o9tmm+VOzrRPolQ1O9wZeJ84AXkyvlX7ePPv8EHhekh3Tm9/gIOA7c+z3LeDVSR4OkGSHDuJVH1X1W3ot/OOa8ihJljQt8bXAJfTKaNOOAC6uqrVJ9m7KcdOJfw/g2vF+gm6Y4EfjYcBJ6XV1vIxeze9v6N2d/0hzI+6bwBYLOOfZzXktz3SgmaPg28Bp8/0CrarrgcOa/S4FVlXVWXPsdwW9ktx3mu/6Ab10NFKza/DT9zaOANbRK4etAc4Fplvzbwee1HSRvBp4UrMO4BHAOc0xlwH3AB8d14fpkkMVLFJJpoAPV9Ufb+hYJlHTUrsYeHVVXbWh45G6YAt+EUrvwY0vAYdv6FgmUXpTR64FLjC5a5LZgpekCWULXpImlAlekiaUCV6SJpQJXpImlAlekibU/wc3dIaywUV9lgAAAABJRU5ErkJggg==\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" + "name": "stdout", + "output_type": "stream", + "text": [ + "prec = 52.80 40.69 30.92 24.52 20.00\n", + "recall = 52.80 81.37 92.76 98.07 100.00\n" + ] } ], "source": [ - "from matplotlib import pyplot as plt\n", - "plt.bar(\n", - " [1,2,3],\n", - " [run_time_scipy, run_time_pytorch, run_time_pecos],\n", - " tick_label = [\"SciPy\", \"PyTorch\", \"PECOS\"])\n", + "from pecos.xmc.xlinear.model import XLinearModel\n", + "xlm = XLinearModel.train(X_trn, Y_trn)\n", "\n", - "plt.ylabel(\"Matrix Multiplication Time (seconds)\");" + "Y_pred = xlm.predict(X_tst, beam_size=10, only_topk=5)\n", + "metrics = smat_util.Metrics.generate(Y_tst, Y_pred, topk=5)\n", + "print(metrics)" ] }, { @@ -800,9 +786,9 @@ "id": "1f833846", "metadata": {}, "source": [ - "## Hierarchical Clustering\n", + "## 2. Hierarchical Clustering\n", "\n", - "Hierarchical clustering is an essential function for tree-based XMR models and plays a role of the indexer in PECOS. Accordingly, PECOS also implements hierarchical K-means algorithms in the manner of efficient C/C++, which can also be considered as useful functions for arbitrary tasks. The Python interface of PECOS hierarchical K-means algorithms is as follows:\n", + "Hierarchical clustering is an essential function for tree-based XMC models and plays a role of the indexer in PECOS. Accordingly, PECOS also implements hierarchical K-means algorithms in the manner of efficient C/C++, which can also be considered as useful functions for arbitrary tasks. The Python interface of PECOS hierarchical K-means algorithms is as follows:\n", "\n", "```python\n", "from pecos.xmc import HierarchicalKMeans\n", @@ -827,7 +813,7 @@ "id": "3b3b073c", "metadata": {}, "source": [ - "### Naive Clustering as Degenerated Hierarchical Clustering\n", + "### 2.1. Naive Clustering as Degenerated Hierarchical Clustering\n", "\n", "When `min_codes` and `max_leaf_size` as the stopping criteria are large enough, the hierarchical clustering will be degenerated to conventional naive clustering." ] @@ -897,7 +883,7 @@ { "data": { "text/plain": [ - "" + "" ] }, "execution_count": 24, @@ -906,7 +892,7 @@ }, { "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD4CAYAAADo30HgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAC4a0lEQVR4nOydd5xcVdmAn3PvnT7bd9N7JYUeepfeQVHBTwUVQRQLRcSGiogoYEMFARUQFRABkd6LhJKEkISQhPRetu/s9Hvv+/1xZmdndmY3bUMCuc/vF9i95Zxz7+yc95y3KhHBw8PDw8OjC2NnD8DDw8PDY9fCEwweHh4eHkV4gsHDw8PDowhPMHh4eHh4FOEJBg8PDw+PIqydPYDeqK+vl1GjRu3sYXh4eHh8qJg1a1aTiDRsTxu7rGAYNWoUM2fO3NnD8PDw8PhQoZRaub1teKokDw8PD48iPMHg4eHh4VGEJxg8PDw8PIrwBIOHh4eHRxGeYPDw8PDwKGKX9Ury8PDYsYhkkPhdkHoKzMGo6CUo39SdPSyPXQBPMHh47KZI26WQfgNIgT0fSb8Kdf9E+abs7KF57GQ8VZKHx26I2Ksg/TqQ6joCpJHOW3fiqDx2FTzB4OGxO+I2gfL1OCjgbNgpw/HYtfAEg4fH7khZdVEQgid/4EPx2PXoF8GglDpJKbVIKbVEKXV1H9d9QiklSqlp/dGvh4fHtqFUAFX9B1BR/Y8ABA5FRT63s4fmsQuw3cZnpZQJ/AE4HlgDzFBKPSoi7/W4rgL4JvDm9vbp4eGx/ajAITDgDcjOB6MeZQ3f2UPy2EXojx3DgcASEVkmIhngPuDMMtf9FPgF3dYuDw+PLUTEwY3/DbfpTNzmzyDpl/qlXaX8KP++nlDwKKI/BMNQYHXB72tyx/IopfYDhovI4301pJS6SCk1Uyk1s7GxsR+G5uHx0UA6roXYjWAvgOxMpPUbSOrpnT0sj48oO9z4rJQygF8BV2zuWhG5XUSmici0hobtSifu4fGRQdw4JP9N8WY7hXT+bmcNyeMjTn8IhrVA4T50WO5YFxXAVOAlpdQK4GDgUc8A7eGxhUii/HG39YMdh8duQ38IhhnAeKXUaKWUHzgXeLTrpIi0i0i9iIwSkVHAG8AZIuJV4fHw2BKMejCHAqrgoB8CJ+ysERUh9lLctitwm87Gjf0WcTt39pA8tpPt9koSEVspdSnwNGACfxGR+Uqpa4GZIvJo3y14eOweiLMW8KPMrVOTKqWg5g9Iy5dA2kEc8O2Fqrhyxwx0KxB7NdJ8DkgScMFeog3jdQ/pcXt8KOmXXEki8gTwRI9j1/Ry7dH90aeHx4cFcdYjrV8GeyUgiP8AVPUtKCO6xW0oaxw0vAj2YlDhXcaLSBJ/A0kDbu5IGpzlkH0b/PvvzKF5bAde5LOHxw5G2r4B9hIgDWQgMwOJ/Xyr21HKQPkm7jJCAQBnHWD3OKh0yg2PDy2eYPDw2IGI2wnZd+leUQNkIPXMzhpSv6KCJ4EKFR8UG/wH7pwBefQLnmDw8NiRKD/a9NbzeOQD6V7ERiS94zoIngLBU4FALrVGCKp+iTJqdlyfHjscrx6Dh8cORCk/EvpkjziEEEQv2aH9ijhI7BeQ+CdgI779UNW/QpkD+7UfpQxU1fVI9BtarWTtgTLC/dqHxwePJxg8PHYwqvIHiDkUkg+ACkD4Qoxwuawx/YfE74LEfWi7BpB9G2n9Cqr+4R3SnzIHgTloh7Tt8cHjCQYPjx2MUiYqeiFEL/zgOk3eR3GktKNdSZ0NehL38OgDz8bg4bEDkPRrOuHdxoNw2y5DnA/aS6dnER7QVdq8taDH5vEEg4dHPyPZuUjrJTrhnbRC6imk5TxE3M3f3F9EvggUegv5wD8NZdZ/cGPw+NDiCQYPjx6IZBCRbb8/fjd53T4Ajvbrz87e7rFtKSr0Cai4AowB2gMqeBqq+vcfWP8eH268faWHRw7JLkLar8xFF0eQ6BUYkc9sQ0NxtNqmEJVLG/HBoJRCRT4Pkc9/YH16fHTwdgweHuR2CS2fA3sR4ILEoPMXSPq1rW5LhT5BsRoHQIH/gP4Y6g5FnPW4sd/itl+DpKfv7OF47CQ8weDhAZB5i5LUDpJEEvdvfVuB4yB6ERAE/GAMRdX+FaUC/TDQHYfYS5CmUyB+OyTvQ1ovwY15NR92RzxVkocHoKOTy6h/VDnvnr5RSqGiX0MiXwa3A4y6D0WmUYn9Klf7oes9JCF+BxL5Asqo2JlD8/iA8XYMHh6g1TwqSvFXIoAK/982N6mUH2XWfyiEAqBtKz2Fo7JyifI8dic8weDhAShloWr/Cf6DgQCYw6HqJpR/v50yHhFBMnOQ5MOIvbz0vLMet+N63JYv4sb/hkhm+zv1H0xZJYI1avvb9vhQ4amSPDxyKGsYqvaunT0Mnfiu9SuQmQFKgThI5AsYFZfr884GpOn0nNrHhsxMJPUM1N6zXbsTFf0mkn5VFwPCBXGh8vpd3jayOUSSOkVI+iWwxqEiF6OsETt7WLs0nmDw8NjVSD2phQLJbs1O/K9I6EyUNTZXHCdJt7E8Bdm5YM8D317b3K0y66HhWUi/Cm4bBA5HmQO271l2MiKCtJwP2QVAGrJzkNSTUP9flDl0Zw9vl8VTJXl85BFxceP/0Ckqmj+lV9e7MJJ5DegZ82BAJhcgZy8HssWnldEvtgClfKjgx1Dhj3/ohQIA2Tlgv093wKELkkLif9uZo9rl8XYMHh95JHYjJP5B12Qrbd9GKhMY4bO2rh17CRL/O0gMFToLFTi8/wcLYI4HAhRFTysF1kj9Y+DYXHxFgfAQG3xeKc0S3I1AT/WaDc7qnTGaDw3ejsHjI41IBhL3UrwCT0J86/zzJTMLafo4JP8JqUeR1q/hdv6pX8fahQp/EowatHAACIG1J/im6ehsiYOVEx4qCvih8gcos2GHjOdDjf8ALTQLUSFU8ISdM54PCd6OweOjjWQAp/S427F1zcR+SXEa6yR0/gGJnI9Swe0ZYQnKqIT6x3Rwnb0Q5T8MQqchnb+B+F8B0W6k5lCIfgsVOBBl1PbrGD4qKKMWqbwOOn6gY1IkC4FjIXjazh7aLo0nGDw+0igjilgTdabTfN1ln54ctoayqgfRRtodUN9AGZWo6Je7e7JXQ/wv5NVLktY2BXejJxQ2gxE+Ewl+TNfeNoehrOE7e0i7PJ4qyeMjj6r+LZjD0CkqLF28XoWQrTHW+g+ipHazUaGzl24FIq52N5XU5i8uxH5X7xKKSEHmjc30J4i9CtnKHdIHhdirkez725XNdktQRgUqcIgnFLaQfhEMSqmTlFKLlFJLlFJXlzn/FaXUPKXUO0qp/ymlJvdHvx4eW4KyRqDqnwXf3oAF0gHJB5Cm0xFn7Za1UXE1GAN1CmsV0Xrqql+j1JZ/hST9JtJ4ONJ4PLLxQNzOW7f8IcxxID1VYgHwTem9v+x8pPEopOk0ZNOhuO0/+GBrQvSBuJ24zf+HNJ2CtHwSaTwGsZft7GF55NhuwaCUMoE/ACcDk4Hzykz8/xCRPUVkH+CXwK+2t18Pj63CWapdF/N2Alsnyev8yxbdrsyBqIbnUNW3oKpuQDW8hgoclD8vmRm47T/Gjf2m7E5E3E6k7WJdl4G0Hkf8NiT9ypb17xsPweNBhXNHgmDUoMKfLXu9iI20fBHcDblnzkDyv9uWFHAHILEbIfsOkNYxGe56pPWrO3tYHjn6w8ZwILBERJYBKKXuA84E3uu6QEQK97ERSrOVeXjsWJw1WhVT9Jdna4GxhShlQRkXVbfzduj8A3oCtpDEXVB7H8q3R/dFmdcocZuUJJJ8BBU4csv6r7oR0i/pdNjWKO0ya0TLX5ydD/RMk5GE1EMQOW+L+tuhpJ+lOBZDwFmDOE1elbldgP4QDEOBQsvcGuCgnhcppb4GXA74gY+Va0gpdRFwEcCIEV7IukffiCQh9TQ4jRA4ongi7olvb+2RUkQQAsds3xjcBHT+nu6dSBYki8RuQtXe2X2hKjeBG6Aqt7gvpQwIfgwVLPv16dF0RRnVE6Cqt7i/HYqqBsrUwc7viDx2Jh+Y8VlE/iAiY4HvAD/o5ZrbRWSaiExraPB8sj16R5xmpPFEpOPHSOevkeZP6ZV7LyijBip+BPj1JK1C4NsHFd7O1bO7UUcd9yQ7F7flC7it30Qyc7Tx2qim2IDtR0XKq4K2F2WNydkf/AVHg6joRTukv61FVXwT7QzQRRBC56AMTzDsCvTHjmEtUGjqH5Y71hv3AVthdfPwKEXit4LbTLc6wobO3yHhc8q6b0r6JYj/Nnd9FCq+iQqdu/0psc2hlHgrga4Al1MfSfp5iHwJqv8E8du0J5E5HFVxFRiDkcSD4G7K7Xr23L7xFKBq7kA6fwupZ8FsQEW/hdpFqsip4IlQE0A6bwfp1EJhO1Kce/QvanvdxJRSFvA+cCxaIMwAPiMi8wuuGS8ii3M/nw78SESm9dXutGnTZObMmds1No+PLm7Tx7ULZyEqiqr5U8nkJ/ZKnY20MEBNhVD1z6DMgds/luSz0H45KDNnw0iUucoAfFB1M0ZIR92K04Q0nwVuDG2Q9kP0Yozo17Z7TB67L0qpWZubXzfHdquSRMQGLgWeBhYAD4jIfKXUtUqpM3KXXaqUmq+UegdtZzh/e/v12M3pcj0tRDJgjS25VFJPUFq204VtTKYn4uDG78ZtPA23+RyUclENL6AqfgiVV1OsIunCBdLQcRUiOkhN73pa0Ok6XCAFnbchbss2jcvDo7/ol8hnEXkCeKLHsWsKfv5mf/TjsfsibgzcVh25qgxU9CtI+mlw4+idQACiX+4lCtikNJGaKhMwtoVjif0MEv8GkuDopHxUXocR/gSIi3T+MecmWg4F9lLwTYbMO5QILOXT5/0fXDSzuAkk+V+wF6MCB0DgOLQXusfuipcSw2OXRsRFYtdB4gHABBVFopfolXb0W3qX4LaiAkei/HuXbUOFTkPifyj2SlImEjgORCDzss5Wao7s2wUUdMRy4gGKXUFTEL8Fwmdoz6HqW5DWL4HEKcnTJHZ3Cg3/fmAvpMhtU7Jldz07CnHjSPOZ2rOLJJJ6UFdyq771w1OS1KPf8QSDx65N6hFI/pv8RCxJiF2rf1ZhMAai6h5CGZFem1DmEKj5M9LxI7BXau8gtwMaD0dUTW4CTwMhJH471D+KMqrLNyYpunMuFVCQckL594YB05Hko9DxE7RwsLUnVOjc/K5GRS7WRWPczlz/Poh+5QPNfSTJf4Ozibz9RRKQfl0HA/r3+cDG4bFr4QkGjx2CZGbnvG+GQfCEbS4PKYmHc9XKyp4EZz2S/BcqckGf7Sj/NKh7RAuH5EPkJ3cp1OcnwU0izZ+EmjtQZWodK6MaMWq1F1H3UQgWJ+VTyo8Kn4P4D0IS/wB3Eyp4CgS6YxCUWQ/1z0D6KT05B45A9ZHiYodgz6c4a2zX8SWeYNiN8QSDR7/jdlwPifuBNKggdP4O6h5CGRVb35hRjbYP9OY9l8qlVihGsvOQ+IMgrRA4GszR2l0183ofbeVwViLNn4KGF0rUSuJs1LaOnpijyzalrOGoyu/02pUywhD6eN/j2YEo30FI6qkewle2q0Sox4cfL7uqR78i9ipI/JO8p40kwNmAJO7dpvZU5ELA18cVwZLKZW7iUaT5PEj9U6/GO66G1nMhM50tzsYiGUg/V3o8OxdKdj8Cmbe2rN1djdBpYE3KRRwHgCCEP4nyTdipwxLJIKmnkPhfkeyinTqW3RFvx+DRv9gLcwVRCspSkobM273eIm470vEzSL+gdwjRb2KETtcnrdGUX78oIADWCFToE91tiUDHTynNE7S18To5odYTc1RpRTB84JuA2Mt1QZ/su+DbE1VxVVl11K6EUn6o/YfeSdnLwL/fB6/O6oG4HUjzJ3IGcRv4NRL9CkbUS7L3QeHtGDz6F2uPMjmJAuDft9dbpOVLkHpcp8N2VkH795HUi/pc6knKGnsB7Svqp9DzRzLTgfatGLAPot+m7BopUJqTSPnGQ+BIbUjWF+kUG8GPa9tE+gWdJiP9AtJ8DuJuzVh2DkoZqMBhqMjndrpQAJD4XeCsRwcKZtDxHX9EnMadO7DdCE8wePQryhoB4U8DIXSSuDCYA3pPD22vAPt9ijNtppD4HYizCWK/pHT1D3oHkAVnARK7WR8RgfaSciCbwYHM/6Dy2lyiuwCoGlT1b1G9VGZT1b9FVf4MgmdA9GJUw5Oo7Bu5MqJdOxNXC8jUk70+t8TvRVJP67rU/YS4LYjTWwzFh4TMm5R85sqfc+31+CDwVEke/Y6q+D4ET9JfcHMoBE/qvS6ypHQSup6aHkkjnX/IuZL2hQOp56Dqx7rWQTnDcJ+4Wo0S/gbU/B1l1oJR32eAl1ImhE5DhbrrBosboyRYjWzZ2tJu/F6I/QIdZGeCUQN1/9aJ/rYBEReJ362N/CQAE7HG6vQg5pBtanOn4tsr51BQGN9RPqrdY8fg7Rg8+h2lFMo/DRX9mg4Y600oAFgTQdVQHJlsgv0eJO+jdzVSAdKEG/sdoqJ6ot0W2s6DljORxhOQ7Jytvz94HKXrLCt3vGCobntOKOSK9UgcnI06mdw2IrFfQ+cvgThawtpgv4+0XrrNbZb04WzA7fgpbvPncDvv0CnPdxAqcmHOG61LXReC8HkfTiH3IcXbMXh84IikkM5bIfUEGLUQ/Rok7tG+87joyW1rSlC6EL9V2yn8R0P6JbRXVF9urr2RhJbPIQNnovJ2hM2jrHFI5Y8gdp3Ow6QMqPihTn9diL24jHE+u9nazb0h4kDibkoirBGwFyFuW+/Belvah9OokxBKHLAhO0e7uNY9uEOio5VZB/VPQ+q/iLMW5T88V3Pb44PCEwweHzjS+vXcRJgGZyVkF6BqbkOscdB4NKUqmS3BAWc5OBvAGqf/uQnIPL8N7dmQfq1ktb85jPAnkNBpegcAqPRLSOI+CJ7YrSYyR+VsEYWY4Ju6lWPswqG8DQa0YPT3cm7LkcQ/c3EOXe8xlSuVOhN2UBpvZUT1LmGHtO6xOTxVkscHijjru4VCHr2D2KYAuBKSeufh27N7hbv1o0Q6rsFNPq4TzLmdW3ynUgEt7JpORWI/QzquQTYdjNt5hz5v1kP4fLrVJEFQFajoJdswzpy7qW9/SpMEGhA6vX8K3zirKCt8ytS2/jAgThOSmbVVn+vuhicYPHpFxEGy72rPof7CbS9vB5BWrbrxH0Zx4ZsAmOMonfj6wobYzyH7v+0YZxO0X4Zs2g/ZdCBuy/lIH4ZtcTsReymum0bar0YLvi41lkDnTUhmBgCq4kpUzZ8gfAFEv4VqeHq79Oeq+mYwx9P9jhQETkJVXrvNbRa1HzimwD03hzi7tHpHxNWfSUG9GRHB7fg50ng00vplZNOhuIkHduIod108VZJHWSS7IJchNAniIL6pqJrb+8w8ukVY40FFegSPBSF4Jm7i/txuomuCsyB0HmTnUd5WYFDeFtEzjmJ7cPW/zAyk9XLENxaS/9HRz+ELUJEvIfE/QudtOYFn6IpkJQgSvxflP0Dr5QMHowIH98sIlTkI8e8JyZXkBVL6KaQ1DpU/QFkjt6+D4MmQekFHkQPgQviLOlYDU+fC2kaPqh2Bm3xCJy+UDjAaoOoGVOBQ7ZacvA/IdKvzOn6KBA5DmUN36ph3Nbwdg0cJIoK0flWvmiVX7yA7F+n89Xa3rZSJqrkDjEFodYpfJ6ALfAw6rkNPbF3qH5/+2e4lalo1QN3L4DtYr2hVRLdXrtTmdmND9jWdA0radRK9+C06KV/n7XrckuhFKHS3ISJIZgaSfAg3uwpx27Y7jkHcdkj+l2L1nKvTiTefidhLt6t9pQwwcnEpZAEFiVuRjp8jseuRxo8h2Xc308oHg2Tf17Es0go44G5AWr+i1Uep58okZDS0PcmjCG/H4FGKuy5XT7mQDKSeRSKXQPp5PREHju0z3XVvKN9kaHhJ6+KNKpRRi8TvKbMnSOVSbvfmWdSBcleg6u7RE5OzFjGnQPOxvVzfHxRMvpLUuwe2xHXTD6FzkJbPgL1Aey6RQjCAABL9Kkb04m0bkhuj1zWepJDO21DVN25b24DYa3LP2fXsXYI7nf9opP0aVP1D29xHfyGpxyi7Y0w/l6uD4afIXqIMvavwKMLbMXiUoiopr6IJ6NVhx/VIx4+0rtZeDuR07JmZiLMBcTtx4/fitv8EST2r9b3OetyWi3E37ovbeBKkX0ZZo7trDxi1ZWwPfvp0N5U0kp2th+ybqgvMZ17r+55tplwVONhit9rwhdpjKjs/p0brSnXtAkno/DXuhqm4jcfiJp/qo6FyQxsKRl0vJ13YXhuRs1K72PaF/f729dFvBCid1gzAjwp9Umf7zZ/3g1EPgSM+0BF+GPB2DB4lKKMCCZ2dWyV2TWCBXO2CroIuAArpuA4JngkdP9ClMiWD/rPKrYiTD+svXnauziGEC84ypO2bUPu37qprweOg/RqKyYIxGNy1vYzUhc5bcVUUpQKIMQhiP+3HFxHJPWdX4FjPuIicYdyZv5l2alAVX0favkHZ2geAfl8ZcFZD+1WIWYfaQldQpRTU3Ia0fhHcnvmEAhAszfm0Vfgmlcl/1YPttWP0Eyp0pi62VLhrUAYEj9deb3UP62p+2YXgPwQVvRi1jSVeP8p4b8SjPNHLwWmG7Ay9qopcCh096wqIrluceYtCtUKxa2MC0i9SaihOI4kCweC2UOoS6fYhFLrbIfYzhEDu/t5W8CaoCpC23Fgs8E/LpVyaTak6SEHN/Tpdt9CjXaXvD56qkwZ2Lu4x9tz5XHpuVfN7lDIRayKkXy7znD1JIfG/bbFgALQqTVTu2SSXRtsFazIq8oXS690EpJ/WGUwDh2v1Xi8ooxap+DbEbqT7c8zkftbvUlX+eIvHuiNR1jCo/QvS8VMd12JNRlX+JO8KrazhqKobdvIod308weBRgrid0Hx2zs6QBicNybso0c8CmLU5I3W6tKE85SKQBezlOXfCLBL7LdvuTST0vhLvGkItSJcu3tT/fNNQgUN16ogSo6SAcrVbZklb9agBL+hdituWqydtoydME1QV1NyKkjT4981Xr1ORzyLJ+3I2gb7eF2xeeBSMNP0SdFxLt3Az9G6n+jaUb0pJdLI4TUjz2d3j6Pw9Er0UI3pRr30Ykc8jgWNyVfkGIcZgnSBQWajg6XpC3kVQ/v1R9Y/s7GF8qPEEg0cJknwkt4LvmrxSOpVD+FxI/AM9aZladRT+Qi5PT1+kKfunZi9EOm/TSeyys9gxtoEc0k73ZOsCWYj/HkncUaa+AoAJRhUo1WNYCnyTuid7oxrqHkI6f6Pdav37oqKXlc/MqsI6uC3+d5BGStNYdBFChc/d8keL303xjscF6dQ5q8qkrJD4bbnPt0sQO9B5CxI+p89608oaDtZw/TOA7+t9j0uSkF0A5pBeM9V67Jp4gsGjFHsJJStwcbTKxYgAVeA/HFVxKRiD9SrYXkbfq9xyk28W4r9Du6X2Z+xBOcpNwm6ZnULBudjNEL0KYjegn80HyoeqKFapKWu4DjLrAxEXaf5MrsZyXwJQQfjTqMDRfbZX3HgvAqaswCNXNKnH+1Y+sJeCv3fBsDW4yWeh49vouI4MEjwZVXVDn1lrPXYd+sUrSSl1klJqkVJqiVKqJCG+UupypdR7Sqm5SqnnlVK7hqXKoxfKTVzpnAG5WRuRU4+D26mLvNT+AyIXanXNVtPbqrm/2dp+BFLPgCRQdfdD5BKIXo6qf1YX69laMm/k6glsblckkHwE2Zyxt+tqZ0POuFyYwdYAVYVYU5HU87ht38Jt/2l3BLtvX0rWhJLtt7TW4nZA+xUFcR0Z/S5Tj/ZL+x47nu3eMSi9BPgDcDywBpihlHpURN4ruGw2ME1EEkqpS4BfAp/e3r49+h8dLFXOH11RPLlmtPG46jodDR06G4n/eRt73d7dQogtiyXYWlKQ+g8qeiHKN6nPKyX7HpL4G7gJVPgTqMCRSHYxErsWsu/l0khvqXCydXnQvqreSQpp/YauY63MXMZWpcfs2xNVdTPS+atcre0kYCKpB6H2n6joV5D0U+B2otV8fohe1KcaaavIzMx5qBUeTCLJJ1Ghs/unD48dSn+okg4ElojIMgCl1H3AmUBeMIjIiwXXvwGUL+flsfPJvlsmLTSUGpBdcNu6f7UX5SaDzRlVe+IH/+GQeZlt3j0oXx8qoe1EVeZ/FHEg9SiSehbMEajI+ShzMJJ+DWm9hC6vKEm/hES+DIm/5gzegBPb8j7FBqOy+JCbQJIPQvZtveJ31mvbDJnuj8UYCvVPYxh+vWpP3E23es8BSSKxX2HU3plLa/2Edm8NHIHy7bVt76ccZj2l3mEmmIP7rw+PHUp/CIahwOqC39cAfWXX+hJQtt6hUuoi4CKAESNG9MPQPLYac3gZn/Ve9MJuo67NHDgarEm967T7JKNz2GxV/YUeSGmVtP6jW9sqbZfl3E2TgIEk7kbMUTnvnkKbTFLXhyh5b1tSH8IHvimonFpHnLVI7I+Q+i/aTmPrvEVdPxfiNqPcDWCM0Ck78nElBTg6IFGntf5USe9u8kno/I2uhBc4BlX5PZRRtZkx98DaE6wJ2vBMGq3aCpR1m/XYNflAI5+VUp8FpgFl4/NF5HYRmSYi0xoavDD1nYGyRuiynPm00D7t+ug/tPTi7Gyk7atI48e0YTr0f300PIDeM6QW1krexci+idv2PdzMu7l4jK6diYuuAbEUZFOZG3PJ94owIHA6mBOAaO6dNKBjHiJgDITw51A1dwLkCuScCakH0YKnoB5Cb8b0rtTl5khKBZMF/iO0IbyMDUPSr0D7d7TwkDZIPYa0fKmPl1MepRSq5i6IXATWFF3atfZfKGvUVrflsXPojx3DWmB4we/DcseKUEodB3wfOEpkq/UNHh8QImkInKDrEDsbwBqNCp+HNJ/Tyx0OuGuR9h9A8Bj0n1SZnUPl96D9W/00SoVOfbCZ2IX+IvWwLkqDxebjD3IY9cWqNgB8EDhEB5aRKUi4Z+hArKqfoKxx+aslcR9IivJCUyh+10EIHpfPcqqUD6p/pYsidXkCqQZQfmTjPkAG8R2Aqr4JZQ7ULcbvpPidZnWJUHtZaSW6zaCMMKri61DRt0urx65JfwiGGcB4pdRotEA4F/hM4QVKqX2BPwEniZRdXnnsAoi9HGk+j0K3UxX9sk7rbNRo9USvpHTAU1kUpLejNkLpSOl/oRBGT/rlVuKOFpKbtYEoIATKQFXfAm4r0v5dbWdQIaj8McTvpGyEd3aGfvcNL3cX13HXlbm2q6uQjkZPv6BrXITOREW+mD8t6VeQ5KMQPB1841HWHoi9JBe9nHt32ZlI64Wo+v/mHnNDmX6MnHDy2J3YbsEgIrZS6lLgafTe9S8iMl8pdS0wU0QeRauOosC/cgE3q0TkjO3t26N/kbYf5PIhFR77NqrhKVT0G0jbFWzbhCw5dciuTGIz59PoXYpDqYdWFzmBVfN3lH8f3OQzILkUFZLRFc+c1WXu67pd55ZSEa2SU4FjkdQTZQzrAQieAuEvoQJHgvKjrNH5s27nrbo+RM4WQsoPtX/N1SIobMsBeyVir9KutM6a0jGpSp32w2O3QhVWONqVmDZtmsycOXNnD2O3QSSZUzH0/HswUAPn6PQP6Zd06gp7EdtWMnNXxGTrvKH8uYnSBbu3GgQR8I2H7DvFh1UIVB24ZSbgPBZUfBcj8jldu6Hjh7nCQDlPseAZqMgFoEJI6wU6gllcsMahav+sj288kBIB7ttPxxXYC3v0F0A1PK29quwFpcOpfQDDv08f4/XY1VBKzRKRadvThhf5vBsj9upcPvoBWhddVpeti9+ICJJ6QafG2B4Pol0JYwi467fypgzY81EDZiCbDqC8UImXCgXQK39zUC4xYG8LMhuJ3YDtPw7LNxhVdR0S/SrYK8E3Oe8h5Dadnau5nPss7IVIx3WoiivLN+ushuiVEPtxwQ7E0m2aQ7R7awkBlNmAZBch8b+AsxZUVHuB+aaiIheizAG9PIfHhxlPMOyGiLNJl+20l6InFj+9GlWtcUjn77W3S+pRtia5265NMFcXYVt2zA6SeBzMseBsZR0CswZC10Dsl+h3WSpYHMly3ewfcuTwb3Lc4D31xO2s1zUwJAuhM3Or+0IBbWuvqaobwYiCW7hjMMF/MCp0FiIdEP+TDm4LHI2qytWFDp0O8b/S/XegwByMOC3Q8ln0DqTgXWXnIKn/Qv1TW+/O6rHL4wmG3RBpuzKnDuqiD7uB/S7Yc9GTwod1pxDO+fQnyZemxEYLxC2JLShD53VQex+0bGUkb/plXflu4AxwW5Dm/wO32O5gAjHb5ifz/sWkqqEMVjOh/Uryn1P6Bcp+Fka1LsNZ9Wuk7WLy7sFGNariO9qNNHI+RM4vuVVFL0XsZZB+CTDAHIiq+RPS8XPKR5VnwY0jyUd0mx4fKTzBsJsh4kD2za24Y0cnt9vRBFC1d4Bvf51eO9M1qdoF8Qdba2cASEPHd7dhPBmIXQuhU3PeXpGSOd4Fko6FAK9sWsCnI7+gWHiXG6sBkW8AoAIHQcP/dDU7FQX/QZstRqOUH1Xze8RtATcO5jCUUkg5g3SelFYveXzk8Ep77gaI04zYS7RQ6CpSs7uggogYOggv8xzldz2iA88CZ6J3EVtI0a5rK3Fyto3Qx9HeTt0YwPdHzKTCdIhYAV1MZ3OoGozwWd2/GlFU8ERU4LCtqlCmjFqdLbYrXXfwWHp9JyqEChy1xW17fHjwBMOHFJEkbudfcFsuxI3djDjNZa6xcduuRBqPQprPQRoP1xlSg2d98APeWUgCWs/LxQT0hqmDztKPs3U2lHIqqN5qQ/e4L1efQIU/C/5pRS0pBXW+FKfVLuNjA6eC/wD6/qqaO6xusYp8BXyTtUdVXkDkfg5+vHxEvMeHnt1o6fjRoTu3/1IgBZnXkcQ/keilqMBReZ92Sfxdpzsmo/3oJYG0fhkKoms/+myJKqz3a7oqPW8xoQty6bp7xisEc/34oOJ7KKXTZCtlacNv5m0KdflBw+WiwYsxOz4LwTN1jQzp0KMRB61OMnKZVSOoisu3ZpRbjDIiUHu/Nna7LTo3lL0crNEYu1DVNo/+xRMMH0Yyr+eSoXXpnbM68V3sl0jsZiR6CUb0q5B8lBLDsnTk0jt4bAlbJRQAkr2kHlcNEDgUwp/F8E8sOiWqhnJxISYdeoeXXQjWXuA7HgIH66A2t1kbslUlBI/NV5TbESilwDdZl2FN/AU6/wjSies7AFV9I2oXzZoqYusCUmZ9/6UU303wBMOHEWctlA1MzGXc7LwV143lM2kW82H1LNoWuorV7wLBeLIaUg+A8z5SexdKhXS96JYv5WJD+vKMyoA9U3uIpR5Fot9ABU/Ol/90k08jnb/SwiJwFKry+ztmIkw9DrHfkd/ZZGciLedD/dNlS4juTCT9JtJ2KXrRZCOhs1CV12qvrV0Yyb6LJO4HbFToU6g+anLsSHbtt+RRHv+B9D3B2zoXfz5J2+6KsCVC4YML/hfIzkXif9W/dVyfi0cozJzaFymd9TT2c6TxGJ0PKf0atH87lxG1A1JPIi0XsCMyGnQX/enC1dX87K2M5djBiCSRtq+AtOdiVTKQ/C+kHtnZQ+sTN/W8VhEn/wXJh5CW83ETj+yUsXiC4UOIskZB9OtobxZfmSscdolV8k5nyybHLVns9t8860DyGSQzC9LPs22fUxZIIW3fzlXNK1QX2uCs3D6PqV4p86Kkl+M7k8zblI4pqZMK7srErkd/li75vFuxn+8QIb85PMHwIcWIXoRqeBYqrwNVg/dRbh0ikHV7n/BFus+lXYO4a9GS9eP2x3fUWaidALqqu20rkuilSJGxDZX0No8OZAsVHDHBHALWNtTA3pEYNZTGehg6FfquTLnsttLGB1cXvRtvNvkQo8xBGOGzofbvO3soHzocFM+3DiPuWmUne6X0P0dgabKSry05koDhYKgQsL0pINwtUPMpCH0Wgp9CezSVWZUrE0LnUTxZK1Bh8E3dzjGW6S54ElRcBaoWXfTnUFTt3bucfQFrEpjjKY6/8KMiW1906APFtxcln7M5bqviUPoLTzDsgogIbuJfuI3H4246HDd2E1JQolFSL+I2nYm7YX/cxuMg+ThbFZi1myMChgjH1qwhaNikHKPXnYOpYGQwxkBfkhfbRoAKUD5FRD+jIqjgMRjV16HqHoTwF9GfsYWePIIQvRIV+jhELtC/Y+YmbZDGo3E7f689c/oRI/J/GAPfwBj0Hkbtn3fJJHpKKVTtXRD5PJijwX84qvZvKN+knT20PlFVPwNVrav5qQioClR12WKXO34sXtrtXQ838S/ouI7uCSgAwRMwqm9GUi8gbd+gNBArwBZXF/MoQmTzdoaEY7I6HWFieEfWly4khKr7F8o3IX9E7NVI8kFwY6jQ6UUeKyIZpPMWiN9D999NEMKfxqj8/gc0Zo+e6BraN+v06P7DUZU/Qpl1vV4vkob0a4ADgcNRKtTrtb3RH2m3PcGwC+I2HgfOqh5HfagBb+oSm86ynTKuXY0tmdD7s/0d3V83JvgPxKi9e6vucjceoD1xigjk6mlsu3LAdh3mtK4EYO+akVhGz1rSHuWQ9Gu6zkXeOcACaxSq7vEdqn7z6jF8VCmp2JU77Kz3hEIPdsRk7YjWsW5zu8aoXM2FzUVdm+idXo/qceYYVM3tRYdEXMi8Ce4GnUK7XFBZWYOzzbalFtesjjdz8Vu3k7T1DjVk+bntwC8zIrKLG3J3AbRbck+PsbXaRdk3eWcNa4vwbAy7IsEzKE6sZoDRAG27R2H1rdnE9qdQEIGUY/Ta7ub66vJkyjrr2JCtRTb39QqcQGk8SgBCZxVFMoubQJo/jrRdgnT8BGk8ATf+j9L2gidRbGuydMCb2vYV/k/mPUhzupO4kybupGlOd3LtvF29TOsuQllBbej0NLs4nmDYBVEV38pltfTpfyqso1q93UIRPdU824MIJFyTfzROQLbBL79r56IU+FSGemsjbp+D8kPVDbrojgqR9yYyB6PC5xW3nbhb58WSRC5gKw2x6xG3teg6VXkNBA4j/3fjPwBVdcNWP0sh77atQgp2HIIwr231TvGt/7Chwp+mvMfYnjtrSFuMp0raBVEqAJXXIeKD9JO7XQTzjlK/9qV2UgrChsMn65egtkH10rNdE1Cq93bSEuKNVbdSGf08+1afgsrOBHMEBE8qzXuU/h+ljgUZJPEgKvrl7jEYUV1cx40Bbr9UVqv2R2jJdPY4Ft71XFR3RYKngrMCOu8AMmCNRVX/Zrt2cB8UnmDYhRC3E1JPgtuMJB/Sf1S7CSJaE25s43zTc54qJwTKzWWF1ykFAcMp2i9sqQ2j53U978mKSYcTpdbUxmHH7aTBuY9vzd5EbXAgdxx0MVX+cPnGrQm5xIc9BE38L0jkSyWGZWVUbH7AW8ilE0/kl/MfJeVqe0nQ8PG18Sf2W/sfZZRSujJe5GKQNMqI7uwhbTGeYNhJiBtHYr/SdQCMah3M1HkTSFfenN0n2Z0IvNNZz6ZsiP0rGqn39VFqtJf7t0QIlLuvJz4l2vjcx/3l+nOxeKJ5CH/dOJkvD5rPCTWrMQuucVyIGrH8fWHTYVQwxqGVq3m+zeL2Jc/y7clnlu1PRS9CkmWCGCUGbiv04f64vZw2dH8GBKv496o3ERHOGXEwB9bvTmnbtx+lfKDKpa7ZdfHcVXcSbsvnczlddn1D1I4i7RrEHYsrlh3OinSFrsQsBt8e+jan1a3sd8PylriebkyHmJ+oZUQwxrhQR9nrur4yhcdXcRbnvWPgYDAu2MYdE14kZOhUBhlX0WYHqDQzBM1igb8yFWGIPwEofJFPoCp/lDNOZlBGLXM3PURd6noG+ztIOSYKIdDVhoqgBrylJx4Pjxy7TByDUuok4Ldo1eqdInJDj/NHAr8B9gLOFZHNujV8lAWDOGuRxpPYXQPSROCnq6bxTNtwHFGAKjL4+pXD41Meo8La8nrTm1P5ZF2wVN/X3LNxAndumIJPudiiOLRyA98a8jYD/Jn8fa7AY03DOaOhuBBP2vVxzcppvNw+FICp4Wa+Ongeg/wJXusYzGsdg7h+1BuEze68NyJ6X9i9s/CBOThf9jOpRoK9jFCBMOkWSrncP+ZQVORiVPBjW/KaPHYD+kMwbLdXktKWlD8AJwOTgfOUUj2ddFcBFwBlfOx2Q9w47OJ54XckL7cN5vm2YdhiIhglXkCmcnk9NoCVqShxp29tp+3Cnesncc6Ck/jL+onYovJuoyJ6IhfRSST6Egpr0hH+vGEyGTGJuz7SYvFGbBCzOgfxQtsQbNEr//80j2ZuooGsW/z5BYwsp9V21794N1HHFcsO47yFJ3Dz2n15IzaIt2IDSThm0Y7DLBpTNhfYmAWyBNwl+I3iHUb3M7jgboLsbKTtWzrC1sOjn+iP2elAYImILBOd0Oc+oEhZKiIrRGQuu5PivC+scUDlzh7FTuMXa/cnLb1P+FVmClcMDITXOwbxr8YxQKlNQERPrBkxWJuJcvemSfxy9b4oBbbAm8njmN6hy0+am1FLvdNZj9HDiyjpWszsHECrU80n3zuRo+aezS/X7MfKdCVZKf7quKJIud3eRAqXqJXl0iFzCSobBfxo5YFcu+oA1qVD/HmDFmJ9YajyX9BSAZeCzlv6fkAPj62gP4zPQ4HCffUa4KBtaUgpdRFwEcCIESO2f2S7KEoZSPA4KGdQ3A3wqZ7rA2GYP8aptatoygb40qCFVFppLAVD/HHiro+3Y/XsV9FUdFfXBPmlQQt4omUUTXaIJ1tHctnQOfiUy8HhlxDsslEJPe0Ee4RbSq7xK4eRwQTnjPsWr8ceZn27Vnu9m6ijMRtkiIrjM3RDhgqQNA8laCTJugZjQh1cN/INRgY7mRpp54X0ZVRk7uWkmhV8e/lhLE9V0uBLc0LNKoJGP6yXpG372+gnxG2F9ItAAALHoIxevK08dll2Ka8kEbkduB20jWEnD2fHYr+3s0ew0/hMwyJ+t25vXK3gARRN2TCDfHH2jTYSNGys3IRtGuAXl6ZssNf2smIwNdLCS+1DESAjJkFDFyvqa01euPIeG4xxcMVGpncMIi0WPuUQMbOcVbcc/Aeyzn4H2ASAoLhkydFcPnQOR1c3YlnDUZXf48zqGKdUfpeMkyqyJUyqnsKU2k/hZvdj9uqrWZWOkhGTm9bsiy0GJ9esxFSC3xqhU17k0ygEWO+Ox7FXMMzf2YcqzA+BXcOFVDJv5WpNdBl0LKi7H2WN2ap20k6WjGtT4dv6JHIe209/qJLWAsMLfh+WO+YBiNOMG/sNbutXcOP34mYXIYl/6C/OR5i+fBqOr16TU9t0v4OUWPy9cSKVZgarh0rHZzjU+FK9p8ZGWJOOYOAyNthBxOjb06vL6FuIUvDTkW9y6ZB5HFa5jv9reJ+/T3yWKstBpZ7iyAGT8Bvd66gWO8gNq/fVOYzctWDUQeAoLGURLvI88kPwFMSNY/jG0Rz6ISqXtiIjJr9csx/HzDubn646UNfVqPieDnQzBkDkfIYOuY+hw2eQiXwfVBSwwBioz6sQEAD/fqiKK/p85v5CxEWS/8Ft+RJu+3eQ7PsF5wRp+3Yu11cCJA7SgXT8ZIvbd8Xl1wse49jnf8qJL/yMz752C2sSzTvgSTz6oj92DDOA8Uqp0WiBcC7wmX5o90OPuK1I8+ngdgAZSL8MuDlj64fL3LKlgV4Z1+DxlpG81TGAdsfPqnQlAcPhvIZFnFW3HMuALEbZFcnaTISZsQZqfakio2tWTIb44mTFwK9cbIH58VpChk3S9dGYDbAqXcHIQIyvD5mz2edwRY/B7KHSsgzhkw1L+WTD0uJ70i9xwYgvMKtlAEs7GzEkgYvi+tFv4DMyIBkkdhNG7Z1Q90+k/QeQnYfOW5SG2HU6x1HVDexXcxg2JoUlPUOGzVGDDsSwGsA6FyLnFo9LgVV5PlLxWZ0SQ+UCpZyloMIoc0j5Z83ORTpv1ZXBgiejIheg1PbV7ZCOH0PqP7nJ30CST0Hd31C+vXSEvtvY8w7Izt3i9h9c9SYPr55BxtXvZ3FsA9+Y8Vf+feQVXrT1B8h2CwYRsZVSlwJPo91V/yIi85VS1wIzReRRpdQBwMNADXC6UuonIjJle/ve1RDJQGYmKD/49kMSD4AboztWoUu98OHSkm1OKHSt5AX42pIjWZysJiVW7oi+8ffr9sbF4FMNSxnkTzI80MnyVAVuXkQISdfHH9fvxb+bx/Hn8S9gKMFSQoft4/YNU1ifjXLRoHexlDAx1EbIdLBdRUYM/jD2RfYIt+EzNp/6whaQLd6xmZB+jlDmf9wxKsv78ina4rPZK7yWUIG6CHuxbt8ah6q7DzfxMHT8CLC7U5q0X0XdgFe5esqZ/GL+Q1hkyIrBx6rX87GGQzc7EqVMUAVRzVbvgWaSnYs0f458bYbOpUh2FqrmT1v43GXadFsg+RDdf88ukERiv0XV/lnnAVLh0nKj5pbbCx9ZMyMfZQ06N1NTJsaKeCOjo7teUaCPKv1iYxCRJ4Anehy7puDnGWgV00cWyS5EWj5PPs2xUZMrr/jhjlWwBRDyOv8uek6+rsCbsYEsSVXlhAKUqIo2TeRTDUvJuoobRk9nevsg7twwmZjrz1+bEotN2RCvxwZxfM0aAAb4U3xh0EL+sG5Pbli9P/fu8VzOhqBX+cp1cDHw5WTM5haWCoPrV+3Hd0e8Tchw+rjeIC/MRev4J6p/QUX3cRFYmKxllXsge0VbGBKqRmK/hsQdlOwKlQ8yMzh1wDAO5wkWJ8IMCcQZ7E9A7B0ksC/KGtn34LcQ6byV4pTPKUhPR+zVKGt46fXZhZCdBeYo8B9Svn6D0wjKKs0O6ujPSSkTqfh+TiCm0etEH6ryB1s87qBRGqwnIgTKHPfYcexSxucPM9J2WbFniBPPf2F2dXpbYa9KRfhP82jqfUk+1bC0xOWz0LPnvsbxPNg4lpTbe4KwpGvmciIphgfinF2/jBY7yN2b9ii6LuVaLE9VknEVfkNwBSJGlhmxgUyOtJS4eZqGVsf0HNt7iRo2ZsPsHWmizqcFdMY1WJKs4jvD30ZQZEVhIbTaAdKuyWB/Iv8u4m4lPmL4jcJi7CnwHQPZ6dhictXyabzdWYehgtjLfsMFwyo4v/YezLKqQpeU1PLs0l9yclWc/SvihSOG9PNgfbHX97dVOBsp2ZkqS1cSo1gwuB03QOIf+nplgjURav9WqnayxlA6ZfghcEz+NyN8NuIbjyQfARVEhc5BWaO2eNifG3MkP5rzQH7X4FMmk6qGMiRck79G3Bawl4E1DmVUb3HbHluOJxj6AXHju1TCu4wLL7QNZ1rFJup9ve9YXNFFaUBh5iwfXZPi9PZBfHfFwWTEwES4a8Mkbh3/MmNCsfz9hcLkxbZhrM/2lSRM2DvShAt5+4HfEPaLNnLvpokUpq4LGg77RpuwxaQtY7I+E+XSJUeAUixI1GL2mPBSrslLbUMZH+pAKZ1q41tLj+C9RA0uClcUXx8yl3MHLGFTJsjYUCsBo/veq5YfxJuxgRgIg/wJfjXmNQb40zzXUsdxNZ2l1bSzb0DtPTyx+nXe7lxHynXpKspz1+omjo9YDA/0fO8+UNX86J2/MiNWz685gyOq1nP50HeotLIIJrhtSPO5OsV68DRU9GKU6t0bC7T3zquNC4nbaQ6rn0B9MBcfEzwZOpdQvGswoEfdY7GX5oRCqutjAnshJP8D4U8WXauUD6p/R2fzN2jOBhkSiGNZo1DRS4uv801F+ab2Oe7eOGbgFJJTzuTOpS/QkU1y1IDJXLbHqfnzbux3EL9D774kg1RcjhHpJ2HqkccTDP2AOE3Qp2PkB88DjWMZ4o9TZ6X7VK2cMO8Man1p7t/jaTqcAHPjdQzyJ/jFmn3zQWg20O6aXL3iEB6Y9AygFSkPbBzHa7EhfGfY2wz2x5mfqCGisnRK11Ra2LGi1kqRFRNT6RX4rFgDj7WMImpmiDs+fIaLiOKoqrVMi24i7lrcsm4vXmgbjoM2HmRFcc3KA7l21Fs4ojCVMLezjvsbx3FS3WqG+zt5uGkM8xO1ZKR79/KbdXuzR6iVfSqai7ybbls/hbdiA8nmrl2VjnLV8kP464TnOal2KbZrkHVVPl4BQEixaN0P+d3SSaTcYhWHQpgbr2d4oHs3IKJQ5gheanF4ub0iH+n9fNsw1qYj3DbuZTDAjN/W3VD8D0j6Oah7tFej68ZkGxe8/keSTgYX4SYRrtv7XI4aOBkVOR/JztL1g5UFGKiaW0t3Adm5Ogq/UNZKEsm8geohGESEP63s5N4VJ2MqvZr/6d6f4ZB+zhp6ytD9OGXofiXHJTMb4n8G0t1FcGK/QfxHoHzj+3UMuzueYNhOxG2Blo9T6GWys/Ep+P24V/N6+N7YmAnzuYHvc0jFBh5vGcnNa/fL5wkqF5m8Nt09AWTF5NGWMaxMR7ng/WMZ5E9QbWWos5KsTpsl9ytcZscb6JqBHm0exa/W7EMqNyH7lMtJ1Ss5u345E8Lt+WNz4g1aKOTbEV7tGMqp757G1HALm7IhVqQrUbhYOfXNax2Di4RCFzet3Yd793i+SFA+1jKq6FrBYFW6gpjjR1B8Z/nB/HHcKxTOnAphfGAx5zUY3L1pUtH9WTFYkaxgUaKaieE2sq4iYwwh4q7k3k2HF6X/yIrJomQNjzSP4ayGMh7e9mI9cfv3Lj0H/G7RU7Rm4rgFY/vJvAd5puH7WIYfVXMbYq/ROxDfpPIeSdbYMr7FQbBKS0++1riIf658jYyr/66SOFw1++88fszVVH4A8QaSfpniHRCAA5lXwBMM/crum7CnHxBnHdJ6pU5/vLPHIsU6/5DhYORijMr5/4tAyMjy2QELGRxIcPPa/YryBJXznBri78QWxepUhGtWHMSKdAWCQcK1WJaqotUOsjxVVZIuAoRBvgSr0pXcvn4Kacfg9+v2zBmpdYBbVkymxwYzNqSFQtIxebl9CJuy3VGzfhxGBzsAIeH6eKtzICvSlbkeFAEjiy2K0cGOvJDoRrEhEyk6sjBRTdwpb9QMGg6/Wrs3S5LV2CXPo4XDOQ1LqTAz+FW3AHYxuK9pPBcvPprvLT8IWyBEI2CTKWN/yYrB64lpmGWdFNxcwFt53mldUSQUAGxx2JBq6x6nNQzl37t3N1VrT131TXW95yCY9bnqY8U8vX4OSac4saGpDN5qXtLrGPsTZQ6guOQtWqVkeN5K/Y0nGLYRyc5Dmk6B7Gs7p/8yk/2KVPeKvq+iMQBpMQgYDn4D3o3XlUlTodDCobuj1ZlKDp/zCb68+GguHjSPhyc/QYWZplBl5GDkIpq7CSiHaivFYH+cfzRO5FtLD6fDKZ2omrMhnm8bxtuxev64birXrjygoH/BxWB5qpJyajufcrl2xYGkxeQzA94naNpYOc+hgNI/jwu1kXK7732qdUQZ8SdMCrUQMW1mxgaQFovfrd2LlGvi5hLpSa5eQ5WV5aHJT/LVwXM5ILoRK+fJkxWTlFi8HhvEW7FBxG3hmZZhnFS7kqDq3lkqXAb54vxixIP0khUJ/Acg2XdxW7+J2/x/uPEHENHPNaac+6aA47rcv3I6/1kzk1g2Wabdgh6UQlXfgqr6BYT+Dyq+g6p7tGxRmWp/BLPMOCutDyg6OXh6ToB17UYtUJUQPP6D6X83wlMlbSUiLpJ6Ajp+qYONtqut/itjqRQYJZN7aX9daZ59ysXMfceH+ONlEroVRyZ3Hfvq4Hl8umEJZi7G4KiqtTzWUpzuIKBsGnwp1mSiDPAluXzIbA6u3ARKeKezge+tOJixwXaWparyqhWFy+RwC42ZEPPjtUwIt/P3hmf45er9mBXXE6DdY1KaEm7mrLplWMrl3Xgtr7YPwadcBvmT/GPiMzzQNI5VqSj7Rps4s245ppJc/ILkxulg9cimZOFyVt1KAAb4knQ4Af7TMob3k9UcX7MaV+D/BizuflbD5dwBS3ExmZMYUrTRSro+ZscbOKBiE9VWmv2im9iUCfFQ8zgcUYwPtfGzUW/mbS4lhC8Ge1XODTqtx519F7Fno6p+ztcnnsy8N/9E1nWwxSVgWJwweG8+O/0WBL2a/+3CJ7jz4IsZEx1Yvg9y8RHBE1HBvtNqfHLEwTy6ZiZJR7urWsqgNhBh/7qtS3exrSijAuof0a642bngn4aKfGWzBnqPrccr1LMVuPZ6aDoFiG/22i1hewRDuXvbsz6qfL3XMHAF3uwYwCFVm0rauHLZocyMDSAlFiZOTq9fKhienvJoUR9ZV3HCvDNIii9/Ta2V4r9THtct9Ggi7Rr8fdME3uwYyPJ0ZV5NE1AOrujJPysmp9cu58ph7zC9YxBXLD+85FmOrlrFj0bMIpCzozii+NnqaYwOdvDJ+iUEDQdbDGxRKFxCpuTfQZf31ep0hM8tPD4fd6FwqTCzPDDpKW5dfxBzO6OsywTJiIFgEDRsrhv5BodXFat3RGB68gR+sKSCZMGOJKhsvjl0DmfULceVbgN2xlFkMYmYfdmlAqjKHyOpxyDTc1fqJ1b9JCmpwBXhsbWz6MgmOXbQnnznnb/Tmun++1TA/rVj+eOBX+qjry1nQfta/vD+06yON3Fg/Ti+OuEEavwfnpKVuwP9UY/B2zFsDW3foL+EAmyfUHABpDuddNI1CZWZaIpqGgN7Rxvzx0GXnDQU3DD6dZ5oGcnL7UNosJI80lJ+FfjdlYfkjLEaSwnH16zm0ZbRKHS8wY2jp/ea5jpguBxbvYbbN0wlSJZRoRjjg6283jGIVjdMlzB6rGUUh1Ru4JHm0nEYuFw1bE5R5LGhhG8Mmcsp809jVmcDR1etozkb4KmW4UQshz+Ne4mQqe0uIrqOwzB/nN+M/R+/Wrs3a9JRJodbuHLYO0QMm9GB9fyneV/GBNoYEkhgIJxet6JEKHRxcP0kRq1ZxPJkkJRYBJRNjS/NiTWrMACzwKvJbwr+zTorpJH08+CsKzmTcFy+8sYNLE9VMyxcy2+nfYGh4VpaM510ZouNswIs6ui/1GWTqoby+wM899CPOp5g2AJEcl9ie/4Oal//vzdBUa7QvCHQ6ViETEfXAnBManylKome94VzK+eu444YvNtZxdRoK6fXrmC/SCMvtA+h3krSZIfo6XL6Tmc9J717KvtFm7h08DxSrsmptSu4etjbZDBwXcFQivXpEE+0jOTQqg3sEWorGsc78Xq006fF+FA73xg6j2+rObTbfn686kBmdQ4gKwZPtowgaDhUmWnanS6jo/C5AYuosUqNtTVWGgPhrdhA3ooNyh8P2Db/ahrL5wfqhG9ZAUdMLMNhn2gT90x8HiAnSEbQ4fgZE9RG8GXpatZnI/x6zFvsGVlf/gNSYGae4bZxy3i8ZQhvdw5gj3ArJ9esQFDYoj3FirHQTr+b2bEHj4f4XRSWgM2IwYpUBS7CqkQzl8+6m/uPuIxKX5ig6SdrF9sV+lIjeXiUwxMMZRBJQvK/SPY9sBdB9h30BNm3++fW4ArM6BzA2nSUqZEmxgQ6MCC/om3MhnilYzAhw+HoqrUlagelYGmqirXpCEdUraXGV5pRdMtUVcIzbSMxlcuEUAcD/EnOa1jCKTWr+PTCE4i7xV4gLoo2O8SLbUN5vWOQ9tFXUG8luWXcKwzyJ1mbClLjy/DZge/zXNswLlt6GDePeY0pkTZSjuKuDXvwg+EzOaZ6LSHDxsiNscGf4qbRr/HZRcdz85jXaPDpCc5UwjUrD+C19sFUmJkyhnL9rGszEU6pWc6zbSOL3GXTYvFWbCCfH/h+/t0O8hevrJenKrjw/Y+RFZWLtejuQwFzOiuYENrYI3Nq93nsBQQM+Hj9cs6uW44ACxLVfG3J0fx5wguMCnTkbTrgh+hXUdYkpO2baMNzGXuVsxZVdROSmQnZBaRcF1scrl5+SJEL7+pEC42pDhqClVw1+XSue/dhbNfBNAwsZXD5pNPKf/QeHr3g2Rh6IG4CaT5bZ6Skb4+ObSXtGlyy5CiWpypxc0bfW8a+wpRIC6aCV9oG88OVutZRVyK5P49/gRHBznwbCcfkJ6sO5M3YACaFWvnDuFfyE2z+WUSvR3seLyTlGFy/en9eaBvGvyc/ycCCCfM/uViD3qqtGbj5JHgGLntFmrlt/MukXCNffCbtGrzSPpifrDyQuyc8S6fjB6WYFG7Bb5T+7SUck3nxOvaNNhadTzgmp7x7GoJij3Arvx3zKoGCDKwuOoXHTWv25e34wKJ4gU/Vvc9XhszP10jIuIrZnQ3sX9GYT/F9xbJDmd4xuKTMKECdmeAno95i/2hTSX6owmjxUnysSNfxTPwTnFrxEIN9jTqvbOgMVOW1KGUhbhuS/A/EfklX9HQXK1JV/GbTZ7hy6EyG+VYxu7OW7ywdR4fTLayjZobTalbytbET8IeOgsBxLI838vz6eQRMPycP2YeGrmhoj90Cz8awA5DkI7li7D0DabaijV5W6iKwMRPg0iVHsS4bzU+qdVaSieE2TKVTVPxs9bTuyVi0UfTXa/fm12NfI+sqHBSvdgzhlfYhCIqFyRrmx2uZGtFVyFSuHQM9CQYMKTueTZkgz7UO44W2YdgY/LtpLJ8fuIhobndyau1KpncM5u2YLnup1TmKkYEO1mUi+Whh0P77c+L1iFBUkSxguBxdtY5rFcyL13FAZSP1vlRZoaAfV8ch9DwfMhz+NP4lrlymdx/BgpV7xlX8ds1e/LtlHCaSm9y1V9UJ1Su5ZOh8QgXBfu90NvD9FQeTFpMJoTauGDabhYmaMkJBuGrYLM6qW1EkAPKqP/oSCgEIn8voAZfxFSMMfFNXNiNQVNFMGdUQ/jyS+Fuu3nN3H36VZUH7ar7cMZD/TJ7F1FAz149q4tKlRwFQaaa5d+KzVFk2vvQcJP04BI9hTPWvGTPeUx95bDu7ZRyDuC24sVtwW7+BJB5EpGClVja6skwbArYoOh2LmN0jyrfMyt0RmBGr5xMLTmFNtrIg3bR2i+zaObTaQZJucXuCwXuJWkTgmdbhfOH9Y/nRyoPyE1nKtTCUDnXq6ttUOnhqdTpS4ooqAg82jubM907ld+v3zrmBKu7dNIEvv39MLn+SNiz/YvTrPLPnf3ly6mPcOf4FBvgS3DL2FSrMUu+nnonsul+IfooOx1/2vi7c3A5ndTqSH0Mhw/0dXD/q9ZLjfkM4rnYtoApULFo4nFO/tEgorExF+fbyQ+l0/WTFZH6ili+9fyzNdghFT2Fkc2L1qnygYP5xVPH/yz+zDxX9Zg8hUIMywojbjsT/hhv7FZKZrWMJav4ChIrsTQ2+JFcOe4e0a/J6xyB8ymZqpIVhfr1zPKd+KVVWhkD+vScg9RRu8tk+BrZtuK5DZ2oFjtt/zhceuy67nWAQtwVpOhXif4L0U0jHT5HWLyMiuE4bZP63+TZyOwJLCUHD5r1ELe901vV6/QONYzjvveP5xrKjcSiNfl2crMrrzautdFkdesiwWZ8JYyphfY8IXkEhqBKVUUYM7tk4mQ7Hj+2SL1Lzj03juGntfl1p8+gyMLuYbMiGmd4+KH89dMVIwORwK7eMfYWIaXPhoPkECwRBUNmcWrMCOxcE1kXaNXi1fQhZMfnThj05Zf6pdDg+3II52BXtwimAX7nsFWnR03rumqyreKBxLMe9+3EeaRlDoIffvwiMCPSoAaBHXuJx+1jLqB6RzAoDIaIyVJkZQkYWv7IJKIfvDptFxNqGgkrmOFTtPWWDxMRZjzSegMRuhPifcJovYPrqm/jnyjm8Fw8ViSafIRxRpQ3eVu6ZTVz2jmjPsomh1iJ1msaB9ssQu/+ikd9vfpzmtftjtpxMZsM0Fqy9il1VBe3RP+x2qiRJ/KNH8ZwkZGYhTWeC8z5bUlmtcKVoKdgv2sgnF5zEw5OfLDrXZvuZ3VnLH9bvXTZ3T35MwMZsiKGBBJYSLh86m1+u2Y+sGJg5HfinG96n2fazIF6di+g1KZz1XmgbxthQR9HqWAGvtA+iyQ4yL1EHKI6vXkWQDCUzZtfbcE1u3zCZ61dPY1igk88NXMSRucnJUDAi0Ikt2shaY6W5r3E8jij2izZyz6Y92CfayLE168jmhMOLbUP5xZr90Kt5BWJw0eJj+NnIN5gYbgP0pH9K7UqqVLakqtqqVITbNkzlhbZhgOLJlhFcPeztks+j0swQNTPahlHAg43jGB+clXdtzYpR4gdkKCFkOty7xzM83zqcWZ0NDPLHOa6mfNr0lGvwRMtIIkaWE2u7rjEhcgmEv4BhVpS9T9KvIe1XgbQCWiB+a+n+DPDN4srhd+biLgws3LyQjzsWe0WaObRyI6AXI98e/g4h02R2fDCHVTVhqZ67sCzSeQeq+hdlx7E1xDItDE5dRaQgdmWE819WNO/L6Prztrt9j12T3U4wYC+h0PVPkwZn4TY32eH4acyGSLgW7bafZ1qHsyoVxTSEoyrX4lN2H4JBOLJqHYP9ifxO5NS6VUwIt/Nc6zAULqfWrSLuWHxz6ZFkc5NHTx5oGs/+0Ub2r2jMBXXBNSsOZEQwzvxEHXau/2dbh+cMruUjm0+rXcFzrSNIiUVrIsg1K6r57vBZnFi7On9VV4zCMdXrOKZ6HUnH5DsrDgG0Kgz0ajfjKgb74/xq9GvYKB5oHM+rHUNYn4nyxcXHUW0msZRQZWU5o25FyTM5As+3DuPiQfP5bMP73LVpIksS1WzKBBkaLHYMsBTsEWxhxqrhOHMjEDdRVTZP7z2KBl+SCwYtJGJkOaZqDQ83jSFd8A4FCBkZ2rIBbl0/Nf+ORwQ6OaV2Zb4mhCNgi8nDzWP4w7o9sZRLrZVm/8pWjNAnUdGv9JqTSFIv5jyQutWUb8QGsTRVyY1jpudW/m7uubUaLOmY3LdpHNePer1IfRVUDpcNm0+8+mms2GfLpHyX7a4FIpl3kM5bMNLvlURmh0yHdPwB8ATDR5bdTzBYewDPsz2V1Qq9bpKOyR3rJxM2bebHa/j28sOYGm7mpjHTESBgODw8+SkuXHwMq9Kl3iGWcvn6kHl5N9WuHfq4YDtjB7fj5ozIn1x6UoEvP3TnMdIzhi0GVyw/nGH+Tup8SRYmahgTaM9FF3cLJRtTV2UrQTi9ZhlPtxZnG02JxR0bpnBi7WrSrsE7nfU020FOqdWGUlfg1vVTeCs2CBOHPXMGcNC6/z0jLfnV79RwC79duxePtIwFoM0J5caexuqxU3AF1qfDnDdgCUHTQQRGB2K82TGITy48mUnhVq4b9aaufoaeMK8YOodP3j8VHN2htPhwXq3h3tAe/KNxAgqXO8a/xNXDZvG79XvTZgcY5I8Ts/2cVLuaW9bvRdy1kJyG9cY1+1FjpTgyF9TWZdN5sW2ozgkliidbR3JAZSMk/4OkX6Y5eg9vt7dS569g39pRGLn8SdL5W3rarhYlqtk70kxWDAIFO1VTCTHb4ua1+/BU60guGLSo5NMySVPlD7AxfBfvr/kco4MtDM2n+g5CQXoLcVuQ+N/Bfh8VOAJCZ/VZ+1my7+bScKQIqdJIC0cgS7jcrR4fEXYbwSBuAmn9cq4wed9Rp65Aqx3IV/0CPWGnxUAJrM1EiTk+HFHc3ziO6bHBfKZ+ETeu3Y+0mPxw5MyiqNyomeXbw2bz9Zw3ScGouGHkdAb7E7ii4xLSjsHESCu+AvV4qx1gU7ZnojKVb6Nw5b8mE2VNJoqBy8JUNcYW1YkQIirL14a8y5Oto0vOttl+2mw/z7UN4/fr9gKBBl+KAyo2kXAtNmQjmDgcUbmOCjNDYzZIg09PgoV2j5DpcNHg9/KCoYtO10djNsjQXPU0Ea3y+emqA5gYauWptpHEHQsXlZ+0FyRq+ObSw7l/j2fyq+nR4RiqMoOxdxwqHEgYuPMiVDQKySEuDiZfXXIUZ9cv48fD36LOSvLtFYfpFbrAokRNvn3QMRt7RVtQuc/CrwRwuHLYbM5//3hAMFTXtJngqaYGrp99G5ahJ92h4Vr+dNBFRK1grnJaMWOD7UzvGERP5ZYjsCBZy1OtowCYH69hWkVj0btszAb5xLM3YYtL0DwU281wau1arho2BxU8FBXWq3lxW5Gm08DtADJI5hVIPga1d/da50E6/0ThwqnrM+m6PCsmNdWXlL3X46PBbmN8lvjvITsH/Qffd6BaRgwuXnwUX1tyJCnXwMl9KYKGm8scKlSbaaZ3DKbel+Lm0a/xZNtIVqejRAybOqt4ZdhluO0xIqaF1zM61M7KZARDwW/X7MnESFtRlKyhIGxkSyaPbsp/uX3KZVKwFRMXvyoUhD3bEQyEe/d4hmpfltHBDlTh6hWXhGtx0rtncNOa/Ui5Fimx+FeTntwDymVBogYB5ibqOXfhiXzivZP5wqJjymaArbQK1Xj6gv9rWES9L1Vkn+l0fMxJ1PFA83g6nAAOZo9J22BTJszKdLc+f2F7DcahHagqB2WAiroYB8dw/ZI3+qfF4r7GCVy14lAWJGu4bfxLjA+1MSzQyaRwS9Gzhwy7rBfViEBn/tnPqlsOaFvAz1fvS0Yg4WRIOBmWxjZy+ay7aUx15DKAFq/SD6tOsjQ9kCXJqnxJ1C43470jzUyLbiSgHBYlavKfsi2KpGNyxbIDyYiDi5BwHDJi8lTraGYat2HU3KarrQGS+GexTU2SYM+B7Gz9q2SQ7AJdVyT/cpvo+XfioOiw/SxJDWS173qGVZXmr/L46LDb7BhIPUNP20Kh142IFggp1+K6VdNYk6lkUzbMPRsnctHgBYD2+7+/cSwXD16A33D5+tB5gPaa+dyA9/nV2n1IuhZJ18JnFE8oa9MRClf3CpcFqXrOXXgyESPL90fM4v1Utc7+qYq/lIaCc+qX8O+mcbmEb+XsA/mnwo/D+QMW8J+m0WQwCWLjU46OiVCSUxV1jQP+NP5FBge0MPv56Nf5xtIjaLGDiMBQf5w16QiZHmsIV1QugE64etgsrlh+OC12MN/ugmQt7yZqmRpuKfL/VwgHVmwoSllxUu2qvGou7lisTkdzwqzbY6r8k+psqF2f3bUrDih7ebyOnCtqgbeUmNy5YQrHVq/mT+NfRoBJkTbe7mwg4foQIOGatNn+ovKoIlo9eN3I10EkHzuyOFmFpVzShd5WCO+0ruTc//2Gew75EoN9i7orpqkoVs1fuP/wgdwyL86JzgymRFqpNDMopZMK/mbcHJAYtihebh+CqVxMhNWZaImRHSDpOrzZ1sFBhSEM9lJK1aYKnFVIqh1pv0K/Scki4U+jKn4AwdMg+x6FAZ6WEaFq0BtU96GCKvpscqnBlerd6cJj12X3EQzGwKIAIgA7Z0x8tnUov16zNxHLodUO5n3hM2LxUvswLhq8gLhjccH7xzEh2MrSZAUvtA8n45qcXLuSYf4YcztrIbfevGH1fvxwxEz8hm5JgN+s3ZvuyVhPUnFXf8naHJMfrDiI4f5Onm4dzvE1a/LV1zakQ9y0Zh+iRoaTalbyXNtwOt3ev5wVRobbJ7zEMH8nI4OdfH/lIaSkuxiN9HC6MpXLWx2D2DOidzRD/XHum/g0y9MVhJWDrz3DF9cdT5uhGOhP0WH7cUTxifqlKAUbMhF+sPLgXGuFs7JiUbyaqeHulahS2pfql6Onc8Gi41iRriRk2AzwaTvBg41juGXdXphKyhT7KUa5LqzLcFnzQayrrgbbxfGbJXtgLZTKqd0UTXaI7644jJvG/A9bDIb6OhkR6GRhsoYuoXTdqmncOHp6PjNql1rpuJq1pF1FxlVYSqj3JYsC/gqJ22nuXv4W35t6L2KvAukEaw+UMhjog5/ufSZu64uoooWLgWUOZHnc5OLFh1NvJblt/Mv4DYeD1UZqzDTXrj6gyH4UNHwMCdUUP3/gCCT1HEVR/OIg5gRoOZciu0fiQfAfiAp/GsnOhtRToHyAhar5Y592iXzTkkLar4HU44AgwZNQldcVxXN47Pr0i2BQSp0E/Bb9vb9TRG7ocT4A3APsDzQDnxaRFf3RdyFpJ8u/Vr3B/zYtZGSkgc+POZKh4VpEBCf0Rea1rsFQ6fxk9WTrSG5evTdpfIAiWWJ6ECJGFkfgqZbhdNg+5iXq+MqSY7DFwEHxr6axuTiCbl5oH86KxRV8vG4Z+0abGOyPFxWvKZd2QYADKzfy25V7sfS9EGdOXMW7ZgM3rN4fv+Gg0LaKv054nsdbRnHXpkklbShcRgRjZF0DnyEcUrmBkt1FToumrK5fTe7ZNJGjqtbyemwQf904iaRrMWZlI3JdEx1NJoZazQVfaeZzV2zEUMLqdJRxIR03cM/GiaTdYtfZLgYEkmWDwHzK5bTa5dy7aQ9+PPItAsphVSrKLev20hHfuZfZe64nQRxIB3ysrgqCYYDfAFdQRvc1fuViKZewkeVj1Wt4oKm4/KODwex4A+szEQTFpYuPoMkJF30+b8QG87NV+/PjUaXpWQKGkHYN3ktU82r7EHpLiOciPLN+DqcN3Y89q0eU6PZt6wCUCqEkSbe7tB+qruFnC/5GzPHzrSFziJjZvEfYx2rWcufGyaxKVwAKnzKJ+oKcNGSf4s6Dp0Hycci+BWIANkS/hnJXI8rqMeQkknwMI3giqvomxLlKq5Ws8XnV1OaQjusg9ST59B6pZxAMVPVNW3S/x67BdgsGpfeKfwCOB9YAM5RSj4rIewWXfQloFZFxSqlzgV8ApbUDtwMR4Rsz/8p77WtIuzZzWlfwzIY5PHDwiVTGvw/OWsYHFX/euAdXLzmEs2qXclfz5NyEXW720d+Y9Zmw9qVfPxUbA7tHecae1cq6WJaq5o/r9+SHI2Yw0J8o8brpiQnwVpLI1St4wg7yhD2B9GlVOBcqkrmC82nX5I4Nk/n+iFk82DyGzoKMo/q/ivmJOr68+BiuGfFWrlC9QokgrkLSBtJiYQzroVJDcV/TeJ5rG07KtSDj0nhlDBXvngwe/1Mtk6fGOfyUdsYGu4PJFieLo7gLmd4xiJblldz/xl5kHYOP7/8e5x40DwMY4Evw78lP5pPoDQnEObRyHS+2j8jfrwqeTaE9uLJiYiKMDLbTUhekzSiYsAxFoSDMiElGTFKuyYpUJbVWiha72IjvUy4dToCxwXYOrNzE4zmDbyHNdu+FYAKGy6hgjAsXT8r9bpF2S50bEk6GS966kxGRem478MtU+fUK+oGVr/OH95+m3jqEK4fNZv+KVixrOKriapT/IBYm/4vg5lOmdGEp4Zaxr3DFitMI+0ewX+1ozht1mDZ0F6CUhaq9A8m+C/YK8O+LMocimRmUxuyYYDR032sOAHMry2YmH6VYdZWB1BOI3NirsRtA0q8jHT8CZ7UWRFXXoXx7bV3fHv1Gf+wYDgSWiMgyAKXUfcCZQKFgOBP4ce7nB4HfK6WU9GP45IKOtSzoWJv/UjoISTvFmg2Xs3e0iVnxBq5dNY1NmTBuxuDPa/ZEBaUPFbZCHNiwoppzkyfrFfaWOPjkEQKGw8EVG0iJlUs13XUqN3mp7mvtDofHL49Cunv/EXiyHWdyCPuwaO6ZDGZ3NhDP+vDND2MvrwZTUOMSGKO7jbdpsfjJqgNxcuoYAaQdJA5qSKZkJW7mDMipXCoO691kyeI3lTR5+p81HH5Kez4X07Otw1mSrKHU5iFYuDw6aw8enR8lk0sZ8scXDmJDewWXnTCdAyoaizLGWkr4zvB3eKl9ePeKXRXbZCaHW6gy03xvxNv4cruBZ1pH8PPV+/cQ0MXjcTGYl6jn7Nql/L1xYnEiPGB0oJ13lg+BRj8HVm+kJpxmQybMnHg9AeVwTNXafKuuq3h2/jheWTSSEXVtnHPAfKojWh0TUE7ZGJMusuKwKt7ErYuf5eopZ/JO6wp+v+gpUm6W1U6Aby49mKDh49Gjv0N1TnAMCFaxLtnK7M56RgRieZUW6NKi9x52NYa5+SR5yjcVsSZD5g0k8ybiOxDMoVpYkNXvS/lRkc9vtq2+OzLKbJz6VguKvQppvZi8WsteiLScDw0voIyaPu/12DH0h2AYCqwu+H0NcFBv14iIrZRqB+qApsKLlFIXARcBjBgxgq1hU6qjxDXTRfHNZYfzp3EvcsXyw/TEp0AFJD8394YIOK9UYx7eTq+7aFewpndiHxTtkXBfGO6LcfPY11ibiXLNygPzQUsgKFWselKAOTeNGKpIKaVSgu+VWF4wALTYQX70yMdoXlINTm7ify+iS3aO6dYX5w3McYfwjRuwZmk9fvbgCMkrBkEwd68IZk7tkh99yKCnS5FSQqTS7Xps5nXWcf3qaUXpn7s8nH484i1GBWN8+bmPEy/II5XK+nhw5hS+cdwbZb19woZNhZmmwwlSdnKP1/H4lEeptroFynHVq3k3Xst/WsZQayY4unodFWaW/3UMYWmqKn9dvS/Jpwcs1oLBFcKWjS2Kbw98h3N/fy7tySDHT1nMjdNew81Jzg12mKWpSo6rWZv/nL774HFMXzKSVNaH37R5YMaefOszLxE2slSbadZl+za2ZsVhSdtM3I73CLTPYFq0kv91DMmfN5Ti9aZFnDxkXwAu3+NUvj/nPv6ycRJHVK0naurocAX8bu2+fGdw+Sjrnogbp33t+dz2bB2vLR7K0NrpXHr66Uwe0gLpV8AapXM7WaO2qL1eCX0SEvfTbbsIQOjsvncLyUcpdSF3tcNIuF8VCx5byC5lfBaR24HbQafd3pp796kZWXYL74jizxsm9ab+7dE/4IDb6kMJqEB59Y9qzmJssvE924H/pRiZY5OkvlyvQ4INwIXVrWHOf/5gkkdWUzjBqXz2z4J+gWylhb/HIMUEqS6eaGzb4PVFI6EwMZ5jIEvCUCAYuuokhH6zCevtBF3Bq763EvDHTSQv115BSik6XT8LkrX6vrQik6onFGzCSKfokhf+gHDWhTpHT8Y1+e3PR5A5W0GR0FQMC8Q4snodCrDLTJKOazC3vQYr47DXoLaicwnXIub46RIwPdV0LornW4dzzoDl+WMbs2GyYjDc387f9ngeE51J9guDFnDL2r14qHksfuVwyaB3dVoNG4w3Kzj2oPc4f8953PbUwZy133ucOHUJQ6pjGAVybiidVPq63+nyxmpeWzyKdE7YZRwLN614cuZE6iakWJ3Z/CQ9MdTKH8e8CAmXCQG4cTS80TGQy5YfkX+HIbM7kPHw6DvcMf4lfrpyKp9ecAKHVW2g1kozIzaEAwceW2qvcFzueXYmj7z2Lj7L5PPH788Zh0zBjd/NRXdOYmVzFVnHZG1bBRf9sYN7rvo044Z9b7Pj3lJUxVUIJiT/BbgQ+gSq4qrN3FXmeyb5/3jsBPpDMKwFhhf8Pix3rNw1a5RSFlCFNkL3G9X+CCcP3ofH1hXn0XFEkXUNlJLiv7NclLEqWBxLs4U7N4zv3RjB+RuhziG530DwFcwWrmDOSRL+1UZw9ZQfeLwdd6iPzKlVYBraYFBjkZpSgbEwjTMxiLskiKwJgqNQA9MYE5Mof/eAnCkh3HoLY30WZefmfZ8ic0Z18YP2VvTLBek6J+gxiOB7o5PCjAYqK/he7SR5eWkTIuC8Wg0Jk/bTpxKevoLA2jZGjIpzyTVr2Di6mqvnH83GbBi1V6qsZXhVqoJPvnE88bRFeoANa60CISb4a9Nc9pOpTFq8npvv78C0BNMUHNPg56v3y8cqBMiSzDkFFIyeWzfsyb4VzYwNdeTrVthi8KMRb+FTbl4Pbynhm0PnknENTq9bwcp0BX9YvSfushCZOpf/GiN4YtFQqkdnOHfMO4RUlrRtEfJ3Ly78PmFDOkpb0mFsqIOVzVWoHunAbUMx063HzGw+I69C8Z1hs/P2pq7Xd3DlRvaNbGROfBBB08ehDRP028rMgdgNTAiluGviBv68YRIPN49DVDVnDDuAr4w/vqSPmx54iUdfn08qq5/jF/e/SDrrML7yHda2jSXrdAvrjG3y9+em86MLxpa0s60o5UNVXg2VV2/5PaHTkfidFO0aFBAofT6PD4b+EAwzgPFKqdFoAXAu8Jke1zwKnA+8DpwDvNCf9oUuLplwAs9umFu0c/Apl0M2rWRupL7YzcUFd24UNSgDjsJdHsI3t5OqZ2ejsrkVzGpIb6jFHerrFg4ZIfhwG8otnp+NpWlCN20ke0RUq36UQsIG1rwEmUwdrAxBLrGcLDNxVoQwD28jmO4k4/fjVpnEbxpO4N5mrFkJnLoAyU8MQIaYyDof0m6hqmzMpSlo6cCpqiBfEswQGJaErIIuYdM1nxo5g0AhhVbMAo2NNPkgbYAoJOQnfuwEUsphjwPmEzlgA1csPigXRwEyIZgTQj0MFg503J8m8MhGQuenyJxcCwbI2gDuyiDZ8R1U/rKd5dkQXzh0D446sw1/UHhpw2CWXjgs30wSP+UkYNz1cdfGPbh25Fv8Ys1++boVk8KtJXWmTeUyJtjO198/glQqgLs8CAkTY98YYmpX5SaCXL7icIbOUNz6uceL7rddWJaq5NGmUfx6/GuMGNxKyi7YBQUczKNbu995uZdK98dxcP04JkYepxwn1m6kMnI0l+1xGn4j945T/6XLkGsquGjwAi4asgJV8UNU+KSi+11xWdy+gYffnkO2QEuXytj89ekZfPfMUQVR2l33GLTGd77SQFmjoeZ3SPuPwF0P5ihU1fUos/eMxR47lu3+q8jZDC4FnkavU/8iIvOVUtcCM0XkUeDPwN+UUkuAFrTw6HcagpX8fJ/PcO28f5Nw0vgUXDZ0AQNmJ1FXrsP82kCcCUEkZeHMi6BMcBeHoM2HyjhUPLMIZRdvayNXrSH9hTqy0yIYm2yCdzdhLtVf1sKvvv/FGMoF34w4mXlJUpcMABNcy4LlpbWTcRXyaIDAw4sIOJDZv4LkdweQvHgA0mmgIi7m+yn4h419ooUxPomywB1govbKYDyRxlrVQeStVahkFnt8kOQVA5ER/qJ+MidV4n+6A5XJeS4FFOlTKyHrYhoOjllgVc+W7gAcMWmJh3ikR+K5ruROXTv+vGwwwK01yRxTQfbUKlQgNxmNTGGEHdRK0X91WWhv9vHoX7QXjDPMgAt79l5OL614N17L0mRFLqBO826iliGBeJFwyIrJbRv2JLE4iiyMAgrjkLa8q25Xe2nX5P1MFU+/O47jpiwl7LfJ2AYZx+SuTZNY1x7iupX783psCGpKHJkfBSUYE+PgL3SR7ULybRce6cimMK1xYM8tfiIFZ438FMH2qVw770EiVpDPjj6cvX1R9FeqUEVqgCqOCVgS28C3Zt5FRzaJe3QWo9GHO6MqvxBp60zw5xfDOG6xN1rI73LKwfuVeccfPCpwNGrAy4i4qNIX6vEB85Es7emIS2smTrUvjKkUuBv4zSUP8/Rdr2LbDvaeQRLfGwJhA4mZONNr8C1uJfrSYoxsqb5zM3bq0ustyE4Jkxw+EntwVc6NskwLjkvdnW/oe0xFdlAFZjyNVCqccT78r3XS/uUpcEK6aDITG4z/OFTcvSIvyERpe0Ts7tE61WgXthC4txn/0x2gIHNqFelza/US1Bb9/64dQ0bhPFOXT0IHEPRluf4Tz/JKsJ7HWkeVfQ5xtRpOVdioIJDObaeCxV9wccB3U4Lwa2spzNIhhiJ9dDXpK+vZ3NsWgYDKsvfSNbw1ZnReIA3yxblr4vP4lYNPuTgY/HbVnjx2dwPW7Di2v4LU2KGYRyVhYKZolxPAxpleRboxxHFTlnLMHstY11bBv5ZNpHkvB3NhUu8wJ+t6CZI0kGYfxsAM+MrtFvQzqHYH1eLgDveDpajxR3jyyM9A08kU69Wj/L7tRzy4+m1Sjl7uBwwfN+x1Aoeor4J0BacZoKpRA15CqWDufQhnvnwjG1JtRe9ZloZwF0SxDANBcFyhIpgiY1s5G5fFOUfuwxXnHNWnYbgntutgKmOr7vH4YPFKe/aCqQzqAwWGQHMIl93+NU6/5BQe+v2TvPzvN0j/2ia7ZzVGawYaDCTs65mJogixDLB7i1rogQ3pmiHYA6votmaWTnhGR7deWjmCb22HvqINzLUZxDCQ2tJpUlkQmN2kdR1dxwRIu5jvJnH2CSMuuAvCyOoQ9uA6Ej9NYowtDTiznm/H90KM5HVDUX4wDujAnVWh5y1RDN2jCf+gJMexisdaRpWds5UBVDo4M6uwDmuHgA42K7nOcQnO36DtIQB+XWBIgn4Shw3FkHRufKXvKm8H6jTIJP1EBoAqUGOtz0T4+Lsnc6yxnurqJP+LD2LjZTGCC5owsi4+K05w7iZip4yEomJJwgB/klgqShrh2fnjeHb+OHyWjbNPHJVx8N3XiXVclF9O+xJPrJvN/PY17D1+JIYy+M+aGThF4eQKXAj+fhP+52NgaaGf/t4Q9j9xKoY1GrfhRWj/EdiLwX8g6fClPDD3DjIFKtC0m+XWpW9z6IF/Qzpu0Kkt/PuhKr6bFwoAa5MttGaKq6opE9TQDOZik5DfoiOhd7ixlL7Pb5n8+YpPMWXUILaUeW2r+MncB1mdaKLGH+XKSadz3OA9t/j+nriuy+K3l+PzW4zeszToz2Pn8pEUDL0xbt/RXPXnr3LVn7+K67ok0lm+cMM/WLqxFXtgBU5lELMtiXKlW7ttGXQePho3GqDiiffYTJxansyYOrCKK4UVuYDaLtH/LSu6p0jZ5Li4fh++WTHsacWpCMRG2wJ6UNi8804U1ga77RoLw9qldVxBagRbsBaloc7Sq/yQiTEwgzqpGeImrg0rayy+tfwIvlD/Hn0hCROafYidi6q2Rbvfdv2FOYKxOoPZ1G0JdwIW8cPHkR1aharvYHyojRHBGG90DNL5itzcS3FANvpxF0QgacLkOHPMerBcJGjosS4N07Hex8PZQaAUVt16Khc2orK5ADkbSLhEv7oKZ2KQ9OfqcBss8Ctu2uNV1Pk+fvjwsby3dgCRYIYLj5jBnffVYqsQycsHUpmxMA2Da/Y6Jz/+FzbM46HVb5a8C/OdBGSFzt+PQKIGvv91EvzZOqaEw6QmpgmGB0Pt7fnrm+NN2G5pYsemdIx4fCyp9O+pH1Jb9r2HVIDMCh/OuhBEHC38wy4TBgzkdz/7It/44yO8t3Jj8fhMA5+15TmMYtkkl874C0lHq6JaMp38ZN6DjIjUMaFyyGbuLmXN++u46rhr6WyLIyIMGj2QG5+/huqGqs3f7PGBsFsJhkIMwyAaCvC7b36CT/zoLlKZLB1nTCU0cxX+FS2QyhI/ZjzZEbV6ghchuf9wwm+vQZUrSJwjr122XcTX48vnCv4FGzCTNv7FjZixvmtC2DUhzOUp7DfCuAfb+Xgv2WChmnuUtwQwTLIDKpEWgTXBMi6tIegSDLnKM75XYkjYoOLiVahmG3ekn8RlA3FHBVHzgkjYQVlwz7+GwdEF/eUEgOTqy7jzc+VGFZBxYX0WwwV3hHa9NJekifxsffFnkLDJjqgB2+VjS9/lxx9/DwH+smES/2ycgOMoRClkXQB3dkWucRdVk6HVDCEPBVFnp7DfrIXO4j9lY7pPu+wWxoXYYDQ7mNPj+KfH9esxYMCiFKG6BHdf+BC2qzCV4Drw90/thWO1Ezs5QGyQy9dm/IX6QJSUk+XogVNYEttQ9nNz9gzhTA3pNB1A5sRKnKE+7r3qPp78zTPUD6tj+dyVjJg0jEt+cz7X28/g9jC0m7ai7g8tfOq5C0EpBo8ewE8fvZohY4tX+dfd9QLuexG9WFCCsypI6GOdfHX/46mtDPPxw/dk2fpmUhm9G1EKqiNBxg+tZ0v5X+PCko1i1rV5fN3sbRIM13361zStbcmXB129cC23XPpnfnh/GVc5j53Cbm/lGVxbyT9/8DmmDR1IyHYhGkBlHExbyA6p6l71K0V64gCS+wzFCVo4PgMnUhx5IIAb0o79wbnrIFsweWcdAvM3UPHaCsJvr8GKpftUS8WOHEPnKZNJHDYap7EaebACZ2YY57VqQj/fiG9jrCg1nD0gSsfHJuL/SwrfbenyFUqzILZAysWcmyR65RqMmIu50cZo0oLHXJEhevUajKUpqn7+Hr7/xjBnxgk+1oGxKAU5G4y7yY+7NoCsDOK8UgOtPtTQFMpET4gj/MgIP8p2Uc/ahH+4CaOpOM7Ergnr97K4EfVOHKdT+Nm1Y7l3zQQdOOfTOw41JI0anNZR3sNTqFoHwhbheSvw/a8TOotVQ2pYCueMnhmsuhVUYoBbqe9RDrz63yrSKf02LUOwMzDjhUpcVyEmWAu0MLVdmw2pdtqyCR5f+zaLY8WCLo/PyAsF0O/DmRoiUwEbVzQy/38LSXQkWfjmYr51+Y2sjjeVNBF+L0vbC+vIpm2yqSyrF67j+6f+HMdxyKS1HWLZ+mbeXLASp+u1igLH4IC2PTm0YSIAZx82lU8dtTcBn4llGowbUs+t3/zEVqlujDLThEJhbsP0kexMsmL+qqKa0Y7tMPOpd7a6LY8dx267Yyhk5MAabr/mswDc/8tHuGfOv8jgUPHCYmIn7JF375SAhYqlMVP6myh2BtdvgqEwsg6ZoVXEjxhD5JWlBN9Zi7IdUnsOAaUILNhA6J2e4R2lCGA3RMmMH9DtjgpIMACbFLhCYEVLkT0kduIeiM+k6skF+XQbHdVh7KGV5BVUjkvgvUZCf18LWRcjme1dMGWE6JWrUS6E796EcvR61v+7OPGzG5CgiRqV0rmXlmo3XDU6iTGpQNfdlUrVVHAkdC4eSeWzSwqMBZAdUkX0hcUEVjYz/qsJfnTBaN4eWKr3VhYYeyRgYgJVlRO2tgvVBqFbG0l/fChupc6BpMYlMSbGUZYi9YU6gn9t1iotR3eaPj5K6gsN4FeohEvo1xv5w/eHEqlwOPC4GABzX49w4zd1aI4ScIf4u58pR1YclAuWcrDZArWMC27QAMPFKLC/ZGsU2axDSRMdDplkt9+piLB++UbOrP48Cb9J9tQppCoCuD1tOQLJ9gLbk1J86+NH8tXTDyWdtakI9573qTcOH7AHxnsFQiDt4n8lxuzATH46biOXnH0W9TXVW9SWP+jH5/fh2MW75aqGLYvg9vhg8ARDD8689GRe+fcbrF6wllBrisBj8xnxf4fQ3Bwj8e463Pcb89cqASPjkJw8kOQBI/Ev3YT4LbIja/Gvbif07gZC75ZXN5RD0IbK2AkTe9gnKI4VUMX2CnNdO8HlzQWutkLFc4uIHT8Re6D+wvmXNxOZvgJcbfnNDq3Cv7Yd1wDl9rRvFA5KcH0mmZE1xI8cCwkTEui4ivEJrBNbtHeTI0WCrAgDqLZoPXdfQq+vwL+mHSNtE5q7TnehFA/9fQhZMVBOtmwdJaMlQejOJnCEzKlV2NPCWPOSqLQQfWUZHSfuARYYExJ5u0bmrBqyh1dgzkuRXVaLeWQH7rhA/l2K3yDxvcGYF2a49sLRhCIOSkG809Q7i4DC3iOIW29Cyi31skq57FvTyIxUVwGEgj2cFHxmrmC02BjrbB1oWfiu58SRnl4PaRd7VluPsj7gZB2ypqLt4/tpz7MyBn6fYXDUnmNKjvt9Fn7ftn3dI1aA2w+6iOvffZhFrWsJ/rGR2FfqmC9x5quFPP7iL/jLEZcwuWHzaWxMy+RTV53JAzf+h1RcC4dA2M8FP90hHuwe28hH0l11exER5r7yHo2rm9n76Ck0DKvLH//hGb/gzcdn6d/R3kqJ/YaRnjII/LkkdCtbqHyqVC+72X4VtJ+9J85mVk+RF94nsLhJT16mouOUyVT+d35JfwK4fhO3KkTiwBE4dRGspk5Cb64kdtIkav4+i86jxhL933KUU5ArSZHfkQiAgrbz9sOt6LHaNF3MYxvxv9CBsS5L+uw6iAJ+VVzTs9PB95M4qYOHU3n/XMzOTNl34/pM7Ao/8a+Mw23XNgs1PIkxrpPKC5djtOVccwOK7IQIqWHDEcsg+N5GzI0xknsPwP1egJ6uV+KCrPajhmUoqRuTdgn+uYnAY+3555UKA7fWJHtIFP+LMYwWBxSkT68i/cV63b4rqE02P+h4lejxPq5eeXhpOvWu71bMJXr5alwniNUUL3rXBAzOfOlz3NvyOm7CBkthvdJJ8LZNGGnJOzsopRAR4vsMIXXQSErcw0TAdvE3xfnPH7/OwF6M1YXYWZs5L83HdVz2PmYq/sDmU2tPf3QGVzQ9gDOk4FpXGGZX89AZ39ns/Xqowov//B//ve0Z/CE/51x2GgectO8W3euxeTx31R2EUoq9j5pS9vh1/72a92ct5ZUH3+DZVWtZUmEhSkHa1uoNy8AeXkN6XB2BJTrrR18Cokssi88kftTYzQoFgPiBIwks1nrp5ORB2PUR3IgfM54pudYeUknnxyboHYhSZIdWkz2jAgScygCZ8Q3EHZfI6yvAESRoEZ82nIr/Le8eu4AbLDNp2Irod9ZgLkkjAZN45XhUxME8LqajsXMr59DvNuFbEMe/IYaRskl/ogZjfQbf9HhJ1qjM5EG4G8P5HZOsCGG9EMsLBQCVFqyFCbKHVINSdDZECb2zluhrq4nPGYqzZ6hb/eeCbPShhmRLhQJoW0yu7JoAhBXxm4cT+epKgv9qBaf78zNfyUKlhXuYQnWkUXHFT/c5RgftlYu/6BJQfkV6rwaSQ4bjW99B9KUlqFQWoib7/+QILj3oNJ45/kk6OjqRgEH6C3V0PjAW3/MdNDyaIpzxYZgGG1c24lb5c4aSHs+hXKr+8y7RtMOcZ+dywvlHl3nYbtYt3cC3Dv8h6aRetVs+i5te/DGjp/a96l+1eB3OpB7ThqFYZ3WUv6EMSik+9pkj+Nhnjtj8xR47BU8wbAMT9h/LhP3H8rH1zXzxpvuxHRffv94hPb6B7LAqzOYERmMnHUeNJfzuesyWRH7K6JmyyY34Sew5mMzkQVDGi0mXiyj0LnKpfKZ7N5IdWQN+i/iRY6l4dlGReqHjpD2wh+QC7LraMBRYBmZLgo6P762N6hMGYA+q1LuOjEN4XqlR1bemTXsQdamLdF1P4lPGE25dRmZULQQsxPFhP+/XxmLDIfrXZfgX6xrJ6dF1GKPjZM6rIfSnxpIJTmVdrI0x0pMK7AxikB7WQJQerr1ZVz+rqcBnktxnKKHZawjfvJH4z4biDM5NoAkTd3YU8+i2Hgn/cgt6w4CNCrfGxBkXIHNUBeFr12H0yMcYP2AEqb0GAwpeB2Wm4cQOlKljMXJ1S8vjM0h9bAAsNcmOqKH1M/sS3auNAXvV8v1DvwiAtTyDtTFFxz2jIaTfcfaEKhpPqOXOQ75Cx2sb+emnbiZFjExkQM7gnutQCSqawmqOY1QECVeGSoZguw6vNi5kYftaJlQO5rGL/kVbYztS8Pfyi8/dwm2zb+zlITR7HTYJtWJ6SXLHBhXt5Q6PDyOeYNgOxgyu48nrv8zLc5dy8+9exnpzJRS4tftfXoq/KoTbUIHqTJNN6BV94fyR3HcomYkDKUn0I0Jw9mqtHljfgVgmqb2H4vgNrMZ4fn1qtiaxB1WSHVFD2yf3IbBgI4FFG/VkP7iyF72/wqkIQiD38RsmTnWIxEEjib6yFKOMG230lWV0nD5Z35fbfaDAHlRJxxlTiT65APbP5VK0DWR1CFyX9IjB+Bcvxq4Nk9p7CNZxzRBUpE+twvdSrGi1roDAsmbwmcSPKEjs1vPVAGIaGKksbiSXidQysAdU4NsYw399B51njdGCNjeBugtDGHt12x/EBlnjx10WofPweoyGVqp/shDfjETJszuVQS0UCnz/xQggm8KokbkgxT68fAKWj7BZSSM2DVURTj5lDAdPHskexkD+c/OT/Ouvz5NY0Ub2yIriYBTAFofv/fE29plVxccvO40nn3iVFfu147xVnUthoiBiE8huxDANgtEQB55SnObCEZdLZ/yFBR1rSToZQqaf7GmdhF8uVn4tm7sCx3ZQpuLNpiUs69zElOph7F09Mu/FNPngCRz+3GBe3XODtnMImCi+f9A5eHx08ATDdhIK+DjpgD14cPIwVr63pujciD2G8qmrziQVT3HHd+4te394+gpU1iE1ZXC3wdkRwq8uJfR+Y5GCwrcxRudho4r7n7OOzPgGbU+oDGJt1OqazNj6oomsCNctFRiG0tHdAK7gBE2MjF6VK0BlHQKz1uAMiJKeMrg47YZSZMY3aAN04XHDwKnWeX2yQ3QxGQkYKAR3XJDEDwYTvKMJozGrBYSrI8ADizYRP3S0HqMIZrP2duqaMp0K7VIcfe597ZFlKLBMknsPwffMIpTtUHX3fIy0Q2ZMLfGDRyGrI7iugTE+qYOTVwSR9/0Y6Qyyv0P0zhUlr6mrP3tgtNTQ6xpIow/fik0E7mrGaHOw9wmRvHQgUtf9tTIxGBmu465vfhUDA8PQtoL//P5JfnzlD3FyLs0KcskOeyjXHKFxWRP/e2ghgZCfax68kocGLeDlyvewO3SOdyNo0/CjDAeeczAX3fj5ElvB643v54UCQNLJoEYHsA+O4Jve7UlWURvFweWil//EsnQjDi6mYXD0gCn8ZK9P5oXDzT/4Fi/PfodHlr1JbUMVn9/vWEZGG/D46LDbxzH0F1f8+auEokGCkQDBaJBQRZCr7vk6J15wDJ1tCbLp0loRAihXiLy5iup/zCL8xHuEZq3GWtuKSmdLtNbKdgnPXvv/7Z13eBTV2sB/Z2a2ZZNNpYQaqvQmSJGiAorS7Hrtvbd7LViu1654bdfeC3asWFFUFLEhRXrvEEIKqZtsn/P9MctmN7sJCaEEv/k9D09mZ6e8cxLOO+ethNIdkfNVt4/UDxdjW7kTAiEseeVGPkKpx1AAMTeU4A9iW1EjUiqk4/p8OclzNkR2BVqnUT6hJ4F2aQSaJVM1uB1Vx3Qh2Cw5QY0O436O+VvBG0Cp9BkTaVDHsq0EwEjmEwK51W4kYwHBw51UPNueyjtbxWaU67J6IhYCy45ywyEeHg+10m+MQVAn9YO/sGwtRVT5CaY5CGkKlh3laGVeFG8A28p8XF+sAF0icx2Efsog9GMGclMSSLDv3EHS4ly0ld64x/L0NpK31BJP/IpA6mi5JTj+V4CaHzT8HvOrcE7ZDlIitvmwvrcLxwM7KDrhD165+e3IJabdPZ0XbnozohR2Y/mzEhGUsUooKLHOMhzjPo+fqec+xcTczjhfLMSyoQzbr8UkX7MVa0GIKW9eGwmUiGZ9xU58odjGSNImoGtSRCZbko3Tbp7EmedOYWXJdrwyQECG8IYC/JC/jI+3zovJPRjeuze3DDiRmweeZCqFvyHmimEf0X1wF97e/By/fbYAIWDY5EGkpBt216zWGVjtlkh43m6kpkRCTBVvEEu51wg9hVpLbyjlvohpPpRqR/GHCGU5jWQ8XTd8CLokadF2/O3Tw7kEipFsF9JBUfB3axFOIpagCKwbi2KiZQRg21KCt08rKo7vEXP/UJYzztyBBNumYhS3D8eyHcbqw6JSNahdJHfDstVQEPpqJ1qxB/svO1HzAvh7ubD+XFZ9qbB5KtrfYl+ZH5O3IXSJpcBNxXHdSJm1BtvGXQQ6Z0FIN2pelVePswC0QjeOXzfgGdIh5rpCgra4Al+gBTa1JK48ebCVC1bkoRVVGj6WNmnG+YEQii+A86ftCF9UZrUOSnEQ9acKnI9WyxwCvnhhFt0Hd2H4yYP56LEv4pQCGE51583b8FzbAqVnMvo2D/bnC1C3VU/qFcVuXp3yLmJVMc4vjX2BlilsHdWBUTc+T/+ubZhyxtG0bZYWOadbamtsqiWyYgBwaFYuPf9ktgVWEgrpTLhsLC/f+jaF/b3giI0+C+ghnlj9FZ/nLuDFIy5j6bfLePi8p/H7giAl5997BqfdOKn6dyglfq8fm8OGyaGJqRj2Ia6MFMZdeHTc/pGnDeWNO98n6A8SDITQLCrJ6U5GPnUen17zOkqREZ2juv0EWrrAqiHKPahlPgjFFu6LRMprCp5OmViLqvB3yMS6tQT7ynyQEqkK1FIPaR8twdM7m1BGEtZNxVQNyQFNqfb36kYxIsvOirhy40iJVugm1LxGlJSm4py1mqpRnZBWDeELkjxnA6o7PBlLICQRoSDOuRtjnO5qqQepKrhe2WA8lwR1rR/dpqKHlWSwWTIVY7rG3DKY6UTxBuJkDKY7DD9HVpKh9FQFPdkG5bEKWACOlQXIJBvevq0jg6gWuPF3bIavSwuEV8f559aY81J+WIu/lQvr9jJSvluDPyeTQGsXSokH+9pCRILJHa/E+Vh+XNSQ3+Pnu7fmsNnjwVcVHz22G2VbgNQXvIy4cDA//efj+OgjYPuaHZHtUKqd8hN6gEUl6Pay9KlZXHjjdDKzXJx1+0lMumocR2R2YkBGBxYVb8KvB7EqGn3S2nHmwGNQR40BjIzk9Ys2oqQ4waNHHOC7CcgQm9wFvL7oO74+/V18nupnmHbXB3Qd2Im+o3ry7bQfefHGN6ksqyK7Ywtufetauh3RpdbnNWmamHkMB4iSgjKmP/wpK35bS89hXTljykmkN0/lh19W8PL1r1GybDvOVCfn/udUTrzmeApK3az9fTX3TPhvjGLY/dsqP7YrwYwkFE+QtC9XxNRv0gGhKUjVcA4qfiMru2JCT5SSKmyrC1C8AbydMgm1TsO2rpCkXzehRE28UlMoP6GH4cCO7DTMHBmv/mEsGqwqwh+qNRgnUjcq/NPfJhW1qBLFG4x9Jk2h7PjuSIuCnpUca7aR4eQ5JM65G7GvLUQqRonyqsHtEQHdSOLb7R8JhnB9thxLUWzFUQn426fjHt3VWFWpUa37hMC6Kp/knzfEyqUIKge1xbEsD8UbjGSVCykJpDlQ3D6UBBV3EwSuGj6gZk6E248IZzTX/L1GgrSsKqWn9EEprsL17ZqE46tqCqGgTuWQ9nh7Z4Oi4PpsGVqBGxE2RdmSbFz//KWMPXcUutRZsGsjq8t30CWlJe3dqWxblUvn/h3IzE4nGAgyOfU8fKEg7ifaoGdbDeVQI7Gv0+8W3E9sxOuOqgwsYPxlYxl7/lHcMuaeGMWX5HLw3rYXSUqJj5SKpjLo47Pt81lVlku/9BwmtB6ATd1zXoVJPPsij8FUDE2cy0f+m/V/rDNKTIeqo0hCTiulp/Uj9ZMlBFumYN1YbJiCok0u4Z9SQOWgtkiXg+TZ62Js2KEUK2Wn9CP1s+UoFV6UoI7UFALZLiqO7149SUdNour2UrTiKuyr8o1qtLXILgUEM5woZR4UXUaq1tY8XreouEd3IdA+g7iOcNEEddKmLyKY6aTyyI44f9+E++gusWG+UmLZuAvX92sTyCNwj+5iOOZrEtJJ/nY1tm2l1ccD3u4tqBrSHuu2ErQd5djWFaIEdHSbRqBFMtatpfFKQBGRyTn6WkQ9e83PMceqClVD2uPtlU3q+4tQy+L9H4qqoId0Ai2SKZ/UC6UqQNr7i+IKPHbqmxMTgiql5OlrXuHb139Es1oI+AKcc+cpnHX7Kbx0y5t88dwsPH4/gZHJhDrZCOVYCfU3kg1VFIaubcH6O//EU1GtGDSLxqk3TqSiuIKvX/4hxhfhSLFz48tXMur0YQme1MAb8nPWr09R6K3ApwewKxZykpvx+tCrUM2mPQ1mXygGc9SbOI9/eTt9hh6G4rQh7FHRLpV+bGsK0FMd2NbtMt5cayiFyOQjwb6+yDDt7I4yCv9TK/w4FmylbHIvKkd0xNO3FRXHdKFiXKxSsC7NxbJypxGx1DoVX+9syk7rh6dXS6MonRZbsk4C0m6hfHx3ys4+HE+PFrFvxVEIXRp+Bagz7BNFUD6+B4FsF6mfL0f4QvH+DiGQSdZIu5yYr6Qkad6WxNdWFTy7w22pfoN3rMon4835JP+0AcfKfJSAjhSGXd+SV0EgJ4Oq7s3xdszE06cV/lauxL0oEnyW0Wa9aKQeKcBYObxjQu2hh/1BSeV+kjeVhO8Zf2AwEBv0sPC7pXz35hz83gBV5VUEfAHefeATtqzcxiVTz+HSR86lU4929Ha3oF/XLlgHpgNgESoOzcr1555BUooDJSqqTbOqnHDpaCxWS9yvTyDQrHVbrGflLaUorBQAvHqArZVF/FYYr9xNDgymj6GJ43Ql8cSceynYVoTfG2DLym28MuVt8jYV4Px9M8GWKYTS7ajFRgVQAeiqUa462jSkFXsSTkICcCzfiVAE3i7N8HfIrM5TACNctKCC5D+2UnLOwNhSFwI8wzrg7d8GpcqPUu4l+bu1oCkEWrmoGt4JHFakLvEMycG+YZdhRhHGW7VUFISUuI/ujLTV/aeo5ZaR8v1aRNBQBr6Omfg6ZcUnBQZCWNcX4umbjWNJfKKe4gnE7duN7rITTHOglnuNSTuko4Rk+C3cWO1IxVA8Vf1bw5AcZDCE6g8Z0Vphx7/wBUn94C/UBN0A4wa/li9sawpIWrCNULqDQEsXlkK3Ucq9xmkBj5+01QVUtulCyGVDLalewdmSrIy/fGzMlRfMWhwXBOHz+Png0c+5+bWrmXTlcUy68jhjPKTODzuXMyd/Ja2S0jm13RCa21N5+o+HeHnKWyyds4p23Vtx6cPnkt2hBSdcOpqZr/0QMSUJIdBsGoPG9atzGDZXFuLVY38vAT3Etqr4qrMmBwZTMRwiNG9rmD/adMnmyMlH8OlTX/HKre8gdlbEHauEEr0v12K2APzt07CtzMe+1JhI3SM64u/aDBBYNxeT/NN6I9IoKYHNN/yGLoqrSJm9LnxRiXVHOd4qP3qKEZlidG6r7hFdPq4bCEGwueEfUAvdqG4fgZYpSIc1ch0wcihc36yKcT7b1hVh2VhM+Ym9DL9E+HhtWwm2VfkEmzmpmZAsAV1VEmeUS4lt1U4CrVMpO70ftuV5OBOsLkLpDpSAjnV7Gd6+rXAs3IZnQNtqBaUaJjT3sd1wfbvacIrvHvyoX4uEOGd6dVCARCszTDXariqkIgglW0ERqOW+mBWJoip4tpeQPu3PuP4fQycNYvLV42L2tcxpjs1hjXEeA8x+9xcmXTWOwwZWJxYqQmFsdh/GZveJObZZm0xuf+eGuLHp0Ls9d318M8/f8DoFW4voNrgLN7xwGVZ7zXKAsQxI78DHW+fFRE2pQqFfek6d55nsP0zFcIhy3IXHMOPpmezcXBgxK9SXmiadyqEdUII6qTOWIYI6yXM3wlyjBEW0MlHKveipiZ2IyXM2VNu3I87iDZSd2g/nLxuwrS2MsX8nz9lI2cl9IBgiZeYqLIVupBAIXafyyI74ujU38iC2Fhs1oGqYjAQgU2xx8oRSHQRbuQi0TMXbuxXJP2+EQKjadBYOsaxuuVqNY1EuAolnQBuELxgXvgrGRC3AcMprKtKixa9ahECp8OLrkGHUz7Ko2LeUIAOhGPs7NapoRPuEYuumSLRyn5HLoQgsdgsBbwChCBRVMa6pS4S/OkpKKIIzbp6MUuM5x547kmn/mR6nGIKBIN++PjtGMewNg47rx6BVTzbonGHNunJks8P4pWA1Qgh0KZnUZiA9Uts0ShaTvcdUDIcoSSkOXlj8KD+8/TO/fPInK35fE4kUyWqdwWWPnsujFz2PPzwBqBbFCCoK6kirYnRZI5wkV+YlmO3Cl5Nh9HoI6oQsSnU2btgvkfzTeiM0Eoyont1mpZCO4o4vo6GWeBDeALY1hXHOWNXtI+3tBejJRvE/EeVYd/66EX/bVGxrC2vtmCeBYMuU2ElZCGRGEhXje0Ym/5L2GVh2lJE8ex1KeOJU3L44hSK8QaPvhEVB3VVF0l+5CaPBpE1DqgKt0E2wRYrx3P4QWGOVg79r84iT27FoO9IXrN1ytAeiQ36FLnFlJuP3BnCXVKKqCkF/fPKkKzOFTv1yIp8rvX6WbsyjeZqTyx89jyevfIlgVLitIkRc854dG3by+KUvsPL3NWRkp3P5I+cx4pQhe/kUtaMIhQf7/YPVZblscOfTPbU1HZNb7PnERiClpHhnKclpSWa+RQJMxXAI43DamXD5sfQe0Z2rj7gtsr+0sJx37v+ELyreoqyoguS0JCxWC0W5u3h5ytvM+n4xFUd1xra+CMUTQGqKkTktQNcUlKCOGtCp6teKYFYySfO3olT60XZWkPbuQvw5GQTapBlF9RRAURJWd9UdFiPLOoEzFjAilcrjO9lJRaDtqiLQOhV/qRfrusKEk6q3V3Zk27qhiKQ/NqNUBQi0SKFyVGf0VDtYVAKt0/D2aGkk2wmjDIl7TNdwfSoBIZ2k3zaFV1EC4Q/Ey6wqVIzuYjyzBLXMA/4g/nZpgKyOpgqGkwx3h8QCnsPbYl+VH/NGXzeJXPThsZGwK7ck8rnmmz+Axarx6Oy7IxP9T0vWc/trM1EVhZCu06t9S2wuB8Fd7upzbBaOv2R05HMwEOSG4XdGCu3lby7k4fOfJrNVOj2GHlbP52gY3VJb0y219X65djSr5q3jvtMfp7SwDEUITrt5MufddVqcYvz/jKkY/gZ88uTXBHzVzrugP0jBlkJW/r6WXkd2i+zPap3JbW9fT8WEh5gdCODp19p449aNiq2KL3biCrRwEczJoKxDJpatJWhFlQTTHTjnbcGxugCpKeh2jWCGE1+XLByLd8RM4GqlH+eCbbVMcQa7p8AYk4oOeoqNUIYTd4YTS8dMw15f4zxRFYA0Ha3QbZSyDtvsLXnlpH62jJKzB6AVVKLllyN8QbyHNUd32VE8fpw/rcfXtRlKUMe2cqdRmFBTcB/TBbW4ZjlwqDyiHYG2aaAqKOVerGsLUSr9VI7qFOus360UopGSQLYL25YS6iIyDgJ0m4YIdwqsz3Slagrtu7dl8IQBTL7meDKzjWgijy/A7a/NjPR8Bli2eSdnPH4265/5gfV/bSSrTSZXPXEhnft1iByz+McV+Dy+mOqrfo+fL16YtU8UQ/HOEnwePy1zmh/QCdnn8XHbuPupLKsulvjRY5/TqW97hp80+IDJ0dRplGIQQmQA04EcYDNwupQy7q9fCPENMAT4RUo5oTH3NImntKAszs8gFEFFsTvh8ffNuIWh037i/TmL2e4PEFhfYIR+1sCxIo+KnAxQBIGcDAI5hs1c/LrJuEdQR3X7Ud1+2Fr7pBft/K0rGW53CGegVSqhDCN23njjTyWQkYQWVb4cIOWndZQf3wPb8jzjTT36fv4gae8vRqmq9k/oTiulR/ZDcftJ/WAxKILKER3RnVaEN0gw24W0qKglsRVWJeDvbBQlVHdVkjpjmVE5tlvL2JLmu28eLjVSvU8YEUVhLVhzDEIpNnSHBakqaLsqkXYLlf1b48p3M3jwYSx877eE5qJorHYrZ91xcky+QGlhGW8/OxPrklz8bdIigQC+QJDFO4t4Y95DtV4vkd9KShKW8mgInkov9536GIt/WoGiCFrkNOP+L28ju8P+NR3tZunPq6iZu+Wt9PHtGz+aiiGKxuYx3Ar8IKXsAvwQ/pyIR4BzG3kvk1oYffYI7M5YO6ke0ul7VHyzITDaK068eDTvvHkjM5+9lmYrCxJO2FqBOxJPD0AwhGV7SZ0hn3tDJDO6bRqVw3KM1qbRSImeao8zsChVAdI+XoI10Zt4SKK6fUbehjTs84ongH3ZTiOXwapi2VZqREa1dBHIyTBCZkO6EUFFTBBRZDWSNG+LoRx10O1a/OogXI48YooKhtDKvfjH98QzJCeSuxD9HKHJfSg/qQ8Vk3oZK5CgTsqcDbSWKuddcwJvrn8Ge3LtdnDNquHKTGHopOqcpvV/beL8ztcy85Ev0H7dSNoHf0XqVSlC0CYrtdbrAfQ7uidqjeq8tiQb4y8bW8sZ9eOVW99m8U8rCPgC+Dx+tq/ZwT2nPNqga4RCIQq2FkaaDDUER7I9ZhUERlitM9XZ4Gv9nWmsYpgMTAtvTwNOTHSQlPIHID6u0mSfMOKUIYy/bCwWmwV7sp2UjGTu+fSWPZYhAMOJ/eSvDyBqTnCAGtRJm7sBtdCNUuHFviyPlAQZxYmoaT7a/Vm3qIlNS6qgql9rfG3S4u37ioKWn3j1U1vYZyJFJ0IStaQK99FdKDl3IIH26aR8u9rIrQiGsC3NJf2t+RFfSXQioH3RdgiEYjK9tfzEf9Kiyo9lczHqrkrsS/NI+m4Nbd3BSJe/3dfcTZtl+Ti2lZLy+TK0baX4O2Wip9jIXbODm0ffQ5LLwaQrx8Xdx5Fip8vhHTnx2uN5dv7UmLDQp65+maoKD35P2LEf1HHOWY8iwGbVuGjcEZFjt6/dwev/fo9XbnuHTcuNelFWu5XHfrybjn3aI4RRkvvqJy+s9WWjvvz84e8xZk9dl2xZuZ3yXfWbHv6avYwzW1/GRd1v4JSsi3j3wY8bdP8eQ7uS2So9RulZHRZOuu6EBl3n706jSmIIIUqllGnhbQGU7P6c4NijgJvqMiUJIS4DLgNo167d4Vu21JKlapIQd2klu/JKaNMlO+5tb0/M+3oRd5/8SMRk4Ui28+A3d/Dje78y9+M/8FX68FZ60WtxJCcicqQCUjPqKulOC6EUO5b8iphM7cg5itHDOtgs2biCopD05xYc4a5ydZmjou+bMGdDU6gc0t7oJwEQ1El/dyF4AmBREIH4ukfR1/R2aYYSCGHdUmx04hNQfMERkV7fgLHiWLYD57ytxsrAolI2qRd6hlHoL/2dhSg1SrCrmkoow0HxCT2M3hJhIZJnryOtqIobXricT5/8itV/ro85z55s44EvbmfVn2sp3FrEkImDOHxsH4QQTEw5Jy6RDUUw5NnzuGjyMDq0NHpCL/phGf+ZPJWg3wiltVg1prx1HSNOrjarBANBVE3dJ76ACw67jtx1sYmHFpvGx0Wv43DaaznLoLKskjPbXB7zXHanjbs+vpmBx/attwwlBWW88K83WPjdUpq3zeSSh89lwOjeDXuQJswBqZUkhPgeaJngqzuAadGKQAhRIqVMr+U6R7EHxRCNWSvpwOOp9LLq97WkNnPRqW9OZP/nz33LS7e8WWdV0ERIoGxST2SKnZSZq9CKq233tdnbd58XbJGMnmJH21mO4vbHlvho2GMZ5ymCUKqdspP6VIe4+oMk/7wB24Zde1Q4td070CKF8hO6G8pBStRCN64vVqAEdfwtknGP7opMqZ7w0l+fFwmbjab8+O5GWe/olZs3QMrqAlKHd8G7o5TAd6vQoooDWu2WSE6D3xvA7rRx/MWjuep/F3LN4FtZM39DzD3SW6QyfcfLMRP8Rd2vZ1tUtVaAjJZpvJ/70n5xCn/31hyevPJlfFXG5G5LsjL67JH888XL93juzx/9zqMXP4+nwhOzf/TZI7j1rev2uayHKvtCMezR+SylHFOHAPlCiGwpZZ4QIhsoaIwwJgcXh9POgDF94vbP/eSPOKUgFEGfUT1o170NGS3SaNmhOf+74sXqcgiKINDKRSjbsGX7O2SgllZF+kxErxbiopIAS74b8t1GFe8UG0pQJ5iZhPAG0XZVRs6vdXUQdS0AdEnloHY1ktEESmX9lF3NnIbdikorcmPZUkyoWQrOnzcYbViBQLoDz+FtY5QCGBVebeuKYp9XEdUlNaKxalT0yqbCXQUuK2JyL1wzlqPtqkSzqNiT7VSWVhEKGorGW+njyxe/4/RbJnP2Hadw/5lPIKVE1yWapnLDi5fHTfZ5m+L/y5bklxIKhtAsxvTw0dylPPf5b5RXeunfuTX3nH8crTJdcefVh7HnjsJqs/D+w5/icfsYd9ExnHbjxHqd60hxJCwmuLvvyf6iqsLDq7e/w6+f/klqMxcX3HsmQyc2at5t8jQ2XPVz4HxgavjnZ42WyKTJkd2xBUvnrIyJVLHaLVzz1MW06tySd+//iDfvnk5m64xIIbUx54wk2K8NT3/+K3LTLmyr8kFPPJHX9V4qMJLhyk/oQbBNGkq5F9fnyxH+ICJQnaRX1zUjiXMLt1HWPt3Ieg6GUEur0KJKivg6ZeHrnIlSFcDxV251j4kE1w60TqVqQBt0hwXLxl2kzF4fF6obynDGVYutGtwe66biiF+kTddWtOrUnO+Lq4z2pzUjnLQoN6Cmoh/ZEcfsdRxxwgC2rcmlvCjWNm+xW3j2uteY9/UiNE3F5/HT/5heXPvsJbTunE1NOvXNYc38WBNVq04tI0rh52UbefyjOZFw17/W53LZEx/y+b0XoSTwS9WHUacPq7Paam0MGN2b5AwnPo+P0O7QZJuFiVcdt1dy1Jc7xj/ImvnrCfiC7NpRwgP/eIJ7Z0xJ+BL1d6GxzuepwFghxDpgTPgzQoiBQohXdh8khJgLfAiMFkJsF0Ls39+kyT7l9JsmYXNYIxU1bQ4rvYZ3J6dnWx78x//48LEvyNtYwI51O9m5KZ+LHjiLM245kVOO6kcb1UrKrDWoVYE4p2t9ERJSFm0HfxA92Ubpaf2MHtlUO4fr4/lQdxm9KLS8chyLtpP6xYrIuZXDO+Ie1YlATia+bi0oPb0fgcykhNfxt06l/LhuBFuloqcnQY3oJAGgS6ybd0EghH3pDpzfr8W2NBfLluKY4ob5WwrI6dWOrkVeQ9EFjT7bicqPS6D9kM58Xv4WJ1wyms0rtsXL5vEz7+uFBLwBPG4vekhn6dxV+L2JI8n++dLlOF1J1W1pnTZuev3qyPfv/7g4JgdCl5JSt4eVW/P3ON7RhIIhfnhnLo9e9CwfPf4FlWWVez6pBqqm8tRvDzLq9GFkZKfTa3g3/vvdnbTrtv+S4nLX57Fu4caY1ry+Kj/vPzxjv92zKdCoFYOUchcwOsH+BcAlUZ9HNOY+JgeXNl1b8ez8qUx/eAY7NuRz5ImDmHjVOEoKyvhz5l8xUSa+Kj/v3P8xQycOxG7VmJzVjDfrqOWkWVRSm7nYtaPu5K9sm412ZSHWbsrFsrMCLa888p2EuIJ5iRBg1IEi3P1Nl5ECdb7Dmle/nSsChELVoHakfrM67jqeAW1iW4QmqKIqQjqKJ0jqp8tQK3xGuY0N8dVCA74gHz/xJVPevJaMzi24/NkZ+G1qnFIAEMEQWz+cz1dZzXnttnfRa0RjKapCp77t43wLAW+Ae097lOZtm3HGlMn0GHoY82f+RSgYYtDx/Xlni9GSNhTSGTZpIK7MGl37EuCt8PDcDa/z++cLSHI56DKgI72Gd6PHsK4sn7saV1YKg8cPwGK1IKXkjvEPsuK3NXgrfdgcVj596mteWvJog8NEM7PTue3t6xt0TmPwVHhjSozvxl3acMV2KGFmPpvUi7aHteam166O2VdYXoWixk9g7pLq0FJ7ki3SbSwaW5KVzFYZnHPnqYw9dxSX9vkXm5fHvwGDEad/+Ng+fPfmHJJ88YleAqO5TSjdgVoYn7WciGCzZMom9DQ63UmZsK9DqEUKUhWRWk26XSPYIsVoHxqFv306SQu2EbNuURSkIlDcvphe2hBv+goFdR675HkenX0PVrePoFdBTwuHGusSgiEjZHdbKdqSXJ68/KW4JK3d41RTKexm+5o8tq/JY9H3S7HaLWiRsGHBf7+7k7HnjUp43plH92PxhtzIqkERglSnnTcueZGNS7YQCO/fuHQL3789h1BQN1aXmkJKRjLPzJvKjvU7WfHb2kg0kc/jp6ywnJmvzubUf9XPv3Cw6NCnHUkuB56ojnW2JBvHXXj0QZRq/2M26jHZa1p1akl6i7SYl1urw8Ix54yMfD76zGFYrLHluu1OG29teJZpa59m7LnGhDRl2jVxNmuhCOxOG+17tOGk68bXGSUjgjqDju7NSbefRFKqwzi2Dg1hW1tI0oJtRnSUzRJfITWkg8dPoFUqUoA/O4WSfwyg4pgu6E5rjCLR05OMDnkY0VZSU6gc2h4R0uudY+Gr8nPtkNuwfrGctI+WkPLVCgjpOOZtxvXVStKm/4Vr1hqjC14tkYTRK7e68HsDVFV48VR48VR4ePi8Z2o9dmTvjvzzlJGkJTtQhKB/59bcedwQNq/YHlEKkSELP6vP48dT4aV4Rwlv3fNBOOopVmafx8/GpU0/HF1VVR765t9kd2xhRIHZLIw9bxQTrzj2YIu2XzFXDCZ7jRCCB766nX9PeIjivBJ0XWfIhIH849YTI8dktc5k6qw7efKql9i+Zgfte7TluucuJb1FWsy1OvfvyBvrnua1298lf0sRR585jLbdWpOSkULXwzsC0LJDc7at2RGXuQqGM7xNposFH/1BVZkHZ2pSTD2cONnBaE1aVEnFhB7VuQhSGkXyiquMsNNAyIg+KqzENXMV5RN6GsXxdJ3d/aiFP4R9d6SRhECWEz3Jarzta0pMuY49jmk4lNWSV4F96Q6kw2pEYSWoMBtznhAJx6U+5K7Pw+8LYLUl7rF82si+nDayOk9gyZwVqNqe3ymDgRCLf1zBideeEJf/Ynfa6L+XuQMVJW68lT6yWmcckDpLHXq1Y9q6pyncXsT6RZvZlVfC1lW55PRsu+eTD1HMns8mjUZKyc5NBSS5HKRm7V0YY33IXZ/HnROnsnNzAQFfEKEYk6GqqThS7EaZhb3ItUBT8B3elsqeRrqO8+cN2NYXxb3ZS03BPaqzUTcJUHPLSP51Y0zXtOjrlp7RD8ei7dg2FRsrBQn+bBfW7aX1M3dlOSk/oTsZ0xejBEK1Jhfm9DImqNpMcXvK0XBlpfBR/qv1nmT9vgCnt7ykTsULxopv1GlDueO9f/Lqbe/wyVNfo4RXct0Hd+HBmXdEop/qQ8Af4NGLnmPux38ghKBF+2bc+/mttOkSH221r/F7/dx49N1sWbEtEp138g3jueiBs/b7vRvKAUlwO1iYisEkEVJKCrYWEQqG+Oa12fw1ezld+negefss3rnvY7xVDa+fA6DaNPwtUhB55Sh1FIrz9M6malgHCOnYVuWTHC4oWBPdplFy7kBQFdSiStTiSkJZTkKpDlyfLEErjlcmMc8JBDtm4h/fk/9MGM7iN+eyduEGdqzPj+QtgOGreXjWndx9yqOU5pfFXSfktBrmrJCOCK+Gos1gikXlllevYszZI+POrYuVv6/h7pMfobzYbRTWE6CqCnpIR0ojjNRqt/DMn1MjE/eWVdtZ+dsa2h7Wip5Hdmvw2/7b93/Eew99GukxIoSgVacWvL7mqf2+cvji+W958ebYJE+r3cLLyx6nVadE+b8HjwOS4GZi0pTY/aYIxLytff3KD3sXCxsm5AvSJd1FbmElqsOKP9ybOrqqqdQUglnJxiQb1ElbV0RtNU9FIGSYklQMhZBlRN+oxVWERnRG+2xZ3c8JjDl3JBddM54/3/uNJT+twFvpo0Ofdmz4a3PEzzD85MFISSSTuCaBNmlUDmqHNbcUxR/C3z4dEQhhW12ACOm0PaZHg5UCQI+hh/F+7kvs2JBP0fYiVv+5gebtsnBlJvPHlwvJaJnOuIuOJqNldSGE9t3b0L773ndl+/7NORGlAMZLQlFuMTs3FZDdse7qrKWFZaz8fS0tc5rTsU/7Bt/7r9nL41ajqkVl9Z/rm5xi2BeYisHkb8Hwk47ghRvfiNsvFIFmMer8+H2BOhMeOvZpzxM/38eO9Ttp2bE5U895igXfLiYU1I0XbU1BWgT2FTtJWVvIozNu5cbRdxNKUOJC2jS0/AqCrVxGQp0ujVpJS/M44fyj+PH3LVQVlMcLEUazqHRLSWHJjAW8cus7kYl//aLYFcovn8yjz4getb4xK24fWBT8XZvH7K8a1gG7VWP0kb144cY32LZmB0PGH864i4+JCxaoDUVRaNMlmzZdsul3dLW/YOCx/ep1fkNJSo3PK9F1iT257hpL37w+m6evfgXNqhEK6vQe0Z17P7ul3s8J0HlAB+Z9tTAmH0QP6bTrvv8bCx0MzKgkk78FrswUHvn+Ljr2bY+qKbTums21z17C6TdNYvylYzESHYxjE8WlAyz4dglXDZzCvK8XYU+yce9nU/jXq1dhT7ahWVQUbxDXrLW0WFXAzU9cyDPXv5ZQKYScVkpP72e0HlUMR7Xi9uH6agXW7aUcO6o3H2x6jtHnjMDpcpDkcsQVPdSsGsnpyXz42Oe1rgbAiGb6/p2fUWtGVYWx5BptTR2/bUQp80TMSBZV4ci22XxxxavMeOYb/vz6L168+S3unDi1HqN9cDjr9pOxJVVXkLXaLQw8ri/pzWsvIV5WVM7TV79iRGKVe/BV+Vg2dyXfvPZjg+498YpjSW3mwuYw7m932jji+AExzY3+Tpg+BpO/PY9d/Byzpv1U78qwNoeVI086IpJIVVlWyfsPz8Dj9jL+sjHk9GzHzaPvYcmcFQlXIO5hOfh6tIy09gQgGCLtg8W4FJVPi9+IvOFLKcldl8c1g2+jsryqWnkpgounns1nz35DwZb4xLhoVE3F7rSR3iKNHRt2YnNYad21FRsWb0LqslpEVdDt2jGcc+lxdG6Vxcf3f8yMp76O6f1sS7Lxv7n30bn//pnwgoEgf878i9L8Mg4/tm/ELFhf5n78B2/d9yHukkqOPnM45919Wp09m3/7fD4Pn/c0VeWxhfcGjevHg1/f0aB7V5ZXMWvaT2xbnUv/0X048sRBKErTe7c2fQwmJvXAXVbZoHLhPo+fuR//wTVPX0zhtl3cdMzdhIJGWeo5H/zO1Fn/Zvkvq2o1S+kOS6xSwOgFEXLZ8RVV8tod79J9cFfSW6by0NlPUbh9F4qq4MpIoby4AqRhInn5lrfrJW8oGKKyvIreI7vz/KL/YrVb8Hn8nNrsIvzeQLXrJSRJWraTId0NG/vCWUtilAIYLUJ3bi5IqBi8VT4WfbcUhJG4OOOZmVSVezj+4tEcc9bwPTqAy3dVcM0Rt1JaVG44qXXJtc9ewrgLj6n1HF3XWfT9MvI25tNnZHdGnDKEEacMqde4ALTMaR6XXKlZVNoe1nATkNOVxEnX/v/o22AqBpO/PWPPO4oF3y6JqeOvqApCEbW2qgz4gjx28XNsWZkb0yLVU+HlygE3oyi197tI/nUjVYrA1zErsk+qCtrOcoIhyUePfYFm0/BX+WMUVrRjNSECugzoQGlBOYXbdsV+J2Hel4uY/81iRpw8mOK8koQms+1rjRLby39ZxdbVuXHf+30BuhzROW7/xqVbuPGou9B1nVAghC9K1jXz17Np+VYueejsOsV/54GPKdy+K0YZPXPNq4w8dWjCplI+j48bj7qLratyCYVDRE+7aSIX3HNmnfeJpmOf9vQe0Z1lc1fiq/KjWVRsSTZO+ef4el/j/yNNbx1kYrKPGTpxIKfeOBGrw4rNYSUjO537vriV4ScdgcWmodm0hAlbf3y5MDKRRiN1kFJHsyZWDoo3hPPH9SRtLjaik3Z3TwsnqQUDIbxuX4NWMQAp6clktc6krDCx01pKyWu3vwsYb8oWa+x7n6qp9DuqFwAfPv5FjFKUAqQqqBjSnkkPvMXNt70R8+wPn/807tJKw05fQ4F5K318+uRXe2y1uXj28vgVikVly8rtCY//5rXZbF6xDY/bi9/jx+/x8+Ejn5O3qWEF/O797BYuf+Q8Bo3rx8Qrj+OlJY/SvF3DTFj/3zBXDCZ/e4QQnH/3GZxxy4mU76ogq3UGiqJwxLj+gPFmev+Z/2PeVwtjsodrmiCi0awaA8b0Ze2C9RTnlcbfM6jTYUs50u5g12/rUNx+6pN6J4Sos+TFwllLaq2UClC805BF1VRufft67j31URRNQQhBcpqTSx4+m+/emsMfXyyMnKNrCr4eLfD0boVMtoGU/FBUxNLR/+HOpy5l2ORBbFxSd/kKKSVV5Z467f0d+7Zn8/KtMQox4A+S3bF5wuP/+iE+RFSzaKydv4HsDnWHp0ZjsVqYeOVxTLzSLOpcX8wVg8n/G+xJNpq3zYpzGNocNu77bAojT423XVtsloSFAlVNY+y5I5me+zJPzL0vYW8C4Q/y9qe38/Q3/+GKx87Hat9zeGRtkyRAs7aZqHVkCquaSs+R3Xn92/k8+9mvpPZqzbtbX+CG5y/j9ndvYNq6p0lt5uK561+P6a0hgjq2lfm4vlkFwRDa9lJcX60kWFTJQ+c8hd8bIL1F7ZE/YJQ+SasjOgjgvLtOJ8mVhCVcesPutHHy9eNJa5b4vM79c7DUGLNQKPS3DRFtSpiKwcQkzAmXjMHujH3jVRTB9S9cjhJlalI1haQUO0MmHA5Az2GHkd2pZYzz1eowWlYCdO7fgYlXHMuYc0ZGrm+xaQkT8jr2zeHmN66O/yL8nZYgLNVi03Ak28lsl8nPLay8+OXvvP7tfC54ZDo/rNrMMWeNYPAJA9AsGpVlVTGVQnWbhp5qRwR11FIvSX9uwfXtaiw7K1B8QfweP3ed+F+uePx8bElWhBAIRSCEQLNqJLkcpGQm8+/p/9yj8zm7YwteW/U/zv3PqUy++jjumTGFix+svaTEpKvHkZqZEhMiOnj84XTo3fAENZOGYYarmphE8d5Dn/DO/R+DENgcVm5753oGHtuXLSu38ert77Jp6VZ6j+zORQ/8g6zWmZHztq8z6jgVbCtC6jrDJg/ilmnXxhSmk1Ky8LulLPxuCaFAiG9en42nwhtz/0Hj+nH3p7cwwXl2XFG89JZp3PXRTfx74kN4KrxGaQxhrCSueOwC3tm6jflrY+31SXYLPz5yJZZwnoSUkgnJ5+Dz+qkc0RFfl+YgjUKAKd+uRnH7UD2xpiqr3cJrq55kV14JM1/9ASEE4y8dg9/r57U73mPlH2tRFIWRZx7JpU9cQNY+bLVZWVbJrGk/sXX1Dg4f24dhk5tmiGhTwqyVZGKyH/BW+SjJL6V5uyxUtfboo2i2rs7lf5e/yJo/15PVLpPrnrmEw8f2rfX4YCDIGa0vi2nNaXfauGXatQwa148T086PqYkEkJKRzCdFrzP9vzOYdvcHBKJ8DRabBf9p/Sl2xpqabBaNT+++gJYZ1c13phx7H7/m5lM1NCe24ZA3gOvDxWhVsYrBkWzn8Tn3xoWw3jlpKgtnLYmU35aqgr93NgMvO4YHLjoeRwMyi032HftCMZiq18SkBvYkG9kdWtRbKXirfPxzhJHb4PcF2LFuJ3ed9F+2rExc7RQMJ+oj399Fm8NaoYZDKP9x20kMP+kI7Ek2+h7VM8ZsZLVbGB3uc/Hdm3NilAIYjmnlg4WoNfbbLCpZNbqknXPnqfi7NovrQSEVweirjo0zV9mSrHTo0y5mn9/rZ/43i2N6MoiQjmVVPr+t2MxTn86t9dlNmj6mYjAxaSTzZ/5FwB+KaQIX8AWNwn510LFPe15f9SQf5b/KjJI3OOv2UyJ2+jveu4E+o3qiqAqKpnDE8f25dKqRJ+BIEPMPoAlBytYyHDYLNouKzaLxn3PGotXIZ+g9ojtaQI/vWqcIjj1jOAPD0VoACAiFdHLX7Yw9VohE3UdBCPzBEN/OX1Pns5s0bUzFYGLSSIy35thJVkpZ745qyWnOuL4EVoeVqgoPFrtRvvrPmX8x+71fAPjHrSclrI0kJVw8diC3nXkMN5w8kk/vvoBj+ndJeM9eWGIbCAV1bCUeDu/bgYDXX50cJ8Fd7GbquU/FymezMOLUoTFRQ1JT8PY0wkgdNsNhHAiG+HDOEq5++hMe/fAndhZXYNL0MRWDiUkjGXxC/7h9VruF4y7Y+77AM56eycalW/BV+vC6ffi9AZ6++hUqStwMmzyIyx89Py4KSFUVjjl9GBOG9OCMo/rF+BVqcttD59JszkYj+qjcS9KaAv5z4ghUTWXpz6tiwlmlNKq6BgOxRcZvfOUKjj3/KFS7BWnT8PRphWdAW+xWjQvHDQLg+udm8MQnP/P7yi18MGcJZ9z/lqkcDgFMxWBi0kicqU6mfnsnrTu3NGoeZSZz/fOXcdig+NIS9eXPrxfFlcjQrBprF2wA4KRrj+euj28itZkLzaKS1jyV29+7IaYvQWlhGe9P/ZQnr3yJeV8vikmc69C7Pe//+hC3Hj2Qf/U8jHdfuo7jzzkKgLTm8V34klLjK8DaHDZueP4yvqh4i9M+vI7U43rSPjuDW04/mlOG92bNtgIWb9iBN+yHCIZ0PP4A7/ywaK/HxeTAYGY+m5jsA3oM6coba5/GW+XD5rA2uqNY+x5tWPn7mpjs66A/GDPxH3niEQydNJCqcg9JLkdMGGfBtiKuGHAzvkpjtfH92z8z9vyjuO6ZSyLHpDdP5aTr4ovCXTL1bB6/9IVI1rE9ycYF951Z6zNZVJXLJwzl8glDY/bv2FWOWiO0NBjS2VpQ0oCR2DeEQiG+fPE7Zr8zl/QWqZwx5SS6D05sZjNppGIQQmQA04EcYDNwupSypMYx/YDnARcQAh6QUk5vzH1NTJoq9qTaS0I0hDOmnMiP7/+Kz+Mn6A9id9oYOmlQXLcwRVFITnPGnT/94RlUlXkiIa/eSh/fvDabM6ecSPO2WXHHR3PMP0aQ0TKdGc/MJBQIMeGKYxl8woAGP0O/Tq0IhGJDbu1WjZF9Ojb4WonQdcmM35Yz49flOGwWzhs7kCN75iQ89rGLn+fnj36PKLsFs5Yw9Zt/02t4930iy9+Nxq4YbgV+kFJOFULcGv48pcYxVcB5Usp1QohWwEIhxLdSytJG3tvE5G9Ly5zmvLL8cWY8M5O8jQUcOXkQo84YVu/z1/+1KS4PwmqzkLsub4+KAaDf0b3od3SvBssdTXpKEreecQwPv/8DWlASsigc3qUNk4b1bNR1d/Pkp3P58OclEVPVsk153Hv+cYwZ0DXmuJKCMn6a/ltMMICvys+0uz/gke/v2iey/N1orGKYDBwV3p4G/EQNxSClXBu1vUMIUQA0A0obeW8Tk781Wa0zueShc/bq3P5j+rBu0aaYyTDgC9Cpb84+kq5+ZBVUkv3+EqrKq3CkOPjHq0Ox1DM/pC58gSDT5yzGH1Wt1esP8tznv8UphrLCclSLGhclVrS9RulykwiNdT63kFLmhbd3AnWWPBRCHAFYgQ21fH+ZEGKBEGJBYWFhI0UzMfn/y2n/mkB2x+Y4UuzYkqxY7RYue+RcXJm1Ryrta3Zs2MlD5zxFxa4KQoEQ7mI3D571P/I2NqxsdiI8vkDCsuUlbk/cvrbdWkXqLe3GYrMw/OTBjZbj78oeVwxCiO+Blgm+iumLJ6WUQoha62sIIbKBt4DzpZQJ6xlLKV8CXgKjJMaeZDMxMUmMM9XJS0sfY8G3S9iVW0z/Mb0bVKp6XzD343noNcxZoZDOL5/M47SbJjXq2mnJDto1T2NTXnEkg0RTFUb2jvdfqKrKPZ/ewr8nPBRpNHTYEZ05645TGiXD35k9KgYp5ZjavhNC5AshsqWUeeGJv6CW41zAV8AdUso/9lpaExOTeqOq6l45jfcVVrvFSJSLMvcoqoK1xtv73vLIZRO58smPcXt86FLSuVUWN546KuGxPYcdxgc7X2bN/A24MlNo180s3V0XjSqiJ4R4BNgV5XzOkFLeUuMYKzAT+EJK+b/6XtssomdicmhTUlDG+V2uiakgm5TiYNr6p2vtwdBQdF2yNrcQu0Ujp2XGPrnmoU5TKKI3FRgrhFgHjAl/RggxUAjxSviY04GRwAVCiMXhf/0aeV8TE5MmTnrzVB7/6V56HtmNJJeDXsO78dice/aZUgCjX0a3ts1NpbCPMctum5iYmPyNaAorBhMTExOTvxmmYjAxMTExicFUDCYmJiYmMZiKwcTExMQkBlMxmJiYmJjEYCoGExMTE5MYmmy4qhCiENiyh8OygKIDIM7+4FCV/VCVGw5d2U25DzyHquxZgFNK2awxF2myiqE+CCEWNDZe92BxqMp+qMoNh67sptwHnkNV9n0lt2lKMjExMTGJwVQMJiYmJiYxHOqK4aWDLUAjOFRlP1TlhkNXdlPuA8+hKvs+kfuQ9jGYmJiYmOx7DvUVg4mJiYnJPsZUDCYmJiYmMTRZxSCEGCeEWCOEWB9uAlTze5sQYnr4+3lCiJyo724L718jhDjuUJBbCJEjhPBE9ax44UDKXU/ZRwohFgkhgkKIU2t8d74QYl343/kHTupGyx2KGvPPD5zUkfvvSfZ/CSFWCiGWCiF+EEK0j/quKY95XXIftDGvh9xXCCGWhWX7RQjRI+q7gzavhO+/V7Lv1dwipWxy/wAV2AB0BKzAEqBHjWOuAl4Ib58JTA9v9wgfbwM6hK+jHgJy5wDLm/iY5wB9gDeBU6P2ZwAbwz/Tw9vpTV3u8HfuJj7mRwNJ4e0ro/5emvqYJ5T7YI55PeV2RW1PAr4Jbx+0eWUfyN7guaWprhiOANZLKTdKKf3A+8DkGsdMBqaFtz8CRgshRHj/+1JKn5RyE7A+fL2mLvfBZo+ySyk3SymXAnqNc48DvpNSFkspS4DvgHEHQmgaJ/fBpj6y/yilrAp//ANoE95u6mNem9wHk/rIXR710Qnsjs45mPMKNE72BtNUFUNrYFvU5+3hfQmPkVIGgTIgs57n7i8aIzdAByHEX0KIOUKIEftb2NrkCtOQcWvqY14XdiHEAiHEH0KIE/epZHumobJfjNE/fW/O3Zc0Rm44eGNeL7mFEFcLITYA/wWua8i5+5HGyA4NnFu0xkprss/IA9pJKXcJIQ4HZgghetZ4CzDZ97SXUuYKIToCs4UQy6SUGw62UDURQpwDDARGHWxZGkItcjfpMZdSPgs8K4Q4C/g3cED9N42hFtkbPLc01RVDLtA26nOb8L6ExwghNCAV2FXPc/cXey13eIm6C0BKuRDDnth1v0ucQK4wDRm3pj7mtSKlzA3/3Aj8BPTfl8LtgXrJLoQYA9wBTJJS+hpy7n6iMXIfzDFv6Ji9D5y4l+fua/Za9r2aWw6U86SBjhYNw5nWgWpHS88ax1xNrBP3g/B2T2KdRBs5cM7nxsjdbLecGA6mXCCjKY151LFvEO983oThBE0Pbx8Q2RspdzpgC29nAeuo4dA72LJjTJobgC419jfpMa9D7oM25vWUu0vU9kRgQXj7oM0r+0D2Bs8tB+Sh9nIgTgDWhv+47gjvuxfj7QPADnyI4QT6E+gYde4d4fPWAMcfCnIDpwArgMXAImBiExzzQRi2zUqM1dmKqHMvCj/TeuDCQ0FuYBiwLPyfbBlwcRMc8++B/PDfxWLg80NkzBPKfbDHvB5yPxn1//BHoibfgzmvNEb2vZlbzJIYJiYmJiYxNFUfg4mJiYnJQcJUDCYmJiYmMZiKwcTExMQkBlMxmJiYmJjEYCoGExMTE5MYTMVgYmJiYhKDqRhMTExMTGL4PytvkTS1l7IEAAAAAElFTkSuQmCC\n", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD4CAYAAADo30HgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAC5EUlEQVR4nOydd5hcVdnAf+feO322l/RGekISWkKo0ntTLIAFBcWGgCCKYEFABAuIiIWigtJFPpFeQodACJDee7JJtu/s9Ln3vt8fd3Z3Zmc22WQ3JIT7ex4esnfOPefcKec9561KRHBxcXFxcelA290TcHFxcXHZs3AFg4uLi4tLHq5gcHFxcXHJwxUMLi4uLi55uILBxcXFxSUPY3dPoCeqq6tl5MiRu3saLi4uLh8r5s6d2ygiNX3pY48VDCNHjuS9997b3dNwcXFx+VihlFrX1z5cVZKLi4uLSx6uYHBxcXFxycMVDC4uLi4uebiCwcXFxcUlD1cwuLi4uLjkscd6Jbm4uOxaRNJI7F5IPgP6QFT42yjPlN09LZc9AFcwuLh8QpHWSyD1FpAEcxGSegOq7neFg4urSnJx+SQi5gZIvQkkO64AKST65904K5c9BVcwuLh8ErGbQHm6XRSwtuyW6bjsWbiCwcXlk4hnUpGLfvCf/JFPxWXPo18Eg1LqJKXUMqXUSqXUVdtod7ZSSpRSB/XHuC4uLjuHUl5U+Z9AhZ3/8IHvEFTo/N09NZc9gD4bn5VSOnAHcDywEZijlHpCRBZ3a1cCXAq809cxXVxc+o7yzYTa2ZBZDFoVyhi2u6fksofQHyeGGcBKEVktImngIeDMIu2uB26my9rl4uLSS0QEST6P3fxN7NYrkMz8fulXKS/Ku58rFFzy6A/BMATYkPP3xuy1TpRSBwDDROSpbXWklLpIKfWeUuq9hoaGfpiai8vegURvR1qvhPTLkHwSafoSknpzd0/LZS9llxuflVIacAtwxfbaisidInKQiBxUU9OndOIuLnsNIimI3Q0kOq4ASaT9N7txVi57M/0hGDYBuefQodlrHZQA+wKvKKXWAjOBJ1wDtItLL7EjOMKgG65rqcsuoj8EwxxgrFJqlFLKC5wDPNHxooi0iUi1iIwUkZHAbOAMEXGr8Li49AatGrSqbhcN8B2+W6bTHUm9gt1wIvaWadjNX0PM9bt7Si59pM+CQURM4GLgOWAJ8IiILFJKXaeUOqOv/bu47A2ICJJZhphrdvhepRSq/DZQJVn30hDoQ1ElP94FM90xJLMQabkErDVAAtJvI83nIpLZ3VNz6QP9kitJRJ4Gnu527Wc9tD2qP8Z0cfm4IOY6pPlrIE0gghhjUZX3oLTyXvehvNOg9i1IzwEVBM9+OOa73YvEHwTSOVdskDik3wbfkbtrWi59ZPd/s1xc9nKk9VKw60ASOAnrliCR63e4H6V8KN/hKO8Be4RQAEDSgN3togL3xPCxZg/5drm47J2IHQVzGfmLpwmpl3fXlPoVFfgsEOh+FXyH7o7puPQTrmBwcdmVKB/QPVkdsANqpL4gknaE0y5C+Q6Gkqsc+wc66KNQlfeiVHdh4fJxwq3H4OKyC1HKgwS/AvF/0hWH4IfQpbt0XBELidwAiUcBC/FMQZXfhtIH9ftYWuhcJPgFkCRKC/Z7/y4fPa5gcHHZxaiSKxBjOMQfBuVDhS5E+Y/dpWNK7G+QeIxOw3BmPtJyEar6f7tkPKU0xyjuslfgCgYXl12MUgoV/DwEP//RDZp4lPy0ZDaYaxFrE0of0tNdLi6Aa2Nwcdkl2IkXsRtOwN66H3bzNxCr7qOdgPIVuSiA96Odh8vHElcwuLj0M5L+ANouB2tt1qf/daTpXESsj24SwQvJ9xbygnc6SndzkLlsH1cwuLjkICKItQWRnc8OL/F/AamcKzZIBNIfXRYYLXgWlF4N2mBQpRA4E1V++0c2vsvHG9fG4OKSRdIfIq2XgN3i/B36Oip8CUqpHewoRWHSO0V+hPCuRwt+AYJf+EjHdNk7cE8MLi6ASBJpuQDsLTi7/RTE/gapF3a4LxX8PIVBXzp4D+6Hme5axFyPHbkRu/VyJPkSIkWyurrs9biCwcUFIP1ukYsJJPH4DnelfEdCyeVOsjt00EejKu/DST685yKZJUjTGRD/l1MMqO1yt+bDJxRXleTiAlkf/CLqH1WyU91pofOR4JeyQV+hPk/vo0Daf5/N55R9HyQB8fuQ8DdRWtnunJrLR4x7YnBxAfAcAFoN+XslHyp0/k53qZT+sREKAFirKRCOyuMWBPoE4goGFxecyF1V+SD4T3WK4hhTUBV/RXkm77Y5iR1FzFWIFBqtxY4h8YexIzcjqVf7xxbgPYLCvE4KjFF979vlY4WrSnJxyaL0KlT5nqFTt6N/huifQOmAhpTehBY4AQCxI0jTmWA149hBHgDfCX2euyr5HpJ+E+x654JYqLLf7fG2ke0h5nok8jNIvw/6AFTJ1Sj/0bt7Wns0rmBwcdnDkNQ7EP0LkOrS7LRdgXhnofQaJH4/WI10xkpIApLPIZmLUJ6xOz2u0iqg+mlIvwN2K/gO3aFiQnsiIhmk+VywmwAbrHVOfYyqf6E8U3f39PZYXFWSy16PiCDJ57GbL8Ru+S6SnrO7p7RNJPk0XZlYsygNUq87/84sJD+ADudkYS7v89hK6SjfoajAKR97oQBAerYTfZ5XDyOFxB/YXTP6WOCeGFz2eiR2p6OWyS62knodKfsdWuD4HevH3IgkHgW7HRU4DeU9YBfMFtAqcHT9uVXQNNBKnX96p2eFRE50tpjgmbJr5vNxRkyc4MK8i26Fue3gnhhc9mpETIj9mfwdeBKiO6aPl8xCpOlUiN0FifuR5q9ix+7r17l2oIKfB+Wla0EznLQWviMdI7P/NNBHZF1svYAfgl9CGcN3yXw+1vgOoVAwBFCBz+2O2XxscAWDy96NJLMpKrphN+5YN5Gbsj7+Jo7iPwnR3yHF+u4jSh+MqnwIvJ8CbSgEzkBVPQrmCqTxJGg4AuyN4DsNVfIDVNWDaKU/6vd57A0o5UdV/hP00YDmxKWUXIXyzdzdU9ujcVVJLns1Sgsj+iiwVuZc1cG7gzWJrdWF10Qco6Y+uE9zLIbyjEdV3pkzVBJpPMVJxgcgMUj+DwJ/2q0utR8HlGcSquaZbGJEr1NUyGWbuO+Qy16PKv89qEogiLMX0kCSSGZJ7zvxHEjBz0X5QRuwQ3MRSSKZBYi1YycW0rPJN6CC46r66I71swchdhxJ/A+JP4xY9bt8PKX8rlDoJf3yLimlTlJKLVNKrVRKXVXk9W8ppRYopT5USr2hlJrUH+O6uPQG5RkHNa+BMRRH35zJ1kg4BzFXbu92p4/SqxyjsAqBCgB+VPmvUUrv9TzsxAtI/Uyk+Xyk4Sjstp/vQGCah+IpO4oV5HEQqwG75ZvYW6Zg1x+GHX+413Pd1Yi5Hmk4Gon8DInciDQch6Re2d3TcsnSZ1WScn4ZdwDHAxuBOUqpJ0RkcU6zB0TkL9n2ZwC3ACf1dWwXl96irJWItYEuTx8BUkjsblTZTdu/Xx8CNS9D6hWwY+D7FEqv7nxdMouR5ItOTqHA6SitMu9+sVuc4j25bqaJ/wPfTPCfvP0H8B7s6MclQdfJwYcKfrFocxFBms8Haw1ggd0AkRsRrQrlP2774+1iJHIDSBu5pyBp/RHUvrVDwtZl19AfJ4YZwEoRWS1O7P5DwJm5DUQ6FKMAhCjc+ri47FrsJqD7gmPvUB4gpfwo/0mo4Nl5QsGOPYA0nQOxPyHtv0UajkfMbjaJ9Luguu/DEkjimV6ObaCqHnTSVqgg6Pugym/rOUjLXA72JiC3alwCif2jV+PtcjLzKFCNSXyHnQJcdg39YXweAmzI+XsjUJB4Xin1XeByHP+6Y4p1pJS6CLgIYPhw1/XOZduI1YQk/g3WZpT/WPAe3nNRHc8B5C+SAAGU/7S+zUGS0H4zXTEFKZA00v5rVMVfuhpqlRTuhwzQe2+jUPoQVOVdvWxtUeimCY5X1R6AMRoy3SraKSMbw+Gyu/nILDEicoeIjAZ+BPykhzZ3ishBInJQTY1bm9alZ8Ta5LhuRm+HxANI68VI+409tldaCFX+h6yNIAz4wH8CBD7dt4lYm6FAGAmk38du/ip2yyVIep5jvNaH4eyLwLEPeFHBL/dt/J4wJjrJAHN/4iqACn5p14y3g6jSH2fjMDw4c/Q7bqQf87xMewv9cWLYBAzL+Xto9lpPPAT8uR/GdfkEI9G/gETpPAVIAuIPIqFvoPTawvbpd5HIr5y4Bm0wlP0YrT907foQiu7MpR3SbwEKSb0EwXOh9NeQehJSr4I+yikbagxHrAZH1WWMQRWom3YOpRRU3Iu0/QAyHziLcOjbTvbYPQDlmQLVTyPx/4DEUIFTUZ59d/e0XLL0x7dwDjBWKTUKRyCcA5yX20ApNVZEVmT/PBVYgYtLX8gspkA1pLxgrYNugkGsOqT5G3RGP9sboO0HiOfFPFvBzqCUFym5FiI/wVEV6dlxOuYmQAbi/4T4w1B6LVr1lc4rYmG3XgnJZ5y6B3ig4g6Ud3qf5tQ5N2MoquohJ/obfcdrV+9ilD4YVXLx7p6GSxH6rEoS51t3MfAcsAR4REQWKaWuy3ogAVyslFqklPoQx86w89VPXFwgm+qgW+0AyYAxvrBt8mkKdOtiQ/LZnRpaJIHd9lPsrQdgb50Jdh1U/Q9V8gMIXwoUcyF1vKCIXIfYcedK/GFIPg+knYA1aUVavlm0/kJfUMrY44SCy55Nv5xbReRp4Olu136W8+9L+2Mcl08uYm11XC6NcSjlRYW+gSSfd66JBQiU/BjVkWgu7+bugWE47QsCxno5l9YfOm6rHa6n0b8AHlT46yCCxP8Bdg/eTkp3XEi1yU7kcvcsquB47PTTqaG3iN0M5gZHnfVxqjrnsktwU2K47NGIWEjbjyD5nOO1IgoJfxdF3NGZayGU3Qzew1DGsOKd+E+B6B/Jy1aqNPCfhEgKid0PqZfB2AcV+nrP/eBUTiM1K78vnNrIhL+e1e3/EWm+IJvuuftJJQ36UOffWjWOfSLHW0nMj9QzR0SQ9t9C/F5HFScmUvoLtGAfjfIuH2tcweCyRyPxRyH5Ao4baMcO/WYE5UQgawOg6j/b3OUqYyhU/BmJ/BSsjaDVggohTZ93ThvS6vSfeQ9JPgnVTzgBbcVnRPEwnC4BoDxTofYtJ0Yhci2OEMk4KTRC33CC4AAVvghJvUqXq6sPPFNRxphevz99Jv0GJP6Fo87KqrAiP0N8h6D0gR/dPFz2KNzEIS67BMksQ2L3IcmXssbPnST5JEXVLYizI7c2I4lHttuN8h2GVjMLSn4FdrOjzrHrQLbSFY1sgbQjTV8sDFDr6EcLgz6y21UN/Pk7bKW8aMEzUTUvQvgSCJ6PqrgbLdxlbFWeKajKe8F7uJP9M3TBDsQp9A+SfCEbTZ07eQ1Sb3yk83DZs3BPDC79jh29HaJ3AbbjbaMPgcqHd053rddCRqNne0ASMh8WXJXky0jsASADgU+jPJOQ5DMQ+9M2+up4gDqk6QtQ87IjCHL7tZvBWt/tBgVa8ROG0qtQ4Yt6HEp590dV/m3b89mVaLU4sRW5Bm8tGwPh8knFPTG49CtibYbonTjqkay3jblup0spqtA32Pb+xZ+Nau7Cjt2LtF4CmVch8xZErkSaToPYH+m1wVkykHq+8Hr6g2wRnVwsSL/cu373MFTwc9lEfB1LgccRCr4jdue0kMxi7JbvYDeeid1+O9L9VOOyS3EFg0v/klmY9cnPJeUUmO8BsRqwm7+OvWUS9taDsWP3dr2oD6B4agcF+EEflleNS8SG6G0U1ETe4fRcdqGKBUAflPWCysUAYwSSmY/deDb2lmnYjWcjmQU7OOZHj9IHoKr+4wS+GeMg+EVU1aP9Fmi3M0hmOdJ0LqReAnMJxO5Emr++2+bzScQVDC79izGmSD1dL/QQ1SoiSMvXIP0mYIK0QPstSOJJ5/XE8/S8qGfdVHOEgFgbsxHRvUWHkmvpSlWRg+/YgkvKMwm8+wP+7BXDMYIHzkSavwLmAiAB5gKk+cs7XndhN6CMEWjlv0OrfhKt9GrUbs5XJLG7cD7Tjs89BZmFSGbZbpzVJwtXMLj0K8oYBYHTgUD2SgC0ClToK8VvsFY7/vPds4DG70OsJojeQr7+u4NsRLG1xnG37KD18h2cse7YKEqvd9Ja4wVV4WQu7cErR1XcCSVXOlXgAl9AVf0XlX6/8CQhVja4rsjsrU1I/BHHFtIX43z3fkWcU9PHGWsrBSo/pbuZVz9CXOOzS7+jSn/pxAik3syqes4qMOJ20dNpwEaidzg5h7aJBclXoAzEbgVz8XbadycNyScg/D2ofgGF7QiybdQEcALsvgyhrgR4kspQaL/ICq9u2PFHIHI9Tg1iDbQaqHoEpZXv4Nyzo0gaifwGEg/h7LQV4jsaVXbTTve5W/Gfmk3LnavKs8Gz/+6a0ScO98Tg0u8opVC+I9FKf4wW+tI2hAKOm6Y+mPxaCQZYzZB4jMJU2UWQRuzITYho7NxX2obGY6FhJtJ0FmJ29zrqBb4TKaz3oGWv50zVjmaFQgpIOMZ5qw6J3cnOIpHrsrEIHSo1gdQrSGv/JRwQSSLx/2C3/xZJvbJLTyUq+FnwHwf4nEy4Kowqvx2lBXfZmC75uCcGl48ckbSzECaecjxgwhdD4pGsgVoHBOyNO9KjE3mcmu14KGU+pCtobAex66Hp08iAd3coBbQyhkH5bUjkJ06chFaJKvulE1yXi7nMMc5LrnE8Dam3oGTHpytiQeJxCgWoDen3EDtSPE3Ijoxhx5Cmz2SLGiWQ+D/BezSq4vd96rcnlNJR5b9DrC2OWskz0U3H/RHjCgaXjxxpvTQbQJUCaxW0zUdV3ImU3wX1B7Fzi7oJ1mKwHE+lTgO0XU+vTh15JJyU2b6jdugu5T8afG+AxB37SPp5xFwH/lNQera+iD68K8K4Ew08E3dwjh1sL+dT35PndRRD6vxcJAGpWUhmEcozuc/994TSB4Ibfb1bcFVJLh8pYm2B1Ovku5MmkdifUKQpppPfMZJgbYCy34H/s+y4UAAQpO06JPU6YjU6c+4lSimnjGfTqU6Zz/YbkYYjsRNPOK/rNRD8ouPJBDjG7jAq9J2dmKeTORXfMRT+lDUnf5S2E8eQ7mQWUCCsleacflz2SlzB4NIjImkkPQfJLEGkn8p0221Fah8DVlM23cRY8ne5Bmgj2eGvasv5EL+9D/PciLRciDQcgTQcj934acSq77G5mGuR5MvYZgMSuQZH8Nk4O3oL2q5EzJUAqJKrUOV/hMC5EP4uqvrZbSbu2x6q7CbwHJl/0TMDVX7LTveZ39dBdHmZZREbjD23sI5YW5HUbMRuyb+emo3deCr2lqnYTed1fiYu+ah++8H3MwcddJC8995722/oskuQzAInQyiW85++D6ry3r7rq8VCGo7o5nroh/C3nbQSkZ/i7E4FpybyROff1sIivXXLTLpL0cHYD4whTh0H5YHAF536C5Gru65JmuLutUDgs2hlPZcf7Qt22zWQ+B95O3ttOKrs56g+RjGLpJCmz4G5gs4TmD7WeV5loIIXoAKn9GmM/sLJFvtrpzCS8jmfR8n30UIXIOYapPFMut4jBaoEVfPqXpVqXCk1V0QO6ksf7onBpQAn6Oy7IG2Orl4SYC5H2n/X576V0lEVdzkumioI+MB3JPhOgsg1OC6KHYu9Dt59wVpUvDNtMFS9Bvokp5Yzfqe/XfK1tsCcmy3uk031Eb8P2n6Yk/01So9CAcDagtgR7OjfnEI/iWexzU19Tvcgdjsk/kuBusdej7R8F0nP6VP/4AU7N+AMsFY4rsGZ+Ujbj7Fj9/dxjH4i/Q7EH8D5jNqBFLTfipirkMR/yE+Dnj3NpT6e6Ux2Ja7x2aUQuw7s1m4XM47B0fqOU49A+cB3wrZdUXtAeSZDzetgrgStDKUPQGL/clJp55GG5P/R46nAbkbJBqh+zImctjYixhRoPnuH59R7chf+pJO2oVfGci94j0YaTwE74tyTeBjQEDxI6Guo8Pd3rtKaROjZyJxEonejKvtQ+CczP5uFticjd8JJThj64s6P0U84acyLCNrUm9nTXLdnEGGbwvwTiisYXApRZRRdjFUAaTg++4cG6kaoehhljHb8883FThEaVYYkngBrNco7E3xHg92MtN/s5P/XB6PCV6J8M7v61sqc6Na8Yb3ZH25PpJDUG2je6c6pA5Do3/v27D2i02Uz2Am8hzvpPuxW8hciG0hB7C4kfj+ij0CVXIHyHdb7vrXBzgmsJxff7QYJbgdJst1TmN3HMfoLfRDOyTFHWCsD9AEo70FI/MFur0nWeO+SiysYXApQWhgJnuPEFnSqOXyOf37uj0oUErke8Z/l2AY6dOzKyC7oCSTxKHiPBHOpUyQHE+wmpOWbUPVAl7uj/zhou7bbTDKO3cHe0MNMbYjdg618KBVCtFqI9l3d1YXXKa4jERzdevdduR+MSWC+v51+PKjym5G2q+h5d+rUgsBciLR8Gyr/ifJO69Usnapxf0JaLnRKnea9GEAF+niC8u5PQX3tPAzwfapvY/QTKnAmEv2TUwkPE/CCVgm+o1HKi5Td6AQYSpuzQSn79cczOnwX4woGl+KEvu0s5Om5Ts7+8Peg7YpujQTS8502uRXWcv30JZ4thamRr99NIbH7UOU3Z9slKcyIam9DKHSQhuhtCB46DeVF6VjUO3b8huNtI5lsGo3u6gcFVf+B5s/nP29nXz4IngvaIIguKjL3bA0JFYDAuSitDPEe5qg0tqt6SiHxv6O8v99Ou1xsMKY52UjtNjrdfgOfh0BhmU6x6pD4v8CqQ/lOAP9JKFX8VKCUFyr/4cSfWJucZ1PB7GcmYIxFlf1iB+a661BaGVT/HxK900lo6D3YKdeaDZDTAqch/lNxPgP/zqnuPgG4gsGlALHboemMLrWHlYTYHRQWdAH0KsfDSLovjLkU8x6ywVyWdYMV54fc21oJhTMunFcBQRzBlML52hsQOA3lOxJp+izYPVWJK6ZSq0bVvoFSCrGbkdjtWWEoOAIhCIHTQCIo/6ngO865Lfg5JPl/jm1F4tt+HjuynefJaZ1Z7KSp7vTm8oFnGqrir0W9bcRc60QySxIwkeTLkH4DtQ2PKeWZCNXPgbRmDf0eJwEiHpQxvNdz/ShQ+kBU2c96fl0pCtxvXfJwBYNLAZL4T3Zh6lhsk07VssC5kLg/e113VEehC6H9pu302LEYd8NcgcTuAHM1JJ8n/0TR36Ry+s+qGSI/R9AobrjVQCvPqsW6Xffu17nTVFolVD2CRH7t7NY9U1ElPyy+WNqtoA0DWUHhCSaXHVP/SPSvdAmF7LNm5jtR39qoIu3/nBVMHYI4AYn/IeFLtlnnWSkFKicltzG65zmJQPo1JPUG6MOziRT7IdjO5SPBFQwuhZjrKFB3iAWZOY5qRJWC71Oo8MWgDUTiD4G5ikJ1Sl6nRa5lIHoHjmF3V3uGFDuNbEv1ZEPbVVDyE4j8jM64CuVFlfwor6UyRqMq/7rN0UXSSONnQBq22Q40CJwN/h2IC7C3UCBglJGNFSkUDJgrKUxr7XFUh/2UgkIi1zgpxyUO+JHYX6H6CUeQuuzx9IvDt1LqJKXUMqXUSqXUVUVev1wptVgpNV8p9ZJSakR/jOuyiyjqV58Cc1HWs6beCaayW51MqpX3Q+gi0AbsxGAWvf8aKgp3973VEe+omkog/R5YG1HVT6NKrkSV/gxVM2vnVCepV533rjek3+r9LDMrshHIvu6vIMa+2PFHsJu+iN1yCZKZ77zkO7KwvZhgTOj1uNuck7nW+X50qsuSYLciuZX5XPZo+nxiUE7i+juA44GNwByl1BMikpsY/wPgIBGJK6W+Dfwa+EJfx3bpf8RuhuT/iryikb+7TiPx+1Blv3L02MHPOrvCnaK3p4ViahfBSUu6K9wlU5B8BlXyPTDO73lWIkjsboj/3bE1+E9DlV4F5mqk7adgLgetgt7lbbKdE4C5BDyTeh7TjiItF0BmmZO3CBvwZOs3gyq/HWm/ARJP4hjWFZJ6BSr/hgpdiKRmgbUOUI5QKL1hp2JSimKuKp5B9mNQ6tTFoT9USTOAlSKyGkAp9RBwJtApGEQkN7RwNvClfhjXZVeQWeIUuy/IANodO993PbOwl/d1xwfGlKzL504an5UGsovSY+i1nf8UO+6kC0+9CPpQVPh7KM9kJP53iP2x66SVeAyxt0J6thMhDVl1T29RdBciYtUh7b93Cth4puFUnluME+GbbaRVQdkdKO++ThR2yzfpErqCE+z2B7TK+6Dq/yDzHlj14J2J0qtyxqoHuwWMMdssWNQjnslFvgc+8M4s2txlz6M/BMMQINencCNw8DbaXwg8U+wFpdRFwEUAw4fvWZ4OnxiMfSis2dzxNem2cFv1SPI58J3gGCIL7usNKTDn06dFXSJ9u39bqDJnCJHsDj3rmmouR1IvI9rArIDMVb+lIPUKhb7/vRFeyhnTcOI7JPU60nYd2Ou6mljrsv1068uOo/Rqx1/fas4azrst0NZmZxSlwJsfDS2SQVp/4ERzq2wt6/K/9DqeovMJ9IFI+FsQvdOZozJAH4IK7v7IaJfe8ZEan5VSXwIOAopGw4jIncCd4CTR+win5pJF6YOQ4Och8e/sDtjv+Kx798+mf8jB/BBp/T7oI1ClV4P/5GwKi2IdV4E0U3xh7KvheRd+VVLPYDe1QOjrzmkqz8AuYG/expy6z0uDwNeyKauXALqTurrzNCGOm2nZb1FKQzLLnZxVBXEPNsVtK5bjSQWgj8KJAM51i/WC7zjEXOXEOnj2zSuAI7H7snmD0o5AkTjSchHUvumk994BtPDFiO9E59SkDwXfETvch8vuoz8+qU1Abs7godlreSiljgOuAT4lsk2nd5fdiNjtzm41oDuCQd8HFfw00nhGD3eYYK1yFjD/6ThfqSIeSCXXQuR7/TjTHsbZFWTegcgqem8kN8Az1Vn882SDB/wzoeV75J8wFKhqKPsVmr8rfbYkHqFnoalwvLk63oMABM7szH6rlA4VdzgLO+J4lRljIPMB0nh/NvW5DhV3d50Ikt2ys4IzvrkEPFN6+ew5M/SMBc/YHb7PZffTH4JhDjBWKTUKRyCcA5yX20AptT/wV+AkEek5qb3LbkUyy5Dm85xFRDkrmqq4G6WVI1rJdvTkSUg+2sNrGmTepn/TZPe3UCgDYj30K2BH6Z3xWDnRtuW/g8wypO1HYG8FVY4quxGJ3k1hlLU4bqytFyO1r3S5dMq2ntELpddB6nnHHhD4NCrwWec2sZyo5sRjYIwF37FOIF/yCYjdS65dQlq/BTVvAFpx+5CYoMp78dwuexN9FgwiYiqlLgaew9nC/E1EFimlrgPeE5EngN8AYeDRbGDQehHpaQvqspuQtp/SmXCtY+Fo+xFUv4QKX4y0/oidK7tpQ+JRdqnKp8+0bef1BM7PRVHooZWLhgp/B6VVYmcWgN0EBBy1TOIRJ01Dj1hI7H7HCwpQgU9nU0V3f899UHIlyn9s1hbgceJKsiktJHItJJ6gUwBlFoMxMhtE2G3xlwRYa5H0fCeIsduz4J3RpyJCLh9P3EI9LgCIJJCt+1FML64GfIhSfiT5nJOgzFxJ30tw7ilUAL2MLwBAc+o260OdTLFFCYAxDsx5hddVCKSx6F0OHgh+Fa30SgDs+H+h/UaQNtCGQvhilP84x/jdcgFdtgYvquph0KqR+pkUCABjnGPUznSvzeBF1b6GNH85W4in27NWz0IzBm9jvi57Gv1RqMe1Bn1CEZFsrhsd9MFI83cpuqNXpXQEQ4lY2WjavUQo6OOy78GOYDs1pSsfgoZDKe5imygiFLLX9XFgbkswZLDif6dJP4Wa0GS04JlI4AwgjVIdn4MgbVd2y7cURyI/Q5X9podpt6LKf440X0jXCSQAgVNQWiXFzX4exwhurs8m3NvgxCZYW8Az2XHX3cPyJLn0D65g+AQi1hak+avODx0L52vQw2LvmQqJR7D1gU6KiJ1SJe2J+LJCbmdsFRbEH3XqIPRUA6En9OEQvgjafp710ioiWMTktoW/YUjFeXx73AmAjUT/6izOWOA9KZvCPO8mJ8ZBqwV9cI5LKzjeSCeivNOh8j4nENFuBv+ZqGA2zjTwuWx6ko7PVwNjFNjtSPPnssFqOeozazWSehmqn8uLgXDZO3AFwycQab2i2055G+6i6TnZ0pBFql99bCjJ2r2TdAnAFMi2agxsh9jvofwuaL1gx+5LPQ/eaWgDnLQXdsPx2UW8Cw2ImooH177J0QMmM057DGJ307lop/5N0dOdPqyrNkPzhY76SSzw7IcquRwA5d0P5f1zwa0qdCFibXbclLGcZIDltyGRG7Juy0Wy40oKSTyOCn99x94Dlz0eVzB8whCxnIjXXtO3esS7Hy+q4o/gPQBpOg/MhXQtctE+9GtB2w924r4UtP8OCX4OpQKgCjOO2kDMMjDF5r3m1YwL3k/+Sa2YXVBDlVwNgDLGQM3Lji1IBVHG0O3OSikDVXYtUno1SLozPYaYq3sYDyBdWBjIZa9gV1RNd9nDEGsrklmCiInzkfdhp/yxw4Pog5CW72Q9gootcspRC+mT2aGfhDTv3JSUAmur8+/AOTiBaHmz4Zrh71Fq2AwMlNMrN1mtFuU7NGcIDeUZ1yuhkD81b37OJN+R9Px98TuGcJe9DlcwfEwRO4Yd/Qt201ewIzc6+W26t5EMdutlSMOxSPN5SP1hkPnA0Sd/YkhD48mQfn0bbbIpqq0l9F1dprP9jK+aYwcAVPCz4Ds2T1xpCmq9ST5bs4kjaydmK7B1z56ai3fH0nTvACr8HcfVVYWccQBHkPkg/A3HbuGy1+Gqkj6GiFhI87lgrsEpyjIHiT+GlFyO8n2qc5co8fshOYvOFAfEnFrL2yiwsvfRGw+qntuIOBv8XuM9wwnmk9xgQOXkHRIbECj9ZWcqCqU08B2RLX/apbbzaxZfq/0A1XQyBL4I/lOzkck2ePZ3MrZiOwFonkmocH9GlefMXCuFqv9Beg5IM2JMRUmzY89wayXvtbiC4eNI+u1sMFKHi6EFtEP7jUj7TUj4MrTwhU5O/IKCO1Hn1ODSK3a4JHD68SIXBaiCwKcgdCGaMaTrFbGyp4VC7yhFwjFMR3/tVE7zHQ/hb6F5JiCSdqq0aeWOTWEX4givbF7MzGIkdg9YWxH/yajguSi1Z6omxY4675E+EGXss7un87HCFQwfR6zNFK1F3LHzjd6KLYkikazQu7QOewsenOfdA7ypZINTFjU9B6l6EKWFHdtP87lOSottRoWbTsqM1DOQehE7fCkq8BmU14lhshNPQ/QWxwXVeziq9Oe7xIVUMoscA35HGdHMIiQ9B1Vxe7+P1VfsxPOOc4AyQEzEdwiq/I97rBDrQNLzkMSDIBYq+PndpqpzbQwfR7wHs+3FzoLYnx13xU80GXojFD664H8Ba4Wz4wYkcn1WyPeUo6nI/aQheivScDSSehVJvQltP3Y2ARKF1ItIy5fZFRkNpDPOoaPvJKReQay6fh+rL4gdzXqMJZ33hCSk3kbiD+/uqW0TO/GCE4GeeByS/0WaL8SOP7Zb5uIKho8hyhgOJVfgGCSL7YAs9pro5I+A3qiLRPpLgNiQnIWkXsuW8NyZE5wJJJHWHyKxv5HvUmyCtSmb1rufseooONkoA6w9zGU182E2e2wuyWyuqD2Y6I0UCN72m3eJkN8ermD4mKKFvoqqeQXKbspmv9xRZfgnGxFIWqrHxT5XEKRsjahlsDJRit0fv1FrGdJ6WXY32wck0ZX0MI8eMqX2Ff8JdHetBQWeif0/Vl/QBhTJTKvBnp4MsIhnoXPq/+jVv65g+Bij9Cq0wOlQ+U9cwbBjWCjejAwkI8V/Akplww0EViTK+PbKTzHYl0ZTFUB5H0e3eyEUFIQugfD1QICiP1WlQfA8x+Mp9z4V2qn6CdtDhS4E74E4xZvCoEJZvb13u/d+lCjPWPAeRJcQ00D5UaE9PELbsx8Fv2Nj3G4pcOQan/dARARJ/Btif3XSOPjPQJVc1vkDFHMl0v4nyLzvVFfzHY/jY7635DHatYiAhnBU+WYUYIvzcyymUtIVjA20MbNkK883D+LTNRv5SNR0KoTyTnXqKARPR5JPQ+Q6HDWShRNH8AOU/0zE2gixu5x8RqoGlA9pPAUJnocKfrkzHXefp6R8qMq/O9HQdhN4pqBU9xPEnoGq+AsSf8BRHxnDUaGLUMao3T2tbaLKfuk4I3QmNNR7Toq4q+fipt3e87Djj0LkBrp0xz7wH4dWfqvjtdD8RfLzGykcwbB3F8bb4ZiCfuw3Yem0mV4G+j6qFCF+VPV/8lxRxdzgbBjsdlTgdJR3/67XxESidzqbic7vjR+C56OVXvERzdmlO3b8iS6PMd+hqNLrUHptj+1F0o47uthOe7WtwMbi9EfabVcw7IEUS6wGHlTtO0jzl8BcXOSu/qyO9vFgVwmKnvrf1eN1YYDvMLSKu3boLnvrzCJpOvyoAfNQfZh4azrGy1sXAXDUgElUeMPbucMFQFJvZGt2dwhq3amPXv1Mnz6P7eHWY9hbkeK7UrE29iAUYG8SCjuyAO+KxdoSR6O/0/1qw7NlULdnAPZk/4vnXzbGoMr/VNBarM2OgdIzofhOUuKF10jTlVp9x1nQup6L5/yt0zPm90uf5vaDLmBqhVuHYXtI7B/ke4xZzvfCXAyeybtpVr3DNT7vifjPJD83jgZaDbRetpsmtOfSn0JBxMlqaosq2u/2xuo4fGesLbRY5Yhs5wb/CRQKdC/4js8LxBIxsVu/jzScgLR8Damf6bi7FvR3DPnuywZ4D+mT8fKXC/9DwkqTtDMk7QwJK80NC3ePb/3HjqKeYQpkz3cldwXDHogquRT8x9O5o1RBx9i3w9XG9m66q3l6S7GYBBGoz/j5xbqDdkrY5J5cPCpNiVaPvU1PsQCU3gy+Q3K8inyglaJCX8zvO/7vbM6rlOPNJDGk5WLEzj8hqNJfZD1bvE5fxsQ+Gy/XRAtjFNbFGnaLb/3HDRU6F8ejrPOK4821CzzG+htXlbQHopQXSq9HCEDyqb77u3/M2FXq147Fu6fTQI0nyQ+Gfojqtovvjbqq++s6oFSxqGvHFtRmlfDsyj9REryQ4ypOxJt5G4zRThoErSL/luTTFNbFSCLxB/KK5CitDFV1fzbTro3SB2570r1gSKCCjYl8u8WgQMUu1ZHvNfhOgvAGiP3J8S40JqLKb0EpfXfPbLu4gmEPQsSC9Gywm5DoX8HqXpx970XEUapoO7nedF+nii3mxday3HaagjIj3ePr26J7u+73ZMRDwg5RorWiFHhpYqb3Pr65pI7b9TLuPeTqbO2FIuiDinvIxu5CQhcUuKNuy+tlR/nR5DO58v1/YYoTZGUonasmn9Vv/e/NKKVQ4YuQ0NeBzE55GO0uXMGwmxBJIO23QfJZ0CogeEHWra0p2+KTE5NgCbwTGUh9JsD0knqG+GI7dH9vhUCx+7rjVTaWsM14wWLjWXh5qnEQf9s6mW8PWsDxFRvQc9pYtoVXRTrvC+gWtST4VNl6nmsZyV0rnuKnU/NVSJ3PEroQSRbJ2ioxxw1Sr972g/aBg6vH8sDhl/Bs3YcAnDR4P4YG3RrPO4IjuD8+QgFcd9Xdht38NSfH/XY9V/Ze6tN+Wk0fP183g62ZILYobBTfHTSfz9es6nfDcm9cT1szHpYnKhjmjzLIGy/arlhA3Go5jy/PS2GhGBto5c4xLxPQnV122la0mj5K9TR+PV+9VJcKUutNAArDfyKq/CYcA6WJ0sK8XfcfqtO/ZoSvhYytMJSNp+OAoIKo2nf3uMhjl93LHhPHoJQ6CbgNR7V6t4jc1O31I4HfA1OBc0Tk39vrc28WDGLVIQ0nsrcHpPWELXDJqiOYH6vGErAc59DO173K4onJT1Fu9F5obk/lY9mgb8fV4vHGUdy6aT88yiIjOidVrON7g+YRNqzOvkXg3fYaDi7NN8qmbA9Xr53OmxGnMtvUUCPfGbSAgd44b0UG8lLLMH47+k0CWlfem0L1mccpMWo7yepiagzKXEFQz78HnBrN6IMd19bQhW4lNZdO9og4BuVYUu4Ajgc2AnOUUk+ISK7D/Xrgq8DOVE/f+7CjoPS9KfRgh3i2eTjzYtVkpLgRzlA270WqGeqPM9Ab36aAyNhw15Z9+V/zSE4qX8e3Bi/Cq2yUcgSQKQqPku2+1VvTAW7dtB9p0Uln5/VC6zBmlNQzIdCc3dXDykQZs1qHsn+4Ca/Wtfv3aRnOrFrTKRjmx6r5/qrDsVGkxPmZPdM8gpMr1qErG68mjiE8/2nA7gpsDNjLEC1/5kp1CAfTSbVtrUdSb0H5bSj/0dt5SheX3tEf7qozgJUislpE0sBDwJm5DURkrYjMZ4+omLIHYIwBQrt7FruN39dN61EoANR44oQMi1pPnBWJMh7YOrZHF1NDQVhL02L6eaxpNDesd9xNTRvWWEcwN+oEYumq8N5c3ovWonfzIkrYHt5uH8i6dA3XrpvBeUtP4IIVx7I2VVaQfE9EYUrXPkthU2Jk+N7g+fiUiUL4/aZpXLb6cNYnw/x24zTS9rZ/fpqSoj/QwpNREoness2+XFx2hP4wPg8BNuT8vRE4eGc6UkpdBFwEMHz43htZqZSG+E+CxD9391R2C2E9Q8TKNcYJfmUys2QrCVvn+pHvEtIz6ArKjQYmBVuYG63moJLGvH46Fsgv1K7kP02j2ZIJ8UrbEOKWjlezGG3MZnRp7+o5j/C1F9ibvcpisCfBYcOv4pHG/2NjugSA+bEqWk0vPs3CUJKdi4+kcTgBLUHK1hgXaOO6ke8w3Bdl31AbL6UupyT9L06oWMMlqz5FXSrISF87p1Wtw6/1Q1rlPagmgkgCUm+C8mYD7PbsqmkuhexRXkkicidwJzg2ht08nV2LuWh3z2C38fUBi7h+w3TEMeF2/rdfuIGJwRY8yu7c4WvKUS01mz1n8cyIxsRgC1vaQkj270AvihXl7rwnB5uZGGhhQbyStBjo2Pg0i7OqV4F3Oo32QmArAILiWyuP5sdD5zKztBFNHwAlV3NqWYYTQ1djShJfjpppQvkkJlV+Dts8mDnrr6Yx48dE59ZN+xG3Dc6oWotXWQQ9Q8HeSpdHmo9GGY2dWcsAb7F0Fx0Y4D9qm8/6USGZ+UjzV7suqCBUPoQyhu5QP1uTbUQycfYJD0Dvp+ywLr2nP97xTUBuBYyh2WsugNhx7NhD2JEbkOTz2OYGJPE4e5hM7ne25dNwcGl9dqfdtTInxeA/TaMpNTJoKv9mXdlUGMke+zSUsCZZio7NhEALAW3bZTJFwOqWrkIpuGX0G5w/YClTQ42cWbWGf45/gSqPiUq+wLED98WrdX1mDZkAv1g/HRsBu9mpsew7El1TeUIBvOCdiVhNaMZwIoEfQtaf3ULjT5unctLCM7hpw3SoehRKrgRtEKgKCJ5D7aCHGDT8Qyi9zbmODsZYUKVO3QUVAGMsquSqbT5zfyGSwI7cjF1/FHbjp5HkrJzXBGm9Ihudnf3PbkQi1/a6/7Rt8oP3/8nZr/2Ob8z+K6e+fBOL2zb2/4O4bJP+WJ3mAGOVUqNwBMI5wHn90O/HHrGjSNNZ2WN+wskPj53dKX/0VZk+CuKWzqONY3g3UoslinozSECzOK9mGSdXrkdTzo67mANRQybA620DOb1yHT6ty+BsisYgT4yMaHiVjSmwKFZJUDOJ2wbrkmE2psNMDDZz2eB56NswNYs4Rmm7SDZan2Zz4cClXDhwaf49qdf40rDvM69lLR+2rMMggaaEX496C0NZQAJpvwWt6l9Q8Q+k7YdgrcVx0hOI/RmJ/gEp/Qkzqk7HEo1cc5tfMzl+yHFoejmEvuz81w0teDIET855DhMy85wduTGhaCSymBuQ+D/B2ozynwj+U/pcm0FaLob0u0AK7DqnEl3F7Sjfp5y4Cqv7ntCGzNxe9//P1a/xTuMK0rZJGohbaS6fex9PHX2Ve3L4COmzYBARUyl1MfAczi/hbyKySCl1HfCeiDyhlJoOPA5UAKcrpX4hInt2esGdQETAWgN4UMYwJPFYtlxfh2qgNwXf9yy63CO338YGvrHiaDamwllPnK5TwW82HkBaND5dvZZqT5LxgRYWxyuzrqoAQsL2cM+WyTzRtA/3jJuFX7MQFFFT5w91jtrlogELsZXG+EArAd3CtBVjA21MDLayjz+Crm3bdVUpx1Op9xiQmoU39Qq3DVdsHPE1Wtv/w/jAFjy5HkOWY2ZT3mmomuew2/8GsVtx8htlVVqRGyitOZYb9zuXn817EJEkpig+W13PETXjdmBOWXdV74E9vi7mKqTps9miL2ZnjWlVduMOjZPXp7WlSyh0kkSif3EEgwqA8hemcNEG9XqM5zfPJ2Xn/04SVpo10XrGlPQ9xYdL7+gXESwiT4vIOBEZLSK/zF77mYg8kf33HBEZKiIhEanaK4WCtQlpPAlp/DTSeAp20xcg/QEf9whmS8AssgHvrtbJCLzVNpC6dKjTPbO7qugfW53awGlbcc3w9/jWoIWEtDS5AiQlBk2mn7cjA/FpNn7NosqT5pLB8whpGa7bMIMJwdbO4DFDE3zKIm4bnXEK281rhOLvm8djSfGEel1oOMI8BSRA4gy172HfcCxPKKxPlfFW/AiaU1FEMthtV0PsZgriVJQHMh9wWEWIpyb/l7vGzuLJyU9y8eC3oO2biLV12xPfASR6RzZ9e8cim4DEE87iXqy9uRFJPIlkFvScIE+iTjnR7thtAE4OoPAVdCWO0wA/qvTHvZ53la+w1oMlNmWeYK/7cOk7e7ei+yNEWi/LFtfJqggyHwAf7MYZ9Z6edtgbkiGeah5BSE9zbu2qTg+c3PvAuff+reP5V/14knbPX6m47cneoxjpjzLIu5J2y8N99RPy2iVtgw2pMLY4xmcbxwA9u30gk0PNBTt+XaPAriACyxLlbE0HmRpupCIbC5G0dFYkSjl/4HJStoauBK8S1ifDxG2DsYHWTsN3zC7FoB1fntdQCrzHQPoVbNH4xbp9eaV1MIZukFn+a74zPMjnyp8ors4SiwQDeGrFzzm9PMXoQE5iPLEh+RyEvtLj+7dDmKsp8A5XXrA2Q7fkenb0zxD9E52qL+90qPhLYbpufR/H9pFXL8QPgU93/qWFvogYo51Kc8qPCp6H8kzq9bQvHH0MC1o3kLKdU5ZPMzikehw1/tJe9+HSd1zB0A+IJCCzkD0lTCNlKV5uG8pBJfVUe3qOrjZthSDYaHiyWvcOAfF22wCuWnsIGdHQEP6xZRJ/GfsKY4ORzvtzhckbkcHE7G2lZhAmBZqwoTMwzKfZHBSu5/768Vg5pwufZnFAuJGEbWDaQkq8fGHpidgolsQrCxbdpK3zausQxgacXERpW+P7qw9jYawKQWGJ4pIhH/KFmtU0ZnyMC7bgy258Y5bBxSsPZ3miHF0JAc3kD6NfZ4Q/xpxImOkl7d2y3AikX4XKx3h202xebVtKSixSpiN47ljXwpEBg8G+7kF5PtAquPr9O/gwWs3tG8/g8NLNXDn0A8qMtGN3Mlcg9ceBtIDvSFTJTxyj9jZoSUd5vm4+7WaCowZM7lK3+I4CcyV5KVfEAmN8/tOY67NCIed7kp4DySchcFZeW6U0qLybRNO3aEu1UeNJoAVOQoW+lt/ONxPlm7nNeffEgVX7cNtB53P3ylm0pGOcOGgaXxx1eNd80x86xmxzNXjGoUqvRXn23amxXHrGFQz9gNh7VlpspYSHG8ZQaSSpMhp6VK1oSjhq/lnUeJI8NOFZLDRWxMup9SS4eeMBnSohGzBF55p1M3lk4vOd1x6rH81b7YO4cugHDPe1syReQYmWps3uWEpzB1YM9MbJiNYZSPZhtJqnmkdSYqRoN734NBtLFCeWr+eAcANx2+DvWyfxfOuITptFRhQ/WzeD60a+iyUKHWF+rIoHG8ZwYtUGhnmjPN44igWx6s4IZoBbN+3PuEAb+4ebyOWPdVNYmqhwAu4E4rbBj9YcyoPjn+Pgkk2kbY0kWl6OI5EEizZexs3LJ5PqFuimIbwfrWGwb11Oe4XShzCr2eLtSGXW+QBeaR3MpnSIv4+bhWChEg93dZR8Ckm/DzUv92gwXhut54LZfyFjm2Rsi3tXv8oVE0/jrGEzUKGvI6lXskZwJ/cSZTehtG4qmcz7RaLwE0jqdVQ3wQBw77qN3LPqMMAmZPi4Ydq5HNSHQkDFOKByH/40Y5+C62JtQVrO7zqxZOYjzV+GmpdQWmW/zuGTjisY+ojYUWg6iz3Jy8ij4LbRrxHWtz2nFtPHtwctYEZJPbPbB/KzdQej4ey4MxRGJm9Kdel/M6LzRPMoViVL+cqyY6nxJAjrGSqNFKm0QVLyv1oKm3mxriygTzUN5zcbDyCZXbw9yubUijV8pmYNo/ztgKM+erltWF4Mg4bN65EhnLrwNPYNNdGQCbAmWYbCRhdn8X4zMihPKHTwu437868JL+Zde6FlWLcobEV9JkCr7SNhGVy55hDuG/9SwbOM9y3ny7Ua99WPJ53zrBnRaMj4qU8HqPUmnPdSG0zIXs/99Ud0CgUAE53ViVLebBvIzLKWgvlib3V2777i8aJ/WPYsMTNFR8KPlG1y69KnOXnw/vj0MFQ97iz8diN4ZxTWeQDQRxTp2ZeNzs9nduMK/rbq5U41Tyod5/L37+Ppo35M2NNznEm/kXzKOfXkIpaToTjoOkL2J67/Vx8QO4pEfgH27o86zTWiKgUluoWmcnPrFLb3KpPPVq9ioDfBT9bOJGF7iNmerFAovGmwN4YINKT93LxhP1YlyxA0YraHtakyIpaPdakS0tL9ayXUeuKsTZXxjy0TSdkaf6iblhUeTnBbRnReiwxlhM8RCglL563IQLZmuna4XizG+FsBIW57eLd9IGuSZdkRFKVGCksUI/zt6AVqPcXmdP5ueXm8jJhdGJUrQEjL8LtN+1GXDmXdS/PRlXB2zSoCmoWR3RQowEbjb1sn8dklJ3HrxqkosQmwFTAL0mgApEXnqbbD0Itm2bVB2otcd1gWqaNYFqiGlKPuU0qhvAei/CcWFwrgVHwzpuZUkfOCVoIKnlPQ9Jm6D0ja+UGDGhrvNH00dUNEMhSqa8U5Dbn0K65g2EnEXI00HO3oYnfH+EUW+42prvxL2yoaA45e3qMEjwbzYtUFhmU6/fy7rm9Il3DovLP5yrJjOadmBU9O/h+leopclZGNVlDS0qcsQlqGaiPBP+oncsWqQ2mzCu0RjRk/r7cNYmGskru3TOSnaw/OGV+w0FierIAiURAeZXP9uhkkRee82hUENLNzwfYqk4CW4YBwA0mr6yv/TMuIIsuqMC3USFC3WByvJCMGf948mYSldyblk6xRvMJI89jEpzmvdjmTAk1oWZVPJpuI74nmfXg/VkvcEh6sH81RZZvwq65FTMNmsDfGDcMep/hPUTnBceZa7LbrsVsuQZLPdXoNTS4bWhARolC0pRPcsexZ7lk1iy2J1iL95rRXClV5N6rkJ9mKY99CVT9ZVJCUGAH0Iu99UP9oag0o/8kUKjlUtna2S3/iqpJ2EBFBUi9C5DcgbX3sy/l/f9QdUIqiu8fu43V4IHmU1el9U2Uk84y/2dYULsDCdwct4HM1K9GVYCjhU2Wb+F9zvj7Yp0xqPEk2psPUehJcPvgDZpZtRQGzIwP4xfrpjPG3sTJZ1jmGwmZisJlliQrWtZQwJtDK3WNf4va6acyNORXJrG6L536hBs6uXoWhbBbEKnmpZRgeZTPIG+dfE17goYaxrE+WML1kK6dVrUXD2el3YCgb3dHud13D5pQKJ9J2uK+dRfEqHm0cy4pEOSdXrsO04TPVazvbhw2L7wxexEMNaVbW1eYpFBO2zuz2AUwJNRHSTcYHWmmu8vFk0yjSojEp2Mz1I99FK1oCFAhfBdYGpPncbDyChaRfhcB7qNJruGTCybzfsoaMbWKKjYbixEFT+da7d5GxTXSlcd/q1/jT9AuZXD6s+BhkS8kGP4cKfq7HNgBnDz+YJza+h5U9NegoSjx+pleN3uZ9/YUyRkD575HIT50CRVo1quxX/VLC1CUfVzDsALa5FRpPAXo+3u8o/VmMJqxvJxUE8EG0igNLmzByVEzjAq2MC7SyNF5OWgxUZ3R2IWdWrc5L+fDDoR/wYsswEtKhkhFKdJOHJz7nVFno1s3BpVs5t2YF70drKNUzmOLseT3KZkMqzJpkKRnRqfQkGR+M8JUBy5i7ekDBPI4vX8fVw97Hpzkqs8NLN9NueXmkYQxnZ9Vj3xm0MJv6Qghkjcd2jnA8rXItjzaM6VQVKYSAbnFgeDO3bzkWhQe/MjFFMT9WxZJ4Bbfu83rRMqIDAsMwlEk6x5XWpywGehJ4NZuTKtbj1YT9wk1cNngeFlq31Bnd8YEykPbfZ42t2Q9LEhB/kK36F0lLgMeP+AGz6hcRzSQ5rGYcF87+S6cNwBaLjGVxy9KnuGfmt7YxVu8YFa7lj9Mv4A/LnmFjvJkDK/fhsgmnYGgfXQ1j5T8GfEfjxAf53drTuwhXMOwIbZewJwgFEUfTKjYY2U10wtIJ6YVJ43JjFBQwIdjSeV0A23bUIn8Y/Tr3bx3La5EhVBoJ3m4fXHTsn607mFtHv9n5t6GE4ys28ETzKBRQoqe5edRbBWmuO/BrNidUbOCerZMJqTQj/W2MD7TyTmQAW8xwpxrqiaZRzAhv5bnWYsZR4ftD5ncGuQF4NOG7gxdy0sLTeT9aw9Hlm2jI+Hm2eTgeTbhr7MuEdNOJixCncM9wX5TfjHqTWzbtR106zPhgC1cNfZ9yT5pSNrIwPoWh3gj7h5so0dOcWLGB8cHWos91eM0kKjesJZP2kBEdA4ugbnJy5Tp0pPNz0lQ2MeB2XZtTkH4r61WUfxKMWTbfn/NrViUrqPKG+dOMCxkZrqU1HSNpFX4H1kTrtzNW75laMYK7+0HI9AVHGAS2285l53EFw46QWbDLut5WGodiheaVQMTyUKpMMqJotzzUegujrLvfFzLsrj6AtGh8GK1gWriZrw1cytHlG3mpdShL4xW0WH66u5y+0z6Azyw6iRmlW/n6gMW0mD6OKt/Ej4a+T9J2jNYeTWjJGLzQMoz9S5oY44/kzWN+rAoQYuJhQrCViwcv5NKh80lYBr9YP4PZ7QOxULwRGURIy1DriVPfaYQWzq9dQoVRGJ9RpqfRsXm7fSBvt3elYfAqi0cbR/PVAcsAJ0o7aXsoMzJML23gwdIXAKdYz/+aRxIxvYwNtAKwMV1KU2uQ20e/zZjstcI3GTzpV/j72PU80DCUudFaJgRaOKdmRed77CtQFxk44n1bAsIPviMgvpHcTLECrEuVIAiN6XaumPtP/n3k5ZR6ApR4AjSnu9ynFTCpbMg2xnBxKcQVDEUQSUPyOSSzHMylkH6HLmNsf40B82JVbEyFmRJqZIg3hoazmxSBVsvLm22DCOgmh5Vuxt9tYdEUbMmEmBcLMCO8pahQ2F65S3DUJ7Nah1GhpxjmjzLUF+f8Acs4o2od5yw9gUQ3rx0bRV0mzP+aAjzbPByUQiFUGin+OOY1BnnjbE35KPdk+HT1Wl5uG8Klqw7nt6PeYlKolYSlcc+Wifxs2ByOKt+EP6sKAvBraW4a9TZfXXYMt45+k1I9jYZwufqQG9cfyAutQwlpJnaRXEcisCEV4tjyDbzSNjTPhTQtOu+11/LVAcsQgeXxCiaH8t1DVyZKuWjF0WREwxQNI+f9Vghz28vYx9+YV2az63XAWkiJAd8ctBiRxQiwJlnCF5cezyMTn6Mm7/PRIfAFlP84pPU7Tg9SJK22tRZVdq1Toc2uI2mZgMXP183Ic7HdnGyhIRWh1l/Gz6d8lh9+cD8AmlIYSucHE08v9tG7uPSIKxi6IZJEmj7nJEUr9mPtBzK24pJVR7I0UQ6AJRq37fMaU8PNgPBO+wCuWnMISjmafq+yuGfcywzxxTr7SFg6/9w6nrfbB7JfqJFb9nmzUPeNozbpSa3TMXbM9vCl5cfz+KRnGZAtYTnAm+DiQQv4Q93UnNxH0HGCsNCxRSFZ/fzmtM6166bz17GvUubJdOrPP1VWh6FsLlpxNPeNf5F208N1I95lQrAFr1YoaG2BHwz5kGojkZeP6Kph7/NamxOf8G50AOdbSwnl1kIG2i2DulSoIIbh81XL+dZgp/6FUjAm0MbSeDkTgq2dQumPdVOI2x3us+QtvCKOu233gjq2kBct3vkOZU9jowNJHt93Nu9lLuIIuRtD4URtG2NQJZejtBKoeRNJPgmRG6Cby2pdvI6nmp7hvKrxhDSd+dEAP1s9lFarK2bAp0xOqNhIWeqviDqMmdWH8viRV/Ba/RJ8uoejBkwmZHw0XkMuew+uYOiGxJ8Acz2Q2G7bHvvoYacuAk0ZL99beSTr0qXYWS+bSiPJpFALuhJsgV+sn+74+GfXxQQ6t26axm/3eQvTVmTQeKd9AK+0DcVG8WGsmsXZXXDH2B15hpKWQsvWF+5OxPTwattgZrUMwUbj6eYRfKGmq/j8GVVreKVtCEsSFYgtxMQLKIb52tmcDmLmLJ42GvNj1Yg4doQOfJrNEWWb0RS8317NIWVbqfYkiwoFcOIRhvvb8zOXAn7N4tf7vM01a2fyh9FvEDa6Fum0rbh5/f481Toqmy6j43SnOL5sHd8evCjPHrEwVsVP1x1MwvYwNtDCZUPmsTrHQyp3NpcMmsc5tSvzBECnNxnbOpH5IPglKkq+zwnKi8hF2TTZJWCM7zSaKi0MgS8gsbs6M7R2jNGWER5e9x6vbk5x3/hl7B/0cOWwBq5ZewgAYT3NveNeosqTxpN4D0neD/5TqS67kc8M36kiii4uwCdUMIjdisQfBnMlyndENk999q1IvUxvhEKHAThhG2hInoqh2GJhCaxKlHLh8mPIdHvbB3rjnYVjWkwfMStffSNozI9VIQIvtQ7lgYZxLEt0+Zkns3PoEAbg/D9jK7amAwzxJfLiFETg2eahXLdhRo73keLuLROZ1TqUe8e/iKYcg+7tY17Hzt66IFbFNWsP5o+jX+OC5cfQZOYbAIPbKJDj5M4TajyJHj2eLNsRaOuTYao8ybyTjlIwLdTIr0a+XVCb2asJJ1dt4KnWfXJcTx3h8LmaVXlCYV0yzJVrDuuMzF4Yr+KbK47GwlEZ5c4tqJmcWbWq8z3Nncv21XQaKnyR4wpK1iXUO915LySNJF8Buwm8h6OMYVBxN9L4GURizolDwT6BCN8evJA76qYwN1rL9JJ6Di/dzEBPjC2ZEJ+pWkWVJ9EliCUBicewPQehBT+zrcntFCkrg1czXE+gTwCfuAA3sVuQxlMh+kdI/hdp+ynS8m0nPsGOQPq17feRXRR05exk58eqWBjrOVfLc81D+Mayo/jK8uOLpppYES/Fq5zFq1RP5+m3OwjpGdotDxnRWJcqyZ8Pju6/+wKWEp376ifSZnmwsou7ZcO/G0bxiw0H45SaV+SqhzamQ7wdGYBpd+2MOzxppoSauH3064T1DN8YuDgvWMuvmXy2eiUZm7wi9ylb45XWIWRE43ebDuS4BWcSs4xOYQOO0ExaGigngd7UcLOzrGddS01b8c+t4zhmwad5LTIIn8pX6YjAPv5iMSWqYPF+snlktwhkp0FAmVQaSYJaBr8y8SmTnw6fQ8jYCbuSqkZV/KVokJjz/TsJafsREvkVdsMpfLj5bzy6qY6F0XxB69NsTq5YjwAtZldA4OiA86yTg815p7PsCBD5KZKet+Pz7oFlze8wb9VRsHUKWzfsz/t1f+y3vl32TD55giH+ANgRurJJJiD9DtJ8PlJ/CNurEwz5O0VDCQeGG/j5uhkF0cjtpofXWwfwqw3TWZysJncRzsVGoyHjLAoeTbh48Hz8ykRhY2BhKIszKlezNhGmLhXMCpHcwRSvtA7Ji+oF58N9sWUw16+fzmmLTuPkBafx1y2TmRetKjoPcE5Ad9Tty6mLTuOC5ccwq7XLo0VTMNLfjleZnFW9hl+MeJf9Qw1MDTZyXs0y7q0fzxuRQXg1m6StkbI1XmgZxi83HIizH1ekxeCC5cewJF7huI0KPFw/hozo6MrZ/Xs1uzOdx7pUmB+sOZQ7Nk/FFI1/N4wpmLlSjkAN64VpJf7dOIaE1SWMM3ZhZLamhJBh8sCE57hy6AccW76BrwxYwlFldUXfo5StMat1CC+35rr06hD8DlTPQtW+ifIdUnCfpN5GGs8CayNIDJEkP1pzAH9YMZtD+S7jg02kResmNBXDvFGOK3eC7rzK5pcj3+Ho8iYWxGsxpdihP4PE/lZ07jtKNB2lInYRk4J1eDSbWm+c8dzBmuan+6V/lz2TT54qyVxOQQEVkpCZvdNdtltetqSDJGydlOi83DqEVfFS4uLhiNJNqG26JAqHlm6m1pvoPIl8pnoN4wJtPNc8DKWEM6vWAHDZ6iOIWh7MIjl3Hm4cy4zSevYNNmHhZB29bt1BjPTH+CBa02lAfrB+LEHdpKfI5hPL1/Fq21CSYtCW8HPduukkLJ1Tq9Z3tupQ8XyqvI5PldeRsHSuXjsTUDRn1Ut+zSZjK4Z4o1wzbC4Z0Xi8cR8WxKvZkglz4YpjqdCT1BoxLKVzZk40cQeWwOON+3D+gGWcXb2aB+vH0mZ6acz4GOjL98LSVTZSeXMt1sIwtOuoygzP7juCWk+c8wcsJaiZfKpsE4837ZNnUBdReLBpN338ftM00qJhicZAT4ITKjbg0aQzMM4GZkcGcsOG6dgCpfobHFDS6tQdKLm0RzWLpF5BWi4ht3DTnGgt70VreGLy0znBidJ5uktYOv9tGslv93mzS0GmwK8srhv5IbGy/2K0nwt2kRLrdmvRefQWMdcisXvIxOdQoqfy1Ho+ZRFv/ztUntKnMVz2XD55gsGYBMyiUDj0nqStdR7hE5bOPVsmEtRNVibKuHT1EUwONPPrfd5CUPg1kwNLnuHC5ceyMV1YncpQNpcNmY+uulQngqMmmBxsds4FAucsO4HGTCBHB96Rx8j5Oy0631t1JKP8bVQbSRbGK6k2EtRngnmLYAaDNkunuFBYy8ttw/O8epJicM/WSZxatZ6UrTEvWs3WTIDTq9YBjk3g7i0Tebt9EDp2nhuoRxOm5aS5Prp8E79cfxAvtjrpGVosPy2Wr6j6TATqUkG+NcgxHIvAklg5DzSM5zNLTmVKqJHrRrzb6UUF8MPBH/ClR74AlnMyk80aVpOH+7wT+Gf9ODSEP4x+gyuHvs/tddNotzzUeBJELC8nV67n9ropRCxPVsUGN288kEpPkpklToCYhcISjXvrJ5CwnQjxZ5pHcGBJI8QfRdLvkSj9Gwvbm6nyljC6pCtiW9pvo3s1v0WxSg4MFyZg1JVjUH+4cQx3bZ7Mp6vXFKjEPESp8Plp5CFWbDiPkf5GBna+FwFUjo1B7GYk9i8wl4H3CFTwM522j2I4ZUHPBklRpqwCJ21NgeGaGfZqPjGCQSSJtHwnW7N226kjbHHy8uemmBCBlGho2GxMhUnaOpZoPNwwhtcjg/lq7WJu3nggCdvgmhFz84zRJXqGK4e+z6Wrj+w+K345YjaDvDFsgTXJUhKWzoRgS2ekrBPT4GNzOtTNaJsrILqur0mWsYYyNGw2pMM96AoLhYJfmXx/yDxeKhJpHLG8tJseXmodym110xCBWm+Cg0vqidsG61Ol6FgcXLKVicH8+IBcu0dAWVwyeH6nYOiYS8TysihWyf7hxk6PqpStc8P66RxVtomHGsfSlPFjoToX7QWxKi5ddQQPTni+01g7PtSKKsugTYtBiQVxhT0/TLgBEoPARuey1UdwYsV6fjDkfWo9cX689hBM0fAoi8WJys7+wQlMGx9o63Q/9SoBLH4w5AMuWHEsToanjgdM8FZzhGve/y2a5sOybfYtH8atB56PT/c4huZujPRH+CBaXXDdFpjbXstfNk8BYF2yhEnd4i7aLY0zXryFlG0R0A8jY6f4dNU6LhuyEBX6PPjPcD5Zu9mxqdntQBpSbyDJ/0Hlv3o+3UTvBEnSEXzX3diesHXCpV8teq/L3sEnxsYg0Tuc3Pak2V6ltYxofGP50Vy1ZiYZWzn+6srR71riZNn0K4sXW4YS0ExuHPk2jzWNYVWylJBmUm3kezVpCvYNNXefEVOC9RwQrmdTKuSkpdg0lfHB1k6h0HFvQDN7sAhAT7YCQwlTAw14lIUnz1jbff/nnDzuGjOLco/J6EAbWs77Y2ATtwyOX3gmN2UFX1IMHmt0Eqd5lc2ieAU2iqaMn6Pmn8XxC87g5vXTimaArfLk7pqdBidWbGBisLnLHRTYlAoxP17FbZunsTUTxETPW7RtNLakg3mG+DXRUrRDIqhSy1nMQ4I2sx3lsTtdg03ReKp5JDdsmM66ZAm3j3mDYb52yvQ0Y/2tqJz3J6BZlBqFdot9Ak5aa5+yOCOr5kvaGj9ZeyAJG2JmiqSd4f3mNVw+914akxHwHQ/k79KPKI+wODGImOXJq6utgANLGpkYbMGrLD6MVne+lyJOZtyfrj2IhG1iI8Qsk7To/Ld5HO8bD6GVXtO56Ev8IbCjdMVIJMBc5NRpAEQsxFyP2F0xMk4ajm4nuOzY7ZaPzfq3GVJ+asH74rL38IkRDCSfpbv6KLeGgYhT5jFh6fx24/6sSZXxVmQgD9SP69z1tlseHmvch1H+dsYEI1wxbB4/HTGXg0u28qXa5YBjvC1W97guFSR3UVbYbM2UcMLCs7hizWF8GK1iWaKcYgu9roQzqlbjU73JOy94MTmnZjn1mRApMTCUkwxbw86qbPJX7D/s8zpjQ04OqF+NfJshvhh+ZeJVFiP9kaL1i7veO+Fnw+cAsCxZQdI2aLe8PN48hmWJsjzhINnV5bDSfKPuOTXLO5PcpW2NunSIEiOVPSEVN9iTfQoNO7tYatyw9qCCpkpBtEoreOaU6PyrYTy1nhh/H/cSZ1at4XtDFhDIGv1BSNgaUauwXoNXWdwy6nWuHDqX/bKqsuWJioJZ2ghzmlfzhTduY7PxNfBMBfygQqBV46l6gIePuIq/NRzHqkRZ/iZEE/46bi4vTPkvp1St479No/hv00jmRmt4omkkq7J1KHJJ2iZvN2/Iv2iuoqja1NqApGYj9Ychjacj9TOx2291Unr7jnXmmYOm/Kja9ygdPJ8xAy4p7K8bIjYie0apW5cd5xOjSkKrBmtd3iVTnJ3nGy0DuLVuP8qNNBvTJZ0Le1oMnmsdzvkDl5G0Nb66/DhG+SIcXLKV1yKDyYjO8eUbGO5rZ0MqBNkl5Xcb9+NHw9/Hp6zOUKvf102jK8W0oFCd+X/Wp0r5/uojGOKN8lLrkGyqCOdH1Zrx8JuN+6NEOKykjjfaB+ele+hOSGX449jXGeNvZWyglZ+uOyQvrYXV7beqK5tliXJmlDq67sG+OA+Pf47VqVKCKoNqsbgwdhwtuh8NR8/uUxafrVmFUrA1HeTqtYfkLOIdKN5vr2F8oMuNtMPF98aRs7lg+bGsSpbhURbDfE5un2eah/GbjQdgo4qeNnJRYqNvTXNNdDprAhVolk3aY/RY1iA36K2DDelSbtm4Hz8e/j66BiN97UwJNTEnWtv5PL9cfyA3jpydF3CnKTi0bCtJS5G2FV5NqNCTRQvxAMTNFPeumcPV+z6AmGtBomBMRCmdGg9cPfkkpG0W+Q9t49XLWRw3uHjlYVQZCe4c9wp+ZbFfuIEyI8316w7CzHF/9mseBvnL8x/ddySSfJG82ByxEH0CtJyTH90f+wd4pqFC5yOpN7IBeTpOWdDfoOml2/pInK4lhUR+Don/AYL4T0aV3YBSbtK7jxP9IhiUUicBtwE6cLeI3NTtdR9wH3Ag0AR8QUTW9sfYuZi2xdN1H/B6/VJGhmr4/IhDqPE7X2YJfptlm69GVynG+NsQYFbrUG7esD/xbERvoxkq6DOgmdgCzzcPoyHtJ2oZfGPFMU7BFhz/esnTNcOzrSNYkyrhM1WrmRZuZIAnnhe0JnkKCwdbYEbJVm7ZMI1NdX7OGLmeJWYl16ybiUc5abAHemL8e+Iz3LVlckENBHB2zxOCLZToaTyacFjpFgq8j7IHho54Phudu7fsyxFlW3g3UstdWycTtTyMXb8V8/pmIo06mtrAOV9v4+Kr1pMUg1WJEqaGHZ33vVsnZJPnFe7qh/ljRYPADGVzauVa/lU/nmuGvUdAmdSlg9y04cBu3kI9BZEJkoG45mGFvyobaKGDdER4O++ugY2eDaobG2jh5bZ824YAL7UN5zvWQuriYW7dOJXlyYpOtRPA65Eh3Lxhf34y4v2CWfh1IW1rrE6EmR+rxlB20ZKiFjYvbVnAWcOmM7F0RIFu3/LMRNG95I4fwlfwq8VPEbc9XDpgHiV6utPoe0zZRu70TmZT2tmQeJRO0PBx8uD9u03yVEg+DanZoDSQDIS/h7I35qnmHBJI4v/Q/Megqu5DMkvArgfPAU76jl4gkRsg8RSdbt/J5xA0VPlvenW/y55BnwWDUkoH7gCOBzYCc5RST4jI4pxmFwItIjJGKXUOcDPwhb6O3Z0rP/gXc5tWk7QzvKV0Ht/4Lo/MPINg7CcY1jIGez38pW4yb7cP5Nyq5fx+y355i0AhQqvppTnj4466KZhotFu9yzuzLFHJbXUl/HjYXKo9yaLZQLujLUxS9qM1PNwS4mFtIslzKsl8Qe/M27MhFeaeLZO5Yug8XmwdlnMScBbCjvQYX1p6PDeOfLsr948ImI4gkwYPalC+3lwQ7q8fywutw53TUsZm8w9iaNEuYTbrb6UcMK2MI05tY0qOIXR1oqTH93BetIrI+jAPvj2VjKVz9oGL+Oz0RWhAtSfBoxOfJaA5qbBrPQmOKNvEiznGb5XzbCq7h7ezQXkhPU2yXMfqHh6dkxLDRMcU2JwOsY8/Qpmeos3KV5F4lEVdKszYQCuDfXGWJgsDFbsHFObi1WyqPUlu2nggAAYaZhEbVruZ5Buz/8qocC1/mvF1Sj3ODvqx9e9w+7Jn2cd3CL8Y8Q6DfVGUCkH4B2iBE1mdfB2w2TfYnOcJ5NGEO8a8ymWrTyHgHcIBlaP40qgjCmovK2WgKv6KZBaDuRa8+6H0wUhqNt3Va6CBVt51r2ciMLHHZy9K4r/kq67SkHwKkV9vM2JaMkuQ9l857uTGVFTp1Shj5I6N7dJv9MeJYQawUkRWAyilHgLOBHIFw5nAtdl//xv4o1JKiWxPYdB7lkc2M7d5dWdN2oxYWJkEm7Z8i4nBJhbHK/nl+gNZkyxFEjq/az0IFbR7Ul8DILZi/eoqTm0/A+WVbbYtcjeaEg4r3YzCyaTa9VJ28crpz4oJ//tOCBJd5wn/Q83Yo32YBzknGROdt9oH8h1zAcHlXtpXV4AuqDFxtOEplHKily2Bq9fOzAoUpy9pA2nWUSMyBYcIQwmL45WdKjR9YZLuwdfJuM5zD1ZyxKmOl44l8GG0hhXJCgpjIgQN4aG5+yILw6QyjoD5w4uHUNdWwqXHzWZ6uJ5QjteXoYQrh85jVuvwrtOX6upXw+bAcD1p0bly6IeM8LeTsTXuqx/P37dOLBg/928LjTnRWo4oreO5lhF5p5CM6AzztbOpsYzqiMlQI0Kpx6TBDNKQCeBXJjNK6juXUBF4deko3lw5jKEVEc46YAklgTQg+JSFKT27gWbEYk20nr+seJ4fTjqT+S3ruW3p0yTtDAvjZZy95AQqPPDI4VdT5nNcm4cGq1gXa2BBrIoR/va81CblRoqHjvhxr1Q8yjMJPJOcBTizDPFMzapXU3R56XlRwS9vt69tD6QVypvtmDLFqkeazwPJGsDTryJNH0DNy04uKZePnP4QDEOAXIvXRqB7Bq/ONiJiKqXagCqgMbeRUuoi4CKA4cOH79AkNiWa0bqlY7YRvrfqcO4eO4vvrTrC2WErUEG7c23uCRGwXi9DPyTiCIUeGulz41j7Bbs5dgtVeoJfjZqNjeKatQcTtz10LJh2wc5J0D9MYKv8n5BKCZ4XI52CASBiernxf5+iYWklZCOdZUEJtij0kV0eP8mOimoxi+Cvt2C87+iSMweHiF8+EAlo2Q22oCP5mUMDWZ/R3LkoIVTS1WZ5oowr1hyWl4W04/l+NvxdhvjifPfFM4lnuk4dyYyHR9+dwsXHvEuJURhhHtBMwnqKiOWncHHXmRsdwP8mP0m5kXZ86XWLr9QuY32qhJdahzHIaOekyvWUe9K80jqUD2I1nfeX6Wm+NmCpIxhsQTQnncnZFau47B+nsbqhkv2G1/G3s1/Eo1touvBWZACrkmV8beDSzpn87PFjeGXpKBIZLz4jw/1vT+OHX5qFT1nUeBJsTG/7J5URi6XNc7EjS/BE3mV6uJTXI10R1Cnby1tNKzpVQldNOpPL5v6De7ZO5IiyOgKaiUfZWGjcuWUqlw7qpYpH0kjLRSxcvYE5a4cwqKyNo2d8FR8LIfUm6MNRJVegPGN71V+PBD4H8YfpitnwQ+CMbZ8WEv91VFxdV5y/k8/DLsj55LJ99ijjs4jcCdwJcNBBB+3QaWJK+XBS0t1rR5ERjTu3TOr9HEyQNgNEoYzipwTVaqLaLDzPRfA920bq9HJS51U6wiHrANPSZHDz0kmsG1fTLQtpcYlkBgx83Z5YNJBQ/m4rmdF5adEYyBWClkJWBCFHMHTo5wO/r8f4IE6Hx6pnTpzgn+tJXJGtk6sU7baXRXHnRCNpRTpZjT/QiJ7uOjl4/cKnv+HI8YSlc8dtw0ker3f7BjmZV48q34xCsDKF+nbT1ljUXoGkbKbWtKDlPF4s69HUKUC7vU8WinfbazmpcmPntUbTT0CZjPa18vfxs9CUo2w6o3It926dwD/qJ+JXJl8fsIjF8UpH3izxM3PSRr48ahEvvTGeqUO38sOTX2fCoEYMvetDODS4hYN8Wzujftc1lTFryWhSpvPQKdODbWs8PWcCw8a2szJVUfC83RkbaOUvo2dB3GacF349Ct5rr+F7q4+kwwPLr3UJ0/19L/Pg+Oe4bv00zllyPEeWb6HSSDInOpT9a08sWHBtW3js9fn8540FeD06Xzn+II7dfywSf4A/PKV4ZM6JZEwNr8fizy+v5P6rL6e09lfbnXdvUSU/dGwXiUcAAf9nUKVXbfsmiVIYW2Rlr7vsDvpDMGwCcq16Q7PXirXZqJw0pmU4Ruh+o9pXwjEDJvPClvwqa6ZoJK0ij5l1t1Q5gWTSpmPPD2EsTuBbtBVVZZI8sAY8ucZbQZ+XIPjbLWA72g7/oy3YlTqZU8pAd5LB2RUeVltVsNKE0Tr2Ri/2ej+YCjUwjbZPotMADGBNDWCX6Ghps3MRx6NIn1aeP2+7h2O5qZCO+xw/TkDwzI6SG8agMoLn9SiJK4q8JQLWa+UQ14mcvi/Bt9fi29jK8BExvv2zjfgm6vxozUzej9SQ7CED7bpUCefNO5Z200uq1oRNRpcQUza+iiSX3DSVCfPquOWRNnRd0HSwDcWN6w/qNIj6yJDAQ3dPp1s27c+EYBsj/e282TaQq9fOxBLFDSPfQVd25yIe0C0uGLiElclSzqxaQ0Cz+OHKQ5EtXsyMh3fNGt5Z9SmGDorxqzFvMdgTx7I1jJzARJ9PWBUpY0AmQZUnxdrGclS3pHUZTfFGZgB6qrBQUncUih8Pfb8zO2zHmn5QSQPTw/W8Fx2ITzc4tGa883mk50L0Dwz2JbhjzCv8c+t4/t04BtGq+PSwmVww+uiCMW7/7xs8/MqHJNPOQvvTfzxLLJnmwJrXeejdSaSzQi2R1mmwNB6aNYuLzug/c59SHlTpj6H0x72/x38yEvs73SPD8R/Xb/Ny2TH6QzDMAcYqpUbhCIBzgPO6tXkCOB94G/gsMKs/7QsdfG/8ybxav4S03bX78CqLwyPrmG90izC1wV4SQg1KOTvuVUH0eQnKnp+HSmXvXw2p5nKk1tOlKkoLvv+2oqx8Vaq+NYN+VyOZQ8OOaklTSKmO57V2kplyZHkou0AK0uLBWh5EP7IVnxYnrXkQv0bslmH472zAmJfArvKQOGcA1kgv0mAgTV5UyERfHoOmdqzykpwc2wJDE5BW4JN81389axDIpYd8BtLogZQGopCAl9gx40gqiwkzFjHokPV8adnxJOxsoNmxHorq40xoui+F7/+2EvhKgtTx1U408wYfssVLenw7pb9tYW3az/kzJ/Kp01vxBoSXGwex9sIulUoC5+TQnYjl5R9bJ/Dz4XP41cYuL6Z9/JGCgkSaEvYPNfDr9QdQnw4iTR5kow/tgHYcc4piI0EuX3c44xdluP7Ts/K/IgJbzCD/WDeBa0fNYeSA5s7TAgA+C/2oFihQNRa+LwqYWTWW8aGnimowjy3fgsd/JFdOOt2JlAYk8Qwdi6Wu4KsDl/HVgetRpdeggvmLpiU2S9vqeHD2HNLprhGSaZO/PjmbK08bgUe3OgUDQNoy+GD19pNG7mqUZwJS+nNovwHEAuWD0htRevG64y67nj4LhqzN4GLgORx31b+JyCKl1HXAeyLyBHAP8E+l1EqgGUd49DsDA+VcO+Wz/HLR49giiJh8Z9AKBr2XRN1Uh3bpQOxRXiShY88LI7ZyFsOIBzIWpc8uQWVydoQC4Ss3kvh2DeZ+QbQmE//fmzCWOj/W3B+498k2MMH7QoTUp8tJfaUaNMdTqEso5NxlAY978T+2FL8o0keXk7i4iviVgyChQcBG25hG/08K83AP2pg4ygB7oAcVSaGe92A0JAjOXocWS2NO9ZO4uAYG5AZkKdInleJ9LoJKZY3QPkXqlFJot9CDgqXnuJpmCpcsS3SaowEeaxxN2ta6XBy9Gog4AVG56a0VSKlOZkaIzOllaD5nB66NjyMlFmqd0OF6395i8OR9jsC2hijHdy2PYkuoYnG8gpaMl6ZMlwfO+9EaBnljefEGaVvjr5unkNwQwF4QhoyOdnBb3klN0IiYXt5qquXD9YPYb/hm/B7LScth6tzbMJHlzaXc6pnGK5GhMCEOS0OgBG18DLzSeerMJ184CJCw0+jGGDDn5z+RgrNGnktJ+1RuXvxfwoafc0cexmS9DOcnmrN4Kw20/OC25ZE6Ln3vH8StNNaRabQmD/a7ZdmcUdAciXHPrCAZM1+AeQ2baWNGF5v8R44WPBsJnA52M2jVXfVRXHYLahds3PuFgw46SN57772dujdjm2xOtFLrL8WnKcRcy6++8jiv/+d9zIxFZmaQxBWDwAPS5sF6pwLPymbCL69AyxS6Gm7HTl3Y3gPpA0OkyoeQGV6VE2DVDcum6m4nq6sYGunhpejxNFKhMMca+J+N0HbhJDg5nb+YmaCeEkrvWY0ynfmKBna1QfSekfm1PC3B968mvM85aRzSp5aROqfSaZMR5/SQbS5phfV8VeeCAuD3ZLjx7Bd4wTeAF1qLOwSIDdJsoEpNlBdI2s4pJZRvYxALjN8nCb26gdwgbtEgfUQ5yR/VsL13WwS8ymTwplbWDq7uFEiVRpK/j3uJsJ5BIehKuH7NgbzynwqMOTFMVUpq1GD0o2MwIF+f7cPEnl2K3eTjy4d+wLGTVrO1Lcxd86exbIwPbVMa1WphTXZcTCWmIY1e1OAkqiAwOuf3FLPR2izsAR7QFeWeIM9+6ovQeDJ5KSdUGX9p/TkPrZ9D0sqgAJ/m4eb9TuFg+VY2CE0AHbQqVM1LOKFBICKc8eqv2ZrsCiQUC2R1AHtxGC17qrRtIeRLkTYNTEvD79UoD4d58OovURrKd3F1+XijlJorIgf1pY+9Uix7NIPhoS7VkfKM5eoHrmTRm0t55NYneeeZ90ndZmNNDKPa0lChEL+nmOaiE9vQUGZ3c2gPZCBj1JIZUtFThBYAWrTL31uZNt7VrU7/W8BYrhClkNrugU9OcJp/bgOYOQXrbdDaLfTFCawpQSdF9MoAss6POayS+E1xtGGpbtMRjNejGG9FSf5oIMoL2vQI9tySbCCcYuiEBvwD4xxmbeaFlmFFn0dpQImF/X4p+swI+LUCr6aOZwws3OLYd8ARzEpDPAaxIwejSSYnQC1/nM79SwbMiMG4sjbWSVXnfJoyfs5edBKH0kB5eYLZsVqiP24lOL8BlbHxemIEFmyl/fQRjiTqfA6h1EjT3qaTMA3ufm06d782HcMwsafEUWYK/b9x/FP8XHfAV3ho3Vus8G5h31HDKPeEeKbuAzKSY8TJxo347mrA91QENBC/RuKaQUw5egKaMQq75lWIXO/47HsPJRX4Fg/O+xOprApUgKSd4U8r5jBzxsNI+2/AXAHe6Y7nkOqKpdmUaKYtnV+bXOmghqTxrDAwdEUs6Zw4YinnPkPT+PrJh3DOMfsT8Bam/CjGskgd1y94jJXtWxjgL+PKSWdweO2EXt3bE5HmdnRdI1RWGFjqsnvZKwVDMZRS7Hv4RPY9fCK2bZOIJomLzYU3PcjGlnbMgSVIyIu0p1DZRU0AdEXskJHYYR8lzy3dpvDIJTW2BoxuXjm5pzPTJvTmmvw55v7bEuyggWdeFPPAbqU+TVCxIvtqBSrtjGEtCMH6QOfuX+aXYFsKfVSOgU9AX5lEeRQkBQIKbUAadVITRHVsC9ZWeLhkzZF8d8C8bT6vJHSk3uvMzQBMQTTVddKxBa3RRNua6Zy3FfAQnzGK9PAKVHWUUpVieCDKskS5kzTPpjOThTQb2CuC0OBFJsd4X6+FtCA+IKFjrwqQrPMyK1MLCjw1WyhZmEBlsgFyGYGYRfi76zEPCJD6cjVSpoPAb8e9QfqLAa58+ESiKS+2rTht2lKe3hjEXuQh/eVKAqaHgOHl9ukXdD7zskgd/9s0l64vhfNkxjsxjOUpEt+vxS438L7cTvCaTcz443FkpmTweAdARVcVtMZoIxk7vyodQH2qDVNGYfpuJ1BZfFcf0vykN+lYm30QstFGJlA+YUx1Lb/+6Ze5+m/PsHDtlrx7PB6dw/Yd1WuhEDWTfOvdu4iZzkZmc7KVH3/4IP845Dt5qcV7S0t9G9d99rcsfXclADNO3p+r/nUJAffkssfwiREMuWiaRqg0SAi44/LP8blf3EvatGg7cwrB2WvxbGhFZUxih+5Dekw1eJx0C4lpQwgsqEN1N+bm0LlE2FIoQwQ8a5rQYml8S7ditGy7trQd8mIsjmHOq0ammWRDf5FmHYkWCSuzNdJDSyACrA0UurQuD0KuYLDB+3oU0RThH2xA25LBHOcn+d1a7AFeWBJAwnGUDnc9OhKOyBkrKwDEzs5pUc6uL2XDljQqoCNVzklMazIJXleXN18tZpIeXQ2WzYwNy7nl7PfJiMZ77bX8bN0M0qIjlsJeGELWBzsnrSrSNGsB5HEf6swk5uwKaM9f5NQbHsRWeclHVAa0ehPfs+34nm133h5DMXxJlMDQCE9ffh91LaWUBZMEjTSvj5qKpWm0Tx9BpNrm4jl/p8obJmVnOLJ2EjEzSbHgPvPAIOYB2dgWTZEY78ec4OPeS+7nmd8+S+WgClbPW8fwiUP49u+/yvXpZ7NuzF3oNgy4J8ZZp52PZVqMmz6Gnz58OTVDq/La3fLg61jvlyKmgCZYqwIEj23neweeyNCacs4+Ygor6xo7vZQ0pagIBxg7pDDdd0+8Ub+U7irnjG3y5Ka5XDphx4v13Hje71kyewWW6QjDOc9+yF+vuJfL/vLNHe7LZdfwycmu2gPDasq590fnMqGyHEPAGliKEkEzhfToKkcoAChFcsogElMHY3t1bF3D9ubnHRXA9jnt/Yu2QCZnF5ix8C3dSumLywm/vRZPS2KbaqnYzBFEzpxC7MjRWOvLsf8dwnrfjz2nDP+NjXjrInQpQ8Aq9dN+3HiM+yz0u+zimcXTCsnW01QtJsGbtqBtNdE3Z9DXpFEJwZifIHT5BtT6JGW/XIzxTBxtQQLP/9pRG9KQtcHYW7zYm73IRh/WG+VIswc1MGsL8Wkw3AdVBipjo+ZA8OI69A35HjBWWQAyFt5VjZTMacaMwq0/GcZVy2Y6iQJ1hfKANiUGFRnQbdTgFFRYSNAgNG8tnndi0J6/v1EDUtgnFmak6r6EK3FOEq//r4x00qmZPbQygl9LM2dWKbalwABjgSPATdtka6qN1kycZ+o+4PWGpUUOkAo8mmOc7/AaC2hkji0l5bPYtGILC15bQqwtzpLZK7j0+79mU7zQczuwzKLpv+tIJzNYps2yd1fyk9Py4w02NLTy4gcrsDqMyrYCU2Nm2/4cUjMOgDMOmcx5R++P32ugaxrjhtbwp0vO3mbAWW/ZGetkKpFi/quLO4UCQCaV4ZWH3+rzfFz6j0/kiaE744fV8sCNjorgX9c/yoPvricNhF9eSfTYcZ3GXPHoiGWjpZ0vtaBhBwyUJaiMhVlbQvToMYReW4l/7gYQSE4eCAp8yxoIvrtuu3MRwKwMkpw8yImJ6LjuD0Cds7D7VzV1qrsA2o8Zg5T4KX1qsaPiBiIlfqwBYTqXQtvGu7yRwEMbwRC0xhRakV+2ctxnKLlsA8qC0F2bUZZz+vH9OkLs7FGIT0eNSCAtBrImALZCDUmiTc0JSFLK8T7SFUyziR0xktIXVuap01Kjqwi/tgrf6gaGfzXJtV8bxdzKQYUrjg76lHbnBFRhOuYB00YqNAK315P6zBDsEkcNoUbG0SbHwFAkv1iJ//5mRCknUE8Ec5yXxHcHYA/xoK9JE/hjPXdcM4SScouDjnZSjy+YHeK3lw3r/EDsWk/XM2XJiOX0qWx6tb8ywa4wUO1pcrJakKlQZDIWdNM6as0m6XhXTivbslm/ZCOfG3ghkVQa/bSptFcFyJjdVFACkaau+5RSXHzW4Xzz9ENIZyxC/p5TdvTE4TUT8lP8pWx8r8RYpM/lNxOauOjzZ1FW2rsIbN3Q0XUNu1uaX19gx+flsutwBUM3Pn3pqbz80Js0bGzCu7Ud37NLqT5zP5ojcdJLN2PM39zZVlk2WsImObGW+Izh+JY3Yge9pEdV46lrJzh3A8G5G7YxWj4CoCmix48jr1oPbNOIbbQk8L23sdNDSQElzy+j/ZSJWOUBEDC2RAi9sQZlOgEYmUGleDdHOkIrutk3cicl2IYiPbyS2FFjIOKsYNLsQY2LYZzalM1fLoVz7kADKgxaPz+NwDvrMLa2o8cyBOdsyD634rknB5CMKWQ6RdJxgLKS+P/VDKaQPrEMa7wPY0EClRRCr6+m/YTxoGtoE+Oddo305yvJHFmCPi+JuaYSY2Qj5omlne+lNcFP9OYhqK+t5dqvjSIYtlAaRCO6Y9rwKqyRXqwJvq5Q8ty3JmUTCpjEpHsgXta6ntNeZQRtU6bTZtL51nwYRajNf7/SNpk5rXRfKs2MRXNLlJYvHgReHVLpbuOCR9M4ZNLIgo/Ao+t49MJI9N4Q9vi5Y8aF3LDgMVa1bSF0WyOx71QxT0WZrxbxn+cWcd8xlzC2avtxB4bH4ISvHcUL971GOuEIMF/Qx2evOH2n5uaya3AFQzdCpUHunPc73n3mA+rXN7L/sVMYPmEIALZtc9kRP2XJ205Rno6F3PZ7QNNITXV+GGbJTu5+FEROnYRdHuy5jaZIjazAt7rZWWM0hVlbQuCD/GBzPZ6m7N/zsMJeJOAhOXUwbWdPxdgaxf/BBqLHjKXi/rnEDh1JaPb6vEINouja1dqgREgcPKJLrQadNgsZGkNfn0JbmcI8rsz5Rmnk1/Q0BX1NFHNaGL0phh7PFKjREvU24tVRCzNYS0qQLQHHmD4igTYuRviqTWibnQSAnjejpPcNEZ8yEjE0fEvrKX9sPvH9arHPzF+kZaCHTK0He7mBjCkrlK+awjwsjPe5CPGooxoUn8Ku0MkcGcaYn6T0jFVIUCP1xUrSZ2XTXtiCarW4ovktFk4Zyn+aRueMm3UB7jgdZYTAjZuxy4PoLQly9Xxak82Xqw7lny1vYcVNMBSeV9rRX4sgepeQVkohIiTH14KvSJpzEUd12Z7ipH1H0RtEhOXvrcKybMZPH42+DcExqWwoDxx+KW/9dw4/+MLDSNBRlQlg2cLVb/yLR8/8Ya/GvfgPF1I5oJxn7pmF4dE5+/LTOfO7J/XqXpePBlcwFEE3dA45vdANWNM0bnvjBt55+n1e//ds3trawLqhYdB1SJvO4qprWMMqSQ8vx7u+tfPenvb7nZtHQyN22CjMwWU9tOwidugovGuaUQLJCbVkBpciAQ8qURjFapX6iZ44wfGQ0hTpEj+Z4eWIUlghL6kJA1EZm+B7G8AWxKMTP3AY4dlru+YtYAeKeLBYCv/djXhfjIBStLaPQPlMtJOioEvn0wVuq8fzXhxvXQQtYZI+owzVaOJ5M5a7lDp1tScMQtb6O08fsjqA9lYMvS7n2VKCZ2GC6JerQddJj6gk+M46wq+sI3baUKzx/i71nw3SaKCNSlI0ZkqB5EaC+yD6+2GEL12P7/FWMLO2iJiN7+/NZDKlWKfo6JtMtDcz3Hz8oYyIxfBgk+muD+qsVSqYFSUkDhiBUR8l/MpKVDIDYY1pPzuM78w8ledPepbWpggSUCQvqCHzyGg8T7RS8b84RlLhC3hp3tKKVeEvOHVkB6H0uaWEI2kWnLqQQecfVeRhu2jc1MQVR19Ly5ZWUBAqC/LbWdcyZMygbd63fmUd5kRPvuDXFBs8bT3f1A3DY/CVa7/AV67t98z7Lv3EJ974vKMopZh56oFc+ffv8qu/fodQKEDAa1D21GJ8yxvQIkk8G1pQ0TSRI0dhh53TQ9cy2fVvAcRvEDtwCM1fPojUhG6uf7m1RzuwbEpeWdlpH0iPqgKvQeywUYiuOX0qEKVoP34c0VMmOTv9zvQZCvEa6JEEkbOngaZI7juIttMnIx4NZVn4l9cXPLdnY2t++TcRsCGtDcEq8ZOYOgQJerH1IOasKqwFYeyFIYLfr8P7ZhRl2lhlAdKnlZI8v9oJfusmLVXGwtjaTl5mPVsjU57viQOgUhadHXh04jOGg4LAb7eimk0kjZOwM6Fhzy2eulkE0DXUShPxKKx9vCS+XUvo5i1oaSDTdXLKDC6l9ZwDyGi12M9VIW8aWOcFSA7xsjxZQWZbPyVDI3HaYMTvITO8gpZz9yd541iCjx/AT77n2La0lUmMZUkS3x+IDPSAoZH5TCUtfx/BjWt/ycV//Dr+kA9dtYO/u2eBoEJJPHURdKUIlBSvlrY10crr9Uupi7dwyzf+ypY19SSiSRLtSZo2tfCrL/2h52fIMvWwiahooWdDhdrGKdflY4d7YugDE4bV8uQNF/Li+8v5y+2vEa7PzwbpfW0NKuRFDSqFphh21midux5Gpw8nPa6WgkQ/IngXb0FZNt5NbYihk5wyCNvQ8Gxs6/Sw0SNJTNsmPbqatvIAvsVb8K5tJjWqkszwijwDdicK7BI/4jOcXa2mY1WHic8cSfj11eithW604TdW03b6vtghb5ctQVNkhpZhnjmF0LOL4cChTn+WjmwKIrZNavhgPGtXYZX5ic8YgXFsA/g10qeU4Xm1HTpSdWSfx7u6yTm1HJGTqqHII4ihodIWEsi+6NExa8J4tkTx/aSV6Of3cQRii1OsSNb6YHTXqUGsbGzE8hDtx1Sjfb6V8uuXEPx9l1Ds+ERsr07kpIl5qjSzuhqtLYJWm+nWuhCPbqClPRg+DxnT4rBP7cMZR0/k4PJ9eOv/5nLn758gsq6NzBHhgpOAic1tT/6H0yLj+cIPz+T+x54lPa0N673yrkaG4A01omkKX9DHwafsXzCHO5Y9x4Pr3sSjdDJioY1swmt1BWyKCMvnrMLMmOiGzpLIJlZH65lcNpRR4S4byKSZ4zj45QHMnlgPXudu3Vb88MAze3x+l48frmDoI2UhP2cfMZUnxw9m47L8IvcDR9Vy8oXHIrbwwK/+Q5rCIKbQW2sBSI+v7VoAbSEwex3BhZtz6pGBd30L0SPzc9sE5tWRGlMNhsKqCqFHkmiJDOnxAwoD7DqwbMSj5xtT9aytJDu+5dXQMoLKnlhUyiTw9hoyg8tJTx7YJRyUQjRFemyNc6LIHVPTMGud3XpmaDkgiF93UnKP8xO/cgD+uxrRmi0nhYYNmiX4l9UTP3SUI9RE0BscgdsRcGiV+NBiacKzlhM9dhyia6BrJKYMxvPScpQIpfcuQ0uapEZXkThoGPayMJoHGJF01EebvdjvhdAiKdgvQ/iBtUXfKumYe/eTm62QDT6Mpc34/96I1mphTguQuHgAUt31s9JEMcBfyt8u+w51De0MrS6jNOTnjf97hy+c/w0S0aRj1waUmXuuzI5v2ix5bSmb7n8H3dD542vXcU3b/7GmsgGrweMUaipNUXlrnBlnzeBbvzsfbzfPo8VtG3l43Vuk/7+98w6Polob+O/MbN90Eggl9N67ggJ2sFBUihXsvdd77eXau9druV6viuUDVOwdLBQRaVKlkxAgQEhPts+c74/ZbHazmxAIUrzzex6e7M6emXn3hJx3zlv1EIFweWsxMgV1TjmWNTV5Lc5kwwR3629vs7hsC0IIdKlzZs5gbul2RmTci3+/hW+W/sbHeb+Rmp7MRf1OpHtqq4TzZ3JkYpqSDhC3vn4VDrcdu9OG3WnD4Xbwt3du4Ly7zkJRFbRgvFIAI7Ipee5mUt9bgvvbtdjW7ETZU4lS4YuJuzcWDh3n0ny0lJqSCGq5j7SPVmDbUAhBDev2MoQEtcwbX5ZCSghp2NbsjF1/dEnyV2tI/mFD5FCwbRMqRnUl2DQJLdWBt29Lqk7ugt7EGf9wrAjUcj/2NbsgqBnVaaUETcey06jRJDwBQ4nkOyLlwUNDk6l4ox1V/2gR2zFOlzWyC4GlxGM4xDEywtVyP6HmqYiqIGnvL8GxfDtqYSVaigNdAeu2Mqx7qlAr/TiX7yDlqz9AA31lMtoXmWifZ6IvSYUg2AsLcK7dhmVFfF6Jr4NhwhJ1/O4seeW4ntuFuiuE8Essiz2479xmOKZ3+LF+XYrzX7upOmspb93yDl1zskhxO/j4n1/xj0nP4a3wxUSEWRZVGTWkon9vIYnl61L8ngCeci8Pnv0014WGYfu1AiXZi8VfhfvZXaRtVbhn+s00a5MVJ+eiok21ynYYEVcMqgkxdbjtjL91DJdd/SDzdqzFpwfxagH8eoiPti7kmx2/xyS5jew3iOdOupTHh19oKoW/IOaO4QDR89huvL3hn8z9aCFCCIaNP5r0poYjOTUzGavdEpPUA0ZEUfUioHqDiMJK7PkliZPTwqilvsgOQnPbUHxBdLslHJYaDqmUEufS7QTaZADCsNmHNCNqxaoSrPZl6BIUgW1LEZadFQitJtzVvrkIX8/mlJ/ZO+b+oayk+KdnKbFvKUYp9+H+Nc8wVTmtVB3TFtcSo6mOLa8EhEBf60b1+rEv3o1SECDYPRX7ZzWN/KSAULPkGLONY0VBTOy/0CXWbaWUn9KVlFnrUCv8+JolQ1BDJtmNXUD1WMBSUI590Vb8/VvFRlYBtt/L8ZdnJHTo+rtnY99SjHVbKUpAQ7coNf6PkI5zYUGkaq0hFyglISyzynA9X1hjigK+f2cOvYZ157hzhvLWPdMSPigIv8R9az7e65uh9kpCy/PieHU36rYax3vBpl28d9t0HMsKqC4gEcxKYtOxzTnxtlcZ1K01t4wfQfOMmnaf2Y40bIoFr1aT3+C0WDlz0sns3L0GLagz+upT+PyV79jUsQicsQEQQanx0MoP+Tj/N14YeDEbF2zk8cn/ZM+2IpzJTq56dgojpxwf931Mjlz+ktVVDzeqyj1M6Xg9FSWV6JqOUASuZCeDn5rE7DumIUqNp1VpUQhmJxs1jEo9KN6gES6a4JpSgKdzFqomCXTMRKkK4FhZgFruBSEQmkRLc+Lp0wIt3YVtSxHegTmxph5dB0XBNW8zztWx9XRkuEaUv0etKBVd4v5hPZ5hHZCqcR/3z5uwb4nP3q2daVwysS9YVVI/+N0oSKhLpEVBt6oIrxHCqmW6KR/VDemuMYckfbUGW0F5JE+jWr6SCX1Jn74Mf+csqo5tDxaVlBm/Yy2JLSpXLUvV0W2M7yPDCnHjHtQyD95+OThW7MC9ODbnRLdbCDZxYdtRjnRZqRrSlmCLVJQyH66FuVh3Vcb9bqrDXYVfxn02cFRfup97NFOnvBonX/T5StsMxt0xlo+veTPhGCUqQUxLsVM63phXQhquxfk4NxXRtm1TLrxvAsPHD8GvBTl3/gvs9pUR0DVsQiXDnsyMYTfhUI159nv9jE2djOfkJHyXZYEz3phgVyxMzjqGz4a/g6+qRvnaXTae+fFBugzqSN4f25h6/3Ry12xjwMm9ueDe8aRkNCz5zeTAYFZXPUJwp7h4efHjvHXfdNYu3EDXozpx0UOTyMrJpFuLLN6+4U2qcvdgt1o4/6KTmPLARPILy1j+zTJevPiVmGtVq/GKER0IZbpRfCFSv/ojJhNaYiy4whPAvTAPxRcimJ2M16KiVPqxbSxE8QTxd2iC1iQJLcNljI9eeIVAq51PEY6Ssm8uwra5CMIhsqKeZ4to5eD6bSuWPZWIgFZjIgvpKBLKR3VFKgKtVVqs70NKKk/pClLinrMJx8Y9SAFaugslqFN5XEcjMius8MrP7EXaR8tRy+I7qlkKK/GHlSGqQqBTZjh7TqC7rEbR1ehq2CGNYJsMrEVVCG+QpNkbjOguXRJqmoRuUVBqVdwVAAmUggQWLt/MgsUbjD5JtZIKo+dJ31HGf+f+jvXoNrh/zYu7lq7pWCwqoZCGr2uzSMRZ8vfrsG4vA02SuyqfJy96CdWicsy4wbw95Fqm5/3C7yW59EprzUmWrqya/Qcd+rQhvVkaiqqgKAq2Hyvwj09HCotRJdenGz8Bvx7ih88WxJXTCPiCfDf1J9KbpXL90Xfhq/IhdcmOjTv57atlvLHmuXpzJAAqgl4+3Pora8q20z+jLeNaDcZpMbOhDxXmjuEwIeALoFrVuD+gC/rews5V4R7Hes2CE0xzUD6uNymfryKUlYRjw55wCGmUaSP8UwrwDMhBy0oyKsRGjQllOCkf05O0j1eiVAUQId3YuTRNouKMHjGx+MZFBdaNhViKPUZ4blWg7hwNETZ3eYIIKREyfhcBoFtVKk/sRLBNRsIM4xphddKmL0N3Wqg8oROuRflUHt8xdhckJbb1u0n+aVOsLGHZK0Z2Me5TG00n+UtjZxJ9jrdvS7y9mmPfXIRa7jN2GV7DfBdsnoIttzheCUQnCEbfP+q7134fMx+qgufoNvh7NiftnUXG/NUaoygCXZcEspOpGN0D4QuR/v6SuAKPnQe251+/PRE1PZKXb3qTL1+fhdVmJRgIMuWBiUy6YxzPXv4Ks9+fh58Q/pEpaJ3tyCQVbaBRIFFBMGh1JnkPLcNbURO5ploUzrzxdCxWCx8++xmhQI2ZzJns5L4PbmXgKX0SfFMDbyjAOfOfp8hfSUAPYVcs5LiaMHXodViU/cvW/l/mQOwYTOfzYYLNYUv4VPXC1/eS3SYLNc2FiEoys5b6sOUWI20WHOt2IzQ9ZtcQbTIXEmx5xbh/2ogIK5fqf5ZiL47l2yk9sxdVR7fF160ZlcM7UHFa9xilYNlchG3dLtAlwQ6ZeAe2pvS8AfjbZyBVgW5R4goKSpuFsrP6UHpOPwKt02MirKIRuk6oadjcUF9xNwXKzuiOv3MWSd+vM5zctSvdhnc6cffAyOB2haPA4lAVvINiGxEJwPX7dtI+WYlz9U7DVBdOItSS7Vi3lRJqnoK/fQaBZkn4O2YSSnPW0XcuwXtViZMTjPmorsdVdUy7hNpDD/+u3WV+3HklYQd5/EBPeezOadkPq/jmvz8Q9AXxlHsI+oJMffAD8tbkc/2/LuOsG08nIzWF1kstdAlmYR1k+BtUoeBQbdwweRJWmyVm12CxWTj10hMpLiiJUQoASEnp7vqT374tWE5JoCrSktevh9jhLWF+4bp6zzP58zBNSYc5TZqnM3XDP1m7cAN+b4Atq7by3sMfUVFiZNAG2qSjpTnDpRaMpUFXQQrDzFGNtbAq8SIEuJbtAJsFf8dM/F2bGqaJKKWgFHlInbWekvP61yTKhX9UndwVX4kHpSoA3gDJP2w08huaJVM1ogM4rUhNpfLkLqS/uxjhM8qHS1Uxnqx1SdXQdshEmdVRWHZWkDR7PYonYCTltc8g2CrdqBkUTUjDmleMt1c2zpU745ZKtSpAXWjpTsPfoesR7SV0iVph2NMjwaQWBU//lki3A81uwVpURbB1WsSZL8r9pH68AjVUTxQBRCKt4hACS0EZae/vItQ0iWBWEtZij7GbI/acoDdAxtpC8k/siO62opT7I5/bnDZOnjIi5tKLvlkW4x8ACHgDfPjsF9z6n6u59NHzuPRRo2V7SNf4ascyfti5mpaudM5reywtXRk8P+9hXrzmddb+tpHsds249oWLad21JcPHH82cDxfEXF8LaQw4OTaAoTb5nj34tNis/aCusd1bXO95Jn8epmI4AlAUhe5DugDQ74RenHXD6Ux74hOmPjAdkVcSP14DmSC0KaETGyOr17UoH9fCrUgBFSd1JtTWiGiy7Cgj5bu1SEWgJ9kTXMGw9wt/iJQ5m8MCCCx7qhDVT4+KwL5ud83CKqHi+E4IXRJsnoJ021DKfSiVfiPqqTpyqNp8FdJJ/mp1TdtVXeJYvwf7pmLKMlzoTdyR42qxB+fKAkJNXBF/QPR31W2qcUwQ58uwryvE160Zvj4tsG0oxL1oa/x3ddsQGIrWm5OOc+k2vP1axZizZIqdypM7k/L9+poue7VtSBDj04n5SJdYt5UhAKXCDxaFUIoDEdJRK/0x30lRFTxbi8l4exG6PfbPufeI7ky8bUzMsWZtsrA5bZECdtXMencOY64ZSaf+7SPHLIrKmFYDGdMq1iqR06UlT81+IG5uBp/Wn9FXn8InL32D1WpB13Vue+Ma0pulxY2NZmBGBz7cujAmakoRCv3T29V7nsmfh6kYjlBOu+xEZj7/BaW7y+OaqEB9ebjEmXSqhrVHBDRSP1uFokljQUtwHaXSHylvXZuknzbWLHRhh6x7zibKz+6D65dcHH/sQmg1T7yu3/IoO6s36JKk79Zi21qCVBSElDUOZSmxbi1BqfTH2c4FxiKtpzhiDuphx3SwTQb+7tkkzdkM4QxfgREWXFclWOfSfISm4+vbwlAqCRoyWcI7jmCrNLCoRnJdgkq4Iqjj65hpZGdbFRzbysAbjPtdxTmuMcKYq82CAiPJzVrsQVcEQoDFbiHkD4UjkQVSGgpQjaqVJRTBlAcmYrHG/omfdMFw3rpvWpxi0IIhvn5jdoxi2FeEEFzx5GQm3TGOXXmFtOneCrsz8cNENEdnduKE7B7MKliFKhRCUmNC66Ppmtpyv2UxaRymj+EIJaVJMq8tf4Zz/jaWnC4tUMIlNYSA1KwUbv3P1SjRC5YwFgswisZFL09KRcBYTFumGtFJgOZQkVYFXakZ656zyciF0PTYJCxdoiSKAir2QEDD8cfOmBwJAEuZj/R3FpPy5RpsW0sRmkQJaoiQTtKPGxBeI/w2efZ6kn7JjfGfVFM73wEhkJlJlJ/ZC1+fFgQ6N6V48kAqT+xk7BSqv2+CYoMioBmLuKKglHhxL4yNBqp9d7XYA5puXCuBySjYJh3Pse3xHtUGabOANxDn36lTedcuO14tty5RhaBFh2ySM5IQQqBYVKP5Ui1cyU46D6zJkg8EQ6zK3UmlrnPlk5Ox1MrnIFy9NZrd+Xu4d+wTjE2bwmW9bmHRN8vqkjiG1MwUOg/o0CClYNxacH+vCbw99Bru7nkm0469kRu6ntqgc03+HMwdwxFMetNULnnkfE6/4hQu730Lfo8fXZN4K7zMencOX1a9x7b1O7A5rTRvl82uvEJevulN5v62joqh7XFs2G1UZJUS2/pdhNJdKMUe1KoAFp+Gt0c2oawknEvyUT1BbNvKSJ3xO4H2TQi2TiPULAUw7Oq6y4bqiX0K1Z1WbJv2JHzyBoxQzz1VcQukDJuiQulOgtkpRjZ37TGAr3dNjoVlRxmuhXmoFX4CLVPxDGmLdNnAZiHQNgO1xItzqRHd5VyYZ/g/qhfHoIZz8dZwJVmZ2KkNeAbm4OtlNFCy5pdCUCeY5Qo7H8LRVCHd2G2Ea1TpKQ58A3JwrtlFvZmLMRMj4pRDNVpIJ3/t9ogyqP3kD0aU0OPf3YsSTsZbuHYrt7/2uWE2DGkc070tNredUGlNvofNYWXUJSdG3UfjpmPvoWh7MbouyVvt4cGzn+bZOQ/ReUCH2rc8ILRPakb7pH3vIb2vFGzZxbOXv8qqeWtJb5bG5U9ewPGTjvnT73skYSqGvwAfPfcFAV8QPbyYBXxB1i3ayJaVW2NMA9ltm/Lgx3dw0/B7WaAKPIPbGGYQKUl7Z2OMKQIg0DaDUKs0Ah0zseUWo+6pQkt34lq0FdeKHeh2C7rbRjDTja97U1yLt8Us4GpVgOQ5m+ptAVlt0ooxqUjQXTa0TDcVLVKxb9pD0s+b4s4TAQ10HbXES8rXf0RMWfZNe7DurqT0nH4opV6suyogpBNsnoyeZEfxBXH9sgV/12YITce+ugB7filSFVSO6ICl2BPfL6JPC3y9W0A4GU8t8WDNLcJzTHtDCYQzzlFEbElqDCdzoEUK9vzSemailolPDe8pND1eKSZQGooqaNmxOQNH9mX8LWfQtLVRGsMXCHHrq5/h8df8bhf8kce5L17Iqme+YfOKPJo0T+fq5y6iS9QOY/nPa6gq80Sin8D4f/XpS99w+5vX1vs9GsLurYX4PAFyurQ4IG1GG4oW0rhlxH0U7yhB1yWF+Xt45tKXadI8nd7Dux80OQ53GqUYhBAZwHSgLZALTJRSxnlDhRDfAEcD86SUZ9T+3KRx7Ni0M67EgqIqFOYXxdmMhRA8+d29TH/hS2YsXccelwrbShKaVxxrdlLZKs1IBuuQCR0yIaTjWmg4ZRV/CMUfQk2wkMbcM/wzUQ4DtT6TqiCU6UbLdBuLrVXF3zETx/IdqKWx9YySftxI+egeOJZvjzHnCAlKVYCUj5ZjKfEaNntNR0t1UHF6dxRviNT3lyBCOlXHtkd0bUawTROCrVKRDiuOFTviZPP1bA5WFaXUS+rHKwxfRKesWEUgwst6ODqp9gzUlbsghTDMdg4LarkfrCqVvZvjLPHStX1zcn/6g1AgVM8MG13QJj8wkeOinny1kMYn7/2MuqUIstyRHZIvGGLZziL+u+xppJQJF+agL34nIqXE7/HHHd8XPBVe7hv3BH8sWI+iKKRnp/Lo1/fQqlP9fSAOFKvmraWqzBuj8PyeAJ++9LWpGKJo7I7hb8BsKeXjQoi/hd/fmWDcU4ALuLKR9zNJwLFnHsXyH1fji/qjDQVC9Dy2a8LxdqedyX87i8lAaWEZ57e9hkRBnNYdZZH6SsZFjaJ4tU1GjX3eq35KDmYlEWibYZiIohcrXaKlu7CUemOUi1rpJ+3/lhqd32pfVNOxFBkKq9o/oVb4sf2xC/uWYoQQRpawKgjmpNecF9SwrzNKb0eWjiiHgGthXiRzW7pscbuDyPiI7DpKIETwhM7IZfnYVu+MmKmqh/kmD8LjMP4UrbnFuH/ZQtLifHK6teS2JyYjpeT6o++qUzlYbBaSM5IYOnZQ5NiuvEJuOvYeKss8WP1B0gWUn9EDLSsJIaBZulH1tq6n9b4n9Iz7zOG2M+rSExOObyiv3/EOa35ZR9BvfJedWwp58OyneH3Fs426bkPRND1hqoymNdDM9z9CY53PY4G3w6/fBsYlGiSlnA1UNPJeJnVw8uQRDBjZB7vThivFhc1h46bXriSlyd5r1KRlpfLgJ4lbMip+jaQFuSjlPoQviH1jIcnfNSzpqLaxozpUVaoisWlJVaga1h5fl6z4kxWBZU9lorMAw6SUMKGt9jFNYt1VScXIrpRMHoSWlUTSrPWG8tMltrW7SXt/SYwCEmHhHat3QlDDUlTjE7EUJpBJCIQviFpUhfCHsOaW4P5+Ha0ddkJpzjilANBuYwmOYg/uWeuwbSo0ggAcVrat2c6tx91Piw7ZjLrkhLhbuVNddBrQnrHXjeLlRU/ElNt+8drXKd5Ziq/ShwhqKAGN5FlGSXK71cLFIwdHxu7M3c17j3zE1AdnkL/OaBFrd9p54vv7aNGhGapVxZns4KKHz6k3g7khzPt4YUQpgLEL2ba+gNLC+pPgqslft52bht3LqfZzuaD9Ncz/5Ld9un/v4d2w2a0xysHusjP6qlP26Tp/dRpVEkMIUSqlTAu/FkBJ9fsEY48DbqvPlCSEuAK4AqB169YD8vLy9lu2/0Xy121nZ24hXQd3JDk9cdeyuvj81e946fo3IsXZFEVw63+v4fupP/P7j6uwWFS0kJ4wNLYuIiMVEWmso6fY0ZzWuAJ0EVOSRaHszF5oSXajppGUOFYX4A6br+ozR9W+Vm10VcE7MAdf33AYZFAj/b0lEAghHRYUb6jOa+sCvINbo+6pwr65yCjvoQiKLxocGxmlS+xrdpI0f4uhCK0qZWN7GrkWAY206UtRPbFmO6vdSiDJSsnpPZBqOLlQQPJ360gt9XHza1fx2cvfsGre2pjzHG47z/78ENvWF7BnezEDR/ahXU8je3tMyoV4K2tFiimC7k9N5Kqzh9GtteHkXf7zau45/TFCwRBSgsWqcve0myOtbaWUVJV5cCY5UOvq77EPXNLtRvJr9S2x2q3MLHoTh6v+KKaAL8B5ra+ivKgikuJid9p45ueHYvwjeyPvj208ceGLbFy2haQ0N5c8eh5nXPnXUQwHoiTGXhWDEGIWkJ3go7uBt6MVgRCiREqZnmBsgxRDNP9rtZIOB3blFTLv44VY7VZGXXx85An0m//+wL9u+G+MqaohSIzieHqak+Rv12EJVz2trluEjC82B8aCG2idjp7iwLqjDMueqphr7o9ikIpAd1opm9DX6FwHEAiR9PMm7JuL9nrdaP9A9NhAThoVp3QxEtykRCn3kfLxSlR/iEB2MpUndEJG5X6k/3chSoKS2+UjuxrZ09FtTb1B0pduI2dsf4r+2E7Vd38YCW5h7C4b7lQ33govwUAIRVWYfP8EJt0xjkt73MTWP7bH3MOV7GRm8ZsxpVcu63Uzeau3xYxr0iKd/8t/LcaUFNQ0/IEQSQ0MQa2LH6fN55nLXsYfNkfaXTZOmXIcN/zr8r2e++sXS3js/BfwRNVpEopg1CUncMu/r9pnWbSQhqIqB9X5fTA4KNVVpZQn1SPALiFEcyllgRCiORDfLNjkiKFZmyzOvileb/84bX68UhDQtkcOrhQn6U3TaN2tJR+/+FVsOYQmbkLhgnWBnDTUMm8klj+6p0NcVJIusecWRz7TrSpKUENLsiNtKmqJJ1KkLtGCHr14R2cTVx7TrkYphFH89Tt1o69ZGwlYt5dh3VJEMCcd18I8HGt3I4CQ24qvT8sYpQAQyk7Gml8a+32FMHIylFqWXbtKyVGtKdlTgtrUhT6hL6kfLTe69KkKVruViqIKglG+h6kPzGDUJSdw+RMX8PCk5wj6g0hdYnfZufjRc+PqcW3fEFtuHaC4oAQtpGGxWpBS8vpXC3nru0UEQzptmqXxxGWn06FFZoPmrTbHn3MMdqeN6U9+grfSx6hLTmDsdaMadK6uJahBL2Vkl7uvNHQHpGka30+dw0/T55PZMoMJt46mTfec/brnkUJjnc+fAVOAx8M/P220RCaHHU1apEeqeVbjcNm5c+r1tOvZmpkvfMn3U38mo3ka5UWVhAIhhowZRNbZA/jPD0vQd1Vg3VkBuqxzIa8LARDSKTutG6GcdOOJ/LNViEDICE+VdSuH2td2LdlGedsMY7ei6SieIJYdNbbtQOt0/O0zUDxBnL9vjxSySyRTKNONp19LdJcNy/YyMt5eFFXNViAQ4d4aMsaRXnVMO9Jm/B7JU3CnumjdrSW/lnoJ1a4XJUQkq1qTIGwqgaPa4Jq9np7HdqWytIrNy2PNrVa7lQ+e+ZwvXv3OKM9ttdB1cEcuuG8C/U/sFfdd2vVqzYYlm2OONWvbNJIx/f3S9bz13SJ8YeWzpaCYq174iK8fvRxLon7iDWDo2EExjvKG0v/k3pHcjGpsThunNtIhvjeeufQV5n70K74qP4oi+HnGLzw392E69m33p973UNJY5/PjwMlCiA3ASeH3CCEGCiH+Uz1ICDEX+AA4UQixTQgxspH3NTmITLhtDNYox6bVbqVdrzZ06NOWpy7+F2/fN50tK7eyY+Mugv4g9390O3e9dyPnnTaYLEUl5dOVWHdX7Hf0kpCSpKXbIKihpzgondAXb79WNZXAifdXJ8JSXGWU2CjzYl+7i9RPVkZKgXuOakPFSZ0JdGmGr3cLSs4fYDiLExBolkzZmJ4E2zZByzZqPUVHJwkpUfwh7HkloOvYNhTiXLAF2/rdWLeWGLum8F+elJJBp/an+fo9RoVUTY/0vagdPiOBdsd355vANC7+x7lsXRNrAgLwewN89PwXVJV5CAVCBP1GTktOlxYJv8tNr16BM9lhtKV12XC47Nz232sin8+cuzKiFKpl8AVCrMotaMCMR8kuJYu/W87b909n9ntzCfjjw6P3hsNl5+kfH6Bdr9ZGhn9mCje+cgU9hnbZ52s1lD07ivlp+i+RnbCuGyG77z70wZ92z8OBRu0YpJRFQJy6llIuBi6Lej+sMfcxObS069ma5+Y8xNQHZrBzy26GjBnIuX8/k/KiCuZ8uCAmysTvCTD1gRkMPKUPboeNc5o1562orX5t5aBaVRwuG1VlXuoj22anZaXOxu27sRaUYwt3jKtWCnXlCEQjJKR8YzhwtSQ7IqQZHeRsqpGnUF1CRFVAEXgGtyYlQRSWd0BOjMNZCWePxxDSw7ub1UbCXK3KqDI8JZ5yL+//40PunnYzlVLnwZk/43dZI5nTta9ZMm8D60/YxEMTniFUy1ehWhTa9GjFpmW5MccD/iCPnPs8rbu1ZPzNZ5DTtSW5q/PRghod+7Xj3S0vM2/mb2ghjWPPHBxT9M5ui18idCkRmmTaE58w96Nfcae6aN+rNV2P6sSg0/qTu3IrqZnJtOpco4weu+AFFny2GF+VH4fbzv89NpN/LnwMpztx7a266NCnLf9e/sxB8w8U7SjBarcQjFJkUsKOTbv+1PseaszMZ5MG0al/ex7+7G8xx4oKSlFUFYi100fX31eEQElQhwcBDpeD8beewYX3TWBKx+vZmZvYRWWxWRhwUm9mvzeHpAQ+AQGgCnSbWm9kUTTBnDS84XwJGc7+jr2oINQ8JaagnVQFoSZu9JRYB2ywZSosUYyn/YjQClIVRvJfKLZOVG3TVyio8fQlL/P8vH+ghnQUT6CmWKGURjgtArXMi++7Ndzwzd0Js59VqxqnFKpvuHr+WlbPX8vX/5lNRvM0POVehBCkZCbz1Kz7Oe2yxOaYC04cwKJ1+ZFdg6oqZKcn83/Xv8nKeWsjJTmWzV6JxW5BC2o43Hb0kE73oZ15+LO/kbdmG798ujiSHOer8rMzdzffvf0TY69pmH8h7rsegAiphtCuZ7wvweqw7pcp7EjCLKJnst+06NCMlCaxYbE2h5URE4dE3g87+yisdmvcmLfX/5PPyqcy5YFJKIrCLf+5KqGjwJnkIKdrC8666fR6nw6FLhlwxgAGjBmA1W7BmuBJNxr7H7uwr9mF7rYh3fbYkFMw+mH7ggRapiAFBLPcFF84kPLTuxuLdpQi0Zom4+3WzNi5CIFUBb7uzRCaTFhaO9G3qCrzcEXvW3F8vIK06ctI/uYP0HQcy7aR9PMmUr5aQ+rMFcbOo446SsEGOtKLC0rxVfnxVvrYvXUPj57/Qp1jB3XJ4eEpo2jdNI0kh43je3fg/tOHsipKKVQT8oeQusRb4cPvDbB6/jr+7/FP2Lxia1xSmd8TYO3CDQ2S91Bic9i4Z9rNONx2XClOHG47Hfu2Y9Kd4w61aH8q5o7BZL9RFIWHP/sbd532CN4KH1pIo/eI7lxw7/jImJwuLbnjzWt54ZrX8VX5cCY7ufm1K2nRITYCut8JvXh+7sO8fPNbFO8o4ajTB3D06AGkZaXQZVBHALJyMtmxsSDu4R7AarOQk5nKHws3IHWJO9VF2Z66cyoF4FxZgFpSReXIbrHtQXWJUuEn9ZNVKP4QElBLvCR/v97obFdd5E7XjaqxEqx7KiPRVv7mqQTaZ6LuqTTMU3tp2FONlDLciQ2s28pwrCxAhHRsW4rq7asN4ejfOhRGvffUJesXbSTgD2KzJ26WdGL/TpzYv1Pk/fKfVqNaVajf+kfAF2TuR79y13s3xu0Y7S4bPYYmzsw/3Bg0qh8zCl5n1by12F12kjOS4qvT/sUwez6bNBpd18ldlU9SmitSvK02mqZRUVxJckbSXhvD18XWtdu569RHKC0si8TBg7EoOtwOpJRx3cn2hgSwKHgH5eDtlg1C4Po1F8eanXGLsW5RqDq+I4H2RqimpaAM129bseyuTFgWvHR8b1y/bsWys9z4XBGE0p1YCuMryiYilOmm4oROpM9cUa9yadkxG5vLxpYV8Y2Fqr9jffdzuO18WjY1LuKnLnwePxOzL4tPoEvAgFP68Pg39/DMZa/w0/T5BP1BbA4bLTs15/l5Dze4NHc13771I+/940OqyrwMO/sornx6Ms6kxEECBxIpJW/c9T4zn/8Si01FURT+/u4NHHX6gD/93vvKQUlwO1SYisEkEVJKtq7djr/Kzycvfc3KuX/QrldrugzswLQnPtlnxVCNarfgb5GKUlCG4qvbJOPt2RzPMe1A17GvKyRpzqaE43SbSsnkQaAYNZnUoipCmUmEmiaROnM5aqlvrwl1oTbp+Ef34rIu7fnhqS/YvXUPopa/xu6y89Ts+3nhmtfZtGxL3HU0lxURCu9sdFmTWBi+hGKzMPne8Zx/99kNmaYIi79bzkMTniboD0VqOAlFRHYtQhihpE9+fx/dh3RBSsnq+WtZNX8drTo3Z8jogfvsJzCS416J+Cqsdgu9h3fn8W/v3afr7A8LPl/Mo+c9H/P/y+608X7+q6Rk7L30zMHkoCS4mZgcTgghaNOtFQB3vHVd5PhXr89qWMxqHWj+EF1Sk9m2qxIluaZDWrTdXrcoaE1cxgKrSdxr6g7ZNHIsJAiFYKs0o+MboJR6CRzbHucXa+r/nsAx5xzDZTeOZvPs1Tjcdpq0SKd5+2asnr8uohyGjz8aZ5KDbeu2J7xOMCcdz4BW2HNLEP4ggXZNUCr8OFfuAE2SOaIL59111r5NFjDwlD58uOsN1i/ZTOG2InJX5ZPVqgneSg9zP1pIRvN0zv3bmREzoBCCnsd2o+ex3fb5XtVMf/KTmOquQX+IFXP+oKighCbNExZciFBV7mH94k00bZ1Jy477Xsn1p+nz4x46VIvKslkrGTFx6D5f73DHVAwmfwmGjhvEK7e8lfAzRVVQVGWvpavbdG/Fc3MfZufmXTRtk8kDZz7Nijlr0MMtSbEohNKdWHOLSFm5k8feu4V7z3gsLmwUAFVgKSgn1CK1JvQ0qJG2soDhZw7m17lbCNYToqtaVfplZbDlxzU8d+VrkQWxaEdsVfs5Hyyg86AOdT59K1V+pN1iNBgKozVxE2ybgd2qctyQnrz/6Efkr93B4FP7MWLS0Aab+mwOGz2PCfsJJtUcn3Dr2Aadv6/4PfE1gIUiCCQoER7Nzx8s4KmLXkK1qoSCGoNH9eOe6Tfv044lrWkqqkVFC9X8riWS5AYUqjwSMaOSTP4SpGWl8tjXd9Oio+HUbtIinfPuPpvh449m+PghqBYVJbxAW6yWhBFOuavzefyCF9m+cSeuZBePf3sPVz93EVaHFVVVUHwh0j5ZRda8XG5/7ALmzPglZqGoRndYKZnUj1B2SsRRLSp8JH+/DntuMWeNGsSM3Ffod1IvrDYLVrsl0na1GqvVgivZyf89NrPeHgh+X4CVc9bUqfSs28pI+mkjzl+2oJT7ItFUDpuFDqnJ/HTTe7z/yExmvzeX5658jYcmPNOwCT8EjLrkeOyumkRLRRE0b9eU7LZN6zynsrSKJy96Cb83gKfcS8AbYPG3v/PtWz/t073HXDMyJrrOYlXJyE6nz3F/zR4O5o7B5C9Dz2O78fb6f6KFtJinwZdueINQIBSpqRMKJl5ENy/PY/PyPJbOWsFZN57Oxf84l3HXncpJFwzni9e+p6KkkkEj+9JrWDeevuRlfnh/bsIIKW/v5khHVJKaAOm0oRZ7sNmt9BjaJdww6T4ASnaXcVGn62OKwwX8QXRdjzmWEAkLPl1MRot0SnaWYrFZCAVCONx2Kks96JqObYtRd8q5ZhetLj2WMecfR+dWWfz2nx/4sMwTqbXkq/Kz5LvlbF6RR/vebRo05/uKFtJY8v0KSnaVMuDk3mS2bNLgc8ffOpqduYV8++YP6Lqkfe823P/hbfWGMa+evxaLVSUQNY0+j5+5M3+tM3cjES07Nue5uQ/x5j3T2LZuOwNO6cOUByftdyDF4Y6pGEz+ctQ2EezKLUz4ZF8Xvio/Hz77Bef8bRyeCh/3jnmc3NX5ICXrftvI3dNu4qcZv8TUjopGS7bHZS4LTUdLd+Iv9DDjqU/pelQnsts25YWr/s2KOWtIzkjC7rZTsrMUMArGvXD16w2SNxgIUbh1DyecP4yTLxxBVk4TsnIyOTvzYgLRrUE1naabijnrWKNm0svfLo8pwAfG3G3fUFCnYtidvwdFERTvLOXjF7+iqszDyIuOZ+jYQXvNQq4oqeSGIXdRVFCC1I3idze+egWnTD6u3vO2rMyjYPNuuh3diRtfvpyrnplM0B8iKc2917lp0jIDrVZEl2pRaN6u7l1GXXTs245Hvvj7Pp93JGIqBpO/PMPOPprff1wV4zwU4Z4HdSaLBYJMf+pTlv+4ms3LcyOLy/KfVnNh+2vrrejp/jUPj91CIKoznFQVLDsrCIZ0pj4ww/B5BDW0kIbUJf7txXv9HkIRtOjYHD0UomBzbJa4rktmvzuXYWcfTU6XlhRs3hUxnUWTv9bohbBu8SY2r8iN/97+IJ0Gxfc2KNldxr2jH2PLyq3ouozIDUbW84TbxjD5/on1yv/+ozPZmVsYY/Z68erXGXbWUQlDTrWQxoPjn2bprBWoFpVQIMSVT09mzDWjGhzm2rFvO7oO7sjahRvwewMoqoLNaWP8LaMbdP7/KqaPweQvz4kXDGPI6IHYHFZcKU7sLhvX/+tSOg/sgMWqIhQRZ+OXuuTDpz9n1by1cU+c1U5QtY7qompVgKTv1uHcWmKUyQhqOH/LQwlfJ+AL4qvyEwqE9ikpzZ3qovfwbhQXlCb8XErJ63e8C0DT1plx2d+KqtB7hGET/+i5z9GCNd9LCqPkR3m/Vpz15Ps89ND/sXtrYeTzp6a8xMZluQR8wTi5fVV+Zjz56V6dwEtnrYjzhahWlS2r8hOOn/3eXJbNXonfE/YP+IK8dttU9uzYuxKN5tGv7mLyg5PocUxXTr5wOC8veiIuwdIkFnPHYPKXR1VV7nr/Jgq27GL31j106t8eV7KT0VeNpKKkkqoyD38/9RF2bt4VE2Hk99a90FntFlp0yCZvTT66Fr+4i5BO2y1leFULnuXbUAsrE/bVjj+ROsNuQ4EQs96ZE1PQrTZFBUbUkmpRufOdG3h4wjPGzkEYiuXyJy9k7syF/DxjQeQc3aLg65GNv3sz9BQnaDqf5G9n/pC/89Dr1zBoVF+WzFpR7y5Jl5KKkiqaNLfVOaZdrzbkrsqPuU7QH6rTrPPr54vjQkQtVgur563dpxBRm8PGxNvGMPG2MQ0+538dc8dg8j9D83bN6DOiB67kGrNFcnoS2W2b8sbq5+g9okfcOdU7itooisLE28fyecW7PPTZnQnHCG+Qjz69lyffuYnJD0zE5khcciKajGZpdaYpZ+U0wWqv+1lOURV6jejG5wtWM+2n32l9dEfey3uFG16+nL+/eyNTN75ERnYaz1/1WsziLDQd5+qduH/eDJqOuruCpFnrCZZ7eeTc5wgFQ9iddS/4AGlZKWRkp9U75sL7xuNMckS+g8NtZ8zVp8RUc42mVZcWcbseXddp1jZxdr3JgcNUDCYmGAv9yIuPx+GOtV0LVeH0K06Oc6wqqsLQsYOwOWwcffoAmraO7WhmdVgZPmEIQgh6D+/OpDvGMWTMoMj11Tpq7XQ9qhNXPTMl4Wfdj+6csOe2xWbBmewgo0UGvzR38vj0H3lh5hzOeuBtFufv4qQLhjNk9ECsNiuVpVVUlXoi50qLgnRaIaRj3V2Bc+k2Uj9fjS2vBLUqgLfCx8MTnuXsW86I6cksBFhsKq4UJ+5UF/fOuHWvzueWHZvzn9XPMenOcZx66QncM/0WrqzjuwKMve5UHEkOLDZjruwuG10GdYwkzZn8eZglMUxMwmiaxqPnvcDCL5YYyVCBEJc/eQHjrjuN5T+t5qXr36Bg8y469mvHja9eQbuerSPnblm1lbtOfYTKMg96SKPX8O48MPP2mMVU0zTmfvgrC79cisWm8vOMBXH1hgac0oeHP7uTM1znx0U9ZbdtyrUvXsLDk55B13RC4Q5zLTpmc/Ej5/Lxrt3MX5OLHvU3neKyM+vJqyLd1nRdZ3TSBfh9QTxHtTH6UACKJ0Dyd2sR3iCKJxizabE5rPx37Qss/HIpn770NUIIxt1wKtltspj+1GesmrcW1aJy2uUncvkTF2C17X1n1FD2bC9i5gtfkbc6n8Gn9+fUS0+ss9ifiYFZK8nE5E8g749t7Ni4k66DO9Zp5qhNwZZdvHT9G6yct5bsNllc/9Jl9BpWd/kHLaRxTqsrKSssi+RCONxGu9QBJ/dmXMZF6LWc3q4UJ5+WTmXG05/y1r3TYsp12JxW/BP6UeyMNb04rBZmPnAR2VH1fG474QF+276bymHta8qNS4nwhUj5cBkWT6yD2Jns5KnZ99NlYGy00sMTn2HB50siPg+pKgT7tOSMO8Zw89nD97v1p0njOBCKwfzNmZjUok23VgwZPbDBSiHgC3DDkLtZ/M3veMu9bFm5lb+f+gj5ddQvAsM5/PQP99OmR2tUi4IzycH595zNsWcaoZvterSOMc1YrCqDT+0HEHZAxy7eAW8QZdpShDfWMa2ogiYprphj599zNr6uTWN7UIT7SBxz0XFxeSCKKujQJzavIeAPMv/TRTGOcKHpWFYX8PH8lbz+1a/1zJjJ4Y6pGExMGsmib34n4A3EmH5CgSBf/ntWvee16Z7D6yue4eOSt/m45C3OufPMyGf3zLiFzFYZ2J02rHYrrbq04Lp/XgoYVUsToQpI2lqCzaKiCIHDauHWs0dgrbXQ9zuhF2qCMFkhBKdeMJxew7vFOMBdSU6KC0rixteFLxBi5ryVDR5vcvhhKgYTk0bi9/iRtWJMdU1vcAlwp9sRV1qhaU4Tsts1RUqJalHYvqGARd/+DsDE28Ykdl7rkskj+nL16KFcPHIQb94+iTPDWc616aXYwi1Dw2g6lgo/g/t3xJ3qisnRKNpRHNflzWa3cszYQTFRUlJV8HVrBoAIa5aQpvP5gtXc9trnvPTJPApLKxs0JyaHFlMxmJg0kkGn9otLVLM5bZx04fD9vuZnL3/LukWbIslwQX+I5654jcrSKoaPH8LFD58TF9aqWlRGnjecKacM5Nqxx9Alp+6yD3c9OpnM+blYij0ITwDn5iLuOW0oVpuVxd/8HpPUp+uSNQvWEwzEmqlue/NaRkwYilAVpMVQCt6BOThsFiaM6A3A7f/+nMen/cAPv2/knVlLmPDwO+w2lcNhj6kYTEwaSXJ6Ev/44u9ktszAYjOqol7x1OSaktT7wYLPF8f1VLbaLKz9bSMAk+4Yx30zbiU5Iwmr3UpyRhK3v3UdrTrVlNf2Vnr5+o3ZvPPQB6z5dX3Mtdr2yOGDuY9y17D+3Na1M//30rWMnnI8AEnp8TWI7E5bnO/B6XZw59Tr+aTiHYa8djGh4zridjs47/h+XDJqMBu272Hh2q14w9nOQU3H6w/w7qwl+z0vJgcHM/PZxOQA0GdED97f+iple8pJSnNjsTbuTyunSwtWzVsbk4gWCoTIjkruGnb20QwdN4jyokpSmsS2TC3ZXcbV/W+nqsyD3xNg+pOfcvZNRsXYalIzUxh33alx977w/gm8cvPbkXLfdpediXeMrbP1p8th4+GLR/HwxaNiju/YU4Za65ygprN5576VtDgQ6LrO91N/ZvZ7c0lrmsrE28bQsV+7gy7HkUKj/vcKITKA6UBbIBeYKKUsqTWmL/AKkAJowCNSyumNua+JyeGIEIK0rNQDcq1Jd4zjx2nz8XsDaEENu8vOoJF9adW5Rcw4VVVJbxp/z+lPfkLZnopIbSK/x88Hz3zOmGtH7bXb2emXn0xKRjIfPvcFWlBj7LWj9sss1rNdNsFaVW0dVgvDeh6YBVlKyexlG5g5bxUuu5XzT+xPv44tE4598ZrXmfXuXPweP0IIfvl0EU9+fy/dh3Q5ILL81WhUHoMQ4kmgWEr5uBDib0C6lPLOWmM6A1JKuUEI0QJYAnSTUpbWd20zj8Hkf52dubuZ+fyXFGzexTHjBnPy5BEN7jp24zF3s2ZBrPnInerivg9vo/+JiR3SfwYzfl7Osx/8hFXT0a0WerTL5qXrzsTWyB0VwGtfLmDqd4sjpiqHzcLjl53O8F7tY8aVF1VwTqsr42pM9TuxF09+f1+j5TjcOBx6Po8Fjgu/fhv4CYhRDFLK9VGvdwghdgNZQGkj721i8pcmu21Trnn+4v06t+ex3diwZHNMv4WAL0i7njkHSrwG0aosQIvpy6kq9+BMdjD59asPiFIIBEO8/e1ifFFNl3yBEP/8eF6cYijZVYpqVeMUw668QkwS01jnczMpZXVH9J1As/oGCyEGAzZgUx2fXyGEWCyEWFxYaP7STEz2l4m3jyE9Ow1nkgPVomJ32jj372c2OGnvQFCweRePnvs8FUUV6EGNquIqHrvgRbZvLNj7yXvB4w+i6fHVXgvL4iOeWnVuEVdGw2q3MGRMox6q/9LsVXULIWYBiYqX3x39RkophRB12qWEEM2Bd4ApUsqE9XullP8G/g2GKWlvspmYmCQmNTOF//7xPHM/WsiebUUMOKUPnfq33/uJB5B5Mxei1SrVrYc05s38jUl3jG3UtVPdDpo3SWHr7tLIMVVVGNItvvOcalG5/6PbuHf04yCMHJM23XP22ljof5m9KgYp5Ul1fSaE2CWEaC6lLAgv/LvrGJcCfAncLaU0c+VNTA4Cdqedky7Y/1yKxmJz2uK6yCmq0qDy43tDCMGTl5/BVS98SDCkI6WkaVoSt088PuH43sO7M2Pn66z+ZT0pTZLo0KftXqvB/i/TWOfzU0BRlPM5Q0p5R60xNuBr4HMp5fMNvbbpfDYxObIpLSxjSsfr8VR4I8ecyQ7eXv/PA2bSCmoaKzYVYLdZ6NGmmbnYc3gU0XscOFkIsQE4KfweIcRAIcR/wmMmAsOBi4QQv4f/9W3kfU1MTA5z0rJSeXbOQ/Q8tiuuFCc9junCsz89dED9HFZVZUDnVvRsm20qhQOIWXbbxMTE5C/E4bBjMDExMTH5i2EqBhMTExOTGEzFYGJiYmISg6kYTExMTExiMBWDiYmJiUkMpmIwMTExMYnhsA1XFUIUAnl7GZYJ7DkI4vwZHKmyH6lyw5Eruyn3wedIlT0TcEsps/Y6sh4OW8XQEIQQixsbr3uoOFJlP1LlhiNXdlPug8+RKvuBkts0JZmYmJiYxGAqBhMTExOTGI50xfDvQy1AIzhSZT9S5YYjV3ZT7oPPkSr7AZH7iPYxmJiYmJgceI70HYOJiYmJyQHGVAwmJiYmJjEctopBCDFKCLFOCLEx3ASo9ud2IcT08OcLhRBtoz77e/j4OiHEyCNBbiFEWyGEN6pnxasHU+4Gyj5cCLFUCBESQoyv9dkUIcSG8L8pB0/qRsutRc35ZwdP6sj99yb7LUKINUKIFUKI2UKINlGfHc5zXp/ch2zOGyD3VUKIlWHZ5gkhukd9dsjWlfD990v2/VpbpJSH3T9ABTYB7QEbsBzoXmvMNcCr4dfnANPDr7uHx9uBduHrqEeA3G2BVYf5nLcFegNTgfFRxzOAzeGf6eHX6Ye73OHPKg/zOT8ecIVfXx31/+Vwn/OEch/KOW+g3ClRr8cA34RfH7J15QDIvs9ry+G6YxgMbJRSbpZSBoBpQO3u4WOBt8OvPwROFEYLp7HANCmlX0q5BdgYvt7hLvehZq+ySylzpZQrAL3WuSOB76WUxVLKEuB7YNTBEJrGyX2oaYjsP0opPeG3vwKtwq8P9zmvS+5DSUPkLo966waqo3MO5boCjZN9nzlcFUNLID/q/bbwsYRjpJQhoAxo0sBz/ywaIzdAOyHEMiHEz0KIYX+2sHXJFWZf5u1wn/P6cAghFgshfhVCjDugku2dfZX9Uoz+6ftz7oGkMXLDoZvzBskthLhWCLEJeBK4YV/O/RNpjOywj2uLpbHSmhwwCoDWUsoiIcQA4BMhRI9aTwEmB542UsrtQoj2wA9CiJVSyk2HWqjaCCEuAAYCIw61LPtCHXIf1nMupfwX8C8hxHnAPcBB9d80hjpk3+e15XDdMWwHcqLetwofSzhGCGEBUoGiBp77Z7Hfcoe3qEUAUsolGPbEzn+6xAnkCrMv83a4z3mdSCm3h39uBn4C+h1I4fZCg2QXQpwE3A2MkVL69+XcP4nGyH0o53xf52waMG4/zz3Q7Lfs+7W2HCznyT46WiwYzrR21DhaetQacy2xTtwZ4dc9iHUSbebgOZ8bI3dWtZwYDqbtQMbhNOdRY98i3vm8BcMJmh5+fVBkb6Tc6YA9/DoT2EAth96hlh1j0dwEdKp1/LCe83rkPmRz3kC5O0W9Hg0sDr8+ZOvKAZB9n9eWg/Kl9nMiTgPWh/9z3R0+9hDG0weAA/gAwwn0G9A+6ty7w+etA049EuQGzgZWA78DS4HRh+GcD8KwbVZh7M5WR517Sfg7bQQuPhLkBoYCK8N/ZCuBSw/DOZ8F7Ar/v/gd+OwImfOEch/qOW+A3C9E/R3+SNTieyjXlcbIvj9ri1kSw8TExMQkhsPVx2BiYmJicogwFYOJiYmJSQymYjAxMTExicFUDCYmJiYmMZiKwcTExMQkBlMxmJiYmJjEYCoGExMTE5MY/h9YOay7NvNeOAAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] @@ -934,7 +920,7 @@ "id": "3acd550d", "metadata": {}, "source": [ - "### Tracing Cluster in Hierarchical Clustering" + "### 2.2. Tracing Cluster in Hierarchical Clustering" ] }, { @@ -1023,7 +1009,7 @@ "id": "080f044a", "metadata": {}, "source": [ - "### Performance Benchmarking\n", + "### 2.3. Performance Benchmarking\n", "\n", "Here we benchmark the efficiency performance of PECOS hierarchicaly clustering and compare with a pure Python implementation based on [sklearn.cluster.KMeans](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html)." ] @@ -1038,7 +1024,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "PECOS takes 1.0543 seconds for hierarchical clustering with a depth 5.\n" + "PECOS takes 1.0039 seconds for hierarchical clustering with a depth 5.\n" ] } ], @@ -1066,7 +1052,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "scikit-learn takes 49.1826 seconds for hierarchical clustering with a depth 5.\n" + "scikit-learn takes 58.2397 seconds for hierarchical clustering with a depth 5.\n" ] } ], diff --git a/tutorials/kdd22/imgs/pecos_label_matrix.png b/tutorials/kdd22/imgs/pecos_label_matrix.png new file mode 100644 index 0000000000000000000000000000000000000000..837cc9fce3393bac75e08e19f2e2d9c4c8febf8e GIT binary patch literal 73040 zcmeEvcRba7|39UWj0j0MR%Av+c8Vk`$;vp9nN_xoQzSAXvR5f&?>$cCNOqDv4%vIp z;~c;DM|EB4y6^jY-}iNWzkmE5SC0qYpU>yL$LswX&)2Ib*OcW*4;?>*hlfWhe?{gx z9^OF|9^O6}(LwM=Q2THg9^PRK^Glbm$zQqzyJl-+Vs2@Shj-;km?oi?S{+Ti&UJaS zhu2;|8X$P4|N7A<7NRTEmXFwiuNf*5aXH?&ad(!*k6Mpf)t>hO50@dozu~<9fv_-# z+!vf&HW&Am&2gi}(85kQ*Re)KeO&1}8Qx6^@zk@l?FV5hA^iS9L^u^&7M8vua4FAJDoso{mn~# ztTb;I6>8o^6U+?mv?iRjKtJ&<6 z@g-%}XM66${>G~4*jD@Bo;G&Ajio>B@yg@;n?xydIneK=WOh6T<*W# zWL``wJ+B^3pHfuI>s@-}Y9%|0amkZlRaabuiBFkYD*jYckiw0cC$uKZmp`O_eHYel5 zj8kRie6Ypub+f&l&S))&W~JpLehaL5zbcG&fP2eS{$M4$SIRY3oR1TurQ0zZKFj(= zL^jhWkG7nbCvRq|9(Fb|C)J{?_I-)EVtBq4U--lRdkFF4`d$Wh)$VuE55=pxsp5~^ zBn~1LCbD`YMpYx5$CP;DKo1O2Qbt3E?%Y@G9UJdV#E93;e5M%{`0&geJ~n3&HqlLW zM`WrbOx*zc7F&F{%JujiF+c7DNyn~vo?4S?{$l@S<%E%H#};PmjbYQV3}Mr|BXT>j z^J)AK+`j~ccd^XC9BITJ#?>Cr6@B~u>Fw1n_L(=7T?N@ty_%!WNuj@G{GVrJ0ndFep z>~npGe~s|?JL0xS>@}k@^-y?=kyI{(`^k2OQSPZV5 zr3`w_GI@oDMf&K+!$yqP{W4@aSaqIpjnIw|+3phwDqwL6mb*f@ba3;LAt!13B_&lI z`(smHd99phk1zz8ybNuIg}ZP(@}SF4sOI|8(3*^tR3O_dN|fwteX176OJJ_H$h1fxMy~)(@^2Y zsqN!9qWSko$_WB0PpWr8Z`lKwjgBN!_z@dFVt7>jsO>I)A=LoY6Pc5(PbWy%4Y>qj z&c~HJDtT5C)FordF3Ic|dWDTOgzD6TqY7W2Ey>rsc6crEdiZtvYsc&4+Fq8*j;ubR zZ67IZkCX;f#JB~xg>A?7DYPbcUEAi&J&JF9+?dW-Ea{E1?u5DAn!&SYM8R_?fu(9`4>yh*dK&$Z5d z8u&bEuXF#jL}0uM{AzY$!fUD5>t=#y_0E6NeX6Vmz9R?GJJD;@IwP`2j>x@@4ljwzHWm-pOU2v+J z6Y38wQ#p35+V7pg0^RblUb+;z+50jQViH;UMf#}&`js@5eC~SgK65u+-h&+ zv9|C!<1xVq0rJx}f>Lym$8$&@9rP!?b2#g`+vyXA$1FL|@(A>qI#(}*9%$ttR{xRkq?d!##~P{d;ST}QJ} zjpIwJxLFdhOJZ4iFZBxh-CfJvgRw@KtC(pr8A2wX6Fz}tnkQ7{ROFuB_8vCo_)^y( zKU2_}>cDC6X8u~;r}~%0vF6Q|(Z69$ZQa>FIX5(y-#^_CcRlYyBq8L|w`S!0bV+xjduFqy zp;uu{eYkzdYBp}J@*O5TC|)JDg)+geiU3W4?yd@~V!~EzFJZ|xu%q)c3B$*MTckz=7~I-9dDWDujjzMOCEo` zNao`H-};#M)8is{+TUOVgakq&%d{vO0`;gdux37M z+rI2qPhJgQ)(r5VkrMZ~yxmQ3`911UKk+=(CCUi)0fju3>ul#ZEZBn-vaaHSDWmfQ zldKwZ1kTjzDxOoOPU%Q__Bo`@@wGz`DSf$#CFXp~y_#X`r^WPA4%?f=AE<0OPQ<-A z_e^;4mMF6XySvktsXkA3NN3!;rmo=Dn(>9sCHNuF1L+4C-s25j9a?UD&=?lQpSLeh z!`Q*sQ^q(^SwOBre_K>c(A93uC$=j#PQ<o z{z+D$*pt~&eeC`rv%#!Oi%VN>5*um=d4p`N7tS{*w63XsI>YW#|1MQjja`jj$u73Y zxo7$P!uy1j;kM1T7*nmrNU`Bo#kI!F{9tXm{Ag31rg*7xcV$$po?WV)-55?|=@s_9^BWaZ9lwp&h_F|?tjzpOMJ?#_PLLNtWh4o@5rF5P5_0G?EpIa0* z6!^B!Zl^3IY=xWfS!s_ITDIAKwXIu<;r7&+xtW@ucXu>Y{gp<0mPsa`HrBccb#A;t zyk00fDN8-WiaUre8*W-@y;;^WEjw*<JNvX>e;eIv|Cz1=#rzag#l zj;up%{5{vK8Pa|o_^@ui1Ct$m(jy0+yAuTej3{-1O7b^kBQ z>9c86?S<_r9mR}vF0V`9A%_;S`H*v#1!fuN(-8LE0#mu0wOdbV8RxA-gdE~OiY~r( zF;lKJ%}k|(b}#1=J8OCr;v>j*#x{G0 zXtKk58U`Y!vp4k3%@|tm&)oUcPStraJFc|3(qr**V_-*&ys!=WxKsLU(cqfDSgDgf znzy9a)u!3~JvM}}hrFIKT0CRBe`wv^O5aM3$y>5)<^0fMCC4@ge?onNhjqeE04`&u zbaOe4)|&sg*n6?8ZGp9*wTRCVm2owQsqNCCFl_oI#DN`$R0~B-j79f!m9WCV@yh}l z`@y@%CMNq9OgQjXZr~kT=)262E=saSk|>H?V_itdnX2gG*T?(vR(X@Y#eCl_Da#@| znA2&zdiVzF)`PNga-qVHXZrhZ=$>_^WLa5Rtvq+&!56%@wAyg^{D-P4{$7I|Cp@$0 z3{fvGZ=#mNIv2wxn$s5T-jXMZCU`^)2T5GJJc#cxvvUCB6jWo&$Cv`74?klqc@rfi zJT8zX!Xv;xj&}g0@WGEHKK*a$%lMpl`*+{(!^872$0PWjM;ZKu{sn^{sLZ#&`$Hb% z5rQX5@ZUomW;XV@XS@Qz3*x(1wC(Wl zXxO12eEI8a3!wf^^BY?BT1twdw{5KW42*0Hjrp9d?n3q8i93seq?NI~0nFLT(%Mec zS%PIZhbTxxulZSEyIJflBv`bRuE8$Z*c!ux_)hbkW|2GugTcgYjZ8$Z%Uu3m96U*| znAzLk73Jr5a&qExI>TpUYsxPmA|k?nT998*kQe0OwR5qyH*n^)wqyNP$&Y$ujO}jQ zn%}iIx3Pvn^%@x3IM_?Dus{v{_U~Iijh)SZZ^_#3`>?wDp!KYlML&JPX!r>Xch&%1BIL`xnL=l^ZdBoEP>pxVGXo-mhDxdHxyT?YNb zrvSe=zx{>M?wzUOg}iup7x3g|q;ELm&-4>V9&ImOSai}lN`2s&tc=VN3fYUnSAv7d zRrm4vY92YFiPF=RvXNY_u{J!1? z^G~M;`T!>B#NU(j_ayy`ZTa_3`r~nXXu_3aR7ZL0=%F-Ibnb1r$<|kLUUqQjp#$3G zj#ggFiNw7&$jX(OZnbj}@_T5)1K9^Klh)h3dnit@+QTqz&9r?HuJ^qN$-`^>Y0J|G zW*?K%zjnv0E!A-(INM}PIBzf3Hq3jhm-kfb%I%>?$Mqh-IwoHCG4HL^{h>DrZT-Sf zb$d327-lBlxTnB;RWi32?V;n?@1VK4z1%V#H*Q_A_U( z&ADTHHpj=Jw-(uzCu#NUpEX9GOP_g4cKqy>WaT(6%xp2dTO?*6$E4}_=D>QDrj0ng zi0$j`F=6Dhe#c&`wj|xld%7L12)b=l&?oR~wLrqq}_y0^x~N?+(TO*CLe|Qo=f6e58OjaM=spL zZms6EDT~!%=87}p! z{4_RMX=x$d`KAp9J`5TshWG8MAg2`EaxprwwU_oRqaf!3*r8WypE9uD$DovuWe(FP zZ9vzPFB$WzLZcX^UR#8spRb}GO|l)~LaYo*&D#v|~A$aO~NL45Pp#G2&lU$&M|Ii9p-e(3Up zJ!kKN(1EPdvE!u+?@6X-`oc8zPk~*m8h1~aqSA9F9ga=Q(68(vt!oyW=TQjfd8QmE zN?lvB=SUE}22-n`5ML|1cM4ACSau6e_7>a3Rr={qj=GL&on7?juL{Plqv15XH*e^6 zmN{5x=iGWY15RI!0N-B5#rzNj>hqWJ+#^5V#18|AXR0uBOH=9D;WO>wI@YJ=kvmnE zm8%`>!>wWSZsTH0{_yQYqiw03WvS7MxZXnn%f=L;|fzfslV*9qZ2Io<~VL2xEf<6z-Y|XUp&;7wC$OGI$!+X_ROndim z08t-e2kY=&v%o@?Ge*?m&OLtg{prjNyK3BK3xcLLu~=t&8otA2)my~-^dA1+&Uk_@ zlE^)D-`kQPOOE60&B;$My)tx)vy6i7uTHTyMDPdI3R!l+ST6?z1Hwi#ZkgXCv=o8d zmR;+y$o_hac5k;b0M*X3W$Pl}v)f>-`v{rbavI~rikk<34yl3C{X-Xfgf5E|rn}{; z^$vC(!C%YhYFT&emy`NWt3z(JXO939<#I0tt-%rvy+9-tW%Q>Z%e8rv~zIO*| zfFqePJYy8dF(J1@doo4Ar18l9uRZbc)d z2e9x$*7vx*n~9FX*Hi20hbc@CIKxJDA?$e#eC1z+1AF&Q&lk{M(((18k>N%*#$?eSAAh<(0RI<)A@407S!Nx#L2@fK}Jl^yE zHQQ;+uG88DW~y;=2;Anladzce+P_;CwopS0{|Ihn4vWq#UY8Ze*`7LZFktHH>g>b1 z(tCDA9v@ijts{mzdl$QjX8~{xb&{AFwQ*JCYNkzHh0F3(z?r=S(+Kb;;j64D+}>r3 z(H`8I&W^}$7ENz{g&QQ43Rd8+EcnB;bwJZ5b#aDi&$Q)49Z)_$L=Yu#`*NII<$M)p zB={0p>K}&g0H&=Yhyt_sVzD%N7%$;k4|o`jaaFYYy53}8sa@Q?U#|QNa4F?^JnE9# zvvqHsIh3K5dl5`qv1m|Np@pV);T`oqlobb*XX)BKLe&ODYfM`>)!qROKoRM@uV~Ap19!cna(}O z2Q>5WAC79z>_GJ@frDmYYsmO(@x6V6DS_#Y-VztuGwHwf@CEDdT?R;=?rxQo#*2E@!HIfH>YT_|A)TeQoqJ zm;udrw{LD#m9eF?e@fymdC&a|Hlw~pL#-|R)>P6Co2q3bdl zhQI!)B4Qg{biPQ$dY_%j<|sd+yFy@JJT}N@4Mb1beixjwq?tL?j4b?;9mQDU{f zq^K~JjzwEU3txVlzf-5;!4f2ebKZ7jhuF@2b1z-^OK0*Lj9L?A7kk^WTb)Sjfn)eB zI8!N1l=xcL?ZsXPUeY9=va)6a#3WrAYjfm zhnaRb8N?i)>sXf_c=l*t>y917Z{}5#iG76B9FNkY7_o6K&9NeR*tOn+g2<@S@;Cp)tfHcNv}hzR1CJddzm{>U06ZdipJb0lmv zEp3VsR(VX$#J4=&Vq>j>-gYCG+!);8FQ|S@8VUrE==to@{&z2Z7?1CGTTXm>xyb)G zrgynm5)B5xgCpqdmW>tn#@Ki1!a+SHP3!yt~pW*8mpa)c%8qgHsYbM2GGq?_RusKUUbmVMhT;IKcMu zZEh_icX7X?&i0OA>rUH3nBw3gD&=6hGbUHiRqmsJ$5RBeZTnfCqXtRz6eryRv>z0p zcA!(LWoFshemU9@!J*r}U8kwjMf!C5@JzT6B>g7*G11D|&;)5g7_y@{DW> zf|)F{`gU)FN2&kz@w%Q8g+dq_Ge{u7?|(lSmYlq#%;xK&{l$ww;9Qs!i*#c{A|sn` zYH9@)v7tWADok2YX*BK%#d6*oLSn-^T=O!XkrA>IS18ofTsSCE6+BbYo{`b@V$Wb> z#;W9$s4q7Lp8TV#)z?l^qVk*+i42QwU!=(2Jc6?HAP|UQl5QN6`!65iY3am+k;3bNxRLD{V%J&DVH7k^*H3 zjpt8OFt9Yp8=6$8>!a(E=l{3J)jt_lSmhIf+!+Qc&&XM5Jb#vgK}!=cb;naHZF_)_ z<$@DcVkHya z@{XgHmd5$PrHK1}xx+QQwDdior*SZyF$xrbkFOmmXj2Z2(7&IG!g!MK1*d0Ml^OBO za%&$pbXfw`G!OC|%oT@P?uELr%O^nJw!U=;N{WJ(i=Zz2OiK9urFm2kI7-mg4}xbb z;5#}AJvri%%E~@_bpic78!g-|x9@H>>2=PdQ(9E0so>{_M_Pkm8n#4K@Ve_14a1-v z1-JSz9++>~UAC%=6bgzOc`yZOo**gEEPIODS8nY?hA!gZ;fYFV$N_>}%BLtz@KAS^ zh5~iu+AL|XCH4G%KujEcZ98qdhfsoSv`1i2NsO$7>W zgthtiT9|#qid*|a!eHT@(CRj6hqsD^9YaAf_!Hc9=kYivp*PAD=?c%CKui6%qm9^R zVd3jw>cdAY)5RiS8l7x)MAAG@H2O=;33DrUwpo4KB~n5HUNfMUpmh(4wQ?AzL`_Ar z(SccMP5TE$K!?&@mD%@vztSVkGY?I-d~;DiQzi=Lh+XY$&^C=zXX#H;poJvik*8vF z!N4q;?MyG7aYeNv80z8TwS#(B#C%7wpdSa~9BX6=a#>_Q9q`nm(7C7jqs|(&Qxg}V z2WSZ<3b^@Le5p@)@=>5XyZro#3}yXZkPzg;ioPZmcxesnY_t1Pt0aAY#p0>;jOANl zshwyRU)hB9*IUsdb|x$Dt}P-Itm$+&EZ^Y=ZlR|LAJg)Q-R^up=wWq83|N26&_!uE zDrjDkPF%vkK&{K|wVe0zKK48_1DWbC^;Y}b*L(8rPOGdfHCxTNla?ZT`KL8kbjs{g z>nmQ$6EZsAa!UZl9W%2#p?w~kqY<?-I-)(9s7YBgt;fX z2^->`9zVx5P&-uZqceBX*=hhwCOYw=%cd*G;N8I{mmy}K6O;VKbtD)4LfY%P%XWp-1kgtqB7q>5ymVs;w`{Y$ zv6zc~fWLpYq;lkG=Cx@w62YayK0l5Imgw8SEV-5RHDaAm0=F@I8(HO!8N;T&&fDRL zcQ%~qOi@X=!6-FPE^7Yybyv1#w!RC3D|KdPq%QO}5@?~5mo{7wpMV9KcWoLV2%uUK zxcGY?=yKse(o%yutFdz%*OjePY`t3&v06KV@&HO#rP0M@+>7yeSL+!~EKA6Z_^Sg5 zoU9V_9TFf>Tb@NK668i$qq9HRrGzhr+O;-WrKcq<1SqWY5bL#|MPGkPy5ujqoPw|k z3UOf`N=xmyVMtLvnH=-=_T9Jmgh%6nv>Fon8JpL`Am^z?tm4K#??I4TR(rN?chin1 zVCXJUDA;ziH?sS-Ydz9s@%4Vwo*@ryO0eHD#&i#rK6j*Oz<@JNA~Joag93Gj=OW(` z)Wt7*)6faR0b;saLWY7wHC_7>MS3pLx6&=hWFkZL)`uBSX-GZ522e;WCi1IFgEOCW zcP+5G8)))AI%tO-p0I7|GR#C@uZ*^i;nEgyEwOAZ^WGlzY8icy&lI7l*&YXHs5dIM z)`_T0IBW9;Rje4G9M_(%so?N3^V0M{b7TKC_oe!FP55k5M)P*I4 zWpi}`k|1~M>8I9Eq#K{G z^-==eWqZ65MLKWDjHonEl;&(VL2eIAT^7|}@qwBibw{eT4S@@2jKqxga&}blmU%RJ zmp4rAbW}U3nbhv~kRt`^A+(Re^4WptoP=Ry_VpH_L_r#1&&W_B(96FhnPx(TvW~61 z-i(|wUawFn3azydfg{Qhby|#B6e#G1R9kQGD1HnZW8*Sa0Ia+Mw9q7WY`)&5rRoF% z97l@S|Dt5mNnh-0%{jsC#-4g>1*Mz%M|JE%3qxNv3ckw zW^fehy8$;=``JB+0`>j`v7v#rHMMl3{3ENsD5;)LY&hLv^IVy*DM*UHQDV~!QyE&` z;KZ>yV9zNWK@9CAj`aV-b;7O6O5Z;r3GZ;bez!$pfUy+e8g*|{lKnQbW+DdNmo3QZ z3$aA$a}ii;QZN%Yn!^Db9X{v!2u3q@Wp|Gg83NnGx9$GtlhH~PV(azeBgN}g$W$S!6$DLNs==%9fxc{MZfbR@Ee6LYj0TVot7A48b19MS7`@{+d(4Wd zpzW`G^8af4Q!}Ax_^}r*C+s|T!+E>bI~0@SrqY)J@foHD2I)2}XAhQcA>o<0d8JKD zy~$qwr9%SbTC8NHaEw*DLW(qex{<`i z;Ge;)BT7Gw3OHrKM3p7k{d>Yxs(tN45xgtBf;U zo9IZDz1JwQ_9@#^-K~$=wfNLbd1tuyREhct|26KL<>;qRf&lcd;8n&pJTqi##U0vNuVQ?ONxaj^A01sMt7LWMMi?VY=FHa%2jZe8POoj zZQL9+@cHnRfj$8N@UP9)b0*eILFd2vcG&=6VpMOj_{^stY`KZGQ-2H1`g$`D4r}~_ zITsv{@L`BPZE$g>6N%VqoV8|bMn0k9_yBnG)rMF&qvOQJa8M9-Yi%~hr@yIBGT!;? zlU#U$q`P3TjZ4={`OuR!9+mf3>vtRGP>DZ*S)J)VV}}X%VGx}yXfs4IO58WHc3NH- zpQkg8J%VwVACR!~03`u_g?nwfBU6XbQP>W19$-7VvH_sxTbE16j9^5<)}6X0ULD_UTrJjUM!5WFEwbxcu*3)c;9^b0rpT=tdffQMRUNMSSh$DM z+O;_a3f>YKkPPL>yd&Vk9lPyOJz|_2|oEE?mS+_A(76K%~a!5~=ocvvJ^1Bi% z2@(Xq>DxY&8M;7Pkw5CbAgv6fYx?~Qly_%SQ$TvAtwreL=T6D+b|w$Le$m2b5>8g! zIt5u-y2e!;PGeWJMxF#>GvC|JZ|+!Hx{i=W}gIU{O<$injAsZ*G+sNHWW>< z%_lM(HL@Qj5QtMN)h5Wb6)d~w8QGlL$K$CbLf3y@T5=Z{i}q?(${gn zVg|$~GTUmWgICnm@4b=d2gtPN!V}HJhSHXM)R#wqmh*`HLl_N8Lsy1^`#OOh%=bVp zHSH0U-%)8=qQ4+}>w-6Rx@0Jm%1*i4MjdjbP4Gy5{J)?DzzGUEY^A45o=6%UVd5T2s6uzXEt}(-N&qH7r!OoV=w|2^)cdoj9qnk|B zW22kEGF@cH(gOmc0#_mBn?;tVfB~eSU;B)1V8ZN+VV?x0w?)y*Eo18d(O3su-I8Se zX+Ec_ar~o75Zk$KYu%=cI|#SM8rmW;OhUW5_Y{AVgdTXdt)bR5OBPJQe^@?WrIUz= zv%&hYXsswX{7aAZhgYfI0GGNzdSRjZw(Eqs=~-o0`N}2|H>5cNW?&3O7X6%JpLFNI zJ(N(_$07MNVeVCsBp2`}L~f?e190CG0FvrK9(Ap~!i!;wutT9X6^oB&fOT2Wy065@ zKZ-Zsv?Uxkk6!J}(gXf?+yI)_XV)UJ1R;-s^^kM8Hb@0P04Xa!JJ$G7Q&wIVx{o;x zGBrk>k<)jbJ+~c_OKpmFLc2qbW*>IS&fUkHgX%!+_~T8F9G4CL}X#Wwt{y(&upx(Kk8A;5C23YLijvY&yLXvP7$ zu~&X&Ee^n!odG{HvN@g*VO89(aZ)Mzxurlp`^Od>PPH)-O!9Kl|EIJ`=tN9e<_a13Vf48Zoi**px8S3_q2Yv=YD5R?|c6fDw# z8ym=QW3{FLBCv57K$#MlO_qoMqfGqmEffG6K*k6d+|Hu_^e3dgmuLQkVEy}#f1@G) zJQ(b6G~~a;=l#u}{mq~K1fc#K4f!uc#lO*ze?#qm_8_PqK#d(mb!5Z&rFJ%3M#&Ky zUclE1g6X6`FryyVCN*>s+0~Bnr%Utje^ZwkpYt6Z1!%DDW@N^} z+=^HPlxq<@R{?G59^n2r^r8)fSOmC`qIo#rFm@|KBX?nsQ&VG;LoL~VgFDiAX~jdDz1i&sLyCqrpg#oq&TrsFDAx#h zcHj9$3AT$@Y(Yc#M+o)^H5>znDuOx-pfm05VT>rrtqIoBjj#TrkLj(0FTT818x7O& zVO!t-1C&@~^N{C2u6X<|fHDFCA%0UH=N&~YIpd50_7I>QO)xlsTKE|t-vB`$6Od3X zD?g?O3033)^%8<_a*Oyj0Z*c2#V8h=%2f^R+n!q5(R~BK5O{F6iPd~c)YAQL!~X$D z1jFA&Fh+n}n$UD$B7T8wa!d1MBXB0{zM}y1_`P3wprS48u8KG{D+>i%TdZYLSc!Nq zHeZo14XhttGNm!WzbRLjAf@mhw9emNe%BX&o6Dcoko%@|M#qIMVa=@$gcmBUw69|zZ4YdL3-Fy)xc3x%Euv3@0Tpe;#N#A-=gS*sCLeMp>(UsL^0gOUWPVr>$0poQ zdY}b)iA;oTjb7B=jLQSS-hd#8^0@T)u>5H12zHwuz}9AjofiggY%k$n$ep++Gb3#F z=>-IAYeC){{!rtwpO8$kh96+*JhnHwY~DV=PZbXXGXkQz_+q!#=Y!x6-u?jO9mp`L z1GCA5S)J{r4ZwQ{;;#s-O}C|}yuA~y7=8BC+H`w5?J@j)KuCCsy}vd?mlMt2o^TH2 zpJ-*J4yf3{js~1*_QjSt%|8Xz0!$_~21u&Bx6Ub~p{aS+rh5vxLGfMq9WAyQ*b4wE z)Ia*-7Y0$snJzOof?NXC21A zNS&^@+f7$ab8y{4t?Q+p|3yrZy0`n5wRXwc&dOR9{ z)1&mu+>sYs;%?Hwtwva`7G$R3Y(6BphCj+0w_&(G@%g1}+-4iHSgx+(gP(TbJt=_f z6|(Nrat1zv6;jl@*2WQPL;Yl3@*k!Y%M|2a-%mP}M$@LrOIK6GVLJiia;vmM7K}Yo zPxZLx7&V0AWnX3pd{tNLlLIrk<#D0}z$I>9p-5NZPACS8Kc&`C{R4LwBlf}kK!{C>_BDy&E)EmFDm{Z>g zsTfe?gmatU1qB?dYUvp0eO0V&=@;CtwJuLl{-SCgdBYmSBe>zlN6I8A9#-3e@dBh& zE*KC%JA@h4{|+Xq=0I)`Bc1BT)olL#T>!UGBW;0mz&Lw#9-u0tsY2MUf3?yn3V)xJ zGIxCq)5u2r@NmAia6uBzEVLbyhzDigXA>{@^Lg!&Ty@x5y-;~qBlTD zqvxlP;xhqaAHIQM7ZqpodWDPCw6`~%Himo}B-Z;_e?&6VGfqGH&G8)2bLpj#P!%ps zt9WyO{dg)m#aTL?Rc9i$)Qh_esr1CIBffhRT4LD8`gA5fOu zG%DR3i%9~2>^!&4CE#i70UW?Tu1vNinYP43x-LARp;~EEiJ-TsPz3>Sn7K7Fu9TFS zr;HEH^P9%IBty;>pDSIRv`cM`+dSDS*2OpCfJdxtzeA^dv336laKKmGI2`5CEHeJF zz`X4>0EOh;Y3BkC%&{Oi4dng|?oWRZwMmRvQ+Reo+-*nSdC1$g>0BoiNNW`M$66c> z1|aU9mXVbNDb_P%Sy>Xg6+a-Fubd^wfjJC>?1)W~Ujq(KGU;==o?3dLY^-zSR-Rgv zh0&!1xySv36$o-E3Zu{cwP^s6>_6|C6pLtY(0^5TTeL14*^hSB2?w6DAi7RM0yF0F zPf+fkVcq{5=`WU^U2wQAYjsNohDH=1!^3OZeB`G!y4@{MwuBH~liSI9#32@N&$fTk z#}K{;i)pErHy_uhS0$FZo_wjaR1qleM#qfQ=QyPmiPcSxOJ^Jou#JQzKjJzQ2-7%t zSwxXS?P6FI=+6FECr_Ot{@2TEI#*+Ob=b7;#4CVKMyVtQ^aH!MB2%dlannzQt$eA>? z-#tT=*3hSX9JnOIT^r3HH0D(8wmlBtTh2ow=kDAF1BvUprCyigblO+GN$%u{OvI_~K#&pI+LHY(&pPLdIk0ao;1XQwfppGjD$gI3Jgp#1eke)bydKc-nI9yxn z0i5wa_RWnAZO44anjyCL5u8+xJ%(AG?&JsD;ZrHxMpw2Cy6P4IQS1+p**=YYqtCD9 zbK%+gm2ZhCBE?;7XF7q}{y{qrBxZd>myxl8>se$t;EW-1uD^Vk7IwEo+qA}`zs!L| zt0@G;65DnAIKX;g5UBS9vA}>rnNwySF)}m`UlA*uShv{q`v_<}kPGa|3{vvz{wSql zUY#pL33l{kS_n+`JMS#90?yKe4&J03_>Pdbo|C1v;6S-6cf0#dL#+bnpjZ^l_XPAE z0b2tG?S*w8q5maxrW=K;eb3wXG?`Uy)IG_}mTe6X%9OM!ae63qzd~YXkjk{G^LW(` zb+*60h$CQK!~1YZj=3*`?%)(|+o^T)2_VZ|y10t|&kA+; zkr}pZ&rN$X^t4x`ao9@i%{T8|WZer#NjfdvAw`gk}*11J5XEvX*YPTlfJwq9tnm4wlBGRin&?!|nZS5LpSdcsSr06hBd^L;wLPw2)7D+~C+v)vp$LPQDP>=-# zExV49cztxIKmU3&X0%}u{0ARtSHt`A5@z~Jo06USCnA2s4`KQA%5H3}PMdlJ0OVeM zHMtZI6l773(FNs(ke|N=h>-1Qm}Wh&%T4oC0=(MDXaw@?Q|lAr&N(ue@<~mAfKA$Z z9S(w$+K5(k^9393m#Z8yLEHp5KQf7xYo#D;pD_Rg z=b5!8Hq8N7I`NsDY9kQ$MbPLYg$pnj3S-*pFuyUDR6CaX70!z$>kXLwmk~^UIQ_F` zpI|N^q{;VuUsUDlKrBCYpfGsOt`KhNh^&wr1m<5}zY zilsv^zxzA#V)ay{IshjL_AEJFR0EYwB zO~pmXY59-o-K-wq>l(I!8`xPvYOMB0B4UjjavHSh$%d36P<1=N2_Xg!J!95wYAP=N zGcc&f6RoIhhXkAFDG~0R=~B1|d&^?)mC33=OQhbkV15HR61R4f&SlQfWxA3de$R2U zyS&bY%8?uS;)qQuO4GL=L^ZYTyH^|aJQ2ZcUK#X>VUXJJr7b*50YeR8R-L^e#N)qC zfA;VOz)~nDCPEWPo*>l}^drGPs|nzeE$9L`bGVa~J~L1hj#7g-5YRJN%w%V8EiTAs zFz~$w65F!6lxdd`tf^_`J3olwaEjOwOLgjsJCKnEREa-BHz|&ox2LIZYj~qKJ_vhu z=m0dGrRQ%to6TL_(KpTQuP^@_eG@_M+hv>8=$3Ri-P$7_olRLc@||^5+$bhxg}9

G#+*=~cOV5B4blhyhqNdF-BvxYBLCy92#}myOE&V$N_wY~ikdvcbtKnvwT0#vsH1?4c(={b${=hSFu#WRj16}w*7f*givHqmQ z`vD#P4}=uTExo(#*EgGH5U6xzhF)zCht>gyIfFT>`MS0S$>(+-=1*D`TWgQ6{jb0n z?Pw~w0ZM$+x#k_lirWci2?vUa6*3LP2LEQyYuTka4E<6lKbY;s)yBU_udZn^iHgWm-zb9O_!Zl7sYY) zdg$ORhGL$mwnz}bATYK2{5t3XaAN!mpt*{ZPMe5I9;x`M>mry2^ZAxuc#oy_G`rE0 zxtF*)h|Bz35(YC-1E9{I%@Dx!@ZUUso_{kherjC`_sq8FV|Y71BWCdB{x8dyK**_! zU!om>v)ve^;Q#L?b5BsD|G&b5{2g=Z&jvpGJLc5Cq!xe2oca$!Ap+hbJ_Y`)$GW%zcjS&L$fmqZdnZ*Mk&2eH1N1ZmVo?HOvQYf(c6S!c}SIhBE zfB^+z5*6>ol88R^VZ483z1F&sQwjMm4^qH(-X!`qw7qlKON)%**(cRgl%iz8T{o@ZIm zkDkvvKtl5o1Onok?I{eMVA6zwHL4)CLMwp2IS+sw>wt@62(L*Ik8a6n9QGXuX35D)u+0WVLz3Ak2>?S>5np^%0c`A_la=Uy;5wmDr$A<)z+{3@ z&#E$MhuY!Mu4RxR-j6%$lcKTm{(S%{>hx-RALo|rXAsHt)7w|~&it!D#KBSz)6%%Q zOYYr%HB%z$i(T-Mdd>u(zl&au^L^Kg%=Fs70boOSS_ZJ*S6t%moscsB5`4CR_Q*_L z7JX)+P7#Yx4)rr@b*>J-8?rN=zE@;3!q_w)2FK_V*o_-cDP6kGN#ONWHcR0G*CPtf z+(9Ls=#I-uN?2Me7L)R2AybFg6tNap4~u4zMedI?u`bQc8!o0FjFB@{!cru(l3QqL z*X1T*(@~C)g9i`32l8snMXB@WH(p$8?m^}+w<*Qt^_^NFq2-G$Y{9&6OKeu@>@T^u zABo!k6wcB&-kXR9F+(XIKYm=GvW7Q@1H0^fEfeCzf`Zt_Mp9{wUaQSOg^K|Q(0VUQ zA?3oj1roSZ5pKucS`Qdy|t64tA>4 zS20NS)|4N|0-22piaa|nG$u(~;=E{Ba)0&CTmIQo58=@|ebiV+kBy-m=f$rjjY(>Q zK!iW6nWam-A@XjHu6mv^I9&4;e0(Pe#E~iNvkEHr>H_Yu$%xGA^6WtGXG68KE&q?b z_ke0DTi=JTBNlW-6vYt)VgW`084IA)!LA6*h#(ye3Q|Huq?e4Kf+CV_u2a`&-*-&)7ReL zO4%Qi8G3#D{;u8?=s1~P(T1rl!B<~ZY^7}0sa=^e=_<0JCc-VmAplMp?fM6&Sm(5^j|4#>PnmdFDnNzP4 z#3=j4*}&9t@X0j*ljO%f`FroIJm3;&Y7#IC#ZYpgzQ8Y6)u6Z; zAzaKudgIfE67pKrh4HAEXnL_*;MK~P)@wov(w+e6>lfavl3WuKjqt@%k>2=-kKlJ5 za;!sNw7Lr0xmEYcQCDS%mLGhD`r?~C3l>wV5C%_#F7ki=e1G@k?|t|=LvkU1JO-p?V&;&)S|cU z`O=%l-a$fy5oBUQdgGtYr@uV;Z@(mG773%xCTle0L#`vYD4FYcV9HJ zV~h`b5^(^;ihKJ%tQ^#V*tg3dd2x`bK6Cou^lr7kU!C>lp#eg&piYZzxdn*hu3o=B z=v^#CkdDkgE5X}053RykL@BiXDhn@4F^_V`phDbT^GXjK5X~6oMWC8FYg_Vf+(wyn zjpE_|TR@yQ(9_D9r<Uk$9L`!jZfLb`S}X40YI-P_Bp7Pq#=Kd;^F@c8PG*YzB)DocrP(NV!F<8tNMZt z1e#_samhjibD&1pt4!Od3aEuGBAz<)0$V#I7&>wx`D#VJb94t?!dNn56l#1&$!bs0 zB9QWW?Pv4?lI@;4ilcz2Rn}t!6a<+vfK#xuWk)t?dpz;YRH zZ^3jPLYZ(!l|4l>0JGN2jdRZ3ChBynNkdO`K*b?95`b%bek}YygwSI4wulw*SLg&3 zkNRGWBQr}?RlU2i+@~-{-f%*1H3Zn#UU$95mNX{^_>XRdS~(i_3T?sZ1(K?p1G>7p z%Ci*5#ah8h=_1+H#0A`HcbV%<%}X1$DDla6RnX%JsbV|ZCji5lPXCr9`IQCH*jBD>(chu=0?Nzx&B`;+`F6_woaQw{SMHKTLO0r)qrr0sIYB(i@pi+KZ{M%Z8at;9@ zyPXV78QlqZ5I4CbNU>Fd&#ET4V3*x81RfL8xO+dNU!-R~y%cCu04B^Izvv1Ws(s4rwL<4{CIxB?Pj_yL(k5lXt5p~(aLejS9_ zvD*jWd76(}NzO%Et;oW6i=gw#0(fPeobFb}-W71$8S; zis-y8!KOf=@(Sk?Y$GGE*y7#qH!-TCy?}%@JSRK9_iPf@Yzb@X=$+nZ-@cL57iVwT zKT6ZF77-Ml)KIk8tgNXqT=S+u#_yIr@eHug3h%zb-2iQ)MYUcXlD~8yQ9i`oH?Kte zR%N4M$XUQ(lREhK>v~1do1}h%cl=scao@yxr#H}oc9EG6B2cOgiw1FkWhFn`FD68| zc8Wuf(Fmy-o9pBySRGPxN{IXG-~1lv8Q@8=QxD_;m*u!xoI@Z=HOfFhgU-4pePn?E zhx8rS-yx`#+lxJ?Uso!{AE~2hRlL5D7ER~(YaF31oRfm(4ircp=}@MNKrq1BCu{Tp z)_THYqAg-}QS-inZK4W2d)mEM2oW^I*UBuS9B3RGTp7|q*faTUz%w90eK!9zD(02A zXAfVPFtH`GdV(54ivDJaFnaj)ZL7tU`u3OW15K~&DOwYP1OtQ!-(L3EE2`%fS~M5^ z<0?kv^lukYZtkGU>MGVK4FTQm`8++)cd}^TIX%VORG~a3sabD$CAhgba;xN{RShkr zoFBXd-CkGK*a|ORl64jg1h`T!+T%DK>36-}5bkB&feZW?e9Qa0_bl>dp<_oyzq?~`KQc|7iziFBPauV{#;i6?JS=iRHjzO`#3Q!Nh&yk5CM znwvO1?MkXTAD=3(s0gz#>zBU^MFHhlpoQh_D(Dbx&2up7B0)r?%Or*)?A%#gFdRP* z{q|7KF}*G|RaLw8IM21xvTn>q|&TnE>Mq;tJ=P<50f0}SDLEb1-*!^*KZK!EO<0&DMzpUw^$M_{T@@W@B+h)1PbtzwhE#{U)7?`W^M|giI{5l*y7jkB>Egz+ z$!UuZZrgl2&e37&hn;6c7wdnkduEU9&681sxZ1t;%BWnoNjrr}(v^J7PF<^=dM@0x_ZTmO$p!pvP=8p2 zKYujIFaoL!d{wm{=V#wvMxH2ZL};Cd#uZi9->uW;-VOkrU5wu)C#%yS4Wv9S2|yHI zt1h_VH$mtaxpm)QcW&7Z-?pce4AjCNtcZ7QEu*o_-SG zZ;M)b3rvLrKukB*hzv314TkI3M1=>;H;)z?@@9Yi&GP^exa$c(Zl?r*?v;O6kmbO< z6=3hC*?)WkT>cwdA3BNw6{!^uXE2l$a$yJnx>RJ&#M_yA~g zo$GL3z5%_Xp9BQWF;cr^r>3@m9?xjid8jI;yBd2V;A~t^OzIx3IL!bpMqldS_8~X9ef4Fgt-n z;1~!a{k$~rD%@}T$gnwe2CLH->-+rloJ?R`8tJZ7Vl>3+fYf3~3l!}igKlf}s65Q6|Nyi|& zQw@|l$8)@Fgae?y=nQZ&)WR(gt=D?=P2wav_la#8y4U6W{{OQuIeTk8~>pg z4{9vuAkI=4(B(azG2j_(eHWI(PsB0CnM1?P3UP$p03cc+bWETmS!e2lKC@SdACYst%5E3~w~x z4K;kxV?Q{x5dD%Q2se!Oi#qL*BfAG)wKM$Gn4(RnFj_A*E60Vq-@ryWQ?+fU{CLiM z!}dbffig?EkZ5|^st||XPmaHmq1{aU4j%ItNpzmCb0>YZ$7|^X%#$*WCNqVeptGTe zS|hhP4?H_2<2(3yLPZhjU0UF#C`sJr^zrBG3NB9 z6^W>r8ClKZKuCA-Yl!+Sf9i-?8PfFd@d+q!_<{$_Bz2#p2VxPT6vqP`dZWc`6X522 zlRrXtr&9|#9v4W5{OBq9#w8jKtrA$zR|!SNah_IlBEj*c}aN?=%oMfaB@)W+#jo{@Hs>9tK zzT#SMhbxu~Q1`t&5LdNlb2OL1`UQ%DC;LeN)_k?Mr_)1rLJIOjX^4_Rd);ZcaJc5k z<3=n%AFl_TBOnl;)=$7$DBO|i5Kj&y`O$^?j;gmLwdw+~NQNG;%c7T>I z>j`+BfG^#{Q|&m_kn`xynIsu#0+0R{kfxp5ICl+MY+ry*W4=?%*F`*~V@oOgfR1ou z1}!_>rOf7x{}j&#H+fBEHbEXo(wk@8?%UI5aWFVfU?6QB?PpoM9#B6GoXp{?LK_70 z??fIFMpN6~rI)_^6lNzyN$DtbYAR1~lx}ioy@+o-Z&LP?v4*u0pPT6NI9-Y&vb?!1 zVR_E>ivBL1YQ}6U zLm$gy&$fu>2KYDIbS0jfjm&Hv?8zb}3xdy0;}=%L24k=5seJDSmuA~8_rcepiA3QD z#dNo;M?I#N(rBADEo+m#)qA^T>B~4PZ8S~Y&53jPu|ouPld_Uwg@MlPU>^FYKm~I- zo6LMhVmxM+UXs4`Mgnc()#=~oRrv~(F{`Eh-smb6pZgPQ22ND<9_fx@^;IW<>hzFP zl$?IfL#L27$N;aB!6!={7Vy|4AaA12!>_c9&|;bDamRY<*!O2H%!7QZ`HQt;3@A&E zjcBPCIq~Mf92!#Qg}g109laD2j6uOIp*@z@cbI;Qe|OJSwC-Zwr*8rgRoylZ96R z!K

lW&j{Yp!m%l~<$xomC%>pK#)#ut&8L$@GN^Wf1rR9KX68;~o-apkC=xI%zk3 zcAUz9n4nvMH^97NWIzC!6AR>AKyZ}fcbo?8lNy)e;a$j?B~X+oJ7hNnb$2A-kT!$U zoNKQ)k>~G}U4H_&nW!MrK0P?V_*q?gx$@xi%-q@>s2{9F+Q6;1mPavO$LpWCNEr! zZHAtf8aLVCv%JIZ(gQr>`v)BBno5h0)e(*7^EYPR=5Fc_EN!aLcX8ujg7#69dY+** zw<&U>S(P=zb2rq)?S>GP<~GQXFG9!JbVi5*Ul=zXC(^>dW?j|uP+cN8UUT3ZXBziT(-J~HM z-@}_r*e4iOX6_C#E~Hc8;7+n^+{O?qZI2pUQ^w63LY7_|yIl(A#-ac;kD~Oz{{Fdid zW}*=B^CqtE{R&(;zOU;c`EHBonGWwcgnjI44R{W)wvd;ReHVsfqD)|vESi3kAk$DCIAO&ZQ#%Zj=iF_|rLQW-5J^9@VyJsp}Fh*x#1 zox^IJFGHQ9m#*kdEknBgth@`Rq~$JM!!AZ8apQ(iDz)R0k(^_x=CPZvDqtf3*@$Hu z$&~`;g$RSQ4PvYt{Z|3In0`{f$Ky*Yudy=XM*^xS@VeZ!VhSA*srwsCgS@MD)F4~X zcW_8BI=z?w_7Bc`C*pp8qB1jef95H2{_vIaP_ZcpxL2Pd@Fggj=C%8H3>>Dpl$PdW z58bg`P^@HCT9xSUa9PQ;_+g=kBi7$H%U%-Gnti&ju=-HDIxWDo^T&gN;fQWQM%$D1)mB}7%wq-s8w?y>)qt8RIqN%0H4zg&(6I50qdmi8dZnd zOF3OBr65)B_#@8?{4tQ%{U@-UUmEXyPtNoylg%c(*iW5`$zQKI^QD{Z>^AZEEDLOJ zol62n=bnQ>VQ0~9$@r}vrX|->EP<0Z*{)$lC@KYT+@B0Mn9t$4yc|${X`fr)cVnhH zwM2*3q@LMO=)-ORbE0&0Dl&uA=iA_Xoitnp)=e$=Fc446zmYB#_33YpB)I2unnci* zF+-C1MBZ$yv(!Ap8(Rj#1k~+@{)aP+uC_VM@g!x`(bju)588q-r-zf1yzc&}cT?

y-tW7QOwn(g$g(s(HcGc}!77>BDQKG?i^mqayjWz**tB9V4MFrY zm&0_FvhSb1)SnH;Y6PHkD1Z%|*6a1WKyW1GB%rLm@a8%)02w9|5J-u^$9|6=bZ7zK zVtT5WouCJMOc=mD4-XCjxaSLR_8>4TvKw?r?dJi!wE$|*a;&G8Dk%V4 zec8?QDFof)3El8(Kz z3HYoanW^u!t@SkSBL+W&O0WBK8j^@I{5QdGBfSW|mo=DM2g!EcV5Hvq%hL`IC_2)h z$k7(E-}&86mr3hFu6t8Uun5cOkT@$un2Ps5L;2KzR@l(5VnguLxk}Ce1i-^FE0B zVGx?(OiyNKjGywTN!)#v7E6Frv#w%@ZFVt!N+Ylf(ArK&MB2aKC@JUNlZ|KCvke2%i918c z3j&$fQMANg`~bOHT6;z=^6N)sghrhSh5f z-GBsGW8Q9y5_hQa?K6rQ1PU<^pyREoaB`TlCvm%$cU7#FM_ZXo5ra7GT4P66^j8{{ zt}*QJ1|RhC+tUJBFM?$Z1d8`yAX-eP2`b&5${S8@JbeE@l4>s3r_%}DTVKogyk1E= z5-c^CT{PtAkW@M~2|7g&`OtSMhGu>+Sl?YQNZY$A5_p80<@R5FTCeRtIjWzC>0Q>A%dS63oDtX7~t9@oP$RpH~Ll}^a>r+CjvWw3aHQNP3c&@UdWYiFHHqK?RMW1 z`Lh<#GcESqlsNtO;mDtPBh?uK!lreur@Lh~%i$BEif2Rkpj3kX_~<(LQF*s}Ya~ho z@-heC=4BdMtRYpGS^^&_gfKKG=Y+Lbd0KHK65(5(b$iF#tE`R}YV#g_6@4#$7-GR{ z{)~=pm2Z3NlIUwtLIk-ZmZQ3J(N3zI+wx#7fU?F%!s6evua*B@Yd>~f{;X>)V4KO7 zfCoO=|5jw35(O9)y|K=>MA zN`C*FRcJw(tkuC)tArR|fCU#}8veC<=f6F|Hxz)&|A;tV7%1v<9~uSBIwL(JJ3Tjs z4fGX^uVIoSpZppfJ;t*L*UgMXkmzjX5AA_yThWZ$N#veyDC;W0ll?WCj)Z{zetLUw zavmDohSCHeqg8cm{5B&!J>3(im&f<43pfpv(?d8OJ;Adu(9K*7SlvI~*+?{l0(V|Q zSqB`;|Lo4fXeda%yJ+>URw%^n>#RF^8P$x^FSFJaZ7{Hc`1d{@UkTLd(x46s;ZXiy z!l;rBG48LUpA<$%1CG4`VT=B?O6TtbBZoV6orhnR#Y`PI(tgi%h?fBsY8hG2G}nt! z@ecdkejfvow|_X`I8n$QYO6KKRx_k3_Q3n4{nw}fBijF33fV( zp*s9BRGDaypl$|uIOZ1J%x|ChbwWe9UHbwJQFeSIP&4@gh0`PMHW$4WyU0&J$^(!d~40z`UfwP(c zVW7@dnEV-V6o*797Z%s_YC&Wr7RhA^_%-j6Ss#4ah#iNF6v5OV4&%+{5M=cZQ5SRp zOJqcCZY7?Kv9z7H>;M!aPAv*GF{gJj811)6dCLAS zQ9ijnccHo?IQIv(Hjw!GT;YHm-a$g)s1Tu0IyvCrML8kDfgY2(UMY%1QA?Sd#r2gT zOJ(Egd?CKLepCb<*1_F3@yl~(5;om?HGsGij?RIo=Bu@F_|+j2+*)GqBE>IX)o3}2r z-Dj8x0Hb&l8#Yr-C+~&5mIGE`Z#-^(^2pduTI-KCug&bUEVWS~(bd=P6v!aNql&R} z(PoW~EK!Q1VvglHzTtdb#n8m%x{C1@t}VbS>j=5o2nJB0s)Bs(%Z<#SuY?GwgEJx0 zJ^OHA^`~#wYz$GdEPm*rH7Ud5XpGyMK0ESg+`R3&@S{xsqGndDZ#8D(T13(C;91(7 zHo-uFi>rlMiF`G9_3WAII^x4-!+;BC{b3OV*;M%#AL48r4+=TR{` zv+Ne?W|pt2mJKu&&yUmkV_SdH57OFzWO*K5Om!@;Vz8jCnsNU|!eDK+2T|U}^0@Ey z4f%<6k5{YZFKEU%>2Nv*`rHTdHWj@&)9FE*>Bg{e)6VgqjXB(fE7U9T@5zC29lPmA zH=Vg?6>+omizqpFngB)LWO}qt3}Y43=< z^BeYcueEEYCLIApTEp!83n232mHMvB{w;YH+&l~RoLB?QP)v#z&#+t zOwX|bURZkCXYtt(aNs*orSqYI?wN&c_|p6mU#qQz#*fETEv zd7Qjl`CSY1Juubmv_+EYWkQ5=M@K9G+QjnsEH+BW-#7UaIC1tDK8q9g z7w}2tV=oZw1M{= z+-VIqd1}Ik7U4h7sgVJoXIC?cU!XIv*5dAQ z$v6_RK4fpH`@kH=9LZTB<#=_8-{Jefjec zs|}+;JF;wlehb}<%T~iHQ)L>Eg0d zU$hPPWp>Af)#IJyk@8^G*BOSbbMzux77QGZ&?2o#fZ6qyjl>S&Sq#hMktGbwzmTYlrwZk8i3wRt`=m5J~oS z>vQyW)$PS=^oHm&Mi$YUuPFq)c~-TKHu?nD+OR3M&BG!uSI>}XZ^L1Y_B9C+ZXT%i z1I%@;#&3=eppA&2LQ(_-Ihe}GCc*c~Ec;C!gQoFXz8>TC_fOBn;7eHP{4E7fv#m5F zLHO7mSnF#!^5ZigJ&qItexxY}tlFt|ig`y;4YN|8-SAf^V)o9Q7xB`ZCvn76^G!;^ z$u_<5SB}cgZMj)mTYW;@2cM>}PLB74x?FW2-?Pom$2zfIhZQZ6b|p57tPC;c=5-_e zF!)|S0QyEu-XO1g#X9V*-4Vp@Wg6ba(Aa=Ge>IJV`sWTwoW3Y`+zL-s!}}+P?}~4> zBU#ncMW`ja$9+a#XJbTgt)1pSupPugXKEGY6VK}iGBFMIyq5xWZ zoL5~+EJ)x+D-D+JrW03_x!cqMh1P!}{|^Hnd=eKj&qL$YRomN!XsCu5*^tvY1DH$- znz-;?+919R83$HrG{{qY=&Zpq03B=6r}M|dPO7?tPCDMTF1*pACW*bRg{*iU)CrB- z%o*%^XtY||^~EOlo2HBFL?k=Q)^Bx}9LV1bK-$LAyN936qm6f0I1d$NIJ4_Aop7a& zm-5#S7WKaH_ISd!*}oek(aX#`0lVq}L@U4gjW9kyZ}OQ3)j=Lgdd*S{Z-b8n5YMBM z8;|Zu;q}V}UX|MeGc5v6R?mqnppv=+C1mBn4Jv+j(>P7cewPL+DsYGy-5Y{b127xeuG{b?sk zOjwPtb(>yf-ay4@N@Tfbm^f^98oR$7pUHci2YDI`C_>sKon~h-61NU1!$C4Ntf$)- zHd#JMzVshmMM~pyQQb`4xTi}&3cx66J6P2Y+YW)zzs)n}VlYf+l}Ley_nZxk&R((x z6BR@L$pdD&Uz_ONqWwFV*Lwgkm3UOjhD>jEnAoemi-BH^XSB{llzYh|c;>ki@# zYe_-m@nV1GZEIG77%4juQw89elt$+`TNv=WKa8sUW4IF14Kv8g@d1ao8~9hntjFg_ zTba|2rM4`vG_12c-V%Y+(5ja^J5Z-p7{TKVZDmo5qdm&ZELzQivC{XV>vzvZC&yA-$EFnoeoPX{;s63P7X78wL~pTnm6*sFd1)eQu6c+0uBpl+Yt*#dFpyBuIr8aI{%pl-K;uP)(f-QsE z=>rQ}Ub1ii*UATAP6LO)Ua1BA{o<cf0mo*(*tU@bVhycO37sl^jyD(e(|>*;-~4&N;qWh90Ke^hyY+pWaGqmR z6=1xY*04iS&1)_uOPxWP#4T%h{|dkR)zc9nNVWjuX^HH@MTVWF-un%qj5HxeUfHi) zxj3iQh2JnWHa9|>b7#TwjXSnb`?D$=*2^tyGECgK%6nof0G-w>B>D9v$2voOP+s?> zrMH4UYI4&`?>hk_S0pPRv5~*M@0}VsxTw;|Wz9l*Z`S71^r=^B2z$M0lD5C1_k=cb z6+_SO{5huvI*LgCU6S-~F`F7#G&;W-Ms&g0TARpJo#*hw?mp|tRL6aZBRz1;!y*0} zkD&TBZI+JZChVD zfd=5;HyR$9vW88a2*|HHQg2^-%>+=vDo$FlK>hT%4(D0Kf7bqp@-7_iP8L^fJv_4r zuuFRFHC`j9*b8;0?q-qs93l{Z%FPhg*{0~771+JAJBpM>`Z_Pb-)jNkK<%%r-NFks>)U<2O#{S1zmMPIdUJuq=8%@Bm2Z+SX<`69*XaZ`s@*E5 zHcrUCg?1IswipT3M~de`+GB92h)G147IksfsVkDTCJWq@kEtGG9`p84J6?tJ93*EY?$ z1mj?i*BZecjmbIQSSV3LYVrp%a}oHWo1AU(O!JO-OK6QM2%(zafQ*kcMMMe3+j>2VlDrDj`s_pS^kBv{F(ppuQ~j$Is9*J z!SB(k|M%uFKq~*mJN{{E;{Wf9#ovLx+UX~KAi!gP3qgAM@uHnXGnV{5#`_lR1<7k_ z3j(hG_z9n|BB&LZHXKtEebRv@F}JpN!^!SFMg8d+{h2*H$XaO3L=atl1Nap|rm_(0 z`T&n)vvoFMA7>K4Ql)$(#{T#Cqrby~0(h$vC-B!0OpQ)c3>h06Gu=QddUe{w*=hLK z_@Q6mUotCyiyM-skBtTa>WlYj9&fd5i2l{@9)Ljc5iIZo1hGLH2?2urxL!B&0GW?q zz0wg_-7UD+FCo7}$T1vXA@cA@I_u-1$mn13U_O8cb1JTb2;*Yo_V1^27X0Z#bcj~+ zAHNsmeha~x15d<3qJe*pH~YJqJb)h|=9LY;f zXFu%UD+7F+$ocPI24O-EuNySGtt&!Dy0hZB;|zHG=BD_QE>FB)|Z+@GF`R`vxnT-z+79Mqnia_@*`}mO1 z-)NSwK!~8eLKX)&m1~)`K)x##9;x#fAYjJEZGp4qXM%k5_>k`~G>XDp^v?6IMzxU) z7SJ*vyazBMa53)z)2FjVZ@&_i0Tl6Q-@c+FQ%2+ z&O6$MzeJ3kht{@bXDtrkFQ!On1UZ#v1Ab-eXiRHBueJmJiIth*4dWv{&7q|H?d;Y( zP9cswc>@ycdwsb&8R;#N^KU6r%fmU_>eQWjMu5l5vUW!e59XTRVSf$`D+7W_(Z^1R z;Ov4BW!~IDWdGGRl!aXnTfV6r(X8BL_vEq1CGlb?22b$&%fLNAu!tvykt89mb%5Cz zab#yU5MCT}llwZC#W1I7FqKFz?6rr=bCR|7ap`I z2DPq@)z8(CHoa7}f{Z;Bugdk0S9S8%N^_0H+H*zvUAVZeB-We8g#%V(*VJYfr^L6> zLOFI;plQMN^Ko3=|0Cen`_QsvyJZyZb;Sm)PaC9jd+n4-GlM;HG?Zy=iGHp(7_Q}w zKH94z+8b3(Nm;ztp7_+3tcKdbhhjc7%)%%6xSGkwtmS#9!vuZn$63+UU;Sn0e;MG{ z`_Qrjx%1IG&r}*MYi*9tvezo|+!IUG8ad*NvF=FyVKC>2g2k9_be3YVvniy`arc}x zXt)!edN--*#>UNs?g~ycIqx#Uh#!K?^8P%?EIUbxcI`=OPXi>0nhbAE3=c#`;I%#V zG#GmR#Tiu|AQ~Q!dhT-pjfz|!uF&XRmTQ$2weYnanbXV~a_@nxd%~uwR zD_XkD2^iQ*J4UPZsPU$)4L0wx4s4Y;uRRPa+b)9-7q;IFsQ(p)gFx!+Z6=F(tkTKx zJPymnsjib#;4{$qP)u1d4{y!7?rA(ur3uU&WE9hKv^>n~hRBXB{o|iErD(oGl-ogR z!BFb#3Jw4>Xoh1c20W+XxO3 zuncRy2ekO-O@J8MPAuI)1Pa&Lt7%)UTKo;-DN!SbicFHQ@rjP{9YxX`mz`vaw# zz4jl2JoM|Vgc7(B5Z(M@GXB@3^LysQZ;$xDp~Qf^0%J}}SBp|?6hY^fFCe-|=6m)= zmNlQX&BJubq(0D>g}MfXMT27B0ATF#ZqqGEl690lU@=tp9PF6^$L* zMHqXH!N4@YG%Fb0=vFD=kA+%gF<(S6Y9=#gnq78zBO=AXwhYAtZ1JP$D4s{twFr2B z*A|zu%VYiw6pMQ1Mst?iBi+o}@_eBA|D5C=00|+!ytQtiw2qExyU>KHw9QTD+c(_O zbK>^u7nwzbwGeJ8`%vXHbMAzjZ`9ylU@)#-3muGn=2kxd)OAKheiWiRFi*K}nYh0Q z6|(hPS;a+^efLl95=M*c69#1P;=9*@y)x*)!^OIpLW=fRP_KTza(9QQ!i?(WveT!7 zy$*c%zmchxNJEUhCQ+-%#~;X+i7nQ?#<=T%56MmlTd}oX-E+v&o)^V|F|-%Z_3L;8 z({}Cet5BwqSxGWr4GiG_3?c?l%?z8>fbs_Rq4mCLotBKVO1{K=`P~Z%;ZoE_>+I1h z=QX!wiY$N;|9tF{jMg8$vyRn5%SDj#fpv&mBF ze;P_2ob8VU=s>VeXqqnhyxZiEA!&jP$lJXf3q{wq_^u z5;$45Q8+E&dUaT;<8?j?KGz4@rvX!Q4x0VbD@HSNxhcNAtt~k1A|Hle7sEM;(USJr z38}8WI%LltS(=eqy-bdlF3|FQsZ1tB`;Y(J>24P{GBeQB$WgV10E2z?-wmVRP2nGo zvG)$*JBO4OCOBi2zSRsyzhKNs9V3oL(9=8j8!5qhx-X|M9E>!@PhL+~hs@%ch)c6n zj+L*d$T!&LE|xJQ5+fgb($ODWuaQ;D1OFaDSr0u)F7_pF1zX$SE!E;Cqx(XU}cs5GMI%iEZu6-=7x_khpR>6LOmNs#+QoX=i8(3Ta1nM`xX zbcACzu2A9v*Rw}aem+zZRQ~OB^*3+#fvxxP!LUyt0r)p7`avK3VgUThxc0H^Z2!-L z1j{UgS`lh6Iv``}HHvKMETT)W>NGMfaHmY$lNOWPOAzYhHRX4(CBAGTV2nD{-TRMf z4Bid$xhGB%O!g*`&dy)0Uc2Gs z%8dc9l#Aa4@_Kay8>Vt*i1V6laQ-yc3aMPF+q0NH!)41bQ0;rphNv8P69IbJYcTSR zh(USJtsr!G7vzjv-c+-8{xD_%U^+~1f@dufO>~0Fd{9d8d!$TT_5g&FkAmh>xcgx1 zGi`2Lot?&>`JHEzJVrtTq|{H{1@fw$8m9Bxd3YSJx-)=_fge>SsW`>d#H-aco7qcA zZaHyy`i48nEC!7R%v97s*&P4=T>CmAsne(WO~Vv5-^o5`_JS(p-UGmmOazi=iGa4! zxsF_r0@I}FY7;YzjbAWqorZ#YlW?V0u4PMy+!bTw-c;!DpGD#l@TEW{Cc9v9ziVs6 zub8mkA-a%1uf%_!VUp$Cly0nllJrn_aHLw91d7k!l{|Z(aVl@z>x}7M`R57j7+RiL zhQceLZaQo;@o?EbDsurOYF+`WGTv-)`NT5S#2>nx47J7&a+IgksK;0H&+2y_9LfhG z>b-)qKE1UGlgn0R|IigWp)!=4;)JMyjoI9hkGRxF(tRI)OoLY*FIG<^=7q+$6xAv{p9Co^t_CVo6K;`Ns!3$+N zpRYJH@m1v}zcswo!%$grP3PcC$1Ncp@Vl5|yImuy<1we3@pT;8&V$v`y(&4SLC zyWacwH0UR=H~q`V^k?F_pOBFMc=o0*$;c!imlQ=1)3Zmn@JnLnU(keqdtTmIH1^%` zZ8sf`hAq(Va?IP@66d;a^Tyb#hL7)=R9E$c*e#5_wdw5bkl9yn96y$|ZTt5tbeAkT zv$kwc{iQjV?eDqO9bwE2=F!*zZ2o_Ild8FVx7x`eW+}V9MVqv~q=-G+tR39F_;vSy zR5j;bFbF3bg8+>9QP+ha1fBjIM(yL5(ulD2gPnn_ePR#%m$0 zu(&rtV*lmFpY@H>z2NmM#ukIt1F%3N~c;kf%X0AASYDzx9R? z$OO=kN({Z&^;sOv@{7xD5mzPM&RgeOv}lGbg8!QLPDu85pZ^8WmG0vrCrkr@c7Bb{ z9~*~qU~AkThwRA@W`iS_16$86kQ_t=+z@p>2}w!Yq?x*kyAG(_*$Mi^O|tuWKcQmc zGW&%BO_%QOpCd$QJZda2N=e^xKnE3)ESI4UbTDO?Il7lYAuB+zTU3u1`Qbj^V#)z> zce(*8#Mt$A0H{3co47*|4LAZIVCk5XE;bi!ydr-b5HH08JE3Fu{utPLarvfcAHX=n zL^fcX5MhTzhA}+Nh|2hil_B0&B0cA#D^(L$3vGxvFe0L>*jDa#2KDO4eVr>qWNBXE zOAu9QHoSXXhN5>cD#p2M>j6YP8tIDPc5M?K{Z^JKN{Pg1Lcy(ca<^2VX^#Gj2q0bZ zJP+onN~J~!Qu<=T*%J2?x!n%rKCVsIfakaghR)!=$yTDvb zc`4@rY)G5Uhphrl+t<{r0eH&*9Rz#pi>WV*gb4Tc#_a`oOEWnDWU;L=@6I}dirJ-5 zswWy6fBTv!rT%!Cr!d+zDh{?Uc&5F3j1EvKgesrc>4rBC;jL9c&=e^S(c({ZMFSzx zyq!G;SwIuEE_P!Byn`IEYA>=gMMH?N-#I#JE_&&O{m@|7TGz?LNlUH`L7lZD4~C0r zN~~4Snn2UDs~QpjPLqX$uG&~K_{(Xcl#K*qgH<8M-)`(!8FCDraeaqq#t)61aF5~a z<;a8xSsx2*NmF^alm&l{goH#DS#=nwsdJzh$aIO`&ALk{@x6luS?=W55(_0&pS-F~ zDC!mKO4R!a^$NYua#e_I{XG#;iskN+9TNE4DllX zxO!-OCqP7q;24n?0jUTiE>N%2k01kK0b1z%=J|tY!<83gpZ)aP8;Vg_mJOTPbI}fY z?|S>M#t1Z!otFcb~O?MpaVPDx0|p={d>{SHLUjLLYq@z!f4-YhdAH&M)IGJ(allQ zg9iELe|zfbcjL&0Bzxz-9m$);0!<~dCf4gJx)3KP6Q?Sz>l2A8$7Zqlm#oGj$zOI_I z7urBJJ|2YusR8>!+Rdes&d)|L8=@UZ zUlZ%T9&lKVPqa5wYF(lca4^=#x9eFhdi3Vk0Msj3?3*}8*EwQm479PLqi^kUAPLbL zricOKLuwn4(dYJlKVzurq0XI~y?aEfS%L8JxrNq(2=n4LXot4;o4Iq*mv;~?5T(o4 zih-tUE;C{PXs4te3hi;UNcMj{J)>zBK}sJdKRBkBL|AsnITxp163QEV6@}qPx_dOg zp7pHPvOsFIOuM-+WHy&og1urNoUbCu##T^g8uH7mtqLC1E^n?32pHI{z};E2T~vX( zbOS7Yh?((;UYjDTgl|1+Y@_z~`|QvQ$GD)hGLuLY+tO9s-Fn z|A=U5oFKi6QSZhL7Fp`COH`pV86a4n;nlaxh4AVwMtk%~DFm9^@SPXNpJ2=hs<@!J zCS)nIAArV)`SWQRFZxO`N}}(vgoMea3Hl;moPfw0q%>qdEK{D#kt*sC3xZjy#A0A|05=I|t=ST9o2_#!{d1=~V z<#xevYx|~ec@pJVK6_(){_~Rb@o>h#pgn!^XaAR1hKxsfU~v)2JIC4JynX%YO|d4h zwPX|hZZ+0I`ii3wmenF(o41-u5|%cUFz|Cu_V3~OoT2*0KOxhQ z{V39ahIb2W4d`iCWHJL~gqf+2H!U5>Ei}`8B`H(~Bsp&O4J3EbcQi zA;O+jG$n+)@szl(V&r2Fy-z(ozaK^P>+w~6IZ)RYH(peg#9Z3TCTgT+g=@2`7$ju| z_K?jfzscd+6rKMxr<7)&bf#-W)OlCel8?B0`*{_qb33f$X}b+x7c;54-k?bP?UOX^ zxw?u=4>?~$iLc(P0=DzZ3rb=X^qREULWF~s1fYt$xTXng?F$OT{i30lZ%c?$4!D-Z z2%|Sd9t2ByU3Tk6u*)MI!GbPZY9SX=w&%ctxiD1G)k1`C_NLGH!^KJID33=G)`XVM z_M=JC%?$?2TbpVni|%)NIoVG}X*=zi<2x08!{g;-u5%NuXr|e1S86bohGkBfPQD1K z?loO|fmcx`PfZPGg_u;4a8|iZ2Blu%CAl{UQxn+(26w@f%Ajt&3cNgXjA(3t9cZWC zRthwYyofar9o;I6Swx9E>gN`CwQj9LC@N-6SG*X-&tR{G5Mj-7I`GZA?W2MPSUK@Z zXhV`wJ@nh&a+q@_uSXK+qW5kr-KzVf_e$PQQN76F8sXnxIzJ9Y^xu7JToXFQojYWh zKeoxbC^y;+?=r5+tdPmI-YsRftaGBcX5tZNT}1s*T?Ou`_tP-(Z#U7{Ow_ zjA_2-Xh}8W)m~WFWaB&B}o+$5Rw|C_J9E%6Bb7U@vqc|kqv>S)w^mi0FP*JnEXaJ^VqFds&mmb zeR+NtQO!9&x=9KVUY#=627S@CI|T~@P2JR$Wd7tfw}5%9*V@e9KEPdf&`pVF{-~mq zdqk3dqXV~k@PYiHwnHw~;xwA?2|xPmcpcY!`8vh=$J$#DwrHPtV8r6|cj&l!aF@;; zEH;$JkmI;9B~ulE1nm0%+Pl)Irp`2spiY5;2%<6=q=(akf=aDJP*w%2vY3bqLNn}w zB1=@5AcZ8dD6O(Mt!za=?64^z1|cXR#9{!;5*8&S1_H`%HVA7%66U+K9d(M#oTLA0 z{@;7Qd-E;t^S;mfJes_Y${cL}1H{ZP$Eq_1P(y<%5s8MFOFow)owanve?+`@8lDIS zPAQ}XcK1jcO_Jf$foo!duVaNtUTKn)oNA#FOG!z~$Z`L^%AJ_y&N$w)%Msn+Nu%rs4}l&W)CofhHY69dee?qDhCv zv0{&Oae?dDBp`N$vdils@h{Nv9M0Ft0{DpFPP+Z29T2pJgcHiFcl1JRDTtt7jS+lp zZs)!R3c4X7Ft3t7!ep?ILvmqE+JDeIM6{a*LE`}HL>Hy!$p`L+!Lg|~um+f@w} z>9y+#L(fef`N&DV%|d>bZkB{{*R6M&;Mc6}<)eN=l;fzJlU!uPNUgCV#G5+ZHR>xN zx4BNwu%A*;qOe4Bc>)WQEgnRX1LeYYH>r6NjS~E=1}vgqk$4_}@}Vts6_8ic123@a zG^66^I#({l$}7Nc9G*xA!~6%73b?cMzqyDs+jJ*IK4{A@uHLwNDv8;YwPKOY>WE&oIRY#b_U=K^?=9Bjk}E8H3q2REMy=`RbKMIhI)g5Nfk8Lz&IC` z<`$9c5{aRrK9(wxwWf&b!L^^AT@Fyj(2ImW1cOzrEISeJ!AD5-QBbfq%QV9 zQaNY2&&2`gJ_ohgF5Cw_Q4;ktZGgu4j_?~&RAZ0efgL7g^{?Y7w3(NIjAT&vA1*@B zsCY5Y0pHCSmJd$#XW_rD{AD#Z$ZH7fYxd_4^?+zMBkP?{dc8crTwFHa_*9e;-dB#- zqdchTt@O{4OGrp}Y(Mgb&E!j~nG#d4Eg~4VjKNh*doT-gy|YG7l}XIOWmh*dsJL1i>bYZ*QU>f5zlA0W zu}KX%-Ru@t!M+C17%A9@d0ITnwy6D^VSO}`=A`9}#@uQHO^9_LVgyoixV(YB`7pj! zm?{2|fIEG%i(Np|gUcSn_~}HAil0?1Pr$6Spb*3UM0DNgAB zgnoNf{4}qN`zjk}$@ihIrE8{FEDS&^KmTx?`=NR}qi-by`|OpZ>sRpyv?fuP;i`V0 z=`JK04p*s#YSZma6ngtw(7Mk7SL?WJS6pVFNrj`9ddb{7@&;>{xUo}m%O{}XbL3;i z=WSs7vEuU{b;gespC!+o|0=tDppxyfHLMo%#kTUY2QRORe~?R~sDy*v7@;xaHrtel zu1Lg9BC=kN73Gn4F?Kry>RDgBN0o`2;lM!<(p?&yjVz|d1Eas^S9MMKz_jo_FrkN? zg)9UC0gXzIkzC;Q(P)6}6(F9sddB_vUyeWUfx~1a@RMXZ_4SC;5&cDW@{;W+atqX$ z3WR+TkzaB!eb`xcXMwXI(?5mOX{UM7+4u_(aiPjjeFwu%b+}0zSD?pM8eHs;70;Ss zrBfv_=q#blrso{Mr6Sc|+kg^rsf4LiQ*NiK^<01aOsBJ%2hoelZ!#sifZIb_uk34P zwP$RiXDWb7mEreCeO^5)S=7)mMKL}hKHyLhS709?56h7QS(u7&E8j|&o<-)v(!6DzMZ@!b8u26}T%}sPU z7zAisSrzz`IN^ z_bSCYAPqAN$>>;98&$4Md0t`Yg~d`A1`}GTMAdGi6(v~{-0qdMz+2jyFEW${HVup< zIJrMIThXwmp-naI-W6}B;KzHtIWztg->ZLN#b;5etJ2gQ9w>pI<5!6N JW%fRC{|57&S8D(O literal 0 HcmV?d00001 diff --git a/tutorials/kdd22/imgs/pecos_pipeline.png b/tutorials/kdd22/imgs/pecos_pipeline.png new file mode 100644 index 0000000000000000000000000000000000000000..634edf2b796516e30303759e7e2fd8a0a471d98d GIT binary patch literal 18041 zcmd43byQrv_V|sv4DLF(dvSLu?#11m;_mKFad#>1ZpEQUao6I-`y1|k?$_3O|9>ZI zI5}Zw@7PIBl20a5>9Z6v0zLv57#OmQw73cw7$gnIuL%bM`fG^rzzn*8yQoNsg4IqF zo`7x=%(P_86%@eeKz=wd7;qRc$PW=PkUTj0KYnnKA0G_*U*TY28sG%~<<|$N`EU7f zD=?^kWXwU=kB>Cy3yS^E6*3?EzcuDV{6`v+CLi)YelX3CXi9B_e?S*_M`>*rFfc@n zk1se_Rt_#07zCn~nwG1Uf;_K@gB_ERse`c@lZTz-M=UUY4_=Vh&dk+_)Wgo!-i6mg zfc#$)ydeKaFf%#nzeHSZ1jw}%lt{%KoXtqNm{^!t$OREdNlE#gP0e{##3lcU4!RQ{ zw{&%NW*S9T@`XA5Rl9v&WM7B*%!Hb#&Hql>4#tC0tzy$kT)O8%oq+|0$q z*~-z?%E6xWL$8ssgPW@WIr)d7|N8tpovv2q|82?M<)3VUGRXY#A7)l27Uuub4T{SD z5z4D#=Hg)M_Mu+g-pW;wjsIW5|2O!*t^GTeqO+A5D7_!i1X=%y@;}4=ue_Lpoujjv zi_3>R=RflQGwlD$D_J_YI)KXB*~&!5-qp+*6!{-(|4HEgx5PiX_?bV7>wgsAzw72- zp`gkUMBr!suZk8#SVTAlwQylD8F3Lc5AchPuiI}+uiTDkjWUw%vhZYlreRSNG{rHd zA>yWL=IC@q;$$2W(MmezB}u8NOTrot)9(UzuRL3O-#6=CI{x@>{pk@pdcXF|`{|kc z#6NTU>!?fP`A_d(-;x`80L}k$T0U^qU5|Px#P6sOnmsqE(Zw(R+cXt2!iFWO8qYs_ z-#H`G(sT*TI%C@lp}&tn8CK|%&cX$fdvFGsHL%I(NnGhLY}b}w?8ecbF5+Z-UA0F9 zvn==t{T(|<$(QFxmF`$NBCzs~YX+w&k*)4=yW7*&vaXONImgN{sU z;R~a85tvUFm;u@|o%B-61T>pn!bxT?1=#CyYtQ$y+Rn?S89|%6Zuc3dG-G;FD98D{ z_q#Hb*~!wc}^^rCaCv zhW*n_%akJ$+$@58$6oSh4 zz%JoZWHy1@GSohTt;dnPw{wf1dS5rb0U3r7pwOYFx%XmV`M1H#a^2S?3G7nXEZCKy z>MCj{@8DO}$~1X(5#TV3t%JP+f zJ#Rny#(0V{yvThG4(v|8och}>_$7dRq5&-&}mkr;|W=qgLA(x z$|6o;LG*U*wodb&^=)S6nnYe_t(M@~7=CRVBoZgm@C#O#p^k~;yKIwV?slza{jTZv z!$w~f&zk?T>vsAgwYC|z=o!?^>`N;aY`*N_Gy7dfFPO+wZv)CnD_zuE+ke=7`_pyw z_R!jkJjQX!H|8_Vb0i*E=W{^+r)^ZA)+P7s*X_4IY934KnxXst@3(m_eV46Ueyv>x zj1JDp*}q;k|Ma}y*TskW+!c}hKZ0YhLEoqjDr8ToPLtx@32o@M(-;^Z=izlo@8u&oedYf1C>mVf6Y5( zY+Cvb&y44Ejf3x5*D?%&ZGw=-d^Bt2Aga(WC%U~z-{@{+7#*|b11CJ;{Gf7>;jcp1 zk&0cnxxZhZX8irm8%DDX_j-~Q6K?+Ok^Fts(6y7u^1rV#Pc-1OByY!`dah%ceNW^#o$uzPe%BA849sl3KeRSw z=K4LK{PzF*l6MtCop5pK^|OEGx z9+6MJ;O3FwpK(Ru7=(&;dPPmi&Lhi}8J|P;p9cPKnfO;vN4)gxL9%)w2f?eJ4C5B&)!b5cJQtlI6iXrF>`x&l0| z+h=^^YUIWhxK>R^iN1DC>RsM^wB6ghzx=nUZH6l8uY2)Af?pC){T~M-jZuA%#SES% zG?QvAsi2stI3P%V$N~u*x>)apC+`ZMu(~$AhU<I1mMEH}O_qwlkCTH+Gzj5jQsaR>+eccZB zJ@0w34D8hi)jK2ScAD=u8LDRl7w1hOPB&b)G?pra>fC#J@9RNIGWyG9XWhujUE*!5 zgU{aO*WMRIgU5mA@>9J}pQ+cL?#hHN7CV&tF^>CP$+mvYDFMm;cJE@aQ7Tws#RG@m`8bO!<%bA~HenF>5GSd@ z`A=jN$ti6+PD*`DDPpAfchS-ev+~Ph9Cs6}#20H?H+7`!^X9!rd-!g~fLOr%(Nh~WX^Sa+Aj)eX`Avw@ms6<9W*4r7j zkRi8U*zJo^97BmiJ-11biu?H6=`P0fA&}u_$%fF62qRiAwz z$@m|U5C|HETkoxx%_8D^n&;=1b^Ej5pj&;y&@0&_VapXid)Fd@(9+udI=}lWDBcDQ z)6$Oz-MKz*lDYSn-N56h>bD!rUAWaaNtCq^f`l^K`9swWXpD7--tGh(1c6;+ipP#b zNBM*cx}r+g4K5;gh<$>jjOm4M4;-@W-oe+=m*jQXIB8)ap-^EVbH&I>S~92Pns{IX zUdILD=o^dZMq&r4S7|b-t&8U@nn7dOy0#75=ZCFeFKOd`JBS7z4^4Rr0I5Sdmq1jD zJYO!W6}HvX;5|h8AJc03Bx%o_>CbeyI$_l~Nql21s4geFZx69IL)e?lEv zRm^jwUWa=Qaas;8>=%m$3^0Xx$u&e|zRJcf=$YsV*Pn2go^biOX3Mf>qY#-;mONc4 zDH6h^UYbA5o%WSy5`7C)pZl6Ny2i@YwMC}lU1u`6TGfovi^$Yw_`Cbhh$>dC?YB6l z!^_mS%-{0tZVk*fTiKxT(1L#t@3-fS_Y_uSFiTWm|Q;id^G`kJ9z63)=0-G_7(cONsB9)D?`E0OLVGwRki!*LFsMeEY zojHk2qQS+4%z3Qp-4N$21aC9V7(AR9*j&+O`)hvWd32xYF}q%4DDtD>6S^%b6WA87A`!Xd#&`^k3eKiPr$+Izd!JnuFV>j9q&1+u4 z^dj?{x}$Us)k8*$iE4I%_e+-Sk4_}C`_UPZk4Gm+Lg|@kNyCMAKR&sCviNJ0()hRbj-$AZEV-wUC>+ZDfkLLRA zMN2*{XjSr^fL{SlOj>3Sh)QZVA_IyO!V?w0JB7Wm(ad0M%PljdfHYX}!4jFX%u~@m zGk4yzBCy>r8^5_=v~t5(4dA`x)&quX(XNd?*GjfvQN87p!R3~t@6yJb>W;rZJtB^h z`xk~@4l>yOb!;?Y;*ZBNYxM8+lzC>E!DxDZVkT^I+>rUN} z9_e3^uM9ZvLNNSBwd6~7BBnQk%37$iW#ZO`Iu3wDDaT|t;JMHDPOOwa9)p?;F+_Zw z@2Oe5c#`IfuQzA~7;#NO`g!X49>mis*xMu4QWH3-?h2qOCReJVSJ*tI42+xr=)Wg> zyNpE?Qqn0L#B*%OVmZLq#gxj_?ZPOiQ;Axu+OVe@qL_XyZmlQ$LxQUwPB_yt;o85u z(z|P$X8H~O7M&}mXK&((o8V6_#6x;7m&EYdDIEgYgQ+&LuBa~V$PG-%1yJz$I7NQu z;78MV%juE&im1meJdIsTf^6{Z&6u+o0mWVOhWj!%S{Mldzx#1U`Zgr93sr#Bn25>O zd-*EThX5M+$D-wB#F!LFIQ~WPi<~ir6^Q(6xh%o;^$cOH`Z_!YQfy-wsnoDLp>Me< zd{Xeg@blyzvfqJH98Zy?X51UknmK1l4Q@g=iSgU{Pw{S9`?9!Z?PxMrWkPT8E*Crt z5tp)|s@0Nrp}G4+GMY#*6~Ow_6l|d@j;{c%FK3ZCr&> z$fn;mBuRP7&dp(RP{_>&YascO@?02Y9&6(OuXN8OQ)jWHl2Dw(gJ0F)WZF%-2t*}I zIvZI$Nac{7OC|6GR%-+tnvLAqCflwB?>ib)Abg(kYab*k;dgs|V!%Kqd(t8n;}sV^ zkI8L@@Yty8JVs!j52F4UpkW+2glGK>6~~>q6f+Czx9(pZ7do;o3oDf$Hb3pT^bD0=5|>LO()}oZR-6)%y_ZHtu4UhGN>@lQfHVZ7lv|1 zmZ81vN){ua-o6FX`eb>&cNNR*-xov;VMVWw4r!#Quz59l!7BQ3ty^mU%4F<-=tS}T*KWrv2FYOT65nyPue+rBO%F@|J!z{xI^q+`6-fC)uV2jG%GDr!Z zi&UDd41Vd_EoRgf!2{Pd-uvb1AcuZ;kqajl67_%?11;%3wDtG>PwhMU38kZPeT)j; z#U07s^J=IoZz5&;HZ+{E?KMBNnvss7TZCiDNaQDT4M}c;_-PqK6B9Pk7BMcc zK=2cW0UvNO$@PXd{m6duT+KtXWhmyxlw66#zPUjX1yr(Fv;q}R?d?D@cZ+p#ttpBTi|LeS=R+>KSUMbL)0Wxc#U z-3|ADGK=l<=EV*g#5@8j-WeDNKBRdR)4dUpXgI4#4@V5Zh_#s_R!c2y6Ber{-PMbf z>V)WN5E|P|;N@&LQOzn5#8fJi=U|$`#Mjr5>jhO{VLM&6o{;%$#SPTobIntVY(ccz z(ZgVMw)N=S>BXlA(SnV1zbZx3ElIPU61Dpb)##z=^UP<^hw@nOu!q(p^==mhZ=T2P z9$_=A>VYLQ=}|Cq zWLx!%ja@#Ph&5@Q`x;ojmY%ahC3;zRlH*-E=B+wvR#NCg^WAFifpd!_HCyJ;=Hf(i zNK__dAbR1cLu}$uXiK_dODOoY8D@(MO zS>$f~iX(HHMQWO%OPFGwY|`r_F5E0kJX8uC)uZ1?oF2WtE?KtJ80xuO5en>;AJR12 zUBeqOIL8E>2_3d10-tgj~t`NB;#Y#(R@aaD5A5q=-c7PPJzxgnb{1QQj2a@>Y( z3(|GJF>#H>k~0tNn-?W5V4B1}f=gy}5lGh{t?C&V)OCckbpPiv($SIx`R$9xn_$H&u8S^Zl&);J5jA!DGf5QD~ z0UW%vv178Z$Y+$^A~kHMD>44WB1YNDR0XNlB#ZstdgS+YmP!BQ2<{eus7>@I&RbMO z{8Nz^3{j+c`)xfG)frrHE^QY>ur@IOfNn}|vTLVyge=1nIMOxcJOm+0%1_Lacrl`E zV<2zZC;E1CYy8=7`;0or5hAg{4=9$UBNGD^7+IZyDq1Wyd`| z5h0UzrQ{qfr50NrZ01b>84(7awNPRsS_kMFrffrtD(CWNH~?D(YhZyT=ZH5iVyZY%hP92OA_*53!iB*De%^Jp zpF26eV-fA^cg1}+`2pGYO{EglO-Ck_MvhE#%nOYXZb%T0GS{5R>`s;|l+0xbH z_28Y$Io0ej*70k=&N>e}dnd^v-a%kU)_mZrh5;_z#NFEEVUP+$xi6P)C+)Xt}WJf6HVRAb%v39;~hWqp|`T2|RC zYj8e#3BWEy*bwF$d_+Y0%5URik~&kIHN8zcWg@{cqwv_o70MHLUC7gKwWDn^uy}%X zZc6U-T+AfqthQN3Cz;eC9|$<{tFDmIvk&|%PsCd+(xcP$_5H zkCuxx18?(pM4Mby^*%%nNC3;8D8D7;(wi>R9%4(5jAAnce|0&diel5!kTz(*bh{5` zz37jj??$0P#i!4!3S{HqxvmL!i0 zvF--3kg(!f+bbpHDOQKZ@7Gs+8n;+gf{KF?Ydzk3QkZ?sv zu_~H-U|)(y={Fv&i{+Q8(i!8`C=u4*$9_H)UY=V3S1Yver2DA{vym}+fD+%OJZ*s1|#?V zu*j9Hg`K znApa86r{pE;rsyO)Tjg2@Q?RwL0GaN1XYsr=$#FdS;3RyWaoT&p&!)) z^!+O=-)qM)WQwmu(1UQ4X-|rJh)gL$n%Wlq`pThNov_6vSSx#$WV>Aa6;)E7IpPj^ zQL$nZHbt*)hC9Ib@V-Zhu}Z>ym#pDly%`C(e5qbYLe!r!JkRZE&ai(XVNu;h9LlJ0 z#1Ok#Rj`zVFxfWsa3^=(^8KSwkm-T+73ZUtOb^I#XuvEGAI-(9&KEUH zvWsUK`D)@bL?uM9q~URWQvt-ZSED*{A8SSIvs_`2mm>BwHThW@Qwp#rJT%S{ZTHY0 zON6CVI(2e`U#f(svzA6 zUUZ{;7C!P&tPKw7OPC@BsS<7#tc6ZZ3jRwlyR1-bpC@wz z27%zHXNe23d2@EUgO$uGj5Z+wEW_~j82*y`Y@6trPVFmrxn9WG+wAz%%QxQ9i}H z3;U~fdm%0ttWVjL{w}_-K%{*%-%zN|sM8aX4>B*pC1?pVr)-P{JJg*8?P*uM8yPcD zlqz4`$kjkPM4;5Cd!J!{1qR|I1B!|hw~``gtX#NuceG$8G5lCZd}&s76&JMQCrFwQ zSYMZu{x8s)xT3(lnjoz?mpdk7IW%%H?Ju-6 z?=`D31q9dNb+Q)YThc>~Yh{9do_b;GpQI1vZaAG?skG<6V0e6^eFF^Zreu)bIwrQQwnTM$FMOmhK>Aj2@3_cxj#L*85o*QR{MsQ2 zUs3RB-(xAv&<>c{?^8a2c-|y!$ai{`AkMc~sU4(zc6e|iY(i)*x>ST z^pKix@rr4(!k2*0E957;jy^u<#gy+Fl86@=hi!aLd@Uib;&%9wA;wy=D)>o6Xr)X_ zq)s=p1p!b8xJx3iL=?7ZucIo3+Koiu%De$4+6v-~BE*5XUL-INP|(K;iIY;qB&K+kNG5@r~=HKGEZo6OF91t#-QNoCai%4SvnFZE_= zni)Xh@N(Fq&q8mmLxF@O<fGvO%1bOz8t_GI`tEslZJB_<5@W zJm_&Fq|8mxLrneBj8n+F4cw`rzc)4*iZVMC62Yc8=f?x+05tBb>l#W_)iYdbEAHiC zouARG*$G2jrY-}%t0QB_D2$~^Lu@C6Tk}+` zuM8b?{bXh^+o>c0r^=e^u6gbe{SF4v3iASyHT7#&IirNoW)MgP7ZP_lQ3VdrLJ3N@ zDv~|NbA@O<7?E)`eY!pxg0=r}-WNSw%b__xms?eMTEs zCMqTN4or2(}iEiB#`KuZ6o=E_Jb#wtZ=_Ms~Ce;x|F`XupOwOZ!m_B zQmn;CITEy+;Kyqgpp(z zsuE)Yzo3ZGh(h)@-KH4bW)|$VVeBIuDC;J)OL|hWQvUX3o*_{lZ1CB-N@DqFHK4ru z8(0ugh1l<;$Xy9eAg>ku?wLh|$* z99QL-O*2;6kf3J~;j{(0wT;E|6O7fuTJ2+19(V#uvM=R`Dm)gFeyuF8Hz#L0S77Al z%zO%Rx)MJ0I2rUvIyMEVr*C!h(5bF`iR$DHmLHSX3(=+Zpau%jG9es